URL Parser and Builder

A link looks like one thing. It is closer to a sentence, with a fixed word order and punctuation that changes the meaning of everything after it.

Read left to right, a full address can hold seven parts. The scheme names the kind of address. Then come optional credentials, the host that says which machine to reach, and a port from 0 to 65535. The path points at something on that machine, the query after the question mark carries extra data for it, and the fragment after the hash sign never leaves the browser. Colons, slashes, an at sign, a question mark and a hash keep the parts apart. Move one of those marks and the same letters describe a different place.

Two rulebooks for one string

An early standard for this grammar was RFC 1738, in December 1994. RFC 3986, which updated it, arrived in January 2005, and tools such as curl have tried to adhere to it ever since. It is a tidy document. It defines the allowed characters for each part, and its appendix gives a single regular expression that splits any string into scheme, authority, path, query and fragment.

Browsers do not follow it. They follow the URL Standard kept by the WHATWG, which is written as a step-by-step machine that says what to do with every possible input, including the broken ones. It singles out six special schemes with built-in ports: ftp on 21, http and ws on 80, https and wss on 443, and file with none. For those schemes the browser forgives a great deal that the older grammar would refuse.

Forgiving means rewriting. Paste HTTPS://User:[email protected]:443/a/./b/../c d?q=caf%C3%A9+au+lait#top into the tool above and the browser hands back https://User:[email protected]/a/c%20d?q=caf%C3%A9+au+lait#top. The scheme and the host were lowercased. The port vanished, because 443 is the default for https. The single dot was dropped, the double dot took the segment before it along, and the space became %20. Leading and trailing spaces would have been trimmed, and any tab or line break inside the string removed, before parsing even started.

None of that is in RFC 3986 as a parsing step, and the people who maintain software on the other side say so plainly. The curl project’s documentation calls the browser standard incompatible with RFC 3986 and warns that mixing different parsers and assuming they behave alike is a security risk. Its author describes the failure in one sentence: check a link with parser A, hand the same string to parser B, and B may draw different conclusions about it.

Here is the smallest case I know. In http://trusted.example\@other.example/ a browser treats the backslash as a slash, because the scheme is special. The host ends right there, so the browser would go to trusted.example and the rest is path. A reader built on the RFC grammar has no such rule. It sees an authority that runs to the first real slash, splits it at the at sign, and names other.example as the host.

Neither answer is wrong by its own rulebook. The danger is a program that validates with one and fetches with the other. The advice from the curl side is simple: parse once, with one parser, and pass the separated parts along instead of the string.

Numbers, names and disguises

The host is the part worth the most suspicion, because it is the only part that decides where a request goes. It also has the most ways to be written. A browser accepts an IPv4 address with parts in hexadecimal or octal, and with fewer than four parts, in which case the last part fills all the remaining bytes. The host 0x7f.1 is 127 times 256 cubed plus 1, which is 2130706433, which is 127.0.0.1. To the RFC 3986 grammar the same text is just a registered name.

IPv6 addresses sit in square brackets, and the browser writes them back in the short form RFC 5952 recommends: lowercase, leading zeros suppressed, the longest run of zero groups folded into a double colon. Zone identifiers are the exception. The URL Standard leaves them out on purpose, and RFC 9844 made obsolete the earlier RFC that had given them a URI syntax, noting that browser implementors found it impracticable.

Names in other alphabets go through a conversion the Unicode Consortium specifies in UTS #46. Each label with characters outside ASCII is mapped, normalized and rewritten with the Punycode algorithm of RFC 3492, behind the prefix xn--. So bücher.example travels as xn--bcher-kva.example. Both forms name the same host. The trouble is the reverse direction, when a name is shown to a person: the URL Standard itself points out how a Cyrillic letter can pass for a Latin one.

The credentials slot is an older disguise. RFC 3986 deprecated passwords in that position in 2005 and described the trick it enables: put a familiar name before the at sign and the real host after it, and a hurried reader stops at the familiar name. The URL Standard now tells browsers not to render the username and password at all, and Chromium’s display guidelines give the same instruction for any place where a person makes a security decision.

Those guidelines also explain what a reader should look at instead: the registrable domain, the name someone actually bought, such as bbc.co.uk under the public suffix co.uk. Finding it needs the Public Suffix List, a maintained data file. No rule of syntax can tell you that co.uk is a suffix and that a similar-looking pair of labels is not.

Percent signs and question marks

A URL is ASCII in the end. Anything else, and any character that would be mistaken for punctuation, is written as a percent sign and two hexadecimal digits per byte. The letter é is two bytes in UTF-8, so in a path it becomes %C3%A9.

Which characters need that treatment depends on the part, and the URL Standard defines a separate set for each. A fragment encodes very little: spaces, quotes, angle brackets, the backtick. A query also encodes the hash sign, which would otherwise end it. A path adds the question mark and curly braces, and credentials add the slash, colon, at sign and more. The logic is local. A character is dangerous only where it would be read as a separator.

One character is in none of those sets: the percent sign itself. A browser leaves %20 alone, which is right, but it means text that was encoded twice survives as %2520. One decoding gives %20 and a second gives a space. Two programs that decode a different number of times no longer agree on what the text says.

The query deserves a warning of its own. The URL only carries a string after the question mark. Reading it as names and values, split on ampersands and then on the first equals sign, is the form encoding convention, a format the URL Standard describes in its own text as an aberrant monstrosity. In that format a plus sign means a space, and MDN shows how that quietly damages a value that held a real plus. Repeated names and square brackets in names are conventions on top of the convention. Servers interpret them; no standard does.

Fragments have gained a convention too. A fragment containing :~:text= asks the browser to find and highlight a passage, with optional context before and after it, as the text directives specification lays out.

Then there are the pairs that exist for someone else’s bookkeeping. Google’s analytics help documents the utm family, such as utm_source for the referrer and utm_campaign for the promotion. Its ads help describes gclid, a click identifier added to the landing address when auto-tagging is on. Microsoft documents msclkid in the same role, and Matomo documents mtm_campaign. Matomo’s own explanation notes that a tagged link and a plain one lead to the same page, which is why such pairs can usually be removed. Usually, not always: a site is free to read any pair it likes.

Read it before anything opens it

The safest thing to do with a strange link is to take it apart as text. Parsing touches no network. Opening is another matter, and the curl documentation lists what an untrusted address can do to a program that fetches it: reach a local service instead of a remote one, pick a port nobody planned for, or use a scheme nobody expected.

Taking a link apart has limits. It cannot say whether the host exists, who runs it, or what the server will send back. It cannot find the registrable domain without the suffix list, and it cannot judge whether an international name is an honest one. It reports how one parser reads the string, and another piece of software may read it differently. That last limit is why the web-platform-tests project keeps one shared file of test cases, each an input and the exact parts expected from it, and why a careful tool shows the browser’s answer and the older grammar’s answer side by side.

The habit worth keeping is small. Look at the host a browser would use, not the one your eye lands on first.