Consider percent-encoding more characters in "fragment state"

[mikewest]

Currently, https://url.spec.whatwg.org/#fragment-state requires that we percent-encode characters in a URL's fragment using the C0 control percent-encode set. Firefox currently encodes more than that, which I think actually makes a good deal of sense.

How would y'all feel about shifting fragment state encoding to match query state? That is, encoding "less than 0x21 (!), greater than 0x7E (~), or is 0x22 ("), 0x23 (#), 0x3C (<), or 0x3E (>)"?

[annevk]

See also #18 (comment) and #291.

cc @achristensen07 @jasnell

[annevk]

cc @valenting

[valenting]

How would y'all feel about shifting fragment state encoding to match query state? That is, encoding "less than 0x21 (!), greater than 0x7E (~), or is 0x22 ("), 0x23 (#), 0x3C (<), or 0x3E (>)"?

That sounds good. It would be nice to converge onto something clear.
Firefox also escapes ` in the fragment, but I don't know if we want to keep it that way.

[jasnell]

I'd be good with this. (ping @TimothyGu)

[TimothyGu]

Sounds good to me.

[achristensen07]

What is the motivation for this change? I wouldn't oppose encoding more characters, but I would oppose using the query encoding set as a model.

[mikewest]

@achristensen07: @annevk linked to some related conversations that discuss motivations (@bzbarsky's discussion of <> in #291 is compelling, for instance). I'm hopping on this due to some marginal risk of code injection when developers reflect url.hash into their pages: that risk would be mitigated if we were a little more strict about encoding these characters.

Based on some spot-checking, Firefox encodes at least ", `, <, and > in addition to everything above 0x7E, which is an indication that we can tighten things up here while maintaining compatibility with the web.

Do you have a set of characters you'd prefer to use instead of the query encoding set? As long as the set includes < and >, I'll be happy. :)

[annevk]

Maybe https://url.spec.whatwg.org/#path-percent-encode-set?

[rmisev]

Do you have a set of characters you'd prefer to use instead of the query encoding set? As long as the set includes < and >, I'll be happy. :)

I'll prefer to encode "less than 0x21 (!)" and 0x22 (") as well:

Code point	Why encode?
0x20 (SP)	See #150 (comment) and https://bugzilla.mozilla.org/show_bug.cgi?id=1148861
0x22 (")	For a safe copy/paste into <a href="http://example.com#with%22quote"> vs <a href="http://example.com#with"quote">

[mikewest]

I'm fine with https://url.spec.whatwg.org/#path-percent-encode-set, FWIW.

[achristensen07]

I’m fine with using the path set for the path and fragment.

[mikewest]

Any objections to the path set? If not, I'll put together a PR and tests.

[rmisev]

It seems a bit stricter set than in RFC 3986, which allows U+003F (?) in non-encoded form.
See https://tools.ietf.org/html/rfc3986#section-3.5

[bzbarsky]

Yes, I'm pretty worried about '?' there in compat terms, esp given it being explicitly called out as allowed in the RFC. Actually somewhat worried about '#' and '{' and '}' too, if no one escapes those right now... How many people pass JSON in fragments, for example?

The actual set of chars the RFC allows in fragments, just for reference, is:

  fragment    = *( pchar / "/" / "?" )

where pchar is the set of characters allowed in path segments. So basically, "everything allowed in paths, plus '?'". Which is definitely more restrictive than what UAs do.