Support bytestring URLs on Python 3.x

[joealcorn]

Hi there folks.

Currently prepare_url will call unicode or str on the url arg depending on the python version. This works fine for most cases, but the one case it trips up on is bytestrings on python 3.x as the string representation of these is "b'http://httpbin.org'". Eventually this will surface as an InvalidSchema exception.

I find this to be completely unexpected, and I'd imagine it's not something that's been done intentionally.

Technically this a breaking change. The possibility of passing non-strings to prepare_url is undocumented and untested, but regardless it may be better to go about fixing this in a different way, that's your call.

[Lukasa]

Thanks for this!

I think we like the ability to pass non-strings to prepare_url. It allows people to use custom url-building classes without running into trouble. It's worth noting that you change also breaks Python 2 behaviour: previously a Python 2 str would get lifted to a Python 2 unicode, which it now doesn't do.

I think we can get around this by simply special-casing string types. The logic is really:

if type is unicode, leave unchanged
else, if type is bytes, decode bytestring to unicode
else, if type is anything else, call the 'to unicode' method

I think that's really the logic we want here. @sigmavirus24, thoughts?

[sigmavirus24]

I agree with @Lukasa that this is the behaviour we want. We absolutely want unicode urls because on Python 3, we can handle IRIs. We would need to adopt something that implements RFC 3987 to support it on Python 2, but once we did, we would be able to support them there too. For example, http://☃.net should be supported. (Your browser, and Python 3 should properly "encode" that to http://xn--n3h.net/.)

[joealcorn]

Ah good point, forgot about that use case.
Have updated the branch, I believe it's doing what it should be now, how's that?

[sigmavirus24]

Would this be acceptable:

try:
    url = url.decode('utf8')
except AttributeError:
    url = unicode(url) if not is_py2 else str(url)

[joealcorn]

That's nice and succinct, have ammended and pushed

[sigmavirus24]

❤️ @buttscicles

@Lukasa this looks okay to me. Thoughts?

[Lukasa]

🍰 Make it so.

[kennethreitz]

Let's not document this :)