The urlparse function in Python urllib returns the wrong hostname when parsing URL crafted by the malicious user. This may be caused by incorrect handling of IPv6 addresses. The bug could lead to open redirect in web applications which rely on urlparse to extract and validate the domain of redirection URL.

The test case is as follows:

>>> from urllib.parse import urlparse
>>> urlparse(urlparse('http://benign.com\[attacker.com]').hostname
>>> 'attacker.com'

The correct behavior should be raising an invalid URL exception.

I can confirm with 3.7.2 on fedora 29

Here is a unittest where you can test this issue and the result on Python 3.8.0a2 and 3.7.2

>>> 3.8.0a2
./python /tmp/test_bug_36338.py
/tmp/test_bug_36338.py:8: SyntaxWarning: invalid escape sequence \[
  url = 'http://demo.com\[attacker.com]'
3.8.0a2+ (heads/master:23581c018f, Mar 18 2019, 09:17:05) 
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)]
test_bad_url (__main__.TestUrlparse) ... FAIL

======================================================================
FAIL: test_bad_url (__main__.TestUrlparse)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/test_bug_36338.py", line 13, in test_bad_url
    self.assertEqual(hostname, expected_hostname)
AssertionError: 'attacker.com' != 'demo.com'
- attacker.com
+ demo.com


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

>>> 3.7.2
python /tmp/test_bug_36338.py
3.7.2 (default, Jan 16 2019, 19:49:22) 
[GCC 8.2.1 20181215 (Red Hat 8.2.1-6)]
test_bad_url (__main__.TestUrlparse) ... FAIL

======================================================================
FAIL: test_bad_url (__main__.TestUrlparse)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/test_bug_36338.py", line 13, in test_bad_url
    self.assertEqual(hostname, expected_hostname)
AssertionError: 'attacker.com' != 'demo.com'
- attacker.com
+ demo.com


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

See also issue20271 that discusses the other format http://[::1]spam where ::1 is returned as hostname. urlparse tries to parse the hostname as IPV6 address when there is [ and parses till ] at [0] thus "benign.com\[attacker.com]" is treated as a URL where attacker.com is assumed as the IPV6 hostname. I am not sure of the correct behavior. FWIW at least Java and golang return "benign.com[attacker.com]" and Ruby raises an exception that this is a bad URL.

Java

> (.getHost (java.net.URL. "http://benign.com\\[attacker.com]"))
"benign.com\\[attacker.com]"

golang: https://play.golang.org/p/q8pTo9ySLby


[0] https://github.com/python/cpython/blob/c5c6cdada3d41148bdeeacfe7528327b481c5d18/Lib/urllib/parse.py#L199

Given a quick scan of RFC 3986[1] I'd say that the behaviour of Ruby seems to be the most correct. That said, I'd also check what the major browsers do in this case (FWIW both FF and Safari use 'benign.com' as the hostname in this case).


[1] https://tools.ietf.org/html/rfc3986#page-17

I found this page to be uesful : https://url.spec.whatwg.org/#host-parsing and following the steps it seems that this should raise an error since at the 7th step it denotes that asciiDomain shouldn't contain forbidden host code point including "[]" . As another data point using 'new URL("http://benign.com[attacker.com]")' in browser's Javascript console also raises exception that this is a bad URL. Even if attacker.com is assumed to be the correct host by Python it's not validated to be an IPV6 address where it should fail.

Ruby seems to use a regex : https://github.com/ruby/ruby/blob/trunk/lib/uri/rfc3986_parser.rb#L6
Java parseurl : http://hg.openjdk.java.net/jdk/jdk/file/c4c225b49c5f/src/java.base/share/classes/java/net/URLStreamHandler.java#l124
golang : https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/net/url/url.go#L587