re.sub, re.Match.expand, etc doesn't allow x, u, U, or N escapes in the template

[mr-nfamous]

BPO	46350
Nosy	@stevendaprano, @mr-nfamous

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-01-11.22:10:12.175>
labels = ['type-bug']
title = "re.sub, re.Match.expand, etc doesn't allow x, u, U, or N escapes in the template"
updated_at = <Date 2022-01-11.22:21:30.194>
user = 'https://github.com/mr-nfamous'

bugs.python.org fields:

activity = <Date 2022-01-11.22:21:30.194>
actor = 'steven.daprano'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2022-01-11.22:10:12.175>
creator = 'bup'
dependencies = []
files = []
hgrepos = []
issue_num = 46350
keywords = []
message_count = 2.0
messages = ['410337', '410340']
nosy_count = 2.0
nosy_names = ['steven.daprano', 'bup']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46350'
versions = []

[mr-nfamous]

The docs use the phrase "unknown escapes of ASCII letters are reserved for future use and treated as errors". That seems ambiguous enough to question why "\x", "\u", "\U", and "\N{}" escapes aren't expanded in the template parameter like they are in patterns.

Since I didn't get a response to the security report I submitted a few weeks ago about \N{} escapes, I'm cautiously assuming it's safe to bring it up here that the "unicode-escape" encoding and re and probably everything else that uses it ignores two obvious clues that a name lookup will fail: length and the presence of invalid characters. I didn't look very hard for a definite length cap in the spec, but 255 seems more than sufficient, based on longest name at present with its 82 characters. Even something as absurd as 65535 would be preferable to the current implementations, which will keep going to the end as in:

>>> r"\N{%s}" % ("\ufb03"*2**30)

searching or a terminating "}" and still perform a lookup of the 2**30 character name.

Another tangentially related "bug" (which probably deserves its own issue) is the inconsistency between group names and standard Python identifiers. The following example shows how the python compiler decomposes a ligature 'ffi' in source code to the ASCII string "ffi", while re merely checks if it could be converted to an identifier:

    >>> ffi = re.search("(?P<ffi>.)", "xxx")
    >>> ffi.groupdict()
    {'ffi': 'x'}
    >>> "\ufb03" in vars(), "\ufb03" in _
    (False, True)

[stevendaprano]

It is better to raise each issue in its own ticket.

You seem to have three distinct issues here:

• The issue listed in the title, which I don't understand. A demonstration of the issue would be helpful.

• The unrelated(?) issue of bad \N{} escapes, which appears to have nothing to do with regexes, but maybe I'm wrong? In any case, whether this is regular expressions or plain old strings, it seems reasonable to me to limit \N{} to ASCII-only (Unicode guarantees that code point names are ASCII) and some reasonable length. I can't imagine the Unicode consortium ever deciding on a name > 255 characters, it wouldn't be practical.

• The difference in normalisation between group names and identifiers.

[arhadthedev]

The following points really hold true for template parameters:

• Unicode guarantees that code point names are ASCII

• I can't imagine the Unicode consortium ever deciding on a name > 255 characters, it wouldn't be practical

So the reported failure happens only when the fixed lookup would fail too. As a result, the bugfix will not change observable behavior and become dead code.