gh-95555: Support Unicode property escapes \p{...} in regular expressions#151969
gh-95555: Support Unicode property escapes \p{...} in regular expressions#151969serhiy-storchaka wants to merge 2 commits into
Conversation
…xpressions
Add support for \p{property} and \P{property} in Unicode (str) regular
expressions, for the properties the engine can resolve without the
unicodedata database. They are matched either as CATEGORY opcodes
(character predicates and combinations of them, see sre.c) or as fixed
sets of character ranges.
Supported properties:
* many General_Category values -- the groups L, N, Z, C and the values Lu,
Lt, Lm, Nd, Nl, No, Zs, Zl, Zp, Cc, Cf, Cs, Co and Cn;
* the binary properties Alphabetic, Lowercase, Uppercase, Numeric,
Printable, XID_Start, XID_Continue, Cased and Case_Ignorable;
* the POSIX compatibility classes alpha, alnum, blank, cntrl, digit, graph,
lower, print, space, upper, word and xdigit;
* the code-point classes ASCII, Any, Assigned, Noncharacter_Code_Point,
Join_Control and the immutable Pattern_Syntax and Pattern_White_Space.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documentation build overview
4 files changed± library/dialog.html± library/re.html± whatsnew/3.16.html± whatsnew/changelog.html |
| Matches ``[0-9]`` if the :py:const:`~re.ASCII` flag is used. | ||
|
|
||
| __ https://www.unicode.org/versions/Unicode15.0.0/ch04.pdf#G134153 | ||
| __ https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G124142 |
There was a problem hiding this comment.
This should be added to:
cpython/Tools/unicode/makeunicodedata.py
Lines 42 to 48 in 868d9a8
There was a problem hiding this comment.
This is unrelated?
There was a problem hiding this comment.
I don't think it is, if you're updating the link in this PR.
| @@ -0,0 +1,267 @@ | |||
| # | |||
| # Secret Labs' Regular Expression Engine | |||
There was a problem hiding this comment.
This wasn't written by the company, nor is it licensed to them?
There was a problem hiding this comment.
I think (but I am not sure), the Secret Labs credit is an internal joke. I can drop it if nobody can confirm this.
There was a problem hiding this comment.
I don't think it's a joke? It was a real company, founded by Fredrik Lundh. See his bio here, for reference.
There was a problem hiding this comment.
It seems I was wrong, it was a real Swedish company of Fredrik Lundh (a.k.a. "the effbot"). References to it are everywhere in the code, even in the module name _sre. So I'll leave it. We're extending their engine, not re-attributing it.
… properties They are complete fixed sets, matched as fixed ranges: Regional_Indicator (the 26 symbols A..Z), ASCII_Hex_Digit (the ASCII hex digits, = POSIX xdigit) and Hex_Digit (which adds the fullwidth forms). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2 participants
Add support for
\p{property}and\P{property}escapes in Unicode (str) regular expressions, for the properties the engine can resolve without theunicodedatadatabase. They are matched either asCATEGORYopcodes (character predicates and combinations of them) or as fixed sets of character ranges, so neither the matcher nor the compiler gains aunicodedatadependency.Supported in this change:
General_Categoryvalues — the groupsL,N,Z,Cand the valuesLu,Lt,Lm,Nd,Nl,No,Zs,Zl,Zp,Cc,Cf,Cs,CoandCn;Alphabetic,Lowercase,Uppercase,Numeric,Printable,XID_Start,XID_Continue,CasedandCase_Ignorable;alpha,alnum,blank,cntrl,digit,graph,lower,print,space,upper,wordandxdigit;ASCII,Any,Assigned,Noncharacter_Code_Point,Join_Controland the immutablePattern_SyntaxandPattern_White_Space.Property and value names use loose matching (UAX #44 UAX44-LM3), and a property may be spelled
\p{Lu},\p{gc=Lu}or\p{name=yes}.The remaining table-based properties (the
General_CategoryvaluesLl/Loand theM/P/Sfamilies,Block, and the other enumerated properties) require theunicodedatatables and are intentionally left out of this first change, to be added separately.reshould support\p{...}character properties #95555