I'm using the PyPI module regex for regex matching. It says
Default Unicode word boundary
The WORD flag changes the definition of a ‘word boundary’ to that of a default Unicode word boundary. This applies to \b and \B.
But nothing seems to have changed:
>>> r1 = regex.compile(r"...
@iBug Thanks for the heads-up. I'll keep a watch on it.
@iBug Yeah, my expectation is that it would have most effect on non-western languages. I suspect that to figure it out would require more research than I was willing to put in. :-;
@Makyen Basically there are two types of languages: Character-based and word-based. Chinese is a char-based while Latin/Greek/Cyrillics are word-based. I think this is a key difference.
@iBug That's very likely a key difference. I'm interested in the details, but, really, only actually need to know enough to determine what's necessary to use in a regex in order to be sure to consume enough characters up to a \b and to be able to know if the WORD flag should be used here. Knowing if it should be used elsewhere probably requires an even better understanding.
Given how it's going, I think we should enable that flag so Chinese keywords match correctly.
> For Thai, Lao, Khmer, Myanmar, and other scripts that do not typically use spaces between words, a good implementation should not depend on the default word boundary specification.
> Chinese character in title, bad keyword in body, bad keyword in title, bad keyword in username, mostly non-latin body, mostly non-latin title ---------- Title - Position 1-8: 办理毕业证成绩单 Title - Position 1-8: 办理毕业证成绩单 Title - Text contains 8 non-Latin characters out of 8 Body - Position 1-8: 办理毕业证成绩单 Body - Text contains 8 non-Latin characters out of 8 Username - Position 1-8: 办理毕业证成绩单
> Chinese character in title, bad keyword in body, bad keyword in title, bad keyword in username, mostly non-latin body, mostly non-latin title ---------- Title - Position 1-3: 毕业证 Title - Position 1-3: 毕业证 Title - Text contains 3 non-Latin characters out of 3 Body - Position 1-3: 毕业证 Body - Text contains 3 non-Latin characters out of 3 Username - Position 1-3: 毕业证
> Definition of ‘word’ character (issue #1693050) > > The definition of a ‘word’ character has been expanded for Unicode. It now conforms to the Unicode specification at http://www.unicode.org/reports/tr29/.
> Default Unicode word boundary > > The WORD flag changes the definition of a ‘word boundary’ to that of a default Unicode word boundary. This applies to \b and \B.
> Chinese character in title, bad keyword in body, bad keyword in title, bad keyword in username, mostly non-latin body, mostly non-latin title ---------- Title - Position 3-5: 毕业证 Title - Position 1-12: 办理毕业证成绩单学位认证 Title - Text contains 12 non-Latin characters out of 12 Body - Position 3-5: 毕业证 Body - Text contains 12 non-Latin characters out of 12 Username - Position 3-5: 毕业证
> Chinese character in title, bad keyword in body, bad keyword in title, bad keyword in username, mostly non-latin body, mostly non-latin title ---------- Title - Position 3-5: 毕业证, Position 9-12: 学历认证 Title - Position 1-12: 办理毕业证成绩单学历认证 Title - Text contains 12 non-Latin characters out of 12 Body - Position 3-5: 毕业证, Position 9-12: 学历认证 Body - Text contains 12 non-Latin characters out of 12 Username - Position 3-5: 毕业证, Position 9-12: 学历认证
@iBug IMO, we should continue to not use the WORD flag. Using it would be significantly more confusing for writing our regexes for English (i.e. Western languages), which is the vast majority of what we write them for. While it may make more sense to use it in some situations (e.g. non-Western languages), the negatives of using it outweigh the positives, by a significant margin. We will just have to live with our regexes that cover things like 毕业证being individually bookended with \w*.
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, pattern-matching product name in body, potentially bad ns for domain in body, +1 more: AmbroSina Skin Cream by mdjfsjad on apple.SE
@TetsuyaYamamoto That pattern looks like it's already caught by Pattern-matching website in body and Pattern-matching website in answer; append -force if you really want to do that.
@TetsuyaYamamoto That pattern looks like it's already caught by Pattern-matching website in body and Pattern-matching website in answer; append -force if you really want to do that.
@TetsuyaYamamoto Added artificialeyesindia\.com to watchlist
[ SmokeDetector | MS ] Blacklisted website in body, blacklisted website in title, mostly dots in title, potentially bad ns for domain in body, potentially bad ns for domain in title, +2 more: .alphaforcetestorev.com/lojesete/ by user48621 on rpg.SE
> Pattern-matching product name in body, pattern-matching product name in title ---------- Title - Pattern-matching product name: Positions 1-3, 7-9, 13-15: x x Body - Pattern-matching product name: Positions 1-3, 7-9, 13-15: x x
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, bad pattern in url body, blacklisted website in body, link at end of body, +2 more: Paltroxt reviews suppliment by gubymoqu on meta.SE
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link following arrow in body, pattern-matching product name in body, +2 more: Where to buy Praltrix? by lancebyrge on stackoverflow.com
tpu- by Shree
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, pattern-matching product name in body, potentially bad ns for domain in body: Viantis Male Enhancement by mdjfsjas on apple.SE
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, pattern-matching website in body, potentially bad ns for domain in body, +1 more: Dermavix Best for Skin care by Darmicoolzz on graphicdesign.SE
@Shree That pattern looks like it's already caught by Pattern-matching website in body, Potentially bad keyword in body, Pattern-matching website in answer, and Potentially bad keyword in answer; append -force if you really want to do that.
@bertieb I am aware of 32 code admins. Currently in this room: tripleee, Glorfindel, Mithrandir, bwDraco, Makyen, John Dvorak, Undo, Byte Commander. Not currently in this room: iBug, doppelgreener, WELZ, CalvT, quartata, NobodyNada, Thomas Ward, Ferrybig, angussidney, K.Dᴀᴠɪs, Suraj Rao, Zoe, Ashish Ahuja, Magisch, ArtOfCode, micsthepick, ProgramFOX, paper1111, Henders, DavidPostill, Tetsuya Yamamoto, Andy, thesecretmaster, Paul Stenne.
[ SmokeDetector | MS ] Blacklisted website in body, link following arrow in body, potentially bad keyword in body, potentially bad keyword in title, repeated url at end of long post: Where to buy Keto Detox Tea? by cathyjames on stackoverflow.com
[ SmokeDetector | MS ] Bad keyword in link text in answer, link at end of answer, pattern-matching website in answer, potentially bad ns for domain in answer, potentially bad keyword in answer: CentOS 7 with Red Hat Satellite 6 by Sam smith on serverfault.com
@tripleee That pattern looks like it's already caught by Pattern-matching website in answer and Pattern-matching website in body; append -force if you really want to do that.
I'm using the PyPI module regex for regex matching. It says
Default Unicode word boundary
The WORD flag changes the definition of a ‘word boundary’ to that of a default Unicode word boundary. This applies to \b and \B.
But nothing seems to have changed:
>>> r1 = regex.compile(r"...
