I was wondering whether we would wind up with FPs, though. We want to catch sneaky cases, but at the same time, we need to be careful not to cast too wide a net.
In this case, unlikely, but something to bear in mind when using broad regexes like this in the future.
Also, we have separate blacklists for keywords and domains.
For the purpose of the watchlist, though, it makes sense.
This whole business of enumerating badness is a walk on the slippery rocks. You know what Smokey needs is a little python scraper that visits the link, gets the html and filters out the words, does a "spam test" baysian filter like in email spam filtering on the words in the link, and if it smells spammy, then bring it up.
Better yet, download the images in the link, do HoughCircles and HoughLines on it, dump it into a neural network, do OCR on it, dump that text into a decision tree, do Harris Corners and color analysis, dump it into a n dimensional logistic regression. and BAM, you've got yourself 8000 lines of brittle gobbledygook that either works perfectly or pukes on the floor.
What we need is 8000 lines of very complex PHD level math code that interprets the spam links, and is better-than-human 97% of the time, but the other 3% of the time it pukes up bugs in the linear algebra guts of the code that nobody understands.
> Bad keyword with email in body, bad keyword with email in title, pattern-matching email in body, pattern-matching email in title ---------- Title - Keyword *loan* with email loan@gmail.com Title - Pattern-matching email loan@gmail.com Post - Keyword *loan* with email loan@gmail.com Post - Pattern-matching email loan@gmail.com
> Bad keyword with email in body, bad keyword with email in title, pattern-matching email in body, pattern-matching email in title ---------- Title - Keyword *loan* with email loan@yahoo.com Title - Pattern-matching email loan@yahoo.com Post - Keyword *loan* with email loan@yahoo.com Post - Pattern-matching email loan@yahoo.com
> Bad keyword with email in body, bad keyword with email in title, pattern-matching email in body, pattern-matching email in title ---------- Title - Keyword *loan* with email loan@abc.def Title - Pattern-matching email loan@abc.def Post - Keyword *loan* with email loan@abc.def Post - Pattern-matching email loan@abc.def
> Bad keyword with email in body, bad keyword with email in title, pattern-matching email in body, pattern-matching email in title ---------- Title - Keyword *loan* with email loan@ab.cd Title - Pattern-matching email loan@ab.cd Post - Keyword *loan* with email loan@ab.cd Post - Pattern-matching email loan@ab.cd
The email part of the regex is copied from keyword_email, so bad keyword with email and pattern-matching email recognize the same things as emails
That is a pretty complicated regex though
I think the (?!(example|domain|site|foo|\dx)\.[A-z]{2,4}) is a lookahead which asserts that the domain does not contain those whitelisted ones, and [A-z0-9_.%+-]+\.[A-z]{2,4}\b matches 2-4 occurrences of alphanumerics, underscore, dot, percent, and plus, followed by a dot, followed by 2-4 letters and a word boundary
