FR: For websites that have been blacklisted, also check "website(dot)tld" instead of just "website.tld". We had an occurence today with proofy.io spam being masked as proofy(dot)io in a post. cc @Undo @ArtOfCode
based on the hits I see in metasmoke, I suppose the code could normalize any occurrence of (at) and (dot) before applying the blacklist, but this seems like a rather gross hack to me
@AshishAhuja If we were to blacklist the word "proofy" then we'd catch thousands of FPs in due time. If we did "proofy" in a website context, well, proofy(dot)io is not technically a valid link so no dice.
@AshishAhuja yes but the occurences of ipubsoft in legit conversation are minimal
"proofy" is a colloquial term for proof-like in mathemathics.
[ SmokeDetector | MS ] URL in title, bad keyword in body, link at end of body, link following arrow in body, pattern-matching website in body, pattern-matching website in title: newmusclesupplements.com/testo-boost-x/ by user129302 on security.stackexchange.com
@DavidPostill the problem with that is you have to repeat that regex magic in every regular expression where you want this to be true
the Metasmoke hits indicate that this is a trick used by multiple spammers, not just a feature of one individual blacklist entry
so it's better to solve that problem once in the code
I was vaguely thinking of adding flags to the blacklist, sort of like some URL rewriting sstems do, so regex [ID] would make regex match case-insensitively and normalize any (dot), <dot>, [dot] etc but it's actually not a system I would be particularly happy with
@tripleee That's exactly what I meant. Do the checks when evaluating the post for a blacklisted entry. I wasn't suggesting adding multiple blacklist entries.
if it's a list of URLs, I would expect the code to do various normalizations to allow for subdomains in the host part, http vs https, etc
currently, it's neither here nor there, and as long as our needs are simple, I suppose that's fine
but if you want to develop it further, there should be a consensus on which direction it should be developed into
eventually maybe we will want both -- a really small but updatable regex collection which we want to keep as controlled as possible, and a URL black/whitelist with some sophisticated features specific to URL matching
spam urls are pasted in very irregular patterns, thats why we need to be specific while blacklisting. For instance, when I blacklist proofy\.io then only exact matches of "proofy.io" will be matched. But all of those. Its an approximation game after all.
Thats why the general guidance is to put <domainname>.<tld> or otherwise regexes following this pattern in there. Its meant to indicate improper mention of a website, not a general blacklisted keyword.
but to me, as a newcomer to the code, the lack of clarity increases the threshold for adding new entries
the specific patterns like //j\.gs/ might miss some legit hits because they are too specific
I'm not saying this is bad or a problem, just that lack of false positives is not the full picture
(j.gs is apparently a link shortener, so this particular case is only indicative of a nonproblem where I had to spend time to find out why it is what it is, but that's one of the things which could be fixed with different semantics for the blacklist)
Smokey is like any piece of software - it's been written by several different Pepe over a large time span. It's inevitable that the code base is going to be sub optimal. If you want to improve that, I don't have an issue with it, just try not to introduce too many new bugs.
@ArtOfCode blacklisted websites have been actively circumventing our filter by using proofy(dot)io instead of proofy.io in spam posts. Multiple spam websites are doing this.
@ArtOfCode So the two floating ideas are currently to either make blacklisted_websites.txt into a true list of websites with no regex escaping/etc and handle that programmatically
or to use (?:[.]|\[dot\]|\(dot\)|<dot>) for future blacklist entries and update existing ones in place of the current literal \. part of the blacklist regex
We can have that cake and eat it, actually. Leave the file as a list of regexes, but do a substitution before compiling the regex - website_regex.replace("\\.", "(?:[.]|\\[dot\\]|\\(dot\\)|<dot>)")
@art when I try to access blaze with https I get an insecure connection, so is it okay if I just put http in the charcoal se website while making the PR?
@ArtOfCode Maybe we can redesign the blacklist command so it turns !!/blacklist proofy.io into proofy(?:[.]|\[dot\]|\(dot\)|<dot>|\{dot\}|\"dot\"|\'dot\')io (I expanded the list of alternate dots a bit)
@ArtOfCode a URL blacklist with specific URL patterns where the code implements some URL-specific things such as perhaps the "<dot>" substitution, and a separate regex blacklist where the entries are applied verbatim everywhere
@ArtOfCode If you can handle the making of the files and changes so it gets the regex from a file, I can start migrating the regex list into the files for you.
Moderators on Stack Overflow are currently discussing the addition of common link shorteners (goo.gl, bit.ly, tinyurl) to the site's blacklist. Before we do so, we'd like to see if the community approves of this.
We have discussed this in the past, and the community has seemed to be in favor of ...
Common link shorteners have been blacklisted.
Not git.io though.
There are currently 100 posts containing a git.io link.
That is more than what most shorteners listed in the cleanup post have.
Is it somehow intentional that git.io is not blacklisted, or has it just been overlooked?
@hichris123 I think making them flat files would benefit contributors a lot
saying "these are just regexes. One regex = one line" makes it very simple and universally obvious how to contribute to it, while changing nothing for the worse
@Magisch I disagree. I think all of these should be stored in a database and not part of the code. Adding a regex shouldn't force an entire CI check, pull, restart. It should refresh the cache of checks in memory and keep chugging along.
I think any regularly changing big list of regexes should be migrated to its own file, to keep a neat tab on how often its changed, make it easy to contribute to & unit test and make it nicely understandable for anyone reading the code.
@Andy Thats already out of scope according to undo
still, I believe outsourcing to a clean list would be superior, easier to manage, easier to track, easier to contribute to and all in all easier understandable.
Undo is reasonable and if we argued the point and convinced him, he'd change his mind. But, I agree with @hichris123, it's not worth that argument right now, so I haven't pursued it.
@hichris123 the point is that there'll be someone who knows regex but hasn't the foggiest how to put it into Python. Magisch can do it; someone else won't be able to.
@tripleee No offense to those developers, but they may need to take a few minutes to look at the code then, before trying to get a new regular expression - which I'd consider vastly more complicated than adding a string to a source file
Python, you could miss the closing quote. You could miss the comma. You could delete the array-end bracket by mistake thinking it's part of your regex.
@Andy do you think this is making it too slow? based on my vague impression of how the system works, this is going to be I/O bound even with thousands of regexss
Honestly, I don't mind how we do it. But moving all our massive lists of stuff into resource files gives us the potential for chat commands like !!/blacklist, and for integration with metasmoke. Those are two pretty big gains.
I can understand the point about pattern-matching regexes being in code. I'm not sure I agree, but I can understand it. There are other lists we can work on, though.
So, what's the next most innocuous list, next to blacklisted websites? I'm thinking keywords? That could be the next target to work on as an experiment.
Okay. Let's go ahead and move bad keywords to a resource file, if @Mag is willing to do that work. We can use that as an experiment - if it goes well, we'll look at doing more in the future.
> Bad keyword in body, bad keyword in title, bad keyword in username ---------- Title - Position 1-12: JobsTribune Body - Position 1-12: JobsTribune Username - Position 1-12: JobsTribune
Backoff received of 10 seconds on request to questions/130072;130117?site=mathematica at 15:10:32 UTC
Common link shorteners have been blacklisted.
Not git.io though.
There are currently 100 posts containing a git.io link.
That is more than what most shorteners listed in the cleanup post have.
Is it somehow intentional that git.io is not blacklisted, or has it just been overlooked?
