@JeffSchaller Given the very short URL path, I probably would have used (?-i:IpMJ)(?<=\bt\.ly/IpMJ), which won't then be shadowed, or shadow, a watch for the domain. (cc @Spevacus)
@Mast I put in a similar one a week or so ago (brought[\W_]*+my[\W_]*+(?:husband|wife)[\W_]*+back) - no TP or FP yet, might be another one to move over at the same time when you're looking at blacklisting that
teward/Osiris: In getting MS post information, recovered from requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='metasmoke.erwaysoftware.com', port=443): Read timed out. (read timeout=10.0)
teward/Osiris: In getting MS post information, recovered from requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='metasmoke.erwaysoftware.com', port=443): Read timed out. (read timeout=10.0)
I'll try to help in the capacity that I can, with respect to those improvements. I've been trying to learn more about the dark art that is regex... While the nuances evade me, I've been starting to get it. Sort of.
@RyanM Yes, it is frustrating. Prior to having write access, I submitted quite a few PRs to make such changes.
@RyanM Yes, but we don't have a way to allow write access just to those files. It's also nice to have a more noticeable record of such changes, under most conditions.
there are some (more potentially controversial ones) that I do actually want to discuss on a PR and get buy-in from others, but others where I'm pretty confident that no one would object (example)
Ideally Smokey could allow a blacklister to !!/approve any PR that requires only blacklister approval in PullApprove, but that integration would likely be non-trivial
(especially because of the need to avoid a TOC-TOU vulnerability)
@Spevacus while we're on the topic: I generated some watchlist stats if anyone is looking for easy promotions/removals
I've been meaning to generate some actionable data from those, but I haven't had the time -- if anyone else wants to take a look at it before I get around to it, you're more than welcome
@NobodyNada Nice! I've had a vague idea of trying to analyze the marginal value of a watch: that is, what exactly is it adding? We have some watches that catch a large number of TP, but are almost always piling onto other detections when they do, while they catch a lot of FP. This means that their stats from a simple search are artificially inflated, and really they're adding almost entirely FP.
The issue here is that actually testing this would require running large parts of SD locally on posts from the DB, and I'd have to build some infrastructure to do that.
also, that may be the first and last time I try to use the GitHub online file editor... it took me way too many attempts, and the result was suboptimal (meaningless branch name, random email).
@NobodyNada Ummm... I haven't looked at these in any detail. However, looking at just the first one, essayssos\.com has me a bit concerned about the data. The data shows 0/0/0, but a
@RyanM Yeah, I've used it a few times, but I definitely prefer to work on a local copy and push back a branch to my fork, or the main repo, from the local repository.
@RyanM Yes, I was lazy and only ran the check on bodies. The only other caveat I can remember is that I used ICU regex instead of Smokey's regex. The results should be the same in almost all cases.
However, notably I had to limit all quantifiers to a maximum of 100 characters, because although ICU regex does support variable-length lookbehinds, it does not support lookbehinds of unbounded length.
(That was because I wrote the script in Swift for simplicity and performance, with the intention of using Swift's Python interop to call into the regex library Smokey uses. Unfortunately, Swift's python interop is not currently thread-safe, and due to time constraints I chose to switch regex engines instead of rewriting everything in Python)
@RyanM When I have time to do a Python rewrite of the script (or once Swift learns how to deal with the GIL), this would be reasonably straightforward to add, although more checks always means less performance. It already takes several hours to test the whole MS corpus, even after filtering on posts reported as "potentially bad keyword"
Yeah...off the cuff, I think the best approach would be to precompute everything by re-running every post through every detection. That's going to take a bit over 3 days, minimum, though, just due to the throughput limit on Perspective. From there, you could work off that more quickly.
@RyanM after thinking about it again, you could just look at the reason weight in MS, since all experimental reasons have a weight of 1
so a watch "adds value" if the reason weight is less than some number like 10 -- i.e. it was caught only by experimental reasons
then you're not re-running all the reasons, you're just seeing which reasons it was caught by at the time
(you could do the same thing by looking at the list of reasons recorded in MS, rather than the weight -- but the weight is easier since you don't have to do another JOIN)
That's fairly true for newer reports, but less so for older ones where the detections have since changed significantly. It may also be the case that additional watches have been added that would catch similar posts, so a given watch may not add any value there. That's still a bit tricky, though, because you can't just say that every watch detecting that post doesn't add any value...at least one of them does, if it was not caught for other reasons.
@NobodyNada both require a JOIN, I believe. The weight isn't on the post record.
(at least, not in Blazer's view of things...)
For maximum ability to analyze the data, you'd want to treat each watch/blacklist entry as its own reason. But ...that would be a lot of reasons. Not sure how practical that would be, even for offline analysis.
@RyanM That's sorta what you have to do for counting watchlists anyway. You could get the bulk of the work out of the way by just making a list of post IDs caught by each watch, then save that for analysis
One trickier thing I want to do is do a deeper dive into some of the more complex regexes. I'm fairly confident that "kill" can be removed from the workplace-troll regexes without ill effect (especially after a few new watches I added), but it's tricky to prove that. But even that wouldn't be hard to query with that pre-processed dataset.
find all posts detected by only those watches, see which would no longer be detected after a change
@tripleee That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; append -force if you really want to do that.
!!/watch calgovcouncil\.org(?#currently appears to be hacked by keto spammers, remove this watch if it looks anything like a government website when you check it)
@RyanM That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; append -force if you really want to do that.
@tripleee That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; append -force if you really want to do that.
