@Makyen Hmm. If rebooting SD fixes it, how can the root cause be an SE API bug? To me, that seems to make it far more likely that it's a race-condition or other multi-threading edge case in SD.
@Spevacus @Jesse Many/most things that we add to the watchlist are trivial to change. Domain names can be easily changed by simply registering a new one. YouTube, Blogspot, Twitter, and other IDs can be easily changed by either creating a new account or uploading a new copy of the same content. Save for something like names of a business/service, which are actually less likely to be unique and thus less good to add to the watchlist (although, don't get me wrong, still should be added).
All of that to say that the watchlist/blacklist working effectively really relies on the laziness of spammers.
@CodyGray speculation: rebooting SD changes something about its state that causes it to not interact with the SE API in the way that triggers the SE API bug
@Makyen And yet, I've seen a fair number of URLs being obfuscated. Presumably, the idea is that a human is going to just type them into the address bar.
But when you have a stateful object interacting with a stateless object, there's no state that could be retained by the stateful object that would cause issues with the stateless object. That means that SD's code is to blame, not the SE API.
@tripleee A normalized version, {{'917065463977'}}, of that pattern is already on the number watchlist. You can use !!/bisect-number +917065463977 to determine which entries on the number lists match that pattern, which would tell you: Matched by +917065463977 on line 3397 of watched_numbers.txt: +917065463977 found verbatim
For instance, consider a hypothetical implementation of an edit-watching feature that watches for grace-period edits, and includes a piece of state about which posts it's watching. If the API returns incorrect information about a post's state, causing that feature to misbehave, that's the API's fault, not SD's.
@tripleee It's doing a different process. The merged commits say "SmokeDetector merged commit xxxx into master n days ago", whereas the closed commits say "SmokeDetector added a commit that referenced this pull request n hours ago", then SD is closing it.
@CodyGray not sure exactly; seems like there is some sort of timing error on CircleCI sometimes when it fetches stuff from Github but I'm basically only relaying what Makyen told me (probably imperfectly) from the previous time it happened to me
@tripleee Yeah, that's an extension of the users.yml issue we've had for a while (since I switched the CI testing to be multithreaded). It used to be only a single user missing. I wrote code that added a bunch of locks which should fix that, and some other issues/potential issues (some are CI only issues, while others are real). My concern is that I may have gone a bit overboard on locks and I need to go back though those changes and see which ones are really not needed. I also need to do at least another fairly long run of a fully active SD instance with those changes to see if I munged up any of the locks. I also stalled out on it because it looks like there's a multithreading issue in the regex package, but I didn't actually find it. I worked around it, but I still need to confirm that what I did really resolved "the" multithreading issue there, or if issue isn't really in the regex package and the code I added just moved other processing around enough such that the issue went away. In other words, I've worked on this issue/these issues, but it needs more work.
@tripleee That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer Append -force to the command word(s) if you really want to add the pattern you provided.
@CodyGray Doesn't rebooting fix everything. :) So far, what we've seen is that it's either "spontaneously" resolved itself (obviously while other things are happening, so not necessarily actually spontaneously) or a user has ignored one or both of the posts that was being reported. I don't believe rebooting has actually been tried, but would, at a minimum, delay things for 30 seconds, which might change the external conditions. Overall, it's an assumption on my part that rebooting will resolve the issue.
The issue could, of course, be entirely in SD. But, if so, why is it only occurring now, starting a month and a half after the last SD changes and being notably more frequent than once/month? OTOH, the only thing we really can affect is SD's and MS' code, so those are the ones we'll be changing in attempts to address the issue.
@CodyGray It does happen, but it's relatively rare.
@CodyGray The SE API itself isn't stateless, as was clearly demonstrated in December when an automatic adjustment in the SE API's internal state resulted in SD getting large numbers of errors, due to the execution plan changing for a database query it was performing in order to respond to SD's requests. In addition, the data which the SE API is delivering isn't stateless. SE API queries are also serviced from multiple servers, which don't, necessarily, maintain 100% perfect synchronization. So, no, interactions with it are not stateless. If the data never changed (i.e. was stateless), then we wouldn't be able to scan posts. :;
One of the possibilities I was describing earlier that might cause the issue is for the SE API to deliver data to consecutive requests which ping-pongs between two versions of the post(s). That could happen if the servers responding to the SE API requests get slightly out of sync. That possibility is something which SD should be taking into account, but it currently doesn't.
However, even that shouldn't really be causing the exact reports which we're seeing, as they should be excluded from being reported again due to not having additional detection reasons.
@CodyGray No, adding the locks resolved the issues. But, each lock adds some risk of deadlocking if not being properly handled. So, I wanted to re-review those changes and perform more thorough testing. In particular, I wanted to re-review them to see where what I'd added was actually already covered by Python's GIL.
@tripleee That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer Append -force to the command word(s) if you really want to add the pattern you provided.
@CodyGray It's not how I initially interpreted what you meant, and it's not how it's typically meant in that context, but the "typical" definition doesn't really cover what happens. The process behind the responses from the SE API isn't necessarily the same each time, which can result in different responses to the same request, even if the underlying data hasn't changed.
@CodyGray That pattern looks like it's already caught by Blacklisted website in body and Blacklisted website in answer Append -force to the command word(s) if you really want to add the pattern you provided.
