> Have you ever broken the surface of something with a hammer? Ever channeled sublime thought into sandpaper? Ever wanted to touch something because you feel things, because touch is the only sense you trust? Incomplete? What is trust? Is making a thing proof that you exist? Is fixing a thing proof that you have transcended mortality? History? Incomplete? Feel things? Feel things? You can do it. We can help. The Home Depot. (source)
Redundant with shortened URL detection; has never caught anything not detected for a different reason (often the other reason is the shortened URL detection)
@JeffSchaller That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; append -force if you really want to do that.
[ SmokeDetector | MS ] Bad keyword in answer, pattern-matching website in answer, potentially bad ip for hostname in answer, potentially bad ns for domain in answer, potentially bad keyword in answer (167): "Bad" Mathematics in Movies by Md Ikram Parvez on math.SE
@Makyen re. your comments about intermediate IDs: I'm not sure I understand what circumstances we would want to fetch more than 100ish intermediate IDs. AFAIK the purpose of intermediate IDs is primarily to make sure we don't miss anything on a reboot, which shouldn't normally be more than 20 seconds or so
I can see that potentially being a useful feature for recovering from brief outages -- however, we couldn't really look further than maybe 10 or 20 minutes back before we start to have a risk of duplicate reports whenever we failover
@NobodyNada I understood it to be our way of trying to make sure we don't miss scanning any posts as a result of any of the ways which might result in not having an SD instance scanning for a period of time (potential examples: an SD dies and we fail over to a new one; the SE WebSocket is broken and we get no notifications for N minutes; we have a problem where all SD instances are down for some significant period of time; etc.).
I think what we really need to decide is what range of issues are we attempting to recover from with no posts going unscanned. If we're trying to cover only a minute or two, then 100 is sufficient. If we're trying to cover much more than that, then 100 isn't sufficient for all sites at all times.
Also, I seem to recall some notes indicating that the WebSocket doesn't, necessarily, get notifications for every post modification on SO. Going back through the intervening posts is the way to assure that any such non-WebSocket-reported posts are scanned.
However, it doesn't actually assure that. It just assures that any new posts get scanned (up to 100), but doesn't actually deal with non-reported edits.
@Makyen We actually use entirely different logic for Stack Overflow posts. We don't pay any attention to what came over the websocket or the interpolated IDs; instead, we just grab the latest 25 active questions (or 50 after a reboot).
@Makyen That makes sense. I'm not sure if it's feasible to extend the logic beyond more than a few minutes, because from SmokeDetector's perspective there is very little difference between a routine failover and recovering from an extended outage.
An alternative solution might be a !!/scan-since command which scans all posts since a given timestamp, so that if we go down for, say, an hour, we can catch up manually.
However, such a command would need to be used only in extreme situations (and probably require a high privilege level), since it would need to make 200-500 API requests.
SO has a queue size of 3, so I think fetching 50 during normal operation (16x the queue size) should be reasonably safe, but I'm not so sure about the 25 on reboot
Why are we not just requesting 100 questions there and dropping any that have a last active time which is before the last active time we have recorded? It costs more processing, and is more of a load on SE for each request, but makes it more likely we don't miss anything (and doesn't cost us any more API quota).
@Makyen Maybe a better solution than intermediate IDs would be to always fetch /questions?sort=activity&min=<10 minutes ago> on the first request to any site (or something along those lines).
