@cocomac as for that...meh. Charcoal has enough regulars to get six flags on the blatant stuff, and sometimes there's a good reason red flags haven't been handled. Fun fact: sometimes even Stack Overflow mods (you know, the ones who have the post gone and the spammer's account suspended or deleted within a minute of posting) sit on spam flags for undisclosed affiliation for a little while until someone has time to go through and handle them.
We often don't want those posts red-flag nuked, since we want more control over the penalty than that - so we delete them without casting our own flags.
@forest There are currently 2466 posts with no feedback on them. That could be a lot of noise, but isn't a terrible idea for how to get them in front of reviewers, actually...
@cigien Thank you. I do know this, but mostly what I've been working on is clearing out the obvious code interpreted as a domain/bifurcated posts where one has feedback but the edit does not. Which I've found easier to do via search.
[ SmokeDetector | MS ] Potentially bad asn for hostname in body, potentially bad ns for domain in body, potentially bad keyword in body, potentially bad keyword in title (4): Công ty TNHH bất động sản Profiland by profiland on serverfault.com
@Undo What types of things are you looking to do? I ask, because I don't want to duplicate effort. Looking at the MS flagging code, specifically preventing us from hitting rate limits on the SE API, of which backoffs are a part, was the next thing on my list of priorities. I'm a little extra frustrated at the moment regarding duplicated and wasted/semi-wasted effort (as I just expended time adding detecting grace period edits to SD, which is now way less of a priority).
I'm happy to have you deal with the MS SE API issues, if it's something you want to handle.
Obviously, if you're looking at infrastructure changes, that's completely your area.
@Undo We're supposed to, but it's unclear how much SE changes them and under what conditions. I have seen differences, at least from post to post. I don't know if they are different from user to user on the same post. It's one of the things I was planning on investigating.
Yes, a significant part of it is that we don't respect backoffs, really at all, or anything else wrt. the requirements of the SE API.
Right now, I'm just getting it so we can at least tell what IP our requests came from. The Lambda environment changes IPs at will, and can have multiple in play when it decides to have concurrent execution.
I believe we also make a lot of requests which we should know in advance that we can't succeed (e.g. we should know if it's possible for a user to flag on a site, but it's not something we track)
If our lambda containers are lasting long enough (I at least think they should), I think we can keep a cache in memory of some useful stuff - flag option IDs if they're stable, etc.
@Makyen Indeed. There may be ways to split requests up between a bunch of Lambdas with different IPs - if nothing else, I can force it by running it in a bunch of regions. Not great because it'd be cross-region access for the DDB queries, but not terrible either
@Undo If they are stable, or not, is something which we will need to determine. It's definitely something the SE API says you should never assume is static, even for a single person over notable time.
The nuclear option, which would definitely work, is to have MS send a "please flag this post with these IDs" request, instead of sending them piecemeal. Then let it run concurrently for each post, respecting backoff. More lambda cost, but probably still in the free tier anyway.
Hoping I don't have to do anything near that kind of complexity.
It's already sent back to MS, and then forwarded to the user who made the manual flag request, so it's already definitely available to MS (and is something I double-checked when I recently touched the lambda code).
My plan had been to build an SE API interface module which handled all of the rate limiting issues and then have the module be switchable between use for requests made from MS and requests made through the lambda (i.e. track everything separately for the two interfaces). I'd expected to build it in a way that it would be easy to expand to N interfaces.
My intent was to store the need for backing off, self rate-limiting, request response caching, etc. in Redis, then respect it in each thread based on the interface which the thread was wanting to use.
Also keep in mind that an actual backoff from the SE API is a per method backoff. It doesn't mean that you need to delay other requests to other methods. OTOH, there are a variety of error responses which indicate you've hit one of the various rate limits, which may happen prior to getting a backoff response, and indicate that you need to pause all accesses. You should also self rate-limit to stay well below the hard limits which the SE API imposes.
@Undo Yes, because we send a lot of those. A substantial issue is to expand our retention and use of the data we get from our scan of a user's sites. Currently, we only use that to determine on what sites a user is a moderator. We should be using that to determine on what sites a user can flag, because we completely ignore that and trust that the user will have only enabled flagging on sites on which they can flag, and that they will manually update that as things change (yeah, right).
So, we send a lot of requests for flagging options for users who have no possibility of being able to flag. Doing so substantially increases the average number of requests we make for every autoflag placed. In some cases, hundreds of requests.
@Undo That's certainly an available option, but doesn't allow for dynamic changes in a user's sites, unless they come back to manually re-enable the site (would be rare). If we do so on a per-site basis, that would be reasonable. If we did it for the user entirely for one issue on one site, then we would probably be down to only being able to use flags from our Core users (i.e. nearly everyone would be disabled)
We could have one field which is "the user has said use these sites" and another field which is a subset which is "these are the ones which the user has said to use where they have an account which could flag".
The second one is generated daily based on the data we already fetch combined with what sites the user has said to use.
The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:06 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/4449899
The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:27 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/702458
The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:42 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/80215
The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:57:09 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/4449832
I'm going to get MS set up to log it, and maybe next week we can hunt down a dev and ask them when it changes.
I mean, it almost can't change over builds either, then you'd blow up cases where the options request was on one build and the flag request on the next
Interestingly, if you read the API docs (1, 2, 3), it doesn't really say the option IDs change, it says the "available flags" change.
We also need to double-check what the responses are to trying to add spam/rude flags under various conditions (e.g. deleted posts). The response to the flag options (only "in need of moderator intervention" option is available) is the only definitive way I recall seeing to know if a post is deleted vs. non-existent.
@cocomac The regex contains an unescaped ".", which should be "\." in most cases. Append -force to the command if you really want to add the pattern you entered.
[ SmokeDetector | MS ] Bad keyword in body, bad keyword with email in body, blacklisted website in body, link at end of body, pattern-matching email in body, +2 more (514): How i become a vampire by Nuraa Idries on gaming.SE
Can I not use FIRE to submit flags on sites where I'm a mod? FIRE doesn't want to raise any flags on SO. The network log indicates an HTTP 500, and flags on other sites are fine
