I'm not sure, yet. There's no way to reliably undo it once done (i.e. we can't know how many encodings SE applied vs how many existed in the original. Setting "unsafe" on the filter we pass to SE might give us good data (untested, yet), but that's not clear. It would definitely be a change to the data we receive. It's also not clear what transformations SE actually does with "safe" vs "unsafe". So, it may run us into other issues, potentially in SD tests, on MS, in FIRE, etc.
The issue that I've seen is that we appear to be re-scanning posts multiple times. We should be doing so, but we only do so if the title, username, and post body aren't the same as the last time we scanned. The problem, from our point of view, is that we appear to be getting multiple different responses for what is supposed to be the same data, but isn't, so we re-scan it.
Thus, we are almost certainly scanning the same post, that's supposed to be the same content, but isn't, multiple times.
@CesarM did I mention that this has been a problem for several months now so the last time a major API change happened started causing this has been turning one of my machines into a portable heater :P
@ThomasWard My problem in the past has been reproducing the issue. I'd tried several times, and many times after seeing that GitHub comment, but until @MartijnPieters also experienced it and asked about it in chat I hadn't seen it reproduced.
@Makyen how hard would it be for us to demultiply escapes and then compare content? That would prevent the same post being rescanned - sure it alters content for SD but it's a potential option if the issue is systemic to the SE API as it appears to be
@ThomasWard There's no way to do it reliably, as we can't know how many times it's been done, so can't determine when it's actually the real post content, which could include things that are escaped. We could repeat the unescape process until there are no changes, but then we can't know if we're testing the real content, or not. We'd also have to choose if we're going to test the multiply unescaped text or the original. If we test the original of the first version we see and not any other version, how do we know that wasn't a corrupted version?
One benefit of testing multiple times is that we probably test the real content at least once. :(
@CesarM I appreciate the offer. I'm not coming up with anything that's chasing down information related. If you could encourage prioritizing fixing the issue, that would help. We're not certain this is the problem which is causing our CPU load issue, but it's a very likely candidate (i.e. if it's doing what we think it's doing, it would cause what we're seeing for CPU loading, along with, potentially, the failure to detect some posts).
also when you find out who made the commit that busted things with the infinite duplicate problem for escapes smack them with this here old as sin moldy pool noodle on my behalf. hands The Weapon to @CesarM
Prior to seeing the encoding issue, and that it's substantially variable as to when it happens, I'd been assuming that the issue was that the SE API servers where not maintaining as good a sync of data as they previously were. In other words, that at times we'd get different data for the same post on subsequent requests when the response happened to be from a particular one, or two, of the SE API servers, then get the "correct" data upon the next request.
If that was what was happening, and it has happened in the past, at least for data such as remaining quota, then we'd see something similar on our end regarding CPU loading.
My expectation had been that I'd need to build both mitigation on our side and the tools to look more closely as to what was really going on. That hasn't happened, yet.
[ SmokeDetector | MS ] Bad keyword in body, bad keyword with email in body, bad phone number in body, pattern-matching email in body, potentially bad keyword in body (324): HACK ATM WITH EASE by Henderson Conor Sean on serverfault.com
[ SmokeDetector | MS ] Bad ip for hostname in body, blacklisted website in body, link at beginning of body, potentially bad ns for domain in body (239): what is Java Assignment by Dwayne Smith on superuser.com
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, body starts with title and ends in url, pattern-matching product name in body, pattern-matching product name in title, +2 more (581): Aizen Power Male Enhancement by elyana aly on 3dprinting.SE
@tripleee That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer Append -force to the command word(s) if you really want to add the pattern you provided.
@tripleee That pattern looks like it's already caught by Blacklisted website in body and Blacklisted website in answer Append -force to the command word(s) if you really want to add the pattern you provided.
@JeffSchaller That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer Append -force to the command word(s) if you really want to add the pattern you provided.
@JeffSchaller That pattern looks like it's already caught by Bad keyword in username Append -force to the command word(s) if you really want to add the pattern you provided.
@SmokeDetector weird word salad in multiple languages. Chinese: "<Guo Fan> <Penguin Q> <25507334>「—cszc3.com—网」「Invitation code: 0989-0009」【top-tip-name-teacher丨letter.yu.ping .Taiwan丨Purchasingpurchasingfirst choice丨one.pair.one丨whole.day.stubble.line.counseling.guidance]" ...
@SmokeDetector ... Japanese: "Akamai Technologies, Inc. (Akamai), a cloud services provider responsible for supporting and protecting your online life, recently launched Akamai Connected Cloud, a cloud computing, security and content delivery service. A large distributed edge and cloud platform. according to the report" Korean: "Founder and CEO Dr. “With 25 years of experience, we are very knowledgeable about the cloud,” said Tom Leighton." ...
@SmokeDetector ... Russian: "local media, Li Sheng, Akamai Regional Vice President and General Manager of Greater China, said that Akamai's business is fundamentally"
