@tripleee That pattern looks like it's already caught by Bad ns for domain in answer and Bad ns for domain in body; append -force if you really want to do that.
@Makyen Just a thought, would making AIM make fewer requests be a relatively easy change? On page load it makes 40+ and that's just a big number. It doesn't cause any problems, but seems like something that could be simplified.
@thesecretmaster Not an easy change. However, it's on track to be done. The plan is to move to GraphQL and make one request. Eventually, the plan is to share data between FIRE and AIM, so they are not duplicating requests, and both have more up-to-date data.
@thesecretmaster Not that I'm aware of at the moment. I'd experimented enough to get to the GraphQL request I expect to be making. I'd been moving towards integrating it in FIRE, but there are still underlying changes wrt. data organization in the script that need to be worked on.
The reduction in requests from FIRE is actually about an order of magnitude higher then for AIM, but the version that collects and actively uses that information hasn't been released, due to my expectation that it would be too high of a load in its current state.I'd planned on doing AIM after, but if it's beter to do AIM first, I can do so.
I had looked at reducing the number of requests in AIM by requesting more posts per request, but the timeouts needed to accomplish that seemed excessive, particularly given that MS has a quite limited number of threads available and uses one per request.
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, link at end of body, link following arrow in body, pattern-matching product name in body, +3 more (590): Where To Buy Keto Pure Canada by darkwalter on superuser.com
@CalvT That pattern looks like it's already caught by Bad ns for domain in answer and Bad ns for domain in body; append -force if you really want to do that.
@double-beep That pattern looks like it's already caught by Pattern-matching website in answer and Pattern-matching website in body; append -force if you really want to do that.
@double-beep That pattern looks like it's already caught by Pattern-matching website in answer and Pattern-matching website in body; append -force if you really want to do that.
@double-beep That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; append -force if you really want to do that.
@double-beep I'm not sure what you want me to review it for. It gets a more FP than we like, but also has a few/several posts where it's the only detection. Thus, it's definitely desirable. It can probably be improved to reduce FP, but doing so would take looking at it in more detail. The go-to change of only detecting when it wasn't in <code> resulted in missing some TP, so I'd need to take a closer look at it.
@double-beep We definitely want it to be in the watchlist. While it does have quite a few FP, that it's the only detection on a number of TP is quite important. One of our goals is to report all TP, even if that means we wade through more FP. Sure, there's got to be some limit to the number of FP we'll accept per TP, but usually that's resolved by refining the detection, so it more accurately picks-up the TP and excludes the FP.
@thesecretmaster My testing indicated getting repeated timeout errors (using the browser's default 30s timeout) when I increased the number of posts requested by AIM to between 20 and 30 posts/GET API call.
@thesecretmaster I'd normally expect rate-limiting to just error-out the request, not make the request take a long time. When I reduced the number of posts/request (thus increasing the total number of requests), the requests received data back within the 30s timeout and none errored-out. My reasoning when making the choice of either increasing the timeout or making more requests was that it was better not to hog a thread on MS. My impression from doing searches was that API calls consume a thread.
@Makyen Yeah, any HTTP request will consume a thread for the duration that it runs for. I would be very surprised if there was any request you could send that'd take a full 30s. I know that the GET requests that AIM and FIRE make now complete in sub-ms time.
@thesecretmaster That sounds like what I expected prior to testing. I'd initially increased the number of posts/request in AIM to request every post on the Chat page. I found that this produced errors when the number of posts requested was > 100, so I chunked the requests to request max 100 per overall request. I then beefed-up the number of posts requested per result page. This always timed-out the browser's default 30s timeout for 100 posts/page.
Dropping it to 30 posts/page usually timed-out. Dropping to 20 posts/page usually succeeded. In order to have almost always succeeded, I left it at 10 posts/API result page (i.e. per API call).
@thesecretmaster No, neither make any requests to search endpoints.
@ArtOfCode I'm kind-of in the middle of tracking something else down at the moment, so it may be a bit. The testing I did was at least a month ago, but I think it was this year. My testing environment has also changed significantly, but I can probably resurrect that config if needed.
@ArtOfCode The hard error (i.e. some number > 100 posts) was due to the URI getting too long ("414 Request-URI Too Large") for requesting all the posts related to the SD reports in The Fire Department. This only happened on occasion, depending on what the IDs are for the posts in the room and how many happen to get shown.
You can duplicate the above error with AIM by going to The Fire Department, requesting "load older messages" a couple (few?) times, then deleting all the AIM info in the DOM with $('.ai-information').remove(); and clicking "load older messages" again. I encountered it upon a normal page-load (transcript?), but it's relatively rare. Chunking the request was an easy thing to do to avoid it.
It may also have only been a non-contrived issue on a transcript page. AIM being functional on transcript pages is something I haven't pushed into the released version, due to concerns over MS load. My hope/plan was to do that at the point of switching to GraphQL.
General slow issue: I've briefly tried some requests to MS, but they all appear to be reasonably quick (maxing out at 4.5s, or so), at the moment. I'd expect the issue I saw to be quite load dependent. There have also been significant changes that have reduced the MS API load on MS (e.g. blocking regex search routes) since I did the testing I reported above (and this is the quite time of day). Thus, that testing and testing now may not reflect the overall current state of things on MS.
If you want, I can turn the per_page value up to 100 in my normal use and watch for issues.
@tripleee That pattern looks like it's already caught by Potentially bad keyword in answer, Blacklisted website in answer, Blacklisted website in body, and Potentially bad keyword in body; append -force if you really want to do that.
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching product name in body, +6 more (791): Keto Slim Weight Loss Pills by blackmon1 on workplace.SE
@tripleee That pattern looks like it's already caught by Blacklisted website in answer and Blacklisted website in body; append -force if you really want to do that.
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching product name in body, +6 more (791): Keto Slim Weight Loss Program by mickieakins1 on askubuntu.com
@SmokeDetector I suggested an edit to replace the copied legitmate phone numbers in the post with a link to the respective contact page on the official IRS website.
@Undo I may have already asked this, but could you give me the output of passenger-status --show=requests (look before you send, not sure if it's all safe to share)
