That's true, but then why record NAA at all? We don't flag them, and the ability to be able to do so through MS is status-declined (which I understand the reasoning for, but it would be convenient for users). What is the value in our capturing that information, other than users are more comfortable with being able to differentiate? There's value in differentiating so users can cast NAA flags, getting some benefit from our having people look at them (assuming they do NAA flag). Is there more?
You can monitor it with the invalidated feedbacks log (tools dropdown, top section). It's going to do a hell of a lot of invalidating tonight.
TL;DR: Every day at 0200Z, it'll run through posts, find any where the feedback is conflicted but one type significantly outweighs the others, and take the majority verdict as truth.
@ArtOfCode I agree that the comfort factor is significant. It does feel weird to say something that shouldn't be on the site is a FP, but we do that with questions, because we can't give NAA feedback to them.
@Makyen I vaguely recall making it possible to send NAA feedback on questions in prelim work for adding a VLQ feedback... but I don't know if that's something I actually did or just something I planned on doing
I did some science on the numbers - there are around 2.2k conflicts, around 1.3k have one type outweighing others by 1, around 400 where it outweighs by 2, and around 200 where it outweighs by 3
soooo there's probably about a thousand feedbacks total on the chopping block for tonight
Now I want to see a list of conflicted posts that don't meet that criteria, so that more feedback can be added :-). It'd be nice if those could show up in review, but I'd want them to include the comments, which last I checked are not shown in review. Including the comments, because, at this point, people will reasonably often include in a comment the additional information that they researched, some of which may not be available to non-moderators, those with <10k on the site, etc.
@Makyen This is kinda maybe preliminary to making a review queue for conflicted feedback. I didn't want to throw another 2.2k items in review because we'll never get them done, but if we can automatically resolve some of them... maybe
@ArtOfCode That kinda sounds like a bad idea -- it'll make accuracy stats look better than they actually are and it'll also bring up peoples invalidated feedback %. I don't love the idea of rewriting history by invalidating them.
Does it clean all is_tp/is_fp/is_naa based on the feedbacks that actually exist? There have been some posts where it's appeared that those don't match the existing feedback (e.g. posts listed under NAA on MS which have only TP feedback).
@thesecretmaster Not really. This is all stuff that could've happened manually anyway if someone had the time to look through the list - if I went through those 400 posts manually, it'd be exactly the same result on invalidated feedbacks.
I'm just making the computers do it for me because I am a god of sentient sand
Rails has it's cacheing thing. You load a page, it takes time to run queries, it caches bits of the page, the next time you load it, it shows you the cached version. When the timer on the cache expires, you have another long load while it recaches stuff. I want to make those cache-updates async so you never have long loads while the cache is being populated.
@thesecretmaster that's... how most caching works. But in principle, no objection to that. If you're working on that you could also use that caching thing I mentioned a while ago
@thesecretmaster Ahh, but that implies you might get significantly out of date information, just not have that data loaded on the page (even when that's the primary info), and/or have quite a lot of speculative updating.
@ArtOfCode Hmmm... I'd suggest that it not do anything if there are comments on the post. Comments could indicate that someone later found some additional information which makes the later feedback more appropriate. Basically, probably a good idea to kick those out and let a human deal with them. Alternately, you could look at the time the comment was placed relative to when the feedback was placed & go ahead with the change if the majority feedback was after the comment.
@ArtOfCode hmm... not sure if this is the right idea. When I go through posts with conflicted feedback, I'd say that about 1 or 2 in 10 I would go against the majority
@ArtOfCode ....okay. I've just gone through all of the elegible posts. (it looks like there's some sort of bug in the job? Metasmoke froze for a minute roughly around 0205, and nothing appears in the invalidated feedback logs). Looks like you were right - out of all of the posts that had at least 2 more of one feedback than the other, there was only 1 post where the automated action would have been incorrect.
When you look at posts with only a difference of 1 feedback, it's a totally different story - many of these required human review.
So 2 feedbacks looks like a good threshold for now.
Side note: should we add a delay before conflicted feedbacks are resolved? i.e. each day only resolve the posts which are more than 24/48 hours old, so that there is at minimum a 24 hour window for people to get in and add more feedbacks
Also, I think there's a very easy way to resolve about half of all the conflicted feedbacks. Some time ago, when we were upping the review requirement to 2 feedbacks per post, we ran a script to duplicate the feedbacks on old posts. However, this script had a bug, and it duplicated invalidated feedbacks as well, causing the post to become conflicted again.
If we can pinpoint when this started being an issue, then we can come up with another script to invalidate them again
So... we'd be invalidating the invalid feedbacks on the conflicted post which became conflicted by invalidated feedbacks being duplicated from when it was previously conflicted
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, body starts with title and ends in url, pattern-matching website in body, potentially bad keyword in body, +1 more: Jovian South Korea by rebeccaerri on graphicdesign.SE
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, pattern-matching product name in body, pattern-matching product name in title, pattern-matching website in body, +3 more: Premium Pure Keto by Jfjbxjxj Nxjc on askubuntu.com
[ SmokeDetector | MS ] Blacklisted website in body, link at end of body, pattern-matching product name in body, pattern-matching product name in title, pattern-matching website in body, +3 more: Premium Pure KetoPremium Pure Keto by liikiwedrrdee on askubuntu.com
> Potentially bad keyword in body, potentially bad keyword in title, potentially bad keyword in username ---------- Title - Position 1-18: zerobugacademy.com Body - Position 1-18: zerobugacademy.com Username - Position 1-18: zerobugacademy.com
[ SmokeDetector | MS ] Url in title, pattern-matching product name in body, potentially bad ns for domain in body, potentially bad ns for domain in title, potentially bad keyword in body, +1 more: www.aristocratholding.com/perfect-keto-max/ by Fowan98 on rpg.SE
> Potentially bad ns for domain in body, potentially bad ns for domain in title, potentially bad keyword in body, potentially bad keyword in title, potentially bad keyword in username ---------- Title - Position 1-21: aristocratholding.com Title - aristocratholding.com NS suspicious dns1.namecheaphosting.com.,dns2.namecheaphosting.com. Body - Position 1-21: aristocratholding.com Body - aristocratholding.com NS suspicious dns1.namecheaphosting.com.,dns2.namecheaphosting.com. Username - Position 1-21: aristocratholding.com
@Makyen That pattern looks like it's already caught by Pattern-matching website in body and Pattern-matching website in answer; append -force if you really want to do that.
@Makyen Added applemacsupportnumbers\.com to watchlist
CI on 4d2db2c succeeded. Message contains 'autopull', pulling...
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, body starts with title and ends in url, link at end of body, +3 more: enduraflex : Considering no longer by Frorning on askubuntu.com
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, body starts with title and ends in url, link at end of body, pattern-matching website in body, +1 more: Complement targets your persistent by samoniguk on meta.SE
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, body starts with title and ends in url, link at end of body, +1 more: I like to use th Le Reviva by Ineackmx on superuser.com
@Makyen That pattern looks like it's already caught by Mostly non-latin body and Mostly non-latin answer; append -force if you really want to do that.
[ SmokeDetector | MS ] Url in title, blacklisted website in body, blacklisted website in title, link at end of body, pattern-matching website in body, +3 more: funmuscle.com/stimulus-rx/ by Conch1960 on drupal.SE
[ SmokeDetector | MS ] Link at end of answer, non-latin link in answer, pattern-matching website in answer, potentially bad ns for domain in answer, potentially bad keyword in answer: Get interresting fields from firebase by Ramin on stackoverflow.com
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, link at end of body, pattern-matching website in body: Stimulus RX health by VXCVCXVXCV on askubuntu.com
@Makyen That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; append -force if you really want to do that.
[ SmokeDetector | MS ] Url-only title, blacklisted website in body, blacklisted website in title, link at end of body, pattern-matching website in body, +3 more: funmuscle.com/stimulus-rx/ by Steedarly on askubuntu.com
Can someone with 10k on SO tell me how this got deleted? My NAA flag got declined as "no evidence found" so I'm not sure if that was supposed to be some sort of hint they thought it was spam: stackoverflow.com/a/51907071
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching product name in body, +4 more: Keto X Factor99 by EttaR Singer on security.SE
[ SmokeDetector | MS ] Body starts with title and ends in url, link at end of body, pattern-matching product name in body, pattern-matching product name in title, pattern-matching website in body, +2 more: feel like super motivated Premium pure keto by Amaslmax on superuser.com
tpu- by PeterJ
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, body starts with title and ends in url, link at end of body, pattern-matching product name in body, +2 more: These preservatives are often added by user399915 on meta.SE
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, body starts with title and ends in url, link at end of body, pattern-matching product name in body, +3 more: Plus Keto After getting disappointed by Unting193 on meta.SE
[ SmokeDetector | MS ] Url in title, blacklisted website in body, blacklisted website in title, potentially bad ns for domain in body, potentially bad ns for domain in title, +1 more: miapiellealpha.com/testrox-ultra/ by Bntak34 on rpg.SE
