the SE app has no clue about Teams... :-(

the same is true for the stackexchange.com network homepage and chat

@Andy I do get almost-instant emails that trigger Gmail notifs, though

ah, I have disabled SE inbox emails

!!/report travel.stackexchange.com/a/125677/28333

[ SmokeDetector | MS ] Manually reported answer (94): UAE Tourist Visa by Dubai on travel.SE

tpu- by Erik the Outgolfer

!!/watch- visauae\.net

[ SmokeDetector ] Blacklists reloaded at rev 9ff7228 (SmokeDetector: Auto watch of visauae\.net by Erik the Outgolfer --autopull) (running on iBug/EC2)

[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, potentially bad keyword in title (197): Get ounces of top shelf buds of weed for 90-100 a pop by kojoe on bitcoin.SE

tpu- by Glorfindel

[ SmokeDetector | MS ] Few unique characters in body, no whitespace in body, repeating characters in body (260): Defining and checking for Poker Hands in Python ✏️ by Newby Coder on stackoverflow.com

tp- by Erik the Outgolfer

@Undo/@Andy re: channels post: I don't think you fully understand what I was getting at. The one post isn't the problem. I think the problem is that it's very unlikely that any of the flaggers did enough research to be certain of spam, and if they're not certain they shouldn't be flagging.

I think that in this case this system did not work because flaggers didn't do research before flagging. Sure, spam got deleted, but we want to be designing a process where only spam gets deleted and 0 fps get deleted. This process could've easily led to a fp getting nuked and flying under the radar.

[ SmokeDetector ] SmokeDetector started at rev 9ff7228 (SmokeDetector: Auto watch of visauae\.net by Erik the Outgolfer --autopull) (running on iBug/EC2, Python 3.6.6)

Restart: API quota is 14321.

Also, finding evidence of nuked fps is really really hard because nothing really gets reviewed after it gets spam deleted. The fact that stuff that's very close like this exists implies pretty strongly (IMO) that there are very similar fps getting nuked.

I'm defining "worked" as "spam got nuked, not-spam did not get nuked"

Not the preferred path to get there, but still the correct outcome.

[ SmokeDetector | MS ] Toxic body detected (69): Why do people on here suck by Sad on math.SE

tpu- by K.Dᴀᴠɪs

Totally understand the concern. It's real. Telling people "do your research" probably isn't the solution; we already do that. It's hard to solve a problem when you can't quantify it.

@Undo Sure, but could've just as easily been wrong. I'd still think it was a fp if Art hadn't spent 20 mins digging up the initials of the founder of the company.

So... have we asked the flaggers yet?

Well, I can't really expose that kind of mod-only info, so no

I can't control flags; I can control feedback.

@Undo Also, most of the answers to the channels question seem to say "Everything works, that was a close one but everything is still OK" which just makes me kinda nervous that nothing will change. And I think something needs to change because I think something went wrong here.

probably post an answer then?

I understand that, but I also understand that we can't change stuff without knowing what the problem is.

Manual flags cast through MS by Tetsuya Yamamoto, Makyen, tripleee.

And right now, I don't know what the problem is.

I know the symptoms, but not the cause.

I think you hit the cause in your answer, @Undo. We see enough spam that people are biased towards assuming that borderline reports are spam rather than not.

@TetsuyaYamamoto @Makyen @tripleee Any recollection of thought process for feedback on metasmoke.erwaysoftware.com/post/142684? I know it's a long time ago, and sorry if I'm not the first to ask.

@ArtOfCode I think it's likely, but I'd like to see more than one case that somehow managed to get handled right.

@Undo I think the problem is that people are too trigger happy with flags rather than giving posters the benefit of the doubt. How to fix that, I have no idea.

@Undo Tetsuya Yamamoto's flag actually failed ("No account on this site"), but there was an attempt to cast.

@thesecretmaster Audits have been proposed before, but holy crap the complexity in so many dimensions

I'm not doing audits

yuck, audits

I do think that the best way to get people to not do something is to make another path easier.

"um" would do that

@Undo Finding multiple examples of stuff like this is hard because once it's gone, it's very gone and doesn't really get eyes on it outside of charcoal. I think that this instance pretty clearly points out that a problem exists.

[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer, repeating characters in answer (263): I didn't get the Pacifist ending, what went wrong? by user221113 on gaming.SE

tpu- by K.Dᴀᴠɪs

I kinda like the idea of the auto-mod-flag on um posts that got deleted. You'd have to be careful about how you word it (it needs to be dynamic based on actual data, not constant for every occurrence), and you'd need to link to some sort of "WTF is this" page for mods, but I think it could work.

@Undo Yeah, audits don't seem like a good solution. I think it's a people problem that kinda needs a people solution.

@thesecretmaster I tend to agree, but we've been doing this for literally years and I'm not aware of a case where we falsely nuked something that stayed that way undetected for a period of time

Seems like someone would have found something like that by now

@ArtOfCode I'd be worried about the volume of such flags

I'm one of the people that provided tpu- feedback and flagged. While I vaguely recall this report and responding to it, I can definitely say that I did not do enough research. If I had done enough research A) I'd remember, and B) there would be an MS comment with the results. So, yes, I agree that not enough research was done on this report. Ironically, I agree that we sometimes do not give people the benefit of the doubt, and usually try to urge people to do so. I clearly did not here.

@Undo And that's even after at least a couple big MS feedback re-review pushes

@JohnDvorak Only on "um" posts, and only on those that (a) had at least one manual flag cast through MS and (b) actually got deleted, should be reasonable volume.

except that "um" posts are probably not going to be just a little few

@Undo I think that's a bit of confirmation bias though -- the only people who are looking at these posts are Charcoal people.

Relatedly: making things easier is absolutely the way we have to do this. MS review is currently standing at 300+ items because it's not frictionless enough to do.

@thesecretmaster Point, but counterpoint: We also survived the autoflagging debates without a found case of "you were wrong here" that wasn't caught early by us.

And that was when very competent non-Charcoal people were motivated to review stuff (which they definitely were)

@Undo related point: I know there were people going through archives looking for those cases

I very very much know it

In short, my gut says there has to be those cases... but my mind says "where the heck are they?"

I'm not sure how to tilt the bias to surface something I'm not even sure exists.

@thesecretmaster I know I'm 20-30 minutes late, but this seems to be assuming bad faith on the part of flaggers. You don't know if they did research, you don't know if they just used experience.

@Andy See Makyen's comment above; we do have at least one point of "I didn't do enough research"

@Undo Conflicting feedback is probably a good place to start. Lemme see if I can query for deleted conflicted posts.

... of course, that's one more user who will probably do enough research the next time. And on our scales, one user doing research tips the scales in our favor heavily for the future.

@ArtOfCode conflicted posts deleted within n minutes of our report, perhaps

probably not that big a difference, but could rule out "we deleted it" from "the community thought it was crap anyway and deleted it"

Most of those will no longer be conflicted.

Another maybe-interesting thing: Retracted tp feedback on deleted posts

I'd also filter for reasonably low score

Vote score?

fwiw, I'm reasonably confident in deletion timings now. Comfortable using them for this kind of thing.

@Undo reason weight, sorry, used the wrong words

I'd be more concerned about the high score ones, actually. I know that my critical thinking level goes down when I see a 200+ score report.

A <150ish weight post tends to make me skeptical upfront, raising the chance that I'll catch an FP

I do know that there have been a few posts reported here that are on SO which have ended up deleted where I didn't feel a spam-delete was appropriate. Where I've seen this, I've custom mod-flagged to have the spam flags disputed, or the post undeleted. I'd have to check my flag history to find specific examples.

I just know that the post that started this was ~70, just caught because of a link at the end.

Link at end is creepily accurate for how simple it is

> (572) FORSKOLIN ... [whoops, turned out to be a question about preventing forskolin spam]

Marketing tactic, whether it's deliberate or subconscious. Make it the last thing you look at so you're more likely to click it. Same thing with repeated link at end of long post.

@Undo Deletion timings will only get worse with this discussion. Research to confirm these suspected cases will cause a delay, as will 'um'. This research/due diligence should already exist and as of yet, I haven't seen a reason it failed here. Spam was flagged. Whatever the process users took to do so, they did not do anything wrong here. They flagged spam. Sure, they should be confident in their flags, but...this ended up being spam.

@Andy Talking about using historical deletion timings for analysis on this

The core question here is "are there FPs we're nuking that we don't know about" and "how do we find them"

I know you mentioned 'naa' in your comment, but I think there is a difference between 'naa' and 'um'. 'naa' has a specific meaning. 'um' triggers gestures to the channel this conversation.

Both are valid, if lacking evidence.

@Andy I do agree with @thesecretmaster that this is a bit of a problem. In this case it was right, but in another case, without the research and/or greater certainty in the flag decision, it could be wrong.

@ArtOfCode I think Andy's point might be that "might be wrong" is kinda irrelevant in a scenario where we've ran hundreds of thousands of posts and the "might" never happened

Yes, it could be. But, we haven't been provided with those cases. We are chasing a hypothetical "could happen"

"might" that never happens is "impossible"

@Andy What I'm saying is that this exact same process could very easily have led to a fp being deleted. By chance this was actually a tp, but if Art hadn't done ~20 mins of research then I'd still be convinced it was an fp.

But does it never happen?

Agreed, which is why I want to look at querying for anywhere that did happen

The null hypothesis here is "we are doing good and nothing is terribly wrong"

The not-null hypothesis is "we're missing stuff and we don't know about it"

It's a pretty easy null hypothesis to reject. Just need a few cases that we can agree should have been caught.

This is sorta spammy because it looks very similar to the previous answer, but possibly an fp. No proven affiliation and the link is relevant to the question.

@thesecretmaster In that case, you'd be doing your job as a moderator. Reject the flags if it's appropriate. Bring it to our attention if you like us ( :) ), but 20 minutes on a spam flag is an unreasonable amount of time to spend. But, this is a very edge case. We either say "whoops, our bad" and adjust what we need to or we do the 20 minutes of research for you and reflag it.

un momento, need to switch machines

@Andy It's a scary edge case, though. I do want to know if we're missing these.

Hm, the laptop didn't shut down. I'll have to check that later. Anywho, back.

@thesecretmaster hm, I remember that my thoughts were "looks like the answer seems valid, but the link contradicts and points to something irrelevant, i.e. whole laptops instead of sound cards, and that's definitely not what was asked here, it's very unlikely the answerer did a mistake, the difference is clear"

@Undo Agreed. But, as you said earlier, we've under gone several waves of scrutiny. I would want to know, but I suspect it wouldn't be a technical fix we'd put in place to eliminate a single mistake. We'd do exactly what we are doing now. Either agree or disagree with the call and figure out how we go it wrong. A mistake will bring more scrutiny and we'd have the post mortem in progress/done shortly.

Anyway, my entire point is, we got this call right. Yes, the methodology is a little suspect on this specific post. Secret makes a valid point that it could have gone either way. We need to be careful. I fully support reminding everyone to be more careful. I don't think a technical solution is needed right now. I am open to changing that opinion if data proves otherwise.

@Andy I know that here in CHQ, we're very data driven, but I think that in this case even just with this one datapoint we can clearly see that a problem exists. I think that saying "no action without data" here is just a way of pretending the problem doesn't exist. However, when I get some time this weekend, I can try and actually find more data, although I don't know how successful I'll be. But I think we shouldn't let a lack of data make us avoid improving the system.

I'm not saying no action without data.

@Undo I'm mostly interested in the "are you comfortable..." question. I think we do a good job of accepting people who want to put on the brakes for a report, but I'm not exactly in a vulnerable position here.

@thesecretmaster I don't think it's "no action without data", it's "we don't even know what action to take without data"

@Undo I think that we know what the problem is and we don't need data to help clarify the problem. However, it's just a hard problem to fix.

@Undo I think I concur with Makyen, though I'll also remark that this is fairly similar to what we have tagged as #blogspam, i.e. very low quality posts generated from a template like "very nice post, keep up the good work, spam.example/";

@thesecretmaster I think that Undo summed it up pretty well by saying that we see enough spam that we assume spam when it's borderline.

@thesecretmaster From your perspective, "people are trusting enough to flag without doing ("enough") research" is the "problem", right?

Because I'd agree that it's a problem.

@Undo Yeah, just about. TL;DR people are too trigger-happy sometimes.

The part I don't have is information on why, how often, or what impact it has when it does happen

That's the data that I think we need to take any kind of action.

And the part that makes it hard is the mountains of evidence saying "somehow the chaos always works out". Only need a few counterexamples to make me say "huh, apparently that's not true"... but we don't have those.

Which is spooky in itself. After a hundred thousand posts we should have gotten something wrong in a bad way.

@thesecretmaster My thinking on that one was very similar to what @EriktheOutgolfer has detailed. I was also biased by the fact that "best laptop" type spam is moderately common with very similar types of posts (i.e. short post with go look at this "review"/"comparison").

@Undo Well, data wouldn't help with the why, but for the rest, digging through data is hard. It really seems to me like the data must exist and just that finding it is really hard. That's just my instinct. And @Makyen said earlier that they thought they possibly remembered some on SO.

@thesecretmaster Same instinct here. Re: data being hard, you should be able to bound the problem. Say that you think this is happening to one out of every x posts, take an n-month window, calculate probability of it happening and not showing up in that window

Then review the posts in that window with whatever methodology we come up with

mods deleting stuff for other reasons often masks the iffy cases, and tends to look like "we did the right thing after all"

@thesecretmaster I should point out that I also have flagged some posts deleted by moderators and suggested that the post was deleted inappropriately. Everyone makes mistakes.

(flagged and the post was undeleted)

let's agree on a tag and add it to posts which should be reviewed as interesting for this investigation

my informal hunch is we are going to have about half a dozen in a month, maybe more

Teams post, anyone?

There's already one teams post related to this discussion: stackoverflow.com/c/charcoal/q/470/18

^ good one, thanks for the pointer

The part that makes this hard is that anything we find in real time is, by definition, not part of this problem

Have to go spelunking to match the "and didn't correct it in a timely manner" criteria

How many posts per month do we get?

but I mean specifically "please tag posts matching criteria A, B, or C with #hmm-interesting" coordination

@tripleee For those that are "interesting", perhaps in that ballpark. What do we consider "interesting" for this problem? For those that are actually FP which are deleted, I'd guess at much lower than that.

@thesecretmaster Because I'm already here...

Sat, 01 Aug 2015 => 121,
Tue, 01 Sep 2015 => 3451,
Thu, 01 Oct 2015 => 3004,
Sun, 01 Nov 2015 => 2670,
Tue, 01 Dec 2015 => 2873,
Fri, 01 Jan 2016 => 3399,
Mon, 01 Feb 2016 => 3379,
Tue, 01 Mar 2016 => 3662,
Fri, 01 Apr 2016 => 3497,
Sun, 01 May 2016 => 3052,
Wed, 01 Jun 2016 => 2934,
Fri, 01 Jul 2016 => 3100,
Mon, 01 Aug 2016 => 3043,
Thu, 01 Sep 2016 => 3184,
Sat, 01 Oct 2016 => 3027,
Tue, 01 Nov 2016 => 3261,
Thu, 01 Dec 2016 => 3405,
Sun, 01 Jan 2017 => 3658,
Wed, 01 Feb 2017 => 3538,
Wed, 01 Mar 2017 => 3500,

@thesecretmaster around 100 per day on weekdays, weekends slightly less

750-1200/wk

So reviewing a month-worth would be about 4k posts.. which is a lot. We could filter out the punctuation ones and the ones which have domains that occur in > 3 posts, and could prolly filter a bit further.

Looking at only TPs gets you to ~3k/mo

2800ish

the vast, vast majority is slam-dunk #drugs

but yeah, reviewing the rest is a thankless task

2200/mo for TPs w/ a recorded deletion date

@Undo take out repeated chars, non-latin, etc

but I would guess maybe down to 200-300 with very modest filtering

@tripleee Don't most of those have domains with 3+ matching posts?

We could knock those out

gotta run, but AR query I'm using if someone wants to pick it up:

ap Post.group_by_month(:created_at).where(is_tp: true).where('deleted_at IS NOT NULL').count

sigh fine

@thesecretmaster Actually, we do get FP on those reasons.

@Makyen But the TPs on those reasons should be slam dunks

now actually leaving

most yeah but add in the product names they use and you can filter out nearly all of the rest too

@Undo Yes/no. We're looking for cases that have been evaluated TP that are actually FP, thus if we get FP on some of them, then we should look at them.

somebody was saying tags aren't very useful but this is precisely why we have them, you can group swaths of posts for exclusion or inclusion easily

[ SmokeDetector | MS ] Potentially bad keyword in body (1): How can I always know what marketable skills are on demand? by Knight OfNights on stackoverflow.com

fp- by K.Dᴀᴠɪs

the ones without obvious keywords but an obvious theme or template like #hacking or #spellcaster woud be hard to filter out otherwise

and #blogspam, with variations

but careful, that's more likely to have FPs

sadly there are probably still many untagged #hacking and #spellcaster posts, and some clear recurring categories which don't yet have a tag

like the TOEFL ones I looked at the other day

@Undo The wording of "Subjectively, do you think there are FP posts slipping through the cracks?" doesn't make it clear what you're asking. You are probably intending something like "Subjectively, do you think there are FP posts which get deleted by spam/R/A flags as a result of SmokeDetector reports where the issue is not caught and rectified in a timely manner?"

I gotta run, but I'll be back in the next couple days.

@Makyen I had to reread that question as well.