Charcoal HQ

8:00 PM

I guess I assume that everyone who trawls through meta and has six digit rep on a site is invested in seeing it be successful, so you can assume good faith

Undo

@Andy Ah, yep.

Should be pretty easily dissuaded with some hard numbers on human accuracy, Smokey accuracy, and post volume.

It'd look like "this raises the expected number of false positives nuked from 0.01 to 0.05 per year"

Magisch

At the point where we're at it's unsafer to let 2 humans flag it rather then 1

because our flags are more accurate then human flags, by a large margin

Glorfindel

That's what I was thinking as well.

SmokeDetector

tpu by DJMcMayhem on official 1800 921 4167 quickbooks enterprise phone number [MS]

Magisch

If we're talking 99,75% (where we're at right now) comparing to the 95,7% number of human flags, human flags are 17,2 times more likely to be FP's (according to my napkin math)

Catija

8:02 PM

Maybe I missed this somewhere... but is it worth pointing out... one way or another... that unless someone is sitting on the site and sees the "x new post" banner, most people don't see a post within five minutes of it being posted because the system has a delay built in?

DJMcMayhem

@Cerbrus That's all fine and dandy, but I have no idea how to tell what the accuracy is given a certain weight

JAD

If someone has as argument that using something that has already been automated is a waste of time, I doubt throwing numbers at them will help.

Mithrandir

This advice is worth reiterating:

Mar 2 at 17:42, by Undo

Charcoal folks - stay away from that meta post. Let BI figure it out for itself.

Undo

@Catija Pretty sure the home page doesn't have a five minute delay

Cache, yes, but not five minutes.

John Dvorak

What's SFF's spam volume?

Catija

8:08 PM

@Undo It does. If you see one of those "x new posts" popups, load it, and then refresh the page, the post often disappears.

Andy

@Magisch Please...he's baiting you.

angussidney

ANyone got a handy link to the post for me? (On mobile and can’t see starboard)

Undo

mother meta post?

Mithrandir

@angussidney Which post?

angussidney

Are there’s more now?

Glorfindel

8:08 PM

1

Q: Do we want Smoke Detector to auto flag spam?

As some of you may have seen already there is this post on mother meta. To save a bad explanation by me here's the TL;DR from the post. Charcoal is the organisation behind SmokeDetector. Since January 2017, we've been casting up to 3 flags automatically on posts that our systems are confident...

Magisch

@Andy I think you're being a bit too cynical

Undo

Here's a big benefit for autoflagging: If you have n autoflags, you always need 6-n manual flags to nuke. That means 6-n human eyes on this junk, at minimum. Timing or not, you can't get around 6-n humans looking at this.

Magisch

We can't assume everyone can read metasmoke and knows the relevant stats

Undo

@Magisch At this point you've got two SO mods saying he's baiting you. We know how to recognize this.

Magisch

alright

whatever happened to me being the cynical one

:/

Undo

8:10 PM

Understand ;)

Magisch

@Undo I've been making an effort lately to not be so cynical because I was seriously losing faith in the voting system with all the serial voting

Undo

ugh, voting

FWIW, outside of SO it's not that bad.

Magisch

You can only see so many people blatantly cheating the assoc bonus and getting away with it

I know

Andy

Speaking of voting...

2

Q: 2018 Moderator Election Q&A - Question Collection

Stack Overflow is scheduled for its tenth election next week, March 12th. In connection with that, we will be holding a Q&A with the candidates. This will be an opportunity for members of the community to pose questions to the candidates on the topic of moderation. Participation is completely vo...

discussion featured election

Undo

\o/

Catija

8:12 PM

:/

Magisch

@Andy just in time to handle my 300 pending custom flags

cackles evilly

Undo

Bets on how long until someone posts a "what do you think about Smokey" question?

Magisch

will be interesting to see who wins this time

Catija

Seems like the last wasn't that long ago... when... June?

Undo

ya know, it'd sure be nice to cut out the middle man on those 300 other flags...

Andy

8:14 PM

@Catija August

Magisch

@Undo You know I got less votes then both of the joke candidates last time

Catija

Plus, you got Jeremy in November/December.

Andy

Well, July was the election process, it ended at the beginning of August

John Dvorak

Any bets SFF opting out of Charcoal entirely and then crawling back a week later?

Catija

Why would they crawl back? They have low numbers... they don't really need Smokey there.

Undo

8:15 PM

Probably not. Initial signs are good on that meta post, assuming it's not our votes

right? riiiiight?

Magisch

don't look at me I have a total of 0 votes on that site

Undo

;)

John Dvorak

right?

Magisch

What's frustrating about SO elections is that unless you have like 20k rep and have been filling your review quotas every day you needn't even bother

Catija

What I don't understand is the voting on the question... People are really weird about voting on discussions.

Magisch

8:16 PM

because there are always like 10 people who have been doing that

John Dvorak

s/about.*//

Catija

I have 105 rep! You think I could win?

Mithrandir

@Undo I'll admit to voting

Catija

@Mithrandir You are allowed.

Mad Scientist

@Undo if I understand it correctly, you can't really argue that having more automatic flags makes the flagging more accurate. The problem is that the automatic flags aren't independent, they're all the same. So you're not adding 5 near-perfect flags and 1 human flag, you're reducing the flag threshold to 2, one automatic and one human one

Andy

8:17 PM

@Magisch I had every intention of continuing my reviews in the LQP queue. But I started in the mod queue and got lost there. Looks like I've dropped from 1st to 4th all time in that queue.

Magisch

I came to peace with the fact that winning an SO election is not on the plate for me

So now I buck the orthodoxy by other methods like causing a flagpocalypse

Catija

@MadScientist Except the rules for the fourth and fifth flags are more strict.

So, in a sense, they are different... quite different.

Glorfindel

@Catija no. You need 3k rep to nominate on SO.

Catija

@Glorfindel Man, I couldn't get there even with edits. :(

Glorfindel

@Magisch if you can't join them, beat them?

Magisch

8:20 PM

aye

Mithrandir

@Magisch I'm pretty sure that this election can be directly traced to you

Andy

@Undo The total scores are pretty 'meh' right now on the question and both answers.

Mad Scientist

@Catija they're not independent, which makes a huge difference in the probability of getting it wrong.

angussidney

Yeah

Magisch

@Mithrandir maybe. If bhargav's numbers are to be believed then I generate around 30 hours of work for the mods per week

Mithrandir

8:21 PM

You raise dozens of custom flags, which stick to the top of the mod queue, and take a long time to deal with. Also factor in time scrolling past to deal with other stuff.

Undo

@MadScientist We intend to have 5 near-perfect flags, yes. 100.00%, one false positive in thirty thousand.

But that's a result of restricting the posts that get five flags to those we're really really sure of, not a result of the number of flags.

It's a higher number of flags because we can do better accuracy, not more accuracy because of a higher number of flags.

quartata

was afk for a while, what happened with SFF?

Undo

nothing :P

quartata

hopefully someone pointed out that was the only FP in all time

Magisch

I posted an answer to the question collection, by the way

quartata

8:24 PM

@YvetteColomb you running again I hope?

Magisch

It's 2 questions in one but both pertain to my speciality

:D

@Mithrandir According to bhargav one of my serial voting flags takes about 20 minutes

so at 80-130 a week that makes for a lot of mod time

Undo

Depends on the flag. Probably a decent average

Mad Scientist

@Undo my point is mostly that the comparison with humans is unfair, if you cast more than 1 flag. As the flawed humans are reasonably independent, the odds are exceedingly low that they'd manage to nuke a post

Magisch

@MadScientist Piling on bias and coordinating humans suggest otherwise

If someone flags and leaves a comment like "this is spam" then others are more likely to flag. Same with downvotes or people reacting in chat

Undo

@MadScientist The math we'd do there is (1/30k) * (chance of a human making a mistake on a given fp) * volume of posts on site.

But yes, we don't do (1/30k)^5. That'd be wrong.

Mad Scientist

8:28 PM

@Magisch yes, and that is one of my major worries with the non-autoflagging part of the project.

Magisch

@MadScientist This isn't exclusive to charcoal though, in fact, I'd argue it's less likely to happen here then in a random unrelated chat room

Undo

@MadScientist Suppose I could give you an expected number of posts nuked accidentally over a given timeframe. What would be an acceptable threshold for you?

Magisch

most people here have plenty of experience with spam by this point

Undo

I.e. 1 post per year, 0.1 posts per year, etc.

angussidney

@Undo you probably need to add quite a few more zeroes in front of that

Mad Scientist

8:31 PM

@Magisch it's a general problem, yes.

Undo

0.1 per year is one in a decade.

Magisch

@MadScientist If it happens once in like a decade, is it really though?

Undo

@angussidney It probably would if we pulled the numbers; I just want to get an idea of what @MadScientist's risk tolerance is.

Magisch

Do you care at that point?

Undo

Careful. Mad's trying to help.

Glorfindel

8:32 PM

@Magisch yeah, but with 5 autoflags and one 'initiate' with the reviewer role a post could be accidentally nuked, and if it happens during a low activity period it could go unnoticed. Which is a good reason to create an additional review queue on metasmoke to have at least two feedbacks per post.

Magisch

@Glorfindel valid point

Mad Scientist

@Undo I'm not actually worried about the "regular" false positive rate, my point was that I think contrasting your 99%+ rate with the human 95% rate directly isn't a fair comparison, if you case more than one flag. The humans improve if they cast more of them

Undo

@Glorfindel That's a decent idea. Want to throw a GH issue for raising the review threshold to two feedbacks?

Glorfindel

sure

Mad Scientist

this isn't a static project, so I'd be more worried about unexpected sources of false positives

Undo

8:33 PM

@MadScientist We've thought about that, yes.

Mad Scientist

and the tools to review posts nuked as spam suck, or don't even exist. Of course mods could use your tools, but that again requires them to know about this project and find the right place. There are no effective SE internal tools to monitor spam-flagged posts (which is not your fault)

Magisch

one step towards solving that was our watch list @MadScientist. It reports, but is capped at 1 reason weight to prevent experimental blacklists from interfering with autoflagging too much. And a reason must have a large enough sample size to count anyways

SmokeDetector

[ SmokeDetector | MS ] Pattern-matching website in answer, potentially bad keyword in answer: How to automate Digital Marketing Assets tests? by Juan yelle on stackoverflow.com

tpu- by Glorfindel

Undo

Some numbers: Assuming humans are ~95% accurate, and completely independent (probably not), 1 in 3.2M chance of a post being nuked. Our chances are 1 in 1M given 1 in 30k and 97% accuracy.

John Dvorak

@MadScientist that's something that should improve irrespective of what Charcoal does

Magisch

8:36 PM

so the chance someone randomly changes something and suddenly many posts get nuked is severely undercut anyways. In addition, if someone's conditions fall below 99,75% accuracy (fast if something like this were to happen), it stops being a valid condition anyways. In addition to that, every false positive on a reason lowers that reason's confidence score, further amplifying the signal to stop autoflagging such.

So yeah, if something messed up happens, aside from everyone having the !!/stopflagging command, the system is almost immediately self correcting due to our high thresholds

Mithrandir

I'm almost tempted to ask for it to be raised to 99.80% just to see how many people's conditions get messed up

John Dvorak

LUL

Magisch

mine, for one. I just raised mine to 185 and that's 99,76%

DJMcMayhem

@Mithrandir I have no idea how to adjust my conditions to be correct

Catija

I think mine are on the default... whatever that is. And I'm not even on every site yet.

DJMcMayhem

8:38 PM