Uh

whoa, hold on - these are the messages that were addressed to you.

Pretty sure I did* say exactly that, @ArtOfCode

There is no obscure technical content there. There is technical content addressed to other people, as a result of me fixing the problem - but that's because chat is async

Then you clicked a link to an internal tool, posted in a message directed at someone else - which is fine, but it's not designed to be easily understandable at first glance.

Let's try this again. Hey there, @Cody! I hear you encountered something that didn't make sense to you. Would you mind explaining how you got there and what you've encountered?

(Also... sounds like everyone here was a bit stressed...?)

Hm. The links I clicked were the ones in the flag message.

So passing those off as being part of an "internal tool" still doesn't make sense to me.

charcoal-se.org/smokey/Auto-Mod-Flags isn't an internal tool

Sigh

It's like talking in riddles

I am more than happy to have a conversation about how we can help address concerns or confusing things in our systems. What annoys me is having my work - and the work of dozens of other people who put their own time into keeping things clean - dumped on just because it's not immediately obvious to those who don't have as much understanding of how things work.

So, let's do the former.

So is there something on https://charcoal-se.org/smokey/Auto-Mod-Flags that isn't clear? I'd believe that

The other link is to this room

I imagine it's a lot less annoying than being treated like an idiot for not immediately embracing a system that is inherently obtuse and poorly explained.

@CodyGray The flag you got should've had a link in it to charcoal-se.org/smokey/Auto-Mod-Flags, is that right? Are there concerns you have about what's on that page?

It also had a link to this chat room, which seems to be the biggest mistake.

You get 5 different people telling you that you're the one with the problem.

No thanks.

@CodyGray So you don't like something that happened here. Can you point to it?

What did the mod flag text say exactly? I this all because I clicked NAA instead of TP?

What I really don't like is the attitude of the people who work on this.

Because it sounds like we're back to "your tool is terrible" and us saying "yeah, because it's designed for us"

@CodyGray Okay, hold up. Let's try and focus on facts, yeah? Clearly we have a difference of opinion here; facts are common ground.

@CodyGray people simply misunderstood you and assumed you were talking about something else, no need to make it heated.

Let me look up the flag and see what it said.

[ SmokeDetector | MS ] Repeating characters in title (64): totalPayment xxxxxxxxxxxxxxxxx ✏️ by user2234937 on stackoverflow.com

> This post had 1 spam/abusive flag(s) cast on it by Charcoal members and has since been deleted, but was ultimately judged not to have been spam. Please review whether spam flags - and the penalty that comes with them - are appropriate for this post - you can let us know in chat.stackexchange.com/rooms/11540 if the flags were inappropriate. If you're wondering WTF this flag is, see charcoal-se.org/smokey/Auto-Mod-Flags for details.

fp feedback received

@PaulWhite You have no fault in any of this, the NAA distinction is purely technical.

It's probably my fault, I just can't figure out how.

Several problems: (1) There is no explanation of what "ultimately judged not to have been spam" means. Who made that judgment? How? Can I review it like I can for other flags? It's asking me to make a decision. I need to have information available in order to make a decision.

It looks like it's spam, but I'm second-guessing myself, because a flag is telling me to second-guess myself.

@Undo Yeah I was just wondering if that is what caused the mod flag? I misremembered the guidance on nonsense is all.

@PaulWhite Automated thing the system does whenever it sees a post with autoflags get deleted while there's any kind of doubt about it.

NAA feedback counts as doubt. It's just part of being super cautious

(2) As already mentioned, there's a very confusing blurring of the distinction between rude/abusive and spam. Internally, the system might treat them as if they're the same (same penalty, etc.), but externally, they are different, and there's different guidance for using them, so reading the message, I'm wondering if the problem is that it was marked as "spam", when it should have been marked as "r/a". I can't tell from this message, or any of the linked content.

@CodyGray Okay, that's useful to know. Would it help if I moved "by Charcoal members" there, so it says "...cast on it, but was ultimately judged by Charcoal members not to have been spam."

Or "not to have been spam/abusive"

@PaulWhite +1

That doesn't really answer the central question of who.

@ArtOfCode Maybe reword "ultimately judged" as well, because it looks like we actually had conflicting feedback

@ArtOfCode I don't think "ultimately judged as" is correct here

@CodyGray What might you do with the 'who' information?

that'd help scope it to be useful

I'm more interested in what those people actually thought, rather than the vague idea that they had a thought.

@CodyGray So the driver behind that is that the spam/abusive flag really could have been either. The system doesn't look up which it is before it sends that other flag. Should it?

Normally, when there are conflicting flags on a post, I can show what each flag was

and thus zero in on where the disagreement was

Perhaps a link to the MS record would be more helpful?

@CodyGray The disagreement is our problem, though. The only role you need to take is "was this wrongly deleted"

There's no "no flag" flag on posts, unfortunately.

> This post had X spam flag(s) cast on it by Charcoal members, but others raised concerns about whether or not it was really spam.

@PaulWhite that's not too easy to interperet for someone who's unfamiliar

@NobodyNada I guess

If this is just a "check the machine's work", then make the flag read: "This post got deleted by the community as spam or rude/abusive. Please verify whether that decision was correct. If so, mark as "helpful". If not, mark as "declined" and undelete the post."

Make it work like other flags, and make the action items obvious.

@ArtOfCode "Spam" should probably be "spam/abusive"

@CodyGray That works

@CodyGray Posts that get that flag didn't necessarily get deleted

@NobodyNada Aye, context-dependent

@ArtOfCode Isn't that the trigger

deleted && (post.is_fp || post.is_naa) && post.flag_logs.manual.successful.count > 0

Is it? Forgotten how exactly that gets triggered

This page says deletion is required

It's triggered at deletion time

My bad, ignore that

Within thirty seconds, in this case. Which seems a bit slow.

Is there a useful distinction to be made here between flags actually cast on the post and feedback given on MS?

How and why are those two things different, @Paul?

@CodyGray One does not necessarily mean the other was done as well. You can flag a post without leaving feedback here on whether or not it was spam; you can also feed back on a post to tell Smokey it was spam without casting a flag.

@PaulWhite MS feedback doesn't always indicate that it should actually be red flagged - for instance, a first offense of self-promotion isn't red flagged but is marked as TP.

@CodyGray Well I can't see actual mod flags on Stack Overflow (and I can't tell if any were cast by Charcoal members from what I can see on metasmoke) but I can see the reviews metasmoke.erwaysoftware.com/post/159775

@Mithrandir Yes exactly - I can't tell if reviewing users actually cast on-site flags.

@ArtOfCode The second case is more what I was asking about. Why would you issue feedback on a post without raising a flag?

@CodyGray FP feedback is triggers that 'other' flag, so you wouldn't be flagging at the same time as issuing FP feedback

As far as moderators are concerned "X spam flags were cast on this post" indicates there were actual spam flags, but "ultimately judged not to have been spam" does not necessarily indicate the presence of any other flags on the post.

@CodyGray when we want to tell the system "yes, good job catching that" but we don't want to flag it. One common case is Mithrandir's example, another is vandalism

This...doesn't make any sense to me

That "ultimate judgement" is based on feedback to Smokey

@CodyGray It's a complex system, modeling a more complex real world. You're hitting the friction in the transition there.

@CodyGray TL;DR: the standards for what is spam-flaggable and what is a true positive for Smokey are slightly different. There are some edge cases where one is appropriate but not the other.

ugh

@Undo Which we try to keep under the hood, and failed this time.

I'm really trying here

But I really don't have time to learn the inner workings of an entirely separate system

And the abstraction is very leaky

@CodyGray Can you be clear about what you want? The "information needed for moderators to do their job" download, or the "information needed for Charcoal people to do their job" download?

I, um, what?

I don't know what that means

I feel like we are speaking two completely different languages here

Are you looking for an explanation of how the Charcoal systems work, or are you just looking for "what do I need to know to handle these flags"?

@CodyGray The set of posts Smoke Detector should pick up for manual review is not identical to the set of posts that should have on-site flags cast. Ideally you wouldn't have to worry (or even see) the former. In this case you did, because I mistakenly assessed that post as NAA rather than abusive. That collision of views raised the custom mod flag.

@ArtOfCode As others have said, "ultimate judgement" is probably the wrong thing to say here. The flag is triggered by having >=1 feedback that's not TP. Thus, there could be 10 people saying TP and only 1 feedback as NAA, so it's more "there was at least 1 person who felt a red-flag was not appropriate".

@Makyen Aye, that needs rewording

"ultimate judgement..not spam" -> "charcoal reviewers disagreed"

"there was a dispute among Charcoal members whether the red flag was appropriate"

Like it

I'm just glad the whole thing was my fault after all 🙂

@PaulWhite seriously, not your fault. Minor mistake just lit the fuse on a discussion that was probably going to happen anyway.

Hence smiley

If the custom mod message gets improved as a result of this all good

Probably no one's fault, actually. Arguments don't have to have fault assigned, just need to get some kind of good result out of them

> This post had X <spam/abusive> flags cast on it by Charcoal members, but further review raised disagreement on whether or not it was <spam/abusive>. Please review whether spam flags are appropriate for this post; you can let us know in <CHQ> if the flags were inappropriate. If you're wondering WTF this flag is, see <CSE.org info page> for details.

Further improvements welcome

I like this

@ArtOfCode Do you need original flag count?

@ArtOfCode 👍

> This post was assessed as <spam/abusive> by Charcoal reviewers, but there was some disagreement. Please review whether spam/abusive flags are appropriate for this post. You can let us know in <CHQ> if the flags were inappropriate. If you're wondering what this custom mod flag is all about, see the <CSE.org info page>.

@ArtOfCode Charcoal HQ: now with 20% less things tested in production!

@thesecretmaster Seems useful to me. Provides a guide on how many flags mods are looking for and potentially helps identify who cast 'em

But the mod will see if there are other flags, and if the flags have been retracted it could be confusing.

I thought we couldn't tell if on-site flags were cast or not?

We can tell for flags cast with the MS API

@PaulWhite we can if they were cast through the MS API, which most of our userscripts use

Ok

We don't have visibility into arbitrary "all flags on any given post", but if it goes through the systems then we can see those

I think a flag count causes more confusion than it remedies. Also, aren't we going for "minimal info needed by mod to handle flag"?

It does seem like it doesn't tell the handling mod anything they can't already see

Was originally going for "enough context to handle it in-place without having to go looking", actually, but whether that's the right goal or not I don't know

Each mod only needs to learn about Charcoal at most once

@JohnDvorak but there's like five hundred of them

I think minimal is best for usability for mods -- don't want to confuse folks.

@JohnDvorak It seems some don't want to (and that's ok).

Hyperbole, or an actual estimate?

Can we boil it down to "The Charcoal project cast spam flags on this, but at least one of us isn't confident it's spam. Please take a look and undelete if the post shouldn't be deleted."

I think it's ~350

w/ a link here at the end

That seems good

@Undo Doesn't solve the spam vs abusive bit.

Spam vs abusive can be inserted dynamically based on what flags were actually cast; we have that data

"spam/rude-abusive flags"

> Charcoal project members cast spam/abusive flags on this, but at least one of us isn't confident it's spam/abusive. Please take a look.

"isn't confident it should have been deleted by those flags"

Do we know for sure it has been deleted at that pont?

"isn't confident of that decision"?

@PaulWhite based on Cody's feedback, something actionable is important

@PaulWhite yes

@PaulWhite Yes

Well that's fine then.

how about changing "one of us" to "one reviewer" because "us" could be confusing?

or "at least one reviewer" lol

messages are hard

"at least one Charcoal member"

How about "Hey Cody! Charcoal here - did we get this right?" Probably not 😃

@Undo We could also poll the TL on what info they care about.

eugh

this is technically something we could do

We'd probably get a bit more information out of that than we could handle

TL is not necessarily the most positive forum right now

Which is not terrible

> Charcoal project members cast <spam/abusive> flags on this post, but at least one subsequent reviewer isn't confident that's the right decision. Please review and undelete the post if necessary; see <info page> for more details.

@ArtOfCode nice

Want to link to that info page, because some mods will have more questions and that's the easiest and most reliably friendly way to field them

@ArtOfCode s/subsequent/other/

works for me

the FP might precede the TP

@JohnDvorak or just omit

next challenge: Implement in first try without Rubocop yelling at you

@PaulWhite true

@Undo Psssh, it's a static string, how hard can it be? :)

you dropped a word last time :P

> 432 errors, 23 warnings

:P

These conversations are why I'm still here - a bunch of people working to solve a problem. We need more of this.

Maybe without the same start, but this part is pretty good

Hey, we can create our own problems if the need be :P

Turn off Smokey on SO for a week and see if that changes the other SO mods' approach to the project at all

SO has plenty of stuff for spam to hide in

a more appropriate victim could be good ole Drupal

@PaulWhite Not sure that'd cause the kind of conversation Undo was talking about...

@Undo run that on the MS server?

juuuust to check I got it right

bah, stick a puts on it

@thesecretmaster Yeah wasn't fully serious

looks good on a dump

[ metasmoke-deploy ] deploy started by ArtOfCode

[ metasmoke ] ci/circleci: rubocop failure on ed4cbc6: Your tests failed on CircleCI

@PaulWhite I've been tempted to do this a few times, but it'd be hypocritical. Since I believe this is a Good Thing, turning it off would be a Bad Thing.