Charcoal HQ

hichris123

18:00

@Sally I was talking about metasmoke review, not whatever discussion was going on in here previously. :P

Sally

Yeah, on Drupal one can robo-review all day.

sd f

Undo

@Sally For metasmoke.erwaysoftware.com/reason/74, 205 where that's the only reason. 75 of those are tps.

hichris123

@Undo metasmoke.erwaysoftware.com/… gives "We're sorry, but something went wrong."

ArtOfCode

repro

Undo

@hichris123 oh, I broke filtering. Just a sec.

ArtOfCode

18:11

100 manual answer reports down, 700 to go

Undo

okay, this is weird.

ArtOfCode

Heh, reviews are back up to 6109.

Undo

norepro on local

Oh yeah I do

Ohhh

these old posts without bodies also don't have links.

so the check to show the NAA button fails

ArtOfCode

Reason 16 filter works, fwiw

Undo

yeah

pending deploy

fixed @ArtOfCode @hichris123

Sally

18:14

@Undo Looks like for posts it can be kept. Stuff like DIE DIE DIE DIE DIE DIE....

Undo

@Sally Yeah. It's also far lower volume.

ArtOfCode

@Undo did you add the cache bust on feedback?

This is what happens when you add an awesome new feature, everyone bugs you about it.

Undo

@ArtOfCode no, but I probably should.

SmokeDetector

[ SmokeDetector ] All-caps body: Can Unreal Engines Blueprint system be used as intricately as traditional programming? by Lovelle on stackoverflow.com …

Sally

It's also possible to keep it for answers only...

Undo

18:20

@ArtOfCode done

ArtOfCode

Awesome.

Sally

But let's see how it works in the present, if all-caps questions are really that annoying. They are rare.

!!/pull

SmokeDetector

@Sally CI build failed! :( Please check your commit.

Sally

Ugh, unit tests

ArtOfCode

@Undo either something's borked, or a manual cache bust is needed for manually reported questions

Sally

18:22

Yes, I noticed that clicking buttons on them did not have a visible effect.

Undo

@ArtOfCode It'll only invalidate the cache on new feedback from the web interface.

ArtOfCode

@Undo There are no more posts in that category to review, so I can't cause an invalidation

Undo

right

Going to manually bust everything, if I can figure out why SSH won't let me in

oh. meatsmoke != metasmoke

closer....

Sally

!!/pull

SmokeDetector

@Sally CI build failed! :( Please check your commit.

ArtOfCode

18:26

translate: vos meres a tous, j'ai dit vos meres a tous lol. jk

(from French) your mothers to all, I said your mothers to all lol. JK

Undo

 => {"fp"=>37, "fp-"=>12, "fpo"=>1, "fpu"=>4, "fpu-"=>9, "ignore-"=>1, "tp"=>275, "tpu"=>4, "tpu-"=>168, "trueu"=>6, "trueu-"=>13}

ArtOfCode

Smoked meat is good though.

Undo

Those last two might be the problem

Huh.

There are 11 posts in a weird state where they have feedback, but it's not true or false.

ArtOfCode

Time to get a-fixing :)

Might wanna check all the reasons for that.

Undo

I'm just going to add them to review.

ArtOfCode

18:30

Sure

Undo

it'll push the count up by a few hundred.

ArtOfCode

That's a fair number... oh well, back above 6k it is

Undo

wait a sec, this is weird.

now it's showing checkmarks on review :P

Going to roll that back, there should be a better way to fix this

ArtOfCode

You can either review them all yourself, or you can let us do it ;)

Undo

no, I'm going to make the system review them

ArtOfCode

18:37

Oh, that's even better

Undo

if it can show checkmarks on the rows, it can just set tp/fp on them directly.

ArtOfCode

Point

SmokeDetector

Restart: API quota is 4225.

Undo

only 555 of them. Shouldn't be too bad

hichris123

!!/pull

SmokeDetector

18:40

[ SmokeDetector ] SmokeDetector started at rev f409620 (Normal Human: There are two lists of tests...) (running on Raspberry Pi)

Restart: API quota is 4210.

[ SmokeDetector ] All-caps answer: C++ thread-safe increment with modulo without mutex using std::atomic by ALCAATWORK on stackoverflow.com …

Undo

@ArtOfCode Does that look any better?

ArtOfCode

I got pinged about 5 times by that one message somehow... checking

Undo

Now we're actually running into issues with rounding errors on the bars :P

ArtOfCode

@Undo I hate to tell you this, but all the red has disappeared.

Something may be seriously borked.

Undo

@ArtOfCode yeah, I know.

I'm just going to floor those and call it good

ArtOfCode

18:46

Yep

Same thing on Chinese char in title, I think.

Nothing in the queue, but there's a gap

Undo

There

Look any better @ArtOfCode?

That gap will be a little bigger than it should be, but it won't be there when it shouldn't be there.

well... maybe it will. Yeah, there's a case where it will

ArtOfCode

Ew, no. Look at the bottom few reasons.

I think you need to ceil one and floor the other

Undo

@ArtOfCode Will that work?

ArtOfCode

shrug

Undo

trying to come up with a case where that wouldn't work.

ArtOfCode

18:52

What's the worst that can happen if you try it..?

hichris123

Or just make it round standard rounding rules.

Ferrybig

@Undo, I created a account at the metasmoke panel, can you activate it? I want to see the new review section

Undo

in a second

ArtOfCode

It should work. Those lower reasons all have a gap of 1%, which wouldn't be there if one had been ceiled.

Undo

right. Deploying

@ArtOfCode there we go, that looks a lot better

I'm favoring the tp column.

ArtOfCode

18:54

Yeah, might as well

Make us feel better about ourselves

OK, next bug: reason 62, numbers-only title, has a space but no posts in review

Sally

@Undo If x+y is an integer, then floor(x) + ceil(y) = x+y

It's math, it works.

3

ArtOfCode

And another: reason 69, black magic in body, has no FP visible at all, no posts in review

Ferrybig

Unless rounding errors?

Undo

@Ferrybig done

@ArtOfCode looking

ArtOfCode

@Undo Same for reason 1, manually reported question

Undo

18:58

ArtOfCode

I've got loads of bugs to give you... this is going to be a fun afternoon.

Undo

The heck did that happen?

Ohhh

some of those have both tp and fps.

ArtOfCode

-5%?

Sally

@hichris123 Theoretically, this is a problem with 1.5+1.5 =3: whatever rounding rule is used, it will give a wrong result. Of course you won't hit that in practice.

Undo

@ArtOfCode It generates the blank space by subtracting the other two from 100

hichris123

18:59

@Sally True. Theoretically though one would be 1.49 and one would be 1.51. Unless it was more evenly divisible....

Undo

okay, should we could a post as a tp or fp when it has both feedbacks?

ArtOfCode

@Undo Ah, I've figured the bug. True has width 91%, false 10%. That's 101%, so false drops onto the next line, which is invisible in the progress bar.

@Undo send it to a moderator for review... oh wait

Sally

@Undo Raise "disputed" flag for a mod.

ArtOfCode

How hard would it be to implement accounts with higher privileges?

Sally

Mark as yellow on the bar?

Or, always go by the last feedback submitted.

Undo

19:03

@ArtOfCode fairly easy

Just a property on the devise model

@Sally That might work

ArtOfCode

@Undo Do it. Make a few trusted people out of here 'moderators', let them review disputed posts.

Sally

I think that going by last feedback makes the most sense. Typical scenarios: (a) someone fat-fingered the first reply (usually with sd). (b) the OP went on and posted the same thing twice more, crossing from link-only to spam.

ArtOfCode

Maybe combine the two? Go by the last feedback by default, but pass them to mods to review anyway.

In case the last feedback is wrong.

Undo

@ArtOfCode Going to do the yellow thing first.

ArtOfCode

Yeah

Ferrybig

19:05

A other reason for double feedback is that something was first marked as false, but after close discovery it was really spam

Undo

another reason is SOCVR being trigger happy

Ferrybig

Example of this problem: metasmoke.erwaysoftware.com/… (Yes, its the same post 3 times)

Sally

@Ferrybig But this is a bit different. Those are three separate Metasmoke entries.

ArtOfCode

Just take a step back and realise what an awesome system this is.

Sally

If I understand correctly, the same SE post may be in multiple MS table rows, and they are treated separately.

Undo

19:08

@Sally Yes

every time smokey reports something, it's a new, unique MS entry

Ferrybig

Metasmoke down? got 502 and 503

Undo

cap failed in the middle of Puma restart

just a sec

ArtOfCode

We have a remotely-controllable, semi-intelligent script running on a Raspberry Pi scanning thousands and thousands of posts a day, classifying them, communicating to a RoR app which implements loads of useful data views and a review queue to catch and feed back on posts that were missed in chat.

5

Spammers don't stand a chance.

Undo

there we go. Adding an orange row also let us remove the rounding foolery

ArtOfCode

@Undo What - if it's ambiguous or disputed, just mark it orange?

Undo

19:11

@ArtOfCode and it costs me... like four dollars a month. Amazing what you can do with $48 per year.

ArtOfCode

Plus whatever's left?

Undo

If it has both tp and fp feedback, it shows as orange in the tp/fp thing

ArtOfCode

For the price of 15 cups of coffee per year, we're making the process of catching spam vastly more efficient.

Oh dear. @Undo metasmoke.erwaysoftware.com/post/19587 404

Undo

that's... odd

why the heck would that 404?

Exists in the database

ArtOfCode

shrug. Does the DB row agree with the details?

Undo

19:14

ohhh

that's a post on Latin. Probably just need to refresh the site list.

ArtOfCode

That would help

Ferrybig

So you used a 404 error when something was missing from the server, should a 505 be better because there is something wrong on the server?

Undo

I didn't use a 404 error

It's Rails generating a 404 error because something wasn't found

ArtOfCode

OK, so when that one's fixed I have two more things in my bug/improvement queue for you :)

Undo

@ArtOfCode fixed

ArtOfCode

19:16

@Undo mod accounts

Undo

An unknown site should probably trigger a site fetch

@ArtOfCode To do what?

ArtOfCode

@Undo also that

@Undo Review disputed posts. Or you can do more fixing of reason progress bar rounding, if you feel like that instead.

Sally

@Undo Opening up a DDOS possibility, since site fetch uses API. Maybe refresh once a week? They launch on Tuesdays.

Undo

@ArtOfCode The bars are working, aren't they?

ArtOfCode

@Undo Nope. Chinese char in title, 62, has a gap, nothing in review

Undo

19:18

@Sally No, no DDOS possibility. MS checks for the Smokey key long before it ever even looks at the post data.

ArtOfCode

@Undo But could someone manually report something a whole load of times on an unknown site?

Undo

@ArtOfCode From EC2, the request for the site would finish long before they got a second or third report out.

Ping times to the API are... insanely fast.

Chat would ratelimit them first, I suspect

But refreshing once a week would probably work just fine

ArtOfCode

Yeah. Refresh on Wednesdays, because then new sites will be completely set up?

Also reason 58 has a gap, nothing in the queue

Undo

@ArtOfCode Two of them that aren't marked tp or fp; one is marked naa and one marked 'ignore'

ArtOfCode

Any of those that can be chucked back into review?

Undo

19:22

@ArtOfCode naa mark again.

from you :P

ArtOfCode

Heh, got myself.

Undo

I suppose NAAs could just be orange?

ArtOfCode

Um, metasmoke is down

@Undo They should be, yeah

Grey says to me 'unreviewed'

OK, back up, but status is showing red

Undo

Smokey will ping and fix that shortly

ArtOfCode

yep

all systems green

Undo

19:25

I... uh... I just crashed vim :P

Gotta go do something, just a sec.

SmokeDetector

[ SmokeDetector ] Offensive body detected: How to download pedo stuff? by pichipichi on stackoverflow.com …

Sally

@SmokeDetector k and gone

SmokeDetector

API quota rolled over with 3970 requests remaining. Current quota: 9999.

stackoverflow: 982
superuser: 435
askubuntu: 354
unix: 164
drupal: 158
english: 154
electronics: 146
math: 143
gis: 101
dba: 98
mathematica: 93
gaming: 91
apple: 88
ell: 83
academia: 81
mathoverflow.net: 77
scifi: 77
physics: 73
travel: 71
meta: 68
magento: 66
diy: 63
programmers: 62
salesforce: 61
android: 59
webapps: 58
worldbuilding: 56
wordpress: 56
chemistry: 55
puzzling: 54
money: 52
tex: 50
stats: 49
security: 49
ru.stackoverflow: 48
codegolf: 48
rpg: 45
graphicdesign: 40
gamedev: 39
aviation: 39

Ferrybig

Should we add the fourth word of that title to the bad keyword list?

SmokeDetector

[ SmokeDetector ] Link at end of body, pattern-matching website in body: That Great to all men by Dixonxyx on drupal.stackexchange.com …

Undo

19:36

weird time

sd k

@ArtOfCode done

ArtOfCode

Awesome. Reasons are filled up from the bottom of the page to about halfway up now.

Could you add tooltips that give the percentages?

Sally

@Ferrybig Yes, looks selective enough.

SmokeDetector

[ SmokeDetector ] Bad keyword in title: #888 352 9606 ms Microsoft office 365 intallation tech support number? by john5s on askubuntu.com …

Ferrybig

@SmokeDetector k

Sally

No phone number detection, because 1 is omitted?

By the way, AU is way ahead of the pack in terms of support numbers, although we don't collect data for that.

There is a very incomplete slice of these posts here: metasmoke.erwaysoftware.com/…

Undo

19:50

Seeing a lot of fps for connectify

metasmoke.erwaysoftware.com/post/14059

epic

Sally

I guess they are big enough to not spam anymore, even if they did?

Undo

@Sally Data for what?

Sally

@Undo I've since whitelisted "ubuntu" for link at end.

@Undo "support number" category of spam.

Undo

@Sally Yeah, but look at the answer and the question title

@Sally It's not in why data?

Sally

Ah, that. Yeah, trying to be funny

Undo

19:55

(going to have to leave in a few minutes)

This is probably the most productive few hours we've had in a loooong time.

Sally

In principle, but there are many keywords. It's just a remark, I'm not saying we have to do something about support numbers specifically.

Yes, huge progress. Thanks for your work on this.

Oh, and I removed connectify a while back: chat.stackexchange.com/transcript/message/27398753#27398753

The database contains a number of past mistakes; the current accuracy of SD is higher than historical average.

Undo

@Sally I might make the bars and graphs and thing default to the last month or so to combat that

Sally

That would also get rid of retired reasons, which is good. But maybe 2-3 months.

Ferrybig

There should be a way to add polls to the SE chat

Sally

Why?

Ferrybig

20:02

So we can vote who wants to make the bars time limited, instead of having 10 messages from people who agree and disagree, I agree with the feature

Undo

too complex, too few use cases.

we're a team, not a democracy.

Sally

In a message one can also give a reason.

1 person who sees how the proposal will blow everything up vs 4 who don't see it.

That said, one can post two messages and ask for stars. So it kind of exists.

Undo

star the one you agree with:

- stars are good

4

- stars are bad

there, see? 100% of the voting populace likes stars.

Ferrybig

How should (self) vandalism be marked at the smokey review? With true positive because the detection was at the time the post was garbage? or with false positive because the post keeps on living

Sally

I think tp because it served the purpose. MS doesn't blacklist users, so we don't have to worry about side effects.

Or, put another way, vandalism is an abusive action.

sd f

ArtOfCode

20:28

We've broken into the 4000's

Ferrybig

Is it just me, or is the "contains X unique characters" reason always off by one? metasmoke.erwaysoftware.com/post/19014 (should be 1 instead of 2) metasmoke.erwaysoftware.com/post/19681 (should be 2 instead of 3)

Sally

@Ferrybig Strange. I'll investigate

Currently I'm trying to improve the phone number regex; the added pre-processing steps allow it to be simpler.

Sally

20:54

@Ferrybig Maybe there is a new line in there.

I recall that body returned by API is <p>actual text</p>\n

.rstrip() should fix that

!!/test #888 352 9606 ms Microsoft office 365 intallation tech support number? by

SmokeDetector

> Bad keyword in title
----------
Title - Key phrase: tech support

Sally

!!/pull

SmokeDetector

[ SmokeDetector ] SmokeDetector started at rev 2b030ea (Normal Human: preprocessing simplifies phone regex) (running on Raspberry Pi)

Restart: API quota is 9593.

Sally

21:09

!!/test #888 352 9606 ms Microsoft office 365 intallation tech

SmokeDetector

> Phone number detected in title
----------
Title - Phone number: 888 352 9606

Sally

Phew, the updated regex works.

hichris123

Heh some spammer titled their question "Weight Loss Off topic" metasmoke.erwaysoftware.com/post/6472

ArtOfCode

Are we truncating posts to ~200 chars now?

That view, and a number of others, appear to be cut off and replaced with ellipsis

hichris123

@ArtOfCode Those were posts solely from the websockets.

And the websockets only give a 200 char body summary.

ArtOfCode

21:18

@hichris123 Oh, before we fetched from the API?

hichris123

Now we have "Universal Bodyfetcher", so everything goes to the API.

ArtOfCode

Sense made.

hichris123

@ArtOfCode Previously, if a post was caught based on just the title & body summary, it never went to the API, so metasmoke only recorded the data it got from the websockets. Now, even if a post is caught from the websocket data, it still goes to the API so we get the full body, title, rep, etc.

ArtOfCode

Yeah, makes sense

Sally

!!/pull

SmokeDetector

21:24

[ SmokeDetector ] SmokeDetector started at rev ee00fbc (Normal Human: remove trailing \n when counting chars) (running on Raspberry Pi)

Restart: API quota is 9478.

ArtOfCode

There are only 15 reasons left with unfilled progress bars

and we're almost down into the 3000s

Ferrybig

Maybe we can complete it today

hichris123

I think I've misclicked a few of these. :( Out of several hundred though, 2 or 3 misclicks isn't bad.

Ferrybig

Nobody of us has a 0% declined flag rate on their main site. A few mistakes are oke. It can be useful if we can change the outcome of a post directly from the posts page if something is wrong

ArtOfCode

3824 tasks left.

We won't get it done by the end of my day, given that's only an hour or two max away.

Then again, we have burned 200 in the past 10 minutes...

SmokeDetector

21:52

[ SmokeDetector ] Bad keyword with email in answer, email in answer: At what point should I begin paying off student loans? by Rose Silva on money.stackexchange.com …

hichris123

@SmokeDetector tpu-

Sally

Some of the remaining ones are really hard to judge. A short answer with a link to a blog...

hichris123

@Ferrybig I'm quicker. :D

@Sally yeah. We're getting to the bottom of the barrel.

SmokeDetector

[ SmokeDetector ] Korean character in title: 썬시티카지노사이트〈♯M ＶＰ 222．Cοm♯〉윈스바카라 by kangsukjung on gaming.stackexchange.com …

ArtOfCode

sd k

Been a while since we had one of those

Ferrybig

21:59

With us all active here, posts are getting nuked in no time

Undo

@ArtOfCode This is... really impressive.

Ferrybig

Bug 404: metasmoke url Image of post in review

Undo

probably irritated because it doesn't have an associated site.

Although we should have had those in late Dec.

@Ferrybig fixed that one, going to look through remaining ones.

Only three others that are broken like that.

fixed all of them

ArtOfCode

22:17

Broken down to the 2000s.

Less than an hour to do 1000 tasks

Ferrybig

Lets do a sprint

Undo

that's what we've been doing :P

ArtOfCode

Could we bring in a few more from SOCVR? People who we know are sane

Undo

@ArtOfCode sure

ArtOfCode

um

who's sane?

I'm never there :)

Ferrybig

22:18

@MsYvette Can you help us?

Undo

@ArtOfCode rene

ArtOfCode

@rene Fancy helping us burn down the review queue of classifying Smokey reports?

Undo

@ArtOfCode You doing pattern-matching in answer?

ArtOfCode

@Undo Manual reports, actually.

Ferrybig

Or praveen or tuna could help us

ArtOfCode

22:19

I did those pattern matches a while back, though.

Undo

@Ferrybig Not comfortable with that for a few reasons.

ArtOfCode

> Man i need to go fuck my sister in the ass and get hur to suck my dik.

um

Undo

@ArtOfCode Will metasmoke.erwaysoftware.com/… load for you?

ArtOfCode

Nope. What's reason 20?

oh wait

Yes, it takes a while to load but it does load.

Undo

huh

weird. Won't load at all for me.

Ferrybig

22:28

@Undo Seems like the page contains one of those extremely zalgo texts, I can feel it in the kind of lagg the page gives

Undo

@Ferrybig Ohh. Maybe it's killing Chrome.

Yeah, I can load it fine in Safari

Interesting

Sally

I think there is no particular need to get the reviews done today, or tomorrow.

Undo

nah

Although keeping momentum is a good thing.

Ferrybig

@Undo Is the lag fixed now, I marked the following zalgo post already: metasmoke.erwaysoftware.com/post/7537 This seems to be the cause

Undo

@Ferrybig Yep. Wow.

I can't even load that page.

Sally

22:31

Loads in Chrome for me.

Undo

yeah, all kinds of crap in it when I pull up the record.

@Sally version?

ArtOfCode

Works fine in Chrome Canary

Ferrybig

Loads slowly in Chrome for me - Windows 7 - 48.0.2564.116 m

Sally

Chrome 49 on Chrome OS.

May be related to OS / its fonts

Undo

oh

Just updated to 49.0.2623.75 (64-bit), still can't load that one post.

weird

ArtOfCode

22:35

I'm on Chrome about 51.

hichris123

Hmm, how should all-caps titles be marked? fps, most of them?

SmokeDetector

[ SmokeDetector ] Repeating characters in answer: Key won't turn to off position on 2010 Chevy Aveo by user15521 on mechanics.stackexchange.com …

Undo

@hichris123 Would you red-flag them?

that's my test.

hichris123

@SmokeDetector tp-

@Undo And for most, the answer is no. :)

Ferrybig

In what reasons is everyone reviewing the posts? to prevent 2 people from going through the same posts

Undo

22:40

I'm taking a kinda-break.

hichris123

@Ferrybig If you review the same post someone else has, it just does nothing. So if the posts don't fade out, then you know to choose another reason. :)

ArtOfCode

translate: А я хуй его знает. ТЫ, Я ХУЙ ИГО ЗНАЕТ НАХУЙ ВАЩЕ.

(from Russian) And I know Dick. TY, I PRICK THE YOKE KNOWS НАХУЙ VASCHE.

Undo

Don't review deleted posts with no body, I have plans for those.

Undeleted ones you can open on the site are fair game.

ArtOfCode

@Undo Heh, sneak previews on your plans. Not even if the titles make it incredibly obvious?

Undo

@ArtOfCode Oh, sure, if you're confident.

But don't waste time guessing.

ArtOfCode

22:46

Spam titles are surprisingly easy to distinguish. If it's ambiguous I'll just leave it.

Undo

I'm really more worried about answers; they're kinda impossible to guess at.

ArtOfCode

Yeah, those are difficult.

Undo

Green line is hugging the grey line much more closely.

Ferrybig

Seems like the overal avarage of false positive keeps the same, while the true positives follow a pattern

ArtOfCode

The FP's do have a very weak correlation to total reports, but it's so minor that it barely matters.

On that note, all the manual reports, questions and answers, are gone.

And it turns out that regexes are actually more accurate than we are.

3

Sally

22:54

@ArtOfCode offensive

hichris123

@ArtOfCode Now you get to move on to all-caps... :P

ArtOfCode

@hichris123 Pretty sure I've done a whole load of them too :) and the code block reason is gone through.

Sally

But given that all-caps titles are retired, should we care about the accuracy?

hichris123

@Sally no, but gotta get to inbox zero! :P

Undo

We could just bulk-mark the ones caught only by that reason orange and say "we don't care"

Sally

22:56

I marked all non-spammy all-capsy things as fp.

ArtOfCode

We can't have reasons with grey regions :)

Even the reasons down the bottom of the dashboard are fully reviewed.

The old retired ones that haven't seen the light of day in years.

Oh hey, we've got through all the bad keyword in body reports! That was the biggest category.

hichris123

yeah, I spent a lot of time there. :)

Transcript for