Charcoal HQ

12:01 AM

204 out of 51564 don't have a deletion date in the dump for one reason or another (some of those correct). Not going to worry about 0.3%

Frequency of deltas between -10 and 10 (seconds, actual - ours)

~~Full version of the above table; 0 is at L957~~ this is borked

So we have very accurate numbers for more than half of the 51360 posts where the following is true: (1) MS had deletion information, and (2) Shog's dump had deletion information

Now I'm going to look at one of those +/- 10 second delta posts and see what's going on. Hoping it's clock skew that can be corrected historically without changing our systems.

MS post 46907 is stackoverflow.com/a/40637555. MS records deletion at 2016-11-16 16:53:10, Shog's dump claims deletion at 2016-11-16 16:53:20 (10 seconds later). Shog is right according to the web interface.

Nothing unusual in the post timeline

Report is this:

Nov 16 '16 at 16:52, by SmokeDetector

[ SmokeDetector | MS ] Manually reported answer: How to improve performance of map that loads new overlay images by Francisco on stackoverflow.com (@dorukayhan)

Hah

Chat report was at 16:52:55Z. MS creation date is 16:52:47

darn near a 10 second delta

Real creation date on site was 16:41:39 - we were slow on that one, but it doesn't really matter.

quartata

SE was having garbage websocket issues that day

Undo

k, we'll look at a later one

going out to 9-11 second delta to get a recent example

well that's interesting

There are only 3 in that range since 2016-11-16 19:31:06 (MS 46919)

All on sites I don't have 10k for

quartata

What are the sites?

Undo

Apple, Cooking, Money

MS 76284, 84703, 101895

quartata

@Glorfindel

Undo

12:16 AM

I don't really need 10k, I suppose. I trust Shog's dump.

Jul 14 '17 at 8:24, by SmokeDetector

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, pattern-matching product name in body, pattern-matching website in body, +1 more: It allows to beautify the dimensions of the penis and makes it more strong by eamber juri on apple.SE (@CalvT븃)

8:24:25Z actual report; 08:24:21Z MS creation date

Within tolerances on that end

Looking at all deltas between 5 and 30 seconds. There are only a few after 47xxx series IDs.

26

1107 / 1136 occurred before 2016-11-28

I'm going to limit my analysis on these to posts after 01/01/2017

oh what the heck

apparently subtracting dates in MySQL isn't that simple

2018-02-15 13:12:59 - 2018-02-15 13:13:00 = -41

not buying it

Okay. With that fixed....

Running the previous report, but with fixed timestamps and only posts after 2017-01-01

An interesting - and encouraging - sudden dropoff in the negatives

That pattern continues - a long tail of singles in the negatives, while the positives have a long tail of not-singles

The timestamp fix brings up way more posts where we're late with deletion logs

quartata

inevitable

Undo

101661 / stackoverflow.com/questions/48229219 - we're ten seconds late with the deletion log

Jan 12 at 15:25, by SmokeDetector

[ SmokeDetector | MS ] Offensive body detected: How can you get current packages for linux? by user2762996 on stackoverflow.com (@iBug @TetsuyaYamamoto)

But on that one, the MS report is created waaaaay before the post here

15:25:15 MS, 15:25:56 chat

MS clock is currently in sync, as far as I can tell

SmokeDetector

12:32 AM

[ SmokeDetector | MS ] Repeating words in answer: On an MSI laptop, how do I disable the backlite keyboard by user799607 on askubuntu.com

Undo

sd n

@Undo Could just be chat ratelimiting on that one.

MS 106659 / stackoverflow.com/a/48806358 was deleted twice - we were right on the first deletion; Shog's dump takes the last one.

Ditto on 106200 / stackoverflow.com/a/48746619

ArtOfCode

Art's Midnight Musings #2: Apparently I am writing that paper now... not quite sure how that happened, but apparently the fingers decided to type and here we are

Undo

And stackoverflow.com/admin/posts/timeline/48711539

ArtOfCode

@Undo mod link

Undo

oh. stackoverflow.com/questions/48711487/…

105670 / stackoverflow.com/a/48681086 was off by 12s. Looks like a natural delay

These are insanely rare recently. I think our deletion logs are actually very solid.

Outside of the ones we're just missing, of course. quartata's fixes should make that way better.

Still, though, I think we have very good log data on this. Rows and rows and rows of 0/1/-1 deltas with Shog's data.

ArtOfCode

12:50 AM

@Undo sounds good... next steps/improvements, etc?

Undo

@ArtOfCode I'm thinking the next step is running a rand(1, 5) flag count experiment on posts with score >= 300. See any problems with that?

Run it for a week or so, collect data.

ArtOfCode

You wanna put the meta post out before or after that?

Undo

after

ArtOfCode

Okay

No problems per se, but probably a couple things to think about

Undo

If we do it before, we're going to be fighting the "but you have no data" crowd. I think I'd rather fight the "site agency" crowd.

ArtOfCode

12:52 AM

whatcha wanna do about core flag count?

Undo

@ArtOfCode I was thinking we use core for the first 2 flags, then fill with commoners past that

ArtOfCode

heh

yeah, works for me

Undo

Was going to add a FlagSetting for five_flag_experiment_weight - set to ten thousand to turn it off

angussidney

When we do the actual 5-flags we should probably scale core flags appropriately

Undo

Definitely

ArtOfCode

12:54 AM

@angussidney yeah, just flag_count / 2

rounded up like now

aye

so 3/2 for those

Okay. Probably a good thing I put better log tooling in before we do this :P

Undo

eh, shouldn't be dangerous. We'll review it before I fat-finger new SO questions into oblivion.

(not saying that'd be a bad thing...)

ArtOfCode

12:56 AM

@Undo you want me to leave writing the code to you for this or do I just go ahead?

Undo

I'll take a stab at it, unless you really want to

ArtOfCode

Nah, go for it. Just working out whether to get going now or let you do it 'cuz you know how you want it to work ;)

Undo

I got it ;)

you have a paper to write :P

ArtOfCode

my hobby is giving myself new things to do and never finishing them :P

6

I also have haskell to learn, Java to write, and JS to finish, but I'm messing around with papers instead :)

Undo

Haskell? School thing?

ArtOfCode

1:03 AM

Aye

may have missed the fact that our coursework is in haskell for the past three months and now have three weeks to learn it and do coursework

Undo

hah

WELZ

֑

SmokeDetector

[ SmokeDetector | MS ] Shortened URL in answer: basement framing with metal Studs Best way to fasten them to the floor by Jim Watson on diy.SE

fp- by WELZ

K.Dᴀᴠɪs

I don't get why people use shortened URLs for legit sites here.

SmokeDetector

1:27 AM

[ metasmoke ] ci/circleci failure on b1da0b6: Your tests failed on CircleCI

[ metasmoke ] ci/circleci failure on f877204: Your tests failed on CircleCI

quartata

@ArtOfCode let me know if you have questions

ArtOfCode

What is life?

How big is the universe?

What is the largest object known to man?

oh haskell questions :)

probably many

SmokeDetector

[ metasmoke ] ci/circleci failure on 5be5519: Your tests failed on CircleCI

iBug

@Undo MS record 108024 was created for testing purposes. Good to destroy?

Undo

You sure?

ArtOfCode

1:37 AM

looks like a TP to me

iBug

Yes. I was testing my new command.

Undo

o.O

iBug

See the last line of "why".

ArtOfCode

still caught a useful post, though :P

Undo

oh, I see. Duplicate

iBug

1:38 AM

don't we cleanup tests?

ArtOfCode

sometimes

Undo

yeah. @ArtOfCode he ran it on an already-caught post.

108022 was the original

ArtOfCode

@Undo aye, I hit the domain list

iBug

@Undo If you search the context you'll find I !!/scanned two new reports

ArtOfCode

don't destroy

Undo

1:38 AM

gone

quartata

you didn't need to do that fwiw

Undo

... too late

quartata

I had already tested it on real posts

iBug

Strangely, one got "does not look like spam", the other went to 108024

Undo

actually, never mind :P. @ArtOfCode review results keep us from nuking posts.

quartata

1:39 AM

need a dependent

ArtOfCode

@Undo hehe. I got it from local cache, anyway.

iBug

@quartata Didn't know that, thanks anyway

ArtOfCode

Anyway, there might be a bug there somewhere

those two records have different weights

same text

Undo

blacklisted user

ArtOfCode

oh

rolls eyes

SmokeDetector

1:44 AM

[ metasmoke ] ci/circleci error on dfa468c: Your CircleCI tests were canceled

[ metasmoke ] ci/circleci success on dfa468c: Your tests passed on CircleCI!

ArtOfCode

\o/

Undo

There we go. When you get a chance, github.com/Charcoal-SE/metasmoke/pull/331

ArtOfCode

Heh, the comparison was already open in another tab :P

K.Dᴀᴠɪs

@AJ stackapps.com/questions/7033/…

8

Q: A clarification request for declined spam flag

I flagged this post for obvious spam link inserted in the end when posted in Charcoal HQ. Check the first version here. After looking at the user's profile and the link in the end, I flagged it as spam. I also posted a comment asking the user to state any affiliation to the site. In reply to tha...

Never tried it, but it looks promising

SmokeDetector

2:03 AM

[ SmokeDetector | MS ] Bad NS for domain in answer: Remove / Rename Uncategorized Category In Wordpress by Sahadath Hossain on wordpress.SE

[ metasmoke ] ci/circleci failure on b453c16: Your tests failed on CircleCI

Undo

>:(

ArtOfCode

Bahahaha one char over :D

SmokeDetector

[ metasmoke ] ci/circleci success on 4fbb960: Your tests passed on CircleCI!

ArtOfCode

Gonna deploy that tonight?

...says he, as it's already deployed

Undo

Deployed, threshold set to 10,000

So it's disabled. Whoever is awake when we start getting spam next can try it out. Probably me or you

SmokeDetector

2:14 AM

[ metasmoke ] ci/circleci success on b15ce72: Your tests passed on CircleCI!

ArtOfCode

@Undo spam hour's only a couple hours out

Undo

I should be around

ArtOfCode

also I may have broken metasmoke... how big is the prod log at the moment?

Undo

Usually weekends are iffy

ArtOfCode

@Undo Monday morning should be okay

Undo

2:16 AM

1.7G

iffy for spam volume, I mean. Not my schedule.

ArtOfCode

@Undo yeah, maybe grepping through that wasn't the best idea I've ever had

Miiiight have to pipe tail output to it first

Undo

hah

tail -n 10000 or something

ArtOfCode

aye

Undo

In fairness, it shouldn't be that big. I just can't figure out why s3cmd doesn't want to work.

ArtOfCode

uh

bahahahaha

you're gonna need to kill the server :D

that request is literally never going to return because it's waiting on the command line for input

grep -E '/api/v2.0/posts/[0-9]+/flag' -C 30 --color=never

SmokeDetector

2:23 AM

Restart: API quota is 17126.

fp by ArtOfCode

[ SmokeDetector | MS ] No whitespace in answer: Is this an instance of personification or is it a simile? by fgvnhf on english.SE

Undo

Restarted nginx

SmokeDetector

tpu- by Tetsuya Yamamoto

ArtOfCode

@Undo Oughtta do it. There's a commit there to make that not happen now :)

I think that takes the cake for the most I've managed to brick metasmoke

Undo

At least grep hanging isn't really resource heavy :P

deploying

SmokeDetector

[ metasmoke ] ci/circleci success on 76f18a5: Your tests passed on CircleCI!

Undo

2:26 AM

done

WELZ

@ArtOfCode just wondering (for my own personal use) how much were the stickers? (I'm wondering what specs you did as well)

ArtOfCode

@WELZ in total? Both orgs?

WELZ

just one I guess

ArtOfCode

@WELZ one org would've been £90

WELZ

how many stickers was it?

ArtOfCode

2:33 AM

340 per org

Total was £157.61, because they give you a discount for doing multiple designs in one order

A J

@K.Dᴀᴠɪs Thanks will check it for sure.

WELZ

and they were 2x2 die cut stickers?

ArtOfCode

@WELZ 5x5 cm

so yeah, 2x2 in

Undo

@ArtOfCode Is that minimum order?

ArtOfCode

They adjusted the sizes slightly to get the proportions right, but at least one side is 50mm

WELZ

2:35 AM

@Undo no, it was just extra for money left

ArtOfCode

@Undo nah, minimum is 10, 340 is just what we had the money for

WELZ

minimum is 10

Undo

Does it scale linearly down to 10?

That'd surprise me

ArtOfCode

Cost? Nope. Dunno what the algorithm is, but it's not linear

Undo

Yeah. They have tooling cost or something

ArtOfCode

2:36 AM

£13 for 10

WELZ

The more you order, the less you pay per sticker. (as with just about everything)

Undo

Probably have to make the die to cut them.

WELZ

or $17 usd

ArtOfCode

(but if you want 10 just get a sample pack for about £7)

Undo

How many sample packs can you get? :P

ArtOfCode

2:37 AM

50 is £46, 100 is £55

@Undo Not more than one for the same design in one order, at least :P

SmokeDetector

[ SmokeDetector | MS ] Bad keyword in title: how to Watch The Walking Dead Season 8 Episode 8 Online by nafodis on scifi.SE (@Mithrandir)

tpu- by WELZ

WELZ

aha! this wasn't caught for sites.google...

SmokeDetector

2:57 AM

[ SmokeDetector | MS ] Repeating characters in title: My main partition goes FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF by DaveMartinezMX on apple.SE

iBug

@WELZ It was removed

SmokeDetector

fp- by Tetsuya Yamamoto

iBug

@SmokeDetector f

WELZ

@iBug it was just funny that right after being removed... (at least it was caught for something else - though it was obviously spam, so prob would've been caught anyhow)

iBug

The Chinese name of Winnie the Pooh (a Disney cartoon character, I bet you all know) is banned in China (blacklisted keyword)

quartata

3:24 AM

@SmokeDetector my main partition goes skraaaaaaa

Tetsuya Yamamoto

@iBug Just banned in mainland or includes HK too? Do you know which reasons behind banning that term?

Rob

3:35 AM

@iBug Hmm.. and to think, I get frustrated at my ISP for throttling youtube at peak time :\

Or more dishonestly.. unthrottling all data to speed tests

@iBug A couple options. One could register their FKey with MS, rather than just the oauth token. Or potentially less dangerous; a userscript which detects flagged posts which need to be retracted

Granted, the second option requires people to be online

Makyen

@Rob Just a machine which is on and has an internet connection through which they are logged in, which could be a mobile device. Doing it on the user's equipment isn't as convenient as having it done via MS, but it's certainly not that difficult.

SmokeDetector

[ SmokeDetector | MS ] Bad keyword in title: How to host machine learning in alibaba cloud by Bhargava Ganti on stackoverflow.com

fp- by K.Dᴀᴠɪs

Rob

@Makyen Well, yeah, that could work. Though then we'd need mobile apps, desktop apps, and so on. Rather than just a userscript

And a further alternative, if we're worried about nuking non-spam, would be to post a message to a mod chat room, so that the flags can be declined/cleared

SmokeDetector

3:52 AM

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, potentially bad keyword in body: Passcode lock feature to prevent iPhone from being by Shf ldvhj on apple.SE

Undo

@Rob fkey also requires cookies. It's not feasible

Makyen

@Rob You can use userscripts with Firefox for Android.

SmokeDetector

tpu- by Makyen

Rob

@Undo Ah, right

SmokeDetector

[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +1 more: fornatgaex.com/primal-alpha-beast/ by spmlxhgvap on meta.SE

tpu- by Makyen

Undo

4:03 AM

Turned on five flag experiment. If it goes nuts, !!/stopflagging

13

Should flag posts over weight 300 with a random number of flags between 1 and 5, in an attempt to collect deletion time statistics.

SmokeDetector

[ SmokeDetector | MS ] Mostly non-Latin answer: ASP.NET MVC 3 Adding comments in article view by Ruchi Kumari on stackoverflow.com

tpu- by Rob

[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +3 more: realcoloncleansingworks.com/andras-fiber/ by guro on askubuntu.com

tpu- by Undo

[ SmokeDetector | MS ] Bad keyword with email in body: javascript code to white list gmail and yahoo email users for registration by G.D on wordpress.SE

fp- by Makyen

[ SmokeDetector | MS ] URL-only title, link at end of body: fitnessexperttips.com/elite-biotics/ by warunqwik on askubuntu.com

tpu- by Makyen

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, pattern-matching website in body, potentially bad keyword in body: ny way to Add a reminder more t ahead of a Goo event by user179510 on webapps.SE

tpu- by Makyen

Undo

4:26 AM

@SmokeDetector Four flags on that one

11 second deletion time

Makyen

@Undo Yep. It was gone well before I provided feedback/tried to flag.

iBug

@Undo Testing or random mistake?

Undo

Testing

autoflagging doesn't make random mistakes :)

iBug

@Undo Are posts randomly chosen for testing, or scheduled?

Undo

weight >= 300; rand(1..5) flags

iBug

4:31 AM

All that are over 300 are chosen?

Undo

yes

K.Dᴀᴠɪs

Just curious. Is there any particular benefit of using a random number of flags?

iBug

@K.Dᴀᴠɪs To collect the data of the correlation between flag count and TTL.

Undo

Want to get numbers on deletion time relation to flag count. We don't have statistically valid data on that across large numbers of posts.

Best we have is from the 0 -> 1 and 1 -> 3 moves, both time-delimited.

K.Dᴀᴠɪs

That makes sense

Are these flags manually deleted?

Undo

4:35 AM

Hm?

iBug

@TetsuyaYamamoto Probably HK not affected. It's because Winnie is "visually similar" to the current chairman, whose proposal to remove the limit of consecutive terms that a chairman can serve for.

People have used Winnie to refer to the chairman, Xi, as a circumvention of keyword filters.

SmokeDetector

[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +4 more: nutritionofhealth.co.uk/total-age-repair/ by user799642 on askubuntu.com

tpu- by K.Dᴀᴠɪs

Undo

@SmokeDetector One flag, 23 seconds

SmokeDetector

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, +2 more: Your body renews every day a kilogram of muscle by ubxkgcqcu on askubuntu.com

tpu- by K.Dᴀᴠɪs

iBug

4:59 AM

@Undo How are flagging candidates chosen?

We have 2 core + 1 any for 3, but what about 1 2 4 5?

Undo

@iBug github.com/Charcoal-SE/metasmoke/blob/master/app/models/…

SmokeDetector

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, +1 more: Enduraflex Canada Colored contact by andrflt12 on askubuntu.com

[ SmokeDetector | MS ] URL in title, pattern-matching website in body, pattern-matching website in title: wellnessdietsolution.com/elite-biotics/ by Rinengex on english.SE

tpu- by Makyen

tpu- by Makyen on Enduraflex Canada Colored contact [MS]

iBug

ELU and Workplace also in test?

Undo

every site.

oh, well, not those.

iBug

Looks no.

github.com/Charcoal-SE/metasmoke/blob/master/app/models/…

Only 6-flag sites are in test

Undo

5:06 AM

yeah

SmokeDetector

[ SmokeDetector | MS ] Blacklisted username, offensive answer detected: Potentiometer to change voltage? by niggery on electronics.SE

tripleee

!!/watch 9785347272

SmokeDetector

@tripleee Added 9785347272 to watchlist

[ SmokeDetector | MS ] URL in title, link at end of body: fitnessexperttips.com/elite-biotics/ by Tayastillo on drupal.SE

tpu- by tripleee on Potentiometer to change voltage? [MS]

tpu- by tripleee

CI on c26d06d succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev c26d06d (SmokeDetector: Auto watch of 9785347272 by tripleee --autopull) (running on teward/Lunar Eclipse)

Restart: API quota is 16187.

tripleee

!!/watch champcash

SmokeDetector

@tripleee Added champcash to watchlist

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, pattern-matching website in body, repeated URL at end of long post: Spread your protein over 3 to 4 shots a day and you're done! by ubxkgcqcu on apple.SE

tpu- by K.Dᴀᴠɪs

tripleee

5:18 AM

!!/watch fitnessexperttips\.com

SmokeDetector

@tripleee Added fitnessexperttips\.com to watchlist

tripleee

!!/blacklist-keyword elite\W?biotics

SmokeDetector

@tripleee Blacklisted elite\W?biotics

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link following arrow in body, pattern-matching website in body: Tryvix Creme, at that point you will be happy by Arleneberg on graphicdesign.SE (@TetsuyaYamamoto)

tpu- by K.Dᴀᴠɪs

CI on d135020 succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev d135020 (SmokeDetector: Auto blacklist of elite\W?biotics by tripleee --autopull) (running on teward/Lunar Eclipse)

Restart: API quota is 16169.

K.Dᴀᴠɪs

So in the event an autoflagger is not set up for a site, how does that work?

Does MS throw an exception? Does it think it went through?

angussidney

The API throws an error and MS just tries to flag using a different account

iBug

5:26 AM

@K.Dᴀᴠɪs Use another flagger

SmokeDetector

5:37 AM

[ SmokeDetector | MS ] Bad keyword with a link in answer, link at end of answer: What do you call these snips/shears? by Mike on crafts.SE

tpu- by tripleee

Suraj Rao

@SmokeDetector k

Laff-O-Tron

[open] Hi o/

tripleee

!!/watch forgica\.com

SmokeDetector

@tripleee Added forgica\.com to watchlist

CI on 23e4286 succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev 23e4286 (SmokeDetector: Auto watch of forgica\.com by tripleee --autopull) (running on teward/Lunar Eclipse)

Restart: API quota is 15962.

iBug

5:54 AM

how do you pronounce git

"dgit" or "ghit"?

SmokeDetector

[ SmokeDetector | MS ] Pattern-matching website in body: Getting scraped data using api with python by Ismail Chouiti on stackoverflow.com

fp- by tripleee

tripleee

@iBug your orthography doesn't look correct for how I pronounce it. In Italian /gh/ is used to mark a hard g (the normal is for g to be soft before e and i) but in some other languages it specifically spells a soft g, or an uvular R, or a number of other sounds ... but I guess this is what you are getting at?

hard g, like the English word git

merriam-webster.com/dictionary/git has a speaker icon you can click on

SmokeDetector

[ SmokeDetector | MS ] Blacklisted username: How can i show vieabag values to table automatically using jqury in asp.net mvc by Akhil Bajjuri on stackoverflow.com

Transcript for