Charcoal HQ

@SmokeDetector v

[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +4 more: tophealthmart.com/enduraflex/ by tyfiugiu on askubuntu.com

tpu- by iBug

Can anyone find me a post in MS that has weight of 199?

11:17 AM

There seems to be a significant dip in the speed of voting in SO primaries between 23 PM and 3 AM. Have people been sleeping the entire time?

Have I killed MS?

@CalvT븃 it's down

downforeveryoneorjustme.com/metasmoke.erwaysoftware.com

Maybe it didn't like me trying to fetch the last 1M posts in data explorer :-(

Not sure if it was my fault or not - was on the post tab going through the pages

cc @Undo @ArtOfCode

an SQL interface would be much appreciated

11:19 AM

[ SmokeDetector | MS ] Blacklisted website in body, blacklisted user: How smart contracts will help reduce inefficiencies in the healthcare industry by Gaurav Vinaik on bitcoin.SE

sd k

sd k

@SmokeDetector k

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, link following arrow in body, +3 more: HOW DOES EnduraFlex WORKS? by Hoper 19 on drupal.SE

ERR_CONNECTION_REFUSED

11:26 AM

[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, link at end of body, pattern-matching website in body, repeated URL at end of long post: whey, soy and rice are regular parts of sustenance bars? by endeaun on askubuntu.com

panic = true

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link following arrow in body: TryVexan Men Execution Upgrade pill by Marlaaher on graphicdesign.SE

MS API reliably down?

Let's all calm down and eat some roadkill-pasta

eat some waffles

11:29 AM

@Henders I will eat some Churma that I saved from lunch. :P

Petter Friberg

both metasmoke and sentinel seems offline

[ SmokeDetector | MS ] URL in title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +5 more: getsupplementhelp.com/vigoriax/ by mpdyvexvs on apple.SE

@JohnDvorak I can give you a data dump if you want

@ArtOfCode me too please! but getting this routinely automated would be great so that we could just grab something reasonably fresh whenever we feel the need

It is automated. Someone just needs to make it accessible to something lower than developers

I'll grab a dump for both of you as soon as MS is back up

11:40 AM

Why does repeated scanning not work?

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, +3 more: Tryvexan Male Enhancement: boost up reproductive system of men by niamhharris on askubuntu.com

@SmokeDetector k

@SmokeDetector k

why was that not autoflagged?

@SmokeDetector autoflagged

@JAD I hit an error while trying to run that command; run !!/errorlogs for details.

!!/errorlogs

11:42 AM

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-03-20 11:38:35.613048 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-03-20 11:39:05.700193 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-03-20 11:39:35.730562 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.

uh oh

ok, so, metasmoke's down so smokey's broken

@SmokeDetector k

Ok, I know people have asked it, (I just came on) but what happened to MS?

@SmokeDetector k

Mithrandir

Cool, panic time

11:43 AM

ahjhhhhhhhhhhhhhhhhhh

Eyyyyyyyyyyyy

@WELZ goto :43508502

Mithrandir

I don't remember - are we going to have instances switching under our feet or not?

@tripleee huh

[everything-is-broken] [panic-driven-development] [dev-on-prod]

11:47 AM

[undo-is-sleeping] :D

[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +5 more: top10sideeffects.com/premier-t-boost/ by Exaled64 on askubuntu.com

@SmokeDetector k

oh good

We're manually flagging spam now?

@WELZ Don't think that'll do anything

11:49 AM

at least smokey is still alive

All smokey can do now is report

good point.

No failover, no autoflags

@Glorfindel I got the access to that room back.

!!/blame᠎ ‍⁣᠎⁣‌

11:54 AM

@WELZ It's CalvT븃's fault.

Probably - either me or John

This will actually be good for TTD data

Can we monitor TTD atm then?

[ SmokeDetector | MS ] Bad keyword in link text in answer, blacklisted website in answer: Static IP address on Wi-Fi at work and dynamic at home? by john on superuser.com

@CalvT븃 probably not :S Lol

12:05 PM

@SmokeDetector k

Glorfindel

@AJ great, thx

@WELZ endless loop

[ SmokeDetector | MS ] Bad keyword in title: Amazon customer service number by user386422 on meta.SE

sd k

@SmokeDetector k

12:17 PM

I'm a little worried about folks feedbacking, that may just cause a huge backlog once it does come online. (maybe I'm wrong, idk too much about this)

idk, too

@SmokeDetector k

Magisch

@WELZ What do you mean?

I suppose the assumption is that the feedbacks stay in a queue somewhere, but I don't believe that is the case

I just submitted two new PRs.

12:25 PM

@Glorfindel Well, you learned something new today. ;-P

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, +2 more: In spite of sex charge, adaptability to case penis go settle by Lifere19 Virk on superuser.com

Did anyone ping Undo and tell him MS was down?

Wait, MS is totally down?

yep

@Andy yes chat.stackexchange.com/transcript/message/43507934#43507934

12:30 PM

Ok. Then I won't do so as well.

It was up at 11:10 UTC, at least.

@SmokeDetector @iBug

seems to have crashed not long after

uh, 11:36

yh

[ SmokeDetector | MS ] Blacklisted website in body, pattern-matching website in body: Renown Skin Labs by MarinaShorter on apple.SE

12:34 PM

There's no more autoflags...

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, bad keyword in username, bad pattern in URL body, blacklisted website in body, +1 more: XLR8 Plus Review: Natural Nootropic Supplement| Buy Online by XLR8 Plus on superuser.com

so the spammers first brought down MetaSmoke and are now bombarding the site with their spam payloads

sd 2k

@rene Interesting theory XD

that sounds somewhat exciting actually

Thanks, I'm a conspiratist

12:37 PM

@Andy I think he's probably sleeping o.O

s 2k

sd 2k

@WELZ Probably, but I didn't need to throw another ping at him if he'd already been informed.

ping him till he's awake

@JAD That's what I'm thinking :S

honestly, I'd guess the chat pings (if you can call them that) would wake most people pretty quickly

12:40 PM

b*st*rd sleeps with sound muted

why don't we have 2 sysadmins for MS?

I think Art would be a good MS sysadmin

@iBug because that would be 2 leaders without a hierarchy, and they would kill eachother

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, pattern-matching website in body, repeated URL at end of long post: It contains commonly all the natural basicsgive of substance, by Lifere19 Virk on graphicdesign.SE

it's been brought up in the past, the technical stumbling stone was that Undo didn't want to make the access tokens exposed to more people than just himself IIRC

but I'm unclear on how hard a constraint this is; we should perhaps try to figure out a workaround or an implementation where this particular information would not have to be leaked to whoever is running an MS instance

After this.... it may be reconsidered.

12:43 PM

this is not a bigger catastrophe than any of the previous times this happened

And because MS being down is a rare problem. An outage is not the end of the world. Smokey is still working. We're just slower at removing spam

but it does seem to bring a fair amount of frustration and bad atmosphere every time

and a fair amount of panic

[ SmokeDetector | MS ] Few unique characters in body, repeating words in body: Journal reccommendation: review of methods in medical statistics by Pub on academia.SE

@SmokeDetector v

12:44 PM

@SmokeDetector v

Magisch

@Andy I reckon at least 4 times since the last autoflagging post

I've been meaning to bring this up in Channels so perhaps now would be a good time

@SmokeDetector v

anyway, see you tomorrow

I'm going to do my homework now

@tripleee where is the SLA and the contract that states the fines for not meeting said SLA? Let's sue them!

12:49 PM

@rene yeah, exactly my line of thinking

I'll let my lawyer handle this ...

@tripleee The access tokens are transferable, my trust in Undo isn't.

[ SmokeDetector | MS ] Bad keyword in answer: How Gmail spam filter works? by JMoser on stackoverflow.com

sd f

@SmokeDetector why

@tripleee Body - Position 377-383: Viagra, Position 390-406: Male Enhancement

12:54 PM

that's really interesting

it isn't spam, but it includes all the spam keywords Lol

yep, that is a good guess by that OP, we can confirm

yeah, it's the sort of thing you just have to live with as long as your architecture is keyword-based

> The output classification is "Spam" or "Not Spam," and the inputs, I'm sure, are top secret at Google, but I'm sure certain email text phrases such as "Buy Now," "On Sale," "Viagra," or "Male Enhancement" are all factors in their model.

No, that's our classification!

:D

[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, link at end of body, pattern-matching website in body, +1 more: I was insignificant to shut end speculation organization by Lifere19 Virk on drupal.SE

12:59 PM

@WELZ heh, we pretty much already have that

@ArtOfCode and you guys fight like dogs every day!!! :P

pun intended

I'm not a dog :P

Rob

@ArtOfCode Stickers have made it to Australia! Thanks very much for your time getting this all organized, and thanks @Catija as well :)

@Rob \o/

@ArtOfCode you're the alpha--threat

1:01 PM

@ArtOfCode you'd think "un-dog" isn't either

[ SmokeDetector | MS ] Repeating characters in title, title has only one unique char: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx by Hassan on electronics.SE

sd v

[ SmokeDetector | MS ] Repeating characters in title, title has only one unique char: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx by Hassan on electronics.SE

sd v

[ SmokeDetector | MS ] Bad keyword with email in answer: What can I do with inconsistent sentiment detection provided by cognitive services? by Ferdinand Fejskid on stackoverflow.com

1:07 PM

@SmokeDetector why

@tripleee Post - Keyword spell with email mlapi@microsoft.com

Magisch

@ArtOfCode you're a .... shape?

ghost in the machine

@SmokeDetector f

[ SmokeDetector | MS ] Username similar to website in answer: Change entity position based on which link is clicked by deckewandboden on stackoverflow.com

@SmokeDetector naa

1:13 PM

@PrincessLuna Recorded answer as an NAA in metasmoke.

@SmokeDetector in metasmoke?! you lying b*st%rd

Relax, he's just blowing smoke

Lmao

@PrincessLuna instantrimshot.com

lol

1:16 PM

[ SmokeDetector | MS ] Bad keyword in body: CSV inputs to Variables by Zachary Stansfield on stackoverflow.com

@SmokeDetector why

@JAD Body - Position 854-862: alphaMax

hah

@SmokeDetector f

@SmokeDetector fp

1:17 PM

@PrincessLuna Registered question as false positive.

not in metasmoke this time

... What got broken this time?

FIRE stopped working for me

@SmokeDetector naa

@WELZ Recorded answer as an NAA in metasmoke.

... load of websocket failures

1:18 PM

@JAD it has to do with the feedback type, it's weeeeeeeird

@Cerbrus MS is down

\o I'm back all. Watch out

I see

@K.Dᴀᴠɪs What should I watch? ;)

@AJ out

just out

so !!/watch out?

1:19 PM

@WELZ What he said @AJ

that might give some fps

@JAD bunch of smartasses this morning ;P

@K.Dᴀᴠɪs Is this? :P

:(

@AJ chutzpah xD

1:20 PM

@AJ Yes, definitely watch that

lol

Hmmm FIRE and SDS not working

@K.Dᴀᴠɪs yes, MS is down.

Ahhh hard drive space again?

[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +5 more: getsupplementhelp.com/vigoriax/ by monadi on askubuntu.com

1:23 PM

MS is down, please have patience ...

@CalvT븃 since this time

@tripleee Most of us are not known for our patience. You will lose your patience too if you watched Out

inside joke if you weren't here for the last 5 mins

I saw that yea, sorry for the noise, just realized we probably needed something on the starboard and I couldn't find anything suitable to pin

@K.Dᴀᴠɪs now you ruined it 😒

@WELZ I'm good at that

apparently. :P

1:26 PM

clearly

like when WELZ wanted to explain my own joke to me (^8

[ SmokeDetector | MS ] No whitespace in answer: Debug adapter process has terminated unexpectedly? by Walid on stackoverflow.com

Has Rob broken 10k yet?

@K.Dᴀᴠɪs 6k atm

@tripleee I don't recall, when was that :O

1:27 PM

@K.Dᴀᴠɪs 6690

Nice

@tripleee Is it back yet?

@Cerbrus nope, we are waiting for Undo to wake up

@tripleee Is he back yet?

What about now, @tripleee?

1:29 PM

:D

[ SmokeDetector | MS ] Few unique characters in body, repeating words in body, repeating words in title: DELETE.DELETE.DELETE.DELETE.DELETE.DELETE.DELETE.DELETE.DELETE.DELETE.DELETE.DE‌LETE.DELETE by Pub on academia.SE

wait

@SmokeDetector tp-

@SmokeDetector v

@SmokeDetector tp-

1:30 PM

@chade_ You are not a privileged user. Please see the privileges wiki page for information on what privileges are and what is expected of privileged users.

oh right, only on SO ^^

@chade_ Nah they revoked that too :p

@K.Dᴀᴠɪs Of course. wouldn't know for what

Everyone ping @Undo. @Undo @Undo @Undo fix it

@Undo We need moar disk space :D

1:32 PM

w3 neeed teh codez

@K.Dᴀᴠɪs No da hard drivezz

Guys, it's not nice to ping him like that, he's been pinged multiple times and he'll see it when he logs in.

2

Give undo space - he's been pinged already

@chade_ moar?

hu+r+y+\s(?:up)?\sfix\it

1:34 PM

somebody with the privileges, please remove those extra pings ^

moar

Adverb: moar (not comparable)

(Internet slang, humorous) Alternative form of more

Noun: moar

(hypocoristic, usually childish) Mother, female parent.
moar m
plural indefinite of mo
moar
indefinite plural of mo...

Verb: moar

present tense of moa and moe

2

@WELZ what WELZ said

"w3 neeed moar 0f teh codez" = "I go to stackoverflow to ask for teh freee codez"

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, link following arrow in body, +2 more: Are there any EnduraFlex side effects? by Aden 19 on workplace.SE

@WELZ Do that and expect downvotes and a quick close

1:42 PM

@WELZ Are you saying he doesn't show up if you say his name three times?

@chade_ and also free codez from rep hogs.

@Cerbrus They said it four - broke the magic spell

Now the spell needs a reboot before it can work again

[Shouts from the top of the mountain] "I SUMMON UNDO"

2

@WELZ What mountain?

The mountain made from Charcoal, duh!

1:44 PM

@CalvT븃 Obviously they didn't read the instructions: "Four shalt thou not count..."

!!/whois_admin

@CalvT븃 No such command 'whois_admin'.

out of all the burnt spam

!!/whois admin

@CalvT븃 I hit an error while trying to run that command; run !!/errorlogs for details.

1:44 PM

@Cerbrus you are confusing him with @ArtOfCode

@SmokeDetector bahahaha

Undo's here anyway :)

last seen 3m ago

2:03 PM

!!/location

@Henders teward/Solar Flare

!!/whois admin

@CalvT븃 I hit an error while trying to run that command; run !!/errorlogs for details.

!!/errorlogs 50

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-03-20 14:01:47.283865 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-03-20 14:02:17.343337 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-03-20 14:02:47.354024 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.

[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, link at end of body, link following arrow in body, pattern-matching website in body, +1 more: Are there any side effects? by Evic 19 on superuser.com

2:07 PM

@SmokeDetector k

It's not disk space. Something just became unhappy

wHAAAA UnDo

Saving the dayz

@ArtOfCode I think I'm gonna blame Data Explorer:

I, [2018-03-20T11:14:32.248211 #20913]  INFO -- : Started GET "/data/retrieve?types%5B%5D=posts&limits%5Bposts%5D=1000000" for x.y.z.a at 2018-03-20 11:14:32 +0000
I, [2018-03-20T11:14:32.249012 #20913]  INFO -- : Processing by DataController#retrieve as */*
I, [2018-03-20T11:14:32.249064 #20913]  INFO -- :   Parameters: {"types"=>["posts"], "limits"=>{"posts"=>"1000000"}}
D, [2018-03-20T11:14:32.250691 #20913] DEBUG -- :   ^[[1m^[[36mSiteSetting Load (0.7ms)^[[0m  ^[[1m^[[34mSELECT  `site_settings`.* FROM `site_settings` WHERE `site_settings`.`name` = 'require_auth_all_pages' LIMIT 1^[[0m

3 hours ago, by John Dvorak

Maybe it didn't like me trying to fetch the last 1M posts in data explorer :-(

also blaming @JohnDvorak, then :P

2

Can probably put a timeout on those queries or something.

Looks like it ate all the memory, triggered the OOM killer. Redis, nginx, and MySQL were all dead.

2:15 PM

ouch

I mean, if it's gonna fail, that's a decent way to do it

!!/blame

@JAD It's Mithrandir's fault.

Smokey disagrees :/

!!/blame᠎ ‍‌‌⁣⁣

2:16 PM

@CalvT븃 It's John Dvorak's fault.

@JAD :P

Mithrandir

@CalvT븃 That command checks MS for who has the role, so if MS is down...

@SmokeDetector :(

@Mithrandir Aye, that's why I was running it - a simple way to check from chat basically

@Undo I prefer catastrophic failure over half measures :P

@Undo thanks for saving us

2:19 PM

@Henders so hook up any exception with a call to rm -rf /

If it fails, start with a clean slate :D

FIRE and SDS still appear to be down

@K.Dᴀᴠɪs working by me

hmm

but all of the posts reported before have no data in MS

I don't even see the FIRE report button on the posts... And SDS at all

2:29 PM

New metasmoke user 'PhillipChers' created

MS has been up for all of about 10 minutes and we already have a new user!

2:44 PM

@Catija stay safe in Austin (with all these package bombs... scary)

@JAD Or at least a random chance.

heh, I like it. keeps you on edge

Hey guys, how do you report to Smokey stuff that's already been nuked? Or does report work either way?

@Machavity it'll complain about not being able to find the post

so !!/report doesn't work

Can someone add stockmetrix.net to the blacklist then? Had some soft spam float by but a mod caught it before I could report

2:54 PM

@Machavity Best thing is to post the text in here, let the regex wizards figure out why it wasn't caught

This is the post in question

[ SmokeDetector | MS ] Bad keyword with email in answer, email in answer: How do I download different blocks of bitcoins blockchain by Diego Lucas on bitcoin.SE

tpu- by PeterJ

PeterJ

Hey my FDSC worked now :)

@Machavity can you please screenshot it for us under 10k folks

3:01 PM

[ SmokeDetector | MS ] Bad keyword with email in answer, email in answer, blacklisted user: What would happen if two miners were mining two blocks with two different set of transactions? by Diego Lucas on bitcoin.SE

tpu- by WELZ

Keep in mind that the whole Q was strange. Closed the Q as Off-Topic but the asker and the three answers were all by new accounts. The other two looked NAA but this one stood out

!!/watch stockmetrix\.net

@WELZ Added stockmetrix\.net to watchlist

Mod nuked all answers and Q as spam, so probably more fishy stuff behind the scenes there

@WELZ Thanks

[ SmokeDetector | MS ] Bad keyword with email in body, blacklisted user: Bitcoin Mining ETH sales by Diego Lucas on bitcoin.SE

tpu- by PeterJ

CI on 6e99bae succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev 6e99bae (SmokeDetector: Auto watch of stockmetrix\.net by WELZ --autopull) (running on teward/Solar Flare)

PeterJ

3:05 PM

!!/report bitcoin.stackexchange.com/questions/72667

Restart: API quota is 7181.

[ SmokeDetector | MS ] Manually reported question: Bitcoin Mining ETH sales by Diego Lucas on bitcoin.SE

@SmokeDetector k

3:19 PM

@SmokeDetector K

!!/scan bitcoin.stackexchange.com/questions/72667

@WELZ Post Bitcoin Mining ETH sales does not look like spam.

@WELZ mister scanner, if the question did match a reason, it would have been posted

API quota rolled over with 6983 requests remaining. Current quota: 19999.

math: 43
stackoverflow: 41
superuser: 10
physics: 8
unix: 5
serverfault: 5
askubuntu: 5
english: 4
codegolf: 4
skeptics: 4
apple: 4
drupal: 3
es.stackoverflow: 3
academia: 3
chemistry: 3
rpg: 3
salesforce: 2
anime: 2
mathematica: 2
sharepoint: 2
security: 2
pm: 2
ell: 2
stats: 1
mythology: 1
quant: 1
sqa: 1
mathoverflow.net: 1
puzzling: 1
tex: 1
retrocomputing: 1
gamedev: 1
interpersonal: 1
japanese: 1
codereview: 1
writing: 1
gaming: 1
chinese: 1
dba: 1
ru.stackoverflow: 1
expatriates: 1
spanish: 1

@SmokeDetector This API quota is a real parkour expert

3:21 PM

@M.A.R.ಠ_ಠ It might have missed scanning

Glorfindel said there is no good way to check either.

Well, Smokey is increasingly human

So mayhaps

!!/watch bitcoin\W?global\.?org

@WELZ Added bitcoin\W?global\.?org to watchlist

CI on 5b455cb succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev 5b455cb (SmokeDetector: Auto watch of bitcoin\W?global\.?org by WELZ --autopull) (running on teward/Solar Flare)

Restart: API quota is 19951.

[ SmokeDetector | MS ] Bad keyword in body, potentially bad keyword in body, potentially bad keyword in title: Produk Dewasa Di Jakarta 081390514444 by chin hooo on coffee.SE

graphic ^

3:37 PM

tpu- by WELZ

Happy Nowruz all!

It's something you don't know about but you should feel good anyway