[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +2 more: bettercoloncleansingguide.com/nutra-prime-cleanse/ by taniyshqik on puzzling.SE (@Mithrandir)

@SmokeDetector k

Appears MS is down again

That or Makyen running another query

Works for me

[ SmokeDetector | MS ] Potentially bad keyword in answer: Why did Ross Ulbricht only use public WiFi? by msdundar on tor.SE

sd f

hmmm can't feedback in FIRE.

sd f

Now I got an error at MS link

@K.Dᴀᴠɪs yep, MS seems slow

@SFTP And you wanted to call me a liar :)

We'll have to feedback in chat like savages. If that even does anything without MS.

Huh: Mysql2::Error: Lost connection to MySQL server during query

Is the user blacklist local to Smokey?

yes

@SFTP Yep

Which means, by the way, that it doesn't sync between instances

MS seems to be loading now...still a bit sluggish though

I think that instances should switch out every hour or so just to pull an update

Which would mean in the worst case scenario that an instance is ~ 7 hours behind

I don't think it's worth it -- it's not that hard to !!/pull

What if someone isn't paying attention, as in most of every one is asleep

I would be fine with posting a message on failover saying "This instance is x commits behind," but I don't think we want any other automatic instance updates

@GrantGarrison It doesn't show users that gained reputation. It shows users who had > 1 reputation when it was reported.

teward/Solar Flare received failover signal. (@quartata)

Yup.. Called it, Makyen running a query.

@GrantGarrison Are you doing large queries (i.e. with a large number of records)?

Restart: API quota is 19999.

everything's red :/

!!/location

@K.Dᴀᴠɪs teward/Solar Flare

@K.Dᴀᴠɪs Henders/EC2

!!/standby henders

@K.Dᴀᴠɪs My queries have been small (e.g. 50 or 500). I've tried slightly larger, but they fail.

[ SmokeDetector ] SmokeDetector started in standby mode at rev ba5809f (metasmoke: Merge pull request #1966 from Charcoal-SE/auto-blacklist-1522985900.7603998 --autopull) (running on Henders/EC2)

@Undo

!!/alive

@K.Dᴀᴠɪs Watching this endless list of new questions never gets boring

got two with pings now

!!/rev

@K.Dᴀᴠɪs teward/Solar Flare 2a4d55c (ArtOfCode: Merge pull request #1951 from NobodyNada/patch-3)

!!/pull

that's old

[ SmokeDetector ] SmokeDetector started at rev ba5809f (metasmoke: Merge pull request #1966 from Charcoal-SE/auto-blacklist-1522985900.7603998 --autopull) (running on teward/Solar Flare)

Restart: API quota is 19983.

@K.Dᴀᴠɪs I've been staying away from searches, which is what was associated with the problem.

I was just messing with ya :P

Did henders sneak back on?

!!/location

@K.Dᴀᴠɪs teward/Solar Flare

Does SDS go through MS?

[ SmokeDetector | MS ] Link at end of body: Encountered BSOD PAGE_FAULT_IN_NONPAGED_AREA after installed windows update KB4088875 (Windows 7 32 bit) by TSH91 on superuser.com

BTW: Is MS on a really slow machine? I can run a search using JavaScript and an identical RegEx on the bodies of ~60k posts at > 100 times the questions/s rate that MS can run a search, and JavaScript isn't a fast language (rates are based on actual tests).

@SmokeDetector f

@Makyen There's something more here.

#1967: revert new user message restrictions

I would expect there's something more to it, but I'm not familiar with what architecture (HW & SW) is used for MS. If it was just 10x, I wouldn't be as concerned. At 100x, it implies that there are some significant bottlenecks, somewhere.

@Makyen Are the posts all in memory already before you run you search from JavaScript, or do they have to be loaded from disk?

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching website in body, +2 more: Golden Farms Forskolin the organization offers? by willettetucker on askubuntu.com

sd k

Also, I wouldn't expect language speed to make much of a difference -- the bottleneck should be in the regex engine, which is hopefully implemented in native code

@SmokeDetector k

@Makyen Also FYI we were talking yesterday about why MS locks up completely when someone runs an expensive regex search; the TLDR is that Rails database access is blocking, so on a single-core machine one search ties up the only worker thread, doing nothing while waiting for the results from the database

@NobodyNada Yes, that can make a significant difference. However, I'd expect that a large portion of the post database would remain in memory, depending on MS' capabilities.

It seems none of the Smokies are pinging MS

nvm

I just did a manual ping and it worked, so it must be a Smokey-side problem

!!/location

@NobodyNada teward/Solar Flare

!!/reboot

[ SmokeDetector ] SmokeDetector started at rev ba5809f (metasmoke: Merge pull request #1966 from Charcoal-SE/auto-blacklist-1522985900.7603998 --autopull) (running on teward/Solar Flare)

Restart: API quota is 19887.

Hmm, doesn't make a whole lot of sense though. What could have happened 18 minutes ago?

Yeah I think there's a Smokey bug here. Get one blip in resolve and it freaks out and stops pinging

It's not like it's on a keep-alive

Reboot fixed Solar Flare

Ah well doesn't matter

...oh

Restart: API quota is 18961.

Huh

Yeah OK

@SmokeDetector That's probably RPi; it doesn't seem to have pinged though

I don't think that thread has excepthook installed

And there's no try

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching website in body, +2 more: Golden Farms Forskolin the organization offers? by willettetucker on askubuntu.com

tpu- by Tetsuya Yamamoto

tpu- by Tetsuya Yamamoto

Hm no there's a try

RPi's pinging again, after a reboot

!!/standby NobodyNada

NobodyNada/Raspberry Pi is switching to standby

Let's see if this will fix it

!!/failover teward

[ SmokeDetector ] SmokeDetector started in standby mode at rev 28bc2f6 (SmokeDetector: Auto watch of hackerusa\.com by iBug --autopull) (running on NobodyNada/Raspberry Pi)

@NobodyNada No such command 'failover'.

...is there any way to failover from chat?

yeah you did it correctly

not sure if it's listening

no

there is no !!/failover despite what people think

everyone tries it

charcoal-se.org/smokey/SmokeDetector-Statuses.html#standby-mode

it's wrong

not sure who wrote that

Then wouldn't smokey say invalid command?

!!/blah

but !!/failover has never existed

@K.Dᴀᴠɪs No such command 'blah'.

@K.Dᴀᴠɪs It did

@K.Dᴀᴠɪs it did

Oh I'm blind then :)

@quartata Lemme guess: Smokey doesn't watch chat in standby?

!!/rev

@K.Dᴀᴠɪs teward/Solar Flare ba5809f (metasmoke: Merge pull request #1966 from Charcoal-SE/auto-blacklist-1522985900.7603998 --autopull)

@NobodyNada course not

In case it helps, all I've got in my logs are a couple of these:

[08:18:49] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

Those seem old though

Maybe it should, only reacting to a failover request

ignoring everything else

@NobodyNada yeah that's old

standby is a boolean value, so if it's true, then ignore all chat except failover messages

Shouldn't be that complicated

[ SmokeDetector | MS ] Bad NS for domain in body, blacklisted website in body, link at end of body, pattern-matching website in body, potentially bad keyword in body, +1 more: The food l Keto Tone Diet t consists by oopu on askubuntu.com

tpu- by K.Dᴀᴠɪs

Actually, it might not be that difficult

it'd be pretty annoying, and I like that Smokey doesn't consume much CPU in standby

[ SmokeDetector | MS ] Potentially bad keyword in answer, blacklisted user: How do I answer a question about not enjoying a party without revealing sensitive information by Suhail Khan on interpersonal.SE

tpu- by JAD

Maybe it wouldn't even have to watch real time... Maybe look at chat every 30 seconds. CPU would stay low till it processes those messages

Cor actually I just realized something

I need to move deletionwatcher past the standby checks

@SmokeDetector k

[ SmokeDetector | MS ] Blacklisted website in answer: Changing language setting/interface of QGIS (to English) using Windows XP in French? by kiransha on gis.SE

tpu- by K.Dᴀᴠɪs

@SmokeDetector haha. doesn't matter if you create a sub-domain; we're watching you.

!!/blacklist-website careerconvey\.com

@AJ You don't have code privileges, but I've created PR#1968 for you.

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, bad pattern in URL body, blacklisted website in body, pattern-matching product name in body, +1 more: Increases your degree of testosterone by user22567 on astronomy.SE

tpu- by K.Dᴀᴠɪs

@GrantGarrison BTW: the JSON.stringify that I used was just to format the entire object. It's not needed for comparisons, or conversions. You'll probably be happier with something like:

store => {
  let results = [];
  results.push( ['Reputation', 'User name', 'User URL']);
  for(let index = 0; index < store.posts.length; index++) {
    if (store.posts[index].autoflagged === true && store.posts[index].user_reputation > 1 && store.posts[index].is_tp === true){
      results.push([store.posts[index].user_reputation, store.posts[index].username, store.posts[index].user_link]);
    }
  }
  return results;
};

@GrantGarrison BTW: If posting code in chat, I suggest using, or even over-using, spaces within in your lines of code. If there are aren't spaces in the lines to allow the words to wrap, SE chat will insert invisible characters to facilitate where line breaks will be. This can result in the code being broken when people copy-&-pasted it out of chat, unless they know of the issue and specifically remove those invisible characters.

Yeah that code isn't working for me... TypeError: Cannot read property 'length' of undefined

tpu by K.Dᴀᴠɪs on https://bettercoloncleansingguide.com/nutra-prime-cleanse/ [MS]

try {
    something
} catch(e) {
    window.location.href = "stackoverflow.com/search?q=" + e.message;
}

This is called Intuitive Debugging.

lol

[ SmokeDetector | MS ] Few unique characters in answer: How do I liferay on mac os X? by bernieg on stackoverflow.com

tpu- by iBug

tpu by A J on Was there ever a black Sith Lord in Star Wars canon or "Legends"? [MS]

[ SmokeDetector | MS ] Blacklisted user: Wordpress Title in Post by Rangwa on wordpress.SE

fp- by iBug

[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +2 more: www.thegarciniaeliteslim.com/luna-trim/ by denripto on english.SE

tpu- by iBug

@SmokeDetector I am not removing you from blacklist for now.

[ SmokeDetector | MS ] Bad NS for domain in body, blacklisted website in body, pattern-matching website in body, repeated URL at end of long post: Often consume handiest one meal a day by oopu on apple.SE

tpu- by iBug

troll?

Very likely

!!/scan scifi.stackexchange.com/questions/134774/…

[ SmokeDetector | MS ] Blacklisted username, blacklisted user: Was there ever a black Sith Lord in Star Wars canon or "Legends"? by MICHAEL DAVID MILLS on scifi.SE (@Mithrandir)

tpu- by A J

There's your answer, in the username

It's a known troll Rand asked to red flag

Yup so I red flagged it already.

only 3 yet. 3 more needed

Did we drop the "all-caps" reason?

[ SmokeDetector | MS ] Potentially bad keyword in answer: compare two pdf files (approach) using java by Tarun Kumar Nayak on stackoverflow.com

@K.Dᴀᴠɪs Did you "Add..." the "posts" table (under "Using this data")?

tpu- by iBug

@SmokeDetector It's his own product, left a comment

@SmokeDetector tpu-

@Undo MS and Redunda "can't connect to MySQL server on '127.0.0.1'"

[ SmokeDetector | MS ] Blacklisted website in answer, blacklisted user: Content script on chrome error page by gethelp on stackoverflow.com

sd k

sd k

[ SmokeDetector | MS ] Pattern-matching website in body: How to upgrade my kernel version to latest version by akash on unix.SE

@SmokeDetector f

@NobodyNada you're so fast... I was just working on the PR

[ SmokeDetector | MS ] Blacklisted website in body, link at end of body, potentially bad keyword in body, repeated URL at end of long post: Find out About Health Halt Cream by healthusa on superuser.com

@iBug Ah, I see.

Anyway, I'm off to bed -- see you later!

@SmokeDetector sd k

just getting back up to speed after an OS update ... is MS still ill?

@Makyen I doubt your feedback worked

I'm just shutting it down for tonight. Have to get some sleep

Redunda and Sentinel will also be dead.

[ SmokeDetector | MS ] Offensive answer detected: Word for people easily influenced by propaganda by Carlos Sun-Tana on english.SE

@iBug I wouldn't be surprised. I'd already gone to the post and flagged.

@SmokeDetector f

@SmokeDetector f

[ SmokeDetector | MS ] Bad NS for domain in body: Communication Help(Arduino Mega+Ethernet Shield+Raspberry+Node-Red+MQTT) by Alvaro Justus on stackoverflow.com

@Undo can you arrange to redirect to a temporary instance somewhere before you go if we arrange to set one up somehow?

Really can't right now. Mobile.

@SmokeDetector VLQ/NAA, quoting urban dictionary isn't going to cut it on ELU

You should still be able to deploy. I'm mostly concerned it'd a DOS of some flavor that's gonna balloon an AWS bill. No way to tell for certain right now

Really? I quoted Urban Dictionary in all my essays back when I was in school

[ SmokeDetector | MS ] Blacklisted website in answer: How to get online date in vb.net? by google play music on stackoverflow.com

[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, pattern-matching website in body: A Somewhat More Gentle Structure Of Kilos Loss by vxoreoemb on puzzling.SE (@Mithrandir @micsthepick)

@Undo sleep well then, new challenges tomorrow (-:

sd 2k

sd 2k

sd 2k

@SmokeDetector f

train wreck

:-)

We don't need more than 2 feedbacks on a post. I'd already gone to the sites and flagged.

more than two feedbacks is not a bad thing

because someday Undo or Art may raise the required feedback to 3

Definitely not a bad thing. I just didn't see much need for a 4th feedback this time.

[ SmokeDetector | MS ] Offensive answer detected: Is there a better way to say: "My question is, is..." (e.g. "The question is, is it the right time") by Mr. Kwalala pussy on english.SE

Is MS having a day off?

I think Undo turned it off for the night.

So I just disabled half my userscripts xD

Needs R/A flags: english.stackexchange.com/a/440255/288699 It's gone

Will feedback in chat continue to be recorded? Or, picked up once MS comes back online?

[ SmokeDetector | MS ] Luncheon meat detected: An interview question - implement Biginteger Multiply by AR Akash on stackoverflow.com

sd f

Metasmoke is turned off for today; userscripts, feedback, and blacklist updates will work poorly

@Makyen there is no facility for that at the mo AFAIK

one of the things a "smoke and mirrors" thing could help improve, but maybe making metasmoke itself more resilient is the way to go

@SmokeDetector k

@tripleee Thanks for confirming. That's what I thought. I'd assumed that, currently, feedback here would only provide hints to other users.

Someone give me the 10 second explanation for why MS is down?

@ArtOfCode Undo was exhausted and went to sleep about 30 minutes ago

Undo killed it

Aye, but why?

that's about a 1 second explanation, so falls within your guidelines

dunno if that actually counts as an explanation but I guess that's what we have

I honestly think it was another query that brought MS down

shrug mkay

there were a few posted in chat here

so likely everyone rushed in to run it

what I'm thinking is that we should have access to the A record so we can point it to a different IP address if this one goes crazy for whatever reason

[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title: freemp3download.xyz/Titli-Zaar-Mp3-Song-Download/ by AddOrd on puzzling.SE (@Mithrandir @micsthepick)

there's a bunch of things I can think of to improve resiliency but one of the reasons I posted a question about this on Teams is that I think we need a more thorough discussion about risks and priorities

risk is a priority

uh, no, these are two different concepts

not really, "discussion about priorities" would have included "risk"

@SmokeDetector k

Just sayin' :)

@tripleee Something like a failover? Yes please!

there are many things we could prioritize which are not related to risk so I was attempting to frame the discussion around the general direction of those priorities which are related to risk management and resiliency ... but can we instead concentrate on the actual question?

@tripleee plenty room for malicious actors in there

[ SmokeDetector | MS ] Potentially bad keyword in body: Reading an array using a randomly generated number by Joshua Sears on stackoverflow.com

@chade_ it's one of the things which would improve resilience in this particular scenario but I'm more thinking along the lines of, how do we make sure all of our ducks can be lined back up in a row if $critical_core_member falls off a cliff or (more chillingly) burns out and/or gets alienated and/or depressed?

[ SmokeDetector | MS ] Potentially bad keyword in answer: Reading an array using a randomly generated number by Milos on stackoverflow.com

@SmokeDetector why

@tripleee Body - Position 818-824: Turkey, Position 2243-2249: Turkey

@SmokeDetector f

@SmokeDetector f

@Makyen It's not particularly slow - not as fast as a good desktop, but not slow. What's taking the time isn't the regex search, but pulling all 100k posts (30MB of text) out of the database and into memory - I'd be really surprised if JS could do that as quickly.

I've seen Anonymous Teacher voluntarily delete their posts as soon as they incur downvotes to evade filters. Perhaps it's time to stop applying automatic downvotes to red flags?

@gparyani The automatic downvotes do have a lot of benefits

3 of them will remove a post from the default homepage of a site, for one

@ArtOfCode not sure what you are getting at, we already have some shared trust around some of our assets and it's working out well so far AFAICT? I don't mean give everyone access, just make it possible for us to fix things somehow if the primary owner is unavailable

@gparyani If you notice that, just raise a custom flag

@Rob On what?

Their posts, if they have any. Or your own, and include a link to their profile + explanation

@Magisch you could discouple that from downvotes, by turning that to trigger at 3 red flags as well

@Magisch No, three pending red flags will explicitly kick posts off the homepage. Also, the kick threshold for downvotes is -4.

@JAD That's how it's set up currently

@gparyani TIL

@Rob What if I don't have any posts on the site, and all of their posts are deleted?

Feels like stuff is falling apart lately

@gparyani barely noticable because of the dowvotes though :P

@JAD It's documented in the MSE FAQ

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link following arrow in body, pattern-matching website in body: Keto Advanced Weight Loss Drink a lot of water no less by Hellderko on graphicdesign.SE

interesting

@SmokeDetector tpu-

@ArtOfCode I can only partially agree with that as the root cause. We can assume that plain text is pre-indexed, which explains why those searches are ~500 times faster, so we can discount that difference. However, the difference in times for different regex based searches (even for not that complex regexes) can easily make a difference of an order of magnitude, or more, in the time required for the search, which can be, by far, the dominating amount of time.

That implies that the cost of pulling the database off disk is not dominant over the cost of the actual regex.

@gparyani To be honest, you can really raise it on any post. If you can flag one of theirs, or the question they answered to, or your own post, that'd probably be better. But we do see who raised the flag and the text in the mod queue. The post it's on isn't that important, if it's a general flag

@Rob I know, I'm just making a pedantic joke

Also - are you sure that they're self-deleting the post and a moderator didn't nuke it?

@Rob "This question was voluntarily removed by its author"

Ah, gotcha

vs. "This question was removed from [site] for reasons of moderation"

[ SmokeDetector | MS ] Bad keyword in body, link at end of body, pattern-matching website in body: Phen375 Avis,Phen375 and the famous Phentermine by pedromitchhel on meta.SE

@SmokeDetector tpu-

Does this look okay? xD

> answer not found

@Makyen I'm 99% that the DB pull is the dominant factor. Any query for all posts is sloooow.

[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, link at end of body: www.testoultra.co.za/ by bbbdff on askubuntu.com

@ArtOfCode Are we talking about queries listing all 60k posts?

Or does any query pull all posts and then do the filtering on the rails / js side?

@ArtOfCode That's possible, but then I wouldn't expect to see what I did observe: an order of magnitude difference in the total time the search takes depending on the regex used (e.g. going from a simple regex to, at most, moderate complexity).

[ SmokeDetector | MS ] Blacklisted website in answer: Changing the SQL Server of a SharePoint Farm by google play music on serverfault.com