gist.github.com/superplane39/e7ec5236c5452d6c857a5194d5a53446

That's at nearly 300 and I'm nowhere near done. ._.

That is a lot of spam

@Mithrandir Does Medium have an API to get a list of users?

There are some patterns in those usernames that instantly jump out at me

dunno. all of those would be caught by our filters.

Nobody does any flagging in medium I guess

@ArtOfCode I see the same 'juriya' pattern as we have here.

...since i'm literally going through domains tagged #drugs on metasmoke and searching google for site:medium.com "domain.com"

@Mithrandir weeeell... if medium wanna give us a realtime feed...

@ArtOfCode I'm going to ask them if they'd be interested in working with us when I give them this list.

We require stickerz payment - monthly :P

@Glorfindel juri seems to be the repeating unit. juri(?:[yr]am?)?

(And if they do, request that stuff be automatically deleted when it gets enough flags, and add a thing to their API to flag stuff :P)

There's also tosan

but yeah, run those posts through Smokey and you'd get most of them high-weight hits without even changing the filters

repeated link at end of long post would have a field day

and every single one of them would be caught by "blacklisted website in body"

[ SmokeDetector | MS ] Potentially bad keyword in body: How do I deal with my over expecting boss who scolds me everytime? by DG4 on workplace.SE

fp- by Glorfindel

This gist is just the usernames

!!/test exoslim

regex to match them all: .*

@WELZ Good idea!

[tag:waffles]

[ SmokeDetector | MS ] Bad keyword in answer, offensive answer detected, potentially bad keyword in answer, toxic answer detected: How best to approach the Mahabharata? by Umar on hinduism.SE (@NogShine @Pandya)

tpu- by Zoe

[corn-for-horse] and [roadkill-pasta] can probably go, TBH.

[ SmokeDetector | MS ] Link at end of answer: MySQL connection through jump server (Sequel Pro or Workbench) by Manu chacko on stackoverflow.com

in your dreams buster

Pretty simple regex that catches 50% of them: (?:(?:\d{3}|_\d{5})$|juri(?:[yr]?am?)?|(?:bond|max|vaxer|jems|tz?osan)$)

That's simple....

naa- by Zoe

@ArtOfCode nginx. And yeah, just Sentinel and Redunda

tpu- by Zoe

@Undo what command runs the Rails server, though?

dunno

Passenger figures it out

Okay, know what the server is? Passenger? Puma?

Nginx/Passenger

(so Passenger)

kk

cc @thesecretmaster that's why puma doesn't run with more workers... it ain't puma

nginx does, though

Should be four

or maybe more

Aye, but if we have 4 nginx workers passing requests to 1 passenger worker... that's where the problem is

4 passenger workers

...huh

there goes that theory

@ArtOfCode hmm, maybe put that in our current filters

âzo  ~ passenger-status
Version : 5.2.1
Date    : 2018-05-09 14:42:43 +0000
Instance: 5s3gZDpC (nginx/1.12.2 Phusion_Passenger/5.2.1)

----------- General information -----------
Max pool size : 6
App groups    : 5
Processes     : 6
Requests in top-level queue : 0

----------- Application groups -----------
/var/railsapps/metasmoke/current (production):
  App root: /var/railsapps/metasmoke/current
  Requests in queue: 0
  * PID: 15751   Sessions: 0       Processed: 18664   Uptime: 16h 34m 45s
    CPU: 0%      Memory  : 228M    Last used: 3s ago

Right now it's running 2 workers, I think that auto-scales somehow

Think it's worth putting MS behind a caching CDN?

[ SmokeDetector | MS ] Blacklisted username: How can I hide a second narrative within my story? (using time travel) by Anonymous Teacher on writing.SE

Might give us a bit more capacity

@SmokeDetector k

Honestly, it really doesn't feel like a capacity problem. Vast majority of individual requests come through just fine. Feels more like something gets sideways occasionally

[ SmokeDetector | MS ] Bad keyword in title: Rock song from album with greenish cover, muscle car, Apple Music A-List, 2017 or 2018 by jonas-schulze on musicfans.SE

fp- by Glorfindel

My current theory is that it's probably a database thing, but I'm not sysadmin enough to figure it out

@Undo I know what it is - it's the userscript API requests. Something makes a bunch of requests all at once, MS stops what it's doing to serve those, and humans lose out.

running ~10 req/s

[ SmokeDetector | MS ] Blacklisted user: Осветлить фон яндекс карты by exe.c00ller on ru.stackoverflow.com

Wonder if we can tell nginx how to prioritize

Thing is, we should still be able to take more than that

fp- by Glorfindel

Yeah, those should be 10ms requests. And they often are.

If 8 workers can hit 2k req/s, then 4 should be able to hit at least a few hundred

metasmoke itself is just slow, it's not a problem with the container

What about it is slow, though?

I guarantee you those few lines of Ruby don't take more than a few ms on their own.

I don't know, hook up a proper profiler to it? my guess is queries

(for most pages)

and why is it necessarily slower than the app that I hit 2k/s with?

Didn't make any special optimizations to that

I don't kno, but Redunda and Sentinel run on the same containers and have almost no problems so I'm opetty sure it's not Passenger

They're also far simpler

[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer: Python - Order nested list in alphabetical way by eeeeeeeeeeeeeeeeeeeeeeeeeeeeee on stackoverflow.com

tpu- by Glorfindel

Redunda takes comparable load I would imagine

At a minimum, we know that regex body searches are very slow cause other requests to timeout. We've also seen that from the Data Explorer.

Can the API run on a separate instance of nginx? With fancy forwarding the user could still hit port 80 and it be off loaded to the proper backend instance of nginx?

@WELZ I'll take stickers (I'm hoping for some from work too), but I'd also like a "Thanks to Charcoal for helping rid us of spam."

Of course, that means we have a bigger target on us from annoyed spammers

Worth a try. Or just the same nginx (that doesn't seem like a bottleneck) but different Passenger apps

tpu by Zoe

fp by Nisse Engström on Rock song from album with greenish cover, muscle car, Apple Music A-List, 2017 or 2018 [MS]

@Undo That'd probably be less resource intensive than having a second nginx. I don't know how to config nginx though. I've worked almost exclusively with apache

It's basically the same

@SmokeDetector is this a known issue that the link to MS for reported localized SO... doesn't really work? (or did I miss anything?)

@Undo i'm behind on the discussions, what did you mean by 'prioritize'? As in prioritize requests? (Like a QoS thing?)

yeah. Probably not a good solution

@Andy same

[ SmokeDetector | MS ] Link at end of answer: how to schedule cronjob for every 45 days? by Sissi_00_20 on unix.SE

Anyways there are a few things to consider

naa- by Glorfindel

[ SmokeDetector | MS ] Offensive answer detected, potentially bad keyword in answer, toxic answer detected: VirtualBox - show only 32bit by A. Haidari on stackoverflow.com

@AndrewT. Just realized Comment on MS... this message got included on MS >_<

1. Swiching to Postgres + Thin with Nginx in front of it would allow us to do better async including the queries themselves -- that means while waiting on a DB query other requests can be serviced

@Undo Probably not, but you also have a problem where that's not an in-built function of NGINX

it'd require some advanced scripts and such.

fp- by Makyen

fp feedback on autoflagged post: VirtualBox - show only 32bit MS (@Makyen @quartata @River)

@Makyen Your flag conditions were manually disabled by @System: Total accuracy must be >= 99.75%.

@River Your flag conditions were manually disabled by @System: Total accuracy must be >= 99.75%.

2. Regex body searches coud be optimized by loading the posts into memory if we don't already have them and then either in a separate thread or on each tick of EM check a post and "yield" it back to the browser if it matches

carnage

@SmokeDetector Already edited by a moderator and spam flags marked disputed.

@quartata This sounds intriguing

I have no idea if this is helpful or not, but I found this - stackoverflow.com/questions/25798558/… or this - stackoverflow.com/questions/13722718/…

fp by Zoe

3. Move our assets to a gh-pages branch on MS

[ SmokeDetector | MS ] Offensive answer detected, potentially bad keyword in answer, toxic answer detected: Error with pre-create check: "This computer is running Hyper-V. VirtualBox won't boot a 64bits VM when Hyper-V is activated by A. Haidari on stackoverflow.com

fp- by Makyen

fp feedback on autoflagged post: Error with pre-create check: "This computer is running Hyper-V. VirtualBox won't boot a 64bits VM when Hyper-V is activated MS (@ArtOfCode @ByteCommander @CalvT)

tpu- by rene

after build of course

@SmokeDetector this is now a pattern, deleted

4. The SOBotics people would probably rather run Redunda on their own server anyeays, so you could free some resources up

5. Cloudflare

thats all I got for now. 1 would be the most work but the most payoff

oh I almost forgot 6. nice MS to -19 for maximum swag

I think it's gotta be something simpler then that.

Is there a place where I can get a current db dump?

That's what I'd think too

I wanna try some stuff

@thesecretmaster metasmoke.erwaysoftware.com/dumps

Should be able to hit that

I can give a demonstration of what happens when workers get blocked doing a query

MySQL does not have non blocking queries but Postgres does

@Undo K. Imma try some stuff locally tonight.

now, we could still use Thin and run queries in an EM.defer

that would be a decent solution

@Art does MS only ever query SE API for user details once after OAuth?

Is this spam? meta.stackexchange.com/q/310035/158100

heh, finding Medium spam led me to find spam on Goodreads, which i reported. how far does this rabbit hole go?

blatantly off topic i'd say, but not spam

Thanks

[ SmokeDetector | MS ] Manually reported question: Quisiera saber tips para tramitar mi visa americana by luis fernando garcia quijano on meta.SE

OK I was wrong MySQL can do non blocking queries. Remains to be seen whether the mysql2 gem does

@SmokeDetector k

[ SmokeDetector | MS ] Blacklisted user: I would like to know tips to get an tourist US visa by luis fernando garcia quijano on meta.SE

@SmokeDetector f sorry

@AndrewT.: It seems this is your first time sending feedback to SmokeDetector. Make sure you've read the guidance on your privileges, the available commands, and what feedback to use in different situations.

@SmokeDetector probably the first time here, but thanks :)

@SmokeDetector f

Restart: API quota is 17581.

[ SmokeDetector | MS ] Blacklisted username: Chemical reaction , blue to red and again to blue by Anonymous Teacher on chemistry.SE (@Loong)

@Mithrandir There is a website included with prices.

naa- by DavidPostill

@quartata We have 6 six workers right now though.

I don't think it's a concurrency issue.

@Undo I might've missed this, but is there a reason we use passenger instead of e.g. puma? Is it a capistrano thing?

Also, how do you deploy MS? I want to try and replicate what you've got, and then tweak stuff?

[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer: How fast were Bucky, T'Challa and Steve running in Civil War? by user100287 on scifi.SE (@Mithrandir)

tpu- by rene

That's the second time Google has rate-limited me. No, I'm not a robot, Google, even if I'm acting like one!

tpu- by Mithrandir on Chemical reaction , blue to red and again to blue [MS]

@Mithrandir Doing a lot of site:<domain> <search> requests?

yes

Yeah, happens to me whenever I do a lot of those :P

Just throw in a how much cheese is too much? every now and again

and the result is currently... 619 Medium spam posts collected so far

Nice!

BTW, if anyone wants to help, add more domains to the #drugs tag

[ SmokeDetector | MS ] Bad keyword in answer, offensive answer detected, potentially bad keyword in answer, toxic answer detected: How do I send an email directly from a gmail add on? by user9765932 on stackoverflow.com

tpu- by Makyen

tpu by Zoe

@ArtOfCode FYI: There's an inconsistency between FP shown in the flag conditions edit previews and the Overall flag condition accuracy query. I currently have 4 conditions active. The sum of the FP in the edit previews (assuming no duplicates) is 118. However, the Overall flag condition accuracy query shows a total of 154 FP. If these queries are all accurate, then the overall total of FP should be <= the sum of those found for each.

@thesecretmaster ActionCable

A year or so ago, Passenger seemed like the only sane way to support it

That doesn't play nice with puma?

Oh

Seems fine now?

Can look again. Historically, Puma has been a faster server when we've been able to use it

@thesecretmaster cap production deploy, everything except config/deploy/production.rb should be there.

I think passenger might've been saying "6 processes, one for each app"

MS was showing two

2 is few enough that I'd blame that. Can you manually increase that number?

Think so. There's a config file somewhere

Wanna try that?

yeah, will give it a shot

Looks like Puma is saner about ActionCable now, might try that too

Well, first let's see if passenger will do it with moar threadz.

Will look at that when I get SSH access again

Adding moar threadz to clog-up a clogged CPU even moar ...

Work firewall does... weird things. Sometimes works, sometimes doesn't

OK. Hopefully that's it.

The instance isn't at 100% CPU or anything, is it?

It's a t2.small, so shouldn't be

If it is for more than a few minutes it'd crawl to a halt

k

[ SmokeDetector | MS ] Potentially bad keyword in body: retriving longurl from json data in php by Kautuk Bhatnagar on stackoverflow.com

So, let's try threads, which IMO should fix it.

sounds more like it runs out of sockets or any other scarce resource that is governed by the kernel.

fp- by DavidPostill

fp by Zoe

[ SmokeDetector | MS ] Bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, blacklisted website in body, +1 more: MORE INFO @>> wellnessfeeds.com/flawless-keto-diet/ by Aenaton on askubuntu.com

@Undo What should I stink in production.rb?

tpu- by Makyen

@thesecretmaster gist.github.com/Undo1/c015bc8ed51d33a442382952ad7de9a8

Thank you!

Yep! It's running on a t2.small, Ubuntu 14.04.5

How do I turn sign up back on?

You already got it running?

Yeah, on an ubuntu VM.

Good grief, that's impressive.

Should be /admin/settings

Well, it's not the first time I've done it.

I think it's... the 4th?

Still pretty darn fast for getting all the dependencies and stuff going

I don't have an account. I can't get to the admin panel.

Rails console or something I think

Oh. Could either change the registration_enabled row in the site_settings SQL table, or just open the rails console and create/hijack an existing user

I remember the horrors when I set it up for the first time. I've never re-set it up since :p

u=User.find(1); u.password = "newpass1"; u.password_confirmation="newpass1"; u.email = "my@email.com"; u.save! makes you me

Heh, the username is "undo" by default

[ SmokeDetector | MS ] Pattern-matching website in body: I'm trying to add Android app install button on my website like playstore by World Info on stackoverflow.com

fp- by rene

@ArtOfCode Can you send me your loadtest frequencies.yml file?

Also, since the JS isn't running, that will signifcantly speed things up.

[ SmokeDetector | MS ] Bad keyword in answer, offensive answer detected, potentially bad keyword in answer, toxic answer detected: How sexual relations with multiple partners affect us? What are after effects of promiscuity? by Yaqub on hinduism.SE (@NogShine @Pandya)

tpu- by NobodyNada

So... it's not metasmoke or mysql. It's threading, based on my local tests.

How can you tell?

Curious what tools you use to figure that out

It hits 80 reqs/sec without an issue.

Load test script I used a while back

What endpoints does that script hit?

And what are the specs on your VM?

gist.github.com/thesecretmaster/…

0.05 req/s each for /reasons /posts /graphs and a body regex search for ultra\W.

@thesecretmaster A MySQL query blocks an entire worker

from what I can tell

Source? That'd be a horrible design choice.

@ArtOfCode How about Trello?

@thesecretmaster igvita.com/2010/04/15/non-blocking-activerecord-rails

Now, that's from 2010, which does post-date Ruby 1.9 meaning they're referring to native threads and not green threads

So it could depend on how many threads each worker is running, I dunno

But, AR and Rails are inherently synchronous is what I know for sure

The arms race of adding more workers to handle more concurrent requests is dumb

That's a ruby bottleneck, not on the mysql side

Adding more workers gets around that.

orrrrr we could do proper non-blocking

and handle as many as we like simultaneously

without eating all of the memory on the EC2

OK, you submit the patch to mysql2 and then it'll all be fixed.

It doesn't need that

Sorry about my tone, that wasn't exactly a nice way to phrase that.

from what I've seen mysql2 already has EM support

I just have a hard time believing that all of these high-performance Rails apps did craziness around async stuff

But wouldn't we have to a) patch AR and b) rewrite metasmoke to deal with that nonblocking IO safely.

Also that. I think more workers is fine.

hey SSH works now

Yay!

in fact

> Mysql2::Client takes advantage of the MySQL C API's (undocumented) non-blocking function mysql_send_query for all queries

Why don't you scale up to ... 8?

Per app?

apparently you just add :async => true

@Undo For MS

        passenger_max_pool_size 8;
        passenger_min_instances 8;
        passenger_pre_start metasmoke.erwaysoftware.com;

Let's give that a try

Ok.

How do you want to test it?

I guess just hammer the thing

Hold on, doing config dance

OK. I've got my script ready to go.

You sure you want to do that on prod?

Worst case, I've got to reboot the server

K. Wanna watch htop while I do it?

[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer, repeating characters in answer: Building word Thesaurus in Python by aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa on stackoverflow.com

Hold on, it's not running

Gotta figure out some config stuff

OK

Ohhhhhh

Hah!

phusionpassenger.com/library/config/nginx/reference/…

> Default passenger_max_pool_size 6;

And apparently that's global

So since there are five apps running (three + two actioncable), each was getting only ONE worker, and juggling around that extra one

That's what I was trying to say

So more threads is the fix

(fingers crossed)

@quartata Could use Netlify — you should be able to upload a ZIP of files to host.

teward/Solar Flare received failover signal.

intentional dead switchover?

!!/location

@ThomasWard teward/Solar Flare

@ThomasWard ArtOfCode/EC2

!!/standby teward

Restart: API quota is 19999.

[ SmokeDetector ] SmokeDetector started in standby mode at rev e0d87a3 (SmokeDetector: Auto watch of xboxsupport\.org by WELZ --autopull) (running on teward/Solar Flare)

@Undo MS maintenance/serverproblems?

Yeah, it was down for a minute or two

@Undo Hammer ready

cool cool, i was afraid we were going to have everything fail over incessantly heh

New metasmoke user 'loadtest' created

^^ me

Still don't think it works -- I sent off two /reasons requests and the second only loaded after the first

Are you sure that's the config for processes and not threads within processes

ActionCable is trying to eat all the processes

Oh

not surprised, AC internals suck from what I've seen

I gotta step out for 5

Wait

does AC spawn a thread per subscription?

no

ok ok good

I thought maybe something like that was going on

@quartata Try that again now

It feels speedy to me

Metasmoke has 6 workers now, up from 1-2

Oh, I remember now. Couldn't do this before because it would eat all the RAM.

tpu by Nisse Engström on MORE INFO @>> wellnessfeeds.com/flawless-keto-diet [MS]

Now we have more RAM (2GB up from 512M) so it's nice and happy

OK

Want me to wack it now?

Aye

Watching stuff

fp by Nisse Engström on How do I deal with my over expecting boss who scolds me everytime? [MS]

[ SmokeDetector | MS ] Potentially bad keyword in body: Difficulty performing POST - Mean Stack by Fred on stackoverflow.com

Hrm

Queue isn't building up

Getting a 403 on /graphs

o.O