« first day (1555 days earlier)      last day (2481 days later) » 

00:13
Initial results from Shog's dumps are encouraging. Analyzing how much delta there is here
2
On the last post, 106698, it's 2018-02-15T17:23:55.513Z (Shog) v. Thu, 15 Feb 2018 17:23:55 UTC +00:00 (MS)
My main concern is whether the MS clock has been significantly out of sync before - if so, we're going to have issues
00:28
Will MS work on sqlite?
I guess not.
Theoretically you can point it to a SQLite database, but I doubt everything would work
I just tried and... no.
@Undo Aren't we most interested in the total time the spam was live (i.e. from SE post time to SE delete time)? Both time's should rely only on clocks at SE. It's interesting to see how long between SE post time to MS detect and MS detect to (various other MS events) and then SE delete, but the main one is how long the spam was live overall (which is what we claim is improved by using SD).
@Makyen yeah, I'm looking at it
Median difference between Shog's data and ours is 0.0 seconds
Are there instructions for setting up a MS dev env anywhere?
Anonymous

Setting up Metasmoke development

Dec 15 '16 at 22:24, 40 minutes total – 147 messages, 6 users, 0 stars

Bookmarked Dec 16 '16 at 18:58 by NobodyNada

Ooo thanks!
I was just playing guess and check :)
fp- by thesecretmaster
@Undo BTW, both of those are missing bundle exec rake webpacker:install
And asset precompilation and yarn
Asset precomp doesn't happen in dev
webpacker also doesn't happen in dev, IIRC
00:40
Hm. Then I have some problem with my env vars. I'll sort out what I did.
As long as it's running in dev, it shouldn't need precompiled assets
Well, yeah. That. But webpacker complained when I tried to start it up.
Could be that I have something else screwed up though.
You're running foreman start, right?
Now yes and I have different problems. I originally wasn't following any instructions and just running rails s -e development and fixing errors.
Yeah, you'll go down a bad path that way because the asset server won't be running
Foreman is the 'right' way to run it in dev. Production does precomp and uses the standard asset pipeline
00:45
Yeah. I think it's about time for me redo it the right way, now that I'm getting real errors.
@Undo It's alright for me to start making PRs for issues, right?
I'll take all the help I can get :)
K. Warning: I do a lot of ruby development, but pretty much no rails.
Rails is pretty easy to learn, especially if you know the language already.
Metasmoke code gets reviewed pretty thoroughly anyway due to PII in the database.
Yeah. I already had to learn rails for a different project (thanks to ArtOfCode).
That was just to say that I'm not a rails expert.
Foreman starts in dev by default?
Yep, on :5000
00:59
Yeah, you do need webpacker
It tries to start the webpacker ... thing ... in the Procfile.
Usually :install actions are one-time. Odd.
Good to know, though. Whatever makes it work.
Hm? I never ran webpacker:install because it never said to do so in the instructions.
I didn't think it is required
It might not actually be required, but foreman tries to start it because it seems like it just executes everything in the Procfile.
Interesting. Does running webpacker:install work?
01:03
Yep
Would adding it between 20 and 21 work?
Yeah. Has to be after 20 because AFAIK it requires yarn.
Also, systemctl enable redis-server and systemctl start redis-server are skipped.
Redis isn't a core dependency - everything except ActionCable will work without it
It should be in there, though
@thesecretmaster Usually installing redis-server in Ubuntu Server automatically starts it for me
@M.A.R. ok so I may or may not have just figured out what your avatar is
I've never manually started it.
01:07
I'm stuck on my horrible mac laptop right now so I had to just redis-server &
On my arch install it should work better.
I'm on macOS, and I think I installed Redis with Homebrew
Out of curiosity, why use foreman?
@JF could answer that best
@Undo Same here. It might autostart if I restarted my computer. I'll report back.
Usually Homebrew gives you some instructions after the install
01:08
Restart: API quota is 17432.
you know forget gems from activerecord
gems from tensorflow is what we need
@Undo Grrr I'm dumb
tensorflow.count_nonzero(tensorflow.logical_or(tensorflow.logical_not(tensorflo‌​w.greater(y_pred, 0.7)), tensorflow.cast(y_true, tensorflow.bool)))
and I know I'm stupid for not doing import tensorflow as tf
but it's stupid for not including a logical_conditional
so take that
@Undo If you send me a CSV of reasons and a pile of sample posts, I could make a PR adding those to the seeds.
@thesecretmaster Want a full SQL dump instead?
01:15
Sure! Just, y'know, not PII or anything.
Should be able to pull that right into MySQL
I think it'd be good to add the reasons to the DB seeds, and some more sample posts. Would you accept that as a PR?
Definitely
Great. That'll be my first one. Then on to issues.
01:27
For reasons: I'm going to omit last_post_title and inactive:true. Should I leave weight, or is that not static enough?
think I'd take weight out
[ SmokeDetector | MS ] Repeating characters in answer: How do you calculate Byte Offset? by hhhhhhhhhh on electronics.SE
sd rude
Wow, these sql queries are taking 1000+ ms each. My computer is sloooow.
01:38
@thesecretmaster Development is often slower for me too
rpg SE gets the strangest rule-lawyering questions
4
Q: If I turn into a Gas Spore and use Death Burst, do the other spore count as "me"?

HunterIn Circle of the Seedling you can use Wild Shape to turn into plants. If I am a Gas Spore and use Death Burst, assuming the target fails the save and eventually dies from the infection, are the resulting Gas Spores that grow from their corpse considered my character?

deep
@Undo really? page loads are maybe a touch slower because lack of caching and no cache_classes, but SQL is lightning on local
[ SmokeDetector | MS ] Potentially bad keyword in answer: C# custom reporting by Satya on stackoverflow.com (@iBug)
@ArtOfCode Probably depends on dev machine.
sd why
@thesecretmaster [:42931767] Body - Position 424-435: wixsite.com
01:45
@thesecretmaster true, mine ain't bad
no ssd, but plenty RAM and CPU
fp- by thesecretmaster
Hm. Will I break FK stuff by removing smoke_detector_id from posts?
Shouldn't
@thesecretmaster do show create table posts and check whether smoke_detector_id is nullable
it is
01:49
you're good then
My computer just froze for 10 seconds because of the load from AR queries.
that... is a remarkably slow machine
mid 2011 mac air
4GB ram and 1.7 GHz Intel Core i5 cpu
so mid-ish range
shrug well it's either the machine or the queries and it could be either
Oh oops I forgot to close my 20 tab chrome window
02:01
might do it
free some ram up
I've got so used to not having to close... uh... anything
The RAM is probably the issue. My 2012 Pro w/ 8GB handles it fine. Longer pageloads, but never freezes
SSD helps too
five IDE instances, 30 chrome tabs, database connections, all at once, it's beautiful
Yeah. I'm used to 32gb of ram on my normal dev machine.
Once I ran 8 vms without issues
[ SmokeDetector | MS ] Bad NS for domain in body, blacklisted website in body: Why do browsers not render pictures in native pixel density on by Sic jcj on apple.SE
tpu- by thesecretmaster
[ SmokeDetector | MS ] Offensive body detected, offensive title detected: I'll go to England to fuck you by Aerals on money.SE
tpu- by thesecretmaster
Wait... Post#tags is MS tags or real tags?
02:20
!!/allnotificationsites 11540
You will get notified for these sites:
android.stackexchange.com, codegolf.stackexchange.com, meta.stackexchange.com, stackoverflow.com, superuser.com
!!/unnotify 11540 meta.stackexchange.com
@iBug I will no longer ping you if I report a post on stackoverflow.com, in room 11540 on chat.stackexchange.com
@iBug I will no longer ping you if I report a post on superuser.com, in room 11540 on chat.stackexchange.com
@iBug I will no longer ping you if I report a post on meta.stackexchange.com, in room 11540 on chat.stackexchange.com
You will get notified for these sites:
android.stackexchange.com, codegolf.stackexchange.com
@SmokeDetector More flags.
02:23
@SmokeDetector n
@SmokeDetector f
[open] Hi o/
Trying to get box/whisker plot for deletion deltas. Excel doesn't like files this big.
All data, excluding posts we didn't have timings on already. Scale in seconds; delta between our value and new value
Shog value - MS value
02:46
Grr... seeding posts is a pain because I don't know the reason IDs.
You should be able to seed reasons and specify IDs there.
Take a look at tests/seed for how it's done in test
Yeah, but seeding ids feels kinda wrong.
Things should work regardless of those IDs.
How are you seeding these?
From a yaml file
Rob
Rob
@thesecretmaster Seeding IDs is pretty common. Especially if you use the seeded ids in code
Oh, that is what fixtures are for.
If it's set up like fixtures in test, you can name objects in one fixture and Rails figures out the IDs
OK. I'll look around.
[ SmokeDetector | MS ] Bad NS for domain in answer, username similar to website in answer: If someone asks to borrow your phone to make a call, what could they do? by Prexblog on security.SE
sd k
history on SO
03:00
[ SmokeDetector | MS ] Few unique characters in answer: Creation of script inside a javascript by gwapo on stackoverflow.com
Okay, this is interesting:
@Undo I think I'll use made-up titles/bodies instead of using real ones (e.g. body = "This post would be caught by <reason>"). Is that alright?
Depends on what your goal is
For DB seeds.
For a long time in there, we tracked reality very closely - 0 or 1 second deltas.
@thesecretmaster DB seeds for test, right?
Or for "new app, want data without dump"?
03:08
The second.
Why not use real ones?
My idea was to make it possibly to just clone, setup the db, and then go, instead of needing a dump.
I could also use the ms dump you just gave though.
I guess fake data is really more appropriate for tests. But reasons should probably be included in seeds.
The full dump is probably overkill. You might look at sampling the data in there - take posts where id % 1000 = 0 or similar.
You could take everything associated with those posts (will be feedback, probably all reasons) and probably all of everything else.
Posts and Feedbacks is the bulk of that dump, IIRC
J F
J F
@thesecretmaster Because or lets you run multiple things at one. I think. I don’t really remember.
Foreman is running Rails and the asset server at the same time. It was some kind of JS change
allowing... ES6, maybe?
@Undo @quartata might be interested in this. It falls apart at a very defined point - trying to find it chronologically.
We were fine for several tens of thousands of posts, then something happened
J F
J F
03:16
@Undo Yep. It bundles all of the JS files together and also lets you use anything you install from npm in the browser.
tpu- by Mego
Hmm... should config/webpack/* be .gitignored? My webpacker:install overwrote all of those files?
Rob
Rob
@JF So it's a competitor to webpack, basically?
Oh, nevermind, it shouldn't have.
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body: obat herbal alami herbal by geradarasinta on ux.SE
03:31
@SmokeDetector naa
@Catija Recorded answer as an NAA in metasmoke.
tpu- by Catija
@Undo Using the existing webpacker files, I had to npm install coffeescript and ajv
You might want to add that to the tutorial
Thanks. I don't remember having to do any of that.
Interesting
You probably already had them installed?
Perhaps, but I don't know why I would have
03:58
[ SmokeDetector | MS ] URL in title, bad keyword in body, blacklisted website in body, blacklisted website in title, link at end of body, +3 more: rhinorx90eveningblog.com/phytolast/ by spmlxhgvae on meta.SE
tpu- by Catija
[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, link at end of body, pattern-matching website in body, +1 more: wrinklerewindfrance.com/reducelant-garcinia-france/ by Tarwiksqik on askubuntu.com
tpu- by Mego
[ SmokeDetector | MS ] Offensive answer detected: In Sichuanese does 日本人 mean “fuck me”? by user19131 on chinese.SE
fp- by Mego
[ SmokeDetector | MS ] Blacklisted user: Get Android Phone Model Programmatically by Quantum4U on stackoverflow.com
fp- by Mego
@Undo I have a PR for #315. Do you want me to push it on my fork and then make a PR, or do it some other way?
04:20
That's the usual procedure we use
319?
Yep
Sorry, GH username doesn't match SE username.
[ metasmoke ] ci/circleci failure on 23f92ee: Your tests failed on CircleCI
[ metasmoke ] ci/circleci failure on 23f92ee: Your tests failed on CircleCI
Uh oh
I think that's my fault
Oh, it is. Typo
Testing now, specifically testing for HTML injection... which I don't think it would, due to the pre block, but it's possible
Oh, that too. I thought I avoided that but I didn't.
04:27
[ metasmoke ] ci/circleci failure on cff460d: Your tests failed on CircleCI
[ metasmoke ] ci/circleci failure on cff460d: Your tests failed on CircleCI
[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +2 more: www.ifirmationantiagingcream.com/reducelant-garcinia-es/ by erotwern on english.SE
yep, p.why = "<script>alert('foo')</script>" allows injection
tpu- by Undo
A J
A J
@SmokeDetector k
@Undo Yeah, I forgot to sanitize the non-modified part. Also rubocop.
Should be good now
Small regression - the sanitize drops HTML tags, leaving only content. Before: <script>alert('foo')</script> is shown; after it only shows the inner content: alert('foo')
04:32
[ metasmoke ] ci/circleci failure on 8be2b05: Your tests failed on CircleCI
[ metasmoke ] ci/circleci failure on 8be2b05: Your tests failed on CircleCI
[ metasmoke ] ci/circleci failure on 9cc6427: Your tests failed on CircleCI
[ metasmoke ] ci/circleci failure on 9cc6427: Your tests failed on CircleCI
Not sure how to overcome that - almost needs html escaping instead of sanitizing
Yeah. Maybe rails has magic?
Looking
Anonymous
Jinja2 has the |e filter, which performs HTML escaping. I'd imagine Rails has something similar.
There we go
Anonymous
04:34
Why are we getting double CI posts?
@Undo I gotta step away from my machine for now, but I'll fix it later tonight.
@thesecretmaster I've got a fix. html_escape does it. I'll push and merge, thanks!
The regex is the valuable thing here
I think it should work everywhere.
We are never going to cast 6 autoflags, right?
Unless SE change the nuke requirement to 7
The annoying thing is that the close paren is a valid char in a url.
04:39
@iBug Why not?
It's a scary proposition, but... the numbers say we could do it. There are some other things I'd want to change procedurally and technically, but I won't rule it out
You said we don't want to, and won't unilaterally nuke anything.
[ metasmoke ] ci/circleci failure on de006bf: Your tests failed on CircleCI
Isn't Smokey already better than human reviewers, @iBug?
[ metasmoke ] ci/circleci failure on de006bf: Your tests failed on CircleCI
Not yet, not with current systems and procedures. But if we set that goal, it's achievable technically to very good tolerances.
04:41
I see.
Anonymous
I don't want to say hard no to instakilling any post with spam flags, but we would have to have a really, really, really good system
As a mod, I'd want a nuked-posts list, but I trust Smokey to nuke things (given very safe parameters).
Anonymous
The first FP to get autonuked would bring an immediate halt to autonuking
We have a really, really, really good system. If I'd asked people three years ago whether 99.99% was good enough to automatically block posts, they'd say "heck yes"
a nuked-post list is a good idea no matter who nuked it
04:42
The difference is now we can do it.
Metasmoke is a nuked list of sorts, actually
Plus Smokey has a 23/7 supervision
Technically, there are no outstanding unsolved problems preventing us from going to 4, 5, or even 6. 4 and 5 are doable now outside of social issues, and I have plans for what we need to do to get 6 comfortably.
Anonymous
Rationally, I can understand that the system is good enough to autonuke. But I'm still hesitant about the idea of flipping the switch.
It's the red button. Hit with extreme caution
[ metasmoke ] ci/circleci success on 18d079c: Your tests passed on CircleCI!
04:45
@Mego I'm sure you know the trolley problem?
@quartata Removed them
Anonymous
Also, rationally, I know that people wouldn't complain about SE autonuking posts on their side, and so we're in a better logical position, since our system is 100% open.
Anonymous
@JohnDvorak Yes, I know :)
[ metasmoke ] ci/circleci success on 49e14bf: Your tests passed on CircleCI!
[ metasmoke ] ci/circleci success on 49e14bf: Your tests passed on CircleCI!
[ metasmoke ] ci/circleci success on f9b6885: Your tests passed on CircleCI!
Are they now hyperlinked?
04:57
server choked on webpacker step, rebooting
teward/Lunar Eclipse received failover signal.
!!/standby Lunar
teward/Lunar Eclipse is switching to standby
!!/location
@paper1111 bwDraco/Linode
05:01
@WELZ Looks like linking works. metasmoke.erwaysoftware.com/post/107104
Yay!
Great job done. Who did it?
Thanks a bunch @thesecretmaster and @Undo
@iBug @thesecretmaster's first patch, I think
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, link at end of body, pattern-matching website in body: Alpha Testo Maxxx - Hikes your stamina and endurance by annasutherlab on superuser.com
tpu- by iBug
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, pattern-matching product name in body: When I reply-all on iPhone, how can I ensure it by Sfhyelnj on apple.SE
tpu- by iBug
fp- by iBug
[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +8 more: needsupplementhelp.com/renux-forskolin/ by cfrf on askubuntu.com
05:12
@SmokeDetector k
@NogShine You are not a privileged user. Please see the privileges wiki page for information on what privileges are and what is expected of privileged users.
@SmokeDetector Not auto flagged :/
@Undo Autoflag down
in Charcoal Test, 2 mins ago, by PulseMonitor
halflife:2018-02-19 05:11:26,440:Connection is already closed.
Something about this?
05:14
Why isn't that one autoflagged?
That should be from the deploy
I was looking at that error message last week as well, not sure but I was thinking it was related to the skipped messages
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, +3 more: How Dead Sea Spa Elixir Serum Work? by gracereed on askubuntu.com
is Smokey quietly restarting and missing a message which it already started processing, or something like that ...?
tpu- by Mego
[ SmokeDetector | MS ] Bad keyword in answer, link at end of answer: VLC not working on Linux Mint Mate 17.1 by dixzint company on unix.SE
[ SmokeDetector | MS ] URL-only title, link at end of body, pattern-matching website in body, pattern-matching website in title, potentially bad keyword in body, +1 more: acaiultralean-france.com/phytolast-france/ by trokdolton on askubuntu.com
05:16
That one wasn't either. Putting debug back in
Metasmoke is also acting up for me ... related?
@tripleee probably related to me having a 1GB log in RAM on that server
I hope those debug messages aren't causing it to work
@quartata I am now!
not sure what you were proposing there, could you rehash?
@paper1111 Actually, that one was probably from a cert refresh I just did
tl;dr quartata put in debugging to find the issue, we thought we fixed it, I just took out debugging, and now it's broken
05:19
@Undo I see
I'm trying out this "being a competent sysadmin" thing
oh I was wondering about the comment where quartata was saying "which pattern matched was too slow for Smokey but could be suitable for Halflife" ... I guess this means figure out which regex was responsible for a match?
I thought you were talking about metasmoke. Ignore that last comment, then
I'm vaguely interested in that too, but not really competent to help at this point
[ metasmoke ] ci/circleci failure on 69dcbdc: Your tests failed on CircleCI
05:23
@SmokeDetector Interesting - this one was flagged, eventually. Didn't show at first
I might have just loaded the page too fast
@SmokeDetector This one definitely wasn't, though
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, pattern-matching website in body, +1 more: Vars Performance The ingredient is set to by Johnnmurph on astronomy.SE
tpu- by iBug
[ SmokeDetector | MS ] Bad keyword in answer, blacklisted user: VLC not working on Linux Mint Mate 17.1 by dixzint company on unix.SE
[ SmokeDetector | MS ] Bad keyword in answer, blacklisted user: VLC not working on Linux Mint Mate 17.1 by dixzint company on unix.SE
tpu- by tripleee
!!/test vars-performance
> Bad keyword in body, bad keyword in title, bad keyword in username
----------
Title - Position 1-17: vars-performance
Body - Position 1-17: vars-performance
Username - Position 1-17: vars-performance
!!/blacklist-keyword agen\W?poker
@paper1111 Blacklisted agen\W?poker
05:28
!!/blacklist-keyword situs\W?poker
@paper1111 Blacklisted situs\W?poker
!!/blacklist-website pokerolb\.com
@iBug Blacklisted pokerolb\.com
CI on d9f7f8c succeeded. Message contains 'autopull', pulling...
@iBug I believe 1/1 sites shouldn't be blacklisted... You should !!/watch first
Restart: API quota is 16021.
05:32
When it's sure there's no legitimate reason for that site to show up, it's fine to blacklist.
So reverting that for now. Watching is technically the right thing to do.
CI on 2a85da4 succeeded. Message contains 'autopull', pulling...
Not a big deal either way
Restart: API quota is 16000.
Apply !!/watch ?
!!/test pokerolb.com
05:36
> Would not be caught as a post, title or username.
:42933836 > Would not be caught as a post, title or username.
!!/watch pokerolb\.com
@iBug Added pokerolb\.com to watchlist
CI on 003b140 succeeded. Message contains 'autopull', pulling...
Restart: API quota is 15993.
[ SmokeDetector | MS ] URL in title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +6 more: needsupplementhelp.com/renux-forskolin/ by user276910 on apple.SE
tpu- by iBug
!!/test renux
> Potentially bad keyword in body, potentially bad keyword in title, potentially bad keyword in username
----------
Title - Position 1-6: renux
Body - Position 1-6: renux
Username - Position 1-6: renux
16/0, good to blacklist?
05:43
absolutely
Google search for renux is... Not recommended. I'd say it's a good candidate
!!/blacklist-keyword renux
@iBug Blacklisted renux
CI on cb5d8f6 succeeded. Message contains 'autopull', pulling...
[ SmokeDetector ] SmokeDetector started at rev cb5d8f6 (SmokeDetector: Auto blacklist of renux by iBug --autopull) (running on bwDraco/Linode)
Restart: API quota is 15945.
[ SmokeDetector | MS ] URL in title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +5 more: needsupplementhelp.com/renux-forskolin/ by cfrf on apple.SE
tpu- by tripleee
05:48
@NogShine you have to link back to the message you respond to, or say "sd k" (prefer the former because that way you avoid a race condition if somebody else posts a message just before you hit enter)
but I guess you don't have the privileges yet ...?
@tripleee It says I am not a privileged member.
Yeah.
I didn't recognize you at first but I see now that I vaguely offered you privileges back in November ...
Nov 9 '17 at 5:42, by tripleee
@NogShine if you are a sock puppet of Nish Gone I'm sure we could add you to the priv list
[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +3 more: www.supplementoffers.org/herzolex-ultra/ by bellybhargay on astronomy.SE
tpu- by iBug
@tripleee I was not active at that part of the year. So, I was not sure then.
[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, bad keyword in username, pattern-matching website in body: juniviveserum.fr/nutralu-garcinia/ by Nutralu Garcinia on askubuntu.com
tpu- by iBug
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, bad keyword in username, bad keyword with email in body, blacklisted user: Why best Vashikaran Specialist in kolkata 101% free guaranteed by Aslivashikaran Khan on dsp.SE
tpu- by iBug
05:59
!!/watch juniviveserum\.fr
@iBug Added juniviveserum\.fr to watchlist
@SmokeDetector For some reason FDSC says this report is not autoflagged...

« first day (1555 days earlier)      last day (2481 days later) »