Charcoal HQ

Thomas Ward

3:17 PM

@ArtOfCode you did a pull for the most recent rev too right?

ArtOfCode

@ThomasWard no, you did last night :P

Thomas Ward

we did add code that deletes queue pickles so we could clear the other instances last night, right before spamfest.

@ArtOfCode there's a commit after that that we didn't get on yours because spam floods happened.

!!/location

SmokeDetector

@ThomasWard ArtOfCode/EC2

Thomas Ward

!!/rev

SmokeDetector

@ThomasWard 10425ad (Thomas Ward: TEMPORARY: Add "Destroy Queue Pickles" to ws.py init.)

Thomas Ward

3:17 PM

yeah you're behind.

!!/pull

ArtOfCode

shrug it's not crashing

SmokeDetector

[ SmokeDetector ] SmokeDetector started at rev d74ccd6 (ArtOfCode-: Revert "Commenting in the queue_timings analysis script") (running on ArtOfCode/EC2)

Thomas Ward

by only a few commits, but we wanted to get that "destroy pickles" out so that the pickles behave right ;)

so instead of nuking every start... :P

SmokeDetector

Restart: API quota is 17317.

Thomas Ward

in any case all is now well :)

except for my stomach it's demanding food, back in a few

ArtOfCode

3:24 PM

@Cerbrus I'm pretty much happy with the multiple urls thing now. Did you remove the testing call?

J F

@Glorfindel Your commit email is incorrect in that last commit on the userscripts repo.

Thomas Ward

@ArtOfCode I'm just glad we fixed the explosions in your code, it took some doing, and some 'convert list into dict' magic but eh it's working now.

ArtOfCode

@ThomasWard I seemed to manage okay for my instance :P

just nuked the pickles manually and let the code create the dicts correctly

Thomas Ward

@ArtOfCode yep that's what it's supposed to do :)

but i'm leaving the improper handling code in place, because we still have lists when we needed dicts in some places

which is why i had to add that if type(BLAH) is... stuff

ArtOfCode

that shouldn't happen

Thomas Ward

3:34 PM

that was the core problem yesterday, prompting the coding spree

ArtOfCode

just nuke the bodyfetcherQueue.p and bodyfetcherQueue.txt files, and it'll recreate the queue correctly

Thomas Ward

@ArtOfCode yep. That's the idea, now, but just in case we have something going wrong (population via an id list that's returned returned a list, when it needed to be a dict, inside the self.queue object), the code is there :)

ArtOfCode

@ThomasWard if you've nuked the files, that won't happen.

Cerbrus

@ArtOfCode Didn't remove it yet

Thomas Ward

indeed :)

(but i'm an alarmist - i'd rather have the 'fix it' code in there still) Nothing's blown up yet so we're all good :)

Cerbrus

3:36 PM

And removed... Push inc

ArtOfCode

Nice, good to merge @Cerbrus?

Cerbrus

I'd say so, yea

Ferrybig

4:57 PM

!!/alive

SmokeDetector

@Ferrybig Of course

Thomas Ward

5:34 PM

@Ferrybig is it dead?

did it die?

SmokeDetector

[ SmokeDetector | MS ] URL in title: www.xaddition.net/gc-fresh by user88930 on graphicdesign.SE

tpu- by teward

J F

@SmokeDetector k

Ferrybig

@ThomasWard I found it weird that nothing was posted to the room over an hour, so I wanted to check if he was alive

Thomas Ward

@Ferrybig yeah we had that oddity yesterday but for several hours

discovered the queue explodified and we needed to fix :)

SmokeDetector

6:00 PM

[ SmokeDetector | MS ] Manually reported answer: How do I link to wechat from a webpage? by Ceceliading on stackoverflow.com

ArtOfCode

wanna bet it was still in the bodyfetcher queue?

DavidPostill

!!/isblu stackoverflow.com/u/7732578

SmokeDetector

@DavidPostill User is blacklisted (7732578 on stackoverflow.com).

ArtOfCode

the thing where we queue up posts to fetch the post body

cuz we have to do that a bunch at a time

the last time I explicitly used a stack was... never, so no :P

@QPaysTaxes /

Thomas Ward

6:25 PM

@QPaysTaxes stacks and queues aren't the same.

just saying :P

implied by...

6 mins ago, by QPaysTaxes

s/stack/queue

just saying ;)

nope :)

ArtOfCode

6:45 PM

So, it's just taken me two hours to work out how to add tests to metapi...

Thomas Ward

@ArtOfCode o.O

ArtOfCode

@ThomasWard blame npm, jquery, and mocha.

Thomas Ward

!!/blame

SmokeDetector

@ThomasWard It's QPaysTaxes's fault.

Thomas Ward

no it's ^ that one's fault

ArtOfCode

6:47 PM

ah, right

quartata

@ArtOfCode don't do it

ArtOfCode

@QPaysTaxes I mean I know testing is meant to be thorough but did you have to make it take two hours to set up? ;)

@quartata don't do what

Thomas Ward

@ArtOfCode it

ArtOfCode

ah, much clearer, thanks

Thomas Ward

:P

quartata

6:48 PM

NPM is from the pits of hell and Mocha is slow as molasses and as hard as bricks to use

Just write your own test harness you'll be done faster

ArtOfCode

well, given that I've already done it...

quartata

Next think you know you'll be using Gulp and Webpack

it's a trap

ArtOfCode

those ain't so bad either

quartata

I guess I'm too late

you will be forever ensnared in 5-tool deep toolchains

SmokeDetector

[ SmokeDetector | MS ] Offensive body detected: how to get http header response for zeep client by Joe on stackoverflow.com

tpu- by DavidPostill

ArtOfCode

6:54 PM

@quartata Nah. I don't use 'em at the moment, but they have their place

SmokeDetector

[ SmokeDetector | MS ] Bad keyword in title: Golovkin vs Jacobs Live Stream by Kalabox on puzzling.SE

tpu- by DavidPostill

Rand al'Thor

@SmokeDetector k

I just came to report that.

Thomas Ward

o/ @Randal'Thor

Rand al'Thor

@ThomasWard Don't mind me, just came 'cos I saw spam :-)

goes back to obscurity

Thomas Ward

@Randal'Thor drags you back into the light and chains you to the pole at the center of the room

Rand al'Thor

7:00 PM

The Dragon Reborn will not be chained!

balefires pole

Thomas Ward

makes a note the pole is tunsten-carbide composite, and resistant to the heat of reentry through the sky, and is what's left over from an orbital strike, and therefore cannot be burned or lit aflame or melted

Rand al'Thor

Looks like a mod caught that PSE spam: it was nuked with a score of only -3.

@ThomasWard Balefire can destroy anything.

Except cuendillar.

ArtOfCode

Yep, I see three flags, one Rand's and two from others here

therefore the last must have been a mod

Rand al'Thor

I see Emrakul's avatar lurking.

ArtOfCode

There are plenty mods who lurk around here

Thomas Ward

7:03 PM

@Randal'Thor well considering that the pole is also SmokeDetector you can't kill SmokeDetector ;)

do that and we all get angry at you

ArtOfCode

I count 12 mods here at the moment

Rand al'Thor

@ArtOfCode TL #2?

ArtOfCode

@ThomasWard nah, we just click the failover button and laugh because it takes more than that to kill Smokey

pretty much

Rand al'Thor

Only 12? I'm surprised it isn't more, tbh.

Thomas Ward

@ArtOfCode But public.

ArtOfCode

7:05 PM

@Randal'Thor the whole room is pretty quiet at the moment. We're only just past one page of avatars, normally we're closer to 2

Rand al'Thor

The weekend effect?

ArtOfCode

probably

People take Saturdays off, and spammers take Sundays off. That works, I guess

Rand al'Thor

I guess all the free publicity on main meta did wonders for this room's user count.

ArtOfCode

it also means I am not a people, since I'm still here, and neither are spammers.

Rand al'Thor

@ArtOfCode So ... spammers are Christians and Charcoal people are Jews?

ArtOfCode

7:07 PM

@Randal'Thor it actually didn't have much impact - we got a bunch of people signing up to autoflagging (something like 130), but the room population didn't change much

Rand al'Thor

@ArtOfCode Well, there was nowhere near 2 pages of avatars last time I was around here.

ArtOfCode

@Randal'Thor A majority of spammers are Indian, actually, which would imply religions other than Christianity.

Hovercraft Full Of Eels

7:20 PM

Hello, Pete S here. Question: How do I request Charcoal privileges?

J F

@HovercraftFullOfEels charcoal-se.org/smokey/Privileges

Thomas Ward

define "privileges" :)

Hovercraft Full Of Eels

being able to interact with smoke detector for example

!!/amiprivileged

SmokeDetector

@HovercraftFullOfEels No, you are not a privileged user. Please see the privileges wiki page for information on what privileges are and what is expected of privileged users.

rene

@ThomasWard you can trust @HovercraftFullOfEels, they are a regular member of SOCVR

Thomas Ward

7:23 PM

@rene does he have privs in SOCVR?

just curious :)

rene

Not yet, I just offered do some dry-runs before I grant privs there

So I'm not sure what your policy is

Thomas Ward

yawns

where'd i put the privileged lists...

FOOOOOEY i lost my data

@HovercraftFullOfEels you have read the privileges wiki page and understand what we expect of you right?

rene

expects no as the answer

Hovercraft Full Of Eels

@ThomasWard: yes, I have

SmokeDetector

CI on 613e689 succeeded. Message contains 'autopull', pulling...

ArtOfCode

7:29 PM

@rene "do you look sane" is a good description of it

SmokeDetector

[ SmokeDetector ] SmokeDetector started at rev 613e689 (Thomas Ward: Add hovercraft to CHQ privs. --autopull) (running on ArtOfCode/EC2)

Thomas Ward

@HovercraftFullOfEels blah blah don't abuse blah blah familiarize with commands blah blah hail satan and give your soul as sacrifice

SmokeDetector

Restart: API quota is 19964.

Thomas Ward

basically Smokey owns your soul now

Hovercraft Full Of Eels

Thank you.

Thomas Ward

7:30 PM

@rene should he have SOCVR privs with Smokey too?

Hovercraft Full Of Eels

but my soul has already been given/corrupted, etc...

ArtOfCode

what nice birthday presents you give @ThomasWard

Thomas Ward

@ArtOfCode i'm stealing his neural energy either way so blah :P

rene

@ThomasWard I'll add those myself after I have seen their interactions here, so not yet.

Thomas Ward

mmkay

returns to lurking and stabbing Smokey code

@ArtOfCode speaking of presents

give me coffee

ArtOfCode

7:31 PM

you wish

more to the point, you assume I actually have coffee

which is where you'd be wrong, because I hate the stuff

Rand al'Thor

@ArtOfCode Well, @HovercraftFullOfEels doesn't look sane - he looks like a Pythonian, and they're all loonies ;-)

Hovercraft Full Of Eels

@Randal'Thor: :)

ArtOfCode

@Randal'Thor Obviously, I mean they only protect the entire network from spam, and what sane person would do that? ;)

Rand al'Thor

@ArtOfCode Oh hey, someone else who hates coffee!

@ArtOfCode Monty Python invented spam. They should be Charcoal's greatest enemies.

ArtOfCode

I take that back, I do have a jar of coffee. Terrible Tesco-cheapest coffee, exists only for the once a year a guest turns up

@Randal'Thor heh. One Python invents it, another crushes it.

speaking of, I wonder how deletion times are doing

Jan Dvorak

7:35 PM

@ArtOfCode I'd recommend that you don't use it

Undo

Anything on fire?

ArtOfCode

Hmmm. Could do with some more improvement, really. But hey, that's what 20k quota is for.

@Undo only Smoking.

Undo

boring :P

Rand al'Thor

Undo

@ArtOfCode Still consistently under that 200s mark, except for the hiccups we've had in the last couples days

SmokeDetector

7:37 PM

[ SmokeDetector | MS ] Pattern-matching website in body: Integrating Wordpress Content into HTML site by Cobra on stackoverflow.com

fp- by J F

Undo

@SmokeDetector why

SmokeDetector

@Undo Body - Position 127-146: naturalsolutions.co

ArtOfCode

@Undo Aye. We really should re-work the queue thresholds, though, the stats are not encouraging. There are posts that have sat there for 4 hours before being scanned.

Undo

@ArtOfCode Oh, you have that data?

Awesome. I'd like to see it at some point

ArtOfCode

@Undo yep

how about right now

genealogy: min 0.000501, max 0.000501, avg 0.000501
buddhism: min 0.000553, max 3000.934901, avg 761.78633775
gamedev: min 0.000383, max 2779.184745, avg 372.324236118
hermeneutics: min 0.000413, max 2994.602917, avg 487.0607365
ru: min 0.000334, max 2014.973346, avg 553.382721229
arduino: min 0.000426, max 1863.557049, avg 351.9857215
ja: min 0.000641, max 1385.866027, avg 692.933334
opensource: min 0.000609, max 444.935946, avg 222.4682775
drupal: min 0.000253, max 0.002859, avg 0.000524018181818

Undo

7:39 PM

Happen to have q1/q3 on that?

ArtOfCode

Nope, but not difficult to get once I find the math for calculating it

Undo

k

ArtOfCode

Might as well get stddev while I'm there, gimme 10

Undo

@ArtOfCode It's just "what's halfway between median and minimum"

Median, not mean

ArtOfCode

fair

Rand al'Thor

7:40 PM

@ArtOfCode Are these stats about time to deletion?

Undo

Time to scan after enqueuing

ArtOfCode

@Randal'Thor no, that's how long in seconds a post sat in our queue before we got the body from the API.

Rand al'Thor

Because if so, I'm disappointed at how high Hardware Recs is, you pair of slackers :-P

@ArtOfCode Oh OK. pretends to know what that means

Undo

Also, it'd be great to know volume distribution there

I.e. what happens if we take the n highest time site and just set the queue to 1?

ArtOfCode

@Undo on it

M.A.R.

7:45 PM

@Randal'Thor IKR, that's me in half these conversations

SmokeDetector

7:55 PM

CI on 5146da4 failed.

Thomas Ward

what

ArtOfCode

me

ignore

@Undo

genealogy: min 0.000501, max 0.000501, avg 0.000501, q1 0.000501, q3 0.000501, stddev 0.0
buddhism: min 0.000553, max 3000.934901, avg 761.78633775, q1 0.000596, q3 1375.803787, stddev 1094.15015412
gamedev: min 0.000383, max 2779.184745, avg 372.018168972, q1 0.000619, q3 513.761358, stddev 714.871742247
hermeneutics: min 0.000413, max 7683.108413, avg 822.51011495, q1 0.000629, q3 775.219645, stddev 1738.81327298
ru: min 0.000334, max 2014.973346, avg 564.587593877, q1 183.737, q3 867.672132, stddev 468.573014447

or with counts:

genealogy: min 0.000501, max 0.000501, avg 0.000501, q1 0.000501, q3 0.000501, stddev 0.0, count 1
buddhism: min 0.000553, max 3000.934901, avg 761.78633775, q1 0.000596, q3 1375.803787, stddev 1094.15015412, count 12
gamedev: min 0.000383, max 2779.184745, avg 372.018168972, q1 0.000619, q3 513.761358, stddev 714.871742247, count 36
hermeneutics: min 0.000413, max 7683.108413, avg 822.51011495, q1 0.000629, q3 775.219645, stddev 1738.81327298, count 20
ru: min 0.000334, max 2014.973346, avg 564.587593877, q1 183.737, q3 867.672132, stddev 468.573014447, count 220

SmokeDetector

[ SmokeDetector | MS ] Shortened URL in body: Craig David thank you😏 by user3855 on musicfans.SE

Undo

8:12 PM

@ArtOfCode What kind of timescale are we looking at?

(also, count is post count not req count, right?)

ArtOfCode

@Undo around or just under 24h

@Undo yeah

Undo

Huh, surprised we only saw 1772 on SO

ArtOfCode

I've got a CSV of that too

Undo

Oh, but that's websocket dark magic. Nevermind

But yeah, this'll be useful. Thanks :)

On sites like boardgames, we could just set the bar to 1 and be done with minimal API usage

ArtOfCode

@Undo gist.github.com/ArtOfCode-/4ac9b49db07c66fa9df8c0842c0b8ecf

Jan Dvorak

8:14 PM

Speaking of black magic, do we still want to keep that reason?

ArtOfCode

Hang on, something's off here. 6527 posts total.

!!/apiquota

SmokeDetector

@ArtOfCode The current API quota remaining is 19745.

ArtOfCode

roll'd over. But anyway, we should have seen way more posts than 6.5k over 24 hours

Do we only queue a post if we're already partially sure it's suspicious, or something?

J F

@SmokeDetector What happened here?

Jan Dvorak

Apart from that it killed Fire?

Undo

8:22 PM

@ArtOfCode We used to dump queues immediately if we were suspicious, but we always enqueue everything

J F

@JanDvorak There’s no MS report for that one.

Undo

And the metasmoke stats show ~100k/day, and they're only recorded from successful API calls

ArtOfCode

@Undo Huh. So we've lost a bunch of posts somewhere

Jan Dvorak

The question itself is a bare shortened URL that I dare not follow

Glorfindel

@JanDvorak nothing bad, just a bunch of concert photos.

Jan Dvorak

8:27 PM

Oh. Not abu then?

SmokeDetector

[ SmokeDetector | MS ] Blacklisted user: Increase in testosterone can reverse the execss fat by Champion on health.SE

Jan Dvorak

seed?

ArtOfCode

wait for it

Cerbrus

8:56 PM

@JanDvorak What killed fire?

Jan Dvorak

47 mins ago, by SmokeDetector

[ SmokeDetector | MS ] Shortened URL in body: Craig David thank you😏 by user3855 on musicfans.SE

Cerbrus

Hm, MS api isn't returning data for that one.

> "Post not found for //musicfans.stackexchange.com/questions/4973. It may have been reported during a period of metasmoke downtime."

On the MS page

ArtOfCode

bet you it's the emoji

SmokeDetector

fp- by Jason C

Cerbrus

@ArtOfCode m.erwaysoftware.com/posts/by-url?url=//…

MestaSmoke just has no data for that url

Which actually might be a bug

I'm fixing fire to show a proper message for those cases, though.

ArtOfCode

9:04 PM

3 mins ago, by ArtOfCode

bet you it's the emoji

Cerbrus

Added a popup message for those cases where no report is available.

@ArtOfCode The emoji in the title? Is it still registered in MS and is the search just failing, or does it fail before that?

ArtOfCode

@Cerbrus I'd guess there isn't even a record, because the database rejected it because it hates anything that isn't US-ASCII

Even making it put up with Unicode can be a fight

Jan Dvorak

URL encode?

Undo

it's MySQL that causes issues

J F

9:20 PM

Store all Unicode stuff as utf-8 base64?

Jan Dvorak

URL encoding might be more efficient

or, perhaps, recode the database to UTF-8?

SmokeDetector

9:47 PM

[ SmokeDetector | MS ] Mostly non-Latin body: What is a good font for code presentations? by frank on codereview.SE

bwDraco

rofl

Undo

10:11 PM

@JanDvorak The database is already coded as UTF8MB4

Pretty sure the posts table is too

But then you've got to update individual columns

I learned that today for sure

An issue would be nice, otherwise I'm gonna forget about it before I'm done with this event thing.

ArtOfCode

@Undo there's one there already from David a couple months ago

Undo

oh, okay

angussidney

o/ everyone

ArtOfCode

o/

you're up early

angussidney

9:19 am

Better than yesterday I suppose :P

ArtOfCode

10:20 PM

Ah, yesterday was just late then :)

angussidney

I don't develop or use FIRE, but you could ask @Cerbrus about that

@QPaysTaxes though I think the info in this should be the most accurate: github.com/Charcoal-SE/userscripts/blob/master/fire/…

And ctrl+space sounds like the key to open the last report IIRC

ArtOfCode

@QPaysTaxes different people develop each script

you can read the userscript header to find out who the author and contributors are

angussidney

@QPaysTaxes Cerbrus is part of charcoal too :)

Cerbrus

Eh, yea, the readme is a little outdated

That.txt file angus linked should be correct

Submit selected feedback: [Enter] / [Space]

Not [Enter] / [Esc]

(Fixed that)

M.A.R.

10:47 PM

Wow, this room has 29 stalkers now

And you wonder why you don't get to flag stuff in time . . .

angussidney

I don't wonder why.

I know it's @Glorfindel and @tripleee's faullt.

DavidPostill

!!/blame

SmokeDetector

@DavidPostill It's angussidney's fault.

DavidPostill

hahahaha

angussidney

@SmokeDetector of course, I ~~nearly always~~ occasionally get the last flag

ArtOfCode

11:16 PM

3

NobodyNada

heh

o/

angussidney

@ArtOfCode is that yours?

ArtOfCode

@angussidney nope

straight off imgur

quartata

11:51 PM

ugh, no spam...

trying to test a websocket thing

SmokeDetector

Does this help at all?

quartata

no

PostsChannel

angussidney

Ah, right

ArtOfCode

@quartata what channel you need?

oh, posts

quartata

I wanted an actual post though

ArtOfCode

11:57 PM

yeah, lemme see if I can figure out how they're structured

quartata

to see how they're formatted

ArtOfCode

oh

@quartata you get the post partial rendered

Transcript for