Charcoal HQ

12:00 AM

...what?

Anyways

@NobodyNada It is a little weird -- the user has legit answers and questions on other sites, but he's just joined today on a bunch of sites so it's odd

Andy

12:11 AM

@ArtOfCode do you have an example or a way to get that pickle? I won't get to it over the weekend, but I might be able to next week before vacation, and figure out something for that analysis

ArtOfCode

@Andy it doesn't exist, yet, it's not being run

Andy

OK. I'll wait then. :) I can look at it in a few hours

ArtOfCode

The basic analysis script I threw in just gives you min, max, and average for each site, so anything better would be good :)

Andy

That's a good start

ArtOfCode

@Andy actually, hang on. I can get at EC2, so I'll just pull in that branch and run it for a while

!!/reboot

SmokeDetector

12:15 AM

Goodbye, cruel world

[ SmokeDetector ] SmokeDetector started at rev 4d62414 (ArtOfCode-: Analysis script) (running on ArtOfCode/EC2)

Restart: API quota is 18605.

[ SmokeDetector ] SmokeDetector started at rev 95ee2fe (Thomas Ward: Changes to Post class, bodyfetcher.) (running on ArtOfCode/EC2)

ArtOfCode

that's got a bug

SmokeDetector

Restart: API quota is 18604.

[ SmokeDetector ] SmokeDetector started at rev c44e0bc (ArtOfCode-: Fix index out of range bug) (running on ArtOfCode/EC2)

[ SmokeDetector ] SmokeDetector started at rev 95ee2fe (Thomas Ward: Changes to Post class, bodyfetcher.) (running on ArtOfCode/EC2)

Restart: API quota is 18602.

ArtOfCode

that also has a bug

angussidney

Bug, Bug, Bug, Goose

ArtOfCode

[ SmokeDetector ] SmokeDetector started at rev c44e0bc (ArtOfCode-: Fix index out of range bug) (running on Continuous Integration. If you're seeing this, create a config file with sane stuff in it.)

Restart: API quota is 19989.

SmokeDetector

12:26 AM

[ SmokeDetector ] SmokeDetector started at rev c44e0bc (ArtOfCode-: Fix index out of range bug) (running on ArtOfCode/EC2)

[ SmokeDetector ] SmokeDetector started at rev 95ee2fe (Thomas Ward: Changes to Post class, bodyfetcher.) (running on ArtOfCode/EC2)

ArtOfCode

bah, that's not fun

bug that I can't reproduce locally

angussidney

!!/gitstatus

SmokeDetector

On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

Restart: API quota is 18575.

angussidney

@SmokeDetector on master

not deploy

Not that it should make much of a difference

ArtOfCode

oh, whoops, that's just me checking out the wrong branch

!!/master

SmokeDetector

12:28 AM

[ SmokeDetector ] SmokeDetector started at rev 95ee2fe (Thomas Ward: Changes to Post class, bodyfetcher.) (running on ArtOfCode/EC2)

ArtOfCode

!!/gitstatus

SmokeDetector

On branch deploy
Your branch is up-to-date with 'origin/deploy'.
nothing to commit, working directory clean

Restart: API quota is 18569.

quartata

github.com/quartata/SmokeDetector/commit/…

I'll make a pull request if everything looks good

SmokeDetector

[ SmokeDetector ] SmokeDetector started at rev c44e0bc (ArtOfCode-: Fix index out of range bug) (running on ArtOfCode/EC2)

ArtOfCode

totally won't mess anything up if I just nuke the entire bodyfetcher queue right

angussidney

12:34 AM

Not much spam ATM, so it shouldn't matter much

SmokeDetector

Restart: API quota is 18562.

quartata

!!/gitstatus

SmokeDetector

On branch time-in-queue
Untracked files:
(use "git add <file>..." to include in what will be committed)

[31mbodyfetcherQueueTimings.p[m

nothing added to commit but untracked files present (use "git add" to track)

quartata

Ooh, we should strip those ANSI color codes.

ArtOfCode

eh

angussidney

12:36 AM

[tag:dev-on-production]

quartata

flake8 doesn't like my code?

ArtOfCode

@angussidney best kind of development

@quartata flake8 doesn't like anybody's code.

quartata

So it likes the spaghetti code with no extra line breaks

but not my pristine code

so mean

ArtOfCode

Turn thee thrice ninety degrees anticlockwise, then flip thee head over heels backwards. Proceed to sacrifice thine firstborn child at the altar of Flake, and it shall forevermore accept thine code.

4

quartata

I don't even understand what it's complaining about

Ooh woops

OK that's valid

angussidney

12:39 AM

@ArtOfCode don't forget to create three extra commits saying 'Thanks flake for making me put in those extra blank lines'

quartata

@angussidney That's exactly what happened

ArtOfCode

@angussidney "thanks flake" also works, as proven by my commits to the time-in-queue branch earlier

quartata

OK, github.com/Charcoal-SE/SmokeDetector/pull/599

angussidney

$ git log | grep [Ff]lake | wc -l
105

NobodyNada

:D

angussidney

12:42 AM

Just over 2% of our commits are thanking flake

ArtOfCode

so, @Andy, what data was it you wanted? The file itself? I've got a file now, the data is still inaccurate because it hasn't averaged out over a long time yet, but it's something

quartata

We really really need to refactor findspam

Andy

@ArtOfCode I really just needed the file so I have something to play with. Accurate data, right now, isn't important

angussidney

TBH, we really need to refactor Smokey for a lot of things. Python 3 support, re-write it to have better MS support (but keep it optional), use OOP, etc

ArtOfCode

@Andy gotcha. Same email?

quartata

12:50 AM

Yeah but findspam is the thing we modify the most

Andy

Yeah

quartata

And man test_post makes no sense.

Andy

@angussidney "thanking"

quartata

It generates why data for regex reasons without checking if it is a regex reason

And it tests on title username and body first and then checks if it needs to

ArtOfCode

On its way, @Andy

quartata

12:55 AM

@ArtOfCode Ooh time_in_queue would be really nice

combined with the rule performance data

ArtOfCode

@quartata that's... sort of the point of me writing that code

quartata

I know just giving encouragement

I hadn't noticed the branch yet actually

angussidney

Looking at the queue code, it seems to focus on the sites which used to be spam magnets but aren't anymore

Maybe that should be the first thing we update once @ArtOfCode gets us some data

ArtOfCode

@angussidney let's let this run for a while to let the data average out, then we can look at it in a day or two

angussidney

!!/location

SmokeDetector

1:02 AM

@angussidney ArtOfCode/EC2

Andy

@ArtOfCode, what do these numbers in the lists represent? Seconds? Milliseconds? I ask, because some of these are really small: u'health.stackexchange.com': [0.000291],

However, you can clearly see when each queue is full, because you get a tiny value like that as it calls the API

ArtOfCode

@Andy seconds. The minimums are tiny because we fire the API call right after adding a post to the queue if the threshold is met, which takes ~0.0003s.

Andy

That's going to mess with averages :)

ArtOfCode

@Andy should be okay, actually, because it's balanced out by the posts that were added in right at the start of the site queue

i.e. the max values

SmokeDetector

[ SmokeDetector | MS ] Manually reported answer: Tkinter / ttk on Mac OS X Proper System Background Color (canvas) by blala on stackoverflow.com

quartata

1:08 AM

@SmokeDetector k

ArtOfCode

:D I can set Windows' taskbar to behave like Ubuntu's

SmokeDetector

[ SmokeDetector | MS ] Pattern-matching website in body: Map of neighbourhoods for Canada by Tom on gis.SE

angussidney

@ArtOfCode be on the left hand side?

ArtOfCode

@angussidney yeah, and hide itself automatically until I hit the left side with the mouse

SmokeDetector

CI on 92adc01 succeeded.

ArtOfCode

1:21 AM

so I now get the full screen dedicated to app windows

SmokeDetector

fp- by angussidney

CI on ec0b1fa succeeded.

NobodyNada

@Art I've fixed the regex thing. Should I use destroy!, or check the return value of destroy?

ArtOfCode

@NobodyNada the former will throw an exception if it fails; the latter lets you display a more friendly error message

SmokeDetector

[ SmokeDetector ] SmokeDetector started at rev ec0b1fa (ArtOfCode: Merge pull request #598 from Charcoal-SE/time-in-queue) (running on ArtOfCode/EC2)

NobodyNada

@ArtOfCode I think we'd rather throw an exception here; destroy shouldn't fail, so the detailed error logging seems more appropriate

SmokeDetector

1:24 AM

Restart: API quota is 18448.

ArtOfCode

Fair enough

quartata

@ArtOfCode Thanks for the merge

waits for spam

NobodyNada

pushed, thanks @Art!

SmokeDetector

CI on 33ab6fe succeeded.

ArtOfCode

!!/pull

SmokeDetector

1:27 AM

[ SmokeDetector ] SmokeDetector started at rev 33ab6fe (ArtOfCode-: Make analysis script python 2) (running on ArtOfCode/EC2)

Restart: API quota is 18437.

CI on a46f747 failed.

ArtOfCode

how much you want to bet me it's flake?

quartata

That's cheating

It's always Flake

Also why do you guys have Travis and CircleCI

ArtOfCode

we race them

quartata

Spoiler alert Travis wins

SmokeDetector

CI on 13f9301 succeeded.

quartata

1:35 AM

Which one does Smokey use then?

And why doesn't Circle trigger on PRs

ArtOfCode

@quartata no, actually Circle wins more often than not

@quartata whichever wins the race

quartata

@ArtOfCode Really? Huh I guess this time was just a fluke

ArtOfCode

@quartata nah, it's being slow today

!!/pull

SmokeDetector

[ SmokeDetector ] SmokeDetector started at rev 13f9301 (ArtOfCode-: Ain't flake great) (running on ArtOfCode/EC2)

Restart: API quota is 18424.

ArtOfCode

math: min 0.000364, max 1284.30336, avg 463.017850933
mathoverflow: min 0.00045, max 1622.969337, avg 681.6722635
biology: min 0.000449, max 5.41631, avg 2.7083795
askubuntu: min 0.000298, max 615.706043, avg 99.8058600625
gaming: min 0.000429, max 1312.966034, avg 656.4832315
stackoverflow: min 0.000217, max 306.296978, avg 69.2360307909
mathematica: min 0.000299, max 1768.700984, avg 461.55807175
ru: min 0.000371, max 2614.520416, avg 1558.4881014
puzzling: min 0.000424, max 412.697364, avg 206.348894

angussidney

1:40 AM

58 mins ago, by angussidney

$ git log | grep [Ff]lake | wc -l
105

106*

ArtOfCode

Bahaha. Line in the referring sites table for the Smokey repo view stats: 820915.rdfx.org 3 3

3 views, 3 unique visitors

NobodyNada

that's your site for school, isn't it?

ArtOfCode

which means that some poor bastard reviewed my website, clicked a link, took one look and went "I understand nothing" and promptly left

angussidney

lol

quartata

Man I wish there was a better way to test rules other than praying. I tested it on a post object I made myself but I have no idea if it will work on wild posts from the API

We should have a !!/test <se or metasmoke id>

angussidney

1:48 AM

Also, I'm about to apply for work experience at <insert large-ish software company here>, nobody minds if I list my contributions here on my application form?

ArtOfCode

go for it, and good luck

angussidney

Thanks

ArtOfCode

and if they come to us for references, I'll be sure to tell them how much of a pain you are ;)

angussidney

I can guaratee that the spammers will say that I'm a horrible person :)

ArtOfCode

I'm probably applying for GSoC in a few days' time, and I fully plan to list my stuff here on that application too

angussidney

1:51 AM

Good luck

ArtOfCode

ta

NobodyNada

@ArtOfCode neat!

ArtOfCode

Applications open 20th, so I'll have to see if there's anything interesting there

(well, either interesting or not-boring enough that I'm willing to work it for a summer for money)

angussidney

Random rant of the day: backlit keyboards can be bad, as you can see all the dead skin and hair underneath the keys in the lights

ArtOfCode

time for a can of compressed air

angussidney

1:57 AM

Tried that, but it isn't very effective if you don't take off the keycaps

I've ordered a keycap remover online, it should arrive soonish

quartata

I wonder if it would be worth asking Pops about getting data on posts scanned and ignored by Smokey but then deleted later by red flags

ArtOfCode

@quartata I suspect that'll be a small number

quartata

I'd love to get false negative data but we can't get deleted posts

ArtOfCode

Might be worth having the data, though.

angussidney

We could probably provide them with a list of every post/site ID we have on MS, then ask for all data since we started using MS

quartata

2:02 AM

@ArtOfCode not so much number as what and more importanly where. if site mods and 10ks don't know about smokey things might not get manually reported (and habitual spammers tere won't end up in our rules)

ArtOfCode

@Pops are you able to get hold of some data for us that we can't get? Ideally, we'd need site, post ID, and post creation date for every post that was deleted by red flags, for as far back as you can get it.

@quartata The majority of mods know about Smokey, if only because we had a pinned message in TL for most of a month

quartata

whoa whoa whoa not every red flag

ArtOfCode

@quartata Why not?

angussidney

@ArtOfCode because then we'd have posts that we actually caught, but we didn't catch because we didnt have MS

ArtOfCode

@angussidney sure, but that's going a good two years back, and I doubt we'll get data for that long :)

quartata

2:06 AM

Also I was hoping for a more "foot in the door" approach in terms of CM diplomacy

ArtOfCode

"foot in the door"?

quartata

Asking for something small first.

Anywho how do we know if Smokey scanned it?

ArtOfCode

shrug I'd rather ask for what we could do the most with and then have Pops tell us that getting that isn't possible

quartata

Could have been during rollover or downtime

ArtOfCode

@quartata just assume Smokey scans everything. It's near enough true as to make no difference.

quartata

2:10 AM

All right.

When I get a chance I'm going to rewrite the magic eight ball with Keras

Thomas Ward

2:35 AM

i was asleep what's up?

@ArtOfcode ^

Riker

~~the sky~~ pls don't kill me

Thomas Ward

punts @Riker into orbit

Riker

now the sky is down

Thomas Ward

@quartata I may want to reapproach your handling - i did a very thorough refactor to check that the Parent object is also a Post and such

must compare to my work

because i already had a branch in place for that refactor in the post object

angussidney

If it works well in practise, why not leave it?

Don't just revert Quaratata's work because you've already done some

quartata

2:38 AM

No it's fine, I didn't know there was a parent prop in another branch

basically it's @Undo's fault

!!/blame

SmokeDetector

@quartata It's ArtOfCode's fault.

quartata

Meh that works too

Riker

haha

quartata

@angussidney It's a three line getter

Thomas Ward

@angussidney i did not mean revert

i meant improve

type checks, fix a major post init issue, etc.

not a HUGE issue

but first, pizza

though keep in mind I am also working on refactoring apigetpost to return a Post object too so...

... may result in some reverts if stuff conflicts, but ultimately not a major revert - just a different Post object approach slightly in unit and passing things

quartata

2:47 AM

!!/alive?

SmokeDetector

@quartata Of course

quartata

Quiet. Are things getting scanned?

Thomas Ward

@quartata huh, interesting, nice work. It's missing some things in some places - I've started getting anal about type hinting, etc. but other than that nice job

let's hope it merges cleanly ;)

(I have to merge into my branch to rebase)

quartata

Oh I would have done type hinting but didn't see it on any of the others. I value code consistency more than strict typing

Thomas Ward

@quartata uhm just one question

where're you setting parent= in the post object?

because I don't think that's implemented anywhere and would probably be implemented in apigetpost

which is what i was working on

quartata

2:53 AM

Undo did it in bodyfetcher

Thomas Ward

ah, nice, missed that, though that may have been me.

Glad it's there though :)

yawns I need more coffee

i thought that was in a branch, only/

@quartata I'm slowly becoming paranoid BTW about some PEP rules :)

quartata

Oh oops I didn't check blame on it

Thomas Ward

doesn't matter it's there

I hope answers are getting checked

@quartata it may be that it's not evil spam hour yet

!!/apiquota

SmokeDetector

@ThomasWard The current API quota remaining is 18244.

quartata

@ThomasWard Not a bad thing really. PEP is good

Thomas Ward

2:55 AM

okay then checks in 3 minutes

!!/apiquota

SmokeDetector

@ThomasWard The current API quota remaining is 18244.

quartata

MS says things are being scanned.

!!/rev

SmokeDetector

@quartata 13f9301 (ArtOfCode-: Ain't flake great)

Thomas Ward

indeed.

@quartata Has it been checking answers in the past several hours? Since I pushed the fix to test answers proper

@ArtOfCode ^

!!/location

SmokeDetector

@ThomasWard ArtOfCode/EC2

quartata

2:58 AM

!!/test-a asdfasdf

SmokeDetector

> Would not be caught as an answer.

quartata

?

Thomas Ward

no i meant from the system ;P

test creates its own post struct and submits that for testing :)

angussidney

!!/test-a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa d

SmokeDetector

> Few unique characters in answer, repeating characters in answer
----------
Post - Contains 3 unique characters
Post - Repeated character: *aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa*

Thomas Ward

2:59 AM

i meant like the API and such, not the tests.

i.e. actual spam items

quartata

Just checking if FindSpam is OK. it was when I pushed

Transcript for