Charcoal HQ

« first day (263 days earlier) ← previous day next day → last day (3564 days later) »

SmokeDetector

4:00 AM

[ SmokeDetector ] Offensive title detected: Why did Drax call Gamora a whore? on scifi.stackexchange.com

1 hour later…

SmokeDetector

5:03 AM

[ SmokeDetector ] All-caps title: SIMPLE SCALAR SIMULATOR FOR BRANCH PREDICTION on stackoverflow.com

[ SmokeDetector ] All-caps title: MY DREAM IS TO BE A DOCTOR. WHAT IS THE NOUN IN THIS SENTENCE on english.stackexchange.com

7 hours later…

SmokeDetector

11:48 AM

[ SmokeDetector ] All-caps title: MEDICINE FOR BODY BUILDING on meta.stackexchange.com

3 hours later…

SmokeDetector

2:23 PM

[ SmokeDetector ] Offensive title detected: What does “idiot-proof-ness” mean? on english.stackexchange.com

Annonomus Penguin

@Seth As long as it is not a penguin, I won't question it q:

SmokeDetector

2:53 PM

[ SmokeDetector ] Offensive title detected: What does “idiot-proof-ness” mean? on ell.stackexchange.com

2 hours later…

SmokeDetector

4:44 PM

[ SmokeDetector ] All-caps title: RUNTIME ERROR DURING RECURSION on stackoverflow.com

1 hour later…

Andy

6:05 PM

I have a gift for you guys. I've been running a slightly modified version of SmokeDetector (the old one, now that I see it's getting upgraded) and storing stuff in an SQLite database.

It's been running since July 27th and except for a few outages while testing, should have almost everything that's been on the real time feed since then. This does not use the API in any way, so it only contains data that is available on that feed.

I know you guys wanted to build some machine learning stuff to help smoke detector out. I figured this would be easier than parsing some log files.

This is using RegEx's I've seen others using - named SmokeDetector itself and Jan Dvorak's user script - and running each post against all those reg exs. If you know of others, let me know and I'll get them added to this.

This pushes to BitBucket (because GitHub limits a file size to 100MB, and I passed that a day or two ago), every 1000ish records on the real time feed. This averages out to about once every 90 minutes during the week. The README is updated with each push giving a few break downs of the dataset.

A few things of note:

- SmokeDetector has relatively few hits compared to Jan's script. Granted, his has a ton of false positives, but I was surprised by the smaller number of hits SmokeDetector actually has.
- The "system_add_date" in the post_types table isn't accurate until 8/6/2014. I was caching the date when I added a record. That's why it looks like those dates in July have 50K questions a day. Oops. Other than the date on those records, the detection rules are accurate.

(aww...bullets don't work?)

My goals for releasing this is to help you have an easily querably dataset, and to see if you can generate patterns for either more accurate RegExs or a machine learning algorithm. I fully suspect that of the 140K-ish records that didn't trigger a rule that some of those should have been flagged for a reason we don't have a rule for yet.

Link: Websocket Database

If you have questions about the table, let me know.

As of right now: There are currently 141586 posts (in the database) and 2374 have been classified as something other than "Good".

hichris123

@Andy <3 Thanks so much! :D

Andy

Glad to help.

hichris123

@Andy This will totally help a lot for ML. I've been trying to work on it, but I didn't have enough of a dataset to get an accurate algorithm.

Andy

6:23 PM

I'm not sure this is enough (yet). I was hoping Tim would release his spam dataset, but alas, it's been 6-8 units already. I'll keep adding to this though, so it should help as it gets more data

And, until I hit a bitbucket file size limit, I'll keep it hosted there too

5 hours later…

SmokeDetector

11:16 PM

[ SmokeDetector ] All-caps title: HDMI NOT WORKING: EMERGENCY! on raspberrypi.stackexchange.com

SmokeDetector

11:27 PM

[ SmokeDetector ] All-caps title: TAG VIDEO HTML5 LARGURA 100% E ALTURA CONTROLADA on pt.stackoverflow.com

[ SmokeDetector ] All-caps title: HTTP CALLOUT IN FOR LOOP on salesforce.stackexchange.com

« first day (263 days earlier) ← previous day next day → last day (3564 days later) »

all rooms