« first day (593 days earlier)      last day (3222 days later) » 

12:14 AM
[ SmokeDetector ] Email in answer: What is size 2 pasta? by Steve Sapato on cooking.stackexchange.com
 
12:29 AM
[ SmokeDetector ] Repeating words in answer: How to parse a string to an int in C++? by user3925906 on stackoverflow.com
 
1:17 AM
[ SmokeDetector ] Offensive answer detected: Remote Backup System needed by Topkek007 on softwarerecs.stackexchange.com
 
 
1 hour later…
2:44 AM
[ SmokeDetector ] Repeating characters in answer: Preservative for pickles by user36579 on cooking.stackexchange.com
 
3:00 AM
Hmm, so we're still kinda slow... but it's all because of regex.findall.
 
Did someone say performance?
 
@TimStone I know, I know, regexes aren't fast...
 
That's not always true
I was more curious than coming in to make any sort of point. I enjoy prodding things to go faster :P
 
Well... I guess, but in terms of Python's module...
@TimStone Methinks you should go make SEDE faster then. (indexes anyone?) :P
 
In my defense I do not have access to the underlying database so that's on Nick :P
 
3:03 AM
Anyway, here's my benchmark script: gist.github.com/hichris1234/3ca8725f0a786505980f. @Undo @ProgramFOX
 
@hichris123 Aren't most of the regexes fixedstring1|fixedstring2|fixedstring3|... anyway? Why not not use regex?
 
@TimStone You can poke with a blunt stick...
 
(unless performance is fast enough that it's not really an issue)
@hichris123 Why not a sharp one? :P
 
@Doorknob Some are, a lot aren't.
 
@hichris123 I think he's on vacation this week, so not right now
 
3:05 AM
in Tavern on the Meta on Meta Stack Exchange Chat, Jul 18 '14 at 17:18, by Pops
@hichris123 It was my understanding that I would not be poked with any sharp sticks as a Stack Exchange employee. Your request for room unfreezing has been put on indefinite hold until HR completes its investigation.
 
lol
 
@TimStone Oh right, "vacation".
Wherein Nick still tweets like he's on the job.
I think we just have some poorly optimized regexes... but I have very little regex knowledge, so...
 
Can you explain the process briefly? You just have a big list of invalid tokens that you concatenate into a huge regex that runs on the post body/titles?
 
3:20 AM
[ SmokeDetector ] Offensive answer detected: Why should I buy Apple? Everyone says it's overpriced by xor on apple.stackexchange.com
 
3:32 AM
[ SmokeDetector ] Blacklisted website in body: That goodbye from brush p on it by richard cain on superuser.com
 
4:00 AM
[ SmokeDetector ] Manually reported question: Are you experiencing issues with your fat by jiiknaacho mani on unix.stackexchange.com
 
4:49 AM
[ SmokeDetector ] Bad keyword in body: I saw Brainfire bunched together with it by Steven Chavis on superuser.com
 
5:08 AM
[ SmokeDetector ] Bad keyword in body: Fun Brain Games: Mind Over Matter by user49262 on drupal.stackexchange.com
[ SmokeDetector ] Manually reported question: What the essential part of good essay writing papers by pehummo on askubuntu.com
[ SmokeDetector ] Bad keyword in body, bad keyword in title: Which ingredients used in Neuroluma? by haroldbeane on superuser.com
[ SmokeDetector ] Bad keyword in body, blacklisted website in body: what is Safe and easy hacking tool? by Bernard Williams on superuser.com
[ SmokeDetector ] Bad keyword in title: Neuroplex Pro – Is it effective? by haroldbeane on superuser.com
[ SmokeDetector ] Email in body: issues after the rooting process by Avi Patidar on android.stackexchange.com
 
5:53 AM
[ SmokeDetector ] Manually reported question: When you're wanting to shed weight? by Amy white on superuser.com
 
6:26 AM
[ SmokeDetector ] Bad keyword in body, blacklisted website in body: What kind of unbelievable circumstances will I get by utilizing Dermakin? by Jacqueleni Toseen on drupal.stackexchange.com
[ SmokeDetector ] Bad keyword in body: Gamon D. and Bragdon A Alpha ZXT by eewaloi45 on drupal.stackexchange.com
 
7:02 AM
[ SmokeDetector ] Email in answer: Remote Backup System needed by Avijit Srivastava on softwarerecs.stackexchange.com
[ SmokeDetector ] Blacklisted website in body: Skin and Beauty Glossary by boleybond on meta.stackexchange.com
[ SmokeDetector ] Bad keyword in body, blacklisted website in body: Boost the stamina? by jicsyruvia on drupal.stackexchange.com
 
7:25 AM
[ SmokeDetector ] Email in body: Suggestions for name of email to use for Online Survey by cool_kid on stats.stackexchange.com
 
7:39 AM
[ SmokeDetector ] Bad keyword in body, blacklisted website in body: How to use this product? by queeni juriya on drupal.stackexchange.com
[ SmokeDetector ] Bad keyword in body: Studies tally want fostered the use of task-specific by welaidking on drupal.stackexchange.com
[ SmokeDetector ] Blacklisted website in body: Creates a protective layer to enhance your complexion by juliarowes on meta.stackexchange.com
 
8:06 AM
[ SmokeDetector ] Bad keyword in body, bad keyword in title: Natural Male Enhancement Pills Ingredients by morrin bailey on meta.stackexchange.com
 
8:23 AM
[ SmokeDetector ] Bad keyword in body, bad keyword in title: Muscle Building Diet-Its Essence by Belinda Torres on superuser.com
[ SmokeDetector ] Email in body: About PhD in Biology in Germany by PULKIT SHARMA on academia.stackexchange.com
 
9:02 AM
[ SmokeDetector ] Bad keyword in title: Strategies For Effectual Muscle Building by jerrye phenson on superuser.com
 
9:15 AM
[ SmokeDetector ] All-caps title: NEED HELP FOR CALCULATING TWO DERIVATIVES ANALYTICALY by Hossein on math.stackexchange.com
[ SmokeDetector ] Offensive body detected: Python serial fails reading large amounts of data by user3052315 on stackoverflow.com
 
10:02 AM
[ SmokeDetector ] Manually reported question: ProFactor T-2000 Preceding and following by cao suat on meta.stackexchange.com
[ SmokeDetector ] Nested quote blocks in answer: extract first three numbers from a string by xXIAmInFazeCanYouTakeItXx on stackoverflow.com
 
10:34 AM
[ SmokeDetector ] Offensive answer detected: How to control manual double-side printing on Mac OS X Lion? by nig on apple.stackexchange.com
 
10:57 AM
[ SmokeDetector ] Nested quote blocks in answer: Datastax cassandra object mapper setting consistency and if not exists by Den Roman on stackoverflow.com
 
11:18 AM
[ SmokeDetector ] Nested quote blocks in body: Alogrithm for balancing the load in multiple worker threads by Jijo Thomas K on stackoverflow.com
[ SmokeDetector ] Bad keyword in body: Web Performance using Spring by Leanne on stackoverflow.com
[ SmokeDetector ] Bad keyword in body, bad keyword in title: Which ingredients used in Neuroluma? by Jacklyn Wilson on drupal.stackexchange.com
 
11:34 AM
[ SmokeDetector ] Bad keyword in body, blacklisted website in body: There are many myths that surround weightlifting by majuoran mani on unix.stackexchange.com
[ SmokeDetector ] Repeating characters in body: Need help on save method to save all in a page block table with custom controller by sfdev on salesforce.stackexchange.com
 
11:51 AM
[ SmokeDetector ] Bad keyword in body: Exciting Games That Can Enhance Your Mind by andreas innis on superuser.com
[ SmokeDetector ] All-caps title: NEED HELP IN CONVERTING MULTYPLE PNP SIGNAL TO ONE by prateek chaudhary on electronics.stackexchange.com
[ SmokeDetector ] Offensive answer detected: Why do I hear beats through headphones only at low frequencies? by NIGGERASSRAPE on physics.stackexchange.com
 
12:20 PM
[ SmokeDetector ] Repeating characters in body: CORS configuration not working by Vishal on stackoverflow.com
 
12:41 PM
[ SmokeDetector ] Bad keyword in title: Is Gmail Phone Number @0800 O86 86 76 UK? by Thomas Reid on android.stackexchange.com
 
12:53 PM
[ SmokeDetector ] Bad keyword in answer: How do I recover from MySQL InnoDB table/index corruption? by lucyfranklin on dba.stackexchange.com
 
1:31 PM
[ SmokeDetector ] Repeating characters in body: Oscillator Error and PIC24F stopped working by Bob on electronics.stackexchange.com
 
2:01 PM
[ SmokeDetector ] Repeating characters in title, title has only one unique char: hhhhhhhhhhhhhhh by Gray on stackoverflow.com
 
2:47 PM
So I figured out which regex is the slow one... the repeating words one.
(?i)(?P<word>[a-zA-Z]+).*((\\b| )+(?P=word)){5,}
 
That could be done without regex.
 
@Doorknob How?
 
For loop over the characters in the string, built a dict (is that what it's called in Python?) of all the 5-letter combinations, select the ones that appear more than n times.
 
That... doesn't seem fast either.
 
3:02 PM
Regex backreferences are sloooow though. I guess some benchmarking is in order.
 
Well, you're the regex person, optimize that regex! :P
 
Pretty sure that's as fast as it's going to get :P
 
> Please hold while we fetch the debug data. This might take some time.
...
If this ever finishes fetching, I'm going to be surprised.
 
4:11 PM
Maybe it's just trying to do too much, allowing arbitrary stuff in between. Back-to-back repetition like lalalalala or penis penis penis would be caught by this simpler regex stackoverflow.com/a/9079897
These seem to fit most abuse patterns.
 
[ SmokeDetector ] Email in body: DDNS Hostname Address? by threehappypenguins on android.stackexchange.com
[ SmokeDetector ] All-caps title: HOW TO BAKE AND RENDER by Che Lio Vincey on blender.stackexchange.com
[ SmokeDetector ] Phone number detected in title: SQl Inserting 2147483647 by Divyank on stackoverflow.com
 
@TimStone We have various regexes that are run against posts. Some search for keywords usually found in spam, some search for blockquotes within blockquotes, some search for repeated characters, some search for email addresses, etc.
@1999 Hmm... I'm going to have to defer to @ProgramFOX on this one...
Without repeating words regex:
average: 0.00650596750067
With repeating words regex:
average: 0.554242632284
 
4:31 PM
[ SmokeDetector ] Email in answer: DDNS Hostname Address? by threehappypenguins on android.stackexchange.com
 
@hichris123 Part of a problem is that sometimes being limited to regex forces unnatural choices. Is it easy to add a filter that uses a bit of Python logic of non-regex kind?
 
[ SmokeDetector ] Offensive body detected: Calling method wrong by Home-owner on stackoverflow.com
 
In case of repeated-link regex it would be: get all matches for <a href="([^"]+)", check that there are exactly two of them, and they are equal. Boom, spam found. Trying to do the same just as a regex match is much less efficient.
 
@1999 Well... we'd have to mess around with some stuff...
It could be done, but I don't think it'd be easy.
 
4:47 PM
Change the array of regexen(?) to an array of lambdas. Then just change the logic that checks to see if they work.
You could even allow plural(regex) and lambdas, and check the type.
Specifically, line 176. Looks pretty simple.
Why wouldn't it be easy?
            if callable(rule['regex']):
                matched_title = rule['regex'](title)
                # etc.
            else:
                compiled_regex = regex.compile(rule['regex'], regex.UNICODE)
                matched_title = compiled_regex.findall(title)
                matched_username = compiled_regex.findall(user_name)
                matched_body = compiled_regex.findall(body_to_check)
^ Backwards compatibility and arbitrary code-execution for testing posts.
(@hichris123)
 
5:10 PM
Hm... I'm not too familiar with lambdas so...
 
f = lambda x: x*2; a = 10; b = f(a); b # => 20
 
You have semicolons for Python? You monster.
 
:P
 
@hichris123 I was using fancy widget.
 
@hichris123 I can't. I've already pinned both of them. :P
 
@Doorknob ... so?
 
so I can't put newlines
 
You never made Shift + Enter work?
 
nope :P
It's an <input>, not a <textarea>.
 
There, repinnified.
 
5:33 PM
!!/pull
 
6:09 PM
[ SmokeDetector ] Manually reported question: Hair Volume Products? by Volume Products on askubuntu.com
 
6:33 PM
[ SmokeDetector ] All-caps title: ERRO CONEXÃO AJAX COM DATASNAP by Furlan on pt.stackoverflow.com
 
6:53 PM
[ SmokeDetector ] Nested quote blocks in body: Recommended setup for F5 with multiple EXIM servers behind by Tru64Gurus on serverfault.com
 
@1999 @hichris123 I'll certainly work on the repeated-links-thingy and on the improved repeated words, but I'm off most of this weekend so it'll be something for next week.
 
You're off? On an adventure? :P
 
Going with the train to Germany is an adventure, indeed :P
 
Oooh. Have fun.
 
The train which we had reversed tickets for was cancelled... so we had to use several replacement trains.
@hichris123 Sure thing! The brother of my grandmother who lives here organizes a party for the whole family :D
 
7:05 PM
Nice. :D
 
7:50 PM
 
8:01 PM
[ SmokeDetector ] Offensive answer detected: What is Sauron saying when Galadriel defeats him at Dol Guldur? by user1983877 on movies.stackexchange.com
 
from termcolor import colored
^ What's that? PyCharm underlines it as read for me.
ah, it's a package
Alright, I have got some time now to work on the repeated-word-speedup.
 
8:31 PM
Aha, benchmarking gave positive results already:
inp = "This, is.  a test: string."
t = timeit.Timer("has_repeated_words(inp)", "from findspam import has_repeated_words; from __main__ import inp")
print "timeit: non-regex"
print t.timeit(1000)
t2 = timeit.Timer("repeated_words_regex(inp)", "from __main__ import repeated_words_regex, inp")
print "timeit: regex"
print t2.timeit(1000)
> timeit: non-regex
0.00817723822082
timeit: regex
0.181189519473
And the results:
@hichris123 ^
 
[ SmokeDetector ] Manually reported answer: What's the best way to have Indeed.com scrape a jobs site? by wardadu on stackoverflow.com
[ SmokeDetector ] Manually reported answer: How complicated is it to scrape web pages with C++? by wardadu on stackoverflow.com
[ SmokeDetector ] Manually reported answer: What tools are available to collect data online? by hindaz on stackoverflow.com
 
@ProgramFOX claps
 
Now working on putting that method into rules.
 
8:48 PM
Er... what happened to next week @ProgramFOX?
 
It appeared I had more time right now than expected :P
Also I had a bug in my method, and fixing it even improved the performance of the method O_o
timeit: non-regex
0.00796499703601
timeit: regex
0.187825238063
 
If you can, can you push to a branch, @ProgramFOX?
 
timeit: non-regex
0.00851509991346
timeit: regex
0.178866771651
So there was another bug, but now it entirely works, and this is the final benchmark:
 
This will not identify hahahahahahahaha, will it?
 
@1999 No, but the previous regex didn't either.
For such things, the gibberish classifier will be useful once it's out of beta. That was its point, anyway.
 
9:01 PM
@ProgramFOX Is \w in a regex just splitting at spaces?
 
@hichris123 No, it splits at non-word characters.
So that includes punctuation and such.
 
Oh, I see.
@ProgramFOX You should probably move github.com/Charcoal-SE/SmokeDetector/commit/… up so the title & username checks don't fail.
 
@hichris123 That's what L193 is for.
 
@ProgramFOX Ohhh nevermind. :P
@ProgramFOX Final as in good to use?
 
@hichris123 As in "good to use" + "all tests passing".
 
9:15 PM
Alright, I'm just going to make a simple change then, if that's okay. (moving the if w/ body_is_summary into each individual check)
 
Sure, that's fine.
 
I'm happy with it. Go ahead & merge, @ProgramFOX. :)
 
darn it, I merged a stale branch -_-
 
lol
With my benchmark.py:
Old: average: 0.583327280013
New: average: 0.00836260195648
 
okay, merge fixed
!!/pull
 
9:23 PM
[ SmokeDetector ] SmokeDetector started at rev 8ff02b6 (ProgramFOX: Merge branch 'speedup-repeatedwords') (running on closet-server)
 
Did someone rewrite history? :P
 
Uhh... maaaybe :P
Well, I'm out for today, cya!
 
cya. Thanks so much! :)
@Undo We're much faster now. :D
 
wow, I would never have guessed that a pure code implementation of that would be faster than a regex.
 
I didn't think so either.
But it does. :)
 
9:36 PM
Besides being faster, having methods as an option ... can get more creative with heuristics.
 
yep.
Okay, I'll look at your double link thingy, @1999.
 
@Undo For that, absolutely. Even for that test post body that @hichris123 linked to last night, that regex is doing hundreds of thousands (maybe millions) of match steps
 
@TimStone Huh. I learned something today
 
@SmokeDetector fpu-
 
9:40 PM
@TimStone Yeah. Does a better regex engine handle it better?
 
[ SmokeDetector ] Repeating words in body: solve nonlinear equations with NDSolve by Ali on mathematica.stackexchange.com
 
@SmokeDetector Huh, why?
 
0 counts as a word
 
oh
 
Perhaps it shouldn't.
No, wait, Fox already put not w.isdigit() there...
 
9:43 PM
@hichris123 That regex is just generally expensive. You might have been able to tighten it up some, but only looked at it briefly.
 
I guess it tripped on i? because the loop skips over digits, it sees that formula as i i i i i i...
 
Heh
 
lol
 
@1999 yep.
 
9:50 PM
Moving the digits check inside the loop would fix that.
words = [w for w in words if w != ""]
curr = 0
prev = ""
for w in words:
    if w == prev and not w.isdigit():
        curr += 1
 
@1999 Yeah, I just did that.
!!/pull
 
[ SmokeDetector ] SmokeDetector started at rev c39e187 (hichris1234: Move not w.isdigit() to the word loop) (running on closet-server)
 
10:13 PM
Hmm, I think we can just use beautifulsoup.
Might be a little slower than matching <a> tags without bs4, but the performance hit is, at the most, 0.01 seconds. So...
 
10:38 PM
Makes sense, but a regex would be just as reliable because SE generates html markup in a predictable way.
 
[ SmokeDetector ] All-caps title: SUGGESTION ABOUT PRAKIM CHART by user9710 on judaism.stackexchange.com
[ SmokeDetector ] Email in answer: It is legal if company asks for scans of my Credit Card? by clarisse on money.stackexchange.com
 
11:02 PM
[ SmokeDetector ] Email in answer: Bank balance for getting a Schengen visa? by help on travel.stackexchange.com
 
11:54 PM
[ SmokeDetector ] Repeating characters in title, title has only one unique char: aaaaaaaaaaaaaaaaaaa by ramon on electronics.stackexchange.com
 

« first day (593 days earlier)      last day (3222 days later) »