« first day (998 days earlier)      last day (1042 days later) » 

9:00 PM
[ SmokeDetector ] Blacklisted username: Query make error by fuck you nik mok on stackoverflow.com
 
In short: metasmoke is our web dashboard where we collect all the feedback. That's probably your best bet for getting signal out of.
 
@ArtOfCode Yeah, that's the term we're using right now. The feasibility of moving forward with any kind of real project at all. The exact shape of that real project isn't defined yet. In other words, does such a project exist.
 
Yep, makes sense
 
metasmoke is a web dashboard (metasmoke.erwaysoftware.com) for said thing, and it's really our only secure point of contact.
Which is an implementation detail.
 
@Pops It works like a real time review queue for spam, which is what its best at. We catch a broad net of posts, and let those that aren't spam through. The strenght of it lies in the fact that we can have human eyes on all potential spam with minimum hassle
 
9:00 PM
sd k
 
Hi all!
 
@Pops the real power of smokey is that anyone can help adjust its DB. That would need to be factored into any SE implementation for it to be effective.
(sorry for jumping in)
 
^ that
 
Yeah
 
This works because we can be agile.
 
9:02 PM
@Seth well, anyone as long as one of the team approves them first.
 
@ArtOfCode well yes
 
In essence, its all just regex matching. But having 10+ people keep ontop of any spam on the network and blacklist websites and patterns as they are used, thats what makes it effective
 
I see two categories of 'integration' here: One is minor nice-to-haves that'd give us a good deal more potential for catching stuff with minimal developer work. The other is 'real' integration - two-way communication, feedback stuff back to the system itself.
 
@Seth just noting that because I think that's also a strength - it's not literally everyone and their dog, it's people who we think are sane. Most people, in effect, but there are social filters in place.
 
For example, it'd be nice if we had network IDs on user objects we get from the API. That kind of stuff.
 
9:03 PM
@undo someone was talking about having undeletion bump posts to realtime the other day
 
@ArtOfCode right. I was more 'anyone can help' because they can submit a PR on GH. Not that it would necessarily get approved ;)
 
aye, that
 
I'd think that something like allowing metasmoke to feed into spamram or reducing the number of spam flags needed for nuking when a post gets feedback on MS would be possible
 
In the middle, it'd be nice if we had a realtime feed of spam we missed. That's stepping into some iffy territory around deletion, but it should be surmountable if we really wanted to do it.
 
@Magisch this is what I was thinking.
 
9:05 PM
Aye, that seems to have been the dominant integration idea: if Smokey says "this post is spam", the system says "okay, so we're only gonna require 3 spam flags on that to nuke it instead of 6".
 
We're already pretty good at identifying spam and getting eyes on it. Its just a number from then on out. Many posts get from 0 score to nuked in <30s when people get on it
 
@Pops Going back to this, I believe it'd be feasible to do with less keyboard-poking. By nature of the system, we have false positives. I don't want to turn this thing loose with power to unilaterally block stuff, there should still be a human involved somewhere.
@ArtOfCode this
 
Really the main thing about this is that we are able to cast such a broad net because we have users that filter out the false positives
 
[ SmokeDetector ] Blacklisted username, offensive body detected: Qury make error by fuck you on stackoverflow.com
tpu- by Ferrybig
 
@undo ^ second post from that user, might wanna destroy it if you can
 
9:07 PM
@SmokeDetector k NSFW imagery /trolling
Also that profile pic is wildly inappropiate
 
mine, gone
 
I still forget undo can do that
 
gone
 
Now that we've summarized, @Pops, what would you like to know? How can we help?
 
9:09 PM
(Note to self: new question for mod elections: how okay is your workplace with what's typically considered "NSFW" content?)
 
this would be a very, very good question.
 
we get some variation of this weekly here
 
But this clearly shows the effectiveness of the smokey system on stackoverflow, where there is also a large group of flaggers in the SO Close Vote Reveiwers. However on the smaller sites, deletion is usually to late for compared to google indexing
 
I actually don't have any questions right now. Was happy to listen to what you had to say, all sounds good so far. I had intended to read up on smokey before coming in here but between out-of-office backlog and the usual pop-up tasks, got swamped with other stuff so far this week. So you have some idea, I am a former CS person but it's been a few years since I've done anything code-y (plus networks were never my strong suit) so I may have lots of stupid questions.
 
I was thinking maybe we could make a point of contact for metasmoke at stack
 
9:13 PM
I'm flipping through meta at the moment trying to see what all the pretty colors and numbers mean.
 
@Pops we've got plenty of people who don't write the code :)
 
We've got a plan sitting around somewhere for automated flagging, which would effectively send signal to the system without any dev work on the SE side. All we need is the go-ahead on that.
 
where metasmoke can put feedback it gets, and if there is positive feedback, the amount of spam flags it needs to delete is reduced. That would help to bring the time-to-deletion during prime time for smaller sites down
 
I'm one of the people who doesn't write code for smokey. I just feed back throughout the day. I have it set up so I get every SD report as a desktop notification while at work and then act on it
 
9:15 PM
As I see it, we've got two potentially separate ideas. One, we automatically cast a spam flag on posts we think are spam (we've got a list of accurate criteria we can use) - that's no dev work, but only weak signal. Two, we set up between SE and metasmoke, and when MS tells SE that we think something's spam, SE reduces the number of spam flags required to, say, 3 - that's more dev work on SE side, but more signal.
 
We already have quality control implemented btw, our reporters are expected to stay above a 96% accuracy rate for feedback and currently everyone is
 
@ArtOfCode Those both make sense.
 
@Magisch that system isn't quite as simple as that, but that's a decent intro :)
 
@ArtOfCode We could expand on your last idea, more reasons a certain post is caugth by (smokey works with a few rule sets, like "blacklisted user", or "blacklisted website"), the more automatic flags are thrown, up to the maxium of 3
 
well yeah
but thats the gist of it
 
9:18 PM
I was going to go looking, but I guess I might as well just ask, how are you set up for outside interaction, on a scale of "you'd have to scrape the page" to "fully documented API and 24/7 customer service with a real human on the phone"?
 
@Pops metasmoke does have a fully documented API.
It currently doesn't include anything in the way of integration services, but they can be written :)
 
@Pops We can do pretty much anything. Metasmoke is a secure point of contact, with a domain name and everything.
 
So, no 24/7 customer service then, huh? Okay... okay... making a note....
 
there definitely won't be any page-scraping going on. I can get you JSON of... just about anything.
 
most of the time there is someone around
 
9:19 PM
(Seriously though, that's crazy impressive.)
 
be it hichris, undo or artofcode
 
@Pops sort of 18/6 support. Basically, whenever I'm here (for the existing API, anyway - Undo's your expert for general stuff)
 
[ SmokeDetector ] Pattern-matching website in answer: Org-mode: protect emphasis by Nicolas on stackoverflow.com
tpu- by ArtOfCode
 
@Pops We even made an userscript to feedback when flagging directly and signal that happened (thats what the tpu- by ArtOfCode you see means)
 
It really depends on how many resources ya'll are willing to devote to this project. We can go anywhere from us handling everything (automatic flagging) to you handling everything and us just providing the data.
 
9:22 PM
Aye. If you flag something as spam, the script uses the API to send the feedback, which gets passed to here in a websocket.
 
Essentially you could have that in any way you like. You can have SE get the feedback from the site, or we could send it to SE directly via an api. Its open to use for you guys
 
[ SmokeDetector ] Blacklisted username: How to add conditions in my Query zab fik by fuck you on stackoverflow.com
 
We already have a system of people making accounts, feedback accountability, quality control and manual approval of flaggers
 
We do have a lot of data on spam. What is it now, 37k posts?
 
sd k
Something like that.
 
9:23 PM
Whoa. Are we that high?
 
yep, 37077
 
37,081. Just since metasmoke has been around.
 
which is what, coming up on a year?
 
@undo But not all of those are spam, some are still false positives, :)
 
Those are valuable too.
28659 true positives.
 
9:25 PM
Okay. This will probably be at least 50% old info to me at 100% old info to you but can you walk me through what happens how smokey handles a single post from start to finish (or tell me why that question doesn't make sense, and what a better question would be)?
 
Makes sense. Just a sec.
 
@Pops, can you provide a bit of info on how spam detection on your side works? Are all detection rules built by SE, or do you use third party services too?
 
Post gets posted or updated and goes to the realtime tab
Smokey reads it and matches it against various patterns, a user blacklist, a website blacklist, etc
Posts who are caught get posted here and a report is created on metasmoke
Posts on specific sites get posted in site relevant rooms too (SO posts in SOCVR, for instance)
 
@Andy You know, I had a moment of mild panic when I came in because I thought I should explain that and then I realized I wasn't sure if that was one of the few non-public things about our company or not.
 
@Pops Smokey watches the realtime websocket. When it gets a post, it retrieves the full details of the post from the SE API. Runs that through our extensive regexes and check methods. If they come up positive (i.e. this is spam), sends it to metasmoke and posts a report in chat.
 
9:27 PM
users then feed back to confirm if a post is actually spam/abusive or not
 
@Pops spamram is kinda obscure to me, and I'm an SO mod. It seems to be somewhat non-public.
 
and then users click through to flag
 
1. We get it from the websocket (the one that feeds the realtime page on stackexchange.com)
2. We run it through a bunch of filters, some regexes and some code things.
3. If it matches, we post it to chat and send it to metasmoke.
4. People tell the system if we got it right or not. We send that to metasmoke too.
 
I've already asked for clarification and will let you know what I can later on.
 
Step 1.5: the post is put in a queue, and posts are batched up before asking the SE api, because else we wouldn't have any api quota left at the end of the dat
 
9:28 PM
Thank you
 
So you do end up automatically examining every single post (question or answer, at least) made to any Stack Exchange site?
 
@Pops Pretty much.
 
Yes
 
On the smaller sites it could be a few minutes before we see it (because of API limitations), but we see everything eventually.
We also see every edit.
 
literally everything that comes through that websocket, so new posts, edits, etc etc
 
9:30 PM
Yeah we catch lots of self-vandalism too with smokey
 
Just popping in here and having to leave again: How about making Smokey “official” by having it feed an SE-wide spam/rude/abusive queue? Should have some strong audits though to kill the robo-reviewers.
@undo Except on deleted posts, IIRC.
 
Except on deleted stuff, yeah
 
@Wrzlprmft that's... also quite a nice idea, with the downside that on big sites spam doesn't live long enough to be put in a review queue
 
@Wrzlprmft It's an interesting idea, lots of dev work attached.
 
And the chat works how, someone on the site manually sets up a bot account and you just say "hey, from now on, if there's a hit on site X, tell chat account Y"?
 
9:33 PM
Thats one of the opt-in options yes
 
Would be quite nice for smaller sites, though, and if we could do it network-wide so that reviewers from any site see posts on any site, then we could dramatically reduce spam lifetime on small sites.
 
You can opt-in to get pinged when a report happens on a certain site
 
@Pops There's only one SmokeDetector account, we add rooms manually and give our account access.
 
@Pops yeah, Undo set up the Smokey account and we give the code the creds so it can post to chat
 
...that makes more sense.
 
9:34 PM
also possibly of interest: this thing is running on a Raspberry Pi. It's stupidly light.
 
We also have a chrome browser extension, that reads reports from this room and desktop-notifies people when reports come in
 
Oh wow.
 
Follow up to @undo's earlier mention about network Id being added to the user object:
25
Q: Add users' network ID to the API shallow_user object

ArtOfCodeA while ago, Tim Post started an initiative to improve spam protection on SE. One improvement that was almost immediately suggested was to add network IDs to the API's shallow_user object. Doing this would allow existing tools such as SmokeDetector to better pick up on network spammers, which wo...

 
yeah its just lots of regex matching
 
basically, most people here have chipped in something, whether that's just feedback, or pull requests, or userscripts, or extensions, or hardware and servers to run it on...
 
9:36 PM
@zaq isn't around right now, but he's a genius with regexes and such.
 
we also have people spam flagging the reports (most of us have accounts on nearly all SE communities, mostly with 101 rep. Some people have 500+ flags on tons of sites due to smokey
 
You almost don't realise how big this project has got until you run through it all like this
 
Yeah its a very iterative effort but it has extended to dozens of people devoting time to nuke spam on all of SE, not just the sites they're usually active in, as fast as possible
 
Nor do you realise the interactions with site specific chat rooms, until you see the history of smoke detector's messages, he posts in many different rooms
 
9:39 PM
@SmokeDetector why
 
@SmokeDetector why
 
hmm
 
I can see the decision juices flowing
 
[ SmokeDetector ] Blacklisted username: Casting List of nested type in sql zabi by fuck you on stackoverflow.com
tpu- by Undo
 
9:42 PM
@SmokeDetector f (genuine question, though off topic. cv-pls sent to SOCVR and VTC'd)
Also edited
 
To conclude (?), there are a few ways to implement this ranging from automated flagging on our side to our providing data to SE's systems.
 
@Pops In essence, we have a pretty good gig going here and its working well, we could use some SE help to make it more effective, though
 
Also, a few cheap things that could make our lives easier.
 
Well okay! This all sounds amazing, and that's really saying something because knowing you all I had high expectations to begin with. (And it makes me feel even worse for not having much to say back to you today.) I'm going to keep poking around both at smokey and on our end. Feel free to ping me if you have further questions/requests/ideas, I will definitely be back in the coming days with more info about my thoughts.
 
Great, thank you! :)
 
9:46 PM
Thanks again @Pops
 
Awesome
 
Thank you Pops
 
@Magisch @undo anything specific you have in mind?
 
Alright, have a good night
 
tpu- by Gothdo
 
9:47 PM
@Pops :316In respect to make stuff easier for us?
 
@Pops the network ID thing for one
 
@Pops There are a few more events (undeletion, for one) that would be nice to have in the websocket. It'd also be nice to have a bigger API quota (20-30k would be awesome, although not entirely necessary). Probably the cheapest thing is adding the network ID thing ^.
network ID is almost certainly the best ROI
 
@Magisch Yes.
 
There are meta posts about most of them somewhere
 
Network ID would let us track spammers' accounts across the network, instead of just on one site. Don't really have data on how many, but I see them somewhat often.
 
9:51 PM
@ArtOfCode How about just getting the entire question/answer from the websocket, so we don't have to query the API at all?
 
@FrankerZ also possible, though we get a number of other useful details from the API too
 
The whole point of it having minimal data was for the masses: It would be nice to get an official websocket endpoints used for developers who need access to full questions/answers.
 
@FrankerZ API querying works fine, though, and it helps us be a little flexible on what we look at (score, for example)
 
@Pops one thing, though minor, is educating people to flag spam, rather than close vote it.
 
But it also slows down our ability to catch posts in near realtime
 
9:53 PM
@GregP gotta make a case that it's a problem first ;)
@FrankerZ More API quota would let us do that. It's only really a problem on the smaller sites.
 
@undo I just think it's a good idea people know how to use the site ;)
 
Just a recommendation. Lot's of ways to implement it
 
@FrankerZ The websocket truncates long posts, meaning we usually miss the link at the end
 
@Ferrybig Franker is suggesting that SE give us a different websocket that doesn't truncate.
 
@GregP See text box...fill it in...press submit. Right?
 
9:54 PM
@GregP Yeah... I was just thinking about that yesterday. That's really a failure on our end as builders of the system; in a practical sense, of course you're right, but on the other hand, it's technically within the correct definition for close voting (and downvoting).
 
There really isn't a frame limit on websocket connections. They're only limited, because less data going out to the public the less strain on the servers. 10 bots/developers using this special websocket wouldn't add that much more load
 
[ SmokeDetector ] Blacklisted username, offensive answer detected: Verb describing chainsaw making that bursting sound when you push the throttle trigger by Just A Nigga With A RPG on english.stackexchange.com
 
The ideal solution probably involves some dev work from both sides, to make sure that we've got a flexible system that's powerful enough and accurate enough to get rid of things fast. It's what undo said - it really depends how much work SE are willing to put into it.
 
tpu- by Yvette
 
10:09 PM
Is it possible to get rid of the error notification when flagging non-reported posts?
 
It was mentioned earlier
 
yeh I don't read the transcript I've missed, maybe I should start doing that
 
11 hours ago, by FrankerZ
@ArtOfCode Would it be possible to remove the error messages of NAA answers that I mark that aren't in metasmoke? API should return like a 200 response, not in metasmoke, and should fail gracefully.
 
200? For a not found resource? I would think 404 would be appropriate.
 
There could be a legitimate reason why 404 would happen on a case of a server error
 
10:13 PM
I find that fairly unlikely. Besides, returning a 200 for a "You messed up, client" response seems... wrong.
 
The client didn't mess up. It was attempting to submit a report of a possible post on meta smoke. Meta handled the response, and IMO should respond with a 200 and a json object with {"error": "no_post_related"} or something similar. I'm against using standard error codes like 404, when simply moving a folder could trigger apache to server 404's, which would cause the app to fail silently, when there was indeed an error server side.
 
[ SmokeDetector ] Manually reported question (batch report: post 1 out of 2): Online Prestashop 1.6 - Issues - Send gifts to Pakistan by Mr. Khan on stackoverflow.com
 
Nah, that's definitely a 404 situation, where the post just doesn't exist. It should be caught earlier by the client, really.
 
[ SmokeDetector ] Manually reported answer (batch report: post 2 out of 2): Bulk removing Wifi networks from an iOS device by Mr. Khan on apple.stackexchange.com
tpu- by ArtOfCode
 
10:23 PM
I don't feel that the first one is spam for me, its just a question of visit the linked website to view the code
 
zaq
Consider both posts together, though.
 
Okay, didn't see both posts coming from the same person
 
11:17 PM
[ SmokeDetector ] Offensive answer detected: Chunked HTTP POST request: by Mcbush on stackoverflow.com
tpu- by Yvette
 
11:35 PM
@zaq Fight spam for Stack exchange just crashed, google chrome showed an error message that it got stuck, and this tab showed the google chrome crash page, no detailed error message
 
zaq
11:50 PM
That's annoying but... Who knows why.
It's not like I'm allocating a lot of resources there...
 

« first day (998 days earlier)      last day (1042 days later) »