Charcoal HQ

sd why

@micsthepick [:42144381] Body - Position 1360-1372: ouseful.info, Position 1537-1549: ouseful.info

sd - fp-

12:34 AM

@JakeSymons It's not spam - you can tell because there's nowhere to send money.

They can send it to me.

[ SmokeDetector | MS ] Bad keyword in body: Php submit button not working and error messages not showing up by Angelo Rodriguez on stackoverflow.com

fp- by Tetsuya Yamamoto

[ SmokeDetector | MS ] Mostly dots in answer: Increasing Google Chrome's max-connections-per-server limit to more than 6 by M0d-X on stackoverflow.com

naa- by micsthepick

Nisse Engström

@SmokeDetector tp-

Conflicting feedback on AJAX js from the loaded part of the page doesn't work.

[ SmokeDetector | MS ] Bad keyword in body: Daniel Fast - Are eggs allowed? by DeveloperMCT on christianity.SE (@Nathaniel)

@SmokeDetector why

12:48 AM

@TetsuyaYamamoto Body - Position 355-366: lose weight

fp- by Tetsuya Yamamoto

1:01 AM

not every post containing "lose weight" is spam...

Makyen

Did someone make changes that would affect the API endpoint: api/v2.0/posts/{postId}/domains? I'm suddenly getting "Internal Server Error" for all requests to that endpoint. The requests were working fine earlier today. If so, what changed? Filters?

@micsthepick The post asks about fasting, it's unrelated with promotion of "lose weight drugs".

@Makyen nothing that I know of

yeah, last commit 6 days ago

@TetsuyaYamamoto excactly

"got error 28 from storage engine"

um

oh of course, it's out of disk

@Undo

1:06 AM

nice

ummmm but it's not

oh, maybe

Makyen

Hmmm... looks like I'm now getting errors for basically all API endpoints :-).

yeah, I know

Makyen

@Undo :-) :-(

rails log backup script failed

@Undo google told me error 28 from mysql was no space left on device :)

1:12 AM

yeah, Ubuntu's login screen is misleading since it counts all attached "storage"

Turns out the rails log backup thing is important.

@ArtOfCode Redunda sure likes logging stuff

@Undo could probably turn it down if you wanted

Just backed up 300+ MB of Redunda logs. After gzip.

that would do it

one more reboot for good measure

[ SmokeDetector | MS ] Bad keyword in answer, bad keyword with email in answer, pattern-matching email in answer, potentially bad keyword in answer: Set Bluetooth audio device to always use AD2P on Linux by mrss jessica on superuser.com

ArtOfCode/EC2 received failover signal.

1:16 AM

lol. We're back to 6GB/12GB usage

six gigabytes of logs

Restart: API quota is 19999.

!!/status

@Undo Running since 23:04:51 UTC (2 hours, 18 minutes, 2 seconds)

[ SmokeDetector | MS ] Bad keyword in body: Php submit button not working and error messages not showing up by Angelo Rodriguez on stackoverflow.com (@micsthepick)

@Undo Running since 02:33:59 UTC (5 days, 22 hours, 42 minutes, 55 seconds)

o.O

@SmokeDetector k

1:17 AM

@Undo No such command 'locatio'.

!!/location

@Undo Henders/EC2

@Undo ArtOfCode/EC2

!!/standby art

ArtOfCode/EC2 is switching to standby

[ SmokeDetector ] SmokeDetector started in standby mode at rev 72cd902 (SmokeDetector: Auto blacklist of titan\W?blast by tripleee --autopull) (running on ArtOfCode/EC2)

1:32 AM

Does everyone who runs an instance have the Smokey user username and password?

[ SmokeDetector | MS ] Link at end of answer: How to write VBA code with UTF-8 characters in it? by Michael Yun on stackoverflow.com

fp- by Tetsuya Yamamoto

[ SmokeDetector | MS ] Potentially bad keyword in answer: EFCom GPRS/GSM Shield by tclauss on arduino.SE (@NickGammon @Majenko)

Rob

sd why

@Rob [:42145455] Body - Position 1379-1401: sites.google.com/site/

fp- by Rob

1:53 AM

!!/errorlogs 420

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-01-06 02:10:33.266615 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-01-06 02:11:03.297108 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.
2018-01-06 02:11:33.664495 UTC

WARNING: Last metasmoke ping with a response was over 120 seconds ago, forcing SmokeDetector restart to reset all sockets.

presumbly from when MS was down

ohh right @Undo remember that it always does a failover when it comes back up

yeah

We should fix that, wait a minute on startup to let the pings and WS connections come back

architecture makes that harder than one would think

1:56 AM

@thesecretmaster yes

Thanks

I was just curious

@Undo OK status_update really confuses me

and I

link and I'll tell you how it works

Why doesn't this make MS fail over whoever gets the ping in first

github.com/Charcoal-SE/metasmoke/blob/master/app/controllers/…

I guess I need to see should_failover

yeah, that's where the logic is

2:07 AM

Do all the smokey instances break when MS is down?

OK I see the problem

MS is failing over to whoever gets the ping in first -- the trick is that the tolerance for the active instance being considered dead is 3 minutes and the pings happen every minute so even if another instance comes first it's not enough... except for when MS comes back up and it really has been longer than 3 minutes

The solution would really be to not store the times in the DB and reinitialize them to now on startup, but we want to keep the status page "last seen"

sooo maybe a hack

just have a boolean flag that gets set after 60 seconds

set by what, 60 seconds after what?

startup

dunno if Ruby has a Timer or just use a Thread

definitely a hack

and how are you going to reset that flag on shutdown?

Not in the DB

I'm pretty sure Rails isn't like PHP in that regard

the VM is always running no

yeah, I used a class variable for the git lock and that worked fine

you can have a state like that

Q: Is there a way to get the elapsed time that a Rails server has been running for?

2:22 AM

oh

5

I'd just like to print it out in a HTML view somewhere in my Rails project, along with a bunch of other helpful info. Something like "Current up time = 4d 11h 54m 2s" or "Up since 23rd Aug 2013". Couldn't find any info on this anywhere, surprisingly. Maybe there's a generic Rack way of doing it.

@quartata ^ That's what you want, right?

I like that. Don't failover anything if that time is less than 120s away

I mean I dont need the time exactly. Just if it's been 120 seconds

which is why I wanted to schedule some kind of task after 120 seconds

Right. Storing the time and checking it seems cleaner to me than starting a timer and setting a bool

hmm 120 or 60

Anything over 60 should work

I could see a freak scenario where a smokey pinged just before MS came online, then the task didn't get scheduled for like 61 seconds

2:26 AM

70 would be fine, guarantees at least one cycle for all instances

Wait so every actually makes a crontab? How on earth does that work

In config/schedule?

It's in the Capistrano deploy cycle

but then how does cron call into the rails context

does it send an HTTP request

@quartata Calls the Ruby methods directly

hold on

# Begin Whenever generated tasks for: metasmoke at: 2018-01-02 02:38:41 +0000
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /bin/bash -l -c 'cd /var/railsapps/metasmoke/releases/20180102023740 && bundle exec bin/rails runner -e production '\''SmokeDetector.check_smokey_status'\'''

0 0 * * * /bin/bash -l -c 'cd /var/railsapps/metasmoke/releases/20180102023740 && bundle exec bin/rails runner -e production '\''ReasonsHelper.check_for_inactive_reasons'\'''

0 0 * * * /bin/bash -l -c 'cd /var/railsapps/metasmoke/releases/20180102023740 && bundle exec bin/rails runner -e produc…

rails runner was the bit I didn't know about

2:37 AM

[ charcoal-se.github.io ] continuous-integration/travis-ci/push success on 22cf5c7: The Travis CI build passed

2:57 AM

API quota rolled over with 11252 requests remaining. Current quota: 19999.

math: 308
stackoverflow: 146
superuser: 64
codegolf: 41
physics: 39
askubuntu: 38
serverfault: 30
electronics: 28
security: 28
bitcoin: 28
english: 26
apple: 22
unix: 19
arduino: 15
ell: 15
ethereum: 13
scifi: 13
webapps: 12
mathoverflow.net: 11
drupal: 11
diy: 10
music: 10
gaming: 10
rpg: 10
gamedev: 9
worldbuilding: 9
money: 9
history: 8
android: 8
gis: 7
es.stackoverflow: 7
academia: 7
salesforce: 7
magento: 6
tex: 6
softwareengineering: 6
sharepoint: 6
space: 6
judaism: 6
mathematica: 5
bricks: 5

Andy

3:20 AM

@Undo How do you want this source file? It's an SVG

[ SmokeDetector | MS ] Offensive answer detected: Why do the Shuttle's external tanks appear to be rusted? by skeeter on space.SE

@thesecretmaster Do you have a GH for your chat gem?

GitLab, but yes: gitlab.com/izwick-schachter/chatx

why gitlab?

Because ... I really don't know.

3:27 AM

is github blocked in your country?

Nope

I honestly don't remember why.

I should move it over to GH at some point.

Also, if you want to use it, the docs are kinda bad and/or from older iterations, so ping me if you have questions.

can confirm

might be time to write some docs before more features ;)

Yes...

[ SmokeDetector | MS ] Bad keyword with a link in answer, link at end of answer: How to find the indices of the visible rows in a TableView in JavaFX 9 by Yuichi Sugimura on stackoverflow.com

still pondering the allure of the central post scanning server and I figure it'll be Rails so

3:31 AM

@quartata central post scanning server?

I talked about it with Undo a while back. Basically the server would consume various SE content (posts, edits, comments, CVs) and send HTTP requests to subscribed bots (running as lambdas)

@quartata ah, is that why you were asking about apicache

why make bots go through setting up lambdas though, when websockets are a thing

Because AWS Lambda

but websockets

No more Redunda or worrying about hosting

3:38 AM

almost certainly easier

However the design I have in mind is very configurable so yeah you could have WS if you wanted

@quartata true, might as well offer both

yell at me if you start making it, sounds interesting

In its final form it could also include Helios and a generalized Metasmoke

Actually, the ChatX (sorry they name is so similar to ChatExchange) docs were pretty accurate (as far as they existed)

@quartata FYI, I also have a gem for reading the realtime WS and for the API. The API gem is really incomplete, but I'm working on it.

[ SmokeDetector | MS ] Blacklisted website in body: The NBA Portland trail blazers by user134178 on wordpress.SE

3:45 AM

oh huh we missed MS's birthday

@SmokeDetector k

Andy

You monster!

[ SmokeDetector | MS ] Few unique characters in body: How to get \`browser-sync reload\` to use https by Michael Prescott on stackoverflow.com

What does one get for something like MS... a new host?

@Catija more disk space, usually

3:46 AM

tp- by Tetsuya Yamamoto

@ArtOfCode Ah. That makes sense.... do you actually store it all locally somewhere?

@Catija well, locally on the server, yes :)

haven't checked how big it all is recently, but Pretty Big (tm)

@SmokeDetector n (link only?)

@ArtOfCode But a server you own or you buy space on a server somewhere?

EC2

3:49 AM

@Catija Undo's AWS server - rented VPS

Ah.

Hmmm the tough choice is do I start from Metasmoke or Sentinel as a base

@quartata metasmoke, for the love of everything good

I meant either

or you could start fresh, but metasmoke is a reasonably good option

sentinel is definitely not

3:57 AM

Eventually it will have to do what Metasmoke does

@ArtOfCode Was this the thing you were telling me about a while ago?

but if I start from Metasmoke itself I'll have to do a lot of pruning

@thesecretmaster nah, different things, but sounds like a similarish principle

well maybe not

@quartata starting fresh wouldn't be a terrible idea

3:58 AM

[ SmokeDetector | MS ] Misleading link: Camera app closes once the Take Photo button is pressed. The code in the tutorial vid was working by arkanaxz on stackoverflow.com

fp- by Tetsuya Yamamoto

devise, rolify, will_paginate+bootstrap, webpacker, foreman are the gems you need

I don't know metasmoke, otherwise I'd volunteer to help where I can. If you're starting fresh, I'd be happy to help.

It's just that Metasmoke has all these nice views that Id like to hve. I just need to take out the SmokeDetector logic and make it so that each table has a column for which bot it came from

@quartata cp -r metasmoke/app/views ./app/views

and drop the dirs for the controllers you don't have

on second thought though I had a plan for generating the views from the config

4:03 AM

I can do frontend stuff, anyway. What views in particular are you thinking?

@quartata It might be better to do views from scratch but just copy paste the bits you need from MS

Depending on how similar they're gonna be.

Well the idea is that the reply from the bot contains a boolean for if it's spam and the why data. The server sends the reprt to chat but then also puts it on a web dashboard

That's Metasmoke right there basically

@quartata doesn't have to be spam either I guess

just a "report to chat" bool, regardless of what it is the bot detects, and the why data has the details

So we'd have post view/review/search/dashboaed at minimum

easy enough

there's some queries you're not gonna want to copy from metasmoke

because I haven't spent nearly enough time optimizing them

4:07 AM

I would add that if we pull this off this is something everyone and their dog will use: Natty, FireAlarm, Queen, Bellisarius, ...

It isn't unique to spam

s/is/could be/

I'm trying to keep it as general as possible

Guttenberg is the only bot with exotic dashboard needs but

This was a sample yml file I made representing a bot while thinking this through:

Jan 2 at 4:22, by quartata

reports:
  types:
    - [questions|answers|comments|edits]
    - ...

  query:
    method: [POST|GET]
    route: ...
    response:
      mime: [json|yaml|xml|protobuf]
      is_spam_key: name of key containing bool...
    report_templates:
      chat: "Bad bad {post_type}: {reasons}, {post_url}... keys can come from scanner response or SE API response"
      web: "...something MS can turn into a view idk"
feedback_types:
  [feedback name]:
    conflicts_with: ...
    icon: <unicode char>
    aliases:

Need to click on it to see the whole thing

I left out how you'd specify the rooms, although that might go in a separate file

So this is a fancy report-to-chat api with a dashboard?

We want to make a central server that scans for posts and then sends off requests for the contents to be analyzed

Ok. So kinda like a real SE websocket api?

4:16 AM

[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, link at end of body, pattern-matching website in body, +1 more: viarexlabsblog.com/outback-vision-protocol/ by wlmllyskahgl on meta.SE

Yeah, but it also does the posting to chat parts

@SmokeDetector k

@quartata if you're gonna do it in YAML, store it in the DB as a serialized column, so that you can let bots update their config via an API

usecase?

Oh I see why

So they don't need PRs

?

It doesn't have to be in YAML. It could be configured via a webpage

@quartata sure, but you can have both

4:20 AM

@ArtOfCode Just have a bot config table, if you're doing it that way, right?

chuck the YAML in the database, let people edit it from a webpage, and expose it on an API

Also how do I dynamic queue size

@thesecretmaster I like the look of your gem but I have a few questions

Ask away

1. Is say asynchronous? Do I get a message ID back?

If not is there another function for sending synchronously to a given room?

You get a message ID back, I forget if it's synchronous or not. If it isn't, I could make it async

4:25 AM

If I get the ID back it must be synchronous

That's what I want

2. What's available in a User object?

Ok. I could also make it async with a callback

3. How many threads do you use?

Just one per chat host?

And a master thread IIRC

@quartata gitlab.com/izwick-schachter/ChatX/blob/master/lib/chatx/models/…

Only thing missing is is_moderator

All right you passed

Yay!

I've been upgrading it as needed when I build bots.

4:32 AM

Oh, one other thing -- it only fetches the usercard when I need a User object right? (Lazily)

I think I saw that somewhere else but making sure

... maybe? I don't remember.

I'd look if I wasn't on mobile

No, it fetches it when a User is instantiated

Right, but does that happen for every new message?

Or does it construct the user object on demand

It constucts one for every message (mostly)

Well all right

I should change that

4:35 AM

Not a huge deal

If you could add is_moderator though that'd be great

All right now all I need is a name

@quartata It does it on entrance, exit, and mention

Kinda spent on that one

@thesecretmaster Ohhh nice the cache

Assuming it was there for the entry that is

I think @ArtOfCode wrote that bit

Might need something to invalidate the cache occasionally though

In case someone gets a diamond

or a name change which is more common

Although I think you might be able to detect that one

Anyyywho

It's kinda rough, but I was happy when I got it to stop dying randomly.

That was my threshold for it working enough.

4:40 AM

Oh is the gem on rubygems?

Not the latest version. If you want to try it, use from git

All right now for the gazillion dollar question

Uh oh

Can I fix my friggin Ruby install

Are you using rvm?

4:42 AM

No because on OS X it tries to install homebrew and then just execs it

!!/alive

@AJ Yup

It might as well not exist

@quartata I like rbenv. It works on my 2011 mac air

The one that came with is 2.0. I built 2.4 from source but OpenSSL doesnt work

@thesecretmaster don't remember if I tried it already but I'll clean up and try again

4:43 AM

@quartata RVM tries to install homebrew? That's... strange.

I know right? It totally broke my symlinks in /usr/local too

Before I was able to control c

I have working ruby on my mac, but python is horribly destroyed.

You can't use control c to send sigint anymore?

That's ... impressive

No it worked it's just it had installed a good chunk of Homebrew before I reacted in time

Ooohhhhhhh. That would explain it. That does not sound fun.

Rbenv and ruby-build just get cloned into your home dir. No serious config beyond that IIRC

Oh ruby-build. That's the one I tried

I think it had the same OpenSSL problem

4:47 AM

brew install openssl?

kidding

Did you try rbenv with ruby-build?

It probobly still won't work, but it might

I'm going to try rbenv

Not right now though

@ArtOfCode It's occurred to me that some boots (Natty) aren't as simple as true/false and output a score

I think I'll handle that by making a threshold in the room data

This also doubles as fulfilling a dream I had for it being able to interface with a completely foreign API like say Perspective without a lick of code

Also: autoflagging. We're currently the only folks that do multiple autoflags

Everyone else has their bot account cast a single autoflag

So we need two options for that

What does it take for autoflagging?

Unless

We could just set the flags cast cap to 1 and make a System user with the access token for the bot account be the only eligible flagger (via permissions)

Hm

@AJ not sure what you mean

@quartata I'm going to sleep, but ping me with chatx/ruby questions and when you start on this project

mkay

4:58 AM

Well, you need users for flagging. And they have to be logged-in for flagging. So, how does Smokey do that automatically?

You're asking about Metasmoke? It uses API write tokens

They're issued once for an app

Restart: API quota is 19403.

naa by suraj on Why do the Shuttle's external tanks appear to be rusted? [MS]

fp by suraj on Php submit button not working and error messages not showing up [MS]

@quartata Oh

5:13 AM

[ SmokeDetector | MS ] URL in title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +6 more: wellnesssupplement.com/testro-t3-reviews/ by viefebrem on apple.SE

[ SmokeDetector | MS ] URL in title, bad NS for domain in body, bad NS for domain in title, blacklisted website in body, blacklisted website in title, +3 more: healthpurelives.com/erx-pro/ by user271157 on apple.SE

Suraj Rao

sd 2k

@ArtOfCode One last thing that I put forward somewhat hesitantly... should we do this in ASP.NET MVC?

We talked a lot about what would happen if SE ever adopted SmokeDetector in a more official capacity, but I think we all agree that as currently designed SmokeDetector is decidedly external

It's a standalone headless program and it's not something that could really fit right into whatever SE's codebase looks like

This, on the other hand, is a different story. It's basically a new SE API for content filtering bots

If we made it in C# it could almost be a drag and drop if that ever happened

Just the various sinks and sources could be replaced with more internal equivalents -- write messages straight to the DB, have feed accounts for bots, ...

This is firmly in the realm of fantasy and yet it's the kind of future think I can't resist which is why I'm bringing it up

cc @Undo

5:32 AM

Nooooooo

I've been working on an ASP.NET MVC Enterprise app. It's not awesome. Seems like open source support just stopped at some point in 2014.

And perf is better on Rails for some bizarre reason that I can't figure out.

If we actually integrate with SE, any scenario where that happens will involve enough investment from them to overcome language differences. Don't design into a corner hoping for that corner case.

sd 2k

The chances of that happening are slim anyway

sd 2spam

[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +3 more: testosteronesboosterweb.com/grs-ultra-cell-defense/ by grevinfogg on askubuntu.com

@Andy SVG works, I guess. I've never really worked with those.

5:40 AM

[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +5 more: healthflyup.com/primal-xl/ by sxjyadkhs on askubuntu.com

Suraj Rao

sd 2k

sd 2k

@SmokeDetector why