« first day (1747 days earlier)      last day (2071 days later) » 

@SmokeDetector n
 
here we go again, this feel familiar
 
Back to December MySQL?
 
something is eating all the credits
can't get in to see what
 
@Undo Good morning
 
2:10 PM
@thesecretmaster I unpinned this since Undo's back.
 
o/
When'd it start?
 
Let me check...
Might be briefly before this:
Ah hah, exact time available here:
 
@Undo Can't you check the credit graph to find out?
 
5 hours ago, by SmokeDetector
File "/usr/lib/python3.6/ssl.py", line 814, in __init__
self.do_handshake()

File "/usr/lib/python3.6/ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()

File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()


ConnectionResetError: [Errno 104] Connection reset by peer
2018-08-30 09:05:21.574440 UTC
File "/home/smokey/SmokeDetector/excepthook.py", line 54, in run_with_except_hook
run_old(*args, **kw)

File "/usr/lib/python3.6/threading.py", line 864, in run
 
@thesecretmaster that'd be too smart
 
2:14 PM
ConnectionResetError: [Errno 104] Connection reset by peer
 
11a UTC?
suspiciously round
 
Isn't there some chronic job that runs then?
 
bet so
if I could get in and see
 
We could also just check the codebase
 
so ... MS still not brought up yet?
 
2:15 PM
@Undo It'd be in prod logs
 
db backups run at 11a.
So that'd make sense
 
MariaDB
 
Succeeded, though
 
Backup started, maria expanded to eat memory, system kaboom, maria finished?
 
You probably want to verify backup integrity
 
2:16 PM
Kaboom should have stabilized it, though, not pegged the CPU for hours
 
teward/Solar Flare received failover signal.
 
It'd be nice to know what process what eating the CPU
 
Restart: API quota is 18877.
 
@Henders @Mithrandir Y'all can re-add MS to your configs.
 
spawns
 
2:17 PM
!!/location
 
@Henders teward/Solar Flare
@Henders Mithrandir/Alqualondë
 
@Undo Try starting the db backup manually and see what happens?
 
Worked
No issues
 
@thesecretmaster Done, thanks :)
 
!!/standby Mithrandir
 
2:18 PM
[ SmokeDetector ] SmokeDetector started in standby mode at rev 7c06a99 (iBug: Switch to broad exception for MS downtime) (running on Henders/EC2)
 
!!/location
 
@thesecretmaster teward/Solar Flare
 
!!/loc
 
[ SmokeDetector ] SmokeDetector started in standby mode at rev 7c06a99 (iBug: Switch to broad exception for MS downtime) (running on Mithrandir/Alqualondë)
@iBug No such command 'loc'.
 
!!/location
 
2:18 PM
@Henders teward/Solar Flare
 
Only other thing that could possibly run at 11:00 is check_smokey_status, and I kinda doubt it did this
 
OK, we're all back now.
 
@Undo I want to blame Maria, but I'm not sure how
 
Feels like it should be
Don't know how to do any backwards-looking analysis, but going to throw cpu-process usage in a file every few minutes and hopefully catch the next one
 
Sounds good
 
2:21 PM
sensing that the next problem will be caused by htop :)
[ Undo: sorry for that, this time htop exploded ]
 
We should also stop Smokey from grinding to a halt without MS
5
 
yes please
 
@thesecretmaster I doubt it's caused by MS
because after that it still has some significant lag/delay
 
It didn't work until we disabled MS in the config, right?
@iBug I don't think so
 
uhm, right
 
2:22 PM
No more delay than delay that is attributable to queue size.
 
queuing what?
 
API posts
 
Posts come down the socket, go into a queue, then get sent to the API in batches
 
[ SmokeDetector | MS ] Url in title, bad keyword in body, blacklisted website in body, blacklisted website in title, link at end of body, +4 more: www.7supplements.org/vital-progenix/ by Hineseepthis41 on drupal.SE
tpu- by iBug
 
Ah, I forgot that
 
2:24 PM
!!/watch \d+\W?canada\W?inc
 
@Glorfindel Added \d+\W?canada\W?inc to watchlist
 
Before we turned of MS in the config, Smokey was being slow on everything.
Not just reporting
 
Restart: API quota is 18777.
 
Could be done in some dirty and clean way
 
2:26 PM
One more reboot
 
The problem seems to be communication with MS blocks the main thread
 
Also, some useful things to have would've been !!/no-ms and !!/unstandby, although the latter is prolly hard because I believe it doesn't listen to chat in standby.
 
@thesecretmaster not hard to tweak that (standby not listening chat)
 
Or specifically !!/reboot no-ms
 
neither looks difficult, provided I have spare time
 
2:27 PM
I don't develop smokey, so I don't know how good/bad/hard/easy these ideas are, but they'd be nice.
 
If some more significance is attached to this, I will spend some time on it.
 
[ SmokeDetector | MS ] Potentially bad keyword in answer: Can my attitude kill you? Part 2 by RespectBoundaries on philosophy.SE
fp- by iBug
fp by doppelgreener on How to install MuMuDVB [MS]
fp by doppelgreener on How to install MuMuDVB [MS]
tp by doppelgreener on Crawling customer reviews from Amazon [MS]
 
@iBug It's not urgent or even really necessary
Nice to have just in case this happens again.
 
I can think of a quick patch that disables MS once it's found down
Then when MS is back up again, reboot Smokey
 
2:33 PM
@iBug Maybe retry n times, then disable if it fails.
 
yeah
 
[ SmokeDetector | MS ] Potentially bad keyword in answer: Can I delete unnecessary device simulators of Xcode? by Ricardo Barroso on apple.SE
 
It should also post a message in chat, like "Failed to communicate with MS, disabling MS"
 
fp- by iBug
[ SmokeDetector | MS ] Manually reported answer: Counting the number of words in a string in eclipse by yoyo on stackoverflow.com
[ SmokeDetector | MS ] Link at end of body: IPv6 firewall configuration ✏️ by Alberto on superuser.com
fp- by iBug
CI on 8becde9 succeeded.
[ SmokeDetector | MS ] Username similar to website in answer: Crawling customer reviews from Amazon ✏️ by Octoparse on datascience.SE
tpu- by iBug
 
Hey I wasn't able to find the community for History what is it called?
 
2:45 PM
@Adam History.SE?
 
@SmokeDetector they very specifically added attribution, though it's arguably vaguely NAA
 
[ SmokeDetector | MS ] Potentially bad keyword in body: Pac proxy configuration and libraries by Darknjan994 on stackoverflow.com
 
ok cool I typed History in the search bar and it came up with a bunch of unrelated stuff about powershell etc
 
fp- by iBug
 
2:47 PM
@Adam weird, for me it works:
 
Ah all good my bad I was obviously entering keywords in the wrong search window it comes up from the pull down menu search bar that says "search for other stack exchange communities"
 
obviously I should tried that first. I have a tendency to ask for customer assistance whenever a website has a chat engine. Literally anything other than the thing I feel like studying in the present moment I sometimes don't even google sorry guys
 
CI on 6cf8923 succeeded.
 
I'll make a brief PR for the MS down detection
Basically, consider MS is down upon N consecutive connection failures, and disables MS afterwards. A reboot will bring it back again
 
[ SmokeDetector | MS ] Blacklisted user: I am a programmer, how do I get into field of Data Science? ✏️ by Octoparse on datascience.SE
naa- by iBug
 
3:01 PM
@iBug I'd suggest only enabling it again on manual reboots, otherwise it'll do this every time someone adds a watched keyword
 
@ArtOfCode I doubt it's possible to watch something when MS is down.
 
Prolly not, but there are a bunch of automatic reboots and we probably don't want to have to detect MS down every time
 
Is there any good approach to make a boolean value survive a reboot?
 
Why would we detect MS as down after a automatic reboot?
Oh nevermind
 
@iBug use exit codes, nocrash keeps track of those
 
3:05 PM
I'll look into that when I have time, thanks for the hint
 
@ArtOfCode By the way, I think I've finally given up on integrating CSS into the pack
 
Zoe
@SmokeDetector why
 
@Zoe Blacklisted user - blacklisted for //datascience.stackexchange.com/a/37588 (m.erwaysoftware.com/posts/uid/datascience/37588) by the metasmoke API
 
Alright, got some heavy work tomorrow so sleeping now
Night all
Oh, by the way, am I eligible for a Smokey runner? @ArtOfCode
I'm on EC2 free tier now and I'll probably buy a reserved EC2 T3 instance after it expires.
 
[ SmokeDetector | MS ] Pattern-matching website in body: Analytics service-to-service google-auth-library-nodejs by AHousker on stackoverflow.com
fp- by iBug
 
3:19 PM
@iBug thanks for your help today, sleep well!
 
tpu- by iBug
fp- by Zoe
 
@ArtOfCode Any objections to me adding fontawesome to metasmoke?
Actually maybe I can just take the one icon I need
 
[ SmokeDetector | MS ] Potentially bad keyword in answer: How to bulk delete all users with no posts? by aman on wordpress.SE
tpu- by doppelgreener
[ SmokeDetector | MS ] Bad pattern in url answer, blacklisted website in answer, potentially bad ns for domain in answer, blacklisted user: Ubuntu vs Windows : Virus definition by bean987 on askubuntu.com
[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer: Is there any word mean `split text into sentences`? by user81451 on ell.SE
tpu- by Glorfindel
tpu- by Glorfindel on Ubuntu vs Windows : Virus definition [MS]
tpu- by Glorfindel on If you expected to smother your [MS]
[ SmokeDetector | MS ] No whitespace in body: Jkkkkbhgjcvvvcgcxjvchbcfhc ✏️ by Jonert6544 on superuser.com
tpu- by ByteCommander
[ SmokeDetector | MS ] Blacklisted website in body: Get Ounces of buds for half price all AAA buds by Gosh44 on bitcoin.SE
tpu- by Zoe
 
@thesecretmaster Post 1: This does not look like spam
 
Hrm. Not sure. Thoughts?
 
yeah that's spam
title bears no relation to the body
 
4:57 PM
@thesecretmaster No, if it's not already there it should be.
Should already have the Bootstrap version, right?
Or did it get clobbered somehow
 
@Undo It's not there, at least in dev
Had to manually add the 1 thing I need in
 
@thesecretmaster None in the slightest, I've been wanting to do that for ages.
 
fp- by Glorfindel
 
Just go ahead and add the whole thing, much easier than just getting single icons :)
@iBug provisional okay with a cc @Undo
 
@ArtOfCode I actually think I'm just gonna add the one, because it makes more sense for how I'm packaging this thing. But I can also just add fontawesome anyways.
 
Aye, please do
I keep running up against not having enough icons
 
5:22 PM
@iBug yes
 
@Undo @iBug granted; check your user dropdown for new shinies
 
[ SmokeDetector | MS ] Manually reported answer (batch report: post 1 out of 3): YouTube account suspended, what now? ✏️ by Kevincrowe on webapps.SE
[ SmokeDetector | MS ] Manually reported answer (batch report: post 2 out of 3): How to disable a YouTube comment notification? by Kevin Crow on webapps.SE
[ SmokeDetector | MS ] Manually reported answer (batch report: post 3 out of 3): How to disable a YouTube comment notification? ✏️ by Kevin crowe on webapps.SE
 
Three different accounts, with different usernames cleverly chosen to pretend to be three different people.
 
tpu- by Mithrandir on YouTube account suspended, what now? [MS]
tpu- by Mithrandir
 
@ArtOfCode Any chance you've got some free time to make my new domain tagging thing a bit prettier? About to push changes, but it's kinda ugly.
 
5:41 PM
@thesecretmaster possibly, I'll have a look tomorrow
 
Alright. Want me to hold off on pushing, or let it be ugly until tomorrow?
 
5:52 PM
Eh I'll push+deploy and let people poke at it, if it's horrible we can always rollback.
Speaking of which, @Undo any chance you could travis-ify a cap rollback?
 
6:07 PM
[ SmokeDetector | MS ] Pattern-matching website in answer: What is the HTML5 alternative to the obsolete meta http-equiv=content-language. by Ehaam on stackoverflow.com
[ metasmoke-deploy ] deploy started by thesecretmaster
[ metasmoke ] ci/circleci failure on 0ced9b1: Your tests failed on CircleCI
 
@SmokeDetector Build failed, trying to sort out why.
 
[ SmokeDetector | MS ] Url in title, bad keyword in body, bad keyword in title, blacklisted website in title, pattern-matching product name in body, +3 more: www.healthfitcenter.com/keto-blast/ by jave on meta.SE
 
Do not forget. I'm a "good guy," regardless of what shows up by the auto-detector.
I'm a garbage collector and humane deflector.
Bye for now.
 
@ArtOfCode Ever had webpack fail in the deploy job with no useful output at all? Can't seem to figure out what happened.
 
6:26 PM
@thesecretmaster possibly
 
6:58 PM
[ metasmoke-deploy ] deploy started by thesecretmaster
 
(trying a fix)
 
[ metasmoke ] ci/circleci failure on 22751f8: Your tests failed on CircleCI
 
Deploy failed again
 
@thesecretmaster Theoretically possible, but rollbacks are complicated.
 
Ah because DB stuff
Right?
 
7:10 PM
It's not smart enough to undo migrations (because by the time it hits the migration step, it's already reverted the whole migration file)
yeah
If you're just adding rows or something it probably wouldn't hurt much, but it's a big can of stateful worms
Might be worth it
 
Well, either way, a rollback button with a "WATCH OUT FOR MIGRATIONS" warning would be neat
Maybe a separate rollback migrations button. But that may be too many buttons.
 
could enforce a migration file check in Git, I suppose.
messy, but kinda neat
 
While you're here, any idea why my deploy is breaking?
 
have about thirty seconds, but yes
afk
@thesecretmaster discourse.roots.io/t/yarn-build-error-exit-code-2/10225/2 is the best I can find quickly
 
[ SmokeDetector | MS ] Bad keyword in link text in answer: Problem finding an element by id with Selenium by Hitchman34 on stackoverflow.com
 
7:22 PM
@Undo My first problem is that I can't repro locally. bin/webpack or bin/webpack-dev-server run fine, so does yarn install.
And assets:precompile does as well
 
8:01 PM
sd f
 
Mithrandir/Alqualondë received failover signal.
Restart: API quota is 19991.
 
!!/status
 
@ThomasWard Running since 15:38:56 UTC (4 hours, 29 minutes, 55 seconds)
 
!!/location
 
@ThomasWard Mithrandir/Alqualondë
 
8:09 PM
oops looks like Solar explodified xD
time to fix
 
Restart: API quota is 15026.
 
!!/location
 
@ThomasWard teward/Solar Flare
@ThomasWard Mithrandir/Alqualondë
 
!!/standby Mithrandir
 
8:14 PM
lol doublerun xD
 
[ SmokeDetector ] SmokeDetector started in standby mode at rev 7c06a99 (iBug: Switch to broad exception for MS downtime) (running on Mithrandir/Alqualondë)
fp by Nisse Engström
 
9:54 PM
[ SmokeDetector | MS ] Few unique characters in body, repeating words in body: matching tokens (punctuation) logic ✏️ by A. Bill on stackoverflow.com
 
rofl
> snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip snip
 
tp by Tetsuya Yamamoto on Why Malwarebytes not cleaning virus [MS]
tp by Tetsuya Yamamoto on Trevulan-fitnesstalkzone.com/trevulan [MS]
 
10:09 PM
looks like we are a bit low on instances
 
tp by Tetsuya Yamamoto on Jkkkkbhgjcvvvcgcxjvchbcfhc [MS]
tp by Tetsuya Yamamoto on YouTube account suspended, what now? [MS]
tp by Tetsuya Yamamoto
fp by Tetsuya Yamamoto on get posts orderby custom field in wp json [MS]
 
10:26 PM
[ SmokeDetector | MS ] Few unique characters in answer: jQuery AJAX-request with JSON to PHP ✏️ by dprogramz on stackoverflow.com
[ metasmoke ] ci/circleci failure on 52652be: Your tests failed on CircleCI
[ metasmoke ] ci/circleci failure on 52652be: Your tests failed on CircleCI
tp by Tetsuya Yamamoto on May a man become a gynecologist? [MS]
 
@SmokeDetector While I can guess at what this rubocop problem is, someone with some experience with the language should fix it. But, the PR does what was intended: fix the xo (JavaScript) issues for CI.
It would be nice if we could get MS to the point that CI actually passes.
 
@Makyen There's a larger problem with MS CI -- we've got mystery possibly db related issues that've existed for about a month now (more?).
 
Also right now I've broken MS deploying... somehow... so that's a problem too.
 
10:38 PM
By the way, if anyone has actual urgent changes to MS, feel free to revert my stuff while I sort out what I did wrong
 
@thesecretmaster Take a look at the CI timeout issue, I suspect it'sbone of your commits (I commented).
 
Yeah, the query timings log thing
 
@thesecretmaster Yeah, there are other issues. I must admit I find it frustrating, mostly because when I get a CI error on a PR/commit I make I feel an obligation to resolve it. It just feels like I'm leaving something incomplete that I should be doing.
 
@iBug But CI also passed once after those changes occurred, so I don't think that's the problem. My feeling is that something changed in CircleCI.
Maybe I'll try reverting them in a branch
 
@Makyen I've been ignoring CI failures and deploying them for a month or two now, you're probably good :)
 
10:40 PM
Can we employ Travis CI?
@ArtOfCode Hidden failures were also deployed :)
 
@iBug Yes, but we'll need to set up Stuff again
 
@ArtOfCode Yeah, it's just one of those nagging feelings. :-;
 
[ metasmoke ] ci/circleci failure on b13de82: Your tests failed on CircleCI
 
MS key vs MS token??
 
10:55 PM
@SmokeDetector Usable output from CI again! Gonna see if I can make the job get along with CI now. cc @ArtOfCode @Undo
 
@iBug hm?
 
also, @thesecretmaster link me to your tag lib repo?
 
I'd love a code review if you could give me one
Example is in GH pages
 
[ metasmoke ] ci/circleci failure on 1016b78: Your tests failed on CircleCI
[ metasmoke ] ci/circleci failure on 2979453: Your tests failed on CircleCI
 
11:05 PM
@thesecretmaster Can you give me push privs for a sec so I can create an empty branch to review off?
 
[ metasmoke ] ci/circleci failure on 9b8117c: Your tests failed on CircleCI
 
ta
 
@ArtOfCode Do you have access to artifacts on circle?
 
[ metasmoke ] ci/circleci failure on 832c542: Your tests failed on CircleCI
 
11:12 PM
@thesecretmaster MS?
 
I have a tab for it, your latest commit doesn't have any
 
@ArtOfCode On this page I see "Key" and an action named "New Token", I wonder what's the difference
 
@iBug absolutely nothing :)
 
So the "new token" actually invalidates the current key and generates a new one?
 
11:13 PM
[ SmokeDetector | MS ] Offensive answer detected, potentially bad keyword in answer, toxic answer detected: What is the difference between user name, display name and log-in name? by Troy Simpson on unix.SE
 
@SmokeDetector k
@ArtOfCode What about SE chat and GH creds?
 
@iBug oh those things
Are you in the Keybase team?
 
uh, no
 
Have you got a Keybase account?
 
11:20 PM
 
@ArtOfCode logs/something_something_errors.log from this build
 
Is that required?
 
@thesecretmaster No artifacts
@iBug Kinda
 
[ metasmoke ] ci/circleci failure on a0927ff: Your tests failed on CircleCI
 
It's how we manage secure kex
 
11:21 PM
@ArtOfCode Check now?
 
@thesecretmaster I have coverage artifacts, but no logs
 
Oh huh
 
did you declare the file to be collected?
 
I declared the dir to be collected
logs
 
@ArtOfCode I was expecting some (encrypted) files via e-mail.
 
11:23 PM
@iBug Can be done, but keybase is easier if you're willing to sign up
 
[ SmokeDetector | MS ] Potentially bad keyword in answer: Support chatroom triggers conflicts. How to build a code of conduct? by Maria Brown on communitybuilding.SE
 
@thesecretmaster maybe try an absolute path?
 
Uploading /home/circleci/Charcoal-SE/metasmoke/logs to home/circleci/Charcoal-SE/metasmoke/logs
  No artifact files found at /home/circleci/Charcoal-SE/metasmoke/logs
 
@ArtOfCode I'm doing that
 
naa- by Makyen
 
11:32 PM
@ArtOfCode keybase.io/iBug
 
[ metasmoke ] ci/circleci failure on b54f1b9: Your tests failed on CircleCI
 
@iBug You're in both teams now. In the charcoal team filesystem, there's a sample config file containing all the credentials - kbfs/team/charcoal/sample-config
 
[ metasmoke ] ci/circleci failure on 23f5e50: Your tests failed on CircleCI
 
Gawd I'm so close to actual usable logs from the CI issue...
 
[ SmokeDetector | MS ] Mostly dots in body: Não sei apagar isso ✏️ by vinicius on pt.stackoverflow.com
[ metasmoke ] ci/circleci: js failure on 843f3c2: Your tests failed on CircleCI
 
11:50 PM
@thesecretmaster ya might wanna just set aside tomorrow for going through this code review...
 
I wrote a lot of code? Or I wrote a lot of bad code?
 
[ metasmoke ] ci/circleci: js success on b942615: Your tests passed on CircleCI!
[ metasmoke ] ci/circleci failure on e950c05: Your tests failed on CircleCI
 
@thesecretmaster You wrote a lot of code, and I wrote a lot of comments
 
Ah thanks
Trying to speed up tests, btw
 
Which doesn't necessarily mean you wrote bad code, it's entirely likely to be half your misunderstandings and half mine :)
 

« first day (1747 days earlier)      last day (2071 days later) »