« first day (1774 days earlier)      last day (2039 days later) » 

2:00 PM
Makes debugging kinda hard
Might just give ya'll a reboot button
 
that sounds more like an OOM problem if it's killing off cronjobs
 
@ThomasWard I'm mostly clueless about the OOM killer, but is it plausible for it to kill off cron before Maria (which I assume is the big player here)?
 
Restart: API quota is 18510.
 
!!/status
 
@Undo Running since 14:01:16 UTC (51 seconds)
 
2:02 PM
!!/master
 
[ SmokeDetector ] SmokeDetector started at rev f268a85 (SmokeDetector: Auto watch of telecoms\W*supermarket by Glorfindel --autopull) (running on Henders/EC2, Python 3.4.3)
Restart: API quota is 18484.
 
@Undo it could, but sounds unlikely, it'd require some digging.
 
I also kinda don't understand why it wouldn't kill off Maria first, and leave nginx and Passenger to return messages about the db connection being dead
It's like it nukes the whole thing at once
... or maybe not. I'm not sure if Maria is still running, since I can't get a session
It could nuke nginx and stop there, I suppose.
 
[ SmokeDetector | MS ] Manually reported answer (95): Submitting issues to a public GitHub project anonymously by Eheh on stackoverflow.com
tpu- by geisterfurz007
[ SmokeDetector | MS ] Bad keyword in link text in body (64): RTMP to HLS conversion frame rate issues with ffmpeg/nginx ✏️ by tommy61157 on superuser.com
[ SmokeDetector ] SmokeDetector started in reverted mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on ArtOfCode/EC22)
 
oh dear a hard crash o.O
 
2:08 PM
@Undo You could always check /var/log/syslog.1 after reboot to see if theres anything there
 
[10:57:09.814] Couldn't bind to MS websocket hardcrash o.O
 
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Solar Flare)
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse)
 
um why are we going for Py3.4 compat if we don't have features in it that require us to be on newer Python?
i'm confused :|
didn't an updated Python become a thing a while ago (newer than 3.4 and 3.5)?
 
[ SmokeDetector | MS ] Link at end of body (99): SMTP Service STARTTLS Plaintext Command Injection ✏️ by Esskay on security.SE
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Solar Flare, Python 3.6.5)
 
@Ferrybig aha
 
2:11 PM
Restart: API quota is 19997.
 
Sep 26 02:02:47 ip-172-30-0-53 kernel: [650839.942973] ntpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
 
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse, Python 3.6.5)
 
erm?
!!/location
 
@ThomasWard teward/Solar Flare
@ThomasWard Henders/EC2
Restart: API quota is 19624.
 
!!/standby teward
 
2:12 PM
teward/Solar Flare is switching to standby
@ThomasWard teward/Lunar Eclipse
 
!!/standby Lunar Eclipse
 
Restart: API quota is 19622.
Restart: API quota is 19991.
 
@Undo Quick cheatsheet: /var/log/xxx.log: Current log file; /var/log/xxx.log.1: log file from previous rotation; /var/log/xxx.log.3.gz: gzipped log file 2 gens back
 
Restart: API quota is 18910.
 
@Ferrybig This would explain why nothing ever shows up in the not-.1 version
 
2:14 PM
[ SmokeDetector | MS ] Bad keyword in answer, bad keyword in link text in answer, blacklisted website in answer (255): Buffer polyline to kilometres in custom projection in QGIS, Python, R by saimouni on gis.SE
tpu- by Glorfindel
tpu- by Glorfindel
tpu- by Glorfindel
 
@SmokeDetector f
 
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Solar Flare)
 
Are there lots of instances bouncing around this morning?
 
tpu- by Glorfindel
 
I think MS had an explode
 
2:18 PM
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse)
 
and Solar just had a revert for no reason
so i restandby'd it
 
sshd died?
 
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Solar Flare, Python 3.6.5)
Restart: API quota is 19949.
 
wait, i forgot this
 
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse, Python 3.6.5)
Restart: API quota is 19566.
 
2:20 PM
!!/standby lunar
 
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse)
[ SmokeDetector | MS ] Potentially bad keyword in body (1): Dealing with double standards from my wife by Plamen on interpersonal.SE
[ SmokeDetector | MS ] Potentially bad keyword in body (1): Dealing with double standards from my wife by Plamen on interpersonal.SE
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse, Python 3.6.5)
[ SmokeDetector | MS ] Potentially bad keyword in body (1): Dealing with double standards from my wife by Plamen on interpersonal.SE
Restart: API quota is 19560.
 
!!/blame᠎ ⁠‌‌⁣​
 
@Cerbrus It's Undo's fault.
@Cerbrus It's Undo's fault.
@Cerbrus It's Undo's fault.
 
Who was it that was responsible for SD? He's broken :P
 
2:24 PM
@Henders May I ask you to upgrade Python version? Yours is the only one running Python 3.4 and all others are on 3.6 now.
I would greatly appreciate if you can get Python 3.5 or newer, 3.5 is a big update to 3.4 IMO
 
@SmokeDetector f
!!/rev
 
!!/standby Henders
 
Yeah, it's vaguely on my list, sorry. It's been pushed down a lot recently in favour of more pressing issues but I'll try and increase priority.
 
!!/standby Art
 
ArtOfCode/EC22 is switching to standby
fp- by Glorfindel
fp- by Glorfindel
 
!!/standby Solar
 
Ah, there were 3 instances running...
 
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Solar Flare)
 
2:27 PM
4 actually
!!/location
 
@Glorfindel teward/Lunar Eclipse
 
@Henders you should be able to do a release upgrade to 18.04 and get it for free, so to speak
 
@iBug cc @ThomasWard
 
[ SmokeDetector ] SmokeDetector started at rev f268a85 (SmokeDetector: Auto watch of telecoms\W*supermarket by Glorfindel --autopull) (running on Henders/EC2, Python 3.4.3)
[ SmokeDetector | MS ] Link at end of body (99): SMTP Service STARTTLS Plaintext Command Injection ✏️ by Esskay on security.SE
Restart: API quota is 18225.
 
@SmokeDetector ignore-
 
2:29 PM
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Solar Flare, Python 3.6.5)
Restart: API quota is 19869.
 
@ArtOfCode Hmm, I think something went wrong last time I tried to upgrade which is why I haven't done it yet. I'll try again
 
shrug
 
Yeah, my general linux upgrading / fixing skills are not great :/
 
If you don't have a lot of data, you can get a new instance and migrate to it
 
2:33 PM
fp- by Glorfindel
fp- by Glorfindel
fp- by Glorfindel
 
noooooo .... not again :(
 
!!/location
 
@iBug teward/Lunar Eclipse
@iBug teward/Solar Flare
@iBug Henders/EC2
 
!!/standby-except solar
 
Henders/EC2 is switching to standby (@quartata)
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse)
Restart: API quota is 18161.
Restart: API quota is 18765.
 
2:35 PM
@Henders You can get a new instance with stock 18.04, detach the EBS volume from the old instance and attach it to the new one and copy data.
 
Also, t3.nano is worth trying. Cheaper but with 2x CPU credits.
 
@iBug ProTip - you can mount the EBS volume from a broken instance onto a new one. Works great for copying stuff over quickly.
 
!!/location
 
@iBug teward/Solar Flare
@iBug Henders/EC2
@iBug ArtOfCode/EC22
 
2:36 PM
!!/standby-except sol
 
ArtOfCode/EC22 is switching to standby
Henders/EC2 is switching to standby (@quartata)
 
@Undo And I even "shrinked" an EBS volume in a similar way.
 
[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer (184): How to display a fading alert box by an event trigger by Aleksandr Zorin on stackoverflow.com
tp- by tripleee
 
@iBug Huh, never thought of that. Clever.
 
@SmokeDetector tp- (self-deleted()
 
2:36 PM
tp- by tripleee
Restart: API quota is 19431.
 
so we're switching to Solar then as the primary?
!!/location
 
@ThomasWard teward/Solar Flare
@ThomasWard teward/Lunar Eclipse
 
!!/standby Lunar Eclipse
 
teward/Lunar Eclipse is switching to standby
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse)
[ SmokeDetector ] SmokeDetector started at rev f268a85 (SmokeDetector: Auto watch of telecoms\W*supermarket by Glorfindel --autopull) (running on Henders/EC2, Python 3.4.3)
 
@Undo that's just about dare doing resize2fs -M and dd-ing between volumes.
!!/standby hend
 
2:38 PM
Restart: API quota is 18145.
Restart: API quota is 19417.
[ SmokeDetector ] SmokeDetector started at rev f268a85 (SmokeDetector: Auto watch of telecoms\W*supermarket by Glorfindel --autopull) (running on Henders/EC2, Python 3.4.3)
 
!!/standby-except sola
 
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse)
Restart: API quota is 19332.
Restart: API quota is 18131.
[ SmokeDetector | MS ] Link at end of answer (61): Fast PSD files preview by Milan Vlach on askubuntu.com
 
@Undo MS is still randomly failing over instances
 
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse, Python 3.6.5)
 
@iBug Ohhh, I'll check that out, thanks :)
 
2:44 PM
Restart: API quota is 18755.
Restart: API quota is 19395.
Restart: API quota is 19326.
[ SmokeDetector ] SmokeDetector started at rev f268a85 (SmokeDetector: Auto watch of telecoms\W*supermarket by Glorfindel --autopull) (running on Henders/EC2, Python 3.4.3)
Restart: API quota is 18123.
 
!!/standby-except sola
 
teward/Lunar Eclipse is switching to standby
ArtOfCode/EC22 is switching to standby
Restart: API quota is 19359.
Restart: API quota is 19312.
Restart: API quota is 18110.
teward/Lunar Eclipse is switching to standby
iBug/EC2 is switching to standby
 
I'm going to sleep, hope there won't be 5 instances running together
 
Restart: API quota is 19299.
 
like Fifth Harmony
 
2:48 PM
Does stuff like this warrant a spam flag?
Just consulting.
 
@SmokeDetector ignore-
 
fp- by Makyen
fp- by Makyen
 
!!/location
 
@Makyen iBug/EC2
@Makyen teward/Solar Flare
 
@TheDarkSide looks fine to me, though perhaps I would consider downvoting
 
2:49 PM
Restart: API quota is 19341.
 
Is it intended for the SD instances to be in reverted mode?
 
@tripleee Thanks, but sometimes I find it difficult to understand the line between spam and multiple link-only stuff. Maybe it is instance-specific.
 
!!/master
@Makyen ^ no
 
Restart: API quota is 19257.
Restart: API quota is 19322.
Restart: API quota is 19592.
 
@iBug I looked at pricing for these the other day - ridiculous
when my T2 reservation runs out I might well move to a T3
 
2:54 PM
[ SmokeDetector | MS ] Potentially bad ns for domain in answer (1): How to make a Windows service with parameters? ✏️ by a deleted user on stackoverflow.com
[ SmokeDetector | MS ] Potentially bad ns for domain in answer (1): How to make a Windows service with parameters? ✏️ by a deleted user on stackoverflow.com
 
!!/location
 
@ArtOfCode teward/Solar Flare
[ SmokeDetector | MS ] Potentially bad ns for domain in answer (1): How to make a Windows service with parameters? ✏️ by a deleted user on stackoverflow.com
@ArtOfCode teward/Lunar Eclipse
 
!!/standby-except Solar
 
Restart: API quota is 19292.
Restart: API quota is 19232.
tp by Filnor
tp by Filnor
 
!!/location
 
2:56 PM
@Makyen teward/Lunar Eclipse
@Makyen iBug/EC2
tp by Filnor
@Makyen teward/Solar Flare
 
!!/standby-except solar
 
Restart: API quota is 18720.
 
@SmokeDetector f
 
Restart: API quota is 19220.
Restart: API quota is 19280.
 
!!/location
 
2:57 PM
@Makyen teward/Solar Flare
@Makyen iBug/EC2
@Makyen teward/Lunar Eclipse
iBug/EC2 is switching to standby
Restart: API quota is 19211.
 
!!/standby-except solar
 
iBug/EC2 is switching to standby
Restart: API quota is 19267.
 
!!/master
 
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse, Python 3.6.5)
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Solar Flare, Python 3.6.5)
Restart: API quota is 19243.
Restart: API quota is 19460.
 
!!/standby-except solar
 
3:02 PM
teward/Lunar Eclipse is switching to standby
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse)
Restart: API quota is 19194.
 
!!/master
 
Restart: API quota is 19189.
Restart: API quota is 19444.
 
!!/rev
 
!!/master
 
3:04 PM
!!/location
!!/standby iBug
uhm broken?
 
Restart: API quota is 19426.
Restart: API quota is 19166.
 
!!/location
 
@ThomasWard teward/Solar Flare
@ThomasWard teward/Lunar Eclipse
@ThomasWard iBug/EC2
 
wtf?
 
Restart: API quota is 19230.
 
3:05 PM
!!/standby Lunar Eclipse
!!?standby iBug
!!/location
 
@ThomasWard iBug/EC2
@ThomasWard teward/Solar Flare
[ SmokeDetector ] SmokeDetector started in standby mode at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse)
 
!!/standby iBug
 
@ThomasWard They have been repeatedly coming out of standby for quite some time.
 
sounds like MS is busted?
 
3:07 PM
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Lunar Eclipse, Python 3.6.5)
 
That's what I've assumed.
 
grabs a phaser rifle and shuts off Lunar Eclipse
 
Restart: API quota is 19225.
Restart: API quota is 19151.
 
!!/stappit
 
[ SmokeDetector ] SmokeDetector started at rev c634171 (𝐢𝐁𝐮𝐠 ♦ͦͦͦͦͦ: Python 3.4 compatib) (running on teward/Solar Flare, Python 3.6.5)
Restart: API quota is 19383.
 
3:10 PM
right, so problem solved temporarily by nuking all instances. Since active/standby doesn't work properly right now, please only launch secondary instances if the running instance stops working.
prods @Undo and @ArtOfCode ^
didn't have a choice but to halt all instances and then only launch one, or we'd have multi-active-instance hellstorms
I have to assume MS is at fault but meh
 
30 mins ago, by iBug
@Undo MS is still randomly failing over instances
 
indeed.
so long as we only run one instance at a time right now, we can avoid the issue
it's not ideal though
 
trust your solar flare, it's very stable
 
indeed.
it's why i killed all then started it back up
so long as MS doesn't blow up and kill it for some reason all should be good
 
In case it does, I have a testing instance running in room 80124. Latest spam will continue to be reported, as purging spam is our primary work.
Really going to sleep now, night all.
 
3:20 PM
naa- by tripleee
[ SmokeDetector | MS ] Manually reported question (93): Og weeds 111#, available by josh ben on stackoverflow.com
tpu- by teward
 
@SmokeDetector k
 
3:36 PM
[ SmokeDetector | MS ] Repeating characters in answer (83): fetching data into dropdownlist and get the selected data in php ✏️ by Learner2016 on stackoverflow.com
fp- by Makyen
 
@SmokeDetector fp-
 
3:53 PM
[ SmokeDetector | MS ] Bad keyword in body, potentially bad keyword in body (98): Hormone or androgenic hormone by woxugiwi on meta.SE
[ SmokeDetector | MS ] Potentially bad keyword in body (1): Simple HTML and CSS Question - Text Align Not Working? ✏️ by NoobLearns on stackoverflow.com
fp- by bertieb
tpu- by bertieb on Hormone or androgenic hormone [MS]
 
4:28 PM
@iBug Have we ever seen this before?
Like, in the last few weeks specifically?
Because nothing should have changed in the last couple of days at least around that spot
 
um that sounds like a failed detection pattern o.O
@SmokeDetector fp-
 
5:31 PM
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title (195): Video streaming on movies123 &putlocker by Murray Wright on askubuntu.com
fp- by Erik the Outgolfer
fp feedback on autoflagged post: Video streaming on movies123 &putlocker MS (@angussidney @Floern @Makyen @iBug)
 
@SmokeDetector I'm wondering about how that managed to put a 195 weight on a next-to-obvious FP (I checked the markdown).
and...Flo and Mak didn't even autoflag!
 
@SmokeDetector hmm? didn't use one of my flags
 
yeah, it's a bit stressful when you're pinged for an autoflag you didn't cast :P
 
Retracted the Smokey flag
Not sure why it chose those folks to ping
 
!!/location
 
5:43 PM
adrenaline rush... followed by confusion
 
@EriktheOutgolfer teward/Solar Flare
 
@Undo did you just login to SD's account while Thomas is also logged in? :P
 
@EriktheOutgolfer SE accounts handle multiauth fine
 
ik, SE isn't afraid of multiple IPs logging in concurrently
it's just that it sounds a bit dangerous :P
 
@SmokeDetector @ArtOfCode - mean to use eligible_flaggers here?
 
5:46 PM
heh, whoops
 
dun dun dun...
 
prolly changed that mistakenly while I was changing validation after FPs
 
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, bad keyword in username, pattern-matching website in body, potentially bad keyword in body (386): Binance Lost Wallet: Support Phone Number by Binance Support Number on bitcoin.SE
tpu- by Erik the Outgolfer
 
Would be kinda neat to figure out how to retract Smokey flags
 
yeah, looks like that ping over there sent a nerve up Flo's and Mak's spines... :-(
 
5:48 PM
Just need to fake the login process w/ an email and password. Metasmoke can have that info
 
SD's credentials and then go to the post and bam?
(like, JS)
 
not even JS; just log in, get cookies, fake the POST request for retraction
 
fake POST request for login, then fake POST request for retraction?
 
@EriktheOutgolfer No need to navigate to the post. If you're logged in it's easy to fake. The issue with MS retracting flags has always been that you have to have actually gone through the login process to get the cookies, not just an auth token from the SE API.
 
5:53 PM
^ FP or TP(u)? ...
 
SO mod says it's R/A
 
tpu- by Erik the Outgolfer
 
alright, I trust the mod, whoever may it be, canine or not
:P
 

« first day (1774 days earlier)      last day (2039 days later) »