« first day (1481 days earlier)      last day (2333 days later) » 

4:00 AM
!!/tea
 
@quartata brews a cup of passionfruit tea for @quartata
[ SmokeDetector | MS ] Bad keyword in answer, blacklisted website in answer: Memory Usage statistics different in free and htop by oycebrewton on askubuntu.com
 
yep
just as I feared
ok good thing I was already working on a fix
this instance will need a restart though because now the locking is out of sync
!!/alive
 
@quartata Watching this endless list of new questions never gets boring
 
!!/alive?
 
@quartata Yup
CI on 04d0e74 succeeded.
 
4:10 AM
!!/pull
this should fix it
 
Restart: API quota is 19665.
tpu by quartata
 
sorry about all this
should be good now
 
[ SmokeDetector | MS ] Bad keyword in answer, bad keyword with email in answer, email in answer, pattern-matching email in answer: Why is it "to take someone seriously" and not "to take someone serious"? by Dr.abel Justus on english.SE
 
phew
sd k
 
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, link at end of body, pattern-matching website in body, potentially bad keyword in body: Fortunately, this marketplace has been recognized by plmllyskahjs on meta.SE
 
4:12 AM
right in time for spam o' clock
let me test the deadlock condition
@SmokeDetector tpu-
in English Language & Usage, 3 mins ago, by SmokeDetector
[ SmokeDetector | MS ] Bad keyword in answer, bad keyword with email in answer, email in answer, pattern-matching email in answer: Why is it "to take someone seriously" and not "to take someone serious"? by Dr.abel Justus on english.SE
that worked
 
!!/watch drayasinhealinghome\.com
!!/alive
 
Restart: API quota is 19593.
 
!!/watch dejustushealinghome\.com
 
@tripleee Added dejustushealinghome\.com to watchlist
 
!!/errorlogs 150
 
4:21 AM
ImportError: No module named 'ChatExchange.chatexchange'
2017-11-10 19:22:41.178073 UTC
File "ws.py", line 22, in <module>
from bodyfetcher import BodyFetcher

File "/home/smokey/SmokeDetector/bodyfetcher.py", line 2, in <module>
from spamhandling import handle_spam, check_if_spam

File "/home/smokey/SmokeDetector/spamhandling.py", line 4, in <module>
from findspam import FindSpam

File "/home/smokey/SmokeDetector/findspam.py", line 19, in <module>
from globalvars import GlobalVars

File "/home/smokey/SmokeDetector/globalvars.py", line 5, in <module>
 
@quartata only if there's a better alternative
 
@SmokeDetector what
no no no
I can't deal with this right now, just work
!!/alive?
 
@quartata Kinda sorta
CI on 884a37d succeeded. Message contains 'autopull', pulling...
 
but did that actually finish right then
 
4:23 AM
or were you deadlocking because you suck
 
Restart: API quota is 19573.
 
looks like the latter
this is literally all CE's fault, I shouldn't need a lock when the result is synchronous
!!/alive?
 
@quartata ... did I miss something?
 
!!/queuestatus
sd kaboom-force
!!/tea
 
@quartata brews a cup of green tea for @quartata
 
[ SmokeDetector ] SmokeDetector started at rev 88fc511 (tripleee: Auto watch of claim-lumen\.ga by tripleee --autopull) (running on teward/Solar Flare)
Restart: API quota is 19497.
 
no no no no
what the hell is going on
!!/alive?
 
@quartata You doubt me?
 
!!/errorlogs 250
sd kaboom-force
I'm furious rn
OK I can fix this by going directly to post_fkeyed but it will take time, time I don't have tonight
@angussidney are you around?
 
is there something we need to do differently in the meantime?
 
4:45 AM
we could failover to an instance without it
then we couldn't pull though
 
@ArtOfCode: with regard to the people list: As an instance operator, can you add me to the list?
 
I'm not sure what's causing the lock issues
!!/alive?
 
@quartata plz send teh coffee
 
@bwDraco you mean in the wiki? anyone can edit that I guess
 
@quartata yes, but I'm about to disappear for the next 4 hours
@tripleee no, it's a MD file on the website
if someone makes the change, I'll merge if I get the chance
 
4:51 AM
!!/alive?
 
@quartata Kinda sorta
[ SmokeDetector | MS ] Mostly dots in body: How to hide symlink file? by kst on stackoverflow.com
 
A J
@SmokeDetector Edited
 
!!/alive?
 
@quartata Of course
[ SmokeDetector | MS ] Bad NS for domain in body, blacklisted website in body, link at end of body, pattern-matching website in body: Synoptic Boost As noted earlier by user768754 on askubuntu.com
 
Oh OK
 
maybe it's working now
 
!!/blacklist-keyword synoptic\W?boost
 
[ SmokeDetector | MS ] Offensive title detected: SNL "Can a bitch get a donut?" by D.W. on ell.SE
@tripleee No such command 'blacklist-keyword'.
 
hhhhuh?
 
Thats not right
 
4:56 AM
@SmokeDetector f
 
@SmokeDetector tpu-
@SmokeDetector fp-
 
er um, in which section should bwDraco go?
 
@tripleee sorry ts with an underscore
 
!!/blacklist_keyword synoptic\W?boost
@quartata like that? ugh, that's unattractive
 
I know I know
I seriously don't know how this ended up so fucked up after so much testing
did it actually make the commit
 
5:01 AM
did it take or is something wrong? it seems to provide less feedback than it used to
 
!!/alive?
 
Restart: API quota is 19313.
 
@tripleee its buggy as heck rn
I can't fix this right now
major CE issues
Don't know what to do
This was a mistake, I have no idea how such obscure bugs could totally sink thjs
 
OK this is good
 
5:09 AM
@quartata Take a deep breath and take a break if you need to. Put an issue on GitHub describing the problem so that someone else can take a look. Assume that didn't read enough of the transcript today to see what the current problem is. If no one picks it up in the next few hours, come back to it then.
 
I think if we just don't run commands it will be OK
@Andy Well normally I'd just reset to a previous revision except then we can't blacklist
 
@SmokeDetector f
 
a one line change to CE would eliminate the need for all this
 
Rob
@quartata Why's that?
 
@Rob it'll fast forward to the bad commits
 
5:11 AM
I'm getting "invalid .pullapprove.yml" too
 
Rob
Sure, but you could commit a revert commit all the way back to the good version
 
that's not me
 
Rob
Then when you're working on the fix; you revert the revert, and you're back to where you are now locally
 
[ SmokeDetector | MS ] Bad NS for domain in body, blacklisted website in body, pattern-matching website in body: 15 days, and then you can by user266976 on apple.SE
 
@Rob can't do that on mobile, no one else here has push
 
5:12 AM
"'hichris1234' is not a collaborator of this repo"
it's really hard to find the error message
 
sd k
 
it seems to be keeping itself together now, so it's definitely because of commands
 
@quartata Make that change then and submit a pull request
 
which is weird because I definitely fixed that
@Andy I will but it probably won't be accepted any time soon
ok let me think this through
 
I'm sure we can find someone to pull it in this room
 
5:14 AM
#1345 invalid .pullapprove.yml
 
Manish works at Mozilla now
 
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, link at end of body: The book actively discourages the by Bevedavis on askubuntu.com
 
@SmokeDetector k
 
OK here's a compromise: unless it's a blacklisting command no one run anything that generates output
it should stay stable through spam o clock
if someone does do something do a !!/reboot
here's what's happening basically:
 
@quartata Jeremy Banks can pull PRs in CE. He's pingable here.
 
5:17 AM
when we're about to post a message we wait on an event, clear it and then post the message
when we receive the messge back we set the event and record the ID
if for some reason the event isn't set again when we get it back (nevermind how that's happening) the thread gets stuck on it
initially this was easy to miss because replies to commands were sent outside of this message sending thread , and so they could set the event and wake it up out of the deadlock
 
@quartata I just approved a PR, we have a bunch of watches pending, should I stop doing that?
 
I fixed the wrong problem and made it wrse: now replies are sent through the usual queue which means that no messages will wake it up because all the messages are handled by that thread
@tripleee no that's fine
 
there would normally be the merge notice and then of course the restart when it's pulled
 
the real question is what is happening that's causing the event never to be set again
@tripleee the restart will still occur which will restart the threads
 
augh, CI failed, the watch was already blacklisted
 
5:24 AM
this is not a great start to my first day of direct git access
classic Smokey used a lock too but we shouldnt have to: the message ID is returned synchronously by the request, so if CE would just return that rather than feeding it back into the on_msg callback everything would be fine
it would elininate all of this
ughh
 
reverted PR, CI in progress
 
thank you
did you revert all my other commits too
 
just the blacklist, this isn't progress
 
oh
heck
 
[ SmokeDetector ] SmokeDetector started at rev ccb4b4a (tripleee: Revert #1341 --autopull) (running on teward/Solar Flare)
 
5:28 AM
I don't think I understand enough to decide on a rollback
 
Restart: API quota is 19142.
 
No just do it please I'm begging you
revert the merge of 1135 and all of my commits afterwards
tomorrow I'll reopen the PR, clean up the git history and fix this for real
maybe we can get Art to turn off protected branch so I can expunge the history of the first merge too
 
[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, bad keyword in username, link at end of body, +1 more: votofelforcefinland.com/ultavive-garcinia/ by Ultavive Garcinia on askubuntu.com
 
@SmokeDetector k
 
[ SmokeDetector | MS ] Blacklisted website in body, pattern-matching website in body: globe with its acronym name by uahdbanpqama on askubuntu.com
 
5:33 AM
@SmokeDetector k
 
bad history starts at 51d93, includes Nobody's PR
I have to go now, I am so so sorry
I should have noticed this during testing, I remember seeing somethig very similar about a month ago but I thought it was due to my MacBook falling into Power Nap suspension or whatever
now I know what it was, once the event got set reports flooded in
 
23 files changed in the PR, not sure I'm confident I can roll this back properly
 
@Andy OK that's good, I'll make it less invasive with a kwarg option though
@tripleee git revert <shas starting from merge commit> will handle it fine
it was a regular merge
 
Rob
@tripleee If you're struggling - you can revert all commits until the known good commit, and cherry pick in unrelated and good commits
 
@Rob that's what I was thinking
 
5:42 AM
I feel like an idiot for time and time again putting this burden on people
 
Rob
Don't stress; happens to everyone eventually @quartata
 
Calm down. Take a deep breath. Look at the room description. You're not the first person who's ever badly messed up Smokey before ;)
Remember - it's where we break things by developing on production.
 
\o morning... seems like I walked in on a bad day? SD having issues?
 
!!/alive
 
@Mithrandir don't run commands that generate responses uness it's something that triggers a reboot eventually (blacklisting)
!!/reboot
 
5:50 AM
[ SmokeDetector ] SmokeDetector started at rev ccb4b4a (tripleee: Revert #1341 --autopull) (running on teward/Solar Flare)
Restart: API quota is 19020.
 
@quartata I've pinned your earlier message about that, it seems important :)
 
@quartata hmm, more broken than I thought then
 
@Mithrandir honestly I know it seems ridiculous but the problem is that I'm loathe to just let the cards fall when it's really all I can do, I can't fix it where I am
 
@quartata I may or may not have a few minutes; with a decent testing setup, how hard should it be to fix?
 
Excruciatingly. Angus might be able to do it
 
5:55 AM
PR#1346 temporarily revert #1135 merge + follow-up commits
 
On the bright side I'd say my "fix" actually did make things better
in that it made t more obviously broken
 
@quartata if you're still around, can you review that briefly?
I just cherry-picked everything not by you or referencing you
which turns out to be pretty much my own blacklist charade from the last hour or so
 
You have to do a revert commit, I can't merge that
Since t's a reset
 
@quartata there was a large number of follow-up commits, so this seemed the simplest
 
the histories diverge
merging that will just reapply the blacklist commits
because it's just going to go back to the common ancestor and apply it on top of what's there
 
Rob
6:00 AM
@tripleee It doesn't look like that pullrequest reverts anything; it's just applying the cherry picks?
 
ugh, what did I do
 
you'd have to update the ref to point directly to that new history, which can't be done because of branch protection
 
Rob
I'm pretty decent with git surgery, if you'd like me to put up a pull request
 
@Rob please do -- I'll reject that one
 
I can pick out the SHAs
 
Rob
6:02 AM
Just need the last good commit. I'm guessing 5ed63bb2aeb90a961ceccc127e68f321addc7d25 is the last good one?
Merge pull request #1342 from Glorfindel83/master --autopull … by Undo
 
that's what I concluded too
 
I've got a testing instance up, I'll try to repro
 
good morning
 
o/ morning
 
6:06 AM
@Rob well here are the follow up commits anyways: 04d0e ecae8 089d5 d89fe 3907b 823e2 15673 9baa1 5f486
then get the parent number and revert the merge separately
really have to go now, this was horrendous timing
 
@quartata If you have time (don't bother replying if you don't), are there any steps which are guaranteed to trigger the bug?
 
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, pattern-matching website in body: It is not designed to curing barring no one medically detected illness. by Notat19 00 on graphicdesign.SE
[ SmokeDetector | MS ] Bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, blacklisted website in body, +2 more: getmenshealth.com/titanax/ by user266980 on apple.SE
tpu- by NobodyNada
 
does report work?
 
6:13 AM
[ SmokeDetector | MS ] Bad keyword with a link in answer, link at end of answer: Why are motor capacitors huge in physical size even though their values are in uF range? by Tibcon Capacitors on electronics.SE
tpu- by Videonauth
 
[ SmokeDetector | MS ] Manually reported question: look of maturing signs, by Rufusmato on graphicdesign.SE
 
6:14 AM
@SmokeDetector oh thanks SD
 
@SmokeDetector tpu-
 
@Rob looks good to me, with one minor adjustment (see comment on PR)
thanks for the effort!
 
Rob
@tripleee Done
I've sanity checked that 5ed63bb2aeb90a961ceccc127e68f321addc7d25 (commit by Undo) and ec288d056995bf376580f3bd8747788cad9da99a (my first commit) are identical versions. The rest are just cherry picks
Not sure why the build fails, though
 
6:21 AM
same reason it failed when I first wanted to merge that -- the blacklist entry should not be accepted
 
Rob
Yep, but I've removed that last commit
cf2fae6 is gone
 
that's the latest failure I see
"Your build was cancelled because you have the 'Auto-cancel redundant builds' setting turned on for this project" on the latest build in CircleCI, dunno what that is
 
Rob
Hmm
 
well it is spam
 
Rob
6:25 AM
@tripleee It is curious, as there's a bunch of linting errors
However, the only difference between Undo's commit and the pull request is this:
Though Undo's commit successfully passed CI
Anyway, gotta head home
 
I'll merge that; it's looking to me like we have a broad consensus that this is what we want
 
Assuming that I understood what's going on correctly, I may have figured out a way to repro the bug.
SE drops duplicate chat messages. Using FireAlarm's say command, I was able set up a situation in which Smokey posts the exact same content as the previous message sent from the same account.
although that may not be triggering the same deadlock condition, it seems SmokeDetector recovered after ~30 seconds :/
 
!!/pull
 
erm? did someone get smokey gagged?
 
[ SmokeDetector ] SmokeDetector started at rev d7c8393 (tripleee: Merge pull request #1347 from rjrudman/master) (running on teward/Solar Flare)
Restart: API quota is 18730.
 
@tripleee Post 1: Could not find data for this post in the API. It may already have been deleted.
 
@Videonauth I'm not sure entirely what's going on, but the TL;DR is that we've replaced the chat system, and the new one seems to have a strange deadlock.
 
I'm hoping the latest PR fixes things
I unpinned quartata's message
here it is in case we would up needing it again after all
1 hour ago, by quartata
OK here's a compromise: unless it's a blacklisting command no one run anything that generates output
 
Ah :)
 
"fixes" as in "reverts back to how things were before"
 
6:40 AM
:)
hotboxing does not work as well?
 
@Videonauth It should work...maybe it didn't like the ?fb at the end of the URL?
 
@NobodyNada I was thinking the same thing
so yeah
 
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, link at end of body, repeated URL at end of long post: The dynamic fixings consolidate by mpbellkiil on askubuntu.com
tpu- by Videonauth
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title: Skin Care Creams: How You Can To Cover Your Creams by okokores on askubuntu.com
 
A J
@SmokeDetector autoflagged
 
@AJ That post was not automatically flagged by metasmoke.
 
6:44 AM
@SmokeDetector k
 
Rob
@quartata When you return, you'll need to git revert ec288d056995bf376580f3bd8747788cad9da99a - and you'll be put back to where you were
 
Rob
Looks like it's back to normal
 
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad pattern in URL body, blacklisted website in body, link at end of body, +1 more: When you will overview about a product by Vistenklonem on gaming.SE
tpu- by Rob
[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +3 more: getmenshealth.com/titanax/ by Agiationeinrot on askubuntu.com
tpu- by Videonauth
 
I wonder why this not gets deleted :) askubuntu.com/q/983999
probably some who just voted it down
 
6:54 AM
[ SmokeDetector | MS ] Bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, blacklisted website in body, +2 more: www.getmenshealth.com/titanax/ by maymie vemae on physics.SE
 
@Videonauth it is
 
tpu- by Videonauth
 
@SmokeDetector k
 
now it is :) it was around for almost 3 minutes or so
and sits at -8 on deletion
 
[ SmokeDetector | MS ] Bad keyword in link text in body: How do I set up live Twitch streaming from RetroArch on a Pi 3? by LeoPGiles on raspberrypi.SE
[ SmokeDetector | MS ] Bad keyword in link text in answer: How do I set up live Twitch streaming from RetroArch on a Pi 3? by LeoPGiles on raspberrypi.SE
 
6:57 AM
@SmokeDetector why
 
@SurajRao Post - Bad keyword live-stream in link text
 
@SmokeDetector f
@SmokeDetector f
wonder why it picked old posts
 
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching product name in body, +1 more: Do you have to stress over the symptoms of the PTX Male Enhancement? by RobynWitte on superuser.com
tpu- by Videonauth
[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +5 more: beaudermaskincare.com/krasa-anti-aging-creme/ by user81393 on drupal.SE
tpu- by Glorfindel
tpu- by Glorfindel on www.getmenshealth.com/titanax/ [MS]
tpu- by Glorfindel on Sudden Drop on Indexed URLs [MS]
fp- by Glorfindel on Sudden Drop on Indexed URLs [MS]
 
ok the last two where confusing :) i was wondering over this one on MS for a while now if its fp or tpu and now it is both?
 
@Videonauth Glorfindel overwrote the feedback. So fp
 
7:11 AM
ah ok
 
[ SmokeDetector | MS ] URL-only title, bad keyword in body, blacklisted website in body, blacklisted website in title, link at end of body, +2 more: alphajackedhelp.com/jacked-alpha-test/ by Ninaraftyi on askubuntu.com
tpu- by Videonauth
[ SmokeDetector | MS ] Potentially bad keyword in username: What are steps to perform integration testing? by TestOrigen Pvt Ltd on superuser.com
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, link at end of body, pattern-matching website in body, +1 more: The idol site moreover records by Boug ereb on drupal.SE
tpu- by Videonauth
 
@SmokeDetector f
 
not sure about the android.se one so i keep my fingers of that
 
@SmokeDetector k
 
[ SmokeDetector | MS ] Bad keyword in body, potentially bad keyword in body: dynamic and best in class things. by MarkBooze on drupal.SE
tpu- by Videonauth
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, link at end of body, potentially bad keyword in body: your announcement, you can receive by comecver on askubuntu.com
[ SmokeDetector | MS ] Bad NS for domain in body, blacklisted website in body, link at end of body, pattern-matching website in body: Hair Loss W Optimal Choice CBD h natural re by user768806 on askubuntu.com
 
7:24 AM
@SmokeDetector k
@SmokeDetector k
 
[ SmokeDetector | MS ] Misleading link: Simulate cat Linux command in python 3 by Ahmed H. Elsers on stackoverflow.com
 
@SmokeDetector f
 
fp- by Videonauth
[ SmokeDetector | MS ] Potentially bad keyword in body: Buy Flowers For Valentines Day by valentines0618 on askubuntu.com
tpu- by Videonauth
[ SmokeDetector | MS ] Bad keyword with email in answer: How do I fix a broken mspaint? by Mike Adams on superuser.com
[ SmokeDetector | MS ] Bad keyword with email in answer: Google snippet tree structure by Bharti on stackoverflow.com
 
7:40 AM
!!/watch kharchabachao\.com
@SmokeDetector k
 
@tripleee Added kharchabachao\.com to watchlist
 
@SmokeDetector k
 
CI on a6b658a succeeded. Message contains 'autopull', pulling...
Restart: API quota is 18290.
 
!!/watch download25\.com
 
@tripleee Added download25\.com to watchlist
[ SmokeDetector | MS ] Link at end of body, pattern-matching website in body: Renuvaline cream by samienerel on meta.SE
tpu- by Videonauth
CI on b2b1a95 succeeded. Message contains 'autopull', pulling...
Restart: API quota is 18244.
 
7:49 AM
!!/windstream\.net
 
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, +1 more: Mass in bones. Noticeable massive testo by Gack197 on workplace.SE
[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, blacklisted website in body, blacklisted website in title, +4 more: freesupplementrial.com/ultra-potent-libomax/ by mkhzydbqb on askubuntu.com
tpu- by Videonauth
 

« first day (1481 days earlier)      last day (2333 days later) »