« first day (1481 days earlier)      last day (2524 days later) » 

12:29 AM
[ SmokeDetector | MS ] Bad keyword in answer: target= _blank is Follow or NoFollow by Nur Roni on stackoverflow.com (@JakeSymons)
Restart: API quota is 16506.
[ SmokeDetector | MS ] Bad keyword in answer: target= _blank is Follow or NoFollow by Nur Roni on stackoverflow.com (@JakeSymons)
Restart: API quota is 16504.
tpu- by Tetsuya Yamamoto
Conflicting feedback across revisions: current, #1
@TetsuyaYamamoto brews a cup of green tea for @TetsuyaYamamoto
1.2x damage factor... while red tea only 0.6x.
12:45 AM
Conflicting feedback across revisions: current, #1
@ArtOfCode Was it a post on StackOverflow?
@quartata yup
If not it won't be posted there
Link?
(I know report works so it might be a configuration issue)
@quartata 1 and 2
[ SmokeDetector | MS ] Pattern-matching website in answer: HTML/CSS/Javascript: Open/Close Menu when Clicking Button by Travor Liu on stackoverflow.com (@JakeSymons)
Restart: API quota is 16365.
12:52 AM
@SmokeDetector why
Restart: API quota is 16363.
fp- by Tetsuya Yamamoto
@ArtOfCode ok I'll repro it when I get home
@SmokeDetector @quartata you broke it
or someone did ;)
!!/errorlogs 150
sd poof-force
2 issues going on
1 is most likely pickle Rick
The other is deletion watcher
Incidentally those are the two things I don't unit test pretty much
!!/threads
140030095042304: MainThread
140029991765760: message_sender
140029322049280: bodyfetcher post enqueing
140029982848768: message_sender
140029974456064: message_sender
140029733095168: Thread-4
140029296871168: Thread-29
140029305263872: bodyfetcher post enqueing
140029741487872: Thread-3
140029330441984: bodyfetcher post enqueing
140029724702464: pickle ---rick--- runner
140029749880576: Thread-2
140029355620096: Thread-27
140029758273280: Thread-1
140029347227392: Thread-6
140029338834688: metasmoke websocket
1:02 AM
Meta Tavern hasn't noticed yet
I give it about 3 hours
!!/master
Restart: API quota is 16254.
1:36 AM
[ SmokeDetector | MS ] Few unique characters in body, no whitespace in body: Cannot be resolved to a type by John Smith on stackoverflow.com (@JakeSymons)
tp- by Tetsuya Yamamoto
@quartata I think this is already included on some instances - Solar Flare already has pyyaml, as does Lunar Eclipse, maybe a side effect of certain packages installed by python3-all on Ubuntu systems?
Mine has it
interesting
OK so SOCVR is getting reports
I'm not sure why manual reports wouldn't work then
@ArtOfCode stand by to delete an MS record
@ArtOfCode do you have SOCVR privileges?
1:51 AM
i have all teh privs
@ArtOfCode k, can you report stackoverflow.com/a/38362924
in there
@quartata IIRC every Smokey maintainer has privs in every channel for Smokey... myself included.
Restart: API quota is 15976.
!!/errorlogs 50
File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 35, in check_websocket_for_deletion
post_type = post_site_id[2]


IndexError: list index out of range
2017-12-07 00:51:33.495669 UTC
File "/home/ubuntu/charcoal/SmokeDetector/excepthook.py", line 54, in run_with_except_hook
run_old(*args, **kw)

File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 97, in post_message_if_not_deleted
1:53 AM
i think it asploded?
yes but those logs are old
I need a tail
those ones arent
IndexError: list index out of range
2017-12-07 01:52:50.342687 UTC
File "/home/ubuntu/charcoal/SmokeDetector/excepthook.py", line 54, in run_with_except_hook
run_old(*args, **kw)

File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 97, in post_message_if_not_deleted
was_report_deleted = self.check_websocket_for_deletion(post_site_id, post_url, 300)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 35, in check_websocket_for_deletion
!!/errorlogs 150
for i, char in enumerate(reversed(x)):


TypeError: descriptor 'add' requires a 'set' object but received a 'tuple'
2017-12-06 21:40:55.066115 UTC
File "/home/ubuntu/charcoal/SmokeDetector/chatcommunicate.py", line 316, in f
result = func(*processed_args, **({"alias_used": alias_used} if give_name else {}))

File "/home/ubuntu/charcoal/SmokeDetector/chatcommands.py", line 333, in watch
return do_blacklist(website, "watch_keyword", msg, force=False)

File "/home/ubuntu/charcoal/SmokeDetector/chatcommands.py", line 299, in do_blacklist
^ latest error log
@quartata you've got an IndexError problem
1:54 AM
that wouldn't cause the restart
that's the last crash
that's a deletion watcher error
!!/location
@quartata angussidney/EC2
remind me how many times we've said "that can't be it" and it was again
if it's an uncaught or unhandled exception it could trigger a restart
@ArtOfCode fifty billion
of which 20 billion were my saying it
1:55 AM
It literally triggers on every post.
Watch
okay, so fix that and you'll probably fix the manual report too
[ SmokeDetector | MS ] Manually reported answer: Generating a random alphanumeric string of length N by quartata on codegolf.SE (@Mego @quartata)
@quartata Trust me i said the same thing about some of the original chat post watchers.
!!/errorlogs 150
@quartata I presume you'd like a full debugger trace or something then
1:55 AM
Restart: API quota is 15958.
because you have a crash there too again
!!/errorlogs 25
...but it didn't happen last time?
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 97, in post_message_if_not_deleted
was_report_deleted = self.check_websocket_for_deletion(post_site_id, post_url, 300)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 35, in check_websocket_for_deletion
post_type = post_site_id[2]


IndexError: list index out of range
2017-12-07 01:55:21.592360 UTC
File "/home/ubuntu/charcoal/SmokeDetector/excepthook.py", line 54, in run_with_except_hook
another index out of range crash
oh half a mo
ok it did happen on all of these reports
ok well I'll just fix that first since I needed to do that anyways
I didn't update check_websocket_for_deletion to use get_report_data()
oh actually
it's because it wants the whole fetch_post_id_and_site_from_url
not just the site part
that's numero uno
I think there was something else with a why failing after restart?
that's probably because it wasn't pickled properly after the death from the deletionwatcher exception
hopefully that won't happen again
2:00 AM
CI on 3907b6c succeeded.
!!/pull
[ SmokeDetector ] SmokeDetector started at rev 3907b6c (quartata: Fix check_websocket_for_deletion) (running on angussidney/EC2)
Restart: API quota is 15915.
[ SmokeDetector | MS ] Manually reported answer: Generating a random alphanumeric string of length N by quartata on codegolf.SE (@Mego @quartata)
don't feedback that
!!/threads
2:01 AM
140016408282880: MainThread
140016304756480: message_sender
140015724128000: metasmoke websocket
140016034494208: message sender
140015715735296: Thread-7
140016042886912: pickle ---rick--- runner
140015707342592: delayed post
140016051279616: Thread-4
140016279054080: Thread-1
140016287708928: message_sender
140015732520704: Thread-6
140016296101632: message_sender
140016026101504: auto restart thread
140016017708800: Thread-5
140016059672320: Thread-3
140016068065024: Thread-2
> 140015707342592: delayed post
all set
that's deletion watcher doing its thing
[ SmokeDetector | MS ] Manually reported answer: Escaping double quotes in TF2 scripting by quartata on stackoverflow.com (@JakeSymons)
!!/threads
140016408282880: MainThread
140016304756480: message_sender
140015724128000: metasmoke websocket
140016034494208: message sender
140015715735296: Thread-12
140016042886912: pickle ---rick--- runner
140015707342592: delayed post
140016051279616: Thread-4
140015698949888: Thread-14
140016279054080: Thread-1
140015690557184: bodyfetcher post enqueing
140016287708928: message_sender
140015732520704: Thread-6
140016296101632: message_sender
140015598302976: bodyfetcher post enqueing
140016026101504: auto restart thread
1, 2
oatmeal
most excruciating 5 minutes of my life
!!/coffee SmokeDetector
2:04 AM
@quartata brews coffee for @SmokeDetector
maybe that'll make you hurry up
!!/threads
140016408282880: MainThread
140016304756480: message_sender
140015724128000: metasmoke websocket
140016034494208: message sender
140016042886912: pickle ---rick--- runner
140015707342592: delayed post
140016051279616: Thread-4
140015698949888: Thread-21
140016279054080: Thread-1
140016287708928: message_sender
140015732520704: Thread-6
140016296101632: message_sender
140015598302976: Thread-22
140016026101504: auto restart thread
140016059672320: Thread-3
140016068065024: Thread-2
140016017708800: delayed post
ok
we're all set
@ArtOfCode you can destroy both now, it posted the first to Tavern
actually
can you put an fp on it in the Tavern?
so that we can see if it deletes
Metasmoke post 96510 (Generating a random alphanumeric string of length N) destroyed by ArtOfCode
Metasmoke post 96511 (Generating a random alphanumeric string of length N) destroyed by ArtOfCode
Metasmoke post 96512 (Escaping double quotes in TF2 scripting) destroyed by ArtOfCode
2:07 AM
@quartata User removed from blacklist (45151 on codegolf.stackexchange.com).
@quartata User removed from blacklist (4766556 on stackoverflow.com).
@ArtOfCode hurry before the 2 minute window elapses
do sd f in tavern
hmm my first instincts tell me it's because messages posted from deletionwatcher get posted directly and the report data isn't fed back
and said instincts would be correct in fact
quick fix
although
it looks like I do it before the thread spawns
hmmm
that's still a problem either way tho
oh, and go give yourself privs in all teh places :P
^
i was going to say that
@quartata if you don't give yourself privs in each of the places in Smokey's user sets I'll do it :P
and then you'll gain -1 sanity
2:14 AM
ugh that's bad style though
I don't need it in SOBotics but I'll add myself to Meta Tavern and SOCVR
@quartata I don't even think I'm in the SOBotics privilege list there, but Tavern and SOCVR it's kinda key to have it in if we do debugging tests and stuff
Especially if it dies here and we need to kick it :P
CI on d89fc9b succeeded.
!!/pull
2:20 AM
!!/blame᠎ ​᠎​‌
Restart: API quota is 15790.
what
!!/blame᠎ ​᠎​‌‍⁠‍
@quartata It's SmokeDetector's fault.
there you go
@SmokeDetector poof-force
@ArtOfCode ready to try again?
AoC is busy hitting people with the hammer :P
2:25 AM
well I guess we'll do it first and ask questions later
[ SmokeDetector | MS ] Manually reported answer: GO OUT AND VOTE by quartata on codegolf.SE (@Mego @quartata)
shoot em up
@SmokeDetector why
@quartata Post manually reported by user quartata in room Charcoal HQ.
bueno
NO ONE FEEDBACK THAT POST
feedbacks the post
2:31 AM
sd .........
all right we're all set
@ArtOfCode @Undo can you destroy that MS record
I think we're all set.
the why thing seems to work too
[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer, repeating characters in answer: How do electrons jump orbitals? by user177852 on physics.SE
This came up this morning:
Your Linode, smokedetector1, has exceeded the notification threshold (30) for CPU Usage by averaging 31.8% for the last 2 hours. The dashboard for this specific Linode is located at: <https://manager.linode.com/linodes/dashboard/smokedetector1>

This is an automated message, please do not respond to this email. If you have questions, please open a support ticket.

You can view or change your alert thresholds under the "Settings" tab of the Linode Manager.

This is not meant as a warning or a representation that you are misusing your resources. We encourage you to modify the thresholds bas
tpu- by Rob
Has something changed that has increased server load?
2:38 AM
!!/whois codeadmins
@paper1111 I am aware of 23 code admins. Currently in this room: bwDraco, quartata, Thomas Ward, Mithrandir, NobodyNada, Henders, Magisch, Andy. Not currently in this room: Ferrybig, doppelgreener, angussidney, Byte Commander, Suraj Rao, tripleee, John Dvorak, Yvette Colomb, ProgramFOX, Ashish Ahuja, DavidPostill, Paul Stenne, ArtOfCode, Undo, Glorfindel.
!!/whois admin
@paper1111 I am aware of 5 admins. Currently in this room: Andy. Not currently in this room: ArtOfCode, Undo, angussidney, tripleee.
!!/test invite.stellar.org.claim-lumen.ga
> Would not be caught as a question, title or username.
2:40 AM
!!/watch invite\.stellar\.org\.claim-lumen\.ga
@paper1111 You don't have code privileges, but I've created PR#1344 for you.
@bwDraco OK this time I doubt that's on me. Angus's post scan rate is pretty much identical to what it normally is
Got interrupted by something urgent. Hold on...
@ArtOfCode pullapprove.yml is screwed up again
!!/watch quickbookspayrollsupport\.com
2:42 AM
@paper1111 That pattern looks like it's already caught by Potentially bad keyword in answer and Potentially bad keyword in body; append -force if you really want to do that.
@ThomasWard I'm going to failover to your instance
!!/standby angus
angussidney/EC2 is switching to standby
@SmokeDetector holy shit that actually works now
bwDraco/Linode received failover signal.
Restart: API quota is 19999.
teward/Solar Flare received failover signal.
Restart: API quota is 19999.
!!/standby Linode
2:43 AM
!!/location
@paper1111 angussidney/EC2
@paper1111 teward/Solar Flare
what
!!/standby
@quartata I hit an error while trying to run that command; run !!/errorlogs for details.
2:44 AM
oops
[ SmokeDetector ] SmokeDetector started in standby mode at rev 19fb004 (Undo1: Undo insulation commit) (running on teward/Solar Flare)
!!/alive?
@quartata ... did I miss something?
what the hell
ArtOfCode/EC2 received failover signal.
2:44 AM
what the fuck is going on
metasmoke please
!!/location
@paper1111 angussidney/EC2
Weird
teward/Lunar Eclipse received failover signal.
!!/standby angus
2:45 AM
angussidney/EC2 is switching to standby
!!/standby ArtOfCode
Does smokey have a mind of its own now?
I think Art's instance is like dead
Restart: API quota is 19999.
2:45 AM
which is not my fault
!!/alive?
@quartata Yup
!!/location
@quartata angussidney/EC2
@quartata Yup
@quartata teward/Lunar Eclipse
The instance was on standby the whole time but there were bursts to 100% at about 0830 UTC and 1500 UTC.
angus's instance won't shut up
2:46 AM
Server load is normal now; did something happen earlier?
[ SmokeDetector ] SmokeDetector started at rev 603e3b1 (ArtOfCode-: Not the usual Flake) (running on ArtOfCode/EC2)
Restart: API quota is 19999.
@bwDraco I'm pretty sure that's when you installed pyyaml
!!/standby Art
[ SmokeDetector ] SmokeDetector started in standby mode at rev 603e3b1 (ArtOfCode-: Not the usual Flake) (running on ArtOfCode/EC2)
I installed this morning and restart the instance. I don't think starting up Smokey requires that much CPU power, though.
!!/location
2:47 AM
@quartata angussidney/EC2
@quartata teward/Lunar Eclipse
hm
!!/reboot angus
[ SmokeDetector ] SmokeDetector started at rev 19fb004 (Undo1: Undo insulation commit) (running on teward/Lunar Eclipse)
what
you git
Restart: API quota is 19986.
Restart: API quota is 15630.
!!/stappit angus
2:48 AM
Goodbye, cruel world
!!/location
@quartata teward/Lunar Eclipse
@angussidney I had to kill your instance, sorry
there is something funky with standby going on rn tho
I'll fix it
!!/pull
[ SmokeDetector ] SmokeDetector started at rev d89fc9b (quartata: Be sure to set report_data after acquiring lock) (running on teward/Lunar Eclipse)
Restart: API quota is 19979.
OK I see the problem
I didn't do this quite right
2:55 AM
Hmm. It looks like my instance was terminated at around 1810 UTC. What happened?
it wasn't
ok figured it out
chatcommunicate.init is called before GlobalVars.standby_mode is initialized
oh
it failed over for like a few seconds
but it wasn't ever killed
uhm so what happened lol
ohhh
sorry it's because I did a !!/reboot
that's probably it
2:58 AM
I presume that Lunar Eclipse is working right but Solar Flare isn't?
Stop (Ctrl+C) and restart the instance in standby mode?
sure whatever
@ThomasWard Nothing major. There was a bug with standby
it was still listening for commands
Starting up.
@quartata fixed now?
2:59 AM
fixing rn
The instance is not up to date.
just need to swap the order
[ SmokeDetector ] SmokeDetector started at rev d89fc9b (quartata: Be sure to set report_data after acquiring lock) (running on teward/Lunar Eclipse)
kicks Lunar Eclipse
!!/coffee
3:00 AM
Restart: API quota is 19904.
one second one second
now why did you go and shut off
@paper1111 brews coffee for @paper1111
stop harassing it
@quartata i was fixing a kernel hangup on Lunar Eclipse
3:00 AM
wait really
was getting some hung process issues
@quartata not related to your stuff, but it was trying to take a good percent of the processor power, so I killed it and a few other unrelated things
that happened?
ok phew
just wanted to unhang the processor :P
huh Solar Flare was out of date badly o.O
Is it normal for an instance to pull only when it starts in active mode?
3:02 AM
instances only pull when told to
Ah, okay.
@quartata And if instructed to by MS/CI
with autopull and such
right
or with TCOM if i'm running smokey in a debugger
because TCOM's PyCharm master handling process has 'git pull' in it
and logic to restart the process if i do a git pull
oh woops
I did this wrong
I wish I could amend commits
3:05 AM
CI on 089d5c8 failed.
@quartata ಠ_ಠ
come on at least it caught it
I honestly wasn't expecting it to
good unit testing
CI on ecae8ac succeeded.
!!/pull
[ SmokeDetector ] SmokeDetector started at rev ecae8ac (quartata: Fix standby logic pt 2) (running on teward/Lunar Eclipse)
Restart: API quota is 19845.
3:09 AM
!!/location?
@quartata teward/Solar Flare
@quartata teward/Lunar Eclipse
!!/standby Lunar
teward/Lunar Eclipse is switching to standby
Restart: API quota is 19998.
[ SmokeDetector ] SmokeDetector started in standby mode at rev ecae8ac (quartata: Fix standby logic pt 2) (running on teward/Lunar Eclipse)
failing over to solar flare
!!/alive?
@quartata plz send teh coffee
3:10 AM
!!/pull
[ SmokeDetector ] SmokeDetector started at rev ecae8ac (quartata: Fix standby logic pt 2) (running on teward/Solar Flare)
Restart: API quota is 19992.
!!/location?
@quartata teward/Solar Flare
All set
@angussidney be sure to pull your instance when you boot it up again
3:26 AM
!!/queuestatus~?
sd kaboom-force
all quiet
on the western front
post scan rate looking fairly normal
34.6287 from 6 minutes ago
43.9889 now
Definitely normal
No performance hit here
and I think we fixed all the obvious bugs
@Andy by the way NG is live if you want to do Helios stuff
I saw that :) I'll start either tomorrow or Friday depending on how work goes in the morning.
How's it working so far? I briefly glanced at the room when you rolled it out but got pulled away so I missed a lot
it's all working so far
@ArtOfCode So...how many people can you give charcoal-se email addresses?
we fixed a bunch of stuff
!!/alive?
@quartata ... did I miss something?
3:38 AM
@quartata Nice :)
[ SmokeDetector | MS ] Bad keyword with email in answer: Do I need to say "I give you divorce" to divorce my wife? by Jessy on islam.SE
tpu by quartata
!!/threads
139624313890560: MainThread
139624194713344: message_sender
139623305029376: Thread-69
139623917401856: message sender
139623296636672: bodyfetcher post enqueing
139623925794560: pickle ---rick--- runner
139623934187264: Thread-4
139623942579968: Thread-3
139623338600192: metasmoke websocket
139624168748800: Thread-1
139624177403648: message_sender
139623909009152: auto restart thread
139623321814784: Thread-93
139624185796352: message_sender
139623950972672: Thread-2
139623313422080: Thread-92
Conflicting feedback across revisions: current, #1
Hmm
I think there might be a lock issue here
That report didn't come in until I did the !!/alive? which released the room lock
this would be sooo much easier if ChatExchange just returned the friggin' ID
honestly think we should ditch ChatExchange
I'm not sure why the lock would still be acquired
it could be from reply

« first day (1481 days earlier)      last day (2524 days later) »