Charcoal HQ

@TetsuyaYamamoto brews a cup of green tea for @TetsuyaYamamoto

Tetsuya Yamamoto

1.2x damage factor... while red tea only 0.6x.

12:45 AM

Conflicting feedback across revisions: current, #1

@ArtOfCode Was it a post on StackOverflow?

@quartata yup

If not it won't be posted there

Link?

(I know report works so it might be a configuration issue)

@quartata 1 and 2

[ SmokeDetector | MS ] Pattern-matching website in answer: HTML/CSS/Javascript: Open/Close Menu when Clicking Button by Travor Liu on stackoverflow.com (@JakeSymons)

Restart: API quota is 16365.

Tetsuya Yamamoto

12:52 AM

@SmokeDetector why

[ SmokeDetector ] SmokeDetector started in reverted mode at rev 6d504a4 (ArtOfCode: Merge pull request #1343 from NobodyNada/master) (running on angussidney/EC2)

Restart: API quota is 16363.

fp- by Tetsuya Yamamoto

@ArtOfCode ok I'll repro it when I get home

@SmokeDetector @quartata you broke it

or someone did ;)

!!/errorlogs 150

sd poof-force

2 issues going on

1 is most likely pickle Rick

The other is deletion watcher

Incidentally those are the two things I don't unit test pretty much

!!/threads

140030095042304: MainThread
140029991765760: message_sender
140029322049280: bodyfetcher post enqueing
140029982848768: message_sender
140029974456064: message_sender
140029733095168: Thread-4
140029296871168: Thread-29
140029305263872: bodyfetcher post enqueing
140029741487872: Thread-3
140029330441984: bodyfetcher post enqueing
140029724702464: pickle ---rick--- runner
140029749880576: Thread-2
140029355620096: Thread-27
140029758273280: Thread-1
140029347227392: Thread-6
140029338834688: metasmoke websocket

1:02 AM

Meta Tavern hasn't noticed yet

I give it about 3 hours

!!/master

[ SmokeDetector ] SmokeDetector started at rev d59572d (SmokeDetector: Auto watch of quickbookspayrollsupport\.com by quartata --autopull) (running on angussidney/EC2)

Restart: API quota is 16254.

1:36 AM

[ SmokeDetector | MS ] Few unique characters in body, no whitespace in body: Cannot be resolved to a type by John Smith on stackoverflow.com (@JakeSymons)

tp- by Tetsuya Yamamoto

@quartata I think this is already included on some instances - Solar Flare already has pyyaml, as does Lunar Eclipse, maybe a side effect of certain packages installed by python3-all on Ubuntu systems?

Mine has it

interesting

OK so SOCVR is getting reports

I'm not sure why manual reports wouldn't work then

@ArtOfCode stand by to delete an MS record

@ArtOfCode do you have SOCVR privileges?

1:51 AM

i have all teh privs

@ArtOfCode k, can you report stackoverflow.com/a/38362924

in there

@quartata IIRC every Smokey maintainer has privs in every channel for Smokey... myself included.

Restart: API quota is 15976.

!!/errorlogs 50

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 35, in check_websocket_for_deletion
post_type = post_site_id[2]

IndexError: list index out of range
2017-12-07 00:51:33.495669 UTC
File "/home/ubuntu/charcoal/SmokeDetector/excepthook.py", line 54, in run_with_except_hook
run_old(*args, **kw)

File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 97, in post_message_if_not_deleted

1:53 AM

i think it asploded?

yes but those logs are old

I need a tail

those ones arent

IndexError: list index out of range
2017-12-07 01:52:50.342687 UTC
File "/home/ubuntu/charcoal/SmokeDetector/excepthook.py", line 54, in run_with_except_hook
run_old(*args, **kw)

File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 97, in post_message_if_not_deleted
was_report_deleted = self.check_websocket_for_deletion(post_site_id, post_url, 300)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 35, in check_websocket_for_deletion

!!/errorlogs 150

for i, char in enumerate(reversed(x)):

TypeError: descriptor 'add' requires a 'set' object but received a 'tuple'
2017-12-06 21:40:55.066115 UTC
File "/home/ubuntu/charcoal/SmokeDetector/chatcommunicate.py", line 316, in f
result = func(*processed_args, **({"alias_used": alias_used} if give_name else {}))

File "/home/ubuntu/charcoal/SmokeDetector/chatcommands.py", line 333, in watch
return do_blacklist(website, "watch_keyword", msg, force=False)

File "/home/ubuntu/charcoal/SmokeDetector/chatcommands.py", line 299, in do_blacklist

^ latest error log

@quartata you've got an IndexError problem

1:54 AM

that wouldn't cause the restart

that's the last crash

that's a deletion watcher error

!!/location

@quartata angussidney/EC2

remind me how many times we've said "that can't be it" and it was again

if it's an uncaught or unhandled exception it could trigger a restart

@ArtOfCode fifty billion

of which 20 billion were my saying it

1:55 AM

It literally triggers on every post.

Watch

!!/report codegolf.stackexchange.com/a/119166

okay, so fix that and you'll probably fix the manual report too

[ SmokeDetector | MS ] Manually reported answer: Generating a random alphanumeric string of length N by quartata on codegolf.SE (@Mego @quartata)

@quartata Trust me i said the same thing about some of the original chat post watchers.

!!/errorlogs 150

@quartata I presume you'd like a full debugger trace or something then

1:55 AM

Restart: API quota is 15958.

because you have a crash there too again

!!/errorlogs 25

...but it didn't happen last time?

File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 97, in post_message_if_not_deleted
was_report_deleted = self.check_websocket_for_deletion(post_site_id, post_url, 300)

File "/home/ubuntu/charcoal/SmokeDetector/deletionwatcher.py", line 35, in check_websocket_for_deletion
post_type = post_site_id[2]

IndexError: list index out of range
2017-12-07 01:55:21.592360 UTC
File "/home/ubuntu/charcoal/SmokeDetector/excepthook.py", line 54, in run_with_except_hook

another index out of range crash

oh half a mo

ok it did happen on all of these reports

ok well I'll just fix that first since I needed to do that anyways

I didn't update check_websocket_for_deletion to use get_report_data()

oh actually

it's because it wants the whole fetch_post_id_and_site_from_url

not just the site part

github.com/Charcoal-SE/SmokeDetector/commit/…

that's numero uno

I think there was something else with a why failing after restart?

that's probably because it wasn't pickled properly after the death from the deletionwatcher exception

hopefully that won't happen again

2:00 AM

CI on 3907b6c succeeded.

!!/pull

[ SmokeDetector ] SmokeDetector started at rev 3907b6c (quartata: Fix check_websocket_for_deletion) (running on angussidney/EC2)

Restart: API quota is 15915.

!!/report codegolf.stackexchange.com/a/119166

[ SmokeDetector | MS ] Manually reported answer: Generating a random alphanumeric string of length N by quartata on codegolf.SE (@Mego @quartata)

don't feedback that

!!/threads

2:01 AM

140016408282880: MainThread
140016304756480: message_sender
140015724128000: metasmoke websocket
140016034494208: message sender
140015715735296: Thread-7
140016042886912: pickle ---rick--- runner
140015707342592: delayed post
140016051279616: Thread-4
140016279054080: Thread-1
140016287708928: message_sender
140015732520704: Thread-6
140016296101632: message_sender
140016026101504: auto restart thread
140016017708800: Thread-5
140016059672320: Thread-3
140016068065024: Thread-2

> 140015707342592: delayed post

all set

that's deletion watcher doing its thing

[ SmokeDetector | MS ] Manually reported answer: Escaping double quotes in TF2 scripting by quartata on stackoverflow.com (@JakeSymons)

!!/threads

140016408282880: MainThread
140016304756480: message_sender
140015724128000: metasmoke websocket
140016034494208: message sender
140015715735296: Thread-12
140016042886912: pickle ---rick--- runner
140015707342592: delayed post
140016051279616: Thread-4
140015698949888: Thread-14
140016279054080: Thread-1
140015690557184: bodyfetcher post enqueing
140016287708928: message_sender
140015732520704: Thread-6
140016296101632: message_sender
140015598302976: bodyfetcher post enqueing
140016026101504: auto restart thread

1, 2

~~oatmeal~~

most excruciating 5 minutes of my life

!!/coffee SmokeDetector

2:04 AM

@quartata brews coffee for @SmokeDetector

maybe that'll make you hurry up

!!/threads

140016408282880: MainThread
140016304756480: message_sender
140015724128000: metasmoke websocket
140016034494208: message sender
140016042886912: pickle ---rick--- runner
140015707342592: delayed post
140016051279616: Thread-4
140015698949888: Thread-21
140016279054080: Thread-1
140016287708928: message_sender
140015732520704: Thread-6
140016296101632: message_sender
140015598302976: Thread-22
140016026101504: auto restart thread
140016059672320: Thread-3
140016068065024: Thread-2
140016017708800: delayed post

ok

we're all set

@ArtOfCode you can destroy both now, it posted the first to Tavern

actually

can you put an fp on it in the Tavern?

so that we can see if it deletes

Metasmoke post 96510 (Generating a random alphanumeric string of length N) destroyed by ArtOfCode

Metasmoke post 96511 (Generating a random alphanumeric string of length N) destroyed by ArtOfCode

Metasmoke post 96512 (Escaping double quotes in TF2 scripting) destroyed by ArtOfCode

!!/rmblu codegolf.stackexchange.com/users/45151

2:07 AM

@quartata User removed from blacklist (45151 on codegolf.stackexchange.com).

!!/rmblu stackoverflow.com/users/4766556

@quartata User removed from blacklist (4766556 on stackoverflow.com).

@ArtOfCode hurry before the 2 minute window elapses

do sd f in tavern

hmm my first instincts tell me it's because messages posted from deletionwatcher get posted directly and the report data isn't fed back

and said instincts would be correct in fact

quick fix

although

it looks like I do it before the thread spawns

hmmm

that's still a problem either way tho

oh, and go give yourself privs in all teh places :P

^

i was going to say that

@quartata if you don't give yourself privs in each of the places in Smokey's user sets I'll do it :P

and then you'll gain -1 sanity

2:14 AM

ugh that's bad style though

I don't need it in SOBotics but I'll add myself to Meta Tavern and SOCVR

@quartata I don't even think I'm in the SOBotics privilege list there, but Tavern and SOCVR it's kinda key to have it in if we do debugging tests and stuff

Especially if it dies here and we need to kick it :P

github.com/Charcoal-SE/SmokeDetector/commit/…

CI on d89fc9b succeeded.

!!/pull

[ SmokeDetector ] SmokeDetector started at rev d89fc9b (quartata: Be sure to set report_data after acquiring lock) (running on angussidney/EC2)

2:20 AM

!!/blame᠎ ᠎‌

Restart: API quota is 15790.

what

!!/blame᠎ ᠎‌‍⁠‍

@quartata It's SmokeDetector's fault.

there you go

@SmokeDetector poof-force

@ArtOfCode ready to try again?

AoC is busy hitting people with the hammer :P

2:25 AM

well I guess we'll do it first and ask questions later

!!/report codegolf.stackexchange.com/a/99009

[ SmokeDetector | MS ] Manually reported answer: GO OUT AND VOTE by quartata on codegolf.SE (@Mego @quartata)

shoot em up

@SmokeDetector why

@quartata Post manually reported by user quartata in room Charcoal HQ.

bueno

NO ONE FEEDBACK THAT POST

feedbacks the post

2:31 AM

sd .........

all right we're all set

@ArtOfCode @Undo can you destroy that MS record

I think we're all set.

the why thing seems to work too

[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer, repeating characters in answer: How do electrons jump orbitals? by user177852 on physics.SE

This came up this morning:

Your Linode, smokedetector1, has exceeded the notification threshold (30) for CPU Usage by averaging 31.8% for the last 2 hours. The dashboard for this specific Linode is located at: <https://manager.linode.com/linodes/dashboard/smokedetector1>

This is an automated message, please do not respond to this email. If you have questions, please open a support ticket.

You can view or change your alert thresholds under the "Settings" tab of the Linode Manager.

This is not meant as a warning or a representation that you are misusing your resources. We encourage you to modify the thresholds bas…

tpu- by Rob

Has something changed that has increased server load?

2:38 AM

!!/whois codeadmins

@paper1111 I am aware of 23 code admins. Currently in this room: bwDraco, quartata, Thomas Ward, Mithrandir, NobodyNada, Henders, Magisch, Andy. Not currently in this room: Ferrybig, doppelgreener, angussidney, Byte Commander, Suraj Rao, tripleee, John Dvorak, Yvette Colomb, ProgramFOX, Ashish Ahuja, DavidPostill, Paul Stenne, ArtOfCode, Undo, Glorfindel.

!!/whois admin

@paper1111 I am aware of 5 admins. Currently in this room: Andy. Not currently in this room: ArtOfCode, Undo, angussidney, tripleee.

!!/test invite.stellar.org.claim-lumen.ga

> Would not be caught as a question, title or username.

2:40 AM

!!/watch invite\.stellar\.org\.claim-lumen\.ga

@paper1111 You don't have code privileges, but I've created PR#1344 for you.

@bwDraco OK this time I doubt that's on me. Angus's post scan rate is pretty much identical to what it normally is

Got interrupted by something urgent. Hold on...

@ArtOfCode pullapprove.yml is screwed up again

!!/watch quickbookspayrollsupport\.com

2:42 AM

@paper1111 That pattern looks like it's already caught by Potentially bad keyword in answer and Potentially bad keyword in body; append -force if you really want to do that.

@ThomasWard I'm going to failover to your instance

!!/standby angus

angussidney/EC2 is switching to standby

[ SmokeDetector ] SmokeDetector started in standby mode at rev d89fc9b (quartata: Be sure to set report_data after acquiring lock) (running on angussidney/EC2)

@SmokeDetector holy shit that actually works now

bwDraco/Linode received failover signal.

Restart: API quota is 19999.

teward/Solar Flare received failover signal.

Restart: API quota is 19999.

!!/standby Linode

2:43 AM

!!/location

@paper1111 angussidney/EC2

@paper1111 teward/Solar Flare

[ SmokeDetector ] SmokeDetector started in standby mode at rev 5ed63bb (Undo1: Merge pull request #1342 from Glorfindel83/master --autopull) (running on bwDraco/Linode)

what

Loll

!!/standby

@quartata I hit an error while trying to run that command; run !!/errorlogs for details.

2:44 AM

oops

[ SmokeDetector ] SmokeDetector started in standby mode at rev 19fb004 (Undo1: Undo insulation commit) (running on teward/Solar Flare)

!!/alive?

@quartata ... did I miss something?

what the hell

ArtOfCode/EC2 received failover signal.

2:44 AM

what the fuck is going on

metasmoke please

!!/location

@paper1111 angussidney/EC2

Weird

teward/Lunar Eclipse received failover signal.

!!/standby angus

2:45 AM

angussidney/EC2 is switching to standby

!!/standby ArtOfCode

[ SmokeDetector ] SmokeDetector started in standby mode at rev d89fc9b (quartata: Be sure to set report_data after acquiring lock) (running on angussidney/EC2)

Catija

Does smokey have a mind of its own now?

I think Art's instance is like dead

Restart: API quota is 19999.

2:45 AM

which is not my fault

!!/alive?

@quartata Yup

!!/location

@quartata angussidney/EC2

@quartata Yup

@quartata teward/Lunar Eclipse

The instance was on standby the whole time but there were bursts to 100% at about 0830 UTC and 1500 UTC.

angus's instance won't shut up

2:46 AM

Server load is normal now; did something happen earlier?

[ SmokeDetector ] SmokeDetector started at rev 603e3b1 (ArtOfCode-: Not the usual Flake) (running on ArtOfCode/EC2)

Restart: API quota is 19999.

@bwDraco I'm pretty sure that's when you installed pyyaml

!!/standby Art

[ SmokeDetector ] SmokeDetector started in standby mode at rev 603e3b1 (ArtOfCode-: Not the usual Flake) (running on ArtOfCode/EC2)

I installed this morning and restart the instance. I don't think starting up Smokey requires that much CPU power, though.

!!/location

2:47 AM

@quartata angussidney/EC2

@quartata teward/Lunar Eclipse

hm

!!/reboot angus

[ SmokeDetector ] SmokeDetector started at rev 19fb004 (Undo1: Undo insulation commit) (running on teward/Lunar Eclipse)

what

you git

Restart: API quota is 19986.

Restart: API quota is 15630.

!!/stappit angus

2:48 AM

Goodbye, cruel world

!!/location

@quartata teward/Lunar Eclipse

@angussidney I had to kill your instance, sorry

there is something funky with standby going on rn tho

I'll fix it

!!/pull

[ SmokeDetector ] SmokeDetector started at rev d89fc9b (quartata: Be sure to set report_data after acquiring lock) (running on teward/Lunar Eclipse)

Restart: API quota is 19979.

OK I see the problem

I didn't do this quite right

2:55 AM

Hmm. It looks like my instance was terminated at around 1810 UTC. What happened?

it wasn't

ok figured it out

chatcommunicate.init is called before GlobalVars.standby_mode is initialized

oh

it failed over for like a few seconds

but it wasn't ever killed

uhm so what happened lol

ohhh

sorry it's because I did a !!/reboot

that's probably it

2:58 AM

I presume that Lunar Eclipse is working right but Solar Flare isn't?

Stop (Ctrl+C) and restart the instance in standby mode?

sure whatever

@ThomasWard Nothing major. There was a bug with standby

it was still listening for commands

Starting up.

[ SmokeDetector ] SmokeDetector started in standby mode at rev 5ed63bb (Undo1: Merge pull request #1342 from Glorfindel83/master --autopull) (running on bwDraco/Linode)

@quartata fixed now?

2:59 AM

fixing rn

The instance is not up to date.

just need to swap the order

[ SmokeDetector ] SmokeDetector started at rev d89fc9b (quartata: Be sure to set report_data after acquiring lock) (running on teward/Lunar Eclipse)

kicks Lunar Eclipse

!!/coffee

3:00 AM

Restart: API quota is 19904.

one second one second

now why did you go and shut off

@paper1111 brews coffee for @paper1111

stop harassing it

@quartata i was fixing a kernel hangup on Lunar Eclipse

3:00 AM

wait really

was getting some hung process issues

@quartata not related to your stuff, but it was trying to take a good percent of the processor power, so I killed it and a few other unrelated things

that happened?

ok phew

just wanted to unhang the processor :P

huh Solar Flare was out of date badly o.O

[ SmokeDetector ] SmokeDetector started in standby mode at rev d89fc9b (quartata: Be sure to set report_data after acquiring lock) (running on teward/Solar Flare)

Is it normal for an instance to pull only when it starts in active mode?

3:02 AM

instances only pull when told to

Ah, okay.

@quartata And if instructed to by MS/CI

with autopull and such

right

or with TCOM if i'm running smokey in a debugger

because TCOM's PyCharm master handling process has 'git pull' in it

and logic to restart the process if i do a git pull

oh woops

I did this wrong

I wish I could amend commits

3:05 AM

CI on 089d5c8 failed.

@quartata ಠ_ಠ

come on at least it caught it

I honestly wasn't expecting it to

good unit testing

CI on ecae8ac succeeded.

!!/pull

[ SmokeDetector ] SmokeDetector started at rev ecae8ac (quartata: Fix standby logic pt 2) (running on teward/Lunar Eclipse)

Restart: API quota is 19845.

3:09 AM

!!/location?

@quartata teward/Solar Flare

@quartata teward/Lunar Eclipse

!!/standby Lunar

teward/Lunar Eclipse is switching to standby

Restart: API quota is 19998.

[ SmokeDetector ] SmokeDetector started in standby mode at rev ecae8ac (quartata: Fix standby logic pt 2) (running on teward/Lunar Eclipse)

failing over to solar flare

!!/alive?

@quartata plz send teh coffee

3:10 AM

!!/pull

[ SmokeDetector ] SmokeDetector started at rev ecae8ac (quartata: Fix standby logic pt 2) (running on teward/Solar Flare)

Restart: API quota is 19992.

!!/location?

@quartata teward/Solar Flare

All set

@angussidney be sure to pull your instance when you boot it up again

3:26 AM

!!/queuestatus~?

sd kaboom-force

all quiet

~~on the western front~~

post scan rate looking fairly normal

34.6287 from 6 minutes ago

43.9889 now

Definitely normal

No performance hit here

and I think we fixed all the obvious bugs

@Andy by the way NG is live if you want to do Helios stuff

Andy

I saw that :) I'll start either tomorrow or Friday depending on how work goes in the morning.

How's it working so far? I briefly glanced at the room when you rolled it out but got pulled away so I missed a lot

it's all working so far

Andy

@ArtOfCode So...how many people can you give charcoal-se email addresses?

we fixed a bunch of stuff

!!/alive?

@quartata ... did I miss something?

Andy

3:38 AM

@quartata Nice :)

[ SmokeDetector | MS ] Bad keyword with email in answer: Do I need to say "I give you divorce" to divorce my wife? by Jessy on islam.SE

tpu by quartata

!!/threads

139624313890560: MainThread
139624194713344: message_sender
139623305029376: Thread-69
139623917401856: message sender
139623296636672: bodyfetcher post enqueing
139623925794560: pickle ---rick--- runner
139623934187264: Thread-4
139623942579968: Thread-3
139623338600192: metasmoke websocket
139624168748800: Thread-1
139624177403648: message_sender
139623909009152: auto restart thread
139623321814784: Thread-93
139624185796352: message_sender
139623950972672: Thread-2
139623313422080: Thread-92

tpu by quartata on target= _blank is Follow or NoFollow [MS]

tpu by quartata on What's cold and hard and doesn't bend? [MS]

Conflicting feedback across revisions: current, #1

tpu by quartata on What's cold and hard and doesn't bend? [MS]

fp by quartata on Dynamic URL rewriting using codeigniter/php/htaccess when change in url [MS]