Restart: API quota is 19974.

second time in a week that the SE websocket went down. I'll fix DeletionWatcher to catch those exceptions (and also not spawn during standby, I already did that locally)

!!/location

@WELZ teward/Supernova

@WELZ ArtOfCode/EC2

!!/standby teward

teward/Supernova is switching to standby

while I'm doing that @Art what's your opinion on angry meta posts

Depends...is it a constructive angry or a "you all suck" angry?

[ SmokeDetector | MS ] Potentially bad keyword in username: Lucene cartesian product diferent documents? by John on stackoverflow.com

I'm not Art, but IMO depends on the exact post.

I'm also not Art

fp- by PeterJ

But I can draw a stick figure, so that's got to count for something

twas a joke

Just an FYI, @Zoe

there's issues with the SE websockets

Restart: API quota is 19960.

what angry meta post have we got now

so instances may be 'rebooting' because of that

just want someone to mention that the WS has problems

@ArtOfCode not an angry meta post, but an angry me right now

websockets are working for my instance

@quartata staff is poking other staff

The SE Websockets are having problems. SmokeDetectors may not run properly, or may show as 'offline' when they actually aren't.

fp by Nisse Engström

@ArtOfCode then your instance is the only one not having those issues. All other instances are having WS Already Closed errors/exceptions.

and I already poked staff on it, to prod the other staff who work on it

503s across the board on all my instances now that Zoe rebooted one.

pat's Art's SmokeDetector on the head Good bot

Ah, there we go, I got one

BAD BOT!

two

@Mithrandir good enough. be sure to mention the "second time" part because Thomas doesn't give forth chances

I take it back. You are a bad bot.

@ThomasWard oh, oki

@Art can I see the full stack trace? I want to know which socket

I suspect the instances were rebooting because DeletionWatcher does not catch these

whereas the other one does

you would be correct

would you prefer to wait an hour until I can fix it or would you like me to tell you how to fix it now

[14:09:10] websocket._exceptions.WebSocketConnectionClosedException: Connection is already closed.
2018-04-24 14:09:10.991655 UTC
  File "ws.py", line 76, in <module>
    GlobalVars.deletion_watcher = DeletionWatcher()

  File "/home/ubuntu/ws/SmokeDetector/deletionwatcher.py", line 27, in __init__
    self.socket = websocket.create_connection("wss://qa.sockets.stackexchange.com/")

  File "/home/ubuntu/.local/lib/python3.5/site-packages/websocket/_core.py", line 487, in create_connection
    websock.connect(url, **options)

@quartata if there's an easyish fix, I'll do it

[ SmokeDetector | MS ] Potentially bad keyword in body: For Looping doesn't change variables in Katalon Studio by Todor Srbinovski on stackoverflow.com

first open up ws.py and move the DeletionWatcher() line down so it comes after the part where it goes if "standby" in ...

then open up deletionwatcher.py and wrap the __init__ in a try. just pass on exception for now (maybe log it), I'll make it retry later

[ SmokeDetector | MS ] Offensive answer detected, toxic answer detected: Unable to format pen drive by Ashish Kamble on askubuntu.com

@quartata roger, pending server rebooting

tpu- by ByteCommander

@ArtOfCode that was fast

fp- by ByteCommander on For Looping doesn't change variables in Katalon Studio [MS]

it's not done yet, I need the server to reboot before I can do it

oh, right

dont commit on deploy tho

@quartata do you mean within the if GlobalVars.standby_mode clause, or after it?

oh there, I see it

after

should be top level

it's just that I was dumb and put it with the bodyfetcher init, which means it triggers before standby is checked. move it after to fix

Done, I suspect this is still not going to work, because the failure on create_connection will mean self.socket isn't a valid Socket object when we call it elsewhere in DeletionWatcher

CI on 2a0232e succeeded.

Yup. I'm running in live mode, not standby, so this is a problem. ws.py is also failing to create a connection.

[ SmokeDetector | MS ] Bad keyword in link text in body: Remove restrict to start in android lollipop by Andriod Ateam on stackoverflow.com

fp- by Zoe

oh, yes, right. forgot a step

in subscribe wrap the part that does a send in a try, but not the rest. we want it to save the action anyways in case it restarts and successfuly starts listening

that should do it, the rest are called from the thread which wont spawn

although im not currently looking at the code

post the contents of def subscribe hre before you commit @Art

NobodyNada/Raspberry Pi received failover signal.

ssshhh my child

the following may or may not work

Restart: API quota is 19278.

the following did not work

[14:34:56] DeletionWatcher failed to create a websocket connection
[14:34:56] DeletionWatcher failed on sending 1-question-50003699
[14:34:56] AttributeError: 'DeletionWatcher' object has no attribute 'socket'
2018-04-24 14:34:56.975121 UTC
  File "/home/ubuntu/ws/SmokeDetector/excepthook.py", line 54, in run_with_except_hook
    run_old(*args, **kw)

  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)

  File "/home/ubuntu/ws/SmokeDetector/deletionwatcher.py", line 45, in _start

@quartata ^

Restart: API quota is 19554.

!!/location

gimme te code, the whole thing

@Zoe NobodyNada/Raspberry Pi

@Zoe teward/Lunar Eclipse

@quartata all on GH

this piece of garbage Im on cant handle TLS420 or whatever github is using now, one second

def subscribe(self, post_url, callback=None, pickle=True, timeout=None):
        post_id, post_site, post_type = fetch_post_id_and_site_from_url(post_url)

        if post_site not in GlobalVars.site_id_dict:
            log("warning", "unknown site {} when subscribing to {}".format(post_site, post_url))
            return

        if post_type == "answer":
            question_id = datahandling.get_post_site_id_link((post_id, post_site, post_type))

            if question_id is None:
                return

Ok thats right. this error was from _start which shouldnt be getting called. can I see init

[ SmokeDetector | MS ] Potentially bad keyword in username: Lucene cartesian product diferent documents? by John on stackoverflow.com

fp- by PeterJ

fp- by PeterJ

def __init__(self):
        DeletionWatcher.update_site_id_list()

        try:
            self.socket = websocket.create_connection("wss://qa.sockets.stackexchange.com/")
        except:
            log('error', 'DeletionWatcher failed to create a websocket connection')

        self.posts = {}

        if os.path.exists("deletionIDs.p"):
            with open("deletionIDs.p", "rb") as fh:
                for post in DeletionWatcher._check_batch(pickle.load(fh)):
                    self.subscribe(post, pickle=False)

oh no you gotta wrap the entire method in the try

!!/location

@PeterJ teward/Lunar Eclipse

@PeterJ NobodyNada/Raspberry Pi

sorry

@SmokeDetector fp

@Zoe Registered question as false positive.

@Zoe Registered question as false positive.

[ SmokeDetector | MS ] Potentially bad keyword in answer: twrp app bricked my huawei y3 II phone and it refuses to be flashed again...any ideas? by mina nageh on android.SE

[ SmokeDetector | MS ] Potentially bad keyword in answer: twrp app bricked my huawei y3 II phone and it refuses to be flashed again...any ideas? by mina nageh on android.SE

we dont want it to go any further if it cant spawn the WS. you could also return there @Art

yeah just stick a return in the except, thats better

tpu- by Zoe

tpu- by Zoe

!!/standby lunar

teward/Lunar Eclipse is switching to standby

CI on 5070c7c failed.

[ SmokeDetector | MS ] Blacklisted user: My phone doesn't boot at all after restarting bootloader by mina nageh on android.SE

[ SmokeDetector | MS ] Potentially bad keyword in answer, blacklisted user: My phone doesn't boot at all after restarting bootloader by mina nageh on android.SE

Restart: API quota is 19869.

Restart: API quota is 19204.

@SmokeDetector k

CI on 3b56dcc failed.

CI on 3b56dcc failed.

!!/location

CI on 3b56dcc failed.

@ArtOfCode teward/Lunar Eclipse

@ArtOfCode ArtOfCode/EC2

fp by Zoe on My phone doesn't boot at all after restarting bootloader [MS]

@ArtOfCode NobodyNada/Raspberry Pi

what

fp by Zoe on My phone doesn't boot at all after restarting bootloader [MS]

!!/standby teward

teward/Lunar Eclipse is switching to standby

!!/standby Nobody

NobodyNada/Raspberry Pi is switching to standby

fp by Zoe on My phone doesn't boot at all after restarting bootloader [MS]

@ArtOfCode standby-except is quicker :>

why is travis failing but not circle

[ SmokeDetector ] SmokeDetector started in standby mode at rev aeceb9d (SmokeDetector: Auto watch of airroofventilator by Glorfindel --autopull) (running on NobodyNada/Raspberry Pi)

CI on 60eb5b5 succeeded.

!!/pull

what was going wrong there

flake

Restart: API quota is 19787.

oh, ok

I threw some reconnect logic into ws.py too, which is useful already:

[14:48:13] DeletionWatcher failed to create a websocket connection
[14:48:21] ArtOfCode/EC2
[14:48:21] metasmoke.erwaysoftware.com
[14:48:21] WS failed to create websocket connection. Attempt 1 of 5.
[14:48:21] WS failed to create websocket connection. Attempt 2 of 5.
[14:48:21] WS failed to create websocket connection. Attempt 3 of 5.
[14:48:21] WS failed to create websocket connection. Attempt 4 of 5.
[14:48:21] WS failed to create websocket connection. Attempt 5 of 5.
[14:48:21] Max retries exceeded. Exiting, maybe a restart will kick things.

@ArtOfCode what's interesting is brand new instances of Smokey fail to cojnnect still

after 10 separate runs

even on one system that didn't have a Smokey before

so I have to conclude something's up with the websockets

looking good

i trust you to put out any fires in non-me code

[ SmokeDetector ] SmokeDetector started in standby mode at rev 60eb5b5 (ArtOfCode: Hi, Flake, it's been a while) (running on teward/Solar Flare)

hmm

!!/brownie

@quartata Brown!

hello flake8 my old friend

Dandruff back?

@ArtOfCode i can confirm 503s still are happening by the way

@ThomasWard Yup, likewise for me, but my last few commits mean we don't crash on them

indeed.

question: do we actually need the hybrid intermediate IDs logged?

fp- by doppelgreener on My phone doesn't boot at all after restarting bootloader [MS]

@quartata Brown!

I'm tempted to remove them

!!/location

!!/location

Restart: API quota is 19186.

@ArtOfCode ArtOfCode/EC2

@doppelgreener ArtOfCode/EC2

oh good

fp- by doppelgreener on My phone doesn't boot at all after restarting bootloader [MS]

that seems to have been happening a lot recently

double instances of smokey

@ArtOfCode teward/Lunar Eclipse

!!/standby teward

@Art theyre helpful for me. maybe put them on a lower log level

Restart: API quota is 19127.

lol

@quartata they're literally as low as they can go

what do you use them for?

@ArtOfCode aside from the fact that I generally would like to know if Smokey is slogging through 100 posts, they also show if a post on a site was missed via the WS

[ SmokeDetector | MS ] Blacklisted user: My phone doesn't boot at all after restarting bootloader by mina nageh on android.SE

fp- by PeterJ

[ SmokeDetector | MS ] Blacklisted user: twrp app bricked my huawei y3 II phone and it refuses to be flashed again...any ideas? by mina nageh on android.SE

fp- by PeterJ

@SmokeDetector fp-

Restart: API quota is 19474.

[ SmokeDetector | MS ] Blacklisted user: twrp app bricked my huawei y3 II phone and it refuses to be flashed again...any ideas? by mina nageh on android.SE

!!/location

@Zoe ArtOfCode/EC2

@Zoe teward/Lunar Eclipse

!!/standby teward

teward/Lunar Eclipse is switching to standby

@ThomasWard Lunar keeps booting back out of standby

@ArtOfCode probably because you have something in there that reboots the bot regularly

did you make sure it keeps the command line arguments?

or that it's exiting on the mode which doesn't restart it out of standby in nocrash?

you are the last one who added code to change how WS is handled with reboots

:p

@ArtOfCode the reboots were caused by your 'Restart" code you put in

so either you're exiting horribly wrongly

or it's your fault.

(just checked on the console)

I added an os._exit(4), which I believe is the exit code for a socket failure, but which shouldn't run in standby mode

[ SmokeDetector ] SmokeDetector started in standby mode at rev 60eb5b5 (ArtOfCode: Hi, Flake, it's been a while) (running on teward/Lunar Eclipse)

give me a minute

grabs the flamethrower and the hammer

Restart: API quota is 19435.

Bah, no, 4 is the crash code

We really need to document these

!!/location

@Zoe ArtOfCode/EC2

@ArtOfCode well, exit code 4 is the crash code

@ArtOfCode ... they are documented if people actually read the code :p

I'm happy to add some of those, but reboot code is 5

code isn't documentation

it can be used instead if there is no documentation, but it itself is not documentation

i'm happy to make an exit_codes.txt while I have the code up, unless you want me to put it as a Charcoal team internal question?

in which case I'll have to find a Linux box that isn't broken or in the middle of a full disk copy

(i'm stuck on my Windows work computer right now)

@ThomasWard exifjs is a javascript library (weakly typed) that gives me documentation for using one function in one way and very little else about what i should expect from it when. Its documentation then says "Please refer to the source code for more advanced usages such as getting image data from a File/Blob object (EXIF.readFromBinaryFile)."

I can tell you for sure, the code does not count as actually documenting how to use it in that situation...

CI on ace0fbe succeeded.

@ThomasWard I've got it, dw

(did you know it requires several specific conditions to be met when called the normal documented way, which aren't themselves documented, and if they aren't met it will fail silently and return false? wasn't documented either, including how it expected me to handle that, so I had to figure out what they expected me to do to handle the error case and hope it was correct.)

@ArtOfCode the only problem is the persistent 'standby' argument isn't kept if it's kicked back into 'active' mode

[ SmokeDetector | MS ] Offensive answer detected, potentially bad keyword in answer, toxic answer detected: The female equivalent of "don't break my balls" by SeneJerry on ell.SE

it probably should but it's been a while since I messed with nocrash.py

New metasmoke user 'Sam' created

@ArtOfCode we could use exit code 10 which adds a 5 second wait for socket failures. I think that was written for chat socket or web socket timeouts, but I don't exactly remember...

been an eon since we had any WS problems :P

!!/pull

[ SmokeDetector ] SmokeDetector started in standby mode at rev ace0fbe (ArtOfCode: Correct exit code) (running on teward/Lunar Eclipse)

[ SmokeDetector ] SmokeDetector started at rev ace0fbe (ArtOfCode: Correct exit code) (running on ArtOfCode/EC2)

[ SmokeDetector ] SmokeDetector started in standby mode at rev ace0fbe (ArtOfCode: Correct exit code) (running on teward/Solar Flare)

now that's interesting...

@ArtOfCode DeletionWatcher is still trying to make a websocket connection even in standby mode

that's probably not a good thing

Restart: API quota is 19355.

@ThomasWard yup. Blame @quartata

[ SmokeDetector ] SmokeDetector started at rev ace0fbe (ArtOfCode: Correct exit code) (running on The Call of Midnight)

throws @quartata into /dev/null

and yes I know TCOM is up

was testing the WS issue

it's off now :P

[ SmokeDetector ] SmokeDetector started at rev ace0fbe (ArtOfCode: Correct exit code) (running on ArtOfCode/EC2)

Restart: API quota is 19316.

[ charcoal-se.github.io ] continuous-integration/travis-ci/push success on 5c30010: The Travis CI build passed

Restart: API quota is 19229.

fp by Makyen on Integrate javascript plugin SkylinkJS in Angular/typescript [MS]

[ SmokeDetector | MS ] Luncheon meat detected: Am I really this dumb? What's the difference between this? by user9692152 on stackoverflow.com

@SmokeDetector fp-

[ SmokeDetector | MS ] Bad keyword in username: H2 database connection pool issue by Binfer on stackoverflow.com

sd why

@Zoe [:44191245] Username - Position 1-7: Binfer

@SmokeDetector fp

@Zoe Registered question as false positive.

[ SmokeDetector | MS ] Blacklisted website in answer, repeated URL at end of long post: polynomial goal programming? by user555396 on mathoverflow.net

tpu- by Zoe

@ThomasWard I thought Art already fixed that

I fixed it locally and forgot to commit, which is why I told Art to fix it

oh, yeah, Art didn't do it right

!!/pull

@quartata CI build is still pending, wait until the build has finished and then pull again.

zzz

CI on 0735f6c succeeded.

!!/pull

zzzzzzz

whatever'

Restart: API quota is 18780.

finally

!!/listening

@quartata I hit an error while trying to run that command; run !!/errorlogs for details.

!!/errorlogs 50

ok

sd poof

that's an easy fix

[ SmokeDetector ] SmokeDetector started at rev b566d0f (quartata: move .posts init up --autopull) (running on ArtOfCode/EC2)

Restart: API quota is 18756.

there

fp by Nisse Engström on The female equivalent of "don't break my balls" [MS]

fp by Zoe on twrp app bricked my huawei y3 II phone and it refuses to be flashed again...any ideas? [MS]

fp by Zoe on twrp app bricked my huawei y3 II phone and it refuses to be flashed again...any ideas? [MS]

fp by Zoe on What is the difference between T, E & P series Intel processor [MS]

fp by Zoe on What are the use cases for Apache Spark vs Hadoop [MS]

CI on ecc9965 failed.

CI on 26a8e13 failed.

tpu by Nisse Engström on twrp app bricked my huawei y3 II phone and it refuses to be flashed again...any ideas? [MS]

tpu by Nisse Engström on My phone doesn't boot at all after restarting bootloader [MS]

use tuples

not lists github.com/Charcoal-SE/SmokeDetector/commit/…

every time you use a list instead of a tuple a server kitty dies

congratulations you contradicted yourself

CI on 125cb37 failed.

but it doesn't matter

@ArtOfCode i haven't had any coffee, you expect me to make sense

so unfair

[ SmokeDetector | MS ] Potentially bad keyword in answer: mock-up prototyping tool by Grace on stackoverflow.com

[ SmokeDetector | MS ] Potentially bad keyword in answer: mock-up prototyping tool by Cersei Zou on stackoverflow.com

tpu- by quartata

tpu- by quartata on mock-up prototyping tool [MS]

er wait

one of those is from 2015

hmmmm

yeah

The new answer triggered SD detection of the second one

hm

need a second opinion

Looks like spam to me; attempted promotion of the software at least

All though the question is tool rec

fp- by ArtOfCode on mock-up prototyping tool [MS]

fp by Zoe on mock-up prototyping tool [MS]

tpu by ArtOfCode on mock-up prototyping tool [MS]

the new answer definitively looks like spam though, the one from 2015 doesn't look as spamish

Bah, how do I remove my feedback completely?

you don't, but you can overwrite it

or you can invalidate it

CI on 5fbb8bc succeeded.

@ArtOfCode Close enough. How?

Invalidating does mark that as an incorrect feedback

visit the post in MS, and hit the clear link

There. Thanks

@SmokeDetector The question explicitly asks for tool recommendations. As such, these answers are not spam. The definitions SE uses for spam require that it must be unsolicited.

fp- by Makyen

fp- by Makyen on mock-up prototyping tool [MS]

@SmokeDetector fp-

fp by ArtOfCode on mock-up prototyping tool [MS]

@SmokeDetector fp-

[ SmokeDetector ] SmokeDetector started at rev b566d0f (quartata: move .posts init up --autopull) (running on ArtOfCode/EC2)

Restart: API quota is 18427.

that's a lot of errors

Restart: API quota is 19652.

[ SmokeDetector ] SmokeDetector started at rev 5fbb8bc (ArtOfCode: Bugs everywhere) (running on ArtOfCode/EC2)

Restart: API quota is 18399.

[ SmokeDetector | MS ] Bad keyword in title, link following arrow in body, phone number detected in title, potentially bad keyword in body: AOL Gold Desktop Support Number +1-844-443-3244 by David Clarke on photo.SE

tpu- by John Dvorak

[ SmokeDetector | MS ] Potentially bad keyword in body: Do you remember the mushroom that used to live next to me? by Cardio Rabbit on superuser.com

[ SmokeDetector | MS ] Bad keyword in title, link following arrow in body, phone number detected in title, potentially bad keyword in body: AOL Gold Desktop Support Number +1-844-443-3244 by David Clarke on photo.SE

!!/watch-force (1[\W_]*)?844[\W_]*443[\W_]*3244[A-Za-z_]*

tpu- by John Dvorak

@Makyen That pattern looks like it's already caught by Mostly non-latin body and Mostly non-latin answer; append -force if you really want to do that.

@Makyen You don't have code privileges, but I've created PR#2089 for you.

[ SmokeDetector | MS ] Potentially bad keyword in body: Do you remember the mushroom that used to live next to me? by Cardio Rabbit on superuser.com

@Makyen That pattern looks like it's already caught by Mostly non-latin body and Mostly non-latin answer; append -force if you really want to do that.

!!/standby-except Art

@Makyen You don't have code privileges, but I've created PR#2090 for you.

[ SmokeDetector ] SmokeDetector started in standby mode at rev f8e3576 (SmokeDetector: Auto watch of outloei\.com by tripleee --autopull) (running on teward/Supernova)

!!/lick

@quartata licks ice cream cone

@ArtOfCode what was this science

tpu- by Makyen