Charcoal HQ

sd 2- 3k

!!/ms-down-force

Metasmoke status autoswitch is now disabled.

@ThomasWard Metasmoke status is now: down; Auto status switch: disabled.

!!/ms-up

7:03 PM

Metasmoke status is now set to up.

@ThomasWard Metasmoke status is now: up; Auto status switch: enabled.

still doing maintenance on my stuff by the way so MS is still likely to have glitches

while power be balanced on my network data storage is not yet

:P

:)

the UPSes are humming happily with 35min on first UPS and 20-25 min on the second UPS

which is Good

just waiting on the SSD now heh

it SHOULD have been here by now (USPS)

That sounds like a quite respectable amount of time on the UPSs and reasonable balance between the two.

!!/watch- fun211211\.com

@Makyen well everything was on one with a 10min rundown time so yeah

it's not like i've got the money or space for a $1000 UPS unit so

these 'off the shelf' units work decent for what they are

7:08 PM

@ThomasWard For tracked packages, USPS is usually quite good, although actual deliver time during the day can be quite variable.

though I pulled the wrong cable earlier and everything went dead :|

so

@Makyen yeah it's an Amazon one day ship though

i'm in no hurry yet

so i mean

@ThomasWard Yeah, at that point you're talking quite a bit of room and $.

Quite a bit of $ just for the batteries.

yeah that's also datacenter grade 240V circuits which I don't have so

:P

[ SmokeDetector | MS ] Link at beginning of answer (30): Where can I find software that was designed for Windows 1.0? by lockheed silverman on retrocomputing.SE

Dharman

!!/allspam stackoverflow.com/users/8948953/prasad-kuppa

7:10 PM

[ SmokeDetector | MS ] Manually reported answer (batch report: post 1 out of 4) (94): Sort XML elements in predetermined order using XSLT by Prasad Kuppa on stackoverflow.com

@ThomasWard Well... you can get 240V... :)

@Makyen the BIG thing was Power

[ SmokeDetector | MS ] Manually reported answer (batch report: post 2 out of 4) (94): How to remove namespace prefix leaving namespace value (XSLT)? by Prasad Kuppa on stackoverflow.com

i needed to balance those

[ SmokeDetector | MS ] Manually reported answer (batch report: post 3 out of 4) (94): How can I remove namespace prefixes in xml without removing the xmlns tag? by Prasad Kuppa on stackoverflow.com

7:10 PM

@Makyen not with this apartment

[ SmokeDetector | MS ] Manually reported answer (batch report: post 4 out of 4) (94): How to create JSON file from postman collection(s) by Prasad Kuppa on stackoverflow.com

i don't have the authority to request the wiring and installation of a 240V circuit

user435118

sdc watch myxml\.in/xml-utils\.html

@Daniil Added myxml\.in/xml-utils\.html to watchlist

and this apartment is old so the existing wiring can't handle that

7:12 PM

@ThomasWard If you're getting 120V, then you can run 240 on the same pair of wires. You won't have a neutral, so can't split that into two 120V circuits, but it's possible to do. :;

@Makyen yeah well i need the 120V for things too so

also not getting a 100lb unit for power unless i have to :P

those things expensive

@SmokeDetector Looks like something new

@ThomasWard bah.. details :)

lol

well in other news

face full of dust is BAD for allergies

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Will sleeping with a chastity device harm my sexual health? by marty on medicalsciences.SE

7:16 PM

@ThomasWard Yeah, that's not good.

[ SmokeDetector | MS ] Potentially bad keyword in body (1): How to print transformed dataset with ColumnTransformer in Keras with fit_transform? by Ace on stackoverflow.com

Is MS up now?

!!/reboot

Makyen/EC2-linux: Goodbye, cruel world

user435118

sdc location

it is for now but you don't have to reboot systems @user12986714

we have ms-down and ms-up for a reason

7:18 PM

[ SmokeDetector ] SmokeDetector started at rev bcd1aa5 (SmokeDetector: Auto watch of gonyeaso@dslextreme\.com by Makyen) (running on Makyen/EC2-linux, Python 3.6.10)

!!/location

@ThomasWard Makyen/EC2-linux bcd1aa5 (SmokeDetector: Auto watch of gonyeaso@dslextreme\.com by Makyen)

Restart: API quota is 19676.

!!/pull

26 mins ago, by SmokeDetector

Cannot initiate MS websocket. Manual !!/reboot is required once MS is up

that's because Makyen's is slow

7:19 PM

[ SmokeDetector ] SmokeDetector started at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on Makyen/EC2-linux, Python 3.6.10)

and old rev

Restart: API quota is 19673.

iBug/Coral received failover signal.

@ThomasWard The rev was current as of when I restarted it shortly after MS when down.

shrugs

Restart: API quota is 19989.

7:20 PM

Although, I did push a change and without MS being up at the time, SD won't be notified that it needs to pull a new version.

fp- feedback received on Where can I find software that was designed for Windows 1.0? [MS]

yeah but that doesn't explain why the MS Websocket part went fubar except that on reboot it tried to connect and failed

[ SmokeDetector ] SmokeDetector started at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on teward/Osiris, Python 3.6.9)

Osiris exploded only because I pulled the wrong power cable and the entire farm here took a crap

!!/standby Makyen

Makyen/EC2-linux is switching to standby

fp- feedback received on Where can I find software that was designed for Windows 1.0? [MS]

7:22 PM

but it's back now

[ SmokeDetector ] SmokeDetector started in standby mode at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on Makyen/EC2-linux)

@ThomasWard Ouch.

!!/location

@Makyen teward/Osiris 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil)

@Makyen iBug/Coral 4dc08ac (SmokeDetector: Auto watch of qatestingtrainings\.com by Makyen)

!!/pull

[ SmokeDetector ] Blacklists reloaded at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on teward/Osiris)

user435118

7:24 PM

!!/location

@Daniil No such command 'locarion'.

@Daniil teward/Osiris 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil)

[ SmokeDetector ] SmokeDetector started at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on iBug/Coral, Python 3.8.0)

Restart: API quota is 19959.

!!/standby coral

[ SmokeDetector ] SmokeDetector started in standby mode at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on iBug/Coral)

There's been no scan activity for 4 status pings. There may be a problem.

There's been no scan activity for 5 status pings. There may be a problem.

There's been no scan activity for 6 status pings. There may be a problem.

There's been no scan activity for 7 status pings. There may be a problem.

um?

!!/errorlog

### JSONDecodeError on teward/Osiris at 2020-07-02T17:54:12Z: Expecting value: line 1 column 1 (char 0)
File "/home/smokey/SmokeDetector/metasmoke.py", line 580, in get_post_bodies_from_ms
response = Metasmoke.get('/api/v2.0/posts/urls', params=payload).json()
File "/home/smokey/.local/lib/python3.6/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode

7:30 PM

... that... doesn't look good

There's been no scan activity for 8 status pings. Going into standby.

iBug/Sandy received failover signal.

[ SmokeDetector ] SmokeDetector started in standby mode at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on teward/Osiris)

Restart: API quota is 19995.

hmm

[ SmokeDetector ] SmokeDetector started at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on teward/Osiris, Python 3.6.9)

i'm live-tailing things now

!!/reboot

7:35 PM

it's weird it's like SD is not connectiong to the SE websockets...

teward/Osiris: Goodbye, cruel world

iBug/Sandy: Goodbye, cruel world

heh i had JUST restarted Osiris xD

[ SmokeDetector ] SmokeDetector started at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on teward/Osiris, Python 3.6.9)

@Makyen any chance SE is broken?

Restart: API quota is 11007.

[ SmokeDetector ] SmokeDetector started at rev ca0760e (SmokeDetector: Merge pull request #3980 from Charcoal-SE/auto-blacklist-1592031427) (running on iBug/Sandy, Python 3.8.1)

Restart: API quota is 19960.

7:37 PM

@ThomasWard Yeah, I think there might be an intermittent problem with the WebSocket, which is the likely cause of the "There's been no scan activity for 5 status pings. There may be a problem."

yeh

so it's NOT osiris' fault technically lol

oh good the mail person's outside

bet my SSD and such will be delivered shortly

@ThomasWard I consider it unlikely to be Osiris, but possible.

@ThomasWard Cool.

well i'm clean-deploying Isis shortly so

on an Ubuntu 20.04 system lol

The solution to the "There's been no scan activity for 5 status pings. There may be a problem." issue it to reboot. I'm probably just going to change the code so it reboots itself instead of just reporting.

[ SmokeDetector | MS ] Potentially bad ns for domain in body, potentially bad ns for domain in title (2): ¿como importar archivos a visual studio (VBA.net)? by David Chavero on es.stackoverflow.com

7:43 PM

!!/location

@ThomasWard teward/Osiris 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil)

@ThomasWard iBug/Sandy

!!/standby iBug

iBug/Sandy is switching to standby

[ SmokeDetector ] SmokeDetector started in standby mode at rev ca0760e (SmokeDetector: Merge pull request #3980 from Charcoal-SE/auto-blacklist-1592031427) (running on iBug/Sandy)

[ SmokeDetector | MS ] Potentially bad asn for hostname in body (1): php script using too much memory and returning fatal error by Von on stackoverflow.com

[ SmokeDetector ] SmokeDetector started in standby mode at rev ca0760e (SmokeDetector: Merge pull request #3980 from Charcoal-SE/auto-blacklist-1592031427) (running on iBug/Sandy)

[ SmokeDetector | MS ] Pattern-matching website in body (96): ImportXML XPath issue using Google Sheets on a simple web scraping query by blueantz on stackoverflow.com

[ SmokeDetector | MS ] Few unique characters in body, luncheon meat detected, no whitespace in body, repeating characters in body (304): Can I compare array[i] to 'letter' ✏️ by T. M. on stackoverflow.com

!!/ms-down-force

um...

oh right i just rebooted my firewall's host

7:58 PM

BTW: Sandy currently can't access GitHub, so it should be considered a last resort. I've pinged @iBug, but I'm assuming busy in RL.

[ SmokeDetector ] SmokeDetector started at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on teward/Osiris, Python 3.6.9)

!!/location

@Makyen teward/Osiris 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil)

Cannot initiate MS websocket. Manual !!/reboot is required once MS is up

!!/pull

[ SmokeDetector ] Blacklists reloaded at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on teward/Osiris)

8:04 PM

really?

!!/pull

[ SmokeDetector ] Blacklists reloaded at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on teward/Osiris)

oh, right the deploy branch...

!!/pull

[ SmokeDetector ] SmokeDetector started at rev 403e757 (Makyen: Change scan activity detection to auto-reboot and much quicker.) (running on teward/Osiris, Python 3.6.9)

Restart: API quota is 10806.

!!/ocation

@ThomasWard No such command 'ocation'.

8:06 PM

!!/location

@ThomasWard teward/Osiris 403e757 (Makyen: Change scan activity detection to auto-reboot and much quicker.)

Cannot initiate MS websocket. Manual !!/reboot is required once MS is up

!!/restart

teward/Osiris: Goodbye, cruel world

that's because my network took a dump ya ninny

[ SmokeDetector ] SmokeDetector started at rev 403e757 (Makyen: Change scan activity detection to auto-reboot and much quicker.) (running on teward/Osiris, Python 3.6.9)

Restart: API quota is 10794.

8:07 PM

I'm not recalling why we had the MS WebSocket not just recover itself, even if there was a long time of it not being reachable. I think there was a reason...

Cannot initiate MS websocket. Manual !!/reboot is required once MS is up

@Makyen Because the thread is dead

well MS is still starting up heh

@user12986714 I know why the code runs that way. I was talking about why we chose to have it run that way.

In getting MS post information, recovered from

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='metasmoke.erwaysoftware.com', port=443): Read timed out. (read timeout=10.0)

[ SmokeDetector | MS ] Potentially bad asn for hostname in body (1): Button on Wordpress with Elementor Builder by echelonxtr on stackoverflow.com

8:10 PM

sd fp

@user12986714 [:54840713] Registered question as false positive.

!!/ms-down-force

Metasmoke status autoswitch is now disabled.

teward/Osiris: Metasmoke status: set to down.

@ThomasWard Metasmoke status is now: down; Auto status switch: disabled.

migrating MS data over now. may take some time

what i don't understand is how the system is using 340GB of storage for this...

@ThomasWard That seems very high for this site.

8:14 PM

well the drive is thick-provisioned 250G

so we can store DB backups, etc.

the extra 100 COULD be snapshots but i'll check

i had provisioned more than enough space for data expansion, etc.

[ SmokeDetector | MS ] Potentially bad ns for domain in body, potentially bad ns for domain in title (2): CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:354) by Jhony Rentería Rodríguez on es.stackoverflow.com

Yeah, but a DB backup isn't even 500MB.

no but i'm saying so we could store MULTIPLE backups

it's probably snapshots

Yeah, of course we want to store multiple backups, but that shouldn't get us to 340GB of used space.

Although, snapshots could do it.

If you are looking through and find some snapshots, keep in mind that we could still use something with a copy of the database between the date we restored from and when we lost data, if there happens to be something around.

[ SmokeDetector | MS ] Mostly punctuation marks in body (27): How to prove that L is not a context-free language? by Amir Hossein Shekari on cs.SE

8:21 PM

i'll check before we're back up

[ SmokeDetector | MS ] Potentially problematic ns configuration in body (1): Is there any way to clear Local storage Using Server side code such as c# etc by Manish Kumar Pandit on stackoverflow.com

That would be between 2020-02-13 and 2020-06-08 (I believe; I'd want to double check when MS is up).

@Makyen but i checked this the last time you asked and didn't see any recent snapshots, only really REALLY old ones

before January

(autoshapshot died in December)

@ThomasWard Ahh... I though you might have discovered a new stash of them. :;

don't think so.

gotta check what's there after teh migration before starting it up

8:23 PM

If it's what you looked through already, then no need to bother doing it again.

it's at 89% migration now

yeah the autoshapshot died in December

so it's not around anymore

so i don't have anything in the timeframe you're after

OK. Sorry, I keep hoping. :;

Not that it really matters. I just don't like loosing data. :;

yeah nor I which is why i'm keeping daily private DB backups now :p

:) That's a good thing.

and my prototype of the processor for the sanitizer is almost done

almost because I actually have to test the sucker :P

8:26 PM

cool. Yeah, that does need to be tested. :;

yep it was snapshots

cleans old snapshots

that'll take some time with disk consolidation i think heh

@Makyen well the 'prototype' test will be the 'get it to metasmoke' part

and then having you or Art or Undo alter the codebase to simply scan the directory

and process accordingly

all the heavy SQL stuff is being done on the other system :P

disk consolidation at 2%

it's an evil process lol

That will very likely be one of them to write code to scan the directory; definitely not me. I'm still at the pattern matching stage for programming in Ruby.

heh

i think it's simple enough to just disable some of the code while letting the rest run we just have to stagger it

so what i'm thinking is I'll let my stuff run and see what happens

get a good timing on it

[ SmokeDetector | MS ] Link at end of answer (60): Roof Replacement Done Wrong? by yeashir frahim on diy.SE

and then we move forward once we have that

disk consolidation at 5%

8:32 PM

Sounds reasonable

this be slow

Yeah, this sort of thing isn't fast.

!!/location

@Makyen teward/Osiris 403e757 (Makyen: Change scan activity detection to auto-reboot and much quicker.)

7% now

but yeah it's worth the wait :P

True :)

That the % is going up is reassuring. :;

8:34 PM

9% but yeah disk consolidation is a pain as it basically compares teh delta between LastSnapshot and now. And applies the delta :p

[ SmokeDetector | MS ] Link at end of answer (60): Can any body able to see offline sqlite db data from apk using android studio in flutter? by Ramy Makram on stackoverflow.com

thereby having one singular VMDK

which should remove the extra 100GB heh

21% now.

@Makyen i'm not hard-pressing getting the processor done either, i want to make sure it works sanely first.

@SmokeDetector fp-

@SmokeDetector tpu-

@SmokeDetector fp-

i should point out the MS integration is offline right now ;)

Looks like the WS channel remains functional

8:43 PM

!!ms-status

or it's just off

because i shut off MS integration with ms-down

38% done now

@ThomasWard SD thinking MS is down shouldn't affect the MS WebSocket, but SD will only try to restore the WebSocket for a limited time (that's probably going to change shortly). When SD thinks MS is down, it will continue to try to send status pings. It will not send: post stats (i.e. reports), feedback, post deletion stats, scanning statistics, and won't try to update its list of code privileged users.

[ SmokeDetector | MS ] Bad keyword in link text in body, blacklisted website in body, potentially bad keyword in body, potentially bad keyword in title (170): do you need to make money online please follow me on this video by John Siwavula on stackoverflow.com

!!/watch- shrinkme\.io

@Makyen That pattern looks like it's already caught by Blacklisted website in answer and Blacklisted website in body; append -force if you really want to do that.

[ SmokeDetector | MS ] Pattern-matching website in answer (78): how to save full code with php on google sports? by Felix on stackoverflow.com

9:17 PM

[ SmokeDetector | MS ] Potentially bad ns for domain in answer (1): Export individual documents from mail merge by Yogeshwar Patel on stackoverflow.com

[ SmokeDetector | MS ] Link at beginning of answer (34): Computer has right arrow key constantly 'held down' by Hyder on superuser.com

[ SmokeDetector | MS ] Pattern-matching website in body (96): ImportXML XPath issue using Google Sheets on a simple web scraping query ✏️ by blueantz on stackoverflow.com

9:52 PM

!!/location

@Makyen teward/Osiris 403e757 (Makyen: Change scan activity detection to auto-reboot and much quicker.)

10:22 PM

[ SmokeDetector | MS ] Bad keyword in answer (86): Can you sub collagen protein for whey protein isolate in baking? by Sara on cooking.SE

[ SmokeDetector | MS ] Bad keyword in answer (86): Can you sub collagen protein for whey protein isolate in baking? by user85464 on cooking.SE

πάντα ῥεῖ

@SmokeDetector f

10:38 PM

!!/pull

[ SmokeDetector ] SmokeDetector started at rev d732ad0 (Makyen: As long as MS WebSocked is defined, always retry the MS WS) (running on teward/Osiris, Python 3.6.9)

Restart: API quota is 9632.

Cannot initiate MS websocket. Continuing to retry at longer intervals.

In getting MS post information, recovered from json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): High memory usage on iMac MacOS Catalina by Radim on apple.SE

!!/ms-down

teward/Osiris: Metasmoke status: set to down.

@Makyen Metasmoke status is now: down; Auto status switch: enabled.

Notice: teward/Osiris: Last 2 connection(s) to metasmoke succeeded. Setting metasmoke status to up.

11:03 PM

wow that took an eon to finish

!!/restart

teward/Osiris: Goodbye, cruel world

[ SmokeDetector ] SmokeDetector started at rev d732ad0 (Makyen: As long as MS WebSocked is defined, always retry the MS WS) (running on teward/Osiris, Python 3.6.9)

@Makyen well, let's see if this works now better

Restart: API quota is 9493.

it's moved to SSD storage

booted up a lot faster it seems

Great!

11:04 PM

and yea the extra 100GB was snapshots

it's down to a more sane 250.15GB

(the 150MB is things like config files, etc.)

@Makyen if you want to test system responsiveness you can :P

That's better, but 250GB still seems like a lot for the base config.

that was my doing

base image I deployed with for Ubuntu 18.04 was 250GB

but who's complaining xD

NOT ME

:P

Yeah, no reason to complain. It's just larger than I'd expect.

But, I'm not the one handling it, so my expectations don't, and shouldn't, matter.

@ThomasWard Feels faster. :)

@Makyen well it should be, it's still a SATA SSD but the I.O should be magnitudes faster than spinnydisk

:)

11:10 PM

though I may add a second 10GB storage disk for 'static files' such as DB backups and MS datafiles

and 'static assets'

but overall responsiveness should improve DRASTICALLY

cc @ArtOfCode @Undo about the migration to SSD

and my maintenance on it should be done now for the day

goes to find booze

@ThomasWard The important part of a project. Thank you for the new hardware and all of the work. I'm sure everyone will appreciate it.

well I mean

if someone WANTS to donate to the Hardware Fund they can... whistles nonchalantly

[ SmokeDetector ] SmokeDetector started in standby mode at rev d732ad0 (Makyen: As long as MS WebSocked is defined, always retry the MS WS) (running on Makyen/EC2-linux)

[ SmokeDetector | MS ] Potentially bad asn for hostname in answer (1): Install packages behind proxy using saltatack by RogueOverride on stackoverflow.com

tp- feedback received on Can I compare array[i] to 'letter' [MS]

11:36 PM

[ SmokeDetector | MS ] Potentially bad ip for hostname in answer (1): Changing slave address of mlx90614 with bcm2835 via SMBus / I2C ✏️ by Fux on stackoverflow.com

[ SmokeDetector | MS ] Bad keyword in body, post is mostly images (96): extract email address from String using regex by Shahryar Rafique on stackoverflow.com

[ SmokeDetector | MS ] Bad keyword in answer, blacklisted website in answer, potentially bad keyword in answer (180): Comparing observed and expected diets of two different species by Rom Ilani on stats.SE

[ SmokeDetector | MS ] Potentially bad ns for domain in body (1): I would like to share our QC workshop materials with community by Ayvenas Wulff on stats.SE

[ SmokeDetector | MS ] Bad keyword in answer, blacklisted website in answer, potentially bad keyword in answer (180): How bad is it to eat a mostly keto diet, but still eat carbs and sugars? by Rom Ilani on medicalsciences.SE

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Nginx and php authentication with m3u hls by rodrigo956 on stackoverflow.com

tpu- feedback received on Comparing observed and expected diets of two different species [MS]

[ SmokeDetector | MS ] Bad keyword in answer, blacklisted website in answer, potentially bad keyword in answer (180): What is the best diet for healthy eyesight? by Rom Ilani on medicalsciences.SE

11:52 PM

!!/watch-force- 776b4g2quo8w7ncuv6ycfufz23\.hop\.clickbank\.net

!!/watch-force- 56d6fj-d0neseueolpwdn10m2j\.hop\.clickbank\.net