@Makyen That pattern looks like it's already caught by Pattern-matching email in answer and Pattern-matching email in body; append -force if you really want to do that.
$ dig @8.8.8.8 +trace metasmoke.erwaysoftware.com
; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> @8.8.8.8 +trace metasmoke.erwaysoftware.com
; (1 server found)
;; global options: +cmd
...
. 86981 IN NS b.root-servers.net.
...
;; Received 525 bytes from 8.8.8.8#53(8.8.8.8) in 27 ms
com. 172800 IN NS b.gtld-servers.net.
;; Received 1215 bytes from 199.9.14.201#53(b.root-servers.net) in 20 ms
erwaysoftware.com. 172800 IN NS ns-cloud-e1.googledomains.com.
erwaysoftware.com. 172800 IN NS ns-cloud-e2.googledomains.com.
@Undo yep looks good with the trace :)
I should be able to get the TXT records up, i'll do that test tomorrow
@Undo if we need to do the same for ms you may want to just CNAME it to metasmoke.erwaysoftware.com if it lets you, because that'll get any IP updates on the delegated DNS
[ SmokeDetector | MS ] Bad ip for hostname in body, bad keyword in body, bad keyword with email in body, bad phone number in body, blacklisted website in body, +1 more (453): custom pins for sale by customenamelpin on mathoverflow.net
@tripleee From a high level POV, there were two problems that hit at the same time.
The first problem was the tld package SD uses relies on this Mozilla URL, which used to provide a list of TLDs. Mozilla's strategic plan was to move custodianship of the list to publicsuffix.org. The tld package has not yet been updated with the new URL.
Mozilla decided to let the URL die early yesterday (UTC). As SD was written at the time, the death of that URL caused SD to crash during boot prior to any ability to command the instance to pull a new version of code from GitHub. So, all SD instances were stuck in a reboot-crash-reboot loop, which no SD instance could recover from without intervention via the instance's console (i.e. the instance owner had to manually pull new code from GitHub). So, all SD instances were down hard.
Note: SD automatically reverts to the next older commit on GitHub if there are multiple passes of a reboot-crash-reboot loop. Unfortunately, the code that caused the crash was added about 3 years ago, so each SD instance would have had to go through on the order of 200k loops prior to getting to something which wouldn't have that code. With each loop taking on the order of 40 seconds, that's on the order of 93 days to maybe auto-recover.
[We may want to consider having a mechanism where in addition to reverting, SD also tries commits which are newer than the one on which it originally had problems. Overall, it's likely that we're going to be working on fixing the problem and that new commits will be where the real fix will be found.]
The currently in use interim solution for the tld package problem is to catch and not re-raise the exception, along with catching an additional exception elsewhere in the code, which results from not being able to perform the task that's broken. The full resolution will be for the tld package to update, or for us to switch to using a clone the tld repo, so the new URL is used. Either solution will require SD runners to manually install the updated/alternate package.
The code for the interim solution shouldn't need to be changed or removed once the tld package is updated.
The 2nd problem was that the SSL certificate from Let's Encrypt for MS expired without being renewed. This also happened early yesterday (i.e. 0:00UTC). This resulted in anything which required verification of the certificate (e.g. SD, browsers, etc.) refusing to connect to metasmoke. If this had been the only problem, then all SD instances would have degraded into their normal MS-down state.
The intermediate solution was to push code which disabled SD verifying the SSL certificate for both HTTPS and WSS communication (done, now reverted in favor of renewing the certificate). The mid-term solution was to renew the SSL certificate and install it (done). The longer term solution (currently in-process) is to adapt the DNS configuration such that the certificate can be automatically renewed.
Though... @Makyen, I don't know what you do for a living (privacy by design, I know), but I'd love to be the employer/customer you're writing code / documentation for.
$ dig @8.8.8.8 ANY knox.metasmoke.erwaysoftware.com
;; QUESTION SECTION:
;knox.metasmoke.erwaysoftware.com. IN ANY
;; ANSWER SECTION:
knox.metasmoke.erwaysoftware.com. 21599 IN CNAME d1mqu4nm867bat.cloudfront.net.
@Undo That's very true. Yeah, the exposed email address on GitHub was something I wasn't happy about when first setting up GitHub, but I had a gmail.com address which could be used, specifically for the privacy. The domain registration is a significant hole. There's not much which can be done about that other than the various whois privacy offerings, which, of course, cost money.
@Makyen That pattern looks like it's already caught by Pattern-matching email in answer and Pattern-matching email in body; append -force if you really want to do that.
@Undo thanks for working with me on the DNS delegation, i know it's a bit busy / crazy BUT this will in the future make things better and more automated with automated SSL cert generation
would this be a good time to talk (again) about moving Metasmoke away from Undo's domain? We used to have metasmoke.charcoal-se.org which would make more sense and be easier to guess if you don't remember exactly
@tripleee true. We can do the DNS delegation wherever, or we can reg a new domain. Hell, I'm running the thing, payday's in the next 12 hours and I could buy a new domain for this
@tripleee That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; append -force if you really want to do that.
@Undo @tripleee @Makyen the question though does sit - do we want to use a different domain. if we do then we can create a domain and I can do that myself specific for metasmoke. But any hardcoded URLs we are using would break :P
so i mean, that's more your guys decision, I'll execute whatever changes we need :)
(if I buy a new domain for this it's ending up on Cloudflare because that is snap-simple to integrate with my existing LE configs to get it working with DNS verification)
Well, the headaches of changing the domain are A) changing the places where it currently exists and are reasonably editable, and B) the old URLs that will be left hanging (e.g. MS links in all the old SD reports). We'd need to make a choice as to do we want to leave the old domain active for some period of time, or cut-over.
@Undo There was supposed to be a paragraph in my response to this saying that I, also, would be happy to work with anyone here. Somehow, that got dropped from what actually ended up being sent as chat messages.
halflife:2020-04-20 05:46:08,875:https://on.mktw.net/2VjqXm9 redirects to https://www.marketwatch.com/press-release/primal-grow-pro-reviews-grow-up-your-size-in-just-weeks-2020-04-13
halflife:2020-04-20 05:46:12,917:http://tiny.cc/j0qcnz redirects to https://www.marketwatch.com/press-release/granite-male-enhancement-ultimate-testosterone-booster-for-men-2020-04-11
@Makyen That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; append -force if you really want to do that.
I completely missed that there was a watch in that set of messages, then the grep I did for it failed, because I still had the deploy branch checked out, which I don't keep automatically updated to match GitHub in my dev fork.
@Makyen That pattern looks like it's already caught by Pattern-matching email in answer and Pattern-matching email in body; append -force if you really want to do that.
@Makyen That pattern looks like it's already caught by Bad ip for hostname in body, Pattern-matching website in body, Bad ip for hostname in answer, and Pattern-matching website in answer; append -force if you really want to do that.
