[ SmokeDetector | MS ] Username similar to website in answer (70): How to initialize HashSet values by construction?‭ by Marinipaving Andmasonry‭ on stackoverflow.com

[ SmokeDetector | MS ] Toxic answer detected (80): How to download video from url and hide/secure this video from other apps and Gallery‭ by tom kraan‭ on stackoverflow.com

[ SmokeDetector | MS ] Potentially bad ip for hostname in body, potentially bad ns for domain in body, potentially bad keyword in body, potentially bad keyword in title (4): dlinkrouter.local : How do I get my Dlink to work?‭ by linneaivanov‭ on superuser.com

@SmokeDetector k

!!/watch marinipavingandmasonry\.com

tp feedback received on [MS] How to download video from url and hide/secure this video from other apps and Gallery‭

@SmokeDetector k

tp feedback received on [MS] dlinkrouter.local : How do I get my Dlink to work?‭

[ SmokeDetector ] Blacklists reloaded at rev da63c7fd85 (SmokeDetector: Auto watch of marinipavingandmasonry\.com by Jeff Schaller) (running on teward/Thoth)

[ SmokeDetector | MS ] Link at beginning of body, pattern-matching website in body, potentially bad asn for hostname in body, potentially bad ns for domain in body (138): Best Graphic Design Training institue in Jaipur?‭ by Creative Desk India‭ on meta.SE (@Ollie @Ethan)

tpu- feedback received on [MS] Best Graphic Design Training institue in Jaipur?‭

!!/watch creativedeskindia\.com

[ SmokeDetector ] Blacklists reloaded at rev eca92fa198 (SmokeDetector: Auto watch of creativedeskindia\.com by Jeff Schaller) (running on teward/Thoth)

[ SmokeDetector | MS ] Pattern-matching website in body, potentially bad asn for hostname in body, potentially bad ns for domain in body, potentially bad keyword in body, blacklisted user (169): Best Graphic Design Training institue in Jaipur? ✏️‭ by Creative Desk India‭ on meta.SE (@Ollie @Ethan)

tpu- feedback received on [MS] Best Graphic Design Training institue in Jaipur?‭

[ SmokeDetector | MS ] Bad ip for hostname in body, bad ns for domain in body, bad keyword in body, bad phone number in body, blacklisted website in body, +2 more (488): Blue Prism Training in Medavakkam‭ by jonydipp44‭ on askubuntu.com

tpu- feedback received on [MS] Blue Prism Training in Medavakkam‭

fp- feedback received on [MS] Error when i try to enter a password: Uncaught TypeError: Cannot read properties of undefined (reading 'apply') throws at exp....aura_prod.js:13:40835‭

tpu- feedback received on [MS] HashiCorp Vault 403 Permission Denied issue with Kubernetes Auth‭

fp- feedback received on [MS] MySQL WorkBench 8.0 Keeps Crashing MacOS Ventura 13.1‭

[ SmokeDetector | MS ] Potentially bad keyword in body (1): I need to create a login and autentication‭ by rabibi54‭ on stackoverflow.com

[ SmokeDetector | MS ] No whitespace in title (71): DeclareUnicodeCharacter‭ by Alois Pichler‭ on tex.SE

CI on 5bfa46e succeeded.

!!/watch spos\.pk

[ SmokeDetector ] GitHub Actions: success on 5bfa46e by tripleee in the master branch

[ SmokeDetector ] Blacklists reloaded at rev f51f18ed8c (SmokeDetector: Auto watch of spos\.pk by tripleee) (running on teward/Thoth)

tpu- feedback received on [MS] Why is SEO so important for a website in 2023‭

naa- feedback received on [MS] Why does not work ASP.NET Core Localization‭

!!/watch dwellfox\.com

[ SmokeDetector ] Blacklists reloaded at rev b78c891d0b (SmokeDetector: Auto watch of dwellfox\.com by tripleee) (running on teward/Thoth)

[ SmokeDetector | MS ] Potentially bad keyword in body (1): How to make a "scroll-system" with buttons Unity?‭ by Not Available‭ on stackoverflow.com

Serial spammer here: askubuntu.com/users/1688441/nilesh-choudhary

fp- feedback received on [MS] DeclareUnicodeCharacter‭

[ SmokeDetector ] GitHub Action workflow build (3.11): failure on 54f3237 by tripleee in the master branch

!!/watch intellimindz

@tripleee That pattern looks like it's already caught by Bad keyword in answer and Bad keyword in body Append -force to the command word(s) if you really want to add the pattern you provided.

[ SmokeDetector ] GitHub Action workflow build (3.11): failure on 3ae9e35 by tripleee in the master branch

!!/watch codelearno\.com

CI on 54f3237 failed.

fp- feedback received on [MS] How to make a "scroll-system" with buttons Unity?‭

@ArturMeinild You are not a privileged user. Please see the privileges wiki page for information on what privileges are and what is expected of privileged users.

CI on 3ae9e35 failed.

[ SmokeDetector ] GitHub Action workflow build (3.8): failure on 3ae9e35 by tripleee in the master branch

[ SmokeDetector ] GitHub Action workflow build (3.8): failure on 54f3237 by tripleee in the master branch

!!/watch codelearno\.com

[ SmokeDetector | MS ] Potentially problematic ns configuration in body (1): When you click on a day in the calendar, the list of appointments only for that day should appear as in Google Calendar ✏️‭ by Carlo Monetta‭ on stackoverflow.com

@tripleee Added codelearno\.com to watchlist However, the blacklists were not reloaded at this time. The most likely issue is another change was made on SD's master branch on GitHub and everything is waiting for CI to pass and MS to update the deploy branch. That could take a few to several minutes. If SD doesn't automatically reload the blacklists or automatically reboot after that time, then someone should investigate why that hasn't happened.

@SmokeDetector don't worry, you're welcome to apply for privileges in this room if you want to become a regular contributor

!!/allspam stackoverflow.com/users/21521574/nilesh-choudhary

[ SmokeDetector | MS ] Manually reported answer (batch report: post 1 out of 2) (94): How to fix scaling of a Java-based application on a high DPI display in Linux?‭ by Nilesh Choudhary‭ on stackoverflow.com

[ SmokeDetector | MS ] Manually reported answer (batch report: post 2 out of 2) (94): Installing python and relevant packages in a PC with no internet‭ by Nilesh Choudhary‭ on stackoverflow.com

!!/allspam askubuntu.com/users/1688441/nilesh-choudhary

[ SmokeDetector | MS ] Manually reported question (batch report: post 1 out of 2) (93): How can I remove a specific item from an array in JavaScript?‭ by Nilesh Choudhary‭ on askubuntu.com

[ SmokeDetector | MS ] Manually reported answer (batch report: post 2 out of 2) (94): Fix scaling of java-based applications for a high DPI screen‭ by Nilesh Choudhary‭ on askubuntu.com

[ SmokeDetector ] GitHub Action workflow build (3.11): failure on 0efce34 by SmokeDetector in the master branch

[ SmokeDetector ] GitHub Action workflow build (3.8): failure on 0efce34 by SmokeDetector in the master branch

[ SmokeDetector | MS ] Link at beginning of answer (36): How to dynamically render TeX equations in input text field in Flutter? ✏️‭ by Ghedo44‭ on stackoverflow.com

[ SmokeDetector | MS ] Potentially bad ns for domain in body (1): How to calculate effect size of moderation analyses on repeated-measures data?‭ by Sarah‭ on stats.SE

@metasmoke copy/paste error; fix committed

fp- feedback received on [MS] How to calculate effect size of moderation analyses on repeated-measures data?‭

@ArturMeinild thanks very much for the report!

fp feedback received on [MS] I need to create a login and autentication‭

if you would like to become a regular contributor here, please review the introduction materials linked from the room description

[ SmokeDetector | MS ] Username similar to website in answer (70): iOS VoIP Services Certificate‭ by ayanshtechnology‭ on stackoverflow.com

tpu- feedback received on [MS] iOS VoIP Services Certificate‭

[ SmokeDetector ] GitHub Actions: success on 369a505 by tripleee in the master branch

!!/watch ayanshtechnology\.com

[ SmokeDetector ] Blacklists reloaded at rev 0911c7a362 (SmokeDetector: Auto watch of ayanshtechnology\.com by tripleee) (running on teward/Thoth)

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): opencv manager package was not found?? how install automatcally?‭ by MOMO‭ on stackoverflow.com

[ SmokeDetector ] GitHub Actions: success on 3effbe2 by tripleee in the master branch

CI on 3effbe2 succeeded.

CI on 08c208f succeeded.

naa- feedback received on [MS] opencv manager package was not found?? how install automatcally?‭

[ SmokeDetector ] GitHub Actions: success on 08c208f by tripleee in the master branch

!!/watch (?-i:^Iris Caleb$)

[ SmokeDetector ] Blacklists reloaded at rev 4d14e0317d (SmokeDetector: Auto watch of (?-i:^Iris Caleb$) by tripleee) (running on teward/Thoth)

[ SmokeDetector | MS ] Potentially bad ns for domain in body (1): Как работать с асинхронностью в Python для Telegram-бота?‭ by Laminaria‭ on ru.stackoverflow.com

@tripleee Thanks ! 👍

CI on 9f2d2fb succeeded.

[ SmokeDetector ] GitHub Actions: success on 9f2d2fb by tripleee in the master branch

CI on a73e8f7 succeeded.

!!/pull

[ SmokeDetector ] GitHub Actions: success on a73e8f7 by tripleee in the master branch

[ SmokeDetector ] Blacklists reloaded at rev c9d805ee8f (tripleee: Watch NS hostiran.net.) (running on teward/Thoth)

[ SmokeDetector ] GitHub Actions: success on c9d805e by tripleee in the master branch

[ SmokeDetector | MS ] Blacklisted user (71): What are the Importance of Responsive Website‭ by mindmadetechnologies‭ on stackoverflow.com

!!/report stackoverflow.com/q/76005174

[ SmokeDetector | MS ] Manually reported question (93): What makes a good SSD network drive? I am seeking a good SSD for network storage. Any recommendation?‭ by Memory clearance‭ on stackoverflow.com

tp feedback received on [MS] What makes a good SSD network drive? I am seeking a good SSD for network storage. Any recommendation?‭

!!/watch memoryclearance\.com

[ SmokeDetector ] Blacklists reloaded at rev 281f8b0824 (SmokeDetector: Auto watch of memoryclearance\.com by Jeff Schaller) (running on teward/Thoth)

:63366379 There was a problem with this command. Was the chat message edited or deleted?

!!/test-a mindmade.in

> Would not be caught as an answer.

!!/watch mindmade\.in

^^ from profile of m.erwaysoftware.com/posts/uid/stackoverflow/76005238

[ SmokeDetector ] Blacklists reloaded at rev 7e2d8e4fdd (SmokeDetector: Auto watch of mindmade\.in by Jeff Schaller) (running on teward/Thoth)

!!/report stackoverflow.com/a/76004564

@Machavity Post 1: Already recently reported [ MS ]

tp feedback received on [MS] How to fix scaling of a Java-based application on a high DPI display in Linux?‭

tp feedback received on [MS] What are the Importance of Responsive Website‭

[ SmokeDetector | MS ] Potentially bad keyword in title (1): Replace values with 0 or blanks after drill down in Power BI‭ by ewa_07‭ on stackoverflow.com

fp- feedback received on [MS] Replace values with 0 or blanks after drill down in Power BI‭

fp- feedback received on [MS] Installing python and relevant packages in a PC with no internet‭

naa- feedback received on [MS] How to dynamically render TeX equations in input text field in Flutter?‭

CI on ed34990 succeeded.

[ SmokeDetector ] GitHub Actions: success on ed34990 by tripleee in the master branch

fp feedback received on [MS] BJT vs MOSFET switching speed‭

[ SmokeDetector | MS ] Potentially bad ip for hostname in body, potentially bad keyword in body (2): Recover From Being Scammed By Hiring A Bitcoin Recovery Expert‭ by Graeme Lesley‭ on bitcoin.SE

tpu- feedback received on [MS] Recover From Being Scammed By Hiring A Bitcoin Recovery Expert‭

[ SmokeDetector | MS ] Blacklisted user (71): Почему я не могу получить ip адреса при запуске сканера на scapy‭ by Глеб Ершов‭ on ru.stackoverflow.com

fp- feedback received on [MS] Почему я не могу получить ip адреса при запуске сканера на scapy‭

tpu- feedback received on [MS] Fix scaling of java-based applications for a high DPI screen‭

!!/blacklist-number +1 (917) 725-3296

@tripleee No such command 'blacklist-nunber'.

[ SmokeDetector ] Blacklists reloaded at rev 3b7ac6b423 (SmokeDetector: Auto blacklist of +1 (917) 725-3296 normalized numbers: {'9177253296', '19177253296'} by tripleee) (running on teward/Thoth)

Unable to launch scan thread due to exhausted general thread limit of 4 for 181.76 seconds.

[ SmokeDetector | MS ] Potentially bad asn for hostname in body (1): What types of legal resources and information can I find on GagLawyers?‭ by Manish‭ on stackoverflow.com

tpu- feedback received on [MS] What types of legal resources and information can I find on GagLawyers?‭

!!/watch gaglawyers\.com

[ SmokeDetector ] Blacklists reloaded at rev 9028ac8086 (SmokeDetector: Auto watch of gaglawyers\.com by tripleee) (running on teward/Thoth)

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Intercepting Form Submissions to Webflow via Jquery or JS ✏️‭ by DTech18‭ on stackoverflow.com

[ SmokeDetector | MS ] Potentially bad ns for domain in answer (1): Is it recommended to perform Qada' of nafl/sunna prayers?‭ by a deleted user on islam.SE

fp- feedback received on [MS] Is it recommended to perform Qada' of nafl/sunna prayers?‭

[ SmokeDetector | MS ] Link at beginning of body (41): I want to label vehicle detaset on kaggle. How can I Iabel vehicle set of 10000 images using code?‭ by Ayush Billade‭ on stackoverflow.com

fp- feedback received on [MS] I want to label vehicle detaset on kaggle. How can I Iabel vehicle set of 10000 images using code?‭

[ SmokeDetector | MS ] Bad keyword in answer, potentially bad ns for domain in answer (89): How did they film the Invisible Man being invisible in 1933?‭ by gracia‭ on scifi.SE

tpu- feedback received on [MS] How did they film the Invisible Man being invisible in 1933?‭

!!/watch gotoassignmentexpert\.com

[ SmokeDetector ] Blacklists reloaded at rev 43e2469187 (SmokeDetector: Auto watch of gotoassignmentexpert\.com by tripleee) (running on teward/Thoth)

CI on 85c8308 failed.

→ 22 messages moved to Trashcan

[ SmokeDetector ] GitHub Actions: success on 85c8308 by tripleee in the master branch

[ SmokeDetector | MS ] Potentially bad keyword in body (1): What's best cryptocurrency accepting payment gateway for 2023, alternative to coinpayments?‭ by Biz Data Serv‭ on stackoverflow.com

fp- feedback received on [MS] What's best cryptocurrency accepting payment gateway for 2023, alternative to coinpayments?‭

[ SmokeDetector | MS ] Potentially bad keyword in body (1): TypeError: _firestore.default.initializeApp is not a function‭ by Subhrajyoti Mahato‭ on stackoverflow.com

fp- feedback received on [MS] TypeError: _firestore.default.initializeApp is not a function‭

[ SmokeDetector | MS ] Link at beginning of body, potentially bad keyword in body, potentially bad keyword in title (43): Hello. www.behance.net/gallery/167850663/Learning-Mobile-AppI will be grateful for reactions and comments under my new project‭ by Oleksandra Baranchuk‭ on stackoverflow.com

teward/Thoth: Executing automatic scheduled reboot.

[ SmokeDetector ] SmokeDetector started at rev 43e2469187 (SmokeDetector: Auto watch of gotoassignmentexpert\.com by tripleee) (running on teward/Thoth, Python 3.10.6)

Restart: API quota is 73273.

[ SmokeDetector | MS ] Potentially bad ip for hostname in body (1): Dendrogram with overlapping branches produced in R (comparative phylogenetic analysis)‭ by JatNTU‭ on stackoverflow.com

[ SmokeDetector | MS ] Mostly non-latin answer (50): “ssh_exchange_identification: read: Connection reset” error‭ by Keaton L‭ on stackoverflow.com

naa- feedback received on [MS] “ssh_exchange_identification: read: Connection reset” error‭

fp- feedback received on [MS] Dendrogram with overlapping branches produced in R (comparative phylogenetic analysis)‭

[ SmokeDetector | MS ] Pattern-matching website in body, potentially bad keyword in body (96): 2009 Macbook Pro w/ broken Catalina OS won't recognize bootable OS X USB‭ by Gabriel Sandoval‭ on apple.SE

fp- feedback received on [MS] 2009 Macbook Pro w/ broken Catalina OS won't recognize bootable OS X USB‭

[ SmokeDetector | MS ] Mostly punctuation marks in title (37): Deleted! ..............hdhdhdj ✏️‭ by Alper Gül‭ on askubuntu.com

tp feedback received on [MS] Deleted! ..............hdhdhdj‭

@SmokeDetector tp I’ve rolled the vandalism back

@cocomac Registered question as true positive. If you want to blacklist the poster, use trueu or tpu.

[ SmokeDetector | MS ] Bad keyword in answer (88): How might the NIV justify putting "husband" in John 1:13?‭ by Trevor‭ on hermeneutics.SE

[ SmokeDetector | MS ] Bad keyword in link text in body (70): Подскажите апи для работы с криптовалютой‭ by Takeda‭ on ru.stackoverflow.com

[ SmokeDetector | MS ] Potentially bad asn for hostname in answer (1): Understanding XLR phantom power adaptor‭ by inductor‭ on electronics.SE

When I access my feedback page on sentinel, it tells me the page does not exist. Why?

wait, I think I am in the wrong room :)

I was about to ask whether you meant metasmoke or meant to ask SOBotics :-)

but then your metasmoke feedback page seemed fine so I was leaning toward the latter

Makyen/EC2-linux: Executing automatic scheduled reboot.

[ SmokeDetector ] SmokeDetector started in standby mode at rev bf976f407e (SmokeDetector: Auto watch of hphelp\.info by Nisse Engström) (running on Makyen/EC2-linux)

@tripleee If we don't have any issues then I want to take this as a multiphase approach rather than an "all at one time" approach. My intention was to have a worker task that runs daily(?) to check to make sure that all the domains in watched, blacklisted are in fact alive (i.e. not NXDOMAIN) and when one is NXDOMAINed just push automatically to the system direct from the Git tree, and then call a restart on SD. Assuming autopull in the git message still works that solves that.

as for commands that can do things

lets start by doing individual component functions first - ns, asn, etc. commands (ASN, etc. require GeoIP libraries though)

then start integration with other more fully fledged components. If you don't mind me trashing your work and starting it piecemeal I mean :P

(not really trashing but you get the idea)

build the lookup commands first, make sure those function, then build out more in-depth blacklisting capacity

@tripleee can you give me a rundown of the lookup commands that would need to exist first (then you can just call the functions programmatically or such internally) for your intended lookup types? we obviously have an ip search function, and I think an NS function, but any additional ones you want me to write while I'm in here? :P

oh this is a concern. @Makyen any idea why this fails on Circle but works on GH Actions? app.circleci.com/pipelines/github/Charcoal-SE/SmokeDetector/…

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, bad phone number in body, bad phone number in title, phone number detected in title (481): Cool Rupee Loan Customer Care Helpline Number={9040112470}= ✏️‭ by Raju Kumar‭ on salesforce.SE

@SmokeDetector k

[ SmokeDetector | MS ] Potentially bad ip for hostname in body (1): Qt Creator stuck repeating Android SDK cmdline-tools update via SDK Manager -- why?‭ by McKee‭ on stackoverflow.com

@ThomasWard It would be helpful to be able to kick something off with a command that would auto-fix these issues. However, before you put a lot of effort into running an auto-test every 24 hours on SD, keep in mind that we're already testing all of the NS/ASN/domain entries every time the CI testing runs, which, typically, is run many times per day. [Actually, the full test of all those is run three times every time the CI testing runs: 2x in GitHub Actions and 1x on CircleCI.] When those fail, we get notices in here and those people who are subscribed will get emails.

If a domain goes NXDOMAIN, then we get lots of notifications and continue to get notifications upon every commit until the issue is resolved. The issue is resolved by someone looking at the CI testing, figuring out what happened and then manually pushing a fix disabling the domain. This reporting and fix is likely to happen way before a once per 24 hour test is run to auto-fix the problem.

Is there additional testing that we're not doing in CI testing which should be done on the domains?

Note: we don't test all entries in the watched/blacklisted keywords or websites. To do so, we'd have to automatically or manually determine what entries on those lists actually are domains that should be tested. I'm also not sure we should auto-remove entries in those lists immediately upon a domain going NXDOMAIN, because it can still be recovered (and doesn't cost us much as just another part of the regexes).

It's unclear to me if you're meaning that you intend to test those regex entries which are actually domains.

@Makyen non-regexes won't be tested, only the stuff in the yml files that've been failing right now. It'd be too complicated to pull in everything there. We need a few commands either way though

we have NS and IP lookup commands, we need to turn those into the mechanism to blacklist/watchlist the domain, do the DNS lookup, and then add that to the existing watch/black lists

i also want to get ASN checks integrated anyways

because it might be prudent to have that available anyways (I didn't see ASN lookups present in SD now, and it's trivial for me to add GeoLite2 ASN DB integration just needs a few new things)

this might be where we decide to redesign Smokey :P

but i digress.

[ SmokeDetector | MS ] No whitespace in answer, potentially bad keyword in username (98): Submit formulario sin salir de la página‭ by fgdfhhdhdff‭ on es.stackoverflow.com

@ThomasWard Yes, I know why those tests tend to fail on CircleCI and not on GitHub Actions. It's one of many threading issues. Both CircleCI and GitHub Actions use 1 thread more than the number of CPUs in their respective compute instances. [The additional thread above CPU capability is specifically allocated to the DNS tests, as they are the longest running tests, even when highly parallelized, and the most network-bound CI tests.] GitHub Actions uses compute instances which have 2 CPUs, while CircleCI uses compute instances with 4 CPUs. Thus, there's higher parallelism on CircleCI, which is why that specific threading issue shows up on CircleCI way more often on CircleCI than on GitHub Actions.

There are a lot of things which need to have locks in order to prevent the threading problems which are seen in the CI testing (multiple types of failures, with the one you've linked just the most common), and are potential issues with SD actually running.

I've written the code to add (lots of) threading locks where needed and done some testing on it. the changes need another pass to see where I added locks that aren't really needed, primarily due to the GIL being sufficient, and needs a notable amount of testing running on a fully active test instance to make sure I didn't fubar it and introduce a deadlock and that it's not too high a performance cost.

I also made modifications to the regex package and our handling of regexes in findspam.py. At higher parallelism, there's some definite issues that show up in the CI testing. Those seem resolved with the changes I've made, but I haven't tracked down the actual issue and need to verify that the regex and findspam.py changes really resolve the issue and are not just slowing things down such that the issue is hidden.

All of that is on hold at the moment, behind needing to resolve the CPU usage issue, or at least substantially reduce it. I'd thought it looked like SE was going to pounce on fixing the underlying issue that's causing the CPU usage problem, but it's now clear they aren't, even not getting me the information I requested that would help limit the amount of work needed on our side to have a workaround.

@Makyen kinda curious if they gave any explanation for why they won't pounce on the issue

because it's clearly a "Their Fault" problem.

Because 6-8 weeks has slowly been expanding into 6-8 years.

i mean we can be glad that SD and MS are running on my infra rather than AWS or something which eats CPU credits

but still

i'm almost at the point where I have to put SD on a dedicated RPi or something

If either of you can provide a timeline on it, I'm willing to poke some bears (Philippe) about it.

@Mast you mean "This should have been fixed by SE 8 weeks ago"

@Mast tell them that they're burning up any CPU infrastructure we have to run SmokeDetector on because of their API screwups

and that i've had to go and replace a burned out CPU already becasue of it

(I technically did by having to move SD off of my common infrastructure twice already and onto dedicated hardware)

@ThomasWard I've gotten no further response. I assume that it's not causing them issues, so isn't a high priority. Our last problem was actually causing them both a lot of errors and really high loading on the servers, so probably got a higher priority. That prior one also actually broke SD, so that might have contributed to increasing the priority of people working on it.

I should probably note that the last one wasn't actually fixed by them, but was primarily resolved by us not requesting a separate piece of data (total question count) that we don't actually use, at this time, but which was a separate (very long) database request and was what was causing their servers to be so highly loaded.

possibly. but the constant emphasis expander shit is what's really breaking SD now

and i'm not kidding i bought a shitty old desktop computer and repurposed it to put SD on it now

so that I don't have to burn out my core infra

@Philippe Charcoal is literally burning through resources to keep up with the API problems. Who can we contact to get this more clearly on the agenda?

@Mast and not cheap resources, either, CPUs are expensive

I know, I've looked into it a while back whether it was sensible to start running something myself.

i mean

This is still a volunteer project supporting a major corporation.

if SE would funnel some money over my way like $100/month that'd solve some things

and I know they can afford that

because then I can get like a dual octa-core Xeon machine that won't burn out any time soon if I give 4 CPU cores to SD

Maybe supporting the project with hardware directly would work.

@Mast not against it

even if they pay for an OVH Dedi or something that'd be sufficient

something to run SD specifically in a way that won't burn out our resources.

@Mast I'm also not against that. And, frankly, they should be.

i don't disagree

@Makyen That's more ore less the realization that hit me a while back.

if SE is capable or willilng to give us the hardware to run SmokeDetector (or at the very least PAY for it if they go a dedi somewhere) that'd solve a lot of my headaches

And with the above in mind, apparently it's time to act on that realization.

for obvious reasons Metasmoke is fine where it is (lots of PII that I keep encrypted at rest when the systems are down)

Alternately, paying for the resources that are being used by volunteers to run this. (primarily Thomas Ward)

@Makyen Yea, but it's probably easier for them to go the hardware route I imagine.

@Makyen also an option, and if they can literally just give $100-$200/month that would allow for just a dedi. Instead of paying for a new CPU every few months

there's a few different ways to approach this

But, there are also resources being used by @Undo and myself, but at a lower cost level.

but unless SE is willing to work with us and fix their things that're causing issues which result in us burning through our stuff...

@Makyen true.

@Makyen The backups?

either way, though, SE has to start supporting us long term I think at some level

even if it's minimally financially

[ SmokeDetector | MS ] Potentially bad ip for hostname in answer (1): How to refactor common EJB code as a Microservice while refactoring a monolith application‭ by Angie Bonilla‭ on stackoverflow.com

Honestly it's atrocious SE is so reliant on the project still. A few years ago, I thought this was a stopgap measure. But if I see what kind of obvious spam still leaks through, their filter clearly isn't working. We're it. There's only so many conclusions you can reach from that.

tpu- feedback received on [MS] How to refactor common EJB code as a Microservice while refactoring a monolith application‭

@Mast For Undo, it's backups and the AWS infrastructure for flagging (an AWS Lambda). For me, it's the backup SD instance and the MS-Relay instance (which, aren't much, but would be expensive if it was the active SD full time).

For me, it's probably cents. My total monthly AWS bill is like $7, and the vast majority of that is elsewhere

the issue with me is the CPU spike due to the API failures on SE's part and their failures to fix it timely

that's burning the equipment through hard

For me, primarily because I'm taking advantage of AWS' offer of a free t4g.small for the year, it's only storage, which is a small amount. If the SD instance was active 100% of the time, it would be hundreds/month in CPU credits.

that's where the cost is

and that's entirely SD nowadays

@ThomasWard like, actual smoke?

[ SmokeDetector | MS ] Offensive answer detected, potentially bad keyword in answer, toxic answer detected (159): VLANs vs subnets pros and cons‭ by gr0und‭ on networkengineering.SE

@Undo one CPU died, so I had to pay $175 to replace it (older Xeon CPU) on the one hypervisor. It kept the CPUs on the 2nd hypervisor it was on at 100% for a very long time and kept triggering temp overload errors

so now it's on a dedicated old 4-core CPU desktop I bought off work for like $75

well that's not great

no, and we know why

well, we're fairly certain we know why, anyways

I'm OOTL, there's a simple connection between high CPU for us and some SE API issue?

@Makyen has specifics I think, but it's because rescanning has increased substantially in amounts, and we THINK it's because of an escaping issue that happens from the SE side of things

at least if I remember right

so we're constantly pegged

that didn't used to be the case with scanning

well, if that's the case, it's not our job to throw hardware at their problems

@ThomasWard I'll see if I can get something of a work-around today/tomorrow. I need to do some testing/investigation in the full known chain of data use (at least SD, MS, and FIRE), so time is more indeterminate. I'd hoped to get a response from SE, at least to my request for information, but I'm giving up on that.

Realistically I'm not benevolent dictator here any more, but my suggestion would be to just pause stuff until we figure it out.

@ThomasWard Can we limit CPU usage by increasing the amount of time between scans or simply allowing the queue to grow during peak hours? To smooth the curve, so to speak?

@Mast I mean, it'd require code to do that, but theoretically we could.

Or is that exactly what we're normally doing and is that breaking due to the current API issues?

it'd potentially result in us skipping things though when it's necessary

@Makyen knows more about the internals on this part than I do

i mean

Well, changing the queue depth is actually easy.

@ThomasWard Cheaper than replacing CPUs.

@Mast accurate.

in which case I should be billing SE for replacement infra

@Undo technically speaking, you still hold all the core things that really allow Charcoal to work

And if they're willing, we can talk about that. If not, I'd definitely deeply consider the limiting route.

meh, not really

@Mast not against either option

If there's a software fix, great - but IMO there's zero reason to be spending real-life money of people already volunteering time on something that I assume is trivially fixed on their side.

@Undo To keep the theme going, I'm not against that.

@Mast Changing the queue depth should notable help. I'll do that now.

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Filter Xero Invoices by Tracking Category‭ by Ethan‭ on stackoverflow.com

fp- feedback received on [MS] Filter Xero Invoices by Tracking Category‭

To be clear - I think it's great that SE relies on us, makes it more fun. It's less great that we apparently can't get any priority for anything

If they're truly reliant on us, that seems easy to point out

I know Philippe is a fan, hopefully he's got enough clout to prevent more costly issues.

If not...

(I resisted doing anything like this for the Monica stuff, because that was kinda tangential. This one is actually causing cost, directly; it's logical)

Exactly.

Someone had to pull the trigger and I'm willing to be the diplomat about it catching flak about it.

i don't have any issue with that

i also don't have any concerns pausing SD indefinitely and let SE suffer to prove a point

but i sit purely on the border between Chaotic Neutral and Chaotic Evil on alignment so :P

I have some experience with getting managers to yell at IT to get things done.

[ SmokeDetector | MS ] Mostly punctuation marks in body (28): How to make an element smoothly expand itself?‭ by havaka‭ on stackoverflow.com

[ SmokeDetector | MS ] Potentially bad asn for hostname in body (1): How to debug c ncurses program with keyboard input on Ubuntu and VS Codium‭ by kronaldo‭ on stackoverflow.com

fp- feedback received on [MS] How to make an element smoothly expand itself?‭

I mean it's less prove-a-point, I think everyone relevant at SE knows the impact here. It's more "guys, we can't compensate for this anymore"

fp- feedback received on [MS] How to debug c ncurses program with keyboard input on Ubuntu and VS Codium‭

(If that is true - I'm obviously not deep in tech details)

In hindsight, breaking the API rate-limit should've been a sign.

We can make the sign more obvious.

And yes, this has nothing to do with proving a point. If necessary, we can keep this going for a long time. But not at any cost.

Which is getting quite literal here.

Though I do want to understand the link between API issues and CPU for us, feels weird. Not that ya'll need my permission to do anything.

@Mast Change pushed. It's in CI testing.

@Undo It's not a direct link. However, it does show the project was functioning for smaller amounts of traffic and the traffic keeps going up so the limit had to be raised. Handling more traffic can't be good on your CPU regardless of the API issues. The API issues simply made it a lot worse.

If that makes sense.

[ SmokeDetector ] GitHub Actions: success on f787329 by makyen in the master branch

Charcoal hasn't been a simple hobby project for quite some time.

@Undo We limit the times SD rescans posts to when the post has actually changed. The SE API is now munging the post data, under some circumstances, so the data has actually changed, so we re-scan the posts.

Got it, that sounds right

So they're literally sending wrong stuff down the API pipe

yes.

@Makyen Oh, that makes it way worse than I made it out to be.

[ SmokeDetector ] SmokeDetector started at rev f78732980c (Makyen: Increase Stack Overflow queue depth to 2) (running on teward/Thoth, Python 3.10.6)

Restart: API quota is 71589.

I'm a hardware engineer, not a sysadmin.

I'm good at burning up electronics, not at preventing it.

The MSE post is: "SE API multiple-escapes the returned data (at least display names, userlinks, and post content)"

We're not sure this is causing our problem, but it's been seen, is reproducible and would cause what we're seeing.

Well, technically, knowing how to burn things down does tend to help with preventing it. But that's details...

At some point there's an argument for pausing simply because we can't trust what they're sending us. Not a very strong argument, because escaping issues probably aren't going to break too much in findspam, but still

@Makyen Thanks. I scanned that, but didn't read it thoroughly enough due to shortage of time at that moment.

@Undo That feels one stop short of the nuclear option, but if we haven't heard anything by next week I'd definitely put it on the table.

[ SmokeDetector | MS ] Potentially bad ip for hostname in body (1): Click anywhere on page show fullscreen div ✏️‭ by Jens Kold‭ on stackoverflow.com

@Mast It's been status-review for three weeks, and that tag was added by non-staff anyway. I'd say another week won't do any good

From a detection POV, the potential breakage is that we won't detect things that we should detect. It mostly won't affect detection, because it shouldn't be affecting the post HTML. It should be affecting the post Markdown, username, and some other fields. My request to SE was what exactly they are doing (without the bug), but I haven't gotten a response.

@Mast Magic smoke-smith?

@Undo I'm currently stepping on some toes to get the ball rolling. Please have a little patience :-)

@ThomasWard Has SD's CPU load reduced? It shouldn't be all gone, but it should be better.

!!/stats

@Makyen "uptime" stats: Posts scanned: 579, Q(208), A(371); Scan time: 847.82, Q(674.61), A(172.87); Posts per second: 0.68; Grace period edits: 199; Unchanged posts: 341, Q(244), A(97); No post lock: 33; Errors: 0; Max scan time: 58.26; Max scan time post: stackoverflow.com/76007730; Post processing lock: 0.03; Check unchanged: 0.15; Threads: 114, API(37), 155QA(63), EW(51), BFrr(0); High CPU: 0; Thread limit: 94; Site limited: 23, SO(23), nonSO(0); All errors: 0; Started: 2023-04-13T16:23:52.010920

!!/stats list

@Makyen The currently stored stats sets are named: "all", "uptime", "ms", "Makyen-2022-12-16_at_11:26", "Makyen-2023-01-27T19:09", "tmp".

@Mast ;) Don't worry, I'm not doing (and can't do) anything rash. Mostly wanting to encourage the real decision makers here to use the tools available to keep TW from burning silicon & dollars to compensate for this

!!/stats create Makyen-2023-04-13_SO_queue_depth_2

@Makyen The create operation succeeded on Makyen-2023-04-13_SO_queue_depth_2.

!!/stats Makyen-2023-01-27T19:09

@Makyen "Makyen-2023-01-27T19:09" stats: Posts scanned: 17769978, Q(5768717), A(12001261); Scan time: 21461543.76, Q(15827941.28), A(5621238.8); Posts per second: 0.83; Grace period edits: 5708708; Unchanged posts: 7165488, Q(5626478), A(1539010); No post lock: 498626; Errors: 0; Max scan time: 1433.35; Max scan time post: stackoverflow.com/75620359; Post processing lock: 628.17; Check unchanged: 4963.65; Threads: 1636106, API(866738), 155QA(904479), EW(731440), BFrr(0); High CPU: 55997; Thread limit: 1244007; Site limited: 373216, SO(373216), nonSO(0); All errors: 0; Post errors: 2; Started: 2023-01-27T19:10:39.186305

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): Importing a CSV or JSON file into age viewer Apache‭ by Waleed Ahmed Shahid‭ on stackoverflow.com

I'm being summoned, I believe?

Ah, yeah. I see the scrollback.

Makyen/MS-Relay: Executing automatic scheduled reboot.

[ SmokeDetector ] SmokeDetector started at rev 6b79b1731 (Makyen: Ping Makyen for some new metasmoke users) (running on Makyen/MS-Relay, Python 3.11.0)

OK, so: first, long time listener, infrequent caller. Huge fan of the work that you folks do here, and I sing your praises absolutely every chance that I get internally and externally. It's a model of ingenuity, and it's truly amazing.

With that said, I wasn't aware of the need for resources to support it. Probably should be self-obviious, but i never stgopped to think about it.

We're out of the budget cycle, so getting additional resources - in this economy - is not the easiest thing in the world to do. I'll start asking and figuring out where I can beg, borrow, or steal some money. In the meantime, if someone will get me a cash app address or paypal or something, I'll personally donate to pay for a couple of months of expenses.

@ThomasWard is the one with the most significant expenses ^

[ SmokeDetector | MS ] Linked punctuation in answer, potentially bad keyword in answer, blacklisted user (146): How can one reset or clear a lost MFA verification method‭ by Irfan Ali‭ on stackoverflow.com

@Philippe No problem, that's why I thought it important to share the need.

If budget is a problem, we can think in hardware instead.

tpu- feedback received on [MS] How can one reset or clear a lost MFA verification method‭

The current situation is not sustainable, but I'm sure we can find a solution both short and long term.

So I just talked to Prashanth.

[ SmokeDetector | MS ] Potentially bad ip for hostname in body (1): How can I programm an personalised survey‭ by mustafaakz‭ on stackoverflow.com

He's fine with me working with you to find some way to cover the budget. I just need to run it through legal (for liability and risk management) and through our CTO since it would probably impact his budget.

So I'm going to get the ball moving there.

That might take a bit so, seriously, get me a cash app address or something and I'll send something to cover cost for a month or two

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): Are there any fonts included with Windows 10 that only contain capital letters?‭ by YTKme‭ on superuser.com

@Philippe That's great to hear, thanks for stopping by. If there's any information you need, @ThomasWard probably has it. If need be, I'll make sure people keep communicating about what's necessary.

fp- feedback received on [MS] Are there any fonts included with Windows 10 that only contain capital letters?‭

I knew you were a fan so I'm glad we poked the right man for the job.

fp- feedback received on [MS] How can I programm an personalised survey‭

Perfect. I'll see what I can do and keep in touch. :). Yeah, I can't sing the praises of this project enough. I'll always do what I can to support reasonable asks. :)

Well, that's pretty cool

I don't always see chat pings, btw - if I don't respond, I'm not dodging you. Just send me an email (philippe@) or ask any CM to find me

Bella went looking for me this time. :D

I know, I asked about you in TL :-)

hah.

@ThomasWard So for this, I expect eventually you'll want to have some math-backed idea of what we cost you. I suppose if you're doing this as a business for other people anyway, that number wouldn't be hard to come up with?