« first day (3943 days earlier)      last day (171 days later) » 

16:00
@tripleee No, I'm sure it wasn't. This had to be a manual review, especially when they went shutting down forks. Very concerning.
fwiw i have a full git copy of what's in smokedetector/cerberus right now on my own git hosting so.
It's odd there's no warnings from GH
yay for private repos xD
@Machavity their moderation doesn't give warnings. Zero warnings, unless you successfully appeal the reason for the suspend/ban
yay for open source in general
@Machavity If there were warnings, they likely would have been sent to the owners of the main repo, not the forks, so we may not have seen them.
@ThomasWard Argh, what? Wow.
16:02
@CodyGray welcome to Microsoft.
they did the same to multiple MS365 tenants without warning or ways to contact them too
@CodyGray ^ Yeah, they didn't send anything
yeah they don't do warnings
BUT i'm glad to see it's only certain repos than the entire 'Company" on GH
I like how the PayPal reciepts show up there, it kinda shows what kind of relationship this is
@CodyGray Yup. The good news is that I'm pretty much completely rehabilitated back to QWERTY now. One aspect that I think is underrated is that [a-gx] (to borrow regex notation) are accessible with one hand. That's fantastic for typing hexadecimal numbers and music notation software.
how funny would it be if it's a copyright strike from SE, for some 'good' examples we have in tests for CI
16:05
@Undo i think that'd come across as a DMCA takedown notice warning instead
because DMCA rules require a much more specific takedown notice and not an "unspecified" termination of things
i've seen those requests so :P
... also made those requests too
@Machavity That seems completely opposite of what one would expect. Microsoft has a much larger infrastructure and way more experience running one. It should not make them lower their tolerances for high usage.
@Undo But surely a human being reviewed this, and they were not fooled by having a few bad words in a file of regexes...
@GeneralGrievance Even more fantastic that your hex numbers include g :)
Cow
Cow
16:07
967 unread
@CodyGray idk, have you seen humans lately?
Cow
Cow
that's a first
I'm putting $1 on "the SmokeDetector repository was a false positive in a spam detection system," just for the irony.
(I don't have any insider info here.)
@Slate Yeah, that's my guess
That's where my money's at.
16:08
I would like to somehow blame GenAI
Striking everything with "genuine-nike-shoes-delhi.com" is the kind of thing that appears safe at first glance to a new PM
@ThomasWard Those are paid services, right?
M365 is, yeah
Though the lower tiers are basically paid Gmail
@CodyGray yup. Including my own tenant for "suspicious mail volume" which locked the entire tenant and their notifications never got to me
so I had to Raise Hell
@Undo Aside from those in my house, not really.
16:10
after six hours on the phone and three circle-jerks around the call chain, I ended up demanding to speak to a supervisor who handled things for me. Got me unlocked AND they made notations on the account
@CodyGray So you don't have proof they still exist
haven't had a problem since
now in MY case
I pay Microsoft Top Money for an E5 license so they prefer to actually help me than hurt me.
@ThomasWard I mean, yeah. That's absurd. People who are benefitting from free services tend to expect to get screwed, but to not even warn people who are paying you for a service is just terrible business. Wow.
@Undo No, it could all be AI out there for all I know. That would explain a lot, wouldn't it?
@CodyGray Yeah, well they did the same for my workplace too. Except my workplace works very closely with US and Foreign law enforcement, so we pulled a few strings to make the workplace reactivation of mail done in less than an hour.
@CodyGray Sure would
16:12
Beep boop.
when the FBI and such say "We really rely on this partner please fix their email already" it kind of carries some massively heavy weight xD
you pay Microsoft? That's your first mistake right there
No, the mistake seems to be continuing to subscribe to the service after getting colossally and inexplicably screwed.
@tripleee Microsoft paid me for a while and I think even that was a mistake
@tripleee welcome to companies on this planet
16:12
Microsoft loves me xD
muahahahahahahaah
@ThomasWard I'm in DC right now, do you want me to nip down and say something
[ SmokeDetector | MS ] Manually reported question (93): How many people have lived since the Sassanid Empire‭ by Artem‭ on stackoverflow.com
@iBug Well, the g is for the music notation, since I do a lot of that too :). I guess the point I was going for is that it's surprisingly useful to have that much of the first part of the alphabet available without having to put both hands on the home row.
@Mithical lol. you could but you wouldn't get far. I have direct contacts I can raise hell with :p
IN THE INTERIM THOUGH
16:13
@Mithical DC, huh? You're missing all the fun back home!! :-)
if this isn't solved soon then I'll work with @Makyen and others to get the code base updated to use alternative git hosting
even if it's on my git temporarily xD
@CodyGray Almost as if that's part of the point. ;)
@Mithical Yeah, probably not an accident that you picked the one time that the US is less crazy to come visit. Hope you're enjoying your stay!
we have a fair bit invested in Github's API etc so I'm really hoping we can get this resolved
16:15
Been good so far, hoping it stays that way :)
@ThomasWard I mean, we could just move to GitLab, but... ^^ what tripleee said.
@tripleee agreed. either way i have Gitea as a backup for my own copy in case GH decides to tell us to go screw ourselves
@CodyGray I think GitLab would strike us for similar reasons
but yeah we have a lot of git integrations with GH and such
but regardless, I have a full git copy ready to deploy if needed :P
The most convincing theory I've heard so far is a "load on infrastructure" one. Not that we're anywhere close to extreme, but we do have a relatively high load in terms of commits, pushes, etc. If they wanted us to back down on that, I'm sure there are ways we could work on it.
@ThomasWard You don't think they're any more sane? That's disappointing.
surely that would manifest as something else than having the repo disabled for ToS violations
I'd imagine the much more prudent way to address this would be to reach out privately, or just directly rate-limit us. (pseudo-ninja'd)
16:17
That would be the more prudent way to address any concern that they might have, wouldn't it?
@CodyGray nah, the way we interact shouldn't even be a blip
we aren't telling the world to clone stuff all the time
I actually still think it is possible that I was manually reported and the rest was all automated
sdc watch- theeditorshub\.com
@VLAZ That pattern looks like it's already caught by Potentially bad keyword in answer and Potentially bad keyword in body Append -force to the command word(s) if you really want to add the pattern you provided.
@Undo I'm not sure I agree with that. The SD repo is far more active than the average repository on GH, don't you think?
16:18
@CodyGray not if there was an underlying complaint that led to the strike
@CodyGray Than the average, but not in a way that causes load
@CodyGray points at ComfyUI and Automatic1111 AI repositories Nope, those're way WAY more active than Smokey in terms of clones, etc.
Reading/writing from the tip are trivial O(1) ops for GH
The places that have been backed off tend to have "telling the public to clone the whole thing, as part of a devops process" in common
@VLAZ probably refrain from trying to watch anything until we get our repo back (see surrounding discussion)
I just noticed...
ugh i'mma have to carve $350 out for what i need to upgrade.. RIP my wallet.
"Blocking or disabling the offending content" keyword blocking is implied by this?
16:21
What's crazy if this is an automated thing is that this is a 400+ star repo. Not a crazy number, but they should be able to derive a reputation score from who's starring repos and apply some sanity based on that
@micsthepick I'm guessing that's just standard boilerplate anyway
How large was the repo, in terms of file size?
< 50 MiB
when I created a detached clone on Github the other day, uploading it took on the order of 1 hour
@iBug Sorry this was a tree:0 clone
16:25
y'all forgetting the commit history, etc. Uploading the complete copy of all branches, etc. on teward/Cerberus SmokeDetector equated to just over 1.3GiB
I wonder what it looks like in windirstat or in wiztree
"detached" as in not a fork, full history
The biggest folder I have in an old-ish fork of mine is the github history folder (.git).
Approaching 1GB, but that’s because GitHub doesn't have a way to compact the database, which is, normally, considered routine maintenance, so storage is quite inefficient.
> We recommend repositories remain small, ideally less than 1 GB, and less than 5 GB is strongly recommended.
16:26
single branch size though is tiny
OK, I figured it would be large, but ~1 GB is totally fine, per the TOS.
it's only when you include artifacts, etc. in the commit history, compressed diff chains, etc.
that is where it gets large. the actual single-commit slice size is tiny
Yeah, I'm just perusing the TOS trying to find something that they might hang their hat on.
Look in community guidelines
Better save your stamina for something more productive than speculating
16:27
Which is linked from the TOS IIRC
If the git database is compacted through normal git commands, it goes down to about 200 MB.
Yes, I've read all of that. Even linked to a section from the Community Guidelines above.
Just glancing through their Community forum, I don't see any posts about other repos being disabled, so it doesn't look like this could be a widespread issue (like the Internet Archive had a few weeks ago with randomly removing people's uploads and accounts), it's localized to Smokey.
@iBug In general, I would probably agree, but it's good to be prepared.
@Mithical Devil's advocate: if it just hit, like it did for us, there might not have been time for stuff to show up on the forum.
It's been over an hour already, plenty of time for people to post.
Cow
Cow
16:30
I see Downdetector has a big spike also
funny
Have we ever received a DMCA takedown notice for any content related to the Charcoal project?
nope, I don't think so
@Mithical link for curiosity? News to me.
Something on MS is the only thing I could think of
16:30
there was one "reformed" spammer who wanted us to remove their domain from the repo
Cow
Cow
Comment from Downdetector
@micsthepick I don't have a link, that's from personal communication and glancing through Twitter search results at the time
@Spevacus But that MS content isn't hosted on GH, is it? I'm not too familiar with the MS architecture, but I would assume that the posts in its database are not part of the MS GH repo. (And obviously I can't check now because... well, you know.)
16:32
Correct.
The MS repo is still up.
Ah, interesting. OK.
@CodyGray metasmoke is hosted on Thomas' infrastructure
the metasmoke Rails code is on Github and still available
but the contents are separate
Right, that's what I thought.
There is notable load from SD. Each watch or blacklist is a commit, which launches CI testing.
16:34
github.com/Charcoal-SE/metasmoke in case you want to check (-:
So hear me out: someone flags a fork for vulgar language e.g. and the mods decide to press the big red button, what would we expect to happen?
But the thing is, the SD repo isn't really filled with vulgar language.
There are some expletives hidden in regex patterns, but that's not really the kind of thing that I would expect a human moderator who reviews any case to hit a button over.
Right now it might not be
$ fgrep -sc fuck * | fgrep -v :0
bad_keywords.txt:1
blacklisted_usernames.txt:2
findspam.py:5
@micsthepick What I'd want to happen is an investigation into this first. Which will probably show that the vulgar language is not there just to be vulgar. But what would actually happen is another matter.
16:36
My fork would be very old
I do love my personal instance of GitLab for things like this. Alas, it doesn't have anything charcoal related in it :(
@tripleee So, way lower than the Linux kernel. :-D
Cow
Cow
In any case, if bad language is the reason, it could be fixed in various ways
my money is still on one or two of the inappropriate URLs in the repo
perhaps as a precaution we would really want to obfuscate those going forward
Cow
Cow
encrypt/hash it
dunno
16:43
that's probably overkill, just making it more regexy is probably enough
Cow
Cow
hehe
Oh, I was not aware that we had that. Yeah... That would be definitely something to suspect.
How old is the keywords?
Cow
Cow
regex is encryption
I don't remember off the top of my head but the longer ones are probably more likely to be trouble
tangentially @gevcyrrr would be a cool sock puppet name (-:
1665664961 is October 2022
16:45
Seriously didn’t need to post them here.
even with rot13?
rot26 is even more secure, I hear.
@tripleee wasn’t aware of this, might be prudent for us to drop those completely
teward/Cerberus: Executing automatic scheduled reboot.
@tripleee Just the fact that they're in the repo is enough info, no need to reproduce them in any form here
16:47
CP is so radioactive that it might be counterproductive to try to prevent it
fair, thanks for removing
Restart: API quota is 92393.
I mean, yeah. I'm not sure we lose any generality by just watching any .onion link. There's no real need for any of those links to be posted on SE.
Cow
Cow
@CodyGray I agree
16:47
we have an entire site for discussing those so that's probably not going to fly
They’re .onion links?
Cow
Cow
yes
well these ones are
The ones that tripleee posted were all .onion links, yes.
@CodyGray maybe tor.SE could result in some problems there
16:48
I'm in agreement. When we had an abusive poster that posted that kind of content a bit ago, I was uncomfortable with those being in MS at all. A change was made to MS to make images load on-demand by clicking them, which was great because I do NOT want those images in my cache (I promptly deleted and cleansed everything I could), so it makes sense we should not preserve links to it anywhere ideally.
If we come across it, we escalate it, and we forget about it.
I mean, OK, then exclude that site. I have no idea what goes on on that site and what might be on-topic, but that's a solvable problem.
I could see LEs having a workflow that looks like “kill Tor site, kill anything linking to it”
What are LEs?
law enforcement
@Spevacus I’m not involved much anymore, but my policy has always been that CP, and to a lesser extent abuse generally, doesn’t need to be in metasmoke. We have a button for nuking stuff specifically for that
I've somehow managed to overlook the most important button ever.
16:51
it’s at some level of admin
I’d be fine with writing off tor.se
Cow
Cow
@CodyGray I added that to watch list like many hours ago here
There really aren't that many *.onion links there that I can see, and regardless, the post volume is very low, so I don't think having that as a general watch would be an issue.
@Cow That would make some sense as a trigger
16:54
There is some value in keeping the IDs at least for a time, as that allows detecting and nuking the posts when multiple are posted.
Yes, but then they should be on the blacklist, not the watchlist, right?
Cow
Cow
Sorry I was not aware of Onions going directly on the blacklist :)
@Undo Indeed, I'm surprised that string of gibberish wouldn't have been caught for some other reason already, like luncheon meat.
@Cow No, not all Onion links. Onion links to known CSAM.
It wouldn’t surprise me much at all if GH has a list of onion hashes that are blacklisted. Those are long and unique enough that they could do it without any real FP risk
Cow
Cow
@CodyGray ah got it
16:56
I’m guessing no one actually followed the onion link and saw actual CSAM?
@Undo Indeed. So, perhaps @Makyen's point is best addressed with a local-only, time-limited abusive materials blacklist?
Cow
Cow
@Undo I hope not
I'm guessing that someone did, at one point when it was first added. Or, at least, it was included in a post that made it very obvious what is contained behind that link, so we can be relatively certain.
Otherwise, you would not be adding them to watch/blacklist...
Yeah, it takes real effort to follow an onion link, so I’m guessing it was just from context that we’re assuming it was CSAM
I don’t think I’ve ever actually used tor
I actually have no idea how to do it, so you are probably right about that.
The people who post this stuff usually aren't subtle about it. That was the whole motivation behind hiding images in MS.
Cow
Cow
17:00
turns out Browserling has a Tor remote browser for viewing onions
I just tried the one that Cody linked, good idea to remove it
Err, really? I thought it was just relatively innocuous spam.
Do we need to purge that report from MS, too?
Cow
Cow
no it's not that bad
we've seen that one a bunch of times: metasmoke.erwaysoftware.com/domains/73102
@Cow to be direct - if it contains CSAM of any kind, I'll drop all those references from MS.
Cow
Cow
17:04
@Undo Please remove it then
Wait, how did you go from "it's not that bad" to "it contains CSAM of some kind"?
Metasmoke post 291771 (Hi i will show the link of dark web its for educational purpose only surf it safely don't do anything ilegel,) destroyed by Undo
@CodyGray guessing from context it's an "index" that contains links to many .onion sites, some of them are probably CSAM
which is good enough for me
17:06
Metasmoke post 293028 (have you seen hidden wiki?) destroyed by Undo
Metasmoke post 371515 (How to get URL, title and body from a onion link) destroyed by Undo
Metasmoke post 387524 (How are the Hidden Wiki and onion links made? (ALSO THE NEW HIDDEN WIKI V3 LINK IS NOW AVAILABLE READ DESCRIPTION!)) destroyed by Undo
Metasmoke post 398617 (Enjoy where you can earn 15 $ per post have a good Day Visit http://....) destroyed by Undo
Metasmoke post 409386 (Clean Architecture vs Onion Architecture) destroyed by Undo
Metasmoke post 411497 (How to access the dark web) destroyed by Undo
Metasmoke post 425916 (Why do I use HidenWiki?) destroyed by Undo
Metasmoke post 444729 (Why is axios not finding onion addresses while using SocksProxyAgent?) destroyed by Undo
Metasmoke post 467788 (Why does useState trigger rendering with the same value) destroyed by Undo
Metasmoke post 472676 (Have you tried deepweb wiki which links are safe as per your experience?) destroyed by Undo
Metasmoke post 479384 (How Can I import Multibit Private Key into Electrum or Blockchain.com) destroyed by Undo
Metasmoke post 479450 (V3 The Hidden Wiki) destroyed by Undo
Metasmoke post 495793 (Do you know the new v3 hidden wiki) destroyed by Undo
Cow
Cow
come on Github
poke
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, phone number detected in title, potentially bad keyword in body, potentially bad keyword in title (283): Gunakshi creations.in Customer Care Number 8116280876‭ by Muskan143‭ on stackoverflow.com
Should we be compiling a watchlist elsewhere?
[ SmokeDetector | MS ] Link at beginning of body, potentially bad keyword in body, potentially bad keyword in title, potentially bad keyword in username (49): How i can watch movies on HD streamz APK | Is it trusted?‭ by HD Streamz APK‭ on stackoverflow.com
@CodyGray not a bad idea, I often go over the Halflife transcript to pick out things from the recent past but it's kind of messy
I mean specifically due to the, uh, unplanned downtime.
[ SmokeDetector | MS ] Phone number detected in body, phone number detected in title, potentially bad keyword in body (189): +1-833-633-1571 What is the cancellation policy for Alaska Airlines?‭ by Flightsvilla‭ on meta.SE
!!/watch 18336331571
17:23
@Vickel An invalid pattern was provided, please check your command.
@Vickel git pull has failed. This shouldn't happen. Details have been logged.
@Vickel See the pinned messages.
@Spevacus ohhh sorry, thanks...
No problem :)
[ SmokeDetector | MS ] Bad keyword in body, potentially bad keyword in body (95): How do I complain on PhonePe?‭ by Sonam Kumari‭ on sharepoint.SE
Cow
Cow
I have made a manual watch list for whenever Github is back, do you guys want me to share it before I got to bed later?
tpu- feedback received on [MS] How do I complain on PhonePe?‭
Cow
Cow
17:30
or maybe it doesn't matter
Have we looked at the idea of watching "How do I complain..."?
@Cow As a heads-up, this is another instance where a mod flag would go a long way, rather than multiple spam flags.
Cow
Cow
argh oh no!
I did it again
fixing now
I've mod-flagged the Magento ones.
Cow
Cow
17:34
retracted, thank you @Spevacus sometimes I run on autocow
No worries.
@GeneralGrievance Yes, something like that has been blacklisted, and also watched. Is it not catching the posts you'd expect?
Some modifications could be made to more align with this recent report, which wasn't detected by those reasons.
Obviously they'll need to wait... But... :)
@CodyGray Oh, I see. I was thinking "Bad keyword in title" would have shown up in this report.
Ah, yeah, that watch needs a tweak.
17:39
@Undo I’m not sure what your criteria for destroying those was, but an onion link isn’t inherently CSAM, or even bad. It is more likely to be nefarious of some sort, but it could be anything.
@Makyen The conclusion is that the Onion link was to some kind of "index"/ToC/directory page, and that included links to CSAM. (I'm not completely sure whether that last statement was empirically verified, but the evidence seemed to suggest that it was true.)
[ SmokeDetector | MS ] Mostly punctuation marks in body, repeating characters in body (81): How many times will the following summation run--> \sum_{i=0}^{0}?‭ by Joseph Igwe‭ on stackoverflow.com
17:42
@CodyGray yes, that one, but quite a few reports were destroyed, including at least one from years ago.
I believe they all included the same Onion link. Or, at least, that was my understanding...
@Cow maybe we just add watch xyz chat commands here (as plain text, not SD commands) for easy copy/pasting later?
Cow
Cow
@JeffSchaller good idea, how do I make plain text commands?
watch not-a-real-keyword
By typing with your normal human hands. Oh, I see the problem already...
Cow
Cow
17:47
rofl
ok ok, I'm not so fast ok :D
watch- test
ok good
!!/watch- edutrainex\.com
Or something nice and obvious for chat search, like: GITHUBDOWN blacklist something
Cow
Cow
!!/watch- (?-i:qAx1muGCWJg)(?# youtu.be)
!!/watch- hdstreamz\.tv\.in
!!/watch-number- 18336331571
!!/blacklist-number- 8919779921
!!/blacklist-number- +91 8919 779-921
what I got so far
thanks @JeffSchaller
Oooh, there's also a chat feature I never use -- conversation. I think we'd mark the beginning and end; could be a way to find this batch in the future
Cow
Cow
17:50
;test;
?
--test--
hehe
Strikeout text is three dashes before and after the text you want to strike out. But Jeff is talking about conversations, which are also, confusingly, sometimes called "bookmarks" in the UI.
Makyen/MS-Relay: Executing automatic scheduled reboot.
[ SmokeDetector ] SmokeDetector started at rev ee0f84fecd (Makyen: Log as debug MS WebSocket messages) (running on Makyen/MS-Relay, Python 3.11.3)
Time to rename them to "saves" ...
Only problem is that you cannot continue to add to that. Well, unless you abuse mod privileges to edit chat messages.
@Spevacus Let's keep the hands off the one thing that hasn't been broken for now.
17:56
LOL
yeah; it wanted an "ending" chat message; I don't see a way to edit the "conversation" afterwards. it's at least a way to find one batch
Well, I guess we could create a Gist. Just kidding.
[ SmokeDetector | MS ] Link at end of answer (63): Аренда виртуального сервера VPS/VDS‭ by anon‭ on ru.stackoverflow.com
Makyen/EC2-linux: Executing automatic scheduled reboot.
@CodyGray If it really was the same one for years, that argues that we really should be detecting it.
@JeffSchaller there’s no way to edit. You have to delete and recreate.
I started with a small ripple on LinkedIn to GitHub. So far it's caught Tim and Jon's attention
Cow
Cow
@Andy thank you
[ SmokeDetector | MS ] Bad ip for hostname in answer, bad ns for domain in answer, bad keyword in answer, bad keyword in link text in answer, potentially bad asn for hostname in answer, +2 more (427): How to serve marketing content and web application from the same domain?‭ by Aslam Ashraf‭ on stackoverflow.com
18:23
@SmokeDetector tpu-
[ SmokeDetector | MS ] Bad keyword in answer, pattern-matching website in answer, potentially bad ns for domain in answer (167): Why is my fragment not working when im clicking the button for it to switch fragment?‭ by Matthew Mark‭ on stackoverflow.com
[ SmokeDetector | MS ] Potentially bad ip for hostname in body (1): Why are my DIV cols not working in a new Shopify template?‭ by Kelly Rossi‭ on stackoverflow.com
Cow
Cow
has the SD Github been disabled before?
Cow
Cow
19:10
:-/
[ SmokeDetector | MS ] Potentially bad keyword in body (1): Failing To Setup Dynamic Enum Popout Menu‭ by Vona‭ on blender.SE
[ SmokeDetector | MS ] Username similar to website in answer (73): Omitting the subject in German? Hegel‭ by Lars van der Miesen‭ on german.SE
19:38
[ SmokeDetector | MS ] Blacklisted website in answer, link at beginning of answer, potentially bad ip for hostname in answer, potentially bad ns for domain in answer (134): How do I make a code that, when the input is a list of your TikTok followers, make a list of all of them that have over ex. 10 000 followers?‭ by Ahmed Sani‭ on stackoverflow.com
@SmokeDetector tpu-
[ SmokeDetector | MS ] Manually reported answer (93): How to detect N+1 problems in a Hibernate/JPA application‭ by jeyong‭ on stackoverflow.com
[ SmokeDetector | MS ] Potentially bad keyword in body (1): API RETORNANDO 404 AO ADICIONAR O PARÂMETRO ID‭ by Everton Hinckley De Azevedo‭ on pt.stackoverflow.com
Cow
Cow
19:49
gnite all :)
[ SmokeDetector | MS ] Potentially bad ns for domain in answer (1): How to fix badly scratched wall? (Paint scraped off)‭ by David Ahrens‭ on diy.SE (@Anerdw)
@SmokeDetector tpu-

« first day (3943 days earlier)      last day (171 days later) »