What exactly does an EC2 SmokeDetector instance do. Like, if I make one, does it let me run a private SmokeDetector instance in some chatroom I make? Or does it help the main smoke detector instance?
@tripleee Your docker setup is awesome; I wanted to set it up locally and your instructions worked perfect. 8 minutes start to running on an M1, including full DB+Redis dumps. And I think Docker was restricted to only 2 cores, too.
Hello! I'd like to run a Smoke Detector instance - I read through this page on the Wiki. Once I get it setup, to connect it to the main @SmokeDetector bot and to @metasmoke, I just need to ping an admin to get the role, and then it will help out scanning posts? Or are the instances for flagging posts?
@cocomac On an EC2, you would be able to run an SD instance which is operating under GitHub and SE accounts you specify and which communicates into rooms which you specify. In general, running your own instance can be used for running a specialized version of SD, testing changes to SD's code, or, yes, an EC2 could be a full instance of SmokeDetector which could communicate with metasmoke.
However, more recent testing indicated that running a full-time active SD instance really needed quite a bit of CPU capability, which required selection of a larger, more capable base compute instance. At the time, to be assured of being able to keep up with scanning, it appeared that it would be necessary to have 4 CPUs available, which put the smallest matching EC2 t3 instance at t3.xlarge, or other similar instance (e.g. I was looking at cost trade-offs vs. c6i.xlarge).
I'd want to do more cost trade-offs and testing prior to saying what size of instance is really needed.
The above testing was prior to the change in SD's code for the number detections moving away from a regex based solution. Ballpark testing indicated that changing that code resulted in substantial compute savings under various conditions, which may drop a full-time instance back down to needing only 2 CPUs (which used to be the case).
There have definitely been some quite substantial performance improvements after the most recent testing I did with respect to what's needed for CPU capability (i.e. required compute numbers are outdated). Basically, at that time, an "unlimited" t3.small fell substantially behind real-time processing during peak traffic periods (i.e. it swamped 2 CPUs).
That testing was part of what indicated the need for the number detection code to change (in addition to the need for improved capability) and identified some areas where there were input types which would cause quite significant consumption of CPU resources. All of the issues which have been identified have been resolved, but there's almost certainly more which should be found in other areas of the code.
I have some prototype code to look into performance issues further, but haven't gotten to the point of deploying it.
Overall, it appeared that a more cost-effective solution would be to just purchase a lower end PC and run SD on that.
@Makyen Good to know. Would it be fine if it wasn't on always, like if it was only on during certain times of day? I have a device in mind (a powerful server I have at home). It almost certainly meets the requirements. It's older, maybe 2010-ish, but the specs are great. It has 48GB RAM and two Xeon E5620s for a total of 8 cores/sixteen threads.
@cocomac I appreciate your interest and willingness to run an active/standby SD instance. However, it's quite unlikely that we'll take you up on the offer to do so at this time. At the moment, our need for additional backup instances is low, due primarily to Thomas Ward providing both MS and the Osiris SD instance, which has been the primary SD instance for a couple years now. We've had some other people offer to run an instance, if we need it, but we haven't taken them up on it, either.
@cocomac That sounds like a machine which would be more than capable of running an SD instance, if not more than one. :)
@Makyen Thanks! That makes sense. If I run my own instance (in a chat room just for me, not connected to the main one), would I be able to have it autoflag directly from there? Bonus question: Because Metasmoke is open source, would I be able to run a private Metasmoke dashboard just for me on my LAN?
@cocomac But, yes, that is indicative of the cost trade-off of using EC2 for an active SD instance. That hardware which you have lying around and not in use should be way more than sufficient. You would have to pay for the electricity and internet access, but that's not all that much (and the internet access is almost certainly something you already have, as the needs for SD are quite low and easily can be on a home broadband connection, but you might want a proxy or VPN for a separate IP).
Whereas a full-time instance running on EC2 would cost in the ballpark of $60 to $120 per month, which is just a tad more expensive than using your otherwise unused equipment for the price of electricity. :)
@Makyen If you can tell me, how much bandwidth does SD typically use? I don't mind a few gigabytes, but if it's 500 GB, that's... not so reasonable. I'm not concerned about energy usage
@cocomac You could run your own instance in a chat room just for yourself, if you want. There are multiple people that have their own instances. The majority of those have been for testing or code projects, but there's nothing which keeps you from using it for reports. Yes, MS is open source, so you could run your own. As to autoflagging, MS is what decides when autoflags will be raised.
The actual flagging is done by yet another compute instance (an AWS Lambda) which is controlled by Undo. It's what holds the user's tokens. It's set up that way such that even MS developers don't have access to the tokens to other user's SE accounts.
@cocomac I've never actually measured it, but I'd expect it to be so low such that it would be hard to measure on a connection which is shared with a "normal" household usage. IIRC, we did do some calculations at one point.
[ SmokeDetector | MS ] Bad keyword in body, blacklisted username, blacklisted website in body, link at beginning of body, potentially bad ip for hostname in body, +1 more (396): do you play the hocky game? by lallu lal on stackoverflow.com
[ SmokeDetector | MS ] Link at beginning of body, pattern-matching website in body, potentially bad asn for hostname in body, potentially bad ns for domain in body, potentially bad keyword in body (139): do-poodles-shed by Mirriam Duke on tex.SE
I'm setting up a local SD instance - I cloned the repo, did the pip install, and I'm editing the config file. I've set me SE username + password. I haven't set any of the other values, as I'll setup Metasmoke later. I got this error: ``` [05:37:25.317] chat.meta.stackexchange.com: Login error LoginError: failed to get `acct` cookie from Stack Exchange OpenID, check credentials provided for accuracy ``` But... I'm pretty sure those are the right credentials, as they are what I use to sign in.
Hmmm. I tried a few times, but now I get this: ``` [05:52:19.516] KeyError: 'time' 2022-02-16 05:52:19.515093 UTC File "/home/me/SmokeDetector/ws.py", line 192, in <module> chatcommunicate.init(username, password) File "/home/me/SmokeDetector/chatcommunicate.py", line 138, in init join_command_rooms() File "/home/me/SmokeDetector/chatcommunicate.py", line 158, in join_command_rooms room.join() File "/usr/local/lib/python3.9/dist-packages/chatexchange/rooms.py", line 52, in join return self._client._join_room(self.id)
@Makyen Both. Ran it, let it try a few times, let it fail, did ctrl + c, and tried again. I gave up after a few attempts, so I'll try again in the morning (I should go to bed soon)
Of course, I'll also double-check those credentials in a private window to ensure it isn't an "I gave it the wrong credentials" issue
@cocomac IIRC, one issue is that SE may be wanting you to solve a captcha, which is generally solved by just retrying, sometimes several times. Once it logs in with those credentials from that IP, it tends to be better behaved for later attempts.
@cocomac As I recall, when first starting mine, I ended up checking the credentials multiple times and swapping credentials between accounts a few times because it wasn't logging in, but it eventually started working after not tooo long (although it did feel a bit long at the time).
[ SmokeDetector | MS ] Bad keyword in link text in answer, link at end of answer, potentially bad keyword in answer, potentially problematic ns configuration in answer (144): RoR: Mailing Lists? by JKMS9009 on stackoverflow.com
@Someone You are not a privileged user. Please see the privileges wiki page for information on what privileges are and what is expected of privileged users.
@SmokeDetector fp- while this is somewhat off-topic, low-quality, and ought to be removed expeditiously, I'm not sure there's concrete evidence that it's spam. The domain has never been seen before, and it's a plausible question. I would, of course, change my mind if seen again or it turns out to be a spam seed.
@tripleee That pattern looks like it's already caught by Potentially bad keyword in answer and Potentially bad keyword in body Append -force to the command if you really want to add the pattern you entered.
@tripleee That pattern looks like it's already caught by Potentially bad keyword in answer and Potentially bad keyword in body Append -force to the command if you really want to add the pattern you entered.
@Someone hmm, okay, thanks for clarifying ... can you just briefly introduce yourself and your interest in this project? We welcome new participants but we want to be a bit careful
@Someone You are not a privileged user. Please see the privileges wiki page for information on what privileges are and what is expected of privileged users.
@Someone thanks, and welcome! I have enabled Flagger and Reviewer on 1823
unfortunately, with chat search being broken, I can't find the usual spiel delivered by Makyen to new users; the wiki is obviously worth keeping open as you follow along here, and please don't hesitate to ask if you are unsure about something
admin presence in this room can be spotty, depending on your time zone, but at most times multiple regular users are around, and happy to help figure things out
@Someone You are not a privileged user. Please see the privileges wiki page for information on what privileges are and what is expected of privileged users.
