« first day (1416 days earlier)      last day (2702 days later) » 

09:00
Deal.
[ SmokeDetector | MS ] Few unique characters in answer: Order date ascending by jhbvc on stackoverflow.com
@angussidney you need a JokeDetector.
Or I need to make better jokes.
Henders/EC2 received failover signal.
Restart: API quota is 19977.
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, +1 more: Vigorous Muscle Maximizer the alternative dietary by waynemrodri on askubuntu.com
!!/standby Henders
09:07
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, link at end of body, pattern-matching product name in body, +3 more: [PTX Male Enhancement Since abounding water-based by user77633 on workplace.SE
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching website in body: Natural Your Skin Care When Considering by Lagantoo on drupal.SE
sd - k - k - k
1. [:40328013] <skipped>
2. [:40327999] <processed without return value>
3. [:40327997] <skipped>
4. [:40327993] <processed without return value>
5. [:40327992] <skipped>
6. [:40327990] That message is not a report.
09:10
SD - 2k
sd - 2k - k
Henders/EC2 received failover signal.
This time I'll let Henders stay, I'll failover tewards
!!/standby teward
[ SmokeDetector ] SmokeDetector started in standby mode at rev 681c48d (SmokeDetector: Auto watch of virplexin by tripleee --autopull) (running on teward/Lunar Eclipse)
Restart: API quota is 19971.
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, bad keyword in username, blacklisted website in body, link at end of body, +2 more: Geneticore Boost Review: Is it safe to use? Get Result by Geneticore Boost on askubuntu.com
09:19
[ SmokeDetector | MS ] Bad NS for domain in body, blacklisted website in body, pattern-matching website in body: These fixings keep up a sound and brilliant skin consistently. by Jessica zozu on graphicdesign.SE
sd k
I'm running out of flags in askubuntu... just give feedback instead.
@Andy yeah, I'm all for it, just I don't think I have the expertise to do it myself.
Old school flagging makes me appreciate FIRE
teward/Solar Flare received failover signal.
Oh dear...
!!/location
09:26
@Henders teward/Solar Flare
@Henders Henders/EC2
!!/standby teward/Solar
!!/location
@Henders Henders/EC2
[ SmokeDetector | MS ] Bad keyword in body, bad pattern in URL body, blacklisted username, pattern-matching product name in body, repeated URL at end of long post: Maximize your ability to stay longer and harder inside the bed room by ghaby juri on astronomy.SE
[ SmokeDetector | MS ] URL in title, bad NS for domain in body, bad NS for domain in title, blacklisted website in body, blacklisted website in title, +3 more: healthpluscogni.com/xtraperf/ by pipr on astronomy.SE
[ SmokeDetector | MS ] Manually reported question: phase for your radiant skin by paulorsuch on meta.SE
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, pattern-matching website in body, repeated URL at end of long post: Do whatever it takes by wexasasan on apple.SE
[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, blacklisted website in body, blacklisted website in title, +4 more: freesupplementrial.com/force-fit-xl/ by zoprert on askubuntu.com
tpu- by suraj
tpu- by suraj on Do whatever it takes [MS]
sd - - k k
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching website in body: Ultra Slim Garcinia Cambogia Exactly by dfgfdgewewerse on askubuntu.com
[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +4 more: nutrasunnaturalgreencleanseblog.com/alpha-x-factor/ by QuinnGendron on askubuntu.com
09:45
MS is loading pages now but very slowly
tpu- by Tetsuya Yamamoto on Ultra Slim Garcinia Cambogia Exactly [MS]
tpu- by Tetsuya Yamamoto
[ SmokeDetector | MS ] URL in title, bad NS for domain in body, bad NS for domain in title, blacklisted website in body, blacklisted website in title, +3 more: healthpluscogni.com/xtraperf/ by alexis loyau on astronomy.SE
tpu- by angussidney
@SmokeDetector why
09:50
@angussidney Post manually reported by user Floern in room Charcoal HQ.
(just testing something)
0
Q: Do spam flags and RA flags count together?

iDebugWhile I was looking around Android Enthusiasts, I saw 3 notably inappropriate answers (10k/Mod) in-a-row under the same question. The first few lines contained an obvious spam link, but its superior length (than regular spams) made me curious and I read the whole post. Realizing its talking about...

[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +6 more: www.oralhealthplus.com/hyperfit-slim/ by Tunt1982 on askubuntu.com
:40328447 Hello :)
09:56
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, link at end of body, pattern-matching website in body, +1 more: At first it manufactured the reason by Gonly plam on drupal.SE
[ SmokeDetector | MS ] URL in title, blacklisted website in body, blacklisted website in title, link following arrow in body, pattern-matching website in body, +1 more: gomusclebuilding.com/virplexin/ by spence more on apple.SE
[ SmokeDetector | MS ] URL-only title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +3 more: nutrahealthtrimsite.com/forskolin-rt/ by Finchkor on askubuntu.com
[ SmokeDetector | MS ] Blacklisted website in body, link at end of body, pattern-matching website in body: Thrive Max - Provides better memory, focus and concentrationThrive Max by amychapa on workplace.SE
10:11
[ SmokeDetector | MS ] Bad keyword in body, pattern-matching website in body: The Unexpected Signs Of Aging by Kveraljzpq on meta.SE
tpu- by J F
!!/rev
!!/pull
!!/rev
10:12
Restart: API quota is 19430.
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, link at end of body: advisorwellness.com/youthful-balance-serum/ Youthful Balance Serum :=> In any case, by Rockhandsome on askubuntu.com
Is Metasmoke dead?
FIRE does not open any reports.
J F
J F
@ByteCommander Kinda.
No, not at all
5 hours ago, by tripleee
Metasmoke is DOWN! Expect erratic behavior from stuff which depends on it (Smoke Detector, userscripts, etc)
10:22
Oh....
J F
J F
ninja’d
Should have a look at the starwall every now and then :P
Sep 19 '16 at 10:16, by angussidney
Why people don't read the star list, I don't know
^^ I wondered the same
Is the low number of SD instances related? I see only 4, it used to be twice as many
10:24
[ SmokeDetector | MS ] Hindi character in title, mostly non-Latin body, mostly non-Latin title: मैर प्रणाम सभी भाईयो को by user8713308 on android.SE
@ByteCommander mine's offline while I sort out git issues, Art's died and he hasn't read my pings about it, one of Nobody's instances was having trouble, and Undo hasn't brought his back online since we required the dnspython package
tpu- by ByteCommander
translation:

My dear brother to all the brothers


Hello to my friends

Date -3 / 10/2017. Mangalore

(Little age of love) //////////////// __ // Naughty eye made this worry at this age. This sweet smile, which I smiled, said the friend to Nelly, what you know about friends, love is not a small, it is love.

Pawan Sharma Kosi
Makes few sense to me, therefore R/A
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, pattern-matching website in body, potentially bad keyword in body, potentially bad keyword in title: Tribulus terrestris things on splendid, however Force Fit XL by lisagloria9 on astronomy.SE
tpu- by angussidney
10:29
Are they the people that make heavy metal music from hard drives and optical drives?
I don't think so
Must be thinking about something else
Still, I really like their music
@ByteCommander Henders/EC2 should be running fine. As should teward/*
Just a MS issue I guess
Your EC2 reports as active and Nobody's Raspi and Thomas' Solar Flare and and Lunar Eclipse are on standby, according to the status userscript.
It used to be around 8 instances recently though.
Now it's 5 :)
> Art's died and he hasn't read my pings about it
two or three should be sufficient if they were reasonably stable
@Glorfindel I got a ping from you around 3h ago, but it disappeared from the transcript (or I am too stupid to find it). What was it about?
See, I said he was a ghost.
10:41
Yeah, most were voluntarily taken down
having more has been beneficial when there were crises but having many unmaintained instances did not uniformly reduce stress levels
@ByteCommander check your inbox maybe?
I see it in the inbox.
@tripleee By 'unmaintained' you mean?
@angussidney oh I read 'em alright, just couldn't be bothered to do anything about them at that time
10:41
But not in the transcript. Probably it got deleted.
@ArtOfCode :P
@tripleee Conditional II doesn't sound too confident...
Also: waffles for breakfast is the best idea I've had this week
@angussidney Art's died - no wonder he couldn't be bothered
@ArtOfCode waffles are the best
10:42
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, +1 more: PrimeX Testo Max Learning to lift weights correctly and safely c by NormaPenaloz on security.SE
@Henders standby but not able to successfully take over when the main instance goes down
Has anyone seen this before? thekeyboardwaffleiron.com
3
tpu- by ByteCommander
@tripleee Are there many of those?
@angussidney yup
10:43
@ArtOfCode Homemad?
e
@Henders no, homecrazy
@ByteCommander we needed the contents of a deleted post on AU, but suraj already found it in Metasmoke. Thanks for pinging back.
sigh - predictable :P
Ah, okay.
10:44
@Henders this morning teward's was behaving erratically, in the past there were instances which didn't have the correct git credentials or the right Metasmoke key or whatever
Probably mine in it's first days of running...
maybe the erratic behavior was because Metasmoke died but it seemed like teward's instances were particularly prone to popping up from standby when the main instance was still up
@Henders I remember that but it was not an isolated incident, we had that with other instances many times in the past
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, pattern-matching website in body: Dietary Excessive Fat - The Particular Bodybuilder's Friend by nenkiric on askubuntu.com
@SmokeDetector k
but now of course the real question is, how can we reduce the impact of MS going down
Well the chat exchange one with zombies was a bit of a rare case
10:46
being able to kick up a second instance would be super useful
but I am also thinking with Art and Undo being here less now than in the past, at the very least we should have more people with access to Metasmoke
@tripleee force Undo to be awake 24/7/365
What caused the failure? Maybe that would point to an issue we can fix
I guess based on seniority Angus and/or Thomas Ward should be nominated
@angussidney Actually yeah, that fixes the problem, nice.
@Henders who knows
10:47
@tripleee define "access"?
@angussidney Can we find out?
@Henders there will be failures in the future, regardless of how robust you make your Single Point of Failure
@tripleee the problem is that syncing data after the main instance comes back up will be hard
Or we can get a dedicated RDS server
@ArtOfCode dunno exactly, eventually having more project owners on Github is also one thing I have been meaning to bring up
Seniority? Thomas? lol
10:48
@Henders wait until Undo wakes up :P
@angussidney but, but, I thought we agreed he should just be up 24/8 now
@ByteCommander that's based on my observations here over the last year or so, if you want to propose others as well or instead, feel free
He's not so active with us anymore
@tripleee The only person with SSH access to metasmoke is Undo. I have the dev role, but not console access.
Nah, just kidding about his funny behaviour with lazers and hammers and the stuff. He's fine :P
10:49
@ArtOfCode not requiring console access would be a nice feature
or providing it without access to a physical console if making it completely headless is a problem
@tripleee it doesn't, generally, except when it dies entirely. You have to have SSH to bring it back up, there's no way around that
The rails console gives you complete access to the DB and everything anyway, so it isn't a softcore solution
What's the usual cause of a Metasmoke instance dying?
Also, while we're talking about it, I'd like to nominate @tripleee for admin, he's been pretty active with us for a while, knows what he's doing, and I think we need another admin now that 2 of us are starting to become a bit inactive. I've been meaning to send an email to the admin mailing list for a while, but never got around to it.
5
@Henders I don't think there is a "usual" cause
10:52
[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, pattern-matching website in body: China, the skin care review by parisness on graphicdesign.SE
@angussidney ping it to the list
@angussidney Aye!
tpu- by ByteCommander
I'm not sure I entirely agree, but that's not because of @tripleee :)
@Henders used to be lack of disk space, but now that we have logrotate on for both MS and Sentinel this issue is completely new
10:53
@Henders there will always be architectural failures (disk dead, network gone, motherboard + coke) which we cannot predict or work around
@angussidney oh well thanks guys (-: though I'm not sure I know what I'm doing
I just meant that if an RDS instance would in someway fix the issue then is it not an avenue worth investigation?
@Henders I was thinking that if we had a dedicated database server, then if MS itself failed it means that one of us can bring up another instance and connect to the same prod DB. Though that all comes with the issues of privacy and security and stuff that needs to be discussed separately
@Henders To an extent, it would, because it would allow us to have a second MS instance. I'm not sure it's worth it, though.
@angussidney I was thinking the same thing
(a) extra cost
10:57
^ of course
(b) that requires giving out database credentials, which isn't something we've ever done and not to be taken lightly
But if you removed cost from the equation, it could fix the issue?
Or at least help to mitigate
@Henders it would bring a number of new issues -- the code is built on the assumption that there is a single instance
we have some failover logic for the chat client but I don't think it's going to be directly applicable to running two instances of the back end server
and (c) switching instances requires changing the metasmoke URL on every Smokey instance
By the time that change has propagated, Undo is likely to have fixed it
@ArtOfCode AWS fallback would take care of that
if you go the AWS route
10:59
@tripleee oh?
Can AWS do switcheroos with the IP addresses?
@ArtOfCode yeah, they have pretty much instant switchover to another instance if one goes away, though I'm not sure if RDS specifically is suitable for that
@tripleee EC2 or RDS?
@ArtOfCode And remember, we have had a couple to people say that they'd be willing to donate if we had a bitcoin address or similar. Something that we can always consider doing.
@ArtOfCode well I am more familiar with EC2 but I think the scaling logic is a separate component which can probably handle both
11:01
@angussidney Aye, worth considering. It's not a favourite option of mine, but it's there
Or we could threaten SE by saying we'll bring down their spam detection if they don't pay up :P
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, pattern-matching website in body, repeated URL at end of long post: When you see the correlation for the skin in the more by Hicall19 on graphicdesign.SE
tpu- by Cai
!!/watch uneeq
@tripleee Added uneeq to watchlist
@angussidney I'm thinking maybe we should not even be joking about this
11:01
@tripleee AWS would need to take requests aimed at one EC2 instance and pass them on to a different EC2 instance. Can it do that?
@angussidney I'll make sure to tell Pops
@ArtOfCode again I'm not actually using that in production yet but yes, it's designed to do that
Although asking SE to help might actually be an option.
simple internal redirection, the front end proxy which receives the request figures out if the back end is up and if not, starts it up, and similarly starts up a second instance under some specific conditions and balances the load between them
@tripleee Huh, that could work.
11:03
[ SmokeDetector | MS ] Bad keyword in answer, blacklisted website in answer, pattern-matching website in answer: Is it a good idea to place app configuration files into "~/Ubuntu One" folder? by lillieterry3 on askubuntu.com
@SmokeDetector k
@tripleee Ah. Now there's the catch - it'd need to forward to an already running instance on someone else's account.
@SmokeDetector Woah, pharma spam in an answer
CI on 8c812ce succeeded. Message contains 'autopull', pulling...
Restart: API quota is 18989.
@Mithrandir if you have a new question spam post, please click the Ask Question button
11:05
@ArtOfCode or what about if we set up multiple instances of MS permanently, and add some load balancing between them?
Still ignoring cost for the moment
That way, when one goes down, all traffic will got to the other
@angussidney that sounds like over-complexifying
Let's think about this in real terms: how much does MS actually go down?
About once a month or so
@angussidney Really? I've not seen it that often - I was thinking more like every three
We've had logrotate issues about once a month for about 3 months now, but now that that's fixed we should be seeing it less often
[ SmokeDetector | MS ] Blacklisted website in answer: VueJS pass props to children in render function call by ng4free on stackoverflow.com (@quartata)
11:08
Once every three months is around 99.7% uptime
Once every month is around 99.2%.
Honestly, above 99%, I don't think we really need to do much mitigation. As a risk assessment, impact is 7 or 8, but likelihood is 1. A risk that low doesn't, in corporate environments, require any mitigation at all.
(Okay, we're not corporate, but it's a pretty good system)
And of course, how much of an issue is it when it happens? People complain about their userscripts not working, and we lose records of a few posts, and that's just about it
We should focus far more work on stopping Smokey crashing so much :)
That's an impact 9-10 likelihood 5
[ SmokeDetector | MS ] Blacklisted website in answer: Vue template or render function not defined yet I am using neither? by ng4free on stackoverflow.com (@quartata)
tpu- by suraj
@ArtOfCode email in your inbox once it arrives in 10 mins
:P
I fixed that most of a year ago now, dammit
11:12
Doesn't matter, the meme will live on :P
[ SmokeDetector | MS ] Blacklisted website in answer, blacklisted user: How do you completely remove Ionic and Cordova installation from mac? by ng4free on stackoverflow.com (@quartata)
tpu- by Yvette
@SmokeDetector looks like we will be seeing a lot of live4u. meta.stackoverflow.com/questions/357292/…
[ SmokeDetector | MS ] Bad keyword in answer, bad keyword with email in answer, pattern-matching email in answer: Can a converted woman to Judaism divorce her Gentile husband and marry a Jew? by linda jane on judaism.SE
@Glorfindel are you having trouble with the scripts?
11:22
Everybody is having trouble with the scripts.
Metasmoke is behaving erratically, that's why
@ArtOfCode I have a feeling you were about to say that :)
Heh :)
teward/Lunar Eclipse received failover signal.
tpu- by Yvette
Eugh!
11:24
!!/location
@ArtOfCode teward/Lunar Eclipse
@ArtOfCode Henders/EC2
!!/standby teward
!!/standby teward/
Oh MS, how we miss thee
11:25
[ SmokeDetector | MS ] Mostly dots in answer: How to control Independent developers activity by memo on stackoverflow.com (@quartata)
@ArtOfCode RE your last point, I have noticed a number of occasions where it would have come in use
@SmokeDetector tp-
@angussidney Aye, so have I, but it's only a few hours' difference at most
When I've been away from a computer for a few days, or while I'm still on the bus on the way home
Yeah, exactly
[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, blacklisted website in body, blacklisted website in title, +4 more: healthpluscogni.com/xtraperf/ by redhouane on askubuntu.com
tpu- by Yvette
[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +8 more: www.hairlosspropecia.org/uneeq-serum/ by Onsted82 on askubuntu.com
11:30
@SmokeDetector k
is the feedback going to MS?
Nope
so no point
pk
just falg
^ nice keyboard skills there ;)
@YvetteColomb if you feedback via chat, it is. Metasmoke is not entirely down.
11:32
@SmokeDetector that post was even autoflagged:
@Henders tired and multitasking :)
@SmokeDetector autoflagged
@Glorfindel That post was automatically flagged, using flags from: Owen Hines, Andy, DForck42.
spasmotic
11:34
@SmokeDetector what was this one's weight?
@ArtOfCode I would imagine you can set up an organization to do this, though it makes more sense for the organization to be the owner of all of this
[ SmokeDetector | MS ] URL in title, bad keyword in body, blacklisted website in body, link following arrow in body, pattern-matching website in body, +1 more: apple.stackexchange.com/questions/281593 by froing on apple.SE
NobodyNada/Raspberry Pi received failover signal.
11:44
!!/watch xtraperf
@tripleee Added xtraperf to watchlist
@tripleee HEAD isn't at tip of origin's master branch
Restart: API quota is 19392.
anyway, do we really need to run Metasmoke in Undo's cupboard? It would be somewhat easier to redeploy in various failure scenarios if that wasn't a requirement
if we don't want to marry this to AWS, we could look into putting it behind Cloudflare or something -- I believe they offer fairly flexible fallback options, too
haven't looked into Azure or Google Cloud but they probably have something for this as well
ArtOfCode/EC2 received failover signal.
[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +2 more: www.healthsupplementsreviews.info/thrive-max/ by karenchun on apple.SE
[ SmokeDetector | MS ] URL in title, bad keyword in body, bad keyword in title, blacklisted website in body, blacklisted website in title, +2 more: www.healthsupplementsreviews.info/thrive-max/ by karenchun on apple.SE
Restart: API quota is 19999.
!!/location
@tripleee ArtOfCode/EC2
@tripleee NobodyNada/Raspberry Pi
@tripleee Henders/EC2
11:49
oh please
now which one should we prefer?
!!/rev
riiight
!!/standby Henders
!!/standby NobodyNada/Rasp
!!/rev
11:50
!!/pull
@tripleee It's not in Undo's cupboard - it's on AWS
@ArtOfCode oh good
still the requirement for it to run under Undo's credentials is problematic
Someone's gotta run it
Restart: API quota is 19968.
11:52
@ArtOfCode there are acceptable (not good but acceptable) ways to share credentials these days
@tripleee not when API tokens are involved
If we wanted other people to have access to the machine, that can be quite easily done.
We don't, because it contains API tokens.
how so? put all of the shared secrets somewhere where you only have the privilege to decrypt, not the privilege to actually access the decrypted keys
Those... are the same thing
blog.cloudflare.com/pal-a-container-identity-bootstrapping-tool is one approach which I tried to discuss here before at some point
If I have access to the machine, the idea is that I still shouldn't have access to the secrets.
11:55
quite so
[ SmokeDetector | MS ] Bad NS for domain in body, bad keyword in body, blacklisted website in body, link at end of body, pattern-matching website in body, +2 more: item functions in countering age spots? by Obs tande on drupal.SE
tpu- by Yvette
You can chmod/chown the secrets file, but then I can't use rails commands because rails needs to be able to open it
@ArtOfCode the trick is to build the deploy instance so that it not only works without manual intervention, but so it prevents manual login etc
then inside that container the code can have everything decrypted, but you can't (easily) get in
@tripleee ...but then you can't administrate it.

« first day (1416 days earlier)      last day (2702 days later) »