@quartata nope

I mean, yes, code admin

Oh

not Admin admin

What's the privilege for failing over

!!/whois admin

I think code admin. Admin =! code admin :P

Do you have whatever you need so that you can tell MS to fail Lunar Eclipse over

@JF I am aware of 5 admin. Currently in this room: Andy, ArtOfCode. Not currently in this room: angussidney, Undo, tripleee.

@quartata possibly, although I never learned how

Ok it's not that cause I saw Nobody do it once

I'll do it the old fashion way

!!/standby Henders

[ SmokeDetector ] SmokeDetector started in standby mode at rev 11c52ba (tripleee: Watch q\W?soft\W?ltd --autopull) (running on Henders/EC2)

zzz

teward/Lunar Eclipse received failover signal.

!!/location

@Mithrandir teward/Lunar Eclipse

First try!

Restart: API quota is 19991.

You have to be a code admin, and the link only shows up if the instance has pinged recently (within the last 3 minutes)

!!/remote-diff

@Mithrandir Go to /status and click “Failover”

!!/pull~

[ SmokeDetector ] SmokeDetector started at rev 11c52ba (tripleee: Watch q\W?soft\W?ltd --autopull) (running on teward/Lunar Eclipse)

Restart: API quota is 19978.

@JF ah, thanks

Ok we're good now

Should I put an announcement on MS asking people to be careful?

> metasmoke is not reliably up at this time. Please avoid putting too much load on the server so other people can use it too.

Nah it's not Metasmoke

fp by Mithrandir

Pretty sure at least

k

@quartata what's up? :)

I dunno your instance was missing status pings but now in standby it seems ok

I have it on Lunar Eclipse for now

maybe take a peek at the error logs

That's just because MS is down.

It's doing it for all instances

Nope

It was making MS fail over to another instance even though yours was up

Yeah, that's because MS is down

Now that Lunar is running it hasn't happened again

It's being doing that all day

but MS couldn't have failed over an instance if it was down

Here I'll see if I can get it to do it again

Yeah, until it pops up and realises that the main instance hasn't pinged

!!/standby Lunar

[ SmokeDetector ] SmokeDetector started in standby mode at rev 11c52ba (tripleee: Watch q\W?soft\W?ltd --autopull) (running on teward/Lunar Eclipse)

Need me to failover to henders?

Yeah just to see

Bah, it's gone down again

teward/Solar Flare received failover signal.

tfw no code admin

Henders/EC2 received failover signal.

Restart: API quota is 19431.

!!/location

@Henders teward/Solar Flare

@Henders Henders/EC2

!!/standby a

Restart: API quota is 14883.

[ SmokeDetector ] SmokeDetector started in standby mode at rev e02ec67 (SmokeDetector: Auto watch of 9686826939 by tripleee --autopull) (running on teward/Solar Flare)

wtf

What is going onnnn

MS isn't loading but this stuff is working

MS being down triggers the failover every time it comes back up and realises that nothing has pinged it (because it's been down).

yeah, it's not dependant on MS, but MS going down causes this

it was happening like every 5 minutes on the nose, when Lunar was running it was fine though

If you read from there, you'll see this has been happening all day

This also happened when MS was down last week too

That just caused instances to restart constantly

OK so maybe now it's time to talk again about opening up ports and decentralizing this

I've been doing !!/standby <whatever> all day. I don't know the ins and outs. If you're concerned about my instance specifically, feel free to standby it and I can double check with Angus in the morning. It's been running as main all day though. I gotta shoot but I'll check back before I go to bed and thanks for the ping :)

!!/rev

@Henders Henders/EC2 11c52ba (tripleee: Watch q\W?soft\W?ltd --autopull)

We have to have something central for autoflagging and it's fine to have something intermittent for statistics but managing instances is something we should be able to do without Metasmoke, especially when it starts affecting uptime like this when we're fiddling with the standbys

@Henders it seems fine now

I don't know what the deal is so I'll leave it alone

[ SmokeDetector | MS ] Potentially bad keyword in body: how to indent for tab space after writing a statement by Chaitanya on stackoverflow.com

@SmokeDetector why

@DJMcMayhem Body - Position 75-80: Xtext, Position 111-116: Xtext

@SmokeDetector f

Why is xtext a bad keyword?

I see that it has 5fp and 3tp

Maybe having xtext-reviews be a bad keyword would be better

huh

if len(reasons) == 1 and ("all-caps title" in reasons or
                          "repeating characters in title" in reasons or
                          "repeating characters in body" in reasons or
                          "repeating characters in answer" in reasons or
                          "repeating words in title" in reasons or
                          "repeating words in body" in reasons or
                          "repeating words in answer" in reasons):
    datahandling.add_auto_ignored_post((post.post_id, post.post_site, datetime.now()))

I didn't know we did this

we automatically ignore posts (i.e report them only once) if the only reason is one of the repeating characters ones

that is

really weird?

@Undo Do you know why we do this?

I'd guess that's for vandalism things

^

but wouldn't we want to know if it got vandalized again

although I guess it also tracks the revision

> datetime.now()

Best chance there is to trace back the blame, probably

I remember us adding something like that, don't remember why

@quartata IIRC it just looks for an 'f' in there

it's no big deal, I was just going through spamhandling to add a few things and noticed

@Undo ok cool

yep. github.com/Charcoal-SE/metasmoke/blob/master/app/models/…

@Mithrandir Yes cc @YvetteColomb :)

@angussidney Looked at that this morning - it's very much not a MySQL problem, as far as I can tell. Today's freezing wasn't because of read starvation from anything I can see

Yeah I doubt it's a hardware thing

I think Undo's instincts about Passenger going awry after some update seems sound

[ SmokeDetector | MS ] Blacklisted website in body: How to open a form in pdf file in android studio and make user to fill the form from app by Syyam Noor on stackoverflow.com

sd f

I'm tempted to just throw hardware at it, but ultimately these are the facts I keep coming back to: (1) it worked for months without intervention, then (2) suddenly failed with no discernible human-driven action, and (3) apt-get was all messed up on some kind of auto update at that point.

It lines up really well

Tempted to go back to running on 14.04 and see what happens

I didn’t necessarily mean it to reduce downtime, rather simply to speed it up a little. DB access seems to be the slowest thing for me, but maybe it’s my AU internet connection

^

DB seems to be taking a long time in the US too.

Miniprofiler says so, yeah

MiniProfiler is showing me simple SQL queries taking far longer than they should

I'm talking queries that should have sub-millisecond timings having a few hundred milliseconds.

In metasmoke?

Granted that's because MS is particularly slow today

whoa, it's up. I thought it was dead.

what the heck

Usually the little thing at the top is about 200 ms

well that's an improvement, at least.

@Undo it's been going up and down constantly

Ping me next time it's down, I want to try to get a backtrace out of it

We'll do. You just have to be online when it's down.

@Undo you're in luck pretty sure it just went down

@Undo yep

It took 32 seconds to load dashboard, and a simple SELECT * FROM api_keys WHERE key = 'keystring'; took 563.25ms.

haha

@ByteCommander I have expedited chat pings enabled, so I see everything... even if I don't act like it

heck, maybe it is mysql

did you update it

SELECT COUNT(*) FROM `posts` WHERE `posts`.`is_tp` = 1; took a whole second for me. SELECT COUNT(*) FROM `posts` LEFT OUTER JOIN `feedbacks` ON `feedbacks`.`post_id` = `posts`.`id` AND `feedbacks`.`is_invalidated` = 0 WHERE `feedbacks`.`post_id` IS NULL; took 4s.

haven't touched it since this morning

Get a MySQL console and run some raw queries on it. I can run the same queries locally and tell you how long they should take.

@JF That's always a long query, but not usually that long. 800ms is about normal for that.

The longer one is Posts.without_feedback.count. Maybe we should cache it — we know when we can bust the cache.

Shouldn't need to. It used to be fast

(post created or feedback created on post w/o feedbacks or feedback deleted)

I'm blaming MySQL. In a "mysql is doing something stupid" way, not in a "we outgrew mysql" way

Heeeey

Anyone remember that issue I had before with MySQL on 16.04?

? (before my time)

sounds right, thanks

well... what happens if we move to MariaDB?

that worked last time

Nginx stopping

@ArtOfCode Yeah, but we moved back for some reason. Forgot what.

@Undo it was a bug with MySQL not working with 16.04; it got fixed.

I'd be really impressed if we outgrew MySQL. Legitamately outgrew it, not "stupid bug" outgrew it

@ArtOfCode supposedly. I'm skeptical :P

We went back to MySQL because it's a touch faster than MariaDB

also, what the heck

@Andy there's no way a 100MB database is doing this. It could keep the whole thing in memory and be fine :P

*doing this legitimately

Also in today's Science: a t2.micro can handle way more load than I expected. 2k requests/minute is apparently fine.

Only 33/s

Keep them under 30ms each and you don't even have concurrency

@Undo The last one was apparently in January 2017.

I have to try it anyway, because if it works I can say "the tides broke my web app"

lol

@Undo They're a touch more than that. Anyway, you hit the problem of running out of CPU credits before you hit concurrency issues

Oh, right

Still, I'm impressed. I'd expected more like 200 req/min max

um

I don't think this email went through somehow

Hahahaha

also, we're moving back to Ubuntu 14.04 in hopes of more sanity

Why do you think it didn’t go through?

@JF look at the address it got sent to

Oh, I see.

@Undo how’s it going?

[ SmokeDetector | MS ] Link at end of answer: Таблицы в веб приложении by Georgy on ru.stackoverflow.com

@JF It's always a long process

with new issues each time. This time my backup script is unhappy

SO Documentation badges have just been converted to a special badge.

Restart: API quota is 13630.

Can I suggest that everyone disable their Charcoal userscripts, the API calls may be giving the server a hard time whenever it just recovers

@JF that's been awarded twice apparently

@angussidney no need

It's dead right now while I move stuff over

mysqldump is being slooooooow

Restart: API quota is 13543.

!!/errorlogs 25

  File "/usr/lib/python3.4/threading.py", line 1186, in run
    self.function(*self.args, **self.kwargs)

  File "/home/ubuntu/SmokeDetector/metasmoke.py", line 361, in send_statistics
    data=json.dumps(payload), headers=headers)

  File "/usr/local/lib/python3.4/dist-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)

  File "/usr/local/lib/python3.4/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)

sd gone

ermm

Oh well

!!/status

@Undo Running since 21:50:50 UTC (4 minutes, 9 seconds)

!!/reboot

[ SmokeDetector ] SmokeDetector started at rev 11c52ba (tripleee: Watch q\W?soft\W?ltd --autopull) (running on Henders/EC2)

Restart: API quota is 13512.

Metasmoke working for anyone else?

I do get an error for the system status: Internal Server Error

don't know why we're not getting Smokey connections

@Undo It’s up, except the TTD graph is broken.

NobodyNada/Raspberry Pi received failover signal.

Restart: API quota is 18776.

huh. Sideways Dictionary explains technical terms with real-world analogies.

rebooting some stuff

!!/standby nobody

[ SmokeDetector ] SmokeDetector started in standby mode at rev 475b3a6 (SmokeDetector: Revert "Merge branch 'master' of github.com/Charcoal-SE/SmokeDetector";) (running on NobodyNada/Raspberry Pi)

[ SmokeDetector | MS ] Manually reported answer: add site url if it is missing by skrotbil on stackoverflow.com

scooter repair spam. wut.

Hey, some people on SO may need broken scooters fixed. :P

[ SmokeDetector | MS ] Email in answer, pattern-matching email in answer: Should I use short term loans to pay for student loans? by rate setter loans on money.SE

[ SmokeDetector | MS ] Link at end of answer: Always show text cursor while moving it in Word by Megan on apple.SE

tpu by Jamal on Should I use short term loans to pay for student loans? [MS]

@Undo how time flies.. sniff :D

tpu- by Yvette

@SmokeDetector spammy website link in email address

@Jamal and testosterone and weight loss and muscle gain and smooth skin and scooter repair. It's all a rich tapestry

I remember seeing one piece of chimney cleaner spam, too.

well we all need those. O>O

[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer, repeating characters in answer: Using Lightning headphones in my Dell laptop by yooyo on apple.SE

tpu- by Yvette

naa- by Yvette on Always show text cursor while moving it in Word [MS]

@Undo we up on 14.04?

MS seems to not be moving at glacier speed

!!/location

@quartata Henders/EC2

Interesting. Was looking at the post scan rates for our instances and Nobody's RPi is really slow

Maybe it was during down time or something else was going on

Like it averaged 3-4 posts per second, that seems impossibly slow

The numbers are deceptive; the raspberry pi has 4 threads in comparison to EC2's 1, so it's closer to about 20

Yeah, those numbers aren't terribly useful for cross-instance comparison. Probably more useful for "did a change make instances slower"

Hi, everybody! I'm new to this chat (the only chat I've been on here so far is the Worldbuilding one).

hi

This might be a little off topic, so tell me if this question belongs somewhere else. I was just wondering whether there are real-life examples of people who have become famous despite their parents originally disliking the career the child wanted.

Specifically, I'm asking because I wanted to think of a story about someone who became a famous video game designer even though his father did not like video games. I'm wondering whether that could ever realistically happen. After all, to be a video game designer requires a lot of study. How might someone be able to get that sort of knowledge without their parents' support?

You're in the wrong place. You're probably looking for Parenting or even Worldbuilding

Okay, @Undo. I was thinking about that. It was just that I wanted real-life examples, and are real-life examples really meant for Worldbuilding?

Don't really know. I'd ask there.

@Undo Ok.