@ThomasWard / @Undo Just dug up the redis setup guide. Should still be accurate.

I can also talk through / ssh in and do things if needed?

[ SmokeDetector | MS ] Username similar to website in answer (60): Position fixed CSS3 by String Manolo on es.stackoverflow.com

@thesecretmaster Guessing there's special config for us running two Redis instances, right?

(running through guide now though, thanks for digging that up... I could see it in my head but couldn't find it)

@Undo Oh yep, that's true... I'm not fully sure how we did that actually. I think there's a config.yml file that has info for both?

@Undo Me too, found it via the team which had a link to the GH wiki. No idea why there's no better way to get to that page on the wiki

@thesecretmaster kk, gonna take a shot at setting it up

Really it's just two Redis instances running on 2 ports

Shouldn't be impossible

Yep

Oh yeah, redis should be very fine with it

I just don't remember how to tell MS which is which

Also pretty sure I wrote systemd jobs for them

So maybe try and steal those from the current instance?

Already stole config files, yep

Do you know the easiest way to get current Redis dumps and move 'em across?

Heard of .rdb files somewhere

Yep

You can get daily (hourly?) ones from the db dump page

on MS

if you want the live ones, you'd have to kill the running MS and then copy them over

Then start MS on the new instance

[ SmokeDetector | MS ] Few unique characters in body (94): grub-efi-amd64-signed package post-installation script subprocess returned error exit status 127 ✏️ by Vearax on askubuntu.com

@thesecretmaster And those are full, not with private stuff censored out or anything, right?

90% sure redis doesn't touch any PII

Buuut

You'll want to bring down the current instance of MS, copy the files over, then start the new one, same as the with the mysql DB or you'll get consistency issues.

Yep

that's the plan

Once it's ready to go, we have to do a bunch of DNS magic anyway

Yeah

Nice big shutdown window

Prolly the best plan is to point new MS at old redis & mysql, wait for the traffic to fully transition over, then have downtime and copy the databases.

Yeah, I was thinking an even harsher approach

Just have 20min of flat downtime

service nginx stop on the old, copy SQL + Redis backups over, do DNS, bring it up on new server

Dunno if that's enough for DNS to propagate. I always hear that a good number for that is 24-48 hours.

Although I have no practical knowledge so don't trust me

I set TTL on DNS to as low a number as I can get

Should help a bit

But there'll be pain, of course

Yeah...

Are the IPs hardcoded anywhere?

I doubt it, but who knows

We haven't changed IPs in forever

Actually no they can't be

Current implementation of nginx wouldn't recognize a straight IP

IIRC

Requires the hostname to be in the request

I think it defaults to some server block by host name

I forget

there's a default_server directive

undo@metasmoke:~/zhregex$ make
clang -o module.so module.c -g -fPIC -shared -std=gnu99 -I. -Wall -Wextra -Wno-unused-parameter -lc -lpcre
make: clang: Command not found
Makefile:18: recipe for target 'module.so' failed
make: *** [module.so] Error 127
undo@metasmoke:~/zhregex$

Ah

Well, you can change the Makefile to point at gcc instead

Should compile with either

Or you can install clang

ah

yep

Bunch warnings but who cares

All unused parameters, meh

They should be fine

I mean it's what we're currently running

I've gotten so much better at C then when I wrote that code...

:'(

How do I import dump.rdb?

It's pointed to in the config file

@Undo this might be a good time to move it to metasmoke.charcoal-se, while we're at it

I think it's like /etc/redis/redis-*.conf or something

@ArtOfCode Might as well have both domain names, at least

Bah, gotta run, but thanks @thesecretmaster, that helps a lot

light at end of tunnel in sight

@Undo Alrighty, if you're back later feel free to ping me and I can walk thru the rest.

[ SmokeDetector | MS ] Bad keyword with email in answer, pattern-matching email in answer (184): TypeError: Cannot read property 'chat' of undefined, problem inline keyboard by Samri nazeer on stackoverflow.com

[ SmokeDetector | MS ] Username similar to website in answer (60): `us altgr-intl` layout on Windows 10 by Pieter_Degroote on superuser.com

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): The meaning of 気もない by JohnSmith13345 on japanese.SE

fp- feedback received on us altgr-intl`us altgr-intl layout on Windows 10 [MS]

tpu- feedback received on TypeError: Cannot read property 'chat' of undefined, problem inline keyboard [MS]

!!/watch-force- info\.acefinanceltd(?:@gmail\.com)?

@Makyen That pattern looks like it's already caught by Pattern-matching email in answer and Pattern-matching email in body; append -force if you really want to do that.

[ SmokeDetector ] Blacklists reloaded at rev 05fa2fb (SmokeDetector: Auto watch of info\.acefinanceltd(?:@gmail\.com)? by Makyen) (running on teward/Isis)

[ SmokeDetector | MS ] Mostly non-latin answer (49): Fatal error: Class CI_Session_files_driver contains 1 abstract method by Jawad EL Marzouqy on stackoverflow.com

[ SmokeDetector | MS ] Repeating words in body (43): Why isn't finala not initalized? by Izvan on stackoverflow.com

tpu- feedback received on Fatal error: Class CI_Session_files_driver contains 1 abstract method [MS]

@SmokeDetector This is just a translation of the error which is in the question.

@SmokeDetector Actually, it looks like it's just a Google Translate of the question's error text into Arabic.

fp- feedback received

[ SmokeDetector | MS ] Potentially bad ns for domain in body (1): can the decimal precision of token contract be changed after launch to mainnet? ✏️ by Corey Cottrell on eosio.SE

@thesecretmaster Okay, so on the old server I've got a /etc/redis/redis.conf config file

But the directory doesn't exist on new server

Gotta apt install redis, I suppose?

Or actually probably install it from the built version?

So, I think I installed redis over the system version on the old server

Like copied the executables from redis-git/bin/ to /bin/ or wherever they want to go.

kk

Although that may not be the best way to do it

Is the best path to install a system redis & then copy over?

Hah, beat me to it

You'll want to copy redis-server and redis-cli

Eehhh

ah

redis.io/topics/quickstart

Well, if you install the packaged version under it, if it gets updated, it'll overwrite

sudo make install

There ya go

fp- feedback received

fp- feedback received on The meaning of 気もない [MS]

tp- feedback received on grub-efi-amd64-signed package post-installation script subprocess returned error exit status 127 [MS]

naa- feedback received on Position fixed CSS3 [MS]

fp- feedback received on How to fight browser fingerprinting? [MS]

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): How can I disable a pass code that I didn't intentionally set? by Mian Jiang on apple.SE

!!/watch- stringmanolo\.ml

tpu- feedback received on Transición fade al cargar la página [MS]

tpu- feedback received

[ SmokeDetector ] Blacklists reloaded at rev 86169a8 (SmokeDetector: Auto watch of stringmanolo\.ml by Eliah Kagan) (running on teward/Isis)

fp- feedback received on Why were so many more Jewish men than Jewish women killed during the Holocaust? [MS]

fp- feedback received on How do I change the default google search country for the location/search bar in Firefox 24+? [MS]

ummmmm

why is my MS dump only four megabytes

this seems wrongish

naa- feedback received on Script to extract selected entries from a bibtex file [MS]

@Undo Redis or mysql?

naa- feedback received on Spotify API: How to get access token for only myself [MS]

fp- feedback received on принцип работы паттерна Monostate [MS]

fp- feedback received on using 'and' / 'or 'inside while structure. python [MS]

@thesecretmaster SQL

@Undo The last one I downloaded was 107MB for the MS dump and 153MB for Redis.

The one from the public dump page

@SmokeDetector Still around.

Yeah, 4mb is wrong

@Makyen That sounds about right

fp- feedback received on Is it possible to set dates across a row of cells where each cell date counts back different amounts of weekdays from a completion date? [MS]

fp- feedback received on Playing music over Bluetooth in Android car unit that only accepts Bluetooth phone call [MS]

bahahaha

gotta unzip it

redis doesn't like raw gzip files

!!/watch- creditquickfix2019(?:@gmail.com)?

@EliahKagan That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer; in addition, the regex contains an unescaped "."; in most cases, it should be "\."; append -force if you really want to do that.

(that's not the SQL issue though)

@thesecretmaster i'm lazy and haven't set up an account for you though :/

i do have your key though :)

@ThomasWard No need right now

Eh, it's fine if Undo can just do it

I thiiiiiink I almost have it

No reason I need access

!!/bisect creditquickfix

mmkay

@EliahKagan Matched by creditquickfix\d*(?:@gmail\.com)?(?#seen: creditquickfix2019@gmail.com) on line 15200 of watched_keywords.txt

@thesecretmaster cough future redis maintenance Cough

If I need it at some point later I can ping you

Or you can give me access, if you want to

I have your key too, can add it to my account if I need to

^ that would work

Are we still going to have a way to reboot MS? It's been very helpful to be able to do so.

or I can give you your own

Hah, got redis on :6379 to work

now it's just the logging one

@Makyen If not, we should

@thesecretmaster how do we currently have it operating?

@Makyen Hoooopefully we won't need that nearly as much, there's a lot more headroom here... Thought if we do, not sure how it'd work

@ThomasWard Currently there's a lambda service that just reboots the underlying EC2 instances

@Undo We can hope, but ... :-)

Yeah

Exactly

No one has ever accused MS of being inherently stable

I feel like it should be fine?

Understand that I've been trying to run it on a ridiculous budget, though. This should be far, far more power than it's ever had

Also having a competent sysadmin involved might be... good

But I think this is a question for @ThomasWard for how it could be implemented. My thought would be a second instance that can reboot the first.

or ultra lightweight service on the first with highest priority

That's riskier, but yeah

lets get the new MS up before we work out the force-reboot mechanism

^

I can think of a few ways to make it work, but lets get the 'force reboot' part after we get the new instance working :P

Also depending on how Thomas has it set up, it might still make sense to keep redis, mysql, and the application on different vms

@thesecretmaster currently Undo's been spinning it all on the same one

Yeah, I guess this is getting a bit ahead of ourselves here

if we're going that route though we may as well spin them as independent LXD containers within the same VM

they can be restarted independently of each other

but again, we do have a lot more CPU and RAM power running this here.

Isn't nested virtualization Bad?

so that should help a bit

fp- feedback received on Change Windows explorer font size [MS]

it can be, but i've got a lot of things (Isis and Osiris even) in LXD containers on a VMware system

LXD isn't full virtualization either

it uses linux containers instead of full VMs

so it' snot like we have a QEMU instance inside a VMware VM

Ok. I mean, you definitely know more about this than I do so I trust you to know what the best thing to do is.

i think the core issue with the multiple EC2s was because of free tier or cost restrictions

it's up to @Undo actually who's been spearheading the migration

not sure how far it's gotten so far.

I thought it was cost and/or single point of failure stuff.

Aaaactually

once we get the instance fully ready to go

@ThomasWard alllllmost done

i can clone the VM and we'll have a backup

Where "done" is "can probably be switched to, then we figure out database backups and that boring stuff"

but as for the DB if we WANT to migrate the DB i can

While we're talking about this, how do we feel about having multiple rails containers, 1 redis and 1 sql?

I think that'd fix a lot of the downtime concerns

Inclined to try it as all services on the same box for a bit

Easy enough to split out if we need to later

True, but it'd be wonderful to be able to let one rails instance die and still have MS be up

well

if we move the Redis and DB into its own individual box

we can have two MS rails boxes

HAProxy can failover to the second when the first goes AWOL

Oh wait

but then we still ahve DB/Redis backup concerns

Could we use HAProxy to do health checks and auto-reboot MS?

we could

though I'd almost rather write my own watchdog service instead

since HAProxy isn't designed to do that.

But there's a few ways we can approach it.

Or the failover thing works too, then someone can manually reboot the dead one

@thesecretmaster there's a few different ways to approach it

but again lets make sure we get something stable before we consider failover

Yeah, you're right

Okaaaaay. There's some chance this is ready to go.

@thesecretmaster if we can get MS functioning stable first, then we can do failover and split DB, etc.

SQL, Rails, 2x Redis, theoretically good to go

want me to snapshot it now before we have fun testing?

That'd be sweet, yeah

oooh

this'll be interesting

you don't have IPv6, right?

Not sure how removing an AAAA record propogates

v6 is 'available' but not configured because Comcast's a bitch

removing an AAAA record just means AAAA has no responses

but we do have the test domain, if yo uwant to use that first with a singular instance

Yeah, it's running there

ms-migration.erwaysoftware.com

unrelated, the disk snapshot is complete

i didn't snapshot memory but i snapshotted the disk

that's all it needs

ok. do we want me to throw Isis or Osiris to point to this instance first?

Looks like Redis setup fixed the web socket issue I was talking about earlier

@Undo Heh, sign in redirects back to metasmoke.erwaysoftware.com

make sure it works with sockets?

@thesecretmaster Yeah, redirect URIs

@thesecretmaster didn't on my side o.o

room topic changed to Charcoal HQ: Where diamonds are made, smoke is detected, and we break things by developing on production. 150k true positives and counting. For more information, see charcoal-se.org/about. Handy links: metasmoke.erwaysoftware.com, github.com/Charcoal-SE [dev-on-prod] [panic-driven-development] [plastic-knives] [programmatic-crowbars]

@ThomasWard I'm pretty sure it'll work fine

@Undo Is that an AppConfig thing or a stackapps thing?

WS works in browser, no reason instances would be different

@thesecretmaster both

Ah

SA has a registered redirect URI

Well, registered redirect domain