Root Access

Bob

00:00

My latency to the US is lower than Europe

qasdfdsaq

Interesting

I have very little clue how the internet works on the other side of the world

tereško

in PHP on Stack Overflow Chat, 48 secs ago, by EnglishMaster

@choz Sorry, I can't barely speak English. I bought this account on Ebay

qasdfdsaq

LOL

Bob

@qasdfdsaq It asks for a keypair for Windows anyway

Whoops. Looks like Windows doesn't like EBS < 30 GB

:\

> Please wait at least 4 minutes after launching an instance before trying to retrieve the auto-generated password.

:S

:S :S

> In order to retrieve your password you will need to specify the path of this Key Pair on your local machine

DavidPostill

@tereško Hmm. Selling accounts is against the T&Cs

31

Q: Selling Stack Overflow accounts

I was just asked if I was willing to sell my Stack Overflow account. Of course, I won't agree to do this, but I was just wondering if there is any official rule regarding this. Do you guys pretty much hand off our accounts to us for us to do with them as we please? Or is there an official guid...

Anonymous

00:13

@tereško "EnglishMaster"

Anonymous

fokin' lol m8

tereško

it's likely a troll

DavidPostill

@tereško Please don't mention that word in this room ;)

Bob

s/troll/joke/

tereško

huh ?!

Anonymous

00:17

@DavidPostill hey that's trolleism. I'm getting a CM in here to suspend you for discrimination.

Anonymous

We trolls are humans just like you

bwDraco

Uh, what's going on?

Seems to be a joke, but I hope there isn't a dispute here...

DavidPostill

@bwDraco No dispute. I asked tereško not to mention the word troll ...

Given our recent fly-by issues ...

Anonymous

@bwDraco yes, it is a joke. Though I didn't think I'd have to explicitly explain it was one.

Dave

00:39

Stack Exchange accounts on Ebay coming soon, you heard it here first.

That's just too funny.

Having to restore a database on a Sunday evening is not funny though.

I hate my life

Man my rep velocity was sick last week.

30 to 131 in 1 day.. Boing

Let's Sing Along 5 Hours

►

Related to work tonight.

Havin' Fun?!? YEEEAHHH How about another one? Another one... YEEEEAAAH!

Ramhound

01:54

Anyone have the keys to the SE nuke launch system? We have a need for them

-5

A: Is there a keyboard shortcut to highlight the selected text in MS Word 2007?

☺ ok ☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺

bwDraco

Voted to delete and flagged as very low quality.

Anonymous

@Ramhound ☺ flagged ☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺‌☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺

bwDraco

Preparing to submit screenshot to GFSE...

Ramhound

GFSE?

is that SE's spy agency?

bwDraco

GemsFromStackExchange, a Tumblr feed run by Sathya.

Ramhound

01:57

Ahhh

bwDraco

Question protected as well.

!!/wat

ChatBot John Cavil

bwDraco

!!/learn oksmiley <>https://i.sstatic.net/ISs60.png

ChatBot John Cavil

@bwDraco Command oksmiley learned

Ramhound

I think we have moved into the Phase 2 of Comic Book Movies, where a good majority will not be PG-13

Deadpool has earned 300 million

That is really really impressive

bwDraco

02:10

alright ☺☺☺☺☺☺☺ — koreankyle 1 hour ago

☺☺☺☺☺☺☺yep☺☺☺☺☺☺ — koreankyle 1 hour ago

☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺ — koreankyle 1 hour ago

Still LMAO.

Dave

neato picture finally updated. It took about 20 minutes for it to propogate across the network though.

bwDraco

> ☺ ok ☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺‌☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺

Can't stop laughing.

Anonymous

02:41

@Dave nice pic, shame it's overexposed tho

qasdfdsaq

03:05

Did Bob fix it?

!!info oksmiley

ChatBot John Cavil

@qasdfdsaq Command oksmiley, created by bwDraco on Mon, 07 Mar 2016 02:09:09 GMT

qasdfdsaq

!!oksmiley

ChatBot John Cavil

Bob

@qasdfdsaq Fix what?

qasdfdsaq

@Bob Your server

Bob

03:06

The ZFS?

Oh. Pool is fine.

I'm in there now from Ubuntu live.

But I need to get the deb install back up...

It took forever to load the damn iso

qasdfdsaq

Ah, still trying to get it to boot again?

Is this the first time you've done a pool upgrade?

Bob

@qasdfdsaq I think so, yea.

qasdfdsaq

Hmm. Wonder if this sort of breakage is... normal

Bob

I more or less haven't touched ZFS since installation. Well, not the kernel modules anyway.

qasdfdsaq

Built yourself or from PPA?

Bob

03:08

@qasdfdsaq From ZoL's debian repo.

I don't run Ubuntu

qasdfdsaq

Oh there's a repo now?

Ah

Bob

(just using it for recovery cause Debian live images don't support boot from UEFI)

@qasdfdsaq There's been for at least a year, I think. Probably longer.

qasdfdsaq

Mine update themselves once in a while. The kernel modules and SPL anyway. Pool itself... I think I'm deliberately running an older version because I'm stupid and paranoid

Bob

I haven't actually done a pool upgrade.

I haven't touched the pool.

It was just the kernel modules that upgraded.

qasdfdsaq

Oh?

Interesting, I thought you upgraded the pool version cause you were talking about it last week maybe?

Bob

03:09

Pool is fine and alive. I can see everything in there.

@qasdfdsaq I was talking about maybe upgrading it, but then decided it was risk for no gain :P

qasdfdsaq

Never had an update of ZFS itself break the system but again I don't boot from the pool.

Bob

I'm already past the legacy pool versions.

qasdfdsaq

Heh, you decided not to take the risk and it broke anyway

Bob

I'm at the point where it's feature upgrades - so I can basically enable them whenever I need them.

qasdfdsaq

Weird it's the first time you've had an update though. Mine update every 2-3 months AFAIK

Bob

03:11

@qasdfdsaq It's more that it's the first reboot

...I think

qasdfdsaq

Hmm.

Bob

Ok. I still have a clean snapshot.

I wonder if I could just copy the kernel and initramfs off the snapshot...

that should get me back into debian, albeit with an older kernel

qasdfdsaq

> The following packages will be upgraded:
apt apt-transport-https apt-utils base-files biosdevname cpp-4.8 g++-4.8
...
python3-software-properties software-properties-common spl spl-dkms
systemd-services sysv-rc sysvinit-utils udev zfs-dkms zfs-doc zfsutils

Another set of ZFS updates since my last update and reboot.

> root@qasdf-servage:/home/qasdfdsaq# uptime
03:12:58 up 57 days, 7:01, 1 user, load average: 0.25, 0.60, 0.73

Which wasn't even that long ago.

Bob

qasdfdsaq

I really need to sort out some automatic snapshotting

Bob

03:15

same :P

qasdfdsaq

> root@qasdf-servage:/home/qasdfdsaq# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
[email protected]:43 409K - 690K -
[email protected]:46 358K - 792K -
seagate-64t@2015-05-25-23:54 Post-Snapshot 256K - 818K -
seagate-64t@temp1 256K - 818K -
[email protected]:52 383K - 844K -

And so on it goes....

Slowlaris at least had automatic snapshotting built in.

That was awesome

Automatically kept hourly, daily, weekly, monthly, etc. and pruned them as it went.

I'd have to write like, at least 20 lines of bash script to do that myself

And it would still be super kludgy

Bob

> The program systemd-nspawn is currently not installed. You can install it by typing: apt-get install systemd

wot.

I was reasonably certain that Ubuntu 15.10 was a systemd release.

qasdfdsaq

ubuntu live? Which version?

Ah

Bob

crap.

I might have to fall back to chroot

which isn't ideal for this

qasdfdsaq

I knew they were dropping systemd somewhere between 14.04 LTS and 16.04 LTS... Never actually use the intermediate releases myself

Bob

03:18

Wait. dropping systemd?

wtf are they using now?

qasdfdsaq

Or was it upstart

bwDraco

@Bob Upstart, if I'm not mistaken.

Bob

> If you are running Ubuntu vivid (15.04), you can easily switch between upstart and systemd at will since both packages are installed at present. As of March 9 2015, vivid was changed to use systemd by default, before that upstart was the default.

qasdfdsaq

They were moving from something to something else

Bob

@bwDraco Upstart was the old one.

bwDraco

03:19

Odd.

qasdfdsaq

Yeah my brain isn't working right today

Bob

Might have something to do with it being 3 AM

qasdfdsaq

3 hours ago, by Bob

@qasdfdsaq Good point. I'm not braining well today.

Bob

urk. chroot it is.

qasdfdsaq

Or you're just contagious

Bob

03:20

possibly.

I wonder how much ec2 has cost me so far

probably mostly networking charges

qasdfdsaq

Even the highest instances are only a few dollars per hour

I did once manage to get a box with multiple Tesla cards and 16 CPU cores all to myself for less than $0.01

Yay for pricing mishaps.

Bob

@qasdfdsaq Yea. Spot m3.medium was something like 0.05

I just bid 0.1 so the chance of losing it is minuscule :P

qasdfdsaq

Surprised you haven't managed to get hold of quixy yet

Bob

haven't really tried

I'm already neck deep here

bwDraco

Spot instances can be shut down without warning. Be sure your application is designed to cope with sudden termination of the underlying VM.

Bob

03:23

@bwDraco ...yes, I am very aware of that, thanks.

bwDraco

...or a two-minute warning, it seems.

qasdfdsaq

I'm surprised neither of us has mentioned using an OVH cloud instance yet :-P

Bob

@qasdfdsaq Can those be started/stopped on demand?

qasdfdsaq

Yeah

Bob

dammit

probably should've done that

would probably be faster too

qasdfdsaq

03:24

They've been pushing cloud services to compete with the likes of EC2 quite hard a few months ago.

Probably because nobody ever thinks of them

Bob

0.221/hour

costs a fair bit more

bwDraco

aws.amazon.com/blogs/aws/…

qasdfdsaq

I participated in the beta a few years back

Bob

but it's within their network so the ipmi iso sharing should be faster

qasdfdsaq

It was like $2 a month for the bottom end ones then

bwDraco

03:25

Be sure your application checks every few seconds and closes gracefully once it detects that the instance is about to shut down.

qasdfdsaq

> 2 vCores x 2.4 GHz
30 GB RAM 200 GB
HA block storage 250 Mbps
Bandwidth
£25 excl. VAT/month*
£0.068 excl. VAT/hour

Where did you get the 0.221 from?

Bob

@qasdfdsaq Windows

qasdfdsaq

Ah

Bob

and $, not £

oh the cpu instances are cheaper

wait. 30 GB RAM O_O

even the EG-7 CPU instance has 7 GB RAM

qasdfdsaq

Yeah exactly

Chuck the whole ISO in a RAMdrive

Bob

03:27

that's so overkill for me it's not funny

bwDraco

Fully-elastic cloud platforms like AWS is designed with the assumption that your application is designed to scale automatically.

Bob

@qasdfdsaq eh, it's not the drive that's slow

the ipmi impl seems to request blocks on demand rather than preloading it

qasdfdsaq

Well, booting off ISOs mounted over IPMI clients is very much latency bottlenecked

Bob

my server has plenty of space for a ramdrive, but there's no way to do it

qasdfdsaq

> VPS SSD 1
OpenStack KVM
1 vCore
2.4 GHz
2 GB RAM
10 GB SSD
Local RAID 10
£2.49 excl. VAT/month
(or £2.99 incl. VAT)

Bob

03:28

heh

but setup time

qasdfdsaq

That's the ultra "Low end"

bwDraco

They're designed such that the storage is completely decoupled from the machine itself.

Bob

I just needed it now, and it's pretty much a one time thing

qasdfdsaq

Enough for hobbyists with their first Frontpage site or whatever

bwDraco

Traditional VPS cloud platforms have storage, RAM, and compute resources bundled and are best for more predictable workloads, but are much less complicated to work with.

qasdfdsaq

03:29

Well even OVH's bare metal hardware machines are automatically provisioned, I'm pretty certain their virtual instances are too

bwDraco

Linode typically creates new instances in less than a minute.

Deploying an image and starting the instance only takes a few minutes.

Bob

@qasdfdsaq With a Windows install ready to go?

qasdfdsaq

> Desktop Operating systems
Ubuntu 14.04 Desktop
Windows 2012 R2 std (Cloud range only)
OVH Release 3

heh

Bob

My dedi took over half a day to provision iirc

qasdfdsaq

Well, yeah. Since Windows is only available on the "start on demand" Cloud services

Bob

03:31

ah

bwDraco

@Bob Dedicated servers are typically set up by hand, unlike cloud systems.

qasdfdsaq

There's not much Amazon can do that OVH can't

Other than having five-nines global resilience.

> EG-7 2 vCores x 3.1 GHz
7 GB RAM 200 GB
HA block storage 250 Mbps
Bandwidth
£35 excl. VAT/month*
£0.098 excl. VAT/hour

That's for the Windows ones, so, yeah about $0.14 USD or $0.19 AUD

Lol OVH's Hubic cloud service seems to be down

Bob

4 mins ago, by qasdfdsaq

Well even OVH's bare metal hardware machines are automatically provisioned, I'm pretty certain their virtual instances are too

@qasdfdsaq Considering the power failure that took out my server also apparently affected their cloud offerings ... yea

@bwDraco deficient?

bwDraco

I meant your system taking longer than expected to set up.

qasdfdsaq

s/def/eff/

bwDraco

03:35

I'm very tired.

Bob

Have you tried sleep?

qasdfdsaq

Urg. SYS still haven't updated their first-gen 2009-era Xeon servers.

Bob

lol...

bwDraco

@Bob So why did it take so long?

Bob

@bwDraco Hell if I know.

They also apparently knocked an "electric connector" loose and had to fix it. That took another day.

hm... this isn't good

qasdfdsaq

03:37

LMAO. They managed to fuck up their "electric connector" quite a bit.

bwDraco

Wow. Absolutely stupid. Machines should be provisioned automatically these days.

Bob

I wonder if it'd work if I just update the initrd...

bwDraco

All this talk about lights-out datacenters...

Bob

ok. moment of truth. *reboots*

qasdfdsaq

I'd be looking more at trying to get a dmesg log from a failed boot to see what's going on first

> HOST-64H Intel Xeon D
D-1540 8c/16t
2,0 / 2,6 GHz 64 GB of RAM
DDR4 ECC 2133 MHz 2 x2TB SOFT vRack 10 Gbps *
£83.99 excl. VAT/month

Tempting. Still feels too expensive, despite the fact I'm no longer paying that monthly for my mobile phone, so it'd effectively be... zero net cost.

Bob

03:39

@qasdfdsaq That's hard.

Mostly because I don't even have access to the rootfs at that point.

@qasdfdsaq ...that seems to cost more than my SP-64

qasdfdsaq

But if you can boot it over IPMI can't you just dump some debug flags into the bootloader?

Bob

hm. maybe?

qasdfdsaq

@Bob Your SP-64 has half the cores

Bob

ah.

> Btrfs loaded

wut.

qasdfdsaq

No wonder. btrfs decided to silently substitute itself for zfs.

Bob

03:40

oh whoops... forgot to export the zpool

qasdfdsaq

Taking over your systems!

Bob

Nup, ZFS modules still not loaded

oh look module zfs not found now

dunno if that's better or worse lol

back to ubuntu we go

qasdfdsaq

Didn't you say you had your own weird manual script to put ZFS into the initrd?

Bob

@qasdfdsaq yea, it's in that screenshot

oh, it's just to copy the initrd over

qasdfdsaq

Oh I see

Speaking of, time to update mine and reboot

Bob

03:45

good luck

qasdfdsaq

While I'm at it I should probably update my public webserver as well, seeing as it's a lot more exposed than my home media box

Course I have no idea what the *****ing password is

Bob

> I am a maroon

qasdfdsaq

@Bob He also wants to RTEFM?

> Errors were encountered while processing:
linux-image-extra-3.13.0-46-generic
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@s4a:~#

Urgh fuck you Ubuntu

Urgh who keeps starring my misfortunes. I bet it's the secret flagging lurker innit

Bob

@qasdfdsaq Well, he is a maroon.

qasdfdsaq

Are you actually getting to an initramfs prompt at least or just a glaring wall of blackness?

Bob

03:53

@qasdfdsaq Oh, I get dropped into a shell.

I think my next step is going to be pulling the kernel and initrd from te snapshot.

That would make the second time this snapshot has saved my arse

bwDraco

theregister.co.uk/2016/03/06/amd_microcode_6000836_fix/?page=1

qasdfdsaq

And you can't get at dmesg from there?

@Bob Heh, that sorta recovery is exactly why I wish you could run Windows from ZFS

Bob

@qasdfdsaq hm... I just heavily suspect it'd just end up telling me that the zfs module is missing

qasdfdsaq

> root@qasdf-servage:/home/qasdfdsaq/ddrescue# cat ddrescue.sh
#!/bin/bash
DATE=$(date +%Y-%m-%d\.%H%M)
ls -1 /dev/disk/by-id/ | grep ata- | grep -v part | while read -r line; do smartctl -x /dev/disk/by-id/$line | tee "/home/qasdfdsaq/smartctl/smartctl ${line:4} ddrescue-start $DATE.txt"; done

ls -1 /dev/disk/by-id/ | grep ata- | grep -v part | while read -r line
do
ddrescue -B -d -v -f /dev/disk/by-id/$line /dev/null ~qasdfdsaq/ddrescue/ddrescue-$line-$DATE.txt &
done

ls -1 /dev/disk/by-id/ | grep ata- | grep -v part | while read -r line; do smartctl -x /dev/disk/by-id/$line | tee "/hom…

Lol. I use my maintenance scripts so infrequently that I have to read the contents every single time to figure out which script it is and to make sure it does what I think it does

Bob

o.O

bwDraco

04:00

<!--
       .o                                .o88       .o88
      .88                                "888       "888
    o8888oo  ooo  oooo  ooo. .oo.  .oo.   888oooo.   888  oooo d8b
    ""888""  888  "888  "888P"Y88bP"Y88b  d88' `88b  888  "888""8P
      888    888   888   888   888   888  888   888  888   888
      888 .  888   888   888   888   888  888.  888  888   888    .o.
      "888Y  `V88V"V8P' o888o o888o o888o 88`bod8P' o888o d888b   Y8P

                                                                        -->

Found this in the source code of GFSE, lol

unwrapping.tumblr.com/post/55143439090/tumblr-html-ascii-art

qasdfdsaq

The above is my surface-scan script, I used to run it weekly and had a nice colourful spreadsheet to track any changes in my drives' health metrics over time.

bwDraco

9

Q: Ascii art in HTML comments - fad or function?

Lately I've seen several websites with large ASCII art headers in their HTML. I don't get the point and it seems like this must just be a fad. For instance, tumblr.com has it: <!-- . .o8 oooo .o8 "888 `888 ...

html tumblr ascii-art

qasdfdsaq

Showed clearly how many retries each drive needed to read through for example, and any slow sectors, yadda yadda. But these new drives have been so reliable none of them have had a single bad sector in the several years I've had them. So I end up only running the check once or twice a year.

Bob

ah

qasdfdsaq

3.8 GB/sec. Yum

Ever since "the chmod incident" I've just been extra paranoid about running the wrong command/script/etc. anytime I'm root.

Plus, reading each script before I run it gives me extra time to admire my kludgey coding handiwork.

Bob

04:08

I don't like how easy it is to use zfs destroy

one of these days I'm going to accidentally nuke the wrong FS

qasdfdsaq

alias "zfs destroy"="echo Are you sure?!"

(Also zpool destroy is just as dangerous, although you can often undestroy)

Bob

@qasdfdsaq I don't type that nearly as often

qasdfdsaq

And like chmod, and rm, you definitely don't want an extra -r in there when accidentally destroying the wrong thing

Eh, I virtually never type zfs destroy into a console

(Another benefit of having automatic snapshot management is that you never have to at all... you can even remove the ability to use the command if you so desire)

Y'know that'd be a cool idea. Remove the ability to use the zfs destroy command from your user, and instead, make it so the only way to destroy filesystems is to write the name of the filesystem into a text file somewhere, and have a cron job that runs daily to check the file and then schedule a destroy 24 hours later

The cron job can also email you/all admins on a system and/or delay the action pending the uploading of a certified sobriety test

Only once all the above conditions are satisfied, plus a cryptographically secure authentication code from a secondary authorizer is provided, will the actual delete go ahead.

Cause to be fair, the "nuclear option" on your data needs to have all the same safeguards as an actual nuclear bomb launch IRL.

Eh. By mrain has clearly gone wonky. I should go to bed. G'night @Bob

By mrain? My brain. Point prvoen

Proven. Urgh.

Bob

04:28

@qasdfdsaq 'night

Bob

05:48

github.com/zfsonlinux/pkg-zfs/issues/181

ffffffffffffffffffuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu‌uuuuuuuuuuuuuuuuuuuuuuuu

2

@qasdfdsaq ^

Also, @allquixotic, ^ may be worth reading. Ubuntu is insulated from that (because they have zfs-native) but if you ever decide to fiddle with debian & ZFS ... make sure you have the right SPL version :\

such a massive waste of time...

maybe I should go ahead and set up that recovery environment on the flash drive :\

Bob

06:37

hm. not done yet. now fuse is broken

Bob

07:19

:O

so close

cmon dev-zvol-rpool-swap.device, load dammit

I'm ... in?

on the login screen but typing doesn't work :S

Bob

07:35

There we go. Had to restart the ipmi console

Burgi

08:48

there are a lot of random chat stars this morning...

Transcript for