The Comms Room

12:07 AM

I fucking rue the day I installed MS TMG 2010

From 10:30 last night till 1am this morning, and then again from 6am till fucking now it's been fucking up routing and not obeying its own rules

Bob

ouch

Mark Henderson

and what caused all this fucking shennanigans? Adding a new subnet to the DMZ network adapter

And of course undoing the change hasn't fixed the problem

At midnight I decided to fuck it, moved a bunch of addresses to terminate on the TMG box and did NATting to the actual destination boxes

So at least shit was online this morning

I fucking hate this thing

BigHomie

@FalconMomot I hate Java just as much as the next man, but I have to disagree with the bit about OO programming sucking. He's outlined extreme examples of how OOP can get carried away, but for skilled programmers it's awesome, simply put. Hence the reason he's complaining about Java and not the .NET Framework.

Mark Henderson

Its logging shows it as passing the traffic through the interfaces, but a packet capture on the interfaces shows no traffic and the previous hop shows traffic being routed to TMG but nothing ever coming back

Magellan

@MarkHenderson ok. Makes me feel better about our Juniper firewalls.

BigHomie

12:11 AM

although .NET has it's complications as well

Mark Henderson

I think tonight I am rolling this cunts backups out

And scheduling a few hours of maintenance

Restore has been scheduled

I think that's an appropriate restore note

According to the restore logs, this is not the first time this particular VM has been rolled back due to it doing some shitty things

Bob

12:34 AM

It's bugs like this that make me wonder what the Java devs were smoking. Must've been some good stuff.

Also, reported in 2002. Ignored for ten years. 2013? "Fixed in JDK 8."

Gee, thanks.

Magellan

@Bob Because Java Devs. That explains 70% of the crap that I put up with each day.

ewwhite

1:02 AM

Ubuntu question!

0

A: Server won't start. IPMI showing Blue Screen with Ubuntu Linux

I managed to get it back up. This particular 1U server has 12 regular hard drives and an SSD in it. Somehow during a recent 'apt-get update && apt-get upgrade' operation grub got corrupted. The solution was to boot (over IPMI virtual CDROM) the Ubuntu Mini.ISO. I then selected the rescue opti...

Michael Hampton

You should weigh in on the latest meta question.

Magellan

1:19 AM

stupid supermicros.

Michael Hampton

@MarkHenderson You'll love this. The campground WiFi service I run actually helped me answer a question this weekend.

Mark Henderson

@MichaelHampton Haha nice

Michael Hampton

@MarkHenderson ...And it's about Wi_Fi on long distance trains in Australia

Mark Henderson

@MichaelHampton Sounds like pretty poor end-user experience

Michael Hampton

@MarkHenderson Yeah, for his application I can't recommend it. With so much of the web's largest sites on HTTPS now, it doesn't make much sense for me either.

ewwhite

1:30 AM

@MichaelHampton I dunno... I'd need some examples.

Michael Hampton

0

Q: Redis server Memory Increased to 22GB and Crashed (OOM)

A redis server v2.8.4 is running on a Ubuntu 14.04 VPS with 8 GB RAM and 16 GB swap space (on SSDs). However htop shows that redis alone is taking up 22.4 G of memory! I do not think the redis database is this large, so why is it taking up so much memory? Redis version: Redis server v=2.8.4 ...

The first one's fine, the second one....not so much

ewwhite

I don't think either are a problem.

especially since devops love htop and atop

Michael Hampton

@ewwhite I think his point was that he could have copied and pasted, but chose to take screenshots instead.

ewwhite

I would have screenshotted as well.

Jacob

1:46 AM

oooo Fedora 20 is nice

Bob

@MichaelHampton Yea, no, unless a lot has changed in the last 5 years then you're gonna be out of reception for 10+ minutes at a time.

And that's just unusable, caching or not.

Michael Hampton

@Bob Haha, don't I know it. If my service was down, by the fifth minute I would have a crowd at my campsite :)

Bob

I'm scared of long distance country drives - cause if you crash you can pretty much forget about calling for help :\

Jacob

@MichaelHampton Maybe for next year you can get a GoGo ku band sat receiver... 70Mbps/

Michael Hampton

@Bob In the US all the motorways have some coverage, but go off into the countryside and you're pressing your luck.

Or, more likely, the desert

ewwhite

1:51 AM

@MichaelHampton I answered

meh

Bob

@MichaelHampton At least you were only trying to reduce bandwidth usage. Worst case of a cache miss just slows everything down a bit.

Michael Hampton

@Bob Naa, I had an unmetered connection. I was trying to make it faster :)

Bob

@MichaelHampton And here I thought bandwidth meant rate, not caps :P

Michael Hampton

@Bob Well, it sort of worked... last year anyway.

I was quite surprised when I processed the squid log this year and only came up with 744 MB

ewwhite

@MichaelHampton any undelete solutions for ext4?

Bob

1:56 AM

@MichaelHampton Different users? Different uses?

Michael Hampton

@ewwhite extundelete

Bob

I suppose you didn't log the requests themselves.

Michael Hampton

@Bob YouTube went all HTTPS

ewwhite

just had someone rsync over a live filesystem.

Bob

@MichaelHampton That much of a difference? Ouch.

Michael Hampton

1:57 AM

And other websites, too. But no, I am not really paying much attention to what sites people visit. I might log that more closely next year, just to see what in hell is going on.

Mark Henderson

2:07 AM

@MichaelHampton Gunna put that in your privacy policy for your wifi at a libertarian conference?

Michael Hampton

@MarkHenderson Believe it or not I had quite a few people actually read my privacy policy this year.

Mark Henderson

@MichaelHampton How do you know?

Michael Hampton

@MarkHenderson Because I was standing there when they did so.

Mark Henderson

@MichaelHampton Fair enough

Michael Hampton

@MarkHenderson Really I only ever looked at the logs in detail if someone was having a problem. Like being unable to type their password on their phone's on screen keyboard or whatever.

Or...Windows Vista

I'm happy to say, though, the one person who had an ancient XP laptop last year got rid of it.

Mark Henderson

2:21 AM

@Magellan I found the problem with the TMG firewall. Get this. Adding the new subnet to the DMZ network caused the NAT configuration to change the IP address it chose for outbound addresses. The IP address it chose had a 1:1 NAT mapping to a standby link. The standby link was offline, so inbound traffic that came in on the correct interface left on the correct interface, but new outbound connections tried to go out on the wrong interface and got shafted

The most fucked up thing is that there's no reason why adding a new subnet to the DMZ should have touched the NAT configuration at all.

I hate NAT

5

Magellan

@MarkHenderson why the hell does it not use a static IP assignment for its outbound interface?

Mark Henderson

@Magellan It does now

It originally had a pool and some rules to rotate between the pool for things, oh I dunno, working around per-IP address API restrictions

And the address it chose was one not in that pool

Magellan

@ewwhite heh. try rsync'ing BerkeleyDB "database files" that are in live use by the front-end?

So pisses me off when people complain about data being corrupted when they require us to do that with the files live.

Mark Henderson

@Magellan Hah I once was handed a system that I had to manage the application level, but someone else handled the platform. It was doing DRDB replication on a flat file database (ISAM). Of course on the first failover the destination database was corrupt because the logs had been half committed.

Magellan

@MarkHenderson flat file is just plain evil.

Mark Henderson

2:27 AM

@Magellan Pervasive 8.2 ftw

Fun fact: We still support the application but have since taken over the platform as well

Magellan

bbl. screaming babies

Mark Henderson

Almost 20 years since the last modification, perhaps even older.

Falcon Momot

3:27 AM

@BigHomie OOP is a tool that some people can use effectively for those situations where it makes sense.

that's not all situations, either.

Jacob

Can't find flight sim joystick :(

@FalconMomot Isn't that any tool though

Pants

3:44 AM

@MarkHenderson I was 4 years old.

Mark Henderson

@Pants I was 10

dawud

6:27 AM

morning

pauska

morning

Falcon Momot

@Jacob yup! you'd be surprised how many people think that doesn't apply to OOP though.

MichelZ

6:52 AM

good morning perverts

Jenny D

morning

Dennis Nolte

7:36 AM

mornin

Dan

8:44 AM

'sup

@MarkHenderson I thought you were much older than me

pauska

8:56 AM

Am I missing something here? I used robocopy per Microsoft's instructions to pre-seed a replicated folder.. still I get thousands of conflict&deleted event messages when replication kicks in..

Dan

DFRS?

dawud

timestamps? (admitedly, I don't know how can robocopy be configured)

pauska

DFSR ye

dawud

...and I'm thinking a bit in rsync-mode

pauska

I just followed this: technet.microsoft.com/en-us/library/dn495044.aspx

Dan

9:00 AM

Used their command line, I presume?

Dan

9:17 AM

Oh yeah, we went off-road Segway-ing this weekend

Thoroughly good fun if you get a chance, and nowhere near as difficult to drive as I thought given the amount of crash videos out there.

faker

9:45 AM

I like how one of the most requested file on our webapp is example.com/text/javascript

faker

10:06 AM

(which happens because of src='text/javascript' instead of type='text/javascript' ...)

Mark Henderson

@Dan Nah I'm just cynical

@faker Hah I have a similar issue. The top 404, every month, without fail, is a missing favicon

Dan

@MarkHenderson That's just because most browsers try and grab it, regardless

Mark Henderson

@Dan yep

Fills up mah logs and I'm too lazy to filter them

Same as my firewall log has about 1,000 requests a minute for port 6881 - someone long ago ran a bit torrent seeder or something on our IP address and there are clients that are notorious for never giving up trying to contact old peers

pauska

@MarkHenderson how about getting a favicon then

:)

Mark Henderson

@pauska Well I guess that would be one way of fixing it

Dennis Nolte

10:36 AM

simple, either make a favicon or just touch the file.
i was annoyed at that 404 aswell, some script to have touched the favicon file and problem solved.
you could even go that far and autogenerate that file (with no input) in something like a skeleton dir for new website instances.

personally i find it more worse if you know the cooling of the server room will fail soon, but because of managment there won't be any action until it actually fails.

Stack Exchange

10:55 AM

Standalone Sysadmin: Kerbal Space System Administration

posted on July 28, 2014 by Matt Simmons

I came to an interesting cross-pollination of ideas yesterday while talking to my wife about what I'd been doing lately, and I thought you might find it interesting too. I've been spending some time lately playing video games. In particular, I'm especially fond of Kerbal Space Program, a space simulation game where you play the […]

Dan

Ever had a good idea and then wish you'd never mentioned it!

faker

@DennisNolte same here, it actually failed and we had to split our testlab up into a different room. Like a unused office room :/

Dennis Nolte

basically when our is failing, the servers will eventually shut down and i get a shitload of mails/error messages.
still nothing for me to do until i get the OK to call the company fixing the cooling..
its great if that happens friday night.. and over the end of the weekend the servers get stuck because of heat.. Its great when the monitoring system is the first one to fail.

but well, not my fault.. i just deal with the consequences :)

AdamC

12:23 PM

Good morning, as it were.

Transcript for