The Comms Room

Mark Henderson

00:07

I fucking rue the day I installed MS TMG 2010

From 10:30 last night till 1am this morning, and then again from 6am till fucking now it's been fucking up routing and not obeying its own rules

Bob

ouch

Mark Henderson

and what caused all this fucking shennanigans? Adding a new subnet to the DMZ network adapter

And of course undoing the change hasn't fixed the problem

At midnight I decided to fuck it, moved a bunch of addresses to terminate on the TMG box and did NATting to the actual destination boxes

So at least shit was online this morning

I fucking hate this thing

BigHomie

@FalconMomot I hate Java just as much as the next man, but I have to disagree with the bit about OO programming sucking. He's outlined extreme examples of how OOP can get carried away, but for skilled programmers it's awesome, simply put. Hence the reason he's complaining about Java and not the .NET Framework.

Mark Henderson

Its logging shows it as passing the traffic through the interfaces, but a packet capture on the interfaces shows no traffic and the previous hop shows traffic being routed to TMG but nothing ever coming back

Magellan

@MarkHenderson ok. Makes me feel better about our Juniper firewalls.

BigHomie

00:11

although .NET has it's complications as well

Mark Henderson

I think tonight I am rolling this cunts backups out

And scheduling a few hours of maintenance

Restore has been scheduled

I think that's an appropriate restore note

According to the restore logs, this is not the first time this particular VM has been rolled back due to it doing some shitty things

Bob

00:34

It's bugs like this that make me wonder what the Java devs were smoking. Must've been some good stuff.

Also, reported in 2002. Ignored for ten years. 2013? "Fixed in JDK 8."

Gee, thanks.

Magellan

@Bob Because Java Devs. That explains 70% of the crap that I put up with each day.

ewwhite

01:02

Ubuntu question!

0

A: Server won't start. IPMI showing Blue Screen with Ubuntu Linux

I managed to get it back up. This particular 1U server has 12 regular hard drives and an SSD in it. Somehow during a recent 'apt-get update && apt-get upgrade' operation grub got corrupted. The solution was to boot (over IPMI virtual CDROM) the Ubuntu Mini.ISO. I then selected the rescue opti...

Michael Hampton

You should weigh in on the latest meta question.

Magellan

01:19

stupid supermicros.

Michael Hampton

@MarkHenderson You'll love this. The campground WiFi service I run actually helped me answer a question this weekend.

Mark Henderson

@MichaelHampton Haha nice

Michael Hampton

@MarkHenderson ...And it's about Wi_Fi on long distance trains in Australia

Mark Henderson

@MichaelHampton Sounds like pretty poor end-user experience

Michael Hampton

@MarkHenderson Yeah, for his application I can't recommend it. With so much of the web's largest sites on HTTPS now, it doesn't make much sense for me either.

ewwhite

01:30

@MichaelHampton I dunno... I'd need some examples.

Michael Hampton

0

Q: Redis server Memory Increased to 22GB and Crashed (OOM)

A redis server v2.8.4 is running on a Ubuntu 14.04 VPS with 8 GB RAM and 16 GB swap space (on SSDs). However htop shows that redis alone is taking up 22.4 G of memory! I do not think the redis database is this large, so why is it taking up so much memory? Redis version: Redis server v=2.8.4 ...

The first one's fine, the second one....not so much

ewwhite

I don't think either are a problem.

especially since devops love htop and atop

Michael Hampton

@ewwhite I think his point was that he could have copied and pasted, but chose to take screenshots instead.

ewwhite

I would have screenshotted as well.

Jacob

01:46

oooo Fedora 20 is nice

Bob

@MichaelHampton Yea, no, unless a lot has changed in the last 5 years then you're gonna be out of reception for 10+ minutes at a time.

And that's just unusable, caching or not.

Michael Hampton

@Bob Haha, don't I know it. If my service was down, by the fifth minute I would have a crowd at my campsite :)

Bob

I'm scared of long distance country drives - cause if you crash you can pretty much forget about calling for help :\

Jacob

@MichaelHampton Maybe for next year you can get a GoGo ku band sat receiver... 70Mbps/

Michael Hampton

@Bob In the US all the motorways have some coverage, but go off into the countryside and you're pressing your luck.

Or, more likely, the desert

ewwhite

01:51

@MichaelHampton I answered

meh

Bob

@MichaelHampton At least you were only trying to reduce bandwidth usage. Worst case of a cache miss just slows everything down a bit.

Michael Hampton

@Bob Naa, I had an unmetered connection. I was trying to make it faster :)

Bob

@MichaelHampton And here I thought bandwidth meant rate, not caps :P

Michael Hampton

@Bob Well, it sort of worked... last year anyway.

I was quite surprised when I processed the squid log this year and only came up with 744 MB

ewwhite

@MichaelHampton any undelete solutions for ext4?

Bob

01:56

@MichaelHampton Different users? Different uses?

Michael Hampton

@ewwhite extundelete

Bob

I suppose you didn't log the requests themselves.

Michael Hampton

@Bob YouTube went all HTTPS

ewwhite

just had someone rsync over a live filesystem.

Bob

@MichaelHampton That much of a difference? Ouch.

Michael Hampton

01:57

And other websites, too. But no, I am not really paying much attention to what sites people visit. I might log that more closely next year, just to see what in hell is going on.

Mark Henderson

02:07

@MichaelHampton Gunna put that in your privacy policy for your wifi at a libertarian conference?

Michael Hampton

@MarkHenderson Believe it or not I had quite a few people actually read my privacy policy this year.

Mark Henderson

@MichaelHampton How do you know?

Michael Hampton

@MarkHenderson Because I was standing there when they did so.

Mark Henderson

@MichaelHampton Fair enough

Michael Hampton

@MarkHenderson Really I only ever looked at the logs in detail if someone was having a problem. Like being unable to type their password on their phone's on screen keyboard or whatever.

Or...Windows Vista

I'm happy to say, though, the one person who had an ancient XP laptop last year got rid of it.

Mark Henderson

02:21

@Magellan I found the problem with the TMG firewall. Get this. Adding the new subnet to the DMZ network caused the NAT configuration to change the IP address it chose for outbound addresses. The IP address it chose had a 1:1 NAT mapping to a standby link. The standby link was offline, so inbound traffic that came in on the correct interface left on the correct interface, but new outbound connections tried to go out on the wrong interface and got shafted

The most fucked up thing is that there's no reason why adding a new subnet to the DMZ should have touched the NAT configuration at all.

I hate NAT

5

Magellan

@MarkHenderson why the hell does it not use a static IP assignment for its outbound interface?

Mark Henderson

@Magellan It does now

It originally had a pool and some rules to rotate between the pool for things, oh I dunno, working around per-IP address API restrictions

And the address it chose was one not in that pool

Magellan

@ewwhite heh. try rsync'ing BerkeleyDB "database files" that are in live use by the front-end?

So pisses me off when people complain about data being corrupted when they require us to do that with the files live.

Mark Henderson

@Magellan Hah I once was handed a system that I had to manage the application level, but someone else handled the platform. It was doing DRDB replication on a flat file database (ISAM). Of course on the first failover the destination database was corrupt because the logs had been half committed.

Magellan

@MarkHenderson flat file is just plain evil.

Mark Henderson

02:27

@Magellan Pervasive 8.2 ftw

Fun fact: We still support the application but have since taken over the platform as well

Magellan

bbl. screaming babies

Mark Henderson

Almost 20 years since the last modification, perhaps even older.

Falcon Momot

03:27

@BigHomie OOP is a tool that some people can use effectively for those situations where it makes sense.

that's not all situations, either.

Jacob

Can't find flight sim joystick :(

@FalconMomot Isn't that any tool though

Pants

03:44

@MarkHenderson I was 4 years old.

Mark Henderson

@Pants I was 10

dawud

06:27

morning

pauska

morning

Falcon Momot

@Jacob yup! you'd be surprised how many people think that doesn't apply to OOP though.

MichelZ

06:52

good morning perverts

Jenny D

morning

Dennis Nolte

07:36

mornin

Dan

08:44

'sup

@MarkHenderson I thought you were much older than me

pauska

08:56

Am I missing something here? I used robocopy per Microsoft's instructions to pre-seed a replicated folder.. still I get thousands of conflict&deleted event messages when replication kicks in..

Dan

DFRS?

dawud

timestamps? (admitedly, I don't know how can robocopy be configured)

pauska

DFSR ye

dawud

...and I'm thinking a bit in rsync-mode

pauska

I just followed this: technet.microsoft.com/en-us/library/dn495044.aspx

Dan

09:00

Used their command line, I presume?

Dan

09:17

Oh yeah, we went off-road Segway-ing this weekend

Thoroughly good fun if you get a chance, and nowhere near as difficult to drive as I thought given the amount of crash videos out there.

faker

09:45

I like how one of the most requested file on our webapp is example.com/text/javascript

faker

10:06

(which happens because of src='text/javascript' instead of type='text/javascript' ...)

Mark Henderson

@Dan Nah I'm just cynical

@faker Hah I have a similar issue. The top 404, every month, without fail, is a missing favicon

Dan

@MarkHenderson That's just because most browsers try and grab it, regardless

Mark Henderson

@Dan yep

Fills up mah logs and I'm too lazy to filter them

Same as my firewall log has about 1,000 requests a minute for port 6881 - someone long ago ran a bit torrent seeder or something on our IP address and there are clients that are notorious for never giving up trying to contact old peers

pauska

@MarkHenderson how about getting a favicon then

:)

Mark Henderson

@pauska Well I guess that would be one way of fixing it

Dennis Nolte

10:36

simple, either make a favicon or just touch the file.
i was annoyed at that 404 aswell, some script to have touched the favicon file and problem solved.
you could even go that far and autogenerate that file (with no input) in something like a skeleton dir for new website instances.

personally i find it more worse if you know the cooling of the server room will fail soon, but because of managment there won't be any action until it actually fails.

Stack Exchange

10:55

Standalone Sysadmin: Kerbal Space System Administration

posted on July 28, 2014 by Matt Simmons

I came to an interesting cross-pollination of ideas yesterday while talking to my wife about what I'd been doing lately, and I thought you might find it interesting too. I've been spending some time lately playing video games. In particular, I'm especially fond of Kerbal Space Program, a space simulation game where you play the […]

Dan

Ever had a good idea and then wish you'd never mentioned it!

faker

@DennisNolte same here, it actually failed and we had to split our testlab up into a different room. Like a unused office room :/

Dennis Nolte

basically when our is failing, the servers will eventually shut down and i get a shitload of mails/error messages.
still nothing for me to do until i get the OK to call the company fixing the cooling..
its great if that happens friday night.. and over the end of the weekend the servers get stuck because of heat.. Its great when the monitoring system is the first one to fail.

but well, not my fault.. i just deal with the consequences :)

AdamC

12:23

Good morning, as it were.

cole

Morning.

Thunderstorm going on here.

pauska

THUNDER! Huaahuaahaahuhaaaaa

cole

and it's POURING

pauska

@MDMarra ugh, you need discrete graphics in order to use more than two external monitors..

Journeyman Geek

@pauska: er... I think sb and IB will do 3

oh, you're probably talking about laptops, scratch that

pauska

12:37

yah, Lenovo laptops and docks

Nathan C

So one of our backup hard drives went missing...

cole

uh oh

Nathan C

I think I left it on my desk by mistake friday afternoon, but it's not here -_-

cole

Meanwhile in Massachusetts

BigHomie

wassup Comms

Nathan C

12:38

@cole Ha

BigHomie

12:54

@pauska so apprently my issue w/ those machines was that my time was off on my image

Nathan C

@cole Found the drive...it was in the bathroom. Lulz.

BigHomie

Apparently sysprep doesn't reset the clock to pacific time zone w/ no dst

and sccm takes the current time on the image and 'updates' it based on it being pacific time zone w/ no dst :|

cole

@NathanC da faq?!

pauska

Weird :)

@NathanC coworker who misunderstood the concept of digital porn?

Nathan C

@cole After I swapped the drives I went to the bathroom...and forgot to take the drive with me

Score one for "things I don't normally do so I forget about them"

cole

12:59

Whoops

Nathan C

The drives really should be encrypted but synology doesn't offer it

MDMarra

sup sup sup

pauska

HELLO INFIDEL

cole

@MDMarra FFL of 2003 and DFL of 2008 yeahhhhhhhh we're almost current!

MDMarra

2008 or 2008 R2?

cole

13:05

2008

MDMarra

hm

cole

R2 in September

MDMarra

nice

cole

I have to rebuild all the DCs around the world.

So, that's going to take some time.

HopelessN00b

Don't rain on his parade.. he's like halfway to 6 years behind the times.

MDMarra

13:06

So no Recycle Bin yet, but you can flip SYSVOL from FRS to DFS-R

cole

yeah going hold on that

I don't know enough about DFS to be honest, want to make sure I can support it and understand it.

Nathan C

@cole One does not simply understand DFS

HopelessN00b

@cole For SYSVOL replication, you don't really need to understand or support much. It's like FRS replication, but works.

MDMarra

@cole And you do know and understand and can troubleshoot FRS? :)

Because I don't

cole

I'm already overwhelmed

I don't need to add another thing that may break right yet.

On projects alone, my utilization percentage is 120%

That doesn't count tickets, and on-call support.

MDMarra

13:10

UTILIZATION

gaaaaah

cole

?

MDMarra

I fight the utilization battle at work all of the time

what time codes count towards it, which dont, why the fuck does anyone care, etc

cole

I'm pretty sure our Unix admin is going to quit

MDMarra

If we're busy and my team is hitting their goals, then utilization means exactly nothing

cole

so, there's another thing.

Domino admin's last day is Friday.

MDMarra

13:12

Exchange, it is!

cole

lol nope

too many notes DBs that run the company.

My biggest issue when looking for a new job is they want someone who knows Exchange and i haven't touched it since 2003.

HopelessN00b

@cole Stand up an Exchange environment to ~~practice on~~ study the feasibility of migrating off Domino?

cole

@HopelessN00b we won't migrate from Domino, ever.

They already looked at it and it's too costly.

Man I slept all weekend, my body aches.

BigHomie

@cole heh, You guys use Sametime don't you

Nathan C

@cole What about a virtual lab?

cole

13:27

@BigHomie we do.

@NathanC yeah I've been looking at those.

It's like I want Exchange experience but I can't get it without knowing Exchange.

>.<

ewwhite

@cole Exchange isn't too hard once you get past the initial installation.

HopelessN00b

...and the initial install's gotten loads easier since the 2003 days.

cole

I feel like I've just lost a lot of knowledge

or maybe I just don't give a fuck anymore.

AdamC

@cole is height to your advantage during a rainstorm

BigHomie

XD

AdamC

13:31

Sun is out finally, rain was horrible this morning.

HopelessN00b

@cole It's this one. The fuck-giving thing.

cole

@AdamC still a little dark and dreary here

AdamC

►

cole

Huh $15 Match.com Groupon

lol jk

BigHomie

t-5 hopefully until vaca and I'm at my desk enjoying a bowl of grits.

Nathan C

13:38

Pulling data off a dead XP machine to put on ...well...another XP machine

le sigh

MDMarra

bro, don't you even file server?

Matt Simmons

bro, how do I fileserver?

;-)

Nathan C

@MDMarra Folder redirection? You make it sound like we do things the right way here

:p

MDMarra

@MattSimmons Install-WindowsFeature FileServices

I just made up that parameter, not sure what the actual one is off the top of my head :)

Matt Simmons

lol

MDMarra

13:40

@NathanC Folder redirectionnnnnnnnnnnnn

Nathan C

@MDMarra It's more of "computer's toast, hook old IDE drive up to a USB->IDE controller and port data to a new drive...

MDMarra

Or just get O365 and do OneDrive for Business and all of your users get 1TB

Nathan C

It's a bandaid really until we get new machines :/

MDMarra

cloudy cloud cloud cloud cloud

Matt Simmons

weird. It's like you type "Install-WindowsFeature FileServices" and the chat channel converts it into "apt-get install nfs-server"

;-)

MDMarra

13:41

That's how I live my life now

Nathan C

@MDMarra I've been pushing at O365...maybe this coming budget year if I can sell it to them, lol

MDMarra

@MattSimmons NFS. Ewwwwwwwww

ewwhite

NFS is proven.

(snicker)

Matt Simmons

well over here in the Linux world, we don't have fancy Nice CIFS yet

at least, I don't think samba has done SMB3 yet

cole

Samba4 does SMB3

Matt Simmons

13:42

oh, actually

samba.org/samba/history/samba-4.1.0.html

yeah

that's cool

cole

Hey I knew a thing!

2

Matt Simmons

and October of last year. Shows how out of touch I am

MDMarra

If NetApp can do SMB3, I assume that means SAMBA4 supports it

cole

That doesn't happen often.

I only know because of working at NetApp >.>

Matt Simmons

lol

that still counts

MDMarra

13:43

Can you get me some NetApp filers that run 8.2?

I want to play with Project Shift

cole

@MDMarra I can give you my old FAS2040

that's sitting in our datacenter.

it's a piece of shit with one controller.

MDMarra

Does it run 8.2 code?

cole

lol god no

Matt Simmons

In the land of JBOD, even an array with one controller is king

ewwhite

@MattSimmons Oh?

Matt Simmons

13:45

oh?

MDMarra

@cole 8.2 is required for shift

:(

Matt Simmons

I mean. what's the response to that? Oh!

?

cole

Huh, tornado warning

AdamC

@cole just sayin'

Matt Simmons

warning?

crap

cole

13:48

Hey @MattSimmons you said your apartment complex is right outside of one of the Norwood T-stations?

Which Norwood stop?

Matt Simmons

yeah

well

Yeah

AdamC

Cloudy cloud cloud cloud

Matt Simmons

Windsor Gardens

AdamC

Cloud just means your server is in a different building! Durrr! (pokes @MDMarra)

cole

cause the Franklin T-station is a 5 minute walk from my front steps.

Matt Simmons

13:49

the apartment complex has its own train station

cole

Whattt

Matt Simmons

yepo

cole

so it's not Norwood Central or Norwood Depot?

Matt Simmons

nope

Windsor Gardens

cole

Ohhh ok

Matt Simmons

13:49

is the name of the stop

AdamC

Anyone here like O365?

cole

Mine is the Franklin-Dean College stop - not Forge Park

Matt Simmons

ah, ok

cole

I took the ride to South Station alone the other day, it's really not that bad.

Matt Simmons

@AdamC I just subscribed

to O365.

I like the convenience

BigHomie

13:51

@AdamC we rolled it out to the students here, haven't had that many problems with it

AdamC

@MattSimmons for yourself?

Matt Simmons

yeah

BigHomie

they love the space so meh

AdamC

@BigHomie I'm a Google Apps admin, I felt like when I was a Windows admin a year ago I heard too many complaints from people trying to do a real O365+Active Directory setup

MDMarra

@AdamC :(

faker

13:52

@AdamC we use it, I don't like it...

AdamC

Interesting.

MDMarra

@AdamC It ain't hard

BigHomie

@AdamC really? We did ours about a year ago, I didn't handle it personally but the admin who did kept us abreast, migrating accounts over wasn't much trouble, there was only one or two accounts that didn't automatically migrate

MDMarra

DirSync if you need same-sign-on takes 5 seconds to deploy

AdamC

Like I said no personal experience with it.

MDMarra

13:53

AD FS takes like a week to read up on and follow the guide for if you've never done it before

Now, Notes to O365

cole

I was going to say, with ADFS isn't it pretty straight forward?

lol Notes

MDMarra

That is a pain in the dick

cole

Dude Notes in general is a pain in the dick

My dick is in constant pain.

BigHomie

the only thing was that if we deleted an account in our AD, their email in the cloud wasn't recoverable, I think he opened a case w/ MS

AdamC

Sometimes you can chock it up to poor planning I bet.

There was a good Reddit thread though.

MDMarra

13:54

The major problem with people doing their own O365 deployments is that a lot of Exchange admins want to treat it as hosted Exchange and it's really not. It's similar enough to get yourself in trouble but different enough that you'll cause a service interruption if you dont plan properly

AdamC

/me can't find it.

/me doesn't care that /me isn't a thing here.

@MDMarra so when I used to be in that crowd I felt like it was better just to shop for hosted Exchange.

cole

if it was on /r/sysadmin - that's why

MDMarra

Microsoft will give most companies of decent size (that have an EA) $25k+ to bring in a partner to help with migration planning to O365

AdamC

@cole herr hrrrrr

MDMarra

There's no reason not to leverage a partner if you qualify for funding

cole

13:56

I've seen some real dumbassery on /r/sysadmin

AdamC

There needs to be phrase for point and click Windows admins without making fun of regular Windows admins, Evan got mad the last time I made that joke.

cole

Like I thought I was an idiot, but my god.

AdamC

Everybody star that comment please.

MDMarra

@AdamC Depends on what you want. If you're only doing Exchange Online, then it's 6 of one. If you're doing SharePoint/Lync Online, OneDrive for Business, Office 365 Pro Plus, etc there's a value prop there

AdamC

@MDMarra word.

cole

13:57

Well, I am an idiot but they make me feel smart.

MDMarra

Ok, I need to go talk about "training plans" for my team. Sometimes I miss real work. Byebye

cole

later homes

AdamC

Speaking of shitty Reddit threads, you reminded me of /r/talesfromtechsupport which reminds me of Airz23, which is actually pretty epic. This guy started writing about his experiences and people turned it into a web comic and an audio book. It's not one of these "look at these idiots can't remember their 28 character password" it's really funny.

BigHomie

@MDMarra I'm actually interested in what you come up w/, if you don't mind

cole

@AdamC I love Airz23

His writing is actually really good

AdamC

13:58

Yeah.

The people doing the comic adaptation are really good.

Nathan C

@AdamC I usually binge read his stuff every few weeks

Gambette's posts are pretty epic too

cole

I wish these apartments had freakin prices on them

Nathan C

I spend way too much time on this subreddit though...

HopelessN00b

So... turns out that we use some desktop application made by a two-employee company to track our recipe history... and guess what? The desktop got wiped out. >:/

Transcript for