The Comms Room

Iain

20:00

it's ok that's mild by the comms room (low) standards

voretaq7

@JoshGitlin Set your firewall to drop everything on the floor for 10 minutes?

Josh Gitlin

LOL, but as a mod elsewhere I should know better :-)

MDMarra

Yeah, Iain's right. Nice to meet you.

voretaq7

Also how do they bill you for bandwidth?

Josh Gitlin

@voretaq7 That's what I was asking in my question... but I don't believe ESXi 4 has a firewall

MDMarra

20:01

Oh, you're a mod elsewhere............ to the edit button

voretaq7

@MDMarra That's how I would describe @JoshGitlin's provider, yes...

Josh Gitlin

@voretaq7 Monthly, I get 500 GB in + 5000 GB out

@voretaq7 Same here!

MDMarra

;)

voretaq7

@JoshGitlin OH it's an ESXi box... yah you'd have to stop (or disconnect) all the guests, but if they're hitting your management IP too you're screwed...

Josh Gitlin

I'm just shutting down / ifconfig eth0 down ing all VMs until I can find the cause, so far no luck

@voretaq7 they might be, can't tell

MDMarra

20:03

If it's an ESXi box and you want to pull the cable, but can't...why not just disable the vswitch that the VMs are on?

same thing

Let it sit for a couple of minutes and then turn it back on

voretaq7

@JoshGitlin If your management IP is on a separate NIC just kill the NIC with the guests on it

MDMarra

See if the attack goes away

@voretaq7 stop stealing my goddamn ideas

I have too few of them, they're precious.

voretaq7

if the attack tapers off you know it's not hitting the management IP (then you just have to isolate which guest(s) are targets)

@MDMarra the precious?

MDMarra

basically

Josh Gitlin

@MDMarra Great idea, thanks!

MDMarra

20:04

Do you have vSphere Enterprise?

Josh Gitlin

DO you know if I can do that through SSH? Because the vSphere console is useless

MDMarra

If so, you can replace the existing standard vSwitch with a Cisco V1000 and do a virtual port mirror and watch the traffic with that

Josh Gitlin

@MDMarra no, sadly, vSphere 4 ESXi free

MDMarra

@JoshGitlin Um, I'm sure you can do it through SSH or PowerCLI. I just have no idea how off of the top of my head.

Josh Gitlin

We're a small company, which is why we don't have two thirds of what I want to have :-(

voretaq7

20:06

@JoshGitlin you can do it with their Perl interface but that involves writing some scripts -- pain in the ass it is

Joel E Salas

@JoshGitlin There's always more to want

MDMarra

@JoelESalas Unless you buy Enterprise Plus. Then you're the king of the world

Joel E Salas

@MDMarra If you have that kind of money, why are you wasting it on VMware instead of going to space

MDMarra

I love space

edu pricing for it isn't bad though

8 sockets of enterprise plus is only like 20k I think including support

Josh Gitlin

I can't believe they can't just unplug me. WTF do I pay them $1000/mo for!?

Joel E Salas

20:12

@JoshGitlin So they can have enough ivory back-scratchers

Josh Gitlin

oh, of course!

Adrian

@JoshGitlin Executive lapdances, I bet.

Joel E Salas

Is there a special "I'm not a douchebag" checkbox I can set somewhere in Pandora? Its science-fair experiment of a matching algorithm plays me shit like Staind now and again

Josh Gitlin

OK well this tech support guy thought Apache runs inside VMware ESXi and he could ssh in to the host and stop Apache. Wow. But he also did say he could have them unplug it for me

Adrian

@JoelESalas That's why I quit using Pandora. They wander off into all sorts of strange directions with that algorithm.

voretaq7

20:21

@JoshGitlin seriously, they won't send a guy in to pull the ethernet cable, or down your port on the switch for 10 minuts?

I mean THEY have to deal with this traffic too - a spike that big would make me move my ass (and have someone blackholed) back in my ISP days

@JoshGitlin well he's competent enough to know that the bits travel through the cable... that's a plus :)

(also he's TECHNICALLY correct Re: the service console's web server, but that's probably not your problem :-)

ewwhite

Still interviewing....

MDMarra

I can only do one of these three things and I'm not sure which one I should do:

Upgrade my MBA's SSD to a 256GB model

Purchase a 2TB Time Capsule

Purchase a 27" 2560x1440 IPS Korean monitor to replace my current 22" at my desk at home

They're all roughly the same price

ewwhite

Getting deep. Looks like I missed @ethabelle....

Joel E Salas

@ewwhite Try to freshen up before the swimsuit competition

MDMarra

@ewwhite How long has the interview been?

ewwhite

20:24

@mdmarra don't do the time capsule...

Live been interviewing places today since 9am EST

MDMarra

I have a DLINK DNS323 with funplug and the AFP module installed so I can do unsupported time machine backups to that

Josh Gitlin

@voretaq7 This second guy just unplugged me from the switch, fingers crossed

Justin Dearing

I'm going to wait a little bit to see if Jeff Atwood livetweets a house fire before buying Korean displays.

3

MDMarra

but it fails too many times and has to do a whole new 100GB backup over the air like every other week

Josh Gitlin

@voretaq7 They hate to deal with the traffic unless they get to bill em for it, lol

voretaq7

20:25

@JustinDearing I'll stick with my Chinese and Taiwanese displays

@JoshGitlin depends on how big their pipe is

Josh Gitlin

@voretaq7 Oh, true, he is correct there. But he was looking for a cPanel Apache server, I know these guys too well, lol

MDMarra

@voretaq7 thats what she said

voretaq7

some ISPs a 60Mbps spike would seriously endanger their margins.

Josh Gitlin

@voretaq7 liquidweb.com/datacenter/network.html

voretaq7

@JoshGitlin but they don't tell me the link speeds, weights, and (fiscal) costs

Justin Dearing

20:28

@voretaq7 My philippino wife has an affinity for Korean spas and korean soap operas . I like korean food. I sayI need a korean monitor

Josh Gitlin

@voretaq7 yes, true

voretaq7

@JustinDearing probably remembers dealing with the CA DDOSes (though that was all tiny packets, almost no bytes) -- If you saturate your fast, cheap link you wind up routing traffic over the more kerspensive ones

so what they're charging you may not make up for their transit bill that month

ewwhite

@MDMarra Build an ESXi server that has ZFS passthrough and presents storage to your MBA via iSCSI. then use time machine on top of that.

Ethabelle

@voretaq7 I think I got it working! :D

ewwhite

There you are!

Ethabelle

20:32

@ewwhite Hi mistah!

ewwhite

@Ethabelle Hey boo!

Did you catch Chopper?

Ethabelle

@ewwhite Yar! I grabbed his e-mail and sent him a message ^__^!

ewwhite

@Ethabelle Excelente. Still in NY. Finishing the 12th hour of interviews in two days...

Ethabelle

@ewwhite You're interviewing in New York o_O?

Iain

@ewwhite who you interviewing with ?

voretaq7

20:39

@Ethabelle what was it?

@@quanta how in the hell did you discover that gzip thing? And go post it as an answer so I can remove my CW version of it :-)

Ethabelle

@voretaq7 Hold on, my victory dance may have been pre-mature. It did it once, but I put the same lines on another job and it keeps referring to my previous volume even though... it's not referencing hat ANYWHERE ...

voretaq7

Also did anyone else know you can apparently cat gzip'd files together and they uncompress to the concatenated output of the input files?

Ethabelle

@voretaq7 Seems to be working again shifty eyes

ewwhite

@Ethabelle All sorts of fancy places

voretaq7

@Ethabelle you really MUST leave out enough fruit/blood for the bat to feast on...

but what did you change to make it get Un-Wonky?

(in case I decide to change the way we handle backups here in the next 6 months)

Joel E Salas

20:47

@voretaq7 Hopefully your budget allows for ritualistic sacrifice

But seriously, it's a great tool. Start with an existing complete config though, or you'll go nuts trying to make your own.

Josh Gitlin

12 hours ago, by Roomey

Happy system administrators appreciation day!

That just occured to me

I wonder if that is related to my issues

Some hacker's stupid idea of a prank...

voretaq7

@JoelESalas heh I wrote ours from the ground up

It's actually very clean and not at all painful. I'll offer it up to anyone who wants it.

Joel E Salas

@voretaq7 Me too, couldn't find a complete one. It's like a rite of passage almost

voretaq7

we aren't all fancy like @Ethabelle over there though -- One pool, automatic tape rotation, fire and forget.

Iain

@JoshGitlin is the attack on one particular customer ?

Joel E Salas

20:51

@voretaq7 I wanted to turn two drives into a virtual autochanger, never found the time.

Josh Gitlin

@Iain I still haven't been able to determine what they're attacking, because it's saturating my port so badly that I can barely get anything done

voretaq7

Eugh -- can anyone help this dude out?

0

Q: Terse, documented, correct way to create Kerberos-backed user shares in Greyhole

As a migration strategy away from Windows Home Server (which is currently out of support and intractable for our needs, for a variety of reasons), our little cloister of nerds has targeted Greyhole for our shared use at home. Despite the documentation's terseness, getting the system set up for si...

oh wait, home question...

ne-ver-mind

Lucas Kauffman

135 reviews left

comon guys

voretaq7

@LucasKauffman now now, there are over 1000 questions with close votes on 'em too...

Lucas Kauffman

@voretaq7 I spit on your close votes!

not really

:p

Ethabelle

21:00

@voretaq7 Yay! It's working! Albeit. I actually have to change all the jobs to new times, etc because this somehow makes Bacula want to use the correct Volumes. Not a big deal, annoying, only have to do it once... like it's stuck somehow, but yeah. Yay!

MDMarra

@ewwhite nuclear warhead, meet nail.

Iain

@voretaq7 not all of em needclosing

Adrian

@Iain Why yes, I do have a problem keeping my beek closed. Poor Impulse Control.

voretaq7

@Ethabelle you have to change the job times <?> funky...

@Iain Most of them don't actually

which is what the "do not close" button on that tab is for :)

Iain

@voretaq7 indeed

voretaq7

21:12

(I don't think that actually clears the votes though -- I wish it did when mods bonk on it)

Adrian

Does Bacula do usable point in time backups for MSSQL, MySQL, and/or PostgreSQL?

voretaq7

@Adrian kiiiinda?

Iain

@voretaq7 it just removes them from the list

voretaq7

There are no agents that I'm aware of ((actually there may be one for MSSQL))

Adrian

@voretaq7 ok. Boss wants PiT for those 3 before he'll green light the new backups

voretaq7

21:13

for Postgres what I do is I back up one slave server. I stop the slave, run the backup, then restart the slave.

Michael Hampton

@adrian Sort of... wiki.bacula.org/doku.php?id=application_specific_backups:mysql

Adrian

@voretaq7 Hmm. That may be good enough. I can do WAL files with PostgreSQL. Not sure about MySQL, but it's the least important of the 3.

voretaq7

(This is the official Postgres-Sanctioned way of doing the backups)

I know the same method would work for MySQL.

Point-in-Time backups are a vague concept -- some people think it means continuous backup -- it doesn't.

Adrian

Nice. There's even a specific MediaWiki script, which is half of my MySQL databases too.

Joel E Salas

@Adrian The real deal is you'd make your backups using another tool like the Percona tool or the Zmanda tool, then back up THOSE backups as part of your Bacula schedule

There's no real DB-to-Bacula intimacy

voretaq7

21:16

@JoelESalas nor should there be really -- your DB backups should make something that can be picked up and moved to tapes. At least that's how I treat 'em.

Adrian

@JoelESalas Is Bacula that great that it's worth not just going with Zmanda then?

Joel E Salas

@voretaq7 Agreed, though I do handle DB backups with a Bacula pre-script :D

@Adrian Unabashedly yes

voretaq7

@Adrian Zmanda is just Amanda -- Bacula was created (partly) to address problems with Amanda :-)

Adrian

@JoelESalas Ok. Boss is thinking about dropping $25k on Iron Mountain or some other solution, but I talked him into adding Bacula to the evaluation list.

Joel E Salas

@Adrian You can bacula to EC2, spend your money there

voretaq7

21:17

@Adrian It all depends on your requirements, just like anything else

@JoelESalas We bacula to rsync.net :-)

Adrian

@voretaq7 Yeah. That's what I'm working on this summer. My Systems Analysis classes are a lot longer ago than I'm comfortable with though. =/

voretaq7

For me I've got a shit-ton of FreeBSD machines and a couple of Redheaded Stepchildren running Debian. The office is all Macs when I get around to that.

@Adrian it's like riding a bike - you never forget how

the systems just get more complicated

Joel E Salas

@voretaq7 DEBIAAAAANNNN

voretaq7

@JoelESalas Lesbian Linux

Joel E Salas

webhostingtalk.com/showthread.php?t=1166050

You might wonder, why'd he link that? It's just exim bullshit. Then you get to the last post

voretaq7

21:21

how can i find a good windows server?
You don't.

Michael Hampton

@voretaq7 I can find a good Windows server down at the shooting range.

voretaq7

@MichaelHampton 's not good anymore!

Michael Hampton

I cannot begin to tell you how thoroughly satisfying it is to put a bullet through a Windows machine.

Ethabelle

@voretaq7 Yeah... this entire adventure was ... ( •_•) bat ... wait for it... ( •_•)>⌐■-■ (⌐■_■) shit crazy. YEEEAAH!

2

voretaq7

@MichaelHampton About as satisfying as it is to baseball bat a printer office-space style

Michael Hampton

21:26

@voretaq7 Yeah, but the poor printer probably didn't deserve it. You can't say the same about Windows.

voretaq7

no, the printer did not deserve it. It was a target for the rage we all felt toward a Netport Express print server.
But the print server was a tiny thing and it broke up into unsmashable bits after the first baseball bat to the head, and we weren't keeping the printer so.... collateral damage.

Josh Gitlin

is so ready to kill people

Iain

@JoshGitlin we'll make a sysadmin out of you yet !

voretaq7

@JoshGitlin And we was both jumpin' up and down yellin' KILL kill KILL kill....

Josh Gitlin

21:55

@Iain haha thanks

84104

@JoshGitlin I'm not so sure that's a good thing.

Josh Gitlin

@Iain this must be what @voretaq7 meant when he said "baptism by fire" yesterday in the TL

voretaq7

@JoshGitlin Has anyone made you clean the underfloor yet?

Josh Gitlin

I guess you mean daily baptisms by fire @voretaq7? :-)

voretaq7

@JoshGitlin Dealing with incompetent hosting companies isn't fire - that's just part of the regular river of shit we wade through every day :-)

Josh Gitlin

21:56

lmao

voretaq7

though a DoS attack certainly qualifies if it's knocking down stuff your customers are using.

Josh Gitlin

I'm calling to see if they can block all connections except those from my IP...

fingers crossed they don't try and sell me DDoS protection again

Is it reasonable to expect a datacenter could do this (at the switch or the core network) and reasonable to expect they would?

I believe so, please correct me if I am wrong...

voretaq7

@JoshGitlin mmmaybe -- they can definitely do it at their router or the firewall for your machine, but the problem is if they do it for one user they start doing it for everyone and the router or firewall configs get nasty.

If you have firewall services as part of your contract though they should be able to put that rule in (as part of your management or for whatever fee they normally charge for minor ruleset changes)

Josh Gitlin

yeah they said they can't, but they can null route my esxi IP

maybe that will help

Well, damn, that wasn't it

OK, back to the drawing board

OK, I could use a huge favor. Can anyone let me know if you can reach 209.59.187.18 via SSH?

I believe I firewalled that off for everyone but me, so if you get an SSH response, I messed it up :-)

Iain

22:13

@JoshGitlin I can't connect to port 22

Josh Gitlin

Thank you!

WesleyDavid

@voretaq7 Especially if it's from an IP.

Josh Gitlin

22:32

OK, so, all my IPs are down except my pfSense firewall. The firewall is denying all connections from everyone except my local office IP. I am still seeing massive packet loss and very sluggish SSH / HTTPS to the firewall. If the only IP that is active is my firewall and the attacker is hitting that and the firewall is configured to drop all traffic from any IP besides my residential IP, would the attacker still be able to saturate my link?

Michael Hampton

22:43

2

Q: Linux I/O bottleneck with data-movers

I have a 24 core machine with 94.6GiB RAM running Ubuntu server 10.04. The box is experiencing high %iowait, unlike another server we have (4 cores) running the same types and amounts of processes. Both machines are connected to a VNX Raid fileserver, the 24-core machine via 4 FC cards, and the o...

linux ubuntu-10.04 io bottleneck

mgorven

@JoshGitlin The link from your firewall to the Internet can still be saturated, since the packets still get to the firewall before being dropped.

Josh Gitlin

@mgorven that's what I thought and that must be what's happening

voretaq7

@JoshGitlin yup. Your inside systems are being protected but the firewall always has to handle the traffic to decide if it should or shouldn't drop it

can you view your FW logs -- Is it one IP, or multiple?

Josh Gitlin

Multiple IPs. I will review the logs shortly, debugging some things with a new datacenter tech who is much more helpful

OK, they think they found the offending IP!

lsiunsuex

23:00

building a custom blog. build the cms - database done - simple - goto integrate it into the main site - no menu item for the blog. um... wtf does it go?

ewwhite

23:10

@MichaelHampton wrong place...

Joel E Salas

Who's at DEF CON?

ewwhite

Nobody cool?

Michael Hampton

@ewwhite Oh, where do you think it should be?

ewwhite

@MichaelHampton No, it should be in SF.

Michael Hampton

@ewwhite That's what I thought. I flagged it already. But I don't know any mods over there...

Joel E Salas

23:20

@ewwhite DEFCON is cool ;_;

ewwhite

Psssh... maybe. I'm too much of a non-dork

Okay, maybe I am a dork.

Michael Hampton

Sorry, I do NOT go to the desert in the middle of the summer.

Joel E Salas

@MichaelHampton There's pedestrian bridges everywhere. You just move from air-conditioned building to air-conditioned building

@ewwhite The hardware village is way too much fun. One year someone soldered an ethernet port onto an android phone

Michael Hampton

@JoelESalas And what happens when the power goes out?

Joel E Salas

@MichaelHampton That would mean that the Hoover Dam collapsed. In that case, we've got bigger problems, in the entire American Southwest

Michael Hampton

23:25

And that's why I don't go to Vegas in the summer.

Joel E Salas

@MichaelHampton It's the time-honored tradition of Nevadaians to blow up the Hoover Dam every summer

ewwhite

One of yesterday's interview questions...

Michael Hampton

@JoelESalas SOMEBODY will do it sooner or later.

ewwhite

"What does the X in UNIX stand for..."

Michael Hampton

And it won't be with explosives.

@ewwhite What was your response?

ewwhite

23:28

"I dunno.. I don't think it means anything"

it was part of a written exam

Joel E Salas

@ewwhite In 1970, Peter Neumann coined the project name Unics (UNiplexed Information and Computing Service) as a pun on Multics, (Multiplexed Information and Computer Services).[9] Unics could eventually support multiple simultaneous users, and it was renamed Unix.

The X stands for Computing Service

Michael Hampton

The X in UNIX indicates where the treasure is buried.

ewwhite

I met the person who write the test this morning.

he said it was a trick question

Michael Hampton

@JoelESalas So while you have Wikipedia open, who owns UNIX this week?

Joel E Salas

@MichaelHampton The Open Group, an industry standards consortium, owns the UNIX trademark.

Michael Hampton

23:31

Wow, they've owned the UNIX trademark for at least a month now. That's gotta be a record.

Joel E Salas

There isn't a single FOSS project that is certified by the Single UNIX Standard

Iain

@ewwhite didn't UNIX come from MULTICS so it onomatopoeic for the ICS

Michael Hampton

@JoelESalas I get the feeling the Open Group has pretty much made themselves irrelevant at this point.

Joel E Salas

@MichaelHampton Through extortionate certification costs, yes

Michael Hampton

Well, you know, if they only sell one...

Sysadmin1138 Expounds

23:40

SysAdmin1138 Expounds: The difference a year makes

posted on July 27, 2012 by SysAdmin1138

I already wrote about the main reason I left WWU. Specifically, the paltry excuse for summer I had out there. Well, it's been a year since that article and I figure it's time for a weather update.Anecdotal reports held the...

ewwhite

In the course of one day, I've trained myself to run the MacBook Retina at full resolution.

Michael Hampton

DO WANT

Joel E Salas

@ewwhite Are you Gnome-in' it on OSX? hi-5

ewwhite

Yea, I keep a Gnome panel in the lower left to launch eTerm with ssh connections

Joel E Salas

@ewwhite I think I'm going to get that same machine and put Fedora on it.

ewwhite

23:45

Let me add gkrellm

@JoelESalas Don't you dare.

It wants to be used as a Mac.

Transcript for