« first day (685 days earlier)      last day (4224 days later) » 

8:00 PM
it's ok that's mild by the comms room (low) standards
@JoshGitlin Set your firewall to drop everything on the floor for 10 minutes?
LOL, but as a mod elsewhere I should know better :-)
Yeah, Iain's right. Nice to meet you.
Also how do they bill you for bandwidth?
@voretaq7 That's what I was asking in my question... but I don't believe ESXi 4 has a firewall
8:01 PM
Oh, you're a mod elsewhere............ to the edit button
@MDMarra That's how I would describe @JoshGitlin's provider, yes...
@voretaq7 Monthly, I get 500 GB in + 5000 GB out
@voretaq7 Same here!
@JoshGitlin OH it's an ESXi box... yah you'd have to stop (or disconnect) all the guests, but if they're hitting your management IP too you're screwed...
I'm just shutting down / ifconfig eth0 down ing all VMs until I can find the cause, so far no luck
@voretaq7 they might be, can't tell
8:03 PM
If it's an ESXi box and you want to pull the cable, but can't...why not just disable the vswitch that the VMs are on?
same thing
Let it sit for a couple of minutes and then turn it back on
@JoshGitlin If your management IP is on a separate NIC just kill the NIC with the guests on it
See if the attack goes away
@voretaq7 stop stealing my goddamn ideas
I have too few of them, they're precious.
if the attack tapers off you know it's not hitting the management IP (then you just have to isolate which guest(s) are targets)
@MDMarra the precious?
@MDMarra Great idea, thanks!
8:04 PM
Do you have vSphere Enterprise?
DO you know if I can do that through SSH? Because the vSphere console is useless
If so, you can replace the existing standard vSwitch with a Cisco V1000 and do a virtual port mirror and watch the traffic with that
@MDMarra no, sadly, vSphere 4 ESXi free
@JoshGitlin Um, I'm sure you can do it through SSH or PowerCLI. I just have no idea how off of the top of my head.
We're a small company, which is why we don't have two thirds of what I want to have :-(
8:06 PM
@JoshGitlin you can do it with their Perl interface but that involves writing some scripts -- pain in the ass it is
@JoshGitlin There's always more to want
@JoelESalas Unless you buy Enterprise Plus. Then you're the king of the world
@MDMarra If you have that kind of money, why are you wasting it on VMware instead of going to space
I love space
edu pricing for it isn't bad though
8 sockets of enterprise plus is only like 20k I think including support
I can't believe they can't just unplug me. WTF do I pay them $1000/mo for!?
8:12 PM
@JoshGitlin So they can have enough ivory back-scratchers
oh, of course!
@JoshGitlin Executive lapdances, I bet.
Is there a special "I'm not a douchebag" checkbox I can set somewhere in Pandora? Its science-fair experiment of a matching algorithm plays me shit like Staind now and again
OK well this tech support guy thought Apache runs inside VMware ESXi and he could ssh in to the host and stop Apache. Wow. But he also did say he could have them unplug it for me
@JoelESalas That's why I quit using Pandora. They wander off into all sorts of strange directions with that algorithm.
8:21 PM
@JoshGitlin seriously, they won't send a guy in to pull the ethernet cable, or down your port on the switch for 10 minuts?
I mean THEY have to deal with this traffic too - a spike that big would make me move my ass (and have someone blackholed) back in my ISP days
@JoshGitlin well he's competent enough to know that the bits travel through the cable... that's a plus :)
(also he's TECHNICALLY correct Re: the service console's web server, but that's probably not your problem :-)
Still interviewing....
I can only do one of these three things and I'm not sure which one I should do:

Upgrade my MBA's SSD to a 256GB model

Purchase a 2TB Time Capsule

Purchase a 27" 2560x1440 IPS Korean monitor to replace my current 22" at my desk at home
They're all roughly the same price
Getting deep. Looks like I missed @ethabelle....
@ewwhite Try to freshen up before the swimsuit competition
@ewwhite How long has the interview been?
8:24 PM
@mdmarra don't do the time capsule...
Live been interviewing places today since 9am EST
I have a DLINK DNS323 with funplug and the AFP module installed so I can do unsupported time machine backups to that
@voretaq7 This second guy just unplugged me from the switch, fingers crossed
I'm going to wait a little bit to see if Jeff Atwood livetweets a house fire before buying Korean displays.
but it fails too many times and has to do a whole new 100GB backup over the air like every other week
@voretaq7 They hate to deal with the traffic unless they get to bill em for it, lol
8:25 PM
@JustinDearing I'll stick with my Chinese and Taiwanese displays
@JoshGitlin depends on how big their pipe is
@voretaq7 Oh, true, he is correct there. But he was looking for a cPanel Apache server, I know these guys too well, lol
@voretaq7 thats what she said
some ISPs a 60Mbps spike would seriously endanger their margins.
@JoshGitlin but they don't tell me the link speeds, weights, and (fiscal) costs
8:28 PM
@voretaq7 My philippino wife has an affinity for Korean spas and korean soap operas . I like korean food. I sayI need a korean monitor
@voretaq7 yes, true
@JustinDearing probably remembers dealing with the CA DDOSes (though that was all tiny packets, almost no bytes) -- If you saturate your fast, cheap link you wind up routing traffic over the more kerspensive ones
so what they're charging you may not make up for their transit bill that month
@MDMarra Build an ESXi server that has ZFS passthrough and presents storage to your MBA via iSCSI. then use time machine on top of that.
@voretaq7 I think I got it working! :D
There you are!
8:32 PM
@ewwhite Hi mistah!
@Ethabelle Hey boo!
Did you catch Chopper?
@ewwhite Yar! I grabbed his e-mail and sent him a message ^__^!
@Ethabelle Excelente. Still in NY. Finishing the 12th hour of interviews in two days...
@ewwhite You're interviewing in New York o_O?
@ewwhite who you interviewing with ?
8:39 PM
@Ethabelle what was it?
@@quanta how in the hell did you discover that gzip thing? And go post it as an answer so I can remove my CW version of it :-)
@voretaq7 Hold on, my victory dance may have been pre-mature. It did it once, but I put the same lines on another job and it keeps referring to my previous volume even though... it's not referencing hat ANYWHERE ...
Also did anyone else know you can apparently cat gzip'd files together and they uncompress to the concatenated output of the input files?
@voretaq7 Seems to be working again shifty eyes
@Ethabelle All sorts of fancy places
@Ethabelle you really MUST leave out enough fruit/blood for the bat to feast on...
but what did you change to make it get Un-Wonky?
(in case I decide to change the way we handle backups here in the next 6 months)
8:47 PM
@voretaq7 Hopefully your budget allows for ritualistic sacrifice
But seriously, it's a great tool. Start with an existing complete config though, or you'll go nuts trying to make your own.
12 hours ago, by Roomey
Happy system administrators appreciation day!
That just occured to me
I wonder if that is related to my issues
Some hacker's stupid idea of a prank...
@JoelESalas heh I wrote ours from the ground up
It's actually very clean and not at all painful. I'll offer it up to anyone who wants it.
@voretaq7 Me too, couldn't find a complete one. It's like a rite of passage almost
we aren't all fancy like @Ethabelle over there though -- One pool, automatic tape rotation, fire and forget.
@JoshGitlin is the attack on one particular customer ?
8:51 PM
@voretaq7 I wanted to turn two drives into a virtual autochanger, never found the time.
@Iain I still haven't been able to determine what they're attacking, because it's saturating my port so badly that I can barely get anything done
Eugh -- can anyone help this dude out?
Q: Terse, documented, correct way to create Kerberos-backed user shares in Greyhole

MrGomezAs a migration strategy away from Windows Home Server (which is currently out of support and intractable for our needs, for a variety of reasons), our little cloister of nerds has targeted Greyhole for our shared use at home. Despite the documentation's terseness, getting the system set up for si...

oh wait, home question...
135 reviews left
comon guys
@LucasKauffman now now, there are over 1000 questions with close votes on 'em too...
@voretaq7 I spit on your close votes!
not really
9:00 PM
@voretaq7 Yay! It's working! Albeit. I actually have to change all the jobs to new times, etc because this somehow makes Bacula want to use the correct Volumes. Not a big deal, annoying, only have to do it once... like it's stuck somehow, but yeah. Yay!
@ewwhite nuclear warhead, meet nail.
@voretaq7 not all of em needclosing
@Iain Why yes, I do have a problem keeping my beek closed. Poor Impulse Control.
@Ethabelle you have to change the job times <?> funky...
@Iain Most of them don't actually
which is what the "do not close" button on that tab is for :)
@voretaq7 indeed
9:12 PM
(I don't think that actually clears the votes though -- I wish it did when mods bonk on it)
Does Bacula do usable point in time backups for MSSQL, MySQL, and/or PostgreSQL?
@Adrian kiiiinda?
@voretaq7 it just removes them from the list
There are no agents that I'm aware of ((actually there may be one for MSSQL))
@voretaq7 ok. Boss wants PiT for those 3 before he'll green light the new backups
9:13 PM
for Postgres what I do is I back up one slave server. I stop the slave, run the backup, then restart the slave.
@voretaq7 Hmm. That may be good enough. I can do WAL files with PostgreSQL. Not sure about MySQL, but it's the least important of the 3.
(This is the official Postgres-Sanctioned way of doing the backups)
I know the same method would work for MySQL.
Point-in-Time backups are a vague concept -- some people think it means continuous backup -- it doesn't.
Nice. There's even a specific MediaWiki script, which is half of my MySQL databases too.
@Adrian The real deal is you'd make your backups using another tool like the Percona tool or the Zmanda tool, then back up THOSE backups as part of your Bacula schedule
There's no real DB-to-Bacula intimacy
9:16 PM
@JoelESalas nor should there be really -- your DB backups should make something that can be picked up and moved to tapes. At least that's how I treat 'em.
@JoelESalas Is Bacula that great that it's worth not just going with Zmanda then?
@voretaq7 Agreed, though I do handle DB backups with a Bacula pre-script :D
@Adrian Unabashedly yes
@Adrian Zmanda is just Amanda -- Bacula was created (partly) to address problems with Amanda :-)
@JoelESalas Ok. Boss is thinking about dropping $25k on Iron Mountain or some other solution, but I talked him into adding Bacula to the evaluation list.
@Adrian You can bacula to EC2, spend your money there
9:17 PM
@Adrian It all depends on your requirements, just like anything else
@JoelESalas We bacula to rsync.net :-)
@voretaq7 Yeah. That's what I'm working on this summer. My Systems Analysis classes are a lot longer ago than I'm comfortable with though. =/
For me I've got a shit-ton of FreeBSD machines and a couple of Redheaded Stepchildren running Debian. The office is all Macs when I get around to that.
@Adrian it's like riding a bike - you never forget how
the systems just get more complicated
@JoelESalas Lesbian Linux
You might wonder, why'd he link that? It's just exim bullshit. Then you get to the last post
9:21 PM
how can i find a good windows server?
You don't.
@voretaq7 I can find a good Windows server down at the shooting range.
@MichaelHampton 's not good anymore!
I cannot begin to tell you how thoroughly satisfying it is to put a bullet through a Windows machine.
@voretaq7 Yeah... this entire adventure was ... ( •_•) bat ... wait for it... ( •_•)>⌐■-■ (⌐■_■) shit crazy. YEEEAAH!
@MichaelHampton About as satisfying as it is to baseball bat a printer office-space style
9:26 PM
@voretaq7 Yeah, but the poor printer probably didn't deserve it. You can't say the same about Windows.
no, the printer did not deserve it. It was a target for the rage we all felt toward a Netport Express print server.
But the print server was a tiny thing and it broke up into unsmashable bits after the first baseball bat to the head, and we weren't keeping the printer so.... collateral damage.
is so ready to kill people
@JoshGitlin we'll make a sysadmin out of you yet !
@JoshGitlin And we was both jumpin' up and down yellin' KILL kill KILL kill....
9:55 PM
@Iain haha thanks
@JoshGitlin I'm not so sure that's a good thing.
@Iain this must be what @voretaq7 meant when he said "baptism by fire" yesterday in the TL
@JoshGitlin Has anyone made you clean the underfloor yet?
I guess you mean daily baptisms by fire @voretaq7? :-)
@JoshGitlin Dealing with incompetent hosting companies isn't fire - that's just part of the regular river of shit we wade through every day :-)
9:56 PM
though a DoS attack certainly qualifies if it's knocking down stuff your customers are using.
I'm calling to see if they can block all connections except those from my IP...
fingers crossed they don't try and sell me DDoS protection again
Is it reasonable to expect a datacenter could do this (at the switch or the core network) and reasonable to expect they would?
I believe so, please correct me if I am wrong...
@JoshGitlin mmmaybe -- they can definitely do it at their router or the firewall for your machine, but the problem is if they do it for one user they start doing it for everyone and the router or firewall configs get nasty.
If you have firewall services as part of your contract though they should be able to put that rule in (as part of your management or for whatever fee they normally charge for minor ruleset changes)
yeah they said they can't, but they can null route my esxi IP
maybe that will help
Well, damn, that wasn't it
OK, back to the drawing board
OK, I could use a huge favor. Can anyone let me know if you can reach via SSH?
I believe I firewalled that off for everyone but me, so if you get an SSH response, I messed it up :-)
10:13 PM
@JoshGitlin I can't connect to port 22
Thank you!
@voretaq7 Especially if it's from an IP.
10:32 PM
OK, so, all my IPs are down except my pfSense firewall. The firewall is denying all connections from everyone except my local office IP. I am still seeing massive packet loss and very sluggish SSH / HTTPS to the firewall. If the only IP that is active is my firewall and the attacker is hitting that and the firewall is configured to drop all traffic from any IP besides my residential IP, would the attacker still be able to saturate my link?
10:43 PM
Q: Linux I/O bottleneck with data-movers

BenjaminI have a 24 core machine with 94.6GiB RAM running Ubuntu server 10.04. The box is experiencing high %iowait, unlike another server we have (4 cores) running the same types and amounts of processes. Both machines are connected to a VNX Raid fileserver, the 24-core machine via 4 FC cards, and the o...

@JoshGitlin The link from your firewall to the Internet can still be saturated, since the packets still get to the firewall before being dropped.
@mgorven that's what I thought and that must be what's happening
@JoshGitlin yup. Your inside systems are being protected but the firewall always has to handle the traffic to decide if it should or shouldn't drop it
can you view your FW logs -- Is it one IP, or multiple?
Multiple IPs. I will review the logs shortly, debugging some things with a new datacenter tech who is much more helpful
OK, they think they found the offending IP!
11:00 PM
building a custom blog. build the cms - database done - simple - goto integrate it into the main site - no menu item for the blog. um... wtf does it go?
11:10 PM
@MichaelHampton wrong place...
Who's at DEF CON?
Nobody cool?
@ewwhite Oh, where do you think it should be?
@MichaelHampton No, it should be in SF.
@ewwhite That's what I thought. I flagged it already. But I don't know any mods over there...
11:20 PM
@ewwhite DEFCON is cool ;_;
Psssh... maybe. I'm too much of a non-dork
Okay, maybe I am a dork.
Sorry, I do NOT go to the desert in the middle of the summer.
@MichaelHampton There's pedestrian bridges everywhere. You just move from air-conditioned building to air-conditioned building
@ewwhite The hardware village is way too much fun. One year someone soldered an ethernet port onto an android phone
@JoelESalas And what happens when the power goes out?
@MichaelHampton That would mean that the Hoover Dam collapsed. In that case, we've got bigger problems, in the entire American Southwest
11:25 PM
And that's why I don't go to Vegas in the summer.
@MichaelHampton It's the time-honored tradition of Nevadaians to blow up the Hoover Dam every summer
One of yesterday's interview questions...
@JoelESalas SOMEBODY will do it sooner or later.
"What does the X in UNIX stand for..."
And it won't be with explosives.
@ewwhite What was your response?
11:28 PM
"I dunno.. I don't think it means anything"
it was part of a written exam
@ewwhite In 1970, Peter Neumann coined the project name Unics (UNiplexed Information and Computing Service) as a pun on Multics, (Multiplexed Information and Computer Services).[9] Unics could eventually support multiple simultaneous users, and it was renamed Unix.
The X stands for Computing Service
The X in UNIX indicates where the treasure is buried.
I met the person who write the test this morning.
he said it was a trick question
@JoelESalas So while you have Wikipedia open, who owns UNIX this week?
@MichaelHampton The Open Group, an industry standards consortium, owns the UNIX trademark.
11:31 PM
Wow, they've owned the UNIX trademark for at least a month now. That's gotta be a record.
There isn't a single FOSS project that is certified by the Single UNIX Standard
@ewwhite didn't UNIX come from MULTICS so it onomatopoeic for the ICS
@JoelESalas I get the feeling the Open Group has pretty much made themselves irrelevant at this point.
@MichaelHampton Through extortionate certification costs, yes
Well, you know, if they only sell one...
11:40 PM
posted on July 27, 2012 by SysAdmin1138

I already wrote about the main reason I left WWU. Specifically, the paltry excuse for summer I had out there. Well, it's been a year since that article and I figure it's time for a weather update.Anecdotal reports held the...

In the course of one day, I've trained myself to run the MacBook Retina at full resolution.
@ewwhite Are you Gnome-in' it on OSX? hi-5
Yea, I keep a Gnome panel in the lower left to launch eTerm with ssh connections
@ewwhite I think I'm going to get that same machine and put Fedora on it.
11:45 PM
Let me add gkrellm
@JoelESalas Don't you dare.
It wants to be used as a Mac.

« first day (685 days earlier)      last day (4224 days later) »