« first day (626 days earlier)      last day (4641 days later) » 

17:03
It will be nice to get out of working with Windows 99% of the time.
Monday can't come quick enough.
17:22
If someone has some time to educate a new user or possibly answer the question? serverfault.com/questions/393677/dbd-and-dbi-and-eirp-oh-my
Laters...
"Code runs fine on localhost but doesn't execute on gatorhost" Of course, gatorhost is not in his /etc/hosts file...
There's not nearly enough adorable in this room.
Stand by...
!!!!!!!!
We're supposed to club it, right?
@Adrian I just realized this was carpet. I thought it was a road.
Impressive contrast indeed
17:40
Anyone see @Pauska around?
@pauska You've got a new comment on your post: thenubbyadmin.com/2012/04/05/…
@tombull89 Well then!
@gparent Seriously. The strange shit that goes on around wouldn't be remotely believable in a work of fiction.
We have an electronic shift log where staff can type in reports of incidents and whatnot regarding our client to pass on to future shifts later that day or in future days.
Some of the stuff posted in there is pretty impressive.
Crazy people pooping on the floor is a rather mundane occurrence, generally speaking.
@Adrian there's got to be one with water or fire involved.
Oh yes. lemme pop that up on flickr. brb.
17:46
@WesleyDavid The answer is that the guy forgot to pass the -a option.
@Zoredache Wanna answer him? I didn't even read what he asked. =)
here you go @gparent :)
AHAHAHAHA
Pro!
Also I found something to listen to at work
Something that won't get me tired as much as music - Fire Department radio feed!
18:01
@Basil hahaha
Finally lunch, at 2PM.
I don't necessarily love my job, but it's not dull:
@Adrian what's going on?
They're probably washing the squad SUV, which would explain why they rented 2 big fire trucks.
@vCole Client got drunk and lit a cigarette. Put it in the ashtray sitting their their bed and then went down the hall to visit another client to bum another beer.
I know now if the pipes start banging to not wait for the fire signal, just get up and leave.
yikes!
18:10
My desk is 6' away from the Sprinker main on the front of the building.
Lmao, one of our coworkers' desk just started letting out music at a moderate volume
But the best log in our system I ever saw was one where the night shift counselor reported that someone had pooped in the mens dorm and instead of "feces in men's dorm" listed it as "Fetus found in Men's dorm".
2
She's not there, though, and we can't spot what makes the noise
LOL
@Adrian Ahahahahahahah
18:15
My girlfriend works at Walgreens.
@Adrian My dad worked as level 2 tech support some place, one of the clients had a plant on top of their computer case. They fed the plant regularly, eventually the computer had a short and stopped working. Dad wrote the ticket: "Replace computer, do not replace plant."
She's told me so many stories about people pooping themselves, or poop logs left in the store in random aisles.
@gparent hahaha
My best ticket was when I worked level 1 at the same place. One of the clients is the most technically illiterate you can imagine. he writes down his user password on his laptop's Intel sticker, and basically refuses to do most things on the computer he's supposed to do.
So one day his laptop encryption software locks him out due to bad password entry.
I get the ticket from the call center with the following two comments:
1. The client identifies himself as the missing brain cell
2. Client refuses call center instructions, dispatch local technician
@vCole I moonlight with fire / rescue. We get some... unpleasant bodily responses.
I guess the key point is that Walgreen's staff shouldn't on a regular basis.
Still, I bought a small jar of vicks to smear over my nose for future incidents.
@JeffFerland Somehow that doesn't surprise me. I was wondering if you worked for a volunteer fire dept. somewhere. I can usually tell.
18:19
@Adrian Some sort of personality thing that seeps through?
@JeffFerland Oh yeah. 10 years here has taught me all sorts of strange skills.
I guess I'm a little crazy too, considering that I worked here for 6 years as an entry-level technician, left, and then came back as the senior tech.
Ah... I seem crazy! Got it :)
@JeffFerland yeah her manager tried to make her clean up some poop.
@Adrian wow, 10 years?
/var/log/poop
2
I've been at my current employer for 7 straight.
@rnxrx You sure? Seems like that belongs in the /proc hierarchy
18:25
@JeffFerland wow.
@vCole Not all in one shot, thank goodness. 6 years last time. 2.7 this time.
Longer than I've been in the field.
@JeffFerland excellent
About 4.5 years in the field right now/
As screwed up as things are here now, it was far worse when I agreed to come back. I think our average server uptime was 6 weeks and the ratio of uncontrolled vs. controlled outages was 6:1.
18:26
@JeffFerland naw, everyone knows /var/log is full of shit...
3
@vCole I've been doing this for almost 15 years. But years in the field don't mean shit if you're not learning the right lessons.
@Adrian I suppose that's true.
@Adrian That's actually a neat metric - controlled vs uncontrolled... leaves room for people doing changes and just utterly shitting the bed
@rnxrx /usr/local/bin/chsheets
and controlled is pretty loose here. Means it didn't barf on its own and it was shut down by us, even if that was only to keep it from freezing up which it was on its way to already.
18:30
so if someone does a change and yanks three drives out of a RAID-5 setup is that considered controlled or uncontrolled?
@Adrian sounds pretty rough.
@JeffFerland /sbin/dump should be run every day
However, when I arrived here 7 months ago
All the server hardware was 6-7 years old.
Early in my career I worked in a big financial setting up some of their first Internet presence - multihoming at a point where half of our carriers couldn't spell BGP. The change control guys were about 700 years old and weren't quite sure why anything beyond CICS was ever written. As long as the change control didn't involve the mainframe, it was OK. We submitted a ticket stating that we were taking down the network core @ 3PM on a Tuesday just because.
When we got to the change, the guy looked at us and one of our people just said "On target" and it was approved.
"on target" eventually became a euphemism for a change control going straight to hell
lol
Why do we still use tapes? Q_Q
18:40
@vCole Why shouldn't we still use tapes?
@vCole Their cost per byte is quite low and they store on the shelf for very long periods of time.
Because it's a pain in my ass having to change 3 tape drives when they'd all fit onto one LTO4 drive.
*LTO4 tape
Thus having 3 copies of ARCServe.
all different (re:old) versions.
@vCole So your complaint about tape is that you can't get (afford) to get tapes/drives large enough for your requirements? I can certainly understand that.
@Zoredache basically, yes.
I think I'm just antsy, since I have about 3.5 days left here.
the same place with the bad change control had 8 igloos for their mainframe backup and something like 1500 Solaris and AIX boxes each with its own tape drive... Everything from DAT to 8mm to DLT, etc.. Ops just went around with a cart and collected tapes for a living. Hilarious.
18:44
@rnxrx oh wow lol.
some lunatic from Iron Mountain would show up in a van to take the tapes somewhere... never saw him at less than 50 mph in the parking lot. I think he actually flipped the van over at one point. It was suggested that this was potentially a problem as far as backup strategies went, but that was dismissed as being silly.
still - hard to argue with the bandwidth of a van full of tapes, even if the latency sucked
As long as his dangerous driving doesn't induce packet loss...
@vCole Not terribly rough. Most of the staff don't really care, but we have limited resources to replace due to tight budgeting, so its stressful when you can't do much about the problems.
@Adrian same issue here. We're in the housing industry. (Door & Window hardware)
@rnxrx Oddly, I'm annoyed that they won't do bi-monthly tape pickups.
18:52
They guy seemed like he had packets of -something-
@Adrian IM won't do bi-monthlies? That seems like a pretty glaring omission.
19:04
@rnxrx Yep. Most infrequent is monthlies. Yeah, I'm not really happy with any of this, frankly. Very little of it suits our needs. Of course, the biggest problem is that while our mgmt. understands that IT costs money, they really just want the entire problem to go away.
Frankly, we'd be a perfect candidate for a total out-source of IT stuff. Mgmt. wants to focus on core strategies and doesn't want to have to deal with any details.
Hell, at this point in my career, I'd probably still sign on with whoever took over.
@Adrian sounds like an opportunity
@Iain I'm half tempted to try and talk our boss into looking that direction. He sure doesn't want to deal with it since the database side is enough to keep him busy too.
@Adrian I hear you... I think a lot of management teams get that IT costs money, but that's different than seeing actual value. No matter the industry it kind of sucks to be viewed simply as a cost-center. The thing about a lot of outsourcing contracts is that they end up structured in such a way that you end up needing smart guys to sit there and try to squeeze value out of every nickel billed while the outsourcer tries to bill as many nickels for the least amount of their own output.
It's not a comment on outsourcers or their honesty, just the basic underlying point of tension in just about all of these situations I've seen.
@rnxrx Pretty much how any given business relationship works, at least as far as I've ever seen.
Sure - and it's kind of the way of things, but somehow outsourcing makes it murkier than just about any industry I've seen.
Sam
Sam
19:12
evening all
You bargain for the best price on hardware - it's a pretty easy equation. The vendor would like to charge more, you'd like to pay less. If you buy more stuff (or commit to buy, or whatever) then there's some incentive for them to charge less. If you have another player who can sell the same thing, it's still the same motivations.
howdy @Sam
but when you have a third party who is essentially both assuming your best interest and selling you the thing to meet that interest... well...
It's like hiring two auditors from the same accounting firm to sign off on one another's work
hello @Sam
Not bad. VMWare Support Rep had excellent English skills and didn't require a script.
Now just hoping I get a call back from the sales guy.
19:23
@Adrian Hey be indulgent, I use scripts all the time.
Who on ServerFault do we know is a MySQL expert? Specifically in the field of analyzing database performance, table structure and database design?
I'm thinking Tom O'Connor is one.
@gparent Indulging others is not in my nature.
@WesleyDavid RolandoMySQLDBA, but he hangs out over on DBA these days
@voretaq7 Ahhh, yes. Of course.
I try to kick the performance questions over to DBA as much as possible - all the guys who are good at it who hang out here also seem to spend time there
19:30
good $TimeOf Day
This afternoon is going to be fun. I'm working on an not-entirely-authorized project to port our database app from Debian Lenny to CentOS 6.
@Adrian better to ask forgiveness?
@Adrian a smop I'm sure
@WesleyDavid Hi. Me.
@Zypher Pretty much. If it works, I'm all set to take over Admin'ing their servers for them.
19:33
@Adrian Not a Debian fan?
Arg, I hate when someone calls you but doesn't leave a voicemail.
(from a number you don't know)
@vCole then it wasn't important, and was actually probably a telemarketer
@Iain Heh. The most entertaining part is going to be figuring out whether I can port their crap into Postgres 8.4 or have to backport.
@Zypher most likely, or a recruiter.
@vCole ... the difference being?
19:34
@JeffFerland Not a non-fan either. But a short support cycle isn't the best for our Dev team. They need at least a 5 year support cycle like CentOS.
@Adrian just a quick hack then
@Iain We'll see. Our Devs write some pretty craptastic SQL and PHP.
@Adrian how old is their Postgres release?
@JeffFerland Really? Awesome. Am I connected to you on LinkedIn?
(also why not jump to 9.x?)
19:35
@Zypher true.
@Adrian having read about some of the things you have to put up with it doesn't surprise me
@voretaq7 It's 8.3.11-0lenny1
@Adrian 8.3 -> 8.4 is a total non-event
@JeffFerland Found you. Added you. Might have a need for you. =)
All the business logic is in stored procedures. They'd have to go validate everything for 9.x
19:36
they'll never even notice :)
8.3 -> 9.x is probably not a big deal either - pl/pgsql didn't change in any shockingly incompatible ways
@voretaq7 That's what I told them. But nobody over there understands what the impacts are to an upgrade. They do it when they're forced to by outside pressures.
some automatic (implied) casts changed between 8.x and 9.x though
they are, in fact, wonderful people. And it does help if I repeat that to myself.
@Adrian haha
the impact of not doing an upgrade is you get bitten by bugs that got fixed years ago.
(with Postgres) the impact of an upgrade is (within major versions) is virtually nil - maybe a minute of downtime while the server restarts.
(that's how I explain it here)
19:39
But yeah, I have to explain a lot of things to them that they should, one would think, be arguing with me about from a position of Superior Knowledge, not a total lack thereof.
@voretaq7 To them, the bugs are just part of the ecosystem. They have no expectation of how it works other than the way it works now.
@Adrian oy... kittens
@voretaq7 Yeah. It makes me a little crazy.
@WesleyDavid accepted
@voretaq7 mew?
@WesleyDavid yes.
19:47
claws sofa
@WesleyDavid meh its being replaced...
Arg, I hate the Dell Premier site
6.8 million SQL queries in 20 hours. One a dedicated server hosting a handful of sites. Sounds legit.
300,000 select statements an hour, on average. 90%+ of the total MySQL usage.
Methinks some tuning is in order.
@WesleyDavid 100 per second... not bad.
OUTER JOIN ALL THE THINGS!!
19:54
/me has visions of fixed-fee, $100 engagement on oDesk
need dis quik. must be xpert. years of xperience.
also, server broken. filesystem won't mount. won't tell you until after engagement starts.
I don't get jaded :D
@WesleyDavid Heh. The infamous staff direct query page where they get the idea that they can check every single table for every client. OUTER JOIN TO THE UNIVERSE.
@WesleyDavid with subselects!
300k select statements and there is zero query cache. I am disappoint.
Unindexed joins... lurvely.
Databases make my head explode.
Bullets do the same thing to mine
20:07
@WesleyDavid Beware the join on views...
oh that's fun
So, is the username from this screenshot... "pornbust"?
0
Q: Sudden load average increase with same number of users and having changed nothing

TebbI'm very novice in termos of administrating a server. Website was running fine till yesterday, load averages under 1. Suddenly, yesterday it stopped working fine, I was getting load averages to up to 50 (!) I haven't changed anything in the server nor in the website, and the users in the websi...

@voretaq7 Hahahahaha. Yes. Which is probably what makes our entire app still run like shite on a x3550M2
@JeffFerland looks like it
"But you must remember that Dropbox has problem with synchronize truecrypt files because adding new staff to truecrypt files doesn't change any of truecrypt file so dropbox doesn't know is it already modified file and need to by synchorized."
What
20:12
I feel so strange creating the interview materials for the people interviewing for the vacancy I'm creating. Who has some totally fucked up trick questions relating to AD, VMWare, or iSCSI storage?
@gparent When you update a file within a truecrypt volume, the file the volume is in does not appear to change at all
So many people don't know that you can dual screen VMWare by launching it twice
@MDMarra I always try to think back to weird fucked up shit that's happened to me
You might as well ask it
no change of size, or timestamp
So dropbox can't detect the change
20:13
@Zoredache So no bytes are written. But you changed the file.
How exactly does that work?
@voretaq7 et tu, SCAN ALL?
@WesleyDavid no no, it uses the indexes...
@gparent the description is bad. The file IS changed, it just is not changed in a way that dropbox can detect.
AFTER it does the joins on the full, un-limited views.
Dropbox only checks timestamp and size?
20:14
There is an option in Truecrypt to make it update the modification times
@gparent correct.
oh, did not know that
@MDMarra Oooh, interview questions!
@MDMarra This is very basic, but it's surprising the number of people who do not know that you can join a machine via AD just with login credentials by default.
As in non-admin ones
@gparent By default a user can only do that 10 times.
Only 10? That's okay then :)
20:17
@gparent Unless you have a good sysadmin like me that set that value to 0 :D
You can change the AD group too AFAIK, but it's pretty funny.
@MDMarra Well, we've proven that's not statistically likely
So far I just have basic questions like What are the 5 FSMO roles and what do they do? What is a Fine-Grained Password Policy and how is it implemented? What are AD Sites, how do they impact logins and DFS namespaces? What order are GP object applied?
I feel like those are all pretty straight forward
I really want to come up with a question that's going to ruin someone's day.
@MDMarra why ? what do you gain from that (other than sysadminy goose pimples) ?
@MDMarra Why is it everyone's goal in interviews to gain "gotcha" questions?
20:20
@JeffFerland superiority complex
Yeah, I was just asked "which operation makes RAID5 possible?"
@ewwhite Parity?
XOR
And I just rattled the answer out... but it strikes me as an odd question.
concept -> binary math.
@Iain Well, I don't want to ask a question that's misleading looking for a wrong answer.
I meant more like the type of question that has no right answer
20:22
@MDMarra Think about the people in here that you consider to be knowledgeable folks you'd like to work with. Think about why you pick them. Try to use that as a basis for your interviews.
Something like that would ruin my day, because the whole day, I'd be thinking to myself "Did I get it right? Shit, am I an idiot?"
@MDMarra You are also going to give them some real life scenarios right? Like 'the CEO says the Web site is down, what do you do'?
I didn't mean that to be a question where you're trying to trick someone into giving a wrong answer.
Open-ended questions... one firm a few weeks ago drew a diagram with a server and a black-box appliance connected to a router. The black-box had a fixed IP/subnet. The server was on a different subnet. The task was to give as many possible methods of connecting the two systems (goal: ping the black box from the server)
@Zoredache Put up dummy website that's mapped to his computer, then go fix the real problem.
20:23
@Zoredache Yeah, I already set up a little environment for a hands on portion for that kind of thing
@ewwhite Yeah, that's more what I meant
So, crossover cable, aliased eth0:1 interface, vlan routing, NATing...
@MDMarra Ask them popular questions from serverfault. Ask them to give you a nice long list of all the various features that you would look for in a switch. Ask the about what they would do if the security auditor is an idiot. :p
A router! Two routers! Three routers! An Internet! Do I win an Internet?
A question where you're on the train home and you're like "Shit, I should have said VPN too"
"Ask the about what they would do if the security auditor is an idiot."
+1
20:24
@MDMarra Yeah, everyone but you is welcome to my resume -.-
hahaha
the classic wget http://serverfault.com what happens
@JeffFerland :D
@Iain I think those are mean questions.
@ewwhite Those are good ones, I think. Understanding the underlying protocol... good thing?
20:25
there is a lot of scope for discussion
... and scope is discussion is good!
@Iain yes, but it's easy to dig a hole, too...
yep
"Keyboard interrupt..."
20:26
Or if you get TCP information off..
nothing ! you didn''t press return/enter
@ewwhite I always start with that... :)
but I generally don't have hardcore interview questions to give... I let the person talk.
@JeffFerland s/gotcha/difficult/ -- because how the candidate gets to the answer is as important as the answer itself, and if they just spit out the answer you don't learn much.
I ask about what their first computer was/how they started...
20:27
video genie
I really haven't had to interview anyone.
If it's a Linux position, I ask them to talk about which distributions they use at home... and which projects they contribute to...
I was thinking about making magnets that said "router, switch, firewall, pc, printer, domain controller, web server" and giving them a marker and saying, draw how you would arrange these into a network.
And then, "how do you find answers?"
If he answers "I ask them on ServerFault and they do all the work for me", hire him
20:29
Seems too simple though, but adding more magnets would make it convoluted
when I see people who don't look at experts exchange, serverfault, mailing lists, etc... that's a bad sign
experts-exchange is a bad sign too.
They can stand that website.
I only ever look at ee or mailinglists if google sends me there
@ewwhite I have setup experts-exchange in my Google blacklist.
@Zoredache Ouch... It's useful every so often...
20:30
Sure but the useful / not useful ratio..
Ah, remember when it was expertsexchange.com
Yep. That's how I initially discovered the site.. wait what
you too?
wat.
I mean
@vCole I still read that as 3 words.
@JeffFerland That rocks
20:32
That's bad.
@MDMarra do they need to script/program ?
@Zoredache I block EE too. The useful ratio for me is generally <10%, and my questions aren't all THAT deep.
Just watched Iron Sky. Pretty damn funny
@DJPon3 Is that one of those Space Nazi things?
oh yes
:-)
20:45
Looked like it would be fun.
yeah its fun. Doesn't take itself too seriously and has bad reviews from people who took it too seriously, but if you're happy to have some fun, it does the job
@DJPon3 I find it difficult to take anything involving space seriously.
then you'll be right at home with this one
05/26/2012 10:27:09
	(37000/701), [Microsoft][SQL Native Client][SQL Server]There is insufficient system memory to run this query.
	SQLColAttributes
What does it all mean?!?
aliens
20:51
It means you need more memory?
It's a SQL 2005 server running on Windows 2003... 4GB given to the VM. 32-bit OS.
It means that when using that system you should think smaller thoughts. Duh.
accidently put a memory reservation on the VM?
*put
@vCole up arrow allows you to edit
20:53
Ah good to know, thank you.
@vCole No reservation or RAM limits.
Well, time for me to start my 1.5 hour drive home, bbiab.
ouch
over in 3 days
well 4 if you count today.
so, there's a light at the end of the tunnel. ;)
Later!
:)
why when faced with the obvious do people try and circumvent it serverfault.com/questions/393717/… ?
21:03
Spotted this on reddit's r/sysadmin page. I'm almost impressed with the tone.
We are not a vendor of end-user software. We don't work with users. Consequently we don't have obligation to explain anything to you here, on third-party forum. As I specified, if one is interested in answer, he is welcome to use our support channels for assistance.
did code blocks just get highlighting?
@TomH you mean this thing?
ooh, we have moved out of 1990s and are progressing rapidly into early 200x
you know code blocks ps -ef
hmm that just nicked my backticks
@TomH some tags have code highlighting enabled
backticks are used for code blocks by markdown
@TomH has code highlighting enabled
Now it just needs to check that the code is actually the expected language, so that logs and CLI transcripts don't randomly get highlighted.
21:15
ah
that would explain it
@mgorven most code highlighting is disabled fro that reason on SF
10
Q: Vote: Do we want code highlighting?

Bart De VosCurrently, code highlighting is turned off (see balpha's comment here). It is possible to turn it on, if we want so. I think this would be nice to have, maybe on a per-tag basis, but turning it on is definitely the first step. Because it is disabled site-wide, we can't force it by manually spec...

@Iain I thought so. Enabling it when necessary is usually good enough.
@Iain Script
PowerShell is listed as a strongly preferred skill
In the hands-on section I have "Using the language of your choice (PS preferred), return a list of user names, samaccountnames, distinguished names, and SIDs in text file"
@MDMarra something simple like fizz buzz or determine if a point is inside a shape - simple oblong (anything else is very complex)
it's a one-liner in powershell after you import the activedirectory module
21:21
why have all these windows user come and started asking questions?
i cant answer those
reboot it
Careful, there are plenty of Windows admins here :p
I am allowed, I am qualified in the art of rebooting computers, I have an MCP
haha
there is much to be learned about the humble reboot,
there is the basic reset, the cold boot
the warm boot.
the "hold the power button for 7 seconds"
reboot
@TomH Reseting the CMOS to make the registers sane of the BIOS after a power spike can make a difference.
21:27
exactly
fml twitter just suggested "Justin Bieber"
@TomH chances are high that that guy is using bash - many systems just link sh ->bash these days
a real power user would have a switch wired externally directly to the CMOS reset
@rnxrx I've got one on my motherboard, but I have to open the case to get to it. I haven't had to do this on my machine for years though.
@84104 I think most motherboards have some mechanism. A bunch of my servers just have a couple of solder pads that need to be bridged with a screwdriver. My point (and I was generally kidding) was setting up an external switch to bump it when an OS completely noodles your machine.
21:39
@rnxrx My motherboard as a huge aluminium-looking backlit button labelled "CMOS Reset"
That's cos its meant for overclockers :p
is there a stat for processes?
ie something to give a useful summary if some running pid?
@tombull89 I love the last comment on the page from the same guy: Go ahead. You can as well write to your congressman or to Pope as well.
@Mark Do you have cold cathode tubes to make it extra awesome?
@rnxrx Yes shameface
But in my defense I havent turned htem on in years
They are from when I was younger :P
21:44
Awesome... some Red Bull stickers too, I hope? <grin>
@rnxrx Nope. V is my choice of sugar-laden energy drink
It also has a home-made window kit because back in '99 windowed cases didn't really exist, especially not in full tower
Weirdos. :D
@Adrian Its the nerd equivalent of lowering your car :p
Plus in '03 I got my first Dual-Socket system and I wanted nerds to flip out when they casually looked through the window
Hard-core is when you not only go with water cooling but also make it fluoresce with black lights
well, or you completely submerge the motherboard in a fish tank full of mineral oil or somesuch
I was always tempted to try watercooling but I just epically suck at plumbing
@rnxrx Ditto
And when you have a $800 motherboardw ith two $1200 CPUs...
21:57
heh - yeah... kind of my thought.. Nothing quite like splashing cold water on a really hot CPU. ouch.
@rnxrx My wife was frying some stuff in oil the other day and she put the frying pan in the sink when she was done and turned on the water to wash her hands without thinking. Big explosion. I found her in shrunk in the corner of the room as hot oil rained down
It's hilarious in retrospect

« first day (626 days earlier)      last day (4641 days later) »