So that's the downside... I don't know the application architecture. I'm just here to try to fix its crashes

Then a bunch of...

check the post queue, there must be gazillions of mail waiting in line

INFO: task varnishd:11260 blocked for more than 120 seconds.
 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
varnishd      D 0000000000000007     0 11260  11258         11261 11259 (NOTLB)

postfix lets you see the subject etc (postqueue -p afaik)

and the system dies.

@HopelessN00b Sucker!

@HopelessN00b trust me, NOBODY but IT staff will understand the true cost of storage

@ewwhite Ouch.

@pauska yes, gazillions... to all sorts of random domains

allthough it's a bit alarming that a CIO doesn't know the difference

@pauska Nobody but IT staff will understand the true cost of cheap storage.

This is the underbelly of DevOps...

@ewwhite I'm kinda hoping you're serious and know of a better deal on those 2TB nearline SAS drives... I could use some for my home server... and someone will notice if I steal any of those. :)

@ewwhite The fact that they don't understand the engineering to build their shit right?

@pauska Doesn't, but postcat -q $QUEUEID will show you the message.

@ewwhite the underbelly of devops is most of SO and the sub-continent

@voretaq7 No, everyone learns it when the company loses all its databases for 3 days because of cheap SATA drives.

@Adrian The fact that they're probably an open-relay and don't know it

@84104 not the entire message, but doesn't postqueue show you subject/from/to?

@HopelessN00b they still won't understand.

@ewwhite sigh If it's not their public MX, why is it listening on an external interface?

they'll blame the systems/storage team

@pauska Whole message. -h -q will show you just the smtp headers. -e -q will show smtp evnvelope info.

@HopelessN00b ~$250/ea without carriers.

@voretaq7 No, they get the cost of cheap storage then.... doesn't mean they're willing to pay for real storage. Just means they buy the expensive, better SATA drives next time. :)

@ewwhite Oh? New? Any warranty?

FFS, my system engineering/kernel skills might be thin, but that's just bonehead.

@pauska 209.xx.xx.89 is a suspected spam relay by way of a very low SenderBase Reputation Score , see senderbase.org/… for details.)

@HopelessN00b because they don't understand - there's a difference between knowledge and understanding :)

@HopelessN00b New.

@voretaq7 Right, fair enough.

@ewwhite OK, I'm interested. Where?

@Adrian My MSAs listen to the world, even on port 25. Damn legacy users.

@ewwhite Hm. Sounds like they are mass-mailing without getting whitelisted by the big ones first

ST32000444SS is the part... I have a guy on Long Island

@84104 Old stuff with no iptables?

@ewwhite At that price, I'd want 12. Doable?

@HopelessN00b probably.

@Adrian Hardware old, OS/packages new. Postfix anvil rate limiting YES!

No system level firewalling.

They do live on their own little subnet.

I'm asking supplier online now

@84104 Yeah, I don't let that stuff play with the rest of the network here.

and the secondary source also has them for $272...

I don't even let the receptionist's stamp machine talk to the staff network. It's over there on the client network fending for itself.

@HopelessN00b Just go direct... serversupply.com/HARD%20DRIVES/SAS-6GBITS/2TB-7200RPM/SEAGATE/…

I've dealt with them for a long time as well...

@ewwhite Oh, yeah, the ones I bought at the office are all the rebranded, official "HP" drives that they markup 100% or whatever (and warranty for 3 years).

@HopelessN00b oh, do you need the HP disks?

@ewwhite Nope, that's perfect. The home server I'm putting this in is actually a Dell disk shelf, so Dell OEM is perfect.

So what's the official word? Is the Plesk, WHM, Cpanel stuff off-topic or not?

@voretaq7 You... you...

@Adrian as with most things, it depends on the context.

If you're installing it, then I'd say yes. If you're using it, I'd say WebApps or Webmasters. Maybe even SU.

@Zoredache that's not a very helpful metric.

@ewwhite Sweet, I'll place the order as soon as I get done with this work shit I'm on now.

If a person can ask a decent question in spite of using one of those, it might be on-topic.

@HopelessN00b They're also known as NYCE - nyceonline.net - Ask for Mayunk and mention "Ed White"

@Zoredache so, effectively off-topic but it won't be acknowledged by the mods as such?

@ewwhite Will do. Make sure you get some nice swag or something for sending me their way. :)

@Adrian do you have a question in mind that is bugging you? It is a lot easier to talk about a specific example.

@Adrian they are more likely to be crappy questions than OT. What @tombull89 says for the rest except I'm not convinced webapps considers them on topic

@Zoredache No, the problem bugging me is the avalanche of those things.

@tombull89 See, I can buy that one. I'm tired of the HALP ME SET UP DOMAINS. URRRRRRGENT!

derpaderpaderpaderpaderpaderp

@ewwhite And the order of that is very important. I once tried beer tasting, then Exchange, then building an AD domain... it did not go well.

that sounds like a easy job

@pauska Doesn't it?

except the part about intergrating AD into their devops cloud blah blah

but it's listed as optional

Honestly, I was almost tempted to apply and say I could do it from remote, it sounds so easy.

I'd do some research first and find out what they're looking for in detail. Suddenly "migrating all local user profiles from windows xp to the new AD domain so that windows 7 can use them" is a part of the project - who knows

yeah a job like that can be done remote, but it sounds like they want someone who can show up (the beer part)

@pauska Not so much research, as make sure you get an ironclad SoW out of them first.

@pauska I'll go to CA for beer. I've gone further for worse reasons.

yeah, guess you could take the trip and just say hello (and get hammered with them), then do the rest over remote, and another trip when you're done

Last thing you want is to sign on and then suffer - what the term? Feature creep?

See, I like engagements like that.

It seems to be my future since I'm an awful employee

@tombull89 I think you mean this: theoatmeal.com/comics/design_hell

@ewwhite You didn't get that new job you were talking about?

Oh, I did...

but I'm still a bad employee

sysadmins are generally not the easiest people on earth

so don't banter yourself :P

I'm just too demanding..

@pauska Hell, if it pays what it ought to, I could tell the current place I need to go out of town for a couple weeks, do 80% of the CA job, come back and do the rest remotely, or fly in on the weekends.

we're looking for new storage to provide clients... and I want ZFS... and there's a guy who likes a Linux-based solution

hahaha

@ewwhite just tell him that the rest of the world either A) buys a expensive SAN/NAS, B) Uses ZFS, and in the future C) uses windows storage spaces

@pauska Oh, right, thank you. Yes, I was "joking." I've never done that at least two or three times.

so I'm trying to learn how to be more humble and such.

We have EMC, Isilon, but need a lower-tier

And ZFS is a lower tier that could outperform the EMC and Isilon

@ewwhite One approach. I find it easier to figure out where the hookers and drug dealer hang out, and what car the other SAs drive.

Man, it's NY... these fools don't drive!

HowTF does a 2.3G PDF maintain any kind of stability? I'd think it'd be a sufficient mess than the app would crash before it got to the printing part.

@ewwhite is what the train is for.

or cabs

@Adrian Unfortunately, it's making it to the print server, which its crushing. The fact that there's not just one of these (and two people tried to print a 2+ GB file at once) is causing me huge headaches.

Honestly, I think ServerFault's primary problem is it doesn't really have a community of people who both answer and ask questions like SO. Lots of low-rep people crawling in looking to suck on the Answer Teat and couldn't give 2 shits about anyone else or contributing

You're not GOING to get quality when 75% of the questions are mediocre at best.

I don't know how so many high-rep people ask so few questions

@ewwhite because they probably don't need to ask?

They know the stuff they do, and they don't often work on stuff they don't know?

^^2.3 GB print job. Freaking lusers.^^

the joys of PDF/PowerPoints

100kB file magically turns into a 3MB print job

@ewwhite I don't know how so many low-rep users ask so many questions. Do they not know how to use Google?

@Adrian My guess too. Of course, the more I whore myself out on the side to the technically clueless, the more I find myself with questions I can't find the answer to, or get support on, and have to ask here.

@Adrian I ask all sorts of stuff... "How do I make sure Playboy's email makes it to their destinations"

@ewwhite 86 questions. 1,275 answers.

@ewwhite I though tit was brazzers email. :)

@HopelessN00b different client

careers.stackoverflow.com/jobs/24381/…

"Come work on a high-performing, passionate, and bleeding-edge DevOps team"

now THATS how you attract lots of sysadmins!

Well, maybe they pay SAs half a million a year to compensate.

"You need to be our systems administrator + argue with developers on how to do your job"

Oh, it's in Utah... maybe a full million a year to compensate for DevOps + Utah.

@pauska ALthough I get to do that anyway, and they're definitely not DevOps.

@HopelessN00b They'd have to do helicopter drops of booze to get me to work in Utah.

64 jobs for “sysadmin devops”

holy shit

are we becoming the same type of people we laughed at 15 years ago? those old farts who were programming in AS400 and said everything else sucked?

@pauska 64 employers added to personal blacklist. <-- FTFY.

@pauska Yes

@pauska Who gives a crap? You know how much those guys make these days?

@HopelessN00b now that's a valid point

@HopelessN00b what, all 10 of them?

@HopelessN00b a lot?

@Zoredache Please. Government, finance, manufacturing, logistics... you know how many companies still have an AS400 at the heart of their IT infrastructure, and at least one team of AS400 "programmers", all making over $100,000 a year to do glorified scripting and DBA work?

Straight up, if I wasn't so easily bored, or was 20 years older, I'd learn COBOL, RPG and OS400, and be making 200k for 20 hours of actual work a week.

I've worked with dozen of people much dumber than I who've done just that.

Honestly, if you want to make bank COBOL programming was still pretty good last i checked 5 years ago. 120k because there were so few doing it compared to the need to support legacy apps. I'm pretty sure WSU's billing and class scheduling software was written in COBOL up until 3-4 years ago.

@Adrian Only downside being having to program in COBOL everyday. I think I'd rather replace my car seat with a pile of broken glass and dirty needles.

@HopelessN00b In 20 years, you'll be 20 years older, that's probably enough time to learn enough COBOL to be nearly pro.

The money is in helping people move from that legacy tech to the newer stuff.

@HopelessN00b Can't be any worse than working here.

@jscott COBOL just isn't all that damn hard either.

I was pretty good at COBOL and I'm a mediocre Dev at best.

@Adrian Syntax wise? Sure, nothing really is. Learning the intricacies of the deployments you'll need to support?....

@ewwhite Today. Give it 20 years, and we'll be the old farts writing the legacy Windows and Linux apps the young kids are making piles of money to move off of. :)

@jscott Exactly. COBOL takes "spaghetti code" to a new level. It's practically all spaghetti code by design. <shudder>

@HopelessN00b can we hope that PHP will go away in this future you imagine?

Producing bug-compatible migrations sounds like a completely infuriating mission. I have the utmost respect for those that are able to eat the old to produce the new.

@Zoredache But then how will the DevOps MAMP stacks function????

@HopelessN00b I'd have to say it's more like piles of 2x4. There's nothing that light about COBOL.

@Zoredache Can they take Flash and the client-side JRE with PHP? I'm pretty fucking tired of the update-treadmill/exploits/version-incompatibilities.

There's been 4 COBOL Dev positions advertised in the Seattle area for over 80k in the past month or so.

@HopelessN00b I might need a liver transplant by the end, but I wouldn't need a glass septum either.

@Adrian If the market's anything like the one here, they can expect to be paying at least 1.5x that for anyone with any experience too.

OK, so if we can switch gears from one dinosaur technology to another for a moment...

This PITA print server, on Win2k3 has a very large folder on the system drive named Printing Subsystem Backup. Anyone ever heard of such a thing?

@HopelessN00b what is the full path? Is it right off the root, or what? What is in it?

@Zoredache Right off root - C:\Printing Subsystem Backup\

It almost sounds like a 'print migrator' directory.

Yeah, but we don't use anything like that.

@HopelessN00b It's probably about the same or a little better. Like any other job listing, it's like a kid's Christmas list. They want everything and a pony too.

It would have been used to upgrade a print server 2000 to 2003.

It isn't like it would be something you use after the system is setup.

Are the files in the directory ancient?

2010, 2011. Could have been the time frame for an upgrade of this box, sadly.

I would probably move it off to a temporary directory instead of just nuking it. But I might be a bit overly cautious.

@Zoredache Got a month worth of Avamar backups, plus previous tape backups of this thing, so not worried about "losing" that stuff... though I secretly hope the system mysteriously dies, so I'll have to replace it with the three 2008 R2 machines it should be.

@HopelessN00b Kinda like that old SCSI160 RAID system that's creaking along for us.

@Adrian Sounds like it, only more ridiculous for us, since this is a VM I'm being prevented from upgrading/replacing because... well, "because," that's why.

@HopelessN00b Ah. And their opinion on cloning it so that you can actually mock up the upgrade and test the plan?

@Adrian Can't do that either. Same reason. Any time I feel the tiniest bit bad for my bill rate relative to the basic shit they're having me do... I remind myself of the last 5 reasons I couldn't do it the right way, like I suggested.

@HopelessN00b Then they're just being bull-headed. So they get to pay for it.

Exactly. Idiot Tax and all that.

@Iain Ya know, it's tough to do my part in downvoting crap when the downvote penalty could easily outstrip the upvotes I get.

@Adrian Damn your reputation. Full downvotes ahead!!!

cackles

My fingers refuse to cooperate on days that in in 'y'

@Adrian: if its really bad, it'll get deleted and you get the rep back.