Part of why I quit $job[-1] was the 2 on / 2 off on-call rotation, and when I was off I was OFF (Not Available For Call)

@Cole they must like you more than the others

@DennisKaarsemaker I'm not a dick, that's why.

Or nobody else answers their phone when on call

@Tanner that too.

Also it's a one stop shop with me.

@Cole that's a sackable offence

@Cole If they're calling you before the primary tell them to knock it the fuck off.

seriously. There are very few reasons for me to insta-fire someone. That's one.

They cal for vmware, unix, linux, windows, storage, load balancers, DNS, dhcp, etc I can field

@voretaq7 they're not.

@DennisKaarsemaker can you come be my manager?

anyway, time to go home. Almost weekend

@Cole sure, make me an offer I cannot refuse :)

No one else in my group has that breadth of knowledge to handle more than 1-2 things.

(hint: you don't have the budget :P)

bye

@DennisKaarsemaker lol I know we dont. later!

I'm trying to think of a way to have a visual representation of on-call people. Right now they write it on the board in the help desk area - and email out the "list".

@Cole dun dun dunnnn

@NathanC so this goes back to ticketing thing.

@Cole A picture of shit rolling down hill on to each on-call's name?

Groping around a Wordpress installation for a radio station because no one knows how to paste an HTML element in to a page. What am I doing with life.

@Tanner lol

seen the GTV V online trailer?

@Chopper3 Is there a newer one than last months?

bcove.me/hfkm264t

So, 1st on call does tickets from 8-10, 2nd does 10-12 and so on like I've said before. My boss would like a visual representation who's on call so they know when they should be doing tickets.

@DennisKaarsemaker You better have a REALLY good excuse for not answering if you're on call. Like "My dog died. Really, here's the vet bill!"

@Cole Ah...the only time I really get calls is early morning when I'm on the way to work in the morning or before 5 when everyone leaves.

Instead of you know, doing your job like you should.

@Cole . . . you guys have a fucked-up on call system

Yes.

Headshot. Someone kill me now.

@Wesley por que?

This would be comical if it wasn't me.

I'll explain in a minute

I'm trying to make this work, with what's handed to me.

(and usually when you do something like that the off-hours people are not "on call" during business hours)

none of this BS "If it's 10 AM call person A, but if it's after noon call person B" bullshit. On call means we call YOU.

@Cole your on-call rotation:

you can't make sense of what's been handed to you @Cole, you need to replace the broken system.

@voretaq7 well it's not really my call - it's my managers. He wants it done this way - I've tried to argue with him about it.

Then just ignore it and let his fucktarded system fail

Otherwise tell whoever routes the calls not to call you unless they've tried everyone ahead of you in the rotation first, and if they won't listen stop answering your phone unless you're primary.

@Chopper3 I'd not seen that one. Thanks.

@Tanner I was thinking that - but like today, I'm not on call at all, so I'm not supposed to be dealing with tickets but they just keep pilling up.

Ok I think my explanation wasn't clear enough.

@Cole Then refuse them

Ok so you have 3 people on call - 1 primary, 2 backup 2 off completely - this rotates weekly. This applies for after hours 4:30PM till 8AM.

Seriously, at @job[-1] if I got a ticket and I was not on call I would reject it back to the helpdesk with a comment saying "[X] is on call, he can come talk to me if he needs me."

When we're in the office, from 8-4:30PM. My boss would like the people who are on call to handle incoming tickets in a shift like manner. All tickets get dumped into a "SysAdmin" pool - no one gets tickets signed to them directly.

Which again, mean no metrics and obviously no one sees who's doing more tickets than most. (Here's a hint, it's me)

@Cole Your boss is actually right here.

@Cole You're DOOOOOOOMED!

There should be an "on call" for business hours, and that person should get all tickets.

otherwise shit sits in a queue and nobody checks it

@voretaq7 I agree - people should be responsible for looking at them.

So he wants to use the existing weekly rotation to accomplish this.

@Cole that's fine too.

His hope is, that people will be like me - When a ticket comes in in that 2 hour range, that if you do not know how to do it....you get up from your desk and go ask the person that does to show you - not do it for you.

Currently, everyone (except me) ignores tickets they don't know how to do.

The Unix admin refuses to touch anything Windows.

@ewwhite I use raid 6 a lot, so no cringe from me.

So when he's 1st on call, I'm 2nd - so I get the shit end of the stick when a call comes in for Windows after hours and during the day with Windows tickets.

@Cole Funny, I refuse to let Unix admins touch anything Windows because they invariably fuck it up

No "shift" bullshit - you're on or you're off, otherwise your help desk will fuck shit up

@RyanRies welllllllllllll I'm both. My title is Unix/Windows Systems Administrator.

@voretaq7 how do I propose this to him so he gets it?

@RyanRies and we refuse to let Windows admins near the Unix machines because when they see a shell they piss themselves :-P

@voretaq7 that's how the other 3 sysadmins are.

And that is the story of why there will always be a divide between Windows and Unix admins :(

PLEASE tell me why we have 5 sysadmins, 3 that are windows only, 1 that is Unix/Linux only and then me. But I'm the lead Active Directory/Windows guy.

Except Cole, he's the missing link

@Cole If he doesn't get why shifts will lead to the helpdesk misrouting shit and tickets getting lost he's an idiot. Get him fired or at least castrate him so he can't breed.

>.< and yet I sit here comfortable with Windows/Unix. Well, I'm unfamilar with powershell but people really should be willing to LEARN in IT.

@Cole Everyone else is incompetent? :)

@voretaq7 well the help desk just dumps them into the queue

@Cole yeah, they can't do that

they need to be assigned to someone (or your queue needs to know about the on-call rotation and notify ONE person.

So should I just give up, and continue my search for a new job - or try to fix this?

try to fix it

but don't stop your job search :-P

:P

I'm not sure how to approach him and say "look your way sucks - here's a better way and we're going to do it this way because I said so."

@Cole Depends, what does your sanity meter say?

@NathanC I pretty much can do whatever I want. My boss loves me -so it's fine with me. My ability to handle both stress and multitask is much higher than the average person.

@NathanC he's hanging out in here. What do you think it says?!?

"I found who does our website. At the bottom it says 'Wordpress' but I don't see their number. There's no one to call!"

@voretaq7 Oh...right. throws sanity meter into the pile

@Wesley LOL

@Cole Tell him a $10M/yr company used the system I described above successfully for 4 years (and AFAIK they're still using it)

@Cole Then I'd just be up-front with him and say that it's not working.

@NathanC we haven't even tried it yet

He wants me to get everything together first to "try" it.

@Cole I guarantee you it won't work -- it's too much work for the helpdesk

Or not going to work because people will just not follow it

he's violating the KISS principle anally with @Wesley's concrete vibrator, and not even buying it a barrel of l00b.

@voretaq7 well the help desk doesn't really do anything other than assign the ticket to the queue

Unless it's a P1

Would any of you consider having unbalanced VMware clusters? For instance, 6 x dual-socket servers and 2 x quad-socket servers in the same VMware cluster?

@Cole and one person needs to be responsible for the queue (and ESPECIALLY for P1 incidents). There needs to be a direct path

@ewwhite Sure

@Chopper3 I also have some blade questions for you.

@voretaq7 which he'd like it to be the first person on call.

you're using DRS right?

The issue really is - no one person can handle everything. The only thing I can't help with is Domino

@Cole During off hours it should always be 1st on-call

@Wesley the only way I could feel the level of pain I think you feel is by sandpapering my whole body, and rolling in salt.

@voretaq7 The client has 576GB RAM in the cluster, and now wants 2TB in the cluster.

and yes, @voretaq7 DRS and all of the goods.

we had the backup person be the business hours ticket bitch just so one person wasn't JUST doing tickets

@ewwhite Trust DRS - it's good at dealing with asymmetric clusters

^ how i imagine the "on call" calls going

It's BETTER if your cluster is symmetric (because you can deal with failures easier if you don't have to worry about losing one of the "big" machines), but it works pretty well with asymmetry.

@voretaq7 well back to my example - Unix admin is 1st on call - doesn't do tickets all day, me second on call handles tickets all day. At night he gets calls for Windows - gets deferred to me every time and then when I'm first on call - I'm still getting calls since then it's like being on call 1st two weeks in a row. That's not fair, right?

@Cole Honestly you should have two groups of on-call -- Unix and Windows.

Well then I'm in both

So I get fucked either way

nope, you have to be one or the other :P

you'd go in the group with the lowest staffing

@voretaq7 Well, the scale is messy... The client's requirements changed substantially.

Won't happen

you can still do both jobs, but you can only be in one on-call rotation

the way you're doing it now wastes time: If a unix-only guy gets a Windows call it's gotta get bounced around.

It's like they baby these guys.

Grow some balls and tell them to do their jobs.

Our rotations were Unix, Windows, and Storage

See - I'm responsible for all of them.

I'm the only person on the team responsible for 98% of everything.

Only reason I don't do Domino is because the Domino Admin is a info hoarder.

(yes, Storage got its own group - usually a storage ticket meant also waking up the Unix or Windows team, but it was WAY better when we had the SAN guy already awake, caffeinated, and then we got called to log in to the server and reset the HBA :-P)

Complains no one helps him then won't share information.

@Cole I thought you didn't do Domino because that would violate the Geneva Conventions.

Fun Fact: I've been a Domino admin twice in my career

@Cole 3 times is torture.

Honestly, I learn it because I hate waiting for someone else to do it for me.

@ewwhite sure, blade away

I've become my manager's right hand man. Or little bitch - either way you look at it.

@Chopper3 You may be able to answer the Vmware question, too...

@ewwhite not sure I've seen either

Hmm need to come up with a naming standard for DNS/DHCP/File sever/print servers for our smaller sites.

prepping for a small holiday

So VMware first... As I just posted, client has 6 hosts in a cluster. 576GB total. 96GB per host. Platform is DL360 G7.

Things I learned today: Our on-call rotation blows.

@Chopper3 They now want 2TB of RAM available in the cluster.

I'm trying to figure out how to get them there...

@Cole two/three-char location identifier, three-char job class identifier, sequence number - i.e. nycweb01

Option #1 - Add more G7 hosts and bring them to 192, 288 or 384GB RAM. Downside is power and space.

@Chopper3 some of these sites are actually sub companies with different names. But I'm wonder if we do go that route

@ewwhite what have they got now

but what would you call a 4 in 1 server?

DEALLTHETHINGS01

@Cole an opportunity to virtualise :)

@ewwhite so what's the blade part

@Chopper3 these are single sites though - one server.

This is the VMware question

@Cole esxi it

Option #2 - Add a couple of DL560 Gen8 servers with 750GB RAM each - Minimal rack footprint. Low cost.

@Chopper3 I thought about it but they don't want to since it'll only ever be a single file share

They'll never add another server. It's for 10 people.

@ewwhite so what's the requirement, 2TB memory across the board right, how many cores and how much storage?

@Cole oh well, call in SevenZarkSeven then

@Chopper3 This is an existing environment, built a few months ago. Cores and storage aren't issues. RAM is.

@Chopper3 @Cole We do something similar -- purpose code & node number, <DOT>, site, <DOT> example.com

Client has basically quadrupled the RAM requirement... and out-scaled the servers they have

@ewwhite ok, well the cheapest thing would be to take the hosts up to 384/768 then

@Chopper3 lulz.

Not really... @Chopper3

eg app01.ny.example.com ; or app.qa.example.com

@Cole do you get the ref though?

6 hosts.

@ewwhite but 6 x 384 = ~2TB

@Chopper3 lol yes

@voretaq7 ours are horrible - 2chars location, 1 char OS, 5 digit 'platform ID', 3char job, 3 digit identifier - hence rde10082app001

I think from now on for these we'll do location, FP(file and print) and number. FP = DNS/DHCP/File/Print in our doc.

(I have site as a domain component for LDAP reasons, also our v4 scheme is 10.<site>.<vlan>.<host>

@chopper3 384 requires 12 x 32GB DIMMs. Across 6 servers == 72 DIMMs. DIMMs are $1000 each

$72,000 to upgrade.

@Chopper3 we don't have to worry about OS in my environment - it's all Unix (mostly FreeBSD ; there's gonna be one Debian box)

$_$

@ewwhite 24 x 16GB surely?

@voretaq7 corporate thang

18 slots

ah - G7, soz

@Chopper3 If you have a heterogeneous environment encoding the OS makes sense

so something like appU01.<site>

or NYAPPU01

I've started naming our servers...since we only have one location I can be like SQL1 or FILESTORE

@ewwhite c3k and 2 x BL660?

Caps to make it stand out in reports from workstations

Just don't become like <mumble> - WPLSNYA1, CBETNIS01, etc...

@NathanC Plan on site names

because you'll have a dev environment one day :)

Ours are pretty straight forward here

@voretaq7 What about the "mash keyboard to get characters" method?

FILESHARE, NEWPRINT

@Cole GNUPRINT?

:-P

CORPDC# (for local DCs) and CORPDC-<location> for DCs in other sites.

ugh, dashes

I don't like dashes

@voretaq7 I know

@voretaq7 Even if we have multiple sites, everything will come back to our main plant since we have an actual "datacenter"-like room in our new building. Raised floors, conduits, the whole thing.

Dashes piss me off.

we use dashes for stuff like "app-db" or "backup-db"

but those are virtual IPs and the name is convenience

Luckily, my manager seems to value my opinion on how we should fix a lot of stuff.

no dashes here =]

I'm starting to work with the help desk/desktop to improve their processes too.

e.g. right now APP01 is the APP pool's DB master, so it's got the app-db address

I was thinking of creating the new DNS names as a CNAME so things are a bit more sane

@Cole I use dashes for "convenience names"

@ewwhite so that's only 6U used, plenty of resilience, space for two more (i.e. 4TB in 6U), use virtual connect flex-10/10D's and you only need to wire up 2 x SFP/+'s and 2 x gigE's for the OAs

I also use unqualified names (like "db" is our production clinical database)

Sometimes I feel like I'm getting bent over and railed for all the shit I do.

actual systems are XXX##.<site> though, and "convenience names" are NAME.<site>

@ewwhite or a couple of 380pg8's?

No blades

@voretaq7 SO YOU DO WORK AT BRAZZERS!

@voretaq7 Do you use CNAME records for stuff like that?

@NathanC usually the convenience names are load balancing pools or some other VIP

We use CNAMEs for some things.. Mostly when users might be typing the name in

"db" points to a CARP address that's shared by the primary and standby DB - so after a failover we don't need to change anything

Like mail.$domain.com is a CNAME for exchange02.$domain.com

"app" points to a load balancer pool of all the app servers

Can you guys tell me if I am getting the shit end of the stick here?

@Wesley When I first moved to Boston, a room-mate possibility was a guy who worked in Marketing to a company in Cambridge named something like "Adult Buddies Inc" (forget exactly). They run manhunt.com

@Cole Yes.

I seem to be the only one responsible for learning more than 1-2 things. Everything from Websphere to storage to AD to Unix etc

You shouldn't be responsible for planning a sane on-call cycle. Your manager should be.

He likes you because you probably do a good part of his job lol.

@NathanC ...though sometimes it's better if we do these things. Management incompetence is dangerous.

Then on top of that, I will be assisting the help desk/desktop manager to get their processes in order and ours.

@Cole that's actually a good thing

I pretty much do .......everything.

because if your manager and that manager get together it becomes a process & procedure circle jerk and everyone's workload goes up EXCEPT theirs.

Didn't really occur to me until I talked about it in here.

@Cole are you well compensated? :-)

@voretaq7 $85K/year

$2600 on call stipend a year. So at $87,600.

@Cole yeahhh they should give you about 10-15k more

@voretaq7 I am in the suburbs of Boston though - with 5.5 years experience, no degree and no recent certs. I'm assuming that comes into play.

@voretaq7 They should, but some companies in metro west can get away with paying that or even less for a good jack-of-all-trades

that's not a bad on-call stipend (figure you're on call half the year - it's about $100/wk)

@ewwhite ok, well what's wrong with 2 x 380's then? that's about as low as you can go isn't it?

@voretaq7 yeah it's not.

@mfinni I'm making 98K doing about the same work but for a much smaller company

@mfinni ah you're in Boston - see I'm in Milford (live in Franklin) so that's my thought process.

(we don't have clients making helpdesk requests, but I do our integration and deployment so it balances out)

@Cole That's where I was at, when I started out in Boston 5 years ago. Got some good jumps in the last few years, both raises and job-hopping

And here come the "PTO guyz, 3-day weekend nah-nah-nahnahnah!" emails

@mfinni I was commuting to Waltham - but it was killing me.

I need a raise in order to pay for my girlfriends Kindle habit... never should have bought her that thing... >_<

I have a ~10 minute commute now. 2 exits up on 495.

@NathanC and at the moment I'm working from the lobby of my car dealer

I could do most of my work from home, but here I am in the freezing office >.>

@Cole I bet. There's plenty of good-paying tech jobs out near you though. Hell, EMC is all over your hood, they're not the only one but a good example.

I like living and working in the city, myself

@mfinni yeah I've applied a thousand times. I don't particularly like the city. Franklin was recently rated the #1 safest city in america :)

I'm originally from Northern RI - so I'm from the 'burbs.

neighborhoodscout.com/neighborhoods/crime-rates/top100safest

lol I live in Franklin Center which is the most unsafe part of town - but it's still super safe.

@RyanRies Or resort to uh...other ways of "acquiring" the books

Is this book worth the $$

amazon.com/Active-Directory-Cookbook-3rd-Edition/dp/0596521103

Maybe @RyanRies could recommend a good AD book

@Cole I'm not a fan of the "cookbook" books myself

@voretaq7 poop.

Wait maybe it was this one:

amazon.com/Active-Directory-Designing-Deploying-Running/dp/…

@Cole I'm from East Lyme CT myself; loving the city. Would prefer to live in Camberville, but my girlfriend works for City of Boston so we live in JP

that one may be good - I just find personally that things work better when I read the books about the theory and such, talk to smart people, and come up with my own "recipe"

@mfinni oh snap! I lived in Gales Ferry for 7 or so months.

@Cole that one I'd seriously consider -- page through it at a bookstore if you can first.

Worked in Chester, CT for that time.

@voretaq7 what is this "bookstore" you speak of?

@Cole Heh - SECT. The land of ... traffic jams on I95 from May through August

@mfinni hated it down there.

@Cole it's like a library but they charge you money and you get to keep the book

Nothing to do.

@mfinni FUCK I95.

@voretaq7 huh...I think there's a Barnes and Noble on the way home actually

@voretaq7 werd.

@voretaq7 With a rake.

I spent a year on I95 driving back from boston one weekend.

WORST. ROAD. EVER.

Seriously, totally worth $150 for the fucking ferry.

@Cole I was able to have a lot of fun in New London : the downtown music and arts scene has a lot of good friends of mine

@mfinni yeah we used to go to this dive bar in New London

@Cole Ernie's? Brass Rail?

God can't think of the name of it.

My new policy is if I have to go anywhere that involves "Take I95 through Connecticut" I'm taking the plane.

Been to the Brass Rail - oh The Dutch

hmm $35 for the newest edition of that book..

@Cole I love the Dutch. had some friends that lived above it. Surprised we didn't fall through the floor into the bar some night.

@Cole if it's as comprehensive as the cricket book? TOTALLY worth it.

@mfinni me too!

@voretaq7 I'm hoping it is - I really need to get more in depth with AD

You know Rich Martin and Matt Potter?

@ewwhite what was the blade question?

Right...clearing cookies kicks me off everything.

@NathanC here ?

@mfinni I do not - my friend Leo used to go there a lot, lived in New London and worked there too

@Iain Yeah. Logs me out.

@NathanC yep

you'll probably need to log in to the main site too.

Alright brb - I need to go eat lunch and take a break from trying to brain.

If you logout of the main site it warns you that you'll be logged out on all devices

Yeah, I logged in at the main site then the chat let me log in too.

@Cole Good idea, about that time for me too.

@Cole There are some OK Active Directory books out there, but what you will find is that they will all cover the basics that we already know just for working with it in our jobs. So they probably won't satisfy you. Honestly, there is an entire book's worth of reading right here and I would tell anyone interested in AD to read every link on this page: blogs.technet.com/b/askds/archive/2010/07/27/…

@RyanRies Annnnd bookmarked.

@Cole The other thing is that there are bits of AD that have remained constant since NT 4, and yet there are other constantly evolving features that even a 1 year-old book won't know of, which makes blogs a really important staple for us if we want the most up-to-date info.

Yeah. Most IT-related books are outdated by the time they're published.

I'm still sad that Ned Pyle left AD Support, he was pretty much the life of that blog, and a damn smart guy.

Which reminds me, I keep forgetting about the Network+ test -_-

Wow, this place brings back memories: http://www.jeah.net/

devopsreactions.tumblr.com/post/44054856083/… ...the guy in the blue slide is me when i started here lol

@Wesley Jeah is still around?

They're the ones we sold our shell provider to when we decided to cash out :D

So far I have a link to a document without the information I need, and a suggestion to use NFS instead of FC ;)

@voretaq7 Apparently so, I just got contacted by someone using a jeah email address for some SMTP redirection.

@voretaq7 Also, their Twitter and Facebook accounts are surprisingly active for a rather neckbeardy service.

@Wesley IRC eggies are "neckbeardy"? :P

I'm not even sure what to say... I guess their audience can do the math anyway...

@ChrisS Not bad for an 89 year old.

^Related

Also, there's good reason the guys are paying attention in high school math....

@ChrisS One of our math teaches in HS was a Penthouse model.

Also, semi-interesting article on why TLC is probably going to be doing better in the future: theguardian.com/commentisfree/2013/aug/07/…

@Chopper3 We've decided to double the RAM in existing DL360 G7s... adding 16GB DIMMs to the existing 6 servers. That gives us 1152GB. Then add 4 new DL360 G7s with 192GB RAM. That brings us to 1920GB cluster-wide across 10 identical hosts for... $33k

Good afternoon, folks.

@ewwhite cool

So the blade question... @Chopper3

k

My HP vendor needs a way to test blades before shipping them.

being a vendor, they typically run standalone servers through a smartstart disc

but for the blades, they don't want to go blade-to-blade to setup.

Is there a nice way to get KVM control of all blades?

@ewwhite well if they can get a KVM that does that via USB they could just connect it via a bunch of the standard 'W' cables at the front

There's no easier way?

Without logging into Onboard Administrator, etc.