@Chopper3 - you have some VMWare suggestions, right?

Four more days at my current job, thank god.

@vCole ?

@BartSilverstrim Start a new (sorta) on Monday.

Ah. What's the new place?

NetApp

my previous employer.

Long story short, moved for a girl, left a good job, we broke up.

Moved back home, and luckily was able to get my job back.

morning

@vCole You work at netapp? In the states?

@Basil yes

@MattSimmons what's shakin?

Hey, not too much

I'm awake, and that's all you're getting ;-)

How are things here?

Still standing.

I'm a zombie. yawn

s/yawn/BRRRAAAINNNNSSSSS/g

haha

@PeterGrace muffled crpumph in the distance...

There's not enough coffee in the world today.

/me backs out of the room slowly

Lotus Notes issue and the template is all in German -_-

@vcole scheissekopfs!

@PeterGrace lol thanks.

empathy, it's what's for breakfast

mmm. I like poached empathy

what is this em-path-ey you speak of?

Meanwhile, time to check trello for my list of "Crap I've Been Meaning To Do."

I took a look at licensing for Server 2008 R2 earlier - I lasted about 20 minutes before my brain fried and I went and rocked back and forth for a bit in a corner.

tl;dr, get someone who knows their stuff.

@tombull89 MS Licensing is the devil.

I'd rather peel my fingernails off with a screw driver.

It's not the devil so much as it's a schizophrenic patient.

Licensing for one person is different for another, depending on how much the agent selling the product likes you, so you can't get peer assistance in sorting that crap out. And $DEITY help you in an audit if you didn't have proof that someone sold you what they sold you.

Looks like I'll be training my replacement this week.

That should be fun, seeing as he doesn't have much server experience.

but they're cheap.

Maybe it's because we're a MS Reseller, but I don't think most of their licensing is all that bad. Some of it is terrible, don't get me wrong, but not most.

Whenever I'm in doubt about MS licensing, I always try to imagine the scenario that would be worst for me, and that seems to line up pretty well with what they want.

@MattSimmons Isn't that a summary of most interactions with companies/other people?

Hey, so I know snapping a DC is bad news, but I'm setting up a single-DC environment for hands-on testing during interviews for my replacement. Is it OK to take a snap of a DC if it's the only one and you really do want to roll AD back to a state immediately post dcpromo or should I just script the dcpromo and run the script after the snap is restored?

I understand why you shouldn't restore a DC from a snap in a production environment (non-authoritative, time out sync, etc), but I don't really know the implications of doing it in a single DC test environment where I just want it to be clean for the next user a few minutes later

It'll work

we used to do it all the time in our college lab

Ok cool. That's what I thought. I couldn't imagine a reason why it wouldn't, but I wanted to make sure.

Just make snapshot right after dcpromo

For what it's worth, it was a 2k8 functional level

If nothing is answering to it, I don't see what it matters to snapshot a DC.

You might even enjoy torturing it a little.

haha

ROLL BACK TIME, YOU LITTLE !@%!!

MUHAHAHA!

haha

Oh look I'm gonna create this user and he'll have this prof-PSYCHE! ROLL BACK! TRA LALALA!

What does this look like to you guys? stoneyforest.net/images/icons/tree.svg

Just make sure nothing is relying on kerberos tickets

because that will fail :)

also tested :D

@ChrisS Euhm, well, a tree?

@ChrisS a...tree? a birch tree?

I just saw this on a client system I sometimes manage...

for i in A B D E F G H I J K L M N O P Q R S T U V W X Y Z ; do echo ${i} ; rm -f ./${i}*.* ; done

@BartDeVos @Tombull89 Does it move?

Nope :)

can .svg's move?

@ChrisS ah, no. (FF 11, Win 7)

Safari, OSX

@BartDeVos Yes

@ewwhite ...why?

upload.wikimedia.org/wikipedia/commons/4/4f/…

@ChrisS Does in chrome

@BartDeVos upload.wikimedia.org/wikipedia/commons/4/4f/…

@BartSilverstrim I'm not sure... I go through root's command history every so often and see gems like this.

@tombull89 who knew :)

@MDMarra Well there's one at least. It's working for me in FF/Win7 too.

@tombull89 That one does move :)

@ewwhite That's an impressive combination of "I know what I'm doing" and "I don't have a clue" in the same line. Unless he really does mean "only remove filenames with extensions".

@MikeyB I think she ran into the "rm -- too many arguments"

I have all sorts of oddness on these client servers...

@ewwhite Yep, that's not a surprise. Teaching her about xargs now? :)

it's a lot of internal IT people who need to justify their existence...

grep "Apr 13 08" messages | grep -v named | grep -v n2n | grep -v dhcp | grep -v xinet | more

wtf?

Do they have named logging set to debug?

@ChrisS No clue, man...

Then I look at the last root login, and I see:

And this server is in Los Angeles... then I realize the other people who manage the system come from IP's that look like 1.0.5.x

(Linux is dangerous)

That's...a lot of trust...how many people are "managing" these servers?

For years, I used to. But I'm competing against my old firm, and we share a few clients.

but then, there are internal IT people at a lot of these places... so they do things themselves, too

That would still give me the heebie jeebies, but I'm a bit of a control freak about some things.

Oh, they mess up... I'm quite good at data recovery.

Remote data recovery?

Yep.

What, you edit things at the inode level?

and disk-level...

Remotely.

Dude, if you have writeups/tutorials on that, I'd love to have a copy.

I like reading up on those things just in case :-)

or my favorite was the application developers trying to build command lines in their program... dumping whitespace in the wrong spot (essentially rm -fr / directory)

@BartSilverstrim step 1 - vodka

But my go-to tool for that type of recovery is UFS Explorer

lol

@voretaq7: step 2, more vodka?

I used to write all sorts of crazy postmortems... and charge for it.

@JourneymanGeek step 2: fsdb and cojones the size of basketballs

@ewwhite serverfault.com/questions/393613/how-to-backup-file-system up your street

@ewwhite So it doesn't install on the machine as a utility, you get a disk image and open the image from a Windows workstation?

@BartSilverstrim Or a VMWare image... a Mac, Windows or another Linux system. The first rule of data recovery is to isolate the disk; prevent writes and mount it elsewhere.

@ewwhite Nice graphs...what was causing the full Var partition in the post-mortem?

@ewwhite I was just trying to figure out how you normally approach this recovery remotely.

Normally I'd shut down the system and mount the disk elsewhere or copy it with dd or something like that...

But remote repair/recovery would be like operating on the heart through the urethra.

@BartSilverstrim There was a motherboard issue on the system and the server sent an email to root every 30 seconds for 20 days.

Nice.

Root was aliased to the address of the people I compete with... and their Barracuda reject each of those messages sent every 30 seconds for 20 days...

So now it's aliased to an actual account? :-)

Oh...it was actually trying to send it somewhere but they never got the message.

lol

So...email + bounces...nice.

@ewwhite . . . . oy

The bounce message went back to root, which was aliased to the address that was rejecting messages.

After several thousand attempts to send to XXX.com, many of these messages began to bounce, creating an email loop as the bounced messages were redirected to the same address that was rejecting them. That’s how the system’s mail queue grew exponentially to 119,772 messages.

So a lot of my business at that time was capitalizing on the failures of other engineers.

@BartSilverstrim As for the remote repair, I'd dd over ssh to a local server or sometime tell the client to go get a USB disk from the local office supply store...

Remote recovery via UPS'ing the drive to your house...

Then mount the USB drive (after dd-ing) to a VMWare server... use USB passthrough and run the recovery that way.

Hmm- I wonder if I have to use getopt in AIX5.3 or whether I can use getopts...

Sometimes I wish we were using linux, if only because it's easier to find support for it on the web.

the worst one was the 4 million tiff files... 400+GB. I lost ~60,000 images.

@Basil Mindshare is important.

that's a lot of lost midget porn

Amputee porn was found on the system, actually... a samba drive.

D:

That helps sell a lot of Barracuda Web filters.

@Basil AIX 5.3 or later should have both

Maybe @basil can help me explain the RAID 5 problem to my client. I have a phone call with them in 20 minutes.

(might have it before 5.3 too, but I know 5.3 does)

@ewwhite "Well for starters you're using RAID 5"?

@voretaq7 But they need to understand in the plainest terms why it's bad for what they're doing. I don't want to sound too harsh.

@ewwhite I might not have an answer you'd like...

I use raid 5 for just about 900TB of storage

and have for years

@Basil VDI, Exchange, web and Linux servers... 80+ VMs... on a Dell Equallogic 14-disk RAID5 array (iSCSI).

It's only bad if you're a server admin. Storage admins like it.

@basil, you're probably using RAID 50?

@ewwhite nope, raid 5. We're on a 5 year old box that doesn't do 50.

@ewwhite You can tell them about silent errors on drives.

@Basil or 6?

We got bit by that.

@ewwhite That equallogic sounds underpowered for that kind of load... are they using 15k drives? Any SSD?

Drive failed. Replaced. Wouldn't rebuild due to another drive having an error that none of the diags ever caught.

@voretaq7 We use 6 for our journal volumes for replication as well as sata drives

We hates the RAID 5...

@BartSilverstrim That doesn't happen on storage boxes, only on servers

@BartSilverstrim Well, I'm trying to remember what the RAID 5 random read and write representations are... is the best read speed the speed of the stripe, and best case write speed equivalent to that of one disk?

@Basil Why's that?

@ewwhite The best case write is as good as the best case read, assuming the parity calculation is done as fast as the disks go

@BartSilverstrim I assume because storage devices cost so much more than raid cards.

@Basil No cache, no tiering, all 7200 RPM SATA in a single RAID 5

@Basil I don't know. Maybe you're right...I haven't played with storage devices. But I would imagine if the issue is with data density on the platter, the problem of undiagnosed errors would still exist even on high-end mondo expensive drives, wouldn't it?

@ewwhite bad setup. They need to use raid 10 here, if only for the writes. Even assuming the dell box can calculate parity as fast as their slow disks can go (which I wouldn't take without verification), they're still dumping a hell of a lot of IOPS onto some pretty slow ass drives.

In his case it's certainly bad, RAID 5 on 7200 RPM drives.

@Basil So there's no recourse other than doing it over, right?

Running multiple VM's. It's surprising you don't have disk queuing problems.

@BartSilverstrim Expense of the drive doesn't matter that much. It's the controller's job to tell when it's failing, and the drive's self reporting is only a small subset of the work that needs to be done to catch failures

@BartSilverstrim They do have problems. Really high read and write latencies.

@ewwhite Add another shelf with fast drives in raid 5, move the heavy hitters there, then migrate the old shelf to raid 10

@Basil True enough...I just don't know if they're immune to the problem until you get caught in a rebuild.

@ewwhite That's not because of the raid choice, that's because their storage is underpowered for the job they've given it

This is low-end Dell, but they paid more than 20K for it, so to them it was a big expense...

@BartSilverstrim They all have ways around it, mostly involving pretty regular scrubbing and pretty aggressive pre-failing.

it's hard for people in the SMB space to understand that product at that tier is still bare-bones.

@ewwhite They can get horsepower on a budget by using server attached storage.

They need 10 or 15k 2.5 inch drives

lots more than 14

or an SSD

@Basil I offered a NexentaStor box with an SSD tier as a proof of concept...

Price tag?

This is his reply to "You don't know what you're doing": actually i have installed malware bytes full PRO version and it blocks some ip addressed which is attacking on DNS, i m running this server from 2003 but this is the first time some hacking attemp on the server.

@ewwhite that would work, but again, SSDs are only cheaper per IO/s, not per GB

Poor TomTom.

@ewwhite What about zfs? Doesn't that do some sort of sub-volume tiering?

wha?

@gparent What are you replying to?

@BartSilverstrim One of these servers - About $8 or $9k all-in.

they could put some slow disks, some fast disks, and an SSD

@BartSilverstrim Oh, sorry, TomTom answered some guy who got hacked by telling him to secure his stuff and the poor guy responds that he installed Malwarebytes Pro.

@Basil 16 drive bays. 2 for boot. 12 for data in RAID 1+0. One MLC SSD for read cache. One small SLC SSD with a super-capacitor for ZIL (write cache).

@ewwhite Anyways, just tell them to ask their admin friends with similar workload how much they had to pay for their storage. I'd expect a proper setup for that would cost at least 80k, and require upgrades as they grow

Um...okay...that probably won't help much.

@gparent oy

@ewwhite hey, did that server you were trying to get rid of fall through?

I stumbled on the AD topology mapper a few days ago. It's a sweet tool for anyone interested in that type of thing.

@PeterGrace I have three systems now... Two ML370 G5's and a DL580 G4.

I never new it existed

@ewwhite Surely you mean flux capacitors

It's always a flux capacitor.

@MDMarra I used it when I left a job. just to document things for the next guy.

@ewwhite ML370's rackmount or tower?

@Basil So I can support that workload for $20k.

@PeterGrace 5U Rackmount

@ewwhite do you know what the weight are on those?

@PeterGrace 90lbs.

@PeterGrace And the DL580 G4 is ~115lbs. 4U rackmount.

@MDMarra oh I always forget about using that.

@ewwhite The G5's are newer than the G4 I take it?

I just used it to map out our sites/site links/replication topo for whoever comes here to replace me

BTW, if anyone wants my old job, it just got posted haha

higheredjobs.com/search/…

The department has it's quirks, but you'd be working for a great guy.

@PeterGrace 2 x quad core E5400-series. The G4 has 4 x 3.4GHz dual core hyperthreaded 7100-series CPU

Didn't know there was a "higher ed jobs" site.

@ewwhite When I was in sales for an IBM reseller, that would have been at least 40 drives

@MDMarra where are you going to?

@MDMarra $$$?

If you can do it cheaper, go for it :)

Lots of people getting new jobs :D