@ewwhite: If I type in uptime and press [Enter], I should expect to see the load averages in less than 0m0.010s, not after 6m49.891s. (Yes, that is an actual result from time.)

Considering you are using a decent PERC, perhaps try disabling OS write cache

This discusses an issue where the os write cache is so large, when the system flushes it, the system grinds to a halt. Where as, if you make it smaller, it is only the application that will grind. linux-kernel.2935.n7.nabble.com/…

@Patrick: Disabling the write cache (sysctl vm.dirty_background_ratio=0 and sysctl vm.dirty_ratio=0) eliminated the OS freezing as a side effect of reducing I/O throughput by a factor of 165. I tried playing with the ratios, but the whole server I/O still crumples at the hard drive's 200MiB/s sustained write.

Yes, try small ratios or set dirty_bytes directly. dirty_background_bytes is when to begin async flushing, dirty_bytes is when to lock up and go synch. Try as low as 16. Double check settings took hold. Sorry thats all i've got :) LUKS would be the next culprit? Keep in mind, PV is not 'realworld' usage.

@Patrick: I initially suspected LUKS (specifically, the dmcrypt_write kernel process), but it wasn't bottlenecking, and the running kernel wasn't affected by known issues such as this one. The server should not buckle from erasing a hard drive with pv. (ATA Secure Erase is not available for other reasons.) I've wiped all sorts of drives with pv before, but the other computers and drives chugged along just fine.

Ok. "problem free control disk is conected to the motherboard" - So, the disk that won't flush is on the PERC, and the disk you are wiping is on the PERC. Perhaps the PERC is locking up?

Unlikely. I'm reading at a total 1600MiB/s with all eight drives right now. There's no evidence that the Dell PERC can't handle I/O.

PERC 310 is a very basic hardware raid

Does this occur without LUKS? FYI, i dont know what luks is

LUKS is part of an encryption layer provided by dm-crypt. There is some CPU overhead, but it's less than 10% of one core most of the time at the fastest speed the hard drive can push, which is how I figured the bottleneck wasn't there.

Sure, PERC H310 is the entry-level hardware RAID, but I'm not using its hardware RAID features, and it shouldn't be struggling with just one hard drive maxing out sequential writes.

Yep for sure.

When the IO locks up, try some other methods of confirming the lock up. Im thinking linux bug report.

Chuck one of the fast drives onto the SATA?

Sure, I can try that.

Oh man, it really could be the PERC.

I'm trying that pv /dev/zero > /dev/mapper/data1 again, and the rest of the server is staying responsive.

I hate raid cards :P

No kiddin'. :P

Or to be more precise, the fact that dell and hp sell these things. for example, the b140i from hp.. raid5 pulls 12 megs a second. 12megs a sec from a 3grand server. Just a bit of a joke

That's ridiculous. I've never worked with that Smart Array before, but I'll take your word for it.

google "hp smart array slow esxi" - b120 and b140, all issues.

of course, the answer is dont use raid 5 for a vm host.

I once had to oversee thousands of VM hosts running in RAID-5…

Total data loss a few times…

urgh. raid 5. ha ha

Well, thank you for suspecting the PERC H310. Would you like to write an answer about it?

Just did. Fishing for points :)

i got slammed yester with down votes, for making a post about azure CPU only giving me 800mhz. Touchy subject ha ha

+1 to you. I'll see if there are any loose ends to tie up. If not, I'd probably award you the check mark.

cheers