« first day (930 days earlier)      last day (4086 days later) » 

12:06 AM
6 months :)
@Cole Called it
@DennisKaarsemaker when you know, you know eh?
@JoelESalas dude, I put my foot down lol
@Cole kinda
I was like here's this long list of shit I will not put up with, if you pull any other this shit BYE
@DennisKaarsemaker That's long
12:17 AM
fucking review tests
I had the same test twice in one review burst
@Cole yeah... thats a good idea, ill do that for my second marriage
I was surprised she was like "okay, that's cool"
Bleh, marriage.
Ok, done my daily dose of review. Bedtime.
Wife comes home... "Fix my Cisco switch stack! Why is my Exchange server slow!?! Why can't Jane's password be 'jane'?!?"
12:20 AM
A Cadbury Creme Egg is a chocolate product manufactured in the shape of an egg. The product consists of a thick milk chocolate shell, housing a white and yellow fondant filling which mimics the albumen and yolk of a real egg. Creme Eggs are the best-selling confectionery item between New Year's Day and Easter in the UK, with annual sales in excess of 200 million and a brand value of approximately £50 million. Creme Eggs are produced by Cadbury UK in the United Kingdom and by Cadbury Adams in Canada. They are sold by Kraft Foods in all markets except the USA, where the Hershey Company has ...
Over the years, Cadbury has introduced a number of products related to the original Creme Egg, including:
Border Creme Eggs. The first variant, wrapped in various colours of tartan foil and containing chocolate fondant. Introduced as "Fry's Border Creme Eggs" in 1970, rebranded as "Cadbury Border Creme Eggs" in 1974 and discontinued in 1981.
Mini Creme Eggs (bite-sized Creme Eggs)
Caramel Eggs (chocolate egg with a caramel filling), launched in 1994[3]
Mini Caramel Eggs (bite-sized Caramel Eggs)
@ewwhite my wife has called me 8 times so far today... and texted me 20 + times... its really annoying
so I agree... bleh
@MDMarra IMO Cadbury eggs are too sweet.
@ewwhite Oh god you're getting it from both ends
@JoelESalas Her company is a client, too.
Well in all seriousness, why is her Exchange server slow?
12:23 AM
@Zoredache agreed
@jscott Because the VSS writers were in a weird state, and the BackupExec backups weren't completing, allowing the log partition to slowly fill up.
@ewwhite And I suppose a completely rational, detailed reply such as that doesn't suit her? :)
@jscott It all started with one of her users who sent a 30 megabyte email to 2,000 people...
@ewwhite Oh dear, even gmail caps at, I think, 25MB.
12:26 AM
which blew up the journaling solution... this was months ago... and all Exchange problems since have been in relation to that.
"Matt, build me a sharepoint server, Matt why isnt it done yet, MAAAATTTTTTT"
"because im playing silly flash games, and I dont wanna"
ok... quick question about sharepoint
if I add a server to the farm, can I run another web application on port 80 on the new server, or is that shared?
Why would you want to run anything other than sharepoint on a sharepoint server?
@MDMarra 2x sharepoint web applications on port 80
that should work
12:32 AM
I love testing in production!
@MattBear Production is a test. Most just won't admit that.
Here's a career question for you all
How busy are you at work? Feel challenged?
@ewwhite to the extreme
Are you satisfied working on one environment?
@ewwhite lol
12:39 AM
@ewwhite im working on... 10+ enviroments lol
one environment
I would LOVE one enviroment
one environment is for suckers
I would LOVE to be a sucker
I have a client at work.. the devops place. And they are demanding and rude.
12:41 AM
So give them what they want
They just sent a snippet of a monitoring graph and asked when OUR staff would initiate an alert.
Stop trying to tell them how bad their designs are
As in, "why didn't you notice?"
They need an in-house person.
No managed provider would ever give them what they're asking
@ewwhite are they asking for 24x7 365 monitoring?
So I check their website and they're looking for someone...
12:43 AM
@ewwhite what, alerts aren't automatic?
Alerts are auto...
So who was wrong here?
Were your alerting thresholds wrong or were they being little babies?
Anyone who knows monitoring knows that you have to baseline...
I'll give context.
Client emails saying, "I see lots of red and lots of spikes... why weren't we notified?!?"
We have comprehensive monitoring... Heck a $300k package.
12:47 AM
@ewwhite And I'm guessing they didn't actually notice any problems.
We use Sciencelogic EM7 with lots of customizations.
@ShaneMadden well, these aren't problems.
@ewwhite This is a LW client?
@ewwhite Yeah exactly - especially the spikes last week look like just regular use during the day.
Is 250K read a lot for them?
12:49 AM
I only have data collection going back to mid-february.
Color the lines/fill green?
the nature of their business is very bursty... so there's no regular pattern. This is measured from the SAN.
Client says, "I am confused because it's all red and we've never received one alert... Also lots of spikes that are not normal so again, why not one alert ever?"
@ewwhite "It's red because that's the UI fill color, not because it's bad. The spikes aren't normal based off of what judgement? They could be normal use patterns, but since we don't have data further back than February we can't tell. You were not alerted because nothing bad was happening."
I can be rather cold sometimes....
But it's true...
So they're hiring
and I'm trying to think of who would be a good fit for them.
oh shit... the job ad is gone!
who knows who I'll end up interfacing with next?
1:17 AM
wow... just got asked a question
and I kid you not
"How do you spell IV?"
@MattBear As in Intravenous or as in Roman Numeral Four?
I'm assuming intravenous.
thats as bad as asking how to spell IT
We'll be moving the #stackexchange network back over to the primary database nodes in a few minutes, the outage should be < 60 seconds.
And we're done!
I love how efficient rsync is
1:36 AM
@WesleyDavid I wish it could do a better job at handling directories full of small files.
Not sure how it would do that, but that seems to be its biggest problem.
2:09 AM
@Zoredache Yeah. Not so great with large file-per-email datastores
2:25 AM
@StackExchange That was 6 minutes! Where's my refund?
Not so great with non Posix file permissions/attributes either, be nice if it support NFS4 and NTFS... Might as well just support a flexible attribute system so would could sync extended attributes and other file systems.
@MDMarra The Giant Creme Eggs were the best. Not only were they bigger than what you get now, the caramel and creme filling mixture was delicious.
3:16 AM
@Zoredache How small and how many?
Sweet. Flaky. Biscuits.
3:49 AM
@ChrisS yuuuup
Hey @ChrisS. Any idea why I add a CSV through Failover Cluster Manager but SCVMM doesn't see that volume?
Do I have to add all storage through SCVMM or what?
I don't remember the details right now, but I remember SCVMM getting all pissy when you don't use it for everything. I'm still on 2008, not sure if 2012 is significantly different.
I couldn't figure out how to add the storage through scvmm since it doesn't control the SAN through a storage provider
oh well
4:14 AM
Gawd. Using OpenBSD is like going back in time 20 years.
Hell, the same seems to be true of NetBSD and FreeBSD.
hmm. How is it that I can never figure out how to do Unicode intentionally, but this stupid Mac is always putting it in things I type...
4:30 AM
Weee! This is fuñ!
I think I saw that on a pyramid in Egypt...
5:00 AM
☭ In Soviet server room, DevOps you!!
☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹
Zed's dead, baby. Zed's dead. ☠

♟♟♟♟ ♟♟♟
@Adrian 1. e4
Your move.
Shoot, I used the wrong color. Fack.
Oh well - time to peruse Facebook.
You won't get that 7 minutes back either.
@Adrian Only 30 seconds. That's how long it takes for me to remember I don't have any friends.

♙♙♙♙ ♙♙♙
There, 1. e4.
That only took 3 minutes. I'm getting more efficient at wasting my time.
5:16 AM
Oh, VMware, how you annoy me.
@MichaelHampton Let's start with their licensing.
@WesleyDavid Tonight's fun is with VMware Workstation. Which decided that since I plugged in my cell phone via USB, I must want to bridge all my network connections to it!
OS: I have a USB interface and a new device was plugged in.
VMware Workstation: -.-
OS: I'm giving you a heads up to get in the IRQ to use it
VMware Workstation: -.-
OS: Here it is. What do you want to do with it?
VMware Workstation: IDUNNOLOL ¯\ (ツ)/¯
We need more nyan cat in here:
I'm really getting sick of all the cPanel-fucked-up-my-system questions.
@MichaelHampton beats cPanel with markdown
My cPanel server: 🚽
Actually, I have one Plesk server for shared host clients and so far one dedicated server client that I installed Webmin for. Someone else might want a server with webmin here shortly.
As long as I don't have to deal with it, I'm fine with it.
Plesk is a royal PITA. Clients love it to click-click-done through life, but wow is it junk in its underbelly.
6:18 AM
I think I got a phone interview with the replacement for the recruiter who contacted me in the first place ;p
/me will reply in a few hours, once I have RAIDED the goddamned butterflies in my stomach. Not the not-a-backup. The Bug Spray ;p
2 hours later…
8:47 AM
@KennyRasschaert meta.serverfault.com/questions/5316/… and did you get RHCE yet ?
You're referring to the mysql question?

> The answer is asking for additional information or making a comment on the question - we can convert to a comment.
And yes :)
@KennyRasschaert people often phrase answers as question - did you try x or y meaning ensure ...
@KennyRasschaert congrats
I see your point
Haven't heard anything from @WesleyDavid in a while. I wonder if he is still going for his RHCSA.
I don't think so
Actual work often gets in the way of certification, sadly :(
I'm lucky my employer really wanted me to get certified (read: the customers want certified engineers handling their servers) and was willing to pay for training.
8:55 AM
The review system is poorly designed for people with very fat fingers.
Can't tell you how many times I've had to go look for a poor question that I just reviewed as "looks good" so I can vote to close and redeem myself.
9:15 AM
@WesleyDavid bad kitty! Professionals don't use webmin/cpanel/plesk. There's enough crap on systems already, no need to add another steaming pile of shit.
Has anyone checked out Docker? docker.io github.com/dotcloud/docker
9:48 AM
@MichaelHampton can't we close those as off-topic? CPanel is not used by professional sysadmins worth their paycheck.
10:00 AM
@DennisKaarsemaker what do you mean with prepend the setting?
Anything appended to a key is part of the comment. The format is <options> <keytype> <base64 data> <comment>
so you'll need to prepend it, not append
So, we both just edited our answers to essentially say: "What that other guy said is right too"
I just edited mine :)
oh well, at least we agree :)
10:02 AM
never had much luck with stunnel personally
and VPN might be overkill for his use case
I use it for bitlbee, never had a problem
There are legitimate use cases for persistent ssh tunnels. Since he mentions his client is also a server, ssh tunnels are probably a terrible idea for his use case.
Some days, I feel like I don't know anything.
I have a customer who's very confused about the notion of IOPs.
And in trying to explain and help clarify, I've managed to confuse myself!
@ewwhite confuse them even more with bandwidth and latency. And seek times on rotating rust. And bus types/speeds. And all the caching that's possible :)
Customer is upset that their SAN is busy...
And has resorted to abusive emails.
10:16 AM
Are they upset their san is busy or upset about their overall throughput?
I find pleasure in taking people's simple view on things like storage and adding all the gory details, so the simple view no longer makes sense.
Cause you can always just turn off the SAN.
@DennisKaarsemaker There's a very very limited number of things I can do that with. Storage is not one of them, sadly
So there's a threshold...
Even a breakdown of disks...
These are just 32 nearline 7200rpm SAS drives...looks like RAID6 groups.
What on earth are they donig on it?
10:22 AM
and they're just busy. But when people complain about storage, it's not common to phrase it in terms of IOPs, right?
They don't know... They said something about backing up to the cloud (S3)
How are they backing up?
Backing up databases or tar or something?
No clue... something S3-related
We're a private-cloud provider... and I don't have insight into how they're really using the system
I know that they're not using our backup service.
The setup appears to be a bunch of KVM hosts running RHEL 6
and a bunch of guest VMs running CentOS 5
the VMs sit on NFS provided by an EMC VNX 5400.
@ewwhite Some of our customers do, actually
those VMs are performing operations on data that also resides on the same NFS export.
We've got some storage machines filled to the brim with tiny disks just to get more spindles to churn out more IOPS
10:26 AM
From this graph/chart, it's apparent that random I/O is hurting... throughput is low 25MB/s on that LUN.
Do you know what the random rw is for just one disk on this SAN?
Aside from these stats your seeing, that is.
but I don't know that I'd use NFS as the backing store for KVM virtual machines either...
@MIfe I mean, these are 3TB SAS disks... maybe 100MB/s sequential and no more than 150 IOPS
But the disk utilization per disk is not 100%.
but IOPs depend on what you're doing and the I/O patterns, no?
So how do you account for that?
10:28 AM
I mean, no more than 200 IOPS
I mean, based on knowledge from other installations
But the customer speaks in terms like, "So, if I send 1,000 NFS IOPS, why can't this SAN keep up?"
I think so.
I see a problem here then. The san isnt under 100% utilization in disks or CPU. Even though its labelled as 'critical', it stands to sense that there is more margin for performance.
but the nature of what they're doing can have an impact...
@ewwhite if rotating rust, then yes. Seek times will kill your random I/O.
10:33 AM
Storage processor utilization is low...
queue length is low as well
I think theres more going on in the KVM hosts here.
posted on March 29, 2013 by Wesley David

In the process of tracking down a Java applet problem, I started troubleshooting by tracking down log file entries. The problem I discovered was that there were dozens of log files available and I didn’t know which one would hold clueful error messages. I could reproduce the applet error in a browser, so started using tail -f to watch some of the log files that I thought were the best ca

@StackExchange @WesleyDavid multitail
@MIfe definitely... but I don't know how to find out.
If you dont have access to these KVM hosts whatsoever it is unlikely that you will be able to easily find out.
If your unable to access the entire utilization chain you can provide some suitable evidence I guess that theres unaccounted for performance. I think its fair to mention that the SAN reports that utilization should be lower than what it is already however the SAN has about 30% more iops it could do.
10:36 AM
I'm logging into a host to see.
Initial complaint was "kvm hostX" is slow... operations that normally take 5-10 minutes are taking an hour.
k. I'm on a guest...
Is it IO bound on the guest?
doesn't look it
just high CPU
What about the hypervisor itself?
Not particularly...
but being NFS... getting real stats is difficult.
plus, they're not optimized for synchronous writes.
oh my...
10:46 AM
Is it cpu/io bound on the hypervisor?
I descend into a directory and type "ls"....
stalls the server
OK but statisically is there it IO/CPU bound?
@MIfe I don't know how to tell. No real metrics.
is the hypervisor exporting NFS from this lun?
The SAN has a NAS head.... so the SAN exports NFS to the hypervisors...
the VMs reside on the NFS exports.
those VMs, in turn, write to NFS exports from the same SAN
It's like giving yourself a blowjob...
10:49 AM
whats nfsstat show?
pretty darn impressive but dosen't change the fact that you fucked yourself?
From VM...
# nfsstat
Client rpc stats:
calls      retrans    authrefrsh
206314583   123796     0

Client nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 1715110   0% 79753     0% 310790    0% 89116     0% 0         0%
read         write        create       mkdir        symlink      mknod
107882932 52% 95956999 46% 32018     0% 1025      0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus
23031     0% 167       0% 13732     0% 0         0% 73493     0% 12084     0%
from host...
# nfsstat
Client rpc stats:
calls      retrans    authrefrsh
77884026   0          77914416

Client nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 1364264   1% 34        0% 55635     0% 195540    0% 0         0%
read         write        create       mkdir        symlink      mknod
5260232   6% 66973071 85% 6         0% 0         0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus
10        0% 0         0% 0         0% 1         0% 0         0% 314069    0%
Welcome to serverfault! I'm afraid that this is a Q&A site, not a 'do my homework for me' site. If you're going to try to make us do your homework, at least rephrase the question so it's not too obvious :) — Dennis Kaarsemaker 1 min ago
too harsh, or ok?
It's fine.
The host seems to be the anomaly. But if you pass nftsstat -Z1 and get like 3 seconds of output
that might help track down stats for now, those values go back since boot.

« first day (930 days earlier)      last day (4086 days later) »