The Comms Room

Dennis Kaarsemaker

12:06 AM

6 months :)

Joel E Salas

@Cole Called it

Cole

@DennisKaarsemaker when you know, you know eh?

@JoelESalas dude, I put my foot down lol

Dennis Kaarsemaker

@Cole kinda

Cole

I was like here's this long list of shit I will not put up with, if you pull any other this shit BYE

MDMarra

@DennisKaarsemaker That's long

:)

Dennis Kaarsemaker

12:17 AM

fucking review tests

I had the same test twice in one review burst

Matt Bear

@Cole yeah... thats a good idea, ill do that for my second marriage

Cole

I was surprised she was like "okay, that's cool"

ewwhite

Bleh, marriage.

Dennis Kaarsemaker

Ok, done my daily dose of review. Bedtime.

ewwhite

Wife comes home... "Fix my Cisco switch stack! Why is my Exchange server slow!?! Why can't Jane's password be 'jane'?!?"

5

MDMarra

12:20 AM

Cadbury Creme Egg

A Cadbury Creme Egg is a chocolate product manufactured in the shape of an egg. The product consists of a thick milk chocolate shell, housing a white and yellow fondant filling which mimics the albumen and yolk of a real egg. Creme Eggs are the best-selling confectionery item between New Year's Day and Easter in the UK, with annual sales in excess of 200 million and a brand value of approximately £50 million. Creme Eggs are produced by Cadbury UK in the United Kingdom and by Cadbury Adams in Canada. They are sold by Kraft Foods in all markets except the USA, where the Hershey Company has ...

Over the years, Cadbury has introduced a number of products related to the original Creme Egg, including:
Border Creme Eggs. The first variant, wrapped in various colours of tartan foil and containing chocolate fondant. Introduced as "Fry's Border Creme Eggs" in 1970, rebranded as "Cadbury Border Creme Eggs" in 1974 and discontinued in 1981.
Mini Creme Eggs (bite-sized Creme Eggs)
Caramel Eggs (chocolate egg with a caramel filling), launched in 1994[3]
Mini Caramel Eggs (bite-sized Caramel Eggs)

Matt Bear

@ewwhite my wife has called me 8 times so far today... and texted me 20 + times... its really annoying

so I agree... bleh

Zoredache

@MDMarra IMO Cadbury eggs are too sweet.

Joel E Salas

@ewwhite Oh god you're getting it from both ends

ewwhite

@JoelESalas Her company is a client, too.

jscott

Well in all seriousness, why is her Exchange server slow?

Cole

12:23 AM

@Zoredache agreed

ewwhite

@jscott Because the VSS writers were in a weird state, and the BackupExec backups weren't completing, allowing the log partition to slowly fill up.

jscott

@ewwhite And I suppose a completely rational, detailed reply such as that doesn't suit her? :)

ewwhite

@jscott It all started with one of her users who sent a 30 megabyte email to 2,000 people...

Matt Bear

uggh...

jscott

@ewwhite Oh dear, even gmail caps at, I think, 25MB.

ewwhite

12:26 AM

which blew up the journaling solution... this was months ago... and all Exchange problems since have been in relation to that.

Cole

ouch

Matt Bear

"Matt, build me a sharepoint server, Matt why isnt it done yet, MAAAATTTTTTT"

"because im playing silly flash games, and I dont wanna"

ok... quick question about sharepoint

if I add a server to the farm, can I run another web application on port 80 on the new server, or is that shared?

MDMarra

Why would you want to run anything other than sharepoint on a sharepoint server?

Matt Bear

@MDMarra 2x sharepoint web applications on port 80

MDMarra

oh

that should work

Matt Bear

12:32 AM

I love testing in production!

jscott

@MattBear Production is a test. Most just won't admit that.

ewwhite

Here's a career question for you all

How busy are you at work? Feel challenged?

Matt Bear

@ewwhite to the extreme

ewwhite

Are you satisfied working on one environment?

MDMarra

@ewwhite lol

Matt Bear

12:39 AM

@ewwhite im working on... 10+ enviroments lol

MDMarra

one environment

Matt Bear

I would LOVE one enviroment

MDMarra

one environment is for suckers

Matt Bear

I would LOVE to be a sucker

ewwhite

I have a client at work.. the devops place. And they are demanding and rude.

MDMarra

12:41 AM

So give them what they want

ewwhite

They just sent a snippet of a monitoring graph and asked when OUR staff would initiate an alert.

MDMarra

Stop trying to tell them how bad their designs are

ewwhite

As in, "why didn't you notice?"

They need an in-house person.

No managed provider would ever give them what they're asking

Matt Bear

@ewwhite are they asking for 24x7 365 monitoring?

ewwhite

So I check their website and they're looking for someone...

Zoredache

12:43 AM

@ewwhite what, alerts aren't automatic?

ewwhite

Alerts are auto...

MDMarra

So who was wrong here?

Were your alerting thresholds wrong or were they being little babies?

ewwhite

Anyone who knows monitoring knows that you have to baseline...

I'll give context.

Client emails saying, "I see lots of red and lots of spikes... why weren't we notified?!?"

Matt Bear

@ewwhite esentire.com

ewwhite

We have comprehensive monitoring... Heck a $300k package.

Shane Madden

12:47 AM

@ewwhite And I'm guessing they didn't actually notice any problems.

ewwhite

We use Sciencelogic EM7 with lots of customizations.

@ShaneMadden well, these aren't problems.

WesleyDavid

@ewwhite This is a LW client?

ewwhite

Yes.

Shane Madden

@ewwhite Yeah exactly - especially the spikes last week look like just regular use during the day.

jscott

Is 250K read a lot for them?

ewwhite

12:49 AM

I only have data collection going back to mid-february.

jscott

Color the lines/fill green?

ewwhite

the nature of their business is very bursty... so there's no regular pattern. This is measured from the SAN.

Client says, "I am confused because it's all red and we've never received one alert... Also lots of spikes that are not normal so again, why not one alert ever?"

WesleyDavid

@ewwhite "It's red because that's the UI fill color, not because it's bad. The spikes aren't normal based off of what judgement? They could be normal use patterns, but since we don't have data further back than February we can't tell. You were not alerted because nothing bad was happening."

I can be rather cold sometimes....

ewwhite

But it's true...

So they're hiring

and I'm trying to think of who would be a good fit for them.

oh shit... the job ad is gone!

who knows who I'll end up interfacing with next?

Matt Bear

1:17 AM

wow... just got asked a question

and I kid you not

"How do you spell IV?"

WesleyDavid

@MattBear As in Intravenous or as in Roman Numeral Four?

I'm assuming intravenous.

Matt Bear

yeah

thats as bad as asking how to spell IT

Stack Exchange

We'll be moving the #stackexchange network back over to the primary database nodes in a few minutes, the outage should be < 60 seconds.

Tweeted by StackStatus on March 29, 2013 at 1:02 AM

And we're done!

Tweeted by StackStatus on March 29, 2013 at 1:08 AM

WesleyDavid

I love how efficient rsync is

Zoredache

1:36 AM

@WesleyDavid I wish it could do a better job at handling directories full of small files.

Not sure how it would do that, but that seems to be its biggest problem.

Adrian

2:09 AM

@Zoredache Yeah. Not so great with large file-per-email datastores

Chris S

2:25 AM

@StackExchange That was 6 minutes! Where's my refund?

Not so great with non Posix file permissions/attributes either, be nice if it support NFS4 and NTFS... Might as well just support a flexible attribute system so would could sync extended attributes and other file systems.

@MDMarra The Giant Creme Eggs were the best. Not only were they bigger than what you get now, the caramel and creme filling mixture was delicious.

WesleyDavid

3:16 AM

@Zoredache How small and how many?

Adrian

Sweet. Flaky. Biscuits.

MDMarra

3:49 AM

@ChrisS yuuuup

Hey @ChrisS. Any idea why I add a CSV through Failover Cluster Manager but SCVMM doesn't see that volume?

Do I have to add all storage through SCVMM or what?

Chris S

I don't remember the details right now, but I remember SCVMM getting all pissy when you don't use it for everything. I'm still on 2008, not sure if 2012 is significantly different.

MDMarra

hm

I couldn't figure out how to add the storage through scvmm since it doesn't control the SAN through a storage provider

oh well

Stack Exchange

4:14 AM

I'm so bad at carrying a tune, those 'find a song by humming its melody' websites throw an HTTP 406 error as soon as I start to hum.

Michael Hampton

Gawd. Using OpenBSD is like going back in time 20 years.

Adrian

†

Michael Hampton

Hell, the same seems to be true of NetBSD and FreeBSD.

Adrian

hmm. How is it that I can never figure out how to do Unicode intentionally, but this stupid Mac is always putting it in things I type...

WesleyDavid

4:30 AM

√√∫∫œ∑´®†¥¨ˆøπåß∂ƒ©˙∆˚¬…Ω≈ç√∫˜µ≤≥

`¡™£¢∞§¶•ªº–≠œ∑´®†¥¨ˆøπ“‘«åß∂ƒ©˙∆˚¬…æΩ≈ç√∫˜µ≤≥÷

Weee! This is fuñ!

Michael Hampton

I think I saw that on a pyramid in Egypt...

Adrian

5:00 AM

å∫ç∂´ƒ©˙ˆ∆˚¬µ˜øπœ®ß†¨√∑≈¥Ω

ÅıÇÎ´Ï˝ÓˆÔÒÂ˜Ø∏Œ‰Íˇ¨◊„˛Á¸



WesleyDavid

☭ In Soviet server room, DevOps you!! ☭

☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹ ☹

Zed's dead, baby. Zed's dead. ☠

    ♟

♟♟♟♟ ♟♟♟
♜♞♝♛♚♝♞♜

@Adrian 1. e4

Your move.

Shoot, I used the wrong color. Fack.

7 MINUTES OF MY LIFE I WON'T GET BACK!!

Oh well - time to peruse Facebook.

Adrian

lol

You won't get that 7 minutes back either.

WesleyDavid

@Adrian Only 30 seconds. That's how long it takes for me to remember I don't have any friends.

    ♙

♙♙♙♙ ♙♙♙
♖♘♗♕♔♗♘♖

There, 1. e4.

That only took 3 minutes. I'm getting more efficient at wasting my time.

Michael Hampton

5:16 AM

Oh, VMware, how you annoy me.

WesleyDavid

@MichaelHampton Let's start with their licensing.

Michael Hampton

@WesleyDavid Tonight's fun is with VMware Workstation. Which decided that since I plugged in my cell phone via USB, I must want to bridge all my network connections to it!

WesleyDavid

OS: I have a USB interface and a new device was plugged in.
VMware Workstation: -.-
OS: I'm giving you a heads up to get in the IRQ to use it
VMware Workstation: -.-
OS: Here it is. What do you want to do with it?
VMware Workstation: IDUNNOLOL ¯\ (ツ)/¯

We need more nyan cat in here:

·.,¸,.·*¯·.,¸,.·*¯[;::;‹]･ᴥ･)

Michael Hampton

I'm really getting sick of all the cPanel-fucked-up-my-system questions.

WesleyDavid

@MichaelHampton beats cPanel with markdown

My cPanel server: 🚽

Actually, I have one Plesk server for shared host clients and so far one dedicated server client that I installed Webmin for. Someone else might want a server with webmin here shortly.

As long as I don't have to deal with it, I'm fine with it.

Plesk is a royal PITA. Clients love it to click-click-done through life, but wow is it junk in its underbelly.

Journeyman Geek

6:18 AM

I think I got a phone interview with the replacement for the recruiter who contacted me in the first place ;p

/me will reply in a few hours, once I have RAIDED the goddamned butterflies in my stomach. Not the not-a-backup. The Bug Spray ;p

Iain

8:47 AM

@KennyRasschaert meta.serverfault.com/questions/5316/… and did you get RHCE yet ?

Kenny Rasschaert

You're referring to the mysql question?

> The answer is asking for additional information or making a comment on the question - we can convert to a comment.

And yes :)

redhat.com/wapps/training/certification/…

Iain

@KennyRasschaert people often phrase answers as question - did you try x or y meaning ensure ...

@KennyRasschaert congrats

Kenny Rasschaert

I see your point

Thanks

Haven't heard anything from @WesleyDavid in a while. I wonder if he is still going for his RHCSA.

Iain

I don't think so

Kenny Rasschaert

Actual work often gets in the way of certification, sadly :(

I'm lucky my employer really wanted me to get certified (read: the customers want certified engineers handling their servers) and was willing to pay for training.

Iain

8:55 AM

indeed

Kenny Rasschaert

The review system is poorly designed for people with very fat fingers.

Can't tell you how many times I've had to go look for a poor question that I just reviewed as "looks good" so I can vote to close and redeem myself.

Dennis Kaarsemaker

9:15 AM

@WesleyDavid bad kitty! Professionals don't use webmin/cpanel/plesk. There's enough crap on systems already, no need to add another steaming pile of shit.

Kenny Rasschaert

Has anyone checked out Docker? docker.io github.com/dotcloud/docker

Dennis Kaarsemaker

9:48 AM

@MichaelHampton can't we close those as off-topic? CPanel is not used by professional sysadmins worth their paycheck.

Kenny Rasschaert

10:00 AM

@DennisKaarsemaker what do you mean with prepend the setting?

Dennis Kaarsemaker

Anything appended to a key is part of the comment. The format is <options> <keytype> <base64 data> <comment>

so you'll need to prepend it, not append

Kenny Rasschaert

So, we both just edited our answers to essentially say: "What that other guy said is right too"

Dennis Kaarsemaker

I just edited mine :)

Kenny Rasschaert

I saw

Dennis Kaarsemaker

oh well, at least we agree :)

Kenny Rasschaert

10:02 AM

never had much luck with stunnel personally

and VPN might be overkill for his use case

Dennis Kaarsemaker

I use it for bitlbee, never had a problem

Kenny Rasschaert

There are legitimate use cases for persistent ssh tunnels. Since he mentions his client is also a server, ssh tunnels are probably a terrible idea for his use case.

ewwhite

Some days, I feel like I don't know anything.

I have a customer who's very confused about the notion of IOPs.

And in trying to explain and help clarify, I've managed to confuse myself!

Dennis Kaarsemaker

@ewwhite confuse them even more with bandwidth and latency. And seek times on rotating rust. And bus types/speeds. And all the caching that's possible :)

ewwhite

Customer is upset that their SAN is busy...

And has resorted to abusive emails.

MIfe

10:16 AM

Are they upset their san is busy or upset about their overall throughput?

Dennis Kaarsemaker

I find pleasure in taking people's simple view on things like storage and adding all the gory details, so the simple view no longer makes sense.

MIfe

Cause you can always just turn off the SAN.

Kenny Rasschaert

@DennisKaarsemaker There's a very very limited number of things I can do that with. Storage is not one of them, sadly

ewwhite

So there's a threshold...

Even a breakdown of disks...

These are just 32 nearline 7200rpm SAS drives...looks like RAID6 groups.

MIfe

What on earth are they donig on it?

ewwhite

10:22 AM

and they're just busy. But when people complain about storage, it's not common to phrase it in terms of IOPs, right?

They don't know... They said something about backing up to the cloud (S3)

MIfe

How are they backing up?

Backing up databases or tar or something?

ewwhite

No clue... something S3-related

We're a private-cloud provider... and I don't have insight into how they're really using the system

I know that they're not using our backup service.

The setup appears to be a bunch of KVM hosts running RHEL 6

and a bunch of guest VMs running CentOS 5

the VMs sit on NFS provided by an EMC VNX 5400.

Kenny Rasschaert

@ewwhite Some of our customers do, actually

ewwhite

those VMs are performing operations on data that also resides on the same NFS export.

Kenny Rasschaert

We've got some storage machines filled to the brim with tiny disks just to get more spindles to churn out more IOPS

ewwhite

10:26 AM

From this graph/chart, it's apparent that random I/O is hurting... throughput is low 25MB/s on that LUN.

MIfe

Do you know what the random rw is for just one disk on this SAN?

Aside from these stats your seeing, that is.

ewwhite

but I don't know that I'd use NFS as the backing store for KVM virtual machines either...

@MIfe I mean, these are 3TB SAS disks... maybe 100MB/s sequential and no more than 150 IOPS

MIfe

But the disk utilization per disk is not 100%.

ewwhite

but IOPs depend on what you're doing and the I/O patterns, no?

MIfe

So how do you account for that?

ewwhite

10:28 AM

I mean, no more than 200 IOPS

I mean, based on knowledge from other installations

But the customer speaks in terms like, "So, if I send 1,000 NFS IOPS, why can't this SAN keep up?"

MIfe

Is "SP UTILIZATION" a CPU %?

ewwhite

I think so.

MIfe

I see a problem here then. The san isnt under 100% utilization in disks or CPU. Even though its labelled as 'critical', it stands to sense that there is more margin for performance.

ewwhite

but the nature of what they're doing can have an impact...

Dennis Kaarsemaker

@ewwhite if rotating rust, then yes. Seek times will kill your random I/O.

ewwhite

10:33 AM

Storage processor utilization is low...

queue length is low as well

MIfe

I think theres more going on in the KVM hosts here.

Stack Exchange

The Nubby Admin: Can I tail Multiple Logs Simultaneously on a Linux Machine?

posted on March 29, 2013 by Wesley David

In the process of tracking down a Java applet problem, I started troubleshooting by tracking down log file entries. The problem I discovered was that there were dozens of log files available and I didn’t know which one would hold clueful error messages. I could reproduce the applet error in a browser, so started using tail -f to watch some of the log files that I thought were the best ca…

Dennis Kaarsemaker

@StackExchange @WesleyDavid multitail

ewwhite

@MIfe definitely... but I don't know how to find out.

MIfe

If you dont have access to these KVM hosts whatsoever it is unlikely that you will be able to easily find out.

If your unable to access the entire utilization chain you can provide some suitable evidence I guess that theres unaccounted for performance. I think its fair to mention that the SAN reports that utilization should be lower than what it is already however the SAN has about 30% more iops it could do.

ewwhite

10:36 AM

I'm logging into a host to see.

Initial complaint was "kvm hostX" is slow... operations that normally take 5-10 minutes are taking an hour.

k. I'm on a guest...

MIfe

Is it IO bound on the guest?

ewwhite

doesn't look it

just high CPU

Dan

Morning

MIfe

What about the hypervisor itself?

ewwhite

Not particularly...

but being NFS... getting real stats is difficult.

plus, they're not optimized for synchronous writes.

oh my...

MIfe

10:46 AM

Is it cpu/io bound on the hypervisor?

ewwhite

I descend into a directory and type "ls"....

stalls the server

MIfe

OK but statisically is there it IO/CPU bound?

ewwhite

@MIfe I don't know how to tell. No real metrics.

MIfe

is the hypervisor exporting NFS from this lun?

ewwhite

The SAN has a NAS head.... so the SAN exports NFS to the hypervisors...

the VMs reside on the NFS exports.

those VMs, in turn, write to NFS exports from the same SAN

It's like giving yourself a blowjob...

MIfe

10:49 AM

whats nfsstat show?

Journeyman Geek

pretty darn impressive but dosen't change the fact that you fucked yourself?

ewwhite

From VM...

# nfsstat
Client rpc stats:
calls      retrans    authrefrsh
206314583   123796     0

Client nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 1715110   0% 79753     0% 310790    0% 89116     0% 0         0%
read         write        create       mkdir        symlink      mknod
107882932 52% 95956999 46% 32018     0% 1025      0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus
23031     0% 167       0% 13732     0% 0         0% 73493     0% 12084     0%

from host...

# nfsstat
Client rpc stats:
calls      retrans    authrefrsh
77884026   0          77914416

Client nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 1364264   1% 34        0% 55635     0% 195540    0% 0         0%
read         write        create       mkdir        symlink      mknod
5260232   6% 66973071 85% 6         0% 0         0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus
10        0% 0         0% 0         0% 1         0% 0         0% 314069    0%

Dennis Kaarsemaker

Welcome to serverfault! I'm afraid that this is a Q&A site, not a 'do my homework for me' site. If you're going to try to make us do your homework, at least rephrase the question so it's not too obvious :) — Dennis Kaarsemaker 1 min ago

too harsh, or ok?

ewwhite

It's fine.

MIfe

The host seems to be the anomaly. But if you pass nftsstat -Z1 and get like 3 seconds of output

that might help track down stats for now, those values go back since boot.

Transcript for