The Comms Room

ewwhite

11:00 AM

Not sure what I'm looking for.

Dennis Kaarsemaker

things that use io. Try iotop.

MIfe

Well i want to see what that commit% shows over the last few seconds

ewwhite

@DennisKaarsemaker iotop won't be useful here...

Dennis Kaarsemaker

why not? If you're on the host causing the problems, you should see the problem

ewwhite

@MIfe commit hovers between 0 and 7

pauska

11:04 AM

@ewwhite replied to the mail now. maybe you knew everything already, but hey :)

MIfe

and write?

pauska

just invite to another room if you want to discuss it

ewwhite

@MIfe 90-100%

@pauska thank you... you see some of the issues...

the SAN is an interesting one... they have another group of 96 enterprise SAS disks.

Iain

@wesledavid - multitail ?

ewwhite

@DennisKaarsemaker the guests are also reading/writing to NFS... bypasses any host I/O.

pauska

11:07 AM

@ewwhite the one you sent me only had 3TB disks..

ewwhite

@pauska right, that's where all of this virtual machine activity resides.

the faster disks are running their databases.

pauska

well

there should have been tiering here

yes, RAID6 is recommended by EMC for 2TB/3TB.. it doesn't mean that it's good enough

MIfe

I think I see the problem here..

ewwhite

This certainly came before I arrived. I don't know storage that well... other than Nexenta and ZFS ;)

but I think the client's design is also a concern.

pauska

also true

ewwhite

11:09 AM

DevOps

Completely...

pauska

35TB of warm data just doesnt make any sense

unless you're doing medical research, graphing and so on..

MIfe

Its about 100% if you add SP utilization and disk utilization, nearly all of it is writes.

ewwhite

they are incredibly write-heavy.

pauska

@MIfe yeah, the thing here is that the SP doesn't even sweat

it's just the LUN's being raped on a raid6 subsystem

but like I said in the mail

new shelf with tiering and see if that evens out the load

MIfe

I'm betting the SP is stuck in IOwait waiting for writes to commit to respond back to the clients.

ewwhite

11:11 AM

And the RAID6 is really 4 groups of 6+2, I think

pauska

yeah, that is how storage pools carves disk up

MIfe

One of the things that anomalous here though. Is if it is 'backups from s3' that data should be sequential, not random.

If its backing up TO the cloud then sure t hat might be relatively random. But writing from the cloud should behave sequentially.

ewwhite

@MIfe THe client stopped the backups... no change.

pauska

this does not look like its sequential at all

ewwhite

but I have no idea what they were backing up or what data they're protecting.

it sucks not owning the infrastructure to the point where you can really help.

MIfe

11:13 AM

OK so now your not doing backups but still have the IO?

ewwhite

exactly.

pauska

does the storage pool have any free space? is there any chance to convince the client into carving this into smaller LUNS to get better granularity?

MIfe

If you want to account for it then it'll be a survey of all hosts needed to see where the writes get generated from.

pauska

it could help tremendously to pinpoint the problem

ewwhite

On the KVM guests...

# nfsstat -Z1
nfsstat: invalid option -- Z
Try `nfsstat --help' for more information.

pauska

11:14 AM

if not, then tiering is the way to go here.. I really don't see any other solution

ewwhite

@pauska I don't think there's free space

pauska

just remember that EMC block tiering is 1GB slices

ewwhite

so the solution is a VNX 5300

and they're just about maxed out on disks.

pauska

how many disks right now on this one?

the report said 32

ewwhite

let me see if I can get into unisphere..

32 + the 96 SAS drives.

It's a little sad because I don't know that we had a storage expert help design this.

pauska

11:16 AM

you're maxed

ewwhite

but the client's activity and business has also grown.

pauska

btw, this is $mainjob right? your company owns the hardware?

ewwhite

@pauska yes, we own.

Client sent this earlier in the day before the problems...

11 hours ago, by ewwhite

MIfe

nfssstat is a bit gash I think its nfsstat -Z=1.

pauska

so, you could get them to buy a VNX5500, and just move other clients over to the VNX5300 when time comes

MIfe

11:19 AM

Unless you can account for these IOPs your treading on thin ice.

You could spend $$$s on more hardware and still have no idea if thats enough.

pauska

(VNX5500 250 disks, VNX5300 125 disks)

ewwhite

@MIfe on the host nfsstat with that parameter worked... the guest is EL5, though

MIfe

Or if your overspending.

ewwhite

@MIfe right, we want a cause... and an explanation.

but this system should be capable of more, right?

MIfe

Your probably going to have to survey every guest.

pauska

11:20 AM

thats impossible without gaining full access to their systems/data..

MIfe

you can do iostat -mtx 1 on every guest and gain an idea as to what the io utilziation is like, which ones do a lot of writes and write iops

pauska

@ewwhite like I said, the SP/NAS is not sweating. You're using up all the random I/O on the raid6 pool

MIfe

then figur eout what processes cause it.

pauska

just do the math.. 32 NL-SAS disks in RAID6

MIfe

@pauska I disagree. If this is all writes, and its syncronous, the SP can only work as fast as the disks.

ewwhite

11:21 AM

@MIfe iostat won't give the NFS info.

remember, the guests are just doing operations on the same NFS mounts as the host.

MIfe

@ewwhate sure bu tthe guests are images on nfs?

pauska

@MIfe They have FAST Cache - I can promise you that the SP is not overloaded

but you do have a point

MIfe

@pauska how do you account for the fast that not all disks are 100% utilized?

pauska

this client could probably rape a VNX7500 if they got the opportunity

MIfe

fast=fact

pauska

11:23 AM

@MIfe a report that @ewwhite sent me

ewwhite

So the hosts have NFS mounts that contain the guests and data. The guests have NFS mounts from within them that point to the same location as the host.

so the guest IO isn't really accounted for.

pauska

about 12 of the 3TB disks are at max IOPS

the rest is at half util

ewwhite

1 hour ago, by ewwhite

MIfe

But those stats dont say they are at max iops.

pauska

they do.

MIfe

11:24 AM

utilization = < 100

pauska

next column

you cant JUST look at util

MIfe

Why?

pauska

they are clearly doing more random i/o than they should

because the disks are getting IO bound

if this was all sequential then sure, util would be the deciding factor

MIfe

Fine, but util is the %age of time the disks spends doing stuff. And 20% of the time its not.

Sequential or random doesnt matter. The disks spend 20% of their time idle.

I'm not suggesting this is good utilization by the wya, and that they should be 100%, but they arent and there are possibly more iops to be had.

ewwhite

and the service times are high

pauska

11:27 AM

you're right in one way

there is 20% of idle time here

but i'm pretty sure thats the SP waiting for clients to ACK

ewwhite

KVM over NFS in this case is stupid, right?

pauska

wasn't this NAS > KVM > NFS > GUEST

or something like that?

MIfe

thats what I said but you said it had FAST cache.

pauska

I said that the storage processor is not overloaded

MIfe

And I said that the SP is waiting for the disks, you said it was not because of the cache.

ewwhite

11:29 AM

It is... KVM hosts mounting NFS from the VNX... the guests reside on that NFS share. The guests also mount NFS shares from the same pool directly. The host also does processing on data that resides on the NFS shares.

pauska

then I misunderstood, sorry

ewwhite

It'd be like running a database on your VMware ESX 4.0 host while mounting your VMs from an NFS share...

(e.g. the host is doing more non-VM stuff than it should)

MIfe

I think at this stage your going to need to look at what is being written, by what guests.

ewwhite

@MIfe it's just data... stuff... being written directory to the NFS shares by the guests

MIfe

I'm not sure if pidstat will account for writes over NFS, but that should work on EL5 if you download a later sysstat.

ewwhite

11:32 AM

@pauska Would this be better in RAID 1+0?

pauska

Who knows really

for all we know that setup of theirs could eat 25 ssd's for breakfast if they got the chance to

ewwhite

@MIfe NFSstat shows...

Client nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 1722822   0% 80317     0% 311219    0% 89809     0% 0         0%
read         write        create       mkdir        symlink      mknod
108874533 52% 96826440 46% 32226     0% 1026      0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus
23233     0% 168       0% 13822     0% 0         0% 73692     0% 12171     0%
fsstat       fsinfo       pathconf     commit

this is on a guest

@pauska so what would EMC do?

pauska

@ewwhite I'm pretty sure they'd recommend the same thing I emailed you - move the hottest data to a different and faster tier

which you can't, since you're maxed out on disks

ewwhite

But the system is at capacity.

pauska

one thing strikes me as odd though

or - let me ask - did they add more disks recently?

you have a lot of disks doing nothing at all

and check if you have a FAST license on it

ewwhite

11:37 AM

@pauska I don't believe so... but you're right... drives at the bottom of the chart are inactive.

not active, that is.

@pauska any idea where I can find that?

pauska

let me log in and see

or bleh, right click the storage pool and go into properites

is tiering a tab there?

ewwhite

one moment

pauska

and if it is - does it list "data to move within"

ewwhite

soooo slow to login

MIfe

Personally I think your not necessarily going to know if any of this will work unless you can determine what causes all the random writes.

pauska

11:42 AM

I agree there, but getting all disks utilized is something they should do either way

MIfe

It could be the way they are writing data, a bad option in a db somewhere, maybe they can store the data they are writing far more efficiently.

ewwhite

@MIfe their application is the cause for the writes.

pauska

yep

ewwhite

and it's a homegrown, devops-style application... and they work in an industry where the workloads are unpredictable

MIfe

If you can turn this around from "my san is the problem" to "your application is the problem" and why/how it is the problem you'll be in a much stronger position to assist them.

ewwhite

11:44 AM

bad design,

MIfe

Sure and they they store all their data into millions of little files, maybe what they really need is a database server so files become rows.

pauska

I forgot to ask - how are these (too large) LUN's carved up at the NAS head?

ewwhite

@MIfe it's hadoop, too

MIfe

Or if they have something even worse (like nfs holding database data) then move that to use a LUN directly instead or something.

ewwhite

@pauska tell me where to look

pauska

11:46 AM

wait a minute.. hm

the NAS head requires a set of two identical LUNS for storage

I don't see that in the report..

ewwhite

I just found something...

Domain : 	Local
Created : 	Mar 29, 2013 3:19:52 AM
Message : 	Slot 2: FSID:35 SavVol:159 Inactive
Full Description : 	A checkpoint is inactive due to limited space on the save volume.
Recommended Action : 	Delete old checkpoints to free up space. Or, check the system configuration to see if more space can be allocated for the save volume.
Event Code : 	0x1310460002

pauska

the NAS stripes two luns at all times

are you really sure that they are using NFS on this? that it's not some other file server connected via block?

ewwhite

Created : 	Mar 29, 2013 2:53:02 AM
Message : 	Slot2:1364540403: /nas/sbin/rootnas_fs -x ckpt_chk_archive_daily_001 QOSsize=20000M Error 3024: There are not enough free disks available to satisfy the request..
Full Description : 	The system encountered an error while trying to perform an autoextend operation on the specified checkpoint storage.
Recommended Action : 	Locate the error code in the brief description of this message. Run the nas_message -info command for the error code, and perform the recommended action.

@pauska no filesystems... all NFS

Created : 	Mar 26, 2013 10:49:31 PM
Message : 	Slot 2: Deduplication of file system id 24 was aborted after 503946 failure(s), 3528129048 KB available on FS.  Last reported error was: SavVolAtCapacity.
Full Description : 	The deduplication process on the specified file system was aborted.
 This could be caused by insufficient disk space or other system resource issues.
 If no corrective action is taken then the scan will be started again based on the current schedule.  The default schedule is weekly.  If you take corrective action then you can suspend and then turn on deduplication for the …

I'm really not responsible for this...

pauska

ok man

I think we're a point here where I would need access to help you :P

Or where you hire a EMC engineer, because im not that good

(are nobody looking at alerts on this vnx? it should be filled with errors)

ewwhite

I just checked... 50 errors

@pauska so there need to be two LUNs?

pauska

11:53 AM

well, yes

when you create storage for the NAS head you assign it two identical LUNS which it stripes

my itap license just expired here on the mac.. fuck.. give me 5 minutes

Dennis Kaarsemaker

@ewwhite they use a central storage for hadoop data? That's so utterly wrong and completely defeats the point of using hadoop...

ewwhite

@DennisKaarsemaker No, there's hadoop activity on the hosts.

Oh, no, that's a different group of servers...

phew...

Dennis Kaarsemaker

@ewwhite you said they don't have local storage, just VM's on shared storage and other shared storage mounts

ewwhite

the hadoop systems are just systems with local RAID0 volumes presented.

not the same servers as the ones in question

pauska

@ewwhite so, did you find a tiering tab on the storage pool?

Dennis Kaarsemaker

11:57 AM

ok, sounds better :)

ewwhite

pauska

right click, properties

look for a tab called tiering

ewwhite

I see a "Manage auto-tiering"

pauska

no, go to properties

ewwhite

okay.. there's a tiering tab

pauska

11:59 AM

hahahah

look at the move within counter

ewwhite

Seems we have no space.

pauska

thats the data it wants to spread out on the disks

no, the raid6 pool has free space

ask your storage engineer why the tiering is paused

ewwhite

@pauska which means... we're getting hot-spots and the data wants to redistribute itself?

pauska

correct

someone should get shot for this..

Cole

Morning

pauska

12:01 PM

look at the schedule and see if it makes sense for your client

to me it looks like you have certain periods where you do not want tiering to run (when IO go through the roof)

adjust the schedule and enable it..

ewwhite

right.

Dennis Kaarsemaker

damn. @Cole is awake. Run!

ewwhite

There is no storage engineer.

pauska

that would explain it

Cole

@DennisKaarsemaker unfortunately.

pauska

12:02 PM

trust me on this one.. enable it

it moves 1GB blocks around

just dont run it when you're very IO intensive

ewwhite

@pauska So it's basically a maintenance task that most people would have enabled

is there any way to see when it was last run?

pauska

someone has enabled it, and then paused it

no idea

ewwhite

yeah, but I'm just trying to help educate, too

pauska

perhaps someone saw that the data was just moving too much around and disabled the tiering..

but like the reports shows.. there are times during the day where the disks are mostly idle

just run relocation at those times on a low priority

ewwhite

Would @basil shake his head in disapproval?

pauska

12:06 PM

of what

MIfe

Good luck with your NAS :). Gonna go now, was interesting. Dont forget to IO profile your guests, theres possibly some underlying IO behaviors they need to fix on their guests which may be able to recover an order of magnitudes performance.

pauska

enabling it or leaving it disabled?

ewwhite

of the state of this setup

pauska

yes, he would..

btw, I figured out your NAS setup.. they are using volumes (LUN's) directly.. there is no file storage pool set up

ewwhite

@MIfe thank you for your help

@pauska how can you tell?

Cole

12:07 PM

Only reason I see to have auto tiering disabled is if your data in unpredictable when it goes from hot to cold to hot again.

pauska

@ewwhite cause file storage pool requires sets of luns for striping

actually, I think you can stripe more than 2 volumes if you want

its just a way of even out the I/O across the system

doesn't fit everyone

ewwhite

@pauska Everything on the nearline drives is for file

actually, EVERYTHING is file access.

they didn't want to use block storage for anything

pauska

their databases run on NFS?

must be fun during maintenance of the NAS heads..

ewwhite

@pauska Maintenance? Psssh.

pauska

(they're disruptive)

ewwhite

12:12 PM

oh, never mind. I found some block devices.

Host: scsi5 Channel: 00 Id: 01 Lun: 00
  Vendor: DGC      Model: VRAID            Rev: 0532
  Type:   Direct-Access                    ANSI SCSI revision:

pauska

click on hosts -> storage groups

Cole

God dammit forgot to mail out my rent check

pauska

filestorage is the NAS

everything else is block

ewwhite

7 hosts + ~filestorage

pauska

just click on filestorage, and then look at the luns below

are the monstrous TB luns there?

ewwhite

12:15 PM

@pauska I guess so.

pauska

yeah, 7,8,9

ewwhite

and those are not configured properly?

pauska

bah.. these should have been in a file storage pool and striped across the SP's..

but you can't change that now

ewwhite

as they should be in lun balanced across each head unit?

pauska

to me it looks like a old RAID engineer designed this

well, like I said with file storage pools you attach identical luns (2 or more)

those luns are evened out across the sp's

and then you create volumes from the file storage pool

ewwhite

12:19 PM

would that have had an impact on the performance problems we're seeing?

pauska

that, combined with auto-tiering (which is paused, remember) evens out the load across disks and storage processors

ewwhite

@pauska how did the data get so bunched up and not evenly distributed?

pauska

@ewwhite they started out with a smaller amount of disks, I guess

or they just got unlucky and got their most demanding luns on the same raid groups (6+2)

it doesn't automatically place luns on the least utilized ones.. it just chooses the ones with the most space available

this is why you use move-within tiering..

ewwhite

@pauska so would that have a positive impact on performance.

pauska

of course.. like you saw, there are disks idling

and the tiering properties showed that it want to move 15TB of data around..

ewwhite

12:22 PM

and fast cache should be enabled as well, right?

pauska

that depends.. you have two storage pools

maybe they want to reserve fast cache for the fast one

ewwhite

yes, we do...

it's in use on the fast one.

but was never really enabled on the slower pool

pauska

it would probably help tremendously, but it could hurt the db performance..

ewwhite

but it's read cache, right?

pauska

no, it writes aswell

this is why its so fucking great

sp -> controller cache -> fast cache -> storage pool

the monitoring software you used should give you details on how the fast cache is doing

if you have a high percentage of dirty pages then its probably not a good idea to enable it on the slower pool

ewwhite

12:27 PM

yeah, there's some info there.

pauska

if its around 40-60% dirty pages then you could probably enable fast cache and see how it goes

also, how large is it?

ewwhite

so what type of improvement can come from the tiering?

pauska

the bottom post here is worth reading: community.emc.com/thread/123274?start=15&tstart=0

ewwhite

@pauska oh, where can I check that?

pauska

@ewwhite click on system, and then "manage cache" on the right side

there is a fast cache tab there

@ewwhite it distributes the load across the pool. You get more IOPS. :)

ewwhite

12:32 PM

@pauska 4 x 100GB FAST cache

pauska

how are the pages right now?

ewwhite

but in RAID... so 183GB usable

pauska

they have to be, fast cache is always raid10

ewwhite

pauska

uhm

ewwhite

12:33 PM

haha

pauska

what the fuck

are your luns trespassed?

system -> reports -> trespassed luns

that, or someone did a terrible job creating luns on the fast pool..

ewwhite

why is the report still running!?!?

pauska

you have lots of luns

it takes a while per lun

is it filling up with luns?..

ewwhite

Yes.

across all of the VNX devices.

I only see two specific to this client.

pauska

that should be sorted.. just note that sensitive hosts might freak out if you move a lun

Ryan Ries

12:37 PM

Wait a second... Linux doesn't do DNS client caching ??

By default?

ewwhite

@RyanRies no. should it?

Ryan Ries

Well... yeah... imho

pauska

go to storage pool, and click the fast one.. look at the lun list below

are the luns evened out across sp's?

Ryan Ries

@ewwhite I was wondering why the Linux guys got all pissy when I rebooted my DNS server... I didn't realize they have no caching

ewwhite

@pauska 10 LUNs 6:4

pauska

12:40 PM

ok

and none of those were the trespassed ones?

ewwhite

nope

we need a pro, right?

pauska

well

it's not like its burning to the ground either, but yeah, some optimization should be done..

ewwhite

well, a client saying they can't work... seems like it's burning to the ground.

also, why does it seem like this thing is out of space?

pauska

the NFS volumes are out of space

ewwhite

the pools are 93% and 86% full

pauska

12:43 PM

thats because of thick luns

for some unknown reason you're not using thin provisioning

it makes sense on ultra-sensitive low latency stuff

like on that DB tier.. perhaps..

ewwhite

also, I see that all NFS exports are on datamover 2

pauska

to be honest - it looks like this was set up by someone who's stuck in 20 years ago designing raid arrays

the datamovers are active/passive, so thats normal

(this is why they're so insanely beefy with a ton of cores and ram)

ewwhite

Running Ubuntu?

pauska

its some kind of linux

there is also a hidden supervisor there.. hyper-v or esxi, cant remember

ewwhite

So I now know that I'll never be a storage guy.

pauska

12:46 PM

the block os runs on windows

and so it goes

ewwhite

so hearing this type of thing makes me lose confidence in my staff.

pauska

well - you could hire me for some hours :)

but - like that other guy said earlier - the client should be researched.. You need to find out what kind of I/O it is, and when it happens, and how it happens

FAST Cache won't help a bit if there are 20 vms doing sequential I/O at the same time

ewwhite

@pauska they don't know

DevOps

pauska

ok, well, then do stuff for them and hope it helps

ewwhite

and when I say "DevOps" derisively, I mean that they don't know what to measure, how to measure it, and they're not following good engineering practices...

but then, neither did we...

pauska

12:50 PM

I can tell you what I would do..

ewwhite

burn it all?

pauska

1) Enable thin provisioning across the board, unless they have a very specific need for thick luns
2) Move LUNs around so that the storage processors are equally busy
3) Enable FAST Cache on both storage pools and watch statistics very closely
4) Enable auto-tiering with a low-priority schedule that does not effect I/O when they need it the most
5) Grow the NFS volumes that are out of snapshot/dedup space (solved by point 1)

ewwhite

wow.. that's a lot

pauska

and MAYBE a 6'th point about enabling compression on LUNS with mostly inactive data

ewwhite

will turning thin prov. on immediately free up space?

(I'm a fan of ZFS compression)

pauska

12:53 PM

it will during relocation (tiering)

yeah, compression on the VNX is a breeze if your storage processors arent overloaded

compression forces thin provisioning

the point is - if the client actually needed data to stay on a specific set of disks (tiering disabled) then they shouldnt be using storage pools at all - old school raid groups should have been used

which is why I'm thinking that the dude who set this up didnt know what he was doing

ewwhite

right...

:(

pauska

not using file storage pools..

but again, those are the things I'd do right away

Hey mark - read up and get a good laugh (sorry edmund, but its not you fault)

I'm going to watch a top gear episode now.. give me a mail if you want to hire me for this

MDMarra

Yeah I was just reading

ewwhite

@pauska heh.. okay. I'll take this to the team.

@MDMarra No bitchassness - chat.stackexchange.com/transcript/message/8731399#8731399

MDMarra

I heard this at work yesterday when I asked for more storage for my HV cluster

"Oh shit. We have that VNX 5500 sitting around. Lets just hook that up"

"Oh yeah. I forgot about that"

We have so much vendor-supplied SWAG here that people forgot about a VNX5500

ewwhite

12:57 PM

I like simple storage... ZFS.

Transcript for