The Comms Room

Farseeker

12:00 AM

Around $100k/each if I remember

Chopper3

yeah, had one for a while, long time back

Farseeker

@Chopper3 - so you just do the planning, and you have minions who actually take care of the deployment for you?

sysadmin1138

I love minions. They're so useful.

Farseeker

I have 1.5 of them but they're always so busy :(

Chopper3

generally I'll get my hands dirty on the first one or two of everything, just to write/test procedures - then yeah someone else gets to do the iterative work - not a good idea for me to do it

Farseeker

12:02 AM

@Chopper3 - sounds like the way it should be

Chopper3

most of mine are either MS, HP, TCS, Tech Mahindra or Infosys though

sysadmin1138

Ayep. Keeps your hand in enough to keep sharp, but not so much you get terribly, horribly bored.

Chopper3

yeah, why pay me for experience and my thinking time then get me to repeat stuff - plus it helps these 20-odd year old guys too

Blog.SF

Server Fault Blog: What to do about Storage?

Posted by Kyle Brandt on January 24th, 2011

If you are really observant you may have noticed in my RRD graph post that our write times on our database server are not so hot:

Also looking at this from SQLs perspective confirms this:

Our sequential log writing however is really fast (<10ms), but there is a write performance problem when it comes to our database files that we need to fix.

Figuring out what direction to go in with our storage is proving to be quite challenging. The main reason is that there are just so many variables:

Right now we are thinking about 3 main options: …

Chopper3

I'm pretty lucky, once you've been around for a while some companies will pay you to pop in once a quarter to appraise their own people on what they could be doing better, where the industry is going, risks etc. Obviously you have to form these opinions but once it's done for customer #1 it doesn't take much changing to suit #2, 3 etc.

Sam

12:04 AM

I need a new minion, to much repetitive stuff to do. Plus It'd help in the plans for world domination

Chopper3

right then, bed time, remember, let's be careful out there

Sam

Night

Ben

Night

Warner

Finally got approval for hiring, pretty excited.

Might be around later, time to go home.

Sam

Time for me to sleep too, night all.

Ben

12:17 AM

Yes me too, night Sam, night all

sysadmin1138

G'night.

sysadmin1138

12:31 AM

Time to go home.

El Yobo

1:01 AM

Morning,

I've got a question over @stackoverflow that I was thinking of migrating here - stackoverflow.com/questions/4516089/…

Is there a way to migrate it, or should I just copy-paste?

Farseeker

If enough people think it should be on SF they will vote to migrate it.

If you want to speed it up, you might want to leave a comment asking people to vote for its migration

El Yobo

I've added a comment asking for migration; will see how it goes :)

Chopper3

1:06 PM

howdy

Hey Chopper!

hello

Gentlemen.

SO tired - not even funny :(

Tom O'Connor

hey

Sam

1:15 PM

Afternoon

Bart Silverstrim

Late night?

I hate questions like this serverfault.com/questions/226608/…

Chopper3

yeah, and an early start too

Bart Silverstrim

When I see them I wonder if it's another case of a school skimping by having a student in charge of a spare system in the closet that runs something that's supposed to be public-facing.

Chopper3

stupid brain's not working - plus I started the day with a long call to my agent, I just mumbled a bit and may have committed to doing an extra week's work in a month when I'm going to be busy already :(

Bart Silverstrim

You have an agent?

Tom O'Connor

1:18 PM

@BartSilverstrim Me too.

Bart Silverstrim

OR do you mean you're talking to your backup agent on a server? I do that sometimes.

Tom O'Connor

@BartSilverstrim Heyy.. I was that student once.. and look how I turned out (!)

Bart Silverstrim

I cringe when I see schools yet again treating technology as an afterthought or a glittery toy that just magically works when thrown into the classroom.

Chopper3

@BartSilverstrim yus, can't be trusted to schedule my own time basically

@BartSilverstrim You cringe whereas I simply ignore them :)

Bart Silverstrim

Train me as your sidekick in ruling the world by your side and I'll schedule your time for you :-)

Chopper3

1:21 PM

plus they get me gigs too of course and arrange stuff when doing mediawhoring etc

Bart Silverstrim

Does the UK education system get as much flak as the US education system?

Chopper3

in what way?

Bart Silverstrim

We're hearing it in the news periodically now, which is usually a bad sign.

Our students are far behind, we're stupid, ignorant on science and achievement in math/reading.

The big push is for accountability of schools.

So there's a standardization push being made for schools across states and possibly the country eventually.

If a school doesn't meet requirements X number of years in a row, the state "moves in" and threatens to fire everyone and replace them.

So now we have a system that focuses on teaching to the test, slowly but surely.

And everyone has to hit certain scores on these tests so that by...2014? 2013? I don't remember...all students must be 100% passing, as I understand it, even special ed students.

jscott

@BartSilverstrim "100% passing" is a simple tweak of the SIS data. I am sure districts will become more creative with their grading/assesments.

Bart Silverstrim

Dat be me.

@jscott, you're not entirely that far from me I see.

Ah, yes, another weakness of accountability and teaching to the test. Tweaking of data.

jscott

1:28 PM

We hear some um, interesting sounding records added to the SIS. I am not privy to all of it but it is surprising

Bart Silverstrim

You're in public education or private/college?

jscott

Public, K-12

Bart Silverstrim

So you're seeing NY version of standards pushes.

jscott

I've heard the SIS liaisons describe unique ways of recording things to keep the graduation rates high.

Bart Silverstrim

I'm wondering how long it'll be before teaching is treated as a mcjob.

Really? I've heard nothing other than the standard pity-passing whispers that have always been around since I've been in school.

jscott

1:31 PM

I am not sure if it is a wide spread issue, but, apparently, it does happen.

Bart Silverstrim

PA has a new thing they're calling the keystone exams, but I think NY has had some sort of exit exam required for graduation for awhile?

jscott

Regents, sort of like that. You can graduate without a Regents diploma

That's a whole other issue :)

[Some] teachers get stuck teaching to-the-exam rather than "true" curricular topics.

Bart Silverstrim

Another case of a good intention gamed for the system, eh?

jscott

Right.

Bart Silverstrim

I miss the idealism of star trek. Every planet had a homogeneous culture and set of values, similar dress style no matter where you went, and every being had the same general attributes that neatly summed them up by their race...ferengi were always X, all Klingons acted Y...only time there was a deviance was when it served a greater good to the plotline.

We can't even cross a state border without finding people acting like aliens to our hometowns.

Chopper3

1:39 PM

sorry - was away - certainly come newspapers say that the schools are getting worse all the time, kids are getting stupider etc etc. It's easy to assume they are when you speak to kids because we forget how dumb we were then too but I know for a fact that my kids are WAY smarter and better educated right now than I was at that age

Bart Silverstrim

problem is that by the time I have the wisdom to see how good or bad things really are I'll be on life support or stuck in a corner of the "home" in my wheelchair. :-/

Chopper3

I've spent a lot of time in the US and it is clear that the 'smarter', more 'switched on' guys tend to be closer to the seaboards but I know that's a real generalisation - but don't feel bad about the US, the UK is FULL of fucking idiots too

the difference is that with the advent of instant knowledge and communication transfer systems such as this we can see the gulf between smart and dumb in much clearer detail - the same as we saw the gulk between rich and poor, healthy and starving etc more with the advent of worldwide TV coverage

it was always there but it was easy to ignore, these days we see the world with a much wider view and so those who focus on the negatives of the world can more quickly do their 'job' but nothing's changed really.

Bart Silverstrim

That's more depressing. Technology was supposed to be empowering, instead it's a spotlight on our stupidity.

Tom O'Connor

@BartSilverstrim Empowers the stupid too.

Bart Silverstrim

No, enables the stupid, there's a difference. :-)

It empowers the people with initiative and ambition. It enables the stupid to do more stupid things to a wider audience (much faster, too).

Ooh! I have a phone that takes pictures! I know, I'll take a picture of myself naked and forward it to my new boyfriend that I really really like so much and be surprised when he forwards it to his buddies on the <sport of choice> team! Tee hee! </facepalm>

Chopper3

1:49 PM

@BartSilverstrim It's a cup half-full/empty thing though - without the internet I wouldn't know you (which would be a bad thing) but then again I'd never have heard of Wildchild either :)

Bart Silverstrim

Chopper does have a point. Perhaps we've always been lazy, greedy, self centered and stupid and newer technologies just highlight how lazy, greedy, self centered and stupid we are.

jscott

"Your ideas intrigue me. I'd like to subscribe to your newsletter."

Chopper3

@TomOConnor Stupid can't be empowered, but smart people can use the ill-formed opinions of the stupid for their own benefit

Robert Moir

Working in a college, I tend to remember the students that stick out as being exceptionally smart or exceptionally.... uh as i'm at work, what's the politically correct term for "as stupid as a sack full of wet mice"

so the point of that is you remember the exceptionally dumb and smart people more than you do the average ones

and there's less really smart people than there are really dumb ones

Chopper3

@BartSilverstrim Human thought processes, instincts, needs and wants haven't changed for thousands of years - what's changed is our ability to comprehend (and judge) others outside our direct sphere of knowledge - this can be used for good or for bad - basically it depends on the individual

packs

1:52 PM

@BartSilverstrim Hah! Keystone exams in Penn. I see what they did there.

Robert Moir

don't think I've seen you in a while @tom, everything going ok i hope

Bart Silverstrim

@packs: Is there a push for the oHIo "Howdies" exams? :-)

@chopper3: you seem wiser than you should be...I can almost hear a British accent from your text.

packs

@BartSilverstrim I wouldn't know, I'm in Higher Ed. Back in Kentucky we fixed the glitch by making teaching so low pay and unglamorous that the only people who went into teaching lowered the bar for us.

Robert Moir

Quite.

More tea, Vicar, what?

packs

@RobertMoir Wot?

Robert Moir

1:57 PM

one must not drop one's "h"s, don't you know. It's just not cricket

Bart Silverstrim

@Packs: that is a scary unintended consequence that I've been trying to ignore.

packs

@RobertMoir I was going to attempt to type in a good ole fashioned cockney accent. Unfortunately, I found even the idea rather distasteful.

Robert Moir

gawd blimey guvner, what a lark eh

Bart Silverstrim

It seems with all the hassles in public education the only people that stick it out are people that fit the environment, turning it into a self-serving loop to the system. More people who would be good teachers run away screaming. The incentives to have an actual academic system aren't there, but there are plenty of incentives to keep things as they are, even as there are cries that the system must change.

packs

@BartSilverstrim My parents were both teaches (k12 and university). I had the fortunate distinction of being able to watch it happen from a young age. It rather jaded me to the industry rather early.

Bart Silverstrim

2:00 PM

@packs-my mom was a teacher.

Robert Moir

well this brings me back to my theory that staff who have experience outside education are far more valuable than people who have always worked in education

Bart Silverstrim

Wife is one now.

I don't know...a lot of people who get into teaching, if you listen to student teachers, have no freakin' clue what teaching really entails.

They see it as a job where you work 8 to 3 and get summers off.

packs

@BartSilverstrim Then you still get to see it first hand. I am good friends with some of those few teachers whom I would actually trust to teach my children.

Robert Moir

consider the teacher who studies in school as a child, who goes through college and university and then returns to school to teach. They've never left school in a way. I'm not saying that makes them bad people or bad teachers or anything, just that a wider base of experience is a good thing

Bart Silverstrim

They shift. As students in school and ignorant college students, they're exposed to teaching systems as a student, not as a teacher.

packs

2:02 PM

I also had the unfortunate distinction of being a TA and tutor during college, and spent way more time than I would care to remember attempting to teach education majors high school algebra.

Bart Silverstrim

Or administrator.

Robert Moir

oh they shift

I'm not saying that they're like the children, I'm saying that their experience is limited to schools only

Bart Silverstrim

I've seen way too many student teachers who are shocked...*shocked*...to see how much of their life is gone if they become a teacher.

No more partying at the bars (without risking getting fired for something stupid...)? Correct these at home? What do you mean I have to take courses over the summer?! That's my VACATION!

And worse, colleges don't teach teachers how to teach!

packs

And the administrative culture pushes those few, truly good, educators out the door with meaningless bureaucracy, seniority rules, and complacent tenured faculty.

Kyle Brandt

<rant>How the hell can all these storage reviews not include the 4 variables I mentioned .... useless!</rant>

Robert Moir

2:06 PM

i know my brother used to teach and hated all the paperwork

Chris S

@packs Teaching certs really need to be more specific. I think it varies by state, but here there's a single certificate to teach in high school; no topic qualifications or anything, if you pass you are legally allowed to teach anything offered to grades 9 to 12. Schools normaly try to keep teachers in their areas of knowledge and comfort, but when push comes to shove (read 'budget cuts') that doesn't always happen.

Bart Silverstrim

You either have it or you don't. Student teachers are like deer in headlights when they start teaching. And they don't seem to get a fundamental grasp on communication skills for teaching. It's great that you know every detail of American history to teach History class, but if you can't communicate it you might as well become a historian, not a teacher.

packs

@RobertMoir My mother did Special Ed. Most of my childhood memories are her filling out the weekly mountains of paperwork per child. It can get rather obscene.

Robert Moir

yep

packs

@ChrisS @Bart may be able to speak more to it, but I seem to recall that being the case as well.

Now, after I graduated with my bachelor's, I looked at a local tech school for employment.

They required a certain number of college credit hours in a subject in order to teach it.

Chris S

2:10 PM

Having half a dozen area specific certs (mostly for the core topics) would help differeientiate the teachers, might push pay up a touch (good teachers are underpaid), and have a positive effect all around. Might also help to get rid of tenure, which I consider a plague of the uneducated past.

Bart Silverstrim

Depends on the school, probably. Private schools have more leeway in their "Standards." Some schools will have local business people just come in and teach things if they have "equivalent experience".

Chopper3

phone went, glad you guys got that stuff out of your system :)

Bart Silverstrim

We did?

packs

@ChrisS I find the concept of tenure at the college level to actually make sense. In the absence of actual research, not to mention federally/state mandated curriculum, it seems a lot less useful.

Chopper3

@KyleBrandt and kyle - the reason they don't quote things is because of the literally billions of combinations of array build, config and use

so they just post the biggest numbers they can as it's a competitive world

Bart Silverstrim

2:12 PM

Maybe we should automate teaching. Find a way for it all to be done by AI routines.

Kyle Brandt

@Chopper3: But those variables on say a 15 drive RAID 10 (for 16 shelf) seems like a reasonable number to have

Bart Silverstrim

If they could I'm sure they'd do that for fast food chains.

packs

@BartSilverstrim We just need to find something to store his software and copy @Ben a bunch of times.

Chopper3

you'd have thought so wouldn't you, certainly a single disk can be pretty predictable - it's how they're grouped/plexed/whatever'ed that can make the difference for different types of performance

Chris S

@packs Maybe at small colleges where an authoritarian and overzealous administrative staff tries to dictate how professor teach students... Otherwise, it's an antiquated cousin of nepotism that allows teachers to get rusty.

Kyle Brandt

2:14 PM

@Chopper3: Actually screw that, that should have those numbers for whole craploads of combinations

I'm sure they do, they just keep them in the vault

Chopper3

for instance on most HP arrays if you have multiples of 8 shelves and group 'vertically' as opposed to horizontally you'll see much better random write performance

packs

@ChrisS It also depends quite a bit on the field. Research in Engineering is significantly less likely to be politically/socially questionable, whereas something such as Biology of Sociology might be.

Bart Silverstrim

@kylebrandt:until they claim that new firmware or other tweaks changes the numbers, or the time they get published they've come out with a new product? :-)

Chopper3

it's purely a commercial thing - if one company publishes one set of results for one combination of parts and use-case then someone else will come along and make that look stupid by changing their parts and use-case even though the first solution will be better at something the second one doesn't talk about

Chris S

@packs Perhaps we need a different way to protect free thought then. Tenure is used far less often to protect legitimate new thought/research and much more often as a crutch for lazy "teachers" to stay on the payroll.

packs

2:18 PM

@ChrisS It was always explained to be that tenure was largely designed to protect the professor from retaliation based on teaching or researching polarizing areas.

Chopper3

it's the same with any sufficiently complex system - i.e. is Oracle faster than MSSQL? who knows unless you all agree a full top-to-toe test system and methodology

packs

@ChrisS Right. If I am right in my understanding, then tenure could be considered almost useless at non-research institutions. Or in roles where research is not required.

Chris S

@packs I know that tenure is supposed to protect teachers from administrative staff who (for whatever reason) disagree with their opinions... the problem is, having working somewhere X amount of time has little to do with future opinions or research.

Chopper3

then if something changes in th test system or methodology all bets are off until both are retested

packs

@ChrisS Uncontested.

Kyle Brandt

2:20 PM

@Chopper3: Well I don't agree with the sql vs oracle comparison. But that aside SPC1 made a standard for OLTP systems (To HPs credit they submitted some of their systems)

Chris S

@packs That makes sense. I have to admit, I hate when I find a problem but can't think of a sufficient solution.

Kyle Brandt

Take for example Intel's download.intel.com/design/flash/nand/extreme/…

packs

@ChrisS I know how you feel. All too often any potential solutions I do think of scare the crap out of me because of what consequences the changes might have.

Kyle Brandt

You can go into all the flaws with that document, but at least it is something

Chopper3

either way kyle I think you know that sooner or later you need a block-level san as you will be MS Clustering at some point - in my experience if you have to do that I'd go 10Gb FCoE first then 8Gb FC then 4GB FC then 10Gb iSCSI then 1Gb iSCSI in that order - today unless you have silly budget I'd put logs on 15k disks, data on 15 or 10k disks, backups and other files on 10k disks and wait for the next generation of SSDs - unless you can afford a 3Par box that'll auto-TSM clusterlets

Kyle Brandt

2:23 PM

@Chopper3: I'm not sure we will ever cluster

Next SQL you can read off of a asynchronous mirror

Chopper3

well if you're sure of that then I'd be tempted to go for a big box of 2.5 inch 15k 146GB disks over SAS with a decent controller - plenty fast enough, cheaper and consistent performance over the box's lifetime

Kyle Brandt

It really comes down to writes for us I think. For reads we can just keep buying more memory until the technology changes. The only problem I see with that might be ramp up time

Right now we pull off about 90-100 MBs random writes off of a 10k SAS Raid 10 with 6 drives

I don't even know if that sucks or not :-(

Maybe I should see if I can get someone at Dell to tell me if that is right

Chopper3

how about a HP MSA70 with 25 x 15k 146GB disks - set one as a hot spare and R10 the remaining 24

I'm sure dell do something similar

for random anything those 2.5's beat 3.5's hands-down - I use them for VoD

Kyle Brandt

@Chopper3: I don't know ... I need data

Chris S

@KyleBrandt I can get 130MBps off 4x 500GB SATA 5400RPM in a RAID-5 at home. =P

Kyle Brandt

2:28 PM

@ChrisS: That is random writes?

And what is the response time?

Bart Silverstrim

@KyleBrandt it's taking him awhile to answer...see what I did there? HA! I slay me.

Chris S

@KyleBrandt Random writes (512MB cache might help it...) 10GB data total, 1MB block size. Not actually a setup I would recommend for DBs, just messing with you a bit. We actually have a MSA70 setup very similar to Chopper's suggestion here at work.

Kyle Brandt

So I should have been clearer. We get 97 MB/s with a 2-3MS response time with 2 threads and 8 outstanding requests for random writes with a 13G test file and a 512 MB cache.

Chopper3

@KyleBrandt sorry - got a confcall - back in a mo

Bart Silverstrim

Silly question thrown out here for @Kyle\@Chopper\etc...if you have a standard test or a standard whatever for measuring something, won't that encourage vendors to game the system by tuning to that test? I thought that was a criticism of graphics card companies/drivers for games.

Kyle Brandt

2:35 PM

When I was stupid enough to run a second benchmark at a later day that fit only into the cache I got 30k IOPs ;-)

Chopper3

actually Kyle I have a DL380 with two MSA70's full of 10k disks in my lab - I'm sure I tested them with iozone a while back, let me see what stats I got out of them, back after this call

@BartSilverstrim yes and yes

Kyle Brandt

@Chopper3: Thanks, I would love to see that!

Chris S

@BartSilverstrim yeah, but it does help a bit, and if the test is sufficiently broad then it might actually be helpful. For instance, disk IO performance, I've been quite a few places throw out numbers like 1 zillion MBps, without qualifying the number with the test parameters (random/sequential, read/write/mix, block size, total count - those are about the most important) or they pick crazy numbers (like my above reference, my home array can get great performance if I game the parameters enough).

Kyle Brandt

Well in that Intel example I have, they listed their testing methodology with instructions on how to repeat it.

Bart Silverstrim

@Kyle/@chriss:I didn't mean to criticize the idea in general, just was musing that I had heard something about vendors having tweaks compiled in to drivers to boost certain ratings if it meant better reviews and sales. I agree that certain broad things can be done to get an idea of performance...

it just occurred to me that a vendor could tweak things here and there to make the numbers not so relevant if they wanted, so it wouldn't be easy to have something like a universal test for performance in comparing, say, two database products.

Chris S

2:44 PM

@KyleBrandt Yep, the more responsible/professional companies will detail how they got their numbers (and will usually pick reasonably 'real world' parameters for the tests). Though I see they used different controllers cards in that test, not just different drives.

Kyle Brandt

Got an email from someone at ramsan.com -- looking past the Texas theme their page has, there are some good whitepapers and information in there

Chris S

@BartSilverstrim I've heard quite a bit of such 'optimizations' for the very popular general benchmarking suites... Generally resulting in inflated numbers. Video cards are pretty bad for this (in my opinion), and it sometimes shows when they do a benchmark followed by benchmarking a bunch of actual games.

Evan Anderson

@KyleBrandt: 90-100MB/sec from six 10K SAS drives in a RAID-10 on, presumably, a Dell H700 RAID controller isn't too shabby based on my experience benchmarking eight 15K SAS drives in a RAID-10 on a Dell R710 w/ an H700 RAID controller (benchmarked using IOmeter-- I don't have all the test parameters handy, though).

Kyle Brandt

@EvanAnderson: Ah thanks for that -- yes that is our current card

Chopper3

Kyle - this is a heavily edited email I sent a few weeks back - pastebin.com/b89f7u4e

Evan Anderson

2:49 PM

I've got a pair of R710's each w/ eight 15K SAS 2.5" drives on H700 controllers running RAID-10. I was able to get about 130MB/sec random read/write performance at approximately 3,500 IOPS off of them. I don't recall what the latency was and, stupidly, I appear to have misplaced the test output. (They're production now, so no repeating that test... >smile<)

Kyle Brandt

@EvanAnderson: Ya that is interesting, I would be interested to see how Random Writes scale with the size of a Raid10 array in different systems

Evan Anderson

I only tested with an 8 disk RAID-10, unfortunately, so I can't tell you what the curve looked like building up to 8 disks.

Kyle Brandt

I think ours are 10k. Sounds like 15k doesn't buy that much

Evan Anderson

My latency is probably a little better than yours, but that ought to be about it.

I wouldn't expect massive differences in IOPS or throughput.

I'm mulling where to go w/ the storage with that Customer next, but it'll probably be 3 - 4 months before I'm ready to do any benchmarking on the new kit. I'm strongly considering shared SAS in lieu of FC or iSCSI. We're not planning on expanding beyond the 2 node VMware cluster anytime soon (i.e. wildly over-purchased the CPU and RAM in those boxes), but it would be nice to have shared storage.

Sam

@KyleBrandt We've just tested one of the bottom of the range RamSans (I say bottom of range at $20K) and it did produce some good results

Chopper3

2:55 PM

@KyleBrandt if you look at that pastebin the answer is 'not that well'

Kyle Brandt

So when you add outstanding requests, is the latency then the time it takes for the outstanding queue and completion ?

@Chopper3: Still parsing that :-)

Chopper3

lines 68 and 133 are what you care about I guess

Kyle Brandt

@Chopper3: Well since we don't have that much data is just lives in memory

That will break eventually of course though, it is all about when

Chopper3

cool, I was testing this spec for a very specific requirement, just happens to be on the kit I was mentioing

Kyle Brandt

But with the projection, not in the next 36 months :-)

Well this is really my first time going that deep into storage, so the more data I see the better framework I have

I have had and looked at SANs but not at this depth really

Chopper3

2:59 PM

essentially a 24/25 disk MSA70 will give you ~190MBps random read when configured as I did (i.e. with 100% read cache - which is what I wanted)

Kyle Brandt

They just did the job as is

Chopper3

@KyleBrandt it's a big part of what I do :)

Kyle Brandt

@Chopper3: I was wondering about that, with SANs can you control how much cache is read and how much is write?

Chris S

@KyleBrandt On all of the nicer equipment, it's very configurable

Kyle Brandt

So with a SAN, slow disk but a giant cache might work well for us

Robert Moir

3:02 PM

we were looking at the coRAID box for a SAN recently

there's some clever clever stuff out there

Evan Anderson

@KyleBrandt: There may be a formal latency definition that I'm ignorant of, but I think of it as the amount of time an outstanding request is "in flight".

The faster seek latency on a 15K disk should translate into lower latency, overall, for random operations. For sequential operations, the 10K versus 15K shouldn't make much difference.

I'd expect that you'd have to have a deep queue of random operations, though, to really see a marked performance improvement between 15K and 10K disks just on the basis of decreased seek latency.

Kyle Brandt

@EvanAnderson: Not sure what you mean by in flight...basically, does the time in the queue count?

Evan Anderson

@KyleBrandt: Oh, yeah-- I'd count the time in queue for sure. "In flight" just meaning, colloquially, that the request has been made from the application layer and hasn't returned. In the case of reads I'd measure latency from the time that the application issues the read request until the data is returned. Working with a data set larger than cache and issuing enough simultaneous requests to consume 100% of the disk's time would be necessary to get a feel for the disks' worst-case performance.

Chopper3

@KyleBrandt sorry - was on a call - it depends on what model you buy

Evan Anderson

My (possibly ignorant) storage benchmarking methodology has been based on measuring the expected read/write percentage and average I/O size of the candidate data (when it can be measured), constructing a test (typically using IOmeter) that replicates those conditions as much as possible using 100% random access, and then piling on more and more "workers" to generate simultaneous I/O requests until the IOPS flatlines (or decreases) and the latency begins to increase.

Kyle Brandt

3:12 PM

I guess large cache at a certain point doesn't do much though. My thinking is that all a write cache does really is smooth out spiky shapes to a sustained pattern. Eventually, if the disks don't keep up with that sustained rate, then the cache will voerflow

Evan Anderson

When I get to that point (where the IOPS are flat or falling and the latency is increasing) I feel like, for the given read/write workload in a worst-case 100% random scenario, I've found the edge of the performance envelope of the physical disks.

Kyle Brandt

@EvanAnderson: That makes sense. I have all that data -- just nothing to benchmark yet :-P

Chopper3

@KyleBrandt If you don't need shared storage (i.e. no clustering) going for a san will give you nothing over a nice big well-spec'ed DAS box

Kyle Brandt

@EvanAnderson: Brent has a sqlio benchmark that does that with the 4 variables I had in my post

@Chopper3: That has crossed my mind. If we are to get two sans, is it really any different from DAS except for the features?

sysadmin1138

@Chopper3 Hey... I know that output.

Chopper3

3:15 PM

@EvanAnderson I use 15k's because I have thousands of movies, in both SD and HD formats and >600k users pulling at different films at different points in the films - so basically no caching will ever work so I just need fast random read over large(ish) data set - for now only lots of 15k's will do that for me

Evan Anderson

@Chopper3: That makes good sense. You've got a cache nightmare there!

sysadmin1138

@KyleBrandt Two SANs just give you potentially better-featured controllers than you'd get DAS. Is that worth the expense? I personally don't think so.

Chopper3

@KyleBrandt I don't really get the two-sans thing kyle - we have dual-sans but that's just for DR, one syncs with the other and act in an active/passive style

Kyle Brandt

@Chopper3: Well, it is if the san does fail, you are kinda screwed

Zypher

@Chopper3 by commandment of Jeff no other reason

Evan Anderson

3:18 PM

There's something attractive about using DAS and having a "shared nothing" cluster using application-layer data replication.

Zypher

i maintain it's overfill

Kyle Brandt

Or you can just come out and say that @Zypher :-P

Zypher

and mornin all :)

sysadmin1138

G'morning, Zypher. I'm even eating breakfast!

Zypher

oo damnit goes to get cereal from kitchen

voretaq7

3:19 PM

breakfast? Wossat?

packs

@sysadmin1138 Wow, I was eating mine almost 3 hours ago.

Evan Anderson

Probably time to get back to "real work"... ugh. Talk to you guys later.

Kyle Brandt

@EvanAnderson: Thanks for your input!

packs

@EvanAnderson Enjoy.

sysadmin1138

@packs Well, I should get dressed and go to work.

packs

3:22 PM

@sysadmin1138 I thought Tuesday was 'Go To Work Pantless' day for you left-coasters.

sysadmin1138

Aaaahm, it is but some of us object to kilts in this kind of weather. We're not natives, and it shows.

Robert Moir

its cold here today

i don't know about numbers but i'm feeling it

Chopper3

bloody chat timeout problem again :(

Robert Moir

doh

sysadmin1138

It's 46 right now, which is unseasonably warm. My friends in the middle of the contentment are dealing with arctic weather thanks to that.

Chopper3

3:24 PM

@EvanAnderson we do application-level clustering for our DR sites that are outside latency limits, works well to be honest

Zypher

my god, i walked outside into 23 degree weather and thought to myself "isn't this pleasant"

Chopper3

@KyleBrandt How do you see the two-sans thing working? how would you sync/backup and how would it failover/back?

packs

@sysadmin1138 In my rolling out of bed haze I think I heard that upstate New York was looking at -35F. Jeebus!

Sam

it's grey and windy here, which is not unusual for England

Kyle Brandt

@Chopper3: Interesting you should ask that. I see it delivering the enterprise advantage in a forward thinking IT environment.

sysadmin1138

3:26 PM

@packs That's the kind of thing I'm talking about.

Zypher

@kyle don't my my eye bleed this early

please

Bart Silverstrim

Only sysadmins normally kept in the basement would go online and compare weather reports they probably get from weather sites in the first place since they don't have windows to view the outdoors... :-/

Chopper3

@KyleBrandt huh?

packs

@sysadmin1138 I was rather happy to get my raining 40s. Now skeedaddle.

Kyle Brandt

@Chopper3: Sorry, the translation is, I have no idea at this point in time.

Zypher

3:27 PM

but in honesty at least with the stuff we've been looking at thinks like block level replication or snap mirroring are options

Robert Moir

I'm thinking of the evening plans while some of you are talking breakfast: go home, fire up the kinect, drink beer and watch the soccer with a friend who's bring round fish and chips... then the SF chat!

Chopper3

ah - I see, I have to do a decent amount of DR planning/design and it's always the fallback that's the tricky bit

Kyle Brandt

@Chopper3: Jeff really does not like the idea of one device (no matter the builtin redundancy) being a single point of failure. Brent has mentioned in his blog he has seen SANs fail.

packs

@RobertMoir It has been my experience that the UTC-5 evenings in the security chat are often drunken conversations.

Kyle Brandt

@Chopper3: Right, so the question is if the SAN does fail, what is our time to recovery

Sam

3:29 PM

@Robert thats the joy of being in the UK and living in the future, home time soon!

Robert Moir

yes indeed

Chopper3

ok, we use block level rep too - that's nice and all - let's forget about replicating bad data for now - so the array falls over - how do you recover?

Kyle Brandt

If we have a contract that says we can have a SAN in our datacenter in 4 hours from the moment someone answers the phone we might be less concerned. I think we are just quite at that stage yet.

Robert Moir

with those kinds of fails, you can usually get all the big ticket stuff into place and find yourself hamstrung by silly little things.

Chopper3

so do you present the replicated lun to the servers all the time or only at failover time and change your mount-points after switching the luns to live mode?

Kyle Brandt

3:30 PM

@Chopper3: I think if it really has to be Two sans, than I think SSDs or FusionIO in the servers makes a lot more sense right now

Chopper3

yes, me too

Zypher

@Chopper3 off the top of my head it'd have to be some sort of manual re-presenting of the LUNS from the snap mirror to the server

Chopper3

I frickin LOVE FusionIO - but as I've said before I'm personally struggling for a valid use-case

Kyle Brandt

Which is why I think that is geting a little ahead of ourselves. Jeff might be okay with a single SAN if we got an great contract

Zypher

but ... since i think having two sans for that sitting next to eachother in the DC is kinda silly never gave it much thought

Kyle Brandt

3:32 PM

@Chopper3: Well the use case I see is a FusionIO in say each server with asynchronous mirroring.

Chopper3

and forget about your logs?

@KyleBrandt on the open db files you mean?

Kyle Brandt

@Chopper3: Not sure what you are asking me there ... can you expand?

Chopper3

@KyleBrandt or using mssql mirroring? (I'm no expert sorry, not really - we have embedded MS techies to help me at that level :) )

Kyle Brandt

SQL Servers support asynchronous mirroring. So it is like a mirror but the writes don't have to wait for the second server

The advantage is speed, the disadvantage would be say 90 seconds of data loss and the time it takes to manually failover

Chopper3

ah ok, like I said I wouldn't consider myself an expert

your business can easily support 90 seconds of lost data though right?

Bart Silverstrim

3:34 PM

@kyleBrandt: you mean there's a window before database servers A and B are in "sync" with each other?

Zypher

@Chopper3 the free public sites yes

Kyle Brandt

Neither am I, but I consider Brent Ozar one and he helped us out with our failover and availability options :-)

@BartSilverstrim: Right

Zypher

need a different strategy for stuff like careers

Chopper3

yeah, he's a hell of a guy

Bart Silverstrim

In cases like that...if you don't mind schooling someone...is database server B like a backup, or is it part of production? i.e., are there read requests coming into both database servers or is the B server just a backup in case A fails?

Chopper3

3:35 PM

really? 90 seconds? who's going to moan about that really?

Kyle Brandt

@BartSilverstrim: Not until the next version of SQL

@Chopper3: That is the way Jeff and I feel. I mean people will moan, but what-ev

@Zypher is more respectful of your data :-)

Zypher

sorry if i against pissing off people who give us money :)

Bart Silverstrim

(irony would be if the data lost in the failover for the careers db are resumes for DB/cluster admins...)

Kyle Brandt

@Zypher: Oh I think a different plan for Careers totally makes sense

But that could be sync mirror, not sure that needs quite the speed

Or log shipping

Chopper3

I bet if you asked jeff how much business impact losing 90 secs of careers would cost compared to how much you'd need to spend to reduce that I think you'd quickly find how this async solution would be perfectly adequate

Kyle Brandt

3:39 PM

Actually we have a Taller Jeff for that ... I wonder what his feeling is

Chopper3

shows how I've lost track of the whole SE business team - you mean it's not just J&J any more :)

Zypher

haha

ehh happens ,nah we have VP of Products who is in charge of the pieces that ... make us money

"Tall Jeff"

Kyle Brandt

@Chopper3: But one of the points that drives the SAN is that we don't want to back ourselves into a wall either. It is far easier ot learn stuff now that we are 2x the size

For instance learning to break up the DB into different files etc

Chopper3

well it's his budget this would have to come out of then I guess - usually when you give a business person the choice between minor inconvenience and big bill they go with the former option - that's why I love working in the finance sector they always go for the less risky/more expensive option :)

sysadmin1138

Time to go to work.

Zypher

3:42 PM

haha

yea ... uhh budgets

we asked about that once

Kyle Brandt

"Make it awesome"

Zypher

"and don't get raped"

Chopper3

@KyleBrandt That's true - and why I suggested that particular model/parts - very easy to setup but lots of 'headroom'

Kyle Brandt

More seriously we want a balance of value and cost more than just a fixed cost

Warner

hahaha

That's my kind of budgeting. ;)

Zypher

3:44 PM

it's called venture capital :)

Chopper3

@KyleBrandt I'm really not convinced by that dell 10Gb iscsi/SSD thing though you know - seriously - think you can get what you need for less

Kyle Brandt

I also need to look at non-dell SSDs with a new controller in each server

Zypher

well that and every C-Level is a tech not a suit

Kyle Brandt

@Chopper3: Neither am I ... I actually feel that it might be in a year though -- hence my idea of waiting

Eventually they will all support SSD right? Not just the fancy models

Chopper3

@KyleBrandt there's a huge gap between enterprise SSDs though - general desktop ones have about 7% overcommit space, good ones 28%, enterprise ones more like 400%+

Kyle Brandt

3:46 PM

So are SSDs in the high ends because of technology, or just because they CAN

Warner

I debating wearing a suit to work everyday in a technical role before. I like keeping people on their toes.

Kyle Brandt

@Chopper3: No we want enterprise, just not 5k for 100GB enterprise rape

Chopper3

@KyleBrandt totally - 5 years from now we'll only use spinny metal things for archive

Kyle Brandt

@Chopper3: So if the only reason they are only in the high end models is because they can -- then that should change fast

Chris S

@Chopper3 I have to admit I cringe a little when people say 10GBe and iSCSI.... If you want that kind of performance you should probably be looking at FC or similar.

Chopper3

3:47 PM

@KyleBrandt exactly - not when a bunch of 2.5" 15k's will give you all the random writes you need for 25-40% of the cost for a lot more space too

Kyle Brandt

@Chopper3: Well non-dell enterprise SSDs are I think 750-1k

Sam

@KyleBrandt or in some cases 20k for 300Gb

Chopper3

@ChrisS iSCSI is a stop-gap, it's kind of a 'something for nothing' thing while FCoE gets out there and drops in price - there's no competition when it comes to functionality and performance

Kyle Brandt

So if a 4k for an SSD 4 disk RAID 10 array might be totally doable depending on the performance

Just not at Dell's list pricing

Chopper3

@KyleBrandt I have literally no idea when it comes to non-HP/IBM/Dell/Sun/apple kit - total corporate whore sorry

Kyle Brandt

3:49 PM

@Chopper3: Well the thing is we just don't need much capacity

As I said, in terms of IOPS. Number 1 is Stack Overflow and number 2 is Super user

Chopper3

how much are you writing per day do you think? for non-enterprise SSDs they usually start cutting into your overcommit when you're going over about 35GB/day

Kyle Brandt

Number 1 has 30x the iops of number 2

Chopper3

not tempted to put them on different disks?

Kyle Brandt

@Chopper3: Good question I can find that

Bart Silverstrim

Yay cloud!

Zypher

3:51 PM

Yay astersk documentation !

Kyle Brandt

@Chopper3: Oh we totally are

Zypher

... oh wait

Chopper3

@KyleBrandt you might be surprised, I put a ocz vertex 2 in my MBP last week - I was doing something stupid like 8GB/day just browsing and emailing stuff - might be an OSX thing though?

@BartSilverstrim too sunny for you?

Bart Silverstrim

It's always a sunny day when I don't have Windows.

Kyle Brandt

@Chopper3: Give me a sec and I will get some BS average extraction :-P

Chopper3

3:56 PM

just looking on my W7 laptop I'm struggling to find which perfmon counters to look at - nothing's leaping out at me, might be different if I could be arsed rdp'ing into a w2k8 box but there doesn't seem to be a 'how much per day/since-x' counter??

Dan

Ugh, dealing with a customer who is clueless -which is ok- but their one and only admin is out on paternity leave for 3 weeks, and when called for help, actually told them not to call because they were on vacation. Fucking lazy admins.

Kyle Brandt

@Chopper3: 1.8GB by taking the average bytes written a second and multiplying by the number of seconds in a day

So on a 4 disk raid 10 that would be half that per disk right?

Sam

@Dan I would say that was more the companys problem for not have a backup solution in place. If someone called me while on vacation (especially paternity leave) with anything but an emergency or a very quick answer, i'd be annoyed (I'd prob help, but I'd be annoyed).

Chris S

@Dan Wow, I sort of understand if they're calling every 5 minutes... But I don't think I'd pull that.

Transcript for