The Comms Room

Basically it's not a HP or Dell thing, it's a SAS and ESX thing, SAS is very clever etc. but is still a single SCSI bus, and all SCSI buses can have a 'pc' at each end so long as they manually arbitrate all non-read commands. That's how you can make a two way MS cluster just about work over this kind of thing. ESX doesn't support it at all, it supports SAS on a one-to-one basis but not for sharing - whether that's HP/Dell etc.

so the san compability guide i linked to is incorrect? or am I reading it wrong?

and 1Gb should hurt your brain, it's ok for lab/dev/test environments or if all of your VMs are very low IO - but if you want to do decent performance and vMotion you need either 4/8/10Gb FC or 10Gb iSCSI/NFS

@pauska which line are you referring to?

vmware.com/resources/compatibility/pdf/vi_san_guide.pdf

bottom of page 5, continuing on page 6

"For SAS Arrays, if listed, VMware supports the following configuration, unless footnoted otherwise:"

* LUN sharing - The ability of multiple ESX hosts to share the same LUN.
* Multipathing - The ability of ESX hosts to handle multiple paths to the same storage device

and if you scroll down to the p2000 g3 sas at page 1303

it doesnt have any footnotes

I see that older msa's do have footnotes, where its explained that you need a new firmware to have vmfs sharing working

Ok, well it states it works there - so maybe it does, I'm happy to be wrong, but I really thought lun sharing wasn't supported - oh well, off you go :)

6:13 PM

communities.vmware.com/message/1710901

guess I'll have to trust them and not you this time Phil :D

not an issue

6:40 PM

btw, did you see what I wrote about the 3750-x the other day?

I've never been so confused about anything.. blamed my ISP for being incompetent and not giving enough bandwith

turned out that the routing table was full.. had to switch over to the sdm template for routing :(

oh really? how big was it?

SAS SAN? ick.

I know, but apparently it will work

I've seen SAS virtual switches, those look somewhat interesting. Take a large pool of drives and assign them to hosts via the switch. I don't think they do concurrent access though.

@chopper3: over 2k E2 ospf entries

the routing template supports 8k

6:43 PM

HP do SAS blade switches but they're not for me

The virtual bus architectures are starting to take over, seems like the next generation will be unified communications/storage devices with inherent virtualization

@ChrisS: Trust me, I'm aiming for something similar to UCS in about 2 years

yes, yada yada I should have a 8gbit FC or 10GigE FCoE san etc

the thing is that you sometimes have to do wonders with less money

We have a pretty large 1gbit iscsi emc san, but its running out of support in 1,5 years

Ethernet won the physical standards race... FCoE and various MPIoE are already available.

We've had management and KVMoIP for a while now...

so this is just band-aid until then.. I'll just re-use this SAS thingy for backup to disk or something like that

Next step is just unifying them, allowing multiple discrete interfaces to bond/team at the hardware/firmware level, and provide virtual interfaces for the overlying OS.

6:47 PM

isnt that what Cisco is already doing?

the VIC card

Some level of that is already occurring in various Cisco and HP products... not terribly familiar with them yet, and I think most of them are in their infancy.

can you eloborate?

my brain is bleeding from reading about this strange SAS SAN...

cause I got the impression that Cisco UCS was full blown unified communications

I know some companies are already touting converged adapters, but most of them only do storage and communications, not management/kvm/monitoring/etc. Also I don't know of any PCIe cards where I can stick two of them in a server and they will appear to the OS as a single one (yet).

I know Myrinet has adapters that do MPI, Ethernet, and maybe storage (memory is getting fuzzy); all in a single PCIe card - Don't know of any Cisco or HP products that do MPI yet.

NUMA would be an acceptable replacement for MPI I suppose, but still don't know of any systems that incorporate that either.

Nyuszika7H

6:52 PM

Hi! :)

in The Tavern (General) on Meta Stack Overflow Chat, 2 mins ago, by Nyuszika7H

SELECT * FROM Users WHERE Site = 'Stack Overflow' OR 'Meta Stack Overflow' AND Reputation ON 'Meta Stack Overflow' > Reputation ON 'Stack Overflow' AND DisplayName ON 'Stack Overflow' = DisplayName ON 'Meta Stack Overflow'

in The Tavern (General) on Meta Stack Overflow Chat, 2 mins ago, by Popular Demand

@Nyuszika7H returns me.

This was true until I edited my message (added AND DisplayName ON 'Stack Overflow' = DisplayName ON 'Meta Stack Overflow') ;)

Hi. I'm not sure why you're asking the serverfault room about this?

yea is that even a question?

@pauska x3

@ChrisS sorry, what type of pcie card?

@Chopper3 Two network cards, or anything along those lines.

6:56 PM

@ChrisS I think NUMA has been used in VMware ESX since version 2.0 or something like that

all of HP's NICs let you do that but only with their SW installed

@pauska But finding a server/blade system that supports NUMA is still very rare. If the hypervisors can take better advantage of it, I think it'll catch on more.

aren't all dual cpu =>nehalem-based boxes NUMA?

@Chopper3 Yeah, to the OS you still get multiple devices. I mean something where the two NICs talk to each other themselves, figure out redundancy and such, and report a single NIC back to the OS (no special software or drivers)

ah, balls yeah

6:59 PM

@Chopper3 sorry just got back from lunch

I think you guys might be confused on what NUMA is... It allows multiple computers that would normally be independent to cluster into a single system, similar to many supercomputer architectures.

OK. I don't need that then

All the implementations I've seen of this require very specialized hardware, usually designed into the mobo.

thought you meant this NUMA; en.wikipedia.org/wiki/Non-Uniform_Memory_Access

I thought the same as @Chopper3

7:01 PM

@Chopper3 wait yes, that was my understanding also

given that the error message says to check the memory

@ChrisS: Erm, NUMA is the article Chopper3 has linked to

its how Cisco can deliver those insane blades with 1TB ram or whatever they have

1 hour ago, by Josh

The system has found a problem on your machine and cannot continue.

The BIOS reports that NUMA node 1 has no memory. This problem is either caused
by a bad BIOS or a very unbalanced distribution of memory modules.

The latest NUMA research I was reading a few months ago concerned 'clusters' where the 'nodes' could come online and offline live, as well as fault without crashing the entire cluster.

@Chopper3 Perhaps I've got the wrong acronym... it's happened before.

@pauska That's Scalable Memory Interface (SMI) that lets you have lots of memory, and it's not Cisco specific

@ChrisS ah ok, no probs :)

@Chopper3 Really? I read that its their Catalina chip in conjunction with intels memory controller that enables it (the switch between banks)

more specifically:

rodos.haywood.org/2009/06/nehalem-memory-with-catalina.html

7:05 PM

@Josh to be fair it can be quite hard to configure memory optimally on these new 55xx/56xx series xeon systems, in particular the 75xx series can be a swine - I actually find it easier to buy the right number and type of memory for those to ensure that whoever installs it for me is more likely to do it instinctively right than to actually document it for them

@Chopper3 thanks, that's good info. I finally was able to open the BIOS (using a KVM over IP) so I am going to try and disable NUMA

@pauska whether it's the same thing or no is moot really, they do the same thing

@Josh NOOOOOOOO!!!!

@Chopper3 OK. I am not then

:-)

@Josh don't DO that, the system needs that, just configure it right - let me know the EXACT model of server you've got, how many of what CPUs and your planned memory layout and I'll tell you how to lay them out - might take me 20-30 mins though

Ok. thank you very much

7:08 PM

@Chopper3 Huh.. I've never even heard about it before.. trying to google it, and getting all kinds of weird results (with an intel patent being at the top)

Let me see what I can gather; I will post a true question for this

It's a dedicated box at my datcenter

Ah, was reading about SSI Clusters implementing DSM via ccNUMA techniques/hardware. Well I had some of it right. =)

@pauska ok, well it's an Intel tech for de/re-synching memory timings to allow for more ranks per channel

WesleyDavid

Reading about how Pinboard survived the surge of traffic from The Great Delicious Exodus. pinboard.in/blog/173 I was wondering how it managed to handle 50 hits per second, and he mentioned that Apache had been tuned due to a crushing a few years earlier. Oh, then he mentions "Oh yeah, I have a 16GB main webserver and two MySQL backends, one with 32GB and another with 4GB." YA THINK that helped?!

@ChrisS :) I'll have a read sometime

7:11 PM

@Chopper3: Is that the same as the catalina chip? I'm trying hard to understand this, hope you're not taking offense by me questioning you all the time :)

from what I can understand from the catalina chip its a 5ns ASIC switch

sales drones are really pushing the palo alto firewall thingy hard at the moment.

god, I have to go into another room.. god damn girlfriends with their nail polish remover...

jscott

@pauska Maybe if you only had one girlfriend, it wouldn't bother you so much.

Holocryptic

"girlfriends"? Isn't that a bit masochistic?

@pauska not at all, love you guys, functionally it is, don't know that catalina but it "just lets you have more than 3 dimms per channel", which is what SMI does too

@RobertMoir I can't remember the last work-day I didn't tell a sales drone to go fuck themselves, it's second nature now

7:14 PM

well i try to be civil about it but they get fairly short shrift from me too.

@Chopper3 you must be right, server must require it, because there's no way in the BIOS to disable it ;-) I am still gathering information and will post a question as soon as I have finished

if i want to buy your product, I'll call you. See how this works salesdude

WesleyDavid

Multithreaded girlfriends are awesome!

@Josh it's a major architectural design aspect of your server :)

@RobertMoir put him on to me, I'll tell him for both of us :)

hahaha, sorry, it's a new acronym for me :-)

7:16 PM

don't suppose anyone's going to HP's Tech@Work in Madrid this year are they?

@Josh don't worry, nobody's born knowing any of this stuff, if you can get me that detail I'll help if I can :)

i had one call on monday that told me that if i didn't agree to at least evaluate the product he would call my boss and go over my head. He was told to cram his product up his ass with walnuts and then fuck off.

Thank you again. I am going to ask the datacenter for more details like what RAM is installed and where

@RobertMoir :), edit - who's this again?

my boss won't even dream of buying stuff in the area i'm responsible for without my agreement, any more than i'd go over his head and order a bunch of shit outside of my area, so this guy's lost a sale for life

some sharepoint admin tools company

@Josh great, feel free to email me directly if I'm not about ok

7:19 PM

Got that, thanks, in case you want to delete

@RobertMoir what are they called so I can make a note to never deal with them ?

trying to remember, might have been deliverpoint

k

lightningtools?

i think so, not 100% sure. I was too angry to remember who they were, tho i know our dev made a note - the idiots rang him too

:)

7:24 PM

he had a big struggle keeping a straight face before asking the guy if he'd just been speaking to me then telling him get lost

@RobertMoir "If the person responsible for approving your software for purchase says no, call everyone else at the company" -- Because that won't get you put in the "Blackballed Vendors" pile... no... not at all...

just found out that they're trying to work with a mate of mine :)

@voretaq7 precisely

You are however fortunate that your boss will simply tell them to get stuffed.

Mine would bother me for a week with every false promise the vendor offered, and I would be forced to refute them point by point

i appreciate they have to do a job and that a job in sales is probably tough but there's ways of doing it that are good and ways that are bad

7:27 PM

almost certainly a Richard Young

I am - he's learnt over the years that if he buys something i don't want to use he'll either have to install, configure and fit it himself or it will remain on a shelf unused

Apparently having done this whole IT thing since I was 16 and taking their infrastructure from "house of cards" to "No unscheduled downtime in 2 years" doesn't quite prove I know what I'm doing...

@RobertMoir Oh they know that much -- I should rephrase: The CTO (my technical boss) lets me do whatever I feel is best. Our CEO likes to be hands-on, which is where the problem comes in.

(small company, lots of dotted/diagonal lines on the org chart and all that)

Jacob

Hello guys

Crap.

"Scrap" as they say on Transformers.

I'm thinking the PA budgeting thing is worse and worse now that I get multiple stories coming in about it. Never a good sign.

@RobertMoir The best "professional" salespeople will sit down, listen to what you need, then (a) Sell you a product that does exactly that, (b) Explain some items you didn't consider & sell you a product that does more & suits you better, or (c) Tell you they have no product that meets your needs and gracefully exit.

In my entire career I think I've met... 5 of those.

7:32 PM

Our IU (kind of like a support agency, for outsiders to our happy system here) has had most of their budget cut. Now I get an email from the university (college) my stepdaughter is attending telling us that 110 million is cut from support to the parent college.

Q: How can I correct the NUMA setup of the memory on my server?

0

I am trying to install VMware ESXi onto a new dedicated server. However, when I boot from the VMware ESXi Installer CD, I am given the following error: The system has found a problem on your machine and cannot continue. The BIOS reports that NUMA node 1 has no memory. This problem is e...

Which means higher tuition...no matter what they're saying about trying to cut costs...and soon, more mandates that won't be funded. Bloody @#%

I am on the line with them now as they look up additional information

And most likely people will soon be losing jobs.

I have some more screen shots of the BIOS if that helps

7:34 PM

@chopper3 - certainly can't remember the guy's name. I want to say that sounds about right but i'm honestly not sure.

@voretaq7 - we have a few like that on our list, both our admin person for day to day stuff and myself for network infrastructure have people like that and we try hard to put as much work their way as possible. They do very well out of the "partner" approach with us.

@Josh might be helpful -- also the type of server (make/model) & where your RAM modules are installed (which banks/rows)

@Josh so it has 4 physical CPUs right? and how is that 24GB configured? 12 x 2, 8 x 4 etc?

@RobertMoir our colo (NYI) and the company I buy monitoring software from (Dartware/InterMapper) are like that. Also the InterMapper folks listen when you whine at them for new features :-)

gah nevermind missed the motherboard type in your Q

@voretaq7 i'm lucky i guess, i work in a college and the principal is interested in IT only in so far as making sure we're doing things that benefit lecturers and students with it, and aside from that he doesn't care how it works as long as it does

our current Dell TAM and internal sales contact are like that, really make the effort to understand what we're doing and so-on instead of just selling us stuff

@Chopper3 asking for more specs. It's apparently custom hardware, not a name brand like Dell etc :-(

@Chopper3 4 physical CPUs, yes

7:39 PM

they've pretty much single handed stopped us (and about a dozen schools who copy everything we do) switching to HP over the years those two

cool, see my answer

just by giving us great service

@RobertMoir yeah, our account manager at iron systems is good like that too -- I could get slightly cheaper prices going to SuperMicro direct but he's been useful in the past & the rare support incident has always been resolved quickly, so we'll pay an extra $1-200 per box

yep, i don't mind paying for good service if i can see where the money went

met Sir Alec Jeffreys today

on a different note

that was nice

I know where the money goes.

<flush>...swirlies!

7:43 PM

the inventor (or is it discoverer) of DNA fingerprinting and profiling

heh

Ok. Thanks very much @Chopper3. They said they doubt it can support 16 DIMMs, they don't think it has that many slots. They need to open the box to be sure but they believe it has 4x 6GB DIMMs in there now

I'm reading that document you linked to

@Josh ok, well it's not the model you mentioned in the question then

but to be honest I don't know any 4 socket systems that CAN'T take 16 DIMMs - that's only 2 per channel

Argh, I hate not being able to pull it off the rack and loot at it myself

I also think this guy I was talking to didn't know what he was saying :-)

@Josh Quite possible :-)

at the risk of sounding like an AOLer /me too-ing, I have to admit that every 4 processor socket machine I've seen has had at least 16 memory slots

usually more

7:46 PM

:)

If I have two physical hosts running several VMs using DAS, what options are there (cheap or free) for getting HA (or near HA) without adding shared storage into the mix?

always amazes me how much they can cram into 4 socket blades now

what hypervisor

@Chopper3 you're right, the document you linked to clearly states it has 32 DIMM slots!

And I have a screenshot from the BIOS showing it's an H8QG6

@Josh which means either the guy in your DC is confused, or doesn't know what model of motherboard is in the box (or both)

7:47 PM

i can't help thinking HA without shared storage is crazy talk :-\

@Chopper3 intentionally omitted

@hobodave If SAS SAN counts as DAS then ESX will do it.. Just talking with @Josh about that earlier.

Now I have enough to send thema very detailed support ticket. THANK YOU

@RobertMoir 32 on the AMD based BL685c G7 and 64 on the Xeon based BL680c G7

well a SAN just isn't in our budget

7:48 PM

yep, same on the dell poweredge m910s that we just got, mindblowing really when you stop and think about it

512Gb in a blade...

@hobodave so what are the details - come on, spec me up!

the BL680c G7 lets you have 1TB! and the single-slot dual-CPU BL460c G7 does 384GB :)

You can always run two hypervisor systems and appropriate HA software between the virtual machines (or in a load balancer) -- basically treat the VMs like physical hardware. It's kinda crude, and probably more work than HA at the hypervisor level, but it should work.
Anything better than that gets into what kind of hypervisor you want to use tho :)

well they're not set in stone, but basically we're looking at getting 2 x PowerEdge R810's with dual X5670's and 32GB of RAM each, storage is still undecided. Not sure if we're going to go SATA or SAS

and I'm still torn between hypervisors, you guys praise the hell out of VMWare, but it just gets stupid expensive fast

so are you after a DAS box linked between hosts or did you mean each host having its own DAS - also what hypervisor are you looking at

each with its own DAS

7:52 PM

it's free if you don't want the HA features

right

I guess I'm wondering what is done to protect against the loss of an entire host

without dropping $20k in software

and the expensive stuff is expensive for a reason... its the gold standard.

I'm not aware of a solution that'll let you use local storage but offer HA features

I realise thats no help if you don't need or can't afford it, but i just mean to say they're not charging that price for funzies

what type of guest VMs are you talking about?

7:53 PM

You could use one the hacky backup solutions

RHEV offers most of VMWares features for much much much less :P @RobertMoir

@RobertMoir also they drop their trousers if you're after quite a few licences too

There are a couple that will take a snapshot, then download the vmdks to somewhere else.

true enough chopper

Chopper3 I'm concerned with the availability of the web server guests

apache, memcached VMs

7:54 PM

@hobodave - a skoda offers the same number of wheels as a ferrari and an aston martin too. Doesn't mean its in the same league as them though

so why not have two or more, one or more per host with a VIP in front

right, what voretaq suggested

I was only focussing on my posts, as usual :)

that was the only idea I had come up with, I was just wondering if that is what I was limited to

what about VMs that aren't "critical" e.g. a monitoring VM that I wouldn't want two running at the same time. What options are there to keep it at least cold swappable on the secondary machine?

@hobodave there are other options but I think all of them involve shared storage.

7:57 PM

is something like DRBD poorly suited for that?

we have one ESX web server farm with over 400 identical VMs, although they do have proper ESX licences their actual normal failover mode is via loadbalancers - it works great

backup, and restore on the other host

@hobodave If you're running Linux (IIRC you are?) you can use heartbeat/HADaemon and DRBD if you need to clone storage (or rsync / backup&restore if they're relatively static)

yes Linux

Our monitoring server rsyncs its critical stuff to a cold standby box every night -- it's "good enough" for us :)

7:58 PM

@hobodave backup host 1's VMs to a central NAS or similar, restore them to host 2 but to a disconnected vSwitch so it can't actually talk, do the opposite (2 to 1) then if 1 goes down just enable the vSwitch

@voretaq7 would you suggest rsyncing from within the guest, or rsyncing the vmdisk?

@Chopper3 that's not a bad option, and you could have a canary VM that checks to see if the other host is running.

@hobodave I'd do it within the guest, otherwise you really have to do like @Chopper3 said and do a vm backup/restore - you don't want to pause the VM while the rsync runs

@voretaq7 yeah, I wouldn't know specifically how to do that right now but I'd get there :)

and having your vmdk change while you're syncing could be a Bad Thing(tm)

communities.vmware.com/docs/DOC-8760

8:00 PM

@voretaq7 but that means I'd have to have two monitoring servers run at least while the rsync occurred

@Chopper3 I know you can twiddle VMWare's VSwitches from the perl API, so it's possible -- never seen it done but I'd like to :-)

@voretaq7 if it was ESX/i you could copy from a snapshot with the VM still up

I wish shared storage wasn't so damn expensive

@Chopper3 true -- never tried that in practice though (whereas I have rsync'd guest-to-guest like they're physical boxes, which is why that's what i'd do :-)

@voretaq7 easy - just flip the vswitch's vlan tag from something invalid to valid

8:01 PM

these solutions all sound very complicated

@hobodave not really -- it's just that you need to write some glue software to automate the transition. If you do the switch from A to B manually it's a lot simpler

just thinking of the simple case of the monitoring server

(downside being if you're on vacation in Tahiti and Host A takes a shit you now have to either walk someone through the manual bits of the failover or find a machine with internet access - that's where documentation comes in to play tho :-)

@voretaq7 esxcfg-vswitch -v <VLAN> vSwitch0

I'd have to have the VM setup on the second host, but not capable of actually probing anything because I wouldn't want double monitoring. I'd need to automatically start the VM prior to doing the rsync, then I'd rsync everything over and shut it down.

8:04 PM

@Chopper3 yup that's the manual part pretty much :)

And because its configured to not monitor anything, that means that's another step I'd have to take if I ever neded to bring it up for real

I wouldn't use rsync, I'd just snapshot it and copy that to a central NAS, then restore on the other side

@hobodave Run the VM but don't start the monitoring daemon. Much simpler.

How quickly does your system need to boot up?

fuck it, just buy enterprise plus and FT it

8:05 PM

Or do the snapshot trick like @Chopper3 suggests :)

You could create a second VM that also includes the monitoring virtual disks...

that was a joke btw, ent+ still costs me >$3k/cpu :)

That will be the backup box. When a failure happens, remove the backup virtual disk from the dest box, and then boot up the backup vm

@Chopper3 pocket change! (If you have really big pockets :-)

@voretaq7 I am very sure now that the guy in my datacenter is confused. I am looking at pictures of this motherboard online and it clearly supports 32 DIMMs

8:10 PM

shared storage makes everything easier

get him to send you a photo and check the mobo part number

@Chopper3 Good idea. Would the BIOS be wrong?

yes, just focus on finding out what it really is rather than what he says it is

@Chopper3 Cool, thanks

@hobodave ...except closets.

8:12 PM

lol

so I haven't actually looked into a DAS device that would be shared by both servers, is that worth looking into?

I forgot such a thing was possible

@hobodave It would definitely be cheaper than a SAN

downside is it hobbles you (most are SCSI so only 2 boxes can be connected to it), but it will last you a few good years at least

how do you guard against the DAS going down?

How do you guard against the SAN going down?
(You put all your eggs in one basket after building a good basket)

so just rely on RAID 10 and dual power supplies?

my brain tells me get two of everything

dual controllers too remembr

can't you just pick something up second hand and make sure it's on a support contract

8:17 PM

I don't know

I'm asking you guy :P

initially I was looking at Nexenta SAN boxes

pogolinux.com/quotes/editsys?sys_id=124510

but my hosting provider advised against it unless we get two

because if it goes down it takes everything with it

and he suggested using internal storage on two separate VM hosts

HA costs $$$

unless we can afford to go HA with the SAN, which we can't

how many VMs do you NEED to be working after a host failure, in disk terms?

I guess i just don't know when to say "Ok, I have made this expensive single point of failure as resilient as humanly possible without getting two of it"

coredump

@hobodave I am frustrated with DA2

8:20 PM

@Chopper3 minimally? two

and what's your budget?

I hate my boss

that's my budget

she refuses to tell me a number

different problems

its always "cheap cheap cheap"

but the words that come out of her mouth are "Just tell me what the best is"

ok, well get on ebay.com, type in 'HP MSA iscsi' and see how you get on

8:21 PM

which she will then try to approach in a piecemeal fashion

coredump

@hobodave invite her for drink and dinner and cook something nice.

...

which I don't think that approach will work because you can't half-ass your way to HA

So what you have to do is give her two quotes. One with what you want, and a second that looks similar by is insanely overpriced.

the problem is I don't know what I want

She will go with the cheap option that is what you actually want.

8:22 PM

and I don't know what is best

I thought going with a single SAN that had dual controllers was a reasonable thing, but my hosting provider shot it down

so that leaves internal storage and what sounds like ghetto, high maintenance, and error prone HA workarounds

Your hosting providers may be staffed with idiots...

I consider him very knowledgeable, he explained his reasoning

if the SAN goes down then I lose every VM on both Hosts. Hard to argue with that

We run that setup

1 SAN dual controllers, all VM's on there

So buy a good SAN, and make sure you have a good support contract.

Yes if it goes down it sucks, but how often is that and what is the replacement / repair agrreement?

8:25 PM

mission critical @ITHedgeHog ?

your hosting provider is being very limiting, their "SAN" is nexentastor right? which has its problems, can't you just pick up a second-hand HP/Dell/IBM iSCSI or FC box? it'll be like $2k or something but it'll work

We can live with 4 hours down

@ITHedgeHog and I have no idea how often that happens

@Chopper3 they were just suggesting Nexenta

@ITHedgeHog I run fricking LOADS like that, like thousands

:)

You can have parts on site

8:26 PM

@hobodave but were they 'poo-poo'ing other options too?

WE've had 1 SAN outage in 4 years

That was because some retard yanked the power

@Chopper3 No. I told him I had looked at Sun Storage and EqualLogic and it was way too pricey, he suggested Nexenta from there.

I've dealt with probably 200-300 separate SAN arrays from loads of vendors and never had a total loss of service on any of them, lost single links/controller etc but never whole service

how much was too pricy?

Sun Storage started at $16K

EqualLogic starts at $10K with crap everything and no HA

you can build a quality Nexenta box for $8-$10k

and that includes the Enterprise license ($2K)

we could use the CLI version for free

it depends on the size of the VMs but there's nothing wrong with something like a HP P2000 G3 FC with a pair of mirrored 600GB 2.5 SAS disks - that'll be fine, you can expand as you need - that's no where near that money

8:30 PM

isn't FC expensive?

@Chopper3 this thing? buy.com/prod/…

that's the iscsi version but that would work just fine

search for AP847A for a full kit

is FC overkill if we wouldn't even saturate a 1 Gbps line?

@hobodave: I would avoid Sun storage at ALL costs

sure, if 1Gb is fine go for that

@KyleBrandt why? I've heard stellar things about it.

8:36 PM

Had a tape device with them, after the Oracle take over they essentially just dropped a support contract with them at our last job

Don MacAskill of SmugMug has several very detailed blog posts about how amazing it is

So really I don't recommend Oracle storage devices :-P

Oracle seems to be maintaining the storage line.

but Nexenta is basically the same, but much much cheaper

We purchased a Sun right before the merger was announced. They have still be releasing updates.

8:38 PM

we looked at sun kit for streaming servers, they spec'ed out a specific machine/config, it was the one with the ~24 internal disks mounted vertically, tested it and it max'ed out at about 60% of what we were getting with untuned out of the box PC kit with Windows 2003 ffs!

@Zoredache: Well they didn't with me. They had me fill out forms and send them to black hole emails. We had to have someone come and re certify it, after they did that they said the person never came -- Need to get the access logs from the colo to prove it to them. Whole thing went on for several months

There was even a month in the middle where the support managers were NOT ALLOWED to offer any support during the period

@Zoredache are you using ZFS?

Ah, we haven't had any hardware issues yet. Everything so far has been software, which they seem to be updating.

Well my experience before the takeover was fine, so that was just not handled well. Can't speak to how it is now a year or so later

Yes, I ZFS is the only option. We have a 7210 oracle.com/us/products/servers-storage/storage/disk-storage/…

8:45 PM

so you guys would agree that a single SAN with redundant controllers is a reasonable risk to take?

Generally, I don't believe you have told us what service you are providing.

What kind of monetary loss will you incur if you have a 8-24 hour outage?

It's our core business. We're a hosted web application servicing hospitals

our SLA doesn't actually have provisions for monetary compensation, but 8-24 hours would be a very very bad day

we probably hit three 9's a year in uptime

@hobodave medical field application without funding for HA; isn't that an oxymoron or something?

hrm... If it is for medical stuff, I would be tempted to tell your bosses they need to cough up some cash...

its not for medical stuff

its for every other piece of keeping a hospital running, facilities personnel, housekeeping, patient relations

nothing directly involving patient health

8:51 PM

Ah, a single SAN may be adequate then, so long as you have a system in place to make backups.

I'd agree that a single SAN with dual all-the-normal-stuff would easily get you 3 nines with a 4-hour 13x5 or 24x7 warranty.

OK @Chopper3, got an answer back:

> The motherboard that is in there is the Asus KGPE-D16

> It is currently configured with 4 banks of 3 sticks to make the 24gb of Ram.

The only single points of failure in our SAN are backplanes, as arrays are not spread across enclosures (design decision). But the chances of that part going bad are extremely low and we've got 4 hour 24x7 warranties on them.

damn Nexenta only gets as good as 9x5 NBD

@Josh firstly - the dude on site is a dick, secondly that's only a 2 socket mobo???

8:58 PM

wait a minute

wtf?

my brain is cooked... been "refactoring" a perl script all afternoon.
I'd start ranting about the author's crappy programming, but that would be me.

@Chopper3 yeah... seriously

Ok I'm pissed now. Sorry for wasting your time @Chopper3 I should have followed that link before I posted it LOL

's'ok

goes to bring the pain