The Comms Room

ewwhite

8:00 PM

@Chopper3 They want clustered storage for 25TB of data...

I think that's still in proper SAN territory.

but @JoelESalas wants Gluster, I believe.

so they want to do it with a bunch of storage nodes... likely DL380p Gen8 25-bay units

Joel E Salas

@ewwhite We're trying to achieve an economical equivalent, with the understanding that we'll be lacking some nice features

Chopper3

@ewwhite I'd got for that DL380p with 24x1.2Tb 2.5" 10k's in R60 with an 800GB MLC for Smartcache myself

@ewwhite or just go for a bog-standard 380p with two small boot disks, a P822 and an MDS 6000

Joel E Salas

@Chopper3 We're going for more self-contained boxes vs adding shelves (allegedly Gluster does better this way)

Chopper3

@JoelESalas that SL can only have 192GB though

ewwhite

@JoelESalas right... 25TB is still in SAN territory. You don't gain much by using a clustered filesystem

Joel E Salas

8:04 PM

@Chopper3 I'm deferring to expertise here, whatever you guys recommend I'll take it up

@ewwhite What about the cost?

also we're trying to stick to the OpenStack model as closely as possible

Chopper3

right - I've been TOLD to go and pack, hard-stare off wifey :(

have a great christmas guys, see you in January

Joel E Salas

@Chopper3 Happy holidays!

ewwhite

@JoelESalas A proper EMC VNX with that capacity would be $115k. A Nexenta dual-head box with that capacity running ZFS would be 50k. A Single head Linux box with your filesystems of choice would be $20k

Iain

@Chopper3 have a great time

ewwhite

I don't know where Isilon would be price wise

Joel E Salas

8:06 PM

@ewwhite so we could get 5x storage nodes?

"in theory":

Matt Bear

grrr... do I need to reboot a server to get snmp to work, or am I missing something?

ewwhite

And the DL380p boxes @Chopper3 is talking about are $17k

but all cluster filesystems kinda suck

I'm trying to help you guys avoid the mistake my DevOps client at LW made

jscott

@MattBear Windows? No. Just make sure the SNMP service is enabled and running.

Matt Bear

ubuntu

jscott

@MattBear service snmpd start I think

Matt Bear

8:07 PM

followed it-slav.net/blogs/2009/02/05/… exactly

I've done the same exact thing on other servers, and I couldn't get it to work until after a reboot

Joel E Salas

@ewwhite Was it a similar idea? OpenStack and the like?

Matt Bear

I need to get these monitoring, but I cant take them down right now

jscott

@MattBear I certainly didn't reboot any of these Debian boxes, and they're all monitored via SNMP.

Matt Bear

@jscott am I missing something then?

ewwhite

@JoelESalas They were RHEL6 boxes with KVM atop them... running CentOS 5 VMs. Everything over NFS back to an EMC VNX with ~100 disks providing DB and VM storage.

Mathias R. Jessen

8:10 PM

Noooooooo, I forgot to order a CPU cooler Dx

84104

@MattBear On Linux the only things the requires reboot are changing your init and changing your kernel.

Joel E Salas

@ewwhite So what was the mistake?

jscott

@MattBear Could be? You've verified snmpd is running? Maybe try restarting the service then snmpwalk it again.

ewwhite

@JoelESalas KVM, NFS, bad SAN design...

84104

@MathiasR.Jessen What's the worst that could happen?

ewwhite

8:10 PM

they ended up trying Ceph

then Gluster

84104

Kidding. Don't do that!

ewwhite

all bad...

and it was because nobody wanted to pay attention to the I/O patterns

Joel E Salas

@ewwhite We wouldn't be using NFS, we'd be creating RO images

jscott

Also check your iptables to make sure it's not blocking.

Joel E Salas

pulling from gluster using the native client

pauska

8:11 PM

@MattBear remember that snmp just silently ignores you if you're using the wrong community string (or authentication if using v3)

ewwhite

@JoelESalas I don't understand what Gluster is buying you over real storage.

MikeyB

@ewwhite except GPFS

ewwhite

@pauska Can you help?

pauska

@ewwhite with what

Mathias R. Jessen

@84104 If I'm lucky I can get my hands on a proper cooling unit by monday, if not I'll have to wait for the 27th :( Christmas is ruined

ewwhite

8:12 PM

@pauska @JoelESalas and his firm need 25TB of usable storage to back a virtualization cluster (KVM) and for logging and other miscellaneous storage. They already have a 10GbE infrastructure.

25TB is still in SAN territory...

but they want to use GlusterFS and cluster across basic storage nodes.

@MikeyB well, what's your thought?

MikeyB

@ewwhite GlusterFS is NOT GOOD for this access pattern.

Joel E Salas

We're totally open to throwing out the idea if it's dumb

84104

@MathiasR.Jessen All I want for Christmas is a CPU heatsink. ♪

Joel E Salas

@MikeyB Not good for read only images?

Matt Bear

@pauska hah... I can snmpwalk the internal ip but not the external...

MikeyB

8:13 PM

@ewwhite However, using GPFS, this works awesome.

pauska

I really don't have any experience with clustered filesystems other than VMFS/CSV..

MikeyB

@JoelESalas Ah! Read-only, that's different. GlusterFS is good for write-once read-many.

pauska

@MattBear there you go - check the config. maybe it's only listening to localhost.

Joel E Salas

@MikeyB That would be our design pattern

ewwhite

@pauska But you DO know that an Isilon or a VNX can do this.

84104

8:14 PM

@MattBear Firewall?

pauska

@ewwhite Yes, but those have a lot of features that he probably doesn't need (when I read the chat log here)

MikeyB

@ewwhite @JoelESalas shared storage or no? (I mean from the perspective of the storage server. Local disks only or does each storage host see the same disks?)

Matt Bear

it runs on 161 doesn't it?

Joel E Salas

@MikeyB It would be shared storage to all of the hosts, yes

so fault tolerance is really important

Matt Bear

ohhhh snap

Joel E Salas

8:15 PM

@MattBear TCP and UDP

Matt Bear

its UDP isn't it

ewwhite

@JoelESalas Which is why people purchase SAN storage...

Matt Bear

haha im a tard

pauska

@JoelESalas what kind of throughput/availability are we talking about here? Do you have any IOPS budget?

84104

@MattBear Usually 161/UDP, but can also be 161/TCP.

Matt Bear

8:16 PM

its UDP

ewwhite

They don't have IOPS figures for me... or throughput

Matt Bear

the monitor runs over UDP, it took a call in to them awhile back to figure that out, and I forgot lol

pauska

Everything is read only?

ewwhite

so whatever is fast enough... 900GB SAS disks throughout and caching where applicable. I don't have an idea of working set size.

@pauska no, because logging will go here, too... e.g. SPlunk

so to me, the question is: what's the best way to present 25TB of usable high-available storage?

pauska

@ewwhite how many GB per day in splunk?

Joel E Salas

8:18 PM

@pauska like 50+GB

ewwhite

@pauska @JoelESalas would know. A lot.

Joel E Salas

We need to get Akamai logs in that son of a bitch

and varnish

those constitute probably billions of events

pauska

OK..

Splunk is extremely I/O heavy

Joel E Salas

@pauska We are not doing splunk. With that said, we know ElasticSearch or anything else is also quite heavy

pauska

hrm. this is not an easy one, since you really have no idea what the usage pattern will be like

ewwhite

8:20 PM

@JoelESalas Splunk was mentioned

pauska

how much of that 25TB of data are only written once? You mentioned something about read only

Joel E Salas

@ewwhite We happen to have splunk, splunk is not the long term solution

@pauska I'm hoping to create a library of "gold" images that we create instances off of

pauska

images of what? I'm not sure what the business model is

ewwhite

@pauska they're trying to use KVM virtualization, and boot off of read-only images... kinda like Amazon AMI's

Linux web server instances

pauska

aha

ewwhite

8:21 PM

@pauska but then that usage pattern is like a VDI setup.

@MikeyB ??

pauska

Okay, so that brings up two obvious points.. a lot of read cache and maybe deduplication

ewwhite

tons of read cache because we don't know the working data set size.

pauska

I would take a look at the VNX2.. it has block dedup and is much much faster than the VNX(1)

and can easily scale to 25TB and beyond

ewwhite

Need a disk pool that can handle the sustained write/read IOPS, especially if the data doesn't cache well

pauska

FastCache helps tremendously on VDI based workloads

MikeyB

8:23 PM

@ewwhite I'm hearing two different things here.

ewwhite

@MikeyB right, two or three applications for the storage... no metrics on any of it.

pauska

VM storage (especially in a reseller environment) is very, very, very random and requires fast disks

ewwhite

they want clustered filesystem for resiliency... but the data scale doesn't warrant it. Seems a SAN would be right

pauska

or is this internal IT?

ewwhite

internal IT

for a website geared towards pretty people

pauska

8:24 PM

ok so tiering must be involved aswell

MikeyB

@JoelESalas NAS or block storage?

pauska

(pin stuff to highest tier etc)

ewwhite

@MikeyB NAS... for KVM

I can tell they don't want to deal with block

DevOps hate block storage

MikeyB

@ewwhite can also do iSCSI for KVM directly. Or, since they're read-only, mount the iSCSI image volume read-only from multiple here.

ewwhite

they already have 10GbE infra... NFS seems simpler

but I disagree with the KVM, too ;) That won't change, though.

MikeyB

8:27 PM

@JoelESalas Xyratex or if you want to roll-your-own, GPFS.

ewwhite

There's a budget.

but it's BIG!

Given all of this, would some Linux storage servers clustered together be a better idea?

or is this ZFS territory?

Joel E Salas

@MikeyB Object storage

we do have a block storage requirement but we're planning on using Cinder for that

wiki.openstack.org/wiki/Cinder

We don't want to back Cinder with Gluster. We'd probably back it with NetApp or Isilon

but that's not in the scope of what we're doing right now, I don't think.

ewwhite

I dunno man... there's a lot of stuff that can do what you want. I'm trying to understand the OpenStack connection

Joel E Salas

@ewwhite What would you do instead with respect to KVM?

ewwhite

@JoelESalas You have the budget for VMware... you have VMware... but if you're set on OpenStack, I'd do it the Red Hat way, since they're backing it.

Joel E Salas

8:32 PM

@ewwhite The VMware stuff is specifically for DWH and WMS stuff that we can't get support for any other way. We have absolutely no interest in expanding the VMware deployment.

ewwhite

@JoelESalas why?

Joel E Salas

@ewwhite Cost, and general interest in FOSS tools

ewwhite

I mean, OpenStack is the right option, it seems...

Joel E Salas

We want to be cool like Etsy

ewwhite

But FOSS comes at a cost.

and you're not Etsy!

pauska

8:33 PM

I'd just be careful about any kind of DYI solution for such a large setup

2

Joel E Salas

Not yet, you mean!

@pauska I think we can make the right decision if we understand the trade-offs

ewwhite

@pauska Exactly... this is big gear.

Joel E Salas

Which is why I asked @ewwhite to chime in

ewwhite

and you have a big budget.

so your DB servers are well-spec'd.

Anonymous

oh so your next startup is uber for instragrammers?

ewwhite

8:34 PM

the storage is kinda tricky because I don't have much to work off of

Joel E Salas

@PatoSáinz Uber is already uber for instagrammers

ewwhite

what I would do is get an HA NexentaStor setup.

Anonymous

@JoelESalas elaborate

pauska

I think you should have a serious talk with a independent storage partner.. someone who has partner status with EMC, NetApp and so on

Joel E Salas

@ewwhite They've got FIOs and enough RAM, I don't know that we need more muscle behind it

@pauska We've had those talks, I just wanted Ed's opinion, and he wants our money so it's a bipartite solution

keep the wealth in the family

ewwhite

8:35 PM

$50k. 25TB. Enough disks to give you the right amount of IOPS. Write and reach caching. If you don't like it, tear it down and make it a Linux box.

pauska

@JoelESalas ok, sorry for sticking my nose in it.. he asked me what I think

Joel E Salas

@ewwhite So two head units, some disk shelves

the write AND read cache SSDs

it does sound like a hot option

ewwhite

but your guy doesn't like ZFS... or had a bad experience with Sun

Joel E Salas

but if we're not going to do fun DFS stuff, we may as well go with boring old isilon

@ewwhite He doesn't want to try ZFS on Linux per se, I don't think he believes it's production-ready

ewwhite

@JoelESalas it's good enough. @MikeyB Your thoughts on ZFS now?

Joel E Salas

8:38 PM

@ewwhite in case you're interested: wiki.openstack.org/wiki/CinderSupportMatrix

ewwhite

I don't fully understand object storage.

Joel E Salas

@ewwhite It's homemade S3

We're going to test OpenStack + Gluster next week

I'll let you know if it's horrendous or not

ewwhite

@JoelESalas right... awaiting that info

but again, for 25TB, a single array can do the job

hell, even add some Fusion-io to it

Joel E Salas

@ewwhite Call David and play him this:

►

kce

@cole - You ever have an issue with NetApp VSC where storage systems show up under Monitoring and Host Configuration but not under Backup and Recovery (SMVI)?

ewwhite

8:49 PM

@JoelESalas nope

You Linux people... all devoppy...

Anonymous

ew windows

ewwhite

Ignoring the lessons of your forefathers!!!

Basil

0

Q: Adblock Chrome update frequency- it's letting some ads through

I use Adblock Plus [edit: originally wrote "pro"] for chrome, and occasionally I've noted that some ads will get through. Every time this has happened, updating the filters manually has fixed the problem, and they're almost always 3 or more days out of date. Is there a way to force updates more f...

I added a bounty to that

my filters update every 3 days, and that means I see a lot of ads

ewwhite

So this happened today in my neighborhood...

Basil

Any mods around to help publicise that question, maybe on the twitters?

ewwhite

8:54 PM

@Basil wait! You're a storage guru... Maybe you can help me

Basil

@ewwhite I'd be happy to try... but what we have in common tends to be the fundamentals :)

ewwhite

@Basil @JoelESalas and his firm need 25TB of usable storage.

Basil

@ewwhite what are they doing with it?

ewwhite

It'll be consumed by Linux

for virtual machines... (not VMware) and for logging.

they are interested in cheaper storage boxes and clustered filesystems, but I think 25TB is well within the realm of a proper SAN

or maybe Isilon, if they have an SSD tier

Basil

does linux VM tech require shared storage?

ewwhite

8:56 PM

@Basil it will... it'll probably need NFS.

none of this will really be block

the databases will be protected by clustering and should have local Fusion-io or equivalent.

Basil

@ewwhite ok, we use FC for KVM here but I don't know if that's just because it's cheap or because it's what they wanted

ewwhite

@Basil the infrastructure is already 10GbE

MIfe

@Basil It doesnt require but supports a few different means to import virtual hdd files.

Basil

ok, so we're talking linux virtual machines with no database IO mixed in with the regular hypervisor IO. 10Gb available, nas preferred?

MIfe

@Basil I read that the wrong way, ignore me ;-). Perhaps you mean "does their linux vm tech require shared storage" :D

ewwhite

8:58 PM

@Basil basically. They may have logging activity to the storage... Splunk

Basil

@MIfe Yeah, I also didn't help by writing that clearly ;)

@ewwhite ok, so I think what's needed broadly is something that provides a solid NFS layer, ideally with writeable snapshots like ZFS/netapp

tiering would help, but since there'll be logs, you need to ensure that those parts don't end up on SSD.

Well, you don't need to, but it'd be a waste.

ewwhite

@Basil They have NetApp... but it's older and they don't want to put more money into it.

Budget is over $100k...

Basil

@ewwhite a new one would be my recommendation, then- they're a lot better than they were 5 years ago. You can do a lot more with a lot less, and there'll be less of a learning curve

ewwhite

but $50k buys you a ZFS-based dual-head filer that can do the job... with the right types of SSD adjusted for working set size.

Basil

that said, I think they'd do well with any tier-1 NAS

ewwhite

9:02 PM

Is Isilon worthwhile here?

Basil

@ewwhite yeah, the way I read ZFS is that it does everything Netapp does but you don't get to call netapp if there's trouble

ewwhite

At last job, I had a client who met this profile and they had Isilon and VNX serving NAS

Basil

@ewwhite I've only heard bad things, except from you. But I don't know because I haven't used it myself. On paper, it fits, and EMC is capable of delivering training, support, and implementation

ewwhite

I mean, it was bad...

but it met the needs.

Basil

I've had nothing but shit support from EMC. And attitude

Mostly shit support

ewwhite

9:07 PM

yeah, that's been the case

Basil

it's been pretty conspicuous- it feels like no matter how hard our salespeople try to fix things, someone above them keeps dropping the hammer and preventing them from escalating. Probably because we forced them to take back some storage 9 years ago

that said, we have like 40 sites using data domain now...

you'd think they'd be a little less cavalier about our business

eventually I'll be in charge and I'll clean house, replace them top to bottom with some other dedupe machine

ewwhite

interesting. Bad support is everywhere, it seems

Basil

got no complaints about Netapp, Brocade, and HDS... those are the ones I use

MikeyB

LATER EVERYONE!

Basil

Even HP was pretty damned good

cheers @MikeyB!

ewwhite

9:10 PM

@JoelESalas How about some Violin Memory?

Joel E Salas

@ewwhite pull.vmem.com/img/home-banner-maestro.png loool it's a static image

ewwhite

@JoelESalas one of my former LW coworkers is there now

all flash... screw testing and measuring shit... just give your company ALL THE IOPS!!

@JoelESalas can you email me your company address and contact info?

Joel E Salas

@ewwhite Call the cell, same number same hood. Let me email you

ewwhite

??

Joel E Salas

9:33 PM

@ewwhite Personal question, did you want the dog or did your wife or both?

cole

@kce Been over a year since I've touched any of that stuff, sorry :(

@MathiasR.Jessen I had mentioned I was looking at it.

ewwhite

@JoelESalas I eventually wanted it

Joel E Salas

@ewwhite I've always thought of the small fluffy dogs as White People Practice Children

84104

@JoelESalas Sometimes replacement.

Joel E Salas

And the bad news is that Maria wants one.

84104

9:35 PM

@JoelESalas You're practicing to raise a white child?

cole

So... my girlfriend made a pretty offensive joke about transwomen. I'm kinda at a loss for words.

Joel E Salas

@84104 Well my girlfriend is russian and doesn't want kids

but she's getting to that maternal instinct phase where she thinks babies are cute

Basil

I don't like small dogs. I tolerate large ones when dealing with their poo is someone else's problem, but small dogs are worse than rats.

Joel E Salas

@cole YOU HAVEN'T DUMPED THAT BITCH YET??

84104

@cole "No."

cole

9:37 PM

@84104 She "didn't mean it that way"

84104

@cole Still no.

cole

I'm like blown away right now

Joel E Salas

@cole Do you care to repeat the joke?

84104

Where is one of those nope gifs?

cole

@JoelESalas it involved a picture and stuff. It was more in context than an actual joke you would tell.

84104

9:38 PM

@JoelESalas Sweet totally fulfilled by racism duty for the day, and it's not even 5 yet.

Joel E Salas

@84104 Wait what

cole

sigh

Joel E Salas

@cole Honestly dude if it hurt your feelings it wasn't okay, no questions asked

cole

@JoelESalas yeah she doesn't seem to get that "I'm sorry" doesn't fix it.

84104

@JoelESalas I assumed your other half was the same race as you because of her name.

Joel E Salas

9:39 PM

@84104 loooooooooooool

This is LA

Also I have a certain lovely rapport with the Soviet bloc girls

They love me for some reason, and I love them back because they're tall and white and pretty and crazy

84104

@JoelESalas Joel "From Russia With Love" Sales.

@JoelESalas I've got no problem with it. Just assumed.

ewwhite

@JoelESalas I need your address information

Joel E Salas

@ewwhite Sent

kce

9:57 PM

@cole - Ah. Thanks anyway.

To the live queue we go!

MIfe

10:33 PM

0

Q: CPU load on some apache threads is >10%, other threads are 0.2%

I have an apache box running a WordPress website. For some reason it seems some apache threads are using most of the CPU (10%+), and the remainder little (0.1~). Apart from high user load, what is the reason the threads are shown like this? Is there any set of apache optimizations to help redu...