The Comms Room - 2011-10-05 (page 1 of 3)

« first day (389 days earlier) ← previous day next day → last day (4591 days later) »

6:50 AM

G'day

1 hour later…

8:01 AM

@BenPilbrow meh !

Ha that guy is a total tool

yeah

Seriously dude, just go outside for a bit. I know it's bright, but you can talk to actual people there and stop worrying about your stupid facebook friends.

8:17 AM

Morning all.

Anyone know anywhere I can buy a storage tray for DIMMs? I can only find stuff on ebay, and (apoplectic swearing) PayPal still won't take my money.

@SmallClanger morning, try maplin or RS ?

8:41 AM

Thank @Iain. No joy there, though I did notice Maplin are still delling 128MB SDRAM, which is quite sweet...

8:54 AM

Oh my god I am such a flid. I just typed logoff into my PowerShell window instead of exit. That forcibly closes shit a lot quicker than I'd like :'(

ouch. Didn't know it did that.

No, me neither. By the time I'd realised what I'd done, everything had pretty much finished closing down. I wonder if Visual Studio autosaves like Word....

Grr, now I'm pissed off. Time for coffee and 5 minutes I think

9:40 AM

g'day

@voretaq7: ₒ∩ₒ

3

Unicode is dirty...

10:21 AM

got our new emc san installed yesterday.. the engineer created a 18TB raid5 and said that it would be ready to use within 5 minutes, but that we could expect slow performance for about 7 hours

its now been 20 hours, and performance is shite.. probably initializing (wich they hide from the gui)

15k for the year :)

10:37 AM

@pauska That's crap. I thought these big sans were supposed to be a bit more open about their initialisation times.

indeed.

the only thing I can see is that my newly created test-vm has been put on S-ATA by the allmight auto-tier

time to force a re-tier!

11:11 AM

mysql> sh databases;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'sh databases' at line 1
mysql> show databases;
+--------------------+
| Database |
+--------------------+

I've been doing too much cisco work lately..

See Jeff's comment to my answer meta.serverfault.com/questions/2120/apache-tags-a-bit-of-a-mess any thoughts ?

indeed, I re-commented

2 hours later…

12:58 PM

user image

:: tumbleweeds ::

Its all quiet in here and stuff.

It's because the fucking blog link replaced the chat link again

you can't type chat.stackexchange.com?

@Holocryptic TYPE!?!?

Typing is for pussies. Unless you're on a server, then clicking is for pussies.

Bart Silverstrim

@Iain In regards to...apache versioning?

1:05 PM

I'm all SIRI up in this piece.

Bart Silverstrim

@MarkM I was just bitching in my head about that :-)

@MarkM quiet you and your PS

Hey folks

howdy

@ITHedgeHog Hows married life? Is it boobilicious?

Yeah, very boobilicious thanks

2

1:08 PM

I do like to hear that

@BartSilverstrim in respect of whether apache2-ssl should remain or as @pauska suggests convert it to apache2 and ssl

New SAN:

user image

Old SAN:

user image

spot the difference..

morning!

So we did a DR test yesterday. * puts on glasses *

@pauska Your second screen shot is slightly larger than your first one.

What do I win?

You win a fail image.. I just dont have the time to find one

Bart Silverstrim

1:10 PM

@iain: Okay. I'm pedantic and aspergian in tendency. But I'd say retag as apache1/apache2 and ssl separately.

the only errors were from other teams!

@Basil I don't even want to think about some of our DR stuff...

don't pay for DR unless you're willing to test whether it'll work, imo

@pauska Your second image is more Christmas-y

DR is admitting defeat. Winners never admit the possibility of defeat, therefor I refuse to do DR testing.

3

Bart Silverstrim

1:12 PM

If every day is some variant of a disaster, do you have to schedule an actual DR drill?

2

Make your own hard modes- unplug cables

@BartSilverstrim I'll wait a bit and see if anyone else has anything to say then retag then - there are only 13

Bart Silverstrim

Then go back to your question and comment to Jeff, "I'm retaggin' yo site, I'm retaggin' yo site bitch! YEAHHHHH!"

what is this I don't even

0

Q: freenas storage in esxi to host vm's?

I have previously set up freenas as vm inside esxi using raw disk access. I was wondering if would be possible/sensible to get freenas to serve iscsi targets to be used for other vm's on the same host. This would require booting up with freenas first and once running other vms could be started. D...

vmware-esx freenas

Bart Silverstrim

Throw in a typo and a "Chug-a-lug, chug-a-lug!" and maybe they'll excuse it when they find you're from the sysadmin site.

1:18 PM

Yo dawg, I heard you like storage so I put some storage in your vms so you can storage while you vm

6

@Basil To eliminate the "chicken and the egg" problem, I'm going to just stuff the chicken inside of an egg and then stuff that inside of another chicken.

hahaha

Bart Silverstrim

Turducken. That is all.

turduckenpigcow?

Bart Silverstrim

Only on Epic Meal Time.

Which should probably be suggested to them...

1:21 PM

@Basil Ok so you're looking at this from a different perspective since you're a storage guy. Imagine: 1) You don't know your enterprise storage expertise. 2) You're a fucking moron. 3) You're probably mentally incapacitated.

Now does that question make more sense?

herpa derp

Bart Silverstrim

INCEPTION

More like ESXieption

Bart Silverstrim

ESXEPTION

-1

Q: Where Credentails are save in PC for SQL Management Studio

When we logged into SQL Management Studio(using Server Name, Login and Password) with checked "Remember Password". I need to know, where it save in PC. I need to format my PC. And when we install SQL Management Studio, then I will loss my all Credential which I saved. That's why I need to get th...

wtf question

1:27 PM

@MarkM I wasn't always a enterprise storage guy- I spend a long time being clueless about smb storage, but I think that I would recognize that having a device provide its own iSCSI service would be less effective than simply going to the disk. Oh wait... #2 and #3

@BartSilverstrim Those guys live in the same city as me :)

I don't have a secure place to store my passwords except built in "remember my password" plugin. OHNOES!

Bart Silverstrim

@Basil Heard the host used to be a teacher before EMT took off.

@BartSilverstrim I don't know- I know people who know them, but not well enough that I read up on it. I just recognized the A&W in their burger episode

Bart Silverstrim

Dammit. Google's not my friend for information on proxying web stuff.

1:40 PM

Damn, @MarkM, you type quickly. I had just only started my answer when your's popped up, containing even more info... +1, nice show, etc....

Thanks :)

I was actually editing my answer to fix some stuff and it said you edited it. I was like damn, I thought I was fast to answer and it's already got an edit

@MarkM OIC, you posted at 13:37 :) Quite.

@pauska What are those numbers under Write and Read? # of transfers?

1:55 PM

Does Direct IO bypass any SAN cache? It might just bypass the OS page cache -- if so, you probably need more than 256 MB for a proper test

very vague question- any great software for tracking internal/external/client/nice-to-have projects in 1 place, with updates, assigning, tracking etc?

@AliGibbs: Jira is pretty good for that.

Pricey, though if you go for the many-users versions.

@MikeyB kB/s

@KyleBrandt no it doesn't. You can't usually bypass SAN cache without playing with the SAN controller.

@AliGibbs have a look at Fog Creek's Trello trello.com

2:01 PM

thanks @SmallClanger & @Iain will take a look :)

@pauska: So make sure you so what @Basil said ;-)

it's for our IT team (of 3)

@KyleBrandt ?

ouch - cheapest iphone 4S is £500 ($775)

@pauska: Make sure you are not hitting SAN cache on your tests

2:08 PM

I loved that one of the features of the iphone 4S is the ability to make phone calls

@AliGibbs that's probably the least used feature on my phones

Although doesn't look like you are going off those images .... but not sure I get those images entirely

the cache is probably making the small block writes slow

but it doesnt mean that I should get 30kB/s random read of 0.5KB blocks

I'm planning to disable the ssd cache to get some real benchmark results, but the general feel of VM's sitting on the new SAN is extremely sluggish

opening up sql management studio takes 3 times longer than on our old san

@pauska That's only 60 IOPs on read - UGH.

Can you show IOPs instead of KB/s?

quick formatting disks on win2k8 r2 used to take 3-4 seconds, now it takes a minute

not sure.. going to run more benchmarks when its done formatting the lab disks

nah I can only get kB/s

looks like thin provisioning might be biting me in the ass

2:22 PM

@Holocryptic RSI you inconsiderate motherfucker!

@Iain Raaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaape?

@voretaq7 I have a solution for you. IE favorites.

 0 watchdog, 1310 multicast, 422696 pause input

ORLY FLOWCONTROL

@Holocryptic What's an IE?

Anyone have any tricks to getting a Microsoft ticket escalated?

I told the support guy that's a retard to escalate me or solve my problem now. He said he would do some research and call me back.

@MarkM Threats of violence always worked for our MS team. Also probably helps if you're an IBM triple-diamond alliance member or whatever the fuck we were (some kind of high-tier Gold Partner)

2:30 PM

I didn't actually call him a retard, it was more like "you seem unable to grasp why I have an issue with every poor resolution that you've offered."

@voretaq7 I threatened to punch him in the cock in chat yesterday. He must not hang out in chat...

"You and I are clearly not communicating well. Give me your supervisor. NAO!"
Record that and play it back on loop.

@MarkM Or maybe he's into that sort of thing. He DOES work for MS support!

oh hey I apparently earned the DNS badge finally

Shit, you're right. Maybe he does hang out in chat.

I randomly for the Linux and Active-Directory badges on the same day.

irony

0

Q: Stop an unstoppable openvz container

I have an openvz container that is NOT stopping with the standard vz tools. How can I kill it? I tried a VZ destory but that wont work. Thanks

Bart Silverstrim

MarkM is a...MS shill?!

So tempted to comment "Put an immovable OpenVZ container in front of it and see what happens"

2

2:39 PM

@BartSilverstrim I am not. I have a bronze badge in Linux!

I'm platform agnostic, it just so happens that 85ish% of my servers are Windows and that's where the bulk of my experience is.

Bart Silverstrim

Sorry, nevermind. I only caught part of that line and thought you worked for MS.

I'm juggling five tabs at once, another site of instructions, and some ticketed problems.

While working on the web server over SSH.

haha

im getting better write than read performance against a raid5 setup

How do you get a badge for a tag?

this.does.not.make.sense

2:43 PM

@pauska yes it does- writes are acknowledged by the cache before they're committed to disk

@Basil N answers scored 10(i think?) or more

I forget what N is though

ok

@Basil that is true

@Basil serverfault.com/badges?tab=tags

@MarkM nifty

2:45 PM

@MarkM ah that's much more complex. Did it change or am I remembering something funky with kittens and lollipops?

@Basil I would expect around 1-5ms write response time on a storage array that doesn't have its write cache saturated.

@voretaq7 I don't know. I didn't even know that you could get tag badges until a few days ago when I got Linux and AD at the same time and almost had an aneurism

@MikeyB If its write cache is available, it should really by sub-ms response time, even with cache mirroring

@Basil 100 up votes spread over at least 20 questions for a bronze tag badge 400 for a silver

@Basil: yeah well disabled both write and read cache, my SAN is now with cancer

2:48 PM

@MarkM It's a sign!

@pauska disabled it on the disk controller? Or on the server?

@Basil on the controllers

im trying to figure out why our new SAN gives less performance than our old one

EMC is all "dont use benchmarks, they are invalid!"

@pauska lol

it doesnt help the fact that servers now use 3x more time to boot up and open up applications

@pauska "please don't test out claims- our salespeople will cry"

2

2:49 PM

@Basil From the perspective of the host, it usually ends up being 1-2ms. Of course, things are faster nowadays so it's probably down to the sub-ms range now, yeah.

@Basil :(

@pauska I'd have to ask what's the disk layout, etc.

Post a question!

@MikeyB Any time you see some silly SVC benchmark with 500,000 IOPS, it's done by running small IOs through cache and only cache

I have a big storage pool with only one LUN.. one smart-ass on the EMC forums told me that I need to add more LUNS to get better performance.. wich doesnt make any sense

@Basil 500K IOPs! 99.9% hit rate, yes, but 500K!

2

2:50 PM

@pauska he's right

@pauska each LUN has a single path to a disk, and each path has multiple potential choke poitns

@Basil no, he's not. LUN1 wont get better performance. The overall performance will go up, yes, but not on that single LUN where there is 1 host

vague question- if you had 5/6 Xen heads, and were going to use a SAN, would you need 10G networking?

@pauska More LUNs gives you better parallelism overall, bigger total queue depth, etc.

@pauska Just for shits and giggles, try breaking it into 2 LUNs and putting them into the same storage pool on your host

@MikeyB overall yes

@Basil thats what im doing right now, didnt make squat difference

except that the second lun is thick instead of thin, wich helped alot on the small i/o blocks

2:52 PM

@AliGibbs Not if the SAN is fiber-attached to each host :-)

@AliGibbs Ah, "networking" - that vague term that people use when they really mean "Ethernet"

@pauska You're saying that if you break your raid into , say, 10 LUNs and put them all in an LVM (or windows equivalent) you don't see better performance than with one?

@AliGibbs that depends if the moon is up or not

@Basil oh.. we don't use software mirroring.. this is ESXi

@AliGibbs Screw ethernet, get a real SAN.

but yes, that would ofcourse increase performance

2:53 PM

I know its too vague to know really.

@MikeyB um. I meant STRIP. Radio Modems FTW!

@AliGibbs 40Gb IB oughtta do.

@MikeyB yea, sorry I mean ethernet

@AliGibbs You don't need any ethernet.

@MikeyB Well... maybe a little to actually communicate with and manage the servers

2:54 PM

@voretaq7 Naw, you can do that over sneakernet.

@pauska esxi is based on UNIX at some level, but whether LVM works or not, you can still put as many LUNs into a storage pool as you want

@Basil: I'm seeing a 2000% better read performance in the benchmark tests when I disable the read cache.. latency doing its work (since its random data)?

@voretaq7 EoIB or IPoIB works

@Basil We're not going to do software raid on our virtualized servers man..

@pauska What??? That's messed up.

2:55 PM

the whole point is that our 4 year old EMC AX4-5i performs better than our new $100k EMC VNX

without read cache on 0.5kB random i/o: 2833kB/s

@pauska ... read cache? It's usually only valuable for certain types of easily predictable workloads

with read cache: 23kB/s

@MikeyB on a VMWare box? How? The ESXI ones don't have a real console...

@pauska Kb/s is not usually what you want to measure- you want to measure IO/s

@voretaq7 He said Xen

2:56 PM

@Basil I know, but the tool im using right now doesnt have such modern terms

@pauska so get another tool

@pauska iometer is free

why? the results are the same

@MikeyB hm? Oh so he did. People use SANs with Xen?

i get less packets with read cache

in that case yeah you don't need ethernet. Epoxy up the ports. FOR SECURITY!

2:57 PM

@voretaq7 At the moment we use Xen. Management of VMs? Who needs that?

@MikeyB Mellanox Dual-port 56Gb IB... forgive my drooling

@pauska the performance that matters for vmware on a SAN is the number of IO/s you can get when doing random reads. Sequential reads are uncommon, and sequential AND random writes are cached. Random reads measure the actual performance of your disk

My task for today is: Implement RHEV

hi need an advice which is be better seagate 500 gb portable hard drive or western digital 500 gb portable hard drive?

@pauska Something is fucked up with your storage array. Oh, and check your switches for port errors.

@lovesh One of each, RAID 'em. That way you avoid the inter-vendor static differential.

2:59 PM

@lovesh Don't go with either of those- you totally need an acer drive

@lovesh How do you define "better"?

@Basil wich is why im crying at this horrendous random read i/o

@MikeyB no i just need for some backup of movies or videos of my laptop

@ChrisS i mean performs longer and is fast

2

@pauska how are you generating the random reads?

@lovesh See the room topic. Also this is ServerFault's chat, not SuperUser

3:00 PM

@voretaq7 oops sorry

@lovesh Don't listen to the haters- acer drives are at least 300% better than either of those

Lifespan will be primarily determined by how many times you drop the drive.

discussions in this room are limited to vendor-bashing, bra-bashing, and bra-contents

(we really need to get some women in here to do the bra-bashing)

@voretaq7 Is that close enough to boobs for a starring?

@ScottPack no, but that was

3:02 PM

user image

@ScottPack m..my star sniffle

sniffle — sulk — kick EPO switch

@Basil used ATTO disk benchmark, running iometer now

I managed to lock out my AD account while still logged into Windows. Not sure how that happened, but awesome.

3:18 PM

@Holocryptic Logging into another computer (RDP), any service that prompts for credentials, hackers, stored (wrong) passwords... lots of ways

@ChrisS Yeah, I think it had something to do with that. All my logged in services just quit on me. Had to reboot and get someone else to unlock my account

6581 IOPS.. hm

again, on a single lun?

@pauska That's a little better. Across how many disks of what type?

after 4 hours, it finally logs me into chat, that was odd

3:27 PM

@Basil Yep.

but this is just preliminary, still some to go on the test

@pauska well it's not uncommon to have a single LUN be unable to allow you to achieve the maximum possible performance for the raid

@Basil im not looking for the maximum possible performance for the raid. I'm looking for the performance that our regular vm servers without any form of software raid/striping gets

@pauska Best practices for vmware is to put several LUNs into the same datastore

are you kidding me

vmware extents?....

no sane vsphere admin does that

(wich also isnt striping, it just fills one up and then goes to the next one, unless they've changed that without me noticing it)

I'm not sure what you mean, as I'm not a real VMWare guy, but at some level, VMWare is supposed to be able to use multiple LUNs concurrently. Otherwise you'll never get more than half the performance of an active/active disk array

3:32 PM

oh

yeah, you can have tons of datastores, all connected to different sans if you'd like

you just cant combine them like striping does

so that's probably what they mean.

if you'd want that then you'd have to give a VM two virtual disks (one from each lun), and use software raid inside the VM to stripe it

that's not it then...

then you're probably thinking about VMFS extents

where you add another LUN to a datastore

like LVM?

3:33 PM

its like dynamic disks in windows

you just add more space, it doesnt stripe

@Basil By active/active disk array, you mean a dual active/passive?

wich means when LUN 1 is full it'll start filling up LUN 2, instead of crashing because you didnt re-organize your vm's

well either way, the data stored on the physical VMFS will be located on both LUNs as you use the datastore more

another reason for using vmfs extents before vsphere 5.0 was the 2TB limit on datastores, but thats gone now

no, the second lun will get used when the first one is completely filled up

its not raid0

doesn't need to be- if it stores files like unix does, after a little while, you'll find actual data all over the "drive"

@MikeyB I mean on an array that has a preferred controller "owner" for each volume

3:39 PM

is it beer'o'clock yet?

@Basil Yeah, I call that an dual active/passive. I interpret active/active as "you can/should hit any controller for any LUN."

@MikeyB those are not common (or cheap) yet.

erm, they are

it's called ALUA

but its not recommended to let data hit any SP, because it'll traverse sp's

SP?

ALUA?

Well even in ALUA you still have a preferred path, it just doesn't require a controller failover like the Engenio/DS3000/etc/

ALUA -> Asymmetric Logical Unit Access

3:41 PM

yellow-bricks.com/2009/09/29/whats-that-alua-exactly

SP -> Service Processor

Storage Processor actually, sorry.. EMC term for Controller

these sound like specific vendor terms

ALUA is not vendor specific at all

@Basil XIV, DS8000 (well in the DS8000 case you're hitting the port which routes it to the preferred controller automatically). But yeah, not cheap.

3:42 PM

@MikeyB: our vnx does that aswell

I just havent enabled it in vmware (you need to explicity tell esxi to use non-optimal routes)

@MikeyB I know how those platforms work, but wasn't aware of the name of the algorithm. Anyways, at over 200k minimum for either of them, it's unlikely to see them outside a few large shops

@pauska NetApp for one uses ALUA, I'm not sure who else does. But I know there's a bunch.

I was thinking about first fixing the performance before I start to tweak stuff :(

@Basil VNX starts way below 200k

@Basil DS8000 is mucho $$$.

@Basil Just reading up on ALUA, never heard the term before but even my MSA2300 does it... Not sure what the brochure would call it.

3:44 PM

ours were $100k afaik

iometer has a weird way of saving results

so back to the issue you're having: VMWare can't use two LUNs for the same server without software raiding?

I think its kind of pointless to debate it any further, sorry

I think you're right.

4:04 PM

I swear, the next person who strolls up and tries to solve a problem I'm having by starting a sentence with "Why can't you just..." is going to get the first available obstruction I can get my hands on inserted forcefully in to their mouths.

2

BEER, BOOBIES, BED*

*may not be in that order

NARQ this chump

0

Q: Multiple users extracting the same tar, occasionally some fail

When multiple users attempt to untar the same archive (uncompressed) occasionally some will fail, resulting in no extracted files for that user. The tarball is 3.2GB if that makes any difference. The command to extract is: tar -xmf Surely tar only reads the archive and there shouldn't be any c...

linux debian tar debian-squeeze

that'll be a way to get my 300 flag weight =D

4:20 PM

@MikeyB I chose to leave a mean comment instead. If he doesn't clarify with an actual error message and return code by the time I go to lunch I'll NARQ him :)

# Error handler
print("OMG I FAILED!!!11!!", fd=sys.stderr)
sys.exit(1)

3

@MikeyB "I've seen tar do all manner of strange shit, but I just looked at the source and "fail" is not an error message it ever uses." :-)

Bart Silverstrim

Tell me if I'm modeling this in my head wrong...? We have a camera inside the network. Management wants to know if we can stream it to the Internet via our web server. I'm thinking that there should be a way to proxy the connection so it goes <internal IP camera> -> web server -> Internet.

So we don't need to forward ports or IPs from the public network into the camera's IP.

Does that require a kind of proxy server to do that?

I have a simple way of getting it to display video into our CMS with a snippet of html but it seems to be directing to our internal IP, which of course won't work for people outside our network.

Hmm...

So far no responses on the webmaster QA

because there is no answer!

Bart Silverstrim

No answer, eh?

4:34 PM

so, after two days of struggling: community.emc.com/thread/124266

(if you can see it?)

@BartSilverstrim Yeah you could do a reverse proxy. But if the camera is sending out its internal IP in the HTML, not much you can do. Make a page that hotlinks the video stream or something.

@pauska Oh you're doing iSCSI? Ew. But at least I was right - "something's fucked" :)

Bart Silverstrim

@MikeyB I'm thinking there must be a way to set up a reverse proxy so that the video on 10x will be seen on the web server's IP...so the web server is a middleman to process the stream.

@BartSilverstrim Like so? apachetutor.org/admin/reverseproxies (search for ProxyPass)

Bart Silverstrim

Looks like it. You know how many goofy crap results I got for googling proxy information? O_o

@BartSilverstrim It's a lot easier to find it when you know the name of what to look for :)

Bart Silverstrim

4:45 PM

Hard part will be enabling this with our CMS. I'll be hammering at it until I can get it working, though.

@pauska Broadcom sux

hi.

@Warner hey

user image

Bart Silverstrim

Hello kitty? Have you been on the help desk too long?

Any time on the help desk is too long. But yeah. I <3 Hello Kitty-ware.

user image

Thats my fav

« first day (389 days earlier) ← previous day next day → last day (4591 days later) »

Transcript for

Oct '115

The Comms Room

ServerFault's lobby

join this room
about this room

00:00

06:00

12:00

18:00

all times are UTC

site design / logo © 2024 Stack Exchange Inc; legal

mobile