The Comms Room

6:00 PM

what if you had some software installed on every pc and you had to remove it before you could install the latest version? what would you do right now?

Miguel

Go to about 30 machines and do it the way I described previously

would be a suck day for sure

I've had them, like when windows live stopped working and I had to put skype on everybodies machine

Fitzroy

again - write a script to uninstall the softwre and another to install the new version. and run it against every machine, sit back and relax

Miguel

color me sold, that could save me a headache

dawud

there is puppet for windows too... just saying

Miguel

Same type of thing?

I don't really use alot of automation tools at the moment

dawud

6:03 PM

you should

Fitzroy

admittedly - the software would have to support unattended instllation, meaning you provide all the instructins either in a config file or as command line arguments. but i have found almost all software for windows systems supports unatended installation these days

all the stuiff i use anyway

Miguel

I could prepose it

Fitzroy

what?

Miguel

The corporate network is the one where I actually have authority to make real changes

propose*

Fitzroy

propose what?

Miguel

6:06 PM

implementing some sort of solution for automated installations and such

although that might not be used much either

we keep their machines pretty minimal, just what they need to do their jobs

unless of course there was an update that required re installation

The only instance I've seen being the skype thing

Fitzroy

well - i think you'd find powershell an interesting challenge. but im just trying to think of a few real life projects you could work on.

Miguel

Yeah that's what I've been trying to do

The only excitement I get anymore is when we decide to add something to our network

Like the IDS thing, we never had that on the corporate network till I offered to set it up and monitor it

and WSUS for the secure network

Fitzroy

ah ha!

here's an idea for you...

ms havvnt made it very easy - but i use powershell to install the updates on my servers.

Miguel

I am assuming your company is also very opposed to auto updates ;)

Fitzroy

you know how one set of updates install - a reboot is required and then sometimes another set of updates are needed. you can use poweshell to keep installing the updates even when a reboot is required until there r no more to install

Miguel

6:11 PM

We can never do auto updates cause nothing can suffer unschedules downtime

Fitzroy

i 100% agree

auto update should not be used on svrs

Miguel

I see how that could be useful

Fitzroy

thats why i use this solution.

Miguel

Can you write custom messages to event log to indicate when the last round is finished

or anything of that nature

Fitzroy

absolutely

or send an email

or write to database

or write to a logfile

Miguel

6:13 PM

either would be useful

I keep a central syslog for corporate and secure

so one of my installation scripts that I just wrote lets me know if the server installation finished without errors by writing to syslog

Gone are the days of having to babysit an ubuntu server installation through every step

Fitzroy

powershell has a very good restart command in version three which allows you to perfomrm a reboot, wait for the machine to come back up and then carry on with your script.

good move

Miguel

sounds like script porn to me

Well I had to, I would always have to walk away during the long parts of the installation and guesstimate when to walk back into the datacenter to continue

seemed logical to have it alert me when all was finished so no time was wasted

Fitzroy

how do you build your windows workstations?

Joel E Salas

We have an image

Miguel

Role dependent, I know I will get body slammed in this room for not using an image, but the infrequency in which we need to install workstations has swayed me from that. I do it the old school way

Whoever their manager is going to be gives me a list of software they will need, assigns it to me in our internal work request system, and I do it

Fitzroy

6:18 PM

I've never bothered much with imaging. Mostly because we have PCs of all different models. I use PXE (Pre-Execution Environment). The Windows setup is 100% automated as is the installation of all the software and lastly the windows updates.

I use Windows PE in conjunction with PXE

Miguel

I've heard of PXE but never implemented it

I rarely have to install workstatios

ALTHOUGH

There is going to be a dark week in the near future where I will have to replace something in the neighborhood of 50 windows boxes in our datacenter with the cute little poweredge R210IIs

If I could make that easier I would

Fitzroy

you're moving to VDI?

Miguel

nar

Fitzroy

oh sorry no - i misread

Miguel

Just replacing old boxes with newer boxes

we were using towers for a particular set of servers back in the day, all installed with windows 7 and serving a single purpose, in a distributed fashion

Fitzroy

6:22 PM

why 50 physical servers? do you use virtualisation?

Miguel

now we are gaining back real estate by using the 1U poweredges

no virtualization

Fitzroy

oh. why not?

Miguel

My boss probably has his own reason, but I am assuming that he does it for fault tolerance or something

If one physical server dies, it's not hosting anything but whatever application it is desinged to serve

And he explained something to be before that I was a little to inept to absorb at the time, but I remember him saying something about horizontal scaling vs virtical scalig

So as we expand, we can simply add more devices to handle the distrubuted workload, rather than continue to upgrade one or two mega servers to handle the increase

I was new at the time but it made sense to me

Shane Madden

Eek

Fitzroy

Not flawed but old fashioned

Shane Madden

6:27 PM

Unless all those physical servers run at full load all the time, that's such a waste

Miguel

He's been doing it that way for like 12 years lol

They pretty much do

Shane Madden

Yeah, old school.

Fitzroy

well - keep one of the old servers, install hyper-v or vmware on it and setup a virtual test environment.

Miguel

What does that do in the way of fault tolerance though

Fitzroy

my previous suggestion wasnt aimed at improving fault tolerance - im just suggesting a little project to get your hands dirty with virtualisation

Shane Madden

6:30 PM

Well, when a single piece of hardware fails (which you'll have fewer pieces of hardware, so fewer parts to potentially fail), your virtual machines with application workloads will be automatically rebooted on a working piece of hardware instead of staying down.

Miguel

It would be fun

Shane Madden

And if you need to do maintenance on one piece of hardware, you can move the virtual machines off to different hardware without taking them down.

Miguel

And you could do this seamlessly with minimal downtime?

Fitzroy

@shane - do you use vmware or hyper-v?

Shane Madden

@Miguel No downtime. Milliseconds of network connectivity loss for the switches to figure out where the MAC address moved to.

@Fitzroy VMware mostly.

Fitzroy

6:32 PM

i mostly use hyper-v

Miguel

See that is what sucks about my job

I have little to no knowlege of some of the newer technologies

and methods

Fitzroy

we have a few dev environments that use hyper-v server 2012

Miguel

Still though I am unsure of how it would be implemented in my environment

Fitzroy

not win svr 2012 with hyper-v role - but the 'hyper-v svr 2012'.

Shane Madden

@Miguel Yeah, understandable. If you can't get exposed to the stuff at work, mess with it in your spare time :)

Fitzroy

6:34 PM

thats why i suggsted powershell as something to learn. you dont need to setup tons of VMs or need any physical kit.

Miguel

I have like 10 servers at my disposal that are doing nothing

Shane Madden

@Fitzroy Ah yeah, that's a good one.

Miguel

I intended to use them for failover testing, sort of a live disaster recovery drill

Shane Madden

@Miguel Nice! If you can, get the ok from management to set up a lab.

Miguel

already did, just haven't had time

Fitzroy

6:37 PM

are the old servers still going to have windows licenses or are you transfering those to the new servers?

Miguel

I am interested in virtualization though, since it seems to be the thing to do nowadays

Fitzroy

if not - you're gonna have to find a nice linux project!

Miguel

they are bare, I smashed the hard drives already

Fitzroy

oh no!

why?

Shane Madden

Well, you can also fire up trial versions of Windows for short term lab projects like if you wanted to set up Hyper-V.

Miguel

6:38 PM

Security policy, since they once had sensitive data, upon retirement the hard drives are destroyed

Although I have tons of spare unused drives to play with

Fitzroy

i see

Miguel

so that isn't an issue, but licensing is

Fitzroy

as shane says you can use trial versions of windows

the server versions seem to last ages, 100+ days at least

Miguel

We have reached a point where we are trying to overhaul and modernize our setup that has remained for the most part unchanged for quite some time

but it is a slow process, and done one piece at a time

we didn't get rid of our last PIX till last year

120 days for server I believe

Fitzroy

also, if you use 'Hyper-V Server 2012' its free! BUT you hve to license every guest. BUT..if you run all linux guests you dont have to pay a penny.

Miguel

6:41 PM

OUr environment doesn't allow it, or at least not all of it

Just to paint a picture, the majority of our servers are there for a single purpose, as part of a distributed workload

Fitzroy

im not suggesting any prtactical usages here - just things you can mess about with

Miguel

yeah but me being the curious type I am now trying to work it out in my head lol

before I ever do anything I always spend assloads of time brainstorming, trying to piece the puzzle together before I pull it out of the box

Fitzroy

that approach will serve you well

always plan first

Miguel

Yeah now that I think about it I doubt it would be feasable

for example

Fitzroy

not once you've fowled something up

Miguel

6:45 PM

The windows boxes I mentioned all exists soley to run a program, this program runs an instance of itself on each CPU core, through some programatic witchcraft

the program uses each core seperately to process incoming files, that are randomly selected from a central server, thus distributed

So with 50 some odd windows boxes with 4 cores per CPU, to try and stick all of those on one server and actually have the resources ( 50 X 4 cores each) would be very difficult at best

I am assuming that VM's let you allocate cores and other resources on install time

Or am I incorrect about that

Shane Madden

@Miguel No need for you to run it all one one server; you'd essentially have a cluster of physical hosts running your virtual machines distributed between them.

Fitzroy

1x vCPU does not have to equate to a core

Miguel

not core per say but threads

Fitzroy

you can have more vCPUs than cores

Miguel

Forgive me if I am wrong but I am only vauguely familiar

So if you have 2 physical cores and 2 vCores, the program can run 4 instances

the question being, are VMs capable of allocating something so specific for each VM

Fitzroy

6:50 PM

sorry - thats not quite what im saying

Miguel

oic

Shane Madden

@Miguel You'll assign a certain number of virtual cores per VM, and their execution will be scheduled on physical cores by the hypervisor.

Miguel

I told you I am not pretending to completely understand what I am getting at, it's just the best way I can explan it :)

@ShaneMadden In that case couldn't that potentially cripple the processing power of the individual hosts?

Fitzroy

say you have two physical cores. You could have a 1:1 ratio or cores to vcpus but you do not have to

you could have two cores but four vms with one vcpu each

Miguel

Currently the CPU on each physical host (50+) are maxed out during processing

Shane Madden

6:52 PM

@Miguel If you overcommit (assign more virtual cores than there exist physical cores to run them on), then there can be contention between the virtual machines for CPU time, yes.

Miguel

well more is better in this case, one host with one core is not as good as one with 4

It sounds wierd but this is one of those scenarios where every environment differs

Fitzroy

that may not be for the right reason though

eg if your program is only single threaded - you have no choice but to run it once per cpu/core.

if its multithreaded you may find that one cpu/core would do]

Miguel

It is single threaded, so it is running one instance per cpu/core

That much we know, and my boss is currently coding the new multithreaded release

Shane Madden

@Miguel Right - and a modern virtual machine host might have something like 4 CPU sockets, each with a CPU that has 10 cores with hyperthreading - lots and lots of execution threads available.

Miguel

I told you changes areslow lol

That would be so cool if I could talk my boss into it, he listens if I know exaclty what I'm talking about when I pitch an idea

Fitzroy

6:56 PM

my point being - that the utilisation of each core by your program might be miniscule. So once you make it multithreaded you might actually get away with one core...

Miguel

But the way your explaining it sounds seems feasable

Ok so if your up to it on to the next

Shane Madden

Yeah, it depends a lot on the workload. If they ever back off their CPU usage (like when waiting for a new file?), then you might be able to allocate more vCPUs than you have physical cores in the system.

Miguel

I have I think 60 or so linux boxes that run mysql and are a distributed workforce for searching

their usage is all RAM, and each of them have 24GB installed and use most of it

hard to see how I could shrink that down without spending more money than we already are

not to mention each of the servers themselves are 1U so the real estate isn't an issue there

Is there some situations where virtualization just doesn't make sense, and the old school horizontal scaling method actually makes logical sense

@ShaneMadden yes when they are waiting for files the cpu usage decreases dramatically

Fitzroy

I would say there probably are cases - but unfortunately those are never discussed .

Shane Madden

@Miguel Search? What kind of data? MySQL isn't typically the best for a straight-up search workload.

Miguel

7:00 PM

In a nutshell

it's an OCR type thing

Shane Madden

@Miguel Yeah - pretty much when you're Facebook or Google.

Fitzroy

unfortunately - in IT, as soon as something new comes out, everyone turns nasty on the old way of doing things. Sometimes we have the sense to go back though.

take VDI for example

Miguel

I can explain that

Fitzroy

a few years ago thin clients were the future

Shane Madden

@Miguel Maybe have a look at some of the distributed search stuff out there, like elastic search.

Miguel

7:11 PM

The searching is for OCR. So the documents come in and are initially stored on one of a series of NAS servers. A copy of the document is sent to one of the windows boxes I talked about to get the text of the documents parsed and associated with the file by an ID. Once this is done the results get sent to a large linux db server which has the information on every document, the text associated with each document ( for text searching) and the location of the actual document on the NAS servers.

When somebody searches the system for text within a document, rather than massive amounts of queries htting that one linux servers, we hand a section of the results to one of the other linux servers. So now when somebody looks for text on a document, the results can come from one of 60 servers, which each server gets enough queries a day to make 24 GBs of RAM just barely enough to return the results within subseconds.

never heard of elastic search

Dennis Kaarsemaker

7:23 PM

elastic search is fun

but if you use it: make sure you understand the JVM.

Miguel

Not in my expertise toolkit as of yet

I figured out the correct term for what I was describing though

Shard (database architecture)

A database shard is a horizontal partition in a database or search engine. Each individual partition is referred to as a shard or database shard. Database architecture Horizontal partitioning is a database design principle whereby rows of a database table are held separately, rather than being split into columns (which is what normalization and vertical partitioning do, to differing extents). Each partition forms part of a shard, which may in turn be located on a separate database server or physical location. There are numerous advantages to this partitioning approach. Since the tab...

I'm not a database guy so I don't know it off the top of my head

Dennis Kaarsemaker

haven't followed the discussion. Are you sharding your DB?

Miguel

That's what it seems like

Although the term is new to me that is what I do

Dennis Kaarsemaker

fun

say goodbye to unique keys and foreign keys :)

Miguel

All I know is it is fast, but I couldn't see myself virtualizing it

Dennis Kaarsemaker

7:26 PM

sharding and virtualization makes no sense as a combination

Miguel

I figured as much

Dennis Kaarsemaker

you shard when a single box is too small. Why the hell would you virtualize?

Miguel

I can't think of a reason, I was just looking at different aspects of my architecture and seeing which portions of it would make sense to virtualize

Something I don't do at right now

Dennis Kaarsemaker

what's your goal with virtualizing?

what problem are you trying to solve?

Miguel

As of right now I don't know anything other than the fact that it seems to be all the rage. But with minimal research it looks as if it may be a good way to cut some costs and free up some real estate in the datacenter

Dennis Kaarsemaker

7:29 PM

it's definitely all the rage, but that's not why you should do it :)

Miguel

But it is not a need as of the moment

Dennis Kaarsemaker

if you need to cut cost and reclaim real estate, it's a good thing to look at. If you don't, why bother with the hassle?

Miguel

None in production, although I certainly would love to fill the knowledge gap with practical experience

Dennis Kaarsemaker

that's a good goal :)

so you know what to do when the need arises

Miguel

which I am sure it will at some point

Dennis Kaarsemaker

7:31 PM

are you a windows shop, or something linux-y or BSD?

Miguel

For the moment we can free up like two racks by getting rid of some gay towers and installing some 1U units that can do the same job better

Combination, windows for some tasks, ubuntu server for othres

Dennis Kaarsemaker

then try both vmware and KVM. They both work in hybrid environments, with kvm needing a linux physical host and vmware having the snazzy tools

Miguel

I've only used VMware for pentesting, never got nearly as involved as hosting servers

dawud

8:11 PM

KVM alone is quite spartan, people tend to use RHEV or OpenStack

Dennis Kaarsemaker

kvm wilt virt-manager and virsh is ok

RHEV seems to be decent these days, now it no longer requires windows to manage linux machines :)

ewwhite

You all know my opinion.

RHEV was a disaster for us.

that was one year ago, though.

Dennis Kaarsemaker

who's the 'us' there?

ewwhite

A trading firm I was working at.

Dennis Kaarsemaker

ah

ewwhite

8:21 PM

I basically left over the KVM versus VMware argument.

Shane Madden

RHEV required Windows? Hah.

ewwhite

yes, for management

dawud

It has change a lot since that, though

Dennis Kaarsemaker

yeah, even more than a year ago

Shane Madden

If I were going to do a KVM deployment it'd definitely be openstack though.

ewwhite

8:23 PM

it has, but so has VMware. We looked at oVirt which was release shortly before...

Dennis Kaarsemaker

We use our own scripts for deploying kvm. Still need to look at openstack and see if it would help us

dawud

no NFSv4 support in VMware yet?

ewwhite

@dawud No.

We get by on NFSv3.

dawud

@Dennis [this ](http://archipelproject.org/) project in still under heavy development, not for production, but take a look just for fun.

@ewwhite they promised support back in 3.x IIRC

Shane Madden

@dawud That looks awesome.

Dennis Kaarsemaker

8:26 PM

@dawud looks nice. Wonder how it integrates with our systems

dawud

yeah, looks nice. the demo is quite impressive, and hey claim support for KVM and VMware

ewwhite

I think I just like my virtualization solution to be a slam-dunk... straight-ahead. I don't have to worry about hypervisor stability on the VMware side. The kernel is refined well enough, supports the hardware I need... Good consideration has been given so I can PXEboot hosts or run off of SD card. Performance always improves with each revision.

Dennis Kaarsemaker

@dawud that's just the start of what we need :) It needs to work with our asset mgmt system, ldap role based auth andnot want control of things like dns and monitoring

most 'server management' or 'vm management' software out there wants to control your entire environment, which is annoying to the point of not being usable.

ewwhite

Any look at onapp.com ?

dawud

@Dennis agree. our identity mgmt is an IPA server (includes DNSSEC and NTP), for monitoring, we use zabbix

Though I would really like to learn about AD and group policy in deep

Dennis Kaarsemaker

8:32 PM

For instance, I want just one and exactly one graphing system, so I can easily correlate between any two metrics. So don't make your vm management thing do its own monitoring and graphing for crying out loud.

dawud

/me kinda a microsoft ignorant :(

@Dennis graphite makes that quite beatifully

Dennis Kaarsemaker

@dawud indeed, I love graphite

we dedicate several dozen machines and many terabytes of disk to it :)

dawud

wow

@Dennis do you use [part of] the logstash~redis~mq~elasticsearch~statsd pipeline to feed graphite?

Dennis Kaarsemaker

Being able to plot e.g. pageviews per minute and packet drops on memcached servers in the same graph is an insanely useful debugging aid (in this case it pointed out that the packet drops started well before the drop in pageviews, making it an unlikely cause of that drop)

@dawud we use diamond for symple measuring and the socket interface to pipe lots of custom metrics from our business monitoring system to it

we may be using stats as well, not sure

dawud

dinner time :) @all Gentlemen, always a pleasure.

Scott Pack

9:20 PM

Roar

Dennis Kaarsemaker

'sup lion king?

Scott Pack

Lives. Circles. You know how it goes.

Dennis Kaarsemaker

Hakuna matata.

And a bottle of scotch.

Scott Pack

I could go for a nice bourbon drink.

I've never made a proper Old Fashioned. Perhaps I should seek one out.

ewwhite

WTF!

I need to get off of this site.

Joel E Salas

9:29 PM

@ewwhite what's the bike shop in Santa Monica called?

ewwhite

0

Q: ESXi 5.0 - Software RAID 1

Environment: - Storage: HP P2000 MSA G3 SAS Array with 24 300GB 10k SAS Disks -- Two Storage Controllers with redundant SAS connections to each host - Hosts: Three HP DL380 G7s with a 10GB SD Card, CPU, Ram, etc... ESXi 5.0 is installed on the SD card in each host, this is the only local sto...

@JoelESalas Performance Bicycle.

Joel E Salas

Thank you!

ewwhite

@JoelESalas get some Wetzels for me

Desired result is ability to survive many disk failures. RAID6 gives me the ability to lose 2 disks per vDisk and still be okay. Applications that the servers run cannot go down. This is most important. Performance isn't much of an issue because the system is small. Right now I'm using Storage vMotion to move VMs off of a LUN with a bad disk, so that disk reconstruction performance doesn't affect performance. — Lucretius 24 mins ago

Joel E Salas

@ewwhite three and a half stars on yelp? Where are you sending me???

ewwhite

@JoelESalas it's a mega bike store... with a huge internet footprint

but they acquired the santa monica shop a few years back

Joel E Salas

9:34 PM

@ewwhite How can this compare to the spirit and moxie of a bike co-op in the hipster part of town

ewwhite

@JoelESalas It's clean... you won't need to use Purell™ after you leave...

Scott Pack

@JoelESalas For one your bike will have more than one gear and made out of something lighter than 1/8th inch tubular steel.

ewwhite

oooooh!

Burn!

Scott Pack

I aim to pleasure.

ewwhite

@ScottPack if YOU were a mod, how would you handle a contentious exchange like the one I linked to?

Shane Madden

9:47 PM

If he's so scared of a triple failure that RAID6 isn't good enough, recommend RAIDZ-3? Heh.

ewwhite

@ShaneMadden if only... but the argument is stupid. I want to downvote.

:)

but I can't

Shane Madden

@ewwhite Why not?

ewwhite

because that's a bitch move...

Dennis Kaarsemaker

@ewwhite we still love you, even when you're a bitch

ewwhite

@DennisKaarsemaker Well, you have lots of HP disks die... any advice for the OP?

Dennis Kaarsemaker

9:56 PM

didn't read the question. But fuck raid 5/6. 1+0 and spares is enough. I have lots of disks die because I have lots of disks. Rarely more than one in the same server and so far never lost a box due to too many disks failed.

did lose a box due to it being raid 0 and a disk got pulled because it said 'predicted failure'. That's just human stupidity though.

ewwhite

@DennisKaarsemaker In December, I had a triple disk failure on a proliant...

and had to rebuild very carefully... two failed disks and a predictive failure

the predictive died right as the array rebuilt

Dennis Kaarsemaker

@ewwhite meh, I'd move that to a different box.

or just scrap the box if it's one of many

ewwhite

it was really a bad batch of drives from CDW

everything I bought at that time from that vendor failed around the same time

never forget.

Dennis Kaarsemaker

not 'genuine hp drives'?

ewwhite

@DennisKaarsemaker Just a shit batch.

Dennis Kaarsemaker

10:00 PM

shit batches happen.

We've had it with memory, which is more of a bitch to sort out than harddisks

Joel E Salas

10:12 PM

@ewwhite which road bike was your recommendation again?

ewwhite

@JoelESalas dunno.

go and see what they have.

I'm looking it up

Adrian

afternoon gents

@dinn

ewwhite

@JoelESalas just search chat... or go

Adrian

10:36 PM

@ewwhite that RAID61 guy works for Kroger. He really shouldn't have problems getting the budget for doing his service properly if it's all that critical to operations.

ewwhite

Oh fuck...

Adrian

They only own about 30% of the grocery stores in the country.

There's 4 major chains in this area. They own 2 of them.

ewwhite

@Adrian My produce clients deal with Kroger... I've had to CALL Kroger's IT staff multiple times and walk them through Microsoft Exchange server settings because they would inadvertently do something that caused mail rejections.

Adrian

@ewwhite Ah. And there's days that it really doesn't pay to live in Seattle. Too much talent here to compete with if guys like those can hold a job down. Too bad I'd never want to live in Atlanta.

ewwhite

They're not a sophisticated operation.

Adrian

10:40 PM

@ewwhite They ought to be. They're the largest grocery retailer in the country and 4th largest in the world.

ewwhite

@Adrian Naw, it's like produce.

I've had to walk them through telnet email tests...

Adrian

@ewwhite Weird. I've seen smaller organizations (Wegman's) that have more IT and business patents than they have stores.

Love me some Wegman's. Which I'm sure @jscott can sympathize with.

ewwhite