so now it means that they're going to run phones on the data network...

and need PoE everywhere

across 4 locations and ~180 phones

Well not everywhere, you don't need poe to the servers, just your access switches.

@ewwhite I don't know if that's such a bad thing for a new install

@MarkHenderson I put new switches in 18 months ago

100% PoE on cisco cat 65/67xx's utterly fucks them after about a year

Most phones have switches built into them, so you can PoE to the phone and power it, and then plug the computer into the back of the phone

and asked them if they were planning voip so that I wouldn't have to do this again

@ewwhite Hmmm

Don't forget to think about increased heat and power usage in your IDFs. PoE swithces get a lot hotter.

Oh, I've run many computers off of phones.

or inline with the phone

But usually on Cisco...

I end up with configs like:

interface GigabitEthernet2/30
 description Voice/Data
 switchport trunk native vlan 2
 switchport mode trunk
 switchport voice vlan 3
 auto qos voip trust
 spanning-tree portfast
 service-policy input AutoQos-VoIP-Input-Cos-Policy
 service-policy output AutoQos-VoIP-Output-Policy

@ewwhite that's not a "bad" decision -- it means they're going to need to install POE switches though...

where the phone starts on the native vlan 2, downloads tftp from the phone server, then reconfigures itself on the trunk to use vlan 3

and vlans.

@voretaq7 well, because they can't tell me where they need PoE, I'm installing chassis switches at all sites.

all PoE on every port.

@ewwhite doesn't have to be chassis switches - plenty of 1U POE switches

but yes, you just PoE everything. It's easier.

@voretaq7 I need a lot of ports per location

more expensive too, but easier.

I'd agree, use access switches

@ewwhite well then you already had chassis switches so you're just swapping out for POE blades :)

@voretaq7 I went budget before, so only lowly HP 4200vl... no PoE options on the existing

@Chopper3 I'm OK with doing chassis switches but then you need to be sure your breakout connectors and patch panels are all appropriately rated for PoE (I'm pretty sure anything made in the last 5-10 years would be though (?))

@ewwhite PoE injectors are also an option, but if you need more than a handful the price of a PoE switch is quickly justified (and it sounds like you need WAY more than a handful)

So right now, I have an HP E5412zl chassis with 8 24-port blades, and E5406zl chassis at the other three locations

It would make me very happy if I could re-IP this customer, too...

but not enough time.

yeahhhh having to replace all your switch chassis is a bit kerspensive.

I gotta say though IP telephony fucking rocks compared to the "standard" PBX options

192.1.1.x/24 is the main site's IP scheme :(

@ewwhite . . . KILL THEM.

@ewwhite Can you buy it back for pennies on the dollar?

so my cost on all the switches is... $26k shipped.

Kill them all. VIOLENTLY. Carve "I USED A NON-RFC-1981 SUBNET FOR MY INTERNAL NETWORK. I AM VERY SORRY." on the bodies and hang them outside until the carving is not legible anymore, then burn the bodies.

customer paid $110k for the new phone system

Use the ash to print copies of RFC 1918.

@ewwhite so $150k by the time you're done integrating it.

BARGAIN!

(narf)

@voretaq7 That's less than $1,000 per phone

Beats ours. 20.100.100.x/8.

I've seen worse deals

I'm trying to decide if my markup will be punitive

@Tanner Wow. Way to wipe out an entire portion of the internet

@MarkHenderson Dude. You're in Australia. You gotta divide by 100 to get "reasonable prices" for the rest of the world :-)

@voretaq7 The voip phones we have here are $600/each just for the phone

@MarkHenderson that's..... actually not unreasonable :P

@MarkHenderson I've been hoping and waiting for a conflict. No joy.

they're nice phones I assume?

@voretaq7 Shoretel

I tried to get them onto an Asterisk-based setup.

@ewwhite Yes, punitive markup.

@Tanner You don't do any business with CSC

@voretaq7 Damn straight

@MarkHenderson ooh, shiny. whose is that?

@ewwhite at least you tried to do the right thing for them :)

@voretaq7 They're a Seimens OpenStage phone

But on a german system called Swyx

I also like that we only use 20.100.100.x but keep the mask 255.0.0.0... wat iz network!?

@MarkHenderson ahhh, not SIP then?

WTF!

@voretaq7 There is a SIP firmware for them, but we don['t use it

or SIP compatible by the look of it....

So you can SIP them if you want

and..... <beats on supplier>

@ewwhite And that means she will NEVER EVER see it.

$350 about. So just the standard 2x OzRape :P

that's with SIP firmware so whatever you guys are running might bump the cost up more too if it's proprietary

@voretaq7 Wow. I bought these off ebay from the EU about 4 years ago for more than that

@ewwhite I don't Facebook, is that for real?

@Tanner yes

Wow

@voretaq7 Yeah there is a licensing cost per-phone for the OpenStage firmware, but it's not much. About $50/device.

I heard they were doing that for 'famous' people so that they didnt get spammed.

Terrible.

@MatthewIfe No, they're doing that to everyone.

In reality its called rent seeking.

@MarkHenderson the 2 guys I use are 340ish and 360ish so I split the difference :) IDK if they ship to Australia though.

@MichaelHampton Oh. Definitely rent-seeking.

@MatthewIfe Fuck. That. Noise. You don't wanna get spammed? Make your fucking facebook profile private.

You want a public page expect the public to interact with you.

@voretaq7 I'm impressed you have a supplier. There is only one supplier I could find in APAC and they're actually in New Zealand

@voretaq7 well, they have to make money somehow. Its a pretty bloated company in terms of its real value (leastways for how they are exploiting peoples data at the moment).

@MichaelHampton How dumb do they think we are?

@MarkHenderson we have two "authorized vendors" for phone stuff - they're resellers, but their prices are reasonable even on our tiny (5-10 phone) orders

Still think the amount of information they are gathering has a ceiling on its value though.

They saw how Candy Crush and Pet Rescue charge $1 for powerups and someone thought "Hey! We can do that too!"

@MatthewIfe nah. they just collect more data when they're in danger of a valuation plateau.

like Da Google.

@voretaq7 Yeah, I suspect ifg I had larger orders than 5 phones I could probably get attention from elsewhere

I literally could not get them to quote on a 5-phone system for a church. They just weren't interested.

(they've gone Lync instead)

@voretaq7 Yes, well Google are becoming decidedly more and more evil each year. I really don't want to have to start hating them.

@MarkHenderson If you've been paying any attention at all, you already hate them

@JoelESalas They haven't done anything that I consider terribly bad yet

But I'm not American and I know how y'all love your civil liberties

@MarkHenderson voiplink.com might quote you (if they do business in Australia) - currency conversion & shipping will be brutal though, and after applying your local taxes to the import it might wipe out any savings :-/

we also use voipsupply.com but I'm moving away from them slowly - they're great, but they don't stock higher-end stuff and we're NOT buying any more grandstream crap - that was all supposed to be PoC and it's STILL in production :-/

@MarkHenderson yessir, we love giving them up.

@MarkHenderson Facebook is filled with people who spend inordinate amounts of money on things like Farmville and Candy Crush. So, on average, pretty damn dumb.

@MichaelHampton Apparently not, 50% of Free to Play's income comes from 0.15% of the players

@MarkHenderson They're extra dumb though

@JoelESalas Or really, really bad at basic puzzle games

Damn, I really need to throw some crappy graphics over some Android sample code and make a zillion dollars.

It's less that they suck and more that the devs are really good at slowly and silently turning puzzle games into pay-to-win games

gamasutra.com/blogs/RaminShokrizade/20130626/194933

Ooooh, I get to have some fun with LXC containers! @JoelESalas @MatthewIfe

@ewwhite Ooooh, for what?

@JoelESalas more produce stuff...

I asked the developers what specs they needed for their Pentaho installation... - 24 to 32 cpu's - 32GB RAM

@ewwhite lol Pentaho

cheapskates

@JoelESalas Yes, that's five of them!

hahahah

@JoelESalas Another client dumped Pentaho and added Tableau instead

@ewwhite Thats a pretty skewed spec sheet

24 processors and only 32GB ram?

@MarkHenderson TOO MUCH RAM. The key is to make the number evenly divisible, otherwise the processors start getting jealous of each other :P

so I have a few systems running this app in production... and the specs won't seem to be justified.

I'm hoping my Ubuntu upgrade staging finishes before I have to leave tonight. I think this upgrade will be my least-suicide-inducing yet.

Or system load...

@voretaq7 So tahts what I've been doing wrong all these years

@ewwhite your load is very bursty. Those are like 10-15 minute bursts?

@voretaq7 daily

how quickly does that processing need to be accomplished?

the spike looks to be a daily ETL load.

So I think they're over-spec'ing the solution a bit

so it could safely run twice as long on cheaper hardware

@voretaq7 heck, ten times as long

if it crushed for an hour... who cares?

@RyJones eh you want it done by 8AM so work can start promptly at 9

@voretaq7 I have no evidence of anyone using this software... ever!

@ewwhite oh... then turn it the fuck off and see if anyone notices! :)

@ewwhite I've gotta say I'm impressed. Most contractors would just take the spec sheet, price it, add a nice margin and be done with it. Not too many would look at it and say "Hey, you're wasting money"

the graphs above are from virtual machines.

@MarkHenderson @ewwhite hasn't had his ethics removed yet.

@MarkHenderson well, considering this is an analytics server that's has specs far higher than the main application server... I think it's pretty sick

You really should Ed, an ethics is a pesky thing to have to live with, and the removal procedure is so simple: You just hang around in the Sales and Marketing department for a week.

so I'm trying to combine this analytics app onto my main server using LXC containers

@voretaq7 Oh, I think nothing of marking a server up $3-5k.

free monies.

@ewwhite you need to overspec and then mark up by a percentage. THAT'S how it's done in Sales and Marketing my friend :)

libcrack.... my ubuntu installation is on crack. this explains so much.....

("never watch the upgrade messages")

@ewwhite I suspect a dev just wants some toys to play with

@MarkHenderson but given than profile, virtualization/containers seems like the right way

@MarkHenderson At $job[-1] I used to take new developers into the datacenter to show them the shared hosting server they were working on. Most of them were working on ColdFusion apps, running on a machine that was (then) about 8 years old.

@ewwhite For sure. Very little needs to run on bare metal these days

@voretaq7 Gives them some respect for the hardware?

@MarkHenderson I found that humbling them by showing them that a properly configured server, ancient though it may be, could have performance far in excess of the "dev sandbox" machine they "administered" in their side of the office was very effective in explaining to them why they should kiss the ring when they asked the systems groups for things.

at Microsoft (on Windows, anyway), the dev machines were insanely highly speced so they could rebuild Windows in a day if they had to.

We have clients who want to tour our datacenter but we can't let them at the moment. We're hopefully moving after easter.

"I can support 50 clients running YOUR CRAPPY INEFFICIENT CODE on a 500MHz Pentium with 2 gigs of RAM. And they're all impressed by the speed."

@voretaq7 haha

(of course some of that was because systems would step in and be PROPER DBAs before we let the dev folks roll their shit out -- INDEX MOTHERFUCKER - CAN YOU USE IT?!)

Secretly you took a Dell R910 with quad-octo-core processors and 512GB and a rack full of SSDs and stuck a Pentium Pro sticker on the front

@voretaq7 Except I discovered too late that you can have too many indexes

naw it was a Dell 4300 - you can't fake "Three power supplies in a Barney-purple steel case."

"What? This table with 20 rows and 10 columns has 9 indexes?"

@MarkHenderson it's hard to have too many indexes - if they're the RIGHT indexes.

I have 2 and 3 row tables with 3 indexes on them

@voretaq7 sp_blitz would probably complain about that

(one on each column referenced in a foreign key lookup, and one on the pair comprising the primary key which also gets searched sometimes)

brentozar.com/blitz

@MarkHenderson sp_blits can lick my shweddy ballz - if I'm doing lookups on both columns independently I'm damn well doing an index rather than a half-million row sequential scan :)

the key to using sp_blitz is knowing when it's wrong :)

@voretaq7 Its quite useful. It found a massive heap that plenty of indexes but not a single one of the indexes was actually being used. And being a heap, no clustered index.

Has anyone ported blitz to Postgres yet?

@ewwhite interesting. Never heard of that app before.

@MatthewIfe which one?

Pentaho?

@ewwhite how come they need so many instances, do they analyze different things?

Pentaho

@MarkHenderson Clustering is "special"

@voretaq7 True

Like any turnkey tool I guess

I really need to cluster some of our tables one day

@MatthewIfe there's only one instance.

but the vendor is asking for impossibly-large hardware and can't justify it

@voretaq7 I scanned 890 tables in our database and only found 10 clustered indexes. Needless to say, that got fixed in two days of ungodly "drop index" "create index" statement writing.

@ewwhite "I own shares in {Dell,IBM,HP,SuperMicro}!"

@MarkHenderson ...

What are the containers for then? Or is this for something else?

I also only have myself to blame, considering that I'm the one that set the database rules

So I'm trying to make a decision as to whether I should combine the Pentaho onto their main application server, which is beefy

@MarkHenderson gotta be careful when you CLUSTER - it'll mean about a 20 minute outage for us because a LOT of rows will need to be moved around.

sp_blitzindex?

@voretaq7 I don't run it in the middle of the day

So I'd put Pentaho in an LXC container

I scripted it all in a job to run at 2am

I only do schema changes on weekends :)

@voretaq7 enterprise sql bitcheeeees

@pauska I haven't run blitzindex yet

Just the generic blitz

@MarkHenderson why not?!

@pauska Oh Postgres will continue to support access while it's clustering without a problem

oh

get blitzindex

chances are nobody would even notice

and

@pauska No time. Two devs down.

but I always schedule outages for it in case the performance goes to shit

@MichaelHampton Quick LXC question... back in the day, I'd use OpenVZ OS tarballs to seed the containers. Has that changed?

@ewwhite would they notice/care?

ola hallengren's maintenance script

@MarkHenderson time? It takes 2 seconds download, and then you run it..

@pauska Yeah and then how much time to go through it and review the problems?

@MatthewIfe they wouldn't, except they're asking for:

Does the application have any kind of real-time requirement?

Anecdotal Postgres-kicking-ass story: Our last DB upgrade had a shit-ton of columns dropped so I did VACUUM FULL - I did it after bringing the environment back up and none of the folks I had testing even noticed.

nope

@MarkHenderson You'll spot immediate problems right away

@MatthewIfe It's just ad-hoc reporting software

heh.

it gives you a score of potential improvements

working off of canned data cubes

@pauska Right now I have bigger fish to fry than database performance to be honest

@ewwhite C-Ration Data?

@ewwhite might be even simpler to just run that thing in a cgroup.

I bought a heap of time when I went through blitz at the beginning of the year

@MarkHenderson same here - but I like optimizing databases

@voretaq7 I do too. But I can't prioritise it at the moment.

@MatthewIfe It's Postgres-backed... so I'm wary of mixing it with the main application. Plus security/auth considerations.

I've got a list of un-released features as long as my arm that need reviewing and signing off

Postgres covers so many of our sins it's ridiculous. Especially now that we no longer use our 6-billion-row x 512 column (at the time I dropped it) decomposed test sample table for anything. Pretty much everything fits in the database workmem or the UBC

Also two of our customers have grown beyond the limits of the infrastructure they were put on much quicker than anyone anticipated, so I need to deal with them too

about 24 hours after spinning up everything is cached somewhere and only writes hit the disk :P

@ewwhite whats their distro?

And to top it all off, I need to write new update/deployment procedures because we now have two clients who start work at 11pm and finish work at 5am which destroys my entire maintenance window

@MatthewIfe EL6.5

(DB01 was "appropriately sized" based on the crappy, bloated DB schema. Now it's almost as ridiculous as the machines @ewwhite is describing :-P)

@MarkHenderson Fuck them with an outage pole :P

I deployed Windows Updates on Sunday morning and had 10 emails by the time I woke up complaining about the outage

you don't have your maintenance windows in your contracts?

@voretaq7 More like I move them onto their own infrastructure that is updated out of sync with everyone else

@voretaq7 Yes, but they don't give a shit

@MarkHenderson oh we charge for that :-)

They probably didn't even read it

@voretaq7 Yeah... I don't get to charge more for it

It's not a big deal once it's up and running

It's just getting it running that's the problem

Two devs down, everyone else at a trade show in another city, and a workload as long as my arm, and a baby due in just over a month

@MarkHenderson In turn, I don't give a shit about their complaints until they pony up $250k + $100k/yr maintenance fees for custom cluster management. (I should feel bad about being so brutally overpriced, but these are mega-hospital conglomerates. Pay Up Suckas.)

@ewwhite hope that it works. Containers are so much more efficient.

so far nobody has taken us up on the custom environment -- I REALLY want someone to

@MatthewIfe than what?

Full virtualization.

@voretaq7 These are people that can probably afford that, but they don't buy Porsche Panameras by actually paying that

Oh, of course.

(and yes @Jacob, the Porsche Panamera is an ungodly ugly car)

@MatthewIfe One thing I don't know about LXC is whether anything can be changed on the fly.

@MarkHenderson @jacob Most new Porsches are pretty hideous...

Define 'anything'

@ewwhite like RAM allocations and stuff? Usually yeah.

@voretaq7 New GT3 looks nice. When it's not on fire.

I think I have to destroy the container, reconfigure and boot again

they can be in FreeBSD jails so I imagine containers aren't much suckier.

RAM/CPU limits

I need to read up

Although if I had the money I don't think I'd be brave enough to actually drive one

@ewwhite yeah, well you can.

@MarkHenderson I'm not a fan of expensive cars - people on this island treat other people's cars like shit, so I'm happy in my $15-20k range VW with some performance mods

@voretaq7 Yeah, you wouldn't want to take it to a shopping center car park

The 'ram' is actually a cgruop and /proc/meminfo output is done via some hack (or perhaps bind mounting memory.stat in the cgruop -- cant remember)

cpu is a cpuset cgroup so its changable on the fly.

I saw a GT3 at a shopping center once and the dude had parked in the middle of two spots. Total dick move, but I understand the temptation to do it

Then again I also treat my car as a 10-year investment and take excellent care of it

@MarkHenderson I would have keyed it.

@MatthewIfe I'll play around with it.

@ewwhite just googled, /proc/meminfo is a mounted fuse hack to represent the memory as /proc outputs it.

--- /opt/pentaho/biserver-ce/tomcat --------------------------------------------------------------------------------
           /..
   20.2GB  /logs
  163.5MB  /webapps
    6.0MB  /lib
    1.7MB  /bin

Seriously, no. Park alllllll the way the fuck at the back of the shopping center away from anyone else - fine. Park across 2 spots right NEAR the shopping center? Fuck you, you park like an asshole, and you're lucky I didn't have a hammer.

@voretaq7 That would be ironic if someone did

@voretaq7 The guy probably owned the shopping center

then he can paint big red boxes next to one of the parking spots in the back 40 and hire a worker in a golf cart to ferry his entitled ass to the door.

the newest kernels have user namespaces too. Works in fedora 20

asshole parking just makes me hate the person and wish bad things to happen to their vehicle. I need to get a book of those "YOU PARK LIKE AN ASSHOLE" stickers.

@voretaq7 You can get some really nice classy ones

Embossed business-card style

Someone makes ones like that, on removable stickers

I'm also tempted to get this (Thanks Google Images):

@voretaq7 I like that one

I diagram is worth a thousand words

@MarkHenderson I personally think there should be several diagrams that you can circle, indicating the jackassery in question, and the "Correct" parking should be on the reverse, with circles, arrows, and a paragraph explaining HOW TO FUCKING PARK BETWEEN TWO GODDAMN LINES PAINTED ON THE GROUND.

There should also be ones for parallel parking with marked spots, with meters, and unregulated

@voretaq7 the problem with parking, is that once one person parks badly, other people also park badly since they have to deal with the existing badly parked car. You can't always blame the most recent person you have seen.

@Zoredache I blame all of them. You skip a space if you have to, you don't fuck up the entire row of spots so that when people leave there's this gap-toothed assemblage of misparked cars blocking up half the lot.

@voretaq7 Not always a choice. If there's only one visible park left I am taking it

Regardless of whether or not the car next to me is parked appropriately, or if the spot is marked "Motorcycles Only"

I will fit my full-sized wagon in there

You're getting a sticker, cat!

(Also: You're a cat. We expect you to be an asshole!)

Oh, this is meta... A recruiter from LinkedIn just LinkedIn-me and messaged me about a role working for..... LinkedIn. Maybe now I can figure out the algorithm that produces the People You May Know recommendations!

@ewwhite: monkeys with dartboards.

@JourneymanGeek possibly...

@ewwhite who was the recruiter? A guy named Jerry Fan perhaps?

@Zoredache let me look

I wanted to check in with you to see if this might be a better time for you to explore an engineering role here at LinkedIn. I understand a colleague of mine, Peter Kim, had previously contacted you regarding an opportunity but it appears the timing wasn't right.

Yes, a Jerry

... If you're open to considering other options at the moment, I'd like to set up a time to go over the position that I had in mind for you. I support LinkedIn's Site Reliability Engineering team and I thought your experience with maintaining Linux servers ...

...would make you a good fit for the team. Please let me know if and when you might be available to talk, or at the very least, Suck my ASS!!!

I wonder if he is spamming anyone based on being serverfault.

@Zoredache I didn't get spammed :(

Only inkling of a job I've had from SF was when Zypher asked me if I would interview for Fog Creek and that was 2 years ago

It's nice to know you're wanted

And even @MDMarra got that offer

But you have lots of things in Australia that can kill you.