@DennisKaarsemaker you like other distributions?

Like what? Funtoo? Arch? SLES?

Debian

@MarkHenderson what's wrong with the server?

It rebooted after a lefthand software upgrade last night, and just never came back online

Oh, it's a DL180...

I have plenty of spare memory that will fit in there, just need to make time to go to the clients office

so hmm... do you have physical access?

@ewwhite I do, but not right now

I'm about a 45 minute drive away with a full schedule today

oh wait.. what is the node type... DL380? It's G6-era

you should be able to see RAM status in the ILO config

You lost a DIMM. It's no longer valid since you have a 2-CPU node.

PROC 2 DIMM 2...

@ewwhite Nope only one socket populated

can you verify that it's a 2-cpu node?

anything useful in the IML?

Well we might have lost a RAM, it might have been meant to have 8GB

I'm not actually sure

@DennisKaarsemaker Apart from that message, the last message was from 2012 when a PSU lost power

hm

PROC 1 DIMM4 is the place to check, but it should be able to boot with one DIMM

@MarkHenderson and the health overview thing?

I forgot where that's is in the G6 webinterface

@DennisKaarsemaker I don't see it

"System Status"

That's the one I posted first

right, nothing more comprehensive... except for IML and ILO2 log, and the DIMM status page

@ewwhite Pretty much

I'm just trying to find someone close who can go in there and swap out the DIMM

I know there is modules in the DC that are spare

nothing useful in the get_embedded_health xml call either, they added that in ilo4 only

Well I'm waiting on HP to get back to me on the next course of action. Otherwise I will just have to hike over there and start troubleshooting 101

It looks exactly like this; except the config of the server hasn't changed

h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/…

sup

does anyone know bacula?

@MattBear I think... @voretaq7?

@Wesley well... hopefully he wakes up :p I'm stuck

@MattBear I've used it

didn't like it too much

I know a little bacula

I can't get the director to connect to the storage daemon

@MattBear ...ok it shouldn't have to anyway :-P

is it throwing an error?

when I run #status storage I get an error

@ewwhite Just checked the original manifest; there is supposed to be 6GB of RAM in each of those nodes, so it looks like the 2GB DIMM has shit itself

@MattBear do you see the connection on the SD? might be a mismatched key/password/whatever-they-call-it

JobId 0: Fatal error: bsock.c:133 Unable to connect to Storage daemon on 10.0.1.14:9103. ERR=Connection refused

@MattBear ah, that sounds like a network error :)

IDK what port it uses offhand but can you telnet to it?

IIRC all the daemon chatter is TCP

you would think... but all ports are open, and thats the machines internal IP

Is the SD even running?

yeah

all daemons start, director, SD,

bconsole connects to clients, and director

Moving DB for Vcenter to MS SQL 2008. That's going to be fun.

Is Hacker News down again? This is too good to be true.

@DanilaLadner Throw it out and install the vCenter appliance! (Linux and Postgres)

Awww, it just loaded :(

@DanilaLadner Also, do you know Josh Storck?

@ewwhite I did bring it up on the meeting

yo

yeah I worked with him.

@DanilaLadner After everything I went through in December, that's what I ended up with

The vCenter Appliance is what VMware is pushing people towards.

So where can I read if all my current stuff will migrate seamlessly to appliance.

and honestly, it does work better.

He worked for two sigma for quite some time.

@DanilaLadner tell me about the setup...

@DanilaLadner yeah, met him at a two sigma friend's wedding

he just moved back to Chicago

we have 2 datacenters

How many hosts?

1 datacenter has 4 clusters

are you on 5.1?

so 2 clusters for one phy dc (1 old cluster with 10 ESX4 hosts and 2 cluster with 8 ESXi5.0 hosts) same repeats for other DC

The new appliance runs 400 vSphere Hosts and 4000 Virtual Machines

in about 32GB of RAM

no plan an upgrade to 5.1 for 5.0 ESXi in spring

hmmm... moving slowly.

its yale man

okay, for 5.1, you can stay with vCenter and the SQL Server DB

but if you do 5.5, you really need to go up to the appliance.

i mean 300 years of history, 10 years here, 10 years there, no one cares -))

you'll still need Windows to handle some side functions...

like separate win server for something

For Upgrade manager

or just win VM for Vcenter Client?

actually, 5.5 has a web interface... flash

oh it comes separately not as plugin?

Upgrade manager needs to be on windows. The Linux appliance handles everything else.

@DanilaLadner I'm reaching out to Seva because I would like to be doing what he does.

I miss it

He is in JP Morgan now

@DanilaLadner oh, I thought he was still consulting.

he says he hates it, give it a year

ha!

@ewwhite Fucking flash

I didn't realize we had so many connections @DanilaLadner

well, bureaucracy and shit.

he went as Linux Tech Manager/Lead, they gave him a f XP

haha

took him 2 months to get a linux desktop

big scary bank

he literally told his boss, i do not know how to use this computer.

You know, peoples 'stupid' questions dont irritate me. But people who know a bit, then make crazy assumptions based on it does annoy me.

@MattBear hmmmm

@MattBear and you can telnet to it?

@voretaq7 I've been messing with it for a couple days...

might be some SELinux shit going on (or I might be just cranky and like to blame SELinux :P)

@ewwhite You decided not get after fin full time opportunities?

@voretaq7 yup

I've decided to get RHCE recently, I still think all certs are shit, but just want to get a marketing bit to the profile

@MattBear . . . DaFUQ?!

aside from "network" and "make sure the names / passwords match" I'm stumped :-/

and this SELinux shit comes in, and i get all thrown off.

@voretaq7 it's gotta be something I'm missing in the configuration

@MatthewIfe Doesn't understand subnets.

@DanilaLadner "DISABLED" -- problem solved :P

all i did with it for all the years was SELinux="DISABLED"

yeah, they have an extensive tasks on exam for it

@DanilaLadner yeah, I'm on my own now

@ewwhite I definitely would go on my own, but my wife freaks out with wtf it a good job, you are f emigrant be happy you have this and shit

@voretaq7 all else fails... try a reboot

@84104 Its like trollscience for serverfault. Doesn't understand ip aliasing (or that its been deprecated for years), doesn't understand linux routing, why is LD_PRELOAD anything to do with it? Love to know how he reckons he'll ever process the packet if he ever got it back as removing it from the local routing table makes linux consider it an IP for forward.

@MattBear Windows User.

@voretaq7 guess what...

it fixed it

@MatthewIfe i do not understand what he tries to do there and why?

@MattBear . . .

I quit.

fuck the internet

fuck unix

:P

and fuck bacula in particular

I've been working on this for 2 days...

@voretaq7 I thought we hated Bacula now

@MattBear This cat adequately captures my feelings about the world of system administration

@MarkHenderson yeah, I haven't gotten around to migrating to BaReOS yet

@voretaq7 haha

but it's still "bacula" at the core :P

oh, I negotiated myself a raise :)

@voretaq7 I need to point out thats possibly dangerous waters.

I am wondering what I can convert my sys admins skills into

@MattBear ...by fixing the fucked backup environment with a reboot? :)

without getting a huge pay cut

@voretaq7 theres lawsuits pending.

@MatthewIfe howso?

@voretaq7 haha no

@MatthewIfe meh, the stuff was published publicly under the Bacula license - legally it's "No Backsies"

by saying "I'm tired of being broke, here's my two weeks"

you can't publish something under, e.g. the BSD license, remove it from a later version, and say "AND ANYONE WITH THE OLD VERSION? NO. YOU HAVE TO DESTROY IT. YOU CAN'T USE IT" -- the genie is out of the bottle, and they're covered by the prior licensing

@MattBear Yay. You bailing on them?

the CIO went to the owner to get me a raise, in order to keep me

@voretaq7 BSD licensing is the best licensing.

@Wesley Public domain, motherfuckers

@Wesley nope. they more then matched the offer I got, I'm sticking around awhile

we'll probably be on Bacula until at least Q2/Q3 though because I've got no time to teach radmind that a bunch of files have been renamed :P

If it's written in Javascript, I consider it released into the public domain

BSD: Free as in "WE DON'T CARE JUST GO DO IT"

@Wesley BSD: Free as in "We don't even want credit anymore, just leave us the fuck alone and go write some code."

but I generally agree, there is no license more "free" than the short-form BSD license

@voretaq7 I can tell you now thats not the case.

@voretaq7 Free as in "We don't eat our toe jam."

@voretaq7 theres code in the enterprise bacula I know to have been pilftered and it was not gpl licensed when it was taken.

f java is like syphilis in IT

@MatthewIfe well then that's something for (a) Bacula Systems to prove, and (b) the courts to hash out :)

This: github.com/bareos/bareos/blob/master/src/dird/quota.c was added with a special license from the copyright holder to go into enterprise only. It was never meant to be gpl.

Honestly it doesn't affect me - I don't need any of the "contested" features and if they have to do a BSDI-style rip shit out to be legal I'm not gonna cry over it. What I do need is fucking active development and bug fixes - and Bacula has kinda stopped providing those in the open-source edition :-/

I know cause I wrote it and worked with bacula to get it eventually into bacula enterprise.

omg now HP are making me send the logs from the Lefthand CMC. It's a fucking hardware issue!

@MatthewIfe then you have a cause of action against the folks behind BaReOS

Well dammit, my blog post got posted too early by accident, and I undid it but stackbot picked it up within 60 seconds

Symantec just sent an evil packet over the network that simultaneously BSOD'ed every one in my office all at once.

@voretaq7 I dont. UKFast owns the intellectual property and I dont work there anymore ;).

So it's not live anymore, but will be live in its fullness tomorrow.

I want everyone to star my blog post when it drops in here. =P

Lesson learned: don't shoot my load too soon.

@RyanRies ...how?

That sounds badass!

@MatthewIfe then UKFast has a cause of action :-) either way, that particular chunk of code isn't important to me, so like I said, if they do a BSDI-style "These 6 files must be baleeted" I'm not going to cry. (Hell if I have to baleet them myself in my own git repo I don't care)

I want to hose everyone's PC with one packet. whines

@voretaq7 Dont say I didn't warn you ;). Also dont use an autochanger. That code is broken.

I'm just particularly cognizant of the fact that I'm running... I think it's 3 years old now? software with bugs that actually bite me in production, and Bacula Systems seems to have abandoned anything that's not Enterprise.

@MatthewIfe 's broken in Bacula too. But I use virtual tape files so I don;t care :P

@Wesley I still have some analysis to do on the crash dump, but we all crashed at the exact same instant, and the crash dump says Teefer.sys (which is the antivirus packet filter component of Symantec AV) tried to access memory address 0x0 which crashed the system.

Yeah, not saying choose one or the other. Saying dont use an autochanger ;). We use an autochanger on our virtual tape files too, days of having to look through multi-threaded nightmare code to figure out all the damned race conditions.

But here's the thing -- it crashed the Cisco phones too... so maybe it was just power surge or something? I'm not sure.

But if it was a packet - I so hope I am able to reconstruct it from the crash dump

So I can use it for... uhm... research

If Bacula Systems starts updating their shit again (like they claim they will) the point is moot IMHO

@MatthewIfe why do you use an autochanger on your virtual tapes? Disk is so fast :)

What I really need to do is rewrite the remote-sync-and-locking routines we use (basically our "tape unmount" code) - every time a tape fills our backups block while it gets sync'd to the remote storage dump

but that's our own internal shit that I wrote in an hour and haven't revisited yet. Stupid shell script....

@voretaq7 we do that because we want to be able to the same storage daemon 'device' for multiple jobs.

Otherwise you need a new 'device' and volumes/pools for each one for each client and/or job you want to run in parallel.

@MatthewIfe yeah - I spool everything in parallel and then despool to the virtual tapes (which is the big serial bottleneck, but we're only moving like 100-150 gigs on big nights)

@voretaq7 you can do it using one tape device but it interleaves the jobs block by block, it makes restoring the data or extracting it in desperate situations painful.

I'm shifting 12TB a day :(.

now if Bacula Systems fixes the autochanger code in their open source edition that might motivate my ass to stick with them - it would be a nice improvement.

@voretaq7 Itll be fixed soon. Cause I submitted a patch. I'm sure its not finished being fixed though cause i've seen another locking bug today.

@MatthewIfe that's really why I spool: the despooling locks the tape device so it's one spool chunk (20GB) then the next, then the next

@voretaq7 if you do it in parallel in bacula it doesn't work like that, normally one block on the volume is dedicated to one 'session' or 'job'. If you dont mess with that value the block is typically 64512 bytes in size.

i am out

good night all

@MatthewIfe 20 February 2013: Bacula 5.2.13 has been released. <-- Bacula Systems and I have different definitions of "soon" sir - a year without a patch when I'm getting bugged by bugs :-/

I know way too much about this because I'm having to write a whole bunch of code to verify whats in our volumes matches whats in the catalog (which bacula have told me due to the bugs mentioned above is unlikely to be the case)

@MatthewIfe the spooled behavior is as I noted above - IDK if that's intentional, but it's the empirical behavior at least in my environment and it's desirable :)

@voretaq7 I dont know what baculas intentions are aside from the fact that they said that my patch was 'very interesting' :)

Yeah you can spool really quick and dump to tape. But the volumes that go to tape are likely interleaved jobs.

@MatthewIfe "we'll include it in the enterprise edition. Thanks and give us some money if you want to use it" seems to be their general attitude lately

@voretaq7 Well, in real terms I dont think they'll do this here, its a correctness patch not a feature patch.

Leaving bacula as is means people using autochangers have corrupt catalogs.

@MatthewIfe they definitely interleave if a job is longer than the size of a spool, but my spools are "generously sized" proportional to my data - our biggest machines might be 5 spools, so they're spread across five bulk-writes :)

Ah right do you spool to different locations per job? I suspect that thatll be done sequentially. Only where multiple jobs go to the same spool you get interleaving.

@MatthewIfe I'll give them credit for doing the right thing when I see them doing the right thing -- like I said there are lots of stupid annoying bugs (like the director locking up if you query a SD or FD that's "busy" and doesn't respond) that I've been told don't exist in Enterprise

@MatthewIfe yeah we have per-job spools, and a cap on the max amount of spooled data

so when a job is done spooling that spool gets written out as one bulk dump to the tape

(again, not sure if that's by design or not, but it's what happens and it's what I want so gift horse, mouth, etc. :-)

@voretaq7 well it technically exists less in bacula 5.2.13 in theory too because they removed the device locking and replaced it with volume locking. Albeit now they've introduced a whole bunch of autochanger bugs cause of it ;p)

@MatthewIfe <APPLAUSE>

the FD locking is normally caused by running out of worker threads. Probably could do with upping your max clients limit and that should go away.

yes, break my backups. They're not enterprise critical or anything...

@MatthewIfe the FDs aren't usually my problem - if I do status storage while our unmount command is running though the director lays down and dies until the unmount is done

Max concurrent jobs in the dir.

Yeah - device locking causes that.

(and depending on how much it has to push over the network on the sync that can be a long-ass time)

yeah theoretically volume locking would solve that problem

backups. lol

@voretaq7 what i really really want from the enterprise is sd to sd migrate jobs.

Really would give me a semi right there.

@RyanRies NSA Misstep

@MatthewIfe that would be useful - not in my particular environment, but generally speaking :)

I've LOVE to backup more clients but would want local storage daemons to do it, then migrate to off site locations (or go full crazy and do virtual fulls weekly and ship them to another sd)

I need to spend time with virtual fulls - it would REALLY cut down on the first-friday-of-the-month crazyness with our monthly full.

Yo, @ewwhite, why you creepin on my wife!!

;)

NSFW: Pornhub comments on stock photos

The photos are perfectly SFW. The comment text overlayed on them? Not so much.

@voretaq7 Oh god, love the spelling and punctuation

I particularly like this one

I laughed so hard at that one

Seriously; how do you fuck up an apostrophe so bad? One handed typing I guess...

@MarkHenderson well they are comments from a porn site

(but seriously, you're gonna comment while you're watching? Kinda defeats the purpose no?

That one makes me afraid to look up "ping pong" on urban dictionary...

Todays productivity: Out the window

@freiheit I assumed it was actual ping pong

@MarkHenderson OH YOU'RE WELCOME

@MDMarra I'm thinking that it's better to not verify that assumption

@MDMarra @freiheit I've seen people do.... things with ping-pong balls. Hashtag-#ImJustSayin ; Hashtag-#WhatHasBeenSeenCannotBeUnseen

@voretaq7 Hashtag thailand

@voretaq7 Does it involve tossing dirty ping-pong balls into a beer somebody else has to drink?

@freiheit no, and those are clean. Alcohol is a great disinfectant.

Better get this over with

nice

dooo ittt

Isn't it all 2012 these days?

The 2008 exams are around until July 31st of this year

Ah right.

Get those exams in before you get metro-slapped.

s/metro/modern

They were sued for using Metro