The Comms Room

@MDMarra Still well-compensated here.

@Wesley Freakin' rich kids. :P

I think @MDMarra wants to be my life-coach.

Yeah, you need one

12:02 AM

I'm in the sysadmin twilight zone...

I'm sorry but I think you miss out on an important character-developing life lesson if you're not subsisting on ramen and that low-rent cereal that only comes in bags during college.

@RyanRies Hey man that cereal is just as good.

don't you badmouth my bachelor chow.

@MDMarra this one is right up your alley...

Similar situation as pr0n-db1 -- We're seeing SNMP failures from pr0n-db3. As a preliminary in the troubleshooting process, I would like to schedule a reboot of this system at your earliest convenience. Please advise.

@JoelESalas Marshmallow Mateys for life!

On our Windows servers, SNMP sometimes fails... so we reboot the systems.

...

12:04 AM

@ewwhite Why not use WMI

@ewwhite Microsoft's SNMP engine sucks hard. I avoid it at all costs.

Our monitoring software needs SNMP... which shouldn't be a problem. I used to use snmp-informant.com, but either way, the failure of the SNMP service shouldn't require a reboot of the server, should it?

It's very obvious that Microsoft has had no interest in supporting SNMP for a while. Hence they never developed their own SNMP v3 engine and the entire mess is deprecated today

SNMP on Windows is bad

@ewwhite Seems extreme. Restarting the daemon doesn't do it?

12:06 AM

And if you use it you should feel bad

service, sorry

NetSNMP on Windows is slightly better

@JoelESalas When they restart it, it doesn't come back up.

But leaks memory like crazy making it not a real option

@ewwhite No of course it shouldn't require a reboot

12:06 AM

@ewwhite Have you ever tried NSClient++?

@ewwhite what do you monitor with? It doesn't support WMI or have a native Windows agent?

One of your servers: WEB1.BIG-UNS.LOCAL is having issues sending SNMP metrics over to our monitoring system. I've personally looked into this issue and it appears the snmp service is experiencing a '1067' error (service terminates upon trying to start) -- Is there a good time/date where we can reboot this machine to begin troubleshooting?

@Zoredache know where I can go to do some more research?

@MDMarra We monitor them with... this.

@JoelESalas never tried.

You use all this shit I've never heard of

12:10 AM

@ewwhite Seems like a slam dunk

I think we paid a few hundred thousand for it.

I'm sure it's legit, but it's crazy how many environments you can see and still find things you've never even heard of

@MDMarra I think we're their largest customer...

is that good or bad?

So a quick google of "SNMP 1067 error" shows: blogs.technet.com/b/networking/archive/2010/06/24/…

not sure

@ewwhite LW is pretty big, I wouldn't be surprised. Though at LW's size, you'd either be 100% FOSS or using the HP/IBM/whoever tools

12:13 AM

@JoelESalas oh god, keep it to yourself

I just talked him off (or maybe over) the ledge.

@MDMarra which ledge? ;_;

Zoredache

@MattBear When said nothing is written to the server, does that mean this is not using Mysql/Postgres or any other database server?

@JoelESalas I think we have what's called client/patient confidentiality.

@Zoredache we have a mysql server also, but are not using database cookies

12:15 AM

@MattBear DB cookies would be the easiest, it's basically designed for that

but you WILL get a scalability issue with most RDBMSes

@ewwhite Drugs or distraught?

where you'll either have to shard on the UID or extract the session from the DB and put it somewhere else like redis

Zoredache

@ewwhite whats going on there. Is that a RL life Bioshock Infinite thing, or?

@Zoredache Photographer Juxtaposes Modern Streets with Olden Crimes - ny.curbed.com/archives/2013/09/13/…

12:19 AM

@JoelESalas what would be best/most secure?

@MattBear Most secure is a whole different can of worms, that'll be impossible to get right without an appsec consultant

@RyanRies So it's odd that SNMP crashes and can't be restarted, right?

@MattBear best would be redis, because sessions are meant to be mostly stateless. There's no real persistence required. They expire. So a key-value store is a great fast choice

@JoelESalas oddly enough, thats part of my job description lol

Q: Should Server Fault stop accepting questions from unregistered users?

0

If you've read the front page recently, you're well aware that there's a large percentage of total crap on it. Indeed, more than a third of the questions posted in the last 60 days have been closed or deleted. I'm led to believe this is quite a high number among Stack Exchange sites. This post i...

Q: PHP sessions in a load balancing cluster - how?

12:23 AM

heeeey

17

OK, so I've got this totally rare an unique scenario of a load balanced PHP website. The bummer is - it didn't used to be load balanced. Now we're starting to get issues... Currently the only issue is with PHP sessions. Naturally nobody thought of this issue at first so the PHP session configura...

php session load-balancing cluster-computing

@ewwhite Yes that is odd. I've seen runaway memory leaks in Microsoft's SNMP service before, but I've never seen it crash and not be able to be restarted before.

@ewwhite I ran snmp on Windows for cacti for years without a single problem

that could of been me posting that lol

it's not idea, but it's also not that unstable

@MDMarra exactly... So I think we're doing it wrong... again.

Zoredache

12:24 AM

@MattBear The answers there aren't great. It is a bit disappointing that the accepted answer is basically a link-only answer.

@Zoredache is this devops....

Ry Jones

@MattBear THIS IS SPARTA

@Zoredache That's a terrible question with a terrible answer

the correct way is not sticky sessions on the LB

So, PHP / MySQL heavy app, 220 active visitors on the site right now, averaging a little over 2 page loads per second:

Mem:   3917412k total,  3336852k used,   580560k free,   142212k buffers
Swap:  4128760k total,      112k used,  4128648k free,  2552040k cached

@Wesley Don't use apache :)

12:28 AM

@JoelESalas Oh you.

@Wesley Not too unreasonable. What's your CPU look like?

@MichaelHampton Barely moving.

Uno secondo

You must not be using Drupal.

top - 17:29:07 up 157 days,  7:13,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 281 total,   1 running, 280 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.5%us,  0.7%sy,  0.0%ni, 96.3%id,  0.4%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   3917412k total,  3317324k used,   600088k free,   142216k buffers
Swap:  4128760k total,      112k used,  4128648k free,  2553092k cached

model name : Intel(R) Xeon(R) CPU X5365 @ 3.00GHz

157 days uptime?!

12:29 AM

@JoelESalas redis it is

@MichaelHampton Shuttup.

=P

Let's see.. Kernel bugs in the last 157 days...

@MichaelHampton Maybe I use Oracle Linux! It's unborkable!

@Wesley So you actually DID buy a bike!

Any iPhoney users do an update to iOS 7 GM from a previous 7 beta? Did you have to drop to 6.1.x first?

12:34 AM

nope

straight upgrade

apps, music, etc all in place

you need itunes 11.2 beta 2 though

other wise your phone wont connect to itunes post-upgrade

you can still upgrade without it, you cant cant sync after

@MDMarra Very, cool, thanks!

no problemo

I hate when Win7 taskbar refuses to auto-hide. What's up with that Windows d00dz?

Auto-hide? Is this 1999?

2

I hate when a window ends up behind the taskbar.

12:39 AM

@MichaelHampton This is what is happening. Worst part, I have the task bar at top, so window chrome is obscured by the refusing-to-auto-hide taskbar.

Wait wait wait

You have an auto-hiding top oriented taskbar?

@MDMarra Yes... and? :)

@MDMarra Yeah, he's a top.

FREAK

I think you just blew @MDMarra's mind.

12:41 AM

All the window chrome appears at the top of, uh, the windows. Why put the clickable taskbar, quicklaunch, start button all the way on the bottom where there are no window controls?

The LDAP/PowerShell ninja that I seek advice from is an auto-hiding topper. My world, it is upside down.

Wat

I keyboard shortcut like a mofo, but I hate the bar obscuring the bottom of windows.

@MDMarra So you're running iOS7?

ya

Since the 10th

12:42 AM

@MDMarra Do I have to use a Mac

@MDMarra gimme!

Post-install if you want to sync with itunes you need 11.2 beta, which is only out for OS X

but if you just wanna upgrade and say fuck itunes syncing for a week, you can upgrade on windows

@MDMarra Wooooo!

@ewwhite iphone 5?

Yes.

12:44 AM

ATT or other?

ATT

so it's a 1428?

1428?

the model

it's in tiny print on the back

A 1428

12:45 AM

mega.co.nz/…

doesnt work in safari, use firefox or chrome

Why is my computer laughing at me?

oh

you lose

It's all over Pirate Bay, with hashes

Now my wife wants to cuddle...

:(

awwww

12:47 AM

@ewwhite And that's why your computer is laughing at you.

@ewwhite mega.co.nz/…

will my iPhone fry?

mine didnt

how does the update even apply?

you open itunes and hit "update" while holding option

instead of checking online for an update, it will let you select a .ipsw

12:49 AM

I thought you said a beta was needed.

only if you want to sync to itunes after

...

I can wait.

it preserves your apps and music

But I sync things.

so unless you're going to put new stuff on your phone through itunes before the 18th

So get itunes 11.1 beta 2

it's all over the Internet

12:50 AM

Hmmmm "Phone could not be restored.... Error 2003" That's a new one.

@MDMarra Man, this is worse than scrounging for parts to stuff inside of a Supermicro chassis...

I stopped updating my iPhone at iOS 4.3.2 because I can't bring myself to install iTunes, not even on a virtual machine.

What do you sync all the time?

@jscott don't restore!

Do update

@MDMarra Podcasts, music...

@RyanRies Funny, iTunes is probably the top reason I don't have an iPhone at all.

12:51 AM

Nekkid pix.

Oh, boy now it's gone all DFU on me. :)

@MDMarra Yeah I tried shift+update first, it failed. So then I figured, f-it, let's kick it a restore.

I don't think a restore will activate

not 100% on that

My server is hosting a little Facebook game. The total amount of income from players is EUR 150. The server costs EUR 50 + I have some more expenses. Could you guys please stop chanting "switch the hoster", because I'm actually happy with it and am willing to take a reboot every few weeks? I just need to configure the watchdog properly, so that the server restarts by itself. — Alexander Farber Sep 10 at 9:00

Can I just respond, "you're doing it wrong?!?"

@MichaelHampton Yeah the iPhone 4 was my first iPhone, and I won't be buying another. Not that they're bad, they're obviously extremely popular. It's just that there are alternatives on the market now and I'd like to see what others have come up with by now.

I think i have a shitty USB cable. Wonder if there's another one around ...

12:54 AM

@RyanRies the other platforms suck though

Android is a mess and Windows Phone has no apps

I think the Windows phone looks really slick. Coworker has one. But even he hates the lack of apps.

my boss has one too

My gf brought home a Windows phone. It looked really slick. That's about all you can say for it.

Obviously neither of us want one.

I'll throw another tidbit out there... My datacenter staff doesn't like HP servers because they say that if we ever encounter an audit situation where we need to disable all VGA and USB ports on the server, the ILO (which requires USB) will stop working.

So why did she bring it home?

12:56 AM

Ok, f this usb cable

That's stretching, right?

@ewwhite What fucking audit situation would that be?

@jscott Woooo

@MDMarra Just in case it ever happens... I think they were referring to a PCI audit.

12:58 AM

@ewwhite There's nothing in PCI that would require anything remotely like that.

@JoelESalas iTunes -1'd me, I can't believe it, after all we'd been through.

Q: Emulating a VLAN via VPN - which latency can be achieved?

0

I am trying to setup a private network with multiple hosts and virtual machines within the same datacenter without using direct connections or specialized routing like a vrack or VLAN (non-technical constraint). Currently I am testing tinc (full mesh VPN) and playing around with the settings (...

vpn latency

What..the...fuck

You have to lock your servers in a cage if you're colocating somewhere. That's all.

Also, the lead datacenter engineer tells me that HP Blade servers sometimes take 2 hours to boot... His cousin Vinny says so...

@ewwhite Well, just in case moon men that feast on Nikes ever invade earth, we should all stop wearing shoes.

12:59 AM

@MichaelHampton Yeah, I think we're covered... but they're just making shit up now.

@ewwhite And we never went to the moon either, which by the way is made of green cheese.

Sounds delicious, we should plan a manned mission at once for this "moon cheese".

@MichaelHampton tasty

@MDMarra Well see I find myself not really using apps very much so maybe Windows phone is right up my alley. :P

A: How can I block outside mail FROM postmaster@mydomain.com?

@ewwhite Clearly they got some good stuff and they aren't sharing it with you.

6

The fact that someone can send you mail addressed to your own mail server's IP address has absolutely no bearing on whether the mail server is an open relay. Open relays accept mail for any and all systems outside their administrative domain and forward them onward. This clearly is not what's de...

1:00 AM

Android really is a mess though. My work gave me an Android smart phone and it's almost unusable.

@ewwhite just say "Well, we wont have to worry about passing a PCI scan with SuperMicros, because they won't stay online long enough for the scan to complete."

3

I'd literally rather have a Blackberry. Even an old Motorola Razr flip-phone.

@MDMarra That needs like 10 stars. I coughed up my drink a little reading that.

Ha! I had a Razr. I'm guessing anyone old enough did.

@jscott I had three of them.

►

myotherpcisacloud.com: Today's Thoughts on Windows 8.1 (Will Do Server 2012 R2 Next)

1:03 AM

@MichaelHampton That's a lot of phones to carry.

Stack Exchange

posted on September 14, 2013 by ryan

Guten abend! So thankfully, Microsoft reversed their earlier decision to not release Windows 8.1 and Server 2012 R2 RTM on TechNet or MSDN until October 18th. Both products popped up on TechNet a few days ago. So, I downloaded both and have been playing with them in my lab the past few days. (Which is likely the last good thing I will be able to get from TechNet. Rest…

@RyanRies They gave you a piece of shit, that someone slapped a heavily modified version of Android on.

@jscott Not all at the same time!

@jscott A Razr and a Slvr

My wife says, "If a fat kid in a hoodie can install all of that, your datacenter staff can!!"

2

I still loves my Cisco UCS

Only thing I don't love is the price

1:06 AM

@RyanRies still no Enterprise 8.1 image :(

@ewwhite Note the outfit changes, I don't think he did it in one day.

@jscott But still...

@ewwhite Pedal harder, fattie.

My guys are saying that the HP Gen8's are too hard to rack/unrack alone using a server lift... And that sometimes our datacenters are staffed with only one technician, so if they have to do a chassis swap, they won't be able to.

@ewwhite Just borrow Sandra's rack jack if your guys aren't strong enough to lift the chassis.

1:08 AM

@ewwhite Chassis's eh?

If you can't carry a 70 pound box 500 meters, GTFO.

Well, for datacenter techs, they need to use the lifts. We have good guys... but they are not happy that the installation isn't lift-friendly.

@jscott Oh, Sandra.... swoon

@MDMarra Deployed any Win 8 in enterprise settings yet?

Not I

Plenty of server 2012

1:12 AM

Me either

@MichaelHampton Grats on 47k...

@MichaelHampton Well done sir.

Thank you.

Funny, ever since they put the "start" button back in, I've done nothing but right-click on it. Left-click doesn't act the way I expect...

I'm estimating it will have taken me just about 2 years to get to 20k.

@RyanRies It took me over two to get 10k

Speaking of things that don't act like I expect, Win+Q used to open All Apps. Now it doesn't.

1:15 AM

This really isn't that hard. But I guess it requires two people.

@jscott You're mostly Windows, right?

@RyanRies Yes, but I'll go both ways, as required by employer.

Wow, it's in the documentation: Caution: HP recommends that a minimum of two people are required for all Rack Server installations. Please refer to your installation instructions for proper tools and number of people to use for any installation.

@jscott Same here... although, I admit that even after years of thinking on it, I'm still trying to work out a grand unified theory of the difference between Unix people and Windows people. ;)

Combine that with pr0n and music... naked datacenter dance par-tay!!

@RyanRies There's a difference... Although @MDMarra doesn't think too differently than I do...

1:19 AM

@ewwhite Right, we have differences, and we have similarities, which is what made me confident about speaking to your scripting question a few days ago, because that is something that I feel spans both disciplines.

@RyanRies There's some that won't cross the borders. Not sure why, but they like the team they're on.

I'm odd in that I do a lot of Microsoft Exchange stuff... although I'm supposed to be a Linux person...

or is that VMware?

or ZFS....

Heh

@ewwhite manager

@MDMarra I'm not a manager.

1:21 AM

YET

I've done a bit of Linux; I did a lot of Linux back in high school, but I think that was sort of in the same spirit that a lot of young men try Ayn Rand and Libertarianism when they're in high school. Anyway, I'm fine with both platforms. My main concern right now is that I just want a common management platform. Which is why I'm very excited about CIM.

No, it's not the career path I'd choose.

@RyanRies CIM has a different connotation here... @JoelESalas and his Los Angeles white girls know what I'm talking about.

►

@MDMarra That dude is going to need some new boots and undies.

He totally lost the race but at least he got a camera to watch him while he did it.

1:24 AM

reddit.com/r/nonononoyes

There are so many subreddits... I am especially confused by all the "porn" ones. "earth porn", "space porn", "elevator porn", "car porn", "horse porn" ... you would think that ALL of those things would be dirty, but then you click those subreddits, and it's literally just photos of nature, the cosmos, elevators, cars, and horses.

@RyanRies No, horse porn is on /r/clopclop

I mean <_< so I've heard.

@Wesley I though that was mlp only? Or does clop clop mean something else?

@Wesley I know you just well enough to not test that.

2

@jscott You're right. It's /r/spaceclop

@RyanRies /r/spacedicks

1:32 AM

You'll not fool me

It's turtles all the way down

reddit.com/r/roombaww

Whoa, iCloud restore even brought back text messages I'd deleted. I didn't know it did that.

@jscott texts pix of his "junk"

Ha!

@jscott Apple keeps all your text messages, even deleted ones. It's for the NSA, you know.

1:37 AM

It also "restores" your running apps list it seems. That's... something...

@RyanRies Earth Porn is one of my favorites

@MichaelHampton I wouldn't be surprised at this point. :(

Even before the NSA thing broke, Apple offered to store your filevault key on their servers for you. Nopenopenopenopenopenopenope

Trollolol

How far do you think the Microsoft and Google lawsuit against the NSA will go, to allow them to publish details about exactly to what extent they have cooperated with the government?

1:42 AM

@RyanRies Not far enough to get anyone in the government fired.

I personally doubt it will go anywhere, because to allow the corporations to make that information public, would be to give that information to "the enemy" ... the enemy in the eyes of the NSA being everyone, including their own citizens.

@RyanRies I would presume this to be true... Even if you could make a significant constitutional argument for releasing anything, the US Government has made itself very clear: It doesn't care what it violates, and it knows you aren't going to do anything about it.

It's truly sad.

Nowhere. If you want to know, it's going to have to get leaked.

Journeyman Geek

@RyanRies: or more importantly, that the NSA has basically been ignoring the checks made agains them

which is what this really been all about

1:49 AM

The problem isn't that I don't care, or anyone here more likely than not. The problem is the endless sea of drones who don't care. Peace breeds apathy; adversity breeds action and unity...

Watching Shark Tank...

Hot.

And we can pin this all on what... the 9/11 attacks? That was a horrible tragedy, yes. Absolutely. However, what is also true is that the uptick in deaths caused by the amount of people who decided to drive in cars instead of flying because of 9/11 outstrips the number of people who actually died in the 9/11 attacks.

2

@RyanRies Real-talk.

@RyanRies Automobiles are the real terrorists!

@RyanRies Ain't nobody got time for common sense!

2:22 AM

Well heck. Tonight wasn't such a big night. I was hoping for peaking at 3 pageloads per second. =(

But still, I'm thinking I need to put 8GB in that thing now.

Time to remember what kind of RAM it takes.

@Wesley You're not even using 1GB!

@MichaelHampton Huh wha? It was saying 3.5GB used...

Q: Why is Linux reporting "free" memory strangely?

17

This is a canonical question about how Unix operating systems report memory usage. Similar Questions: Server refuses to use swap partition Memory Usage in LINUX I have production server that is running Debian 6.0.6 Squeeze #uname -a Linux debsrv 2.6.32-5-xen-amd64 #1 SMP S...

linux unix memory-usage

@Wesley linuxatemyram.com

@MichaelHampton Well there's that, but it seemed to be healthily taken up by MySQL

Of course, if the PHP dev would optimize queries...

@Wesley Take a look at free. :)

If the PHP dev could optimize queries, he wouldn't be a PHP dev.

2:25 AM

Ah right

Call me when it starts looking more like this.

$ free
             total       used       free     shared    buffers     cached
Mem:       3924896    3704708     220188          0      46888    1496704
-/+ buffers/cache:    2161116    1763780
Swap:      1015800      46924     968876

A little swappy.

Eh, that's trivial.

What should you do with those 45 MB? Keep a bunch of useless junk in memory that never gets used, or cache some stuff that is being used?

Memory is so cheap these days, it's hard for me to justify having anything less than way more than enough.

Ehhhh, when you pay by the hour, you try to right-size your VMs.

2:33 AM

Well true, I don't mean when you're paying for someone else's GeeBees by the hour, I just mean when you're buying hardware.

Though this particular client is having pretty steady growth and I think he'll have to scale up or out soon.

@RyanRies And... I don't do hardware, except at home. Everything is in @Wesley's butt.

The butt is a very good lab environment in which to test production scenarios.

@RyanRies No, my butt is very good... etc.

@RyanRies And your balls are huge.

Oh, you're right...stealth edit time

GioMac

DDR3 ECC 8GB @ 60 USD :)

2:45 AM

We've been over this a thousand times.

My butt, your balls. This is not difficult to understand.

@MichaelHampton Better.

@Wesley I don't get that compliment often enough

@RyanRies I don't get paid often enough.

s/often //

booyah!

@MichaelHampton =D

I got paid once. It was terrible.

3:07 AM

@MichaelHampton is that 4T of ram there?

@DanilaLadner No, that's 4G.

I mean seriously, who would do a terabyte of swap?

oh it is in k, i just alway create alias for free to be free -m

default it is K

yeah, that why i asked -)))

$ free -m
             total       used       free     shared    buffers     cached
Mem:          3832       3628        204          0        325       1316
-/+ buffers/cache:       1985       1847
Swap:          991         47        944

@MichaelHampton Someone with multiple terabytes of heavy disk activity?

1TB of swap seemed insane

3:13 AM

$ free -m
             total       used       free     shared    buffers     cached
Mem:         16048      15611        437          0        745       8914
-/+ buffers/cache:       5951      10097
Swap:         8031          0       8031

My home PC...

@MichaelHampton had a phone screen today ask me how much swap to set up for a machine with 128G of memory. Was a pretty good trick question, I must say.

@MichaelHampton me too

sigh Microsoft got rid of the regular fdisk and replaced with that kludgy-ass diskpart. How irritating.

but on mac pro

@Magellan Come to think of it, I'd kind of like to know the answer to that one.

3:26 AM

@MichaelHampton I told him no more than 4-8GB, which he found to be acceptable.

because, if you're running out of RAM on 128GB, you got bigger problem.

I've heard the same: a trivial amount of swap to keep Linux's memory management code happy, and if you run out of RAM you fail at capacity planning.

@MichaelHampton Any of my servers that would be up for more than a few weeks would end up with some small amount of RAM in swap for no discernable reason. Something like 192k or 2M or something miniscule and ridiculous.

@Magellan I rather like diskpart other than the disjoint syntax and crazy slow startup.

@Magellan Precisely. I once considered taking Linux's mm code and rewriting it from scratch, but I never found the time...

@ChrisS I'm sure it's good. I'm just used to fdisk and still use it under Linux. I'm formatting a 700G USB drive in Vista now though, so that's going to take all night anyway no matter what I do.

3:46 AM

A facebook ad that @MichaelHampton should love!

I can get an IPv4 for $115 a month!

@Wesley Shit, they're throwing in a /27 too?