The Comms Room

they are

Stack Exchange

After failing over to Oregon for Sandy in late October, we'll be moving back to New York tomorrow: http://stackstatus.net/

Tweeted by StackStatus on February 8, 2013 at 8:27 PM

2

►

9:01 PM

In the middle of a snowstorm, too.

replacing the hard drive in an imac: Still a better love story than twilight.

2

Tomorrow? With incoming blizzard?

@RobM I'm installing PHP on an IIS server. Also still a better love story than twilight.

very good tom

@tombull89 true

@petergrace Well hopefully 111 8th Ave won't flood tomorrow.

9:02 PM

heh, I wasn't even on stackexchange yet in october :)

Oooh, here's a good one... 150 users in a new Exchange server... how much storage space to provision?

@DennisKaarsemaker There was an awesome podcast about what Fog Creek (Joel's other company) had to do since they didn't have a DR facility like Stack Exchange did

StackStatus.net - "Anyone got a bucket?"

@MDMarra the bucket brigade of non-planning?

yep

9:03 PM

@DennisKaarsemaker Some of the SE people ended up carrying buckets of diesel up several flight of stairs

A++ for effort, F- for planning

I was listening to it and it was a great story to listen to, but the point that they all missed was that SE just flipped a switch and went to sleep

@ewwhite I want to say 200mb/user but knowing your clients...

Zypher

@MDMarra not all of us went to sleep

@MDMarra yeah that was a point that should have had more air time

9:05 PM

/yawn

DR planning isn't easy, but it's necessary

►

serverfault.com is 174-176ms away from me today - how much closed will NY be

@tombull89 The client is asking for 1TB of Exchange storage :)

@Zypher did you see meta.serverfault.com/questions/5227/serverfault-com-dns-issue

9:08 PM

@ewwhite Do Exchange databases dedupe well? I'd imagine not, since it's all in a database

THey compress now. Single-instance-storage went away

Zypher

@Iain nope, dns change freeze until after we move though

2

Q: Serverfault.com DNS issue

While developing my DNS checking tool and running test reports I've noticed a DNS issue with ns4.serverfault.com: Full report here: http://www.dnsinspect.com/serverfault.com Different glue is offered by parent name servers and domain name servers: ns4.serverfault.com IP=69.59.196.18 at paren...

Zypher

will fix it then

@Zypher You're doing it wrong :P ^^

9:09 PM

@mdmarra they used to de-dupe but it's been wound down for a while. It was killing disk resources for virtual machines.

@RobM I meant if put on underlying storage that dedupes at the block level like ZFS or a SAN with dedupe

Not the built in exchange dedupe stuff

all bets are off at that point. I'm guessing not so much.

Yeah that's what I thought as well

MSSQL doesn't dedupe well, I didn't expect jet would

This will be on a ZFS-based SAN.

Exchange still uses jet right?

9:11 PM

either that or an HP P2000 G3 fibre)

@MDMarra Yarp.

Microsoft had a test build of exchange on MSSQL once. It wasn't pretty

@RobM Jet is perfect for email storage - the SQL Server dreams were just whiners wanting to over complicate things.

Agreed

Not to mention, Jet has developed so, so much over the years, it's not what it once was.

<-- used to be one of the SQL Server whiners until he got whomped by an Exchange neckbeard.

9:13 PM

They actually tried it though - whether as a serious idea or to make a point to an important idiot in house, I don't know for sure.

they started building it just before the launch of exchange 2000 and got something running after about 3 or 4 years of hard graft, then proved how much it sucked and swept it back under the carpet.

I think it might have been related to the "you know what, even the filesystem is a SQL server" ideas that were first pushed about as part of windows longhorn.

@RobM I thought there was a later attempt as well. Around 2007.

I thought there was a PoC done with white papers or something. I wish I had saved all that information.

@WesleyDavid wouldn't surprise me, but I was less involved with Exchange by then. Maybe it was someone flogging the dead horse that I'd seen for a bit longer or maybe it was a new attempt.

voretaq7

I hate snow :'(

@RobM WinFS!

Was that what it was called?

WinFS, yeah the SQL based file system

9:18 PM

so is 1TB of exchange storage a lot?

@ewwhite We had 3TB for ~1k users with a default 4GB quota at $lastjob

No dedupe/compression

Just DAS

It's not an amount I'd be concerned about as to whether or not Exchange can handle it. I'd want to be splitting it up with lots of small databases though

If they have the requirement for mail stores that add up to that, no problem

@MDMarra this is 150 users and it's a collection agency. They'll need pretty strick storage quotas and policy.

Oh noes! It's actually chilly outside!!

9:20 PM

@ewwhite Sounds high

Tanner

@WesleyDavid Burn in hell.

It's flippin' cold up here.

@Tanner +1

So, just fudging it, that's like 6.67TB per user

Account for 1/3 space for logs, and it's like 4ish GB per user

@WesleyDavid at work today & yesterday i had to wear my coat indoors. Please melt in a kitteh heatwave

Tanner

9:21 PM

At least today it's warm. 40 degrees and feelin' like t-shirt weather.

It's nearly shorts weather here

voretaq7

@WesleyDavid @freiheit I am currently plotting your demise

just so you both know its coming.

@voretaq7 I have heard of snow, but basically have to travel someplace to see it

another one on my list of people who should melt

It's about time to start spring planting in the garden...

9:28 PM

@freiheit we have another 4/6 weeks before it's even remotely safe

I wouldn't do tomatoes, chiles or other nightshade family stuff yet, but anything that can handle a light frost is good now... We had our 2-3 weeks of freezing a while back and now it's spring. I can see little buds forming on the tree outside my office window, so it'll have leaves soon.

There's definitely some nice bits to living someplace that only has 3 seasons

It was almost warm enough not to wear tights on the bike today

@MDMarra working on AD tonight for Davalan

@ewwhite I'd imagine that 150 users in a collection agency could chew through space real fast. Where are you provisioning this storage?

@HopelessN00b I don't think so. The bulk of the users are on a call center floor and have an autodialer and no internet access.

Storage will either be a NexentaStor SAN or an HP P2000 G3 SAN.

9:35 PM

@ewwhite So they aren't going to be mailing horse porn videos to each other?

@ewwhite Oh, grunts. Personally, I'd thin provision a shit-tonne more than I think they'd need, so that you don't have to take down the mailstore to extend the disk or any silliness like that.

I'd probably do 500 GB, and then another 100 GB for a trans log drive, so I wouldn't have to bother with it again until forever.

(We did have a big issue with that at $[job-2] where the mail migration crushed our translog drive and we had to shut down Exchange for a few minutes to provision more disk space... anything to avoid that again, because people get so pissy about email interruptions.)

ooh, speaking of transaction logs, have you seen that iOS 6.1 bug?

No, but that sounds like it'd be good for a chuckle. Do tell.

run away transaction logs because it continuously tries and tries to open connections to your exchange system

if you have a big corporate iDevice rollout on 6.1 then say hello to transaction logs filling up faster than you can clear them

paulrobichaux.wordpress.com/2013/02/07/…

Oh, dear God. Thanks for the heads up, we have a bunch of crApple products for all the Execs and VIPs (naturally)... so I'm sure that's gonna kick our asses when we move them to the new Exchange server. <sigh>

9:42 PM

If you can control those iDevices then don't upgrade them to 6.1 - 6.0.1 is ok and I'm sure it'll be fixed soon in 6.1.1 or whatever

Well, we'll probably just have to put off migrating the VIPs until an iOS update fixes that. Grr. Nothing's easy.

yeah. It bites either way. No arguments here

Apples already seeded 6.1.1 to devs. The fix exchange bugs quick

They like their enterprise penetration, it seems

good

yeah

@ewwhite I'm working on it or you're working on it?

9:49 PM

Trouble is, they still don't really 'get' business networking. Mind you, they don't have to while every CxO in the world is going "oooh my preciousssss" and bring a personal iWhatever in from home, throwing it at their network team and screaming "MAKE IT WORK. NOW"

I do love me some apple stuff as a home user but as a techie I wish they'd get their heads out of their asses

@RobM Yeah. Fucking execs.

to be fair, ours aren't like that. But that might be because they know I like the stuff myself so they just need to wait until my next payday for the latest iDevice to work anyway...

@RobM compress IIS logs

Invest in JAMF or Absolute products. They seem top notch at OS X and iOS management

Just expensive

@HopelessN00b circular logging!

9:52 PM

I evaluated JAMF stuff and got it quoted. It didn't get purchased until after I left that job though

nooo

@MDMarra I'm buying certs now

For?

@MDMarra the mail

Ah ok. So do you have anything for me tonight (or with this project in general)?

9:55 PM

@ewwhite Well, because it was a migration scenario, that wouldn't have helped either (actually, we may have been doing that, but I don't remember). The migrations ran the disk out of space too quick.

Or do you just need someone to talk you out of .local :)

.local? You guys do know about that impending CA change that's gonna prevent certs from being issued to fake TLDs, right?

.local 4 lyfe...

LDAP connection string.. can I connect to a higher level OU amd use objects in a sub OU?

or do I have to connect to the specific OU

No, you can connect to a higher level OU. Domain root comes to mind immediately. :)

9:59 PM

That's called defining a "search base" and it depends on how your app works.

@MDMarra sharepoint :p

FBA

Ya with a custom forms front end :)

Every time you name a Windows domain .local another devop is born.

@RobM Not just Windows domains.

you mean people actually name other things .local?

10:00 PM

yuuup, its turned into a priority because IE likes to try to plug the local computername in as a domain name

Share point speaks AD/Kerberos by default. So it doesn't need all those ldap strings unless you're doing custom forms auth. And I have no clue how SP handles that

@MDMarra external contractors

Good luck!

@RobM Yeah, I've seen it elsewhere. What can I say? People suck.

jesus

10:02 PM

There are dumb admins everywhere. There are just more windows ones because the barriers to entry are lower :(

@MDMarra dont be mean :p

no, it's because REAL work gets done with Linux.

I thought REAL people ran Unix?

@MDMarra On the bright side, I think the DevOPS fad is generating a lot of really terrible Linux admins, so they're becoming a lot more common than they were back when I was primarily a Linux admin.

@HopelessN00b I'm working on the systems for a devops shop now

10:04 PM

Ya devops!

odd design choices

Or maybe it's Ubuntu? Every dunmbass with a PC running Ubuntu trying to turn that into a job as an SA...

and the devops shop has ubuntu :(

@ewwhite Well, that's what happens when you don't design anything.

@HopelessN00b a good example of odd devops. Calculating disk latency per drive in a RAID array.

10:07 PM

I'm thinking that when I read CVs/Resumes I need a rubber stamp that says "NO, installing ubuntu at home, plugging it into your home router, sharing a file with your windows PC and browsing the internet via the built in browser DOES NOT make you a sysadmin. Now fuck off."

I just called SWC for Microsoft stuff... and was mentioning that goatmale guy... My contact there was like, "hmm, I don't know him... are you sure he works here?"

Jeff Ferland

@MichaelHampton Is it not normal behavior to use link-local addresses? I thought your answer was right.

@ewwhite Wow. I don't even know what to say to something like that.

I was scared for a minute... but then someone was like, "oh yeah, he works here still"

@JeffFerland Yes it's normal, but I didn't read his question closely enough...he said it didn't work. So he needs to do some troubleshooting, I think...

10:11 PM

@ewwhite sounds like he's making a name for himself!

@MDMarra I asked the person on the phone to try to find him in Lync.

LYNC

What a terrible name for a product

@MDMarra SSSD is bad, too...

@MDMarra Not that bad. Now "Good," on the other hand is just fucking awful. Some retard in marketing obviously had no clue about search engines.

@ewwhite My latest version of Fedora is encouraging me to join a Windows domain using Samba winbind.

10:14 PM

@MichaelHampton I dare you.

this should be fun

@ewwhite I have a DC up, let's spin up a VM and see what happens...

@MichaelHampton I've done that with Solaris

...but not right now. I'm trying to get 40 more reputation in the next two hours so I can get my badge!

I should complain to HR that my workload is getting in the way of Server Faulting.

Lately 50 rep is a big day for me. :(

10:17 PM

@MichaelHampton which badge?

Epic

@MichaelHampton how far away?

@MichaelHampton BOOM! Done

@MichaelHampton Oh, by the way, I heard someone complaining that it took a few days for his Epic badge to be awarded after he met the requirements... don't think the code checks for that badge with the same frequency as the other badges.

> 40 more reputation

10:19 PM

@ewwhite Days at or above 200 rep: 49 of 150. (32%)

@MichaelHampton bitches!

@ewwhite Hey, you have money. Which would you rather have??

damn @MichaelHampton

I just get a string of 185-point days.

Yeah, I have had lots of those lately.

10:21 PM

It's only because I don't answer Windows questions... total conspiracy.

I usually only ask Windows questions.

I only have 10 days >200 shrugs

That's because you spend all your time chatting and banhammering people.

I increased my rates by $20/hr... About to send invoices and prepare for the outrage...

(instead of being 49/50 days at > 200)

@MichaelHampton I hardly chat and no one is currently banned by my hammer - I just cba answering questions for days/weeks at a time

10:28 PM

@ewwhite What did you start at ?

$180... in 2008.

@ewwhite Where you at now?

$200

@ewwhite Don't hate

but new expenses... I just got hit with a $2700 bill from new data center.

10:29 PM

Oh, that's pretty reasonable. But yeah, I bet people bitch.

nobody bitched at 180 because that what my old company charged.

the new colo charges 30-days forward

When I get close to 200+, I go and and dig up old question in my favorite tag with the answered:0 query string and grind to 200

@MDMarra No Bitchassness!

I....don't know what that means.

I know those words, they are all familiar

P. Diddy speaks the truth.

10:33 PM

@ewwhite Recurring?

@WesleyDavid $1149/mo

@MDMarra I'm surprised you know about the Liverpool University Botanic Gardens

but they prorated the half-month I've been in, plus Feb, plus March

give up

@Iain I was thinking loch ness monster, personally

10:36 PM

@MDMarra Ness Gardens are about 20 minutes from her e

finally

So, @ewwhite. I'm still unclear. Am I doing anything for you tonight or was that just an informational update? :)

Exchange user has been over the send/receive limit for one week... and has not been receiving mail.

Woohoo... I managed to let our Exchange 2003 server limp on for another week. I need a shower.

@MDMarra I'm sending bills and quote stuff... has to get done first.

Ok that's fine. You're just very cryptic sometimes :)

I have a hard time figuring out what you actually want me to do and what's just informational sometimes

10:49 PM

It's tough behind behind schedule.

@ewwhite For one rack? How mach bandwidth? Amps/Watts?

@ewwhite one terabyte? for exchange? do not want.

@WesleyDavid 1Gbps HSRP uplinks, one cabinet, 2 x 20 AMP circuits.

@tombull89 seems to be too much, right?

@ewwhite For $1100 a month? Wow. Nice.

@MDMarra for davalan, you could make the default OU magic work.

10:51 PM

@ewwhite I'm no expert by any means but that's excessive to say the least.

@WesleyDavid oh, um 10Mb commit... $25/Mb overage

@ewwhite So new accounts get put in a sane OU and the UPN suffix gets set automatically?

@MDMarra Right. You probably need VPN access.

yep

Oh, I know what the issue is... I need to swap DC's

10:53 PM

swap?

@MDMarra davalansales.local to the new dc I built in ad.davalansales.com

Oh right

.cc

@ewwhite Just checked my exchange server - 400GB total storage for mail, of which 89GB is used.

after you change the IP address of the new DC make sure you run

ipconfig /registerdns

dcdiag /fix

Similar number of users.

10:55 PM

You'll get weirdness for a short time after if you dont

@tombull89 oh yeah... none of my exchange servers is over 200GB of data

@MDMarra at least this place has normal IP addresses

192.168.10.x

not 199.3.63.x or 192.1.1.x

wow, progress

@ewwhite why do they want that much? ultra-long term plans or they tend to use their email to store a ton of files as attatchments?

@tombull89 they don't know any better.

completely new to the idea of exchange, SAN storage or VMware.

facepalm

10:56 PM

The SAN I spec'd only has 3TB usable...

@ewwhite Ew. Okay, nevermind.

maybe it should be more, but I can just add nearline disks if necessary.

It's all very well having money to spend but there's no point wasting money (unless you're in education. education is very good at wasting money. I'll tell you that from expereince).

@ewwhite Just get a 3TB NL disk and span them

DEVOPS

Nearline disk?

10:58 PM

== 7200 RPM SAS drives.

@tombull89 7.2K SAS

Basically 24/7 use SATA disks with a SAS interface instead

I thought the whole point of SAS was to take advantage of the higher spin/interface speed...

@tombull89 naw, the protocol has better error correction and performance under load.

voretaq7

@MDMarra :kick:

:P

ah.

See, you do learn something new every day.

11:09 PM

@MDMarra easiest way to copy Win 2008 DHCP DB to a new server?

lights just flickered

@ewwhite a netsh export and import command

I swear, my power goes out I'm going to be PISSED.

@ewwhite let me look it up real quick, I have it bookmarked

netsh dhcp server export C:\dhcp.txt all

netsh dhcp server import C:\dhcp.txt all

Then shut down the DHCP service on the old server. Restart it on the new one

Authorize the new one and you're good

It'll copy the leases and everything

looks like I'll need a .local zone temporarily.

Tanner

11:18 PM

@MDMarra That's a neat trick.

@ewwhite why?

@MDMarra Upvote!

@MDMarra existing clients, domain suffixes and some devices that may be referencing .local

ah

I didn't think anything was using it

nothing AD-joined.. but for generic DNS

yeah I figured

Hennes

11:37 PM

The Greek god of plagiarism is Poseidon, as he has the power to Ctrl-C.

@MDMarra reimported DHCP config.. had problems with option 15

What about it? Did it fail or did you just have to change it?

Q: ssh to local network behind a public IP (TCP forwarding)

Had to modify the existing... no biggie

mossy

0

I want to connect (using ssh) two remote computers in a local network using a unique public IP X.X.X.X. | HOST_A---------| 172.1.1.2 | _____________ |---------|ROUTER PUBLIC|-----------|INTERNET|-------------|CLIENT | ___________...

ssh port-forwarding ip-forwarding

Wait... So @MDMarra freelances for @ewwhite?

Q: RDP and SSH running separate Desktop instances

11:46 PM

bling bling

mossy

O_O

0

I'm SSH'ing into an amazon EC2 box running windows to run some selenium tests. However when I RDP into the same box to take a look at them, I don't see any browser windows or anything happening, for that matter. Looking at the task manager, I see all the processes involved running, I see the logs...

windows ssh remote-desktop selenium

@MDMarra should I dcpromo/demote the old domain controller or just turn it off?

Might as well just turn it off in case you need to spin it back up for some reason

It was the only DC in the forest, so it's not like anything will fuck up if you just keep it off

The new exchange server needs a name..

"banana"!!!

How about EXCHANGE-01

11:52 PM

BANANA!

@MDMarra Thats already being used by mine, sorry try again

:(

BANANA it is