The Comms Room

oh...whoops >.>

stupid acronyms

Fun fact, if you want to use System Center DPM to back up Hyper-V hosts, the DPM server must be physical and must have the hyper-v role enabled, even if you don't virtualize anything on it

Enabling the Hyper-V role puts the right magic in place for it to back up Hyper-V, like the Hyper-V VSS APIs, etc.

oi...

Lots to learn :D

yeah...I'm doing that training course too

3:04 PM

People with experience in more than one System Center product can make a lot of money in consulting. They're hard to find

Most people use SCCM extensively, or SCOM extensively. Few people do both, and even fewer people use Orchestrator or Service Manager or DPM

We've been hiring anyone we can that knows more than one System Center product and isn't brain damaged, and so far that's been two people in the last 6 months

heh

So, it's definitely worth the pain of learning all the components

or some

I have to reinstall SCCM because it failed halfway through -_-

Yeah, if you get SCCM and SCOM under your belt to the point where you can deploy them, manage them, maintain them, etc

That's easily worth a minimum of $75-80k

That's the floor for someone that's good with both products around here and has no other skills

$_$

yep

I keep getting an email for a 12 month "Deployment Engineer" contract. It's all SCCM and coordinating an XP -> 7 upgrade with it. 140k in the 12 months

There's really good money to be had if you can master them

damn...I'm at like $33k at this place lol

3:12 PM

Yeah, that's the rub. If you want salaries like that for those skills, you need to be near a major metro :)

my boss is pushing salary for me (and a raise) but meeting some resistance because of how the pay structure is (basically, only managers are salaried)

Boston, NYC, Philadelphia, Chicago, LA, etc

believe me, if I wasn't buried up to my eyeballs in debt already i'd be working towards that lol

but, i'll get to that point since I want to learn all the goodies :P

Depending on how much a place likes you, it's not uncommon for help with relocation either

so keep plugging away

@MDMarra Is this position listed online?

Not for me, another guy here.

3:15 PM

@jscott Not sure, let me look

arg...SCCM uninstall is hanging at the sql server phase...sigh

nuke from orbit is likely in its near future

@jscott What's your email address. I'll forward you the email with the recruiter's info

IT's dated Oct 8th, but they've reached out since then, so it's probably still open

@MDMarra jscott@jasonscott.org

sent

Thanks.

3:19 PM

no problemo

Looks like it was 128k not 140k, but still

close enough :)

@MDMarra Heh, I broke SCCM completely. Since it failed on install the SQL databases, I have to uninstall it. Unfortunately, it's trying to delete the databases ...which don't exist.

fun

so, spinning up another VM to move AD to and nuking the server :>

(yes, I had AD installed on the same box...working with limited resources makes me commit sins)

Who can tell me the difference between a Project Coordinator and a Project Manager?

more money talk?

3:23 PM

Just telling @NathanC that if you're going to work in Windows, learn System Center. It's where the money's at right now

@MDMarra are you available for a call?

@MDMarra Coordinator works with contractors and mostly outside folk, Manager works with internal?

@ewwhite Negative. I was on vacation all last week, but you want to call me at 1030a on my first day back at work?

@MDMarra I didn't need you then.

I need to come up with a name for @ewwhite timing. It's uncanny

Around noon ET, I can talk

I have a few calls before then

3:25 PM

Plus, I was fixing all that was wrong with ZFS, Nexenta, VMware, Linux...

oh, and my domain replication isn't working :(

Is it 2008 R2 SP1?

It's whatever's current for 2008 R2

it's because of that failed DC I forgot to fix.

Did your DC that is working unexpectedly halt as well?

nah... just have to reconfigure my sites and services

Ah. You must not be bridging all site-links then, if a failure doesn't cause a DC to try with the next closest site

3:29 PM

@MDMarra I think I was replicating from dc2 (colo) to dc3 (home)

and dc2 is what failed

Man I get some major cabin fever with the holidays

@MDMarra Do I owe you anything?

Right, say you're set up like this

Site1------Site2------Site3

With two site links. One for 1-2 and one for 2-3. If everything in Site 2 fails, Site 1 and 3 can still replication (as long as the network topology allows it) if site link bridges are enabled

@ewwhite Not sure actually, let me look

dc1 and dc2 are in site1. dc3 is in site2.

Oh, did you make manual replication connections then?

3:31 PM

yes

tsk tsk

maybe?

I don't know

ok, 1 hour at Valley is all I have marked as unpaid

The note is "File Services bullshit"

:)

Anonymous

company has Oracle DB running on CentOS

Anonymous

three fucking minutes of bootup

Anonymous

3:32 PM

for just two LLVM'd disks

Anonymous

unafuckingcceptable

Oh wait... replication is working

@ewwhite That should be working

auto connections

This Active Directory stuff is magic!

2

3:34 PM

That it is. So is fire and the wheel.

In Linux, that would be TEN config files, an SSH key and two reboots.

Weird quirk - significant others don't attend my work's holiday party

That's weird, right?

@MDMarra depends on size of org.

60-70

I've had it both ways.

Anonymous

3:36 PM

@ewwhite rebooting linux? nuh huh

And one firm that just scrapped the party.

@MDMarra Is it actually preferrable to install most System Center stuff on server core installs?

I don't even think most of them are supported on core installs. Never looked

I'm not a big core user, tbh.

ah..

core's probably good for stuff like DCs and things of that sort

since they can be managed remotely easily

The only Server Core install I have is a 2012 R2 RRAS box in a VM on my laptop that connects my private vSwitched VMs to the Internet

Anonymous

3:40 PM

@ewwhite just actually dropping your connection to the ssh server

There were too many caveats with it in 2008 when it first launched, so it was never on my radar, and I haven't been in an environment that's had a use for it

On the phone with a new potential client. Client is running a Terminal server in EC2... and... says something about it not reaching a "KDC server"

ha

KDC = Key Distribution Server = Domain Controller (In AD)

he says "Amazon support sucks... and there are no ways to back up the server"

@MDMarra Client says there's no domain controller.

uh

3:44 PM

And there's a Windows Genuine notice that pops up every time someone logs on.

hehehehehe

@RyanRies why would you get KDC errors in a workgroup?

youredoingitwrong.gif

I'm not crazy that that shouldn't happen, right?

@MDMarra No that should never happen... we're missing some piece of information

3:47 PM

Like they had a DC and then deleted it from EC2

:)

Yeah or they let @MikeyB in and he stood up some Samba4 domain or something

haha

That reminds me

I haven't tested it yet, but I heard you can remotely crash an AD KDC by sending it a malformed PAC during a ticket request

LIES

@_@

Anonymous

o.O

3:52 PM

@MDMarra Weird.

@RyanRies waaat

fun

and the terminal server is in the Amazon Virginia region... but the company is in San Fran... Terminal server latency!! @MDMarra

3

freaky!

4:00 PM

Oh, that's not good.

Really? It's been pretty obvious since Snowden started releasing documents that the NSA
1. Wants access to ===ALL=== your information.
2. Does not care what it has to violate to get access.

Client just asked me what type of CPUs are in their server... no good can come from this

@ewwhite On EC2? dafaq

Android runs on x86 =]

@ChrisS Obvious doesn't make it less freaky.

4:07 PM

Anyone wanna take bets on what the client is trying to do?

I just assume at this point that if it's popular and out of my direct control that the NSA has their hooks in it.

I'd be surprised if my phone isn't sending data back to them, either directly or through Candy Crush or whatever else...

@NathanC nah, physical gear

@ewwhite Shoot themselves in the foot?

the reason is... "the boss wants me to order a spare motherboard for the server"

...

redundancy through ignorance is the phrase I like to use.

So no hardware support then?

4:10 PM

@jscott It's top-notch HP gear. There's hardware support... and two failover servers; one onsite, one in my colo 1800 miles away

So I bought a PS3 Controller on Amazon on Dec 6th. Seller promptly "shipped it". Didn't arrive until today, and they didn't put postage on it (USPS is nice enough to just charge me at my door so I still get the package). But also didn't put the correct tracking number of it or anything.
I'm tempted to be a complete A-Hole and claim I never got the package, especially since I just paid for shipping twice. What would you do?

And I hot-rodded the server... fancy 3.3GHz E5-2643 CPUs, mirrored 1.2TB Fusion-io...

Anonymous

@ChrisS forgive but FUCK HIS REP

@ewwhite "If anything fails a motherboard replacement will work!!"

This is embarassing

Just had to do an authoritative sysvol resync on my own DCs

4:13 PM

Explosions, explosions everywhere

I've had two HP motherboards fail in all the time I've worked on computers... Both were due to Dihydrogen Monoxide corrosion.

@ChrisS That website...my eyes burn!!!

@NathanC Yeah yeah... left over from the 90s

the site's quite funny actually

It's an "educational tool"... Most commonly, students are given the task of researching DHMO dangers. They immediately find that site, write their 3 paragraphs report, turn it in, and promptly fail the assignment.

4:22 PM

ha

4:48 PM

Sup bros

@cole Playing with SCCM, you? :p

pauska

SCCM is a love and hate relationship

it's amazing when you have unattend-friendly software to push out

it's a nightmare when it comes to updates

@NathanC first day back

Just got off of a call. Doing our first ESXi, vCenter, vCOps, SRM --> Hyper-V, VMM, SCOM, Orchestrator migration

It's gonna be awesomeeeeeeee

That sounds dreamy

4:53 PM

yup

Three of us are involved though and my piece is small because of my schedule :(

I only get to do the Orchestrator stuff

Oh well, there will be more of these

Q_Q

Hate having to reinstall all my shit lol

I did not miss Lotus Notes.

hehe

@MDMarra I'll be doing that after our factory moves

That's why I'm learning system center/hyper-v now :)

@NathanC blogs.technet.com/b/privatecloud/archive/2013/02/11/…

thats a lot of orchestrator runbooks to automate DR processes

Oooh

huh...vcenter ops is acting weird

it's reporting "no data" for everything

5:34 PM

welp...on the bright side, vops was basically untouched since I installed it >.>

@ewwhite can probably help you there. He breaks VMware products daily

3

pauska

reporting and statistics on vcenter is always bugging one way or the other

He can probably reconstruct all of the DB tables from memory

heh, I rebooted vcenter to apply an update and now the web client broke

I think I got @ewwhite's curse

ah, there it is

Q: In-place upgrade Windows Server 2012 to Windows Server 2012 R2 with System Center and SQL Server products installed

5:50 PM

0

I'm upgrading a demo environment from Windows Server 2012 to 2012 R2. All of the System Center products in the environment have already been upgraded to 2012 R2 from 2012 SP1. In the interest of my sanity and schedule, I was hoping to do an in-place upgrade of the OS on these servers. The envi...

Interesting Azure fact that I found out recently. Microsoft configured the physical hosts to boot to VHD. They don't ever patch the hosts directly, they prepare a new host image with the patches in place in a new VHD and then swap which VHD the host is configured to boot from, and then they reboot. /cc:@pauska @RyanRies

That is interesting... it's also interesting that they haven't moved to VHDX yet, but Russinovich said that they working on it.

I'd imagine it's way easier to roll back in the event of a catastrophic update breaking something untested.

And they use software load balancers

Which work pretty well for software load balancers

You almost have to at that size

Can you imagine what it would cost for the amount of hardware LBs they'd need

I was kind of bummed when I heard Russinovich was getting out of the core Windows OS and moving to Azure, but I have to admit Azure is quite fascinating in its own right

But that there will be no Windows Internals 7th ed. Q_Q

5:58 PM

Cloud first - means move your best people there, I guess

Yeah, I'd help work on Azure in a heartbeat if they'd have me

Especially with how Microsoft's strategy for enterprise seems to be going, Azure seems to be a safe bet

@MDMarra that's pretty slick

6:36 PM

8

A: Linux oom situation

I dont have a complete answer for you yet, but I suspect this is to do with the zone it is allocating from and the relatively high order (order 3) allocation being requested. I'll probably come back and re-edit this with a longer explanation when I have time. A 'sledgehammer' approach though wou...

This was a really interesting OOM and I got to explain loads of cool memory related stuff. So I'm pimping myself.

Never thought so many people would be happy to see me in my life.

2

Joel E Salas

@cole What?

@JoelESalas back at my old job

@cole because now you get to be blamed for all wrongdoings again.

Meh

Anonymous

6:48 PM

@MIfe you get my prestigious "upvoted by Pato" trophy

If it wasn't for internet points, internet hats and internet badges, I would not be reaching my minimum flair requirement. Hooray for the internet!

@RyanRies If you think that Microsoft as a corporate entity has a single unified strategy, you're higher than a kite.

It's more like the Holy Roman Empire. In bad years, the HRE during the Hundred Year's War, in less bad years it's the HRE during the early years of the Reformation.

Lots of squabbling, and sometimes outright warfare.

Yeah, same as any big corporation, but that's why I said enterprise, which is not even the big part of the company

@DennisKaarsemaker WTH?

6:59 PM

(lets see if you follow my thinking...)

@DennisKaarsemaker she didn't screw lumberg in that video though ;)

hehe

@Magellan "minimum flair requirement" is a quote from Office Space, involving Jennifer Aniston. And Jennifer Aniston in a computer setting reminded me of the above

@DennisKaarsemaker Ah. I so must admit my shame. I haven't watched OfficeSpace in several years now. I assumed the above was a Photoshop job though. But just about any degree of stupidity was possible back in the 90s.

►

Apparently, my flair includes 'chuck yeager' as a secret hat. Ha!, I win!

7:06 PM

It is now safe to turn off your computer.

3

@DennisKaarsemaker Oy.

And now it's time to go to work. Later gents.

enjoy

Office Move Day. Blergh. =/

@RyanRies oh the day I found out that that was a simple image file one could replace....

Anonymous

cyber sitcom?

Anonymous

7:08 PM

oh god...

@DennisKaarsemaker Oh hell yeah, you better believe my 14 year old self had that image file modded all to hell with skulls and crossbones and flames.

@RyanRies hehe. I went for 'You broke your computer' to scare my parents

I remember I almost got "expelled for hacking" from my school for replacing the Start button text in the computer lab to read "Ryan" instead of "Start" ... people were fucking scared of computers back then, especially in my backwoods little town.

I was all 'Jesus Christ keep your panties on I'll fix it back like it was, sheesh"

@RyanRies I saw one kid get expelled for trying to ping a router while the network was down. That was '05. Nothing changes. :)

Black screen + text = HAX!!!

Freakin' Luddites

7:17 PM

@Tanner wow.

I'd be like, "And if you took a calculus textbook with you to West Papua, they might just kill you for witchcraft. Don't you want to be better than them!?"

lolno. burn the witch!

TheCleaner

Anyone ever put a DVD on an SD card for a portable dvd player? I'm mainly curious if I can create my own "movie" with menus/screens and stick it on an SD card and play it in a portable DVD player (one w/ sd slot) from "power on".
(We are wanting to create "demo kits" of our product and ship them with our "getting started" videos)

7:33 PM

sup all

sup @MattBear

on my sftp server, I modified fail2ban on Friday to write all banned IP's to a file, and permaban them...

today there are 2968 unique IP's in my ban list

2

Have you got enough CPU to maintain a permaban? You know that 2968 is nothing and that list is going to explode, right?

and my error log is pretty much free :)

@MichaelHampton im switching it over to a whitelist today

7:45 PM

I disabled SSH on IPv4 on some of my systems. They're so quiet, now...

4

@MichaelHampton hmmm not a bad idea

if only all the clients that use it were IPv6...

I run SSH on port 2222. Nice and quiet.

@ChrisS I would still see the occasional attempt on alternate ports. Of course, it's just security by obscurity...

@ChrisS that's what I do on my other servers, cant do that on this one =/

Oh I don't even consider it "security"... Heck, I practically advertise that it's 2222. It just keeps the script kiddies out of the log files.

Q: Different subnet range of ips are not working - windows server 2012 - virtual machine

7:52 PM

Haha wow, this fool gave me a check mark on this answer, despite his comments

-6

ip , subnet , default gateway The main server is windows server 2012 and main ip is : 198.245.73.82 , 255.255.255.248 , 198.245.73.81 Now i have originally allocated ip range from my server provider as 198.245.73.82-86 (82,83,84,85,86) On my hyper v virtual machine i am able to use all thes...

networking hyper-v virtual-machines windows-server-2012 windows-server

@MichaelHampton obscurity is the first layer of defense

He gave me a downvote and a check. How often do you see that?

Jacob

@MDMarra UPBOAT ALL THE THINGS!!!!

Iain

@MIfe Fab answer

I think of my network as an empire.. and each of my servers as a fort

I try to make them all bullet proof... But if I hide them also, then the walls wont get scratched up by random bullets :p

I also think of myself as a benevolent supreme ruler

I wonder if that's some kind of complex....

8:01 PM

I use the highest IP in a range for the default gateway.... We've got two Internet connections here, one uses lowest the other uses highest... So I don't see how anyone could assume one or the other.

linux kernel-panic oom oom-killer

A mod may want to close this and re-open it to wipe the close votes. ;)

2

Q: Linux oom situation

I have continous oom&panic situation unresolved. I am not sure system fills up all the ram (36GB). Why this system triggered this oom situation? Is it about some other memory? highmem? lowmem? Kernel 3.10.24 Dec 27 09:19:05 2013 kernel: : [277622.359064] squid invoked oom-killer: gfp_mask=0x42d...

@MDMarra They'll expire off themselves shortly. Call me if it actually gets closed.

goes to cast the 5th vote and call you

:D

@ChrisS I take my ip range, add the sum of the first and last IP's together, divide by the square root of pie, roll a d20 3 times and add the sum of the rolls to the number, round to the nearest whole number, add my shoe size, and then use that as my gateway

@MattBear So "9"?

8:07 PM

and if it doesn't work, I blame solar flares

I do all of that and then subtract X where ($current_value - x = 1)

@MDMarra you just made me literally lol

Who here has a lot of AWS experience?

TheCleaner

@MDMarra Especially since technically your answer wasn't the "answer".

@ewwhite I have a 14 month old daughter... I experience "Awwwws" every day

8:12 PM

@MDMarra 36 GB of RAM on a 32-bit OS...just a wee-bit wasteful eh?

@ewwhite I know some general things from messing with on my own, but it depends on what it is really

@MattBear yeah, yeah... Same for my PUPPY

I only have one small server left on AWS, and I think it's time to move. Amazon had it up and down all weekend.

@ewwhite Awww

see, there was another one lol

Well, my AWS question...

I have a potential client who uses an EC2 Windows 2008 R2 Terminal server

@ewwhite my kid does the exact same thing, with the same look

8:14 PM

they want off of AWS

but for a bad reason...

is there anyway to migrate their AWS instance to... um, maybe my hosted VMware?

@ewwhite You could use the converter tool to convert an online VM

AWS just rolled out a VM Export feature

I've been told you can't migrate Windows AMIs off AWS, only Linux (and even that's a b----) - But I'm no expert by any stretch of the imagination.

Ah, there's that too...but it's a pain

@ewwhite aws.amazon.com/ec2/vmimport

8:16 PM

@MichaelHampton I thought it only worked for VMs that had already been imported through the same method.

@ewwhite Hm, that might be. I haven't trried it. I know you have to reactivate Windows with a new key after you get it moved to wherever it's going.

@MichaelHampton If I'm reading right, I think it doesn't work for native EC2 instances

But VMWare Converter should do the trick although it'll incur some costs bandwidth and IO-wise

It's a 500GB disk image... 130GB used

may be a little ugly via VMware converter

Of course, PRO engineers rebuild and copy data...

Give the export a shot and let us know if it works?

I don't necessarily want to take this thing whole...

8:19 PM

I did one that size with a P2V (although Linux) and it worked fine

as long as their connection's stable it shouldn't complain too much :)

I want to rebuild all our servers under 2012 R2 when we switch to Hyper-V, but it's a lot of work for certain things and it would have to be done outside office hours, so may not be worth it

I could enlist @MDMarra and see if he has any ideas on how to potentially improve what's there. The client seems to be missing a domain controller.

Is @JoelESalas available?

Oh, thanks for reading that. Took an hour putting that together.

@MichaelHampton So you want OFF of EC2, too?

@ewwhite Absolutely...I'm done with their crap

pauska

amazin cloud wow iops?

8:23 PM

@MichaelHampton can you elaborate?

So today, in 2013, is there a better cloud option than Amazon?

@ewwhite Azure has been pretty good to me so far

Wow already 3:30

This is a nice quiet server, not CPU or disk intensive.

@MichaelHampton any reason for the blips?

8:27 PM

@ewwhite Amazon's I/O sucks donkey balls.

You should see what happened last week...

So this is a case where the people who want to get their terminal server off of AWS are looking to do so because of cost, the "inability to perform backups" and simply not trusting Amazon.

I don't trust Amazon AWS as far as I can throw them, anymore.

@MichaelHampton what will you do? Where will you go?

@ewwhite Haven't decided. Which is part of the reason why I haven't moved it yet.

This sort of shit should not be happening.

Probably not.. but you know me. I find it easier to just host my own

8:31 PM

But it's pretty routine for Amazon. And you never hear anything about it unless you read their forums.

@MichaelHampton is autoscale overhyped?

@ewwhite Probably... I can see it being situationally useful, but most of the events it would be useful for can be planned for.

@MichaelHampton isn't this the price you pay for using shared infrastructure?

@MIfe it shouldn't be

@MIfe No one expects I/O, which is usually near zero, to look like that for a full day!

8:34 PM

Virtualization solutions can manage resources well. Although I don't know much about Amazon's backend

That's indicative of a problem.

Well, its probably not you but some othe rhost right.

@ewwhite Being first to market, Amazon built practically everything from scratch.

so.. im trying to figure out how to set up whitelisting on my sftp server, never done it before

By user or IP?

8:39 PM

I was thinking just set max retrys = 0 in fail2ban, and whitelist the IP's in the jail.local file

@MIfe IP

Oh, so not within SSH but from the fail2ban system?

@MIfe to the entire server

Erm,. iptables?

thats what I was going to say

I have more reading to do :p

hmm...i have a 2k3 box that keeps hanging..

can't rdp or console login, no apps work, but ping responds

everything else just "hangs"

8:45 PM

O_o

@NathanC Isn't that just normal operation? Perhaps you've been living in a timewarp all this time unable to notice the delays..

@MIfe Its a cloud server so I have to be careful modifying iptables, don't want to nuke it, I have no other way back in

@MIfe Maybe...it's been happening for a few months now and I can't pinpoint the cause of it

Alas, I'm only really skilled in pulling linux apart. Windows is an too much of a terse logging experience for me to handle.

It has the UniFi controller on it...and it turns out if the controller is dead RADIUS auth doesn't work

8:50 PM

awww yeah casual dress tomorrow

so I get infinite hanging...since the heartbeats from the APs still work, but the actual app doesn't respond so they don't go into a failover state

which effectively cripples the wifi

nagios reports all OK too which is even stranger...

@cole I thought you stopped wearing dresses years ago...

@DennisKaarsemaker Sometimes I want to feel pretty.

2