« first day (1307 days earlier)      last day (3960 days later) » 

16:00
@DennisKaarsemaker Hm... Didn't know that.
@BigHomie sizeof() is a compile time construct that can only give you the size of fully defined datatypes.
@DennisKaarsemaker Whelp, I've been doing it wrong for decades then
such as sizeof(char) being 1 and sizeof(int) being 4 or 8
@voretaq7 TWSS
@DennisKaarsemaker :(
@84104 . . . but I'm NOT a gentleman (by any stretch of the imagination)
16:01
@MichaelHampton XO gave us a new IP block to use... but with no guarantees that it wasn't already tainted...
@voretaq7 you're far from gentle. And hardly man anyway :P
@DennisKaarsemaker Reading SO it seems that one could sizeof(x) / sizeof(x[0])
@DennisKaarsemaker ...that's not what my exes say :-D
@ewwhite Weak...
Anyone know a decent DNS provider that allows root level CNAME for use with azure / aws etc?
16:02
@BigHomie I can see how that might work, it'll still be evaluated at compile time, and only on fully defined types.
@Adam DNS doesn't allow root level CNAME.
@voretaq7 Bah language (I'm used to ladies, gentlemen and children of all ages), but I'm going to go with "not all black things are ravens" in my defense.
@DennisKaarsemaker I agree, but that's why the need for a field like payload size
@Adam you can't cname the root of a domain, it's an RFC violation.
Go search the main site, we've yelled at people about this before :)
@84104 <inspects coffee carefully for signs of feathers and a beak>
@voretaq7 What makes something the root of a domain? SOA RRs? CNAME can't co-exists with anything other than CNAME.
I am now officially tired of seeing questions.
16:03
@84104 Basically having a SOA record in this context
@MichaelHampton You're only getting tired of it now!?
@voretaq7 I know, thats why I asked in here! So you cannot have a custom domain on a CDN or anything, we just bought a cdn for our clients, but we have to serve using a subdomain instead of the top level domain :(
@Adam that's what everyone does. e.g. cdn.sstatic.net
You could also have an A record that points to a machine that just serves HTTP/3xx redirects but that's ghetto and if you do it we'll all make fun of you :)
16:05
Noooo. Do not abuse the naked domain.
virustotal.com/en/file/… good to know forefront didn't pick it up...
@MichaelHampton Clothe the naked domains?
made it past clamav too
@voretaq7 I bought cdnx.net, so I'll have to use s.cdnx.net or something
cdnx.co *
@Adam for your current implementation, yup. For the future "cdn.example.com" is free when you already own example.com :)
16:07
@MichaelHampton Let's just start advising people to not use SSL. "Just do everything over http, so it can't be stolen from the server's RAM."
(clients like "free" -- cheap bastards)
And while we're on the topic... yes-www.org
Oh, that's cool. Thanks for disabling our antivirus, Kerio. -_-
@NathanC Really? Is there a .scr file anywhere that didn't come with Windows and isn't a virus?
@MichaelHampton yeah, I know. Forefront said it was clean...
16:09
@NathanC I'd just ban attachments with those extensions. Fuck that.
What about client.com pointing to loadbalancedclient.cloudapp.net, I know you can setup www. but then without www it doesn't work
@MichaelHampton It came inside a .zip file
And I can't ban those :(
@NathanC Untangle extracts ZIP files coming across the network to scan the contents.
@KevinSoviero Yeah...it's not exactly something I can drop into our network to filter this sort of stuff
I mean, our sonicwall does have antivirus built-in, but it's not activated ($$$)
@NathanC Untangle is sort of free... At least the ClamAV based AV is...
16:13
I have clamav on our mail server, but the kerio update overwrote my config file for it
so now i have to reactivate it...yay
I love email.
What's that site we've been using in here to check for heartbleed-yness?
@MichaelHampton I love email, and printers, and users. I especially love users who print their emails.
@84104 And scan them back in.
@MichaelHampton So that they can fax them.
16:19
@MDMarra fix it!
@ewwhite Just gotta redomain it to the join. Or some combination of those words.
@ewwhite I'm not a Windows guy. What does that even mean?
@HopelessN00b It's a domain controller
@HopelessN00b Tired?
@KevinSoviero Means the computer account isn't found in AD, or the computer account password is wrong. Typically happens when someone joins a new machine to a domain using a machine name that's already in use.
@KevinSoviero Nope, I stand by it. Redomain the computer to the join.
@ewwhite Oh, good times. Mark's really gonna earn his money on this one. (Probably... could just be a simple DNS eff up too, I suppose.)
16:25
@ewwhite This is why we don't clone domain controllers?
@HopelessN00b Difference between computers (today) and humans? I still understood what you said the first time I read it.
@ewwhite Replication might be busted
@KevinSoviero Good, you passed my Turing test. :)
On the authenticating DC
@84104 I called the client... and he cloned the domain controller in VMware to create a server for Spiceworks.
4
I don't know how to recover. Now the Spiceworks server is a GC.
16:28
@ewwhite You gotta start revoking some of your clients' admin rights. That's just... oh my god. Have fun with that.
and there's another domain controller, but this impacted Exchange and a bunch of other things.
@HopelessN00b he called me yesterday to ask about cloning VMs... and I said, as long as you don't clone a DC...
Probably gonna have to go to backups... you know, provided they haven't decided to overwrite the backups with the Spiceworks installer ISO.
I didn't think he would use a DC as the source of the clone!
@ewwhite I wasn't expecting to be so on the nose with that comment. =(
So what do you think... DCPROMO the Spiceworks system? Rebuild dc1? Is there any nice recovery process?
16:29
@ewwhite "From now on, let's make this simple. No computering of any kind by any of you!"
he's the IT director for the company!
@ewwhite Well, is that company just in great hands, then?
That's fun...Kerio is regenerating its config file when I restart it
@ewwhite even more of a reason for him not to do IT work
Remember the cocaine bananas from yesterday?
16:30
@ewwhite Those are for the customers, not internal use.
It's an honest mistake... maybe?
costly to fix
and my time is limited.
@ewwhite Only thing I'd be comfortable doing is going to the backups and doing an authoritative restore of the whole active directory, honestly..
May as well pimp Mark out to do that and collect a cut. :)
I'll call @MDMarra
Nah, he's not picking up the phone... he knows
You should start a twitter alias
"Stuff my customers do"
@ewwhite Are you starting a drug running business? If so I'm in
16:35
@JoelESalas I think it's more that his client are all on drugs...
@HopelessN00b That's where he's starting obv
@pauska You mean... Something like this ?? At least there they won't immediately suspect him should they find themselves topics
well, no @MDMarra So I'm going to dcpromo the spiceworks server
@ewwhite HAHAHA
16:38
So, AD people... this is a duplicate-SID problem. If I remove the duplicate server, what happens then?
rejoin the one that got booted
sorry, cant talk I'm already supposed to be in two places at once :|
@MDMarra I can't log into it.
If @ewwhite were calling you, would you answer?
Not even with a local admin account?
Budding is a form of asexual reproduction in which a new organism develops from an outgrowth or bud on another one due to cell division at one particular site. The new organism remains attached as it grows, separating from the parent organism only when it is mature, leaving behind scar tissue. Since the reproduction is asexual, the newly created organism is a clone and is genetically identical to the parent organism. Organisms such as hydra use regenerative cells for reproduction in the process of budding. In hydra, a bud develops as an outgrowth due to repeated cell division at one spe...
time to grow another @MDMarra
2
16:41
@MDMarra He can't find the Post-It note with the local password.
cleaning lady removed it
@ewwhite A whole lotta fail there
@ewwhite Serious?
ultimate boot cd -> nuke the local admin password
@BigHomie no, not serious...
16:42
@ewwhite Then break into it with some boot disk :) or restore the old AD object from backup on the DCs
A giant produce rat scurried away with it.
gotta run. good luck! :D
Backups?
@ewwhite this is why I like the comms room. Gold comedy :)
@ewwhite Wow. Better use Nutcracker then.
16:43
C'mon... I'm better than that. I know how to break into servers.
doing all of this through a VMware window to the VM
I had to submit a support ticket to Kerio to figure out why it's rewriting the configuration file and removing my edits...
So the spiceworks server was demoted. If there's a duplicate SID between a member server and DC, that's still bad, right?
@ewwhite Download something from Tucows
Didn't happen in the previous version :(
@MDMarra is tucows still a thing?
16:46
Grrr... I have a fucking conference call about fucking printers in 15 fucking minutes. Fuck, FUCK, FUCK!!! Gonna miss all the good parts of this trainwreck.
I think @MDMarra did clone himself if he's still chatting here
tucows is still alive unfortunately
@MDMarra @DennisKaarsemaker Yes, Tucows lives. That's where I found this gem.
> Printers were the problem.
Printers are always the problem.
@ewwhite on the thumbnail that looks like "twerking.com"
I'm okay with that, too!
16:49
@ewwhite NuT Cracker, bootable iso?
@ewwhite Oh my god what are you doing.
3
@BigHomie I just use a Linux live CD
@JoelESalas TWSS
(downloads NuT Cracker...)
Alright, I'm off to ask a new question on the main. "Is my DVR system that runs on Windows and uses an Oracle database back-end vulnerable to Heartbleed?
17:00
@HopelessN00b On-topic, because Everything is vulnerable to hearbleed.
@HopelessN00b No. proceeds to hax
Anyone here do network admin at a uni?
@mossy What you need cuz
Sorry @Jacob, the first SE Question on heartbleed is on page 7 right now, and that's Security.SE
...and repcrapped. But I don't wanna go back to work.. I'm dealing with printers. <sobs>
17:15
trying to reset this dang password
@mossy Yes.
urgh... trying to figure out why our custom erp app doesn't want to run on fresh pc's
it has some old VB portions inside it, and of course the installer spreads dll's and ocx files around the filesystem like it was powered by ebola
there must be literally 500 different dll's here.. most of them end up in syswow64..
@ewwhite Remember that you gotta boot into Directory Services Restore Mode mode to use that local account...
@BigHomie huh?
I fucking hate people.
No cant log in and the server log showed Winlogon errors but its vague... — Sal 40 secs ago
17:22
@HopelessN00b oh?
Nevermind, I hate printers more. WHY THE FUCK WON'T YOU SHOW UP IN THE DIRECTORY?!?!??! DIE!!!!!
Dammit PHP. Why do you have to suck so bad?
@ewwhite Yup, no local accounts on a domain controller... exception being when in DSRM, you can access the "local" Administrator account... that's only usable in DSRM.
@Jacob I would have liked to see your SF question on google above the fold, but now it's clogged with..... news articles
That's just googling heartbleed though btw
@Jacob In stark contrast to this
@HopelessN00b how do i DSRM?
17:31
F8 @ boot I believe @ewwhite
@ewwhite At boot time, you hit F8 to get the Windows startup options... and select it from the list.
jack be quick....
F8 F8 F8 F8 F8 F8 F8 F8 F8 F8 F8 F8 F8 F8 F8 F8
I'm trying to help my dad get a job where I work... Is that unprofessional?
No option for it
@KevinSoviero Why do you ask? Is he incompetent, or something?
17:34
@HopelessN00b No, just seems unprofessional to do favors for family...
FInally!
@KevinSoviero As long as he knows what he's doing, there's nothing wrong with it. Employee referrals happen all the time, and generally people refer their friends and family.
@KevinSoviero Where do you work now?
@BigHomie Yeah NPR fucked the story hard
@ewwhite Roight, that one.
17:36
@Jacob A place you've never heard of
"if a site has the padlock, it's affected"
I didn't know that DCs had no local admin
<- ig'nant
@ewwhite No local accounts at all, in fact.
So I'm in Directory Recovery mode
what now.
Ctrl-Alt-Delete
then what?
still not allowing me to log on
@ewwhite It's been so long I forgot what you were trying to do. Which account did you reset?
17:40
Administrator?
I guess
@ewwhite guess? Uh oh.
@ewwhite Which Administrator account? There's also a Domain account, named "Administrator," by default.
I just need to get into the DC
not sure how or what's necessary to do so
that's what I get
Do I need to create a computer account?
do I need to just blow away the DC?
@ewwhite I probably would
@BigHomie Have IM? Need someone's opinion on something.
17:44
Let me get this right.. the customer had a single domain controller, and he cloned it?
@pauska Two DCs
this was a @MDMarra build from last year. Stable
@Jacob hangouts, Cyborg G+
Everything is compromised from Heartbleed, not even Tor is safe.
client cloned one of the DCs so he could save time building a server for Spiceworks
Same SID
I don't know why he did this
@BigHomie Not found
so a day later, Exchange breaks... And DHCP is probably down for the environment, too
17:46
@ewwhite Yeah, nuke it. Walk through removing all the pieces from AD, then stand up a new one in its place.
@HopelessN00b I wanted to get back into the DC1 to see what the DHCP configs were
@BigHomie got it
I already demoted the cloned SPiceworks server.
The issue here is that the cloned server has overwritten/created duplicate AD DS entries
I won't lie, you're in deep shit.. or the customer is rather
2
there is no AD DS backup? No system state backup of either domain controllers?
17:48
@pauska So it's more than SID.
Either a restore, or complete redoing I assume
@ewwhite Yes, a lot more
@pauska My access to VMware and everything (like backups) is limited
@ewwhite Aren't you a Windows admin?
because VMware relies on AD
17:49
0
Q: Windows Firewall rule to mitigate Heartbleed

asudhakI have looked everywhere and I can't seem to find a netsh equivalent of the u32 module for iptables. I want to create a rule to Drop a packet that starts with a specific set of bytes. Similar to: # Block rules iptables -t filter -A INPUT -p tcp --dport 443 -m u32 --u32 \ "52=0x18030000:0x1803F...

nuuuu
So I can only get into the ESXi hosts... can't even do full vCenter work from here
@ewwhite Your access to backups? I don't get it.. elaborate
vSphere Data Protection
@ewwhite Local root vCenter account is disabled?
does the customer have a fresh system state backup of a domain controller or not?
17:50
It's vSphere Web Interface only
@pauska not that I can access
What if I told you... The One doesn't need firewalls? — BigHomie 17 secs ago
Couldn't help it
because the VMware backup software requires the web interface, AD and vCenter to be able to recover
that is the most stupid backup solution I've ever heard of
@ewwhite Cyclic dependency?
ok... can you somehow dig out the snapshot of a healthy domain controller when vsphere data protection last ran?
it must store them somewhere.. I'm guessing vmfs
17:51
@pauska I cannot
it's an apparent flaw in the product
You can't even log in to one of the esxi hosts with a local account?
@pauska I can. That's how I'm in.
but I can't do anything with the backup software.
so scratch any backup recovery options. I'm looking for another path forward.
@ewwhite Possibly stand up another DC and migrate roles?
@ewwhite This is why I hate GUIs, they make people so reliant on them, that when they break no one knows how the underlying system works! No offence @ewwhite, just an observation of the IT industry as a whole.
@KevinSoviero +1 for shells all the world over
17:54
what roles did the cloned DC have? do you know if it held the FSMO roles?
I do not know
I smell a very expensive rework
can you get in on the other DC? the one that weren't cloned?
Yes, I can.
@BigHomie If he's a salaried employee, than it's not very expensive at all...
17:56
@KevinSoviero Ed's a freelancer, they called him, which means he'll probably subcontract it out, $$$
@ewwhite netdom query fsmo
@BigHomie Ah, ok.
on the not-cloned DC
has all 5 roles
@KevinSoviero If the FTEs do it, it still costs b/c that's time spent they could be doing other things, and we all know time == $
17:57
just to be sure, the not-cloned DC has all the FSMO roles?
@pauska yes. There was a DC1 and a DC2. DC1 was cloned to SPICEWORKS.
DC2 is on and was not touched. It has all the FSMO roles.
Ok, you've struck luck
just shut down DC1 and the spiceworks server
reboot exchange
those are off now
on the functional server
open up ad sites and services, and check ntds settings for dc02
does it have global catalog ticked?
18:02
good, good
exchange should function..
If I were you I'd start AD cleanup and just wipe the old domain controller and the spiceworks server
there are lots of articles out there on how to do this
yeah, I've removed dead DCs before.
do you know if the old dc had any extra services? DHCP, dfs, CA etc?
DHCP
So that's missing
check the event logs on the functional DC
if you're lucky then it's not complaining about mismatched ad
Okay. Exchange is picking up dc2
18:05
yeah it uses whatever DC it can reach in the same site.. it just has a habit of crashing when said DC goes away
No eventvwr complaints.
so I need to build a new dc1
run dcdiag as well
and figure out DHCP
and NPS/radius/ca/bleh/blah if it had those running
DGDIAG only complains about earlier attempts of DC1 trying to connect
18:07
Oh, God, WHY?!?!?!
0
Q: store log files in the cloud, when servers are not in the cloud

Florin AndreiWe're looking at a long-term migration towards the cloud. The plan is to start small, and gradually move less essential parts of the infrastructure into the cloud. All good so far. Part of this migration includes log files from web servers and whatnot. Keep in mind that the servers are still in ...

@pauska Is there any way to get insight into what the DHCP ranges/etc. were on the old DC1?
And where should the DHCP service run?
I was always used to seeing it on one DC
@ewwhite It can be clustered
in 2012
yeah, with 2012 you should aim for a dhcp failover setup
18:18
shouldn't need shared storage for DHCP :)
I don't think it stores the DHCP configuration in AD.. backups..
@pauska I'll just add a scope of some sort.
It doesn't use shared storage..
@pauska the link @BigHomie sent for 2008 does.
it's not a cluster either, it's just HA
@ewwhite Applies To: Windows Server 2012
It's not for 2008
18:20
yeah it's failover
don't bother with it now, you can add it later
just get stuff up again
Maybe you can bring dc1 up on a separate network, point dc2 to it for dhcp management and try screenshotting (!) the info
barbaric, but then again, who cloned the DC...
you can boot the dc1 vm with the network adapter disconnected
but it will probably crap out at the conflicts
@pauska Yeah the DHCP config and whatnot is stored on a local database.
maybe you can get in ad recovery mode...
seems like awful alot of work just for a dhcp setup.. did they use tons of reservations or something?
18:24
@BigHomie I can't log onto it.
remember.
@pauska printers
they just told me the range.
I'm here to propose experts.serverfault.com - an exclusive q&a subsite of Serverfault for users with 1,000 rep or higher only.
Cool..the product key that we just spent $800 doesn't work on my existing R2 install
:(
@ewwhite the domain controller was the print server?
18:29
@ewwhite I was thinking that it would use some domain account remotely (like network svc or something), it probably would use the account you logged in as, probably wouldn't auth though
@pauska Strike II
@TheCleaner Experts of the experts?
@pauska Sounds like it was the DHCP server for the printers. Still, printers.
@BigHomie inner sanctum. VIP lounge, Million Mile Club...whatever you want to call it. A place where we can shut the doors and and promote IT classism.
@TheCleaner Bump up rep req on Comms? Is that even possible?
I mean a Q&A place..not "live support" :D
and some clients can't connect because they only had one DNS server entry...
18:32
@TheCleaner lol good one
@TheCleaner Yeah, good luck with that. Also, since Evan Carrol's pretty close to 1,000 rep on ServerFault, I think your metric is fatally flawed.
@HopelessN00b is that your archnemesis?
@BigHomie that would imply someone roughly his equal.
@BigHomie No, just the biggest, dumbest internet troll I've ever seen.
(I actually think he's legitimately got a psychiatric personality disorder, FWIW.)
18:35
@TheCleaner idk, the penguin had nothing on batman.
Honestly, if I actually didn't know @HopelessN00b or @voretaq7 and read Evan's meta post and original main question I would have sided with Evan on at least keeping his question open. I made that comment to Hopeless in chat before those "comments started flying" about Evan's question. Evan needs to realize thick skin is a requirement online though. There are multiple users here that won't tiptoe around people and will call it like they see it, whether that's allowed by the rules or not.
@TheCleaner Yeah, you don't get to be a prolific troll without some skills. He's also obviously relying on people not knowing, or ignoring, his past history.
Doing an in-place upgrade to 2012 R2 datacenter...wheee
look at the video
cool mr hipster devops developer with his fucking hat and nerdy magazine
KILL
@pauska Fuck that fucking hat
Hats on men are NEVER ok
18:45
@JoelESalas that's debatable
@JoelESalas Maybe not that kind of hat, unless you're a gay Frenchman... but hats have their uses. I present to you, the beer hat:
Why are all their I's uppercase?
(Ignore the shitty beer, you can put good beer in it too.)
@pauska Because lowercase "i"'s are sooooo last year. Every hIpster knows thIs.
@NathanC from 2012 R2 standard? so easy! :)
18:51
We've done lots of 2012 datacenter -> 2012 R2 datacenter
hasn't been an issue.. but remember to delete windows.old
@TheCleaner Indeed. dism did the trick, but it's refusing to activate the key now...
@HopelessN00b Yeah, I don't know him at all...but I know that if you plan on surviving on SF you have to roll with what the masses lean towards on any given day. It's not as bad as Meta.SO, but you still can't come in thinking "OMG, everyone is going to love me for posting this" while you stare at yourself in the mirror while typing.
I've thought that on occasion.
@BigHomie you know you've got a giant mirror next to your monitor... :)
@TheCleaner I keep the webcam on me at all times
18:54
@pauska @HopelessN00b The i's are actually not uppercase. Not that it makes a great deal of difference
@TheCleaner ...or, you know, just not get butthurt about downvotes/closed questions.
@HĂ„kanLindqvist What are they then?
lowercase l's
@pauska Lowercase?
new hipster font
18:55
Lowercase I's without a dot over the line
really
@BigHomie lol, there ya go. You need to leave it on for your entire career and then do a "time lapse" video of your entire life at work when you retire.
@pauska Seems to be a font rendering issue or something. Copy paste the text somewhere or look at the source, it's not actually written in a funny way.
@HopelessN00b yes, in summary.
@TheCleaner Sounds depressing actually, and I don't drink anymore
Watching myself get old in front of a computer monitor :(
18:57
Sooo.... auditing our websites because of the heartbleed thing, and so far, of the 14 I've checked, 1 works, 5 don't resolve (or go to a parking page) and 8 have a certificate mismatch, because they use the certificate for the one working site. <sigh>
From a billion dollar company. I hate life.

« first day (1307 days earlier)      last day (3960 days later) »