The Comms Room

WesleyDavid

00:01

@JourneymanGeek I'm of the opinion that people need to buck up and take it like an adult, rather than flag because "Ouchies my feels!" I think flags are only appropriate for things that are totally not okay in chat like porn or posting of personal information.

Then again, the cultures of other chat rooms might differ, but that makes the universal flag notification rather silly because I'm not going to be bothered with learning the rules of each room and flagging based on them. I judge by the culture that we have in The Comms Room.

Adrian

@WesleyDavid most of the Bridge aren't adults, no?

Luka

@MichaelHampton why hetzner is not good?!

WesleyDavid

@Adrian I dunno - I don't bother with them. I assumed many were older than one would think.

Luka

@MichaelHampton then who is good?

Michael Hampton

@Luka Hetzner, OVH and GoDaddy are probably the biggest providers we get a wide variety of complaints about. Usually because their networks suck, though sometimes it's bad hardware like this that they don't want to do anything about, or a variety of other complaints.

So I can tell you where NOT to go :)

Luka

00:04

@MichaelHampton strato.de :P ?

Michael Hampton

@Luka Never heard of them (which is probably good, becuase I never heard anybody complain about them :)

WesleyDavid

@Luka They're a farm that runs on slim margins to compete on price, but as a result they have a huge ratio of customers-to-support techs. Also, I would suspect gravely overcommitted network resources. It appears that that they do not have a great support organization and you'd better make sure that you can migrate your site in a hurry if things go bad.

Luka

I do not have money to pay more from 49 eur monthly

I have more than 200 sites on this server, server is powerful, only hdd is problem :(

WesleyDavid

@Luka In that case, Hetzner is probably your best bet. If money is the deciding factor, and you need a provider in Germany... you're probably stuck there.

Luka

ALWAYS HDD are problem

unbelievable

WesleyDavid

00:06

@Luka Wow, how do you have 200 sites on there and aren't monetizing it to the point where you can afford more €49?!

Luka

I am low pricing hosting provider

9 months existing

WesleyDavid

No kidding.

Luka

I really have problem with this raid arrays

hetzner says it can be synced

...

WesleyDavid

Like, ¢0.50 per site?

Luka

hahahah? :D

10 EUR is smallest hosting package

I told hetzner to do short hdd test, again...

I hope hdd will be faulty by their side this time :P

WesleyDavid

00:08

€10 * 200

€2,000 a month?

Luka

noooo

I have 200 clients in 9 months

smallest packages id 10 eur ANNUALLY

shared hosting

WesleyDavid

Ahhh... yeah. Makes sense now.

Luka

I really did a great tweaking of server :(

I really do not want to move to new provider

MDMarra

@WesleyDavid i love those things

i still have about 45 in my freezer

WesleyDavid

@MDMarra Share!

Luka

00:10

@MichaelHampton hetzner just told me they started with hdd check

WesleyDavid

@Luka I think that problem is going to follow you wherever you go for €49 a month.

Michael Hampton

@Luka Do the extended self test. It takes a few hours but if there is a read failure, it will catch it.

Luka

YAAAAAAAAAAAAAAAAAAAAY

Dear Client,

as requested we've checked the drives of the server and we'd like to replace the
drive with the serial number: 9VPETFD8. Please note that the server runs a rebuild
from this drive to the other one so we've to wait until the rebuild finished
successfully. If it's not possible to rebuild the RAID then please backup you data
and let us know when you're ready for the replacement.

The server stands currently in the rescue system with the following password:
Epvnwm4e

Michael Hampton

@Luka I knew that drive was failing :)

Luka

What now? :D

WesleyDavid

00:11

@Luka You backup your device to something local, give Hetzner the go-ahead, and then hope they don't screw it up.

Cole

sgbljhglijsligjaehgwaer'

None of this makes SENSE

Luka

They said they started rebuilding

WesleyDavid

@Luka How much hard drive space do you need?

Luka

can you give me remote ftp server? :D

Cole

I hate Active Directory right now

WesleyDavid

00:12

@Luka Oh well in that case, you pray.

@Luka How much space do you need?

Luka

100 GB

Michael Hampton

@Luka One thing I like to do is the "sucks test". I google the provider's name and the word "sucks". Like: google.de/search?q=hetzner+sucks

WesleyDavid

@Luka I've got a slow connection in my cabinet at the moment, so that would take a while.

Luka

haha

WesleyDavid

The rebuild would be done by then.

Luka

00:13

SHould i tell them they are wasting time with rebuild?

WesleyDavid

@Luka Err... no.

Wait, you want the rebuild!

I mean, it's RAID 1, one drive is failing, so they replaced it. That's... good. I'm so confused now.

Luka

they didn't replaced it

they are waiting me to do backup

so they could replace drive

i am really confused

WesleyDavid

2 mins ago, by Luka

They said they started rebuilding

Luka

will my data be losted?

WesleyDavid

^ So they didn't start rebuilding?

Luka

00:15

they started synchronizing

WesleyDavid

That means they replaced the bad drive and it's rebuilding the mirror.

Luka

i already did that, no succes

Dear Client,

as requested we've checked the drives of the server and we'd like to replace the
drive with the serial number: 9VPETFD8. Please note that the server runs a rebuild
from this drive to the other one so we've to wait until the rebuild finished
successfully. If it's not possible to rebuild the RAID then please backup you data
and let us know when you're ready for the replacement.

they WOULD LIKE

WesleyDavid

@Luka Ohhhh, so okay. Hmm.

Just tar and scp everything down to a computer at your house.

Luka

i have remote ftp backup server

hetzner gave me it

WesleyDavid

FTP doesn't retain unix permissions

Luka

00:17

i have cpanel

backup

If they replace drive, i will loose data?

Michael Hampton

A cpanel backup is better than nothing, just make sure it's complete.

WesleyDavid

Okay, that'll make a monster tarred file and then push it over FTP. Better.

Michael Hampton

You shouldn't lose data, that's the point of the rebuild.

WesleyDavid

Interestingly, I had a client who had a 1and1 server with a mirror pair and the rebuild destroyed his data.

Michael Hampton

But you should always have backups, automated and on a schedule. Because, what happens if both drives fail at the same time?

WesleyDavid

00:18

Kinda defeats the point, eh guys?!

Michael Hampton

Yeah, 1&1 is one of the others that I would never give a dime to.

Luka

omg ogm mog

I DO NOT GET THIS

crap

SO i will run backup now

I told them this

"You are wasting your time with rebuilding, i already did that from rescue mode, it stops.
Can i reboot server and backup my data and tell you when you can replace drive?
I will need to reinstall system after replacement, right?"

Cole

I think I'm going to have to demote and re-promote this DC

WesleyDavid

I think your server is hosted in an alternate universe and you should call Dr. Who.

2

Luka

xD

WesleyDavid

00:21

Actually, that sentence stands for both @Luka and @Cole

Luka

haha

WesleyDavid

Luka

...

i am desperate xDDD

Michael Hampton

First, take a deep breath and relax. If you can't relax, then drink a few beers.

Luka

It is not funny

WesleyDavid

00:23

@Luka cPanel backup. Download it. Give them the go ahead.

Luka

I have so much settings!! :((

MDMarra

it's a learning experience

WesleyDavid

@Luka ...and this is why bargain basement hosts suck. You lose your hair over stuff like this.

Luka

everything agian ohhhh

aa

MDMarra

when you use cheap hosting, you get what you pay for, and backups are important.

Luka

00:23

clear

Cole

@MDMarra can I ask you a question

Michael Hampton

No, it's not funny, but you really do need to relax :)

WesleyDavid

And @MDMarra is right. If you want to be in IT, this is the kind of stuff that you have to deal with all the time. Stuff breaks. Emergencies happen. Bad admins look good when nothing goes wrong. Good admins handle the emergencies with skill. Great admins avoid emergencies.

Luka

i always have backup

on extrenal ftp server

i will now update it

WesleyDavid

So, @Luka, you're right. This is a bad situation. Be a good admin, handle it with skill, and you'll be good... if not with slightly less hair.

@Luka Good man.

Luka

00:25

i hate downtime!

i had 20 minute of downtiem in 9 months

I HATE THIS

Cole

@MDMarra I just noticed one of the DCs has a non-auto generated connection. Could that be fucking up my shit?

MDMarra

yes

Cole

well fuck me

MDMarra

is it a bridgehead in either site that's broken?

and is the connection on the other side of that manual as well, or just a single manual connection?

WesleyDavid

@Luka I can't imagine that your clients would have much to complain about though, not to make light of it, but if someone pays €10 for one year of hosting, they have to know they're not getting Rackspace levels of uptime and service.

Cole

00:26

Single manual connection within the same site

MDMarra

@Luka so build in redundancy. It ain't cheap

WesleyDavid

@Luka How many concurrent connections does your server typically have?

Cole

RDE-AD1 has a manual connection to RDE-AD2 (all in the same site as CORPDC1-5, and my forest DCs)

Michael Hampton

It's also 2h25 so maybe few people will notice.

MDMarra

oh then probably not but you should still fix it

Cole

00:27

@MDMarra do I delete it..or leave it?

MDMarra

if japan or DC4 aren't the ones with manual connections then it probably isnt the issue

Luka

My server handles over 30 000 unique visits per da

day

Cole

@MDMarra I have no idea what the problem is then. I've literally checked everything

MDMarra

delete it then run repadmin /kcc on each server

or was it dcdiag /kcc

one of those two

Cole

repadmin /kcc

WesleyDavid

00:28

@Luka Okay, so not much - and the hard drives can't handle that?!

MDMarra

@Luka then you should have had more than one server if it was that important

Luka

it seems so

too much mysql requests myabe

WesleyDavid

7.2k SATA drives should be able to handle 30k uniques unless the web apps are crap.

Michael Hampton

@WesleyDavid The drive was failing due to wear and tear, mostly, it seems.

WesleyDavid

@Luka Yeah, bad queries will do that.

Luka

00:28

Joins

WesleyDavid

@Luka How much RAM do you have?

Luka

too much

24 GB

but it is used no more than 2 gb

WesleyDavid

@Luka Holy balls, 24GB for €49 a month?!

Michael Hampton

As TomTom would say, I can do 30.000 visitors/day on my phone...

WesleyDavid

So that's $63 a month in USD

Michael Hampton

00:29

@WesleyDavid Yep, Hetzner is CHEAP...

but you get what you pay for. Which is complete crap.

Luka

wait

WesleyDavid

@MichaelHampton In every possible interpretation of the word.

Luka

i do not pay

vat

serbia is not in EU

MDMarra

Isn't Hetzer the host that told someone they had RHEL but it was Debian with /etc/release changed, or whatever?

I wouldn't put my worst enemy's infrastructure there

Luka

I have this server

hetzner.de/en/hosting/produkte_rootserver/ex5

WesleyDavid

00:31

@MDMarra Wow, that's ghetto as crunk.

@Luka Fricking heck.

Michael Hampton

Ohhh, it's five-year-old desktop tech.

Luka

hetzner.de/en/hosting/produkte_rootserver/ex5

what?!?/

WesleyDavid

@Luka Big clue: i7

MDMarra

yeah you're not running on server-grade hardware

Cole

How the hell do these DCs not have repadmin!?

MDMarra

00:32

you're basically on an old desktop

WesleyDavid

Hosts like that rack up Fujitsu desktop boxes and sell them off as servers. It's... okay. I mean, people host stuff on Mac Minis, but don't think you're getting anything enterprise class.

Luka

What are you guys talking about my dedicated?

MDMarra

@Cole 2003 with no R2?

Luka

i am not that good with english

MDMarra

@Luka it doesnt use server grade hardware

Luka

00:32

big clue - what?

MDMarra

it uses desktop components

Michael Hampton

Yeah, those were basically the specs of my previous desktop that I built at the end of 2008.

Cole

@MDMarra yep

Luka

where is my passphrase OMG

MDMarra

then you need to install the support tools

Michael Hampton

00:33

@Luka You are better with English than many Americans :)

MDMarra

or just wait

the KCC will take care of itself in like 15 minutes or so (hopefully)

WesleyDavid

@Luka Meaning, the clue that you're "Server" is a five year old desktop is that the CPU is an i7.

Take a look at this.

Michael Hampton

I still think you should have a beer, though.

WesleyDavid

Servers4u had a big double page ad in Admin Magazine for a few months.

Same specs as Hetzner.

Luka

Michael

Cole

00:33

@MDMarra yeah I know, I was just like - why the fuck hasn't someone else already installed them

Luka

i

BLANK

WesleyDavid

They're just Fujitsu desktop PCs

Michael Hampton

Isn't server4you just a Hetzner reseller?

MDMarra

@Cole is 2003 with no R2 even still supported?! haha

WesleyDavid

@MichaelHampton You know... I think they might just be!

Cole

00:34

@MDMarra dude, neither is NT!

MDMarra

hahahahhah fair

Cole

FML lol

WesleyDavid

@MichaelHampton Shameful admission: I was considering a monster Servers4u box for a while.

Cole

Why would someone create a manual connection

WesleyDavid

This one

MDMarra

00:35

-1

Q: Using Logicboxes Fully-Branded Name Servers

I have a problem with creating nameserver records for Fully-Branded Name Servers. I have read this: "ns1.your-domain-name.com: Create 4 Child Name Servers for this domain name with the same Host Name, each pointing to a unique IP Address from among 50.23.136.229, 50.23.136.173, 50.23.136.174, 5...

BREAK OUT THE MODHAMMER

WesleyDavid

Best price I can get for 32GB of RAM at a reputable host is $128, and that's with 1 250GB drive. SuperMicro chassis

MDMarra

@Cole Because they don't understand the implications

I just saw it at the client I was at last week

three one way connections between three sites

Luka

I will pay someone to talk with hetzner haha

Michael Hampton

@MDMarra You know you can vote to close, right? :)

WesleyDavid

@MDMarra SET YOUR PHASERS TO ZOT ZOT ZOT!!

Cole

00:36

@MDMarra wtf.

Luka

I would order DDoS on hetzner xD

MDMarra

@MichaelHampton already did

Luka

i would pay 5000 EUR

MDMarra

@Cole yeah it was a one way replication triangle haha

Cole

@MDMarra wow.

MDMarra

00:37

I'd never seen that setup before...for good reason

Cole

I mean I'm no AD expert, but I know better than that

WesleyDavid

@Luka If you have €5,000 then might I suggest you come to me for some real dedicated servers? =)

Cole

I may be after I finally fix this issue.

MDMarra

it looks like 3+ hours for complete replication to happen since each site link had a 60 minute interval set

Luka

What is your deal :P

hah

clear

Cole

00:37

We started having problems when the DC went down (gracefully) for a extended power outage.

Luka

cat /proc/mdstat

Cole

@MDMarra holy hell

Luka

sorry

tell me how to stop this stupid syncing

i want to do backup faster

MDMarra

no you dont, you want to disks to rebuild faster

Luka

???

Cole

00:39

@MDMarra is there ever a case where you should have a manual connection?

Luka

What are you talking about

I need to backup files

they will replace hdd

i need to reinstall os

MDMarra

@Cole If you want very granular control over which servers are bridgeheads

@Luka for a disk failing in a RAID?

Cole

@MDMarra ah probably why it was set. These are two DCs that are on a different child domain within the forest

Luka

I really do not understand NOTHING now

you are confusing me

MDMarra

@Cole basically never unless you have things like satellite connections to a boat or something

Luka

00:40

hetzner told me i need to reinsatll os after they replace failed drive

WesleyDavid

@Luka X3430 Xeon CPU, 32GB ECC DDR RAM, 2 x 250GB Enterprise 7.2k SATA drives, RAID 1, 5TB of monthly transfer on a 1GB port, /29 block of IPs, cPanel/WHM and Softaculous, and one hour of my time per month, $185 a month.

Cole

I think it should still be fine, regardless

WesleyDavid

@Luka Ohhhh, damn.

Luka

what/??

MDMarra

@Luka So then you had a single server with no hard drive redundancy?

Sorry man, you deserve all the downtime you get

Cole

00:41

@MDMarra ran KCC now it's auto-generate. weee

WesleyDavid

@Luka Well if it's a RAID 1 they should be able to rebuild it without you losing anything. I don't know what they're talking about.

MDMarra

@Cole for you, just let it all autogenerate

Michael Hampton

@MDMarra Fine, it's closed.

Cole

I work with idiots!

MDMarra

<3

Luka

00:41

RAID 1 can not rebuild for over 1 month

MDMarra

that doesnt make sense

Luka

What?

WesleyDavid

@Luka Yeah, something is wrong - it should take a matter of a few hours at most.

Luka

take a look onto thread

MDMarra

are you saying you've had a dead disk for a month?

Luka

00:42

no

Michael Hampton

He's had a failing disk for a month.

Luka

hetzner says it has recoverable errors

but now it has unrecoverable haahah

Michael Hampton

It was throwing a metric shitload of recoverable ECC errors.

Luka

they didn't wanted to replace it before 1 month

MDMarra

so if you replace that one disk, you wont have to reinstall anything

Luka

00:42

now they want

MDMarra

thats the whole point of raid

Luka

I think so

I just need to do rebuild raid arrays

but they do not say that

WesleyDavid

@Luka Yes.

Luka

they say i will need to reinstall system, what is the right answer?

WesleyDavid

@Luka Oh and since you use so little RAM, I can do the above server with 16GB for $30 cheaper.

Luka

00:43

@MichaelHampton knows the situation best

Michael Hampton

Hopefully they replace the right disk!

Luka

michael, please stop

:D

MDMarra

@Luka unless they didnt set you up correctly in the first place, you wont need to reinstall anything

Michael Hampton

@Luka Before you let them go ahead, I would run smartctl -i /dev/sdb and make sure the serial number matches the drive they want to replace.

MDMarra

just rebuild

WesleyDavid

00:44

@Luka The right answer is that they're lazy and confused and just want to put new drives in without having to do any kind of rebuild.

Luka

ok just sec

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.12
Device Model: ST3750525AS
Serial Number: 9VPEV294
LU WWN Device Id: 5 000c50 03f874d8b
Firmware Version: JC4B
User Capacity: 750,156,374,016 bytes [750 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Jun 2 02:44:21 2013 CEST

Michael Hampton

> Serial Number: 9VPEV294

Cole

@MDMarra I found something weird on the Japan server in the \HKEY_LOCAL_MACHINE\System\currentcontrolset\services\NTDS\Parameters on the Japan DC

MDMarra

go on

I like weird things

Cole

@MDMarra under the Machine DN Name

Luka

00:45

they want to replace

9VPETFD8.

WesleyDavid

@Luka 750GB beats desk with head

Cole

CN=NTDS Settings,CN=CORPDC-JAPAN,CN=Servers,CN=Singapore,CN=Sites,CN=Configuration,DC=waters,DC=com

Michael Hampton

@Luka That's the OTHER drive! It's the one with all your data on it.

Cole

It's in the Japan site, not Singapore

MDMarra

@Cole huh

Cole

00:45

Right?

MDMarra

when you open sites and services on the japan DC and make sure you're connected to japan, what site does it say its in

Cole

Japan

Luka

/dev/sda is with 9VPETFD8.

Michael Hampton

@Luka OK so you have a serious problem. It seems BOTH drives have errors and are failing.

Luka

who says that?

Michael Hampton

00:46

@Luka And the OS, your smart data and I say the other drive is failing.

MDMarra

@Cole I don't know if/when/what that key does or how it gets set

Luka

they say one drive is falling

ear Client,

as requested we've checked the drives of the server and we'd like to replace the
drive with the serial number: 9VPETFD8. Please note that the server runs a rebuild
from this drive to the other one so we've to wait until the rebuild finished
successfully. If it's not possible to rebuild the RAID then please backup you data
and let us know when you're ready for the replacement.

Cole

@MDMarra nor do I, I was in there checking for something else and I was like "huh..."

MDMarra

@cole is sinagpre an adjacent site?

Luka

they checked DRIVES

Michael Hampton

00:47

Therefore I have to conclude that both drives are failing.

Luka

not drive

MDMarra

and is that DC the only one in japan?

Cole

@MDMarra Yes and yes

Singapore has one DC as well.

Luka

THey do not say the same

Cole

Src Root Domain Srv key shows as the Singapore DC

Michael Hampton

00:48

They're also complete morons.

MDMarra

Maybe it logged into the singapore DC?

Michael Hampton

I bet they never even checked /dev/sdb because they just checked /dev/sda first, and got an error.

Cole

@MDMarra It just doesn't make sense that it went down for a few hours, planned and shut down gracefully - and came back fucked up.

Luka

@MichaelHampton can i give you passphrase, openssh key to take a look, i am really confused now

oh god

Michael Hampton

@Luka I think you should continue with making your complete backup, but be prepared to reinstall from scratch and restore your backup.

Luka

00:51

You are saying me my other drive fails too

Michael Hampton

That is what I would be doing.

Luka

i am doing backup now

Michael Hampton

Yes, I think both drives are bad. Maybe Hetzner will figure it out later :)

Luka

why do you think that??

Michael Hampton

Because of the SMART data that you posted in your question, that I already answered.

Luka

00:52

here is info for both drives

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.12
Device Model: ST3750525AS
Serial Number: 9VPEV294
LU WWN Device Id: 5 000c50 03f874d8b
Firmware Version: JC4B
User Capacity: 750,156,374,016 bytes [750 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Jun 2 02:44:21 2013 CEST

Michael Hampton

And the fact that mdraid rejected it.

Luka

"Dear client,

the values of the other drive looks good so please tell us why you thing that this
drive has also a problem."

Michael Hampton

Gawd, what a bunch of idiots.

Luka

What to tell them?

Michael Hampton

You could copy and paste my answer, since I already wrote it :)

Luka

00:53

i did that :D

before

Michael Hampton

In that case, they can't comprehend English (or maybe they can't comprehend computers) and you need to go find another provider.

In the meanwhile you can run the extended self test on the drive yourself: smartctl -t long /dev/sdb It will tell you how long it expects the test to run. like "Please wait 214 minutes for the test to complete"

Luka

while backup?

you say SDA is OK?

SDB is not? xD

Michael Hampton

Please don't make me repeat myself :)

Luka

ok

you said SDB failed

hetzner says SDB failed

great to know :P

Michael Hampton

No, hetzner said SDA failed.

Luka

00:59

yes xD

Transcript for