« first day (763 days earlier)      last day (4504 days later) » 
00:00 - 20:0020:00 - 00:00

00:00
@MichaelHampton with network cables.
00:33
@Zoredache NOT with one of these
I would love to let a few of those on a spare table in an IT office.
It would be a nice test. One I am sure someone will fail.
00:46
I don't get it!
:)
Once you plug it in, you will get it.
Bzzt!
An ex housemate (20-ish years ago) actually used something like that to power a laptop in a room without powersockets.
But it was extremely well marked.
Still, it was a lyer-8 problem waiting to occur.
I'm adding a 2TB mirror pair of nearline drives to a server
@MichaelHampton This is replacing a 2TB SATA set.
It used to be an LVM volume.... fully allocated
You just reminded me of my backups. DOne to 2TB sata drives. And now half a year old.
00:52
as I rebuild it, should it be LVM or just a normal partition?
/me waits for the system to crash and burn, just before I fix that
No LVM experience here. I know it can be usefull and flexible. But if you are not going to use that flexibilityI would keep it simple.
But with no LVM experience you should take that with a grain of salt
I don't like LVM, to be honest.
it's not my system... but for a pair of backup drives, it's silly to add that layer of overhead.
@ewwhite: are you planning on adding more disks?
(which is the only reason you'd use LVM. With raid, personally it dosen't make much sense)
01:09
@JourneymanGeek Nope. not planning more disks.
I think LVM is a little lazy... especially when using a modern hardware RAID controller
111
A: LVM dangers and caveats

RichVelSummary Risks of using LVM: Vulnerable to write caching in kernel and hard disk on kernels before 2.6.33, or in VM guests Harder to recover data due to more complex on-disk structures Harder to resize filesystems correctly Snapshots are hard to use, slow and buggy Requires some skill to config...

How long should it take for me to mkfs.ext3 a 2TB volume?
@ewwhite: I personally don't like to use it myself. If I need to do any of the things LVM is meant for (other than snapshotting), I haven't planned well enough ;p
exactly... although, people have called me a "partition snob"
but I hate seeing:
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg1-root  255G   50G  192G  21% /
/dev/sda1              99M   49M   46M  52% /boot
tmpfs                  24G     0   24G   0% /dev/shm
/dev/mapper/vg2-data  1.9T  581G  1.3T  32% /data
lol. I like to plan before I even touch a disk
o0
versus...
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              12G  4.3G  7.0G  39% /
tmpfs                  14G     0   14G   0% /dev/shm
/dev/sda1             291M  166M  110M  61% /boot
/dev/sda7             2.0G  123M  1.9G   7% /tmp
/dev/sda3             9.9G  2.5G  7.0G  26% /usr
/dev/sda6             6.0G  854M  5.2G  14% /var
/dev/sdb1             400G  260G  141G  65% /data
>_>
also, /data is new to me.
01:15
I'm obfuscating the real partition name
@ewwhite: I'm a big fan of the single large partition sadly enough
how do you know if there's a problem in /var?
ahh, I was wondering if its some file system hierarchy I wasn't aware of
(CUPS server running awry)
@ewwhite: well, as you guys would have guessed, I mainly run systems to learn for now.
so, not had any fun failures yet
01:16
ah?
Still in school (far too long) ;p
gotcha
and slightly unusual speciality, so most of my *nix learning is self directed, and somewhat haphazard. A good chunk of stuff not to do with DR/IR type things is self learnt
I used to be a EE major, dropped out, did computer security, had a nervious break dowm then dropped out.
so, ya, not much practical experience
oh my...
lol, life sucks. I fall flat on my face, I get up, I get going ;p
I'm almost at the tipping point between 'first world problems' to 'getting on my feet'
01:19
@ewwhite his point about easy of resizing is a bit stupid. His argument basically is. You can break things. Which is true no matter what tool you use to partition/resize things.
@Zoredache I guess I've been through some heinous filesystem recoveries... I couldn't imaging having to deal with LVM on top of that.
My first week here, I saw people with DRBD problems and LVM signatures(?) preventing recovery.
@ewwhite Yeah, gawd, Red Hat seems to always manage to feature-freeze something on just the wrong version.
So I have a question, how did you create filesystem larger then 2TB if you don't like LVM? Did you switch over to GPT or something? Since MBR won't give you a partition larger then 2TB.
@Zoredache I have a pair of 2TB disks...
and mirrored them in the LSI controller
and fdisk /dev/sdc
created one partition
and am waiting a little too long for mkfs.ext3 to run
Right, but lets say you had a pair of 3TB disks. How are you going to create 1 filesystem with that?
01:28
it's been like 20 minutes.
I wouldn't use 3TB disks...
at that point, it's shared storage...
or if I did... it would be carved into HP logical volumes
the reason I'm in the middle of this now is to replace some SATA drives with nearline SAS equivalents.
So are we just hoping that SSDs fall below the $/GB mark of spinning media before they get too much larger?
@MichaelHampton maybe?
30+ min to create that filesysystem!
WHat should I set the fsck interval to?
This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
It's journaled, isn't it?
Then who cares.
If it comes up unclean the journal gets replayed and you go back to work. That code dates back to when ext2 had no journal.
01:33
ah, ok
i've been on XFS since 2001-2002...
I honestly don't know if that even does anything anymore.
I know you have to look up the right argument to fsck to pass to it if you really want to check the filesystem, otherwise it'll just replay the journal and look at you funny.
gotcha
@MichaelHampton On Debian it seems to do a full fsck when that count or time expires.
@Zoredache Hm, I suppose I could spin up a VM and change the count to 1 on the root filesystem, and see what it does..
These nearline SAS drives! much happier...
7200 RPM, but 40% faster than the SATA disks they replaced
01:37
I which they weren't so much more expensive then SATA.
I am tempted to replace+upgrade my home storage with NL-SAS... But would be a lot of bucks.
I'm using SATA green 4k sector disks at home.
cheap shit
but doing it in a way that's kosher with the controller setup
I currently have 8x1.5TB. I want to go to 2 or 3TB drives.
@MichaelHampton November 2009. 3 year old PowerBar.
@WesleyDavid Well, you seem to have survived.
@JourneymanGeek I actually use /data. Alot. It's just a nice scheme of doing things when you have vm's with one os-disk and one data-disk. The extra data disk always starts at E: for windows servers and /data on *nix
and then F:, /data2 and so on
01:55
I put data being served by the system at /srv :)
@MichaelHampton: Thats 'standard' tho, no one does it any more
everyone uses /var for what /srv is supposed to do
Yeah, who would follow a standard?
@JourneymanGeek The standard also says that OS distributors aren't supposed to make /srv the default for anything.
I don't touch /var, as I religiously believe that it should be left alone for critical system logs/databases
/srv is good for LXC
@MichaelHampton: I very vaguely recall reading really old RH distros used to pop what's now /var/www into /srv
01:57
I have git repos in /srv/git, websites in /srv/www, etc.
Only one client where that's not the case, and that's only because I inherited the system that way, and untangling it would be a nightmare the client doesn't want to pay for.
$contract_sysadmin[-2] set up all the web sites in the user's home directory, and $contract_sysadmin[-1] set up a spaghetti tree of chroot jails and bind mounts so that $developer could work on them.
oh, I know someone who does that
runs gentoo too
I find it confusing
Hell, I find it confusing. And I'm the bloody admin! Half the tickets I get on that box are "$DIR/$FILE has the wrong permissions!"
@pauska I got hooked on /storage/prima, /storage/altera, /storage/teritus, etc
02:32
@JourneymanGeek clfags, dawg.
@WesleyDavid: hardcore FOSS guy/ more on the comp-sciency side of things
nearly started his own distro too.
02:43
I ran Gentoo... in 2004. I built my own Linux From Scratch... in 2005. I got over it.
lol
I personally don't mind gentoo if I had a week to set it up right, and it was for personal use
I just find the whole setup method... well, odd
/me would rather start with a system where he knows everything works without needing to recompile the damned kernel
@JourneymanGeek Just break a bear bottle, get a running start, and land on it spread eagle. Same effect, gets it over quicker.
@WesleyDavid: eheh. I did get rid of my gentoo test box for centos
Its just not really worth my time at this point.
@JourneymanGeek Oh, it was for personal use. The only really funny part was, I had it on a Dell business laptop and the keyboard died. The guy who came out to replace the keyboard was completely lost...
@MichaelHampton: I think it was quite heavily cause it was linux
02:49
Anyone want a quick 250 reputation?
then again, My CFISM classmates have no idea what IRC is...
Well, fortunately he just shut up and replaced the keyboard. This is why I never buy "home" stuff, even for home.
"Business" customers get much better service, even when they aren't actually businesses. :)
@MichaelHampton why?
02:51
Don't ask me. I just take advantage of it.
ugh, which reminds me, I really need to find time to drop by the computer mall
/me is getting the itch to get on with building his new system, considering I finally have enough money
@Zoredache Oh, you mean the quick 250 reputation? I really want a good set of solutions to turning off Linux console blanking. The question was pretty good, but all the answers suck.
Can you link to the question? And suck how?
@MichaelHampton: so you want a consolidated answer that covers the more common possibilities?
Pretty much.
21
Q: How do I permanently disable Linux's console screen saver, system-wide?

raldiI've got an Ubuntu server that boots up in text mode. It rarely has a screen or keyboard attached to it, but when I do attach a screen, I usually have to attach a keyboard too, because the darn console mode screen saver will be on and I'll need to hit a key to see what's going on. I'm aware that...

Some are clearly distro-specific and don't mention which distro. The accepted answer is specific to Ubuntu (and maybe all Debian-derived distros). Nothing seems to cover Red Hat-derived distros, and I figured it would be better to offer a bounty than to ask a "duplicate" question.
03:47
Anyone have a Canadian server I can VPN into to get to a link?
I don't, but I do have IPv6 :)
Though I'd be interested in seeing a live example of a link that doesn't work in the US.
@WesleyDavid And, that's what Tor is for.
 
1 hour later…
05:09
Visited a datacenter, got a half rack, fixed customer problems, creeped everyone out on FaceBook. All in all, a magnificent day.
2
@WesleyDavid What'cha need?
Wow… the writers of Criminal minds are sick bastards. I say this with grudging respect.
@MikeyB Oh, just a silly IM conversation with a Calgarian. She sent me some YouTube videos that were blocked in 'mericuh so I got lazy and decided to just ask if a 'nadian friend had a VPN for me.
But all that is past.
 
1 hour later…
06:36
G'day
@MichaelHampton that's probably get closed OT today
 
5 hours later…
11:45
I is alive :)
@Chopper3 been in surgery?
@Chopper3 good to hear - how did it go ?
still VERY groggy ant full of painkillers so difficult to know for sure but not bad I think
good
sleepytime now :)
morpine's ace :)
11:57
good to hear you're ok mate, hang in there :)
12:36
Silly question...
software RAID...
you don't get write caching, do you?
Beyond what Linux normally does?
1
Q: Linux software RAID that uses system RAM for write cache

ensnareIs it possible to configure software RAID 5 on linux that uses system RAM for write cache? I have a file server with 8GB of RAM ... would be really cool if I could dedicate 4GB to write cache. If so, how is this done? Thanks.

@MichaelHampton I'm talking about the equivalent for low-latency write cache... like a flash-backed or battery-backed write cache.
I suppose I do this in ZFS with the DDRDrives and STEC ZeusRAM that I deploy
but I'm working on a client system... big devops type of media firm
ZFS. They thought of everything.
12:44
and this got escalated to me
and the server has been crashing.
Put a real RAID card in it?
it's a database server... and it has software RAID.
and that seems silly to me...
Find whoever did that and escort them to the nearest conference room for a talk.
Invite HR>
Well, I read blogs like this and this..., and wonder why so many people bemoan the cost of hardware RAID.
Bah. Hardware RAID cards are very common and cheap. At least if you find a way to not pay the vendor's list price.
12:48
every server I've purchased in the last 10-11 years has had hardware RAID, (with the exception of this, but that was by design).
This new company is in the business of hosting these things, and maybe I just take premium hardware for granted.
out of ~3,000 servers, there are maybe 8 Dells... and ZERO HP systems.
What, is it all white box stuff?
ALL Supermicro
so when a DB server crashes, and it's software RAID, and it's supermicro, so I don't have an IML log, watchdog or any of the nice tools to examine, I feel like we face more failures than we have to.
Cheap, fast, good. Pick two.
I always bought HP... and when cost is an issue, I have a line of HP-refurb available.
Supermicro motherboards can all take an IPMI card for remote (serial, I think) access. But they never ship you the card with the system.
12:53
we have IPMI and serial over IPMI enabled everywhere.
but it's simply not as elegant.
I bet it still feels kludgey when you're used to iLO
Getting an email from the server that says...
Trap-ID=6025

An 'ASR Recover Complete' trap signifies that the system has
been shutdown by the ASR feature and has just become operational
again.
in HP-land, is extremely helpful
these Supermicro systems just suck-ass
I guess they're all right if you have < 5 of them and the sysadmin is parked right on their ass. But 3,000 of them?!
we have a fully-stocked spares inventory and full-time data center staff onsite 24x7
and pre-tested components.
even have pre-charged LSI and Adaptec RAID batteries ready to go.
yet, all of that shouldn't be necessary :)
I've been here a month, and have seen ~15 LSI RAID controller batteries die.
and I think it's a Supermicro cooling issue.
and then think, gosh, why even bother with systems that you have to assemble in this manner?
@ewwhite because they don't know better ?
13:00
My guess is they started out on < 5 servers on a shoestring budget, and grew too fast?
But this is a hosting company... I think people poo poo the idea of good hardware because of misconceptions about cost.
@MichaelHampton that may be right...
people don't do proper lifecycle costing
How much more does it cost to keep a piece of crap limping along, than buy good hardware in the first place?
I have insight into how much we spend on these systems... so the one I'm working on is a dual E5620 box with 48GB RAM, an IPMI card, 2 x generic SATA disks and an 8Gb Qlogic fiber HBA
$3950
My price on the HP equivalent, DL360 G7, minus the FC HBA, is ~$3500
and it would have had ILO3, RAID, better cooling and power efficiency...
So they pay more for the hardware right off, AND the inevitable maintenance. Definitely time to start replacing all that crap.
And you can make the business case for it.
13:05
I'm going to introduce them to my Long Island guy this week
i don't expect to make much headway...
but the outages I see day-to-day are because of unstable hardware.
Is this even something that should be here?
0
Q: How do I back up my rails database on the fly?

Danny DylaI have been looking for a way to back up my app's database which is hosted by heroku. It uses an sqlite database. What I am looking for is a way to add a button on my site which will allow me to download the database as an sqlite file. I am sure there is some simple fix I am overlooking but it w...

weird
@ewwhite You may have to make it a 3-year project where everything gets replaced gradually, but the numbers are on your side.
the other thing is, I don't touch the hardware... they are techs and first-line support staff who manage things on the daily.
so just because I know how to navigate HP doesn't mean other people would adopt it.
Oh, in that case, you pretty much have to get them on board.
13:10
THey tried to evaluate Dell earlier this year, but decided on Supermicro's new server generation
imagine how much power is wasted...
Excuse me while I go off to find just the right facepalm image.
but I guess that's why Silicon Mechanics is in business. :)
@Iain Oh, and @Iain, why would the question be closed as OT?
@MichaelHampton screen savers on servers seems more desktopy to me
13:16
But it's the text console!
is it ?
ah - indeed - missed that bit - not enough caffeine
And blanking the text console that happens to have a kernel panic or some error on it that you want to look at is...annoying.
13:51
I might be ill and, essentially high, but even I know this is bullshit; serverfault.com/q/438013/1435
 
1 hour later…
15:17
@ewwhite We tried silicon mechanics for one of our servers to see what we thought of their stuff. I wasn't impressed.
Boss liked the idea of going with a local company because it would be quicker to get on-site support. This was about a year before I convinced him that Virtualization had any legs as an operational plan.
15:39
Perhaps I'm just a Linux asshole, but I really don't get the whole reasoning behind WAMP hosting.
16:34
@Adrian What is WAMP hosting?
and it looks like the Stack Exchange datacenter failover is working
sorta...
lol who broke it this time?
@ewwhite Windows, Apache, MySQL, and PHP
Oh, odd.
Very.
@Adrian About Supermicro... Shit?
16:43
@Adrian WAMP shouldn't be seen as anything more than a dev tool and these days it would be easier to set something up with VirtualBox or Similar to run a LAMP stack
@ewwhite I dunno. It's great stuff for piss-ant little orgs that don't bother with monitoring, clustering, and have no real budget or customers to worry about.
but Developers ...
@Adrian Another client was down today... I couldn't see the cause. Another engineer says it's a bad Fibre channel HBA.
@ewwhite That sucks. Lemme guess, no Vhost failover?
And I feel like that is the type of thing to trigger an ASR reboot on HP... or would give better logging. But I'm totally lost on the Supermicro hardware.
it's a physical server with fibre channel connection to SAN storage.
16:45
@ewwhite It's like Linux in the bad old days. You roll your own.
@ewwhite (A bit late but)... My dead 3ware/LSI battery also came from desktop with a partially hot case. (no air stream near the HW raid card).
Oh, chat is back to working. :)
So am I just being a hardware snob?
Yeah, I'm not impreassed by SuperMicro either, but half of that is the shit chassis those cut-rate assemblers tend to use.
SuperMicro is fine for Dev and Test/QA environments. I wouldn't run HA on it though.
I wouldn't want to push the hardware much either.
Hmm. I still can't get Profile information off the main site.
AH. It's in read-only mode right now
Still can't post on SF, can't load SU.
@Adrian but I see lots of it in data centers.
16:50
@ewwhite Because it's cheap? I suppose if you got the technicians to work on it all the time and the SREs to roll your own stuff, it's worth it.
We do.
except there are a lot of failures.
RAID batteries, system boards, backplanes, server crashes.
@ewwhite Think about Google. Are they running top of the line stuff? No, they're getting cheap commodity stuff and rolling their own HA.
they are spec'ing their own stuff.
Yes, but they have the processes and infrastructure in place to handle failures and make sure everything keeps working.
Had an interesting chat at our table with one of the Google SREs last Spring.
SF is back down
@adrian good to know they have processes.
@ewwhite Not that you'd believe it from the outside. I think they're more tightlipped there than the KGB ever managed.
Very nice.
Well I just got my /48 so maybe I'll go renumber my network today.
(I asked for a /56)
/123
@MichaelHampton I remember DeLong talking about how they're not handing out little subnets, switching the thinking from managing scarcity to managing a super-abundance.
Indeed. It feels kind of weird after 15 years of conservation.
17:08
Starting to think that IPv6 and DevOps are going to completely shake our profession out. So many so-called admins that think security == NAT and can't manage even simple scripting.
And there goes one more failed migration back to whence it came from.
@Adrian devops, yes. IPv6, I honestly haven't thought about it
@ewwhite NAT permits a certain laziness of thinking that's more than a little disturbing. Plug this little box in and suddenly your network's secure! Idiots.
ooh palindromic rep 40404
I don't think anyone thinks of NAT as secure
@Iain I know how to spell 40404 it's just I don't know when to stop.
17:16
like bananananananananana
or iaiaiaiaiaiaiaian
Or queueueueueueueue
:)
I'm waiting for the rain to stop so I can do a water change in my fish tank - I put some new bog wood in and now I have water that looks like tea
@ewwhite Perhaps you end up talking with more clueful "admins" than I do. =)
mayybe?
honestly, my only outlet is Server Fault
@Iain Poor fishies.
17:26
@WesleyDavid oooh no - they're all amazon tetras and the like - they will really like the extra tannins
but new wood leeches lots and makes the water a bit too dark
@ewwhite I do quite a bit of non-industry socializing around my motorcycles and home-brewing. The number of NAT & GUI-only "IT Admins" that try to talk shop with me pains me greatly.
Interesting that you group NAT and GUI.
Sadly I seem to belong to the GUI part myself, since that is almost all I have used in places where my coworkers also needed to understand things. And thus the knowledge which is most up to date. (Even though I started with CLI and Linux/Slackware)
@Hennes Mostly because that and a lack of scripting/Powershell knowledge is a common trait amongst many of the admins around here.
I am guilty of lacking powershell knowledge.
I think I started powershell twice.
(Not counting server 2012 install with no GUI)
@Hennes I don't use PowerShell since we're a mostly Linux shop, but I highly recommend learning it.
17:40
Aye, I should. If only for having it ready just in case
Bunch of my friends are Devs at various levels within Microsoft and it's getting a lot of attention finally.
Is it just me or is stackoverflow.com not responding to requests?
It's not just you! http://stackoverflow.com looks down from here.
@MikeyB There's a scheduled fail today.
There are scheduled fails at 16:00
Is my memory off, or is SO one of the few sites that's not hosted in the same method as the rest due to scale issues?
17:44
Uhm, no idea.
Any good tricks for draining a laptop battery? Right now I use full screen brightness, no screen saves, and prime95.
(new battery, 8800mAh, and roughtly a third of the weight of the laptop).
I want to measure how long it can keep its charge, and how long the old (6600mAh, 4 y.o. battery) keeps charge
@Hennes There's battery rebuilders here locally. I just take them in to them and have them rebuilt if I think the battery is getting hinky.
I would love to have some of them locally
But most of my laptops are between 4 and 8 years old.
@Hennes Ouch. I'm not sure I've ever seen a laptop battery still good after 4 years.
They are not. My sister uses my old lenovo 3000 C100 (pentium M, single core, 1.73Ghz. 2GB RAM, replaced HDD (it failed after 5 years)... Battery life about 3 minutes
Which means use it with power plugged in
But it is still good enough to surf the net, check gmail etc
Getting a new laptop seems excessive
And with a good battery it used to last around 5 hours (6 hours fab. spec, 5.5 tested on my own with low screen brightness)
So I rather replace it with a new battery with freshly produced cells than get a new laptop
The middle part is a problem
I get get relative cheap batteries, but without guarantee that those have been shelved for years.
True. If the laptop still runs fast enough to be useful there's no reason to pitch it.
17:53
nods
My own laptop is a Dell E6500. Think it is 4 years old. It lasts about 3 hours (used to be 5 hours)
When I leave home I used it for an average of 4 hours. Which now fails.
So I got a new battery
A bloody heavy/big one, since I expect them to be unavailable when I need to replace it again (say in another 4 years, or 8 years after the model was released)
WTF??? JS has a === operator but no !=== operator? Blark.
@Hennes A resistor.
@MikeyB wire wound ceramic dip ?
@Iain Isn't that a toroid?
@MikeyB You need !==
18:13
that kind of thing - can take a bit of power
18:37
Hmm. It's a bit old, but NARQ?
0
Q: Pretested commit with Jenkins?

SlamiceI got a request from the man upstairs to setup Jenkins for Pretested commit (through a build at Jenkins, if all unit tests and other pass then commit it and build a dist). We're using svn here. Is anyone clear about how this can be done with Jenkins? I am finding next to no information on it.

18:55
That's called DevOps.
It should scare the bejeezus out of you.
@MichaelHampton I really don't understand this whole Tomcat/Jenkins stuff. I probably need to figure out how it works at some point before I start interviewing
They blew up SF again. It just went away.
19:24
@Adrian No SF for you!
19:36
Yay, it's @RebeccaChernoff! When will Data.SE get newer data?
@Ward When will hell freeze over?
@MichaelHampton the day before data.se gets newer data
@Iain Bingo!
It's quite tedious adding up Gilles' votes on about 20 sites and mine on 10 to make sure I'm pulling ahead of him.
19:41
there's a few more than 20 sites, heh.
too many of them (;
@Adrian I don't think you need to know it
He's got about 96 accounts, but only the first 20 or so have significant voting activity.
0
Q: Other services like CloudFlare's Always Online?

user985171Since the website is about server problems i guess it is good to ask about this. I just want my viewers to be able to see a cached version of my website's main page and a couple other pages even if they are a month old when my server at my home is down. I use CloudFlare but i've read for some rea...

Nuke it from orbit
19:59
Who else is at work?
They painted the building
I'm trippin' balls woooooo
@MichaelHampton Thinking that through, guess that makes sense. (== → !=, === → !==)
@JoelESalas I'm at home with a nasty head cold, learning HTML5 & Javascript & Webworks programming.
00:00 - 20:0020:00 - 00:00

« first day (763 days earlier)      last day (4504 days later) »