« first day (1254 days earlier)      last day (4013 days later) » 
00:00 - 23:0023:00 - 00:00

00:01
@ewwhite I feel like you'd be a Ktown kind of guy
do you like walking?
@JoelESalas Well, my clients are just south of Little Tokyo...
@ewwhite Arts district is always an option, though it's unnecessarily expensive
btw we're going to your favorite sausage fest
and I'm trying to decide between just doing my JW/Ritz thing each month or getting a place of my own.
@JoelESalas Love that shit...
I tried to take my wife a week or two ago...
line was out the door
@ewwhite Did you end up somewhere else? Possibly at a pie hole?
@JoelESalas Room service at the Ritz Carlton.
she said, "eff this Hipster shit!"
I'd like to try the Pie Hole.
So I like downtown LA, mainly because it's changing...
but really, walkability
I s'pose Santa Monica has that, too
00:41
Don't we have a canonical "Use puppet, dumbass" answer somewhere?
@MichaelHampton If we don't, we should.
01:03
@MichaelHampton why, where?
0
Q: Sync multiple server configs

tdbui22I have a pool of servers and keeping them the same has been a challenge. I'm running the latest CentOS 6 and I'd like them to all have the same server configurations (updates and installed modules). I don't need my files sync as I use rsync for that. Currently I have to log into each of them and...

Yeah, so... I wasn't getting SNMP data for two months and Zabbix didn't warn me. Oops! And the reason why I wasn't getting SNMP data? snmpd is stopped
@MichaelHampton Hah. You didn't want that data, did you?
@ShaneMadden Well, I would have liked it before tonight when the disk filled up and I actually did get an alert... "MySQL is down"
Heh. I just got a "uh, disk is full" call a few minutes ago. Non-production server, so I'll figure out how badly the crappy monitoring failed on tuesday.
01:19
I read the bloody Zabbix manual cover to cover... it's time to ask Server Fault!
01:42
@MichaelHampton I'm actually working on my puppet configs to make sure my data collectors are always running
Notice: /Stage[main]/Ewwhite/Service[procallator]/enable: enable changed 'false' to 'true'
@ewwhite It's a client system and not (yet) puppetized. Won't happen until I convince him it's worth spending the money on.
Do you charge a lot for it?
It's not that, he's just a real penny pincher
but what's your charge for stuff like that?
Though if he agrees t the work he always pays on time and in full.
01:44
I don't charge specifically for puppet... it's more of a justification of my prices
Hell, I may end up doing it for him for free. He's been with me for almost ever. 2006 or so?
Just so I don't have to get interrupted in the middle of dinner anymore.
'ello
The Zabbix SNMP thing is really pissing me off though.
02:10
Oh, this is new...
@MichaelHampton so long story short, #monitoringsucks
@Andrew Everything in IT sucks. If it didn't, there would be far less need for system administrators to manage the suckage.
@MichaelHampton Only because developers make all IT software.
02:40
@MichaelHampton Wanna help me with a puppet decision?
03:14
damn puppet.
03:25
@ewwhite What's wrong with Puppet? My Linux guys drool over Puppet like the best thing since sliced bread.
@RyanRies I need to do a few things... having a hard time deciding how to implement
it's very powerful
@ewwhite What sort of suckage are you looking at?
@MichaelHampton All HP servers with management agents have a file at: /opt/hp/hp-snmp-agents/cma.conf
it governs the agents and email notification settings.
I'd like to ensure a specific email address across all systems with that file
struggling on where to define this and whether to just push a file or try string manipulation
I was about to try augeas, but there's no lens... and it seems more complex than necessary
I could also do this with templates
Config file content is here: pastebin.com/uuuvL5JN
template makes most sense, right?
03:49
@ewwhite I see exactly two uncommented directives and four other directives. It shouldn't take more than an hour to write a whole puppet module to control all of them and generate the file from an appropriate template.
I already wrote the template.
And if you don't care about them, and just want trapemail, you could do it in about five minutes.
I'm checking EL4,5 and 6 servers just to see if the file's format changed
in the early EL5 days, I had to exclude certain daemons
But yeah, it should be easy, you might even put it on the puppet forge.
and now, HP has a stupid bug that requires me to zero the contents of one of the config files.
Remove current contents of usb-device.sh by performing the following:
Note: The USB data from the hp-snmp-agents (cmastdeqd) will no longer be available until package is reinstalled or updated.

Open the /opt/hp/hp-snmp-agents/utils/usb-device.sh file in any Linux editor.
Delete current contents of the file.
Save and exit.
Restart the cmastdeqd by typing the following command:
/opt/hp/hp-snmp-agents/server/storage/etc/ cmastdeqd restart
There are a couple hp agent modules on Puppet forge
I still wrote my own
03:53
Text editor my ass. > /opt/hp/hp-snmp-agents/utils/usb-device.sh
Then it gets a file resource... needs to notify the agent to restart
That's all boilerplate code though
hmmm, to break this out into params.pp or not...
If you don't, six months from now you're going to look at it and wonder what you were smoking when you wrote it.
I end up looking at puppet modules once every month or two
because once it's in, it really does work well
I do have about 20 more modules to make on my to-do list, though
K. Module is live.
I'd like to add a variable in my ENC for additional alert destinations... hmm
 
2 hours later…
06:32
G'day
this is a love hotline?
2
how?
we ask love questions here?
@Portia I think they were just being fasitious (sp) it's a general chat room that tends to be off topic a lot
okay
lol
@AJHenderson: The comms room has a unique culture...
;p
It's topical at all ?
06:46
Well, its topical in never being about live support?
2
;p
 
3 hours later…
09:17
ugh, claiming credit card insurance is hard
like finding something worth answering on this site
you have to do it at specific times of the day and week, and as often as not it just doesn't happen
they owe me $305.00
so I will fight them on the phones, and I will fight them in the branches, and I will never surrender...
2
 
2 hours later…
10:51
@Iain it's topical as in topical cream
 
3 hours later…
13:25
Why don't you ask Google? — Michael Hampton 8 secs ago
14:07
0
Q: Windows: View network shares without sharing anything

Georges DupéronUsing windows, I would like to be able to browse computers and their shares on the network, without publishing any shares, administrative shares, and not even having my computer listed in the "Network" folder on other computers. When using windows, if I go to the "Network" folder, I get this yel...

14:25
@DennisKaarsemaker and don't mention the lube ...
14:36
@MichaelHampton is too quick on these questions. Not getting a look in!
It's Sunday morning and I'm bored watching a bunch of servers install updates. What else am I supposed to do?
Theres always porn.
2
strava.com/activities/113698521 that was not fun to go down on my bike - 33% is a bit steep
14:57
@Iain I'd get to 20 miles, look up, then get a bus home and a bag of chips.
@MatthewIfe beautiful weather today here - way too good to miss
You know it is still really nice outside here too. Might be worth going out
I may go and potter in the garden - there's a bit of tidying up to do after the recent storms
It's -5 and WINDY. I'm not going outside!
@Iain shouldn't you be riding a waterbike these days? :)
15:11
@DennisKaarsemaker I'm generally ok if I need a water bike there will be arks afloat. At times going up Gadlys lane strava.com/activities/113698521/segments/2502083374 there was water running in streams down both sides of the lane though.
Further south its worse.
I'm currently surrounded by lakes that were not previously there in December.
15:29
@MatthewIfe where are you ?
@Iain I'm still riding indoors here
@ewwhite strava.com/activities/113698521/segments/2502083374 convert that segment for your trainer - you'll love it 800ft in 2.9 miles
@Iain Oxford.
yeah, lake Oxford is pretty damp now
@MatthewIfe ah the Thames floods - could be worse, could be in Somerset
15:40
@Iain yea, I have been pretty lucky in all honestly. Lots of areas in oxford were flooded, I'm just surrounded by water but not submerged in it.
16:07
What. I just got an answer to a question accepeted. I posted the answer over two years ago.
16:23
@tombull89 the OP just came back
@tombull89 must be lag.
Fucking people. The questions are bad, and so are the answers.
@MichaelHampton examples ?
0
A: Can Zabbix alert me when an SNMP device does not respond?

BertYou could create a trigger to fire when the device does not respond, with the action being send an email to the administrator.

ah, but not bad enough to downvote
16:39
@MichaelHampton According to the answer its "when the device does not respond"
I have no idea how much of that answer is an answer :)
@MatthewIfe There's no known item name corresponding to this condition, even though it's obvious from the GUI that Zabbix tracks it. It's impossible to write a trigger without the item name!
Strangely, there IS one corresponding to use of the Zabbix agent, and that trigger already exists and is preconfigured by that template.
But none of the SNMP templates do this.
Is zabbix receiving traps or sending requests?
It's sending requests.
Can you set thresholds that are dynamic?
I mean, can you ask for sys.uptime and expect it ot fall within a given range/
or, does the event not even process in that case
16:57
RPM dependency hell!
WARNING: Can't read module /lib/modules/2.6.9-78.0.1.plus.c4smp/weak-updates/xfs/xfs.ko: No such file or directory
WARNING: Can't read module /lib/modules/2.6.9-78.EL/weak-updates/xfs/xfs_quota.ko: No such file or directory
WARNING: Can't read module /lib/modules/2.6.9-78.EL/weak-updates/xfs/xfs.ko: No such file or directory
WARNING: Can't read module /lib/modules/2.6.9-78.ELsmp/weak-updates/xfs/xfs_quota.ko: No such file or directory
WARNING: Can't read module /lib/modules/2.6.9-78.ELsmp/weak-updates/xfs/xfs.ko: No such file or directory
I'm dirty for even dealing with this
RHEL 4? Take it out back and shoot it.
Just upgraded from 4.7 to 4.9 :)
Selling client a new server...
been awhile since I've had to deal with weak-updates
I have three EL4 systems left.
I mean, getting to EL6 was a goal... but now @MichaelHampton says I need RHEL 7!
Yeah, but RHEL 7 isn't ready yet. You'll just have to limp along for a while longer :)
I think my affair with ZFS on the desktop is about to end.
17:13
tsk tsk, RHEL 4
I'm kicking 5 out the door as we speak
I kicked 5 out the door a year ago!
6 was giving us humongous performance problems a year ago
only after those were solved, we started killing off 5
Yeah, certain hardware had issues, as I recall. But I think @ewwhite knows more about that.
@ewwhite I forgot to ask.. are all the drive slots on a 380 connected to the same internal sas port, or are they split?
I see two sas ports going from the raid controller, but couldn't be bothered to dismantle the server to see where they went :)
@pauska Sure...
so, a DL380 G6/G7 has one or two 8-port drive bays.
these are direct ports... 1:1 to the controller, so no expander or oversubscription
the two SAS SFF-8087 ports you see coming from the backplane are 4-lane 6Gbps SAS connectors.
17:26
yea the ones I bought have one 8-bay
so the P410i controller onboard can handle 2 x 4 == 8 SAS drives.
ok
but how is the mapping done? each disk has one lane to each controller port?
1-4 go to one connector, 5-8 go to the other.
ah
so two ssds in 1+2, and two in 5+6 then
I actually split it the other way...
1 and 5...
17:29
I need more than 1 and 5.. four ssds
Smart Array P410i in Slot 0 (Embedded)    (sn: 500143802335E8F0)

   array A (SAS, Unused Space: 15085  MB)


      logicaldrive 1 (72.0 GB, RAID 1, OK)
      logicaldrive 2 (200.0 GB, RAID 1, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 300 GB, OK)
      physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 300 GB, OK)

   array B (Solid State SATA, Unused Space: 1012121  MB)


      logicaldrive 3 (400.0 GB, RAID 1+0, OK)

      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, Solid State SATA, 480.1 GB, OK)
yeah well thats the same I was going for
you use 3+4 and 7+8, and I'll use 1+2 and 5+6..
so that's a PAIR of OS drives... 300GB SAS. In ports 1 and 5. Split across SAS cables.
Yeah not going to be using OS drives this time
Oh, yes...
I'd also create a logical drive smaller than the capacity of the SSDs.
unless you need all of the space
@DennisKaarsemaker I'm trying to get rid of EL5, too.
@MichaelHampton Y u want to get rid of ZFS?
17:33
@ewwhite It seems to be leaking memory like a sieve. Maybe I'll do a NAS with FreeBSD or something. But it's not working out for me on the Linux desktop
@MichaelHampton Hmm, I'd urge you to try it the Red Hat/CentOS way.
no memory leaks
@ewwhite I used the yum repo from zfsonlinux.org. Is there some other way?
@MichaelHampton Oh, no... I thought you were using another distro
I may give btrfs another look
@MichaelHampton DOn't say that.
terrible!
17:38
@ewwhite What's wrong with btrfs?
I don't see what's wrong with ZFS..
@MichaelHampton unstable.
no momentum.
no mindshare
ZFS is pretty portable and can perform quite well.
I like the flexibility... I use ZFS for NAS/SAN... ZFS to host my Apple Time Machine backups (with HFS+ on top), XFS on top of ZFS for my produce clients, ZFS to back VMware, ZFS for large image stores (4 million+ files).
It only let me down once... and that was a Nexenta bug
Last time I tried btrfs was a year ago. Was unstable over LVM, returned a couple of oops's on my ssds without using lvm too.
Might be different now, but wasn't hugely impressed back then.
WRT storage, I have a very low tolerance for filesystem issues.
It doesn't offer anything that ZFS doesn't have.
It worked quite well for me on FreeBSD
@ewwhite My most serious issue right now is it seems to be eating all my RAM and not giving it back up. Despite I limited the ARC to 2GB.
did you reboot?
seriously, cut ARC down to something like 1GB
but make the change persistent in /etc/modprobe.d/zfs.conf
17:54
I even put it on the kernel command line. zfs.zfs_arc_max=2147483648 No obvious effect.
@ewwhite Why? Speed?
@pauska overprovisioning of the SSDs
because they're consumer disks... better for their health
I don't think this is a problem with the ones I ordered
@pauska always over provision if you can on a controller/combination like this.
unless you need ALL of the space
 
2 hours later…
19:30
@StackStatus
Status updates for the Stack Exchange network, including http://t.co/VY1vdMiR. You can also find more detailed updates on http://t.co/uCcZjNx5.
94 tweets, 513 followers, following 1 users
Chat kind of barely works...the main site is unusable
That's not "a bit" of increased traffic, it's probably some kind of DDoS attempt
I'm getting lots of loss from internap-gw.ip4.tinet.net onwards
posted on February 16, 2014 by ryan

Aloha.  I did some fun Powershelling yesterday and now it's time to share. If you work in an IT environment that's of any significant size, chances are you have firewalls.  Maybe lots and lots of firewalls. RPC can be a particularly difficult network protocol to work with when it comes to making sure all the ports necessary for its operation are open on your firewalls.&

20:14
@ewwhite What's the mechanism behind that?
20:39
@ewwhite you're spot on
they tripled the IOPS output on these ssds at 80% provision
well, on random stuff.. on pure sequential stuff it didn't matter at all
We have partially mitigated a DDoS attack against our network. We are continuing to watch traffic.
@JoelESalas It's just hardware voodoo
@pauska The cheaper SSDs don't guarantee performance as capacities fill. The expensive SSDs are more deterministic, but cost too much...
That's why I'm still doing more with PCIe SSDs
Well, we'll either go raid5 with 1.5TB of raw or RAID10 with 1TB
cutting 20% of either of those will be just fine
thanks for the tip
welcome
21:21
@Iain I ain't got time for that shit!
@MichaelHampton I was aiming to educate them
I guess I could see if unix.se wants it, but unix.se is still down...
@MichaelHampton oh no, don't migrate shit
@Iain No, I was going to see if they had a canonical
I didn't realise I hadn't chucked a close vote on it tbh
21:27
I decided to spend the "outage" time picking up some groceries, and I come back to THIS!
@MichaelHampton what makes the outage time special, just spin on your chair and you'll get that !
@Iain It was supposed to make those people go away, at least for a while!
Jeezus, there are a shitload of bad tutorials out there.
@MichaelHampton we see evidence of that here every day
@Iain I know, I just saw another one. I'm actually seriously considering starting a web site and writing good tutorials. Maybe it'll actually generate some ad revenue too.
@MichaelHampton it probably would
21:41
And then I just found nearly identical directions to the bad tutorial in an accepted SF answer.
do they/would they work ?
@Iain Yes, but it's difficult to maintain, and gives you spurious warnings.
I happen to be reading the front page right now.
-3
Q: Router Switch hanged down

Ali Muhammad BuriroI have a switch placed in department and connected through Media Convertor which is directly connected through optical fiber cable from #com Switch. Now the issue is Some times suddenly Endpoint Switch hanged down and led become slowly slowly blinking,and 5 desktop systems there, loss their conne...

What's wrong with this question (aside from the bad English)?
@MichaelHampton dunno
21:53
Oh, it was a fucking review audit
@MichaelHampton I quit reviewing because of the audits
@Iain Well, at least I passed it.
22:12
@JoelESalas All the btrfs oopses I could find were in reference to years-old kernels. Apparently you'd have to work a lot harder to crash it on a mainline kernel. (And I run mainline on my desktop.)
@ewwhite The only really missing feature from btrfs is it has no equivalent to a ZFS zvol, which I can live without.
@MichaelHampton just converted my filesystems to btrfs (since we talked about it and it was a year ago)
if things go horribly wrong i'll be sure to let you know ;)
@MatthewIfe And i went to ZFS... it isn't quite ready for the desktop I think.
I wanted something that does intelligent backups (my backup directory was consistently full) without going out of mainline.
@MatthewIfe Well, I have to sort out encrypted offsite backups... Easy enough with Bareos but I wonder if I can take advantage of these shiny new btrfs features somehow.
@ewwhite how do I control the over provision space? Is it enough to just leave 20% free on the array?
22:24
@MichaelHampton friend of mine thought it would be clever to use ZFS on his netbook (yeah, apparently they still make those things) - single-core Atom 1.6, 1GB RAM.
I think we can guess how well that ended
@MarkHenderson I've heard of people doing it. Never actually spoke to one, though.
@MichaelHampton This was his facebook quote
> Probably should have done a bit more research into ZFS before attempting to implement it on a Single Core Atom 1.6 ghz with only 1gb RAM...
I tried booting FreeBSD with 512MB of RAM and ZFS at the root system
It did not end well
> Dedupe was off, but compression was on...
Needless to say it's running *a lot* better with UFS
@pauska I had no issues with a 4GB FreeBSD VM and ZFS root. Runs like a dream. Even made me like ZFS.
Oh wait, strike that. It wasn't a ZFS root, the data was on ZFS.
22:29
I have a ZFS-On-Linux that I'm testing at the moment for our subversion repositories
@MichaelHampton I don't have 4GB of extra ram on my home desktop
But not as root
When I tried to install 10.0-RELEASE with a ZFS root, the result was unbootable.
getting two new lab servers with 96GB ram this week, so I'll do another test
@pauska Sigh, lab servers with 96GB... I wish :P
Our lab server has 24GB :P
(singular)
22:30
My desktop, with 16GB of RAM, is my lab server...
@MichaelHampton I have one of them at home too. You can run a lot of Linux VM's in there
Core 2 Quad with 16GB. Old and getting slow, but suffices
Yeah our CTO has new tech as a high priority
he wants all of us to be able to lab and test new products and then deploy them if it makes sense for us
@pauska Wow
Lucky you
and we don't want those tests to affect the production servers/san
Wow, that's like... a blank check!
22:33
The only CTOs I have the misfortune of dealing with are all stingy bastards whose primary goal is to save money and implement as many firewalls as they can
MSDN licenses, TechEd, lab environment etc
@MarkHenderson You might have me beat there... AMD Phenom(tm) II X6 1075T Processor
the only "bad" part is that these servers are going to make our SAN look like it's pre-ww2 tech
4x512GB SSD's in each
and it's going to be hard to actually test how it will perform in production
@pauska One of our clients went and bought a brand new, top-spec Dell R720 with 8x SSD's in RAID10, and then wondered why their software wasn't any faster. They never checked to see if they were disk limited or not - they had just read about how fast SSDs were and went with it
They're mostly on par with your AMD having more cores
@MarkHenderson Aaaugh, I had a Q6600 .... five years ago!
22:40
@MichaelHampton Yes well, the barber is always the last one to get a haircut
And that was the wrong processor, I think yours is actually this:
Which is substantially better
And any upgrades have a WAF of 0.0 which makes them unlikely :p
Yeah, I think it's time to upgrade my box. Or build a new one.
@MichaelHampton zvols are magic
@pauska For you, just create a logical volume smaller than the array's capacity.
Download the HPSSA Offline CD to configure your RAID volumes
@MichaelHampton I mostly only do minor lab stuff on there and play the occasional game, and I only buy games that are on sale on Steam which means that most of them are 3-5 year old anyway
I'll convert all of YOU to ZFS!!!
@ewwhite Then get better support for it in Linux
22:48
It works great
on Linux
Then why do I have to jump through hoops to get it installed?
that's my main platform for ZFS nowadays, since I left Solaris/Nexenta
two commands to install it
(on CentOS/rhel)
Hrm
Mine was substantially more than 2 commands
@MarkHenderson rhel?
@ewwhite CentOS
22:49
@ewwhite Tell me more about this magic.
OK, I lie - it was two commands to get it installed
But then it was a hassle to get a usable volume out of it
Or, maybe I'm just dumb
My Linux skills are more akin to a monkey bashing the keyboard
here's ZFS on a new server...
yum localinstall --nogpgcheck archive.zfsonlinux.org/epel/zfs-release-1-3.el6.noarch.rpm
yum install zfs
zpool create -f vol1 mirror wwn-0x500000e014609480 wwn-0x500000e0146097d0 mirror wwn-0x500000e0146090c0 wwn-0x500000e01460fd60 mirror wwn-0x500000e01460e7f0 wwn-0x500000e0163a9990 mirror wwn-0x500000e014611300 wwn-0x500000e0152a3550 mirror wwn-0x5000c5001cdf0113 wwn-0x500000e014605e30
zpool create vol2 raidz1 wwn-0x5000c5000c245113 wwn-0x5000c5000b307057 wwn-0x5000c5001cc4b45b wwn-0x5000c5000b30d58b wwn-0x5000cca00a0ea20c wwn-0x5000cca00a1a63ec
that resulted in two pools of disks and two filesystems.
@ewwhite What the hell are all those flags in line 3 and 4?
@MichaelHampton zvols are very nice... I have a situation where I have heavily compressible data...
@MarkHenderson the WWNs of each disk
LSI controllers show each SAS disk's WWN
22:54
@ewwhite Ooohhh ok
@MichaelHampton So you can run XFS/ext4 on top of ZFS zvols...
and get compression, snapshots, clones, etc.
@ewwhite So that created Vol1 as five two-disk mirrors, and then vol2 as RAID Z1 on the remaining 6 disks
yep
[root@Testa_Mirror1 ~]# zpool status -v
  pool: vol1
 state: ONLINE
  scan: none requested
config:

        NAME                        STATE     READ WRITE CKSUM
        vol1                        ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            wwn-0x500000e014609480  ONLINE       0     0     0
            wwn-0x500000e0146097d0  ONLINE       0     0     0
          mirror-1                  ONLINE       0     0     0
            wwn-0x500000e0146090c0  ONLINE       0     0     0
So, effectively RAID10 and RAID5
22:58
(only better)
@ewwhite That... is pretty damn nice
It's also simpler than what I did to get it working
so it's a volume manager + RAID + whateevr
most of my installs don't have raw disks like that... so I just use ZFS on top of hardware RAID.
00:00 - 23:0023:00 - 00:00

« first day (1254 days earlier)      last day (4013 days later) »