The Comms Room

voretaq7

22:00

@ewwhite my bands are narrow, but my data is large...

ewwhite

@voretaq7 why don't you come down and interview?

voretaq7

@JoelESalas splunk is pretty nice

@ewwhite OK, but my 50 gig SQL archive is going to need its own pipe...

ewwhite

@voretaq7 we're hiring...

Joel E Salas

@voretaq7 I kind of hate it so far, all proprietary stuff

voretaq7

@ewwhite heh I might consider side consulting, but as insane as where I work is they let me hang out on Stack Exchange all day :P

freiheit

22:02

@JoelESalas There's a lot of python and editable config files hiding out in there...

voretaq7

(though I could use a nice raise.... </greedy>)

ewwhite

same here... remember, my DBA is #2 on dba.se

voretaq7

@ewwhite are you guys an Oracle shop?

Iain

@voretaq7 and Ed's place doesn't

ewwhite

@voretaq7 MySQL, Oracle.

voretaq7

22:03

@ewwhite HISS

MySQL

Bad.

Napster Bad. Only MySQL is worse.

like a Lars Ulrich solo that never stops

ewwhite

We need Windows DBA help... We need DevOps (but not @JoelESalas) and of course, networking and Linux.

Mark Henderson

@ewwhite That's approximately 390Mbps faster than I get at home and 480Mbps faster than we get at our office

Joel E Salas

@ewwhite Is it because I can't figure out OSX?

voretaq7

@ewwhite ...can I force you to replace your Linux environment with shiny fast clean BSD servers? :-)

@JoelESalas oh honey, NOBODY can figure out OS X!

Mark Henderson

And your uploads are 409Mbps faster than we get at both :p

ewwhite

22:04

@MarkHenderson It's a gigabit link, but the ASA 5510 is limited.

@voretaq7 no. there is one BSD server.

Joel E Salas

@voretaq7 A former VP insisted beyond insistence that we should move to Solaris

Michael Hampton

possible duplicate of Long-term storage of Business Critical data — Michael Hampton 27 secs ago

@Iain Can you tell me if you agree with that?

voretaq7

@JoelESalas ...and then Oracle octosquidded Sun and he quit before y'all realized how horrible your fate was?

Joel E Salas

@voretaq7 No, I quit

voretaq7

@ewwhite you have some application that requires Linux?

Mark Henderson

22:05

I know... just stop complaning :p

ewwhite

@voretaq7 yes... being a hosting provider.

Mark Henderson

I know I know you weren't complaining lol I brought it up

voretaq7

@JoelESalas ...because you realized how horrible your fate was? :P

@ewwhite meh, the users at $job[-1] never noticed

ewwhite

@voretaq7 but the clients who pay for Red Hat will.

voretaq7

(actually that's not true, they were happy the server wasn't getting rebooted twice a month for ZOMG CRITICAL SECURITY HOLE IN THE KERNEL patches. But Linux has outgrown that)

@ewwhite eugh

Dennis Kaarsemaker

22:06

@voretaq7 yes, it's only once a month these days...

voretaq7

<- not sure I have the stomach for working for a RedHat Managed Bitch environment again

at least the Microsoft Managed Bitch team got treated nicely by their vendor

@DennisKaarsemaker It's been a good 6 months since there have been any "Oh holy shit I have to patch that or we're all going to fucking DIE!" holes in the Linux kernel.

they mostly stick to breaking things like the Network Manager status icon these days.

Dennis Kaarsemaker

true, the latest one is only local

voretaq7

"Why fuck up the core OS when you can just trash the user experience?"

I really do wish (server) Linux didn't go out of its way to be "Not Unix" though

I can admin BSD, Solaris, AIX, even SCO if I'm paid enough

but I would be totally uncomfortable stepping into a Linux environment.

freiheit

@voretaq7 rhn.redhat.com/errata/RHSA-2013-0830.html

@voretaq7 Don't you mean "16 hours", not "6 months". ;)

voretaq7

@freiheit that's a local

(it's still face-palmingly bad, but I don't have untrusted local users on my servers)

freiheit

22:12

@voretaq7 Yeah, but that plus, say, a PHP app == OMG panic!

voretaq7

I run FreeBSD systems with local issues for as long as 3 months if it's not convenient to a patch cycle.

@freiheit I'd have to look at the exploit PoC, but it doesn't sound that bad -- you'd need to run arbitrary code on the server to launch the exploit and that's already a major breach

Michael Hampton

Yep, you'd have to exploit something else as well

voretaq7

It's not like "Pass off a carefully crafted DNS query and BAM: Root Shell that eats TXT entries!" or something insane

Michael Hampton

Though, on the bright side, it's good news for people who want to root their Androids.

Maybe. I dunno if it works on ARM.

freiheit

@voretaq7 Yeah, I may update the kernel on some servers that run web apps open to the world. Most stuff can wait until the regularly scheduled patch window.

voretaq7

22:15

now of course I have this level of confidence because I know the developers responsible for what we run on our servers, and I audit the code before it goes live, so I know it's at least reasonably secure.

Joel E Salas

Good news! The splunk license expired in February

2

voretaq7

@MichaelHampton it probably does - I think the performance events & performance counters code is shared across all platforms

ewwhite

I dunno, man... MaaS is bad Juju — ewwhite 16 mins ago

voretaq7

@JoelESalas Great, replace it with syslong-ng :-)

freiheit

@voretaq7 PHP. I manage PHP apps. sigh

ewwhite

22:16

@voretaq7 Don't you want a CHALLENGE?!

voretaq7

@freiheit we still have PHP apps here :P

@ewwhite syslog-ng is free

Joel E Salas

@voretaq7 That's the plan! kibana.org/infrastructure.html

voretaq7

(or more accurately "Has a free edition that can be made to not suck without considerable expense of capital or hours")

ewwhite

@voretaq7 I'm talking about the work with @ewwhite challenge!

freiheit

> A number of security related issues were resolved. Details of these issues will be released after a period of approximately one week to allow system administrators to safely update to the latest version.

^^ That's on the release announcement for an update released this week for the biggest PHP app I manage.

Joel E Salas

22:17

@ewwhite Challenge one: Communicating with your future superiors!

voretaq7

@ewwhite Working with you wouldn't be a challenge - doing managed hosting (again, without full control of the target environments) would be the challenge :P

The best projects at $job[-1] were where $client would bring me requirements, I would bring them a solution, and they would pay us to implement exactly what I spec'd.

The projects were always on time, on budget, and worked.

freiheit

@voretaq7 I'm confused. Where's the step where they see 95% of what you've done, throw out the specs, replace them with something completely different, and you redo 117% of the project?

voretaq7

the ones that sucked were when the client had "their own admin" who insisted we swap out the AIX for RedHat, the Oracle/SAP for MySQL and GNU/Something, wanted to run a local firewall....

@freiheit see above ^ :P

Joel E Salas

@voretaq7 How dare they want to use services common to what they already have!

freiheit

@voretaq7 Oh, hey, could you get that to work with our obscure single signon system, while you're at it?

Joel E Salas

22:21

@freiheit The CTO bought it seven years ago when a vendor bought him a steak at Lawry's then gave him a handie in the restroom

voretaq7

Our IBM partner accounts were great.
Me: "What do you want?"
Client: "We have this third-party shipping/logistics software, here's the specs and the vendor"
Me: <brief chat with vendor>
Me: <brief chat with IBM Global Services>
Me: "Here's your environment build. It's going to cost you about 750k and be ready in about a month."
Them: "Here's the purchase order."

@JoelESalas Hey, not my problem - they changed my design, so when it breaks it's $150/hr consulting fees to debug it and put a patch over the gaping holes :P

@freiheit Surprisingly most of those were easy (they were almost always LDAP or Kerberos. One time NIS, which was "interesting" because the thing that had to talk to it was a Windows program...)

@freiheit ...because a week is plenty of time to push out a major update

(or are you talking a tiny little not-breaking-the-universe update?)

freiheit

@voretaq7 It's a point release that shouldn't break the universe, but still needs testing. Luckily, this week is between semesters for us.

@voretaq7 I was thinking more single signon, not unified signon... SAML or something like that.

voretaq7

@freiheit never had the displeasure of working with anything like that (thank nataS)

freiheit

@voretaq7 We support two single signon protocols! :)

voretaq7

@freiheit that.... doesn't seem very single to me...

I'm Just Sayin'!

freiheit

22:30

@voretaq7 Well, see, one of them authenticates against the other... Just a simple matter of editing a few dozen XML files!

voretaq7

@freiheit ...which then falls back to authenticating against the first....

Ward

@Iain We had a tiny bit of rain this morning, currently back to super-nice and sunny now

voretaq7

@freiheit ...lies. nobody edits XML files.

freiheit

@voretaq7 Somehow we avoided having it actually work that way...

voretaq7

they're simply concatenated from /dev/random

freiheit

22:32

@voretaq7 Well, "edits" might be overstating it with XML... more like "curse at until it stop breaking"

voretaq7

(really! If you take the output of /dev/random and strip everything except <>?/=A-Za-z0-9 you get well-formed XML! Try it! It probably works with utf-8 too!)

freiheit

@voretaq7 Actually, it's "A authenticates with B, but pulls extra bits in via LDAP, and other extra bits in with another LDAP feed. B authenticates with LDAP. LDAP knows all."

voretaq7

@freiheit . . . so why doesn't A authenticate against LDAP, and B get sent to the software depot of the useless?

along with a sensible schema consolidation so you don't have two LDAP sources, because god hates multiple authoritative information stores.

Joel E Salas

@ewwhite I'm attempting to reproduce your XFS on ZFS, except on EBS

ewwhite

@JoelESalas It'll fail... servers will catch fire! Pinterest will go offline!

Joel E Salas

22:36

@ewwhite loling

this is so absurd CLOUD CLOUD CLOUD

ewwhite

pogostorage.com/products/pogorack/pogorack-overview.php

freiheit

@voretaq7 Well, making an app authenticate against B (CAS) is really simple. Making anything work with A (Shibboleth) involves editing at least 2 XML files, often 3 or 4... But A has all sorts of options about what it actually tells a given service.

voretaq7

@JoelESalas Hope it doesn't rain, because we put the cloud in the cloud to back up the cloud...

2

freiheit

@voretaq7 It's one LDAP directory, but the SSO software can't use the same connection for both User and Group information, because it hates humanity

voretaq7

@freiheit . . . LDAP WAS SUPPOSED TO SOLVE ALL OF THIS FUCKING SHIT

Hennes

22:37

/me mutters "airheads" :-)

voretaq7

@freiheit oh... well LDAP can't solve incompetent developers. There's only one tool for that

Ward

And, @Iain, White Fang is helping out at work today...

voretaq7

►

Ward

freiheit

@voretaq7 I thought there were many tools for that.... guns... knives... garrotes...

voretaq7

22:39

@freiheit See the video.

(because really, when dealing with developers when DON'T you want to kill every motherfucker in the room?)

creepy-ass doggie!
Probably gonna have nightmares now...

it's go-home time... me and my nightmares. (thanks @Ward!)

Ward

Not a dog!

moshimonsters.wikia.com/wiki/White_Fang

JohnMerlino

What is some good documentation for firewalls and proxies?

voretaq7

@Ward "Musky Husky", "puppies set" "resembles a white wolf", "totally barking" "tail-chasing bone-loving" .... sounds like a dog to me :)

freiheit

@JohnMerlino You need basic firewall and proxy concepts?

@Ward That sounds halfway to furry...

Joel E Salas

FUNTINUOUS WINTEGRATION

Joel E Salas

23:06

@ewwhite This is dog-shit slow (as expected), what do

ewwhite

@JoelESalas hire me!

Joel E Salas

23:23

@ewwhite Should I redo it as a stripe of mirrors??

Joel E Salas

23:50

@MichaelHampton Do you have that tirade explaining why disabling ping is bad?

Michael Hampton

@JoelESalas Not offhand, though I could probably write one.

12

Q: Security risk of PING?

I have been told that PING presents a security risk, and it's a good idea to disable/block it on production web servers. Some research tells me that there are indeed security risks. Is it common practice to disable/block PING on publicly visible servers? And does this apply to other members of...

Joel E Salas

@MichaelHampton Bingo, thank you! Bookmarked

Mark Henderson

@MichaelHampton I find blocking ICMP Echo is not worth the tradeoff

Matt Simmons

As a general announcement, the LOPSA Live Q&A session with candidates for the Board of Directors will be happening in one hour in #LOPSA-Live on FreeNode. It's open to members of LOPSA and the general public.

Mark Henderson

Ok it might make things more secure e against some ICMP payload attack, but it means as soon as I hit the edge of the network my tracerts stop working

Transcript for