« first day (698 days earlier)      last day (4298 days later) » 

4:02 PM
I don't want to make this an SF question...
but I installed a server...
customer site...
and the HP agents are giving me:
Aug  9 11:46:09 Customer kernel: bnx2 0000:03:00.0: eth0: NIC Copper Link is Down
 
driver?
 
and then back up
4 second outages.
 
ip conflicts?
 
@ewwhite in the immortal words of Solaris: NIC and/or cable problem?
 
network storm ?
 
4:05 PM
or what @voretaq7 said
 
Aug  9 11:46:09 Tantalalicious kernel: bnx2 0000:03:00.0: eth0: NIC Copper Link is Down
Aug  9 11:46:13 Tantalalicious kernel: bnx2 0000:03:00.0: eth0: NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
 
@ColdT IP conflicts shouldn't cause link-down (they might make the machine unreachable, but there's still +5v on the copper) -- I'm assuming that "link down" is talking about the physical link though.
@ewwhite Flaky autonegotiation on the switch/NIC could do that too
 
It's HP gear... DL380 G7. The link is actually going down, as the server's agents, dmesg, message log all report the same
Even the telnet sessions are dying...
Aug  5 10:17:20 Tantalalicious telnetd[42226]: ttloop: peer died: EOF
 
faulty cable could even be the culprit :|
 
@ewwhite my money would be on bad cable, followed by dodgy autonegotiation, then bad port (one of the fingers in the RJ45 jack isn't making contact)
Check (1) and (3) first because (2) is a pain in the ass to fix and could be affected by either of those.
(and if it's all HP gear dodgy autonegotiation isn't very likely -- not that it's likely at all these days, GigE gear is pretty rock solid...)
 
4:09 PM
@voretaq7 After I installed the server onsite, the customer moved the server room to another floor.
then replaced the networking... I had to yell at them for installing a 10/100 switch...
 
ouch
 
then there was a flooding problem... (sprinklers...)
they put a Gigabit switch in... I think it's some form of Linksys.
managed, though
 
mmh... same cables and everything, just the Linksys switch changed?
 
no, they moved the server room up one floor... for some odd reason
 
I can't imagine Linksys/Cisco having broken autonegotiation, but if everything else is constant that could be it. Try re-seating or replacing the cable first though - that's a free-to-cheap and relatively easy thing to test :)
@ewwhite . . . ::huddles in a ball:: too many changes... too many variables... JUST BURN IT ALL DOWN
 
4:12 PM
so the cable could be some janky Cat5 cable with a broken clip.
The server's name isn't really Tantalalicious
I'm also surprised nobody picked up on the "telnet sessions are dying"
 
alright everyone place your bet lol a) faulty cable, b) autonegotiation issues or c) nic damaged
 
faulty cable
 
@ewwhite we're just sort of quietly plotting your demise
 
a mouse is playing with the faulty cable which is why the link goes up and down every 4 seconds
 
besides according to the Linux weenies BSD has been dying for about 30 years now - the longer Telnet holds on life support out the better BSD's long-term chances
@lsiunsuex ... don't laugh.
 
4:15 PM
Oh, the motherboard was replaced, too
 
@ewwhite . . . OK so now we really are in the "too many variables" territory :)
 
guess that's more of a jigsaw puzzle then anything else!
 
@lsiunsuex I've had rats chew through my serial cables in some old NYC installations.
 
@ewwhite This is why NYC code requires metal conduit...
 
cats chew on my network cables at home :(
put a cat in the room - the mouse will be dead or another server will show signs of problems
 
4:17 PM
(I don't think it's required for data cabling. Any AC or DC power cabling requires it though, so TECHNICALLY your POE needs to go through a metal pipe...)
 
@voretaq7 Bronx produce market... conduit, pssh...
 
shotgun debugging
(Find the developer, put the shotgun in his mouth and yell until the bug is fixed)
 
Is there anything I can check host-side while I wait for customer to replace cable?
 
any other servers on the same gigabit switch? ping that?
or - have them jack a laptop into the switch and ping it
 
No other real servers.
oh wait...another linux server
iperf...
 
4:26 PM
@ewwhite not really -- the server is telling you you're losing link, so the problem is hardware
 
I'm asking for access to the switch.
 
probably cable, possibly jack, occasionally one of the actual NICs.
 
motherboard-based NICs. Never had one fail.
 
My bet would be a cable that's not quite plugged in all the way, FWIW/if anyone cares.
You know the kind, broken clip, wriggled its way loose or didn't get pushed in all the way in the first place...
 
@HopelessN00b yeah that or one that got kinked/damaged in the move are the most likely problems
 
4:30 PM
So what do I get if I'm right and I win the bet? A real prize, or one of those phony cop-out prizes like "respect" or "satisfaction"?
 
I've messaged the customer.
We'll see. I'll announce results.
 
Do you bill double for troubleshooting faulty cables? Always been my dream to be able to charge quadruple for having to deal with that infuriation.
 
A kitten.
(@WesleyDavid)
 
@HopelessN00b You can have me, but only if you love me and pet me and call me George.
 
Better than "respect" I guess.

I *can* rub your stomach and pet you backwards, right?
 
4:36 PM
@HopelessN00b If sufficient quantities of gushee foods are dispensed, yes.
I'll still poop in your slippers though.
 
I know a guy who is a RedHat employee and knows everything about RHEV (or... close). He has been looking for jobs in America -- anyone need a RedHat guru, not just in RHEV, let me know.
 
@WesleyDavid Why is he leaving?
 
RHEV? Red Hot Endless VD?
 
@ewwhite Wants to move his family to America. Tired of the heat. (Israel)
Or Canada.
 
4:39 PM
@WesleyDavid If I hook him up, will you not poop in my slippers? :p

Seriously, I assume he's looking for a pretty senior sysadmin position?
 
OH... hmm.. Red Hat seems like it may have a presence in the US...
 
He's interested in Canada. RH could relocate him, but right now the positions that RH offered him in north america involve lots of travel.
He has a 3 month old, so lots of travel is not to his liking.
@HopelessN00b And pretty much. I think his skills would best be used as a Sr SysAdmin / Engineer working on very large scale infrastructure.
 
Research labs.
Argonne... Fermi lab.
 
There's a very big webhosting company based where I live that was looking for such a person a few months back. I could inquire f the position's been filled or not, fi you think that would be to his liking.

I can't actually vouch for the company being a good place to work or anything like that, but I was looking around for a different job this spring (before I found the one I'm at), and they came up. Big tech employer in these parts.
 
@HopelessN00b Somewhere in the midwest...
 
4:45 PM
Ohio, to be specific.
 
Let me guess... Jumpline? I'd never do that to even my worst enemy. =)
 
Nope.
 
Oh okay
I suspect that he'd be more interested in a corporate environment rather than a hosting environment, but I'll mention it to him
I thought @ewwhite might know some fast paced, big infra places that a RH wizard would fit in. =)
 
SpaceX is hiring!
I do.
 
SpaceX, lol.
 
4:47 PM
Actually, this fellow probably would have been awesome for that ALL-HAIL-THE-KVM place you left Ethabelle behind to die at.
=P
 
Please, work me 80 hours a week for average wages, because space is so cool!!
 
@ewwhite Also, I found out yesterday that SpiderOak / Nimbus.io is located in the greater Chicago area. I didna'e know that.
 
@WesleyDavid what is that?
 
@ewwhite SpiderOak is an online backup / storage company that is actually different than the rest, it appears
nimbus.io is their cloud storage system that purports to be better than S3 and built 100% on open tools. $0.06 per GB
Also, cool blog: spideroak.com/blog
 
Can anyone help me try to repair a master.mdf ?
Error: 9003, Severity: 20, State: 1. is not a fun error
 
4:50 PM
sigh
 
@Landmine By repair, you mean "restore from backup," right?
 
I wish it was my database so that I could restore it from a backup
Sadly, it is not. They have no backups and there was a power failure, and not the service will not start.
 
Hacking a repair into the master database is something I've become too wise to do since I outgrew my stupid a while back... some of my stupid, anyway.
 
@Landmine . . . hang on, I have a completely unhelpful observation to make!
19 hours ago, by voretaq7
user image
 
I was really hoping I could tell them more, than they are dumb for not making a backup. But that wont help.
 
4:54 PM
19 hours ago, by voretaq7
user image
@Landmine ...you can let Insanity Wolf there do it for ya...
 
@Landmine plenty of software out there that promises to be able to do it...
 
@voretaq7 Dude, you so don't get Insanity Wolf.
 
(seriously, you're pretty well screwed -- you can google up software that will try to fix it, but run it on a COPY of the database, obviously)
 
I'd rather not throw $100 bills at software that may or may not work.
 
@WesleyDavid I do, but even Insanity Wolf agrees that backups are important.
 
4:56 PM
 
(His backups may be on punch cards and paper tape, but by god he has them!)
 
@Landmine Don't throw your money at it, tell your client they'll have to.

Backups don't seem too expensive now, huh?
 
Haha, that is a good argument
 
@Landmine You're charging by the hour, right? Or are they on a managed service contract that this somehow falls under?
 
@Landmine The best way to sell a client on disaster recovery is to have a disaster...
 
4:57 PM
lights @voretaq7's office on fire
 
It's only master.mdf; prop up another copy of SQL server on another computer and see if the actual application DBs are corrupted.... If good, just reinstall SQL on the server, attach the application DBs and Bob's your uncle.
 
Might I interest you in some Unitrends products?
 
@WesleyDavid shrug Nothing of value here.
 
Basically, the company I work for sold a product on a server to the client. The client owns the software and the computer it is on.
 
4:59 PM
@ChrisS can you just copy the DBs over like that with MS SQL?
 
It is on them, but my company wants to try to help since they spend so much money with us.
From what I can tell, it is only the master.mdf
 
@voretaq7 More or less... You'll still lose all the Instance's settings and security; but the DBs will move easily.
 
@Landmine Well then you shouldn't touch the machine, but if what @ChrisS said will work with MS SQL you can pass that advice over to them
 
Help them by convincing them to do backups. :p
 
@ChrisS well that's a pain, but better than losing all your data :-)
 
5:00 PM
@voretaq7 Yeah, PITA, but it could be much worse.
And you'll spend just as much time dinking with those recovery tools.
 
"I never heard anyone in this industry say 'OH MAN I LOVE my application - you can totally throw away all our data, but you gotta keep the application man!"
 
@voretaq7 Is that a challenge? I know some pretty dumb people "in this industry"...
 
So the company I work for has about 75 of these sites and they want me to think of a way to back them up remotely each month. Any software recommendations?
 
@Landmine After you pass along that advice some other helpful phrase-book things you should use in the conversation are "Backups", "restore tests", "UPS", "battery-backed RAID cache", "Do you even CARE about your data?!?"

(well OK maybe not the last one)
 
@Landmine What's the OS?
 
5:02 PM
@Landmine I recommend you don't get into backup as a service
 
(including version and service packs and major applications and RTO and RPO)
 
@Landmine Laugh and suggest they get a backup specialist to walk into that minefield for you?
 
@voretaq7 Unless fistfuls of money are demanded and received.
 
All windows
 
speaking as someone who worked for a company that did it - it's an endless well of pain and misery, you WILL do it wrong, clients WILL lose data, and YOU will get the blame...
 
5:03 PM
Haha, that is the answer I want to give, but I have a feeling they will tell me to get creative and think of something.
 
@Landmine ...but $5 says "not all the same version of windows, and not all with the same network capabilities"
 
Correct
 
@voretaq7 Not so much that you will do it wrong, but when the client install a new firewall, or loses their connection, it's your fault that the backups didn't occur via magic.
 
Some machines are Windows XP, Server 2003, Server 2008 and Windows 7
 
8
Q: How do I stop a bed frame from squeaking?

Tom SquiresI recently bought a second hand wooden bed frame. All is good except it squeaks quite a lot when I move around on it (ahem). Any advice on fixing the annoying squeak? EDIT :: photo of the joint

 
5:04 PM
@Landmine Unless your company is prepared to own the servers 100% end-to-end and dictate inviolable network connectivity requirements you should NOT attempt this.
 
All the connections are whatever they select, and sometimes it's a $15 a month DSL connection
I do not believe that they are.
 
we usually stuff pillows behind the head board. that seams to help soften the blow
 
So maybe what I should do is build a plan for my company to hand out to the clients for them to take care of their own hardware and software
 
@Landmine ...and you want to move how much data over it? :)
 
@lsiunsuex Lol. Resisting the urge to suggest telling his girlfriend to scream loudly during sex.
 
5:06 PM
isn't his a dupe of a dupe serverfault.com/questions/416003/…
 
Hopefully, no more than 200 megabytes a month. These are small databases.
But again, sometimes the databses are SQL 2000, 2005, 2008, 2012 or Access even.
 
you should strongly recommend they make backups, and give them a list of things that must be backed up to restore your software to a working state, but beyond that I would stay the hell away from the issue.
yah if you don't even have a consistent DB platform there's no way dude.
 
Thanks Voretaq, I agree with you there.
 
The first step of designing a backup plan is standardization.
 
I'm only talking about half the systems
Some of them are Windows 98 FE desktops with a modem for dial-up remote connections
 
5:08 PM
@Landmine crashplan.com
 
Guess this is what happens when a company is around for 30 years and I'm the first "IT guy"
I run crashplan on my home server for remote backup
 
@ChrisS I wouldn't even make the recommendation TBH -- it sounds like this software is getting put on everything from Grandma's old 386 to a deluxe g8 HP system
 
@Landmine OK, who in your company is stupid enough to suggest offering backups of a Windows 98 machine connected by dialup?

Kill thm, kill them ***now***.
 
Sales people who sell and make promises to get the contract.
 
@voretaq7 That's "Gen8", They didn't want their servers getting confused with Governmental Forums on Economic Planning
 
5:10 PM
Sales people, even better. Murdering one of those is a victimless crime.
 
@HopelessN00b effing markdown
 
Well, I'll let my boss know that, there isnt anything that we can do about their database.
Thanks for all the input guys
 
@Landmine UNGGHHH - I know those types of products. You're only hope is to set yourself on fire in protest. sloshes gasoline We'll always remember you as having lived life as you died. Screaming in pain.
 
It is refreshing to talk to people with an understanding.
 
@lsiunsuex Silicone caulk works great, though it's hard to separate the parts once the caulk dries.
 
5:16 PM
are there any web design/web savvy people online?
 
me me me
 
Savvy with what?
 
I have a client whose current website is awful...
they've been working with a firm to create a new site.
 
The new site is awful, too?
 
@ewwhite What functionality do they want out of the site?
 
5:18 PM
The new site is parked on one of my systems...
 
So someone made an account on superuser just to irrelevantly nitpick an answer of mine superuser.com/a/459527/2381
 
but the developer has taken 10 months to get to that point..
 
@ewwhite Monkey on a stick, that's criminal.
 
Oh $DEITY, flash....
 
@ewwhite I made that stop after 5 seconds..
 
5:18 PM
and now their phone line is disconnected
 
@ewwhite Is it slow because it's on a test box, or because the developer is a moron?
 
fucking flash
 
so I have angry client who spent $5k on that...
 
Needs more 3d.
 
Oh, there's Flash there? Adblock must have gotten it.
 
5:19 PM
and wants me to help them to find someone to finish it and roll it into production
 
Adblock didn't for me, I run mostly default settings
 
The CMS appears to be joomla.
 
sorry, not flash. was a dumb loading image - immediately assumed it was flash
@ewwhite any money left and timeline ?
 
@ewwhite The problem is he's going to piss off his neanderthal coustomers. Chefs make sysadmin seem like well adjusted people in comparison, and their not gonna like all the website changes at once.
 
Oh god, it's Joomla.
I managed to get an error page out of it.
 
5:21 PM
I have a good friend who's a designer/developer... and he says getting the site prod ready with the outstanding changes will be 40+ hours.
the buttons appear to be images... so now there's photoshop work involved, too.
 
@MichaelHampton And one giant php file generates the whole thing. It makes kittens cry.
 
@ewwhite That doesn't sound unreasonable.
 
Oh god, flash intros. Who is that said flash intros on websites are like blocking customers from entering the store so you can tell about your store?
2
 
It's horrendous. The contractor might have an eye for design, but he probably keeps it in a jar on his desk.
 
so as an honest admin, what do i tell customer?
1). Scrap it?
2). Try to find someone to fix it (even though none of the devs I know want to touch it)
3). ??
 
5:22 PM
4) Profit
 
I'd poke around the Joomla site and see if you can find a dev there that would want to take it on..
 
The original developer profited and skipped town...
 
Ouch
 
I don't know anyone who does Joomla. I wouldn't touch it with a 10 foot pole.
 
Is customer angry at it being unfinished or angry about the fact that it looks like ass?
 
5:23 PM
I'd scrap it, and start over.
 
5). Do I tell them that the design is outdated and tacky?
 
Option 5, start over with someone who knows what they're doing? (Fsking touchpad)
 
@ewwhite They almost certainly wont listen to that no matter how correct the statement.
 
@ChrisS Exactly, because they APPROVED this design.
 
I ran into that with my company's latest redesign (which we paid way more for too). tried to tell them their design looked 5 years old already, got shutdown very quickly that I was being negative and that it was OK because 5 years old still looks better than most webpages.
 
5:25 PM
"hmmm, your car needs more speed holes"
 
I won't even install Joomla. I'd sooner shoot the developer.
 
@ewwhite contact my designer - bondtogether.com
 
Oddly-enough, this deal was done because the original developer owed the company $$ for messing up the original version of their loosely-related site... undrest.com
 
but - hes in toronto on, canada and i'm in NY so not sure if the distance / time difference would be a problem for them
 
(NSFW, maybe?)
 
5:27 PM
@ChrisS Why not? Both produce the same amount of hot air!
 
@ChrisS how did the final product look?
 
@voretaq7 Yeah, sorry, that was sarcasm. =]
 
@ChrisS seems busy, but I don't know web stuff that well.
 
it is a pretty poor design. really nothing special. we redesign those kinds of sites usually, not the original one
 
Yeah... Very busy, unorganized, no flow... I have a lot of issues with the site. I was on the Advisory Committee, but basically got kicked off because I kept bringing up all the problems and most of the solutions to those problems were "start over". The site is one of their cookie cutter designs too, which was "personalized" for us. I'm a little bitter about the whole experience.
 
5:30 PM
@ChrisS ...mine wasn't :P
@ChrisS @ewwhite everyone I know is all Drupal
 
@voretaq7 I had suggested and demoed Drupal for my company's site... We still have hosting at our old provider for $3/mo; I keep it around now for various file hosting and such.
I've made a couple simple Drupal sites for people where they wanted something stupid easy to use but looked fairly professional, and had a budget with 2 digits.
 
Drupal is the EMACS of content management
 
So for my messed up site? No recommendations? The start-over suggestion may be a tough pill.
 
call my designer
 
@ewwhite I'd at least recommend they drop the flash intro and go straight to the front page. Try to convince them that overly-long, blocking intros are a serious turn off that will actually cost them business?
 
5:35 PM
we may be able to fix / finish it
 
@ewwhite Offer them both options - an estimate for T&M to finish the bad design & launch, and an estimate for a start-over
 
I'd still try to find a Joomla dev and see what they think... But start over is a close 2nd place option. =/
 
without access to the original files though (u said the designer isn't answering) it'll be a rebuild / redesign
 
@lsiunsuex I got a copy of the site.
so I have it now...
 
And make sure they know the second estimate will have to be done at some point -- they're just delaying it
 
5:36 PM
and the CMS.
 
so we might be able to finish / fix it
mario g from buffalo referred you - tell him
 
@ewwhite that site is quite hard on the eye
 
@Iain why so?
 
the transparency of the text over the background
the fonts on the drop downs are crappy too
 
@ewwhite BURN IT ALL DOWN! :-)
 
5:48 PM
Apparently Microsoft is trying to make WMI an open standard OMI blogs.technet.com/b/windowsserver/archive/2012/06/28/… collaboration.opengroup.org/omi
 
*
## TODO: if you're looking at this file and aren't busy doing
## something else very important, please consider taking a couple of
## hours to refactor it (:
*
Ah, it's a good day
 
although i think web developers that use joomla, wordpress, etc... are pussies and their doing it wrong. any pro can do it from scratch in half the time. those frameworks are garbage IMO and should be avoided at all costs
but thats my 2 cents.
 
@JeffFerland so... refactoring code are we?
 
I'd actually be pretty excited if that happened in a useful form on linux mac and *BSD
 
I'm still trying to figure out how the whole thing works to begin with.
 
5:50 PM
@JeffFerland waht languafge is this file in and is it a single script or something other files depend on
 
All software sucks. (Axiom)
All web application frameworks are software. (given)
All web application frameworks suck. (by inference)
 
@JustinDearing It has 40 lines of imports and EVERYTHING depends on it.
 
Hello Everyone. I am looking for Books/Resources which explain TCP/IP kernel implementation.
I only know about:
1) Internetworking with TCP/IP: Vol.II, Design, Implementation, and Internals
2)TCP/IP Illustrated, Vol. 2: The Implementation
but these are quite dated. I am particularly interested in Open source implementations. Any ideas?
 
@Bruce TCP/IP code hasn't really changed since iptables was brought into existance
 
@Bruce TCP/IP hasn't changed
 
5:52 PM
@voretaq7 once you understand that, and accept that all SW you ever write will suck, you can begin to suck less
 
(at least not v4)
 
Are there any books which talk about the Linux implementation?
The Stevens Vol 2 is on BSD Reno
 
@JustinDearing WEBM and it's related standards are published, have been for a while, and several other manufacturers already use them for various things... MS just like to pretend like they invented it.. But it's just like AD is an implementation of LDAP and Kerberos (which existed long before AD).
 
@Bruce it... REALLY hasn't changed :-)
Read Vol.1 (The Protocols). All implementations work out to be the same state machine (even if it's implemented as a big switch() with some goto's)
 
@Bruce from an API point of view the linux implementation is virtually identical the BSD implementation. Why do you need to dig deeper? What are you trying to accomplish?
 
5:54 PM
@voretaq7: I am trying to implement my own congestion avoidance algorithm. I know it hasn't changed but I need to know the internals to implement my scheme
 
Once you understand the FSM (flying spaghetti monster or finite state machine -- take your pick) you can read the code and understand what it's doing better than any book :-)
@Bruce ahhhh - OK. So you need to read Vol.1, and then look up how you drop in congestion avoidance protocols for your specific OS-of-choice
(I know FreeBSD has (or will have soon - I've lost track of its development status) modular drop-ins for that with a nicely defined API -- I think Linux does too. Kernel docs would be your best bet there)
 
@ChrisS I know AS is LDAP and kerberos. I did not know about WMI. My question is , is there a linux tool that will let me do "SELECT * FROM Win32_Product" to a windows box i nthe same manner that I can use the openldap client tools on AD?
 
@JustinDearing probably not.
 
@voretaq7: I have read Vol1 (probably the best TCP I know of). I am a total noob when it comes to kernel programming. I don't know what kernel docs are. I have tested my scheme on ns2 till now
 
@JustinDearing Yeah. There's a WMI implementation for Linux
 
5:58 PM
@lsiunsuex Yes, I could implement a full featured CMS in half the time as WordPress, but I can't imagine a client that would wait the four years.
 
I don't remember the name, but I saw it when looking at doing WMI polling with Cacti
 
@MDMarra Works like a champ. I use it daily via credentialed Nessus scans.
 
funny, i write all my CMS's custom for the site i'm building and i get them done single handedly in < 4 weeks
 
@Bruce A wiki or blog on your project would probably be a good thing to keep. Worse case you have a somewhat organized notebook, at best case someone smarter than you might rip some of your ideas apart and save you some time.
 
@MDMarra I know nagios has one as well, but does it support wql queries? Nagios wmi+ exchange.nagios.org/directory/Plugins/Operating-Systems/Windows/…
 
5:59 PM
@Bruce linuxquestions.org/questions/linux-kernel-70/… <- that site has a list of good starting points. Generally the docs are the code though
 
@Zoredache I thought so, might be wrong about that
it def lets you do wmic stuff
 
Anyone besides me keep a separate monitor at their desk with dashboard info (nagios/wug/whatever) running simply to make their boss/coworkers think you are deligent/busy? :)
 

« first day (698 days earlier)      last day (4298 days later) »