« first day (13 days earlier)      last day (5254 days later) » 
00:00 - 18:0018:00 - 00:00

00:38
Yes Apache can control how many children processes are started (or threads if mpm_worker)
Just change MaxClients
01:37
:6640 Sorry, stopped for dinner
I have MaxClients 150
it was 250, I changed it on someone's advice. @KyleBrandt 's I think :-)
It's mpm_preform (because this server has mod_php instead of fastcgi) and this situation will use up all the apache process slots
Killing apache and restarting it solves the issue one time which is the aim of this script
Otherwise, apache spans up to maxclients children and they eventually all just end up hung
and no more clients will be served...
what is your MaxRequestsPerChild?
sec
StartServers 5
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 10
</IfModule>
MaxClients 150
MaxRequestsPerChild 10000
no, sorry... I overide it to 3000 elsewhere
hrm...
I did that to solve this issue, did no good
so I may raise to 10000 again
It's very weird, never seen this before, we have like 4 other apache servers
6 actually
I'd guess 80% of the time it's one given wordpress site
which today I moved to a different apache server
so I'll see if the problem follows it
Here's the last /server-status report from ~3am this morning before it crapped out
and I woke up to /etc/init.d/httpd restart
which I am sick of doing at 3am, lol
An hour earlier it was exhibiting the issue but hadn't hit MaxClients yet
So, we'll see what happens with zachdotsey.com on a different server...
01:58
@Josh, hrm... :|
Actually...
I have a sneaky suspicion this might possibly be related to cPanel's leechprotect service...
on @Warner's advice I've been checking MySQL show processlist when MySQL usage is high, and leechprotect is often in there
I wouldn't know. I don't use cpanel.
which is (obviously) tied in with apache... although I thought as a 403
:6683 I wish we didn't in so many regards :-)
This server has to as it's a web hosting company that needs cPanel's management functionality. My own customers I am transitioning to custom built software.
If the problem returns leechprotect will be the next to go :-)
As always thanks for your advice/suggestions @Zoredache! This has been one of the worst systems admin problems I've had to solve yet, in terms of difficulty to track down!
(and of course thanks to @KyleBrandt and @Warner too, all of you have been helpful :-)
Looks like it to me.
02:09
I concur
but lack the rep to vote :-)
> This belongs on Super User, but we'll move it there. You don't have to do anything. – Chris S 1 min ago
I love it :-)
Well get to answering questions... Don't spend time on the tricky ones they almost never give you any rep
People cross post the instant you tell them it belongs on another site......
:6698 Right
Yeah. Lesson #1, if you have to think hard about the question it'll get you very little rep.
You have to pounce on the ultra easy questions to get the most rep.
:6700 True. But they're so much more fun!
I worked hard pouncing to get over 1k. Then I was satisfied
02:11
Yes; fun and rewarding, but you rarely get more than 1 vote for it; and that's rare.
enough to see splits and also show that I'm not a total noob, to be taken at least semi-seriously :-)
I'll go serial upvote you....
LOL, then I'll probably lose rep, haha
The system will automatically undo that if you serial vote someone (up or down to my understanding).
(assuming you've upvoted at leats one thing I said and you then set off the serial upvoting alarms :-)
:6708 right. IDK what the threshold is, but I know it's set up like that
02:13
For down it's either 3 or 4 IIRC.
I would think up would be more, but I really don't know.
I seriously hope it's not like that for up because I know I've upvoted most of you guys more than 4 times :-)
Well you have to do it within a short period of time; like on the same day or something like that.
ah, yes, good to know
I should spend more time pouncing on easy SF questions... personally I find easy questions on SO much more frequently. Especially for PHP, LOL
And I like to throw an easy one into the mix myself every now and then ;-)
02:31
Hm. I can't reboot my Win7 workstation via RDP, I only get Log Off or disconnect..
click on the 'windows security' button
It will bring up another screen where you can click a dropdown to shutdown/reboot
:6721 Woah really?
I have never been able to reboot windows via RDP...
I so rarely use windows, but...
:6725 Awesome, Thanks. I'll be sure to check that (if I remember!) next time I need to reboot/shut down a Windows VM
02:48
Heh. The method I use for that is to start a cmd window and enter "shutdown -r -t 2" Didn't know about the menu option.
:6728 EVEN BETTER!!!
I'm great with the linux command line, really sucky with windows'
Works on XP too.
:6731 Good to know, as most of our Windows VMs are XP
Well I'm off for the night. I'll catch you guys later!
G'night
03:10
@Zoredache: Ah! Thanks man!
03:43
I wish that shutdown command had something equalent to the reset button on a computer
03:56
What do you mean?
04:11
Well, the shutdown process hung up
should have taken a hint when I couldnt even terminate outlook via task manager
ahhh ... gotcha, yea things got that screwy not much you can do
nope. good thing this is just my own workstation, I'm at the office in one hour and can hard-reset it then
The only suggestion i have for situations like that are network accessable PDU's like these guys apc.com/products/resource/include/…
04:48
Yeah, I'd love to have those on my server room - but a bit overkill for my workstation at the office :)
haha, at my last job we actaully had a bunch of the 0 U 14 port ones laying around and so all my my equipment in my desk was hooked to one
never got around to putting it on the network though
Do you have anything more definitive to identify the error, @Josh? A particular message in the log? Restarting based on Apache children is a bit kludgey. If you don't have a better thing to watch for before taking action, a better approach might be to use Nagios, as it could initiate the action when the Apache instance stopped responding.
@josh: I'm almost certain that you are having a problem with PHP scripts running with way too long keepalives or sendfile streams
 
8 hours later…
12:34
Nice feature-the stackexchange badge indicates when you're mentioned in comments now.
13:13
Sorry, just got back @Warner and @pauska
:6747 I agree wholeheartedly, this is very kludgey. It's just an interim solution while I look for more information. Your suggestion of `show processlist;`in MySQL has already opened some additional avenues
Nagios runs on a different server, and I was afraid that sshd might be slow to respond or unresponsive, plus I'd like to stop this cycle before it happens, which I wasn't sure how to check with Nagios... that's why I did it as a cron job on the affected server. But I can easily switcgh to nagios given the right check command
:6749 Yes, it's probably something like that... I'll quadruple check the php 1max_execution_time` but I'm pretty sure it's 30 seconds, and these scripts run for hours
:6769 Woah, I didn't know that! Example?
The question was closed now, the one about best c compiler.
:6749 This used to happen roughly 80% of the time with one given wordpress site. I've moved that site off this apache server to a different apache server. I'm very curious to see if the problem follows it, or moved to a new site on the original server, that (I'm hoping) will tell me if it's bad PHP code, or some issue with the apache server
I left a comment about grouchy sysadmins on friday.
:6779 LOL. rats. I'll keep an eye out for that new feature :-)
Went back to list of questions, and the StackExchange badge in the upper left had a "1" appear when someone left a comment saying "@bart silverstrim:..." in that comment list.
@Josh: did the badge change now?
13:27
:6782 It did!
That's sweet
Someone help me locate the person responsible for Entourage 2008 EWS edition. I have a sharp knife ready..
Advantage to open source: usually it's easier to find the programmers responsible for the ouchie parts of a program and vent.
2
:6785 Ooooh! That sounds like fun!!! :-D
:6787 LOL, have a star :-)
OK, have a very busy morning, headed AFK. Will check back later
Here's my attempt to redeem myself from yesterday's stupid question :-)
1
Q: Is it possible to change the cmdline of a running process (to more easily identify it in top, for example)?

JoshI run a number of linux servers with VMware Server 2. When their response time is not so hot or the load average goes up, I open up top to see what's going on, or for our production servers I have top open all day long. The issue is, the cmdline of vmware-vmx processes is really long, for example...

I like that one blog entry. "because…well, usually because they’ve spent a lot of money to put themselves in a position where they’re trusted. " Trust us because we've got lots of money! Heh...
13:44
Just invented new term. Multiharassitasking.
2
Thanks to @Josh and @Zoredache
 
1 hour later…
14:48
Ugh ... thats 30 minutes I won't get back...
`DS:in:COUNTER:600:0:12500000
DS:out:COUNTER:600:0:12500000
RRA:AVERAGE:0.5:1:1440 #day
RRA:AVERAGE:0.5:30:336 #week
RRA:AVERAGE:0.5:120:360 #month
RRA:AVERAGE:0.5:1440:365 #year
RRA:MAX:0.5:1:1440 #day
RRA:MAX:0.5:30:336 #week
RRA:MAX:0.5:120:360 #month
RRA:MAX:0.5:1440:365 #year
RRA:MIN:0.5:1:1440 #day
RRA:MIN:0.5:30:336 #week
RRA:MIN:0.5:120:360 #month
RRA:MIN:0.5:1440:365 #year`
Getting closer :)
That is for graphing ethernet traffic in rrd :-) Let me know if you see the bug ;-)
I found it, or at least, I think I have
Hint: DS:ds-name:GAUGE | COUNTER | DERIVE | ABSOLUTE:heartbeat:min:max
Don't speak rrd... walking though calcs for time periods though
The year min:max is swapped order? Or perhaps, mixed units of time?
James Lawrie
15:04
year max < year min
James Lawrie
ah, +1 for repeating
James Lawrie
:) sorry jscott
@JamesLawrie np, I was spinning my wheels calculating all the numbers in different units without even comparing the numbers on the same line....
 
1 hour later…
16:29
Bloody @#$%! Why is Windows keeping something open in profiles when the user has logged out!?
Stupid registry.
l0c0b0x
@BartSilverstrim For XP/2003 at least, we learned to use the "Profile Hive cleanup service": microsoft.com/downloads/en/…
Wow what a fun adventure. Got to work and two of our APC units were just sitting there turned off. No easy Friday for me.
@l0c0b0x: we normally install it, but on these lab machines I think it was overlooked.
16:52
ninety bloody degrees in October...sheesh..
@BartSilverstrim It's already October where you live?
@jscott:Close. :-/
Close enough that it shouldn't be ninety here.
@BartSilverstrim Awwwww that was the only chance you'll ever have to pretend you're from the future.
You didn't ask what year.
What year? o_O
17:04
2012.
They say it's the end of the world.
Omg
Stupid windows 10.
Hahaha
Happy Friday!
@Sagar: quiet friday...
17:12
How come? It's Friday. The only day it should not be quiet! Slow Friday though :\
I'll never make sense of this. Delete a profile that's listed at 300 meg and it took two seconds. Deleted a profile listed at 30 meg, took twenty seconds. Huh?!
Fragmentation?
runs
@Sagar: for us, usually it's the quieter day, because everyone is too focused on just wanting to go home to bother complaining about tech issues. They figure they'll wait until Monday.
:6837 Number of files makes more more difference than the size.
@ChrisS: good point.
"Download complete. You can now disconnect from the Internet." Yeah, goody! Then I can just sit and watch you install .NET!
17:15
@BartSilverstrim good point.
@BartSilverstrim That always makes me chuckle
I like installing WinXP fresh.
Its so clean...
The sales pitch it gives you while it copies crap over is funny.
Because of that awesome music?
Doug Luxem
17:16
When Prodigy is charging you by the minute, you don't want to be on the modem too long. ;)
lol
Yeah, Friday is always slower. Wednesday is boomday.
@Dynamo: It IS awesome! And the way it disables the Enter key for keying in data is ingenious!
That music always makes me dance in my chair.
I'm going to go find the file now
Personally, seriously, I like the way Apple does there advertising in their OS. They give a nice, clean bit of music, catchy, it's pushing their brand on you without being overtly annoying. But that's an opinion and it's not hard to look great after installing Windows.
Doug Luxem
17:18
Trying to come up with a written sysadmin test. Trying to test on concepts, not commands for this...
Doug Luxem
Any tips from you guys?
@DougLuxem: for testing what kind of admin?
Doug Luxem
For Senior-level interview
Hand him a Novell server and tell him to get it on the network. See if he cries.
Doug Luxem
Windows admin with strong network background
Doug Luxem
17:19
hehe....been there, cried too ;)
I pressed ctrl+alt+del on a Novell server once
ouch
Yeeeeeeeeeah
Doug Luxem
ouch
Doug Luxem
lol
17:19
Oops.
Pretty much
l0c0b0x
:6862 LOL
senior leve and strong network skills eh
Quick poll-when you hear another sysadmin you work with say "oops", do you laugh or panic first?
laugh
l0c0b0x
17:20
laugh
its hard to keep a straight face when anyone says oops
l0c0b0x
'oops' from someone with power to take down 'everything' is always exciting!
Doug Luxem
No one says "oops" on the bad mistakes...they just try and work quietly to fix it before anyone notices
You're sadistic and not nice people. I like you all.
Turn your speakers up to 11 boys
C:\WINDOWS\system32\oobe\images\title.wma
2
The Windows XP install music
17:24
I actually like that music. It was different (?)
all gentle...giving you a false sense of security...
l0c0b0x
I hear the next version of the windows welcome screen will include some Justin Bieber melodies
Not that I know of. It makes for the most soothing OS install ever
Doug Luxem
Maybe...Windows 95 came with a Weezer video
@l0c0b0x yeah...I might shoot myself in the foot.
@DougLuxem Oh that was the most amazing CD ever!
Goodtimes, Buddy Holly
Windows 95 came with Weezer!?
17:26
Hover (the game)
Doug Luxem
/me misses 90's alternative
90's alt was great
Was it Win95 or the beta?
Win95
Oh, the beta had a different video.
17:26
It came on the CD which had the extras. My first computer...*sigh*
l0c0b0x
:6896 like grunge?
lunch time..be back
Crap...can't remember whose video was on the beta...anyone remember?
Doug Luxem
racks brain Video on "chicago"...
something like "You don't even have to cry..." for a lyric?
Edie brickell!
Wasn't that the name?
17:28
Oh
yep
but that was also on the win95 disk
Goodtimes
was the name of the song
Thought that was just the 95 beta?
@l0c0b0x I was never huge into grunge. i've heard there was a lot of good stuff in the whole scene but I just kind of skipped over it
Yeah...good times...
Win 95B (OEM v2)
Doug Luxem
17:30
Wow, that song brings back some memories...weird
yep
A more innocent time. when it was better than WfWg 3.11
Doug Luxem
haha
And I accidentally dragged and dropped part of a system directory across the network while exploring the new interface.
Doug Luxem
or win 3.1 with trumpet winsock
17:30
oops.
Trumpet...wow...
Doug Luxem
good times :)
Locking program manager features with a couple alterations to the .ini
Watching the user panic.
Doug Luxem
nice
It was like working with a Japanese style cartoon. Fileman! Progman! Launch!
Doug Luxem
My first job was to setup a networked install of Win3.1
17:33
@DougLuxem: What protocol?
Doug Luxem
The windows directory was on a mapped novell drive, and we just copied ini files to the local HD on running a batch file
I worry. =) I don't want production broken.
Doug Luxem
Networked as in, not installed locally
Doug Luxem
A shared install...
lol novel?
*novell?
17:33
I don't like your written test idea, Doug. This isn't for a position in the service industry.
Doug Luxem
yea, 3.12...the good days
Token Ring! LanMan...whee!
Who bumped the coax cable?!
hahaha
Doug Luxem
@Warner I'm not a huge fan either but it can be tough to see who's telling the truth in an interview
damnit! what happened to those simpler issues...oh wait..
Doug Luxem
17:34
I want to avoid memorization things too
Bleh. Written tests don't do enough.
Doug Luxem
@BartSilverstrim I've never pulled off a terminator cap to mess with a coworker ;)
Google for the Joel Test for admins.
I think there was something like that floating around.
Doug Luxem
@BartSilverstrim That is for employers though
Interviews are crap in general. People who think otherwise are often delusional.
17:35
Think it's better to have someone work with you a few hours to figure them out.
Doug Luxem
Hehe
Doug Luxem
No doubt
If you want to focus on deep technical knowledge and not higher level knowledge, drill down into the subject.
Oooh...reality show. The Admin Apprentice. The loser gets hired.
LOL
17:36
I prefer open ended and high level but will often drill down into the key qualifications. If they can't explain subnetting and routing on an intimate level, they don't know networking.
I have a series of questions I reference but they're not anything special, it's generally an outline.
I'd prefer finding how how well they can find information.
If they don't know the various FSMO roles, they're not the best Windows admin.
difrerences between a switch and a router. really basic...but very important. not very high level though
Doug Luxem
I personally like having them troubleshoot a performance issue...
Memorizing this stuff isn't always going to be the greatest asset. The stuff I run into now...I'd get nowhere without GOOGLE FU!
Doug Luxem
17:37
Those are always open-ended and can cover different areas
Bart: We're talking senior level here. Certainly, that's key.
Doug Luxem
usually no 1 solution
Troubleshooting is for tech support.
Doug Luxem
really?
shrug figuring out problems is for everyone.
Doug Luxem
17:38
We don't work in silos here....small company, you need to do everything
Finding information...unless you don't think senior levels need to keep their skills up in that area.
Doug Luxem
We need someone to provide overall direction on infra projects
Doug Luxem
Can't google that
If you're asking a senior admin to troubleshoot a resource issue during an interview, I'd question the level of the role. I'm not saying that troubleshooting and knowledge of scientific methods isn't key.
Can't google product reviews, current best practices...?
@Warner: Ever work for a school district? ;-)
I'd say these...that personally, the interview can only get rid of the most inept.
Doug Luxem
17:40
@Warner Performance issues often require deep technical knowledge of a product and don't have a simple solution
if you want to know what they're like, work with them an hour or two.
Or a day.
Doug Luxem
Usually that is lvl 3 stuff
And that's where high level knowledge comes in, @DougLuxem. If they can't explain and example best practices from both a business and senior technical level perspective without much prodding from you, they're probably not senior level.
Memorizing stuff...bleh. There's stuff that comes from experience, but chittering off stats and such isn't as important as finding it.
Warner has some good points, but I'd be wary of someone that can't do these things like having considerable google-fu.
I agree, @BartSilverstrim. That's why I find less value in the technical approach to interviewing.
Doug Luxem
17:41
Definately not interesting in memorized commands or specs or any of that
Doug Luxem
* definitely
I still don't know all the commands in Linux or the command line switches. I google it.
Hell...don't even know my cell phone number.
I care about attitude. Ownership, sense of urgency, dedication..
It's taped on the back of my phone. I never call me.
Those traits can override a lack of experience rather quickly.
Doug Luxem
17:42
@warner But who comes in to an interview with a bad attitude or says they are lazy and undedicated ;)
Our intern is one of the smartest people I've ever worked with in IT. He does senior level stuff.
You only know someone after seeing them interact and work and see their philosophy on the subject.
I won't be able to replace him.
There are far too many people I've run into in Tech that don't know what a TARDIS is.
Doug Luxem
:)
17:43
Those traits can be exampled in the behavior but it's really luck of the draw, @DougLuxem. That's why I say interviews are a bit of a joke. ;)
Some markets have better tech people than others.
Doug Luxem
Unfortunately, that's all we have
Doug Luxem
..the interviews
And increasingly, social media.
Another good point, @sysadmin1138
Doug Luxem
That's true...its nice to see participation in the industry
17:45
Develop a high level outline focusing on the core behaviors and knowledge you like. If you care about key technical skills, drill down through all layers of OSI and focus on the particular knowledge. If they fit the culture and meet those requirements, you'll probably be okay.
Your tech people should be googlable.
90-day probation and stick to it.
I'm suspicious of people that don't show up in searches and yet aren't online...at...all...
Establish your expectations. If they don't meet them, fire them after 90 days.
no forums, no posts, no web page, something.
17:46
I didn't use my real name on the Internet for nearly 10 years, @BartSilverstrim. You can't trust that.
Many things can be learned or trained.
:7012 That's why I crawled out of my cave and became googleable.
Doug Luxem
Same here
If you're not prepared to follow-through on it, be prepared to build a mediocre team.
That's about the best advice I can give. ;)
@Warner: Unless it's stuff that needs to be hidden, if someone asks, why not share it at the job?
:7019 Good point too.
17:47
shrug.
If you participated in online questions in newsgroups...here...etc...
shows some of your personality and skill and style.
if you learned more for it, then it's a bonus.
I crawled out from under my rock specifically to distinguish myself from the pack.
I personally am suspicious of people who are big in technology but don't appear to have any digital footprint.
I'm very careful about that myself as well, @sysadmin1138. That's why I do it now.
What have you learned or where did you learn it if you never contribute to other people or interact in the community? That's all I'm saying.
17:49
And why I've been careful about not crosslinking my professional online identity with my personal ones.
You'll find me on BMW forums and technical areas, that's about it.
Laymen are actually catching on to using it as a filter.
When I was first on USENET under my real name, archives aren't around anymore that far back.
There's a weird bug with chat
I can see hiding your 4chan posts, but when you have a linux background, you should have something to show for it.
17:50
only started a few days ago
the title bar (chrome) that shows how many messages have been typed since the tab was last active is not resetting
Beware the tech people who get a quizzical look when mentioning slashdot, serverfault, or other tech forums (or popular online sites, depending on your group's dynamic).
Well, it resets, but on the next new message, instead of saying (1), it says (184) or whatever
viewing 4chan at work?! wow - brave
I like my privacy, @BartSilverstrim. I want my personal life to stay exactly that. I understand your perspective but it's good to be aware of alternate perspectives.
I didn't say anything about viewing it at work. I said I can see why you'd hide your identity for google reasons if you have an account there.
There's a lot of buzz about "anon" pulling stuff. If anyone's part of that stuff, tread lightly.
Yeah... my old Usenet posts are still retrievable if you have the right search terms, but very few people have that. And I'm keeping it that way.
17:53
@Warner: and that would be part of the employer dynamic. They shouldn't care about you puttering around on the BMW forums (unless you're working for another car company, I guess).
@Warner: but I personally just get suspicious if you're a tech that hasn't had much out there. Maybe you could explain it, and that's fine. I personally find it a bonus if there's stuff out there to see what you're like as a potential employee.
Sometimes a good post or dozen on Serverfault gives a good idea of what you're like in your background. What you do (asking questions that could have been googled in moments? harassing people? Making fun of less knowledgeable people?)
This builds a kind of resume stored by the Googlesphere.
Nod.
Deeper than captured on a sheet of half-embellished truths.
All out new hires are taken on as temp to perm - they come on as a 6 month temporary employee. Meet the grade, and you'll get a perm job
I like LinkedIn a lot. I want my career and professional life to be largely public.
@BartSilverstrim "anon pulling stuff" - elaborate
17:56
Yep, I'm fond of LinkedIn as well.
give people a chance to explain or redeem themselves, but still it's good to have these things out there. The Internet is becoming too pervasive to not consider this.
:7055 Anon? On 4chan?
Oh geez...how to do this without stepping in digital poo...
4chan is like the sewer of the Internet, only with piranha instead of alligators.
If you do something that will rile their people up, their members will do things that can have consequences.
@Warner LinkedIn has some major flaws though. Namely, by virtue of what it is, having to add people from your current workplace. That kind of cancels out its usefullness when working somewhere, and looking for a new gig. It's great if you're a) currently unemployed or b) a contractor
if you're perm, it's useless
Or c) unemployed, at which point those 'current co-workers' are 'former co-workers'.
@BartSilverstrim I know what 4 chan is (I spend some time there - some of memes that spill out of the place are bloody hilarious) - I just didn't understand your statement - just reread it and it makes sense. Ignore me
@Izzy: They are like a force of nature. There was something about a kid posting some semi-anonymous thing where he tortured an animal. They tracked him down in real life and he had police at his door within a short amount of time.
Denial of service attacks.
17:58
@BartSilverstrim I know ;)
@sysadmin1138 see a)
real life harassment and stuff inundating mailboxes.
I disagree, @izzy. But I have a meeting now.
@Izzy: Ah. okeydokey.
:7069 Ah, reading fast. So unprecise.
@#%# another bloody reboot.
17:59
Me too!
00:00 - 18:0018:00 - 00:00

« first day (13 days earlier)      last day (5254 days later) »