@Jesse_b Bruce isn't Australian slang. It's a name, see en.wikipedia.org/wiki/Bruce . It was fairly popular at one time (lots of boomers called Bruce), common enough to be used as a generic name in jokes, etc. a lot less common in succeeding generations - baby names come and go in fads, they become "cool" again when they don't remind you of your uncle or your grandma (roughly every 3 or 4 generations)

ps: i have an uncle called Bruce.

Could not understand the significance of /build vs /cache

@ilkkachu Thanks for the edit. I changed things around a bit to be slightly more complete in the check. I was to busy yesterday to do it properly from the start. Thanks again.

@cas I know there are people named Bruce, it's my father's name. I also know it's used as slang in Australia

en.m.wiktionary.org/wiki/bruce

A lot of Australians work for my company and they are all often just referred to as "Bruce" or refer to each other as Bruce

I've just seen it referred to the male equivalent of "Sheila" which is also a name but can be used as Australian slang to generally mean "woman"

@terdon I won't anymore (at least not intentionally). Not really sorry though :( I think if something can't be joked about than it doesn't have any merit

Also can't edit it now :/

@Jesse_b I don't disagree. But you are very likely to get a suspension out of things like that in the current climate. Not by me, mind you.

What one person means as a joke, another may read as a direct attack. So it's really better to avoid jokes at the moment.

@terdon: You live in France right?

NVM I see your profile says Greece

puuh...

...huup

/dev/chat: where all of the homeless UNIX vowels come to roost

hah

@JeffSchaller I imagine someone, somewhere, has run a character distribution analysis over the command names defined in POSIX

@StephenKitt If that doesn't exist it will soon...brb

probably a high overlap with the people that tried to crack the old /etc/passwd file

Don't tempt me...

ed chardist.pl ...

I'm calling it pchrdst

ed to the rescue! It’s been a while, hasn’t it?

To respect the early UNIX innovators that seemed to hate vowels in command names

Sep 20th

@JeffSchaller oh my, nearly a whole month!

It's been a weird month

So close to a month and you're quitting now?! Don't do it man, stand strong, we believe in you!

I have my eye out, but I really need to catch up on Meta.Unix; there's the LQP answer I've been meaning to (re)write for quite a while

4: 1
7: 2
9: 2
a: 53
b: 8
c: 47
d: 32
e: 62
f: 18
g: 19
h: 17
i: 35
j: 2
k: 11
l: 37
m: 33
n: 36
o: 32
p: 31
q: 12
r: 37
s: 46
t: 55
u: 29
v: 6
w: 10
x: 10
y: 4
z: 1

"u" wins as the least-appreciated vowel, unless you count "y"

So vowels aren’t under-represented, although we should really compare those results to typical distributions in English...

E: 1
g: 1
h: 1
i: 1
l: 1
n: 1
s: 1

LOL

L: 2
O: 1

#!/usr/bin/env bash

declare -A alph
dirs=( $(echo "$PATH" | tr ':' '\n') )

for dir in "${dirs[@]}"; do
	commands=( $(find "$dir" -type f -exec basename {} \;) )
	for c in "${commands[@]}"; do
		for ((i=0; i<${#c}; i++)); do
			p=${c:$i:1}
			alph[$p]=$((${alph[$p]} + 1))
		done
	done
done

To be clear, since someone might ask; the distribution above is only the "Utilities" list, not all the reserved words and builtins, key words/tokens, etc

$ cat distrib.pl
#!/usr/bin/perl -w
use strict;
my %chars = ();
while (<>) {
  chomp;
  map { $chars{$_}++ } split //, $_;
}

foreach my $char (sort keys %chars) {
  print "$char: $chars{$char}\n";
}

I should have named the variables har instead

@JeffSchaller I see your month and raise you.

@FaheemMitha I fold, you win

@JeffSchaller Bear in mind I have absolutely no idea what I just said means.

ilkkachu over there looks like they know how to play cards

But yes, it's been a pretty strange month. My bank imploded. My evil cousin got arrested. Definitely up there.

I only need one more thing to happen to make it a trifecta.

Quite the roller-coaster for you! Are you getting by OK?

It's weird to read news articles talking about my family.

@JeffSchaller If you're addressing me, yes, I'm ok. Thank you for asking.

Some reporter apparently approached the family asking for information/comment, as reporters do, I suppose. But nobody was talking.

I'm contemplating issuing a press statement. Though I'm not sure anyone will be interested.

@FaheemMitha your family might be smarter than some at a certain company we know

But it might be an opportunity to get information about what has been going on, assuming anyone reads it.

@JeffSchaller Oh, I wouldn't call them smart. But considering many of them were involved, it's probably best for them to stay quiet.

-	-	901
K	-	1
[	-	1
Y	-	4
W	-	5
X	-	6
F	-	7
+	-	8
P	-	10
V	-	12
B	-	13
U	-	28
T	-	38
q	-	46
O	-	55
z	-	80
H	-	81
G	-	84
w	-	200
_	-	202
I	-	239
L	-	247
D	-	252
S	-	272
R	-	275
N	-	276
C	-	278
M	-	285
A	-	295
v	-	299
y	-	431
x	-	544
u	-	597
k	-	617
h	-	730
g	-	883
E	-	939
4	-	1397
6	-	1410
9	-	1426
l	-	1429
7	-	1439
0	-	1501
2	-	1526
3	-	1526
8	-	1565
1	-	1604
j	-	1628
r	-	1637
5	-	1650
o	-	1738
b	-	1896
t	-	1941
p	-	1952
m	-	1980
f	-	2037
i	-	2061
n	-	2327
c	-	2804
.	-	2897
d	-	3262
s	-	3471
a	-	3643

That's what I got

> I've realized over time that "common sense" is a term we use for things that are obvious to us but not others link

@JeffSchaller wow, if that was a subject where they had some other idea of common sense ... (discussing or not discussing arguments like that with the press, I mean)

I wonder if having bloodthirsty feeling about spammers makes me a bad person.

You know, fantasies of tearing them limb from limb. That kind of thing.

I mean, I'm not a violent person...

@ilkkachu I shouldn't bring it up; I'm happy that the hubbub has largely bypassed U&L.

@FaheemMitha Overreaction, dehumanization, and aggression are all natural human behaviors. Hopefully you can look back on the feelings after you calm down and regret them though

@FaheemMitha it's not much, but you could join the Charcoal room and cast spam flags as they come in

@Jesse_b Thank you for the thoughts. But I do have those feelings periodically. I guess a remedy is to get a better spam filter. My email host has a useless spam filter, so I need to move. But I've been otherwise occupied.

@JeffSchaller Is that just an incoming possible spam room, or does it have other functions?

@JeffSchaller Hmm? I think I missed the cards reference here completely? :)

Any playlist recommendations? Amazon Music seems to have a wide range of stuff.

@FaheemMitha I can't say I have a full grasp on the whole thing myself, but there's a robot there that reports possibly-spammy posts that come in, network-wide. I think the purpose of the room is to provide feedback to the robot to tweak its algorithms & keywords. Side benefit is being able to vote/flag spammy posts to get them removed.

@FaheemMitha Anything with Big Daddy Kane

@ilkkachu your avatar ("gravatar"?)

@JeffSchaller I see. Well, that's very public-spirited work, certainly.

@Jesse_b Who?

@JeffSchaller I thought so, but I didn't see how it came up now :)

@ilkkachu sorry; Faheem said some poker words and I said some back, but that was the extent of our knowledge

It's a weird feeling to actually agree with Trump. At least, somewhat.

Regarding the Syria thing, that is.

@JeffSchaller Yes, I heard those words somewhere. But I don't actually know what they mean.

You agree with Trump on all sorts of things

I think I may have played a card game once. But I forget.

I'm sure he thinks food tastes good, sleep is relaxing, stepping into a warm building on a cold day feels nice, etc

@Jesse_b I think you're being a mite too literal.

@JeffSchaller oh, right! I missed those lines and misparsed your comment to top it off. I thought it was directed to me and missing the @ from the start.

@FaheemMitha I think you're being too critical by pretending this could possibly be the only thing you agree with him on

@ilkkachu I (ironically) didn't want to bother you with very idle chit-chat :)

@JeffSchaller yes, and then I made a monster out of a housefly (or whatever the expression is) :)

But to be honest, I don't play cards much, boardgames yes. I don't remember where the idea of that picture came from. Well, other than seeing those cards.

@Jesse_b Rap? Not exactly my kind of thing.

@Jesse_b I didn't say that's the only thing I agree with him on.

I have no idea what he thinks about most things, so how could I?

@ilkkachu made a mountain out of a molehill, possibly. Unless you're thinking of the sendmail quote "sendmail started off as a sledgehammer used to kill a fly, only to discover that the fly was actually an elephant in the distance"

I don't think an elephant in the distance would look much like a fly.

Wow today is ed day!

@StephenKitt Isn't every day ed day?

It is, when people try to script that other editor: unix.stackexchange.com/questions/547248/…

@JeffSchaller yeah, that's the one. The Finnish equivalent has different sorts of flies. Incidentally, since we glanced that Other Subject, I think part of my confusion was the singular "they". I didn't realize it referred to me, and not some third party. If only English had distinct words for singular and plural "they", the world would be a better place. But I'll be happy leave That Subject now.

@ilkkachu I'll join you in leaving the subject :)

Why OP mentioned to use _exit() in child process? stackoverflow.com/a/42882413. does not it exit automatically?

@Biswapriyo read the comments, it’s explained there ;-)

@StephenKitt The link explains why use _exit instread of exit, I mean why to use exit at all. The child process is automatically terminated.

@Biswapriyo “Because that's equivalent to exit()”, in the comments.

As in, returning from main() is equivalent to calling exit().

@StephenKitt Thanks, this clears the doubt :)

@derobert I visited my bank branch earlier today. The staff were all there, trying to put a brave face on it. I felt quite sorry for them.

It must have been a rough month for them.

So, does SE get a lot of spam?

@FaheemMitha yes; a lot of it gets deleted within seconds of it being posted. The charcoal chat room's "description" box says "141k true positives and counting" (a "true positive" is one where the 'bot said "spam" and ... two? or more humans agreed).

@JeffSchaller 141k true positives since when?

I see metasmoke.erwaysoftware.com might be more up-to-date, saying "142362 (70.09%) TP"

@FaheemMitha great question! I have no idea.

So every once in a while my monitoring system will alarm that a server is down when it's actually up. We have this documented in an SOP because it means the monitoring agent is hosed and needs to be restarted. One of the people in my department completely ignores this fact and just acknowledges the alarm (which hides it) with the reason "Node is up, false alarm"

this means the node is completely not being monitored and it's not okay to acknowledge these alarms without actually fixing the underlying problem. I've explained this to that person several times yet they still do it

@FaheemMitha you mentioned something about being filled with rage earlier? :p

I've got something very similar; there's an agent that reports to a central dashboard; if it's not accessible for any reason, we'll get a ticket "server X is down!"

How many times should you tell someone something and then watch them do the exact opposite before you give up?

@Jesse_b Spammers, yes.

@Jesse_b Where's Batman when you need him?

@FaheemMitha My office needs a hero

I was actually thinking intimidation.

I just sent out a pretty actively aggressive email to my entire department so lets see how that works

@JeffSchaller I have been trying to convince my company to allow me to automatically generate tickets for all alarms. That would make so many things easier to manage and harder to miss

@Jesse_b I'm Australian. I think I know my own culture's slang better than a foreigner, even one who works with a few australians. btw, the male equivalent to "sheila" is "bloke", even though that's not a name. both are fairly old, not quite obsolete yet but heading there. sheila is almost entirely out of fashion these days. bloke is still fairly commonly used. also btw, a lot of australian slang evolved from english, scottish and especially irish slang. sheila, for one.

@cas Seems odd you didn't know about Bruce then ;)

The Bruce thing came from a Monty Python sketch in 1970. Eric Idle thought it was amusing that many of the australians he met were called Bruce.

It's a British thing. like how Americans often think that comedian Paul Hogan's Crocodile Dundee character is some kind of archetypal Australian, rather than a caricature invented for an American audience.

@Jesse_b monitoring and alerting are hard, despite every company's sales pitches

Personally, much of my information about AU comes from Nevil Shute. Who happened to be British.

Though I do get the occasional AU guest.

@FaheemMitha Who happens to be named Bruce :p

Well, I've had other AU guests.

Actually, I've found Australians are generally pretty good guests.

Easy-going, friendly, relaxed.

@Jesse_b why doesn't someone fix the monitoring agent to not crash? Or to automatically restart, at least?

@derobert Well I don't think we should be using the monitoring application we are using (circonus)

To me it seems like an unfinished product in beta development by a small team of engineers that couldn't possibly care less about user experience

However that isn't my call

To be fair though it doesn't seem to happen that often and one of the causes for it has little to do with the agent and more to do with the fact that the server pulled down bad information about itself (which also should be fixed but isn't yet)

But if the fix is just restart the agent... then presumably that could fairly easily be scripted? Even if not fully automated, if hitting the 'restart agent' button is as easy as hitting the 'ack alarm' button...

@derobert It pretty much already is as easy

svcadm restart circonus-agent

Wonder why you coworkers don't do that, then... That's odd.

Also while I agree with you, if one of my employees ever told me that they were purposely sabotaging the company because we should fix our monitoring agents before we expect them to do their job I would probably have to resign because corporate america isn't very fond of punching people in the throat

what's the name for arranging words and pictures to make a phrase? e.g. if there's ball covered in the letter "I", it'd mean "eyeball"

@JeffSchaller mad gabs is the game (but I think that's just arranged words, not pictures)

@JeffSchaller Rebus

rebus looks right

so "circonus" means ... "we're on a circus"?

@JeffSchaller Heh don't take my opinion as gospel. The people in my company responsible for making the decision seem to think it's a good product and they are definitely more experienced and knowledgeable than me so they are probably right

Though, I guess I should confess the Nagios I maintain here manage has issues too. I'm not sure there is a monitoring system that actually works well :-(

@Jesse_b ahh - so it's "the circus is on us" instead

@derobert I believe it's specifically one person doing it

either the "we paid for your circus" meaning or the "this circus is burying us in alerts" meaning

@Jesse_b Ah, that's always fun.

Also note that I am not the manager. I honestly believe some people just severely dislike me (rightfully so if I'm being honest) so they purposely mess with me by doing stuff like this

They're supposed to mess with you by dissing your favorite Linux distro, football team, etc., not sabotaging the company :-P

True and my favorite distro is RHEL so I'm an easy target :p

I'm actually sort of interested in your opinion @derobert because of the argument you made above, although I'm not really sure how to articulate myself so let me try lol

My opinion of...?

However as I said, this is by far the most efficient, squared away, advanced, etc infrastructure that I have maintained so in my eyes I don't necessarily dismiss their complaints but it's more like "Yeah that's valid and we have already identified it as a problem but we don't have the manpower to fix it, so unless you suddenly learned to engineer solutions and have a fix we don't really need your complaints. Also if you think you can find better I believe you are sorely mistaken"

I do believe there is a lot of ill-will towards the current monitoring platform and therefore it may be possible that people legitimately want to sabotage it because they are "fed up" with the false positives

The place I'm at is tiny. So much so that when I run into something that is set up insanely, I get to go yell at myself. Our company is of course much, much smaller, and still getting monitoring right is hard. I've sometimes gotten fed up enough to spend an entire day doing nothing but stopping false positives.

Monitoring systems are damn annoying. Even when they only let you know of real problems.

Nagios, for example, loves to send tons of alerts because it failed to understand the root cause (e.g., it has failed to notice a link went down, and notifies about all the systems on the remote end being down... which its configured not to do, but...)

I've investigated why it does that, and at least once the reason was: the link wasn't just "down", it was unreliable. So when Nagios tested the link, it happened to be working. But when it tested the remote servers, it wasn't.

@JeffSchaller 141K doesn't sound like a lot, unless it's per day. my spam filters reject up to 40,000 spams per week on my tiny home postfix server with only a few users (i do my own mail and spam filtering because 1. i don't trust any corporation with my mail, and 2. i do a better job of filtering spam than any of the available offerings). It used to block a lot more until I started using fail2ban on my mail.log. I used to block 100s of thousands a day when I worked in the ISP industry.

@derobert it's been a while, and pardon the interruption, but could you set up a host dependencies on a "link" service? ahhh, nevermind. You'd need a reliable test of the link

That's... not really solvable. Well, you could test the link more extensively, but then you're introducing latency. (As in, delaying the alert)

@derobert the way I see it though is you are doing what you are supposed to do

global email traffic has been 95+% spam since at least the late 90s / early 2000s. if it wasn't for good anti-spam techniques, email would be unusable.

@derobert 24/7 streaming of your audio library, with an AI program on the audio-input that listens for skips/hiccups

Complain but also try to fix

I have an issue with the people who only complain and never try to fix

@cas agreed, and I don't have any perspective on the numbers for Stack Exchange (spam vs "ok")

@JeffSchaller "I'm not watching TV at work, I'm monitoring the network"

@derobert monitoring! HAH!

Also there seems to be a lot of people in IT lately that want their job clearly defined. In my opinion if you work in IT operations your job is to figure out what others couldn't, you can't really define that. Your job is to know stuff and if you don't know it, learn it.

Though, to be honest, at a previous job, I would sometimes find out about a network hiccup from the person who listened to streamed music before the monitoring system let me know. So that can work. At least if the buffers are short enough. :-P

@Jesse_b is your company large enough to have a human between the monitoring tool and the alerting tool? Documentation can be one way to solve the problem. If the doc is right and the person looks at it.

@Jesse_b That's annoying, some people are just like that. Especially if they've learned from experience that trying to fix doesn't work. E.g., they've become jaded at bureaucracy preventing any fixes.

@JeffSchaller I'm not quite sure what you mean. We have a person in charge of the monitoring systems and I believe I am being transitioned to eventually become that person. Monitoring is done by Circonus and Alerting is done by pagerduty, both managed by the same person/team

The alarm and remediation for it are both clearly documented and I just found three different messages in the chat history where I direct messaged the person in question about this specific issue and asked them to read the relevant SOP

@Jesse_b I mean if there's any brain cells that fire between "alert" and "page" that could handle false alarms

@Jesse_b ahh, so then it can become an educational issue for their manager and/or HR

@JeffSchaller No but honestly I don't even consider this a false alarm. The same alarm title will trigger for two events: 1) the node goes down 2) the node is no longer monitored. Both require action

@Jesse_b well, by "false" I intended to cover "you did the wrong thing"

So... if your coworker is acting in good faith, then something like getting a reminder into the alert itself (or the monitoring tool), something like "do not ack without checking the agent" would help.

The alarm essentially just says it's no longer receiving an uptime value from the machine so honestly I don't even think the wording needs to be changed

And if they're not acting in good faith, then that's really a problem for management, I'd think

bingo

Yeah my manager is already looking into it because of my email. I don't want to call people out to my manager by name ever though so I have just been trying to address the person directly

Also to make things better: this person is my former roommate/boss and was hired because of my recommendation lol

@Jesse_b No. In the 7 or 8 years I've been on SE I've variously been living in France, Greece and now London, UK. I have updated various of my profiles at different times so I now think I have scattered profiles claiming I live in all three of those places.

@Jesse_b Dear boss: I've made a huge mistake :)

@terdon :) I was going to ask if you were into cars at all. I am fascinated by the citroen DS and was just watching a video of a car show last night that was mostly old French collector cars.

There are other technical things to ask, too... like is that particular alert actually of any value? It sounds like yes (I'm guessing host monitoring agent not checking in means a lot of other things aren't being checked), but possibly your coworker thinks its just a redundant ping check? Or, possibly it is.

So someone not understanding that the agent being down means that disk space, etc. isn't being checked anymore would be a lot more likely to just ignore that alert because obviously the host is up

@derobert Yeah unfortunately I mention that every time I bring it up

The uptime check is our only way of knowing if the node is down. We got rid of the ping checks because they actually lead to way more false negatives? (The OS can crash but the node still responds to pings)

@Jesse_b Yeah, it can. It's not your only way of knowing its down, or at least of knowing its not down — e.g., you probably have non-monitoring services on the node too. If they're still up, you know the node isn't down.

Unless of course the local monitoring agent is the only thing checking them.

Have you asked the coworker why he/she ack'd the alert? I mean, in a genuine & polite sense. There may be a reason why (even if crazy). Not sure what else you can do, without being management.

@derobert Yeah I asked. We aren't on the same shift so I wont get a response until next week. Although I noticed when I looked back at previous conversations related to the same issue that this person never seems to respond to criticism. I will say something like "Hey I was looking at ticket # x related to alarm # y and noticed that you acknowledged it as a false positive. While it is falsely reporting the node down, the alarm is still serious because of x y z"

The following week they will just message me about something else completely ignoring the previous conversation

@Jesse_b That's not asking why. That's just politely telling someone the f---ed up.

true

But if there was a reason why wouldn't they respond with it?

Last time I linked to the acknowledged alarm and specifically asked if they had tried following the SOP. No response

@Jesse_b If they're assertive enough, sure.

Well the response was "What meeting are you in right now" two weeks later

@Jesse_b That sounds more like sarcastically telling someone they f---ed up.

@derobert Yeah well that was the 4th or 5th time. Sarcasm is warranted I think

@Jesse_b Ah, no, sorry. I can drive them, but that's about the end of my knowledge about them :)

@Jesse_b Warranted or not, it'd explain why the person doesn't want to talk to you about it. He/she probably expects to be berated...

@derobert Well that just leads me back to the point I had earlier about how they rightfully hate me lol

I have the social skills of a wet cat that someone just let out of a burlap sack

hah, maybe someone else on the team can ask your coworker about it

I found the very first time I talked to him about it

me: *Links alarm*
me: for that we have *links sop*
him: yeah but the agent was fine when I checked it, said it was online
me: yeah it normally still says it's online, did you check the logs?
him: no
him: am I looking for this *links error*
me: sort of but not really, honestly you are just looking for anything odd
him: I thought they were false positives
me: it is a false positive but we need to fix the agent

Part of me sort of feels bad for calling out a "friend" like this but part of me also feels like what kind of friend would ignore my advice for so long...seemingly intentionally too

Hmmm. So there is something wrong there (it's not a false positive, the monitoring agent is actually down, alert title is misleading), and checking the logs for anything odd is a hard instruction to follow... but I'd expect a sysadmin to be able to figure all that out in not too awful long.

@derobert I mean really you just restart the agent regardless of what's in the logs but that sounds even worse to say

We want to at least pretend to try and understand why it's happening

Someone should, of course. But it sounds like you all have already investigated it as much as you deem justified or are able to, so no, there is no reason to investigate it. I'd put "restart the agent" as the first step.

I dunno. I think it's good to have a ticket history of the problem with as much detail as possible so that when it does get tracked down that detail exists

Step 2 can be "log another crash with the vendor" if that's really what the business requires, but really if someone wants to go get logs of crashes.... they'll check the logs, which I presume you keep for at least a bit.

also we are tier 1/2 and a tier 3 (hell even 4 if that is a thing) exist within this company so the primary function of their job is to collect as much information as possible and then escalate

And if someone gets lazy and just restarts it without bothering to capture the logs, sounds like that's a lesser evil than just ack'ing it as a false alarm

@derobert I don't think we should accept either though, and certainly not the acking of the alarm

Honestly, speaking as a programmer too... if I were the one tasked with actually fixing the bug, usually the first few log entries are as good as the next few hundred. Seldom does yet another copy of the same logs help. The only thing that comes to mind as an exception is when I need to search through the logs to make sure I have every instance. That's not going to happen with manually created tickets.

Possibly you need to capture every instance of it happening and report it to the vendor for, err, business BS reasons, but that's it.

(E.g., maybe your company is trying to claim SLA discounts or something. Or trying to make sure the vendor prioritizes it.)

@derobert fair point however the error in the log isn't always the same, if there is an error at all

@Jesse_b There is probably (a) only a few different errors, might mean there are a few bugs. (b) random errors, probably means all the errors are useless noise and it's something more annoying, e.g., memory corruption from overrunning a pointer somewhere

and ultimately we are hired to be monkeys at a keyboard that go out and collect the data we are told to. If they had come up with a solution to automate it and were told that wasn't allowed, that would be a valid complaint IMO. However simply refusing to do your job because you don't agree with it doesn't seem like it would ever be justified

@Jesse_b it's not justified, but it sure is human

And quite possibly the SOP ought to be changed because pretending to debug an issue that no one actually cares enough about to fix is a waste of time the company is paying for.

@derobert The SOP is 5 steps. One of those is "create a ticket" another is "here is how you ssh" (yes we need to be that detailed with some of these people)

the 5th step is only applicable if the 4th step fails

so really it's two steps

lol the SOP also has a comment on it that is obnoxious

It sounded like from your discussion there is an attempt to diagnose why the agent crashed before just restarting the agent. Which strikes me as a waste of time (e.g., because you can't do that, only the vendor can). Though yeah, even with that... if the company wants to pay everyone to waste their time, that's the company's prerogative.

After you restart the agent is says to run svcs -l circonus-agent to verify that it is online. This person commented "When I try to validate the agent is online I get an error -bash: circonus-agent: command not found"

from running half the command? ugh :-(

@derobert I don't really disagree with almost everything you have said about how the process should be fixed. I just think there are more counter arguments

a) the people complaining have never been the person in charge of an unruly application so they don't know what it's like trying to track down these bugs. They are, albeit likely unknowingly, extremely rude the way they complain about these issues as if the people in charge are incompetent rather than just very busy with more pressing issues

b) they don't have any ground to stand on. You can't complain about the process being broken if you simply refuse to participate in the process

In order for them to complain about having to collect logs they first have to do some work collecting logs

c) they don't see the big picture, nor do they even try

I don't think (b) is really true. If you see one guy collect the log and hand it off to another guy who immediately puts it right back where the first guy found it, you've got good grounds to complain about wasting time. Or at least to ask why they're doing that, because it sure looks like a waste of time. Even if you've never personally lifted one.

Nobody is putting it back

Also this issue isn't happening that often. It's like 1-5 times per week, it has definitely happened less than 50 times total in the history of...forever. So there very well could be some new log entries that we haven't seen yet

Fairly recent then, too, I guess? Ideally you'd hear back from the vendor who would honestly tell you if more logs help. But... ha, that's a good joke!

I figured this was a more frequent thing that had been happening forever.

Additionally I have a general rule for everyone that starts here that you should never restart a service without first checking it's log file. We try really hard to not be in the business of applying band aids without (again at least pretending to) understand(ing) the issue

That's an admirable goal. But once you've understood the issue is a bug in the vendor's code, which you just have to wait on the vendor for a fix...

@derobert It's been an issue for a few months now, just not a very regular occurrence

@derobert Well like I said there are actually at least two different situations that cause virtually the same issue with the same symptoms

Well, presumably you find out it was the "node down" case when you try to ssh in :-)

One seems to be the agent itself so definitely on the vendor's side, the other seems to be with our systems themselves. They will have bad data or no data in sysinfo

and the agent requires that data for one reason or another. So the fix for that case is to force sysinfo to update with my favorite command: sysinfo -fu

@Jesse_b If you listen carefully, you can hear a DS rust.

;-) :-)

@Fabby :p

So if you buy one, you'll have to take it apart completely and galvanize the entire thing.