Island of castaway thoughts

7:00 PM

@Videonauth Have you had a chance to read the transcript from when Zanna and I were talking about it? I have a hypothesis for why it is happening, which I think is correct.

Videonauth

I'm still reading and trying to reproduce as far i can

Eliah Kagan

So, to summarize as briefly as I can: crashing programs don't print out the segfault message; shells are responsible for doing so. And yet this result is very nonintuitive if you assume that the shell that runs the crashing program is what prints out the segfault message:

ek@Io:~/source$ bash -c ./crash
Segmentation fault (core dumped)
ek@Io:~/source$ bash -c ./crash | cat
ek@Io:~/source$

In that situation, it is the outer shell that is apparently printing them, rather than the inner one that actually runs the crashing process.

What's happening becomes clearer if the inner and outer shells are shells that print different segfault messages.

When zsh is the outer shell running bash as the subprocess:

ek@Io:~/source$ zsh
Io% ./crash
zsh: segmentation fault (core dumped)  ./crash
Io% ./crash | cat
zsh: segmentation fault (core dumped)  ./crash |
zsh: done                              cat
Io% bash -c ./crash
zsh: segmentation fault (core dumped)  bash -c ./crash
Io% bash -c ./crash | cat
zsh: segmentation fault (core dumped)  bash -c ./crash |
zsh: done                              cat
Io%

Those messages are coming from zsh, even for bash -c ./crash where one would assume bash would be seeing that its subprocess ./crash had been terminated abnormally and print the segfault message.

Videonauth

┌─[19:56:02]─[michael@NEXUS-ONE]
└──> testing $ bash -c 'trap "echo Bye" EXIT; ./crash' | cat
bash: line 1: 22918 Segmentation fault      (core dumped) ./crash
Bye
┌─[20:03:34]─[michael@NEXUS-ONE]
└──> testing $ bash -c 'trap : EXIT; ./crash' | cat
bash: line 1: 22938 Segmentation fault      (core dumped) ./crash
┌─[20:04:28]─[michael@NEXUS-ONE]
└──> testing $ bash -c './crash; :' | cat
bash: line 1: 22954 Segmentation fault      (core dumped) ./crash
┌─[20:05:17]─[michael@NEXUS-ONE]
└──> testing $

look at the line numbers, thats more puzzling

or is that second number the PID?

Eliah Kagan

Similarly, with bash as the outer shell and zsh as the subprocess, bash prints the segfault messages, even though one would assume that the zsh subprocess would print them:

ek@Io:~/source$ zsh -c ./crash
Segmentation fault (core dumped)
ek@Io:~/source$ zsh -c ./crash | cat
ek@Io:~/source$

@Videonauth The big numbers are PIDs, yeah. The line numbers are normal. When you manage to get bash to actually print segfault message for a command that was run noninteractively via bash -c, it shows line numbers as though it were a script file.

So it turns out that my use of trap my have been a red herring. Any command in its place achieves the same goal.

ek@Io:~/source$ bash -c ':; ./crash'
bash: line 1:  4124 Segmentation fault      (core dumped) ./crash
ek@Io:~/source$ bash -c './crash; :'
bash: line 1:  4127 Segmentation fault      (core dumped) ./crash
ek@Io:~/source$ bash -c './crash'
Segmentation fault (core dumped)

Videonauth

I can only do a wild guess here, seesm if you have any other command which producec a pipe the output of ./crash wil get piped too but if youre not it wont

Eliah Kagan

7:10 PM

There, the outer shell and the subprocesses are all bash but you can see the difference because when bash -c displays segfault messages it displays them in a different format to indicate the line numbers.

@Videonauth There is no output of ./crash.

Videonauth

or the system message

Eliah Kagan

Well the shell -- some shell -- is always printing it.

And it's on standard error, so it's not being piped.

Can you explain what you mean by "any other command which produces a pipe"?

Videonauth

if you use trap or other commands they initiate a pipe

as seen in the trap examples, then the system message of the segfault is cought up in that

Eliah Kagan

How would that initiate a pipe?

Videonauth

┌─[20:14:40]─[michael@NEXUS-ONE]
└──> testing $ bash -c 'less crash.c; ./crash' | cat
/* crash.c - simple program that crashes by raising a segmentation fault */

#include <signal.h>

int main(void)
{
    raise(SIGSEGV);
    return 0;
}
bash: line 1: 23163 Segmentation fault      (core dumped) ./crash

while again ./crash all alone does nothing when piped to cat

Eliah Kagan

7:16 PM

Yeah, but the number of commands in the operand to -c does not affect if a pipe is created, does it?

Videonauth

a pipe is only created if a command/script has an actual output to sdtout

Eliah Kagan

I mean if you're talking about the pipe object that the shell pipeline uses in the outer shell, where bash -c ... is piped to cat, that's still a pipe whether there's one command in the operand to -c or more than one.

@Videonauth That's... not true. Can you explain?

Videonauth

second please

Eliah Kagan

Ok.

Videonauth

┌─[20:20:36]─[michael@NEXUS-ONE]
└──> testing $ cat test
#!/bin/bash

cd ..
┌─[20:20:59]─[michael@NEXUS-ONE]
└──> testing $ bash -c './test' | cat
┌─[20:21:22]─[michael@NEXUS-ONE]
└──> testing $ cat test2
#!/bin/bash
echo "I am a test"
┌─[20:21:28]─[michael@NEXUS-ONE]
└──> testing $ bash -c './test2' | cat
I am a test
┌─[20:21:35]─[michael@NEXUS-ONE]
└──> testing $

what happens if you compile the crash script to have an output before raise ?

did this actually with bash scripts just to visualise my thoughts

Eliah Kagan

7:22 PM

Do you want me to do that? Or to predict the outcome?

Videonauth

yes do that, a simple line of text im curious if it gets into cat

Eliah Kagan

Because ./crash raises SIGSEGV, it terminates abnormally. Buffered writes on open file descriptors are not flushed. So if I write to stdout, I expect it might not appear. A write to stderr should appear, as should a write to stdout if I flush it explicitly.

Videonauth

i think this crazy behaviour id about this

Eliah Kagan

If stdout is a terminal, I expect it is line-buffered, so that may still appear.

How do you want me to do it? Write to stdout and then flush the buffer, and then crash?

Videonauth

but is it really the ./crash outputting that signal fault message or is it the system

Eliah Kagan

7:25 PM

Neither. A shell outputs it. That's why it looks different depending on what shell you run the program in.

When a shell waits on a subprocess and the subprocess segfaults, the shell typically prints a message. All the messages we've been talking about are from shells. None are from ./crash itself and none of them are kernel messages.

So, I don't know if you still want this, but:

Sorry, I forgot to update the name!

/* crash2.c - prints a message and then crashes with SIGSEGV*/

#include <signal.h>
#include <stdio.h>

int main(void)
{
    puts("About to crash...");
    fflush(stdout);
    raise(SIGSEGV);
}

ek@Io:~/source$ ./crash2 | cat
About to crash...
ek@Io:~/source$

ek@Io:~/source$ bash -c ./crash2 | cat
About to crash...
ek@Io:~/source$

ek@Io:~/source$ bash -c ':; ./crash2' | cat
About to crash...
bash: line 1:  4643 Segmentation fault      (core dumped) ./crash2
ek@Io:~/source$ bash -c ':; ./crash2'
About to crash...
bash: line 1:  4646 Segmentation fault      (core dumped) ./crash2
ek@Io:~/source$

So it prints the message to standard output, then crashes.

Regarding what causes pipes to be set up: they have to be set up before the commands that use them are run, so they cannot depend on whether or not there is actually output to be piped.

In particular, for Bash, in 3.2.2 Pipelines the reference manual says:

> The output of each command in the pipeline is connected via a pipe to the input of the next command. That is, each command reads the previous command’s output. This connection is performed before any redirections specified by the command.

And section 3.6 Redirections opens with the text:

> Before a command is executed, its input and output may be redirected using a special notation interpreted by the shell.

Since redirections are established before commands run, and pipelines are established before redirections are established, pipelines are necessarily established before commands run, and thus whether or not a pipeline is established can never depend on whether or not there will actually be any read or write operations on it.

Videonauth

but why then is the shell-message about the segfault not being output by cat ? (tested it neither grep, nor less, nor more etc does it)

Eliah Kagan

7:40 PM

Well it wouldn't be output by cat, because it is never sent to cat, because it is sent to standard error.

So, in zsh, it does get printed.

Videonauth

weird

gonna making me something to eat not had anything yet today

Eliah Kagan

ek@Io:~/source$ zsh
Io% ./crash2
About to crash...
zsh: segmentation fault (core dumped)  ./crash2
Io% ./crash2 | cat
About to crash...
zsh: segmentation fault (core dumped)  ./crash2 |
zsh: done                              cat
Io% ./crash2 >/dev/null
zsh: segmentation fault (core dumped)  ./crash2 > /dev/null
Io% ./crash2 >/dev/null 2>&1
zsh: segmentation fault (core dumped)  ./crash2 > /dev/null 2>&1
Io% ./crash2 |& cat >/dev/null
zsh: segmentation fault (core dumped)  ./crash2 2>&1 |
zsh: done                              cat > /dev/null

Videonauth

be back in a while

Eliah Kagan

@Videonauth Whoa you should probably do that!

ttyl

Videonauth

ah well i not feel very hungry, having a cold and so on, but i should actually eat something

ok food is in the stove, takes aout 30 mins

Eliah Kagan

7:45 PM

Ok.

So I do have a hypothesis about what is happening.

I am not totally sure it is correct.

Videonauth

tell me

Eliah Kagan

Usually when a shell runs an external command, it forks and then one brach execs the command.

Fork–exec

Fork–exec is a commonly used technique in Unix whereby an executing process spawns a new program. == Description == Dennis M. Ritchie created fork-exec. fork() is the name of the system call that the parent process uses to "divide" itself ("fork") into two identical processes. After calling fork(), the created child process is an exact copy of the parent except for the return value. This includes open files, register state, and all memory allocations, which includes the program's executable code. In some cases the two continue to run the same binary, but often one (usually the child) switches to...

Shells do support execing without first forking though, which is what the exec builtin does when it is called with at least one command-line argument. This replaces the shell with the new process.

When a shell can be certain that there is nothing it will have to do after the final command it must run completes, some shells will apparently do this as an optimization even when the exec builtin is not used.

I say "apparently" because Zanna and I found that this was actually happening under specific situations, and that the situations were different for different shells.

Videonauth

side note: hell wow wikipedia is missing 7.1 mil, heck i miss that too im only getting ~~400 bucks per month to live of ...

Eliah Kagan

Oh the donation prompt thing. :)

Videonauth

yes :p

i would ba happy to have 1200 euros right now then i would get me finally new hardware

Eliah Kagan

7:53 PM

The way we were able to know that it happened was by observing that the PID of the shell was immediately reused as the PID of the command the shell had run.

This does not always happen, because it can't... but also it doesn't always happen even when it seems like it can. zsh appears to always try to do this for the last command it has to run, but bash appears only to do it when it has one command in total to run, and dash appears never to do it.

Videonauth

ok makes sense different kinds of implementation i guess

Eliah Kagan

So, this populates the cmd shell variable with a command to pass as either all or part of an operand to -c (I used a here document and read to avoid confusing quoting):

8 hours ago, by Eliah Kagan

IFS= read -r cmd <<'EOF'
perl -we 'use English; print "$ARGV[0] $PID\n"; sleep 100' "$$"
EOF

And this displays the command stored:

8 hours ago, by Eliah Kagan

printf '%s\n' "$cmd"

(The reason for use English; is that otherwise the only way to access the PID from a Perl script is $$, which is the same notation as in the shell, which would be very confusing.)

The sleep 100 part could be removed, actually, but I wanted to be able to inspect the running processes externally, too.

First terminal:

ek@Io:~/source$ bash -c "$cmd"
5294 5294

Second terminal:

ek@Io:~$ ps 5294 | cat
  PID TTY      STAT   TIME COMMAND
 5294 pts/4    S+     0:00 perl -we use English; print "$ARGV[0] $PID\n"; sleep 100 5294

Note that I've piped the output of ps to cat so that it doesn't truncate the output to fit my terminal.

But if my bash subprocess runs another command first, even if it's the built-in : no-op, the optimization is not performed, and instead of the PID of the bash shell being the same as that of its child process, they have two separate PIDs. In the first terminal:

ek@Io:~/source$ bash -c ":; $cmd"
5331 5332

In the second terminal:

ek@Io:~$ ps 5331 5332 | cat
  PID TTY      STAT   TIME COMMAND
 5331 pts/4    S+     0:00 bash -c :; perl -we 'use English; print "$ARGV[0] $PID\n"; sleep 100' "$$"
 5332 pts/4    S+     0:00 perl -we use English; print "$ARGV[0] $PID\n"; sleep 100 5331

However, zsh will perform the optimization in both cases. From the first terminal:

ek@Io:~/source$ zsh -c "$cmd"
5414 5414

From the second terminal:

ek@Io:~$ ps 5414 | cat
  PID TTY      STAT   TIME COMMAND
 5414 pts/4    S+     0:00 perl -we use English; print "$ARGV[0] $PID\n"; sleep 100 5414

And then, adding the second command... from the first terminal:

ek@Io:~/source$ zsh -c ":; $cmd"
5422 5422

From the second terminal:

k@Io:~$ ps 5422 | cat
  PID TTY      STAT   TIME COMMAND
 5422 pts/4    S+     0:00 perl -we use English; print "$ARGV[0] $PID\n"; sleep 100 5422

dash never performs the optimization, not even when its "script" consists of a single command. From the first terminal:

ek@Io:~/source$ dash -c "$cmd"
5459 5460

From the second terminal:

ek@Io:~$ ps 5459 5460 | cat
  PID TTY      STAT   TIME COMMAND
 5459 pts/4    S+     0:00 dash -c perl -we 'use English; print "$ARGV[0] $PID\n"; sleep 100' "$$"
 5460 pts/4    S+     0:00 perl -we use English; print "$ARGV[0] $PID\n"; sleep 100 5459

Videonauth

8:11 PM

yep hs even behaves more weird it doesn't optimize at all

testing $ sh -c "$cmd"
24307 24308

testing $ ps 24307 24308 | cat
  PID TTY      STAT   TIME COMMAND
24307 pts/1    S+     0:00 sh -c perl -we 'use English; print "$ARGV[0] $PID\n"; sleep 100' "$$"
24308 pts/1    S+     0:00 perl -we use English; print "$ARGV[0] $PID\n"; sleep 100 24307

Eliah Kagan

Yeah, same as dash (since by default in Ubuntu sh is a symlink to dash).

So if a shell replaces itself with the last (zsh) or only (bash) command is has to run, and then that command crashes, the shell isn't around anymore. It's not waiting on the command. I believe that fully explains the effect of some noninteractive shells appearing to fail to print a segfault message. They've become the crashing program, and then their parent--if it is a shell--may print a segfault message when they crash.

Btw I gotta go soon. And I'm guessing you may be eating now anyways.

Videonauth

no still waiting for the food in oven to be edible

Ah.

fries and schnitzel

finally caught up

8:19 PM

We're possibly ships passing in the night, as it looks like I'll have to run real soon. :)

Zanna

haha that just keeps the suspense going

maybe tomorrow I will read why piping suppresses segfault messages

Eliah Kagan

I'm not even really there yet.

So -- and this is partially a repeat of something I've said before but I want to bring @Videonauth up to speed too -- I made a named pipe in the same directory as crash and crash2. This isn't a shell pipeline and it doesn't suppress messages. It's just so I can conveniently redirect stderr and see what happens in another terminal, without having to constantly clear out a file or constantly remember how many lines were already written.

So to do this I just ran: mkfifo pipe

Android Dev

why did I get a notification I was invited here again?

O_o

Eliah Kagan

It keeps sending them?

Android Dev

yeah I just got one

Eliah Kagan

8:24 PM

Hold on let me look.

I had moved some messages from the Downboat earlier. Some of them were yours. I guess it makes sense that it always tells people when their messages have been moved (at least if the room the messages were moved from is one they've been in recently).

Videonauth

@AndroidDev this is hotel california! you can check out ever time you like but you can never leave ;)

2

Eliah Kagan

I was moving stuff about site cleanup (a question dupe/merge) discussion from here into the Downboat and I moved some stuff from there to here while I was at it, based on the idea that the distraction had already been "paid for." :)

The distraction of moving, I mean.

Zanna

haha

I think like that too.

Eliah Kagan

:)

So -- and all this is with the named pipe open for read in the second terminal, though I doubt the order matters -- if I run ./crash2 2>pipe in bash in the first terminal, nothing appears on the second terminal and the first terminal shows:

ek@Io:~/source$ ./crash2 2>pipe
About to crash...
Segmentation fault (core dumped)
ek@Io:~/source$

Which is as expected.

Similarly, with zsh:

ek@Io:~/source$ zsh
Io% ./crash2 2>pipe
About to crash...
zsh: segmentation fault (core dumped)  ./crash2 2> pipe
Io%

However, with dash:

$ ./crash2 2>pipe
About to crash...
$

And in the terminal where I am reading from the pipe:

Segmentation fault (core dumped)

This is to say that when a command segfaults in dash, it's as though the command had actually printed out the segfault message... except of course that it didn't. I don't know quite how dash does it though.

Zanna

or in ksh

Eliah Kagan

8:38 PM

Yes. At least ksh93 which I have tested.

In bash, enclosing the command in { ;} makes it so I can pipe the segfault message, which actually does make sense. In the first terminal:

ek@Io:~/source$ { ./crash2; } 2>pipe
About to crash...
ek@Io:~/source$

And the second terminal shows:

Segmentation fault (core dumped)

That makes sense because redirections on grouped commands are really redirections on what the shell does while executing them.

However:

ek@Io:~/source$ (./crash2) 2>pipe
About to crash...
Segmentation fault (core dumped)
ek@Io:~/source$

(And nothing in the second terminal.)

Android Dev

I really should get back to that thousands of file renaming thing

Just haven't had the time :(

Zanna

@AndroidDev you'll find it :) no rush

Eliah Kagan

Oh yeah. Well, if you do end up doing it, I'm still interested in the results. It might provide for at least a blurb about performance in an answer on that question, or maybe even a new answer. But indeed, as Zanna says, there is no hurry.

I promise that I haven't been moving messages into here as part of a nefarious plot to perpetually reinvite you so that you perform that benchmark. Honest.

:)

Android Dev

lol

:D

Zanna

lol

Android Dev

8:43 PM

yeah there's a couple things I've been meaning to do like creating a new answer for how to install Android Studio because the current one is pretty bad

Zanna

@EliahKagan so the stderr of the subshell wasn't redirected, or... the subshell didn't write the segfault message?

Eliah Kagan

Whether you group with { ;} for no subshell or ( ) for a subshell, redirections written after the group on the same line as the closing punctuator apply to the group. So standard error was redirected.

But I don't know what happened.

At first I figured the subshell had done the same thing that we observed with the subprocesses befor: that it had one command in it and so it execed that command without forking.

However... in the first terminal:

ek@Io:~/source$ (:; ./crash2) 2>pipe
About to crash...
ek@Io:~/source$

And in the second terminal... nothing.

Same deal with:

ek@Io:~/source$ (:; ./crash2) 2>pipe
About to crash...
ek@Io:~/source$ (./crash2; :) 2>pipe
About to crash...

Zanna

haha where did it go

Videonauth

@zanna this systemctl question which was -1 at the start made it onto page one for HNQ

Eliah Kagan

@Zanna Well I assume this is like the situation where it appears on the left side of a pipe in bash and is not shown.

Zanna

8:56 PM

and we still don't know what causes that, do we?

Eliah Kagan

I don't know what causes it.

The question about what is going on with ( ) vs. { ;} grouping to redirect segfault messages is the original problem that I was trying to figure out.

Zanna

maybe Gilles knows :)

@Videonauth yay for salvage

Eliah Kagan

@Zanna That seems likely. At this point I'd be happy if I could ask a narrowly-scoped question.

Videonauth

i still wonder why it did got a downvote, it was well articulated question, this is why i answered it in the first place

Zanna

I don't really see why grouping should be like piping

Eliah Kagan

9:08 PM

Well with ( ) you get a subshell.

And when you pipe from a command, the command you piped from runs in a subshell.

In Bash, the command you pipe to runs in a subshell as well. In Zsh and Ksh it doesn't.

Videonauth

9:37 PM

holy cow ... 53 uv on my UL answer from yesterday hahaha

Transcript for