8:24 AM
8:59 AM
9:33 AM
8 hours later…
Today I answered a question. But after pLumo's comment, that my answer would fail if there are whitespaces along the path or filename.
I though may be enclosing the output of
find in quotes or escaping the whitespaces would remove the limitation.
6:09 PM
@Kulfy
find isn't doing that--bash is. Specifically, it performs word splitting (which POSIX calls field splitting).
Depending on what characters
find output, the shell may also perform globbing (POSIX), which is also called "filename expansion" or "pathname expansion" and which you also don't want here.
The reason this happens is that word splitting and globbing are performed on the results of some (though not all) kinds of shell expansions, when the expansions are not quoted.
The kinds of expansions that are immediately subject to word splitting and globbing (when not quoted) are parameter expansion, command substitution, and arithmetic expansion.
@EliahKagan This is to say that the expansion that are subject to word splitting and globbing are the ones that can be denoted with a syntax that includes a leading
$ sign.
You may or may not find that mnemonic useful: note that command substitution is subjected to word splitting and globbing both when it is denoted by backticks, as you wrote it, or by
$( ), which is the preferable way to do it (because that form nests less confusingly--people say the backticks form is deprecated, which is not actually true, but is still best avoided in favor of the $( ) syntax).
@Kulfy Though I strongly believe that it is inaccurate to say it is deprecated -- usually that designates an official status, and the backtick form of command substitution has not been officially deprecated by the Austin Group or otherwise -- I do nonetheless agree urge against its use (unless there were some unusual and very compelling reason to use it).
(I'll get to this. I feel I've not yet answered all of your question, especially the part about why quote-pasting didn't work.)
@EliahKagan Changing the value of
IFS doesn't affect globbing, so if you were to proceed that way, you would also want to disable globbing, which could be done with: set -f
However, you would still have the vexing problem that the output of the
-print action of find (whether specified explicitly or triggered as the default action) is actually ambiguous and thus poorly suited for parsing, because it is possible for a filename to contain an newline, and this does actually happen in real life (usually by accident, but someone might also be trying to attack your script).
6:24 PM
@Kulfy Yeah, in addition to not being officially deprecated, I suspect it will not be. Anyway, that something may in the future be deprecated is rarely a reason not to use it. The reason to prefer the
$( ) syntax is mainly that it nests more cleanly. Nesting backticks does work properly in modern Bourne-style shells including bash (I've heard rumors it used to not work right in some shells, but I am not sure about that).
Anyway, what I've said so far is quite abstract. To concretely test what (and how many) words an expansion produces, I recommend adjoining it to the end of something like:
ek@Kip:~$ mkdir tmp ek@Kip:~$ cd tmp ek@Kip:~/tmp$ ls ek@Kip:~/tmp$ touch 'one-word' ek@Kip:~/tmp$ touch 'two words' ek@Kip:~/tmp$ find . . ./two words ./one-word ek@Kip:~/tmp$ printf '[%s]\n' `find .` [.] [./two] [words] [./one-word]
ek@Kip:~/tmp$ touch $'two\nlines' ek@Kip:~/tmp$ find . . ./two?lines ./two words ./one-word ek@Kip:~/tmp$ find . | cat . ./two lines ./two words ./one-word ek@Kip:~/tmp$ printf '[%s]\n' `find .` [.] [./two] [lines] [./two] [words] [./one-word]
This shows something else important about parsing the output of
find -- the output you see on a terminal is not necessarily the same as the output you would be parsing, because find checks if its standard output is a terminal and, when it is, replaces weird characters with ?.
When it is not a terminal -- a situation I produce artificially via the quite useful technique of (what I would call) "postfix UUOC" -- it does not perform that replacement.
6:43 PM
GNU
find has a -print0 action. The null character cannot appear in a filename, so this makes it safe to parse the output of find. That approach is used in pa4080's answer.
You can also make
find run cat one or more times and pass the filenames to it. One way to do that is the approach used in aleksander_si's answer, which I'd say is the best way...though the placement of > testing.txt is extremely confusing because it creates the false impression that it is part of the command find is running.
@Kulfy Neither
find nor cat see > testing.txt or are in any way involved in the redirection. The shell applies that redirection before it runs find, and find runs cat. The cat command(s) run by find just write to standard output, which it/they have inherited from their parent find, which find inherited from a shell process, which happened after that shell process opened the file on standard output (i.e., performed the redirection).
Yeah, the way the command is written in that answer is super confusing. I presume we'll eventually delete comments but since a couple of them are relevant to this conversation I'll onebox them here:
This probably fails with a large number of files because
cat > testing.txt {} will be called more than once, overriding the previous contents of the file each time. — danzel 14 mins ago@danzel That's not a problem, it just looks like it would be, due to the confusing placement of
> cat testing.txt. As pLumo says, it's clearer to put it at the beginning or end of the whole command. The key is, > cat testing.txt is not actually part of any command run by find. find never sees > cat testing.txt. The shell applies that redirection to the command find . -name "configuration_dev.txt" -exec cat {} +. No matter how many times find runs cat--as you say, with enough files this happens multiple times--testing.txt is opened only once, when the shell opens it. — Eliah Kagan 6 mins ago
@EliahKagan That is another flaw in the technique you presented in your answer, though: because it builds a single
cat command and cat is an external utility rather than a shell builtin, it will fail -- albeit safely and with a comprehensible error message from the shell -- if the command being built is too long. In contrast, find with exec ... + and (with its default behavior) xargs automatically run a command multiple times as necessary.
Regarding the problems with word splitting and globbing, I should clarify that sometimes we use the term "filename" to mean "last path component," and in that sense, the OP's specific use case doesn't cause problems with word splitting or globbing from the filename
configuration_dev.txt. But that probably doesn't significantly mitigate the problem, as find outputs paths with other components (i.e., directories leading up to the filename), which may trigger word splitting or globbing.
At least as I understand the OP's question, it doesn't necessarily work for their specific case, because
find could output something like
I don't think I know of a good way to really fix your answer that wouldn't effectively be replacing the method it currently recommends with a different technique. For example, you could change it to recommend using a recursive glob, which would still fail with a huge number of files, but that's really probably not a big issue here... but that would be a quite different method from what you've recommended. It wouldn't use
find at all.
It would be similar in spirit, though, in that it would be a way of using a shell expansion to build an argument list for a
cat command which the shell directly runs.
But if you were to modify your answer to suppress globbing and to make it split only on newlines, and disclose prominently that it breaks in the corner case of filenames (including directory names leading up to the file) containing newlines, then I think that would fix the answer in the sense that the answer would no longer be misleading and the technique it recommends would be one that readers would be interested in and that would be acceptable in a number of real specific cases.
7:58 PM
@Kulfy So anyway, the reason pasting quotes onto the filenames doesn't work is that quoting characters that result from command substitution (or parameter expansion) are not treated specially by the shell, regardless of whether or not the command substitution (or parameter expansion) is itself quoted.
It is good that quoting characters are not treated specially in this context, because if they were, then that would be one more weird thing you usually wouldn't want and would have to take care to suppress or work around much of the time.
@Kulfy For demonstration purposes I'll use this command, which is shorter and (in my opinion) more readable, but which has the same effect as your
sed pipeline:
ek@Kip:~/tmp$ ls 'Test File 1' 'Test File 2' ek@Kip:~/tmp$ printf '[%s]\n' * [Test File 1] [Test File 2]
ek@Kip:~/tmp$ printf '[%s]\n' $(find . -name 'Tes*' | sed "s/.*/'&'/") ['./Test] [File] [1'] ['./Test] [File] [2']
The quoting characters do not perform quoting in the shell when they arise as a result of any of the
$ expansions -- nor when they arise as a result of command substitution written with the backticks syntax.
ek@Kip:~/tmp$ printf '[%s]\n' `find . -name 'Tes*' | sed "s/.*/'&'/"` ['./Test] [File] [1'] ['./Test] [File] [2']
If it did work, that would be bad, because the same characters that could perform quoting often occur in contexts where they either don't signify quoting or don't mean to signify it at the level of processing.
ek@Kip:~/tmp$ touch "Tessa's bar and grill" ek@Kip:~/tmp$ printf '[%s]\n' $(find . -name 'Tes*' | sed "s/.*/'&'/") ['./Test] [File] [1'] ['./Tessa's] [bar] [and] [grill'] ['./Test] [File] [2']
Quotes are syntactically meaningful to the shell in code the shell actually runs. They are not meaningful in the output commands the shell runs, even when the shell processes that output, unless the shell is actually running that output.
You should almost never actually do this, but you can make the shell run arbitrary text that is generated at runtime with the
eval builtin. If you do, it is only safe to paste quotes in situations where you know the exact text you are pasting them on, i.e., in situations where you neither need nor significantly benefit from eval in the first place (and even then, the resulting code is immensely confusing).
In situations where the power of
eval actually brings something to the table, you would need to use a more robust method if you wanted to ensure something was quoted. Bash provides two mechanisms that help with this (though one of them was only recently introduced). Note that I am absolutely not advocating that you should use this, or any technique involving eval, in that particular answer. I'm just mentioning it because it is topically related and interesting.
8:35 PM
In cases where that applies, you can also hand-roll something that performs robust quoting with
sed, based on the insight that you can wrap the text in single quotes if you handle just the single quotes that it contains.
ek@Kip:~/tmp$ printf '%s\0' * | xargs -0 printf '%s\n' Tessa's bar and grill Test File 1 Test File 2 ek@Kip:~/tmp$ b=\\ ek@Kip:~/tmp$ q=\' ek@Kip:~/tmp$ printf '[%s]\n' "$b" "$q" [\] ['] ek@Kip:~/tmp$ printf '%s\0' * | sed -z "s/$q/$q$b$b$q$q/g;s/.*/$q&$q/" | xargs -0 printf '%s\n' 'Tessa'\''s bar and grill' 'Test File 1' 'Test File 2' ek@Kip:~/tmp$ file 'Tessa'\''s bar and grill' Tessa's bar and grill: empty
The other mechanism bash provides to help with this is the new
${parameter@operator} form of parameter expansion. In which the Q operator performs quoting.
« first day (787 days earlier) ← previous day next day → last day (1612 days later) »
