@overexchange @derobert Except for actually putting contents into the files, mtree may create the hierarchy and even touch missing files. I'm on BSD though and I don't knov if Linux uses mtree at all.

... but questions go on the site.

@Kusalananda I have the manpage (it’s part of the development package for libarchive), but no sign of the binary (on Debian or Fedora)

@StephenKitt The man page says mtree is a "textual format". Was it also a binary?

@FaheemMitha I don’t know, that’s the impression I got from Kusalananda’s comment but I might be wrong

Looks like fmtree in freebsd-buildutils

That makes it sounds like an executable.

@MichaelHomer What does it do?

mtree -- map a directory hierarchy

Sorry, mtree is used on BSD to verify that a file hierarchy conforms to a specification.

It may be made to create the hierarchy as well.

... from an existing directory structure.

It takes permissions and everything into account.

My machine runs it nightly to catch things that are missing or have the wrong permissions in the base system.

@FaheemMitha The mtree utility is a binary. It reads a text file. The manual you're reading may refer to the text specification by the name of mtree, I guess.

@Kusalananda man mtree in Debian doesn't refer to the binary

"The mtree utility compares the file hierarchy rooted in the current ..."

Yes it does

@FaheemMitha man.openbsd.org/mtree

@Kusalananda Thanks.

A bit oddly in section 8 though

@MichaelHomer On OpenBSD, that's "system maintenance and operation commands".

See also: man.openbsd.org/intro

Yeah, "System administration commands (usually only for root)" in Debian, but it seems pretty general for that

I suppose if it expects to chown things that makes sense

@MichaelHomer Well, factor(6) is in "games", but one could use it as if it was in section 1...

FACTOR(1) User Commands

@MichaelHomer I think it (mtree) is extremely seldom used as a general command though, even if it may be useful under some circumstances.

@MichaelHomer Oh. On Debian?

@Kusalananda Yes, from GNU coreutils

So it is.

@MichaelHomer All coreutils are in section 1.

I've got a feeling this has been asked before, but how do I find what packages provide a given virtual package?

One of the answers suggests aptitude search '?provides(x-terminal-emulator)' which looks like it sort of works.

@FaheemMitha why “sort of” works? It just works...

@StephenKitt I was hedging... :-)

@FaheemMitha yeah that’s what I suspected!

So, my email address, which I intend to be permanent, is faheem@faheem.info. However, this is frequently regarded by the mail systems of the world to be evil, apparently because it is a .info domain. I myself am, of course, as pure as the driven snow. Now, I came across some measure that I could to reassure everyone that's I'm one of the good guys, but I now cannot remember what it was, and appear to not have made proper note of it.

Does anyone have any idea what it might have been?

@FaheemMitha There are a bunch of them. There are a couple sender whitelists, support varies (most are only relevant if you're running the mail server—they whitelist the server, not the domain). DKIM/etc. might also get a message scored as more-hammy.

Alternatively, you could become Prime Minister. Then you can just threaten anyone who rejects your mail that you'll make their money worthless.

@derobert Hmm, interesting suggestion.

@derobert I thought I ran across something quasi-canonical.

I just heard from someone who said my mail ended up in his spam bin. Apparently Gmail doesn't like me.

You could also see if there is an RFC3514 for email.

@FaheemMitha Not aware of anything quasi-canonical, unless it was SPF or DKIM.

DKIM is something I can implement at my end? And it would make it less likely that my mail be categorized as spam?

SPF sounds familiar. Let me check.

DKIM is implemented on the mail server.

SPF is just a TXT record added to your domain—that you can implement even if you don't run the mail server.

@derobert Which mail server? Can I do something about it myself?

@FaheemMitha The mail server that sends out your email

I'm looking for stuff I can do. With my domain, for example.

@derobert I don't own it. That's Luxsci.

Would anyone volunteer to let me send email to their address, so I can check if it ends up as SPAM?

@FaheemMitha sure, you can send to anthony@derobert.net

Email me there and I'll email back my gmail address (which I'd rather not post to chat)

@derobert Ok.

Sent. How can I make a message to gmail look more spam-like?

@FaheemMitha I donno, I guess tell me that I've got a relative in Nigeria who has left me a large sum of money?

Hmm, ok, I'll try that.

I emailed back. Sent you the headers of the delivered message to—you can see my SpamAssassin didn't thing your message was spammy at all

It might make it more difficult for people who actually have relatives in Nigeria who have left them a large sum of money. The poor things would probably never see the message - it would get eaten by spam filters.

@FaheemMitha I suspect they'll just have to use certified mail, like everyone else with a legal notice to send :-P

mxtoolbox.com/… — doesn't appear you currently have an SPF record

Shall I check with Luxsci?

Let me see what they have on the subject.

luxsci.com/blog/…

Or even leave them a ticket. Haven't done that for awhile. Let them earn that $10 that I give them every month.

@derobert Wow, you're fast.

@FaheemMitha If you're seeing delivery problems, you ought to.

@derobert Leave them a ticket?

I see they suggest DKIM and SPF. And DMARC too.

Yes, though beware those have various issues with mailing lists...

Their regular spam filter is crap. I'd complain if I thought they'd pay any attention.

@FaheemMitha Yeah. See if they have any suggestions about why your mail isn't being delivered.

@derobert Which ones?

@derobert Ok.

I think I'll sleep a bit now. Back in a little while.

@FaheemMitha did you send anything to my gmail? Hasn't arrived yet if you did..;.

@FaheemMitha The ones that key off the "From:" field—mailing lists typically leave that unchanged, but of course the mailing list server isn't one of your authorized mail servers (or for the signed things, does not have your private keys)

@derobert Not yet. Expect a message from a long-lost Nigerian relative (who also happens to be a Prince) shortly.

@FaheemMitha a Prince, or the Artist Formerly Known as Prince?

@derobert So, which of those are they? And how do they manifest themselves? Or do you have a handy link about these?

@derobert The former. The latter is deceased.

To be clear, we are talking about DKIM, SPF and DMARC, right?

@FaheemMitha I didn't know death prevented spamming.

@FaheemMitha yeah, looking for a good link

@derobert Good point. It probably doesn't.

Though I haven't received a spam message from Elvis yet.

@FaheemMitha I don't keep copies of old spam, else I'd search and check....

@derobert Maybe Elvis has money stashed away in old bank accounts.

Here is the best explanation I've found so far: blogs.msdn.microsoft.com/tzink/2015/05/28/… ... yes, from M$. I think my Googling is failing here...

@derobert So just DMARC is the problem? Not the others?

DKIM can be, with mailing lists that modify subject or body

@derobert Oh. What's your experience with it? And where on SE are such matters on-topic?

@FaheemMitha I have an SPF record published, but not with a hard fail. I haven't bothered implementing DKIM or DMARC. It's probably mostly discussed at Server Fault

@derobert Ok. Do you just recommend SPF then? Is that the safest?

Its surely the easiest. But I'm not sure how much it improves delivery.

And SF is that place which will only let "professional" sysadmins post.

ok.

Up-and-coming is ARC, arc-spec.org — haven't looked into it much myself

@derobert oh

Why os.system('echo abc > file1') does not work but `os.system('/bin/bash -c echo abc > file1') works?

> operator

@overexchange Should we assume you're using python?

yes

with any other language, it is the same behavior

I think the problem is around using > operator

It works fine on my Arch system and on an Ubuntu I tried.

>>> import os
>>> os.system('echo abc > file1')
0

ok

What OS are you using?

Ubuntu

16.04

How does it fail?

(and this is why you should be asking question on the site where you can give all relevant information and we don't need to ask for every bit like this)

But never mind, how does it fail?

Di you get an error message? Does it seem to work but nothing is written to the file?

What actually happens?

@terdon How > is being parsed?

It is not an argument right?

@overexchange Please tell me what actually happens when you run it.

Do you get me?

yes it gives 0 and works

Then what's the problem?

using pexpect command an uanble to

pexpect.spawn()

Let me check and get back to u

OK, you mean os.system('/bin/bash -c echo abc > file1') fails. You said the inverse.

That fails because you're not quoting correctly. You want:

os.system('/bin/bash -c "echo abc > file1"')

The same thing happens if you try from the command line:

$ /bin/bash -c echo foo > file1
$ cat file1

$

greetings

o/

arm-wrestles @terdon, and wins

@overexchange I have almost no Python knowledge, but one thing to be aware of: if whatever you call does not invoke a shell (e.g., uses exec directly), then redirection will not work.

@derobert os.system calls the shell

@derobert It does, it invokes /bin/sh (well, $SHELL, I guess) by default.

@ThomasWard yeah, but what about pexpect? No idea what that is...

it's the same as subprocess.call([command,split,accordingly], shell=True) and that uses the system default shell

The problem is the lack of quotes

and I'd have to say, if whatever you're using does invoke a shell, then you're doing it wrong

@overexchange when you run /bin/bash -c echo foo > file1 you run two separate commands: i) /bin/bash -c echo foo and ii) > file1

@derobert from what I can tell?

the pexpect functions use subprocess at its core

However, the > file is part of what you want to tell bash to execute, so you need to quote:

os.system('/bin/bash -c "echo abc > file1"')

Basically, the format of bash -c is: bash -c "whatever commands you want it to run"

so it abides by the subprocess.{call,run,check_call,check_output,Popen}([cmd], shell=True) function, essentially, but if the call is '/bin/bash -c "Some Command > Here"' then it's going to send right to the shell anyways, in a spawned subshell.

Ugh. Spawning a shell to run something from a scripting language (other than shell script) is almost always the wrong answer. If you call (e.g.,) execvp directly, it's really easy to pass variables as arguments. If you instead invoke a shell, you've got to quote and escape everything, or you have a potential security issue. And you've got to somehow quote and escape everything even though /bin/sh can be weird at times—especially if you need to work cross-platform.

Or at lot of times the answer is to do it in the scripting language.

I'm sure echo is just an example, but I've seen `mkdir $foo` in perl scripts! UGH. That's not quoted right, doesn't check for errors, etc. and it's downright silly—perl has a mkdir() function!

@derobert I assume you don't object to mkdir($foo), right? :P

Ah

@derobert if you can give me a viable and simple alternative to readlink -f in perl, I'll be very happy.

@terdon Hmmm, seems POSIX does not export that :-(

Yep

Yeah, why add another process to it, if you can achieve the desired task from within the script itself. I agree with derobert on this

Usually, yes.

I will submit that pipeline scripts are valid exceptions. I mean, if your perl script is already calling all sorts of external executables since it's running a pipeline, a few extra system() calls don't make much fo a difference,

And there are times when they just make the code much much easier to read.

@terdon oh... wait... there is a good reason it doesn't export it....

I had originally written some sort of recursiove function that found the final link target of a link, but that was a dozen lines of code or so instead of a simple $target = `readlink -f $link `

@terdon try readlink() ... it's a built-in function

@derobert Yeah, it doesn't do -f

I also sometimes have things like this in my Perl:

system("/bin/grep '^#' $file1 | /bin/sed 's/intervals=\[[^]]\]//' > \"$stage0\"");

Yes, I could do all that in Perl, but i) using grep will be faster despite the system call and ii) it's just much much simpler than having to write all the code to open the file handle, iterate over it etc.

@terdon that reads to me like you wanted to write a shell script instead. Sometimes it's OK, maybe you really do just need one pipeline like that, but if you're doing tons of them...

And you didn't quote $file1 properly. Maybe it doesn't matter, of course.

@terdon Hmm, I suspect there is a module to do that. I guess File::Spec->rel2abs doesn't resolve further links? Not sure.

@derobert Even if there is, is it really worth it to add an extra dependency and import a module just to avoid a system call?

If you're writing super efficient code it might be yes, but if you are, you're not doing in in Perl anyway.

@terdon Cwd::abs_path apparently does it

@derobert Oh, no I didn't. This is a collection of perl scripts that each run external functions and a master script to coordinate them. We're talking ~5000 lines of code.

@terdon File::Spec is a core module

Not something I would want to do in Perl.

@derobert Ah, true. I still don't see that it's worth it.

Ah, and here's another valid reason to use mkdir:

system("/bin/mkdir -p $outDir")

Sure, you could split whatever path is in $outDir on slashes and iterate over it to create each directory, or you can just have a one-line system call that does it for you.

@terdon File::Path's make_path does that for you, and without the escaping nightmare needed for system

... another core module.

Oh. OK, there goes that argument. One more change for the next release then :)

Nothing is new under the sky, and everything is in modules nowadays

The main issue here though is that I'm working on scripts that can take days to run. They analyze huge amounts of genomic data. So the overhead of the system() call seems completely irrelevant.

@terdon This isn't primarily about performance—it's safer (no shell escaping) and often error checking is easier

Yeah, I know.

It'll surely be much faster too, but very few programs have that as a real bottleneck.

I've inherited these scripts from the CEO of the startup I work for and I've been gradually weeding out his various system calls.

I've gotten rid of most of them but I do still keep some when I feel they're making my life easier rather than harder (as is usually the case with system())

Oh, and we always control the machines this is installed on 100%, so portability etc is not an issue.

BTW—all of those modules have been part of perl core since the 1990s. File::Spec was added in 5.00405 (yeah for corelist)—not sure when that came out...

Cool

Another pretty useful module, which is not core, is IPC::Run3

whazzat?

A very easy way to run a command (w/o a shell) and redirect stdin, stdout, stderr.

So basically like subprocess.Popen in Python

@derobert nice

There is also IPC::Run, which does the same thing and a lot more. Including such interesting things as letting you build pipelines (in the shell sense) where one of the commands in the pipeline is a perl subroutine. Much more heavyweight than IPC::Run3.

Hmm. But no, this is not that sort of pipeline. I mean, the main perl script calls other perl scripts which in turn call C programs.

@SergiyKolodyazhnyy Not sure, since I don't know Python. Does that allow all three? Also, can it do both files and Python variables? (E.g., IPC::Run3 lets you easily pass $foo—a variable containing a bunch of text—as stdin. Or pass $bar as stdout, capturing the c ommand output in $bar) popen(3) (C) doesn't allow such things, of course, but the Python one might.

@terdon Yeah, we have things like that. Though they take minutes to run at most, not days! And I don't think we have any C, only Perl. And at one point, a tiny bit of Java.

But error checking is really important there—e.g., it'd be very annoying to find out that the C program failed to write its results after crunching numbers for 2 days because there was a @!(*# space in the directory name, causing system("mkdir $dir") to behave unexpectedly.

We used to have sillyness like that with our several-minute runtime programs, until I fixed a lot of that. (I too inherited a ton of perl... not all of it well-written)

@derobert Yeah, there's ways to redirect streams with Popen, and add custom environment variables. For instance, I used to do something like this:

O_o , my formatting got messed up, hold on

    def run_cmd(self, cmdlist):
        """ utility: reusable function for running external commands """
        new_env = dict(os.environ)
        new_env['LC_ALL'] = 'C'
        try:
            stdout = subprocess.check_output(cmdlist, env=new_env)
        except subprocess.CalledProcessError:
            pass
        else:
            if stdout:
                return stdout

There. I need to fix my less command. It doesn't like terminator for whatever reason

In short, in perl terms, that makes a hash of all environment variables and changes LC_ALL = C in the hash, then runs the command with env = new_env.

and captures stdout

yup

If process has no output or fails, return nothing

@terdon BTW: tested here, and perl -MCwd=abs_path -E 'say abs_path("ugh")' worked for a symlink ugh, which required a fair bit of recursion to resolve

getcwd("/tmp", 4096)                    = 5
lstat("/tmp/ugh", {st_mode=S_IFLNK|0777, st_size=3, ...}) = 0
readlink("/tmp/ugh", "e/d", 4095)       = 3
lstat("/tmp/e", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/tmp/e/d", {st_mode=S_IFLNK|0777, st_size=4, ...}) = 0
readlink("/tmp/e/d", "../b", 4095)      = 4
lstat("/tmp/b", {st_mode=S_IFLNK|0777, st_size=1, ...}) = 0
readlink("/tmp/b", "a", 4095)           = 1
lstat("/tmp/a", {st_mode=S_IFLNK|0777, st_size=1, ...}) = 0
readlink("/tmp/a", "f", 4095)           = 1

@ThomasWard ok pexpect uses subprocess

so subprocess of echo C program

ok ok

Got u

@derobert Nice! Thanks :)

Folks, I think unix.stackexchange.com/questions/364481/… and unix.stackexchange.com/questions/363357/… are dupes, posted by the same person, a short time apart.

I wish people wouldn't do this, but in any case, how should this be handled?

That's the same OP.

He should have closed one of them himself. It's not really right to mark one as a dupe of the other.

I'm thinking he wrote the second one as a reformulation of the first. But he should have modified the first one rather than posting a new mostly identical question.

@Kusalananda Meta might be overkill. They're clearly dupes.

And yes, I already said the poster was the same.

Not a big deal, but annoying.

@FaheemMitha They are the some, yes. But the issue is that the second question is not resolved by the first (or the other way around). Well, you do have an upvoted answer on one of them, but it's not accepted. What does @terdon think?

@Kusalananda <Shrug.> He got basically the same response both times, from different people.

Hmm... I might vote the older one as a dupe of the newest one. The newest one at least has a proper answer.

So, done.

@Kusalananda The title of the second one is arguably too general. The first one is more accurate - he's asking about a specific issue. But it doesn't matter much either way - people often title their questions badly.

@FaheemMitha :-) I wrote about half of that title (but didn't actually change much of its meaning).

@Kusalananda Oh :-)

In

I think there might be a "not" missing in the second-to-last sentence of the answer. Can anyone confirm, please?

I.e.

> That would be very sloppy, but that would work provided the output of mycommand does contain spaces, tabs, newline, *, ?, [ characters or .. components and doesn't start with - or +.

@FaheemMitha Yes, @MichaelHomer just fixed it.

@Kusalananda ok

Some dodgy English here:

> Misunderstanded, this made some confusion because first live-cd was always using this script forcing a bad geometry for the case the usb key would be used on a very old bios.

Should that read "Understandably..."?

Made an attempt to edit this - lots of errors. What's 1,44Mo? Is that Megabytes, and if so, isn't Mb standard?

@FaheemMitha Should probably be Mb. There's a Go as well, which should probably be Gb.

@Kusalananda Any idea what he meant? And I suppose the comma is a European thing.

BTW, does SE have any mechanism to flag a question/answer as in need of spelling/grammar/format improvements, so that people might take a look at it?

I wonder what he means by "Debian tries to stay essentially useful".

@FaheemMitha Sweden uses , as a decimal comma (decimal point?), but we use "Mb" for megabyte... His name looks vaguely Finnish, but they wouldn't use "Mo"...

@Kusalananda I wondered if Mo was non-standard notation for Megabyte.

Never seen it.

Hmm. Well, I took a shot at cleaning up that answer. It has a truly staggering number of minor errors.

@FaheemMitha He tries to say that Debian aims to provide a base system that works on low-powered hardware, I think.

@Kusalananda That isn't clear. Suggestions for rewording?

Not really, no. And I'm about to sleep now...

Ok.