/dev/chat

10:56 AM

Is there anyway to make cp -v -r do a recursive copy of directories faster? It took 453 minutes to copy 29G of data.

@BlackPanther Removing the -v will make it a tiny bit faster since it doesn't need to print, but it sounds like you're transferring over very slow drives or network.

Stephen Kitt

@terdon if there are lots of files, removing -v can make it significantly faster

terminals are slow

Black Panther

@terdon I'm transferring from an internal HDD to an external HDD via USB 3.0.

@StephenKitt I see. It's way to slow. I've backed up more data than that, and it took less time.

Stephen Kitt

@BlackPanther are there any I/O errors in dmesg?

Black Panther

@StephenKitt I still want a log of every copy operation that cp did. If I keep -v but redirect the verbose output of cp to a file using >> will it be just as fast as if I removed -v?

Stephen Kitt

11:05 AM

@BlackPanther pretty much, yes

Black Panther

@StephenKitt dmesg printed a list of messages but I don't know how to determine an error in dmesg's output.

terdon

@StephenKitt yep

Black Panther

But cp threw the error:

cp: error reading '/home/user/folder' -> '/run/media/user/folder'
: Input/output error

Only 29G was copied by cp before the disk space error, double that amount of data should have been copied. Now I can either start over or attempt to resume the recursive copy from where cp stopped.

Stephen Kitt

@BlackPanther are you sure it was a disk space error? flaky I/O could cause slow copies without failing outright

Black Panther

Is rsync the best tool to resume the incomplete recursive copy or can cp be used?

Stephen Kitt

11:13 AM

rsync is pretty much always better than cp

but you need to address your I/O errors first

Black Panther

@StephenKitt Yup, that disk space error is the reason I posted this question:

0

Q: Why is there a discrepancy in the size shown by df and du for a mounted google drive storage account?

I am using cp to copy directories from a mounted google drive storage account to disk. I need to find out the total size of all directories to be copied from google drive to disk to know the amount of disk space needed. The google drive storage account was mounted using google-drive-ocamlfuse. I ...

@StephenKitt Why is rsync better than cp in this case?

Is it faster? Or are there other advantages to using rsync instead of cp?

Stephen Kitt

@BlackPanther it won’t copy files that are already present in the target

terdon

@BlackPanther Hang on, you're copying from a remotely mounted drive to an external disk? That's always going to be slow. Copying over USB is slow and copying over the internet is even slower.

Black Panther

@StephenKitt I think with the -u option, cp also won't copy files that are already present in the target. However there is a caveat as mentioned in the comment by ilkkachu for this answer

> If the first copy ended abruptly, one of the files may be copied only partially, and cp -u might not care, since it only checks timestamps, not the size... So you'd need to touch the source files first if you have to use it. (Or just use rsync.)

Stephen Kitt

@BlackPanther yes, rsync avoids that

Black Panther

11:33 AM

@terdon Good observation. I'm not really sure whether cp is copying from a remotely mounted drive to an external disk or from the local file system to an external disk. It's not completely clear to me yet whether google-drive-ocamlfuse downloads contents from the remote drive to the local file system, in which case cp would be copying from the local file system to an external disk.

terdon

@BlackPanther Does it take hours to mount? If not, it isn't downloading.

Also, the "fuse" in the name suggests it is mounting the remote and not downloading

Basically it doesn't make sense that it would download.

yep:

> google-drive-ocamlfuse is a FUSE filesystem for Google Drive, written in OCaml. It lets you mount your Google Drive on Linux.

So it makes sense that this would be suuuuper slow. It's slow on both ends

Black Panther

12:05 PM

@terdon It mounts almost instantly, and the Google Docs are lazily downloaded at first access which I think is when the drive is mounted.

terdon

@BlackPanther Of course it mounts instantly, but it won't download the data.

Black Panther

@terdon Yeah, I don't think I will restart the recursive copy from scratch since its so slow

terdon

Presumably it will work just like an NFS mount or a fuse mount over sshfs or any other remote mounting.

Use rsync, not cp. That will both speed up the transfer and it will be able to actually resume from where you left off. Always assuming you use the right rsyunc flags which I don't remember off the top of my head.

You might also want to consider -z to compress while transferring. That will likely speed things up considerably

Stephen Kitt

@BlackPanther you might want to use a file copying tool instead of a FUSE system — for example, rclone

they tend to work faster for bulk operations

Black Panther

@terdon From what I read, rsync is the best tool for the job, not cp. The rsync command I want to use is rsync --append-verify --partial source-directory destination-directory but I read that resuming a copy operation will not gain much unless you called the first rsync with --partial or --partial-dir 1. Is this true?

@StephenKitt Thanks I'll keep that in mind. I've already invested time into learning google-drive-ocamlfuse but I'll find out whether rclone can be used to mount or copy a remote Google Drive.

Stephen Kitt

12:19 PM

@BlackPanther it can, I use it for that (amongst other things)

not for mounting, for copying

Black Panther

@StephenKitt Thanks for clarifying

In this answer:

2

A: Resume transfer of a single file by rsync

By default, rsync will enable --whole-file if transferring from local disk to local disk. This will restart an interrupted transfer from the beginning, rather than checking the parts that are already there. To disable this, use: --no-whole-file Combining this with either --inplace or --partia...

Tom Hale warns that:

> --append and --append-verify have a dangerous failure case: when the receiver's file is the same size or larger but has different data

Is this true?

Stephen Kitt

@BlackPanther yes, see man rsync

Black Panther

man rsync says this:

> Rsync skips any files that exist on the
receiving side that are not shorter than the asso‐
ciated file on the sending side (which means that
new files are trasnferred).

As well as this:

> This special copy mode only works to efficiently
update files that are known to be growing larger
where any existing content on the receiving side
is also known to be the same as the content on the
sender. The use of --append can be dangerous if
you aren't 100% sure that all the files in the
transfer are shared, growing files.

Stephen Kitt

and more importantly,

> Rsync updates these growing file in-place without verifying any of the existing content in the file (it
only verifies the content that it is appending). Rsync skips any files that exist on the receiving side
that are not shorter than the associated file on the sending side (which means that new files are trasnferred).

Black Panther

@StephenKitt am I right in thinking that the concerns raised in the answer unix.stackexchange.com/a/545370/289865 does not apply to me since in my case any existing content on the receiving side is also known to be the same as the content on the sender?

Stephen Kitt

12:29 PM

@BlackPanther if “any existing content on the receiving side is also known to be the same as the content on the sender” is true, then yes, the concerns don’t apply; but only you can know that for sure

i.e. it must be true that no file has changed locally since it was copied to the receiver

Black Panther

@StephenKitt If I want to verify the existing content in the file, should I use append-verify instead of just append? I didn't understand the whole check sum thing but the excerpt from man rsync you quoted is pretty clear.

Stephen Kitt

@BlackPanther --append-verify is only safe if all the source files are the same size or larger than the target files which exist; it will fail if a source file is smaller than the corresponding target file

Black Panther

@StephenKitt I believe that is the case.

@StephenKitt That makes sense because that is how you can verify whether an existing target file that you are resuming an interrupted copy operation is the uncopied part of the corresponding source file.

Going back to this scenario:

16 mins ago, by Black Panther

> --append and --append-verify have a dangerous failure case: when the receiver's file is the same size or larger but has different data

Wouldn't rsync just transfer a new file? I don't see how this is a problem or as Tom Hale put it, "a dangerous failure case"?

Stephen Kitt

@BlackPanther if you sync files, then update your local copy in such a way that it ends up smaller, rsync will skip it

you’ll think your local copy is synchronised with the remote, but it’s not

Black Panther

12:46 PM

@StephenKitt Oh I see. rsync just ignores files that are smaller on the sending side than the receiving side.

What does the following excerpt from man rsync mean by new files are transferred?

> Rsync skips any files that exist on the
receiving side that are not shorter than the asso‐
ciated file on the sending side (which means that
new files are trasnferred).

Stephen Kitt

@BlackPanther with the --append options, yes.

@BlackPanther it means that new files on the sending side are sent to the receiving side (because they don’t meet the requirements to skip them — they don’t exist on the receiving side)

Black Panther

@StephenKitt This would be bad. When rsync completes a copy operation, is there a way to at least verify that a file on the receiving side is exactly the same as the file on the sending side?

@StephenKitt And this is why man rsync warns:

> The use of --append can be dangerous if
you aren't 100% sure that all the files in the
transfer are shared, growing files. You should
thus use filter rules to ensure that you weed out
any files that do not fit this criteria.

Stephen Kitt

@BlackPanther you don’t use --append

Black Panther

@StephenKitt Great explanation, that makes a lot of sense.

@StephenKitt You mean to say the only way to not experience this "dangerous failure case" is to not use the --append option?

Stephen Kitt

@BlackPanther using --append comes with certain limitations; if you don’t want those limitations, don’t use --append

it’s a bit like cutting your fingers off: if you don’t think you’d like having fewer fingers, don’t cut your fingers off

Jeff Schaller

1:04 PM

@StephenKitt you might be thinking of the --truncate option, but yes :)

Stephen Kitt

@JeffSchaller ha ha yes! but rsync always starts by truncating files when it wants to write them

Black Panther

@StephenKitt Is there a way to use rsync to compare destination files to source files, to make sure that they are identical?

Stephen Kitt

@BlackPanther yes, you run rsync, that’s what it does before copying anything

Jeff Schaller

@BlackPanther maybe --dry-run?

been a while; maybe you want -v (or more -vv's)

Black Panther

@StephenKitt Well, the only time the files on the sending side would have changed is if the Google Docs are modified. As the Google Docs are links, not my own Google Docs, I guess I could check their modification time to make sure that they are not modified in between the time I ran cp -v -r and the subsequent rsync.

From what I've gathered, the right command to run to resume an incomplete cp -v -r operation is:

rsync --append-verify source-directory destination-directory

Stephen Kitt

1:13 PM

@BlackPanther what I mean is that if you’re going to run rsync --append, but then check whether anything is out of sync, then you might as well just use rsync, because it does delta updates anyway

Black Panther

@JeffSchaller I'll add this to the command, thanks.

Stephen Kitt

I would just use rsync -avz source dest

(rsync -avzHAX for traditional Linux-to-Linux copies, but not for cloud syncs)

and again, for Google Drive I would use rclone, not rsync

Black Panther

@StephenKitt I don't quite get it. Is it rsync source-directory destination-directory (i.e. rsync without options) that checks whether anything is out of sync, rsync --append-verify source-directory destination-directory doesn't?

@StephenKitt If rsync source-directory destination-directory does delta (i.e. incremental) updates then what is the reason for using rsync --append-verify source-directory destination-directory?

@StephenKitt Despite the data being stored on the clould, it is downloaded to the local file system by google-drive-ocamlfuse, so I believe this is a Linux to external HDD recursive copy.

Stephen Kitt

1:29 PM

@BlackPanther it’s slightly more efficient, for example for log files which are only ever appended to

@BlackPanther it’s a piecemeal cloud to cache to external HDD copy

bulk cloud cloning tools are far faster than cloud FUSE file systems

I move terabytes of data around, I started out with stuff like hubicfuse (which I maintain in Debian) but it would never complete, it’s no trouble with rclone

Black Panther

@StephenKitt Wish I knew this, I definitely have to look into rclone now, because the time copying files mounted by google-drive-ocamlfuse is too slow.

Black Panther

1:52 PM

@StephenKitt rsync's delta-transfer algorithm updates a file if it exists. I don't understand how or when rsync compares destination files to source files to make sure that they are identical?

Stephen Kitt

@BlackPanther when it calculates the deltas

Black Panther

@StephenKitt When rsync calculates the differences between two files, right?

@StephenKitt What criteria does rsync use to determine if two files are identical? It can't be the size of the two files, because the whole point of incremental copying is to update a file only when the file on the sending side is smaller than the same file on the receiving side.

Stephen Kitt

> Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.

Black Panther

2:09 PM

@StephenKitt This sounds like rsync finds corresponding files (i.e. copies) in the receiving side, and then determines if those corresponding files need to be updated based on the condition that the original file on the sending side is bigger than the copy on the receiving side.

Stephen Kitt

@BlackPanther changed in size, they can be bigger or smaller

Black Panther

@StephenKitt However it doesn't indicate how rsync finds files on the receiving side that are copies of files on the sending side, it only indicates how rsync determines which copies on the receiving side need to be updated.

Stephen Kitt

@BlackPanther the docs explain that: it runs rsync on the remote system

Black Panther

@StephenKitt Oh yeah correct, but we don't want original files on the sending side to be smaller than their copies on the receiving side.

Stephen Kitt

@BlackPanther why not? forget about --append

Black Panther

2:14 PM

@StephenKitt That's good to know.

@StephenKitt Because of --append. Without --append what happens when original files on the sending side are smaller than their copies on the receiving side?

Stephen Kitt

@BlackPanther rsync syncs them

Black Panther

@StephenKitt The copy is made smaller to match the original?

Stephen Kitt

@BlackPanther yes, the goal of rsync is that the receiver ends up with exactly the same files as the sender

Black Panther

Say I run rsync -avz source dest and rsync --append-verify source-directory destination-directory on exactly the same source directory and destination directory. What would be the difference in what rsync does?

@StephenKitt For both commands does rsync update any files on the receiving side that are copies of files on the sending side, with the only difference between the two commands being that rsync --append-verify source-directory destination-directory is more efficient than rsync -avz source dest and rsync for files that are known to be growing larger where any existing content on the receiving side is also known to be the same as the content on the sender?

Stephen Kitt

2:36 PM

@BlackPanther I suggest you test it; my quick tests here suggest that -avz --append-verify isn’t actually more efficient than -avz, it’s slightly worse

Black Panther

@StephenKitt I'll look into it. You mean --append-verify isn't actually more efficient than -avz?

@StephenKitt Is it true that for resuming copying that cp didn't complete, rsync has to work with the --append option?

Stephen Kitt

@BlackPanther it seems to depend on the circumstances, but the difference is very slight, around 0.01% of the transferred data

@BlackPanther no

Black Panther

@StephenKitt I think this answer:

0

A: rsync with append smaller to longer file part that existing on receiver

The append option doesn't work like that. From the man page: This causes rsync to update a file by appending data onto the end of the file, which presumes that the data that already exists on the receiving side is identical with the start of the file on the sending side. So, if you "...

Answers this question of mine:

22 mins ago, by Black Panther

@StephenKitt For both commands does rsync update any files on the receiving side that are copies of files on the sending side, with the only difference between the two commands being that rsync --append-verify source-directory destination-directory is more efficient than rsync -avz source dest and rsync for files that are known to be growing larger where any existing content on the receiving side is also known to be the same as the content on the sender?

From that question:

> By passing the --append option you're basically promising rsync that you know the start of the file will never change.

Stephen Kitt

2:54 PM

That Q&A is rather inaccurate. You’ve answered your own question by quoting the documentation previously. --append is a promise made by the user that the portion of the file already available on the receiver hasn’t changed on the sender.

Black Panther

So probably if you don't pass --append to rsync then rsync will do extra work to determine that the data that already exists on the receiving side is identical with the start of the file on the sending side.

Stephen Kitt

But like I’ve said several times, you should forget about --append. The slight efficiency it adds in some cases will never make up all the time you’ve been thinking about it today.

Black Panther

@StephenKitt That's true

@StephenKitt Yeah, I'll just execute the command rsync -avz source dest in my script. At least it was good to know more about rsync and that with or without --append rsync always updates a file on the receiving side that is a copy of a file on the sending side.

Stephen Kitt

@BlackPanther one thing to be aware of is that on local copies, rsync always copies full files; you need --no-whole-files if you’re not talking to a remote rsync

Black Panther

@StephenKitt when using the -z option, are the destination files packaged into a zip file?

Stephen Kitt

3:02 PM

@BlackPanther no, the deltas are compressed

it’s all explained in the docs

Black Panther

@StephenKitt Thanks, that's very useful to know.

Will include that option

@StephenKitt Thanks. I'll use rsync to resume the incomplete copy operation, and time how long it takes

Black Panther

3:17 PM

@StephenKitt Btw, from reading man rsync it seems that -z is used only for `remote sources?

> --compress, -z
With this option, rsync compresses the file data
as it is sent to the destination machine, which
reduces the amount of data being transmitted --
something that is useful over a slow connection.

Stephen Kitt

@BlackPanther yes

Black Panther

@StephenKitt An external HDD is considered to be local, not remote, right?

Stephen Kitt

@BlackPanther yes

Kusalananda

3:45 PM

@BlackPanther Also, -z for compression is only ever useful over really slow links. You will have to be able to compress (and decompress) faster than you can transfer data over the network for it to be worthwhile using -z.

Fabby

3:59 PM

@AndrasDeak Thanks for the tip

@FaheemMitha Staying logged in to ensure the deleted information is available for the least time as possible.

(I.E.: ping me)

Black Panther

4:15 PM

@Kusalananda Thanks

Black Panther

4:52 PM

Can someone help me figure out something. After my system ran out of disk space because I tried to copy Google Docs from a mounted Google Drive using google-drive-ocamlfuse, I can't seem to get back the disk space that was used up, even though I've unmounted the remote Drive.

The Google Docs must have been downloaded to somewhere on the local file system, but I can't find them in ~/.cache which is where I expected them to be.

I've even restarted my system

Stephen Kitt

@BlackPanther in ~/.gdfuse perhaps?

BTW you might find github.com/astrada/google-drive-ocamlfuse/wiki/rsync useful

Andras Deak

5:15 PM

@Fabby my point was that you don't have to synchronise ;)

Black Panther

5:31 PM

@StephenKitt Correct. I found it there. Thanks

@StephenKitt That's really useful. Although I don't know how this can be true:

> using rsync with -W (no delta operations - copy Whole file) speeds it up and handles corrupted files better than cp (see issue #48)

@JeffSchaller If you put the redirection outside the loop does the shell open the file at the beginning of the iteration and keep it open, rather than opening the file in each iteration of the loop when >> is in the loop?

20 hours ago, by Jeff Schaller

that looks equivalent; I'd offer two suggestions: (1) any errors from cp will escape both the |tee and the >>; (2) consider putting the >> redirection outside of the loop so that the shell doesn't have to reopen the file each time -- but only if you want to capture all of the command output inside the loop.

terdon

## The shell opens the file each time it gets to a '>' or '>>'. Here, it will open once per iteration:
while :; do echo "foo" >> file; done

## Here, it will only open it once, at the end, to write everything
while :; do echo foo; done > file

@BlackPanther ^^

Black Panther

@terdon Thanks. Can the second loop also be while :; do echo foo; done >> file?

I mean using the append operator?

Or does it work only for the overwrite operator (if that's what it's called)?

terdon

Ah no, it can be used in both of them. I just used >> in the first one so it would mimic the behavior of the second. Otherwise, with while :; do echo "foo" > file; done, each "foo" you write would overwrite the contents of file. So at the end of the loop (if that loop ever ended) you would have a file with just one foo in it.

If you do while :; echo foo; done > file then file would have everything

terdon@tpad ~ $ for i in foo bar baz; do echo $i > file; done && cat file
baz
terdon@tpad ~ $ for i in foo bar baz; do echo $i; done > file  && cat file
foo
bar
baz

Black Panther

5:46 PM

@terdon if for while :; do echo foo; done > file the shell only opens the file once, at the end, to write everything, is the data stored in memory? In other words, where is the data stored before the shell opens the file to write the data to the file?

terdon

I believe it is stored in some buffer, presumably in memory.

Although I also suspect there will be some mechanisms that allow swapping to disk when the content is too large. @StephenKitt will probably know the details.

But yeah, basically it's stored in memory or temp files or something along those lines.

Black Panther

@terdon I see. So while :; do echo "foo" >> file; done is equivalent to while :; do echo foo; done > file. Is the only difference between while :; do echo foo; done > file and while :; do echo foo; done >> file that the former overwrites the file with everything where as the latter appends everything to the file?

@terdon That's very cool

terdon

@BlackPanther Well, they are only equivalent if the file is empty or non-existent when the loop starts. Otherwise, when using >>, the existing contents of the file before running the loop will be kept and the new content will be appended.

Black Panther

@terdon In my case I am logging the output of an rsync recursive copy of over 50G data. Since $ for i in foo bar baz; do echo $i; done > file && cat file would store the output of rsync in memory, it would be problematic wouldn't it?

terdon

Ah, the whole thing will be problematic. Rsync uses carriage returns to output progress, so it will look horrible in a file.

Jeff Schaller

5:55 PM

terdon's our local expert on shell redirections!

terdon

snort. Not in this company I'm not!

Jeff Schaller

@BlackPanther it doesn't hold it in memory; it changes the stdout file descriptor to point to the file instead of the screen (generally speaking)

Black Panther

@terdon I see. I anticipated that. My disk is almost full, so better not take any chances, and stick with the straightforward way of opening a file in a loop using the shell.

terdon

Jeff seems to be contradicting me, and he probably knows what he's talking about.

Black Panther

@terdon Thanks

terdon

5:58 PM

@JeffSchaller isn't the content stored in a memory buffer and flushed to the file when the buffer is full?

Jeff Schaller

@terdon ehhh, all that guy knows is ed ed ed ;)

@terdon there's a small I/O buffer, yes. same as any write; if you're closing the file inside the loop, that forces a flush

terdon

Looks like it, actually. In which case, @BlackPanther, you can safely do it. It will store a few Kb in the buffer and then flush to the file.

Yeah. So to answer @BlackPanther's question, no it won't store large amounts of data in memory and it won't use temp files as I incorrectly suggested.

Black Panther

@terdon You mean even if I use something like $ for i in foo bar baz; do echo $i > file; done? The line terminator in unix files is usually \n, right? So a log file that contains output from rsync will have \r instead of \n/?

terdon

In fact, I am testing now by running this in one terminal:

while :; do date ; sleep 1; done > file

and this in another:

tail -f file

And I can see a line printed every second, so nothing is being held in the buffer. I guess the ">" functionality is simply printing data as it gets it. I also guess that it will open the file once when the loop starts and keep it open, but I am no longer sure.

Black Panther

@JeffSchaller So that would be like 1> file (I think that's how you redirect stdout, correct me if I'm wrong).

Jeff Schaller

6:03 PM

@BlackPanther yep -- the 1 is silent (assumed), if you don't use a different number

> Redirection of output causes the file whose name results from the expansion of word to be opened for writing on file descriptor n, or the standard output (file descriptor 1) if n is not specified.

(from man bash)

Black Panther

@JeffSchaller I think I get this. It uses that small I/O buffer to store bytes read from rsync's output, then writes the bytes in the buffer to the file. Is this what the small I/O buffer is used for?

@terdon That's great to know. I suspect it uses a buffer size of 4096.

Jeff Schaller

any/all writes are buffered to disk; if/when you close the file (either inside the loop or outside), that buffer is flushed

terdon

I'd have thought so too, but no. It seems to print directly. Try running

while :; do printf '1' ; sleep 1; done > file

And then tail -f file

You will see each 1 being printed. If it were flushing a buffer, I would expect to see batches of 1s printed together whenever it collected enough to fill the buffer.

Jeff Schaller

maybe I'm out of date again!

terdon

@JeffSchaller Or maybe any READ operation on the file causes any processes writing to the open file handle to be flushed?

So I only see them coming out one by one because I am reading it constantly with tail -f?

Jeff Schaller

6:11 PM

well, I'm veering straight into wrongness, but I don't think reads would flush the buffer, so I'm apparently wrong about it being buffered on the way to the file

terdon

And yet all read/writes are supposed to be buffered, as you said.

scratches head

jesse_b

Surge 2013: LightningTalks

►

slybloty

Any idea on how to allow network access to a virt-manager VM while AnyConnect VPN is connected?

Jeff Schaller

unix.stackexchange.com/a/45953/117549 might help

Black Panther

@terdon That's a relief. Updated the script, if rsync is a lot faster than cp, I might have to use rsync as my file copying tool.

@terdon Yeah, it must open the file at the start, instead of at the end.

My rsync command in a loop is:

for destDir in "$destDirA" "$destDirB" "$destDirC" "$destDirD" "destDirE"
do
        rsync -avv --inplace --no-whole-file "$1" "$destDir"
        shift
done >> "$log_file"

^^ should finally work.

terdon

6:26 PM

@jesse_b that's really good! Funny, informative and very apt, thanks!

Black Panther

What is the best way to follow in real time content being written to a log file?

terdon

GNU tail seems to have fixed this though. Works as expected.

Black Panther

Is it watch?

Jeff Schaller

17 mins ago, by terdon

And then tail -f file

Black Panther

@JeffSchaller Wow, this is really nice. Say good buy to printing to stdout.

Is it unneccary to have touch "$log_file" before the for loop?

Does the redirection operator in done >> "$log_file" create a file if it doesn't exist? I don't quite remember.

terdon

6:33 PM

yes

both > and >> will create the file if it doesn't exist.

And you can just do:

   > file

dammit!

Ah, finally.

Black Panther

@terdon Nice, so convenient.

terdon

I mean you don't need a command. Simply > file is enough to create it and/or truncate it if it exists

Black Panther

Keep going @terdon

Is it possible to comment code in a bash script?

terdon

yes. If a line starts with 0 or more whitespace characters and then a #, it is a comment.

Black Panther

@terdon I see, that's similar to Python, I think. I've spent more time with C like languages where // is the way to comment lines of code

terdon

6:38 PM

Yeah, shells (at least sh-family shells), perl and python use #

So do many config file formats in the *nix world.

Lisp uses ";" (because of course it had to be different :P)

Black Panther

@terdon Haha. That would be madness for a C and C derivative programmer

That reminds me. I used a virtual machine on MacOS in the past called VMWare Fusion. It is not on Linux. What is the best virtual machine for a Linux machine that has 8GB and a weakish CPU?

Memory consumption is really important

terdon

@BlackPanther vmware exists for Linux

Other options are VirtualBox and K-something-or-other

Black Panther

@terdon There's another VMWare virtual machine for Linux I think but it's not Fusion

terdon

ah, quite possibly

Black Panther

@terdon Gnome boxes is preinstalled

Jeff Schaller

6:44 PM

@terdon KVM maybe?

Black Panther

Not sure how good it is

terdon

I've been using virtualbox (lightly) for years and it's fine. At work we use (thank you @JeffSchaller) KVM

Black Panther

@JeffSchaller Is that the one without a GUI?

@terdon How virtualbox when it comes to memory consumption?

Jeff Schaller

There...might be a GUI for KVM; I don't know. GUI's are for people that don't know ed

terdon

You can set how much RAM you want to give it.

@JeffSchaller hopeless... absolutely hopeless

:P

Jeff Schaller

6:45 PM

:)

Black Panther

@JeffSchaller Haha, the command line is my playground. Problem is, sometimes I get lost playing around on the command line instead of just using an application

jesse_b

kvm has a gui

Black Panther

@terdon This is a must, thanks

Was setting up KVM quick and easy?

I don't mind if it's a command line app, I'm not sure I want to spend a lot of time configuring it or trying to get it to work out of the box.

@jesse_b What's the GUI called?

Interesting, KVM is not in any of the repos in fedora.

jesse_b

@BlackPanther kvm

terdon

I've never worked with KVM. But setting up VirtualBox was trivial and worked the first time I tried it.

Black Panther

7:02 PM

There's something unexpected in the log file

I used double verbose: rsync -avv --inplace --no-whole-file "$1" "$destDir"

Yet the output is only showing a single file

cp -v would show something like ~/sourceA -> ~/destA.

Where as it looks like rsync is only showing the file being copied, not the file being copied and the path it is being copied to. Is this how rsync normally displays verbose output?

rsync -avv --inplace --no-whole-file "$1" "$destDir" outputs something like:

~/sourceDirA/sourceDirAFile1
~/sourceDirA/sourceDirAFile2
~sourceDirA/sourceDirAFile3
~/sourceDirB/sourceDirBFile1
~/sourceDirB/sourceDirBFile2

From man rsync:

> --verbose, -v
This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single -v will give you information about what files
are being transferred and a brief summary at the end. Two -v options will give you information on what files are being skipped and slightly more information at the end. More
than two -v options should only be used if you are debugging rsync.

I'm not sure what -vv is showing me. Is -vv showing only the files that are being skipped or is it showing both what files are being transferred and the files that are being skipped?

Black Panther

7:31 PM

What is the difference between rsync -v and rsync -vv?

I don't know if me using the --no-whole-file option makes any difference

terdon

7:42 PM

@BlackPanther What it says in the quote you posted. Just try with a single file:

terdon@tpad ~ $ rsync -v file file2
file

sent 168 bytes received 35 bytes 406.00 bytes/sec
total size is 85 speedup is 0.42

terdon@tpad ~ $ rsync -vv file file2
delta-transmission disabled for local transfer or --whole-file
file
total: matches=0 hash_hits=0 false_alarms=0 data=85

sent 168 bytes received 102 bytes 540.00 bytes/sec
total size is 85 speedup is 0.31

But yes, it doesn't print the destination, but that makes sense since you already know the destination, you need to pass it as a command line argument.

Black Panther

@terdon It looks like -vv includes the output of -v, but adds additional information to it.

@terdon file is what is being transferred in both cases.

I think the following is a lot clearer than what is in man rsync:

> -v, –verbose: By default, rsync works silently. A single -v will give us information about what files are being transferred and a brief summary about the data transferred at the end. Two -v options will give us information on the status of delta-transmission and on what files are up to date so as to be skipped and slightly more information at the end. More than two -v options are generally used for debugging rsync.
-v, --verbose https://www.geeksforgeeks.org/rsync-command-in-linux-with-examples/

@terdon That's true. I guess I wanted to check that the contents of the directory passed to rsync was being copied, not the directory itself or the full path to the directory.

Fabby

8:17 PM

@AndrasDeak True. :facepalm:

@FaheemMitha ---^ See deleted message

Andras Deak

8:36 PM

^ specifically chat.stackexchange.com/messages/56676358/history

Black Panther

10:06 PM

I'm getting an input/output error from rsync:

rsync: [sender] read errors mapping "/run/media/user/folder/another_folder/file.pdf: Input/output error (5)
WARNING: another_folder/folder/file.pdf failed verification -- update retained (will try again).

Is anyone familiar with the above error and what it means?

This question is about a similar problem:

25

Q: How to interpret and fix a Input/output error in Linux?

I am running a daily backup with rsync. Starting some days ago, one of the files has been throwing this error during the backup: rsync: read errors mapping "/home/folder/file.ext": Input/output error (5) WARNING: /home/folder/file.ext failed verification -- update discarded (will try again). W...

linux rsync

Andras Deak

@BlackPanther did you swap those two doctored folder names by accident?

Black Panther

@AndrasDeak Crap, good eye. The actual folder names are much harder to comprehend.

Let me try again:

rsync: [sender] read errors mapping "/run/media/user/folder/another_folder/file.pdf: Input/output error (5)
WARNING: folder/another_folder/file.pdf failed verification -- update retained (will try again).

Black Panther

10:26 PM

Another error:

rsync: [sender] read errors mapping "/run/media/user/folder/another_folder/file.pdf": Input/output error (5)
ERROR: folder/another_folder/file.pdf failed verification -- update retained.
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]

^^ It looks like rsync tried to write /run/media/user/folder/another_folder/file.pdf one last time but it failed yet again.

> update retained.

So how do I access this update to try and manually force rsync to apply the update?

The following line appeared after rsync had operated on the entire hierarchy of a directory (i.e. a single directory tree). What do matches and hash_hits mean here?

total: matches=350  hash_hits=350  false_alarms=0 data=17832458389

I'm gonna make the example closer to the real output.

The real output contains special character ü

Something similar to:

rsync: [sender] read errors mapping "/run/media/user/folder/another_folder/folder ü/file.pdf": Input/output error (5)
ERROR: folder/another_folder/folder \#303\#274/file.pdf failed verification -- update retained.
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]

Andras Deak

10:48 PM

Your erroring directories don't all contain unicode chars, do they?

I'd expect that not to matter, but who knows

Black Panther

Maybe the special character ü is causing unforeseen havoc.

@AndrasDeak Now you mention it, that is the only directory that causes an error.

Andras Deak

űnicode problems

The file systems on both ends might also be relevant. But I'm 100% a rubber duck here.

Black Panther

One is an internal disk, the other is an external drive plugged in via USB 30.0

echo ür
ür

Andras Deak

Those are not file systems.

Black Panther

The shell should have no problem with ü since echo shows it has no special meaning

Andras Deak

10:53 PM

it would be überweird if ü were special in any way

Black Panther

@AndrasDeak Right, I think I should say one is extension 4 and the other is vfat or something

Andras Deak

Now you can at least try to create a reproducible example using an accented directory

that should help the experts help you

Black Panther

@AndrasDeak It is not the only source directory named using an accented character, just the only one that fails to be copied.

Andras Deak

Ah, I misunderstood you then

then I guess you're stuck having to figure out what's special about that directory

permissions, inodes, I dunno

Any chance the file/disk itself is broken?

Black Panther

Yep, I'll look at the source code for rsync and see if I can get any clues from line 1330 main.c

@AndrasDeak Hope not. That would be annoying.

Do you know of any way to test if a disk is broken?

Andras Deak

11:02 PM

I wouldn't expect main.c:1330 to contain any hints, it sounds like that's only about logistics. The IO error is reported elsewhere.

Black Panther

I'll try copying it (with rsync)

to /tmp first

Andras Deak

@BlackPanther nah. I only know that SMART tests might give you hints (reallocated sector counts, mainly). But that's no guarantee of anything.

Black Panther

then to the target directory

Andras Deak

@BlackPanther keep your eye on dmesg

An eye. "Your eye" sounds like I'm implying you only have the one eye.

Black Panther

@AndrasDeak The output of dmesg is so large, what exactly should I be looking for?

Andras Deak

11:06 PM

IO errors near the end when you're trying to copy the file

in my bash they are big and red so they're hard to miss

Black Panther

@AndrasDeak yeah, "keep your eyes" sounds better

Andras Deak

or that

I was also trying to imply multitasking

Black Panther

Do you ever use a virtual machine by the way?

Andras Deak

Ever yes, but very rarely. Think every 4 years.

Black Panther

I see. Was looking for one with a low memory footprint

Andras Deak

11:08 PM

I've mostly used virtualbox which is an insane resource hog. Perhaps I always configure it wrong.

My completely uneducated impression is that docker is a lot leaner. Might or might not be true, and it might or might not work for your use case.

OK, I'll have to tip my hat to bash

Black Panther

@AndrasDeak That's good to know. Did you try limiting the amount of memory the VM uses?

Andras Deak

cp -v $k ${k/eps0/eps${eps}}

I didn't expect nested substitutions to work!

Black Panther

@AndrasDeak I need a virtual machine to run windows on Linux so that I can use a Windows application on Linux

@AndrasDeak Docker is a container, how is it similar to a virtual machine?

It's late, and I have to be up early tomorrow. See ya later

Andras Deak

@BlackPanther Yeah, but it's mostly the CPU that I had issues with. But again: I used it rarely and for short times, and I touched very few knobs. Don't let my impressions persuade you in either way. In any case I used a VM briefly again last year, and I noticed that virtualbox is no longer in debian. I used something else.

@BlackPanther that's also my typical use case

@BlackPanther do you know what a container is?

I don't

But I just googled and stackoverflow.com/questions/42158596/… suggests that you can't put a windows container on a linux host

Transcript for