Ask Ubuntu Classroom

Artur Meinild

16:00

Ok we about ready? Do we need to ping anyone?

BeastOfCaerbannog

@terdon unix.stackexchange.com/questions/65115/…

maria

hi

Eliah Kagan

@terdon What surprises me about it is that I don't get the same result in ksh and zsh, which support recursive globbing with **. But perhaps I'm making a mistake somewhere. I'll try and look into it further after this class.

Artur Meinild

Hi

Eliah Kagan

@terdon On my system info ls has:

 For each directory that is listed, preface the files with a line
 ‘total BLOCKS’, where BLOCKS is the total disk allocation for all
 files in that directory.  The block size currently defaults to 1024
 bytes, but this can be overridden (*note Block size::).  The BLOCKS
 computed counts each hard link separately; this is arguably a
 deficiency.

terdon

16:02

@BeastOfCaerbannog Thanks, but I meant a reference to official documentation. I don't doubt that it is the blocks, I just can't find where it is documented. But Eliah just did, so great! (and I'll shut up now and let Artur continue with the lesson)

Artur Meinild

Hehe

Thanks for joining everyone!

αғsнιη

@ArturMeinild I'm ready from "Sun Oct 25 19:32:49 +0330 2020"

Artur Meinild

Awesome

So I'll try and walk you through the following:
- Basic SSH usage
- Basic Rsync CLI usage
- Basic usage of Rsync with cron
- Using SSHFS to mount remote folders

Zanna

great!

Artur Meinild

And it will be rather basic - just ask if in doubt of anything

Hope mods can help as well

I should say that I have a handout on Pastebin, that's also my session guide

I'll share the link afterwards so you have a full reference

Zanna

16:05

excellent

αғsнιη

thanks

Artur Meinild

Also, for the purpose of this session, the following example objects will be used:
User: artur
Host: myserver
Source dirs: /home/artur/documents, /home/artur/pictures, /home/artur/videos
Destination: /mnt/backup

Just fyi

jokerdino

@ArturMeinild woo awesome

Artur Meinild

We also have a small exercise somewhere in the middle ;-)

Ok first SSH

Secure Shell (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network. SSH can be used for many things, including:
- For login to a shell on a remote host
- For setting up automatic (passwordless) login to a remote host
- To backup and copy files efficiently and securely using SCP, SFTP or Rsync
- Plus many other things

SSH operates with a keypair of a private and public key, so you can login remotely without using password (you can check the .ssh folder in your homedir to see if you already have a keypair with: ls ~/.ssh)

αғsнιη

ls ~/.ssh
known_hosts

Artur Meinild

16:07

Please report back if you already have a keypair in ~/.ssh

jokerdino

no such file hehe

Artur Meinild

@αғsнιη No keypair

Zanna

yes, already have id_rsa and id_rsa.pub

Eliah Kagan

I have one:

ek@Grr:~$ ls ~/.ssh
authorized_keys  id_rsa  id_rsa.pub  known_hosts

Artur Meinild

Yup

maria

16:08

I have the same

user.dz

I ahve

Artur Meinild

If you don't have a keypair you can create one with the command:
`ssh-keygen`

ssh-keygen

For me, this would create the files id_rsa (private key) and id_rsa.pub (public key) in the folder /home/artur/.ssh

jokerdino

it did

Artur Meinild

Great

αғsнιη

done

Artur Meinild

16:10

Now, if you have a remote host that you want to connect to, you can copy the public key with this command:

ssh-copy-id user@host

For me that would be ssh-copy-id artur@myserver

After this, I could login to a remote shell (if SSH server is running on the remote host) without a password using:

ssh artur@myserver

jokerdino

got it

Artur Meinild

This was just to give a very basic understanding of SSH keypairs

If you want to script and connect remotely, passwordless is the way to do it

This post explain some more details:
https://askubuntu.com/questions/46930/how-can-i-set-up-password-less-ssh-login/46935

terdon

note that if you don't have a remote host handy, you can use localhost instead and still follow this tutorial.

Artur Meinild

@terdon Thanks terdon

And that's actually it about SSH - the very basics

Zanna

much appreciated

jokerdino

16:14

thank you

Artur Meinild

So on to Rsync ..

rsync is a somewhat advanced copy/sync/backup command with lots of possibilities

However, the basic syntax is simple:
`rsync [Options] [Source(s)] [Destination]`

rsync [Options] [Source(s)] [Destination]

For now, don't mind the options - we'll come to that

Let's see the source and destination syntax

Syncing locally: rsync -zav /home/artur/documents /mnt/backup

Syncing from local to remote: rsync -zav /home/artur/documents artur@myserver:/mnt/backup

Syncing from remote to local: rsync -zav artur@myserver:/home/artur/documents /mnt/backup

jokerdino

ooh

Artur Meinild

For remote path, notice artur@myserver: is prefixed

And that's it

jokerdino

does rsync just copy stuff over?

or does it zip stuff or something?

Artur Meinild

It can compress during transfer

But yes it copies over, but in a smart way, so it only copies the changed parts

jokerdino

16:19

oh, so it is better than cp in that way

Artur Meinild

And can compress - so that's why it's much more bandwidth efficient than for instance SCP or SFTP

yes

So again, prefix for remote paths is user@host:

The username is the user to log in to the remote host with SSH. Typically, this would be the normal user (in this case artur) or root.

The hostname of myserver can be either an IP address or a FQDN (fully qualified domain name). For instance, if I had local DNS, and my domain name was arturmeinild.net, then a FQDN could be: myserver.arturmeinild.net

^^^ Makes sense?

jokerdino

hope so

Artur Meinild

@jokerdino Hehe :_)

And as mentioned, adding a public SSH key to the remote host is the preferred way to login - alternatively a password file can be used.

BeastOfCaerbannog

So it would be [email protected]?

Artur Meinild

Yes with the examples provided

terdon

16:21

@ArturMeinild It doesn't need a fully qualified domain name. Anything that is understood by ssh (including aliases set up in etc/hosts or ~/.ssh/config) can be used by rsync as well. It just needs anything that can be identified as a remote machine.

Artur Meinild

@terdon Correct

So as you can see, there are many options already - but we can focus on the basic stuff still :-)

Ok I'll just fly through some of the more useful options of rsync - all available as reference afterwards - so you don't need to remember now

jokerdino

phew

Artur Meinild

Note I had the -zav option in my examples

But we take a few at a time

(also the manpage has them explained in detail)

-t, --times                 preserve modification times
-p, --perms                 preserve permissions
-o, --owner                 preserve owner
-g, --group                 preserve group

This block of options preserve different attributes

jokerdino

got it

Artur Meinild

    --delete                delete extraneous files from destination dir
-x, --one-file-system       do not cross filesystem boundaries
-v, --verbose               increase verbosity
-u, --update                skip files that are newer on receiver

(The layout is better in my handout)

Oh thanks

Should have thought of that :-)

αғsнιη

16:26

@ArturMeinild just copied my playroom stuff to /mnt/backup directory

Artur Meinild

@αғsнιη Hehe nice

jokerdino

@αғsнιη most important stuff

Artur Meinild

Then you also noticed that rsync wil create directories itself

If a dir doesn't exist, it gets created

αғsнιη

yes, I wanted to ask this question also

Artur Meinild

Ok final block of options for now - again just to give you an idea about possibilities:

-r, --recursive             recurse into directories
-l, --links                 copy symlinks as symlinks (preserve)
-n, --dry-run               perform a trial run with no changes made
-z, --compress              compress file data during transfer
-a, --archive               archive mode; equals -rlptgoD (-D are more special filetypes)

Hmm I can't get code format to work xD

Thx

But some useful options here, recurse into dirs, preserve symlinks etc

jokerdino

16:28

@ArturMeinild Click the 'fixed font' button

Eliah Kagan

When composing or editing a message that consists of multiple lines, a fixed font button appears (to the right of upload...). Clicking that automatically indents every line by four spaces, which makes it render as a code block.

Artur Meinild

@EliahKagan Thanks

As I mentioned earlier, notice the -z option that compresses during transfer

and the -a option, that is a collection of a number the other preservation options

So I usually use -zafor options - and add -v if I want verbose output

jokerdino

I get the -v and the -z

karel

and the -a option, that is a collection of a number of the other preservation options

Artur Meinild

-a is archive

It preseves times, permissions, user, group and symlinks

jokerdino

16:31

ahh

Artur Meinild

so it's just a collection of other options, in short

the same as for cp

also has an -a option

jokerdino

I see

Artur Meinild

I hope you're with me - there are plenty of material, and now we move on again - to cron

Still, the very basics

αғsнιη

yes, please

Artur Meinild

And with cron, we also do the exercise - so you can try out for yourself

Zanna

16:33

this is what I need help with most hahaha

Artur Meinild

There are generally 3 different ways to run a command or script with cron:
- Add a cronjob definition in crontab
- Add a cronjob definition file in /etc/cron.d/
- Put a script file directly in /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/ or /etc/cron.monthly/

At least that's the 3 ways I know of

jokerdino

is there a minute option?

Eliah Kagan

There are also user crontabs.

Artur Meinild

That's the first bullet

crontab

Eliah Kagan

Sorry, I thought you were referring to /etc/crontab.

jokerdino

16:34

so, what's the difference?

Artur Meinild

Yeah ok - see there was another option

terdon

@jokerdino not a dedicated "minute" file, no, but you can use the first bullet point (run crontab -e) and add a job that will run every minute, yes.

Artur Meinild

Well, actually I didn't plan to explain all the different options, just say that each method can have advantages for certain usecases, or sometimes it just comes down to personal preference.
In the following examples we will add cronjob definitions under `/etc/cron.d/`

terdon

@ArturMeinild May I suggest you use crontab -e instead so the format doesn't require a user name and the command doesn't require root access?

Artur Meinild

@terdon We can do that - just be aware then , that if you would like to create a script as root, then you would need root crontab, or the other method

terdon

16:37

(just thinking that using /etc/cron.d/ might make things more complicated, especially for those on WSL)

Artur Meinild

But as long as you just need to do things as your own user, let's go with user crontab, that's ok

jokerdino

WSLmatters

Artur Meinild

But as you can see, several ways to accomplish things :-)

Zanna

:)

jokerdino

ok I am on crontab -e

Artur Meinild

16:38

Under user crontab, a line has the following format:

jokerdino

here, I have a situation that I think cron can help me

Artur Meinild

[Minute] [Hour] [Day of month] [Month] [Day of week] [Command]

@jokerdino There are many situations where cron can help you :-)

The time and date fields are: (first 5 fields)
  Field          Allowed values
  -----          --------------
  Minute         0-59
  Hour           0-23
  Day of month   1-31
  Month          1-12
  Day of week    0-7 (0 or 7 is Sunday)

In addition, for all fields * is wildcard meaning any value.

Ok and now it all comes together with a couple of examples - and then the exercise!

jokerdino

yay!

Artur Meinild

I should just mention, that to edit your user crontab, use this command: crontab -e

You may be asked to select an editor

Eliah Kagan

@terdon Although I agree with the benefits of crontab -e, I wouldn't expect it to have any specific advantages on WSL. If cron isn't running, none of the techniques will work. If cron is running, it should work with /etc/cron.d on WSL -- it is present, and sudo/sudoedit work fine in WSL. Unfortunately it seems cron is not running by default on WSL.

Zanna

16:41

I selected nano

how do we check whether cron is running?

αғsнιη

yes, unfortunately I selected nano which I'm not familiar with it, forgot to use "ed", anyone knows how to change it to "ed", please?

jokerdino

ed????

Artur Meinild

For a cronjob that runs rsync for a single folder each day at 22:30, this would be the cronjob definition to put in crontab (notice the PATH must be included in the job file as well)

Eliah Kagan

You want to use ed?

terdon

@EliahKagan OK. I don't know how WSL works and I was thinking that might add an unnecessary layer of complication. It's also a generally good idea to not use the system-wide files unless absolutely necessary.

@αғsнιη You can't use ed, it's a stream editor, not a file editor.

αғsнιη

16:42

but it was in suggested editors:

Select an editor. To change later, run 'select-editor'.
1. /bin/nano <---- easiest
2. /usr/bin/vim.basic
3. /usr/bin/vim.tiny
4. /bin/ed

Artur Meinild

For a cronjob that runs rsync for a single folder each day at 22:30, this would be the cronjob definition to put in crontab (notice the PATH must be included in the job file as well)
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
30 22 * * * rsync -zav /home/artur/documents /mnt/backup

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
30 22 * * * rsync -zav /home/artur/documents /mnt/backup

^^^ this would go in user crontab in this example

Minute is 30, hour is 22 - and the rest of timefields is *

And the command is rsync -zav /home/artur/documents /mnt/backup

terdon

whaaaaa? Anyway, you should be able to change it by running "export EDITOR=/usr/bin/ed" and/or export VISUAL=/usr/bin/ed

jokerdino

@ArturMeinild full path needed?

Artur Meinild

@jokerdino Yes I would very much recommend so

Cron runs with no environment whatsoever

jokerdino

@αғsнιη run the command 'select-editor`

Artur Meinild

16:45

So PATH is always needed, and always use full paths for files

terdon

@ArturMeinild (with a very limited environment, including annoyingly, its own PATH which is hard coded in the cron binary)

Artur Meinild

@terdon Yeah, result is the same ;-)

terdon

PATH is only needed if you're using a program that isn't in one of /usr/bin or /bin. So it isn't needed in most cases, but it's never a bad idea to add it just in case.

jokerdino

I see

Artur Meinild

@jokerdino Yeah, better safe than sorry I say

Also @terdon

Ok so maybe we should do the cron exercise to see if cron works for you

Zanna

16:47

yes please!

maria

yes :)

jokerdino

yes let's gooo

Artur Meinild

So if we're going with the crontab -e option, please add the following line:

[Minute] [Hour] * * * echo "Hello" > /home/[User]/hello.txt

Where you replace [Minute] and [Hour] so the script runs 15 minutes from now in your Timezone, and [User] with your username

αғsнιη

thanks, next time I should read title line also :D

Select an editor. **To change later, run 'select-editor'.**

Artur Meinild

@αғsнιη xD

The above line will (in the time you put in) create a text file in your homedir called hello.txt

jokerdino

16:49

@αғsнιη XD

Artur Meinild

When you've edited your crontab, save end exit from editor

It will then say: Inserted new crontab (or simething like that)

jokerdino

crontab: installing new crontab

Artur Meinild

Thats it

Zanna

same

BeastOfCaerbannog

Done.

Artur Meinild

16:51

Ok earlier, I gave an example of a simple crontab line that ran a single rsync command at 22:30

αғsнιη

no hello.txt created for me

Artur Meinild

However, if you have several dirs you want to sync to multiple locations, and/or multiple hosts, it might be an idea to create a simple bash script

@αғsнιη What time did you set it to run?

jokerdino

to see the crontab, use crontab -l

αғsнιη

* * * * * echo "Hello" > /home/user/hello.txt

jokerdino

maybe cron is not running

Zanna

16:53

@ArturMeinild and then have cron run the script?

Artur Meinild

@αғsнιη You need to define minute and hour in the first 2 fields - and replace "user" with your username, so it's the absolute path of your homedir

terdon

@αғsнιη that will run the next time the current minute changes. If it hasn't appeared after 50 seconds, then your cron service isn't running.

Artur Meinild

@Zanna Yes

Zanna

:)

terdon

And what Artur said: /home/user/ doesn't exist, so that will just fail.

Artur Meinild

16:54

@terdon How do we see cron service is running?

I just assume it is xD

terdon

'sudo service cron status' probably

Someone said it isn't on by default on WSL.

Artur Meinild

systemctl status cron

jokerdino

* cron is not running

Artur Meinild

Is the default on Ubuntu I guess

jokerdino

sudo service cron start

Artur Meinild

16:55

Try: systemctl start cron

Add sudo to that

αғsнιη

@terdon yes, was not running ...

jokerdino

@ArturMeinild gotta make that script executable?

terdon

@αғsнιη OK. But you also need to change the /home/user as Artur explained.

Artur Meinild

I hope you can get cron running then?

@jokerdino We'll come back to that script very shortly I hope

αғsнιη

@terdon yes, I did that already, problem was "cron" was not started, now I see hello.txt

Artur Meinild

16:56

Ok nice

Zanna any luck with cron?

jokerdino

hello.txt

she scheduled it for 15 minutes from then

Artur Meinild

As you can see, 1 minute is the shortest cron interval

Ok

maria

I've set full hour so I'm still waiting but it seems it works

jokerdino

@maria oh no

Artur Meinild

As I said, if you have more folders, you might want to create a script, and run that from cron

Zanna

16:58

@ArturMeinild active (running) here (and a lot of other text) (I'm on Lubuntu 20.04 FYI)

Artur Meinild

I could create a bash script in my homedir: /home/artur/rsyncbackup.sh (remember to make it executable with: chmod +x ~/rsyncbackup.sh )

Script example:

#!/bin/bash

rsync -zav /home/artur/{documents,pictures,videos} /mnt/backup
rsync -zav /home/artur/{documents,pictures,videos} artur@myserver:/mnt/backup

Zanna

@jokerdino yes, I'll check it in ~7 minutes

Artur Meinild

1st line is the bash "shebang" (so system knows it's a bash script)

Next I put two lines with rsync commands to 2 different locations

Notice I used the brace expansion for multiple source dirs, like Zanna showed last week :-)

jokerdino

:D

Artur Meinild

Now I would add this bash script to my crontab

Transcript for