SSH, Rsync and Cron - Continuation, clarification and questions

Artur Meinild

Nov 1, 2020 16:10

I thought this lesson would be a follow up on the last one, to answer some questions and hear some experiences

Eliah Kagan

I'm here.

Artur Meinild

And maybe elaborate a bit on some topics from my handouts

karel

I'm here.

Artur Meinild

For those who didn't attend last, this is the link: pastebin.com/siaGhWNs

At any point during this session, please ask for clarification, and I'll try and answer - or maybe someone else can :-)

αғsнιη

I'm here too

jokerdino

Nov 1, 2020 16:12

here and back

Artur Meinild

Good - just waiting for the last couple of folks..

Ok so did anyone try something with cron and/or rsync since last time - and did it work? If no, what didn't work?

Let us hear your experiences, if you got any :-)

Or if you have any questions regarding the topics, please ask.

jokerdino

cron worked wonderful

Artur Meinild

Good to hear!

jokerdino

haven't tried out rsync though

Artur Meinild

I think I would elaborate a bit on cron actually

αғsнιη

Nov 1, 2020 16:17

when a new cronjob added by crontab -e; where that job exactly written? I mean in which file path?

Artur Meinild

@αғsнιη Can you elaborate?

Generally, when using cron you should use absolute paths

So all filenames, commands etc. (except those in path) should always be full, absolute names

Or was there something specific? Maybe try to give an example if this didn't answer.

jokerdino

@αғsнιη is asking crontab -e is editing which specific file

as we all know, everything is a file in linux

Artur Meinild

oh

αғsнιη

when we type 'crontab -e' then define a job, these job actually writing into which file? because we only know editor opens a list of jobs, I'm asking the location of these jobs maybe somewhere in a file?

Artur Meinild

The crontab file is a special system file - I actually don't know the exact location

but when using crontab -e you open it and can see the list

And using crontab -l the contents is displayed

But this file should only be edited with crontab -e, and not manually

Eliah Kagan

Nov 1, 2020 16:20

User crontabs are stored in /var/spool/cron but it is not recommended to edit them there and that directory is owned by root without even read permission for other users.

Artur Meinild

@EliahKagan Thanks for elaboration

But remember that "user crontab" as we went through last time, is only one of several types of crontabs

Aside from user crontabs, there are some system-wide cron definition files

One example is the file: /etc/crontab

Eliah Kagan

@EliahKagan Running crontab -e creates a temporary file, which is then edited. When you quit the editor, it will create a file in /var/spool/cron or apply changes to an existing file there or do nothing, depending on whether or not your user crontab existed already and whether or not you made any changes in the editor.

Artur Meinild

Another example are the files located in the directory /etc/cron.d

Zanna

I feel a sort of reference table of cron's files could be useful

αғsнιη

thanks @ArturMeinild @EliahKagan

Eliah Kagan

Nov 1, 2020 16:24

(I should've said more specifically that user crontabs are stored directly inside /var/spool/cron/crontabs.)

Artur Meinild

@Zanna Actually, you kind of get that with the command ls /etc/cron*

This lists all files and dirs that have something to do with cron, except for the user crontabs we talked about last time

Zanna

oh, cool!

Artur Meinild

But right now, I would just like to point out that the syntax of the systemwide cron definitions are a little bit different than the user crontab

Zanna

oh

Eliah Kagan

@Zanna Although it's not a table, the "NOTES" section of cron(8) gives all the locations (including for user crontabs).

Artur Meinild

Nov 1, 2020 16:27

System cron definitions have this format:

[Minute] [Hour] [Day of month] [Month] [Day of week] [User] [Command]

Zanna

@EliahKagan thanks! :)

Artur Meinild

The user crontab had this format:

`
[Minute] [Hour] [Day of month] [Month] [Day of week] [User] [Command]`

Sorry, [Minute] [Hour] [Day of month] [Month] [Day of week] [Command]

The "user" field is unique to system crontab

Which kinda makes sense

Zanna

yes

Artur Meinild

So in the file /etc/crontab you can put any number of lines for cronjobs, and define which user it should run as, including root

Zanna

there are lots of system users it might be useful to run jobs as, right?

Artur Meinild

Nov 1, 2020 16:30

And in the dir /etc/cron.d you can make any file, and include a cronjob definition there

Zanna

@ArturMeinild with the same format as /etc/crontab?

Artur Meinild

@Zanna Yes

@Zanna I usually run as myself or root

I'm not aware if there are specific system users where it would make sense, but I guess there is

αғsнιη

is there any priority of running for the jobs when it's systemwide and when it's user level? which jobs are running at first or maybe they are running in parallels?

Artur Meinild

@αғsнιη Actually I'm not sure - or if everything is just fired at the same time

I mean, fired when it's set to go - and if 2 jobs are set to run at the same time I guess they run in parallet

*parallel

Also, someone asked what would be obvious tasks to do with a cronjob

And I can only answer for myself- and you're all welcome to add to this

Remember, the scenario is that I run a couple of home servers

So the systemwide cronjobs I'm running are the following:

αғsнιη

no, I'm not talking for the same job in both systemwide or user level; I'm asking if there are some system level defined jobs and some user level, so cron daemon execute system level jobs at first priority then user level or what (assuming they have same running time)?

Artur Meinild

Nov 1, 2020 16:37

- Run update/upgrade and cleanup script every week
- Run script to reboot if needed
- Renew certificates for my webserver (letsencrypt)
- Update root hints and adserver lists for unbound
- Automatic ZFS snapshots
- Trim and scrub ZFS once a month

@αғsнιη I don't think there is any priority - I think they go in parallel - someone correct me if I'm wrong

However, since you can define runtime down to the minute, you can define the order in which scripts are run by the time they run

For instance, my reboot script runs 1 hour after my update script

Zanna

a quick look at Unix & Linux suggests cron jobs with the same timing run in parallel if possible

Artur Meinild

The other way around wouldn't make sense

Zanna

but further research needed to confirm

Artur Meinild

@Zanna Thanks

Zanna

@ArturMeinild very useful list

Artur Meinild

Nov 1, 2020 16:39

@Zanna Just some examples from the real world (or my real world at least) :-)

Zanna

(I'm interested in ZFS, if you want to talk more about it)

Eliah Kagan

Yes. cron will start the jobs in some order, and I'd be interested to learn what order that is for jobs scheduled for the sametime... but the order does not actually matter hardly at all, because cron does not wait for one job to finish before it starts the "next" one.

Artur Meinild

@EliahKagan I think that's a good point - if you need a very specific order, either do it by timer, or program some conditions into your scripts - for instance 1st script leaves a file that 2nd script checks for to continue

@Zanna re: ZFS - I'm by no means an expert

But ZFS is really easy and very cool for storage

I wouldn't run it as system disk, but for NAS, definitely

It's incredibly reliable

Zanna

I think I read that it was useful for preserving stuff, like images, that could somehow degrade over time (but how does that work?)... archival storage

Artur Meinild

Yes - it has many features for data preservation and recovery

And most of them automatic, so you mostly don't even have to worry about it

Zanna

Nov 1, 2020 16:45

nice!

Artur Meinild

To put it very short, is has in-build hash-checking of all files - and support for Raid-redundancy

so basically, if the filesystem ever detects the checksum of one of your files doesn

*doesn't match, it can recreate the file as it should be

Zanna

so, what does the trimming and scrubbing in your cron job do?

Artur Meinild

Trimming is because it's an SSD raid

so like normal SSD trim - but for the ZFS pool

and scrubbing is a monthly integrity check - just because

And I keep 1 month of daily snapshots

so with a single command you can always roll back to an earlier filesystem state where you have a snapshot

Zanna

exemplary backup haha

Artur Meinild

I think that's enough about ZFS for now, I could go on and on :-D

@Zanna I would say it's a good addition to backup ;-)

If your server catches fire, you're still screwed - but if not, you're good :-)

Zanna

Nov 1, 2020 16:49

hahaha

Artur Meinild

Ok I think last point for today - something else we didn't have time for last, and one example - SSHFS

SSHFS can be used to mount a filesystem over SSH

First we install it: sudo apt install sshfs

And then, we can mount a remote folder on a local one:

sshfs artur@myserver:/mnt/backup /mnt/backup/remote

The syntax is: sshfs [remote source] [local target]

Zanna

@ArturMeinild We don't get many questions about it and I find it hard to research. I was hoping that more people would try giving lessons to reveal a glimpse of their untapped expertise :)

Artur Meinild

(The folder /mnt/backup/remote must exist and be empty)

αғsнιη

@ArturMeinild at remote side for the "artur@myserver:/mnt/backup" path, there should be any changes to config?

Artur Meinild

@Zanna Hehe point taken into consideration

@αғsнιη Absolutely no config whatsoever is needed for SSHFS - not for me at least

I just fire one command when my server starts to mount the remote folder - that's the beauty of it

Now let's say you have mounted the remote folder /mnt/backup on myserver to the local folder: /mnt/backup/remote

αғsнιη

Nov 1, 2020 16:55

hmm, I see

Artur Meinild

Now we can rsync over SSH, but use the local path instead

rsync -zav /home/artur/{documents,pictures,videos} /mnt/backup/remote

This will then be copied to /mnt/backup as if using SSH with artur@myserver

Did that make sense?

Zanna

yes

Artur Meinild

I have one usecase for SSHFS currently - and there may be others

αғsнιη

wait, when it's mounted locally then we can use mv, cp as well, right?

Artur Meinild

I have a primary server and a support server (raspberry pi)

@αғsнιη Of course

I would like to have all log files of all my servers available on my primary server

so I made the dir /var/log/xo where xo is the name of my secondary machine

And mount with this command: sshfs artur@secondary-server:/var/log /var/log/xo

so now the logs of my raspberry pi is available inside /var/log/xoon my main server

/var/log/xo

This is an alternative to using remote logging, and I think this is easier

So the logs are stored locally on my ras-pi, but available as a network dir on my server

And it's very easy to use

Ok I think that's it for now.. :-)

Zanna

Nov 1, 2020 17:01

that sounds very handy!

Artur Meinild

Anyone still here? ;-)

@Zanna :-)

Eliah Kagan

I am still here. :)

user.dz

yes, it is good

αғsнιη

thank you @ArturMeinild

Zanna

thank you so much for giving the lesson @ArturMeinild

Artur Meinild

Nov 1, 2020 17:02

You're most welcome

Eliah Kagan

Thank you!

Is there time for questions?

αғsнιη

@ArturMeinild what is the command to umount the sshfs mounted path?

user.dz

it is on df list

Artur Meinild

Sure question if you have any?

Let's see if I can answer

Eliah Kagan

@αғsнιη You can just use umount as with other kinds of mounts. You don't have to run it as root (just as sshfs need not be -- and should not usuallybe -- run as root).

@ArturMeinild I've just noticed a problem with sshfs when the local and remote systems don't agree about the group ID of the user's primary group.

ek@Grr:~$ mount | grep sshfs
arg:backup on /home/ek/backup/remote type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1001)
ek@Grr:~$ rsync -zav ~/{Documents,Pictures,Videos} ~/backup/remote
sending incremental file list
rsync: [generator] chgrp "/home/ek/backup/remote/Documents" failed: Permission denied (13)
rsync: [generator] chgrp "/home/ek/backup/remote/Pictures" failed: Permission denied (13)
rsync: [generator] chgrp "/home/ek/backup/remote/Videos" failed: Permission denied (13)

Artur Meinild

Nov 1, 2020 17:06

@EliahKagan You're right that SSHFS is rather picky about permissions

It has to be the same user and group ID on both sides

Eliah Kagan

I'm not sure how properly to solve this.

Ah.

Artur Meinild

I think that's just how it works

Eliah Kagan

Okay.

αғsнιη

@EliahKagan I'm not sure if it's because of rsync that is trying to change the group of the file it's copying while no that group_id in destination side?

Artur Meinild

Is it because UID and GID was different between the machines?

Eliah Kagan

Nov 1, 2020 17:07

Yes. Or, specifically, the GID was different. The UID was the same.

ek is the user I created during installation and has user id 1000 (on both systems), but on the client the ek group has group id 1001, while it has group id 1000 on the server. I don't know why ek has gid 1001 instead of 1000 on the client. Maybe some weird thing specific to the Calamares installer (since the client is Lubuntu which uses that).

Artur Meinild

Hmm okay

user.dz

try without preservation flags

Artur Meinild

@user.dz Good idea - try resync without the -a flag, which is contains UID and GID preservation

*rsync

αғsнιη

@ArturMeinild for your information to edit your messages text you can hover mouse on your messages and an arrow button will appear, click there and select "edit"

Eliah Kagan

It actually works even as run, it just gives the error. Like, it does everything I want it to do (copying/updating files).

Artur Meinild

Nov 1, 2020 17:10

@αғsнιη Thanks - I haven't chatted much here :-)

Eliah Kagan

However, if the server had a group 1001, then I think it would try to change the GID of the files to that -- and if my user on the server were a member of that group, then the operation would presumably succeed -- which I wouldn't want.

Artur Meinild

@EliahKagan That's odd - so it's actually just cosmetic

αғsнιη

@ArturMeinild no problem

Artur Meinild

Yeah but as I said, SSHFS can be a bit picky

Eliah Kagan

@ArturMeinild I've tried a variation on that where I keep everything -a does except the GID part, and that works. Thanks -- and thanks @user.dz.

ek@Grr:~$ rsync -zav --no-g ~/{Documents,Pictures,Videos} ~/backup/remote
sending incremental file list
Documents/
Documents/greeting.txt

sent 227 bytes  received 41 bytes  536.00 bytes/sec
total size is 17  speedup is 0.06

αғsнιη

Nov 1, 2020 17:12

@ArturMeinild thank you for the lesson; and thanks everyone, I have to leave; bye

Artur Meinild

@EliahKagan Nice, that's good to know!

Eliah Kagan

(That is, I added --no-g as a subsequent argument, rather than removing -a, so that I would still have -rlptoD, as implied by -a.)

Artur Meinild

@EliahKagan Good solution! :-)

Yeah thanks for now everyone - later !

SSH, Rsync and Cron - Continuation, clarification and questions

Ask Ubuntu Classroom

Participants