@mikeserv, I want to be very careful when giving security advice after I was less than completely vigilant over here. Thus, your review of this answer would be appreciated, if you like.

@Celada I'm not mikeserv, but FWIW I agree with you - the main thing here is to limit the key rather than to protect its passphrase. I'd add limiting not only what it can do, but also where it can connect from. Have a look at serverfault.com/a/507899/120438 and security.stackexchange.com/a/76882/13705 - in both of those I've given similar answers to yours

@JennyD Super, I added from="..." and no-pty and all that to my answer. I should use that more often myself!

@Celada I have actually used that for backups in a very security-conscious environment - I used to work at a bank, working with the security systems for the on-line banking system

I think too many people rely on "oh I'll just set a password then" and they don't consider what happens if/when the password is compromised

@JennyD Lucky you. Too often I try to do security work and my colleagues/boss don't value it as useful.

@Celada That was a very good place to work - security really was highly important, to everyone. We'd rather have downtime than a security issue... and we had very little downtime. The main thing is to have in-depth security - if you start with the premise that "everything can fail", then you learn to not rely on just one thing to secure your system.