Chris Davies

Unless it's just an example for the question here, cp over sshfs can almost certainly be more efficiently handled with either scp or rsync. And you won't get the side-effect (bug?) you're describing here

When you say you tried write myuser pts/0, did you actually use the six characters myuser or was it really write vtian pts/0?

:-)

@JeffSchaller thank you - much more polite and intentional now!

Thanks Jeff. unix.stackexchange.com/questions/797050/… the second sentence is supposed to read "If you stop creating local accounts…"

@Kusalananda are you and the other mods able to edit comments? If so could you fix one of mine for me please? I'm not ready to delete it but there are a couple of autocorrect typos I'd really rather weren't there

@ilkkachu thank you for editing the shift. I must have copied and pasted an ancient piece of code to have left my original in there!

Not convinced that Different dollars! is a duplicate. To my mind the question being asked shows that the OP doesn't even realise that $1 is a command argument. But I can't just vote for a reopen as I (think I've) got enough privileges simply to hammer it open again. @StephenKitt?

Interesting - I don't see many dependency problems with Debian. I mostly run servers, mind. I do have a couple of desktop-focussed systems that have Backports included, and those can get really upset when upgrading to the next major version

I've been on Debian since I switched from (IIRC) Pink Hat Linux after getting fed up with rpm dependency hell (there was no yum, but there was apt-get)

Official - Arch, Debian, Ubuntu, Kali, Fedora, Alpine

There's Debian and Ubuntu, so we do have alternatives to punt people towards

@terdon Looks like there's network enabled in WSL Kali by default, yes. No raw access to network interfaces, so I'm not entirely sure what use Kali is in this situation

Using your favourite editor (I like setting export EDITOR=vi but you may prefer something different, such as the default), run systemctl edit rsyslog.service

We need to change rsyslog.service so that it is dependent on syslogs.mount

If you run systemctl list-dependencies you'll see two relevant entries, one for rsyslog and one for your filesystem mount, something like syslogs.mount

When you're back to a point where the files are in the right place and you've disk space available on / once more, you can continue...

systemctl stop rsyslog; systemctl disable rsyslog

Stop and disable the syslogger, rsyslog and repeat everything we've done here. Then we'll need to change the dependency so that the logger doesn't start until your disk is mounted

In that case I would say that whatever is logging is starting before you mount the separate filesystem on /syslogs. You need to fix that dependency

@Beckyboo any progress?

Remember that telnet isn't the same as ssh. Did you start the ssh server on a different port, or perhaps it's firewalled off? Have you told Azure that it's to allow inbound ssh traffic?

OK. You can log in to the serial console?

Sure. You've got a recipe now in my answer (with /mnt and /mnt/root) to check underneath the /syslogs mount point to ensure nothing's being written there unexpectedly

Unfortunately I'm not sure enough about the interactions inside systemd to be able to advise you how to do that. It might be sufficient just to have /syslogs in your /etc/fstab and systemd will do the right thing. But I don't know

You need to get /syslogs mounted before rsyslogd starts

You've got rsyslog configured to write its log files to /syslogs. That's fine. But I think it's starting before you've mounted your extra disk on /syslogs, so it's writing to what are now hidden files on the / filesystem instead of visible files in the /syslog filesystem

This may have fixed it properly, or it may have only fixed it temporarily. Here's my theory ...

Last things then: cd / and then umount /lost+found

Is that good or bad?

Yay! You've got your disk space back

Now df -h / again please

Great. That's good news. So there's nothing anymore in there. But you're saying that df -h / is still showing 100% used, so we need to look elsewhere. Please try lsof |awk '/deleted/ {print $1}' | sort -u | xargs and include the output here.

What do you get from running du -hxs /lost+found/syslogs

You're trying to kill the grep command - look at the output - which of course has already finished by the time you get round to trying to kill it

OK. Has the disk space on / reduced at all? (run df -h /)

you want to save, copy, or delete files from /lost+found/syslogs NOT from /syslogs

Careful. You're in your real /syslogs directory there. That's presumably where you DON'T want to delete things

Yes. Because it's hidden by your //logs.windows.net/sys mount on /syslogs

No worries. Call me back with @chrisdavies when you're done

The target is to have /lost+found/syslogs empty

OK. So having exposed the original /syslogs as /lost+found/syslogs you can access the files and move them out of there or delete them. That will reduce the disk used on /

Still with me?

However, /syslogs is not mounted on /lost+found/syslogs so we can see the files that it's hiding

I chose (b), and suggested originally you put it at /mnt/root. You couldn't do that so we have temporarily repurposed a different directory, /lost+found. So now the filesystem on / is ** the same ** filesystem that's mounted on /lost+found.

OK. So I reckoned you have a load of files in the hidden /syslogs, and it's these that are taking up your disk space. But we can't get to them directly. At this point we have two options: (a) unmount /syslogs so we can see if there are (still) files there, and (b) create a "bind mount" that reproduces / at a different point, but without the additionally mounted filesystems

This means that you cannot see what's on the root filesystem (/) under its own /syslogs - you can only see what's in the filesystem that is currently mounted on /syslogs. Yes?