/dev/chat - 2021-10-20

« first day (4029 days earlier) ← previous day next day → last day (919 days later) »

7:22 AM

@MichaelHomer The answer I wrote (which I had forgotten about) is deficient, because it doesn't address deduplicating backups. Perhaps I didn't know about them at the time. I guess I should edit it.

Anyone here still using Restic? I recall @Kusalananda was using it some time ago.

I just thought Ben might like to re-view that question

8:10 AM

@FaheemMitha I'm still using it. It's working well and is stable. Doing hourly backups of five machines to the one and same backup repository. Pruning old backups daily.

8:46 AM

@Kusalananda Does restic compress backups too? How far back are you keeping the backups? Does it ever get "stuck"?

Looks like they are releasing actively, at least.

@MichaelHomer re-view? Has he already seen it?

9:11 AM

@FaheemMitha it dedupes and compresses (it’s all configurable)

9:42 AM

@StephenKitt Do you use it too?

@FaheemMitha I still use Borg

@StephenKitt OK.

9:55 AM

@FaheemMitha No compression. I keep a year's worth, but I also archive the whole backup repository weekly using rsync untilI run out of space (haven't happened yet). Nothing ever gets stuck in my experience.

@Kusalananda Does Restic not have compression, or you just don't use it?

Ah yes, I was wrong about that, Restic doesn’t support compression.

@StephenKitt OK.

10:11 AM

Greetings!

@FaheemMitha Restic does not support compression, but there is an open issue in its Github repository about it. It's currently not implemented due to fiddly security issues if I remember correctly. Or possibly due to data consistency issues.

I'm uncertain I would actually use compression if it ever got implemented. Much of my data on disk that I back up is compressed anyway, so it seems useless to run it through a process that would spend CPU time for little gain.

@MartinTournoij I was wondering if it is safe to run smartctl on my failed hdd?

@Kusalananda You don't backup text, or you compress the text before backing up?

If it is the case that
> If the drive is actually damaged then then the safest thing to do is copy it first since any further reads risk damaging it further. I

@FaheemMitha I do back up text, but compared to any other bit of data that I back up, that's a tiny amount. I back up whole machines, from the root directory down (avoiding the obvious bits like /dev etc.) so that includes installed binaries, kernel and a host of other archives and files.

10:22 AM

@Kusalananda That's a whole lot of backups.

> Smartmontools displays early warning signs of hard drive problems detected by S.M.A.R.T., often giving notice of impending failure while it is still possible to back data up

Can I make use of it for the seemingly inaccessible hdd with ext4 filesystems?

@FaheemMitha Well, it's deduplicated. I use about 300 GB for a year's worth of backups of 2*macOS, 2*OpenBSD, and a Linux machine. Or 4 TB for two years of weekly snapshots of the backup archives.

Also, how would you guys choose a data recovery service?

Do you guys do regular backup at bit level (like dd) or file level (like rsync)?

@Kusalananda I suspect deduplication works well with binary files.

It really works best with text.

@Ben file level, every 15 minutes

10:37 AM

@Ben File level. Hourly. Not sure what you mean by bit level.

@StephenKitt Thanks. I don't know what to do with that failed hdd now. There is only one local store in this town for computer repair, and my instinct makes me doubt whether they have the skills or can recommend other places that can.

If I can do anything on my own, I will try

@Ben IME with a failed HDD, if you’re planning on asking a data recovery service for help, you shouldn’t try anything yourself

@Ben I wouldn't know how smartctl could be of help given that your drive has already failed and your aim is to recover your data from it. As others have suggested, I'd probably 1) hand the disk to a professional recovery service and do nothing else with it, trying not to further damaging it; or 2) use ddrescue or dd_rescue, as others have suggested, to try recovering the data yourself; I have never used them, though.

The beep sounds I heard on and off since Sunday may indicate it is a hardware failure. I have tried to rsync my files out when the system was still on, I didn't get all of them out, there are some copying errors, and i don't know how much percentage that I didn't manage to save

After I rebooted, the Ubuntu system made some checking and recovery. Then reboot and then stuck at emergency mode

Do you have a spare disk with enough room on it to store a copy of the entire failed disk?

10:46 AM

I have to purchase one. so I was also wondering what device you would choose

My only backup device is a portable 1TB hdd

Already have some files saved from the failed hdd

I should have invested on some backup device, if I were financially comfortable

But anyhow the loss is greater

@Ben I have a VPS. Ramnode offers them for USD 5 a month.

Or you can use your own computer for backup.

But a separate machine, preferably in a separate location, is best.

In case there is a hurricane or a wildfire or something.

It won't help with the Zombie Apocalypse though.

@Ben data recovery tends to be extremely expensive

@Ben any disk larger than the one you’re trying to save

The hdd is about 6 or 7 years old. I guess probably too short for its life expectancy. One of the reason of causing the hdd failure, I guess, is that I am still using Ubuntu 18.04 LTS already not supported. I can't update packages, and I don't do that often. I used to keep it running for months without problem, but in recent months, it became frozen often while I was writing, and I had to turn it off by the power button

The OS you’re running wouldn’t cause a HDD failure...

Is that hurtful to hdd?

I mean turning off a frozen OS by power off button

10:53 AM

@Ben not to the drive itself; it can cause data corruption though

I saw more system and filesystem checking when restarting

@Ben yes, because the file systems would be “dirty”; but that wouldn’t contribute to physical failure

Is keeping it running for days, weeks and months hurtful to hdd?

@fra-san I don't know how to choose a professional service

I am now moving to NixOS on a laptop which I gratefully got from someone's trash

@Ben no

I can't find a hdd in trash.

I hope NixOS give me better experience than Ubuntu.

I will see if I can get some $$ for a hdd for mirroring the 1TB failed hdd. Do I need 1.5 TB or 2TB? The / partition is about 60GB, and /home is about 900GB.
How do you guys choose a data recovery service?

Is desktop hdd better for mirroring backup than laptop hdd? cheaper, but the physical inconvenience can be overcomed?

Yesterday, I have installed the minimal version of NixOS. I am still learning how to install other packages for GUI and text editors

11:44 AM

@Ben I have never had to contact one, so I wouldn't know either. Sorry. Anyway, as Stephen said, I'd expect them to be expensive.

@Ben ideally you need at least twice the size of the disk to be recovered, so that you can store an image of the disk alongside ddrescue logs and the recovered files (you want to image the disk and never modify the image), so 2TiB at least.

It is possible to get by with a disk only slightly larger than the disk to be rescued, but that requires a lot of care and familiarity with recovery tools.

12:06 PM

@StephenKitt Thanks. If just for mirroring, is a same size device ok? You are right, if I run recovery tools, then I still have to find much bigger device

@Ben no, because with a flaky disk you need to have somewhere to store recovery logs (so that whatever tool you use to attempt the mirroring knows what it’s managed to recover).

@StephenKitt Do mirroring tools also create recovery logs?

@Ben if they don’t, you won’t know what you’ve managed to recover...

is dd sufficient for mirroring for other tools to recover later?

Put it another way, you’re already kicking yourself for not buying a large enough backup device; don’t repeat the same mistake with your recovery device.

@Ben no, you need something like ddrescue.

12:12 PM

@StephenKitt Thanks, I will look for a 2TiB cheap device. I was wounded by not having invest on another backup device

@Ben Is "mirroring" referring to making a copy of your currently failing drive or to preventing data loss in the future, with other drives?

@fra-san Isn't mirroring bit level copying such as dd?

@StephenKitt How do you feel about NixOS? Compared to Debian?

Worth to switch to or have both?

@Ben I’ve never used NixOS

Why does execlp require the filename twice? First as the filename, second as the zeroth argument. Isn't this redundant?

int execlp(const char *filename, const char *arg0, ... /* (char *)0 */ );

@FaheemMitha the second one isn’t the file name

there’s a Q&A about that somewhere on the site

12:19 PM

Are there cases when the second could be something other than the filename?

A command name can be matched by several files

@StephenKitt Technically, no. But in practice is it ever not?

@Ben True, but how is that relevant?

In my experience, the zeroth argument is always the filename. OK, I'll search.

you run a command mostly by command name only

131

A: Why does argv include the program name?

To begin with, note that argv[0] is not necessarily the program name. It is what the caller puts into argv[0] of the execve system call (e.g. see this question on Stack Overflow). (All other variants of exec are not system calls but interfaces to execve.) Suppose, for instance, the following (us...

@FaheemMitha login shells for example

12:25 PM

That backup question (by "our" Tim, if I'm not mistaken) is a sensible question, but seems to have attracted little interest.

Even though in my experience DVCS is one of the best ways to do backup. Doesn't work well for large files, of course.

@StephenKitt What would the zeroth argument be in that case?

Perhaps more directly relevant:

16

Q: Why do we have to pass the file name twice in exec functions?

I read Advanced Programming in the UNIX Environment by Stevens, 8th chapter. I read and understand all the six of exec functions. One thing I notice is, in all the exec functions: first argument is the file name / path name (depends on the exec function). second argument is argv[0] that we get...

command-line exec

@Ben I was referring to the word "mirroring" in your own sentence, to understand whether you were still talking about recovering your data or about adding a level of redundancy to data in general (e.g. to your future systems).

@fra-san yes I was talking about making a mirror and maybe send it to some people for recovery.

OK. What Stephen said, then. I just wanted to make sure.

Ah yes, goldilock's answer reminded me of the use case of this. I think that in TeX they use it. If the same executable is invoked by different names, that changes the behavior.

@fra-san I was also installing Linux on another computer, so had to consider whether to use LVM. I guess its potential risk of complicating data recovery is more than the flexibility it may bring, for just one hdd

12:33 PM

wurtel's argument seems on point:

> You don't have to pass the file name twice.

The first one is the file that is actually exec'ed.

The second argument is what should be the argv[0] of the process, i.e. what the process should see as its name. E.g. if you run ls from the shell, the first argument is /bin/ls, the second is just ls.

You can exec a certain file and call it something else via the second argument; the program can check its name and behave differently according to the name. This can also be done via hard links (or symbolic links) but this way gives more flexibility.

@FaheemMitha busybox comes to mind as well

So if I invoke an executable via a symbolic link called 'foobarbaz', that's what the process is called, apparently.

@JeffSchaller Busybox?

@FaheemMitha -bash for example

I haven't experienced much with LVM, so I was asking about others experiences

@StephenKitt As opposed to bash?

12:36 PM

bash calls execve()

@Ben Which features (from LVM) do you feel may be useful in your case?

@FaheemMitha see

@Ben I've used LVM for many years. Never had any problems. I use it with software RAID (whatever the Linux flavor is).

446

Q: Difference between Login Shell and Non-Login Shell?

I understand the basic difference between an interactive shell and a non-interactive shell. But what exactly differentiates a login shell from a non-login shell? Can you give examples for uses of a non-login interactive shell?

@StephenKitt Ok, will do.

I used to think that hardware RAID would be more reliable than software RAID. My experience suggests the opposite is true, though it's only the experience of one person, of course.

12:39 PM

@fra-san I tried LVM because I like the fact that it allows me to reorganize filesystems and partitions, without making copies.

I am still open to use LVM someday, but starting with a new Linux distribution is already a lot right now

@Ben LVM is not a big deal. Though experiment before actually using it for something.

I am also uncertain about how to choose a data recovery service.

@Ben You don't need a data recovery service. You need backups. Preferably multiple backups via different methods, and you also need to test them periodically.

and I am open to any information about data recovery service

Via some automated method, because no human is actually going to sit and test backups periodically.

12:44 PM

@FaheemMitha for Ben that’s only a useful suggestion if you also have time travel

@StephenKitt For future reference. Fresh out of time machines here.

@FaheemMitha I see what you did there...

Do companies regularly update their computing devices? If so, do they give out old working devices? for example, to unfortunate people?

@Ben I agree. Though, from that perspective, it may or may not make a big difference, depending on the partitioning scheme you choose. Especially with respect to a laptop with a single drive.

@FaheemMitha I was going to point to some source code, but the level of complexity is too high for me to grok at the moment. My understanding is that the code compiles binaries of various names (cp, ls, etc) that are hardlinked to the same image; that image runs different code based on the name it's called as (cp, ls, etc)

12:49 PM

@JeffSchaller even coreutils can be built like that, it’s not just Busybox ;-)

@JeffSchaller I see.

@Ben in my experience, such companies contract with disposal companies that record the disposal (and possibly destroy data/drives to some degree)

In the case of pdflatex one can see that it's a symbolic link to pdftex.

@StephenKitt having never built coreutils, TIL -- thanks!

But Busybox etc. are somewhat orthogonal to exec’s two arguments: such tools are normally invoked through a link, so the file that’s used actually matches the expected command name.

12:51 PM

though I have found one PC abandoned by the side of the road; I passed the drive along to a friend who was interested in it forensically

Otherwise everyone would have to know how to call Busybox as ls...

@StephenKitt I've been counter-pointed; thanks! :)

@Ben Define "computing device". I think HDDs are physically destroyed.

I hope they can mail the hdds to me

@Ben as far as I’m aware, nowadays, in the US companies tend to lease their hardware; in Europe, companies either lease or buy, but when they buy, they have to dispose of it once it reaches its EOL, and keep paperwork showing it’s been properly disposed of (i.e. recycled).

Anything storing data is typically physically destroyed so that the data can’t be recovered.

12:53 PM

when I was foolishly trying to build my own Beowulf cluster, I found a business that sold machines that had been disposed of by larger companies, but that was decades ago, before additional regulations

@StephenKitt Are there more humane ways of making recovery impossible, and still fullfiling the true meaning of "recycle"? isn't dd zeros make that possible

Right, there are still hardware brokers, and that can be a valid outcome for hardware, but it’s less common than it used to be.

@Ben “recycle” in IT typically means “break down into constituent components”, e.g. recover metals etc., because hardware is deemed obsolete once it reaches its end of life and its re-use value is nil

I seem to remember some companies change their laptops every a few years. But a laptop can last much longer

@Ben yes, that’s common because of warranty concerns and business continuity

I was sometime wondering: Can I have one then?

That is how I was lucky to get one from someone's trash

actually two

1:02 PM

I'm confused why you're specifically looking for a used drive when you just had one fail.

because life taught me so

1:30 PM

Folks, how to I break up tput > /dev/null for execlp?

I tried:

execlp("tput", "tput", "> /dev/null", (char *)0);

and also

execlp("tput", "tput", ">", "/dev/null", (char *)0);

They both give the same results, but different from what is produced by just typing the command directly.

> is a shell thing, not a C thing

In C, it goes like int fd = open("/dev/null", O_WRONLY | O_CLOEXEC); dup2(fd, 1); (with error-checking)

2:03 PM

@JeffSchaller Yes, I realised that after posting this.

@StephenKitt And then call tput?

That worked, thank you.

@FaheemMitha out of curiosity, why do you want to run tput and discard its output?

@StephenKitt Error testing. chat.stackexchange.com/transcript/message/59383566#59383566

Courtesy of @fra-san.

@FaheemMitha ah, instead of sh -c 'exit ...'?

The Stevens book is proving to be very helpful. I recommend it if you are trying to do this sort of stuff.

@StephenKitt Right. I want some "real" situations. :-)

@FaheemMitha they’re just as real

2:12 PM

@StephenKitt Or just as unreal.

@FaheemMitha exactly

2:23 PM

I feel somehow responsible for this. In front of questions I tend to see puzzles (e.g. how can I find, quickly, the standard utilities that...). Happily getting caught in the X/Y problem. :-)

@fra-san Feel responsible for what?

@FaheemMitha For suggesting a way (for you to test error detection/handling) that's less simple than it could. Not a serious thing, thouhg. No worries.

1 hour later…

3:59 PM

@fra-san I think it's a good test option.

@FaheemMitha instead of using execlp and relying on a specific tool’s exit codes, you could also just call exit() directly ;-)

4:13 PM

@StephenKitt I don't follow. I was just trying to check that the exit codes that come up are the correct ones.

@Ben I think I've seen (at least) three different companies in Finland that deal in pre-used business laptops. And if I had to buy a laptop with my own money right now, I'd probably go browse one of them...

@ilkkachu Is used hardware in Finland reliable? Because it probably isn't here.

@FaheemMitha ah, sorry, I thought you want to test your own code’s handling of various exit codes

@ilkkachu So nice to learn that there are ecofriendly people

@StephenKitt I'm just trying to see whether those executables with those option exit with the same exit code as they do in the shell.

4:16 PM

@FaheemMitha why wouldn’t they?

Not a very lofty ambition, but a couple of days earlier I got myself into a very confused state, so I'm proceeding cautiously.

@StephenKitt If my code is wrong they wouldn't.

Just curious. Have most people here read Stevens' book?

@FaheemMitha yes, they would. What your code does doesn’t change how a program you call exits.

@FaheemMitha I’ve read Stevens’ books, not Steven’s ;-)

The latest date on the Indian edition I have is 2000. Which is odd, because I wasn't in India in 2000. But maybe I got someone to send it to me from India.

At any rate, it was a while ago. Though I never read it at the time.

@StephenKitt I think you underestimate my ability to write incorrect code. I have some code here, the last time I checked, gave a different error code every time I ran it. Still not sure how that happened.

@StephenKitt Stephen's read Stevens' books, not Steven's

4:36 PM

@FaheemMitha well, they have to abide by the consumer protection laws, and of course something like that partly works on reputation too

@ilkkachu I wish India had consumer protection laws. And they might even have them - on paper.

Are Finns active posting reviews online?

1 hour later…

5:42 PM

Interesting. I found a case where NULL vs (char *)0 actually makes a difference in practice. Does anyone know why?

The exec call is:

execlp("sh", "sh", "-c", "./nopath", (char *)0);

Here is the complete code.

As written, it gives:

> sh: 1: ./nopath: not found
Child's exit code 127

No, scratch that, I was mistaken.

Figured it out. Put quotes around the NULL. So it was "NULL". Sigh. Silly me.

I guess the question then is what happens when execlp isn't properly terminated with a NULL pointer.

sounds like you have too many computing languages in your head at once :)

6:27 PM

@JeffSchaller Just a silly mistake. Looks like it got it from blindly copying from

geeksforgeeks.org/exit-status-child-process-linux

I assume that using "NULL" is an error, right?

Google really should stop putting links from those sites on top. They are usually not very good.

OK, looks that is where my strange exit codes were coming from. Though I wonder what the compiler was understanding by that.

@FaheemMitha ew, geeksforgeeks

let me fetch the petrol and lighter

@AndrasDeak I assume that's not intended as an endorsement.

indeed

geeksforgeeks, w3schools, realpython... some of the top crap (realpython is a bit different because it's python-specific and it's more of a gamble than uniform crap)

7:16 PM

Am I correct in thinking that the exec family "only returns on error" means that it is unable to execute the command at all? As in, there is a syntax error? It doesn't mean that the command itself returns with an error, right?

1 hour later…

8:35 PM

@FaheemMitha After the command executed by exec has replaced the one exec is part of, there is no running exec to return to anymore. No?

« first day (4029 days earlier) ← previous day next day → last day (919 days later) »

Transcript for

Oct '2120

/dev/chat

General discussion for unix.stackexchange.com. If you have a q...

join this room
about this room

00:00

06:00

12:00

18:00

all times are UTC

site design / logo © 2024 Stack Exchange Inc; legal

mobile