/dev/chat

Jeff Schaller

01:17

I'd be interested in knowing if anyone has coded up a Unix CLI-based Stack Exchange utility. I'm curious to try my hand at an "edited since my downvote" list of posts meta1, meta2 .

I don't have to reinvent the wheel if someone already has something mostly-built. Thinking along the lines of curl or wget (after authenticating) to get my downvoted posts, then see when they were last edited.

Jeff Schaller

01:31

11

Q: Get a list of posts you've voted on that have been edited

About Many posts over the years have asked for ways to systematically ensure up- and down-votes are as up-to-date as the posts they were cast on, whether by notifying as soon as a post was edited, or by listing posts that now qualify for retracting votes. This script was inspired by this sugge...

tripleee

02:09

@JeffSchaller I'm thinking you could do that in SEDE but I'm no good with SQL

Jeff Schaller

02:27

@tripleee not directly as your votes are anonymous. SEDE was a plan B, though, as I considered outputting a custom SQL query that essentially retrieved a predetermined list of posts (based on a previous client-side script that came up with the list of posts that had been edited after voting)

The script from Nathan, above, seems to work (at first glance; will play with it more tomorrow)

tripleee

@JeffSchaller oh silly me, of course

Faheem Mitha

05:22

@JeffSchaller Why would this be useful?

Faheem Mitha

08:45

Isn't 25 M/sec rather slow for a copy from one part of an SSD to another part of that same SSD?

To be clear, using rsync with the -z flag. Does compression slow things down?

Kusalananda

@FaheemMitha Compression in file transfer protocols in general only has a positive effect if the transfer link is very slow.

terdon

Seems pointless when copying on the same device.

Faheem Mitha

@Kusalananda Yes, I suppose so.

terdon

It doesn't even make sense for copying over LAN. The compression will only make it slower. Don't use it unless you're copying over the internet or similar slow connections.

Faheem Mitha

Should I restart without -z?

Kusalananda

08:50

Basically, the rate of compression/decompression has to be faster than the rate of transfer.

@FaheemMitha Yes.

Faheem Mitha

Ok, seems to be faster now.

I see I'm using the -b option, though I can't remember what it is good for.

Is there some way of finding out what the speed of transfer of a rsynccommand is? I'm running iotop, but the speed it shows keeps fluctuating.

terdon

@FaheemMitha That's normal.

Faheem Mitha

@terdon What, that the speed keeps fluctuating?

Kusalananda

@FaheemMitha With --progress you will get some info about individual files. With --stats you wil get a summary at the end.

terdon

@FaheemMitha Yes. That's how transfers work, I've never seen constant speeds.

Never knew why, either, but that's how it's always been as far as I know.

Faheem Mitha

09:01

@terdon If my computer was an AI, I could ask it (her? him?). Of course, it might not tell me.

@Kusalananda There is some new flag since 3.1.

terdon

For instance, transfers over ssh (scp) always start fast then slow down.

Faheem Mitha

30

A: Can rsync display current average speed?

Yes. Starting with rsync version 3.1.0 the --info=progress2 argument will give you progress on the entire transfer, including speed of the entire transfer. You can see a little bit of detail on the rsync man page.

Kusalananda

Ah, cool. Didin't know that.

Faheem Mitha

@terdon This is a local transfer. And it doesn't seem to be slowing down as such. It's fluctuating.

Finished now.

terdon

@FaheemMitha Ah, right. But I think that's normal as well. Makes perfect sense for a magnetic drive (since the head needs to move around). Dunno why with an SSD though.

Faheem Mitha

09:04

Lots of rain over here this last week or so.

@terdon An accurate measurement would be nice.

terdon

@FaheemMitha But what makes you think what you see is not accurate?

Faheem Mitha

@terdon I don't find iotop particularly confidence-inspiring, for some reason.

And different commands keep appearing and disappearing, which makes it confusing.

Kusalananda

09:19

@FaheemMitha You always have this problem. You either show instantaneous values (which means that "things appear and disappear") or you show aggregated values (that may be misleading at ony one point in time).

Faheem Mitha

09:32

@Kusalananda I think one could show continuously changing instantaneous values.

Kusalananda

@FaheemMitha But if one process isn't doing I/O right now it wouldn't show up in a screen update. Then it would show up in the next, but not in the one after. To have it show all the time, you would have to say "this process has done an average of this much I/O over the last x seconds".

That way you get a value that is wrong in the moment, but accurate over time.

Faheem Mitha

@Kusalananda That's true, if it wasn't doing I/O, it wouldn't show up. But is there some reason to believe it isn't doing I/O at some moment in time?

Kusalananda

@FaheemMitha Context switching by the kernel. If several process are wanting to do I/O, they are queued up and if one get high priority it may block others for the duration of the screen update. rsync might obviously also wait for the receiver/sender to do its thing before continuing.

Faheem Mitha

@Kusalananda Good point. But these days a processor is really multiple processors.

I think mine has 6.

Though I suppose there still might not be enough processors to go around.

Kusalananda

I/O to any single device is serialized.

... by necessity.

Kusalananda

14:18

When someone comes along who clearly has specialist knowledge, and writes a good answer to a question, I wish I could upvote it more than once :-)

Pandya

@terdon I'm not sure how much of FSF's campaign and other relevant things would be accepted there. Suppose one wants to do his/her computing with free (as in freedom) softwares (platform) and he is facing some troubles, opensource.SE may not be suitable place, of-course U&L and other sites like SO, SU can be suitable, actually I want to propose an FSF-specific site :P. But I afraid that such site may draw opinions and consequently no-compatible to SE model.

(I think I need help of other peoples/users like you to have a reliable ground for thinking towards it.)

Kusalananda

There are other SE sites built around philosophies that seems to work well. I'm thinking primarily about the religious sites. They would presumably also have to deal with opposing opinions, but they still fall within SE's model.

terdon

14:40

@Pandya I'm not really a good person to ask. I have little patience for the dogmatism of the FSF.

Faheem Mitha

@Pandya Why wouldn't opensource SE be suitable? Questions about software freedeom would clearly be on topic.

Not everyone on the site would necessarily be in sympathy with a specific point of view, like the FSF one, but that's hardly a reason for creating another site.

Eliah Kagan

@terdon But it wouldn't have to be just about the specific free-software views propounded by the FSF, would it? For example, it could also include people who believe software freedom is an important moral and political issue but who usually prefer non-copyleft free software licenses. (I believe this would describe many BSD contributors--at least in OpenBSD, which is the BSD I'm most familiar with.)

Although it's true that sometimes non-copyleft licenses are chosen to cultivate relationships with specific businesses that prefer them, but in general a significant part of (at least what I would call) the free software movement considers them better, and there is thus disagreement within the movement about which respects users' freedom better and which is more strategically and tactically suitable.

A number of free software projects, including OpenBSD, use non-copyleft free software licenses for ideological reasons. I would want that to fall within the scope of such an SE site.

Faheem Mitha

@EliahKagan But I doubt it's possible to create a SE site with a specific point of view.

Eliah Kagan

Well, it would be about an idea/movement. I don't know if it's practical or a good idea but it could certainly welcome people with different perspectives including people who disagree altogether. Still...

The thing is, such a site--at least with the way SE currently operates--would exclude many of the people whose perspectives and contributions would be the most interesting and useful, because you basically have to use proprietary software to use Stack Exchange, and a number of people either won't use it at all or will generally avoid using it. More precisely, the SE backend is proprietary, which would exclude people who strongly prefer free software and include the cloud in that preference.

Faheem Mitha

I suppose if there is a site about Judaism, why not a site about FSFism? For lack of a better word.

@EliahKagan That's a good point. I know that Debian developers, for example, tend to avoid SE.

Though not all of them. We have a couple who are regulars here.

I'm not happy myself that SE runs on MS, but it's not like one has a lot of choice about where to get help on the net.

Eliah Kagan

14:52

The JavaScript-based frontend and official mobile apps are also proprietary, and that's how most people use SE, as well as how SE suggests to use it. I don't think there's any prohibition on FOSS SE API clients but SE still pushes people pretty hard toward running closed code on their own machines. I doubt many of the most interesting/important voices would make it to such a site. I don't just mean prominent people, but people such a site would want to bring out of their shells (pardon the pun).

Faheem Mitha

@EliahKagan Some free software developers are active on SE. But overall, not that many.

terdon

@EliahKagan Sure, but I was replying to what Pandya had suggested:

yesterday, by Pandya

@terdon I'm thinking to propose a site on Area51 about Free Digital Society (based on GNU & FSF Philosophy) where people can question answer regarding freedom in computing and net and so many other philosophical concepts of FSF like surveillance, security, privacy, software freedom, resources for free software, free software in education, hardware that are compatible with free software, tldr; FSF campaign and much more relevant things. How about it?

Which seems more FSF and GNU centered. And don't get me wrong, I have a lot of respect for both organizations, I just also find their positions a bit extreme for my personal tastes.

Either way, I'm not saying the SE wouldn't be useful, only that I personally don't care much.

Eliah Kagan

@FaheemMitha Well one can be a free software developer or proponent without thinking all software should be free. Or think all software should be free without thinking one should only use free software. Or think one should only use free software but still hypocritically use proprietary software. (I don't mean that as an jab to anyone--that's basically my own position/behavior.) Such people might contribute to a software freedom SE. The problem is, we'd also want people who do avoid proprietary.

@terdon Yeah, maybe my suggestion that it should be broader than FSF-specific views applies more to the original idea than to your commentary.

terdon

Actually, it looks like Open Source covers this quite well:

> Open Source Stack Exchange is a question & answer site about a group of related movements which encourage people to give up some of the rights given to them by copyright law, by using licenses which give others the freedom to use and transform their creative works, and which optionally (by using copyleft/share-alike licenses) ensure that these works will always be made available with the same rights. This site is not limited to software, but also covers art, literature, hardware, databases and more.

@Pandya ^^

Eliah Kagan

Yeah, I guess it does. Cool.

LandonZeKepitelOfGreytBritn

15:04

Hello world

Eliah Kagan

@FaheemMitha I can imagine a future where SE uses a mostly Microsoft stack but only free software. I doubt Windows will ever be free software--MS hasn't even published the source of ancient Windows OSes--but Microsoft has a number of free-software projects (not to be confused with "shared source" which usually imposes considerable restrictions). .NET Core and ASP .NET are free as in freedom.

I am not holding my breath on Stack Exchange's own backend code, that we are using whenever we use the site, being made free software anytime soon, though.

@LandonZeKepitelOfGreytBritn Hi. Did you ever get that package installation problem fixed?

Kusalananda

@EliahKagan Hrrm. I don't want to get into an argument, but where did you read that OpenBSD uses "copy-left"? OpenBSD is actively working on removing GNU licensed software from its base system.

Maybe I misunderstood the use of "copy-left" here.

I've always taken it to mean "GNU", but it appears I'm wrong.

Eliah Kagan

@Kusalananda Did I misstate something? I was trying to say that OpenBSD does not use copyleft--that they think copyleft is bad and avoid it.

I was giving OpenBSD as an example of a project deeply committed to freedom but that does not have the same views about software freedom as the FSF.

Kusalananda

It's just that you said "A number of free software projects, including OpenBSD, use non-copyleft free software licenses for ideological reasons."

Ah.

non-

I just woke up from a nap.

Is that a good excuse?

Eliah Kagan

15:20

Sure!

Seriously though, no excuse is necessary. I'd rather you (and anyone) ask about something I've said that might be wrong when it's right, than not ask about something I've said that might be wrong when it's wrong.

Anyway, regarding what I was saying earlier: when talking about about software freedom as an ideology and movement, people often describe it by reference to the FSF. I don't think is inherently bad, and I greatly respect the FSF--I think they are a force for great good. I also think most of their positions are correct (though license-enforced invariant sections being okay in free documentation comes to mind as an example of a position I very much don't agree with).

And I tend to lean toward their views even when they disagree with those of others in the free software movement. But they're not the whole free software movement. OpenBSD is an example of a free software project that is every bit as committed to software freedom as the GNU Project.

LandonZeKepitelOfGreytBritn

@EliahKagan yup, xential-updates and xenial-security were disabled

Faheem Mitha

@EliahKagan Thinking "all software should be free" isn't a well defined term. One might think that in the sense that world peace would be nice. And that we'd like to live in a world with sunshine, rainbows, and shiny white unicorns.

Sorry, the above isn't particularly clear.

LandonZeKepitelOfGreytBritn

15:37

sudo dpkg --add-architecture i386

Can I just add any architecture instead of i386?

Faheem Mitha

@LandonZeKepitelOfGreytBritn I think so. But don't expect the packages to function on your system.

I don't think dpkg knows what archs will run on your system.

But you can just try it and see.

LandonZeKepitelOfGreytBritn

@EliahKagan apparently much software doesn't run on ARM64 architectures

@FaheemMitha what solutions are there if it doesn't function?

Let's say I want to run a generic package on an exotic architecture

Faheem Mitha

@LandonZeKepitelOfGreytBritn I don't think there are any.

Eliah Kagan

You could use an emulator for the architecture...

LandonZeKepitelOfGreytBritn

@EliahKagan like virtualbox?

I thought that you guys would start speaking about multiarch and explain how this would be a solution

Eliah Kagan

15:42

@LandonZeKepitelOfGreytBritn If it's one of the architectures VirtualBox supports. VirtualBox does virtualization rather than emulation. Virtualization is preferable when it is possible, since it is almost always magnificently faster than emulation. But is usually only possible when your actual hardware's architecture is capable of running code for the virtual machine's architecture.

LandonZeKepitelOfGreytBritn

@EliahKagan I think -correct me if I am wrong- your last statement only applies when you do hardware virtualization rather than software virtualization.

TIL software virtualization = emulation

Eliah Kagan

I am not certain that the terms are always used the same by everyone. With that said, under the way I use the terms "virtualization" and "software virtualization" and have seen those terms used, virtualization is where you have a virtual machine in which the machine language instructions that go to its virtual processor are actually executed by the host system's physical processor (so that processor must be able to run it), sometimes with slight modification.

Similarly, under the terms I am familiar with, software virtualization is virtualization that does not use any features provided in the design of the processor to help with virtualization, and hardware virtualization is virtualization that does use such features. Under those meanings, most of the popular type-2 hypervisors (i.e., virtualization software that runs inside a general purpose OS rather than being its own OS) support both software and hardware virtualization.

Some people do construe "emulation" broadly enough to encompass all forms of virtualization, though.

This Super User answer is written in such a way that it seems to support the way you are using these terms. However, there are parts of that answer that are definitely wrong--most of the speedup associated with running guest machine language instructions on the host's processor can be achieved, and has very often been achieved, without the processor having any features to help with virtualization.

For example, when early versions of VMware were commonly used, such processor features didn't exist, but it was still massively faster than emulation.

Hardware-assisted virtualization

In computing, hardware-assisted virtualization is a platform virtualization approach that enables efficient full virtualization using help from hardware capabilities, primarily from the host processors. Full virtualization is used to simulate a complete hardware environment, or virtual machine, in which an unmodified guest operating system (using the same instruction set as the host machine) executes in complete isolation. Hardware-assisted virtualization was added to x86 processors (Intel VT-x or AMD-V) in 2005 and 2006 (respectively). Hardware-assisted virtualization is also known as accelerated...

derobert

16:07

@Kusalananda you can use a bounty for that

(there is a bounty option for "reward an existing answer")

LandonZeKepitelOfGreytBritn

16:19

@EliahKagan indeed that s what I read :)

Kusalananda

@derobert Oh! That is certainly an interesting option. This particular answer might not be worth it, but I know of others...

derobert

The other option is of course to post it here to draw attention to it.

Kusalananda

It was unix.stackexchange.com/a/389086/116858 that I thought was very insightful. I don't know enough to say that everything is correct, but it was just very well written.

... by what looks like a fairly new user.

derobert

Looks right to me, too

Jenna Sloan

16:35

@Pseudohuman Give me a Tux‐like curve, please

Pseudohuman

derobert

@JennaSloan hmmm, what else does it do?

@Pseudohuman give me the equation

Pseudohuman

@derobert I do not understand.

Jenna Sloan

@Pseudohuman Tux‐like curve equation

Pseudohuman

x(t) = ((-22/9 sin(7/5 - 2 t) + 43/3 sin(t + 13/6) + 23/8 sin(3 t + 9/4) - 619/2) θ(31 π - t) θ(t - 27 π) + (172/7 sin(t + 19/5) + 5/2 sin(2 t + 50/11) + 19/5 sin(3 t + 18/7) + 7/8 sin(4 t + 1/4) - 1205/2) θ(27 π - t) θ(t - 23 π) + (-3/4 sin(8/9 - 4 t) + 116/5 sin(t + 17/9) + 64/13 sin(2 t + 22/5) + 16/5 sin(3 t + 27/14) + 5/4 sin(5 t + 13/7) - 46/3) θ(23 π - t) θ(t - 19 π) + (-11/3 sin(5/8 - 4 t) + 157/4 sin(t + 17/6) + 37/7 sin(2 t + 11/3) + 5/4 sin(3 t + 1/5) + 6/7 sin(5 t + 16/5) + 170/7) θ(19 π - t) θ(t - 15 π) + (-5/4 sin(8/9 - 4 t) + 573/14 sin(t + 33/16) + 19/7 sin(2 t + 10/3) + 31/…

2

derobert

16:40

@Pseudohuman Give me a Beastie-like curve, please

Pseudohuman

@derobert I do not understand.

derobert

Guess it only does Linux.

Kusalananda

Pah, so typical.

@Pseudohuman Give me a GNU-like curve.

Pseudohuman

@Kusalananda I do not understand.

Kusalananda

@Pseudohuman Well, +1 to you.

Pseudohuman

16:42

@Kusalananda I do not understand.

derobert

@Pseudohuman make me a sandwich

Pseudohuman

derobert

well... errr... I guess that is technically what I said...

Kusalananda

Ha!

derobert

Good thing I didn't use sudo on that one, or I'd be in trouble!

Jenna Sloan

16:44

@Pseudohuman Nyan Cat‐like curve

Pseudohuman

derobert

@Pseudohuman Squid-like curve

Pseudohuman

@derobert I do not understand.

Kusalananda

16:57

@Pseudohuman I do not understand.

Pseudohuman

@Kusalananda I do not understand.

Jenna Sloan

@Pseudohuman price of tea in china

Pseudohuman

@JennaSloan Your question is unrelated to the current topic of discussion.

Kusalananda

@Pseudohuman two things are infinite

Pseudohuman

The universe and human stupidity; and I'm not sure about the universe.
(attributed to Albert Einstein)

Kusalananda

17:03

Ah, so WolframAlpha it is then.

Jenna Sloan

@Pseudohuman notable facts about Linus Torvalds

Pseudohuman

Software engineer best known for creating the Linux kernel
Initiated the development of and became the project coordinator for the Linux kernel
Developed Git, a file revision control system

Faheem Mitha

So, do we have a bot?

Kusalananda

@FaheemMitha Yeah, it's channeling WolframAlpha I believe.

@Pseudohuman Who are you?

Pseudohuman

@Kusalananda My name is Pseudohuman.

Faheem Mitha

17:16

@Kusalananda WolframAlpha?

Kusalananda

wolframalpha.com

derobert

@Pseudohuman x^2-2x+3

Pseudohuman

derobert

does indeed look like WolframAlpha.

Kusalananda

@Pseudohuman time flies like an arrow

Pseudohuman

17:25

@Kusalananda fruit flies like a banana

Jenna Sloan

You can easily tell it's using WolframAlpha if you check the domain of its images

Kusalananda

@JennaSloan Ah, but that's cheating...

Now, this guy: unix.stackexchange.com/a/389131/116858

Faheem Mitha

17:50

@Kusalananda You mean it's an instance of this Wolfram Alpha software running on the site?

Kusalananda

In the chat, yes.

Although, there is an answer from it on SO too.

Faheem Mitha

Sorry, I meant in the chat. But who is running it?

Kusalananda

@FaheemMitha Jenna showed up at the same time and provoked the first response from it.

Faheem Mitha

The bot code links to someone called Jenna Sloan.

Who I see is in the chat.

So, I'm seeing this slightly weird situation.

I decided to move some files in my home directory to a separate filesystem.

They were in a directory which du -hs said was around 63 GB, I think.

But once I had copied it, the copy is 91 GB.

Kusalananda

Different block sizes?

Faheem Mitha

17:57

Unfortunately, I deleted the original before I realised this. I've got a backup, but I ran out of space on the backup when extracting it to my home directory.

Kusalananda

Sparse files made non-sparse?

Faheem Mitha

@Kusalananda Can you elaborate?

@Kusalananda That I don't follow at all.

How could that make it take up more space?

BTW, is anyone else getting constant streams of imaginary purchase orders? And similar things?

Kusalananda

I don't know what du does with sparse files, but it may show the actual space taken by them, which means that if a sparse file takes 5GB of actual space and then you copy it with something that doesn't preserve sparse-ness, then it may well appear bigger than before.

@FaheemMitha Not purchase orders, but other ... things.

Faheem Mitha

@Kusalananda I copied using rsync.

Kusalananda

`rsync` has `-S`: Try to handle sparse files efficiently so they take up less
              space on the destination.  Conflicts with --inplace because it’s
              not possible to overwrite data in a sparse fashion.
``

Faheem Mitha

18:01

I did actually go to some trouble to verify that the files were the same.

Kusalananda

(forget what I said about block sizes)

Faheem Mitha

@Kusalananda Oh. Well, I've deleted the original, so I presume there is no putting the genie back in the bottle.

Kusalananda

Hmm... probably not.

Faheem Mitha

Unless someone has an idea.

Actually, given that the directory in question contained torrents, that does seem plausible.

Stupid me. Next time I'll be less hasty deleting stuff. Usually I'm not hasty.

Kusalananda

27

Q: Can a file that was originally sparse and expanded be made re-sparse

I know that copying or transferring what was originally a sparse file without using a utility that understands sparse files will cause the 'holes' to be filled out. Is there a method or utility to turn what was once a sparse file back to sparse? For Example: create sparse file: % dd if=/dev/zer...

files utilities sparse-files

@FaheemMitha Oh the files would be "the same" (MD5 checksums and everything would match), but the way they are stored is different.

Faheem Mitha

18:05

@Kusalananda Sure, I understand that. But the 'sparse' files would be more efficiently stored, presumably.

Kusalananda

A sparse file would just have a hole in it, while the copied file would have zeroes.

Yes

Faheem Mitha

@Kusalananda The bottom answer in that question you linked to looks interesting.

Though updating in place might be a bit risky.

rsync -avxWSHAXI . .

Though that's an awful lot of options in one place.

Kusalananda

Yeah. Make a copy of one candidate file and test in isolation.

Faheem Mitha

@Kusalananda I'll probably just run into over the entire directory. But output to a different directory.

Kusalananda

If you have the space.

Faheem Mitha

18:08

Hmm, borg has a --sparse option too.

@Kusalananda I've got plenty of space on my new disks.

Well, perhaps not plenty. But enough.

@Kusalananda Thanks for the pointer. It would have taken me awhile to figure it out myself.

Kusalananda

@FaheemMitha No worries. Thanks for the --sparse option in borgbackup. I was unaware of it.

Faheem Mitha

@Kusalananda Oh, you've heard of borg, then?

Most people haven't.

Kusalananda

I'm using it hourly.

Faheem Mitha

I was just trying to extract from my archives. But I ran out of space.

@Kusalananda Oh, me too.

No, every three hours, I think.

0 */4 * * *     export HOSTNAME=orwell; /usr/local/bin/backup_local_hourly

Kusalananda

@FaheemMitha I back up my whole /home to a backup drive.

Faheem Mitha

18:11

Actually, I think that's every 4 hours.

@Kusalananda Have you tested it much?

I'm hoping the developers don't run out of steam.

Kusalananda

I've run it for a couple of months. It does a good job I think.

Faheem Mitha

@Kusalananda Have you tried running it to a remote location?

I have a basic but incomplete script.

Kusalananda

No. I'm using a separate drive. It should work over SSH though.

I used to use duplicity for remote backups, because it did'nt require any server or daemon running on the remote host (other than SSH).

Faheem Mitha

Someone has a bit of a discussion here.

linuxquestions.org/questions/linux-newbie-8/…

About the sparse thing.

@Kusalananda Why did you switch to borg?

And did you just run across it online, or did someone point it out to you?

I actually found attic first, and thought it would be a good choice if it was maintained.

Shortly afterwards, I came across borg, and thought to myself, Bingo!

I wonder if borg stores sparse files sparsely.

Kusalananda

@FaheemMitha Someone at work said it was working well, and I decided to try it out. I don't take backups too seriously. The really important stuff is backed up to a AirPort from my Mac.

Faheem Mitha

18:16

@Kusalananda I use Mercurial for the important stuff. It gets pushed off the disk.

github.com/borgbackup/borg/issues/14

Though the discussion is a bit over my head.

Kusalananda

@FaheemMitha It probably do if you use --sparse. It does compression, if you tell it to, and that would effectively compress any holes.

Faheem Mitha

At any rate, it looks like borg does not currently store sparse files intelligently. If I understand it correctly.

Kusalananda

@FaheemMitha I was more thinking about photos and stuff.

Faheem Mitha

@Kusalananda Is --sparse both for archive creation and retrieval?

Kusalananda

Ah. Good point. It is only for retrieval.

Faheem Mitha

18:19

Yes, I'm only seeing it as an option to borg extract.

And that GH issue suggests that Thomas wants to get it working for storage too.

Which would be good, of course.

Kusalananda

Well, if you compress the chunks (--compression lz4 for example), holes would compress nicely.

Faheem Mitha

> Update: I put a bounty on this. It is for implementing sparse file support at archive creation time (seeking over holes [not reading them as zeros], storing which parts of a file are holes / contain data) and reproducing sparse files hole/data layout precisely at archive extraction time.

@Kusalananda Sure, borg can use compression, but it's about not storing the zeros in the sparse files at all, I think.

Like the bit I quoted.

Kusalananda

So, reproducing the holes that were there originally, instead of creating the holes at extraction if asked.

I see.

I wonder if that would be important for some applications, that the holes are in specific places?

Faheem Mitha

@Kusalananda I hope not.

And I also hope Thomas doesn't lose interest in Borg. It's unlikely to survive his departure.

derobert

19:28

@Kusalananda if there is an app that cares, it's abusing the Unix API

Sparse files are an implementation detail. Logically, the file contains 0s

Kusalananda

@derobert Yes, that's why I don't quite understand the point of the quote that Faheem posted.

derobert

And actually, detecting the holes in sparse files, at least by POSIX I believe requires reading the whole file and looking for sections of 0s. Linux has a few other ways (e.g., FIEMAP)

@Kusalananda Ticket seems pointless to me. But I guess someone wants the backup to be exactly the same... I suppose if you're paranoid enough that an app might care...

OTOH, there are some files where having holes are a problem. E.g., you can't swapon a swap file which has holes.

Faheem Mitha

@derobert The backup would be smaller.

derobert

@FaheemMitha Not much smaller than just using gzip -1

Faheem Mitha

It's not about it being exactly the same.

derobert

19:33

... or any other compressor

(and that's presuming nothing else compresses—which is unlikely)

Faheem Mitha

@derobert You mean because gzip would effectively get rid of all those zeros?

derobert

Yep

anthony@Zia:/tmp$ truncate -s 128m big
anthony@Zia:/tmp$ gzip -v --fast big
big:     99.6% -- replaced with big.gz
anthony@Zia:/tmp$ ls -lh big.gz
-rw-r--r-- 1 anthony anthony 572K Aug 29 15:34 big.gz

Faheem Mitha

But people don't necessarily want to run a compression algorithm over big binary files. Like torrent downloads.

derobert

There are some compression algorithms that are really fast. Even run-length encoding (which is basically free) would take care of blocks of 0s.

Faheem Mitha

@derobert I guess so. But I suppose they just want to optimize the basic storage as much as possible. Though of course I can't speak for the devs.

You could comment on the ticket. Maybe there is something else we haven't thought of.

And I'm not sure that it's easy to use borg with any compression algorithm.

derobert

19:38

oh... borgbackup is deduplicated. That should also take care of the chunks of 0s

And it should be possible to compress—each chunk should be compressible

Kusalananda

@FaheemMitha I mentioned that I'm using compression with borg. It's easy. It's just a matter of using --compression followed by the compression algorithm (lz4, gzip or xz)

derobert

@FaheemMitha looking at the docs, if you have deduplication on, big sparse files shouldn't cost more than (approximately) the chunk size. And if you additionally have compression on, they're probably costing under a few kilobytes

And if you're not running with compression, lz4 should be fast. Unless its entirely precompressed files, probably faster than running without due to saved disk write.

@FaheemMitha I'm surprised you have that many sparse files... unless those are half-downloaded torrents.

@FaheemMitha also, to re-sparse-ify a file fallocate --dig-holes ought to work...

Faheem Mitha

@Kusalananda No, I meant an arbitrary compression algorithm.

@derobert I'm surprised too.

@derobert I don't know which ones are sparse.

derobert

19:58

@FaheemMitha Well, it should be safe to run on any file. Should give you essentially the same as cp --sparse=always, except w/o having to copy

Though someone on superuser claimed it didn't do as well. No idea if I should believe them, it's superuser after all...

Faheem Mitha

@derobert I see.

Kusalananda

21:35

If I edit a question, the edit is immediate. If I approve a pending edit, it requires additional approvals. Isn't that weird?

derobert

@Kusalananda yep. And if you improve a pending edit, then it applies immediately

I guess it's supposed to be because they figure you take less care reviewing edits than making them yourself, or that you are more likely to miss something.

Kusalananda

... or something like that. Yes maybe.

Eliah Kagan

18

A: Why do we "Approve" a suggested edit when we can "Force Edit" it?

If you have full editing privileges, then you have full editing privileges. It should not, therefore, be surprising that you have privileges to make edits to posts. There are, of course, ways that you can abuse this privilege, but the idea is that the system trusts you not to abuse it because yo...

Kusalananda

Well, ATM, any edit on a question is effectively blocked until the pending edit is approved.

This one: unix.stackexchange.com/questions/389177/…

derobert

@Kusalananda There is a limit of 1 pending edit. If you want to edit it, you can instant-approve or instant-reject by hitting the relevant button

("improve edit" or "reject and edit")

Though I just ruined your example by voting to approve it :-P

Kusalananda

21:42

@derobert I'm not in the review queues at the moment, I'm looking at the question, and I want to edit it...

Cheers, and thanks, and good.

derobert

@Kusalananda you should get that if you hit the edit link

Though maybe if you've already voted to approve you don't

anyway, gotta run

Kusalananda

It possibly did, but hit "approve" instead. When I clicked again, there was no option to approve or edit.

derobert

That makes sense. Once you've voted, it probably doesn't let you change your mind.

Kusalananda

Ah, ok.

Transcript for