do any of the linux wizards here would be able to help with this difficult issue ? unix.stackexchange.com/questions/674670/…

trying to patch glusterfs/fuse so performance is better with big directories with lots of subfolders/files

I read about stuff like FUSE_MAX_PAGES_PER_REQ but find no information in how to set them at runtime

@Freedo just a suggestion on the phrasing of the question; are you really wanting "bigger buffer for getdents" or: do you want to know why it's slow? or do you want to know how you can make it faster? I'm just afraid of an X/Y question that might not solve your problem.

@JeffSchaller I want to know how can i make fuse faster for big directories with lots of files

@Freedo then that's what your question should ask (IMHO)

starting with the title and the first part of the body; then describe the ideas you have and what you know about it, and ask for what you want

@ilkkachu Is this a response to the comment saying that the code leaked file descriptors?

This is (as usual) probably a dumb question, but the Unix system calls for reading prominently feature buffering. In fact, it seems impossible to read with buffering, though I'm not clear why. But it seems writing does not require buffering. Again, not clear why. Which is my question.

What do you mean by "require buffering" here?

Are you referring to the OS buffer mode (unbuffered, line-buffered, chunk-buffered) or something else?

@ToxicFrog So, there are read and write functions, including POSIX versions, which can read from file streams. They both require buffers. But there is the printf family, which writes to file streams (fprintf version), but does not require a buffer.

But all the read functions I've seen require buffers.

Why isn't there a function where you just read everywhere in there at once?

So, reads require a buffer because the data you're reading in from disk has to be stored in memory somewhere for you to work with it.

Scripting languages have them, though I suppose they are using buffers under the hood.

HLLs like Lua or Python will allocate that buffer for you automatically when you do a read; C requires you, as ever, to manually manage memory.

@ToxicFrog OK, fair enough. But why can't the system just read everything out, creating a large enough buffer?

Or is this just something that C doesn't do?

And why doesn't fprintf require a buffer too?

Your first question is basically "why don't fread() and friends automatically allocate the buffer for you and return a pointer to the allocated buffer"

@ToxicFrog Yes, something like that.

Got to run for a meeting. Back in a few minutes.

That's a philosophical question that properly needs to be addressed to the ANSI C standards committee, but my best stab at answering it is that leaving memory management up to the caller is the more general answer; it means you can easily choose whether to stack-allocate or heap-allocate or mmap() your buffer, you can re-use buffers if you want to it, it'll work even on embedded systems that don't have any equivalent to malloc(), etc

If fread() is meant to allocate the buffer itself -- ok, how does that work on platforms that don't have malloc at all and only support compile-time allocation? Is that really what you want if you're just reading, like, 8 bytes that you can easily stack-allocate much faster? etc

If you want a function that's just "allocate a buffer the size of this entire file and copy the contents of the file into memory", you can quickly write that yourself using fseek/ftell/fread/malloc

But for that, 90% of the time you just want mmap() anyways

For your second question -- fprintf formats and prints the string in one go. It doesn't need you to provide a buffer because, if it needs one, it has its own internal buffer it uses; and not all (f)printf implementations may need a buffer at all.

It doesn't need to keep the formatted string around in memory because it gets written to the file descriptor as soon as possible, and after that it's the OS's job to deal with it.

You will note that the sprintf() family does need a buffer, and some systems (BSD, GNU glibc) provide an "asprintf" (allocating sprintf) variant that automatically allocates a buffer for you (it is of course still up to you to free it when you're done)

The broader answer underpinning all of these questions is that, to work with data, it needs to be located somewhere in memory, and in C, it is usually the responsibility of the programmer, not the language runtime, to make decisions about when and how memory is reserved for use and which regions of memory are used for what tasks; thus, functions that move data to or from program memory require the programmer to tell it where the data is coming from or going to.

@ToxicFrog Yes, something like that. Or read the entire output of a pipe. mmap looks relevant, but I don't yet understand it. I should and read about it again.

Reading the entire output of a pipe is a harder question because pipes aren't seekable, so how do you know what size of buffer you need?

@ToxicFrog Yes, I understand that, but convenience functions for (dumping) all the output of a pipe into a file would be nice.

Lua handles this (in io.read()) by allocating an internal buffer and reading into it until it either runs out of input or buffer space, and expanding the buffer if it runs out of space

@ToxicFrog Right. I was about to say that C could provide a convenience function for that. Assuming line-wise reading, for example.

It would be nice, and this is why a great many languages that aren't C and have completely different philosophies of memory management implement convenient functions to do that :P

@FaheemMitha did you just put "convenience functions" and "C" in the same sentence?

@AndrasDeak Indeed I did. But isn't much of POSIX convenience functions, anyway?

Like, this is why I don't generally use C except for things where "being written in C" is a hard requirement for unrelated reasons

(and a lot of the time the requirement is actually "must be C ABI compatible" and these days that means I use Rust instead)

@ToxicFrog Agreed, that's why don't use much either. But in the current circumstance is appears to be the right place for the job.

Namely, writing code to handle the (somewhat) general call of calling a command. Which apparently Lua cannot be bothered to do, and which doesn't exist in C or POSIX, either.

@ToxicFrog But most languages don't have Rust APIs.

Re mmap() -- basically, it reserves a region of memory addresses large enough to hold the entire file without actually allocating memory; reads from those addresses read the file contents and writes to them write back to the file on disk.

Under the hood pretty much any OS will automatically handle copying chunks of the file into memory and back to disk as needed to improve performance, but you don't need to worry about it.

I'll ask you the same question as I asked @ilkkachu earlier. You seem to know a lot about this. So either you studied it in college and remember it well, or you use it regularly.

So on platforms where it's available it's a very convenient function for "I want to access the entire contents of this file in memory"

Both.

Or maybe there's another option?

(although I started using C before university)

@ToxicFrog That sounds handy. Is that (mmap()) something one could use for reading from a pipe? Instead of looping over read calls?

> But most languages don't have Rust APIs.

Rust can both call and export C-compatible APIs, so you anything that can interoperate with C can also interoperate with Rust, if you write the Rust code with an eye to compatibility.

Or is it not intended for pipes? Which, as you say, can't seek on.

@ToxicFrog Hmm. Does that include Lua?

So, is Rust like a "better" C? I thought that's what D was going for. Though maybe it's not an exclusive ambition.

absolutely, Rust is meant to be C done right from what I've heard

C without the footguns?

I don't actually know what happens if you try to mmap() an fd connected to a pipe, but if nothing else mmap requires you to know up front how large the file is, which you can't do with a pipe.

at the cost of thinking real hard about things like lifetimes

@ToxicFrog Yes, I understand.

@AndrasDeak That would be nice, assuming it's possible without a major performance hit.

> Does that include Lua?

Yes; you can both call lua from rust code and write dynamic libraries in rust that are loadable from lua in the same manner as C libraries

Putting protections in place seems to be very expensive in practice.

@ToxicFrog Good to know. I'll keep that in mind, considering I'm probably stuck with Lua, because LuaTeX.

@FaheemMitha I believe Rust is quite performant

> writing code to handle the (somewhat) general call of calling a command. Which apparently Lua cannot be bothered to do, and which doesn't exist in C or POSIX, either.

In POSIX that exists as fork/exec. You seem to be really fixated on having something like Win32's CreateProcessEx instead, and I'm not sure why

Though I have no immediate plans to learn Rust.

Unless you mean "calling an external command and collecting all of its output conveniently" or something?

@FaheemMitha I see no reason for that assertion. You know, there's a thing called compile time ;)

Like, I'm not sure what your actual, underlying goal is here, it feels like there's a lot of XY problem going on

> Putting protections in place seems to be very expensive in practice.

Rust implements most of its safety features with extremely rigorous compile-time checks. This makes the learning curve a bit steep but you spend a lot more time dealing with compiler errors and a lot less time dealing with segfaults, which is IMO a very worthwhile tradeoff.

don't forget undefined behaviour

@ToxicFrog I want something that writes standard output and standard error from the called command. popen from the Lua Posix module just mixes them together. And there's the separate issue of the command itself failing out, in which case the error message gets lost.

> I thought that's what D was going for. Though maybe it's not an exclusive ambition.

It's definitely not an exclusive ambition, although Rust appears to be the most successful so far

I tried D back in the day but, independent of any of the merits of the actual language, it was undermined by having two different mutually imcompatible implementations, so you had two competing D ecosystems that couldn't share code with each other

I can hack posix.popen, but it's ugly.

I wasn't expecting to spend so much time with C, but I suppose I shouldn't be surprised. And I learned something. Actually, quite a lot. The last time I did any C programming was circa 20 years ago.

@ToxicFrog So you think Rust is a better choice than D? It's from the same person who was behind Monotone, I think. Hoare?

Graydon Hoare. Yes, I think I have the same person.

> the command itself failing out, in which case the error message gets lost.

You should get the contents of stderr regardless of the program's exit status, unless it couldn't start the program at all in which case there is no error message to lose

@ToxicFrog Actually, there is. From exec, that is.

But the fact that stderr and stdout are intermixed is annoying, yes.

exec() doesn't output error messages.

@FaheemMitha anecdata: I've never really heard of D, and Rust is all the rage, so probably go for Rust

On error, it sets errno and returns.

The typical way this seems to be handled is to put bash in the middle, and have that return.

You may be thinking of the shell, which, when it fails to exec something, emits a descriptive error message.

But this is a property of the shell, not exec itself.

@ToxicFrog That's what I thought, but if you pass errno to strerror(), it gives you the error message.

Yes, but it's up to you to do that!

printf(stderr, "read() returned %ld, errno is %d (%s)\n", nbytes, errno, strerror(errno));

exec will not do that for you!

@ToxicFrog Right. So I did that.

Anyways

But popen doesn't give you that.

Is there a particular reason you can't just do the quick and dirty approach of having the external program write one or both output streams to a temp file and then reading that in?

I just hate it when valuable information gets thrown away.

@ToxicFrog I was doing exactly that, yes.

That is definitely going to be a lot more straightforward that reimplementing CreateProcessEx or python's Subprocess library in lua ex nihilo

I just thought I would do it "properly". And I didn't understand the underlying mechanisms at all, so thought it would be an opportunity to learn them.

Aah.

sounds like a terrible rabbit hole

I didn't take these topics at university, nor have I had occasion to learn them since.

@AndrasDeak I mean, subprocess management and IPC is a terrible rabbit hole but it's also potentially useful knowledge to have

It took more time than expected, but I do think I understand things better now.

@AndrasDeak That's true of learning practically anything.

I spent two years learning about databases on my own, because it made a terrible project I was working on manageable.

Classic lemonade making.

@ToxicFrog Yes, IPC is basic basic Unix stuff.

Bear in mind that I have zero formal CS education.

I've actually never taken a course with the word Computer in the title.

@ToxicFrog Anyway, thank you for very much for the discussion and insights.

@ToxicFrog C, not Lua.

I don't like Lua enough to actually want to write in it. C sounds more sensible. I plan to wrap it in Lua.

"as a lua-loadable C library" is what I meant to say there

@ToxicFrog Ah, yes. Yes, that's the idea.

You can't implement it in pure Lua because the necessary APIs aren't exposed, unless you use luajit's FFI or something

Most of the people here seem to know about Unix System Programming.

So I'm guessing it comes up a fair amount, one way or the other.

sounds like cognitive bias

@ToxicFrog I guessed that. And the idea of trying to write it in Lua brought me out in hives, anyway.

@AndrasDeak Pardon?

you're in the unix chatroom, among people who probably partly do system programmy stuff

Personally I quite like lua and would use it in preference to C when I have a choice, although their problem domains are so different that circumstances where "lua or C?" is a sensible question to ask are relatively rare

@AndrasDeak Is that cognitive bias?

If your conclusion is that "it comes up a fair amount", then potentially.

@AndrasDeak Not accidental. I doubt I'd get very far on the net in general (or in real life) trying to get help with Unix system programming.

like going to the photography chatroom and concluding that people tend to have DSLRs :)

@AndrasDeak Within people who do Unix, I mean.

Unix is a big tent. Lots of stuff goes on there.

@FaheemMitha maybe, depends on the meaning of "who do unix". I've been a linux-only (end) user for 10+ years.

And I know nothing about any of these things.

Yeah, apart from a bit of dabbling with BSD in university I don't think I've touched an Actual UNIX since my family migrated off SCO in like 2002

@ToxicFrog Hmm. I can't say I feel much love for Lua, and I feel C is something of a solid ground for this kind of application. Because this is exactly its use case, and it's one of the worlds most widely used programming languages. So people tend to know it.

@AndrasDeak But as you've probably noticed, half the people here do. And they're just the people who happen to be here.

@ToxicFrog The proprietary flavor, you mean?

BTW, thought you were the US, but your profile is CA.

@AndrasDeak I learned quite a lot in a week. Something to build on, if the occasion arises.

I wonder how hard it is to get work writing C.

my mostly-uneducated impression is that C++ is a lot more common for general applications

Also, gcc seems to have improved considerably since I last used it. 10 years ago, gcc compiler messages were just gibberish. Now they actually tell you what is wrong.

@AndrasDeak People use that a lot too, sure. But it's a somewhat different domain. C is more common for system applications, I think.

yup, that seems right

I've mostly used C++ myself. My familarity with C is very low.

@FaheemMitha yeah, we used to run SysV, then SCO OpenServer while my dad worked at SCO, then we migrated from that to Red Hat

@ToxicFrog Anyway, thank you again. You've been very helpful.

Although these days the family server runs NixOS

@ToxicFrog A long UNIX heritage. :-)

(it spent a while being Fedora and then SUSE in there)

@ToxicFrog Aaron Hall would be proud

Isn't SO the people who sued the Linux project?

Also they used to be the people who held the UNIX trademark, or something.

@ToxicFrog For an advanced user, you don't actually have much of a presence here. :-)

> Isn't SO the people who sued the Linux project?

The history is a bit complicated, but tl;dr SCO used to make commercial UNIXes and UNIX development tools, then at some point they were bought by the Caldera Group which kept using the SCO name, and at some point Caldera switched from making software to making litigation and then self-destructed spectacularly

My dad got out before that happened

@ToxicFrog I remember reading about it. Sounds like a sad sequence of events.

There was a lot of coverage of it, back in the day.

@ToxicFrog Sensible.

> For an advanced user, you don't actually have much of a presence here. :-)

tbh I'm mostly connected to chat.se because of The Bridge, since some of my friends hang out there, but I like talking UNIX too, so I'm also in this channel. I'm not very active on the site proper.

@ToxicFrog Right. I guess answering questions for free isn't for everyone. :-)

@AndrasDeak You are not a C or C++ user, then?

I assumed for a scientific user, you'd write in it. Or maybe you meant you don't do system programming.

The other option is Fortran...

@FaheemMitha Nope. I learned C somewhat, I'd still claim I can read and maybe even write it if I really have to. Apart from one program in the group that I don't use, we do fortran. And yeah, absolutely no systems programming.

but most of the actual code I write is python

@AndrasDeak OK.

In some way, languages which enforce strong static type checking are easier to write in, because you've just got to submit to the compiler.

"easier to write in", no

It's a bit like living under a fascist govt, I suppose.

@AndrasDeak Well, it tells you what to do, and slaps you if you don't behave.

The whole point of Python is productivity. You can write "working" code in no time. The price is performance and higher chance of subtle bugs (UB not withstanding).

@AndrasDeak UB?

undefined behaviour

Python is actually pretty good for writing bug-ridden code in.

arguably subtle bugs are also rampant in C due to it

Strong static typechecking is surprisingly good at winkling out entire categories of bugs. It's not a panacea, of course.

yup

Though I can't call myself a fan of strong static typechecking. It's a giant pain.

but at times you end up talking for days about (char *)NULL

@AndrasDeak There's that, yes.

Though one eventually gets past it.

@AndrasDeak If I decided to look at one of those, I guess I'd try Rust first, yet.

But I have no such plans currently.

As far as rabbit holes go, few are worse than trying to learn a brand new language. Especially for no particular reason.

@ToxicFrog Actually, at the risk of sounding like a freak, I quite enjoyed learning a bit of Unix Systems Programming. Stevens has been sitting on my bookshelf for 20 years.

> This has been quite exhausting.

^ from the announcement; hopefully it goes well and doesn't ruin too many people's Halloweens!

just don't lock yourself out of your routers

Ah, the TeX SE election started.

Percusse's answers are interesting.

on SO they removed joke candidates

perhaps they are trying to trap him into becoming a mod