/dev/chat

11:12 AM

In the Stevens book, he defines the following macro, and uses it all over the place.

#define MAXLINE 4096

This is (I think) intended to be the number of characters in a line. I just wonder why it's so large. In what universe are lines 4096 characters? Or is this is Unicode thing, perhaps?

terdon

11:26 AM

@FaheemMitha Why would you make it shorter? Isn't the idea to have it as long as possible?

Faheem Mitha

@terdon Well, lines aren't usually 4096 characters. And "as long as possible" isn't meaningful, because integers don't have an upper bound.

terdon

@FaheemMitha Precisely. So a large upper bound was chosen.

Why would you want to force anyone to use short lines instead of coding to allow as large as is practical?

Faheem Mitha

@terdon In the first place, I just wondered how that number was chosen.

terdon

Presumably because it is i) large and ii) a power of 2.

Faheem Mitha

Is the power of 2 important?

terdon

11:45 AM

Yes. Everything related to computers tends to be in powers of two, right? 16, 32, 64, 128 etc?

Stephen Kitt

Also a common page size, so allocating that much memory makes some sort of sense, anything larger could start causing issues. (Most of the examples are on the stack though so that’s not necessarily all that relevant.)

terdon

Remember that computers use binary, so base 2 arithmetic makes sense and therefore it is practical (presumably even essential in some contexts, this is way beyond my comfort zone) to use powers of 2.

user56834

12:01 PM

Hi guys, I was recommended to ask this question here, rather than as a Question on StackExchange, since it is off-topic. I hope someone might point me in the right direction:

I'm looking for an introductory resource on how the Linux kernel and distros work. Specifically I'd like to understand which problems are being solved by different parts of the kernel, and how it works at a level of functionality, rather than details about which commands to use to in order to manage a system. I don't have any particular goal in mind, I'd just like to understand it at a conceptual/functionality level. …

Faheem Mitha

@user56834 Linux kernel and distributions are different topics. Distributions are a more user level topic. But it would probably help if you could describe your use case, if any.

There are books about the Linux kernel, of course. At different levels.

And there are distribution specific books too.

But those are mostly for people who are actually using that distribution. I doubt they would make for a very interesting read if one isn't.

Jeff Schaller

it's now off-topic, but had accumulated some Q & A's before that:

unix.stackexchange.com/…

Faheem Mitha

@user56834 Most of these are more useful and relevant when you are actually trying to accomplish something. My experience is if you just read something, you've forgotten it the next day, if not before. If you've actually made use of it, you have a fighting chance of actually retaining some of the information.

user56834

12:25 PM

@FaheemMitha I don't have any specific use case. I'm just curious about how the Linux operating system works, in the same way one might be curious about how a biological organism or biological cell works on the inside. For example I'm curious about how exactly Linux manages to boot itself (I suspect it's mostly not different from other OSes there, but I don't know how the others do it), and I'm curious about which functionalities are preformed by the kernel vs the distros.

I'm not curious about "tips & tricks" about how to achieve specific things.

Faheem Mitha

@user56834 That's a fairly large range of stuff. I suppose the threads Jeff pointed to would have relevant books.

And there isn't such a thing as the Linux operating system. Linux is the kernel.

And no, I don't want to have a discussion/argument about it.

The kernel is itself a big topic, especially if you don't already have some familiarity with Unix kernels.

user56834

I'm ok with it being a big topic.

terdon

Hi @user56834 and thanks for coming! For context, this is the original (closed) question:

Jeff Schaller

People do get picky about the words, once they matter. "Linux the kernel" is easily confused with the OS ecosystem around it, since they usually go together. Linux distributions end up with different startup daemons, desktop environments, services, etc.

terdon

0

Q: Overview of the functionality of the linux kernel & distros for someone who is just curious

I don't have any particular goal in mind (e.g. not planning to contribute to the kernel, or build my own distro or anything like that). I'm just curious what the kernel actually does, and what is done by the added distros, how the kernel is booted. I'd like to understand which problems are bein...

Faheem Mitha

12:30 PM

@JeffSchaller A Unix kernel is usually a rather small part of the OS, albeit a very important one.

user56834

I guess my ideal book would be something that progressively zooms in on the kernel, listing first all the modules, and what their purpose is, how they interact, then how they work inside, zooming in more and more.

Faheem Mitha

@user56834 I'd go to a bookshop and browse. My experience of computer books (and I've purchased some) is that you mostly don't end up reading them. With some occasional exceptions. So I'd be cautious about buying them. Use a library instead.

Jeff Schaller

I remember picking up a book, something like "Linux Kernel Internals"? that had a CD with it. Probably still have it somewhere.

user56834

What do you mean by a library in this case?

do you mean, use many books as reference?

Stephen Kitt

@user56834 a place where one can borrow books rather than buy them

user56834

12:33 PM

oh lol, i see.

Jeff Schaller

not to be confused with Stephen's library

terdon

1:01 PM

@user56834 I think both you and Faheem just aged yourselves! :P

By the way, don't expect any book to "list all the modules". There is no such thing. Modules are like drivers in the Windows world. There are loads of them and loads of user-specific ones and whatever. You can get a list of the modules available to be compiled into the kernel, presumably, but it will never be a comprehensive list of all modules.

Faheem Mitha

Libraries still exist.

Buying books just for oneself is actually very wasteful. But a common practice.

Unless you plan to use it regularly. In my experience, that rarely happens.

Stephen Kitt

1:17 PM

@FaheemMitha or if you’re interested in books which libraries no longer carry ;-) (old technical books for example)

of course these days archive.org has scans of many old technical books

Faheem Mitha

@StephenKitt Libraries are more likely to carry older books than newer books.

But if you can't get it in a library, and need or want it, I suppose buying it is the only option.

Still kind of wasteful, though.

Stephen Kitt

@FaheemMitha slightly older books, yes, but for my retro-computing interests I have a bunch of books which libraries no longer carry — in fact some of mine are former library books

Faheem Mitha

@StephenKitt I see. For example?

Like the Apple 2e manual?

Jeff Schaller

Oregon Trail for Dummies

Stephen Kitt

@FaheemMitha comparing my nearest bookshelf and my local library catalog, UNIX Secrets (which is obsolete but I still find interesting) and Programming Perl (which isn’t obsolete but is too old for libraries)

and that’s without considering my DOS books ;-)

Faheem Mitha

1:20 PM

@StephenKitt Why is "Programming Perl" too old for libraries?

Stephen Kitt

or books which libraries don’t carry, like the Intel CPU manuals

@FaheemMitha because it was published in the 20th century

Faheem Mitha

@StephenKitt I sounds like you have a large collection. Perhaps stored inside a handy TARDIS.

@StephenKitt Is that bad?

Stephen Kitt

@FaheemMitha for technical books in libraries, yes

my local libraries carry technical books published between ~2005 and 2-3 years ago, so lots of PHP and Java books :-/

Faheem Mitha

@StephenKitt So they automatically throw older stuff away?

Or sell or give it away, more likely.

Stephen Kitt

@FaheemMitha yes, give it away or sell it (for peanuts)

libraries end up being like bookshops: they have limited shelfspace and have to make the best use of it

some libraries also have larger archives, but they’re not all that common

Faheem Mitha

1:24 PM

A lot of the libraries I've used kept books around for ages. And the technical ones they'd purchased were typically not the best.

terdon

@StephenKitt The Camel Book? People are throwing out the Camel Book!?

Faheem Mitha

Not sure where they got the space.

Stephen Kitt

@terdon libraries are, yes :-(

terdon

But the 4th edition came out in 2012!

(no, I don't know that off the top of my head, I just looked it up)

Faheem Mitha

There's a library in Fort, Bombay, called "The J.N. Petit Institute", which, at the time I was using it, had books from the 1930s. It was like a time machine.

Or maybe even earlier.

Unsurprisingly, it has a Wikipedia page.

J. N. Petit Library

The J. N. Petit Library (officially the J. N. Petit Institute) is a membership library in a Grade II heritage structure in Fort, Mumbai. It was founded in 1898 by a group of Parsi students studying at Elphinstone College. Membership is open to residents of Mumbai.The library is one of the finest examples of Neo-Gothic architecture in Mumbai. In 2014–15, it was restored by a team led by the conservation architect Vikas Dilawari. The restoration project won the Award of Distinction under the UNESCO Asia-Pacific Awards for Cultural Heritage Conservation in 2015. == History == The library traces its...

It was quite a strange experience being there. Though it was bang in the middle of South Bombay with traffic roaring all around, it was like being on a desert island.

They've kept the building in good shape, though. Until recently I used to pass by it quite often, though I don't think I've been inside it since the early 1990s, or possibly even before.

Stephen Kitt

1:47 PM

@terdon I’ve got the 3rd edition from 2000. I suspect that the length of time between editions doesn’t help: by 2010, a 2000 Perl book might have triggered the “obsolete tech” flags in libraries, and then in 2012 noone would be asking for Perl books any more...

The nearest library with a decent technical section only has two Perl books: one published in 2015 which is a comparison of multiple scripting languages, and one published in 2020 which is a history of the web

Andras Deak

4:49 PM

those sound like unusual perl books

Faheem Mitha

@AndrasDeak Isn't that the standard reference?

I don't use Perl, so I'm not sure.

When I first tried to learn C in 1997, I recall having a lot of trouble with it, and wondered why it was made so difficult. In 2021 it all seems quite reasonable to me. I guess that's what they call perspective.

terdon

@FaheemMitha The Camel Book (Programming Perl) is the standard reference. Andras was referring to the two books that Stephen mentioned just above which do indeed not sound like usual perl books since one is a comparison of many languages and the other is a history of the web.

Faheem Mitha

@terdon Ah yes, I wasn't reading carefully. Sorry.

Also, half asleep.

Faheem Mitha

6:36 PM

As a matter of style, should one check every system or standard function call in C and return strerror(errno) if there is an error? I don't recall doing this for C programs I wrote earlier, and it certainly makes for a lot of boilerplate.

ToxicFrog

Most of your functions probably don't have a return signature of const char* in the first place and for the ones that do, writing all your call sites to use string manipulation functions to tell if an error occurred seems like a bad idea.

Also, not all libc functions can set errno in the first place!

But like, as to the question I think you were trying to ask -- yes, if you care at all about whether a call succeeded or not (which is probably not always, but you should definitely care if, e.g., you were able to successfully open a file you need to read/write, or execute a subprocess, or what have you), you should check if the call succeeded using whatever idiom that function employs for error checking

Faheem Mitha

Oh, strerror is a POSIX thing, which I why I don't recall using it earlier.

ToxicFrog

And if it's not a recoverable error, you should probably output an error message (which, for errno-based errors, can be conveniently done with perror or strerror) and then exit

And if it's a recoverable error, but one the user should be made aware of (either because it's something they could reasonably fix for future runs or because it affects the operation of the program in some way), you should at least log it

Faheem Mitha

@ToxicFrog Sorry, I don't follow the reference to functions with return signature of const char *.

ToxicFrog

> and return strerror(errno)

> man 3 strerror: char *strerror(int errnum)

Aah, you're right, it's not const

Faheem Mitha

6:43 PM

For example, should one test for error with malloc and memcpy?

@ToxicFrog Well, I would normally just print the strerror to stderr, and exit or return, I suppose.

ToxicFrog

You should read the man pages for both of those.

Faheem Mitha

@ToxicFrog Fair enough.

ToxicFrog

> I would normally just print the strerror to stderr, and exit or return

That's what I'd generally do, but that's not the same thing as returning the error, which is what you originally said (and in some languages is a common idiom for error handling)

Faheem Mitha

I guess _exit is the right thing to use here. return isn't.

@ToxicFrog I wouldn't attempt to handle the error. Just exit.

I wrote:

> should one check every system or standard function call in C and return strerror(errno) if there is an error.

No handling was implied.

Well, there are certainly a lot of things that can go wrong in even a small program.

ToxicFrog

Yes there are!

Also, is there a particular reason you're preferring _exit() to exit()?

Faheem Mitha

6:48 PM

Supposedly POSIX defines a UNIX standard of sorts, but a lot of it doesn't seem particularly UNIXy. Just enhancements of various sorts which could easily be in the standard library.

@ToxicFrog Well, it makes less of an effort to clean up.

ToxicFrog

Yes; personally I generally want my output buffers flushed and atexit handlers to run.

Faheem Mitha

@ToxicFrog Doesn't it disguise problems, though?

ToxicFrog

> a lot of it doesn't seem particularly UNIXy

So, (a), a lot of the stuff in POSIX has, in the ~35 years since it was written, been adopted by a lot of non-UNIX OSes, to varying degrees of success (windows being a notable exception until very recently)

and (b) POSIX originated as an attempt to standardize APIs and user environements across the at the time very heterogenous UNIX ecosystem, specifically

Faheem Mitha

Really, even pipes seem relatively general. Though I don't know if other OS's use them.

So, when Bash uses a pipe, it's just calling the corresponding C function, right?

ToxicFrog

Probably? I haven't looked at bash's internals ever.

Faheem Mitha

6:53 PM

OK. You were saying? specifically...?

ToxicFrog

And, again, POSIX is almost as old as I am; a lot of stuff that looks "general" now is general because it was specified in POSIX and people copied it for their non-UNIX operating systems to make cross-platform development easier

Faheem Mitha

@ToxicFrog I see. I meant, it doesn't appear to be tied to any very specific model of computing.

I suppose there is a lot of "everything is a file", which I realise isn't exactly a universal interface.

ToxicFrog

Like, pipes "seem very generaly" and indeed as a concept they predate UNIX, but as an operating system feature they were a major innovation in 1970s UNIX

Faheem Mitha

@ToxicFrog I didn't know that.

You didn't finish what you were saying about "heterogenous UNIX ecosystem".

ToxicFrog

What? Yes I did.

Oh, I see the confusion

That wasn't "specifically, X, Y, Z"

Faheem Mitha

6:57 PM

@ToxicFrog Well, you forgot the period. :-)

So "specifically" was the last word there? OK.

ToxicFrog

It was "posix was specifically created to standardize the state of UNIX, not as a standard for OSes more generally; the fact that many non-UNIX OSes have adopted it is a convenient accident".

Faheem Mitha

Does Windows now use pipes, then?

ToxicFrog

Windows has had pipes since at least NT -- possibly earlier, I am by no means proficient with windows -- but the API for using them is completely different from POSIX pipes, and as far as I know there is no equivalent to mkfifo to create a pipe in the filesystem -- named pipes exist but have their own completely isolated namespace separate from files on disk.

Faheem Mitha

@ToxicFrog I see. So not for use at the user level?

ToxicFrog

They're usable in cmd.exe in a manner conceptually similar to bash (e.g. dir | find or so), but the semantics are different in some (to me) surprising ways

(which I do not remember the details of, it's been well over a decade since I used cmd.exe for anything but the most trivial of tasks)

[_exit disguising problems] -- what sorts of problems do you expect to be disguised by this? I could perhaps see it making some problems harder to diagnose if the program has gone sufficiently wrong that an atexit() handler stomps the stack or something...

Faheem Mitha

7:10 PM

@ToxicFrog I just thought one might lose some information. So you think exit is better, then?

ToxicFrog

In general I find that exit() is a better default. There are some circumstances where _exit may be preferable. You need to decide on a case by case basis.

Faheem Mitha

@ToxicFrog OK.

ilkkachu

8:48 PM

@ToxicFrog if you're forking child processes, you might not want to flush everything or process exit handles in both the parent and the child. E.g. something like printf("going to fork now.\n"); int a = fork(); if (a == 0) { printf("now in child\n"); exit(0); } would print the first line twice if the output is redirected to a pipe.

Transcript for