/dev/chat

Videonauth

15:00

i still try to understand that, had a discussion lately with someone about linux python programming and he has proven this to me that a // in a filepath doesnt matter

Stephen Kitt

@Videonauth yeah that’s POSIX, there’s a Q&A about it here somewhere.

Faheem Mitha

@StephenKitt The point I was trying to make is that if there is plenty of RAM available (in theory) as there is in my machine, what magic would changing a number do? The OS should be using the RAM, not swapping to disk.

Stephen Kitt

@FaheemMitha yes, and in your case since free shows lots of available memory you shouldn’t be thrashing.

vmstat will give you better info though since you need to measure behaviour over time.

Faheem Mitha

@StephenKitt No, I shouldn't. Though it's more swapping than thrashing.

Any idea what might be going on? Would memory leakage account for it?

@StephenKitt I've kept vmstat running before. It clearly shows swapping.

And my swap is on my older disks. I guess I could move it to my SSDs, though I'm not enthusiastic about that. And it shouldn't be swapping in the first place.

Stephen Kitt

@FaheemMitha you’d have to post a fair amount of info...

Faheem Mitha

15:04

@StephenKitt I also probably don't understand the output of vmstat well enough.

@StephenKitt Like what?

Stephen Kitt

I’m on a train so connectivity is likely to be limited.

@FaheemMitha ps details of your memory guzzlers, for a start.

Faheem Mitha

@StephenKitt I'm not sure what you mean.

Stephen Kitt

@FaheemMitha ps -ef showing the largest RSS values

Faheem Mitha

@StephenKitt ps -ef shows a ton of stuff, but it doesn't have an RSS column.

Forgive my ignorance, but how do I sort by RSS values?

Looks like the v flag shows memory.

Videonauth

I think i found the Q&A

110

Q: How does Linux handle multiple consecutive path separators (/home////username///file)?

I'm working on a python script that passes file locations to an scp subprocess. That's all fine, but I'm in a situation where I may end up concatenating a path with a filename such that there's a double '/ in the path. I know that bash doesn't care if you have multiple file separators, but I'm wo...

gilles answer is a very good read

as well as the Q&A by stephane about leading / or //

Rui F Ribeiro

15:23

@FaheemMitha ps aux --sort -rss

Faheem Mitha

@RuiFRibeiro That gives a lot of stuff. Shall I pastebin it? Gist, perhaps?

Rui F Ribeiro

pastebin seems fine

Faheem Mitha

This is the Gist - gist.github.com/5477c8b0d4e56175efd8de6e6d5feea3

malan

Having a shell set to /sbin/nologin shouldn't preclude sftp login, right?

Rui F Ribeiro

@FaheemMitha loads of stuff lol....how many chrome tabs?

Faheem Mitha

15:38

@RuiFRibeiro Not that many, actually.

Rui F Ribeiro

@FaheemMitha Is tor and chrome a good combination?

Faheem Mitha

@RuiFRibeiro Dunno. Probably not. Am I running tor without knowing it?

Rui F Ribeiro

@FaheemMitha tor is running....

Faheem Mitha

@RuiFRibeiro I see. It that bad? And around 15 Chrome tabs right now.

Rui F Ribeiro

@FaheemMitha I prefer to usually have 3 to 6 tabs open at only given time... Also email MUA tend to eat loads of I/O not quite fond of them

Faheem Mitha

15:40

@RuiFRibeiro I've just got Pine open.

Rui F Ribeiro

@FaheemMitha As for having DBs, it depends on how and often you use them....seeing there postgresql

terdon

@FaheemMitha are you using that extension I'd mentioned a while back? The tab suspender?

Faheem Mitha

@RuiFRibeiro Yes, PG isn't doing anything here.

terdon

chrome.google.com/webstore/detail/the-great-suspender/…

Faheem Mitha

@terdon Yes. Well, the Great Suspender currently.

@terdon Yes, that's the one.

Rui F Ribeiro

15:42

@FaheemMitha You seem to be using KDE. I also prefer using a lighter wm, like fluxbox or xfce

Faheem Mitha

@RuiFRibeiro I am indeed. But KDE isn't really doing anything either.

Rui F Ribeiro

@teron That one is nice.

Faheem Mitha

I've used it since 1999. At this point, it's become a matter of habit.

terdon

@RuiFRibeiro It really is.

Rui F Ribeiro

@FaheemMitha Oh, I see.... I have fluxbox in my corporate linux, xfce in my FreeBSD....

terdon

15:43

@FaheemMitha But it is. That's a source of your confusion, I think. Simply having something like KDE open means it's doing things. Many things.

Faheem Mitha

@terdon Like what?

terdon

Like running.

faheem    6273  1.3  1.9 5666108 318012 ?      Sl   Jan03 561:23 /usr/bin/plasmashell --shut-up

Rui F Ribeiro

@terdon and many more processes, yeah

Faheem Mitha

@terdon Yes, but it isn't engaging in any meaningful activity. And not much memory.

Rui F Ribeiro

@FaheemMitha If you do not have a printer, I would also stop cups

Faheem Mitha

15:46

@RuiFRibeiro I do have a printer. And I use CUPS.

terdon

@FaheemMitha Well, it's the second entry in your ps output, using 1.3% of your memory.

Faheem Mitha

@terdon Seriously? That's nuts.

terdon

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
faheem   17312 31.8  2.7 1196452 445460 ?      Sl   20:24  12:29 /opt/google/chrome/chrome
faheem    6273  1.3  1.9 5666108 318012 ?      Sl   Jan03 561:23 /usr/bin/plasmashell --shut-up

Rui F Ribeiro

@FaheemMitha And you probably can do without exim....ssmtp ;)

Faheem Mitha

@terdon Oh, you mean plasmashell, whatever that is.

terdon

15:48

And pretty much everything KDE does is memory intensive since it's so flashy. Or it used to be.

Faheem Mitha

@RuiFRibeiro Huh? I'm using exim.

terdon

@FaheemMitha That's KDE. The main desktop.

Faheem Mitha

@terdon I see.

terdon

plasmashell is the KDE GUI shell.

Rui F Ribeiro

@FaheemMitha Also having permanently an external ntfs mounterd drive is not the best of the ideas

terdon

15:49

But hell, even its terminal uses more resources than it has any business using:

faheem    6321  0.1  0.6 1527712 111868 ?      Sl   Jan03  47:27 /usr/bin/konsole -session 10dbde7765000149813255800000148740010_1546530870_507505

Faheem Mitha

@terdon I see. The question still remains why the machine is trying to swap with 16 GB.

Rui F Ribeiro

@FaheemMitha and if you do not use bluetooth, kill it

terdon

@FaheemMitha Have we actually established that it is indeed swapping?

Faheem Mitha

@RuiFRibeiro I don't. At least, not intentionally. Maybe you're talking about my backup drive?

@terdon I think so. What proof do you require?

Rui F Ribeiro

@FaheemMitha Oui

@FaheemMitha also, not the best of the ideas running at the same time both MySQL or postgresql

@FaheemMitha stick with one of them.

terdon

15:50

3 hours ago, by Faheem Mitha

:48811075
free
              total        used        free      shared  buff/cache   available
Mem:       16333076     3290196     7785260      110624     5257620    12592628
Swap:       8388604     3150676     5237928

I didn't remember.

Faheem Mitha

@terdon It's definitely swapping. I can hear it. And there's a visible pause.

Rui F Ribeiro

@FaheemMitha ModemManager? do you need that?

terdon

But what is the actual issue you have? The machine (the GUI) isn't snappy and responsive?

Faheem Mitha

And the si and so jump when I try to restart a process that has been sitting there.

Rui F Ribeiro

@FaheemMitha mpv lol

Faheem Mitha

15:52

@RuiFRibeiro Yes, interactive things are sluggish.

@RuiFRibeiro ?

terdon

@FaheemMitha Then change your swapiness and reboot.

Faheem Mitha

@terdon I think we had this conversation already.

Rui F Ribeiro

I have all my setting for UK and Mr google, youtube on twitter either give me portuguese suggestions or portuguese friends....I might consider running a permanent VPN to the uk

depressing

@FaheemMitha ModemManager

have to leave chat for a while.

enjoy

Faheem Mitha

@RuiFRibeiro ?

terdon

@FaheemMitha I know, but you've yet to do it.

Faheem Mitha

15:57

@terdon I'll consider it as a last resort. But I consider that voodoo computing.

terdon

@FaheemMitha You shouldn't. I don't understand why you're resisting this. It only takes a couple of minutes to test and is trivial to undo.

Faheem Mitha

I like to have reason on my side when I do something. Rather than doing random things in the hope it will help.

terdon

1 hour ago, by terdon

@StephenKitt So then it does make sense to lower the value for a home machine, doesn't it?

1 hour ago, by Stephen Kitt

@terdon it’s hard to come up with a general rule but yes, it’s worth trying, but it won’t perform miracles (e.g. when your working set is larger than RAM)

Faheem Mitha

@terdon I guess I'm just ornery.

I didn't say I wouldn't do it. I just haven't yet.

terdon

@FaheemMitha No, you're being irrational. This is the reasonable thing to do, there's no voodoo about it, it's very logical: you change the setting that controls how your machine uses swap.

Nothing random about it.

Faheem Mitha

16:01

@terdon I'm not being illogical. If there is plenty of RAM, that setting shouldn't make a difference.

Machines don't just swap for fun, they swap if they need to.

terdon

@FaheemMitha What are you talking about? That setting is precisely what defines what will be swapped.

Faheem Mitha

@terdon I don't think that is what that setting means.

Rui F Ribeiro

@FaheemMitha We apply that setting for some customers in our corporate cloud....

terdon

The question is what is being swapped and, in your case, you are swapping out things that make the GUI slower and this is what you want to avoid. Well, changing that setting will alter precisely that behavior.

@FaheemMitha Then I can only suggest you read the transcript (especially Stephen's explanation) again. As well as this:

125

A: Why is swappiness set to 60 by default?

Since kernel 2.6.28, Linux uses a Split Least Recently Used (LRU) page replacement strategy. Pages with a filesystem source, such as program text or shared libraries belong to the file cache. Pages without filesystem backing are called anonymous pages, and consist of runtime data such as the stac...

Faheem Mitha

@terdon Ok.

Another option would be to just turn off swap. If there is plenty of RAM, it shouldn't be a problem.

At least for testing purposes.

Videonauth

16:13

@FaheemMitha this can however cause undefined behavior of some programs which might rely on swap space, i would set it maybe then down to 512 MB which is the bare minimum afaik

terdon

@FaheemMitha I would strongly urge you not to do this.

Having swap can save you from a completely unresponsive machine and give you the time you need to kill an misbehaving application.

More importantly, the approach seems conceptually wrong: instead of scratching your elbow to relieve an itch, you remove the arm.

@StephenKitt if that AU answer was nonsense, then my AU answer here is also nonsense, right?

114

A: How can I turn off swap permanently?

May I suggest a safer approach? You never know when swap can save you from crashing. While swapping does indeed slow down your computer, if you use a program that eats up all your RAM, having the ability to swap can save you from a hard reboot. When the computer starts swapping, you'll notice and...

I'd appreciate it if you could give me some pointers or just edit it directly to make it less nonsense.

I had always thought the vm.swappiness value was answering the question "how often should I swap" not "which pages should I swap first".

Faheem Mitha

16:37

@terdon Ok, I won't do that. Though I'll consider moving swap to a faster disk. Possibly my SSDs. Currently the sounds my computer makes are kind of painful.

Reading about swappiness to try to understand what it actually does. It defies logic to think it tells the machine to swap when there is plenty of RAM available. But, OTOH, I don't actually know what it actually does, either.

terdon

@FaheemMitha That sounds like a good idea if you have that option. SSDs should be comparable to RAM in terms of speed, I think.

@FaheemMitha Read that answer I pasted just above.

Faheem Mitha

@terdon I don't think so. But I don't know how they compare, either.

terdon

> The vm.swappiness option is a modifier that changes the balance between swapping out file cache pages in favour of anonymous pages. The file cache is given an arbitrary priority value of 200 from which vm.swappiness modifier is deducted (file_prio=200-vm.swappiness). Anonymous pages, by default, start out with 60
(anon_prio=vm.swappiness). This means that, by default, the priority weights stand moderately in favour of anonymous pages (anon_prio=60, file_prio=200-60=140). The behaviour is defined in mm/vmscan.c in the kernel source tree.

@FaheemMitha Neither do I, fair enough. Maybe not the same, but certainly orders of magnitude faster than a regular HDD.

Faheem Mitha

@terdon Faster, certainly.

@terdon I'm not clear what either file cache pages or anonymous pages are.

I assume the former is saving file content in RAM?

terdon

This claims that SSDs range from 200 to 550MB/s while HDDs go from 50-120MB/s

DDR3 seems to go from 6400MB/s to 12800MB/s, so you're quite right. SSDs don't even come close.

Faheem Mitha

16:43

@terdon Faster, but not really orders of magnitude faster. Barely one order of magnitude at most.

terdon

Indeed.

I knew not of what I spoke.

Faheem Mitha

If there were comparable, then nobody would buy RAM.

I found an WP entry for page cache - en.wikipedia.org/wiki/Page_cache

This looks relevant - stackoverflow.com/questions/13024087/…

It sounds like the significant difference is that anonymous pages are anonymous in the sense that they don't correspond to a file.

Though I'm not sure why that is significant.

I think there is room for a "explain like i'm five" article about this swappiness thingy.

terdon

17:15

@FaheemMitha I think so, yes

They don't correspond to anything on the file system.

Which, presumably, is exactly what the stuff that GUI programs store in RAM is.

Jesse_b

17:38

I see all the monkeys have been cleared from the starred area but @JeffSchaller fed his about an hour ago: unix.stackexchange.com/a/498156/237982

Stephen Kitt

1

Q: Why does swappiness not work?

We have a RHEL 7 machine, with only 2G of available RAM: free -g total used free shared buff/cache available Mem: 31 28 0 0 1 2 Swap: 15 9 5 so we decided to increase...

linux rhel memory swap free

2

Q: Swap memory, and when the OS starts using it

We are running Red Hat Enterprise Linux version 7, and all our machines are virtual machines. Our memory resources are limited and physical RAM costs money so we are thinking of increasing swap instead of adding memory. Is this a good idea? Secondly, when / from which point does the OS start ...

linux lvm memory swap virtual-memory

derobert

@FaheemMitha @terdon SSDs are orders of magnitude faster than HDD in IOPS, not sustained sequential write.

@StephenKitt S3QL almost got me to run machines with swapoff -a ... but I managed to solve it with a cgroup memory limit instead

Stephen Kitt

@FaheemMitha exactly, anonymous pages aren’t file-backed, so the kernel can’t remove them from memory unless it swaps them out. Non-anonymous pages can be freed by updating their backing store if necessary, and dropping them from memory.

@derobert urgh yes, I can see that causing trouble!

derobert

(Somehow, S3QL convinces the kernel that everything should be swapped out to make room for more disk cache. To the point the machine becomes unusable from thrashing. Solution: hard limit the memory it uses)

Faheem Mitha

@StephenKitt If I understand correctly, that parameter alters the balance between swapping out anonymous pages and file cache pages. So, why does this have any bearing on whether the machine uses swap when RAM is still available?

derobert

17:46

... that was even with swappiness at an absurdly low value, in the single-digit range.

Stephen Kitt

@FaheemMitha it doesn’t, which is my point.

Faheem Mitha

@StephenKitt I don't think so either.

Stephen Kitt

@derobert presumably that’s S3QL cache though, not disk cache, right?

Faheem Mitha

Which brings us back to why my machine is swapping. Thoughts?

derobert

@StephenKitt nope, S3QL doesn't maintain its own cache, let's the kernel handle it

Stephen Kitt

17:47

@FaheemMitha not off the top of my head...

derobert

(well, memory cache — it does have a local disk cache)

Faheem Mitha

@StephenKitt Did you take a look at that output I pasted?

Stephen Kitt

@derobert ah, OK, I’m not familiar with it...

@FaheemMitha yes.

@derobert what’s the cache backed by?

derobert

It's mostly (entirely) just Python code, doesn't even run as root (uses FUSE). So was rather surprising.

@StephenKitt local filesystem, in my case ext4.

Faheem Mitha

@StephenKitt Could it be that the RAM is simply not working properly? I.e. could it be a hardware issue?

Anyway, sleep time.

derobert

17:59

Pretty sure if its detected at all, it's either going to work or you're going to have the normal defective RAM symptoms (e.g., crashes & corruption from bit errors).

Rui F Ribeiro

18:30

I am reading all the time "Stack Overflow & InfoJobs: Bringing Opportunities to Developers in Spam" Is my mind that twisted? lol

Fabby

19:33

@Videonauth I didn't take it: I'm not a dev any more...

But that's the same question they posted last year...

@FaheemMitha As long as that made you grin instead of grimace, I'm fine with that! ;-)

P_Yadav

20:06

I got little confused in this:

Consider executing command HOME=foo, so this command will be executed as a child process in the current shell. So, child process just copies environment variables from parent. Why does value of HOME will change in the original parent shell?

Jesse_b

@P_Yadav It will change in the current shell because that is where you are making the change, it wont change globally unless you export it (I think)

$ (echo $HOME; HOME=foo; echo $HOME); echo $HOME
/Users/jessebutryn
foo
/Users/jessebutryn

P_Yadav

I am not talking about subshells

Just consider a simple command in terminal

HOME=/foo, this will change the value of HOME variable of current shell

But this command was executed as a child process, so how does it is changing parent process's varible

Jesse_b

@P_Yadav Because a child process is not a child shell

P_Yadav

But like in the case of fork(), child process has separate varibles from parent

And in fork() there is no effect of changing the variables.

Jesse_b

All commands executed in a shell execute as a child process of the shell, but they inherit the shell variables of the parent shell and environmental variables of the environment

subshell =/= child process

P_Yadav

20:14

If they inherit, then does they get copy like child process in fork(), where they have separate stack.

But in fork(), there is no effect if we change variables in child.

Jesse_b

That's over my head but it simply has to work that way. If the behavior you are talking about existed when you set foo=bar, foo would become unset immediately after the command ended

P_Yadav

Well I am confused, but will think.

Michael Homer

20:48

@P_Yadav No it wasn't.

Jesse_b

That’s right, Dude, the beauty of this is its simplicity. Once the plan gets too complex everything can go wrong.

Kusalananda

21:08

@P_Yadav HOME=foo is executed in the current environment. If it wasn't, there would be no way for any command to change the current environment, and the only way to change a variable would be through starting subshells or child processes to inherit the variable's new value.

Going with that idea, there would be a new subshell for every assignment in a shell script and executing an external command, for example, would have to include spawning a new instance of the script and somehow continuing at the point after the execution of that command, in order to set e.g. $? to the exit status of that external command. It would be dreadfully complicated and slow.

Transcript for