« first day (5022 days earlier)      last day (12 days later) » 

Ramhound
Ramhound
3:52 AM
Well, that was a stupid solution to that problem. I disabled system protection and the "invalid memory location" problem went away
 
Journeyman Geek
Journeyman Geek
4:07 AM
I love it when I make a completely wild guess about an obscure issue based on "some thing I was messing about on" in the comments, and I'm right :D
@Ramhound ....
(though, I never even turn that on generally)
 
Ramhound
Ramhound
Wait, did you suggest that I turn off recovery points (system protection)?
I just wish the error message was more than "invalid memory location", I mean what in the heck, does that even mean?
What is even stranger is that when I converted my physical machine into a VM, I also had a problem, but I ran a third-party application and shrunk the partition. Once I did that, it gave an error about not being able to do it due to the available size. Which also was solved by the same solution.
 
Journeyman Geek
Journeyman Geek
I think there's a memory protection thing
but no, I don't turn on recovery points :D
cause I'm a madlad, and generally have my systems set up so I can nuke em with no real concequence
 
Ramhound
Ramhound
4:25 AM
Yeah, memory protection has more to do with kernel memory.
Honestly that was going to be my next thing to enable then disable
It was either my security software, Acronis, or the fact I disabled a bunch of services to enable a game mode using Killer Intelligent Center 3 years ago
In that case, I enabled it, realized what it did, and tried to disable it, but my system decided to freeze in the process.
Only current or former schnauzer owners can appreciate this t-shirt: gifteclub.com/products/schnauzer-aop-hawaii-shirt
I can't wait for Schnauzerfest 2024!
 
 
3 hours later…
Journeyman Geek
Journeyman Geek
7:59 AM
rofl
@Ramhound currently my family is divided into 'team westie' 'team snauzer' and 'I'd love a dog but I'm too old'
 
 
4 hours later…
Ramhound
Ramhound
11:43 AM
Team Schnauzer 4 Life :-)
 
Journeyman Geek
Journeyman Geek
lol
 
Ramhound
Ramhound
It’s like a gang but only cooler because well Schnauzers…
 
Journeyman Geek
Journeyman Geek
@Ramhound there's two downstairs, one's a bit antisocial, her sister demands pets :D
I mean, clearly the solution is one of each :D
 
Ramhound
Ramhound
I had to get another one recently. The bark, he has so many different barks, and the older one demands that the “stranger danger” bark is his job, feel bad for the little one at times because he doesn’t know why they are barking sometimes.
 
Journeyman Geek
Journeyman Geek
11:58 AM
lol
I think Ash was mostly "PEOPLE! HIIII"
We didn't have/need a working doorbell for years cause he'd enthusiastically greet anyone at the door, even when he was at the other end of the apartment
 
 
3 hours later…
OakBot
OakBot
2:31 PM
Ah, the never-ending "dog discussion". Though, I must interject: having to decipher the meaning behind different barks seems like an exercise in futility. Give me a silent hard drive whirring any day; at least when it's making strange noises, it's a sign to replace it, not decode it.
 
Journeyman Geek
Journeyman Geek
barks @OakBot
 
OakBot
OakBot
@JourneymanGeek Don't start with me, @JourneymanGeek. Unless you're encoding your Wi-Fi password in those barks, I'm not interested.
 
 
2 hours later…
Marko Bonaci
Marko Bonaci
4:34 PM
@Bob reporting for duty :) Have some time now to help with my memory issue?
 
Bob
Bob
@MarkoBonaci sure
wow, I haven't heard that ping bonk sound in a long while
let's start with what you're seeing - what exactly is the problem?
 
Marko Bonaci
Marko Bonaci
@Bob first, let me thank you in advance for this.
@Bob I've just posted a comment there, copying here:
I've never experienced this type of issue. I'm on Win11 and the memory slowly over 2-3 days after a restart grows up to 64GB and then I start getting visible issues, like programs and mouse pointer freezing... The worst thing is that I don't see that in "Processes" in Task manager/Resource monitor. I summed all processes and landed on approx. 20GB while the "Performance" tab shows 60GB, for example. I had no luck using RamMap to identify the offending process/program.
 
OakBot
OakBot
@MarkoBonaci Hi on Win11, I'm Oak!
 
Bob
Bob
ugh
ignore the bot
@MarkoBonaci could you share your rammap results? optimally a file => save, then upload that somewhere I can grab it. otherwise, we can start with a screenshot of the "use counts" tab
(keep in mind that a rammap dump does have some open filenames/paths in it, I think)
 
Marko Bonaci
Marko Bonaci
Here's a public album with a few images: photos.app.goo.gl/nBGLhAGiN2wZYNc16
@Bob forgot to mention you ^^^
As if some program is not releasing memory and thus "Unused" category is huge. Page table is also substantial.
But when I look at individual processes, the highest is WSL (Linux Ubuntu), which is basically my whole dev env, with frontend and backend codebases loaded, so it's not unusual that vmmemWSL is taking 3GB.
 
Bob
Bob
4:51 PM
hmmm
@MarkoBonaci Could you go to the processes tab in rammap and sort descending by page table?
@MarkoBonaci Also, in task manager details, add the columns Handles and GDI Objects (they're somewhat common 'invisible' leaks)
 
Marko Bonaci
Marko Bonaci
@Bob done
 
Bob
Bob
And try sorting desc by those two
@MarkoBonaci Okay, on the surface it doesn't look like a single large process, which leaves the possibility of a large number of processes. Which brings us back to the zombie process possibility that started this conversation :P
After you check handles and gdi objects, could you also post the results of that FindZombieHandles run, preferably as admin and with the -v arg?
(and yes, that Unused is unusually high, but we'll get to that later. hopefully.)
 
Marko Bonaci
Marko Bonaci
@Bob Added the two TaskMgr images. I'm on zombie handles next. Have to download it again, sec, will notify you when that's up.
 
Bob
Bob
those two look normal, so it's probably not those (un?fortunately)
 
Marko Bonaci
Marko Bonaci
5:07 PM
@Bob I think we're onto something, after all:
`211371 total zombie processes`
but the list below doesn't reflect that total (same behavior as in Task Manager?).
 
Bob
Bob
that is a very high number
 
Marko Bonaci
Marko Bonaci
Yeah, 64GB need to be filled :)
 
Bob
Bob
that * 32kB ~= 6.5 GB, which is about the extra PTEs you have, so that all adds up
 
Marko Bonaci
Marko Bonaci
I uploaded
 
Bob
Bob
now we just need to identify which process is holding all of these, while somehow not being visible... that's a new one to me
@MarkoBonaci could you confirm - the run you did was as admin?
if all else fails, what we can do is fall back to killing them until it stops :D
 
Marko Bonaci
Marko Bonaci
5:12 PM
Yes. I ran first as myself and then got the message so I reran as admin.
When I scroll through processes in RamMap, one process is repeated many times (an ability to collapse the list, with counts would've been lovely): docker.exe with identical 32K. I wish I could count how many times. Let's just say: a lot.
 
Bob
Bob
@MarkoBonaci then almost certainly docker is the process that got leaked, but which process is holding those handles open remains a question
now that I think about it, it could possibly be within WSL
 
Marko Bonaci
Marko Bonaci
docker.exe, conhost.exe, com.docker.cli
those 3 appear roughly the same number of times (hard to say exactly, but approx.)
 
Bob
Bob
@MarkoBonaci Could you try lsof within WSL?
I'm hoping they show up there
 
OakBot
OakBot
@Bob Hi hoping they show up there, I'm Oak!
 
Bob
Bob
...goddammit
I wonder if I can kick the bot
 
Marko Bonaci
Marko Bonaci
5:19 PM
@Bob Sure. Something like:
lsof | grep "node" | wc -l
9990
Once I closed VSCode, there's nothing there, so all seems good on the WSL side.
I limited the memory to 20GB a week ago, precisely for this reason - to see whether the issue is on the Win or WSL side
 
Bob
Bob
@MarkoBonaci my suspicion is that something within WSL is launching Win32 processes and not closing them off properly
which would not count as memory used in WSL
could you run ps -e in WSL and see what's still active?
 
Marko Bonaci
Marko Bonaci
I see. Could be. I can tell you that terminating WSL doesn't free memory.
```
PID TTY TIME CMD
1 ? 00:02:05 systemd
2 ? 00:00:00 init-systemd(Ub
7 ? 00:00:01 init
36 ? 00:00:00 systemd-journal
56 ? 00:00:00 systemd-udevd
68 ? 00:00:00 snapfuse
69 ? 00:00:00 snapfuse
70 ? 00:00:00 snapfuse
75 ? 00:00:00 snapfuse
79 ? 00:00:00 snapfuse
81 ? 00:00:01 snapfuse
87 ? 00:00:00 snapfuse
88 ? 00:00:00 snapfuse
111 ? 00:00:00 systemd-resolve
139 ? 00:00:00 cron
141 ? 00:00:00 dbus-daemon
 
Bob
Bob
@MarkoBonaci how do you terminate it?
(I still prefer/use WSL1 so I honestly don't know how WSL2 handles this)
 
Canadian Luke
Canadian Luke
From Powershell as Admin:

```powershell
wsl.exe --shutdown
```
 
Bob
Bob
oh hey luke
 
Canadian Luke
Canadian Luke
5:26 PM
Sup Bob?
 
Bob
Bob
well, if WSL is definitely stopped and you still see those processes floating around in RAMMap (with the associated high page table usage) - my next suggestion is you close individual applications and check counts every time
which will at least let you identify which process is launching or holding these
ideally FindZombieHandles should've told you straight up, not entirely sure why it doesn't for you
 
Marko Bonaci
Marko Bonaci
If I close everything related to WSL on the Win side (VSCode, Idea, Terminal) the vmmemWSL process shuts itself down after 10 minutes or so.
But I could try `wsl --terminate <distro_name>`
 
Bob
Bob
e.g. I'd expect you to see something like this in FZH
 
Marko Bonaci
Marko Bonaci
```
wsl --list
Windows Subsystem for Linux Distributions:
Ubuntu-22.04 (Default)
docker-desktop
docker-desktop-data
```
Docker desktop uses WSL2 as its engine
 
Bob
Bob
@MarkoBonaci try a --list --verbose?
 
Marko Bonaci
Marko Bonaci
5:31 PM
Yeah, that's what I expected, of course, but this is some ghost stuff here.
I could uninstal docker engine completely and restart to test that assumption
 
Bob
Bob
@MarkoBonaci I would expect this to not occur then, but obviously you expect/need to use docker!
It has to be something that's calling docker that's holding them. It could be some other docker component but removing docker isn't a great test because then the caller would fail just because docker's gone?
 
Marko Bonaci
Marko Bonaci
```
>wsl --list --verbose
NAME STATE VERSION
* Ubuntu-22.04 Running 2
docker-desktop Stopped 2
docker-desktop-data Stopped 2
```
 
Bob
Bob
right, so just the Ubuntu system is still running from the looks of it
so if you do a full wsl --shutdown as luke said, you still have those handles floating around?
 
Marko Bonaci
Marko Bonaci
I see. I assumed that something internal to Docker is creating issues, but you could be right, something from the WSL side might be "pinging" docker.exe. I know that docker doesn't work on the WSL side unless Docker engine is running on the Win side.
Let me try wsl --shutdown
Yes, no serious RAM cleanup after wsl shutdown. I uploaded the image from Task manager which shows the impact.
 
Bob
Bob
@MarkoBonaci Could you confirm those 200k+ zombies are still there?
 
Marko Bonaci
Marko Bonaci
5:41 PM
I see docker and those other two processes in rammap, after refreshing.
I'll run the program again, sec.
Still there:
```
FindZombieHandles.exe -verbose
214695 total zombie processes.
1 total zombie threads.
237 zombies held by \Device\HarddiskVolume3\Windows\System32\Taskmgr.exe(14416)
46 zombies of \Device\HarddiskVolume3\Program Files\Google\Chrome\Application\chrome.exe - process handle count: 46 - thread handle count: 0
36 zombies of \Device\HarddiskVolume3\Windows\System32\conhost.exe - process handle count: 36 - thread handle count: 0
28 zombies of \Device\HarddiskVolume3\Program Files\WSL\wslhost.exe - process handle count: 28 - thread handle count: 0
 
Bob
Bob
This is incredibly odd...
 
Marko Bonaci
Marko Bonaci
I think I've just found a similar issue in docker's GH: github.com/docker/for-win/issues/14027
 
Bob
Bob
@MarkoBonaci That number seems to have gone up by 3k since the last check
@MarkoBonaci Huh. Good find! It does look a lot like yours.
 
Marko Bonaci
Marko Bonaci
Right. At least I'm not alone. Thank you very much. You've been very generous with your time.
 
Bob
Bob
It never even occurred to me that kernel-mode drivers can hold process objects open in the same way
If you'd like, you could try to use poolmon to confirm the same thing for yourself
> What I found was that if the iGPU driver is enabled, it will prevent windows from properly exiting from ANY process, and thus the page table slowly gets filled up and fragmented.
That is ... an incredibly nasty driver bug
 
Marko Bonaci
Marko Bonaci
5:51 PM
I'm going to try to downgrade AMD drivers
 
Bob
Bob
If that is the case, then you only happen to be suffering more from it because docker (and/or vscode) constantly opens/closes a lot of processes
And possibly keeping your computer running for longer periods of time
(all of which should be completely fine, absent ... this bug)
 
Marko Bonaci
Marko Bonaci
Right. This is a new machine and I've never experienced it on my previous one
 
Bob
Bob
@MarkoBonaci If that does fix it, I'd suggest you create a self-answered question so we have this available on SU :) Link it here, I'll upvote it
 
Marko Bonaci
Marko Bonaci
Sure. I'll keep you posted.
 
Bob
Bob
...this seems to be a long-running issue?
8
Q: How to identify a driver causing every exited process to become a zombie, polluting page table and active unused memory?

SopelI have a pretty new Windows 11 Pro installation, it has about a week. I noticed that my RAM usage is constantly going up. It was concerned because procexp readouts did not match the total used RAM value that's reported, with the difference factor of 4 or more. I checked RAMMap and I found that pr...

windows memory windows-11 process kernel
!!!
@MarkoBonaci You might find ^ interesting
 
Marko Bonaci
Marko Bonaci
6:02 PM
@Bob thanks, restarting now to install the older driver, will check that out.
 
Bob
Bob
They seem to think 23.10.2 is a good version. Maybe.
 

« first day (5022 days earlier)      last day (12 days later) »