First comment: the display of "kernel memory - Paged", almost 22 GB here, is the virtual size of the paged pool.
The "pools" in Windows are kernel-space heap storage. They're used by kernel drivers and Windows kernel mode code in much the same way as heap is used by user mode programs, but they...
A few days ago i tried to set system managed pagefile but there were no much difference, i mean pagefile was used about 5 - 8 GB and the memory was almost full
A thing which i did not mentioned is that this is a virtual server on the proxmox ve, may be that one of his drivers works like that ?
A VM? Ok, it's possible that you're looking at a "balloon driver". Such drivers runs in the guest to find unused virtual RAM and then allow it to be freed back to the host. But here it's not being freed back to the host because it's not being saved in the pagefile. If that's because of a fault in the "balloon driver" then that driver is broken as designed and should be disabled; or if that's not possible, the VM environment should be scrapped and a better-written one employed.
Or, it might simply be because of a pagefile problem. Did you try my pagefile steps? Also, this answer of mine from long ago may prove helpful: superuser.com/a/786313/348119
I don't understand your wording. Are you saying it's not enabled now? then it's not the problem. If it is enabled and there's an option to disable the "ballon driver", then yes, try disabling it. However then you will be in the situation that your VM could be sitting on many GBs of RAM that it needed at one time, but needs no longer, and can't release to the host OS (because the host OS has no way of knowing what would be safe for it to snatch away and make available to the host system). So this would be more of a diagnostic step than a solution.
I would also ask on forums specific to Proxmox VE if others have had this issue with Server 2008 R2 as their guest OS. But.. Have you tried the things I've suggested re the pagefile? If not, is there some reason for not trying them?
if you mean this steps: • set the pagefile to disabled • shut down and reboot • be sure the old pagefile is gone. If not, delete it. • set the pagefile sizes to something reasonable • if necessary, shut down and reboot ; Yes i tried this. No changes.
Ok. Then set the pagefile (for all drives) to "no pagefile", shut down, reboot. Then go look at superuser.com/a/786313/348119 , set the registry value as indicated there, and reboot again.
George21 - With experts around I may be giving naive ideas. Proxmox forum talks about similar issues. BTW did you try "Empty modified list" from RAMMap (and refresh)? What is the behavior? Jamie is whom I already said he is and Magicandre is also a long time player and moderator of sysinternals. There is always something to learn from them...
The only way to empty the modified list is to write its contents to backing store - which for paged pool is the pagefile. We already know the system isn't writing to the pagefile. So I doubt that that button can work.
@George21 Try deleting the pagefile (set to "no pagefile", reboot, be sure it's gone) and then create one with min AND max size of 30 GB. This won't solve the "ballooning" problem but maybe it will allow those pages to be released.
@Sreejith.D.Menon, I can confirm that Empty modified list "from RAMMap works, it releases 10% - 20% from used memory. But i cant do this manually every time .. you understand..
@JamieHanrahan, sorry i was busy yesterday with over work, i will try this today.
@JamieHanrahan Sure I just wanted to see how persistent this part is. Not as a solution but just to go further and see what pool gets released and what doesn't. Combining what you as well as magicandre said about pools to see if there are specific pools that get "locked" for some reason. Also to see if this is blocked due to "storage stack". Promox promises capability of using SAN etc and seems to be for even storing "system-content". I just didn't want to suggest all this in one go.
@George21 - As I mentioned above, I wasn't giving you a solution. I just wanted to note the behavior. Purely academic interests and if possible another vector to narrow down on the RC, as commented above.
@JamieHanrahan - all that said I completely understand that if a pool was used ....released and soon claimed back then it won't be in modified and soon will come back to Active... yet just wanted to see if the Memory manager is indeed capable of "acting upon" vs "updating stats". I know this sounds as unbelievable as some of the issues presented in promox - ZFS causing issues because Linux claims memory from guest! .. :)
Based on pseudo algo as in Filesys internals book, Mm marks the pte as "not modified" optimistically then goes ahead for freeing but all this happens while pfndb is locked. This is if Mm reached to this point.. but is it possible that it got into a loop of "pagefile not availble" in "MiGatherMappedPages"? @Jamie - Am I going the wrong direction?
@Sreejith.D.Menon I don't know offhand. Keep in mind that the FSInt book pre-dates even Windows 2000; Mm was completely rewritten for Win7/2008R2 - among other things the global pfn lock is gone, replaced with per-page (and per-other-things) locks. So all of those algorithms are different now. docs.google.com/…
@JamieHanrahan, the memory rise up fast if i empy it rises up in 4-5 hours and most importantly, it is already more and if try to empty it will release less. For example: now i have 20GB used, i click on empty it decreases to 17, after 5 hours it is 21 and when i click empty it will decrease to 19 or 20.
@JamieHanrahan, i just made the pagefile min AND max size of 30 GB, i will check the server after a few hours.
btw i configured proxmox ve on two ssd drives in zfs mirrored configuration
@Sreejith.D.Menon, An interesting thing that i observed - the pagefile remains unused(CurrentUsage=0; PeakUsage=0) till the moment when i click in RamMap "Empty Modified Page List" only after that the pagefile starts to "fill"
@JamieHanrahan thanks for the pointers and slides on win 7 vm algo. You are right I shouldn't have kept the archaic algo as ref for the analysis.. I really appreciate the way you started looking into the side where - "Modified list is abnormally full and therefore there is some issue" .. good differential and good learning for us. I have been waiting to see more inputs of yours here. Anyway time to go back to "THE books" on internals.
@George21 - After making changes as Jamie has suggested I see quite a lot of difference. Afaik, the modified list has a certain cap of max size and as per that aspect it growing to this size is kind of "allowed" by OS. but as Jamie pointed out the metrics are unusual and not pragmatic. So when you shared the pool sizes initially and two different screenshots sorted in different order, it was clear that "something" was being "cleared".
Secondly given that the pagefile did show increase, it was obvious that it will get "filled up" and a "trigger" may work. That is why I questioned about "RAMMap - empty list behavior". So that part is confirmed - "few things get cleared when you 'push' it ". It proves an important point though - the pagefile is not "locked" as I thought. It is made available when you "push" it. Memory manager is able to clear something. So something else is holding the memory manager...
Should say It is quite difficult to analyze things without access to the machine... Since you seem to be game for analyzing.. can you take snapshot of pools at different intervals (a few of them we need) using poolmon. Sort for "Paged pool" .... we can at least see "which" pool (tag) is stagnant. Then we can go for next course of action..
@George21 - thanks please share couple of them at different time intervals/days .. but ensure you have cleared the pool every time before you took the snapshot..this is to see what is getting cleared and what isn't
Two more things - Can you also manage the Virtual Machine options. pve.proxmox.com/pve-docs/… , especially the "swapiness". Secondly, does this server run in a container?
@Sreejith.D.Menon i.stack.imgur.com/Zsgb1.png 53 hours of uptime (Pagefile usage - CurrentUsage=9626; PeakUsage=9626) i clicked a couple of times on epmpty before made this screen
@George21 - Before you go for the memory setting (swappiness) can you check two applications running there. a) Try stopping the SoftPerfect file access monitor and see how the memory performs. This would be running some sort of windows service or something you can just try stopping the service(s). (b) If "a" doesn't show you much difference in performance, try stopping Avast services for a while and check how the memory performs. You can use even task manager when you try (a) and (b). I feel one of these may be causing the consistent peak in mem usage.
As for "swappiness" go for "aggressive" setting. I think that was 100 or something. But do this ONLY after you try (a) and (b) above and ONLY when you see that stopping those services didn't help much.
@Sreejith.D.Menon avast service isn't running i uninstalled it a long time ago, i stopped SoftPerfect service.. will keep watch over
@Sreejith.D.Menon THANK YOU !!! seems like the problem in paged pool was in " SoftPerfect File Access Monitor 1.0.2 ", i assume it's a bug in this program... But the pagefile still doesn't "fill" :C i.stack.imgur.com/1s6HZ.png
Great :) Yes there was some heavy parsing going on and that too in kernel mode. Avast may still be running.. I can see a filter driver loaded.. perhaps an unclean uninstall? Just check services and registry..
If you are sure about the "problem services" , that they are completely stopped/decommissioned/disabled and still have issue with pagefile, it is time to try the "swappiness"... Just make it aggressive .
I don't see how the "swappiness" setting in the host OS is going to affect whether or not Windows writes pages on the modified list to the pagefile. If they're on the modified page list then the "page replacement" algorithm in Windows in this case has already decided to remove them from their working set(s). But this may be a "performance tweak" (ugh) that is being done either by the VM guest support code or by something else that is running in the guest system.
I feel, as you said, the performance tweak is doing this... I am not aware of promox internals but apart from its kernel maintaining the "swap space" I think there is something within the guest that works in tandem with the hypervisor to manage dynamic memory. I did find some forum on promox and people talking about "swapiness". I think the "aggressiveness" of swapping is defined by this message-passing between the "tools" in guest and the hypervisor. It is a guess anyway! : )
Discussion on answer by Jamie Hanraha…
Discussion on answer by Jamie Hanraha…