[Crash-utility] Re: vtop returns stale page table entries due to FILL_PUD caching logic

On Sun, May 24, 2026 at 5:41 AM Anderson Nascimento <anderson(a)allelesecurity.com&gt; wrote: > > Hi Tao Liu, > > Thanks! It makes sense. I thought about making a patch to make users > decide if they want fresh data, but if this is intended, that's okay. > I just wanted to make sure it is intended behavior and not a bug, > because this has been happening very often in my classes, and it > doesn't look good for me when I say something and the tool doesn't > confirm it. Now at least I know exactly how it works and will be > prepared during classes. Yes, this is more like a trade-off. For static crash & dump, say we run crash utility against a vmcore, with its data never changing, the readmem() for one address always returns the same results, so a readmem() cache can accelerate. When it comes to kcore, with its data constantly changing, such a cache can improve the performance, however a outdated info might returned. For our work, static crash & dump, aka the vmcore case is the majority one we encountered. So this is more a trade-off than intended for the kcore case. If you are interested, you can improve the page table walking for crash, so that a readmem() is always used instead of cache when kcore, rather than vmcore, is the case. Ofcourse, like I said previously, if we don't do the fix, open a new crash instance is a simple work around. Thanks, Tao Liu

> > Thank you! > > On Mon, May 18, 2026 at 12:26 AM Tao Liu <ltao(a)redhat.com&gt; wrote: > > > > Hi Anderson, > > > > I think you're right, the cache is more performance intended. Glad to > > know machdep->last_pud_read=1 can work, this will force crash to > > readmem() to load the updated data. Also you can use 2 crash > > instances, say open terminal 1, invoke crash utility against > > /proc/kcore for unpolulated address. Then trigger the page fault > > process to populate the address. Then open terminal 2, invoke another > > crash utility to read the populated address.Then you can make the > > comparison of the 2 outputs, the difference is the results of kernel > > fixing page fault. > > > > Personally I don't think this is a big issue worthy fix for crash. To > > me, always open a new crash instance session is a better solution for > > live kcore debug if you think the data is outdated. > > > > Thanks, > > Tao Liu > > > > On Thu, May 14, 2026 at 6:55 PM Anderson Nascimento > > <anderson(a)allelesecurity.com&gt; wrote: > > > > > > Hello, > > > > > > I have been using the crash tool to teach paging. It is an excellent > > > tool for simplifying page table walks for students. However, I have > > > encountered a persistent issue regarding stale data when inspecting > > > mappings that are populated mid-session. > > > > > > When using the vtop command on a mapping that is not yet populated, > > > and then running it again after a memory operation has occurred, vtop > > > continues to return NULL or stale entries. This happens because the > > > FILL_PUD (and similar) macros check if the current PUD address matches > > > the last_pud_read address. If they match, the tool skips the readmem() > > > call, even if the underlying physical memory has changed. > > > > > > In the debugging session below, I demonstrate that rd -p showed the > > > populated PUD entry, but vtop still reported 0. I was able to resolve > > > this by manually forcing a re-read in GDB by resetting the cache > > > variable: > > > > > > (gdb) set machdep->last_pud_read=1 > > > > > > 992 #define IS_LAST_PUD_READ(pud) ((ulong)(pud) == machdep->last_pud_read) > > > ... > > > 1001 #define FILL_PUD(PUD, TYPE, SIZE) > > > \ > > > 1002 if (!IS_LAST_PUD_READ(PUD)) { > > > \ > > > 1003 readmem((ulonglong)((ulong)(PUD)), TYPE, > > > machdep->pud, \ > > > 1004 SIZE, "pud page", FAULT_ON_ERROR); > > > \ > > > 1005 machdep->last_pud_read = (ulong)(PUD); > > > \ > > > 1006 } > > > > > > Steps to Reproduce: > > > > > > 1) Run vtop on an unpopulated user address. > > > > > > 2) Trigger a page fault/memory access in the target process to > > > populate the entry. > > > > > > 3) Run vtop again; it will still show "(not mapped)" despite the > > > physical memory being updated. > > > > > > Is this caching behavior intended for performance, or should a way to > > > invalidate this cache for live sessions be implemented? > > > > > > crash> vtop -c 5084 0x41414000 > > > [Detaching after fork from child process 5085] > > > VIRTUAL PHYSICAL > > > 41414000 (not mapped) > > > > > > PGD: 6e80000 => 7fc8067 > > > PUD: 7fc8008 => 0 > > > > > > VMA START END FLAGS FILE > > > ffff88801ec382b8 41414000 41415000 8100073 > > > > > > crash> rd -p 7fc8008 > > > [Detaching after fork from child process 5086] > > > 7fc8008: 0000000000000000 ........ > > > crash> vtop -c 5084 0x41414000 > > > [Detaching after fork from child process 5087] > > > VIRTUAL PHYSICAL > > > 41414000 (not mapped) > > > > > > PGD: 6e80000 => 7fc8067 > > > PUD: 7fc8008 => 0 > > > > > > VMA START END FLAGS FILE > > > ffff88801ec382b8 41414000 41415000 8100073 > > > > > > crash> rd -p 7fc8008 > > > [Detaching after fork from child process 5088] > > > 7fc8008: 000000000e576067 g`W..... > > > crash> > > > Thread 1 "crash" received signal SIGINT, Interrupt. > > > 0x00007ffff629d141 in pselect () from /lib64/libc.so.6 > > > => 0x00007ffff629d141 <pselect+193>: 48 3d 00 f0 ff ff cmp > > > $0xfffffffffffff000,%rax > > > (gdb) en 2 > > > (gdb) c > > > Continuing. > > > vtop -c 5084 0x41414000 > > > [Detaching after fork from child process 5089] > > > VIRTUAL PHYSICAL > > > > > > Thread 1 "crash" hit Breakpoint 2, x86_64_pud_offset > > > (pgd_pte=<optimized out>, vaddr=1094795264, verbose=0, IS_XEN=0) at > > > x86_64.c:1970 > > > 1970 FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE()); > > > => 0x00005555557f5da5 <x86_64_pud_offset+85>: 48 8b 96 40 01 00 00 mov > > > 0x140(%rsi),%rdx > > > 0x00005555557f5dac <x86_64_pud_offset+92>: 48 39 9e 20 01 00 00 cmp > > > %rbx,0x120(%rsi) > > > 0x00005555557f5db3 <x86_64_pud_offset+99>: 74 32 je > > > 0x5555557f5de7 <x86_64_pud_offset+151> > > > 0x00005555557f5db5 <x86_64_pud_offset+101>: 8b 4e 18 mov 0x18(%rsi),%ecx > > > 0x00005555557f5db8 <x86_64_pud_offset+104>: 41 b9 01 00 00 00 mov > > > $0x1,%r9d > > > 0x00005555557f5dbe <x86_64_pud_offset+110>: be 04 00 00 00 mov $0x4,%esi > > > 0x00005555557f5dc3 <x86_64_pud_offset+115>: 48 89 df mov %rbx,%rdi > > > 0x00005555557f5dc6 <x86_64_pud_offset+118>: 4c 8d 05 6c a8 58 00 > > > lea 0x58a86c(%rip),%r8 # 0x555555d80639 > > > 0x00005555557f5dcd <x86_64_pud_offset+125>: e8 8e e2 f5 ff callq > > > 0x555555754060 <readmem> > > > 0x00005555557f5dd2 <x86_64_pud_offset+130>: 48 8b 35 87 8e a9 00 > > > mov 0xa98e87(%rip),%rsi # 0x55555628ec60 <machdep> > > > 0x00005555557f5dd9 <x86_64_pud_offset+137>: 48 89 9e 20 01 00 00 > > > mov %rbx,0x120(%rsi) > > > 0x00005555557f5de0 <x86_64_pud_offset+144>: 48 8b 96 40 01 00 00 > > > mov 0x140(%rsi),%rdx > > > (gdb) set machdep->last_pud_read=1 <- This forces the PUD to be re-read > > > (gdb) dis > > > (gdb) c > > > Continuing. > > > 41414000 f67e000 > > > > > > PGD: 6e80000 => 7fc8067 > > > PUD: 7fc8008 => e576067 > > > PMD: e576050 => aa1c067 > > > PTE: aa1c0a0 => 800000000f67e867 > > > PAGE: f67e000 > > > > > > PTE PHYSICAL FLAGS > > > 800000000f67e867 f67e000 (PRESENT|RW|USER|ACCESSED|DIRTY|NX) > > > > > > VMA START END FLAGS FILE > > > ffff88801ec382b8 41414000 41415000 8100073 > > > > > > PAGE PHYSICAL MAPPING INDEX CNT FLAGS > > > ffffea00003d9f80 f67e000 ffff8880142d40c1 41414 1 fffffc0040028 > > > uptodate,lru,swapbacked > > > crash> > > > > > > Best regards, > > > -- > > > Anderson Nascimento > > > Allele Security Intelligence > > > https://www.allelesecurity.com > > > -- > > > Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io > > > To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io > > > https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/ > > > Contribution Guidelines: https://github.com/crash-utility/crash/wiki > > > > > > > > -- > Anderson Nascimento > Allele Security Intelligence > https://www.allelesecurity.com >