3:58 p.m.
On Sun, May 24, 2026 at 5:41 AM Anderson Nascimento
<anderson(a)allelesecurity.com> wrote:
Hi Tao Liu,
Thanks! It makes sense. I thought about making a patch to make users
decide if they want fresh data, but if this is intended, that's okay.
I just wanted to make sure it is intended behavior and not a bug,
because this has been happening very often in my classes, and it
doesn't look good for me when I say something and the tool doesn't
confirm it. Now at least I know exactly how it works and will be
prepared during classes.
Yes, this is more like a trade-off. For static crash & dump, say we
run crash utility against a vmcore, with its data never changing, the
readmem() for one address always returns the same results, so a
readmem() cache can accelerate. When it comes to kcore, with its data
constantly changing, such a cache can improve the performance, however
a outdated info might returned.
For our work, static crash & dump, aka the vmcore case is the majority
one we encountered. So this is more a trade-off than intended for the
kcore case. If you are interested, you can improve the page table
walking for crash, so that a readmem() is always used instead of cache
when kcore, rather than vmcore, is the case. Ofcourse, like I said
previously, if we don't do the fix, open a new crash instance is a
simple work around.
Thanks,
Tao Liu
Thank you!
On Mon, May 18, 2026 at 12:26 AM Tao Liu <ltao(a)redhat.com> wrote:
>
> Hi Anderson,
>
> I think you're right, the cache is more performance intended. Glad to
> know machdep->last_pud_read=1 can work, this will force crash to
> readmem() to load the updated data. Also you can use 2 crash
> instances, say open terminal 1, invoke crash utility against
> /proc/kcore for unpolulated address. Then trigger the page fault
> process to populate the address. Then open terminal 2, invoke another
> crash utility to read the populated address.Then you can make the
> comparison of the 2 outputs, the difference is the results of kernel
> fixing page fault.
>
> Personally I don't think this is a big issue worthy fix for crash. To
> me, always open a new crash instance session is a better solution for
> live kcore debug if you think the data is outdated.
>
> Thanks,
> Tao Liu
>
> On Thu, May 14, 2026 at 6:55 PM Anderson Nascimento
> <anderson(a)allelesecurity.com> wrote:
> >
> > Hello,
> >
> > I have been using the crash tool to teach paging. It is an excellent
> > tool for simplifying page table walks for students. However, I have
> > encountered a persistent issue regarding stale data when inspecting
> > mappings that are populated mid-session.
> >
> > When using the vtop command on a mapping that is not yet populated,
> > and then running it again after a memory operation has occurred, vtop
> > continues to return NULL or stale entries. This happens because the
> > FILL_PUD (and similar) macros check if the current PUD address matches
> > the last_pud_read address. If they match, the tool skips the readmem()
> > call, even if the underlying physical memory has changed.
> >
> > In the debugging session below, I demonstrate that rd -p showed the
> > populated PUD entry, but vtop still reported 0. I was able to resolve
> > this by manually forcing a re-read in GDB by resetting the cache
> > variable:
> >
> > (gdb) set machdep->last_pud_read=1
> >
> > 992 #define IS_LAST_PUD_READ(pud) ((ulong)(pud) ==
machdep->last_pud_read)
> > ...
> > 1001 #define FILL_PUD(PUD, TYPE, SIZE)
> > \
> > 1002 if (!IS_LAST_PUD_READ(PUD)) {
> > \
> > 1003 readmem((ulonglong)((ulong)(PUD)), TYPE,
> > machdep->pud, \
> > 1004 SIZE, "pud page", FAULT_ON_ERROR);
> > \
> > 1005 machdep->last_pud_read = (ulong)(PUD);
> > \
> > 1006 }
> >
> > Steps to Reproduce:
> >
> > 1) Run vtop on an unpopulated user address.
> >
> > 2) Trigger a page fault/memory access in the target process to
> > populate the entry.
> >
> > 3) Run vtop again; it will still show "(not mapped)" despite the
> > physical memory being updated.
> >
> > Is this caching behavior intended for performance, or should a way to
> > invalidate this cache for live sessions be implemented?
> >
> > crash> vtop -c 5084 0x41414000
> > [Detaching after fork from child process 5085]
> > VIRTUAL PHYSICAL
> > 41414000 (not mapped)
> >
> > PGD: 6e80000 => 7fc8067
> > PUD: 7fc8008 => 0
> >
> > VMA START END FLAGS FILE
> > ffff88801ec382b8 41414000 41415000 8100073
> >
> > crash> rd -p 7fc8008
> > [Detaching after fork from child process 5086]
> > 7fc8008: 0000000000000000 ........
> > crash> vtop -c 5084 0x41414000
> > [Detaching after fork from child process 5087]
> > VIRTUAL PHYSICAL
> > 41414000 (not mapped)
> >
> > PGD: 6e80000 => 7fc8067
> > PUD: 7fc8008 => 0
> >
> > VMA START END FLAGS FILE
> > ffff88801ec382b8 41414000 41415000 8100073
> >
> > crash> rd -p 7fc8008
> > [Detaching after fork from child process 5088]
> > 7fc8008: 000000000e576067 g`W.....
> > crash>
> > Thread 1 "crash" received signal SIGINT, Interrupt.
> > 0x00007ffff629d141 in pselect () from /lib64/libc.so.6
> > => 0x00007ffff629d141 <pselect+193>: 48 3d 00 f0 ff ff cmp
> > $0xfffffffffffff000,%rax
> > (gdb) en 2
> > (gdb) c
> > Continuing.
> > vtop -c 5084 0x41414000
> > [Detaching after fork from child process 5089]
> > VIRTUAL PHYSICAL
> >
> > Thread 1 "crash" hit Breakpoint 2, x86_64_pud_offset
> > (pgd_pte=<optimized out>, vaddr=1094795264, verbose=0, IS_XEN=0) at
> > x86_64.c:1970
> > 1970 FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
> > => 0x00005555557f5da5 <x86_64_pud_offset+85>: 48 8b 96 40 01 00 00
mov
> > 0x140(%rsi),%rdx
> > 0x00005555557f5dac <x86_64_pud_offset+92>: 48 39 9e 20 01 00 00 cmp
> > %rbx,0x120(%rsi)
> > 0x00005555557f5db3 <x86_64_pud_offset+99>: 74 32 je
> > 0x5555557f5de7 <x86_64_pud_offset+151>
> > 0x00005555557f5db5 <x86_64_pud_offset+101>: 8b 4e 18 mov
0x18(%rsi),%ecx
> > 0x00005555557f5db8 <x86_64_pud_offset+104>: 41 b9 01 00 00 00 mov
> > $0x1,%r9d
> > 0x00005555557f5dbe <x86_64_pud_offset+110>: be 04 00 00 00 mov
$0x4,%esi
> > 0x00005555557f5dc3 <x86_64_pud_offset+115>: 48 89 df mov %rbx,%rdi
> > 0x00005555557f5dc6 <x86_64_pud_offset+118>: 4c 8d 05 6c a8 58 00
> > lea 0x58a86c(%rip),%r8 # 0x555555d80639
> > 0x00005555557f5dcd <x86_64_pud_offset+125>: e8 8e e2 f5 ff callq
> > 0x555555754060 <readmem>
> > 0x00005555557f5dd2 <x86_64_pud_offset+130>: 48 8b 35 87 8e a9 00
> > mov 0xa98e87(%rip),%rsi # 0x55555628ec60 <machdep>
> > 0x00005555557f5dd9 <x86_64_pud_offset+137>: 48 89 9e 20 01 00 00
> > mov %rbx,0x120(%rsi)
> > 0x00005555557f5de0 <x86_64_pud_offset+144>: 48 8b 96 40 01 00 00
> > mov 0x140(%rsi),%rdx
> > (gdb) set machdep->last_pud_read=1 <- This forces the PUD to be re-read
> > (gdb) dis
> > (gdb) c
> > Continuing.
> > 41414000 f67e000
> >
> > PGD: 6e80000 => 7fc8067
> > PUD: 7fc8008 => e576067
> > PMD: e576050 => aa1c067
> > PTE: aa1c0a0 => 800000000f67e867
> > PAGE: f67e000
> >
> > PTE PHYSICAL FLAGS
> > 800000000f67e867 f67e000 (PRESENT|RW|USER|ACCESSED|DIRTY|NX)
> >
> > VMA START END FLAGS FILE
> > ffff88801ec382b8 41414000 41415000 8100073
> >
> > PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> > ffffea00003d9f80 f67e000 ffff8880142d40c1 41414 1 fffffc0040028
> > uptodate,lru,swapbacked
> > crash>
> >
> > Best regards,
> > --
> > Anderson Nascimento
> > Allele Security Intelligence
> > https://www.allelesecurity.com
> > --
> > Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> > To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> > https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> > Contribution Guidelines: https://github.com/crash-utility/crash/wiki
> >
>
--
Anderson Nascimento
Allele Security Intelligence
https://www.allelesecurity.com