12:09 a.m.
Hello,
I have been using the crash tool to teach paging. It is an excellent
tool for simplifying page table walks for students. However, I have
encountered a persistent issue regarding stale data when inspecting
mappings that are populated mid-session.
When using the vtop command on a mapping that is not yet populated,
and then running it again after a memory operation has occurred, vtop
continues to return NULL or stale entries. This happens because the
FILL_PUD (and similar) macros check if the current PUD address matches
the last_pud_read address. If they match, the tool skips the readmem()
call, even if the underlying physical memory has changed.
In the debugging session below, I demonstrate that rd -p showed the
populated PUD entry, but vtop still reported 0. I was able to resolve
this by manually forcing a re-read in GDB by resetting the cache
variable:
(gdb) set machdep->last_pud_read=1
992 #define IS_LAST_PUD_READ(pud) ((ulong)(pud) == machdep->last_pud_read)
...
1001 #define FILL_PUD(PUD, TYPE, SIZE)
\
1002 if (!IS_LAST_PUD_READ(PUD)) {
\
1003 readmem((ulonglong)((ulong)(PUD)), TYPE,
machdep->pud, \
1004 SIZE, "pud page", FAULT_ON_ERROR);
\
1005 machdep->last_pud_read = (ulong)(PUD);
\
1006 }
Steps to Reproduce:
1) Run vtop on an unpopulated user address.
2) Trigger a page fault/memory access in the target process to
populate the entry.
3) Run vtop again; it will still show "(not mapped)" despite the
physical memory being updated.
Is this caching behavior intended for performance, or should a way to
invalidate this cache for live sessions be implemented?
crash> vtop -c 5084 0x41414000
[Detaching after fork from child process 5085]
VIRTUAL PHYSICAL
41414000 (not mapped)
PGD: 6e80000 => 7fc8067
PUD: 7fc8008 => 0
VMA START END FLAGS FILE
ffff88801ec382b8 41414000 41415000 8100073
crash> rd -p 7fc8008
[Detaching after fork from child process 5086]
7fc8008: 0000000000000000 ........
crash> vtop -c 5084 0x41414000
[Detaching after fork from child process 5087]
VIRTUAL PHYSICAL
41414000 (not mapped)
PGD: 6e80000 => 7fc8067
PUD: 7fc8008 => 0
VMA START END FLAGS FILE
ffff88801ec382b8 41414000 41415000 8100073
crash> rd -p 7fc8008
[Detaching after fork from child process 5088]
7fc8008: 000000000e576067 g`W.....
crash>
Thread 1 "crash" received signal SIGINT, Interrupt.
0x00007ffff629d141 in pselect () from /lib64/libc.so.6
=> 0x00007ffff629d141 <pselect+193>: 48 3d 00 f0 ff ff cmp
$0xfffffffffffff000,%rax
(gdb) en 2
(gdb) c
Continuing.
vtop -c 5084 0x41414000
[Detaching after fork from child process 5089]
VIRTUAL PHYSICAL
Thread 1 "crash" hit Breakpoint 2, x86_64_pud_offset
(pgd_pte=<optimized out>, vaddr=1094795264, verbose=0, IS_XEN=0) at
x86_64.c:1970
1970 FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
=> 0x00005555557f5da5 <x86_64_pud_offset+85>: 48 8b 96 40 01 00 00 mov
0x140(%rsi),%rdx
0x00005555557f5dac <x86_64_pud_offset+92>: 48 39 9e 20 01 00 00 cmp
%rbx,0x120(%rsi)
0x00005555557f5db3 <x86_64_pud_offset+99>: 74 32 je
0x5555557f5de7 <x86_64_pud_offset+151>
0x00005555557f5db5 <x86_64_pud_offset+101>: 8b 4e 18 mov 0x18(%rsi),%ecx
0x00005555557f5db8 <x86_64_pud_offset+104>: 41 b9 01 00 00 00 mov
$0x1,%r9d
0x00005555557f5dbe <x86_64_pud_offset+110>: be 04 00 00 00 mov $0x4,%esi
0x00005555557f5dc3 <x86_64_pud_offset+115>: 48 89 df mov %rbx,%rdi
0x00005555557f5dc6 <x86_64_pud_offset+118>: 4c 8d 05 6c a8 58 00
lea 0x58a86c(%rip),%r8 # 0x555555d80639
0x00005555557f5dcd <x86_64_pud_offset+125>: e8 8e e2 f5 ff callq
0x555555754060 <readmem>
0x00005555557f5dd2 <x86_64_pud_offset+130>: 48 8b 35 87 8e a9 00
mov 0xa98e87(%rip),%rsi # 0x55555628ec60 <machdep>
0x00005555557f5dd9 <x86_64_pud_offset+137>: 48 89 9e 20 01 00 00
mov %rbx,0x120(%rsi)
0x00005555557f5de0 <x86_64_pud_offset+144>: 48 8b 96 40 01 00 00
mov 0x140(%rsi),%rdx
(gdb) set machdep->last_pud_read=1 <- This forces the PUD to be re-read
(gdb) dis
(gdb) c
Continuing.
41414000 f67e000
PGD: 6e80000 => 7fc8067
PUD: 7fc8008 => e576067
PMD: e576050 => aa1c067
PTE: aa1c0a0 => 800000000f67e867
PAGE: f67e000
PTE PHYSICAL FLAGS
800000000f67e867 f67e000 (PRESENT|RW|USER|ACCESSED|DIRTY|NX)
VMA START END FLAGS FILE
ffff88801ec382b8 41414000 41415000 8100073
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea00003d9f80 f67e000 ffff8880142d40c1 41414 1 fffffc0040028
uptodate,lru,swapbacked
crash>
Best regards,
--
Anderson Nascimento
Allele Security Intelligence
https://www.allelesecurity.com
10:25 p.m.
New subject: vtop returns stale page table entries due to FILL_PUD caching logic
Hi Anderson,
I think you're right, the cache is more performance intended. Glad to
know machdep->last_pud_read=1 can work, this will force crash to
readmem() to load the updated data. Also you can use 2 crash
instances, say open terminal 1, invoke crash utility against
/proc/kcore for unpolulated address. Then trigger the page fault
process to populate the address. Then open terminal 2, invoke another
crash utility to read the populated address.Then you can make the
comparison of the 2 outputs, the difference is the results of kernel
fixing page fault.
Personally I don't think this is a big issue worthy fix for crash. To
me, always open a new crash instance session is a better solution for
live kcore debug if you think the data is outdated.
Thanks,
Tao Liu
On Thu, May 14, 2026 at 6:55 PM Anderson Nascimento
<anderson(a)allelesecurity.com> wrote:
Hello,
I have been using the crash tool to teach paging. It is an excellent
tool for simplifying page table walks for students. However, I have
encountered a persistent issue regarding stale data when inspecting
mappings that are populated mid-session.
When using the vtop command on a mapping that is not yet populated,
and then running it again after a memory operation has occurred, vtop
continues to return NULL or stale entries. This happens because the
FILL_PUD (and similar) macros check if the current PUD address matches
the last_pud_read address. If they match, the tool skips the readmem()
call, even if the underlying physical memory has changed.
In the debugging session below, I demonstrate that rd -p showed the
populated PUD entry, but vtop still reported 0. I was able to resolve
this by manually forcing a re-read in GDB by resetting the cache
variable:
(gdb) set machdep->last_pud_read=1
992 #define IS_LAST_PUD_READ(pud) ((ulong)(pud) == machdep->last_pud_read)
...
1001 #define FILL_PUD(PUD, TYPE, SIZE)
\
1002 if (!IS_LAST_PUD_READ(PUD)) {
\
1003 readmem((ulonglong)((ulong)(PUD)), TYPE,
machdep->pud, \
1004 SIZE, "pud page", FAULT_ON_ERROR);
\
1005 machdep->last_pud_read = (ulong)(PUD);
\
1006 }
Steps to Reproduce:
1) Run vtop on an unpopulated user address.
2) Trigger a page fault/memory access in the target process to
populate the entry.
3) Run vtop again; it will still show "(not mapped)" despite the
physical memory being updated.
Is this caching behavior intended for performance, or should a way to
invalidate this cache for live sessions be implemented?
crash> vtop -c 5084 0x41414000
[Detaching after fork from child process 5085]
VIRTUAL PHYSICAL
41414000 (not mapped)
PGD: 6e80000 => 7fc8067
PUD: 7fc8008 => 0
VMA START END FLAGS FILE
ffff88801ec382b8 41414000 41415000 8100073
crash> rd -p 7fc8008
[Detaching after fork from child process 5086]
7fc8008: 0000000000000000 ........
crash> vtop -c 5084 0x41414000
[Detaching after fork from child process 5087]
VIRTUAL PHYSICAL
41414000 (not mapped)
PGD: 6e80000 => 7fc8067
PUD: 7fc8008 => 0
VMA START END FLAGS FILE
ffff88801ec382b8 41414000 41415000 8100073
crash> rd -p 7fc8008
[Detaching after fork from child process 5088]
7fc8008: 000000000e576067 g`W.....
crash>
Thread 1 "crash" received signal SIGINT, Interrupt.
0x00007ffff629d141 in pselect () from /lib64/libc.so.6
=> 0x00007ffff629d141 <pselect+193>: 48 3d 00 f0 ff ff cmp
$0xfffffffffffff000,%rax
(gdb) en 2
(gdb) c
Continuing.
vtop -c 5084 0x41414000
[Detaching after fork from child process 5089]
VIRTUAL PHYSICAL
Thread 1 "crash" hit Breakpoint 2, x86_64_pud_offset
(pgd_pte=<optimized out>, vaddr=1094795264, verbose=0, IS_XEN=0) at
x86_64.c:1970
1970 FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
=> 0x00005555557f5da5 <x86_64_pud_offset+85>: 48 8b 96 40 01 00 00 mov
0x140(%rsi),%rdx
0x00005555557f5dac <x86_64_pud_offset+92>: 48 39 9e 20 01 00 00 cmp
%rbx,0x120(%rsi)
0x00005555557f5db3 <x86_64_pud_offset+99>: 74 32 je
0x5555557f5de7 <x86_64_pud_offset+151>
0x00005555557f5db5 <x86_64_pud_offset+101>: 8b 4e 18 mov 0x18(%rsi),%ecx
0x00005555557f5db8 <x86_64_pud_offset+104>: 41 b9 01 00 00 00 mov
$0x1,%r9d
0x00005555557f5dbe <x86_64_pud_offset+110>: be 04 00 00 00 mov $0x4,%esi
0x00005555557f5dc3 <x86_64_pud_offset+115>: 48 89 df mov %rbx,%rdi
0x00005555557f5dc6 <x86_64_pud_offset+118>: 4c 8d 05 6c a8 58 00
lea 0x58a86c(%rip),%r8 # 0x555555d80639
0x00005555557f5dcd <x86_64_pud_offset+125>: e8 8e e2 f5 ff callq
0x555555754060 <readmem>
0x00005555557f5dd2 <x86_64_pud_offset+130>: 48 8b 35 87 8e a9 00
mov 0xa98e87(%rip),%rsi # 0x55555628ec60 <machdep>
0x00005555557f5dd9 <x86_64_pud_offset+137>: 48 89 9e 20 01 00 00
mov %rbx,0x120(%rsi)
0x00005555557f5de0 <x86_64_pud_offset+144>: 48 8b 96 40 01 00 00
mov 0x140(%rsi),%rdx
(gdb) set machdep->last_pud_read=1 <- This forces the PUD to be re-read
(gdb) dis
(gdb) c
Continuing.
41414000 f67e000
PGD: 6e80000 => 7fc8067
PUD: 7fc8008 => e576067
PMD: e576050 => aa1c067
PTE: aa1c0a0 => 800000000f67e867
PAGE: f67e000
PTE PHYSICAL FLAGS
800000000f67e867 f67e000 (PRESENT|RW|USER|ACCESSED|DIRTY|NX)
VMA START END FLAGS FILE
ffff88801ec382b8 41414000 41415000 8100073
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea00003d9f80 f67e000 ffff8880142d40c1 41414 1 fffffc0040028
uptodate,lru,swapbacked
crash>
Best regards,
--
Anderson Nascimento
Allele Security Intelligence
https://www.allelesecurity.com
--
Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
Contribution Guidelines: https://github.com/crash-utility/crash/wiki
1:14 a.m.
New subject: vtop returns stale page table entries due to FILL_PUD caching logic
Hi Tao Liu,
Thanks! It makes sense. I thought about making a patch to make users
decide if they want fresh data, but if this is intended, that's okay.
I just wanted to make sure it is intended behavior and not a bug,
because this has been happening very often in my classes, and it
doesn't look good for me when I say something and the tool doesn't
confirm it. Now at least I know exactly how it works and will be
prepared during classes.
Thank you!
On Mon, May 18, 2026 at 12:26 AM Tao Liu <ltao(a)redhat.com> wrote:
Hi Anderson,
I think you're right, the cache is more performance intended. Glad to
know machdep->last_pud_read=1 can work, this will force crash to
readmem() to load the updated data. Also you can use 2 crash
instances, say open terminal 1, invoke crash utility against
/proc/kcore for unpolulated address. Then trigger the page fault
process to populate the address. Then open terminal 2, invoke another
crash utility to read the populated address.Then you can make the
comparison of the 2 outputs, the difference is the results of kernel
fixing page fault.
Personally I don't think this is a big issue worthy fix for crash. To
me, always open a new crash instance session is a better solution for
live kcore debug if you think the data is outdated.
Thanks,
Tao Liu
On Thu, May 14, 2026 at 6:55 PM Anderson Nascimento
<anderson(a)allelesecurity.com> wrote:
>
> Hello,
>
> I have been using the crash tool to teach paging. It is an excellent
> tool for simplifying page table walks for students. However, I have
> encountered a persistent issue regarding stale data when inspecting
> mappings that are populated mid-session.
>
> When using the vtop command on a mapping that is not yet populated,
> and then running it again after a memory operation has occurred, vtop
> continues to return NULL or stale entries. This happens because the
> FILL_PUD (and similar) macros check if the current PUD address matches
> the last_pud_read address. If they match, the tool skips the readmem()
> call, even if the underlying physical memory has changed.
>
> In the debugging session below, I demonstrate that rd -p showed the
> populated PUD entry, but vtop still reported 0. I was able to resolve
> this by manually forcing a re-read in GDB by resetting the cache
> variable:
>
> (gdb) set machdep->last_pud_read=1
>
> 992 #define IS_LAST_PUD_READ(pud) ((ulong)(pud) == machdep->last_pud_read)
> ...
> 1001 #define FILL_PUD(PUD, TYPE, SIZE)
> \
> 1002 if (!IS_LAST_PUD_READ(PUD)) {
> \
> 1003 readmem((ulonglong)((ulong)(PUD)), TYPE,
> machdep->pud, \
> 1004 SIZE, "pud page", FAULT_ON_ERROR);
> \
> 1005 machdep->last_pud_read = (ulong)(PUD);
> \
> 1006 }
>
> Steps to Reproduce:
>
> 1) Run vtop on an unpopulated user address.
>
> 2) Trigger a page fault/memory access in the target process to
> populate the entry.
>
> 3) Run vtop again; it will still show "(not mapped)" despite the
> physical memory being updated.
>
> Is this caching behavior intended for performance, or should a way to
> invalidate this cache for live sessions be implemented?
>
> crash> vtop -c 5084 0x41414000
> [Detaching after fork from child process 5085]
> VIRTUAL PHYSICAL
> 41414000 (not mapped)
>
> PGD: 6e80000 => 7fc8067
> PUD: 7fc8008 => 0
>
> VMA START END FLAGS FILE
> ffff88801ec382b8 41414000 41415000 8100073
>
> crash> rd -p 7fc8008
> [Detaching after fork from child process 5086]
> 7fc8008: 0000000000000000 ........
> crash> vtop -c 5084 0x41414000
> [Detaching after fork from child process 5087]
> VIRTUAL PHYSICAL
> 41414000 (not mapped)
>
> PGD: 6e80000 => 7fc8067
> PUD: 7fc8008 => 0
>
> VMA START END FLAGS FILE
> ffff88801ec382b8 41414000 41415000 8100073
>
> crash> rd -p 7fc8008
> [Detaching after fork from child process 5088]
> 7fc8008: 000000000e576067 g`W.....
> crash>
> Thread 1 "crash" received signal SIGINT, Interrupt.
> 0x00007ffff629d141 in pselect () from /lib64/libc.so.6
> => 0x00007ffff629d141 <pselect+193>: 48 3d 00 f0 ff ff cmp
> $0xfffffffffffff000,%rax
> (gdb) en 2
> (gdb) c
> Continuing.
> vtop -c 5084 0x41414000
> [Detaching after fork from child process 5089]
> VIRTUAL PHYSICAL
>
> Thread 1 "crash" hit Breakpoint 2, x86_64_pud_offset
> (pgd_pte=<optimized out>, vaddr=1094795264, verbose=0, IS_XEN=0) at
> x86_64.c:1970
> 1970 FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
> => 0x00005555557f5da5 <x86_64_pud_offset+85>: 48 8b 96 40 01 00 00 mov
> 0x140(%rsi),%rdx
> 0x00005555557f5dac <x86_64_pud_offset+92>: 48 39 9e 20 01 00 00 cmp
> %rbx,0x120(%rsi)
> 0x00005555557f5db3 <x86_64_pud_offset+99>: 74 32 je
> 0x5555557f5de7 <x86_64_pud_offset+151>
> 0x00005555557f5db5 <x86_64_pud_offset+101>: 8b 4e 18 mov
0x18(%rsi),%ecx
> 0x00005555557f5db8 <x86_64_pud_offset+104>: 41 b9 01 00 00 00 mov
> $0x1,%r9d
> 0x00005555557f5dbe <x86_64_pud_offset+110>: be 04 00 00 00 mov
$0x4,%esi
> 0x00005555557f5dc3 <x86_64_pud_offset+115>: 48 89 df mov %rbx,%rdi
> 0x00005555557f5dc6 <x86_64_pud_offset+118>: 4c 8d 05 6c a8 58 00
> lea 0x58a86c(%rip),%r8 # 0x555555d80639
> 0x00005555557f5dcd <x86_64_pud_offset+125>: e8 8e e2 f5 ff callq
> 0x555555754060 <readmem>
> 0x00005555557f5dd2 <x86_64_pud_offset+130>: 48 8b 35 87 8e a9 00
> mov 0xa98e87(%rip),%rsi # 0x55555628ec60 <machdep>
> 0x00005555557f5dd9 <x86_64_pud_offset+137>: 48 89 9e 20 01 00 00
> mov %rbx,0x120(%rsi)
> 0x00005555557f5de0 <x86_64_pud_offset+144>: 48 8b 96 40 01 00 00
> mov 0x140(%rsi),%rdx
> (gdb) set machdep->last_pud_read=1 <- This forces the PUD to be re-read
> (gdb) dis
> (gdb) c
> Continuing.
> 41414000 f67e000
>
> PGD: 6e80000 => 7fc8067
> PUD: 7fc8008 => e576067
> PMD: e576050 => aa1c067
> PTE: aa1c0a0 => 800000000f67e867
> PAGE: f67e000
>
> PTE PHYSICAL FLAGS
> 800000000f67e867 f67e000 (PRESENT|RW|USER|ACCESSED|DIRTY|NX)
>
> VMA START END FLAGS FILE
> ffff88801ec382b8 41414000 41415000 8100073
>
> PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> ffffea00003d9f80 f67e000 ffff8880142d40c1 41414 1 fffffc0040028
> uptodate,lru,swapbacked
> crash>
>
> Best regards,
> --
> Anderson Nascimento
> Allele Security Intelligence
> https://www.allelesecurity.com
> --
> Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
>
--
Anderson Nascimento
Allele Security Intelligence
https://www.allelesecurity.com
3:58 p.m.
New subject: vtop returns stale page table entries due to FILL_PUD caching logic
On Sun, May 24, 2026 at 5:41 AM Anderson Nascimento
<anderson(a)allelesecurity.com> wrote:
Hi Tao Liu,
Thanks! It makes sense. I thought about making a patch to make users
decide if they want fresh data, but if this is intended, that's okay.
I just wanted to make sure it is intended behavior and not a bug,
because this has been happening very often in my classes, and it
doesn't look good for me when I say something and the tool doesn't
confirm it. Now at least I know exactly how it works and will be
prepared during classes.
Yes, this is more like a trade-off. For static crash & dump, say we
run crash utility against a vmcore, with its data never changing, the
readmem() for one address always returns the same results, so a
readmem() cache can accelerate. When it comes to kcore, with its data
constantly changing, such a cache can improve the performance, however
a outdated info might returned.
For our work, static crash & dump, aka the vmcore case is the majority
one we encountered. So this is more a trade-off than intended for the
kcore case. If you are interested, you can improve the page table
walking for crash, so that a readmem() is always used instead of cache
when kcore, rather than vmcore, is the case. Ofcourse, like I said
previously, if we don't do the fix, open a new crash instance is a
simple work around.
Thanks,
Tao Liu
Thank you!
On Mon, May 18, 2026 at 12:26 AM Tao Liu <ltao(a)redhat.com> wrote:
>
> Hi Anderson,
>
> I think you're right, the cache is more performance intended. Glad to
> know machdep->last_pud_read=1 can work, this will force crash to
> readmem() to load the updated data. Also you can use 2 crash
> instances, say open terminal 1, invoke crash utility against
> /proc/kcore for unpolulated address. Then trigger the page fault
> process to populate the address. Then open terminal 2, invoke another
> crash utility to read the populated address.Then you can make the
> comparison of the 2 outputs, the difference is the results of kernel
> fixing page fault.
>
> Personally I don't think this is a big issue worthy fix for crash. To
> me, always open a new crash instance session is a better solution for
> live kcore debug if you think the data is outdated.
>
> Thanks,
> Tao Liu
>
> On Thu, May 14, 2026 at 6:55 PM Anderson Nascimento
> <anderson(a)allelesecurity.com> wrote:
> >
> > Hello,
> >
> > I have been using the crash tool to teach paging. It is an excellent
> > tool for simplifying page table walks for students. However, I have
> > encountered a persistent issue regarding stale data when inspecting
> > mappings that are populated mid-session.
> >
> > When using the vtop command on a mapping that is not yet populated,
> > and then running it again after a memory operation has occurred, vtop
> > continues to return NULL or stale entries. This happens because the
> > FILL_PUD (and similar) macros check if the current PUD address matches
> > the last_pud_read address. If they match, the tool skips the readmem()
> > call, even if the underlying physical memory has changed.
> >
> > In the debugging session below, I demonstrate that rd -p showed the
> > populated PUD entry, but vtop still reported 0. I was able to resolve
> > this by manually forcing a re-read in GDB by resetting the cache
> > variable:
> >
> > (gdb) set machdep->last_pud_read=1
> >
> > 992 #define IS_LAST_PUD_READ(pud) ((ulong)(pud) ==
machdep->last_pud_read)
> > ...
> > 1001 #define FILL_PUD(PUD, TYPE, SIZE)
> > \
> > 1002 if (!IS_LAST_PUD_READ(PUD)) {
> > \
> > 1003 readmem((ulonglong)((ulong)(PUD)), TYPE,
> > machdep->pud, \
> > 1004 SIZE, "pud page", FAULT_ON_ERROR);
> > \
> > 1005 machdep->last_pud_read = (ulong)(PUD);
> > \
> > 1006 }
> >
> > Steps to Reproduce:
> >
> > 1) Run vtop on an unpopulated user address.
> >
> > 2) Trigger a page fault/memory access in the target process to
> > populate the entry.
> >
> > 3) Run vtop again; it will still show "(not mapped)" despite the
> > physical memory being updated.
> >
> > Is this caching behavior intended for performance, or should a way to
> > invalidate this cache for live sessions be implemented?
> >
> > crash> vtop -c 5084 0x41414000
> > [Detaching after fork from child process 5085]
> > VIRTUAL PHYSICAL
> > 41414000 (not mapped)
> >
> > PGD: 6e80000 => 7fc8067
> > PUD: 7fc8008 => 0
> >
> > VMA START END FLAGS FILE
> > ffff88801ec382b8 41414000 41415000 8100073
> >
> > crash> rd -p 7fc8008
> > [Detaching after fork from child process 5086]
> > 7fc8008: 0000000000000000 ........
> > crash> vtop -c 5084 0x41414000
> > [Detaching after fork from child process 5087]
> > VIRTUAL PHYSICAL
> > 41414000 (not mapped)
> >
> > PGD: 6e80000 => 7fc8067
> > PUD: 7fc8008 => 0
> >
> > VMA START END FLAGS FILE
> > ffff88801ec382b8 41414000 41415000 8100073
> >
> > crash> rd -p 7fc8008
> > [Detaching after fork from child process 5088]
> > 7fc8008: 000000000e576067 g`W.....
> > crash>
> > Thread 1 "crash" received signal SIGINT, Interrupt.
> > 0x00007ffff629d141 in pselect () from /lib64/libc.so.6
> > => 0x00007ffff629d141 <pselect+193>: 48 3d 00 f0 ff ff cmp
> > $0xfffffffffffff000,%rax
> > (gdb) en 2
> > (gdb) c
> > Continuing.
> > vtop -c 5084 0x41414000
> > [Detaching after fork from child process 5089]
> > VIRTUAL PHYSICAL
> >
> > Thread 1 "crash" hit Breakpoint 2, x86_64_pud_offset
> > (pgd_pte=<optimized out>, vaddr=1094795264, verbose=0, IS_XEN=0) at
> > x86_64.c:1970
> > 1970 FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
> > => 0x00005555557f5da5 <x86_64_pud_offset+85>: 48 8b 96 40 01 00 00
mov
> > 0x140(%rsi),%rdx
> > 0x00005555557f5dac <x86_64_pud_offset+92>: 48 39 9e 20 01 00 00 cmp
> > %rbx,0x120(%rsi)
> > 0x00005555557f5db3 <x86_64_pud_offset+99>: 74 32 je
> > 0x5555557f5de7 <x86_64_pud_offset+151>
> > 0x00005555557f5db5 <x86_64_pud_offset+101>: 8b 4e 18 mov
0x18(%rsi),%ecx
> > 0x00005555557f5db8 <x86_64_pud_offset+104>: 41 b9 01 00 00 00 mov
> > $0x1,%r9d
> > 0x00005555557f5dbe <x86_64_pud_offset+110>: be 04 00 00 00 mov
$0x4,%esi
> > 0x00005555557f5dc3 <x86_64_pud_offset+115>: 48 89 df mov %rbx,%rdi
> > 0x00005555557f5dc6 <x86_64_pud_offset+118>: 4c 8d 05 6c a8 58 00
> > lea 0x58a86c(%rip),%r8 # 0x555555d80639
> > 0x00005555557f5dcd <x86_64_pud_offset+125>: e8 8e e2 f5 ff callq
> > 0x555555754060 <readmem>
> > 0x00005555557f5dd2 <x86_64_pud_offset+130>: 48 8b 35 87 8e a9 00
> > mov 0xa98e87(%rip),%rsi # 0x55555628ec60 <machdep>
> > 0x00005555557f5dd9 <x86_64_pud_offset+137>: 48 89 9e 20 01 00 00
> > mov %rbx,0x120(%rsi)
> > 0x00005555557f5de0 <x86_64_pud_offset+144>: 48 8b 96 40 01 00 00
> > mov 0x140(%rsi),%rdx
> > (gdb) set machdep->last_pud_read=1 <- This forces the PUD to be re-read
> > (gdb) dis
> > (gdb) c
> > Continuing.
> > 41414000 f67e000
> >
> > PGD: 6e80000 => 7fc8067
> > PUD: 7fc8008 => e576067
> > PMD: e576050 => aa1c067
> > PTE: aa1c0a0 => 800000000f67e867
> > PAGE: f67e000
> >
> > PTE PHYSICAL FLAGS
> > 800000000f67e867 f67e000 (PRESENT|RW|USER|ACCESSED|DIRTY|NX)
> >
> > VMA START END FLAGS FILE
> > ffff88801ec382b8 41414000 41415000 8100073
> >
> > PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> > ffffea00003d9f80 f67e000 ffff8880142d40c1 41414 1 fffffc0040028
> > uptodate,lru,swapbacked
> > crash>
> >
> > Best regards,
> > --
> > Anderson Nascimento
> > Allele Security Intelligence
> > https://www.allelesecurity.com
> > --
> > Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> > To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> > https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> > Contribution Guidelines: https://github.com/crash-utility/crash/wiki
> >
>
--
Anderson Nascimento
Allele Security Intelligence
https://www.allelesecurity.com
6:54 p.m.
New subject: vtop returns stale page table entries due to FILL_PUD caching logic
On Sun, May 24, 2026 at 5:58 PM Tao Liu <ltao(a)redhat.com> wrote:
On Sun, May 24, 2026 at 5:41 AM Anderson Nascimento
<anderson(a)allelesecurity.com> wrote:
>
> Hi Tao Liu,
>
> Thanks! It makes sense. I thought about making a patch to make users
> decide if they want fresh data, but if this is intended, that's okay.
> I just wanted to make sure it is intended behavior and not a bug,
> because this has been happening very often in my classes, and it
> doesn't look good for me when I say something and the tool doesn't
> confirm it. Now at least I know exactly how it works and will be
> prepared during classes.
Yes, this is more like a trade-off. For static crash & dump, say we
run crash utility against a vmcore, with its data never changing, the
readmem() for one address always returns the same results, so a
readmem() cache can accelerate. When it comes to kcore, with its data
constantly changing, such a cache can improve the performance, however
a outdated info might returned.
For our work, static crash & dump, aka the vmcore case is the majority
one we encountered. So this is more a trade-off than intended for the
kcore case. If you are interested, you can improve the page table
walking for crash, so that a readmem() is always used instead of cache
when kcore, rather than vmcore, is the case. Ofcourse, like I said
previously, if we don't do the fix, open a new crash instance is a
simple work around.
Thanks,
Tao Liu
I did a quick analysis to implement a patch, but I discovered that the
issue does not occur on newer kernels.
I successfully verified my patch on the affected platforms I initially
encountered: Ubuntu 22.04 (kernel 5.15) and Alma Linux 8 (kernel
4.18). However, on Fedora 42 (kernel 6.19), I couldn't reproduce the
bug. Because the issue persists on newer versions of the crash utility
when compiled on those older platforms, the difference in behavior is
tied to the kernel version being analyzed, not the crash version
itself.
When executing the vtop command, on the affected platforms,
last_pud_read is expected to cache the PUD of the requested user
mapping. However, on newer kernels, a subsequent internal read
clobbers this cache before vtop finishes, masking the behavior. The
sequence on kernel 6.2 or greater happens when vtop dumps the target
page table mapping and then calls vm_area_dump() to retrieve VMA
information. This calls get_task_mem_usage(), which invokes
percpu_counter_sum_positive(). This extra call triggers a kvtop
translation that overwrites last_pud_read and leaves a different value
behind.
Consequently, on newer kernels, the subsequent cache check fails and
forces a page re-read every time, avoiding the behavior I reported. On
older kernels, percpu_counter_sum_positive() is not called, leaving
the user-requested mapping PUD cache intact and exposes the behavior.
Reproduction Data
Kernel 6.19 (Bug masked / Cache overwritten):
crash> vtop -c 344463 0x41414000
VIRTUAL PHYSICAL
41414000 (not mapped)
PGD: 1f2675000 => 2c7626067
PUD: 2c7626008 => 0
VMA START END FLAGS FILE
ffff8d7a7d089b40 41414000 41415000 8100073
crash> help -m|grep pud
last_pud_read: 100040000
pud: 1423710
crash>
Kernel 5.15 (Bug reproduces / Cache intact):
crash> vtop -c 1798 0x41414000
VIRTUAL PHYSICAL
41414000 (not mapped)
PGD: 106876000 => 114be8067
PUD: 114be8008 => 0
VMA START END FLAGS FILE
ffff895e06bbf000 41414000 41415000 8100073
crash> help -m|grep pud
last_pud_read: 114be8000
pud: 5555566bed30
crash>
I just thought about sharing that analysis. I don't know if this is
intended and if the cache should work that way. On my system, with a
recent version of crash on a 6.2+ kernel, it seems last_pud_read is
always 100040000.
>
> Thank you!
>
> On Mon, May 18, 2026 at 12:26 AM Tao Liu <ltao(a)redhat.com> wrote:
> >
> > Hi Anderson,
> >
> > I think you're right, the cache is more performance intended. Glad to
> > know machdep->last_pud_read=1 can work, this will force crash to
> > readmem() to load the updated data. Also you can use 2 crash
> > instances, say open terminal 1, invoke crash utility against
> > /proc/kcore for unpolulated address. Then trigger the page fault
> > process to populate the address. Then open terminal 2, invoke another
> > crash utility to read the populated address.Then you can make the
> > comparison of the 2 outputs, the difference is the results of kernel
> > fixing page fault.
> >
> > Personally I don't think this is a big issue worthy fix for crash. To
> > me, always open a new crash instance session is a better solution for
> > live kcore debug if you think the data is outdated.
> >
> > Thanks,
> > Tao Liu
> >
> > On Thu, May 14, 2026 at 6:55 PM Anderson Nascimento
> > <anderson(a)allelesecurity.com> wrote:
> > >
> > > Hello,
> > >
> > > I have been using the crash tool to teach paging. It is an excellent
> > > tool for simplifying page table walks for students. However, I have
> > > encountered a persistent issue regarding stale data when inspecting
> > > mappings that are populated mid-session.
> > >
> > > When using the vtop command on a mapping that is not yet populated,
> > > and then running it again after a memory operation has occurred, vtop
> > > continues to return NULL or stale entries. This happens because the
> > > FILL_PUD (and similar) macros check if the current PUD address matches
> > > the last_pud_read address. If they match, the tool skips the readmem()
> > > call, even if the underlying physical memory has changed.
> > >
> > > In the debugging session below, I demonstrate that rd -p showed the
> > > populated PUD entry, but vtop still reported 0. I was able to resolve
> > > this by manually forcing a re-read in GDB by resetting the cache
> > > variable:
> > >
> > > (gdb) set machdep->last_pud_read=1
> > >
> > > 992 #define IS_LAST_PUD_READ(pud) ((ulong)(pud) ==
machdep->last_pud_read)
> > > ...
> > > 1001 #define FILL_PUD(PUD, TYPE, SIZE)
> > > \
> > > 1002 if (!IS_LAST_PUD_READ(PUD)) {
> > > \
> > > 1003 readmem((ulonglong)((ulong)(PUD)), TYPE,
> > > machdep->pud, \
> > > 1004 SIZE, "pud page", FAULT_ON_ERROR);
> > > \
> > > 1005 machdep->last_pud_read = (ulong)(PUD);
> > > \
> > > 1006 }
> > >
> > > Steps to Reproduce:
> > >
> > > 1) Run vtop on an unpopulated user address.
> > >
> > > 2) Trigger a page fault/memory access in the target process to
> > > populate the entry.
> > >
> > > 3) Run vtop again; it will still show "(not mapped)" despite
the
> > > physical memory being updated.
> > >
> > > Is this caching behavior intended for performance, or should a way to
> > > invalidate this cache for live sessions be implemented?
> > >
> > > crash> vtop -c 5084 0x41414000
> > > [Detaching after fork from child process 5085]
> > > VIRTUAL PHYSICAL
> > > 41414000 (not mapped)
> > >
> > > PGD: 6e80000 => 7fc8067
> > > PUD: 7fc8008 => 0
> > >
> > > VMA START END FLAGS FILE
> > > ffff88801ec382b8 41414000 41415000 8100073
> > >
> > > crash> rd -p 7fc8008
> > > [Detaching after fork from child process 5086]
> > > 7fc8008: 0000000000000000 ........
> > > crash> vtop -c 5084 0x41414000
> > > [Detaching after fork from child process 5087]
> > > VIRTUAL PHYSICAL
> > > 41414000 (not mapped)
> > >
> > > PGD: 6e80000 => 7fc8067
> > > PUD: 7fc8008 => 0
> > >
> > > VMA START END FLAGS FILE
> > > ffff88801ec382b8 41414000 41415000 8100073
> > >
> > > crash> rd -p 7fc8008
> > > [Detaching after fork from child process 5088]
> > > 7fc8008: 000000000e576067 g`W.....
> > > crash>
> > > Thread 1 "crash" received signal SIGINT, Interrupt.
> > > 0x00007ffff629d141 in pselect () from /lib64/libc.so.6
> > > => 0x00007ffff629d141 <pselect+193>: 48 3d 00 f0 ff ff cmp
> > > $0xfffffffffffff000,%rax
> > > (gdb) en 2
> > > (gdb) c
> > > Continuing.
> > > vtop -c 5084 0x41414000
> > > [Detaching after fork from child process 5089]
> > > VIRTUAL PHYSICAL
> > >
> > > Thread 1 "crash" hit Breakpoint 2, x86_64_pud_offset
> > > (pgd_pte=<optimized out>, vaddr=1094795264, verbose=0, IS_XEN=0) at
> > > x86_64.c:1970
> > > 1970 FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
> > > => 0x00005555557f5da5 <x86_64_pud_offset+85>: 48 8b 96 40 01 00
00 mov
> > > 0x140(%rsi),%rdx
> > > 0x00005555557f5dac <x86_64_pud_offset+92>: 48 39 9e 20 01 00 00
cmp
> > > %rbx,0x120(%rsi)
> > > 0x00005555557f5db3 <x86_64_pud_offset+99>: 74 32 je
> > > 0x5555557f5de7 <x86_64_pud_offset+151>
> > > 0x00005555557f5db5 <x86_64_pud_offset+101>: 8b 4e 18 mov
0x18(%rsi),%ecx
> > > 0x00005555557f5db8 <x86_64_pud_offset+104>: 41 b9 01 00 00 00
mov
> > > $0x1,%r9d
> > > 0x00005555557f5dbe <x86_64_pud_offset+110>: be 04 00 00 00 mov
$0x4,%esi
> > > 0x00005555557f5dc3 <x86_64_pud_offset+115>: 48 89 df mov
%rbx,%rdi
> > > 0x00005555557f5dc6 <x86_64_pud_offset+118>: 4c 8d 05 6c a8 58 00
> > > lea 0x58a86c(%rip),%r8 # 0x555555d80639
> > > 0x00005555557f5dcd <x86_64_pud_offset+125>: e8 8e e2 f5 ff callq
> > > 0x555555754060 <readmem>
> > > 0x00005555557f5dd2 <x86_64_pud_offset+130>: 48 8b 35 87 8e a9 00
> > > mov 0xa98e87(%rip),%rsi # 0x55555628ec60 <machdep>
> > > 0x00005555557f5dd9 <x86_64_pud_offset+137>: 48 89 9e 20 01 00 00
> > > mov %rbx,0x120(%rsi)
> > > 0x00005555557f5de0 <x86_64_pud_offset+144>: 48 8b 96 40 01 00 00
> > > mov 0x140(%rsi),%rdx
> > > (gdb) set machdep->last_pud_read=1 <- This forces the PUD to be
re-read
> > > (gdb) dis
> > > (gdb) c
> > > Continuing.
> > > 41414000 f67e000
> > >
> > > PGD: 6e80000 => 7fc8067
> > > PUD: 7fc8008 => e576067
> > > PMD: e576050 => aa1c067
> > > PTE: aa1c0a0 => 800000000f67e867
> > > PAGE: f67e000
> > >
> > > PTE PHYSICAL FLAGS
> > > 800000000f67e867 f67e000 (PRESENT|RW|USER|ACCESSED|DIRTY|NX)
> > >
> > > VMA START END FLAGS FILE
> > > ffff88801ec382b8 41414000 41415000 8100073
> > >
> > > PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> > > ffffea00003d9f80 f67e000 ffff8880142d40c1 41414 1 fffffc0040028
> > > uptodate,lru,swapbacked
> > > crash>
> > >
> > > Best regards,
> > > --
> > > Anderson Nascimento
> > > Allele Security Intelligence
> > > https://www.allelesecurity.com
> > > --
> > > Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> > > To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> > > https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> > > Contribution Guidelines: https://github.com/crash-utility/crash/wiki
> > >
> >
>
>
> --
> Anderson Nascimento
> Allele Security Intelligence
> https://www.allelesecurity.com
>
--
Anderson Nascimento
Allele Security Intelligence
https://www.allelesecurity.com
18
days inactive
48
days old
devel@lists.crash-utility.osci.io
4 comments
2 participants
participants (2)
-
Anderson Nascimento -
Tao Liu