[Crash-utility] Re: Subject: [BUG/RFC] arm64: stack init functions fail in POST_GDB when percpu data is in vmalloc area

HI Haixiang, Thanks for reporting the issue. On Thu, Sep 10, 2026 at 09:01:42AM +0000, chenhaixiang (A) wrote: > Subject: [BUG/RFC] arm64: stack init functions fail in POST_GDB when percpu data is in vmalloc area. > Hi, > While running crash on a large arm64 NUMA system, crash fails to > initialize the arm64 stack subsystems during arm64_init(POST_GDB) and > aborts startup with SEEK_ERROR on every percpu IRQ stack pointer. > Environment > ----------- > - architecture : aarch64 (64K pages, CONFIG_ARM64_64K_PAGES=y) > - kernel : 6.x, large NUMA, many CPUs > - crash : current master (arm64.c) > Background on percpu allocation on this machine > ----------------------------------------------- > On this arm64 machine the kernel's first percpu chunk does NOT live in > the linear map, it lives in the vmalloc area. The arm64 percpu setup > path in drivers/base/arch_numa.c:setup_per_cpu_areas() first tries > pcpu_embed_first_chunk() and, when that fails, falls back to > pcpu_page_first_chunk() > Observed symptom / reproduction log > ----------------------------------- > With crash started on the vmcore, the percpu IRQ stack pointers are all in the > ffff800100xxxxxx range, which is the vmalloc area on this kernel. > Because vt->vmalloc_start is still 0 at POST_GDB, arm64_kvtop() takes > the early "VTOP()" shortcut and returns a bogus physical address far > beyond max_mapnr, so read_diskdump() reports SEEK_ERROR: > ... > <readmem: ffff800100000050, KVADDR, "IRQ stack pointer", 8, (ROE), 1bfe2ef0> > <read_diskdump: addr: ffff800100000050 paddr: 800100000050 cnt: 8> > read_diskdump: SEEK_ERROR: paddr/pfn: 800100000050/800100000 max_mapnr: 60c000000 > crash: seek error: kernel virtual address: ffff800100000050 type: "IRQ stack pointer" > <readmem: ffff800100021050, KVADDR, "IRQ stack pointer", 8, (ROE), 1bfe2ef8> > <read_diskdump: addr: ffff800100021050 paddr: 800100021050 cnt: 8> > read_diskdump: SEEK_ERROR: paddr/pfn: 800100021050/800100021 max_mapnr: 60c000000 > crash: seek error: kernel virtual address: ffff800100021050 type: "IRQ stack pointer" > <readmem: ffff800100042050, KVADDR, "IRQ stack pointer", 8, (ROE), 1bfe2f00> > <read_diskdump: addr: ffff800100042050 paddr: 800100042050 cnt: 8> > read_diskdump: SEEK_ERROR: paddr/pfn: 800100042050/800100042 max_mapnr: 60c000000 > crash: seek error: kernel virtual address: ffff800100042050 type: "IRQ stack pointer" > <readmem: ffff800100063050, KVADDR, "IRQ stack pointer", 8, (ROE), 1bfe2ef08> > <read_diskdump: addr: ffff800100063050 paddr: 800100063050 cnt: 8> > read_diskdump: SEEK_ERROR: paddr/pfn: 800100063050/800100063 max_mapnr: 60c000000 > crash: seek error: kernel virtual address: ffff800100063050 type: "IRQ stack pointer" > ... (one SEEK_ERROR per CPU) > Note the wrong physical addresses (800100xxxxxx) come from a direct > linear-map style translation (addr - PAGE_OFFSET) which is only valid > for the linear map, not for vmalloc-backed percpu memory. > Root cause > ---------- > In arm64_init() the three stack initializers are invoked in the > POST_GDB phase: > case POST_GDB: > ... > if (!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) > arm64_recalc_KERNELPACMASK(); > arm64_irq_stack_init(); /* <-- readmem on percpu var */ > arm64_overflow_stack_init(); /* <-- readmem on percpu var */ > arm64_stackframe_init(); > break; > These helpers end up calling readmem() on percpu addresses that, on > the machine described above, live in the vmalloc range. The > translation goes through arm64_kvtop() (arm64.c:1963), which for any > vmalloc address requires: > 1. vt->vmalloc_start != 0 -> to reach the IS_VMALLOC_ADDR() branch > 2. vt->kernel_pgd[0] -> to walk the kernel page tables > Both are populated by vm_init(), which runs in the POST_VM phase -- > i.e. *after* POST_GDB. As a result, in arm64_kvtop() we hit: > if (!vt->vmalloc_start) { > *paddr = VTOP(kvaddr); /* wrong for vmalloc addresses */ > return TRUE; > } > and the returned physical address is bogus, so the subsequent > readmem() fails with SEEK_ERROR (as shown in the log above: paddr > 800100xxxxxx is the linear-map translation of a vmalloc address, and > it falls well beyond max_mapnr 0x60c000000). > Proposed fix > ------------ > Move the three stack initializers from POST_GDB to POST_VM, so that > vt->vmalloc_start and vt->kernel_pgd are guaranteed to be ready > before any readmem() that may touch vmalloc-backed percpu data. > diff --git a/arm64.c b/arm64.c > index d177092..654b972 100644 > --- a/arm64.c > +++ b/arm64.c > @@ -787,7 +787,9 @@ arm64_init(int when) > */ > if(!machdep->machspec- >CONFIG_ARM64_KERNELPACMASK) > arm64_recalc_KERNELPACMASK(); > + break; > > + case POST_VM: > arm64_irq_stack_init(); > arm64_overflow_stack_init(); > arm64_stackframe_init(); > Testing > ------- > Verified on: > - the failing large-NUMA arm64 system whose percpu first chunk was > created by pcpu_page_first_chunk() (i.e. percpu lives in vmalloc): > with the patch, the percpu IRQ stack pointers are translated > through the kernel page tables and crash starts cleanly. "bt" on > the panic task produces correct irq/overflow/standby stacks; > - a regular arm64 QEMU VM whose percpu first chunk was created by > pcpu_embed_first_chunk() (i.e. percpu lives in the linear map): > no regression, stack initialization still happens before any > "bt" command. > I'd like to get feedback on whether POST_VM is the preferred phase, > or whether the maintainers would rather guard the readmem() calls > inside the three helpers with a fallback (e.g. defer percpu > translation until later). I can send a formal patch with > Signed-off-by once the approach is agreed. I'm not confident if move it to POST_VM is safe enough so I'd prefer the second approach, still keep them in POST_GDB, and defer the callbasks later in a special case. Also please add more info in patch log, eg. kernel version, error msg, etc.