[Crash-utility] Re: Subject: [BUG/RFC] arm64: stack init functions fail in POST_GDB when percpu data is in vmalloc area

in POST_GDB when percpu data is in vmalloc area Hi Haixiang, Thanks for your follow up and details. On 9/14/26 5:10 PM, chenhaixiang (A) wrote: > Hi Dave, > Thanks for the feedback. > > On Fri, Sep 11, 2026 at 18:53 +0000, Dave Young wrote: >> HI Haixiang, >> >> Thanks for reporting the issue. >> >> On Thu, Sep 10, 2026 at 09:01:42AM +0000, chenhaixiang (A) wrote: >>> Subject: [BUG/RFC] arm64: stack init functions fail in POST_GDB when >> percpu data is in vmalloc area. >>> Hi, >>> While running crash on a large arm64 NUMA system, crash fails to >>> initialize the arm64 stack subsystems during arm64_init(POST_GDB) >>> and aborts startup with SEEK_ERROR on every percpu IRQ stack pointer. >>> Environment >>> ----------- >>> - architecture : aarch64 (64K pages, CONFIG_ARM64_64K_PAGES=y) >>> - kernel : 6.x, large NUMA, many CPUs >>> - crash : current master (arm64.c) >>> Background on percpu allocation on this machine >>> ----------------------------------------------- >>> On this arm64 machine the kernel's first percpu chunk does NOT live >>> in the linear map, it lives in the vmalloc area. The arm64 percpu >>> setup path in drivers/base/arch_numa.c:setup_per_cpu_areas() first >>> tries >>> pcpu_embed_first_chunk() and, when that fails, falls back to >>> pcpu_page_first_chunk() >>> Observed symptom / reproduction log >>> ----------------------------------- >>> With crash started on the vmcore, the percpu IRQ stack pointers are >>> all in >> the >>> ffff800100xxxxxx range, which is the vmalloc area on this kernel. >>> Because vt->vmalloc_start is still 0 at POST_GDB, arm64_kvtop() >>> takes the early "VTOP()" shortcut and returns a bogus physical >>> address far beyond max_mapnr, so read_diskdump() reports SEEK_ERROR: >>> ... >>> <readmem: ffff800100000050, KVADDR, "IRQ stack pointer", 8, (ROE), >> 1bfe2ef0> >>> <read_diskdump: addr: ffff800100000050 paddr: 800100000050 cnt: 8> >>> read_diskdump: SEEK_ERROR: paddr/pfn: 800100000050/800100000 >> max_mapnr: 60c000000 >>> crash: seek error: kernel virtual address: ffff800100000050 type: >>> "IRQ >> stack pointer" >>> <readmem: ffff800100021050, KVADDR, "IRQ stack pointer", 8, (ROE), >> 1bfe2ef8> >>> <read_diskdump: addr: ffff800100021050 paddr: 800100021050 cnt: 8> >>> read_diskdump: SEEK_ERROR: paddr/pfn: 800100021050/800100021 >> max_mapnr: 60c000000 >>> crash: seek error: kernel virtual address: ffff800100021050 type: >>> "IRQ >> stack pointer" >>> <readmem: ffff800100042050, KVADDR, "IRQ stack pointer", 8, (ROE), >> 1bfe2f00> >>> <read_diskdump: addr: ffff800100042050 paddr: 800100042050 cnt: 8> >>> read_diskdump: SEEK_ERROR: paddr/pfn: 800100042050/800100042 >> max_mapnr: 60c000000 >>> crash: seek error: kernel virtual address: ffff800100042050 type: >>> "IRQ >> stack pointer" >>> <readmem: ffff800100063050, KVADDR, "IRQ stack pointer", 8, (ROE), >> 1bfe2ef08> >>> <read_diskdump: addr: ffff800100063050 paddr: 800100063050 cnt: 8> >>> read_diskdump: SEEK_ERROR: paddr/pfn: 800100063050/800100063 >> max_mapnr: 60c000000 >>> crash: seek error: kernel virtual address: ffff800100063050 type: >>> "IRQ >> stack pointer" >>> ... (one SEEK_ERROR per CPU) >>> Note the wrong physical addresses (800100xxxxxx) come from a direct >>> linear-map style translation (addr - PAGE_OFFSET) which is only >>> valid for the linear map, not for vmalloc-backed percpu memory. >>> Root cause >>> ---------- >>> In arm64_init() the three stack initializers are invoked in the >>> POST_GDB phase: >>> case POST_GDB: >>> ... >>> if (!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) >>> arm64_recalc_KERNELPACMASK(); >>> arm64_irq_stack_init(); /* <-- readmem on percpu var */ >>> arm64_overflow_stack_init(); /* <-- readmem on percpu var */ >>> arm64_stackframe_init(); >>> break; >>> These helpers end up calling readmem() on percpu addresses that, on >>> the machine described above, live in the vmalloc range. The >>> translation goes through arm64_kvtop() (arm64.c:1963), which for any >>> vmalloc address requires: >>> 1. vt->vmalloc_start != 0 -> to reach the IS_VMALLOC_ADDR() branch >>> 2. vt->kernel_pgd[0] -> to walk the kernel page tables >>> Both are populated by vm_init(), which runs in the POST_VM phase -- >>> i.e. *after* POST_GDB. As a result, in arm64_kvtop() we hit: >>> if (!vt->vmalloc_start) { >>> *paddr = VTOP(kvaddr); /* wrong for vmalloc addresses */ >>> return TRUE; >>> } >>> and the returned physical address is bogus, so the subsequent >>> readmem() fails with SEEK_ERROR (as shown in the log above: paddr >>> 800100xxxxxx is the linear-map translation of a vmalloc address, and >>> it falls well beyond max_mapnr 0x60c000000). >>> Proposed fix >>> ------------ >>> Move the three stack initializers from POST_GDB to POST_VM, so that >>> vt->vmalloc_start and vt->kernel_pgd are guaranteed to be ready >>> before any readmem() that may touch vmalloc-backed percpu data. >>> diff --git a/arm64.c b/arm64.c >>> index d177092..654b972 100644 >>> --- a/arm64.c >>> +++ b/arm64.c >>> @@ -787,7 +787,9 @@ arm64_init(int when) >>> */ >>> if(!machdep->machspec- >>> CONFIG_ARM64_KERNELPACMASK) >>> arm64_recalc_KERNELPACMASK(); >>> + break; >>> >>> + case POST_VM: >>> arm64_irq_stack_init(); >>> arm64_overflow_stack_init(); >>> arm64_stackframe_init(); >>> Testing >>> ------- >>> Verified on: >>> - the failing large-NUMA arm64 system whose percpu first chunk was >>> created by pcpu_page_first_chunk() (i.e. percpu lives in vmalloc): >>> with the patch, the percpu IRQ stack pointers are translated >>> through the kernel page tables and crash starts cleanly. "bt" on >>> the panic task produces correct irq/overflow/standby stacks; >>> - a regular arm64 QEMU VM whose percpu first chunk was created by >>> pcpu_embed_first_chunk() (i.e. percpu lives in the linear map): >>> no regression, stack initialization still happens before any >>> "bt" command. >>> I'd like to get feedback on whether POST_VM is the preferred phase, >>> or whether the maintainers would rather guard the readmem() calls >>> inside the three helpers with a fallback (e.g. defer percpu >>> translation until later). I can send a formal patch with >>> Signed-off-by once the approach is agreed. >> >> I'm not confident if move it to POST_VM is safe enough so I'd prefer >> the second approach, still keep them in POST_GDB, and defer the >> callbasks later in a special case. >> >> Also please add more info in patch log, eg. kernel version, error >> msg, etc. > > Below are the kernel version, reproduction steps, and error log: > > Kernel: arm64 linux-6.6.0 with CONFIG_GENERIC_ARCH_NUMA=y, > CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y. > The same setup_per_cpu_areas() fallback path still exists in linux > mainline drivers/base/arch_numa.c, so this should reproduce on current > mainline as well. > On a QEMU arm64 VM, add "percpu_alloc=page" to the kernel cmdline to > force pcpu_page_first_chunk(), so percpu lives in vmalloc. > Then: > # makedumpfile -d 31 /proc/kcore vmcore > # ./crash vmlinux vmcore > crash log: > crash: seek error: kernel virtual address: ffff800100000050 type: "IRQ stack pointer" > crash: seek error: kernel virtual address: ffff800100021050 type: "IRQ stack pointer" > crash: seek error: kernel virtual address: ffff800100042050 type: "IRQ stack pointer" > crash: seek error: kernel virtual address: ffff800100063050 type: "IRQ stack pointer" > ... > KERNEL: vmlinux [TAINTED] > DUMPFILE: vmcore [PARTIAL DUMP] > CPUS: 4 > RELEASE: 6.6.0.aarch64 > MACHINE: aarch64 (unknown Mhz) > MEMORY: 4 GB > ... > > I tried the deferred approach you suggested, but it requires a new > flag, a new field, and a new helper, and only fixes the irq_stack_ptr > branch. So I looked at how x86_64 handles this. > x86_64_init() (x86_64.c:655-661) does not defer anything: it just > pre-fills vt->vmalloc_start and vt->kernel_pgd[] before the percpu > readmem in POST_GDB: > machdep->vmalloc_start = x86_64_vmalloc_start; > vt->vmalloc_start = machdep->vmalloc_start(); > machdep->init_kernel_pgd(); > ... > x86_64_per_cpu_init(); /* readmem percpu, translation works */ > > arm64 registers the same callbacks but never invokes them eagerly, > leaving vt->vmalloc_start == 0 until vm_init() runs in POST_VM. > So I'd suggest we follow the x86_64 approach and pre-fill on arm64 as well. > functions in POST_GDB: > case POST_GDB: > ... > if (!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) > arm64_recalc_KERNELPACMASK(); > vt->vmalloc_start = machdep->vmalloc_start(); /* NEW */ > machdep->init_kernel_pgd(); /* NEW */ > arm64_irq_stack_init(); > arm64_overflow_stack_init(); > arm64_stackframe_init(); > break; > One caveat: arm64_init_kernel_pgd() uses OFFSET(mm_struct_pgd), which > is only initialised by vm_init() in POST_VM. Calling it at POST_GDB > triggers "crash: invalid structure member offset: > mm_struct_pgd" because OFFSET() FATALs on INVALID_OFFSET. Adding an > INVALID_MEMBER() guard lets it fall through to the swapper_pg_dir > fallback; the later POST_VM call goes through the init_mm.pgd path as > usual: > if (!kernel_symbol_exists("init_mm") || > INVALID_MEMBER(mm_struct_pgd) || /* NEW */ > !readmem(symbol_value("init_mm") + OFFSET(mm_struct_pgd), ...)) { > if (kernel_symbol_exists("swapper_pg_dir")) > value = symbol_value("swapper_pg_dir"); > ... > } > > Patch > ----- > diff --git a/arm64.c b/arm64.c > --- a/arm64.c > +++ b/arm64.c > @@ -788,6 +788,9 @@ arm64_init(int when) > if(!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) > arm64_recalc_KERNELPACMASK(); > > + vt->vmalloc_start = machdep->vmalloc_start(); > + machdep->init_kernel_pgd(); > + > arm64_irq_stack_init(); > arm64_overflow_stack_init(); > arm64_stackframe_init(); > @@ -1894,7 +1897,8 @@ arm64_init_kernel_pgd(void) > ulong value; > > if (!kernel_symbol_exists("init_mm") || > + INVALID_MEMBER(mm_struct_pgd) || > !readmem(symbol_value("init_mm") + OFFSET(mm_struct_pgd), KVADDR, > &value, sizeof(void *), "init_mm.pgd", RETURN_ON_ERROR)) { > if (kernel_symbol_exists("swapper_pg_dir")) The above proposal copies some vm_init code and looks hacky, looking again at the code, probably moving the irq stack init to POST_VM looks better. I do not have arm64 hardware to test. To ensure no regressions, could you do more tests with live debugging (/proc/kcore) and the crash --minimal?