Hi Dave, Thanks for the feedback. On Fri, Sep 11, 2026 at 18:53 +0000, Dave Young wrote: > HI Haixiang, > > Thanks for reporting the issue. > > On Thu, Sep 10, 2026 at 09:01:42AM +0000, chenhaixiang (A) wrote: >> Subject: [BUG/RFC] arm64: stack init functions fail in POST_GDB when > percpu data is in vmalloc area. >> Hi, >> While running crash on a large arm64 NUMA system, crash fails to >> initialize the arm64 stack subsystems during arm64_init(POST_GDB) and >> aborts startup with SEEK_ERROR on every percpu IRQ stack pointer. >> Environment >> ----------- >> - architecture : aarch64 (64K pages, CONFIG_ARM64_64K_PAGES=y) >> - kernel : 6.x, large NUMA, many CPUs >> - crash : current master (arm64.c) >> Background on percpu allocation on this machine >> ----------------------------------------------- >> On this arm64 machine the kernel's first percpu chunk does NOT live in >> the linear map, it lives in the vmalloc area. The arm64 percpu setup >> path in drivers/base/arch_numa.c:setup_per_cpu_areas() first tries >> pcpu_embed_first_chunk() and, when that fails, falls back to >> pcpu_page_first_chunk() >> Observed symptom / reproduction log >> ----------------------------------- >> With crash started on the vmcore, the percpu IRQ stack pointers are all in > the >> ffff800100xxxxxx range, which is the vmalloc area on this kernel. >> Because vt->vmalloc_start is still 0 at POST_GDB, arm64_kvtop() takes >> the early "VTOP()" shortcut and returns a bogus physical address far >> beyond max_mapnr, so read_diskdump() reports SEEK_ERROR: >> ... >> <readmem: ffff800100000050, KVADDR, "IRQ stack pointer", 8, (ROE), > 1bfe2ef0> >> <read_diskdump: addr: ffff800100000050 paddr: 800100000050 cnt: 8> >> read_diskdump: SEEK_ERROR: paddr/pfn: 800100000050/800100000 > max_mapnr: 60c000000 >> crash: seek error: kernel virtual address: ffff800100000050 type: "IRQ > stack pointer" >> <readmem: ffff800100021050, KVADDR, "IRQ stack pointer", 8, (ROE), > 1bfe2ef8> >> <read_diskdump: addr: ffff800100021050 paddr: 800100021050 cnt: 8> >> read_diskdump: SEEK_ERROR: paddr/pfn: 800100021050/800100021 > max_mapnr: 60c000000 >> crash: seek error: kernel virtual address: ffff800100021050 type: "IRQ > stack pointer" >> <readmem: ffff800100042050, KVADDR, "IRQ stack pointer", 8, (ROE), > 1bfe2f00> >> <read_diskdump: addr: ffff800100042050 paddr: 800100042050 cnt: 8> >> read_diskdump: SEEK_ERROR: paddr/pfn: 800100042050/800100042 > max_mapnr: 60c000000 >> crash: seek error: kernel virtual address: ffff800100042050 type: "IRQ > stack pointer" >> <readmem: ffff800100063050, KVADDR, "IRQ stack pointer", 8, (ROE), > 1bfe2ef08> >> <read_diskdump: addr: ffff800100063050 paddr: 800100063050 cnt: 8> >> read_diskdump: SEEK_ERROR: paddr/pfn: 800100063050/800100063 > max_mapnr: 60c000000 >> crash: seek error: kernel virtual address: ffff800100063050 type: "IRQ > stack pointer" >> ... (one SEEK_ERROR per CPU) >> Note the wrong physical addresses (800100xxxxxx) come from a direct >> linear-map style translation (addr - PAGE_OFFSET) which is only valid >> for the linear map, not for vmalloc-backed percpu memory. >> Root cause >> ---------- >> In arm64_init() the three stack initializers are invoked in the >> POST_GDB phase: >> case POST_GDB: >> ... >> if (!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) >> arm64_recalc_KERNELPACMASK(); >> arm64_irq_stack_init(); /* <-- readmem on percpu var */ >> arm64_overflow_stack_init(); /* <-- readmem on percpu var */ >> arm64_stackframe_init(); >> break; >> These helpers end up calling readmem() on percpu addresses that, on >> the machine described above, live in the vmalloc range. The >> translation goes through arm64_kvtop() (arm64.c:1963), which for any >> vmalloc address requires: >> 1. vt->vmalloc_start != 0 -> to reach the IS_VMALLOC_ADDR() branch >> 2. vt->kernel_pgd[0] -> to walk the kernel page tables >> Both are populated by vm_init(), which runs in the POST_VM phase -- >> i.e. *after* POST_GDB. As a result, in arm64_kvtop() we hit: >> if (!vt->vmalloc_start) { >> *paddr = VTOP(kvaddr); /* wrong for vmalloc addresses */ >> return TRUE; >> } >> and the returned physical address is bogus, so the subsequent >> readmem() fails with SEEK_ERROR (as shown in the log above: paddr >> 800100xxxxxx is the linear-map translation of a vmalloc address, and >> it falls well beyond max_mapnr 0x60c000000). >> Proposed fix >> ------------ >> Move the three stack initializers from POST_GDB to POST_VM, so that >> vt->vmalloc_start and vt->kernel_pgd are guaranteed to be ready >> before any readmem() that may touch vmalloc-backed percpu data. >> diff --git a/arm64.c b/arm64.c >> index d177092..654b972 100644 >> --- a/arm64.c >> +++ b/arm64.c >> @@ -787,7 +787,9 @@ arm64_init(int when) >> */ >> if(!machdep->machspec- >> CONFIG_ARM64_KERNELPACMASK) >> arm64_recalc_KERNELPACMASK(); >> + break; >> >> + case POST_VM: >> arm64_irq_stack_init(); >> arm64_overflow_stack_init(); >> arm64_stackframe_init(); >> Testing >> ------- >> Verified on: >> - the failing large-NUMA arm64 system whose percpu first chunk was >> created by pcpu_page_first_chunk() (i.e. percpu lives in vmalloc): >> with the patch, the percpu IRQ stack pointers are translated >> through the kernel page tables and crash starts cleanly. "bt" on >> the panic task produces correct irq/overflow/standby stacks; >> - a regular arm64 QEMU VM whose percpu first chunk was created by >> pcpu_embed_first_chunk() (i.e. percpu lives in the linear map): >> no regression, stack initialization still happens before any >> "bt" command. >> I'd like to get feedback on whether POST_VM is the preferred phase, >> or whether the maintainers would rather guard the readmem() calls >> inside the three helpers with a fallback (e.g. defer percpu >> translation until later). I can send a formal patch with >> Signed-off-by once the approach is agreed. > > I'm not confident if move it to POST_VM is safe enough so I'd prefer > the second approach, still keep them in POST_GDB, and defer the > callbasks later in a special case. > > Also please add more info in patch log, eg. kernel version, error msg, > etc. Below are the kernel version, reproduction steps, and error log: Kernel: arm64 linux-6.6.0 with CONFIG_GENERIC_ARCH_NUMA=y, CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y. The same setup_per_cpu_areas() fallback path still exists in linux mainline drivers/base/arch_numa.c, so this should reproduce on current mainline as well. On a QEMU arm64 VM, add "percpu_alloc=page" to the kernel cmdline to force pcpu_page_first_chunk(), so percpu lives in vmalloc. Then: # makedumpfile -d 31 /proc/kcore vmcore # ./crash vmlinux vmcore crash log: crash: seek error: kernel virtual address: ffff800100000050 type: "IRQ stack pointer" crash: seek error: kernel virtual address: ffff800100021050 type: "IRQ stack pointer" crash: seek error: kernel virtual address: ffff800100042050 type: "IRQ stack pointer" crash: seek error: kernel virtual address: ffff800100063050 type: "IRQ stack pointer" ... KERNEL: vmlinux [TAINTED] DUMPFILE: vmcore [PARTIAL DUMP] CPUS: 4 RELEASE: 6.6.0.aarch64 MACHINE: aarch64 (unknown Mhz) MEMORY: 4 GB ... I tried the deferred approach you suggested, but it requires a new flag, a new field, and a new helper, and only fixes the irq_stack_ptr branch. So I looked at how x86_64 handles this. x86_64_init() (x86_64.c:655-661) does not defer anything: it just pre-fills vt->vmalloc_start and vt->kernel_pgd[] before the percpu readmem in POST_GDB: machdep->vmalloc_start = x86_64_vmalloc_start; vt->vmalloc_start = machdep->vmalloc_start(); machdep->init_kernel_pgd(); ... x86_64_per_cpu_init(); /* readmem percpu, translation works */ arm64 registers the same callbacks but never invokes them eagerly, leaving vt->vmalloc_start == 0 until vm_init() runs in POST_VM. So I'd suggest we follow the x86_64 approach and pre-fill on arm64 as well. functions in POST_GDB: case POST_GDB: ... if (!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) arm64_recalc_KERNELPACMASK(); vt->vmalloc_start = machdep->vmalloc_start(); /* NEW */ machdep->init_kernel_pgd(); /* NEW */ arm64_irq_stack_init(); arm64_overflow_stack_init(); arm64_stackframe_init(); break; One caveat: arm64_init_kernel_pgd() uses OFFSET(mm_struct_pgd), which is only initialised by vm_init() in POST_VM. Calling it at POST_GDB triggers "crash: invalid structure member offset: mm_struct_pgd" because OFFSET() FATALs on INVALID_OFFSET. Adding an INVALID_MEMBER() guard lets it fall through to the swapper_pg_dir fallback; the later POST_VM call goes through the init_mm.pgd path as usual: if (!kernel_symbol_exists("init_mm") || INVALID_MEMBER(mm_struct_pgd) || /* NEW */ !readmem(symbol_value("init_mm") + OFFSET(mm_struct_pgd), ...)) { if (kernel_symbol_exists("swapper_pg_dir")) value = symbol_value("swapper_pg_dir"); ... } Patch ----- diff --git a/arm64.c b/arm64.c --- a/arm64.c +++ b/arm64.c @@ -788,6 +788,9 @@ arm64_init(int when) if(!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) arm64_recalc_KERNELPACMASK(); + vt->vmalloc_start = machdep->vmalloc_start(); + machdep->init_kernel_pgd(); + arm64_irq_stack_init(); arm64_overflow_stack_init(); arm64_stackframe_init(); @@ -1894,7 +1897,8 @@ arm64_init_kernel_pgd(void) ulong value; if (!kernel_symbol_exists("init_mm") || + INVALID_MEMBER(mm_struct_pgd) || !readmem(symbol_value("init_mm") + OFFSET(mm_struct_pgd), KVADDR, &value, sizeof(void *), "init_mm.pgd", RETURN_ON_ERROR)) { if (kernel_symbol_exists("swapper_pg_dir"))