[Crash-utility] Re: Subject: [BUG/RFC] arm64: stack init functions fail in POST_GDB when percpu data is in vmalloc area

Subject: [BUG/RFC] arm64: stack init functions fail in POST_GDB when percpu data is in vmalloc area. Hi, While running crash on a large arm64 NUMA system, crash fails to initialize the arm64 stack subsystems during arm64_init(POST_GDB) and aborts startup with SEEK_ERROR on every percpu IRQ stack pointer. Environment ----------- - architecture : aarch64 (64K pages, CONFIG_ARM64_64K_PAGES=y) - kernel : 6.x, large NUMA, many CPUs - crash : current master (arm64.c) Background on percpu allocation on this machine ----------------------------------------------- On this arm64 machine the kernel's first percpu chunk does NOT live in the linear map, it lives in the vmalloc area. The arm64 percpu setup path in drivers/base/arch_numa.c:setup_per_cpu_areas() first tries pcpu_embed_first_chunk() and, when that fails, falls back to pcpu_page_first_chunk() Observed symptom / reproduction log ----------------------------------- With crash started on the vmcore, the percpu IRQ stack pointers are all in the ffff800100xxxxxx range, which is the vmalloc area on this kernel. Because vt->vmalloc_start is still 0 at POST_GDB, arm64_kvtop() takes the early "VTOP()" shortcut and returns a bogus physical address far beyond max_mapnr, so read_diskdump() reports SEEK_ERROR: ... <readmem: ffff800100000050, KVADDR, "IRQ stack pointer", 8, (ROE), 1bfe2ef0> <read_diskdump: addr: ffff800100000050 paddr: 800100000050 cnt: 8> read_diskdump: SEEK_ERROR: paddr/pfn: 800100000050/800100000 max_mapnr: 60c000000 crash: seek error: kernel virtual address: ffff800100000050 type: "IRQ stack pointer" <readmem: ffff800100021050, KVADDR, "IRQ stack pointer", 8, (ROE), 1bfe2ef8> <read_diskdump: addr: ffff800100021050 paddr: 800100021050 cnt: 8> read_diskdump: SEEK_ERROR: paddr/pfn: 800100021050/800100021 max_mapnr: 60c000000 crash: seek error: kernel virtual address: ffff800100021050 type: "IRQ stack pointer" <readmem: ffff800100042050, KVADDR, "IRQ stack pointer", 8, (ROE), 1bfe2f00> <read_diskdump: addr: ffff800100042050 paddr: 800100042050 cnt: 8> read_diskdump: SEEK_ERROR: paddr/pfn: 800100042050/800100042 max_mapnr: 60c000000 crash: seek error: kernel virtual address: ffff800100042050 type: "IRQ stack pointer" <readmem: ffff800100063050, KVADDR, "IRQ stack pointer", 8, (ROE), 1bfe2ef08> <read_diskdump: addr: ffff800100063050 paddr: 800100063050 cnt: 8> read_diskdump: SEEK_ERROR: paddr/pfn: 800100063050/800100063 max_mapnr: 60c000000 crash: seek error: kernel virtual address: ffff800100063050 type: "IRQ stack pointer" ... (one SEEK_ERROR per CPU) Note the wrong physical addresses (800100xxxxxx) come from a direct linear-map style translation (addr - PAGE_OFFSET) which is only valid for the linear map, not for vmalloc-backed percpu memory. Root cause ---------- In arm64_init() the three stack initializers are invoked in the POST_GDB phase: case POST_GDB: ... if (!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) arm64_recalc_KERNELPACMASK(); arm64_irq_stack_init(); /* <-- readmem on percpu var */ arm64_overflow_stack_init(); /* <-- readmem on percpu var */ arm64_stackframe_init(); break; These helpers end up calling readmem() on percpu addresses that, on the machine described above, live in the vmalloc range. The translation goes through arm64_kvtop() (arm64.c:1963), which for any vmalloc address requires: 1. vt->vmalloc_start != 0 -> to reach the IS_VMALLOC_ADDR() branch 2. vt->kernel_pgd[0] -> to walk the kernel page tables Both are populated by vm_init(), which runs in the POST_VM phase -- i.e. *after* POST_GDB. As a result, in arm64_kvtop() we hit: if (!vt->vmalloc_start) { *paddr = VTOP(kvaddr); /* wrong for vmalloc addresses */ return TRUE; } and the returned physical address is bogus, so the subsequent readmem() fails with SEEK_ERROR (as shown in the log above: paddr 800100xxxxxx is the linear-map translation of a vmalloc address, and it falls well beyond max_mapnr 0x60c000000). Proposed fix ------------ Move the three stack initializers from POST_GDB to POST_VM, so that vt->vmalloc_start and vt->kernel_pgd are guaranteed to be ready before any readmem() that may touch vmalloc-backed percpu data. diff --git a/arm64.c b/arm64.c index d177092..654b972 100644 --- a/arm64.c +++ b/arm64.c @@ -787,7 +787,9 @@ arm64_init(int when) */ if(!machdep->machspec->CONFIG_ARM64_KERNELPACMASK) arm64_recalc_KERNELPACMASK(); + break; + case POST_VM: arm64_irq_stack_init(); arm64_overflow_stack_init(); arm64_stackframe_init(); Testing ------- Verified on: - the failing large-NUMA arm64 system whose percpu first chunk was created by pcpu_page_first_chunk() (i.e. percpu lives in vmalloc): with the patch, the percpu IRQ stack pointers are translated through the kernel page tables and crash starts cleanly. "bt" on the panic task produces correct irq/overflow/standby stacks; - a regular arm64 QEMU VM whose percpu first chunk was created by pcpu_embed_first_chunk() (i.e. percpu lives in the linear map): no regression, stack initialization still happens before any "bt" command. I'd like to get feedback on whether POST_VM is the preferred phase, or whether the maintainers would rather guard the readmem() calls inside the three helpers with a fallback (e.g. defer percpu translation until later). I can send a formal patch with Signed-off-by once the approach is agreed.