Hi lianbo,
On Wed, Sep 11, 2024 at 2:26 PM lijiang <lijiang@redhat.com> wrote:
>
> Hi, Tao
>
> Thank you for the update.
>
> The following patch is a regression issue, so I tend to discuss it as a separate patch.
> [PATCH v7 01/15] Fix the regression of cpumask_t for xen hyper
Can you also post v2 for this one? I have two comments about it:
[1] is it possible to not introduce the code related to hyper to a common module such as tools.c?
[2] for IA64 arch, I saw the machdep->get_irq_affinity = generic_get_irq_affinity is registered (see the ia64_init())
>
> In addition, I found another issue in my tests(on ppc64le), the gdb bt can display the back trace for the panic task, but when I switch to another task, the gdb bt can not display the back trace:
>
> crash> gdb bt
> #0 0xc0000000002bde04 in crash_setup_regs (newregs=0xc00000003264b858, oldregs=0x0) at ./arch/powerpc/include/asm/kexec.h:133
> #1 0xc0000000002be4f8 in __crash_kexec (regs=0x0) at kernel/crash_core.c:122
> #2 0xc00000000016c254 in panic (fmt=0xc0000000015eef20 "sysrq triggered crash\n") at kernel/panic.c:373
> #3 0xc000000000a708b8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:154
> #4 0xc000000000a713d4 in __handle_sysrq (key=key@entry=99 'c', check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:612
> #5 0xc000000000a71e94 in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1181
> #6 0xc00000000073260c in pde_write (pde=0xc00000000af9cc00, file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:334
> #7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:346
> #8 0xc00000000063c0e0 in vfs_write (file=0xc0000000092d2900, buf=0x10012536f60 <error: Cannot access memory at address 0x10012536f60>, count=2, pos=0xc00000003264bd30) at fs/read_write.c:588
> #9 vfs_write (file=0xc0000000092d2900, buf=0x10012536f60 <error: Cannot access memory at address 0x10012536f60>, count=<optimized out>, pos=0xc00000003264bd30) at fs/read_write.c:570
> #10 0xc00000000063c690 in ksys_write (fd=<optimized out>, buf=0x10012536f60 <error: Cannot access memory at address 0x10012536f60>, count=2) at fs/read_write.c:643
> #11 0xc000000000031a28 in system_call_exception (regs=0xc00000003264be80, r0=<optimized out>) at arch/powerpc/kernel/syscall.c:153
> #12 0xc00000000000d05c in system_call_vectored_common () at arch/powerpc/kernel/interrupt_64.S:198
>
> crash> ps
> PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 c000000002bda980 RU 0.0 0 0 [swapper/0]
> > 0 0 1 c000000003864c80 RU 0.0 0 0 [swapper/1]
> ...
> 8017 923 0 c000000043a20000 IN 0.2 22528 16256 sshd-session
> 8025 8017 6 c000000032271880 IN 0.1 22784 11840 sshd-session
> > 8026 8025 0 c000000043a26600 RU 0.1 9664 6208 bash
> ...
> 11645 2 3 c000000032264c80 ID 0.0 0 0 [kworker/u32:2]
> 11738 6188 2 c00000003811b180 IN 0.1 43520 9408 pickup
> 12326 2 0 c00000003226b280 ID 0.0 0 0 [kworker/0:1]
> 13112 6089 2 c00000000c809900 IN 0.0 7232 3456 sleep
>
> Let's take the "pickup" task as an example:
>
> crash> set 11738
> PID: 11738
> COMMAND: "pickup"
> TASK: c00000003811b180 [THREAD_INFO: c00000003811b180]
> CPU: 2
> STATE: TASK_INTERRUPTIBLE
>
> crash> gdb bt
> #0 0xc0000000a7f876a0 in ?? ()
> gdb: gdb request failed: bt
> crash> set gdb on
> gdb: on
> gdb> bt
> #0 0xc0000000a7f876a0 in ?? ()
> gdb>
There is a bug for ppc64 crash of newer version kernel. The code for
determining the address of pt_regs from stack is outdated, see the
following code from crash:
ppc64.c:get_ppc64_frame()
readmem(sp+STACK_FRAME_OVERHEAD, KVADDR, ®s, sizeof(struct
ppc64_pt_regs), "PPC64 pt_regs", FAULT_ON_ERROR);
The pt_regs is expected to be placed at sp+STACK_FRAME_OVERHEAD, aka sp+112.
However since kernel >= v6.2, the value is no longer appropriate:
linux kernel:arch/powerpc/kernel/process.c:copy_thread():
kregs = (struct pt_regs *)(sp + STACK_SWITCH_FRAME_REGS);
p->thread.ksp = sp;
linux kernel:arch/powerpc/include/asm/ptrace.h:
#ifdef CONFIG_PPC64_ELF_ABI_V2
#define STACK_FRAME_MIN_SIZE 32
STACK_SWITCH_FRAME_REGS (STACK_FRAME_MIN_SIZE + 16)
Good findings, Tao.
If we apply the change to crash, i.e:
readmem(sp+0x30, KVADDR, ®s, sizeof(struct ppc64_pt_regs), "PPC64
pt_regs", FAULT_ON_ERROR);
The stack unwinding can work as expected, you can have a test locally
to see if the above change works for you.
So this bug isn't related to the gdb stack unwinding support to me,
just a bug relating to a newer version of kernel.
Agree. For the [PATCH V7 02/15] -[PATCH V7 15/15]: Ack.
And I will put them in the merging queue, once the current issue gets resolved, we can merge them together. Otherwise it may not work on ppc64 arch.
BTW: another issue is observed when I do some tests, the "gdb bt" command does not work well on old kernels, furthermore it can not display any backtrace on the 2.6 kernels.
Given that, can we state on which kernel version gdb stack unwinding works well in the patch log? Or can it be improved in the next step?
case [1]:
crash> bt
PID: 0 TASK: ffffffff8cc10740 CPU: 0 COMMAND: "swapper/0"
#0 [fffffe0000008a10] machine_kexec at ffffffff8ba5176e
#1 [fffffe0000008a68] __crash_kexec at ffffffff8bb4776d
#2 [fffffe0000008b30] panic at ffffffff8baa6aa8
#3 [fffffe0000008bb8] watchdog_overflow_callback.cold.7 at ffffffff8bb79d94
#4 [fffffe0000008bc8] __perf_event_overflow at ffffffff8bbdc572
#5 [fffffe0000008bf8] intel_pmu_handle_irq at ffffffff8ba0cbc6
#6 [fffffe0000008e40] perf_event_nmi_handler at ffffffff8ba05ffd
#7 [fffffe0000008e58] nmi_handle at ffffffff8ba20663
#8 [fffffe0000008eb0] default_do_nmi at ffffffff8ba20a0e
#9 [fffffe0000008ed0] do_nmi at ffffffff8ba20bd2
#10 [fffffe0000008ef0] end_repeat_nmi at ffffffff8c4014d8
[exception RIP: cfb_imageblit+1102]
RIP: ffffffff8be751be RSP: ffff97587fa036e8 RFLAGS: 00000046
RAX: 0000000000000000 RBX: ffffa4c120650cf0 RCX: 0000000000000006
RDX: 000000000000000e RSI: 0000000000000000 RDI: ffffa4c120650c00
RBP: 0000000000000002 R8: ffff97587d15fb5b R9: 00000000ad55ad55
R10: 0000000000000000 R11: ffffa4c120650c04 R12: 0000000000000800
R13: 0000000000000003 R14: ffffffff8c899170 R15: ffff97587d15fb45
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#11 [ffff97587fa036e8] cfb_imageblit at ffffffff8be751be
#12 [ffff97587fa03750] drm_fb_helper_cfb_imageblit at ffffffffc04c0e12 [drm_kms_helper]
#13 [ffff97587fa03768] bit_putcs at ffffffff8be6f6f1
#14 [ffff97587fa03870] fbcon_putcs at ffffffff8be6baa9
#15 [ffff97587fa038c8] fbcon_redraw at ffffffff8be6bd1d
#16 [ffff97587fa03928] fbcon_scroll at ffffffff8be6d434
#17 [ffff97587fa03988] con_scroll at ffffffff8befe900
#18 [ffff97587fa039c0] lf at ffffffff8befe9b0
#19 [ffff97587fa039e8] vt_console_print at ffffffff8bf00816
...skipping...
#22 [ffff97587fa03af0] printk at ffffffff8bb09186
#23 [ffff97587fa03b50] irq_work_run_list at ffffffff8bbbca5d
#24 [ffff97587fa03b78] irq_work_run at ffffffff8bbbca94
#25 [ffff97587fa03b80] smp_irq_work_interrupt at ffffffff8c401e02
#26 [ffff97587fa03b90] irq_work_interrupt at ffffffff8c401bbf
#27 [ffff97587fa03ba0] irq_work_interrupt at ffffffff8c401bba
#28 [ffff97587fa03c28] update_group_capacity at ffffffff8bae4a2f
#29 [ffff97587fa03c80] find_busiest_group at ffffffff8bae4c6e
#30 [ffff97587fa03e10] load_balance at ffffffff8bae57eb
#31 [ffff97587fa03f00] rebalance_domains at ffffffff8bae62ca
#32 [ffff97587fa03f68] __softirqentry_text_start at ffffffff8c6000e3
#33 [ffff97587fa03fc8] irq_exit at ffffffff8baacaf5
#34 [ffff97587fa03fd8] smp_apic_timer_interrupt at ffffffff8c40241c
#35 [ffff97587fa03ff0] apic_timer_interrupt at ffffffff8c401a7f
--- <IRQ stack> ---
#36 [ffffffff8cc03d68] apic_timer_interrupt at ffffffff8c401a7f
[exception RIP: finish_task_switch+125]
RIP: ffffffff8bad131d RSP: ffffffff8cc03e10 RFLAGS: 00000246
RAX: ffff97587eb18000 RBX: ffffffff8cc10740 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff97587fa22940
RBP: ffffffff8cc03e38 R8: 000001e44b0102cc R9: 0000000000000002
R10: 0000000000000040 R11: 0000000000000000 R12: ffff97587fa22940
R13: ffff97587f3c7380 R14: ffff97587eb18000 R15: 0000000000000402
ORIG_RAX: ffffffffffffff13 CS: 0010 SS: 0018
#37 [ffffffff8cc03e40] __schedule at ffffffff8c20855b
#38 [ffffffff8cc03ea8] schedule_idle at ffffffff8c208dee
#39 [ffffffff8cc03eb0] do_idle at ffffffff8bad9fc5
#40 [ffffffff8cc03ef0] cpu_startup_entry at ffffffff8bada2af
#41 [ffffffff8cc03f10] start_kernel at ffffffff8d1b2055
#42 [ffffffff8cc03f50] secondary_startup_64 at ffffffff8ba000d5