[Crash-utility] Re: [PATCH v5 0/5] Improve stack unwind on ppc64

Hi Lianbo, On Fri, Jan 05, 2024 at 01:53:53PM +0800, Lianbo Jiang wrote: >> ...<snip>... >> >> Q. Is it harmless ? >> >> Yes, all unwinders which were tried before chosing the correct one, are expected >> to fail. >> >> About address being unavailable, that also is harmless because: >> 1. The functions instructions were freed by the kernel, so the data at those >> address being missing is expected >> 2. Even GDB is okay with a read failing, in this case, tramp_frame_sniffer >> simply returns if it's unable to get an address. >> >> ------------ >> >> Btw, this only happens with CPU0 WHEN it has idle task. Since unlike other CPUs >> idle tasks, CPU 0's idle task is called during this initialisation sequence, >> thus it's backtrace having very early init functions. > It's true. >> Since this is expected and harmless (gdb handles such cases), I am thinking to >> ignore this warning when called by gdb, what do you suggest ? > If this does happen and it's rare, it might be good to have a warning for > users. > > At least it reflects the actual situation, although the warning output looks > not very friendly. So I would suggest still leaving it there for the time > being. > Yes, the warning is not very friendly but it does tell that the page is excluded. >> i) Add flag "IGNORE_PAGE_EXCLUDED" to 'readmem' call when done by gdb, and >> only print the error if this is true: >> if ((error_handle & IGNORE_PAGE_EXCLUDED) && (CRASHDEBUG(0)) >> ... >> This way, it will still be visible with -d 1 and above debug levels. >> ii) Append 'quiet' to the 'error_handle': >> This way we don't have to introduce a flag, but it might skip showing other >> warnings/errors which might be of interest >> >> Other than this, there was a bug introduced in V5, where default thread is still >> thread 1 (CPU 0) at initialisation, which has been fixed, and will also fix the > Sounds good. So for now there's no change needed to the 'page excluded' warnings ?

>> 'unavailable' frame being shown. Though it's funny that bug helped in a way to >> uncover these warnings, this would have hit in earlier versions also. > Indeed, it can be triggered conditionally. > > I have no other issues, thank you for the great job, Aditya. Thanks, Aditya Gupta > > Thanks > > Lianbo > >> Thanks, >> Aditya Gupta >> >>> crash> gdb frame >>> #0  <unavailable> in ?? () >>> crash> set gdb on >>> gdb: on >>> gdb> bt >>> #0  <unavailable> in ?? () >>> #1  0xc0000000000f570c in plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:111 >>> #2  0xc000000001004dd8 in cede_processor () at >>> ./arch/powerpc/include/asm/plpar_wrappers.h:37 >>> #3  check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83 >>> #4  0xc000000001005000 in shared_cede_loop (dev=<optimized out>, >>> drv=<optimized out>, index=<optimized out>) at >>> drivers/cpuidle/cpuidle-pseries.c:256 >>> #5  0xc000000001004498 in cpuidle_enter_state >>> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558 >>> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267 >>> #6  0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558 >>> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at >>> drivers/cpuidle/cpuidle.c:388 >>> #7  0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>, >>> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>, >>> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at >>> kernel/sched/idle.c:134 >>> #8  0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215 >>> #9  0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282 >>> #10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at >>> kernel/sched/idle.c:380 >>> #11 0xc000000000011030 in rest_init () at init/main.c:730 >>> #12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827 >>> #13 0xc0000000020051ec in start_kernel () at init/main.c:1072 >>> #14 0xc00000000000e788 in start_here_common () at >>> arch/powerpc/kernel/head_64.S:1039 >>> gdb> info threads >>>   Id   Target Id         Frame >>> * 1    CPU 0             plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>>   2    CPU 1             plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>>   3    CPU 2             plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>>   4    CPU 3             plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>>   5    CPU 4             plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>>   6    CPU 5             0xc00000000028b5e8 in crash_setup_regs >>> (oldregs=<optimized out>, newregs=0xc00000005d3e7958) at >>> ./arch/powerpc/include/asm/kexec.h:69 >>>   7    CPU 6             plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>>   8    CPU 7             plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>> gdb: page excluded: kernel virtual address: c000000002004c80 type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c000000002004c7c type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c000000002004c78 type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c000000002004c80 type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c000000002004c7c type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c000000002004c78 type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c0000000020051ec type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c0000000020051e8 type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c0000000020051e4 type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c0000000020051ec type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c0000000020051e8 type: >>> "gdb_readmem_callback" >>> gdb: page excluded: kernel virtual address: c0000000020051e4 type: >>> "gdb_readmem_callback" >>> >>> gdb> thread 1 >>> [Switching to thread 1 (CPU 0)] >>> #0  plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>> 114             li      r4,0 >>> gdb> bt >>> #0  plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>> #1  0xc000000001004dd8 in cede_processor () at >>> ./arch/powerpc/include/asm/plpar_wrappers.h:37 >>> #2  check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83 >>> #3  0xc000000001005000 in shared_cede_loop (dev=<optimized out>, >>> drv=<optimized out>, index=<optimized out>) at >>> drivers/cpuidle/cpuidle-pseries.c:256 >>> #4  0xc000000001004498 in cpuidle_enter_state >>> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558 >>> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267 >>> #5  0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558 >>> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at >>> drivers/cpuidle/cpuidle.c:388 >>> #6  0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>, >>> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>, >>> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at >>> kernel/sched/idle.c:134 >>> #7  0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215 >>> #8  0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282 >>> #9  0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at >>> kernel/sched/idle.c:380 >>> #10 0xc000000000011030 in rest_init () at init/main.c:730 >>> gdb: page excluded: kernel virtual address: c000000002004c80 type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c000000002004c7c type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c000000002004c78 type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c000000002004c80 type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c000000002004c7c type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c000000002004c78 type: >>> "gdb_readmem callback" >>> #11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827 >>> gdb: page excluded: kernel virtual address: c0000000020051ec type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c0000000020051e8 type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c0000000020051e4 type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c0000000020051ec type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c0000000020051e8 type: >>> "gdb_readmem callback" >>> gdb: page excluded: kernel virtual address: c0000000020051e4 type: >>> "gdb_readmem callback" >>> #12 0xc0000000020051ec in start_kernel () at init/main.c:1072 >>> #13 0xc00000000000e788 in start_here_common () at >>> arch/powerpc/kernel/head_64.S:1039 >>> gdb> thread 3 >>> [Switching to thread 3 (CPU 2)] >>> #0  plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>> 114             li      r4,0 >>> gdb> bt >>> #0  plpar_hcall_norets_notrace () at >>> arch/powerpc/platforms/pseries/hvCall.S:114 >>> #1  0xc000000001004dd8 in cede_processor () at >>> ./arch/powerpc/include/asm/plpar_wrappers.h:37 >>> #2  check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83 >>> #3  0xc000000001005000 in shared_cede_loop (dev=<optimized out>, >>> drv=<optimized out>, index=<optimized out>) at >>> drivers/cpuidle/cpuidle-pseries.c:256 >>> #4  0xc000000001004498 in cpuidle_enter_state >>> (dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558 >>> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267 >>> #5  0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558 >>> <pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized out>) at >>> drivers/cpuidle/cpuidle.c:388 >>> #6  0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>, >>> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>, >>> dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at >>> kernel/sched/idle.c:134 >>> #7  0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215 >>> #8  0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282 >>> #9  0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at >>> kernel/sched/idle.c:380 >>> #10 0xc00000000005f048 in start_secondary (unused=<optimized out>) at >>> arch/powerpc/kernel/smp.c:1680 >>> #11 0xc00000000000e058 in start_secondary_prolog () at >>> arch/powerpc/kernel/head_64.S:885 >>> gdb> >>> >>> >>> Could you please check it again? Or am I missing anything? I did the test >>> based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b). >>> >>> BTW: I did not see the similar issues after applying the v4 patch set. >>> >>> >>> Thanks >>> >>> Lianbo >>> >>>> Known Issues: >>>> ============= >>>> >>>> 1. In gdb mode, 'bt' might fail to show backtrace in few vmcores collected >>>> from older kernels. This is a known issue due to register mismatch, and >>>> its fix has been merged upstream: >>>> >>>> This can also cause some 'invalid kernel virtual address' errors during gdb >>>> unwinding the stack registers >>>> >>>> Commit: https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581... >>>> >>>> Fixing GDB passthroughs on other architectures >>>> ============================================== >>>> >>>> Much of the work for making gdb passthroughs like 'gdb bt', 'gdb >>>> thread', 'gdb info locals' etc. has been done by the patches introducing >>>> 'machdep->get_cpu_reg' and this series fixing some issues in that. >>>> >>>> Other architectures should be able to fix these gdb functionalities by >>>> simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'. >>>> >>>> The reasoning behind that has been explained with a diagram in commit >>>> description of patch #1 >>>> >>>> I will assist with my findings/observations fixing it on ppc64 whenever needed. >>>> >>>> Changelog: >>>> ========== >>>> >>>> V5: >>>> + changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable >>>> code >>>> + changes in patch #3: fixed typo 'ppc64_renum' instead of 'ppc64_regnum', >>>> remove unneeded if condition >>>> + changes in patch #5: implement refresh regcache on per thread, instead of all >>>> threads at once >>>> >>>> V4: >>>> + fix segmentation fault in live debugging (change in patch #1) >>>> + mention live debugging not supported in cover letter and patch #1 >>>> + fixed some checkpatch warnings (change in patch #5) >>>> >>>> V3: >>>> + default gdb thread will be the crashing thread, instead of being >>>> thread '0' >>>> + synchronise crash cpu and gdb thread context >>>> + fix bug in gdb_interface, that replaced gdb's output stream, losing >>>> output in some cases, such as info threads and extra output in info >>>> variables >>>> + fix 'info threads' >>>> >>>> RFC V2: >>>> - removed patch implementing 'frame', 'up', 'down' in crash >>>> - updated the cover letter by removing the mention of those commands other >>>> than the respective gdb passthrough >>>> >>>> Aditya Gupta (5): >>>> ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg >>>> remove 'frame' from prohibited commands list >>>> synchronise cpu context changes between crash/gdb >>>> fix gdb_interface: restore gdb's output streams at end of >>>> gdb_interface >>>> fix 'info threads' command >>>> >>>> crash_target.c | 44 ++++++++++++++++ >>>> defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++- >>>> gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++- >>>> gdb_interface.c | 2 +- >>>> kernel.c | 47 +++++++++++++++-- >>>> ppc64.c | 95 +++++++++++++++++++++++++++++++++-- >>>> task.c | 14 ++++++ >>>> tools.c | 2 +- >>>> 8 files changed, 434 insertions(+), 10 deletions(-) >>>>