2:52 a.m.
The Problem:
============
Currently crash is unable to show function arguments and local variables, as
gdb can do. And functionality for moving between frames ('up'/'down') is
not
working in crash.
Crash has 'gdb passthroughs' for things gdb can do, but the gdb passthroughs
'bt', 'frame', 'info locals', 'up', 'down' are not
working either, due to
gdb not getting the register values from `crash_target::fetch_registers`,
which then uses `machdep->get_cpu_reg`, which is not implemented for PPC64
Proposed Solution:
==================
Fix the gdb passthroughs by implementing "machdep->get_cpu_reg" for PPC64.
This way, "gdb mode in crash" will support this feature for both ELF and
kdump-compressed vmcore formats, while "gdb" would only have supported ELF
format
This way other features of 'gdb', such as seeing
backtraces/registers/variables/arguments/local variables, moving up and
down stack frames, can be used with any ppc64 vmcore, irrespective of
being ELF format or kdump-compressed format.
Note: This doesn't support live debugging on ppc64, since registers are not
available to be read
Implications on Architectures:
====================================
No architecture other than PPC64 has been affected, other than in case of
'frame' command
As mentioned in patch #2, since frame will not be prohibited, so it will print:
crash> frame
#0 <unavailable> in ?? ()
Instead of before prohibited message:
crash> frame
crash: prohibited gdb command: frame
Major change will be in 'gdb mode' on PPC64, that it will print the frames, and
local variables, instead of failing with errors showing no frame, or showing
that couldn't get PC, it will be able to give all this information.
Testing:
========
Git tree with this patch series applied:
https://github.com/adi-g15-ibm/crash/tree/stack-unwind-v5-smaller-gran
To test various gdb passthroughs:
(crash) set
(crash) set gdb on
gdb> thread
gdb> bt
gdb> info threads
gdb> info threads
gdb> info locals
gdb> info variables irq_rover_lock
gdb> info args
gdb> thread 2
gdb> set gdb off
(crash) set
(crash) set -c 6
(crash) gdb thread
(crash) bt
(crash) gdb bt
(crash) frame
(crash) up
(crash) down
(crash) info locals
Known Issues:
=============
1. In gdb mode, 'bt' might fail to show backtrace in few vmcores collected
from older kernels. This is a known issue due to register mismatch, and
its fix has been merged upstream:
This can also cause some 'invalid kernel virtual address' errors during gdb
unwinding the stack registers
Commit: https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
Fixing GDB passthroughs on other architectures
==============================================
Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
thread', 'gdb info locals' etc. has been done by the patches introducing
'machdep->get_cpu_reg' and this series fixing some issues in that.
Other architectures should be able to fix these gdb functionalities by
simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
The reasoning behind that has been explained with a diagram in commit
description of patch #1
I will assist with my findings/observations fixing it on ppc64 whenever needed.
Changelog:
==========
V5:
+ changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
code
+ changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
remove unneeded if condition
+ changes in patch #5: implement refresh regcache on per thread, instead of all
threads at once
V4:
+ fix segmentation fault in live debugging (change in patch #1)
+ mention live debugging not supported in cover letter and patch #1
+ fixed some checkpatch warnings (change in patch #5)
V3:
+ default gdb thread will be the crashing thread, instead of being
thread '0'
+ synchronise crash cpu and gdb thread context
+ fix bug in gdb_interface, that replaced gdb's output stream, losing
output in some cases, such as info threads and extra output in info
variables
+ fix 'info threads'
RFC V2:
- removed patch implementing 'frame', 'up', 'down' in crash
- updated the cover letter by removing the mention of those commands other
than the respective gdb passthrough
Aditya Gupta (5):
ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
remove 'frame' from prohibited commands list
synchronise cpu context changes between crash/gdb
fix gdb_interface: restore gdb's output streams at end of
gdb_interface
fix 'info threads' command
crash_target.c | 44 ++++++++++++++++
defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
gdb_interface.c | 2 +-
kernel.c | 47 +++++++++++++++--
ppc64.c | 95 +++++++++++++++++++++++++++++++++--
task.c | 14 ++++++
tools.c | 2 +-
8 files changed, 434 insertions(+), 10 deletions(-)
--
2.41.0
2:52 a.m.
New subject: [PATCH v5 1/5] ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
Currently, gdb passthroughs of 'bt', 'frame', 'up',
'down', 'info
locals' don't work. This is due to gdb not knowing the register values to
unwind the stack frames
Every gdb passthrough goes through `gdb_interface`. And then, gdb expects
`crash_target::fetch_registers` to give it the register values, which is
dependent on `machdep->get_cpu_reg` to read the register values for
specific architecture.
┌────────────────────────┐
gdb passthrough (eg. "bt") │ │
crash ─────────────────────────▶│ │
│ gdb_interface │
│ │
│ │
│ ┌────────────────────┐ │
fetch_registers │ │ │ │
crash_target◀────────────────────────┼─┤ gdb │ │
─────────────────────────┼▶│ │ │
Registers (SP,NIP, etc.)│ │ │ │
│ │ │ │
│ └────────────────────┘ │
└────────────────────────┘
Implement `machdep->get_cpu_reg` on PPC64, so that crash provides the
register values to gdb to unwind stack frames properly
With these changes, on powerpc, 'bt' command output in gdb mode, will look
like this:
gdb> bt
#0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>,
newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69
#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974
#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358
#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at
drivers/tty/sysrq.c:155
#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99,
check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602
#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>,
buf=<optimized out>, count=2, ppos=<optimized out>) at
drivers/tty/sysrq.c:1163
#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized
out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at
fs/proc/inode.c:340
#7 proc_reg_write (file=<optimized out>, buf=<optimized out>,
count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352
#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00,
buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>,
count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582
instead of earlier output without this patch:
gdb> bt
#0 <unavailable> in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Also, 'get_dumpfile_regs' has been introduced to get registers from
multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS'
has been introduced to tell helper functions to get registers, to not
print registers with every call to backtrace in gdb.
Note: This feature to support GDB unwinding doesn't support live debugging
Signed-off-by: Aditya Gupta <adityag(a)linux.ibm.com>
---
defs.h | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
kernel.c | 33 +++++++++++++++
ppc64.c | 105 +++++++++++++++++++++++++++++++++++++++++++++--
3 files changed, 257 insertions(+), 3 deletions(-)
diff --git a/defs.h b/defs.h
index 5218a94fe4a4..615f3a37935a 100644
--- a/defs.h
+++ b/defs.h
@@ -6023,6 +6023,7 @@ int load_module_symbols_helper(char *);
void unlink_module(struct load_module *);
int check_specified_module_tree(char *, char *);
int is_system_call(char *, ulong);
+void get_dumpfile_regs(struct bt_info*, ulong*, ulong*);
void generic_dump_irq(int);
void generic_get_irq_affinity(int);
void generic_show_interrupts(int, ulong *);
@@ -6121,6 +6122,7 @@ ulong cpu_map_addr(const char *type);
#define BT_REGS_NOT_FOUND (0x4000000000000ULL)
#define BT_OVERFLOW_STACK (0x8000000000000ULL)
#define BT_SKIP_IDLE (0x10000000000000ULL)
+#define BT_NO_PRINT_REGS (0x20000000000000ULL)
#define BT_SYMBOL_OFFSET (BT_SYMBOLIC_ARGS)
#define BT_REF_HEXVAL (0x1)
@@ -7854,4 +7856,124 @@ enum x86_64_regnum {
LAST_REGNUM
};
+/*
+ * Register numbers to make crash_target->fetch_registers()
+ * ---> machdep->get_cpu_reg() work properly.
+ *
+ * These register numbers and names are given according to output of
+ * `rs6000_register_name`, because that is what was being used by
+ * crash_target::fetch_registers in case of PPC64
+ */
+enum ppc64_regnum {
+ PPC64_R0_REGNUM = 0,
+ PPC64_R1_REGNUM,
+ PPC64_R2_REGNUM,
+ PPC64_R3_REGNUM,
+ PPC64_R4_REGNUM,
+ PPC64_R5_REGNUM,
+ PPC64_R6_REGNUM,
+ PPC64_R7_REGNUM,
+ PPC64_R8_REGNUM,
+ PPC64_R9_REGNUM,
+ PPC64_R10_REGNUM,
+ PPC64_R11_REGNUM,
+ PPC64_R12_REGNUM,
+ PPC64_R13_REGNUM,
+ PPC64_R14_REGNUM,
+ PPC64_R15_REGNUM,
+ PPC64_R16_REGNUM,
+ PPC64_R17_REGNUM,
+ PPC64_R18_REGNUM,
+ PPC64_R19_REGNUM,
+ PPC64_R20_REGNUM,
+ PPC64_R21_REGNUM,
+ PPC64_R22_REGNUM,
+ PPC64_R23_REGNUM,
+ PPC64_R24_REGNUM,
+ PPC64_R25_REGNUM,
+ PPC64_R26_REGNUM,
+ PPC64_R27_REGNUM,
+ PPC64_R28_REGNUM,
+ PPC64_R29_REGNUM,
+ PPC64_R30_REGNUM,
+ PPC64_R31_REGNUM,
+
+ PPC64_F0_REGNUM = 32,
+ PPC64_F1_REGNUM,
+ PPC64_F2_REGNUM,
+ PPC64_F3_REGNUM,
+ PPC64_F4_REGNUM,
+ PPC64_F5_REGNUM,
+ PPC64_F6_REGNUM,
+ PPC64_F7_REGNUM,
+ PPC64_F8_REGNUM,
+ PPC64_F9_REGNUM,
+ PPC64_F10_REGNUM,
+ PPC64_F11_REGNUM,
+ PPC64_F12_REGNUM,
+ PPC64_F13_REGNUM,
+ PPC64_F14_REGNUM,
+ PPC64_F15_REGNUM,
+ PPC64_F16_REGNUM,
+ PPC64_F17_REGNUM,
+ PPC64_F18_REGNUM,
+ PPC64_F19_REGNUM,
+ PPC64_F20_REGNUM,
+ PPC64_F21_REGNUM,
+ PPC64_F22_REGNUM,
+ PPC64_F23_REGNUM,
+ PPC64_F24_REGNUM,
+ PPC64_F25_REGNUM,
+ PPC64_F26_REGNUM,
+ PPC64_F27_REGNUM,
+ PPC64_F28_REGNUM,
+ PPC64_F29_REGNUM,
+ PPC64_F30_REGNUM,
+ PPC64_F31_REGNUM,
+
+ PPC64_PC_REGNUM = 64,
+ PPC64_MSR_REGNUM = 65,
+ PPC64_CR_REGNUM = 66,
+ PPC64_LR_REGNUM = 67,
+ PPC64_CTR_REGNUM = 68,
+ PPC64_XER_REGNUM = 69,
+ PPC64_FPSCR_REGNUM = 70,
+
+ PPC64_VR0_REGNUM = 106,
+ PPC64_VR1_REGNUM,
+ PPC64_VR2_REGNUM,
+ PPC64_VR3_REGNUM,
+ PPC64_VR4_REGNUM,
+ PPC64_VR5_REGNUM,
+ PPC64_VR6_REGNUM,
+ PPC64_VR7_REGNUM,
+ PPC64_VR8_REGNUM,
+ PPC64_VR9_REGNUM,
+ PPC64_VR10_REGNUM,
+ PPC64_VR11_REGNUM,
+ PPC64_VR12_REGNUM,
+ PPC64_VR13_REGNUM,
+ PPC64_VR14_REGNUM,
+ PPC64_VR15_REGNUM,
+ PPC64_VR16_REGNUM,
+ PPC64_VR17_REGNUM,
+ PPC64_VR18_REGNUM,
+ PPC64_VR19_REGNUM,
+ PPC64_VR20_REGNUM,
+ PPC64_VR21_REGNUM,
+ PPC64_VR22_REGNUM,
+ PPC64_VR23_REGNUM,
+ PPC64_VR24_REGNUM,
+ PPC64_VR25_REGNUM,
+ PPC64_VR26_REGNUM,
+ PPC64_VR27_REGNUM,
+ PPC64_VR28_REGNUM,
+ PPC64_VR29_REGNUM,
+ PPC64_VR30_REGNUM,
+ PPC64_VR31_REGNUM,
+
+ PPC64_VSCR_REGNUM = 138,
+ PPC64_VRSAVE_REGNU = 139
+};
+
#endif /* !GDB_COMMON */
diff --git a/kernel.c b/kernel.c
index 6dcf414693e6..52b7ba09f390 100644
--- a/kernel.c
+++ b/kernel.c
@@ -3533,6 +3533,39 @@ get_lkcd_regs(struct bt_info *bt, ulong *eip, ulong *esp)
machdep->get_stack_frame(bt, eip, esp);
}
+void
+get_dumpfile_regs(struct bt_info *bt, ulong *eip, ulong *esp)
+{
+ bt->flags |= BT_NO_PRINT_REGS;
+
+ if (NETDUMP_DUMPFILE())
+ get_netdump_regs(bt, eip, esp);
+ else if (KDUMP_DUMPFILE())
+ get_kdump_regs(bt, eip, esp);
+ else if (DISKDUMP_DUMPFILE())
+ get_diskdump_regs(bt, eip, esp);
+ else if (KVMDUMP_DUMPFILE())
+ get_kvmdump_regs(bt, eip, esp);
+ else if (LKCD_DUMPFILE())
+ get_lkcd_regs(bt, eip, esp);
+ else if (XENDUMP_DUMPFILE())
+ get_xendump_regs(bt, eip, esp);
+ else if (SADUMP_DUMPFILE())
+ get_sadump_regs(bt, eip, esp);
+ else if (VMSS_DUMPFILE())
+ get_vmware_vmss_regs(bt, eip, esp);
+ else if (REMOTE_PAUSED()) {
+ if (!is_task_active(bt->task) || !get_remote_regs(bt, eip, esp))
+ machdep->get_stack_frame(bt, eip, esp);
+ } else
+ machdep->get_stack_frame(bt, eip, esp);
+
+ bt->flags &= ~BT_NO_PRINT_REGS;
+
+ bt->instptr = *eip;
+ bt->stkptr = *esp;
+}
+
/*
* Store the head of the kernel module list for future use.
diff --git a/ppc64.c b/ppc64.c
index e8930a139e0d..ea4821d86b9e 100644
--- a/ppc64.c
+++ b/ppc64.c
@@ -55,6 +55,8 @@ static void ppc64_set_bt_emergency_stack(enum emergency_stack_type
type,
static char * ppc64_check_eframe(struct ppc64_pt_regs *);
static void ppc64_print_eframe(char *, struct ppc64_pt_regs *,
struct bt_info *);
+static int ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size,
+ void *value);
static void parse_cmdline_args(void);
static int ppc64_paca_percpu_offset_init(int);
static void ppc64_init_cpu_info(void);
@@ -704,6 +706,8 @@ ppc64_init(int when)
error(FATAL, "cannot malloc hwirqstack buffer space.");
}
+ machdep->get_cpu_reg = ppc64_get_cpu_reg;
+
ppc64_init_paca_info();
if (!machdep->hz) {
@@ -2501,6 +2505,99 @@ ppc64_print_eframe(char *efrm_str, struct ppc64_pt_regs *regs,
ppc64_print_nip_lr(regs, 1);
}
+static int
+ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size,
+ void *value)
+{
+ struct bt_info bt_info, bt_setup;
+ struct task_context *tc;
+ struct ppc64_pt_regs *pt_regs;
+ ulong ip, sp;
+
+ if (LIVE()) {
+ /* doesn't support reading registers in live dump */
+ return FALSE;
+ }
+
+ /* Currently only handling registers available in ppc64_pt_regs:
+ *
+ * 0-31: r0-r31
+ * 64: pc/nip
+ * 65: msr
+ *
+ * 67: lr
+ * 68: ctr
+ */
+ switch (regno) {
+ case PPC64_R0_REGNUM ... PPC64_R31_REGNUM:
+
+ case PPC64_PC_REGNUM:
+ case PPC64_MSR_REGNUM:
+ case PPC64_LR_REGNUM:
+ case PPC64_CTR_REGNUM:
+ break;
+
+ default:
+ // return false if we can't get that register
+ if (CRASHDEBUG(1))
+ error(WARNING, "unsupported register, regno=%d\n", regno);
+ return FALSE;
+ }
+
+ tc = CURRENT_CONTEXT();
+ BZERO(&bt_setup, sizeof(struct bt_info));
+ clone_bt_info(&bt_setup, &bt_info, tc);
+ fill_stackbuf(&bt_info);
+
+ // reusing the get_dumpfile_regs function to get pt regs structure
+ get_dumpfile_regs(&bt_info, &sp, &ip);
+ pt_regs = (struct ppc64_pt_regs *)bt_info.machdep;
+
+ if (!pt_regs) {
+ error(WARNING, "pt_regs not available for cpu %d\n", cpu);
+ return FALSE;
+ }
+
+ switch (regno) {
+ case PPC64_R0_REGNUM ... PPC64_R31_REGNUM:
+ if (size != sizeof(pt_regs->gpr[regno]))
+ return FALSE; // size mismatch
+
+ memcpy(value, &pt_regs->gpr[regno], size);
+ break;
+
+ case PPC64_PC_REGNUM:
+ if (size != sizeof(pt_regs->nip))
+ return FALSE; // size mismatch
+
+ memcpy(value, &pt_regs->nip, size);
+ break;
+
+ case PPC64_MSR_REGNUM:
+ if (size != sizeof(pt_regs->msr))
+ return FALSE; // size mismatch
+
+ memcpy(value, &pt_regs->msr, size);
+ break;
+
+ case PPC64_LR_REGNUM:
+ if (size != sizeof(pt_regs->link))
+ return FALSE; // size mismatch
+
+ memcpy(value, &pt_regs->link, size);
+ break;
+
+ case PPC64_CTR_REGNUM:
+ if (size != sizeof(pt_regs->ctr))
+ return FALSE; // size mismatch
+
+ memcpy(value, &pt_regs->ctr, size);
+ break;
+ }
+
+ return TRUE;
+}
+
/*
* For vmcore typically saved with KDump or FADump, get SP and IP values
* from the saved ptregs.
@@ -2613,9 +2710,11 @@ ppc64_get_dumpfile_stack_frame(struct bt_info *bt_in, ulong *nip,
ulong *ksp)
pt_regs = (struct ppc64_pt_regs *)bt->machdep;
ur_nip = pt_regs->nip;
ur_ksp = pt_regs->gpr[1];
- /* Print the collected regs for panic task. */
- ppc64_print_regs(pt_regs);
- ppc64_print_nip_lr(pt_regs, 1);
+ if (!(bt->flags & BT_NO_PRINT_REGS)) {
+ /* Print the collected regs for panic task. */
+ ppc64_print_regs(pt_regs);
+ ppc64_print_nip_lr(pt_regs, 1);
+ }
} else if ((pc->flags & KDUMP) ||
((pc->flags & DISKDUMP) &&
(*diskdump_flags & KDUMP_CMPRS_LOCAL))) {
--
2.43.0
2:52 a.m.
New subject: [PATCH v5 2/5] remove 'frame' from prohibited commands list
Currently, 'frame' is prohibited on every architecture, even in gdb mode.
Since the passthrough is working on few architectures (eg. ppc64, and
vmss x86_64), so remove it from prohibited list, so that 'frame' command
can be used
This has an implication on all architectures, that 'frame' now outputs
this in default mode:
```
crash> frame
#0 <unavailable> in ?? ()
```
instead of the before prohibited message
```
crash> frame
crash: prohibited gdb command: frame
```
Though, 'frame' should work in gdb mode for architectures that implement
'machdep->get_cpu_reg'
Signed-off-by: Aditya Gupta <adityag(a)linux.ibm.com>
---
gdb_interface.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/gdb_interface.c b/gdb_interface.c
index b14319c66147..8f99a0d7fd54 100644
--- a/gdb_interface.c
+++ b/gdb_interface.c
@@ -711,7 +711,7 @@ static char *prohibited_list[] = {
"watch", "rwatch", "awatch", "attach",
"continue", "c", "fg", "detach",
"finish", "handle", "interrupt", "jump",
"kill", "next", "nexti",
"signal", "step", "s", "stepi",
"target", "until", "delete",
- "clear", "disable", "enable", "condition",
"ignore", "frame", "catch",
+ "clear", "disable", "enable", "condition",
"ignore", "catch",
"tcatch", "return", "file", "exec-file",
"core-file", "symbol-file",
"load", "si", "ni", "shell", "sy",
NULL /* must be last */
--
2.43.0
2:52 a.m.
New subject: [PATCH v5 3/5] synchronise cpu context changes between crash/gdb
Currently, both crash context and gdb's current thread context were
pretty independent, and could be different, for example, crash commands
might be working on thread 6 (CPU 5), but GDB passthroughs will be
working on thread 2 (CPU 1).
This was not a problem earlier since interaction of crash and gdb was
not depending on current context for most part. But for gdb passthroughs
to work correctly, gdb needs register values from crash, which depend on
current context in crash.
Synchronise 'thread' command in gdb with 'set -c' command in crash.
1. crash -> gdb synchronisation:
Everytime crash's context changes, a helper is called to switch to
the thread on that CPU in gdb. The function has been implemented in
crash_target.c, since gdb functions are accessible inside
'crash_target.c', and the thread ID to CPU ID mapping is also done
by the crash_target, during initially registering the threads with
gdb. With this implementation, GDB's default thread initially also
changes to the crashing thread, so a switch to crashing thread
manually isn't required anymore
2. gdb -> crash synchronisation:
gdb has been patched to call 'set_cpu' whenever user switches to any
thread.
Signed-off-by: Aditya Gupta <adityag(a)linux.ibm.com>
---
crash_target.c | 24 ++++++++++++++++++++++++
defs.h | 3 +++
gdb-10.2.patch | 30 ++++++++++++++++++++++++++++++
kernel.c | 8 +++++++-
task.c | 4 +++-
5 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/crash_target.c b/crash_target.c
index 455480679741..a9d7eea8a80e 100644
--- a/crash_target.c
+++ b/crash_target.c
@@ -29,6 +29,8 @@ extern "C" int gdb_readmem_callback(unsigned long, void *,
int, int);
extern "C" int crash_get_nr_cpus(void);
extern "C" int crash_get_cpu_reg (int cpu, int regno, const char *regname,
int regsize, void *val);
+extern "C" int gdb_change_cpu_context (unsigned int cpu);
+extern "C" int set_cpu (int cpu);
/* The crash target. */
@@ -133,3 +135,25 @@ crash_target_init (void)
/* Now, set up the frame cache. */
reinit_frame_cache ();
}
+
+/*
+ * Change gdb's thread context to the thread on given CPU
+ **/
+extern "C" int
+gdb_change_cpu_context(unsigned int cpu)
+{
+ ptid_t ptid = ptid_t(CRASH_INFERIOR_PID, 0, cpu);
+ inferior *inf = current_inferior ();
+ thread_info *tp = find_thread_ptid (inf, ptid);
+
+ if (tp == nullptr)
+ return FALSE;
+
+ /* Making sure that crash's context is same */
+ set_cpu(cpu);
+
+ /* Switch to the thread */
+ switch_to_thread(tp);
+ return TRUE;
+}
+
diff --git a/defs.h b/defs.h
index 615f3a37935a..7f7a12753658 100644
--- a/defs.h
+++ b/defs.h
@@ -7976,4 +7976,7 @@ enum ppc64_regnum {
PPC64_VRSAVE_REGNU = 139
};
+/* crash_target.c */
+extern int gdb_change_cpu_context (unsigned int cpu);
+
#endif /* !GDB_COMMON */
diff --git a/gdb-10.2.patch b/gdb-10.2.patch
index 2f7d585105aa..8c1b43eb07b7 100644
--- a/gdb-10.2.patch
+++ b/gdb-10.2.patch
@@ -10,6 +10,7 @@
tar xvzmf gdb-10.2.tar.gz \
gdb-10.2/gdb/symtab.c \
+ gdb-10.2/gdb/thread.c \
gdb-10.2/gdb/printcmd.c \
gdb-10.2/gdb/symfile.c \
gdb-10.2/gdb/Makefile.in \
@@ -485,6 +486,35 @@ exit 0
return best_pst;
}
+--- gdb-10.2/gdb/thread.c.orig
++++ gdb-10.2/gdb/thread.c
+@@ -58,6 +58,11 @@ static int highest_thread_num;
+ /* The current/selected thread. */
+ static thread_info *current_thread_;
+
++#ifdef CRASH_MERGE
++/* Function to set cpu, defined by crash-utility */
++extern "C" void set_cpu (int);
++#endif
++
+ /* RAII type used to increase / decrease the refcount of each thread
+ in a given list of threads. */
+
+@@ -1896,7 +1901,13 @@ thread_command (const char *tidstr, int from_tty)
+ {
+ ptid_t previous_ptid = inferior_ptid;
+
+- thread_select (tidstr, parse_thread_id (tidstr, NULL));
++ struct thread_info* thread_id = parse_thread_id (tidstr, NULL);
++
++#ifdef CRASH_MERGE
++ set_cpu(thread_id->ptid.tid());
++#endif
++
++ thread_select (tidstr, thread_id);
+
+ /* Print if the thread has not changed, otherwise an event will
+ be sent. */
--- gdb-10.2/gdb/symfile.c.orig
+++ gdb-10.2/gdb/symfile.c
@@ -652,7 +652,26 @@ default_symfile_offsets (struct objfile *objfile,
diff --git a/kernel.c b/kernel.c
index 52b7ba09f390..a8d60507dd95 100644
--- a/kernel.c
+++ b/kernel.c
@@ -6504,7 +6504,13 @@ set_cpu(int cpu)
if (hide_offline_cpu(cpu))
error(FATAL, "invalid cpu number: cpu %d is OFFLINE\n", cpu);
- if ((task = get_active_task(cpu)))
+ task = get_active_task(cpu);
+
+ /* Check if context is already set to given cpu */
+ if (task == CURRENT_TASK())
+ return;
+
+ if (task)
set_context(task, NO_PID);
else
error(FATAL, "cannot determine active task on cpu %ld\n", cpu);
diff --git a/task.c b/task.c
index ebdb5be3786f..3a190cafbacb 100644
--- a/task.c
+++ b/task.c
@@ -5301,7 +5301,9 @@ set_context(ulong task, ulong pid)
if (found) {
CURRENT_CONTEXT() = tc;
- return TRUE;
+
+ /* change the selected thread in gdb, according to current context */
+ return gdb_change_cpu_context(tc->processor);
} else {
if (task)
error(INFO, "cannot set context for task: %lx\n", task);
--
2.43.0
2:52 a.m.
New subject: [PATCH v5 4/5] fix gdb_interface: restore gdb's output streams at end of gdb_interface
Currently for most gdb_interface call with a non-null file pointer,
GDB's output stream is replaced with the passed file pointer
'info threads', which is a gdb passthrough, fails to print all thread
after support was added to get registers from crash_target:
Id Target Id Frame
1 CPU 0 <unavailable> in ?? ()
2 CPU 1
Also, it was noticed that 'info threads' will print all threads when
'info threads' is run 2nd and subsequent times.
This incomplete output of 'info threads' was due to a subtle bug in
gdb_interface. For any passthrough, it switches the gdb output and
stderr streams to another FILE* passed in `struct gnu_request`.
But it does not restore the original FILE* stream gdb was using.
For each thread, which does not have it's registers yet, gdb goes to
crash_target::fetch_registers to ask for registers. Which in turn might
call 'datatype_info', which causes gdb's output stream to be set to
'/dev/null'
So all output after the call to datatype_info will go to /dev/null, and
hence the data to be printed is lost.
When 'info threads' is run a second time, it sets GDB's output streams
to something which is NOT /dev/null, and since it already has registers
for each thread, it does not go to 'crash_target::fetch_registers' and
hence the output stream is consistent, and we get complete output
So what happened is this:
1. GDB prints thread 1 (CPU 0) correctly since it had it's registers
cached during initialisation in 'crash_target_init' (at that times
registers where still unavailable, so it shows unavailable)
2. At CPU 1, to print the frame, it goes to 'crash_target::fetch_registers'
to get the registers
3. crash_target then calls 'machdep->get_cpu_reg' which finally ends up
at 'get_netdump_regs' which uses 'datatype_info' to get info of
'elf_prstatus.pr_reg'
4. Inside 'datatype_info', we set the file stream to /dev/null
(pc->nullfp). Then it passes this request object to gdb_interface
5. gdb_interface replaces GDB's output stream to the passed stream, ie.
/dev/null
6. Now, ALL FUTURE WRITES EVEN BY GDB, SUCH AS FOR THE 'info threads'
COMMAND GOES TO THIS STREAM gnu_request->fp, WHICH is /dev/null,
until someone else sets the non-null stream later
7. Similar behaviour occurs with all other threads
8. And hence we lose all subsequent output by GDB also
When 'info threads' is run a second time:
1. 'gdb_pass_through' passes a default FILE* to 'gdb_interface', which
is not /dev/null
2. Since the registers are cached for all threads now, it does not go
back to 'crash_target' for registers, and hence no call is made to
'datatype_info', hence GDB's output stream is not changing
3. Since the gdb's stream, even though not what it was using originally,
it's still valid, and not '/dev/null', hence we get the complete output
Fix this by restoring the original output streams, after gdb_interface
has handled the output
Signed-off-by: Aditya Gupta <adityag(a)linux.ibm.com>
---
gdb-10.2.patch | 46 +++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 45 insertions(+), 1 deletion(-)
diff --git a/gdb-10.2.patch b/gdb-10.2.patch
index 8c1b43eb07b7..a4e45fa0e6fa 100644
--- a/gdb-10.2.patch
+++ b/gdb-10.2.patch
@@ -1484,13 +1484,29 @@ exit 0
+ }
+}
+#endif
+--- gdb-10.2/gdb/ui-file.c.orig
++++ gdb-10.2/gdb/ui-file.c
+@@ -161,6 +161,12 @@ stdio_file::~stdio_file ()
+ fclose (m_file);
+ }
+
++FILE*
++stdio_file::get_stream(void)
++{
++ return m_file;
++}
++
+ void
+ stdio_file::set_stream (FILE *file)
+ {
--- gdb-10.2/gdb/ui-file.h.orig
+++ gdb-10.2/gdb/ui-file.h
-@@ -195,10 +195,10 @@ class stdio_file : public ui_file
+@@ -195,10 +195,11 @@ class stdio_file : public ui_file
bool can_emit_style_escape () override;
-private:
++ FILE *get_stream(void);
/* Sets the internal stream to FILE, and saves the FILE's file
descriptor in M_FD. */
void set_stream (FILE *file);
@@ -3267,3 +3283,31 @@ exit 0
for (compunit_symtab *cust : objfile->compunits ())
{
+--- gdb-10.2/gdb/symtab.c.orig
++++ gdb-10.2/gdb/symtab.c
+@@ -6964,8 +6964,12 @@ void
+ gdb_command_funnel_1(struct gnu_request *req)
+ {
+ struct symbol *sym;
++ FILE *original_stdout_stream = nullptr;
++ FILE *original_stderr_stream = nullptr;
+
+ if (req->command != GNU_VERSION && req->command !=
GNU_USER_PRINT_OPTION) {
++ original_stdout_stream = (dynamic_cast< stdio_file *
>gdb_stdout)->get_stream();
++ original_stderr_stream = (dynamic_cast< stdio_file *
>gdb_stderr)->get_stream();
+ (dynamic_cast<stdio_file
*>gdb_stdout)->set_stream(req->fp);
+ (dynamic_cast<stdio_file
*>gdb_stderr)->set_stream(req->fp);
+ }
+@@ -7068,6 +7072,12 @@ gdb_command_funnel_1(struct gnu_request *req)
+ req->flags |= GNU_COMMAND_FAILED;
+ break;
+ }
++
++ /* Restore the streams gdb output was using */
++ if (original_stdout_stream)
++ (dynamic_cast<stdio_file
*>gdb_stdout)->set_stream(original_stdout_stream);
++ if (original_stderr_stream)
++ (dynamic_cast<stdio_file
*>gdb_stderr)->set_stream(original_stderr_stream);
+ }
+
+ /*
--
2.43.0
2:52 a.m.
New subject: [PATCH v5 5/5] fix 'info threads' command
Currently the output is like this ppc64le:
crash> info threads
Id Target Id Frame
1 CPU 0 <unavailable> in ?? ()
2 CPU 1 0xc000000000025d00 in ppc_panic_event (this=<optimized
out>, event=<optimized out>, ptr=0x0) at arch/powerpc/kernel/setup-common.c:712
3 CPU 2 0xc000000000025d00 in ppc_panic_event (this=<optimized
out>, event=<optimized out>, ptr=0x0) at arch/powerpc/kernel/setup-common.c:712
4 CPU 3 0xc000000000025d00 in ppc_panic_event (this=<optimized
out>, event=<optimized out>, ptr=0x0) at arch/powerpc/kernel/setup-common.c:712
* 5 CPU 4 0xc000000000025d00 in ppc_panic_event (this=<optimized
out>, event=<optimized out>, ptr=0x0) at arch/powerpc/kernel/setup-common.c:712
6 CPU 5 0xc000000000025d00 in ppc_panic_event (this=<optimized
out>, event=<optimized out>, ptr=0x0) at arch/powerpc/kernel/setup-common.c:712
7 CPU 6 0xc000000000025d00 in ppc_panic_event (this=<optimized
out>, event=<optimized out>, ptr=0x0) at arch/powerpc/kernel/setup-common.c:712
8 CPU 7 0xc000000000025d00 in ppc_panic_event (this=<optimized
out>, event=<optimized out>, ptr=0x0) at arch/powerpc/kernel/setup-common.c:712
There are two issues here:
1. CPU 0 always shows '<unavailable>' frame:
This is due to 'thread 0' being initialised in 'crash_target_init',
but it's very early in crash init, so it could not get any registers
from the vmcore.
And, since GDB caches registers (whether it is there or now), it
keeps thinking that registers for this thread are unavailable, so all
future usage of thread 0 also shows unavailable registers & frame
Fix this by refreshing the register cache of all threads in
'gdb_refresh_regcache', at task_init
2. All other CPUs show the same frame:
For each thread, GDB depends on crash to give it registers, ie.
crash_target::fetch_registers, but this in turn depends on crash's
current context
GDB internally switches it's thread context internally to each thread
one by one, while printing the thread's frame, similarly switch
crash's context when the thread context is changed, so that it
matches the thread context in gdb (such as, CPU 3 in crash for thread
2(CPU 3) in gdb).
Also, since we are switching context multiple times, add an argument
to 'set_cpu' to not print the context everytime
With these fixed, it will show all threads, as well as correct registers
and frame. It might still show similar frame for many threads, due to
all of them being on same NMI exception crash path, but if checked with
'info registers', they will have different registers.
Signed-off-by: Aditya Gupta <adityag(a)linux.ibm.com>
---
crash_target.c | 37 +++++++++++++++++++++++++++++++++++--
defs.h | 5 +++--
gdb-10.2.patch | 40 +++++++++++++++++++++++++++++++++++++---
kernel.c | 8 +++++---
task.c | 30 ++++++++++++++++++++++--------
tools.c | 10 +++++-----
6 files changed, 107 insertions(+), 23 deletions(-)
diff --git a/crash_target.c b/crash_target.c
index a9d7eea8a80e..d06383f594aa 100644
--- a/crash_target.c
+++ b/crash_target.c
@@ -30,7 +30,8 @@ extern "C" int crash_get_nr_cpus(void);
extern "C" int crash_get_cpu_reg (int cpu, int regno, const char *regname,
int regsize, void *val);
extern "C" int gdb_change_cpu_context (unsigned int cpu);
-extern "C" int set_cpu (int cpu);
+extern "C" void gdb_refresh_regcache(unsigned int cpu);
+extern "C" int set_cpu(int cpu, int print_context);
/* The crash target. */
@@ -150,10 +151,42 @@ gdb_change_cpu_context(unsigned int cpu)
return FALSE;
/* Making sure that crash's context is same */
- set_cpu(cpu);
+ set_cpu(cpu, FALSE);
/* Switch to the thread */
switch_to_thread(tp);
+
+ /* Fetch/Refresh thread's registers */
+ gdb_refresh_regcache(cpu);
+
return TRUE;
}
+/* Refresh regcache of gdb thread on given CPU
+ *
+ * When gdb threads were initially added by 'crash_target_init', crash was not
+ * yet initialised, and hence crash_target::fetch_registers didn't give any
+ * registers to gdb.
+ *
+ * This is meant to be called after tasks in crash have been initialised, and
+ * possible machdep->get_cpu_reg is also set so architecture can give registers
+ */
+extern "C" void
+gdb_refresh_regcache(unsigned int cpu)
+{
+ int saved_cpu = inferior_thread()->ptid.tid();
+ ptid_t ptid = ptid_t(CRASH_INFERIOR_PID, 0, cpu);
+ inferior *inf = current_inferior();
+ thread_info *tp = find_thread_ptid (inf, ptid);
+
+ if (tp == NULL) {
+ warning("gdb thread for cpu %d not found\n", cpu);
+ return;
+ }
+
+ /* temporarily switch to the cpu so we get correct registers */
+ set_cpu(cpu, FALSE);
+ target_fetch_registers(get_thread_regcache(tp), -1);
+
+ set_cpu(saved_cpu, FALSE);
+}
diff --git a/defs.h b/defs.h
index 7f7a12753658..ba4a6a606f66 100644
--- a/defs.h
+++ b/defs.h
@@ -5940,7 +5940,7 @@ extern char *help_map[];
* task.c
*/
void task_init(void);
-int set_context(ulong, ulong);
+int set_context(ulong, ulong, uint);
void show_context(struct task_context *);
ulong pid_to_task(ulong);
ulong task_to_pid(ulong);
@@ -6044,7 +6044,7 @@ void parse_kernel_version(char *);
#define SHOW_LOG_AUDIT (0x8)
#define SHOW_LOG_CTIME (0x10)
#define SHOW_LOG_SAFE (0x20)
-void set_cpu(int);
+void set_cpu(int cpu, int print_context);
void clear_machdep_cache(void);
struct stack_hook *gather_text_list(struct bt_info *);
int get_cpus_online(void);
@@ -7978,5 +7978,6 @@ enum ppc64_regnum {
/* crash_target.c */
extern int gdb_change_cpu_context (unsigned int cpu);
+extern void gdb_refresh_regcache (unsigned int cpu);
#endif /* !GDB_COMMON */
diff --git a/gdb-10.2.patch b/gdb-10.2.patch
index a4e45fa0e6fa..9051c83ac784 100644
--- a/gdb-10.2.patch
+++ b/gdb-10.2.patch
@@ -494,13 +494,47 @@ exit 0
+#ifdef CRASH_MERGE
+/* Function to set cpu, defined by crash-utility */
-+extern "C" void set_cpu (int);
++extern "C" void set_cpu(int cpu, int print_context);
+#endif
+
/* RAII type used to increase / decrease the refcount of each thread
in a given list of threads. */
-@@ -1896,7 +1901,13 @@ thread_command (const char *tidstr, int from_tty)
+@@ -1093,6 +1098,9 @@ print_thread_info_1 (struct ui_out *uiout, const char
*requested_threads,
+ uiout->table_body ();
+ }
+
++#ifdef CRASH_MERGE
++ int current_cpu = current_thread ? current_thread->ptid.tid() : 0;
++#endif
+ for (inferior *inf : all_inferiors ())
+ for (thread_info *tp : inf->threads ())
+ {
+@@ -1108,6 +1116,11 @@ print_thread_info_1 (struct ui_out *uiout, const char
*requested_threads,
+
+ ui_out_emit_tuple tuple_emitter (uiout, NULL);
+
++ /* Switch to the thread's CPU in crash */
++#ifdef CRASH_MERGE
++ set_cpu(tp->ptid.tid(), FALSE);
++#endif
++
+ if (!uiout->is_mi_like_p ())
+ {
+ if (tp == current_thread)
+@@ -1180,6 +1193,11 @@ print_thread_info_1 (struct ui_out *uiout, const char
*requested_threads,
+ /* This end scope restores the current thread and the frame
+ selected before the "info threads" command, and it finishes the
+ ui-out list or table. */
++
++#ifdef CRASH_MERGE
++ /* Restore crash's context */
++ set_cpu(current_cpu, FALSE);
++#endif
+ }
+
+ if (pid == -1 && requested_threads == NULL)
+@@ -1896,7 +1914,13 @@ thread_command (const char *tidstr, int from_tty)
{
ptid_t previous_ptid = inferior_ptid;
@@ -508,7 +542,7 @@ exit 0
+ struct thread_info* thread_id = parse_thread_id (tidstr, NULL);
+
+#ifdef CRASH_MERGE
-+ set_cpu(thread_id->ptid.tid());
++ set_cpu(thread_id->ptid.tid(), FALSE);
+#endif
+
+ thread_select (tidstr, thread_id);
diff --git a/kernel.c b/kernel.c
index a8d60507dd95..505cffde1cf8 100644
--- a/kernel.c
+++ b/kernel.c
@@ -6491,9 +6491,10 @@ no_cpu_flags:
/*
* Set the context to the active task on a given cpu -- dumpfiles only.
+ * Optionally show the context if print_context is TRUE
*/
void
-set_cpu(int cpu)
+set_cpu(int cpu, int print_context)
{
ulong task;
@@ -6511,11 +6512,12 @@ set_cpu(int cpu)
return;
if (task)
- set_context(task, NO_PID);
+ set_context(task, NO_PID, TRUE);
else
error(FATAL, "cannot determine active task on cpu %ld\n", cpu);
- show_context(CURRENT_CONTEXT());
+ if (print_context)
+ show_context(CURRENT_CONTEXT());
}
diff --git a/task.c b/task.c
index 3a190cafbacb..32a27420bda2 100644
--- a/task.c
+++ b/task.c
@@ -672,7 +672,7 @@ task_init(void)
if (ACTIVE()) {
active_pid = REMOTE() ? pc->server_pid :
LOCAL_ACTIVE() ? pc->program_pid : 1;
- set_context(NO_TASK, active_pid);
+ set_context(NO_TASK, active_pid, FALSE);
tt->this_task = pid_to_task(active_pid);
}
else {
@@ -684,7 +684,7 @@ task_init(void)
else if (ELF_NOTES_VALID() && DISKDUMP_DUMPFILE())
map_cpus_to_prstatus_kdump_cmprs();
please_wait("determining panic task");
- set_context(get_panic_context(), NO_PID);
+ set_context(get_panic_context(), NO_PID, FALSE);
please_wait_done();
}
@@ -706,6 +706,17 @@ task_init(void)
kt->boot_date.tv_nsec = 0;
}
+ /*
+ * Refresh CPU 0's thread's regcache
+ *
+ * This is required since, it's registers were initialised in
+ * crash_target_init when crash was not initialised yet and hence could
+ * not pass registers to gdb when gdb requests via
+ * crash_target::fetch_registers, so CPU 0's registers are shown as
+ * <unavailable> in gdb mode
+ * */
+ gdb_refresh_regcache(0);
+
tt->flags |= TASK_INIT_DONE;
}
@@ -2985,9 +2996,9 @@ refresh_context(ulong curtask, ulong curpid)
struct task_context *tc;
if (task_exists(curtask) && pid_exists(curpid)) {
- set_context(curtask, NO_PID);
+ set_context(curtask, NO_PID, FALSE);
} else {
- set_context(tt->this_task, NO_PID);
+ set_context(tt->this_task, NO_PID, FALSE);
complain = TRUE;
if (STREQ(args[0], "set") && (argcnt == 2) &&
@@ -3053,7 +3064,7 @@ sort_context_array(void)
curtask = CURRENT_TASK();
qsort((void *)tt->context_array, (size_t)tt->running_tasks,
sizeof(struct task_context), sort_by_pid);
- set_context(curtask, NO_PID);
+ set_context(curtask, NO_PID, FALSE);
sort_context_by_task();
}
@@ -3100,7 +3111,7 @@ sort_context_array_by_last_run(void)
curtask = CURRENT_TASK();
qsort((void *)tt->context_array, (size_t)tt->running_tasks,
sizeof(struct task_context), sort_by_last_run);
- set_context(curtask, NO_PID);
+ set_context(curtask, NO_PID, FALSE);
sort_context_by_task();
}
@@ -5281,7 +5292,7 @@ comm_exists(char *s)
* that pid is selected.
*/
int
-set_context(ulong task, ulong pid)
+set_context(ulong task, ulong pid, uint update_gdb_thread)
{
int i;
struct task_context *tc;
@@ -5303,7 +5314,10 @@ set_context(ulong task, ulong pid)
CURRENT_CONTEXT() = tc;
/* change the selected thread in gdb, according to current context */
- return gdb_change_cpu_context(tc->processor);
+ if (update_gdb_thread)
+ return gdb_change_cpu_context(tc->processor);
+ else
+ return TRUE;
} else {
if (task)
error(INFO, "cannot set context for task: %lx\n", task);
diff --git a/tools.c b/tools.c
index 0f2db108838a..9b149fb27fcb 100644
--- a/tools.c
+++ b/tools.c
@@ -1860,7 +1860,7 @@ cmd_set(void)
break;
}
cpu = dtoi(optarg, FAULT_ON_ERROR, NULL);
- set_cpu(cpu);
+ set_cpu(cpu, TRUE);
return;
case 'p':
@@ -1871,7 +1871,7 @@ cmd_set(void)
return;
if (ACTIVE()) {
- set_context(tt->this_task, NO_PID);
+ set_context(tt->this_task, NO_PID, FALSE);
show_context(CURRENT_CONTEXT());
return;
}
@@ -1880,7 +1880,7 @@ cmd_set(void)
error(INFO, "no panic task found!\n");
return;
}
- set_context(tt->panic_task, NO_PID);
+ set_context(tt->panic_task, NO_PID, FALSE);
show_context(CURRENT_CONTEXT());
return;
@@ -2559,14 +2559,14 @@ cmd_set(void)
case STR_PID:
pid = value;
task = NO_TASK;
- if (set_context(task, pid))
+ if (set_context(task, pid, FALSE))
show_context(CURRENT_CONTEXT());
break;
case STR_TASK:
task = value;
pid = NO_PID;
- if (set_context(task, pid))
+ if (set_context(task, pid, FALSE))
show_context(CURRENT_CONTEXT());
break;
--
2.43.0
8:25 p.m.
Hi Aditya,
On Wed, Dec 20, 2023 at 4:52 PM Aditya Gupta <adityag(a)linux.ibm.com> wrote:
The Problem:
============
Currently crash is unable to show function arguments and local variables, as
gdb can do. And functionality for moving between frames ('up'/'down') is
not
working in crash.
Crash has 'gdb passthroughs' for things gdb can do, but the gdb passthroughs
'bt', 'frame', 'info locals', 'up', 'down' are
not working either, due to
gdb not getting the register values from `crash_target::fetch_registers`,
which then uses `machdep->get_cpu_reg`, which is not implemented for PPC64
Proposed Solution:
==================
Fix the gdb passthroughs by implementing "machdep->get_cpu_reg" for PPC64.
This way, "gdb mode in crash" will support this feature for both ELF and
kdump-compressed vmcore formats, while "gdb" would only have supported ELF
format
This way other features of 'gdb', such as seeing
backtraces/registers/variables/arguments/local variables, moving up and
down stack frames, can be used with any ppc64 vmcore, irrespective of
being ELF format or kdump-compressed format.
Note: This doesn't support live debugging on ppc64, since registers are not
available to be read
Implications on Architectures:
====================================
No architecture other than PPC64 has been affected, other than in case of
'frame' command
As mentioned in patch #2, since frame will not be prohibited, so it will print:
crash> frame
#0 <unavailable> in ?? ()
Instead of before prohibited message:
crash> frame
crash: prohibited gdb command: frame
Major change will be in 'gdb mode' on PPC64, that it will print the frames, and
local variables, instead of failing with errors showing no frame, or showing
that couldn't get PC, it will be able to give all this information.
Testing:
========
Git tree with this patch series applied:
https://github.com/adi-g15-ibm/crash/tree/stack-unwind-v5-smaller-gran
To test various gdb passthroughs:
(crash) set
(crash) set gdb on
gdb> thread
gdb> bt
gdb> info threads
gdb> info threads
gdb> info locals
gdb> info variables irq_rover_lock
gdb> info args
gdb> thread 2
gdb> set gdb off
(crash) set
(crash) set -c 6
(crash) gdb thread
(crash) bt
(crash) gdb bt
(crash) frame
(crash) up
(crash) down
(crash) info locals
It seems we cannot view arbitrary process stack unwinding right? I
mean, "gdb thread X" or "gdb bt" can only view the process which is
currently running on the specific CPU, but for other processes which
are not occupying CPU, we cannot specify it by "gdb bt <pid>" or
"gdb
thread <pid>". However drgn can unwind stack on arbitrary processes:
>>
prog.stack_trace(<pid>)[<frame_id>]["varibale_name"]
It seems crash cmd bt need to be improved in order to view arbitrary
process stack unwinding like drgn right?
Thanks,
Tao Liu
Known Issues:
=============
1. In gdb mode, 'bt' might fail to show backtrace in few vmcores collected
from older kernels. This is a known issue due to register mismatch, and
its fix has been merged upstream:
This can also cause some 'invalid kernel virtual address' errors during gdb
unwinding the stack registers
Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
Fixing GDB passthroughs on other architectures
==============================================
Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
thread', 'gdb info locals' etc. has been done by the patches introducing
'machdep->get_cpu_reg' and this series fixing some issues in that.
Other architectures should be able to fix these gdb functionalities by
simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
The reasoning behind that has been explained with a diagram in commit
description of patch #1
I will assist with my findings/observations fixing it on ppc64 whenever needed.
Changelog:
==========
V5:
+ changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
code
+ changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
remove unneeded if condition
+ changes in patch #5: implement refresh regcache on per thread, instead of all
threads at once
V4:
+ fix segmentation fault in live debugging (change in patch #1)
+ mention live debugging not supported in cover letter and patch #1
+ fixed some checkpatch warnings (change in patch #5)
V3:
+ default gdb thread will be the crashing thread, instead of being
thread '0'
+ synchronise crash cpu and gdb thread context
+ fix bug in gdb_interface, that replaced gdb's output stream, losing
output in some cases, such as info threads and extra output in info
variables
+ fix 'info threads'
RFC V2:
- removed patch implementing 'frame', 'up', 'down' in crash
- updated the cover letter by removing the mention of those commands other
than the respective gdb passthrough
Aditya Gupta (5):
ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
remove 'frame' from prohibited commands list
synchronise cpu context changes between crash/gdb
fix gdb_interface: restore gdb's output streams at end of
gdb_interface
fix 'info threads' command
crash_target.c | 44 ++++++++++++++++
defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
gdb_interface.c | 2 +-
kernel.c | 47 +++++++++++++++--
ppc64.c | 95 +++++++++++++++++++++++++++++++++--
task.c | 14 ++++++
tools.c | 2 +-
8 files changed, 434 insertions(+), 10 deletions(-)
--
2.41.0
10:08 p.m.
Hello Tao,
On Mon, Dec 25, 2023 at 10:25:00AM +0800, Tao Liu wrote:
Hi Aditya,
On Wed, Dec 20, 2023 at 4:52 PM Aditya Gupta <adityag(a)linux.ibm.com> wrote:
>
> The Problem:
> ============
>
> Currently crash is unable to show function arguments and local variables, as
> gdb can do. And functionality for moving between frames
('up'/'down') is not
> working in crash.
>
> Crash has 'gdb passthroughs' for things gdb can do, but the gdb
passthroughs
> 'bt', 'frame', 'info locals', 'up', 'down'
are not working either, due to
> gdb not getting the register values from `crash_target::fetch_registers`,
> which then uses `machdep->get_cpu_reg`, which is not implemented for PPC64
>
> Proposed Solution:
> ==================
>
> Fix the gdb passthroughs by implementing "machdep->get_cpu_reg" for
PPC64.
> This way, "gdb mode in crash" will support this feature for both ELF and
> kdump-compressed vmcore formats, while "gdb" would only have supported
ELF
> format
>
> This way other features of 'gdb', such as seeing
> backtraces/registers/variables/arguments/local variables, moving up and
> down stack frames, can be used with any ppc64 vmcore, irrespective of
> being ELF format or kdump-compressed format.
>
> Note: This doesn't support live debugging on ppc64, since registers are not
> available to be read
>
> Implications on Architectures:
> ====================================
>
> No architecture other than PPC64 has been affected, other than in case of
> 'frame' command
>
> As mentioned in patch #2, since frame will not be prohibited, so it will print:
>
> crash> frame
> #0 <unavailable> in ?? ()
>
> Instead of before prohibited message:
>
> crash> frame
> crash: prohibited gdb command: frame
>
> Major change will be in 'gdb mode' on PPC64, that it will print the frames,
and
> local variables, instead of failing with errors showing no frame, or showing
> that couldn't get PC, it will be able to give all this information.
>
> Testing:
> ========
>
> Git tree with this patch series applied:
> https://github.com/adi-g15-ibm/crash/tree/stack-unwind-v5-smaller-gran
>
> To test various gdb passthroughs:
>
> (crash) set
> (crash) set gdb on
> gdb> thread
> gdb> bt
> gdb> info threads
> gdb> info threads
> gdb> info locals
> gdb> info variables irq_rover_lock
> gdb> info args
> gdb> thread 2
> gdb> set gdb off
> (crash) set
> (crash) set -c 6
> (crash) gdb thread
> (crash) bt
> (crash) gdb bt
> (crash) frame
> (crash) up
> (crash) down
> (crash) info locals
>
It seems we cannot view arbitrary process stack unwinding right? I
mean, "gdb thread X" or "gdb bt" can only view the process which is
currently running on the specific CPU, but for other processes which
are not occupying CPU, we cannot specify it by "gdb bt <pid>" or
"gdb
thread <pid>". However drgn can unwind stack on arbitrary processes:
>>> prog.stack_trace(<pid>)[<frame_id>]["varibale_name"]
Yes, currently gdb features cannot be used with arbitrary PIDs, since GDB
doesn't know of all those threads.
GDB gets the details of threads, registers etc from a registered 'target'.
When we debug a vmcore and vmlinux with GDB (NOT gdb mode in crash), ie. with
`gdb <vmlinux-path> <vmcore-path>`, then the target is 'core_target'.
This target knows all the known PIDs in the vmcore.
While in gdb mode, the target we register with GDB is 'crash_target'. And in
'crash_target_init', we explicitly register only 'nr_cpus' threads, one
for each
CPU. Hence it does not know of any other threads.
It seems crash cmd bt need to be improved in order to view arbitrary
process stack unwinding like drgn right?
Yes, specifically 'crash_target' needs to be improved to support all those
arbitrary threads. crash bt can already print arbitrary PID backtraces, but
improving crash_target will allow printing arbitrary threads in gdb mode also.
Thanks,
Aditya Gupta
>
> Thanks,
> Tao Liu
>
>
> > Known Issues:
> > =============
> >
> > 1. In gdb mode, 'bt' might fail to show backtrace in few vmcores
collected
> > from older kernels. This is a known issue due to register mismatch, and
> > its fix has been merged upstream:
> >
> > This can also cause some 'invalid kernel virtual address' errors
during gdb
> > unwinding the stack registers
> >
> > Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
> >
> > Fixing GDB passthroughs on other architectures
> > ==============================================
> >
> > Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
> > thread', 'gdb info locals' etc. has been done by the patches
introducing
> > 'machdep->get_cpu_reg' and this series fixing some issues in that.
> >
> > Other architectures should be able to fix these gdb functionalities by
> > simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
> >
> > The reasoning behind that has been explained with a diagram in commit
> > description of patch #1
> >
> > I will assist with my findings/observations fixing it on ppc64 whenever needed.
> >
> > Changelog:
> > ==========
> >
> > V5:
> > + changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
> > code
> > + changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
> > remove unneeded if condition
> > + changes in patch #5: implement refresh regcache on per thread, instead of all
> > threads at once
> >
> > V4:
> > + fix segmentation fault in live debugging (change in patch #1)
> > + mention live debugging not supported in cover letter and patch #1
> > + fixed some checkpatch warnings (change in patch #5)
> >
> > V3:
> > + default gdb thread will be the crashing thread, instead of being
> > thread '0'
> > + synchronise crash cpu and gdb thread context
> > + fix bug in gdb_interface, that replaced gdb's output stream, losing
> > output in some cases, such as info threads and extra output in info
> > variables
> > + fix 'info threads'
> >
> > RFC V2:
> > - removed patch implementing 'frame', 'up', 'down' in
crash
> > - updated the cover letter by removing the mention of those commands other
> > than the respective gdb passthrough
> >
> > Aditya Gupta (5):
> > ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
> > remove 'frame' from prohibited commands list
> > synchronise cpu context changes between crash/gdb
> > fix gdb_interface: restore gdb's output streams at end of
> > gdb_interface
> > fix 'info threads' command
> >
> > crash_target.c | 44 ++++++++++++++++
> > defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
> > gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
> > gdb_interface.c | 2 +-
> > kernel.c | 47 +++++++++++++++--
> > ppc64.c | 95 +++++++++++++++++++++++++++++++++--
> > task.c | 14 ++++++
> > tools.c | 2 +-
> > 8 files changed, 434 insertions(+), 10 deletions(-)
> >
> > --
> > 2.41.0
> >
>
3:15 a.m.
Hi, Aditya
Thank you for the update.
On 12/20/23 16:52, Aditya Gupta wrote:
The Problem:
============
Currently crash is unable to show function arguments and local variables, as
gdb can do. And functionality for moving between frames ('up'/'down') is
not
working in crash.
Crash has 'gdb passthroughs' for things gdb can do, but the gdb passthroughs
'bt', 'frame', 'info locals', 'up', 'down' are
not working either, due to
gdb not getting the register values from `crash_target::fetch_registers`,
which then uses `machdep->get_cpu_reg`, which is not implemented for PPC64
Proposed Solution:
==================
Fix the gdb passthroughs by implementing "machdep->get_cpu_reg" for PPC64.
This way, "gdb mode in crash" will support this feature for both ELF and
kdump-compressed vmcore formats, while "gdb" would only have supported ELF
format
This way other features of 'gdb', such as seeing
backtraces/registers/variables/arguments/local variables, moving up and
down stack frames, can be used with any ppc64 vmcore, irrespective of
being ELF format or kdump-compressed format.
Note: This doesn't support live debugging on ppc64, since registers are not
available to be read
Implications on Architectures:
====================================
No architecture other than PPC64 has been affected, other than in case of
'frame' command
As mentioned in patch #2, since frame will not be prohibited, so it will print:
crash> frame
#0 <unavailable> in ?? ()
Instead of before prohibited message:
crash> frame
crash: prohibited gdb command: frame
Major change will be in 'gdb mode' on PPC64, that it will print the frames, and
local variables, instead of failing with errors showing no frame, or showing
that couldn't get PC, it will be able to give all this information.
Testing:
========
Git tree with this patch series applied:
https://github.com/adi-g15-ibm/crash/tree/stack-unwind-v5-smaller-gran
To test various gdb passthroughs:
(crash) set
(crash) set gdb on
gdb> thread
gdb> bt
gdb> info threads
gdb> info threads
gdb> info locals
gdb> info variables irq_rover_lock
gdb> info args
gdb> thread 2
gdb> set gdb off
(crash) set
(crash) set -c 6
(crash) gdb thread
(crash) bt
(crash) gdb bt
(crash) frame
(crash) up
(crash) down
(crash) info locals
After applying the v5, I still got the following errors:
crash> gdb bt
#0 <unavailable> in ?? ()
#1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:111
#2 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#5 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#11 0xc000000000011030 in rest_init () at init/main.c:730
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
#12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
#13 0xc0000000020051ec in start_kernel () at init/main.c:1072
#14 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
crash> gdb frame
#0 <unavailable> in ?? ()
crash> set gdb on
gdb: on
gdb> bt
#0 <unavailable> in ?? ()
#1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:111
#2 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#5 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#11 0xc000000000011030 in rest_init () at init/main.c:730
#12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
#13 0xc0000000020051ec in start_kernel () at init/main.c:1072
#14 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
gdb> info threads
Id Target Id Frame
* 1 CPU 0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
2 CPU 1 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
3 CPU 2 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
4 CPU 3 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
5 CPU 4 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
6 CPU 5 0xc00000000028b5e8 in crash_setup_regs
(oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
./arch/powerpc/include/asm/kexec.h:69
7 CPU 6 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
8 CPU 7 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem_callback"
gdb> thread 1
[Switching to thread 1 (CPU 0)]
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
114 li r4,0
gdb> bt
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
#1 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#4 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#10 0xc000000000011030 in rest_init () at init/main.c:730
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
#11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
#12 0xc0000000020051ec in start_kernel () at init/main.c:1072
#13 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
gdb> thread 3
[Switching to thread 3 (CPU 2)]
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
114 li r4,0
gdb> bt
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
#1 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#4 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#10 0xc00000000005f048 in start_secondary (unused=<optimized out>) at
arch/powerpc/kernel/smp.c:1680
#11 0xc00000000000e058 in start_secondary_prolog () at
arch/powerpc/kernel/head_64.S:885
gdb>
Could you please check it again? Or am I missing anything? I did the
test based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
BTW: I did not see the similar issues after applying the v4 patch set.
Thanks
Lianbo
Known Issues:
=============
1. In gdb mode, 'bt' might fail to show backtrace in few vmcores collected
from older kernels. This is a known issue due to register mismatch, and
its fix has been merged upstream:
This can also cause some 'invalid kernel virtual address' errors during gdb
unwinding the stack registers
Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
Fixing GDB passthroughs on other architectures
==============================================
Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
thread', 'gdb info locals' etc. has been done by the patches introducing
'machdep->get_cpu_reg' and this series fixing some issues in that.
Other architectures should be able to fix these gdb functionalities by
simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
The reasoning behind that has been explained with a diagram in commit
description of patch #1
I will assist with my findings/observations fixing it on ppc64 whenever needed.
Changelog:
==========
V5:
+ changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
code
+ changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
remove unneeded if condition
+ changes in patch #5: implement refresh regcache on per thread, instead of all
threads at once
V4:
+ fix segmentation fault in live debugging (change in patch #1)
+ mention live debugging not supported in cover letter and patch #1
+ fixed some checkpatch warnings (change in patch #5)
V3:
+ default gdb thread will be the crashing thread, instead of being
thread '0'
+ synchronise crash cpu and gdb thread context
+ fix bug in gdb_interface, that replaced gdb's output stream, losing
output in some cases, such as info threads and extra output in info
variables
+ fix 'info threads'
RFC V2:
- removed patch implementing 'frame', 'up', 'down' in crash
- updated the cover letter by removing the mention of those commands other
than the respective gdb passthrough
Aditya Gupta (5):
ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
remove 'frame' from prohibited commands list
synchronise cpu context changes between crash/gdb
fix gdb_interface: restore gdb's output streams at end of
gdb_interface
fix 'info threads' command
crash_target.c | 44 ++++++++++++++++
defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
gdb_interface.c | 2 +-
kernel.c | 47 +++++++++++++++--
ppc64.c | 95 +++++++++++++++++++++++++++++++++--
task.c | 14 ++++++
tools.c | 2 +-
8 files changed, 434 insertions(+), 10 deletions(-)
12:51 a.m.
Hi Lianbo,
Thanks for the reviews
After applying the v5, I still got the following errors:
crash> gdb bt
#0 <unavailable> in ?? ()
#1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:111
#2 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#5 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#11 0xc000000000011030 in rest_init () at init/main.c:730
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
#12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
#13 0xc0000000020051ec in start_kernel () at init/main.c:1072
#14 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
crash> gdb frame
#0 <unavailable> in ?? ()
crash> set gdb on
gdb: on
gdb> bt
#0 <unavailable> in ?? ()
#1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:111
#2 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#5 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#11 0xc000000000011030 in rest_init () at init/main.c:730
#12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
#13 0xc0000000020051ec in start_kernel () at init/main.c:1072
#14 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
gdb> info threads
Id Target Id Frame
* 1 CPU 0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
2 CPU 1 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
3 CPU 2 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
4 CPU 3 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
5 CPU 4 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
6 CPU 5 0xc00000000028b5e8 in crash_setup_regs
(oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
./arch/powerpc/include/asm/kexec.h:69
7 CPU 6 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
8 CPU 7 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem_callback"
gdb> thread 1
[Switching to thread 1 (CPU 0)]
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
114 li r4,0
gdb> bt
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
#1 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#4 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#10 0xc000000000011030 in rest_init () at init/main.c:730
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
#11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
#12 0xc0000000020051ec in start_kernel () at init/main.c:1072
#13 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
gdb> thread 3
[Switching to thread 3 (CPU 2)]
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
114 li r4,0
gdb> bt
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
#1 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#4 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#10 0xc00000000005f048 in start_secondary (unused=<optimized out>) at
arch/powerpc/kernel/smp.c:1680
#11 0xc00000000000e058 in start_secondary_prolog () at
arch/powerpc/kernel/head_64.S:885
gdb>
Could you please check it again? Or am I missing anything? I did the test
based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
I am checking it. Trying to fix the 'page excluded' error. As of now, I found
that it happens when gdb is trying to unwind the frames. It tries
'tramp_frame_sniffer' unwinder which causes those errors, afaik it's normal
for
most unwinders to fail to unwind (gdb tries all unwinders to see which one can
unwind the previous frame), will see how to handle this case.
BTW: I did not see the similar issues after applying the v4 patch set.
There was a minor bug in V5, due to gdb thread still set to thread 0 initially.
Fixed that one.
Thanks,
Aditya Gupta
>
>
> Thanks
>
> Lianbo
>
> >
> > Known Issues:
> > =============
> >
> > 1. In gdb mode, 'bt' might fail to show backtrace in few vmcores
collected
> > from older kernels. This is a known issue due to register mismatch, and
> > its fix has been merged upstream:
> >
> > This can also cause some 'invalid kernel virtual address' errors
during gdb
> > unwinding the stack registers
> >
> > Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
> >
> > Fixing GDB passthroughs on other architectures
> > ==============================================
> >
> > Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
> > thread', 'gdb info locals' etc. has been done by the patches
introducing
> > 'machdep->get_cpu_reg' and this series fixing some issues in that.
> >
> > Other architectures should be able to fix these gdb functionalities by
> > simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
> >
> > The reasoning behind that has been explained with a diagram in commit
> > description of patch #1
> >
> > I will assist with my findings/observations fixing it on ppc64 whenever needed.
> >
> > Changelog:
> > ==========
> >
> > V5:
> > + changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
> > code
> > + changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
> > remove unneeded if condition
> > + changes in patch #5: implement refresh regcache on per thread, instead of all
> > threads at once
> >
> > V4:
> > + fix segmentation fault in live debugging (change in patch #1)
> > + mention live debugging not supported in cover letter and patch #1
> > + fixed some checkpatch warnings (change in patch #5)
> >
> > V3:
> > + default gdb thread will be the crashing thread, instead of being
> > thread '0'
> > + synchronise crash cpu and gdb thread context
> > + fix bug in gdb_interface, that replaced gdb's output stream, losing
> > output in some cases, such as info threads and extra output in info
> > variables
> > + fix 'info threads'
> >
> > RFC V2:
> > - removed patch implementing 'frame', 'up', 'down' in
crash
> > - updated the cover letter by removing the mention of those commands other
> > than the respective gdb passthrough
> >
> > Aditya Gupta (5):
> > ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
> > remove 'frame' from prohibited commands list
> > synchronise cpu context changes between crash/gdb
> > fix gdb_interface: restore gdb's output streams at end of
> > gdb_interface
> > fix 'info threads' command
> >
> > crash_target.c | 44 ++++++++++++++++
> > defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
> > gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
> > gdb_interface.c | 2 +-
> > kernel.c | 47 +++++++++++++++--
> > ppc64.c | 95 +++++++++++++++++++++++++++++++++--
> > task.c | 14 ++++++
> > tools.c | 2 +-
> > 8 files changed, 434 insertions(+), 10 deletions(-)
> >
>
12:57 a.m.
Hi Lianbo,
Thanks for the reviews.
crash> gdb frame
#0 <unavailable> in ?? ()
Will fix this in V6. It got introduced in V5 due to initial gdb thread
still being thread 1 (CPU 0).
crash> set gdb on
gdb: on
gdb> bt
#0 <unavailable> in ?? ()
#1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:111
#2 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#5 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#11 0xc000000000011030 in rest_init () at init/main.c:730
#12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
#13 0xc0000000020051ec in start_kernel () at init/main.c:1072
#14 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
gdb> info threads
Id Target Id Frame
* 1 CPU 0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
2 CPU 1 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
3 CPU 2 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
4 CPU 3 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
5 CPU 4 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
6 CPU 5 0xc00000000028b5e8 in crash_setup_regs
(oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
./arch/powerpc/include/asm/kexec.h:69
7 CPU 6 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
8 CPU 7 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem_callback"
<...>
Could you please check it again? Or am I missing anything? I did the test
based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
I am looking at it. As of now, I found these 'page excluded' error come up when
gdb tries to access some addresses while trying 'tramp_frame_sniffer' unwinder.
afaik most unwinders are expected to fail to unwind, as gdb tries all unwinders
to see which one can unwind the previous frame correctly, will see how to handle
this case.
BTW: I did not see the similar issues after applying the v4 patch set.
A bug got introduced in V5, where the initial gdb thread is thread 1,
irrespective of crashing thread. Found a fix for that, but still pending for
'page excluded' errors.
Thanks,
Aditya Gupta
>
> Thanks
>
> Lianbo
>
> >
> > Known Issues:
> > =============
> >
> > 1. In gdb mode, 'bt' might fail to show backtrace in few vmcores
collected
> > from older kernels. This is a known issue due to register mismatch, and
> > its fix has been merged upstream:
> >
> > This can also cause some 'invalid kernel virtual address' errors
during gdb
> > unwinding the stack registers
> >
> > Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
> >
> > Fixing GDB passthroughs on other architectures
> > ==============================================
> >
> > Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
> > thread', 'gdb info locals' etc. has been done by the patches
introducing
> > 'machdep->get_cpu_reg' and this series fixing some issues in that.
> >
> > Other architectures should be able to fix these gdb functionalities by
> > simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
> >
> > The reasoning behind that has been explained with a diagram in commit
> > description of patch #1
> >
> > I will assist with my findings/observations fixing it on ppc64 whenever needed.
> >
> > Changelog:
> > ==========
> >
> > V5:
> > + changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
> > code
> > + changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
> > remove unneeded if condition
> > + changes in patch #5: implement refresh regcache on per thread, instead of all
> > threads at once
> >
> > V4:
> > + fix segmentation fault in live debugging (change in patch #1)
> > + mention live debugging not supported in cover letter and patch #1
> > + fixed some checkpatch warnings (change in patch #5)
> >
> > V3:
> > + default gdb thread will be the crashing thread, instead of being
> > thread '0'
> > + synchronise crash cpu and gdb thread context
> > + fix bug in gdb_interface, that replaced gdb's output stream, losing
> > output in some cases, such as info threads and extra output in info
> > variables
> > + fix 'info threads'
> >
> > RFC V2:
> > - removed patch implementing 'frame', 'up', 'down' in
crash
> > - updated the cover letter by removing the mention of those commands other
> > than the respective gdb passthrough
> >
> > Aditya Gupta (5):
> > ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
> > remove 'frame' from prohibited commands list
> > synchronise cpu context changes between crash/gdb
> > fix gdb_interface: restore gdb's output streams at end of
> > gdb_interface
> > fix 'info threads' command
> >
> > crash_target.c | 44 ++++++++++++++++
> > defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
> > gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
> > gdb_interface.c | 2 +-
> > kernel.c | 47 +++++++++++++++--
> > ppc64.c | 95 +++++++++++++++++++++++++++++++++--
> > task.c | 14 ++++++
> > tools.c | 2 +-
> > 8 files changed, 434 insertions(+), 10 deletions(-)
> >
>
9:27 a.m.
Hi Lianbo,
Found the reason for the warning, basically it was due to gdb trying various
functions/unwinders, which are okay to fail, one of which tried accessing
instructions of a function marked '__init' in the kernel.
More details follow.
crash> gdb bt
#0 <unavailable> in ?? ()
#1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:111
#2 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#5 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#11 0xc000000000011030 in rest_init () at init/main.c:730
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
#12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
#13 0xc0000000020051ec in start_kernel () at init/main.c:1072
#14 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
These warnings were caused due to gdb accessing some addresses which were freed
by the kernel, and thus are expected to not be available ('page excluded').
---------
Q. Which addresses ?
I noticed that the addresses in the warnings are instructions of the given
function. For example, before frame #13 (`start_kernel`), we see it's PC was
`0xc0000000020051ec`, and it failed accesing `...20051e4` and `...20051e8`,
these are instructions just before the call was made.
Verified this from kernel disassembly also.
Q. Why are the addresses unavailable ?
The instructions are of `start_kernel` which is defined as `__init` in the linux
kernel. Such functions are intended to be called only during initialisation, and
the instructions/code of these functions are put in '.init.text' section in the
ELF file, which "is freed later by free_initmem", to free some memory used by
these functions which will not be called later.
Q. Why did GDB even try to access instructions while unwinding ?
While unwinding the frames, GDB is trying many functions to see if any one can
give it the previous frame's registers (aka unwinders).
One of the the functions is `tramp_frame_sniffer`, which relies on checking the
instructions at and around the PC, to see if it's a trampoline frame, and
proceed further.
This is the sequence of unwinders GDB is trying for every frame:
dummy_frame_unwind
dwarf2_tailcall_frame_unwind
inline_frame_unwind
0x11567ff0 -> tramp_frame_sniffer
0x11567f90 -> tramp_frame_sniffer
dwarf2_frame_unwind ==================> CORRECT ONE
dwarf2_signal_frame_unwind
rs6000_epilogue_frame_unwind
rs6000_frame_unwind
GDB tries all these unwinders in sequence, till one of them is able to get the
registers correctly. Here, for most frames, `dwarf2_frame_unwind` is the correct
unwinder which uses the DWARF debuginfo to unwind.
Q. Is it harmless ?
Yes, all unwinders which were tried before chosing the correct one, are expected
to fail.
About address being unavailable, that also is harmless because:
1. The functions instructions were freed by the kernel, so the data at those
address being missing is expected
2. Even GDB is okay with a read failing, in this case, tramp_frame_sniffer
simply returns if it's unable to get an address.
------------
Btw, this only happens with CPU0 WHEN it has idle task. Since unlike other CPUs
idle tasks, CPU 0's idle task is called during this initialisation sequence,
thus it's backtrace having very early init functions.
Since this is expected and harmless (gdb handles such cases), I am thinking to
ignore this warning when called by gdb, what do you suggest ?
i) Add flag "IGNORE_PAGE_EXCLUDED" to 'readmem' call when done by
gdb, and
only print the error if this is true:
if ((error_handle & IGNORE_PAGE_EXCLUDED) && (CRASHDEBUG(0))
...
This way, it will still be visible with -d 1 and above debug levels.
ii) Append 'quiet' to the 'error_handle':
This way we don't have to introduce a flag, but it might skip showing other
warnings/errors which might be of interest
Other than this, there was a bug introduced in V5, where default thread is still
thread 1 (CPU 0) at initialisation, which has been fixed, and will also fix the
'unavailable' frame being shown. Though it's funny that bug helped in a way
to
uncover these warnings, this would have hit in earlier versions also.
Thanks,
Aditya Gupta
> crash> gdb frame
> #0 <unavailable> in ?? ()
> crash> set gdb on
> gdb: on
> gdb> bt
> #0 <unavailable> in ?? ()
> #1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:111
> #2 0xc000000001004dd8 in cede_processor () at
> ./arch/powerpc/include/asm/plpar_wrappers.h:37
> #3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> #4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> drv=<optimized out>, index=<optimized out>) at
> drivers/cpuidle/cpuidle-pseries.c:256
> #5 0xc000000001004498 in cpuidle_enter_state
> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> #6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
> drivers/cpuidle/cpuidle.c:388
> #7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> kernel/sched/idle.c:134
> #8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> #9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> #10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> kernel/sched/idle.c:380
> #11 0xc000000000011030 in rest_init () at init/main.c:730
> #12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> #13 0xc0000000020051ec in start_kernel () at init/main.c:1072
> #14 0xc00000000000e788 in start_here_common () at
> arch/powerpc/kernel/head_64.S:1039
> gdb> info threads
> Id Target Id Frame
> * 1 CPU 0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 2 CPU 1 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 3 CPU 2 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 4 CPU 3 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 5 CPU 4 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 6 CPU 5 0xc00000000028b5e8 in crash_setup_regs
> (oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
> ./arch/powerpc/include/asm/kexec.h:69
> 7 CPU 6 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 8 CPU 7 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem_callback"
>
> gdb> thread 1
> [Switching to thread 1 (CPU 0)]
> #0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 114 li r4,0
> gdb> bt
> #0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> #1 0xc000000001004dd8 in cede_processor () at
> ./arch/powerpc/include/asm/plpar_wrappers.h:37
> #2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> #3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> drv=<optimized out>, index=<optimized out>) at
> drivers/cpuidle/cpuidle-pseries.c:256
> #4 0xc000000001004498 in cpuidle_enter_state
> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
> drivers/cpuidle/cpuidle.c:388
> #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> kernel/sched/idle.c:134
> #7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> kernel/sched/idle.c:380
> #10 0xc000000000011030 in rest_init () at init/main.c:730
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem callback"
> #11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem callback"
> #12 0xc0000000020051ec in start_kernel () at init/main.c:1072
> #13 0xc00000000000e788 in start_here_common () at
> arch/powerpc/kernel/head_64.S:1039
> gdb> thread 3
> [Switching to thread 3 (CPU 2)]
> #0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 114 li r4,0
> gdb> bt
> #0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> #1 0xc000000001004dd8 in cede_processor () at
> ./arch/powerpc/include/asm/plpar_wrappers.h:37
> #2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> #3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> drv=<optimized out>, index=<optimized out>) at
> drivers/cpuidle/cpuidle-pseries.c:256
> #4 0xc000000001004498 in cpuidle_enter_state
> (dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558
> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> <pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized out>) at
> drivers/cpuidle/cpuidle.c:388
> #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at
> kernel/sched/idle.c:134
> #7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> kernel/sched/idle.c:380
> #10 0xc00000000005f048 in start_secondary (unused=<optimized out>) at
> arch/powerpc/kernel/smp.c:1680
> #11 0xc00000000000e058 in start_secondary_prolog () at
> arch/powerpc/kernel/head_64.S:885
> gdb>
>
>
> Could you please check it again? Or am I missing anything? I did the test
> based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
>
> BTW: I did not see the similar issues after applying the v4 patch set.
>
>
> Thanks
>
> Lianbo
>
> >
> > Known Issues:
> > =============
> >
> > 1. In gdb mode, 'bt' might fail to show backtrace in few vmcores
collected
> > from older kernels. This is a known issue due to register mismatch, and
> > its fix has been merged upstream:
> >
> > This can also cause some 'invalid kernel virtual address' errors
during gdb
> > unwinding the stack registers
> >
> > Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
> >
> > Fixing GDB passthroughs on other architectures
> > ==============================================
> >
> > Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
> > thread', 'gdb info locals' etc. has been done by the patches
introducing
> > 'machdep->get_cpu_reg' and this series fixing some issues in that.
> >
> > Other architectures should be able to fix these gdb functionalities by
> > simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
> >
> > The reasoning behind that has been explained with a diagram in commit
> > description of patch #1
> >
> > I will assist with my findings/observations fixing it on ppc64 whenever needed.
> >
> > Changelog:
> > ==========
> >
> > V5:
> > + changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
> > code
> > + changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
> > remove unneeded if condition
> > + changes in patch #5: implement refresh regcache on per thread, instead of all
> > threads at once
> >
> > V4:
> > + fix segmentation fault in live debugging (change in patch #1)
> > + mention live debugging not supported in cover letter and patch #1
> > + fixed some checkpatch warnings (change in patch #5)
> >
> > V3:
> > + default gdb thread will be the crashing thread, instead of being
> > thread '0'
> > + synchronise crash cpu and gdb thread context
> > + fix bug in gdb_interface, that replaced gdb's output stream, losing
> > output in some cases, such as info threads and extra output in info
> > variables
> > + fix 'info threads'
> >
> > RFC V2:
> > - removed patch implementing 'frame', 'up', 'down' in
crash
> > - updated the cover letter by removing the mention of those commands other
> > than the respective gdb passthrough
> >
> > Aditya Gupta (5):
> > ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
> > remove 'frame' from prohibited commands list
> > synchronise cpu context changes between crash/gdb
> > fix gdb_interface: restore gdb's output streams at end of
> > gdb_interface
> > fix 'info threads' command
> >
> > crash_target.c | 44 ++++++++++++++++
> > defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
> > gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
> > gdb_interface.c | 2 +-
> > kernel.c | 47 +++++++++++++++--
> > ppc64.c | 95 +++++++++++++++++++++++++++++++++--
> > task.c | 14 ++++++
> > tools.c | 2 +-
> > 8 files changed, 434 insertions(+), 10 deletions(-)
> >
>
11:53 p.m.
On 1/4/24 23:27, Aditya Gupta wrote:
Hi Lianbo,
Found the reason for the warning, basically it was due to gdb trying various
functions/unwinders, which are okay to fail, one of which tried accessing
instructions of a function marked '__init' in the kernel.
More details follow.
Thank you for investigating the details, Aditya.
It helps me a lot.
> crash> gdb bt
> #0 <unavailable> in ?? ()
> #1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:111
> #2 0xc000000001004dd8 in cede_processor () at
> ./arch/powerpc/include/asm/plpar_wrappers.h:37
> #3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> #4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> drv=<optimized out>, index=<optimized out>) at
> drivers/cpuidle/cpuidle-pseries.c:256
> #5 0xc000000001004498 in cpuidle_enter_state
> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> #6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
> drivers/cpuidle/cpuidle.c:388
> #7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> kernel/sched/idle.c:134
> #8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> #9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> #10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> kernel/sched/idle.c:380
> #11 0xc000000000011030 in rest_init () at init/main.c:730
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem callback"
> #12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem callback"
> #13 0xc0000000020051ec in start_kernel () at init/main.c:1072
> #14 0xc00000000000e788 in start_here_common () at
> arch/powerpc/kernel/head_64.S:1039
These warnings were caused due to gdb accessing some addresses which were freed
by the kernel, and thus are expected to not be available ('page excluded').
---------
Q. Which addresses ?
I noticed that the addresses in the warnings are instructions of the given
function. For example, before frame #13 (`start_kernel`), we see it's PC was
`0xc0000000020051ec`, and it failed accesing `...20051e4` and `...20051e8`,
these are instructions just before the call was made.
Verified this from kernel disassembly also.
Q. Why are the addresses unavailable ?
The instructions are of `start_kernel` which is defined as `__init` in the linux
kernel. Such functions are intended to be called only during initialisation, and
the instructions/code of these functions are put in '.init.text' section in the
ELF file, which "is freed later by free_initmem", to free some memory used by
these functions which will not be called later.
Q. Why did GDB even try to access instructions while unwinding ?
While unwinding the frames, GDB is trying many functions to see if any one can
give it the previous frame's registers (aka unwinders).
One of the the functions is `tramp_frame_sniffer`, which relies on checking the
instructions at and around the PC, to see if it's a trampoline frame, and
proceed further.
This is the sequence of unwinders GDB is trying for every frame:
dummy_frame_unwind
dwarf2_tailcall_frame_unwind
inline_frame_unwind
0x11567ff0 -> tramp_frame_sniffer
0x11567f90 -> tramp_frame_sniffer
dwarf2_frame_unwind ==================> CORRECT ONE
dwarf2_signal_frame_unwind
rs6000_epilogue_frame_unwind
rs6000_frame_unwind
GDB tries all these unwinders in sequence, till one of them is able to get the
registers correctly. Here, for most frames, `dwarf2_frame_unwind` is the correct
unwinder which uses the DWARF debuginfo to unwind.
Q. Is it harmless ?
Yes, all unwinders which were tried before chosing the correct one, are expected
to fail.
About address being unavailable, that also is harmless because:
1. The functions instructions were freed by the kernel, so the data at those
address being missing is expected
2. Even GDB is okay with a read failing, in this case, tramp_frame_sniffer
simply returns if it's unable to get an address.
------------
Btw, this only happens with CPU0 WHEN it has idle task. Since unlike other CPUs
idle tasks, CPU 0's idle task is called during this initialisation sequence,
thus it's backtrace having very early init functions.
It's true.
Since this is expected and harmless (gdb handles such cases), I am
thinking to
ignore this warning when called by gdb, what do you suggest ?
If this does happen and it's rare, it might be good to have a warning
for users.
At least it reflects the actual situation, although the warning output
looks not very friendly. So I would suggest still leaving it there for
the time being.
i) Add flag "IGNORE_PAGE_EXCLUDED" to
'readmem' call when done by gdb, and
only print the error if this is true:
if ((error_handle & IGNORE_PAGE_EXCLUDED) && (CRASHDEBUG(0))
...
This way, it will still be visible with -d 1 and above debug levels.
ii) Append 'quiet' to the 'error_handle':
This way we don't have to introduce a flag, but it might skip showing other
warnings/errors which might be of interest
Other than this, there was a bug introduced in V5, where default thread is still
thread 1 (CPU 0) at initialisation, which has been fixed, and will also fix the
Sounds good.
'unavailable' frame being shown. Though it's funny that
bug helped in a way to
uncover these warnings, this would have hit in earlier versions also.
Indeed, it can be triggered conditionally.
I have no other issues, thank you for the great job, Aditya.
Thanks
Lianbo
Thanks,
Aditya Gupta
> crash> gdb frame
> #0 <unavailable> in ?? ()
> crash> set gdb on
> gdb: on
> gdb> bt
> #0 <unavailable> in ?? ()
> #1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:111
> #2 0xc000000001004dd8 in cede_processor () at
> ./arch/powerpc/include/asm/plpar_wrappers.h:37
> #3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> #4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> drv=<optimized out>, index=<optimized out>) at
> drivers/cpuidle/cpuidle-pseries.c:256
> #5 0xc000000001004498 in cpuidle_enter_state
> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> #6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
> drivers/cpuidle/cpuidle.c:388
> #7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> kernel/sched/idle.c:134
> #8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> #9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> #10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> kernel/sched/idle.c:380
> #11 0xc000000000011030 in rest_init () at init/main.c:730
> #12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> #13 0xc0000000020051ec in start_kernel () at init/main.c:1072
> #14 0xc00000000000e788 in start_here_common () at
> arch/powerpc/kernel/head_64.S:1039
> gdb> info threads
> Id Target Id Frame
> * 1 CPU 0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 2 CPU 1 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 3 CPU 2 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 4 CPU 3 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 5 CPU 4 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 6 CPU 5 0xc00000000028b5e8 in crash_setup_regs
> (oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
> ./arch/powerpc/include/asm/kexec.h:69
> 7 CPU 6 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 8 CPU 7 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem_callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem_callback"
>
> gdb> thread 1
> [Switching to thread 1 (CPU 0)]
> #0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 114 li r4,0
> gdb> bt
> #0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> #1 0xc000000001004dd8 in cede_processor () at
> ./arch/powerpc/include/asm/plpar_wrappers.h:37
> #2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> #3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> drv=<optimized out>, index=<optimized out>) at
> drivers/cpuidle/cpuidle-pseries.c:256
> #4 0xc000000001004498 in cpuidle_enter_state
> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
> drivers/cpuidle/cpuidle.c:388
> #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> kernel/sched/idle.c:134
> #7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> kernel/sched/idle.c:380
> #10 0xc000000000011030 in rest_init () at init/main.c:730
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c80 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c7c type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c000000002004c78 type:
> "gdb_readmem callback"
> #11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051ec type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> "gdb_readmem callback"
> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> "gdb_readmem callback"
> #12 0xc0000000020051ec in start_kernel () at init/main.c:1072
> #13 0xc00000000000e788 in start_here_common () at
> arch/powerpc/kernel/head_64.S:1039
> gdb> thread 3
> [Switching to thread 3 (CPU 2)]
> #0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> 114 li r4,0
> gdb> bt
> #0 plpar_hcall_norets_notrace () at
> arch/powerpc/platforms/pseries/hvCall.S:114
> #1 0xc000000001004dd8 in cede_processor () at
> ./arch/powerpc/include/asm/plpar_wrappers.h:37
> #2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> #3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> drv=<optimized out>, index=<optimized out>) at
> drivers/cpuidle/cpuidle-pseries.c:256
> #4 0xc000000001004498 in cpuidle_enter_state
> (dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558
> <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> <pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized out>) at
> drivers/cpuidle/cpuidle.c:388
> #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at
> kernel/sched/idle.c:134
> #7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> kernel/sched/idle.c:380
> #10 0xc00000000005f048 in start_secondary (unused=<optimized out>) at
> arch/powerpc/kernel/smp.c:1680
> #11 0xc00000000000e058 in start_secondary_prolog () at
> arch/powerpc/kernel/head_64.S:885
> gdb>
>
>
> Could you please check it again? Or am I missing anything? I did the test
> based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
>
> BTW: I did not see the similar issues after applying the v4 patch set.
>
>
> Thanks
>
> Lianbo
>
>> Known Issues:
>> =============
>>
>> 1. In gdb mode, 'bt' might fail to show backtrace in few vmcores
collected
>> from older kernels. This is a known issue due to register mismatch, and
>> its fix has been merged upstream:
>>
>> This can also cause some 'invalid kernel virtual address' errors
during gdb
>> unwinding the stack registers
>>
>> Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
>>
>> Fixing GDB passthroughs on other architectures
>> ==============================================
>>
>> Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
>> thread', 'gdb info locals' etc. has been done by the patches
introducing
>> 'machdep->get_cpu_reg' and this series fixing some issues in that.
>>
>> Other architectures should be able to fix these gdb functionalities by
>> simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
>>
>> The reasoning behind that has been explained with a diagram in commit
>> description of patch #1
>>
>> I will assist with my findings/observations fixing it on ppc64 whenever needed.
>>
>> Changelog:
>> ==========
>>
>> V5:
>> + changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
>> code
>> + changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
>> remove unneeded if condition
>> + changes in patch #5: implement refresh regcache on per thread, instead of all
>> threads at once
>>
>> V4:
>> + fix segmentation fault in live debugging (change in patch #1)
>> + mention live debugging not supported in cover letter and patch #1
>> + fixed some checkpatch warnings (change in patch #5)
>>
>> V3:
>> + default gdb thread will be the crashing thread, instead of being
>> thread '0'
>> + synchronise crash cpu and gdb thread context
>> + fix bug in gdb_interface, that replaced gdb's output stream, losing
>> output in some cases, such as info threads and extra output in info
>> variables
>> + fix 'info threads'
>>
>> RFC V2:
>> - removed patch implementing 'frame', 'up', 'down' in
crash
>> - updated the cover letter by removing the mention of those commands other
>> than the respective gdb passthrough
>>
>> Aditya Gupta (5):
>> ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
>> remove 'frame' from prohibited commands list
>> synchronise cpu context changes between crash/gdb
>> fix gdb_interface: restore gdb's output streams at end of
>> gdb_interface
>> fix 'info threads' command
>>
>> crash_target.c | 44 ++++++++++++++++
>> defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
>> gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
>> gdb_interface.c | 2 +-
>> kernel.c | 47 +++++++++++++++--
>> ppc64.c | 95 +++++++++++++++++++++++++++++++++--
>> task.c | 14 ++++++
>> tools.c | 2 +-
>> 8 files changed, 434 insertions(+), 10 deletions(-)
>>
1:07 a.m.
Hi Lianbo,
On Fri, Jan 05, 2024 at 01:53:53PM +0800, Lianbo Jiang wrote:
>
> ...<snip>...
>
> Q. Is it harmless ?
>
> Yes, all unwinders which were tried before chosing the correct one, are expected
> to fail.
>
> About address being unavailable, that also is harmless because:
> 1. The functions instructions were freed by the kernel, so the data at those
> address being missing is expected
> 2. Even GDB is okay with a read failing, in this case, tramp_frame_sniffer
> simply returns if it's unable to get an address.
>
> ------------
>
> Btw, this only happens with CPU0 WHEN it has idle task. Since unlike other CPUs
> idle tasks, CPU 0's idle task is called during this initialisation sequence,
> thus it's backtrace having very early init functions.
It's true.
> Since this is expected and harmless (gdb handles such cases), I am thinking to
> ignore this warning when called by gdb, what do you suggest ?
If this does happen and it's rare, it might be good to have a warning for
users.
At least it reflects the actual situation, although the warning output looks
not very friendly. So I would suggest still leaving it there for the time
being.
Yes, the warning is not very friendly but it does tell that the page is
excluded.
> i) Add flag "IGNORE_PAGE_EXCLUDED" to 'readmem' call when
done by gdb, and
> only print the error if this is true:
> if ((error_handle & IGNORE_PAGE_EXCLUDED) && (CRASHDEBUG(0))
> ...
> This way, it will still be visible with -d 1 and above debug levels.
> ii) Append 'quiet' to the 'error_handle':
> This way we don't have to introduce a flag, but it might skip showing
other
> warnings/errors which might be of interest
>
> Other than this, there was a bug introduced in V5, where default thread is still
> thread 1 (CPU 0) at initialisation, which has been fixed, and will also fix the
Sounds good.
So for now there's no change needed to the 'page excluded' warnings ?
> 'unavailable' frame being shown. Though it's funny
that bug helped in a way to
> uncover these warnings, this would have hit in earlier versions also.
Indeed, it can be triggered conditionally.
I have no other issues, thank you for the great job, Aditya.
Thanks,
Aditya Gupta
>
>
> Thanks
>
> Lianbo
>
> >
> > Thanks,
> > Aditya Gupta
> >
> > > crash> gdb frame
> > > #0 <unavailable> in ?? ()
> > > crash> set gdb on
> > > gdb: on
> > > gdb> bt
> > > #0 <unavailable> in ?? ()
> > > #1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:111
> > > #2 0xc000000001004dd8 in cede_processor () at
> > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > #3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> > > #4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> > > drv=<optimized out>, index=<optimized out>) at
> > > drivers/cpuidle/cpuidle-pseries.c:256
> > > #5 0xc000000001004498 in cpuidle_enter_state
> > > (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> > > <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
> > > #6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized
out>) at
> > > drivers/cpuidle/cpuidle.c:388
> > > #7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized
out>,
> > > dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> > > kernel/sched/idle.c:134
> > > #8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> > > #9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > #10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>)
at
> > > kernel/sched/idle.c:380
> > > #11 0xc000000000011030 in rest_init () at init/main.c:730
> > > #12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> > > #13 0xc0000000020051ec in start_kernel () at init/main.c:1072
> > > #14 0xc00000000000e788 in start_here_common () at
> > > arch/powerpc/kernel/head_64.S:1039
> > > gdb> info threads
> > > Id Target Id Frame
> > > * 1 CPU 0 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > 2 CPU 1 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > 3 CPU 2 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > 4 CPU 3 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > 5 CPU 4 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > 6 CPU 5 0xc00000000028b5e8 in crash_setup_regs
> > > (oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
> > > ./arch/powerpc/include/asm/kexec.h:69
> > > 7 CPU 6 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > 8 CPU 7 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > gdb: page excluded: kernel virtual address: c000000002004c80 type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c7c type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c78 type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c80 type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c7c type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c78 type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051ec type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051ec type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> > > "gdb_readmem_callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> > > "gdb_readmem_callback"
> > >
> > > gdb> thread 1
> > > [Switching to thread 1 (CPU 0)]
> > > #0 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > 114 li r4,0
> > > gdb> bt
> > > #0 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > #1 0xc000000001004dd8 in cede_processor () at
> > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > #2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> > > #3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> > > drv=<optimized out>, index=<optimized out>) at
> > > drivers/cpuidle/cpuidle-pseries.c:256
> > > #4 0xc000000001004498 in cpuidle_enter_state
> > > (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> > > <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
> > > #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized
out>) at
> > > drivers/cpuidle/cpuidle.c:388
> > > #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized
out>,
> > > dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> > > kernel/sched/idle.c:134
> > > #7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> > > #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>)
at
> > > kernel/sched/idle.c:380
> > > #10 0xc000000000011030 in rest_init () at init/main.c:730
> > > gdb: page excluded: kernel virtual address: c000000002004c80 type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c7c type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c78 type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c80 type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c7c type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c000000002004c78 type:
> > > "gdb_readmem callback"
> > > #11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> > > gdb: page excluded: kernel virtual address: c0000000020051ec type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051ec type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> > > "gdb_readmem callback"
> > > gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> > > "gdb_readmem callback"
> > > #12 0xc0000000020051ec in start_kernel () at init/main.c:1072
> > > #13 0xc00000000000e788 in start_here_common () at
> > > arch/powerpc/kernel/head_64.S:1039
> > > gdb> thread 3
> > > [Switching to thread 3 (CPU 2)]
> > > #0 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > 114 li r4,0
> > > gdb> bt
> > > #0 plpar_hcall_norets_notrace () at
> > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > #1 0xc000000001004dd8 in cede_processor () at
> > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > #2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> > > #3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> > > drv=<optimized out>, index=<optimized out>) at
> > > drivers/cpuidle/cpuidle-pseries.c:256
> > > #4 0xc000000001004498 in cpuidle_enter_state
> > > (dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558
> > > <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
> > > #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > <pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized
out>) at
> > > drivers/cpuidle/cpuidle.c:388
> > > #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized
out>,
> > > dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at
> > > kernel/sched/idle.c:134
> > > #7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> > > #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>)
at
> > > kernel/sched/idle.c:380
> > > #10 0xc00000000005f048 in start_secondary (unused=<optimized out>)
at
> > > arch/powerpc/kernel/smp.c:1680
> > > #11 0xc00000000000e058 in start_secondary_prolog () at
> > > arch/powerpc/kernel/head_64.S:885
> > > gdb>
> > >
> > >
> > > Could you please check it again? Or am I missing anything? I did the test
> > > based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
> > >
> > > BTW: I did not see the similar issues after applying the v4 patch set.
> > >
> > >
> > > Thanks
> > >
> > > Lianbo
> > >
> > > > Known Issues:
> > > > =============
> > > >
> > > > 1. In gdb mode, 'bt' might fail to show backtrace in few
vmcores collected
> > > > from older kernels. This is a known issue due to register
mismatch, and
> > > > its fix has been merged upstream:
> > > >
> > > > This can also cause some 'invalid kernel virtual address'
errors during gdb
> > > > unwinding the stack registers
> > > >
> > > > Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
> > > >
> > > > Fixing GDB passthroughs on other architectures
> > > > ==============================================
> > > >
> > > > Much of the work for making gdb passthroughs like 'gdb bt',
'gdb
> > > > thread', 'gdb info locals' etc. has been done by the
patches introducing
> > > > 'machdep->get_cpu_reg' and this series fixing some issues
in that.
> > > >
> > > > Other architectures should be able to fix these gdb functionalities
by
> > > > simply implementing 'machdep->get_cpu_reg (cpu, regno,
...)'.
> > > >
> > > > The reasoning behind that has been explained with a diagram in commit
> > > > description of patch #1
> > > >
> > > > I will assist with my findings/observations fixing it on ppc64
whenever needed.
> > > >
> > > > Changelog:
> > > > ==========
> > > >
> > > > V5:
> > > > + changes in patch #1: made ppc64_get_cpu_reg static, and remove
unreachable
> > > > code
> > > > + changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
> > > > remove unneeded if condition
> > > > + changes in patch #5: implement refresh regcache on per thread,
instead of all
> > > > threads at once
> > > >
> > > > V4:
> > > > + fix segmentation fault in live debugging (change in patch #1)
> > > > + mention live debugging not supported in cover letter and patch #1
> > > > + fixed some checkpatch warnings (change in patch #5)
> > > >
> > > > V3:
> > > > + default gdb thread will be the crashing thread, instead of being
> > > > thread '0'
> > > > + synchronise crash cpu and gdb thread context
> > > > + fix bug in gdb_interface, that replaced gdb's output stream,
losing
> > > > output in some cases, such as info threads and extra output in
info
> > > > variables
> > > > + fix 'info threads'
> > > >
> > > > RFC V2:
> > > > - removed patch implementing 'frame', 'up',
'down' in crash
> > > > - updated the cover letter by removing the mention of those
commands other
> > > > than the respective gdb passthrough
> > > >
> > > > Aditya Gupta (5):
> > > > ppc64: correct gdb passthroughs by implementing
machdep->get_cpu_reg
> > > > remove 'frame' from prohibited commands list
> > > > synchronise cpu context changes between crash/gdb
> > > > fix gdb_interface: restore gdb's output streams at end of
> > > > gdb_interface
> > > > fix 'info threads' command
> > > >
> > > > crash_target.c | 44 ++++++++++++++++
> > > > defs.h | 130
+++++++++++++++++++++++++++++++++++++++++++++++-
> > > > gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
> > > > gdb_interface.c | 2 +-
> > > > kernel.c | 47 +++++++++++++++--
> > > > ppc64.c | 95 +++++++++++++++++++++++++++++++++--
> > > > task.c | 14 ++++++
> > > > tools.c | 2 +-
> > > > 8 files changed, 434 insertions(+), 10 deletions(-)
> > > >
>
1:26 a.m.
On 1/5/24 15:07, Aditya Gupta wrote:
Hi Lianbo,
On Fri, Jan 05, 2024 at 01:53:53PM +0800, Lianbo Jiang wrote:
>> ...<snip>...
>>
>> Q. Is it harmless ?
>>
>> Yes, all unwinders which were tried before chosing the correct one, are expected
>> to fail.
>>
>> About address being unavailable, that also is harmless because:
>> 1. The functions instructions were freed by the kernel, so the data at those
>> address being missing is expected
>> 2. Even GDB is okay with a read failing, in this case, tramp_frame_sniffer
>> simply returns if it's unable to get an address.
>>
>> ------------
>>
>> Btw, this only happens with CPU0 WHEN it has idle task. Since unlike other CPUs
>> idle tasks, CPU 0's idle task is called during this initialisation sequence,
>> thus it's backtrace having very early init functions.
> It's true.
>> Since this is expected and harmless (gdb handles such cases), I am thinking to
>> ignore this warning when called by gdb, what do you suggest ?
> If this does happen and it's rare, it might be good to have a warning for
> users.
>
> At least it reflects the actual situation, although the warning output looks
> not very friendly. So I would suggest still leaving it there for the time
> being.
>
Yes, the warning is not very friendly but it does tell that the page is
excluded.
>> i) Add flag "IGNORE_PAGE_EXCLUDED" to 'readmem' call when
done by gdb, and
>> only print the error if this is true:
>> if ((error_handle & IGNORE_PAGE_EXCLUDED) && (CRASHDEBUG(0))
>> ...
>> This way, it will still be visible with -d 1 and above debug levels.
>> ii) Append 'quiet' to the 'error_handle':
>> This way we don't have to introduce a flag, but it might skip showing
other
>> warnings/errors which might be of interest
>>
>> Other than this, there was a bug introduced in V5, where default thread is still
>> thread 1 (CPU 0) at initialisation, which has been fixed, and will also fix the
> Sounds good.
So for now there's no change needed to the 'page excluded' warnings ?
Yes, no need changes this time.
I did not see the similar warnings in other situations, unless there are
a lot of warnings for this cases.
Thanks
Lianbo
>> 'unavailable' frame being shown. Though it's funny that bug helped in
a way to
>> uncover these warnings, this would have hit in earlier versions also.
> Indeed, it can be triggered conditionally.
>
> I have no other issues, thank you for the great job, Aditya.
Thanks,
Aditya Gupta
>
> Thanks
>
> Lianbo
>
>> Thanks,
>> Aditya Gupta
>>
>>> crash> gdb frame
>>> #0 <unavailable> in ?? ()
>>> crash> set gdb on
>>> gdb: on
>>> gdb> bt
>>> #0 <unavailable> in ?? ()
>>> #1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:111
>>> #2 0xc000000001004dd8 in cede_processor () at
>>> ./arch/powerpc/include/asm/plpar_wrappers.h:37
>>> #3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
>>> #4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
>>> drv=<optimized out>, index=<optimized out>) at
>>> drivers/cpuidle/cpuidle-pseries.c:256
>>> #5 0xc000000001004498 in cpuidle_enter_state
>>> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
>>> <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
>>> #6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
>>> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized
out>) at
>>> drivers/cpuidle/cpuidle.c:388
>>> #7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
>>> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized
out>,
>>> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
>>> kernel/sched/idle.c:134
>>> #8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
>>> #9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
>>> #10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
>>> kernel/sched/idle.c:380
>>> #11 0xc000000000011030 in rest_init () at init/main.c:730
>>> #12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
>>> #13 0xc0000000020051ec in start_kernel () at init/main.c:1072
>>> #14 0xc00000000000e788 in start_here_common () at
>>> arch/powerpc/kernel/head_64.S:1039
>>> gdb> info threads
>>> Id Target Id Frame
>>> * 1 CPU 0 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> 2 CPU 1 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> 3 CPU 2 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> 4 CPU 3 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> 5 CPU 4 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> 6 CPU 5 0xc00000000028b5e8 in crash_setup_regs
>>> (oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
>>> ./arch/powerpc/include/asm/kexec.h:69
>>> 7 CPU 6 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> 8 CPU 7 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> gdb: page excluded: kernel virtual address: c000000002004c80 type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c7c type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c78 type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c80 type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c7c type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c78 type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051ec type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051ec type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
>>> "gdb_readmem_callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
>>> "gdb_readmem_callback"
>>>
>>> gdb> thread 1
>>> [Switching to thread 1 (CPU 0)]
>>> #0 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> 114 li r4,0
>>> gdb> bt
>>> #0 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> #1 0xc000000001004dd8 in cede_processor () at
>>> ./arch/powerpc/include/asm/plpar_wrappers.h:37
>>> #2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
>>> #3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
>>> drv=<optimized out>, index=<optimized out>) at
>>> drivers/cpuidle/cpuidle-pseries.c:256
>>> #4 0xc000000001004498 in cpuidle_enter_state
>>> (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
>>> <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
>>> #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
>>> <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized
out>) at
>>> drivers/cpuidle/cpuidle.c:388
>>> #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
>>> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized
out>,
>>> dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
>>> kernel/sched/idle.c:134
>>> #7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
>>> #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
>>> #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
>>> kernel/sched/idle.c:380
>>> #10 0xc000000000011030 in rest_init () at init/main.c:730
>>> gdb: page excluded: kernel virtual address: c000000002004c80 type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c7c type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c78 type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c80 type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c7c type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c000000002004c78 type:
>>> "gdb_readmem callback"
>>> #11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
>>> gdb: page excluded: kernel virtual address: c0000000020051ec type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051ec type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051e8 type:
>>> "gdb_readmem callback"
>>> gdb: page excluded: kernel virtual address: c0000000020051e4 type:
>>> "gdb_readmem callback"
>>> #12 0xc0000000020051ec in start_kernel () at init/main.c:1072
>>> #13 0xc00000000000e788 in start_here_common () at
>>> arch/powerpc/kernel/head_64.S:1039
>>> gdb> thread 3
>>> [Switching to thread 3 (CPU 2)]
>>> #0 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> 114 li r4,0
>>> gdb> bt
>>> #0 plpar_hcall_norets_notrace () at
>>> arch/powerpc/platforms/pseries/hvCall.S:114
>>> #1 0xc000000001004dd8 in cede_processor () at
>>> ./arch/powerpc/include/asm/plpar_wrappers.h:37
>>> #2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
>>> #3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
>>> drv=<optimized out>, index=<optimized out>) at
>>> drivers/cpuidle/cpuidle-pseries.c:256
>>> #4 0xc000000001004498 in cpuidle_enter_state
>>> (dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558
>>> <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
>>> #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
>>> <pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized
out>) at
>>> drivers/cpuidle/cpuidle.c:388
>>> #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
>>> drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized
out>,
>>> dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at
>>> kernel/sched/idle.c:134
>>> #7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
>>> #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
>>> #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
>>> kernel/sched/idle.c:380
>>> #10 0xc00000000005f048 in start_secondary (unused=<optimized out>) at
>>> arch/powerpc/kernel/smp.c:1680
>>> #11 0xc00000000000e058 in start_secondary_prolog () at
>>> arch/powerpc/kernel/head_64.S:885
>>> gdb>
>>>
>>>
>>> Could you please check it again? Or am I missing anything? I did the test
>>> based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
>>>
>>> BTW: I did not see the similar issues after applying the v4 patch set.
>>>
>>>
>>> Thanks
>>>
>>> Lianbo
>>>
>>>> Known Issues:
>>>> =============
>>>>
>>>> 1. In gdb mode, 'bt' might fail to show backtrace in few vmcores
collected
>>>> from older kernels. This is a known issue due to register mismatch,
and
>>>> its fix has been merged upstream:
>>>>
>>>> This can also cause some 'invalid kernel virtual address'
errors during gdb
>>>> unwinding the stack registers
>>>>
>>>> Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
>>>>
>>>> Fixing GDB passthroughs on other architectures
>>>> ==============================================
>>>>
>>>> Much of the work for making gdb passthroughs like 'gdb bt',
'gdb
>>>> thread', 'gdb info locals' etc. has been done by the patches
introducing
>>>> 'machdep->get_cpu_reg' and this series fixing some issues in
that.
>>>>
>>>> Other architectures should be able to fix these gdb functionalities by
>>>> simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
>>>>
>>>> The reasoning behind that has been explained with a diagram in commit
>>>> description of patch #1
>>>>
>>>> I will assist with my findings/observations fixing it on ppc64 whenever
needed.
>>>>
>>>> Changelog:
>>>> ==========
>>>>
>>>> V5:
>>>> + changes in patch #1: made ppc64_get_cpu_reg static, and remove
unreachable
>>>> code
>>>> + changes in patch #3: fixed typo 'ppc64_renum' instead of
'ppc64_regnum',
>>>> remove unneeded if condition
>>>> + changes in patch #5: implement refresh regcache on per thread, instead
of all
>>>> threads at once
>>>>
>>>> V4:
>>>> + fix segmentation fault in live debugging (change in patch #1)
>>>> + mention live debugging not supported in cover letter and patch #1
>>>> + fixed some checkpatch warnings (change in patch #5)
>>>>
>>>> V3:
>>>> + default gdb thread will be the crashing thread, instead of being
>>>> thread '0'
>>>> + synchronise crash cpu and gdb thread context
>>>> + fix bug in gdb_interface, that replaced gdb's output stream,
losing
>>>> output in some cases, such as info threads and extra output in info
>>>> variables
>>>> + fix 'info threads'
>>>>
>>>> RFC V2:
>>>> - removed patch implementing 'frame', 'up',
'down' in crash
>>>> - updated the cover letter by removing the mention of those commands
other
>>>> than the respective gdb passthrough
>>>>
>>>> Aditya Gupta (5):
>>>> ppc64: correct gdb passthroughs by implementing
machdep->get_cpu_reg
>>>> remove 'frame' from prohibited commands list
>>>> synchronise cpu context changes between crash/gdb
>>>> fix gdb_interface: restore gdb's output streams at end of
>>>> gdb_interface
>>>> fix 'info threads' command
>>>>
>>>> crash_target.c | 44 ++++++++++++++++
>>>> defs.h | 130
+++++++++++++++++++++++++++++++++++++++++++++++-
>>>> gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
>>>> gdb_interface.c | 2 +-
>>>> kernel.c | 47 +++++++++++++++--
>>>> ppc64.c | 95 +++++++++++++++++++++++++++++++++--
>>>> task.c | 14 ++++++
>>>> tools.c | 2 +-
>>>> 8 files changed, 434 insertions(+), 10 deletions(-)
>>>>
1:36 a.m.
>
> ...<snip>...
>
> So for now there's no change needed to the 'page excluded' warnings ?
Yes, no need changes this time.
I did not see the similar warnings in other situations, unless there are a
lot of warnings for this cases.
Sure. Thanks a lot for your reviews, I sent a V6 with a minor change in
patch #5.
Thanks,
Aditya Gupta
>
> Thanks
>
> Lianbo
>
> >
> > > > 'unavailable' frame being shown. Though it's funny that
bug helped in a way to
> > > > uncover these warnings, this would have hit in earlier versions also.
> > > Indeed, it can be triggered conditionally.
> > >
> > > I have no other issues, thank you for the great job, Aditya.
> > Thanks,
> > Aditya Gupta
> >
> > >
> > > Thanks
> > >
> > > Lianbo
> > >
> > > > Thanks,
> > > > Aditya Gupta
> > > >
> > > > > crash> gdb frame
> > > > > #0 <unavailable> in ?? ()
> > > > > crash> set gdb on
> > > > > gdb: on
> > > > > gdb> bt
> > > > > #0 <unavailable> in ?? ()
> > > > > #1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:111
> > > > > #2 0xc000000001004dd8 in cede_processor () at
> > > > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > > > #3 check_and_cede_processor () at
drivers/cpuidle/cpuidle-pseries.c:83
> > > > > #4 0xc000000001005000 in shared_cede_loop (dev=<optimized
out>,
> > > > > drv=<optimized out>, index=<optimized out>) at
> > > > > drivers/cpuidle/cpuidle-pseries.c:256
> > > > > #5 0xc000000001004498 in cpuidle_enter_state
> > > > > (dev=dev@entry=0xc0000001ff5910c0,
drv=drv@entry=0xc000000002b8f558
> > > > > <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
> > > > > #6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > > > <pseries_idle_driver>, dev=0xc0000001ff5910c0,
index=<optimized out>) at
> > > > > drivers/cpuidle/cpuidle.c:388
> > > > > #7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized
out>,
> > > > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>,
dev=<optimized out>,
> > > > > dev@entry=0xc0000001ff5910c0, next_state=<optimized out>)
at
> > > > > kernel/sched/idle.c:134
> > > > > #8 0xc0000000001d5d68 in cpuidle_idle_call () at
kernel/sched/idle.c:215
> > > > > #9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > > > #10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized
out>) at
> > > > > kernel/sched/idle.c:380
> > > > > #11 0xc000000000011030 in rest_init () at init/main.c:730
> > > > > #12 0xc000000002004c80 in arch_call_rest_init () at
init/main.c:827
> > > > > #13 0xc0000000020051ec in start_kernel () at init/main.c:1072
> > > > > #14 0xc00000000000e788 in start_here_common () at
> > > > > arch/powerpc/kernel/head_64.S:1039
> > > > > gdb> info threads
> > > > > Id Target Id Frame
> > > > > * 1 CPU 0 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > 2 CPU 1 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > 3 CPU 2 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > 4 CPU 3 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > 5 CPU 4 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > 6 CPU 5 0xc00000000028b5e8 in
crash_setup_regs
> > > > > (oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
> > > > > ./arch/powerpc/include/asm/kexec.h:69
> > > > > 7 CPU 6 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > 8 CPU 7 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > gdb: page excluded: kernel virtual address: c000000002004c80
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c7c
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c78
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c80
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c7c
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c78
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051ec
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051e8
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051e4
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051ec
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051e8
type:
> > > > > "gdb_readmem_callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051e4
type:
> > > > > "gdb_readmem_callback"
> > > > >
> > > > > gdb> thread 1
> > > > > [Switching to thread 1 (CPU 0)]
> > > > > #0 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > 114 li r4,0
> > > > > gdb> bt
> > > > > #0 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > #1 0xc000000001004dd8 in cede_processor () at
> > > > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > > > #2 check_and_cede_processor () at
drivers/cpuidle/cpuidle-pseries.c:83
> > > > > #3 0xc000000001005000 in shared_cede_loop (dev=<optimized
out>,
> > > > > drv=<optimized out>, index=<optimized out>) at
> > > > > drivers/cpuidle/cpuidle-pseries.c:256
> > > > > #4 0xc000000001004498 in cpuidle_enter_state
> > > > > (dev=dev@entry=0xc0000001ff5910c0,
drv=drv@entry=0xc000000002b8f558
> > > > > <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
> > > > > #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > > > <pseries_idle_driver>, dev=0xc0000001ff5910c0,
index=<optimized out>) at
> > > > > drivers/cpuidle/cpuidle.c:388
> > > > > #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized
out>,
> > > > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>,
dev=<optimized out>,
> > > > > dev@entry=0xc0000001ff5910c0, next_state=<optimized out>)
at
> > > > > kernel/sched/idle.c:134
> > > > > #7 0xc0000000001d5d68 in cpuidle_idle_call () at
kernel/sched/idle.c:215
> > > > > #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > > > #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized
out>) at
> > > > > kernel/sched/idle.c:380
> > > > > #10 0xc000000000011030 in rest_init () at init/main.c:730
> > > > > gdb: page excluded: kernel virtual address: c000000002004c80
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c7c
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c78
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c80
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c7c
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c000000002004c78
type:
> > > > > "gdb_readmem callback"
> > > > > #11 0xc000000002004c80 in arch_call_rest_init () at
init/main.c:827
> > > > > gdb: page excluded: kernel virtual address: c0000000020051ec
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051e8
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051e4
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051ec
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051e8
type:
> > > > > "gdb_readmem callback"
> > > > > gdb: page excluded: kernel virtual address: c0000000020051e4
type:
> > > > > "gdb_readmem callback"
> > > > > #12 0xc0000000020051ec in start_kernel () at init/main.c:1072
> > > > > #13 0xc00000000000e788 in start_here_common () at
> > > > > arch/powerpc/kernel/head_64.S:1039
> > > > > gdb> thread 3
> > > > > [Switching to thread 3 (CPU 2)]
> > > > > #0 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > 114 li r4,0
> > > > > gdb> bt
> > > > > #0 plpar_hcall_norets_notrace () at
> > > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > > #1 0xc000000001004dd8 in cede_processor () at
> > > > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > > > #2 check_and_cede_processor () at
drivers/cpuidle/cpuidle-pseries.c:83
> > > > > #3 0xc000000001005000 in shared_cede_loop (dev=<optimized
out>,
> > > > > drv=<optimized out>, index=<optimized out>) at
> > > > > drivers/cpuidle/cpuidle-pseries.c:256
> > > > > #4 0xc000000001004498 in cpuidle_enter_state
> > > > > (dev=dev@entry=0xc0000001ff7910c0,
drv=drv@entry=0xc000000002b8f558
> > > > > <pseries_idle_driver>, index=index@entry=1) at
drivers/cpuidle/cpuidle.c:267
> > > > > #5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > > > <pseries_idle_driver>, dev=0xc0000001ff7910c0,
index=<optimized out>) at
> > > > > drivers/cpuidle/cpuidle.c:388
> > > > > #6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized
out>,
> > > > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>,
dev=<optimized out>,
> > > > > dev@entry=0xc0000001ff7910c0, next_state=<optimized out>)
at
> > > > > kernel/sched/idle.c:134
> > > > > #7 0xc0000000001d5d68 in cpuidle_idle_call () at
kernel/sched/idle.c:215
> > > > > #8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > > > #9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized
out>) at
> > > > > kernel/sched/idle.c:380
> > > > > #10 0xc00000000005f048 in start_secondary (unused=<optimized
out>) at
> > > > > arch/powerpc/kernel/smp.c:1680
> > > > > #11 0xc00000000000e058 in start_secondary_prolog () at
> > > > > arch/powerpc/kernel/head_64.S:885
> > > > > gdb>
> > > > >
> > > > >
> > > > > Could you please check it again? Or am I missing anything? I did
the test
> > > > > based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
> > > > >
> > > > > BTW: I did not see the similar issues after applying the v4 patch
set.
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > Lianbo
> > > > >
> > > > > > Known Issues:
> > > > > > =============
> > > > > >
> > > > > > 1. In gdb mode, 'bt' might fail to show backtrace in
few vmcores collected
> > > > > > from older kernels. This is a known issue due to
register mismatch, and
> > > > > > its fix has been merged upstream:
> > > > > >
> > > > > > This can also cause some 'invalid kernel virtual
address' errors during gdb
> > > > > > unwinding the stack registers
> > > > > >
> > > > > > Commit:
https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef78581...
> > > > > >
> > > > > > Fixing GDB passthroughs on other architectures
> > > > > > ==============================================
> > > > > >
> > > > > > Much of the work for making gdb passthroughs like 'gdb
bt', 'gdb
> > > > > > thread', 'gdb info locals' etc. has been done by
the patches introducing
> > > > > > 'machdep->get_cpu_reg' and this series fixing
some issues in that.
> > > > > >
> > > > > > Other architectures should be able to fix these gdb
functionalities by
> > > > > > simply implementing 'machdep->get_cpu_reg (cpu,
regno, ...)'.
> > > > > >
> > > > > > The reasoning behind that has been explained with a diagram
in commit
> > > > > > description of patch #1
> > > > > >
> > > > > > I will assist with my findings/observations fixing it on
ppc64 whenever needed.
> > > > > >
> > > > > > Changelog:
> > > > > > ==========
> > > > > >
> > > > > > V5:
> > > > > > + changes in patch #1: made ppc64_get_cpu_reg static, and
remove unreachable
> > > > > > code
> > > > > > + changes in patch #3: fixed typo 'ppc64_renum'
instead of 'ppc64_regnum',
> > > > > > remove unneeded if condition
> > > > > > + changes in patch #5: implement refresh regcache on per
thread, instead of all
> > > > > > threads at once
> > > > > >
> > > > > > V4:
> > > > > > + fix segmentation fault in live debugging (change in patch
#1)
> > > > > > + mention live debugging not supported in cover letter and
patch #1
> > > > > > + fixed some checkpatch warnings (change in patch #5)
> > > > > >
> > > > > > V3:
> > > > > > + default gdb thread will be the crashing thread, instead of
being
> > > > > > thread '0'
> > > > > > + synchronise crash cpu and gdb thread context
> > > > > > + fix bug in gdb_interface, that replaced gdb's output
stream, losing
> > > > > > output in some cases, such as info threads and extra
output in info
> > > > > > variables
> > > > > > + fix 'info threads'
> > > > > >
> > > > > > RFC V2:
> > > > > > - removed patch implementing 'frame',
'up', 'down' in crash
> > > > > > - updated the cover letter by removing the mention of
those commands other
> > > > > > than the respective gdb passthrough
> > > > > >
> > > > > > Aditya Gupta (5):
> > > > > > ppc64: correct gdb passthroughs by implementing
machdep->get_cpu_reg
> > > > > > remove 'frame' from prohibited commands list
> > > > > > synchronise cpu context changes between crash/gdb
> > > > > > fix gdb_interface: restore gdb's output streams at
end of
> > > > > > gdb_interface
> > > > > > fix 'info threads' command
> > > > > >
> > > > > > crash_target.c | 44 ++++++++++++++++
> > > > > > defs.h | 130
+++++++++++++++++++++++++++++++++++++++++++++++-
> > > > > > gdb-10.2.patch | 110
+++++++++++++++++++++++++++++++++++++++-
> > > > > > gdb_interface.c | 2 +-
> > > > > > kernel.c | 47 +++++++++++++++--
> > > > > > ppc64.c | 95
+++++++++++++++++++++++++++++++++--
> > > > > > task.c | 14 ++++++
> > > > > > tools.c | 2 +-
> > > > > > 8 files changed, 434 insertions(+), 10 deletions(-)
> > > > > >
>
176
days inactive
192
days old
devel@lists.crash-utility.osci.io
15 comments
3 participants
participants (3)
-
Aditya Gupta -
Lianbo Jiang -
Tao Liu