Hi Alexey, On Thu, Mar 14, 2024 at 6:29 PM Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; wrote: > > Support for GDB debugging of all tasks active and inactive. > Before this commit only active tasks were listed by "info threads" > with "CPU #" as a Target Id. > > "info threads" will now show all tasks, similar to "ps", example: > crash> info threads > Id Target Id Frame > * 1 0 swapper/0 0xffffffffadba19d4 in default_idle () at arch/x86/kernel/process.c:731 > 2 0 swapper/1 0xffffffffadba19d4 in default_idle () at arch/x86/kernel/process.c:731 > 3 0 swapper/2 0xffffffffadba19d4 in default_idle () at arch/x86/kernel/process.c:731 > 4 0 swapper/3 0xffffffffadba19d4 in default_idle () at arch/x86/kernel/process.c:731 > 5 0 swapper/4 0xffffffffadb97292 in context_switch (rf=0xffffbaf0000f3e88, next=0xffff9ecb04908000, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > ... > 730 970325 taskset 0xffffffffadb97292 in context_switch (rf=0xffffbaf006a0fd18, next=0xffff9ecb0aec0000, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > 731 975217 sleep 0xffffffffadb97292 in context_switch (rf=0xffffbaf005743c20, next=0xffff9ecac0692880, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > 732 975228 sleep 0xffffffffadb97292 in context_switch (rf=0xffffbaf00696fb58, next=0xffff9ecac0690000, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > ... > 876 976084 docker 0xffffffffadb97292 in context_switch (rf=0xffffbaf0153dbd10, next=0xffff9ecac0645100, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > 877 976085 systemd-userwor 0xffffffffadb97292 in context_switch (rf=0xffffbaf0153cbc58, next=0xffff9ecac0645100, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > 878 976086 systemd-userwor 0xffffffffadb97292 in context_switch (rf=0xffffbaf0153e3c58, next=0xffffffffaec15a40 <init_task>, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > 879 976087 systemd-userwor 0xffffffffadb97292 in context_switch (rf=0xffffbaf0153ebc58, next=0xffffffffaec15a40 <init_task>, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > Where "Target ID" contains "PID COMM" of the task > I see here, we chose a different tech path of viewing arbitrary tasks. You improved the "info threads" as an alternative to crash "ps" cmd, and use "thread X" to switch to task X and view its stacktrace. However mine is using crash "ps" cmd first, getting specific tasks pid or task struct, then using crash cmd "set <pid>" or "set <task_struct>" to switch to task X and view its stacktrace. I notice there is one problem of using the "info threads as ps" tech path, it is difficult to handle the live debug, see the comment below: > Example of "731 975217 sleep" debugging, real case, trying to > figure out why sleep was stuck in uninterruptable sleep. > Backtrace using crash: > crash> ps | grep 975217 > 975217 969797 3 ffff9ecb3956a880 UN 0.0 0 0 sleep > crash> bt 975217 > PID: 975217 TASK: ffff9ecb3956a880 CPU: 3 COMMAND: "sleep" > #0 [ffffbaf005743ba0] __schedule at ffffffffadb97292 > #1 [ffffbaf005743c60] schedule at ffffffffadb982b8 > #2 [ffffbaf005743c80] rwbase_write_lock at ffffffffadb9aed7 > #3 [ffffbaf005743cc0] down_write at ffffffffadb9b133 > #4 [ffffbaf005743cd0] unlink_file_vma at ffffffffad2b0e2e > #5 [ffffbaf005743cf8] free_pgtables at ffffffffad2a47b0 > #6 [ffffbaf005743d88] exit_mmap at ffffffffad2b3b8d > #7 [ffffbaf005743e80] mmput at ffffffffad08c81f > #8 [ffffbaf005743e98] do_exit at ffffffffad09636c > #9 [ffffbaf005743ef8] do_group_exit at ffffffffad096c78 > RIP: 00007f111c70ddf9 RSP: 00007fff451817e8 RFLAGS: 00000246 > RAX: ffffffffffffffda RBX: 00007f111c8089e0 RCX: 00007f111c70ddf9 > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 > RBP: 0000000000000000 R8: ffffffffffffff80 R9: 0000000000000000 > R10: 00007fff451817b0 R11: 0000000000000246 R12: 00007f111c8089e0 > R13: 00007f111c80e2e0 R14: 0000000000000002 R15: 00007f111c80e2c8 > ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b > > Backtrace using gdb (pay attention, task must be selected by thread Id): > crash> thread 731 > [Switching to thread 731 ( 975217 sleep)] > 5372 switch_to(prev, next, prev); > crash> gdb bt > #0 0xffffffffadb97292 in context_switch (rf=0xffffbaf005743c20, next=0xffff9ecac0692880, prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > #1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6696 > #2 0xffffffffadb982b8 in schedule () at kernel/sched/core.c:6772 > #3 0xffffffffadb9aed7 in rwbase_write_lock (rwb=rwb@entry=0xffff9ecaf1831430, state=state@entry=2) at kernel/locking/rwbase_rt.c:259 > #4 0xffffffffadb9b133 in __down_write (sem=sem@entry=0xffff9ecaf1831430) at kernel/locking/rwsem.c:1474 > #5 down_write (sem=sem@entry=0xffff9ecaf1831430) at kernel/locking/rwsem.c:1574 > #6 0xffffffffad2b0e2e in i_mmap_lock_write (mapping=<optimized out>) at ./include/linux/fs.h:466 > #7 unlink_file_vma (vma=vma@entry=0xffff9ecadd566090) at mm/mmap.c:127 > #8 0xffffffffad2a47b0 in free_pgtables (tlb=tlb@entry=0xffffbaf005743dd0, mt=mt@entry=0xffff9ecb28e3b180, vma=0xffff9ecadd566090, vma@entry=0xffff9ecadd566000, floor=floor@entry=0, ceiling=ceiling@entry=0) at mm/memory.c:431 > #9 0xffffffffad2b3b8d in exit_mmap (mm=mm@entry=0xffff9ecb28e3b180) at mm/mmap.c:3237 > #10 0xffffffffad08c81f in __mmput (mm=0xffff9ecb28e3b180) at kernel/fork.c:1204 > #11 mmput (mm=mm@entry=0xffff9ecb28e3b180) at kernel/fork.c:1226 > #12 0xffffffffad09636c in exit_mm () at kernel/exit.c:563 > #13 do_exit (code=code@entry=0) at kernel/exit.c:856 > #14 0xffffffffad096c78 in do_group_exit (exit_code=0) at kernel/exit.c:1019 > #15 0xffffffffad096cf8 in __do_sys_exit_group (error_code=<optimized out>) at kernel/exit.c:1030 > #16 __se_sys_exit_group (error_code=<optimized out>) at kernel/exit.c:1028 > #17 __x64_sys_exit_group (regs=<optimized out>) at kernel/exit.c:1028 > #18 0xffffffffadb8a327 in do_syscall_x64 (nr=<optimized out>, regs=0xffffbaf005743f58) at arch/x86/entry/common.c:51 > #19 do_syscall_64 (regs=0xffffbaf005743f58, nr=<optimized out>) at arch/x86/entry/common.c:81 > #20 0xffffffffadc000dc in entry_SYSCALL_64 () at arch/x86/entry/entry_64.S:120 > #21 0x00007f111c80e2c8 in ?? () > #22 0x0000000000000002 in ?? () > #23 0x00007f111c80e2e0 in ?? () > #24 0x00007f111c8089e0 in ?? () > #25 0x0000000000000000 in ?? () > crash> f 3 > 259 rwbase_schedule(); > crash> p *rwb > $1 = { > readers = { > counter = 1 > }, > rtmutex = { > wait_lock = { > raw_lock = { > { > val = { > counter = 0 > }, > { > locked = 0 '\000', > pending = 0 '\000' > }, > { > locked_pending = 0, > tail = 0 > } > } > } > }, > waiters = { > rb_root = { > rb_node = 0xffffbaf006977be0 > }, > rb_leftmost = 0xffffbaf00696fbe0 > }, > owner = 0xffff9ecb3956a881 > } > } > > Additional changes: > 1. Allow gdb "frame" command. > 2. Blacklist useless gdb "gcore" command. Use gcore plugin instead. > 3. Move crash_target_init() to later time as crash target requires a list of > tasks to be initialized. > > Known issues and TBD items: > 1. "info threads" may bail out first time throwing errors trying to access > userspace address during unwind process. Following "info threads" > invokations run without issues. > 2. To unwind a stack of inactive task, only modern Linux versions, which use > inactive_task_frame, are supported and only x86_64 architecture. > 3. gdb bt unwinder does not stop properly and may show invalid frames (21-25 > on example above). Not a regression, existed before. > 4. gdb bt unwinder does not work on active tasks in userspace. Not a regression, > existed before. > 5. Only x86_64 architecture supported. machdep->get_task_reg() must be > implemented for others. Not a regression, existed before. > 6. Active tasks registers fetching imlemented only for VMware dumps, see > x86_64_get_task_reg() for more details. Not a regression, existed before. > > Signed-off-by: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > --- > crash_target.c | 39 ++++++++++++++++++++--------- > defs.h | 10 ++++++-- > gdb-10.2.patch | 7 ++---- > gdb_interface.c | 63 ++++++++++++++++++++++++++++++---------------- > help.c | 1 + > main.c | 1 + > task.c | 1 + > x86_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++------ > 8 files changed, 140 insertions(+), 48 deletions(-) > > diff --git a/crash_target.c b/crash_target.c > index 4554806..2fdf203 100644 > --- a/crash_target.c > +++ b/crash_target.c > @@ -2,6 +2,8 @@ > * crash_target.c > * > * Copyright (c) 2021 VMware, Inc. > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" > + * refers to Broadcom Inc. and/or its subsidiaries. > * > * This program is free software; you can redistribute it and/or modify > * it under the terms of the GNU General Public License as published by > @@ -13,7 +15,7 @@ > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > * GNU General Public License for more details. > * > - * Author: Alexey Makhalov <amakhalov(a)vmware.com&gt; > + * Author: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > */ > > #include <defs.h> > @@ -23,11 +25,11 @@ > #include "regcache.h" > #include "gdbarch.h" > > -void crash_target_init (void); > - > +extern "C" void crash_target_init (void); > extern "C" int gdb_readmem_callback(unsigned long, void *, int, int); > -extern "C" int crash_get_nr_cpus(void); > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char *regname, > +extern "C" int crash_get_nr_tasks(void); > +extern "C" void crash_get_task_info(int task_nr, unsigned long *pid, char **comm); > +extern "C" int crash_get_task_reg (int task_nr, int regno, const char *regname, > int regsize, void *val); > > > @@ -60,7 +62,13 @@ public: > bool has_registers () override { return true; } > bool thread_alive (ptid_t ptid) override { return true; } > std::string pid_to_str (ptid_t ptid) override > - { return string_printf ("CPU %ld", ptid.tid ()); } > + { > + unsigned long pid; > + char *comm; > + > + crash_get_task_info(ptid.tid(), &pid, &comm); > + return string_printf ("%7ld %s", pid, comm); > + } > > }; > > @@ -68,18 +76,25 @@ public: > void > crash_target::fetch_registers (struct regcache *regcache, int regno) > { > + int r; > gdb_byte regval[16]; > - int cpu = inferior_ptid.tid(); > + int task_nr = inferior_ptid.tid(); > struct gdbarch *arch = regcache->arch (); > > - for (int r = 0; r < gdbarch_num_regs (arch); r++) > + if (regno >= 0) { > + r = regno; > + goto onetime; > + } > + > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) > { > +onetime: > const char *regname = gdbarch_register_name(arch, r); > int regsize = register_size (arch, r); > if (regsize > sizeof (regval)) > error (_("fatal error: buffer size is not enough to fit register value")); > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)&regval)) > + if (crash_get_task_reg (task_nr, r, regname, regsize, (void *)&regval)) > regcache->raw_supply (r, regval); > else > regcache->raw_supply (r, NULL); > @@ -107,10 +122,10 @@ crash_target::xfer_partial (enum target_object object, const char *annex, > > #define CRASH_INFERIOR_PID 1 > > -void > +extern "C" void > crash_target_init (void) > { > - int nr_cpus = crash_get_nr_cpus(); > + int nr_tasks = crash_get_nr_tasks(); > crash_target *target = new crash_target (); > > /* Own the target until it is successfully pushed. */ > @@ -119,7 +134,7 @@ crash_target_init (void) > push_target (std::move (target_holder)); > > inferior_appeared (current_inferior (), CRASH_INFERIOR_PID); > - for (int i = 0; i < nr_cpus; i++) > + for (int i = 0; i < nr_tasks; i++) > { > thread_info *thread = add_thread_silent (target, > ptid_t(CRASH_INFERIOR_PID, 0, i)); > diff --git a/defs.h b/defs.h > index 98650e8..2b3f247 100644 > --- a/defs.h > +++ b/defs.h > @@ -1080,7 +1080,7 @@ struct machdep_table { > void (*get_irq_affinity)(int); > void (*show_interrupts)(int, ulong *); > int (*is_page_ptr)(ulong, physaddr_t *); > - int (*get_cpu_reg)(int, int, const char *, int, void *); > + int (*get_task_reg)(struct task_context *, int, const char *, int, void *); > int (*is_cpu_prstatus_valid)(int cpu); > }; > > @@ -2263,6 +2263,7 @@ struct size_table { /* stash of commonly-used sizes */ > long pt_regs; > long task_struct; > long thread_info; > + long inactive_task_frame; > long softirq_state; > long desc_struct; > long umode_t; > @@ -8001,9 +8002,14 @@ extern int have_full_symbols(void); > #define XEN_HYPERVISOR_ARCH > #endif > > +/* > + * crash_target.c > + */ > +extern void crash_target_init (void); > + > /* > * Register numbers must be in sync with gdb/features/i386/64bit-core.c > - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg() > + * to make crash_target->fetch_registers() ---> machdep->get_task_reg() > * working properly. > */ > enum x86_64_regnum { > diff --git a/gdb-10.2.patch b/gdb-10.2.patch > index a7018a2..ecf673d 100644 > --- a/gdb-10.2.patch > +++ b/gdb-10.2.patch > @@ -221,7 +221,7 @@ exit 0 > warning (_("\ > --- gdb-10.2/gdb/main.c.orig > +++ gdb-10.2/gdb/main.c > -@@ -392,6 +392,14 @@ start_event_loop () > +@@ -392,6 +392,13 @@ start_event_loop () > return; > } > > @@ -230,7 +230,6 @@ exit 0 > +extern "C" void main_loop(void); > +extern "C" unsigned long crash_get_kaslr_offset(void); > +extern "C" int console(const char *, ...); > -+void crash_target_init (void); > +#endif > + > /* Call command_loop. */ > @@ -316,7 +315,7 @@ exit 0 > } > } > > -@@ -1242,6 +1274,16 @@ captured_main (void *data) > +@@ -1242,6 +1274,14 @@ captured_main (void *data) > > captured_main_1 (context); > > @@ -324,8 +323,6 @@ exit 0 > + /* Relocate the vmlinux. */ > + objfile_rebase (symfile_objfile, crash_get_kaslr_offset()); > + > -+ crash_target_init(); > -+ > + /* Back to crash. */ > + main_loop(); > +#endif > diff --git a/gdb_interface.c b/gdb_interface.c > index b14319c..03178f5 100644 > --- a/gdb_interface.c > +++ b/gdb_interface.c > @@ -3,6 +3,8 @@ > * Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > * Copyright (C) 2002-2015,2018-2019 David Anderson > * Copyright (C) 2002-2015,2018-2019 Red Hat, Inc. All rights reserved. > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" > + * refers to Broadcom Inc. and/or its subsidiaries. > * > * This program is free software; you can redistribute it and/or modify > * it under the terms of the GNU General Public License as published by > @@ -711,7 +713,7 @@ static char *prohibited_list[] = { > "watch", "rwatch", "awatch", "attach", "continue", "c", "fg", "detach", > "finish", "handle", "interrupt", "jump", "kill", "next", "nexti", > "signal", "step", "s", "stepi", "target", "until", "delete", > - "clear", "disable", "enable", "condition", "ignore", "frame", "catch", > + "clear", "disable", "enable", "condition", "ignore", "gcore", "catch", > "tcatch", "return", "file", "exec-file", "core-file", "symbol-file", > "load", "si", "ni", "shell", "sy", > NULL /* must be last */ > @@ -877,6 +879,7 @@ gdb_readmem_callback(ulong addr, void *buf, int len, int write) > switch (len) > { > case SIZEOF_8BIT: > + fprintf(fp, "%s\n", pc->curcmd); > if (STREQ(pc->curcmd, "bt")) { > if (readmem(addr, memtype, buf, SIZEOF_8BIT, > "gdb_readmem_callback", readflags)) > @@ -1063,34 +1066,52 @@ get_frame_offset(ulong pc) > unsigned long crash_get_kaslr_offset(void); > unsigned long crash_get_kaslr_offset(void) > { > - return kt->relocate * -1; > + return kt->relocate * -1; > } > > /* Callbacks for crash_target */ > -int crash_get_nr_cpus(void); > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > +int crash_get_nr_tasks(void); > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm); > +int crash_get_task_reg (int task_nr, int regno, const char *regname, > int regsize, void *val); > > -int crash_get_nr_cpus(void) > +int crash_get_nr_tasks(void) > { > - if (SADUMP_DUMPFILE()) > - return sadump_get_nr_cpus(); > - else if (DISKDUMP_DUMPFILE()) > - return diskdump_get_nr_cpus(); > - else if (KDUMP_DUMPFILE()) > - return kdump_get_nr_cpus(); > - else if (VMSS_DUMPFILE()) > - return vmware_vmss_get_nr_cpus(); > - > - /* Just CPU #0 */ ()> - return 1; > + return RUNNING_TASKS(); > } The crash_get_nr_tasks() will return a fixed number of running tasks, which is the task status when crash is loading. However for live debug mode, there will always be tasks exit and create, so we need to sync the tasks number with the "info threads". I guess it is not an easy work because that will involve: when crash loading: for (int i = 0; i < nr_tasks; i++) { thread_info *thread = add_thread_silent (target, ... } when new task create: thread_info *thread = add_thread_silent (target, ... when task exit: delete_thread(...) We may monitor those task create/exit events and do thread add/delete in a callback function or we just always do all tasks delete/add when "info thread" is invoked. But I guess none of these are as simple as using crash "ps" cmd and "set <pid>", there won't be any problem. Because crash "ps" can always show the current tasks status, and set <pid> can switch the task context to it. What do you think? Thanks, Tao Liu > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > - int regsize, void *value) > +/* Get task information by its index number in TT */ > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm) > { > - if (!machdep->get_cpu_reg) > - return FALSE; > - return machdep->get_cpu_reg(cpu, regno, regname, regsize, value); > + int i; > + struct task_context *tc; > + > + tc = FIRST_CONTEXT(); > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > + if (i == task_nr) { > + *pid = tc->pid; > + *comm = tc->comm; > + return; > + } > + *pid = 0; > + *comm = NULL; > + return; > +} > + > +int crash_get_task_reg (int task_nr, int regno, const char *regname, > + int regsize, void *value) > +{ > + int i; > + struct task_context *tc; > + > + if (!machdep->get_task_reg) > + return FALSE; > + > + tc = FIRST_CONTEXT(); > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > + if (i == task_nr) { > + return machdep->get_task_reg(tc, regno, regname, regsize, value); > + } > + return FALSE; > } > > diff --git a/help.c b/help.c > index a9c4d30..85dbda5 100644 > --- a/help.c > +++ b/help.c > @@ -8520,6 +8520,7 @@ char *version_info[] = { > "Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.", > "Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.", > "Copyright (C) 2015, 2021 VMware, Inc.", > +"Copyright (C) 2024 Broadcom, Inc.", > "This program is free software, covered by the GNU General Public License,", > "and you are welcome to change it and/or distribute copies of it under", > "certain conditions. Enter \"help copying\" to see the conditions.", > diff --git a/main.c b/main.c > index 0b6b927..13acd2d 100644 > --- a/main.c > +++ b/main.c > @@ -794,6 +794,7 @@ main_loop(void) > } else > SIGACTION(SIGINT, restart, &pc->sigaction, NULL); > > + crash_target_init(); > /* > * Display system statistics and current context. > */ > diff --git a/task.c b/task.c > index ebdb5be..5d26c52 100644 > --- a/task.c > +++ b/task.c > @@ -298,6 +298,7 @@ task_init(void) > tt->flags |= THREAD_INFO; > } > > + STRUCT_SIZE_INIT(inactive_task_frame, "inactive_task_frame"); > MEMBER_OFFSET_INIT(task_struct_state, "task_struct", "state"); > MEMBER_SIZE_INIT(task_struct_state, "task_struct", "state"); > if (INVALID_MEMBER(task_struct_state)) { > diff --git a/x86_64.c b/x86_64.c > index 502817d..b6e36a5 100644 > --- a/x86_64.c > +++ b/x86_64.c > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *, ulong, ulong, char *); > static void x86_64_framesize_debug(struct bt_info *); > static void x86_64_get_active_set(void); > static int x86_64_get_kvaddr_ranges(struct vaddr_range *); > -static int x86_64_get_cpu_reg(int, int, const char *, int, void *); > +static int x86_64_get_task_reg(struct task_context *, int, const char *, int, void *); > static int x86_64_verify_paddr(uint64_t); > static void GART_init(void); > static void x86_64_exception_stacks_init(void); > @@ -195,7 +195,7 @@ x86_64_init(int when) > machdep->machspec->irq_eframe_link = UNINITIALIZED; > machdep->machspec->irq_stack_gap = UNINITIALIZED; > machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges; > - machdep->get_cpu_reg = x86_64_get_cpu_reg; > + machdep->get_task_reg = x86_64_get_task_reg; > if (machdep->cmdline_args[0]) > parse_cmdline_args(); > if ((string = pc->read_vmcoreinfo("relocate"))) { > @@ -891,7 +891,7 @@ x86_64_dump_machdep_table(ulong arg) > fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n"); > fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n"); > fprintf(fp, " get_kvaddr_ranges: x86_64_get_kvaddr_ranges()\n"); > - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n"); > + fprintf(fp, " get_task_reg: x86_64_get_task_reg()\n"); > fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n"); > fprintf(fp, "clear_machdep_cache: x86_64_clear_machdep_cache()\n"); > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? > @@ -6398,6 +6398,9 @@ x86_64_ORC_init(void) > }; > struct ORC_data *orc; > > + MEMBER_OFFSET_INIT(inactive_task_frame_bp, "inactive_task_frame", "bp"); > + MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, "inactive_task_frame", "ret_addr"); > + > if (machdep->flags & FRAMEPOINTER) > return; > > @@ -6455,9 +6458,6 @@ x86_64_ORC_init(void) > orc->__stop_orc_unwind = symbol_value("__stop_orc_unwind"); > orc->orc_lookup = symbol_value("orc_lookup"); > > - MEMBER_OFFSET_INIT(inactive_task_frame_bp, "inactive_task_frame", "bp"); > - MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, "inactive_task_frame", "ret_addr"); > - > orc->has_signal = MEMBER_EXISTS("orc_entry", "signal"); /* added at 6.3 */ > orc->has_end = MEMBER_EXISTS("orc_entry", "end"); /* removed at 6.4 */ > > @@ -9070,14 +9070,64 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp) > } > > static int > -x86_64_get_cpu_reg(int cpu, int regno, const char *name, > +x86_64_get_task_reg(struct task_context *tc, int regno, const char *name, > int size, void *value) > { > if (regno >= LAST_REGNUM) > return FALSE; > > + /* > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15 from > + * inactive_task_frame (see __switch_to_asm). Other regs saved on > + * regular frame. > + */ > + if (!is_task_active(tc->task)) { > + int frame_size = STRUCT_SIZE("inactive_task_frame"); > + > + /* Only modern kernels supported. */ > + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) { > + ulong rsp; > + int offset = 0; > + switch (regno) { > + case RSP_REGNUM: > + readmem(tc->task + OFFSET(task_struct_thread) + > + OFFSET(thread_struct_rsp), KVADDR, > + &rsp, sizeof(void *), > + "thread_struct rsp", FAULT_ON_ERROR); > + rsp += frame_size; > + memcpy(value, &rsp, size); > + return TRUE; > + case RIP_REGNUM: > + offset += 8; > + case RBP_REGNUM: > + offset += 8; > + case RBX_REGNUM: > + offset += 8; > + case R12_REGNUM: > + offset += 8; > + case R13_REGNUM: > + offset += 8; > + case R14_REGNUM: > + offset += 8; > + case R15_REGNUM: > + readmem(tc->task + OFFSET(task_struct_thread) + > + OFFSET(thread_struct_rsp), KVADDR, > + &rsp, sizeof(void *), > + "thread_struct rsp", FAULT_ON_ERROR); > + readmem(rsp + offset, KVADDR, value, sizeof(void *), > + "inactive_thread_frame saved regs", FAULT_ON_ERROR); > + return TRUE; > + } > + } > + /* TBD: older kernels support. */ > + return FALSE; > + } > + > + /* > + * Task is active, grab CPU's registers > + */ > if (VMSS_DUMPFILE()) > - return vmware_vmss_get_cpu_reg(cpu, regno, name, size, value); > + return vmware_vmss_get_cpu_reg(tc->processor, regno, name, size, value); > > return FALSE; > } > -- > 2.39.0 >

Hi Tao, Thanks for the details. Agree, my approach is targeted primarily for offline debugging. Create/exit monitoring sounds like an overkill. And from the other side, maintaining a static list of threads (CPUs or Tasks) does not work for live debugging. "ps" followed by "set" where GDB will receive just one inferior/thread at a time is probably the right universal approach. Let me experiment with a combination of my and your patches and come back to you.

> 'set 1' (let's assume it's an INACTIVE task on CPU 5) > > 'info threads' > -> when it switches to CPU 5, it asks crash to sync, ie. `set_cpu(5)` > -> crash will return CPU 5's active tasks registers > -> WRONG registers for CPU 5, since we wanted registers for PID 1 (which > is an inactive task on CPU 5) What do you mean of "WRONG registers"? Isn't the CPU5's regcache be pid 1's context after 'set 1'? If it isn't then it's a bug.

On Sun, Mar 17, 2024 at 03:40:52PM +0530, Aditya Gupta wrote: > Hi Tao and Alexey, > > On Fri, Mar 15, 2024 at 03:15:19PM -0700, Alexey Makhalov wrote: > > Hi Tao, > > > > Thanks for the details. Agree, my approach is targeted primarily for > > offline debugging. > > Create/exit monitoring sounds like an overkill. And from the other side, > > maintaining a static list of threads (CPUs or Tasks) does not work for live > > debugging. > > "ps" followed by "set" where GDB will receive just one inferior/thread at a > > time is probably the right universal approach. > > Let me experiment with a combination of my and your patches and come back > > to you. > > Sorry I noticed a bug late while trying the combination of tao's and my > patches. > > I agree with Tao that current approach might not be good for live > debugging, but I think the approach of having all tasks in 'info > threads' is a good advantage for offline debugging, since then gdb knows > about all tasks (similar behaviour as usual gdb), and makes it easier > to synchronise crash <-> gdb context. > > I had refrained from changing the format of 'info threads' from active > tasks to all tasks, but there are some issues with crash <-> gdb sync, > when doing arbitrary task unwinding. > > 'set 1' (let's assume it's an INACTIVE task on CPU 5) > > 'info threads' > -> when it switches to CPU 5, it asks crash to sync, ie. `set_cpu(5)` > -> crash will return CPU 5's active tasks registers > -> WRONG registers for CPU 5, since we wanted registers for PID 1 (which > is an inactive task on CPU 5) What do you mean of "WRONG registers"? Isn't the CPU5's regcache be pid 1's context after 'set 1'? If it isn't then it's a bug. After crash finished loading and we see crash> prompt, the cpu regcaches will be all active tasks', which are running on each cpu. That is what we saw using "info threads", like: crash> info threads Id Target Id Frame .... * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 .... 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 .... Then we use "set <pid>" to switch to task <pid>, its cpu regcache will be flushed, and if we use command "info threads" again, it will show the status after the flushing. E.g in the following, task pid 1's cpu is #13, so cpu #13 is flushed after "set 1": crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd crash> set 1 crash> info threads Id Target Id Frame .... 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 .... * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^... .... When design, I assume the CPUs/threads are like power slots, its number is constant, but the contents can be changed. We don't need to prepare the "slots" as many as our tasks quantity (I don't know the maxinum threads that a gdb can accept, e.g. on a system which have 65K tasks, but for most tasks we'd never switch to it in order to view its stack trace, I guess it would be a waste to call add_thread_silent (target, ...) 65K times). Currently the gdb CPUs/threads(slots) equals to the system's cpu number. But if a system which have thousands of cpus, which we think is unnecessary to call add_thread_silent 1K times, we can even shrink the slots to be less.

Thanks, Tao Liu > > I can think of solving it in two ways: > 1. Having a array in crash_target, which tells what CPU in gdb maps to > which task in crash, this can be done without requiring changes to 'info > threads', while we do the 'add_thread_silent' > 2. Having all tasks in gdb itself, so it's a one-to-one, so `set_cpu` > will then refer to the task ID, I will have to explore more on this. > > Any comments ? > > Thanks, > Aditya Gupta > > > > > Thanks, > > --Alexey > > > > > > On Thu, Mar 14, 2024 at 9:22 PM Tao Liu <ltao(a)redhat.com&gt; wrote: > > > > > Hi Alexey, > > > > > > On Thu, Mar 14, 2024 at 6:29 PM Alexey Makhalov > > > <alexey.makhalov(a)broadcom.com&gt; wrote: > > > > > > > > Support for GDB debugging of all tasks active and inactive. > > > > Before this commit only active tasks were listed by "info threads" > > > > with "CPU #" as a Target Id. > > > > > > > > "info threads" will now show all tasks, similar to "ps", example: > > > > crash> info threads > > > > Id Target Id Frame > > > > * 1 0 swapper/0 0xffffffffadba19d4 in default_idle () at > > > arch/x86/kernel/process.c:731 > > > > 2 0 swapper/1 0xffffffffadba19d4 in default_idle () at > > > arch/x86/kernel/process.c:731 > > > > 3 0 swapper/2 0xffffffffadba19d4 in default_idle () at > > > arch/x86/kernel/process.c:731 > > > > 4 0 swapper/3 0xffffffffadba19d4 in default_idle () at > > > arch/x86/kernel/process.c:731 > > > > 5 0 swapper/4 0xffffffffadb97292 in context_switch > > > (rf=0xffffbaf0000f3e88, next=0xffff9ecb04908000, prev=<optimized out>, > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > ... > > > > 730 970325 taskset 0xffffffffadb97292 in context_switch > > > (rf=0xffffbaf006a0fd18, next=0xffff9ecb0aec0000, prev=<optimized out>, > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > 731 975217 sleep 0xffffffffadb97292 in context_switch > > > (rf=0xffffbaf005743c20, next=0xffff9ecac0692880, prev=<optimized out>, > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > 732 975228 sleep 0xffffffffadb97292 in context_switch > > > (rf=0xffffbaf00696fb58, next=0xffff9ecac0690000, prev=<optimized out>, > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > ... > > > > 876 976084 docker 0xffffffffadb97292 in context_switch > > > (rf=0xffffbaf0153dbd10, next=0xffff9ecac0645100, prev=<optimized out>, > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > 877 976085 systemd-userwor 0xffffffffadb97292 in context_switch > > > (rf=0xffffbaf0153cbc58, next=0xffff9ecac0645100, prev=<optimized out>, > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > 878 976086 systemd-userwor 0xffffffffadb97292 in context_switch > > > (rf=0xffffbaf0153e3c58, next=0xffffffffaec15a40 <init_task>, > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > > > > 879 976087 systemd-userwor 0xffffffffadb97292 in context_switch > > > (rf=0xffffbaf0153ebc58, next=0xffffffffaec15a40 <init_task>, > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > > > > Where "Target ID" contains "PID COMM" of the task > > > > > > > > > > I see here, we chose a different tech path of viewing arbitrary tasks. > > > You improved the "info threads" as an alternative to crash "ps" cmd, > > > and use "thread X" to switch to task X and view its stacktrace. > > > However mine is using crash "ps" cmd first, getting specific tasks pid > > > or task struct, then using crash cmd "set <pid>" or "set > > > <task_struct>" to switch to task X and view its stacktrace. > > > > > > I notice there is one problem of using the "info threads as ps" tech > > > path, it is difficult to handle the live debug, see the comment below: > > > > > > > Example of "731 975217 sleep" debugging, real case, trying to > > > > figure out why sleep was stuck in uninterruptable sleep. > > > > Backtrace using crash: > > > > crash> ps | grep 975217 > > > > 975217 969797 3 ffff9ecb3956a880 UN 0.0 0 0 > > > sleep > > > > crash> bt 975217 > > > > PID: 975217 TASK: ffff9ecb3956a880 CPU: 3 COMMAND: "sleep" > > > > #0 [ffffbaf005743ba0] __schedule at ffffffffadb97292 > > > > #1 [ffffbaf005743c60] schedule at ffffffffadb982b8 > > > > #2 [ffffbaf005743c80] rwbase_write_lock at ffffffffadb9aed7 > > > > #3 [ffffbaf005743cc0] down_write at ffffffffadb9b133 > > > > #4 [ffffbaf005743cd0] unlink_file_vma at ffffffffad2b0e2e > > > > #5 [ffffbaf005743cf8] free_pgtables at ffffffffad2a47b0 > > > > #6 [ffffbaf005743d88] exit_mmap at ffffffffad2b3b8d > > > > #7 [ffffbaf005743e80] mmput at ffffffffad08c81f > > > > #8 [ffffbaf005743e98] do_exit at ffffffffad09636c > > > > #9 [ffffbaf005743ef8] do_group_exit at ffffffffad096c78 > > > > RIP: 00007f111c70ddf9 RSP: 00007fff451817e8 RFLAGS: 00000246 > > > > RAX: ffffffffffffffda RBX: 00007f111c8089e0 RCX: 00007f111c70ddf9 > > > > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 > > > > RBP: 0000000000000000 R8: ffffffffffffff80 R9: 0000000000000000 > > > > R10: 00007fff451817b0 R11: 0000000000000246 R12: 00007f111c8089e0 > > > > R13: 00007f111c80e2e0 R14: 0000000000000002 R15: 00007f111c80e2c8 > > > > ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b > > > > > > > > Backtrace using gdb (pay attention, task must be selected by thread Id): > > > > crash> thread 731 > > > > [Switching to thread 731 ( 975217 sleep)] > > > > 5372 switch_to(prev, next, prev); > > > > crash> gdb bt > > > > #0 0xffffffffadb97292 in context_switch (rf=0xffffbaf005743c20, > > > next=0xffff9ecac0692880, prev=<optimized out>, rq=<optimized out>) at > > > kernel/sched/core.c:5372 > > > > #1 __schedule (sched_mode=sched_mode@entry=0) at > > > kernel/sched/core.c:6696 > > > > #2 0xffffffffadb982b8 in schedule () at kernel/sched/core.c:6772 > > > > #3 0xffffffffadb9aed7 in rwbase_write_lock (rwb=rwb@entry=0xffff9ecaf1831430, > > > state=state@entry=2) at kernel/locking/rwbase_rt.c:259 > > > > #4 0xffffffffadb9b133 in __down_write (sem=sem@entry=0xffff9ecaf1831430) > > > at kernel/locking/rwsem.c:1474 > > > > #5 down_write (sem=sem@entry=0xffff9ecaf1831430) at > > > kernel/locking/rwsem.c:1574 > > > > #6 0xffffffffad2b0e2e in i_mmap_lock_write (mapping=<optimized out>) > > > at ./include/linux/fs.h:466 > > > > #7 unlink_file_vma (vma=vma@entry=0xffff9ecadd566090) at mm/mmap.c:127 > > > > #8 0xffffffffad2a47b0 in free_pgtables (tlb=tlb@entry=0xffffbaf005743dd0, > > > mt=mt@entry=0xffff9ecb28e3b180, vma=0xffff9ecadd566090, vma@entry=0xffff9ecadd566000, > > > floor=floor@entry=0, ceiling=ceiling@entry=0) at mm/memory.c:431 > > > > #9 0xffffffffad2b3b8d in exit_mmap (mm=mm@entry=0xffff9ecb28e3b180) > > > at mm/mmap.c:3237 > > > > #10 0xffffffffad08c81f in __mmput (mm=0xffff9ecb28e3b180) at > > > kernel/fork.c:1204 > > > > #11 mmput (mm=mm@entry=0xffff9ecb28e3b180) at kernel/fork.c:1226 > > > > #12 0xffffffffad09636c in exit_mm () at kernel/exit.c:563 > > > > #13 do_exit (code=code@entry=0) at kernel/exit.c:856 > > > > #14 0xffffffffad096c78 in do_group_exit (exit_code=0) at > > > kernel/exit.c:1019 > > > > #15 0xffffffffad096cf8 in __do_sys_exit_group (error_code=<optimized > > > out>) at kernel/exit.c:1030 > > > > #16 __se_sys_exit_group (error_code=<optimized out>) at > > > kernel/exit.c:1028 > > > > #17 __x64_sys_exit_group (regs=<optimized out>) at kernel/exit.c:1028 > > > > #18 0xffffffffadb8a327 in do_syscall_x64 (nr=<optimized out>, > > > regs=0xffffbaf005743f58) at arch/x86/entry/common.c:51 > > > > #19 do_syscall_64 (regs=0xffffbaf005743f58, nr=<optimized out>) at > > > arch/x86/entry/common.c:81 > > > > #20 0xffffffffadc000dc in entry_SYSCALL_64 () at > > > arch/x86/entry/entry_64.S:120 > > > > #21 0x00007f111c80e2c8 in ?? () > > > > #22 0x0000000000000002 in ?? () > > > > #23 0x00007f111c80e2e0 in ?? () > > > > #24 0x00007f111c8089e0 in ?? () > > > > #25 0x0000000000000000 in ?? () > > > > crash> f 3 > > > > 259 rwbase_schedule(); > > > > crash> p *rwb > > > > $1 = { > > > > readers = { > > > > counter = 1 > > > > }, > > > > rtmutex = { > > > > wait_lock = { > > > > raw_lock = { > > > > { > > > > val = { > > > > counter = 0 > > > > }, > > > > { > > > > locked = 0 '\000', > > > > pending = 0 '\000' > > > > }, > > > > { > > > > locked_pending = 0, > > > > tail = 0 > > > > } > > > > } > > > > } > > > > }, > > > > waiters = { > > > > rb_root = { > > > > rb_node = 0xffffbaf006977be0 > > > > }, > > > > rb_leftmost = 0xffffbaf00696fbe0 > > > > }, > > > > owner = 0xffff9ecb3956a881 > > > > } > > > > } > > > > > > > > Additional changes: > > > > 1. Allow gdb "frame" command. > > > > 2. Blacklist useless gdb "gcore" command. Use gcore plugin instead. > > > > 3. Move crash_target_init() to later time as crash target requires a > > > list of > > > > tasks to be initialized. > > > > > > > > Known issues and TBD items: > > > > 1. "info threads" may bail out first time throwing errors trying to > > > access > > > > userspace address during unwind process. Following "info threads" > > > > invokations run without issues. > > > > 2. To unwind a stack of inactive task, only modern Linux versions, which > > > use > > > > inactive_task_frame, are supported and only x86_64 architecture. > > > > 3. gdb bt unwinder does not stop properly and may show invalid frames > > > (21-25 > > > > on example above). Not a regression, existed before. > > > > 4. gdb bt unwinder does not work on active tasks in userspace. Not a > > > regression, > > > > existed before. > > > > 5. Only x86_64 architecture supported. machdep->get_task_reg() must be > > > > implemented for others. Not a regression, existed before. > > > > 6. Active tasks registers fetching imlemented only for VMware dumps, see > > > > x86_64_get_task_reg() for more details. Not a regression, existed > > > before. > > > > > > > > Signed-off-by: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > > > > --- > > > > crash_target.c | 39 ++++++++++++++++++++--------- > > > > defs.h | 10 ++++++-- > > > > gdb-10.2.patch | 7 ++---- > > > > gdb_interface.c | 63 ++++++++++++++++++++++++++++++---------------- > > > > help.c | 1 + > > > > main.c | 1 + > > > > task.c | 1 + > > > > x86_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++------ > > > > 8 files changed, 140 insertions(+), 48 deletions(-) > > > > > > > > diff --git a/crash_target.c b/crash_target.c > > > > index 4554806..2fdf203 100644 > > > > --- a/crash_target.c > > > > +++ b/crash_target.c > > > > @@ -2,6 +2,8 @@ > > > > * crash_target.c > > > > * > > > > * Copyright (c) 2021 VMware, Inc. > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" > > > > + * refers to Broadcom Inc. and/or its subsidiaries. > > > > * > > > > * This program is free software; you can redistribute it and/or modify > > > > * it under the terms of the GNU General Public License as published by > > > > @@ -13,7 +15,7 @@ > > > > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > > * GNU General Public License for more details. > > > > * > > > > - * Author: Alexey Makhalov <amakhalov(a)vmware.com&gt; > > > > + * Author: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > > > > */ > > > > > > > > #include <defs.h> > > > > @@ -23,11 +25,11 @@ > > > > #include "regcache.h" > > > > #include "gdbarch.h" > > > > > > > > -void crash_target_init (void); > > > > - > > > > +extern "C" void crash_target_init (void); > > > > extern "C" int gdb_readmem_callback(unsigned long, void *, int, int); > > > > -extern "C" int crash_get_nr_cpus(void); > > > > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char > > > *regname, > > > > +extern "C" int crash_get_nr_tasks(void); > > > > +extern "C" void crash_get_task_info(int task_nr, unsigned long *pid, > > > char **comm); > > > > +extern "C" int crash_get_task_reg (int task_nr, int regno, const char > > > *regname, > > > > int regsize, void *val); > > > > > > > > > > > > @@ -60,7 +62,13 @@ public: > > > > bool has_registers () override { return true; } > > > > bool thread_alive (ptid_t ptid) override { return true; } > > > > std::string pid_to_str (ptid_t ptid) override > > > > - { return string_printf ("CPU %ld", ptid.tid ()); } > > > > + { > > > > + unsigned long pid; > > > > + char *comm; > > > > + > > > > + crash_get_task_info(ptid.tid(), &pid, &comm); > > > > + return string_printf ("%7ld %s", pid, comm); > > > > + } > > > > > > > > }; > > > > > > > > @@ -68,18 +76,25 @@ public: > > > > void > > > > crash_target::fetch_registers (struct regcache *regcache, int regno) > > > > { > > > > + int r; > > > > gdb_byte regval[16]; > > > > - int cpu = inferior_ptid.tid(); > > > > + int task_nr = inferior_ptid.tid(); > > > > struct gdbarch *arch = regcache->arch (); > > > > > > > > - for (int r = 0; r < gdbarch_num_regs (arch); r++) > > > > + if (regno >= 0) { > > > > + r = regno; > > > > + goto onetime; > > > > + } > > > > + > > > > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) > > > > { > > > > +onetime: > > > > const char *regname = gdbarch_register_name(arch, r); > > > > int regsize = register_size (arch, r); > > > > if (regsize > sizeof (regval)) > > > > error (_("fatal error: buffer size is not enough to fit > > > register value")); > > > > > > > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)&regval)) > > > > + if (crash_get_task_reg (task_nr, r, regname, regsize, (void > > > *)&regval)) > > > > regcache->raw_supply (r, regval); > > > > else > > > > regcache->raw_supply (r, NULL); > > > > @@ -107,10 +122,10 @@ crash_target::xfer_partial (enum target_object > > > object, const char *annex, > > > > > > > > #define CRASH_INFERIOR_PID 1 > > > > > > > > -void > > > > +extern "C" void > > > > crash_target_init (void) > > > > { > > > > - int nr_cpus = crash_get_nr_cpus(); > > > > + int nr_tasks = crash_get_nr_tasks(); > > > > crash_target *target = new crash_target (); > > > > > > > > /* Own the target until it is successfully pushed. */ > > > > @@ -119,7 +134,7 @@ crash_target_init (void) > > > > push_target (std::move (target_holder)); > > > > > > > > inferior_appeared (current_inferior (), CRASH_INFERIOR_PID); > > > > - for (int i = 0; i < nr_cpus; i++) > > > > + for (int i = 0; i < nr_tasks; i++) > > > > { > > > > thread_info *thread = add_thread_silent (target, > > > > ptid_t(CRASH_INFERIOR_PID, 0, > > > i)); > > > > diff --git a/defs.h b/defs.h > > > > index 98650e8..2b3f247 100644 > > > > --- a/defs.h > > > > +++ b/defs.h > > > > @@ -1080,7 +1080,7 @@ struct machdep_table { > > > > void (*get_irq_affinity)(int); > > > > void (*show_interrupts)(int, ulong *); > > > > int (*is_page_ptr)(ulong, physaddr_t *); > > > > - int (*get_cpu_reg)(int, int, const char *, int, void *); > > > > + int (*get_task_reg)(struct task_context *, int, const char *, > > > int, void *); > > > > int (*is_cpu_prstatus_valid)(int cpu); > > > > }; > > > > > > > > @@ -2263,6 +2263,7 @@ struct size_table { /* stash of > > > commonly-used sizes */ > > > > long pt_regs; > > > > long task_struct; > > > > long thread_info; > > > > + long inactive_task_frame; > > > > long softirq_state; > > > > long desc_struct; > > > > long umode_t; > > > > @@ -8001,9 +8002,14 @@ extern int have_full_symbols(void); > > > > #define XEN_HYPERVISOR_ARCH > > > > #endif > > > > > > > > +/* > > > > + * crash_target.c > > > > + */ > > > > +extern void crash_target_init (void); > > > > + > > > > /* > > > > * Register numbers must be in sync with gdb/features/i386/64bit-core.c > > > > - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg() > > > > + * to make crash_target->fetch_registers() ---> machdep->get_task_reg() > > > > * working properly. > > > > */ > > > > enum x86_64_regnum { > > > > diff --git a/gdb-10.2.patch b/gdb-10.2.patch > > > > index a7018a2..ecf673d 100644 > > > > --- a/gdb-10.2.patch > > > > +++ b/gdb-10.2.patch > > > > @@ -221,7 +221,7 @@ exit 0 > > > > warning (_("\ > > > > --- gdb-10.2/gdb/main.c.orig > > > > +++ gdb-10.2/gdb/main.c > > > > -@@ -392,6 +392,14 @@ start_event_loop () > > > > +@@ -392,6 +392,13 @@ start_event_loop () > > > > return; > > > > } > > > > > > > > @@ -230,7 +230,6 @@ exit 0 > > > > +extern "C" void main_loop(void); > > > > +extern "C" unsigned long crash_get_kaslr_offset(void); > > > > +extern "C" int console(const char *, ...); > > > > -+void crash_target_init (void); > > > > +#endif > > > > + > > > > /* Call command_loop. */ > > > > @@ -316,7 +315,7 @@ exit 0 > > > > } > > > > } > > > > > > > > -@@ -1242,6 +1274,16 @@ captured_main (void *data) > > > > +@@ -1242,6 +1274,14 @@ captured_main (void *data) > > > > > > > > captured_main_1 (context); > > > > > > > > @@ -324,8 +323,6 @@ exit 0 > > > > + /* Relocate the vmlinux. */ > > > > + objfile_rebase (symfile_objfile, crash_get_kaslr_offset()); > > > > + > > > > -+ crash_target_init(); > > > > -+ > > > > + /* Back to crash. */ > > > > + main_loop(); > > > > +#endif > > > > diff --git a/gdb_interface.c b/gdb_interface.c > > > > index b14319c..03178f5 100644 > > > > --- a/gdb_interface.c > > > > +++ b/gdb_interface.c > > > > @@ -3,6 +3,8 @@ > > > > * Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > > > * Copyright (C) 2002-2015,2018-2019 David Anderson > > > > * Copyright (C) 2002-2015,2018-2019 Red Hat, Inc. All rights reserved. > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" > > > > + * refers to Broadcom Inc. and/or its subsidiaries. > > > > * > > > > * This program is free software; you can redistribute it and/or modify > > > > * it under the terms of the GNU General Public License as published by > > > > @@ -711,7 +713,7 @@ static char *prohibited_list[] = { > > > > "watch", "rwatch", "awatch", "attach", "continue", "c", "fg", > > > "detach", > > > > "finish", "handle", "interrupt", "jump", "kill", "next", "nexti", > > > > "signal", "step", "s", "stepi", "target", "until", "delete", > > > > - "clear", "disable", "enable", "condition", "ignore", "frame", > > > "catch", > > > > + "clear", "disable", "enable", "condition", "ignore", "gcore", > > > "catch", > > > > "tcatch", "return", "file", "exec-file", "core-file", > > > "symbol-file", > > > > "load", "si", "ni", "shell", "sy", > > > > NULL /* must be last */ > > > > @@ -877,6 +879,7 @@ gdb_readmem_callback(ulong addr, void *buf, int len, > > > int write) > > > > switch (len) > > > > { > > > > case SIZEOF_8BIT: > > > > + fprintf(fp, "%s\n", pc->curcmd); > > > > if (STREQ(pc->curcmd, "bt")) { > > > > if (readmem(addr, memtype, buf, SIZEOF_8BIT, > > > > "gdb_readmem_callback", readflags)) > > > > @@ -1063,34 +1066,52 @@ get_frame_offset(ulong pc) > > > > unsigned long crash_get_kaslr_offset(void); > > > > unsigned long crash_get_kaslr_offset(void) > > > > { > > > > - return kt->relocate * -1; > > > > + return kt->relocate * -1; > > > > } > > > > > > > > /* Callbacks for crash_target */ > > > > -int crash_get_nr_cpus(void); > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > > +int crash_get_nr_tasks(void); > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm); > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname, > > > > int regsize, void *val); > > > > > > > > -int crash_get_nr_cpus(void) > > > > +int crash_get_nr_tasks(void) > > > > { > > > > - if (SADUMP_DUMPFILE()) > > > > - return sadump_get_nr_cpus(); > > > > - else if (DISKDUMP_DUMPFILE()) > > > > - return diskdump_get_nr_cpus(); > > > > - else if (KDUMP_DUMPFILE()) > > > > - return kdump_get_nr_cpus(); > > > > - else if (VMSS_DUMPFILE()) > > > > - return vmware_vmss_get_nr_cpus(); > > > > - > > > > - /* Just CPU #0 */ > > > ()> - return 1; > > > > + return RUNNING_TASKS(); > > > > } > > > > > > The crash_get_nr_tasks() will return a fixed number of running tasks, > > > which is the task status when crash is loading. However for live debug > > > mode, there will always be tasks exit and create, so we need to sync > > > the tasks number with the "info threads". I guess it is not an easy > > > work because that will involve: > > > > > > when crash loading: > > > for (int i = 0; i < nr_tasks; i++) > > > { > > > thread_info *thread = add_thread_silent (target, ... > > > } > > > > > > when new task create: > > > thread_info *thread = add_thread_silent (target, ... > > > > > > when task exit: > > > delete_thread(...) > > > > > > We may monitor those task create/exit events and do thread add/delete > > > in a callback function or we just always do all tasks delete/add when > > > "info thread" is invoked. But I guess none of these are as simple as > > > using crash "ps" cmd and "set <pid>", there won't be any problem. > > > Because crash "ps" can always show the current tasks status, and set > > > <pid> can switch the task context to it. > > > > > > What do you think? > > > > > > Thanks, > > > Tao Liu > > > > > > > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > > - int regsize, void *value) > > > > +/* Get task information by its index number in TT */ > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm) > > > > { > > > > - if (!machdep->get_cpu_reg) > > > > - return FALSE; > > > > - return machdep->get_cpu_reg(cpu, regno, regname, regsize, > > > value); > > > > + int i; > > > > + struct task_context *tc; > > > > + > > > > + tc = FIRST_CONTEXT(); > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > > + if (i == task_nr) { > > > > + *pid = tc->pid; > > > > + *comm = tc->comm; > > > > + return; > > > > + } > > > > + *pid = 0; > > > > + *comm = NULL; > > > > + return; > > > > +} > > > > + > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname, > > > > + int regsize, void *value) > > > > +{ > > > > + int i; > > > > + struct task_context *tc; > > > > + > > > > + if (!machdep->get_task_reg) > > > > + return FALSE; > > > > + > > > > + tc = FIRST_CONTEXT(); > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > > + if (i == task_nr) { > > > > + return machdep->get_task_reg(tc, regno, regname, > > > regsize, value); > > > > + } > > > > + return FALSE; > > > > } > > > > > > > > diff --git a/help.c b/help.c > > > > index a9c4d30..85dbda5 100644 > > > > --- a/help.c > > > > +++ b/help.c > > > > @@ -8520,6 +8520,7 @@ char *version_info[] = { > > > > "Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.", > > > > "Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.", > > > > "Copyright (C) 2015, 2021 VMware, Inc.", > > > > +"Copyright (C) 2024 Broadcom, Inc.", > > > > "This program is free software, covered by the GNU General Public > > > License,", > > > > "and you are welcome to change it and/or distribute copies of it under", > > > > "certain conditions. Enter \"help copying\" to see the conditions.", > > > > diff --git a/main.c b/main.c > > > > index 0b6b927..13acd2d 100644 > > > > --- a/main.c > > > > +++ b/main.c > > > > @@ -794,6 +794,7 @@ main_loop(void) > > > > } else > > > > SIGACTION(SIGINT, restart, &pc->sigaction, NULL); > > > > > > > > + crash_target_init(); > > > > /* > > > > * Display system statistics and current context. > > > > */ > > > > diff --git a/task.c b/task.c > > > > index ebdb5be..5d26c52 100644 > > > > --- a/task.c > > > > +++ b/task.c > > > > @@ -298,6 +298,7 @@ task_init(void) > > > > tt->flags |= THREAD_INFO; > > > > } > > > > > > > > + STRUCT_SIZE_INIT(inactive_task_frame, "inactive_task_frame"); > > > > MEMBER_OFFSET_INIT(task_struct_state, "task_struct", "state"); > > > > MEMBER_SIZE_INIT(task_struct_state, "task_struct", "state"); > > > > if (INVALID_MEMBER(task_struct_state)) { > > > > diff --git a/x86_64.c b/x86_64.c > > > > index 502817d..b6e36a5 100644 > > > > --- a/x86_64.c > > > > +++ b/x86_64.c > > > > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *, > > > ulong, ulong, char *); > > > > static void x86_64_framesize_debug(struct bt_info *); > > > > static void x86_64_get_active_set(void); > > > > static int x86_64_get_kvaddr_ranges(struct vaddr_range *); > > > > -static int x86_64_get_cpu_reg(int, int, const char *, int, void *); > > > > +static int x86_64_get_task_reg(struct task_context *, int, const char > > > *, int, void *); > > > > static int x86_64_verify_paddr(uint64_t); > > > > static void GART_init(void); > > > > static void x86_64_exception_stacks_init(void); > > > > @@ -195,7 +195,7 @@ x86_64_init(int when) > > > > machdep->machspec->irq_eframe_link = UNINITIALIZED; > > > > machdep->machspec->irq_stack_gap = UNINITIALIZED; > > > > machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges; > > > > - machdep->get_cpu_reg = x86_64_get_cpu_reg; > > > > + machdep->get_task_reg = x86_64_get_task_reg; > > > > if (machdep->cmdline_args[0]) > > > > parse_cmdline_args(); > > > > if ((string = pc->read_vmcoreinfo("relocate"))) { > > > > @@ -891,7 +891,7 @@ x86_64_dump_machdep_table(ulong arg) > > > > fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n"); > > > > fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n"); > > > > fprintf(fp, " get_kvaddr_ranges: > > > x86_64_get_kvaddr_ranges()\n"); > > > > - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n"); > > > > + fprintf(fp, " get_task_reg: x86_64_get_task_reg()\n"); > > > > fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n"); > > > > fprintf(fp, "clear_machdep_cache: > > > x86_64_clear_machdep_cache()\n"); > > > > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? > > > > @@ -6398,6 +6398,9 @@ x86_64_ORC_init(void) > > > > }; > > > > struct ORC_data *orc; > > > > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_bp, > > > "inactive_task_frame", "bp"); > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, > > > "inactive_task_frame", "ret_addr"); > > > > + > > > > if (machdep->flags & FRAMEPOINTER) > > > > return; > > > > > > > > @@ -6455,9 +6458,6 @@ x86_64_ORC_init(void) > > > > orc->__stop_orc_unwind = symbol_value("__stop_orc_unwind"); > > > > orc->orc_lookup = symbol_value("orc_lookup"); > > > > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_bp, > > > "inactive_task_frame", "bp"); > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, > > > "inactive_task_frame", "ret_addr"); > > > > - > > > > orc->has_signal = MEMBER_EXISTS("orc_entry", "signal"); /* added > > > at 6.3 */ > > > > orc->has_end = MEMBER_EXISTS("orc_entry", "end"); /* > > > removed at 6.4 */ > > > > > > > > @@ -9070,14 +9070,64 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp) > > > > } > > > > > > > > static int > > > > -x86_64_get_cpu_reg(int cpu, int regno, const char *name, > > > > +x86_64_get_task_reg(struct task_context *tc, int regno, const char > > > *name, > > > > int size, void *value) > > > > { > > > > if (regno >= LAST_REGNUM) > > > > return FALSE; > > > > > > > > + /* > > > > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15 > > > from > > > > + * inactive_task_frame (see __switch_to_asm). Other regs saved on > > > > + * regular frame. > > > > + */ > > > > + if (!is_task_active(tc->task)) { > > > > + int frame_size = STRUCT_SIZE("inactive_task_frame"); > > > > + > > > > + /* Only modern kernels supported. */ > > > > + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) { > > > > + ulong rsp; > > > > + int offset = 0; > > > > + switch (regno) { > > > > + case RSP_REGNUM: > > > > + readmem(tc->task + > > > OFFSET(task_struct_thread) + > > > > + > > > OFFSET(thread_struct_rsp), KVADDR, > > > > + &rsp, sizeof(void *), > > > > + "thread_struct rsp", > > > FAULT_ON_ERROR); > > > > + rsp += frame_size; > > > > + memcpy(value, &rsp, size); > > > > + return TRUE; > > > > + case RIP_REGNUM: > > > > + offset += 8; > > > > + case RBP_REGNUM: > > > > + offset += 8; > > > > + case RBX_REGNUM: > > > > + offset += 8; > > > > + case R12_REGNUM: > > > > + offset += 8; > > > > + case R13_REGNUM: > > > > + offset += 8; > > > > + case R14_REGNUM: > > > > + offset += 8; > > > > + case R15_REGNUM: > > > > + readmem(tc->task + > > > OFFSET(task_struct_thread) + > > > > + > > > OFFSET(thread_struct_rsp), KVADDR, > > > > + &rsp, sizeof(void *), > > > > + "thread_struct rsp", > > > FAULT_ON_ERROR); > > > > + readmem(rsp + offset, KVADDR, > > > value, sizeof(void *), > > > > + > > > "inactive_thread_frame saved regs", FAULT_ON_ERROR); > > > > + return TRUE; > > > > + } > > > > + } > > > > + /* TBD: older kernels support. */ > > > > + return FALSE; > > > > + } > > > > + > > > > + /* > > > > + * Task is active, grab CPU's registers > > > > + */ > > > > if (VMSS_DUMPFILE()) > > > > - return vmware_vmss_get_cpu_reg(cpu, regno, name, size, > > > value); > > > > + return vmware_vmss_get_cpu_reg(tc->processor, regno, > > > name, size, value); > > > > > > > > return FALSE; > > > > } > > > > -- > > > > 2.39.0 > > > > > > > > > > >

Hi Tao, I like the idea of having just one thread at a time from the gdb side, and it can be replaced by a "set" crash command. Single semantics will work for both live and offline debugging and for both types of tasks: active or scheduled out. Need to be careful with lifetime for the data (gdb caches). I'm experimenting with live debugging these days. Aditya, Tao, let me know what is the latest patchset I should use from you folks?

Thanks, --Alexey On Mon, Mar 18, 2024 at 2:56 AM Tao Liu <ltao(a)redhat.com&gt; wrote: > > Hi Aditya and Alexey, > > On Mon, Mar 18, 2024 at 11:35:41AM +0800, Tao Liu wrote: > > On Sun, Mar 17, 2024 at 03:40:52PM +0530, Aditya Gupta wrote: > > > Hi Tao and Alexey, > > > > > > On Fri, Mar 15, 2024 at 03:15:19PM -0700, Alexey Makhalov wrote: > > > > Hi Tao, > > > > > > > > Thanks for the details. Agree, my approach is targeted primarily for > > > > offline debugging. > > > > Create/exit monitoring sounds like an overkill. And from the other side, > > > > maintaining a static list of threads (CPUs or Tasks) does not work for live > > > > debugging. > > > > "ps" followed by "set" where GDB will receive just one inferior/thread at a > > > > time is probably the right universal approach. > > > > Let me experiment with a combination of my and your patches and come back > > > > to you. > > > > > > Sorry I noticed a bug late while trying the combination of tao's and my > > > patches. > > > > > > I agree with Tao that current approach might not be good for live > > > debugging, but I think the approach of having all tasks in 'info > > > threads' is a good advantage for offline debugging, since then gdb knows > > > about all tasks (similar behaviour as usual gdb), and makes it easier > > > to synchronise crash <-> gdb context. > > > > > > I had refrained from changing the format of 'info threads' from active > > > tasks to all tasks, but there are some issues with crash <-> gdb sync, > > > when doing arbitrary task unwinding. > > > > > > 'set 1' (let's assume it's an INACTIVE task on CPU 5) > > > > > > 'info threads' > > > -> when it switches to CPU 5, it asks crash to sync, ie. `set_cpu(5)` > > > -> crash will return CPU 5's active tasks registers > > > -> WRONG registers for CPU 5, since we wanted registers for PID 1 (which > > > is an inactive task on CPU 5) > > > > What do you mean of "WRONG registers"? Isn't the CPU5's regcache be pid > > 1's context after 'set 1'? If it isn't then it's a bug. > > > > After crash finished loading and we see crash> prompt, the cpu regcaches > > will be all active tasks', which are running on each cpu. That is what we saw > > using "info threads", like: > > > > crash> info threads > > Id Target Id Frame > > .... > > * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 > > .... > > 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 > > .... > > > > Then we use "set <pid>" to switch to task <pid>, its cpu regcache will be > > flushed, and if we use command "info threads" again, it will show the status > > after the flushing. E.g in the following, task pid 1's cpu is #13, so > > cpu #13 is flushed after "set 1": > > > > crash> ps > > PID PPID CPU TASK ST %MEM VSZ RSS COMM > > 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd > > > > crash> set 1 > > crash> info threads > > Id Target Id Frame > > .... > > 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 > > .... > > * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^... > > .... > > > > When design, I assume the CPUs/threads are like power slots, its number is > > constant, but the contents can be changed. We don't need to prepare the > > "slots" as many as our tasks quantity (I don't know the maxinum threads that > > a gdb can accept, e.g. on a system which have 65K tasks, but for most > > tasks we'd never switch to it in order to view its stack trace, I guess > > it would be a waste to call add_thread_silent (target, ...) 65K times). > > > > Currently the gdb CPUs/threads(slots) equals to the system's cpu number. But > > if a system which have thousands of cpus, which we think is unnecessary to > > call add_thread_silent 1K times, we can even shrink the slots to be less. > > > > I made a trial patch of leaving only 1 gdb CPUs/threads(slots) in [1]. > And it can also view arbitrary tasks stack unwinding. In this case, > "info threads" and "thread XX" will have no meaning to be kept. > > Please see my test results: > > KERNEL: /home/ltao/kernel/linux-next-next-20220914/vmlinux > DUMPFILE: /proc/kcore > CPUS: 4 > ^^^^^^^ there are 4 cpus in the system > crash> ps > PID PPID CPU TASK ST %MEM VSZ RSS COMM > ... > 1 0 1 ffff9908002a6500 IN 0.1 253024 19444 systemd > ... > 5 2 0 ffff9908002a0000 ID 0.0 0 0 [netns] > ... > ^^^^^^^^^^^^^ we use pid 1 and pid 5, which belongs to 2 different cpus > crash> set 1 > crash> gdb bt > #0 context_switch (rf=0xffffbf750001bd68, next=0xffff99080032cbc0, prev=0x0, rq=0xffff990b6fcafe80) at kernel/sched/core.c:5201 > #1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6510 > #2 0xffffffffb3cf0603 in schedule () at kernel/sched/core.c:6586 > #3 0xffffffffb3cf6ab9 in schedule_hrtimeout_range_clock (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS, clock_id=clock_id@entry=1) at kernel/time/hrtimer.c:2296 > ... > crash> bt > PID: 1 TASK: ffff9908002a6500 CPU: 1 COMMAND: "systemd" > #0 [ffffbf750001bdb0] schedule at ffffffffb3cf0603 > #1 [ffffbf750001bdc0] schedule_hrtimeout_range_clock at ffffffffb3cf6ab9 > ... > ^^^ we can view pid 1's stack trace with no problem > crash> info threads > Id Target Id Frame > * 1 CPU 0 context_switch (rf=0xffffbf750001bd68, next=0xffffffffb4c18a00 <init_task>, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201 > > crash> set 5 > crash> gdb bt > #0 context_switch (rf=0xffffbf750003be58, next=0xffff9908002d6500, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201 > #1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6510 > #2 0xffffffffb3cf0603 in schedule () at kernel/sched/core.c:6586 > #3 0xffffffffb3308752 in rescuer_thread (__rescuer=0xffff9908001f7480) at kernel/workqueue.c:2599 > ... > crash> bt > PID: 5 TASK: ffff9908002a0000 CPU: 0 COMMAND: "netns" > #0 [ffffbf750003bea0] schedule at ffffffffb3cf0603 > #1 [ffffbf750003beb0] rescuer_thread at ffffffffb3308752 > ... > ^^^ we can view pid 5's stack trace with no problem > crash> info threads > Id Target Id Frame > * 1 CPU 0 context_switch (rf=0xffffbf750003be58, next=0xffff9908002d6500, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201 > ^^^^^^^^^^^^^^^^^^^^^ now its pid 5's context > > From my view, I think we can keep the current "gdb thread/CPUs = nr_cpus", or > be more ambitious, leaving "gdb thread/CPUs = 1", in case there are > systems with thousands of cpus, and we only need this in task.c: > > + gdb_refresh_regcache(0); > - for (int i = 0; i < get_cpus_online(); i++) > - gdb_refresh_regcache(i); > > Do you have any thoughts? > > [1]: https://github.com/liutgnu/crash-dev/commits/one-thread > > Thanks, > Tao Liu > > > Thanks, > > Tao Liu > > > > > > > > > > I can think of solving it in two ways: > > > 1. Having a array in crash_target, which tells what CPU in gdb maps to > > > which task in crash, this can be done without requiring changes to 'info > > > threads', while we do the 'add_thread_silent' > > > 2. Having all tasks in gdb itself, so it's a one-to-one, so `set_cpu` > > > will then refer to the task ID, I will have to explore more on this. > > > > > > Any comments ? > > > > > > Thanks, > > > Aditya Gupta > > > > > > > > > > > Thanks, > > > > --Alexey > > > > > > > > > > > > On Thu, Mar 14, 2024 at 9:22 PM Tao Liu <ltao(a)redhat.com&gt; wrote: > > > > > > > > > Hi Alexey, > > > > > > > > > > On Thu, Mar 14, 2024 at 6:29 PM Alexey Makhalov > > > > > <alexey.makhalov(a)broadcom.com&gt; wrote: > > > > > > > > > > > > Support for GDB debugging of all tasks active and inactive. > > > > > > Before this commit only active tasks were listed by "info threads" > > > > > > with "CPU #" as a Target Id. > > > > > > > > > > > > "info threads" will now show all tasks, similar to "ps", example: > > > > > > crash> info threads > > > > > > Id Target Id Frame > > > > > > * 1 0 swapper/0 0xffffffffadba19d4 in default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 2 0 swapper/1 0xffffffffadba19d4 in default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 3 0 swapper/2 0xffffffffadba19d4 in default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 4 0 swapper/3 0xffffffffadba19d4 in default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 5 0 swapper/4 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0000f3e88, next=0xffff9ecb04908000, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > ... > > > > > > 730 970325 taskset 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf006a0fd18, next=0xffff9ecb0aec0000, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 731 975217 sleep 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf005743c20, next=0xffff9ecac0692880, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 732 975228 sleep 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf00696fb58, next=0xffff9ecac0690000, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > ... > > > > > > 876 976084 docker 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0153dbd10, next=0xffff9ecac0645100, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 877 976085 systemd-userwor 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0153cbc58, next=0xffff9ecac0645100, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 878 976086 systemd-userwor 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0153e3c58, next=0xffffffffaec15a40 <init_task>, > > > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 879 976087 systemd-userwor 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0153ebc58, next=0xffffffffaec15a40 <init_task>, > > > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > Where "Target ID" contains "PID COMM" of the task > > > > > > > > > > > > > > > > I see here, we chose a different tech path of viewing arbitrary tasks. > > > > > You improved the "info threads" as an alternative to crash "ps" cmd, > > > > > and use "thread X" to switch to task X and view its stacktrace. > > > > > However mine is using crash "ps" cmd first, getting specific tasks pid > > > > > or task struct, then using crash cmd "set <pid>" or "set > > > > > <task_struct>" to switch to task X and view its stacktrace. > > > > > > > > > > I notice there is one problem of using the "info threads as ps" tech > > > > > path, it is difficult to handle the live debug, see the comment below: > > > > > > > > > > > Example of "731 975217 sleep" debugging, real case, trying to > > > > > > figure out why sleep was stuck in uninterruptable sleep. > > > > > > Backtrace using crash: > > > > > > crash> ps | grep 975217 > > > > > > 975217 969797 3 ffff9ecb3956a880 UN 0.0 0 0 > > > > > sleep > > > > > > crash> bt 975217 > > > > > > PID: 975217 TASK: ffff9ecb3956a880 CPU: 3 COMMAND: "sleep" > > > > > > #0 [ffffbaf005743ba0] __schedule at ffffffffadb97292 > > > > > > #1 [ffffbaf005743c60] schedule at ffffffffadb982b8 > > > > > > #2 [ffffbaf005743c80] rwbase_write_lock at ffffffffadb9aed7 > > > > > > #3 [ffffbaf005743cc0] down_write at ffffffffadb9b133 > > > > > > #4 [ffffbaf005743cd0] unlink_file_vma at ffffffffad2b0e2e > > > > > > #5 [ffffbaf005743cf8] free_pgtables at ffffffffad2a47b0 > > > > > > #6 [ffffbaf005743d88] exit_mmap at ffffffffad2b3b8d > > > > > > #7 [ffffbaf005743e80] mmput at ffffffffad08c81f > > > > > > #8 [ffffbaf005743e98] do_exit at ffffffffad09636c > > > > > > #9 [ffffbaf005743ef8] do_group_exit at ffffffffad096c78 > > > > > > RIP: 00007f111c70ddf9 RSP: 00007fff451817e8 RFLAGS: 00000246 > > > > > > RAX: ffffffffffffffda RBX: 00007f111c8089e0 RCX: 00007f111c70ddf9 > > > > > > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 > > > > > > RBP: 0000000000000000 R8: ffffffffffffff80 R9: 0000000000000000 > > > > > > R10: 00007fff451817b0 R11: 0000000000000246 R12: 00007f111c8089e0 > > > > > > R13: 00007f111c80e2e0 R14: 0000000000000002 R15: 00007f111c80e2c8 > > > > > > ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b > > > > > > > > > > > > Backtrace using gdb (pay attention, task must be selected by thread Id): > > > > > > crash> thread 731 > > > > > > [Switching to thread 731 ( 975217 sleep)] > > > > > > 5372 switch_to(prev, next, prev); > > > > > > crash> gdb bt > > > > > > #0 0xffffffffadb97292 in context_switch (rf=0xffffbaf005743c20, > > > > > next=0xffff9ecac0692880, prev=<optimized out>, rq=<optimized out>) at > > > > > kernel/sched/core.c:5372 > > > > > > #1 __schedule (sched_mode=sched_mode@entry=0) at > > > > > kernel/sched/core.c:6696 > > > > > > #2 0xffffffffadb982b8 in schedule () at kernel/sched/core.c:6772 > > > > > > #3 0xffffffffadb9aed7 in rwbase_write_lock (rwb=rwb@entry=0xffff9ecaf1831430, > > > > > state=state@entry=2) at kernel/locking/rwbase_rt.c:259 > > > > > > #4 0xffffffffadb9b133 in __down_write (sem=sem@entry=0xffff9ecaf1831430) > > > > > at kernel/locking/rwsem.c:1474 > > > > > > #5 down_write (sem=sem@entry=0xffff9ecaf1831430) at > > > > > kernel/locking/rwsem.c:1574 > > > > > > #6 0xffffffffad2b0e2e in i_mmap_lock_write (mapping=<optimized out>) > > > > > at ./include/linux/fs.h:466 > > > > > > #7 unlink_file_vma (vma=vma@entry=0xffff9ecadd566090) at mm/mmap.c:127 > > > > > > #8 0xffffffffad2a47b0 in free_pgtables (tlb=tlb@entry=0xffffbaf005743dd0, > > > > > mt=mt@entry=0xffff9ecb28e3b180, vma=0xffff9ecadd566090, vma@entry=0xffff9ecadd566000, > > > > > floor=floor@entry=0, ceiling=ceiling@entry=0) at mm/memory.c:431 > > > > > > #9 0xffffffffad2b3b8d in exit_mmap (mm=mm@entry=0xffff9ecb28e3b180) > > > > > at mm/mmap.c:3237 > > > > > > #10 0xffffffffad08c81f in __mmput (mm=0xffff9ecb28e3b180) at > > > > > kernel/fork.c:1204 > > > > > > #11 mmput (mm=mm@entry=0xffff9ecb28e3b180) at kernel/fork.c:1226 > > > > > > #12 0xffffffffad09636c in exit_mm () at kernel/exit.c:563 > > > > > > #13 do_exit (code=code@entry=0) at kernel/exit.c:856 > > > > > > #14 0xffffffffad096c78 in do_group_exit (exit_code=0) at > > > > > kernel/exit.c:1019 > > > > > > #15 0xffffffffad096cf8 in __do_sys_exit_group (error_code=<optimized > > > > > out>) at kernel/exit.c:1030 > > > > > > #16 __se_sys_exit_group (error_code=<optimized out>) at > > > > > kernel/exit.c:1028 > > > > > > #17 __x64_sys_exit_group (regs=<optimized out>) at kernel/exit.c:1028 > > > > > > #18 0xffffffffadb8a327 in do_syscall_x64 (nr=<optimized out>, > > > > > regs=0xffffbaf005743f58) at arch/x86/entry/common.c:51 > > > > > > #19 do_syscall_64 (regs=0xffffbaf005743f58, nr=<optimized out>) at > > > > > arch/x86/entry/common.c:81 > > > > > > #20 0xffffffffadc000dc in entry_SYSCALL_64 () at > > > > > arch/x86/entry/entry_64.S:120 > > > > > > #21 0x00007f111c80e2c8 in ?? () > > > > > > #22 0x0000000000000002 in ?? () > > > > > > #23 0x00007f111c80e2e0 in ?? () > > > > > > #24 0x00007f111c8089e0 in ?? () > > > > > > #25 0x0000000000000000 in ?? () > > > > > > crash> f 3 > > > > > > 259 rwbase_schedule(); > > > > > > crash> p *rwb > > > > > > $1 = { > > > > > > readers = { > > > > > > counter = 1 > > > > > > }, > > > > > > rtmutex = { > > > > > > wait_lock = { > > > > > > raw_lock = { > > > > > > { > > > > > > val = { > > > > > > counter = 0 > > > > > > }, > > > > > > { > > > > > > locked = 0 '\000', > > > > > > pending = 0 '\000' > > > > > > }, > > > > > > { > > > > > > locked_pending = 0, > > > > > > tail = 0 > > > > > > } > > > > > > } > > > > > > } > > > > > > }, > > > > > > waiters = { > > > > > > rb_root = { > > > > > > rb_node = 0xffffbaf006977be0 > > > > > > }, > > > > > > rb_leftmost = 0xffffbaf00696fbe0 > > > > > > }, > > > > > > owner = 0xffff9ecb3956a881 > > > > > > } > > > > > > } > > > > > > > > > > > > Additional changes: > > > > > > 1. Allow gdb "frame" command. > > > > > > 2. Blacklist useless gdb "gcore" command. Use gcore plugin instead. > > > > > > 3. Move crash_target_init() to later time as crash target requires a > > > > > list of > > > > > > tasks to be initialized. > > > > > > > > > > > > Known issues and TBD items: > > > > > > 1. "info threads" may bail out first time throwing errors trying to > > > > > access > > > > > > userspace address during unwind process. Following "info threads" > > > > > > invokations run without issues. > > > > > > 2. To unwind a stack of inactive task, only modern Linux versions, which > > > > > use > > > > > > inactive_task_frame, are supported and only x86_64 architecture. > > > > > > 3. gdb bt unwinder does not stop properly and may show invalid frames > > > > > (21-25 > > > > > > on example above). Not a regression, existed before. > > > > > > 4. gdb bt unwinder does not work on active tasks in userspace. Not a > > > > > regression, > > > > > > existed before. > > > > > > 5. Only x86_64 architecture supported. machdep->get_task_reg() must be > > > > > > implemented for others. Not a regression, existed before. > > > > > > 6. Active tasks registers fetching imlemented only for VMware dumps, see > > > > > > x86_64_get_task_reg() for more details. Not a regression, existed > > > > > before. > > > > > > > > > > > > Signed-off-by: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > > > > > > --- > > > > > > crash_target.c | 39 ++++++++++++++++++++--------- > > > > > > defs.h | 10 ++++++-- > > > > > > gdb-10.2.patch | 7 ++---- > > > > > > gdb_interface.c | 63 ++++++++++++++++++++++++++++++---------------- > > > > > > help.c | 1 + > > > > > > main.c | 1 + > > > > > > task.c | 1 + > > > > > > x86_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++------ > > > > > > 8 files changed, 140 insertions(+), 48 deletions(-) > > > > > > > > > > > > diff --git a/crash_target.c b/crash_target.c > > > > > > index 4554806..2fdf203 100644 > > > > > > --- a/crash_target.c > > > > > > +++ b/crash_target.c > > > > > > @@ -2,6 +2,8 @@ > > > > > > * crash_target.c > > > > > > * > > > > > > * Copyright (c) 2021 VMware, Inc. > > > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" > > > > > > + * refers to Broadcom Inc. and/or its subsidiaries. > > > > > > * > > > > > > * This program is free software; you can redistribute it and/or modify > > > > > > * it under the terms of the GNU General Public License as published by > > > > > > @@ -13,7 +15,7 @@ > > > > > > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > > > > * GNU General Public License for more details. > > > > > > * > > > > > > - * Author: Alexey Makhalov <amakhalov(a)vmware.com&gt; > > > > > > + * Author: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > > > > > > */ > > > > > > > > > > > > #include <defs.h> > > > > > > @@ -23,11 +25,11 @@ > > > > > > #include "regcache.h" > > > > > > #include "gdbarch.h" > > > > > > > > > > > > -void crash_target_init (void); > > > > > > - > > > > > > +extern "C" void crash_target_init (void); > > > > > > extern "C" int gdb_readmem_callback(unsigned long, void *, int, int); > > > > > > -extern "C" int crash_get_nr_cpus(void); > > > > > > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char > > > > > *regname, > > > > > > +extern "C" int crash_get_nr_tasks(void); > > > > > > +extern "C" void crash_get_task_info(int task_nr, unsigned long *pid, > > > > > char **comm); > > > > > > +extern "C" int crash_get_task_reg (int task_nr, int regno, const char > > > > > *regname, > > > > > > int regsize, void *val); > > > > > > > > > > > > > > > > > > @@ -60,7 +62,13 @@ public: > > > > > > bool has_registers () override { return true; } > > > > > > bool thread_alive (ptid_t ptid) override { return true; } > > > > > > std::string pid_to_str (ptid_t ptid) override > > > > > > - { return string_printf ("CPU %ld", ptid.tid ()); } > > > > > > + { > > > > > > + unsigned long pid; > > > > > > + char *comm; > > > > > > + > > > > > > + crash_get_task_info(ptid.tid(), &pid, &comm); > > > > > > + return string_printf ("%7ld %s", pid, comm); > > > > > > + } > > > > > > > > > > > > }; > > > > > > > > > > > > @@ -68,18 +76,25 @@ public: > > > > > > void > > > > > > crash_target::fetch_registers (struct regcache *regcache, int regno) > > > > > > { > > > > > > + int r; > > > > > > gdb_byte regval[16]; > > > > > > - int cpu = inferior_ptid.tid(); > > > > > > + int task_nr = inferior_ptid.tid(); > > > > > > struct gdbarch *arch = regcache->arch (); > > > > > > > > > > > > - for (int r = 0; r < gdbarch_num_regs (arch); r++) > > > > > > + if (regno >= 0) { > > > > > > + r = regno; > > > > > > + goto onetime; > > > > > > + } > > > > > > + > > > > > > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) > > > > > > { > > > > > > +onetime: > > > > > > const char *regname = gdbarch_register_name(arch, r); > > > > > > int regsize = register_size (arch, r); > > > > > > if (regsize > sizeof (regval)) > > > > > > error (_("fatal error: buffer size is not enough to fit > > > > > register value")); > > > > > > > > > > > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)&regval)) > > > > > > + if (crash_get_task_reg (task_nr, r, regname, regsize, (void > > > > > *)&regval)) > > > > > > regcache->raw_supply (r, regval); > > > > > > else > > > > > > regcache->raw_supply (r, NULL); > > > > > > @@ -107,10 +122,10 @@ crash_target::xfer_partial (enum target_object > > > > > object, const char *annex, > > > > > > > > > > > > #define CRASH_INFERIOR_PID 1 > > > > > > > > > > > > -void > > > > > > +extern "C" void > > > > > > crash_target_init (void) > > > > > > { > > > > > > - int nr_cpus = crash_get_nr_cpus(); > > > > > > + int nr_tasks = crash_get_nr_tasks(); > > > > > > crash_target *target = new crash_target (); > > > > > > > > > > > > /* Own the target until it is successfully pushed. */ > > > > > > @@ -119,7 +134,7 @@ crash_target_init (void) > > > > > > push_target (std::move (target_holder)); > > > > > > > > > > > > inferior_appeared (current_inferior (), CRASH_INFERIOR_PID); > > > > > > - for (int i = 0; i < nr_cpus; i++) > > > > > > + for (int i = 0; i < nr_tasks; i++) > > > > > > { > > > > > > thread_info *thread = add_thread_silent (target, > > > > > > ptid_t(CRASH_INFERIOR_PID, 0, > > > > > i)); > > > > > > diff --git a/defs.h b/defs.h > > > > > > index 98650e8..2b3f247 100644 > > > > > > --- a/defs.h > > > > > > +++ b/defs.h > > > > > > @@ -1080,7 +1080,7 @@ struct machdep_table { > > > > > > void (*get_irq_affinity)(int); > > > > > > void (*show_interrupts)(int, ulong *); > > > > > > int (*is_page_ptr)(ulong, physaddr_t *); > > > > > > - int (*get_cpu_reg)(int, int, const char *, int, void *); > > > > > > + int (*get_task_reg)(struct task_context *, int, const char *, > > > > > int, void *); > > > > > > int (*is_cpu_prstatus_valid)(int cpu); > > > > > > }; > > > > > > > > > > > > @@ -2263,6 +2263,7 @@ struct size_table { /* stash of > > > > > commonly-used sizes */ > > > > > > long pt_regs; > > > > > > long task_struct; > > > > > > long thread_info; > > > > > > + long inactive_task_frame; > > > > > > long softirq_state; > > > > > > long desc_struct; > > > > > > long umode_t; > > > > > > @@ -8001,9 +8002,14 @@ extern int have_full_symbols(void); > > > > > > #define XEN_HYPERVISOR_ARCH > > > > > > #endif > > > > > > > > > > > > +/* > > > > > > + * crash_target.c > > > > > > + */ > > > > > > +extern void crash_target_init (void); > > > > > > + > > > > > > /* > > > > > > * Register numbers must be in sync with gdb/features/i386/64bit-core.c > > > > > > - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg() > > > > > > + * to make crash_target->fetch_registers() ---> machdep->get_task_reg() > > > > > > * working properly. > > > > > > */ > > > > > > enum x86_64_regnum { > > > > > > diff --git a/gdb-10.2.patch b/gdb-10.2.patch > > > > > > index a7018a2..ecf673d 100644 > > > > > > --- a/gdb-10.2.patch > > > > > > +++ b/gdb-10.2.patch > > > > > > @@ -221,7 +221,7 @@ exit 0 > > > > > > warning (_("\ > > > > > > --- gdb-10.2/gdb/main.c.orig > > > > > > +++ gdb-10.2/gdb/main.c > > > > > > -@@ -392,6 +392,14 @@ start_event_loop () > > > > > > +@@ -392,6 +392,13 @@ start_event_loop () > > > > > > return; > > > > > > } > > > > > > > > > > > > @@ -230,7 +230,6 @@ exit 0 > > > > > > +extern "C" void main_loop(void); > > > > > > +extern "C" unsigned long crash_get_kaslr_offset(void); > > > > > > +extern "C" int console(const char *, ...); > > > > > > -+void crash_target_init (void); > > > > > > +#endif > > > > > > + > > > > > > /* Call command_loop. */ > > > > > > @@ -316,7 +315,7 @@ exit 0 > > > > > > } > > > > > > } > > > > > > > > > > > > -@@ -1242,6 +1274,16 @@ captured_main (void *data) > > > > > > +@@ -1242,6 +1274,14 @@ captured_main (void *data) > > > > > > > > > > > > captured_main_1 (context); > > > > > > > > > > > > @@ -324,8 +323,6 @@ exit 0 > > > > > > + /* Relocate the vmlinux. */ > > > > > > + objfile_rebase (symfile_objfile, crash_get_kaslr_offset()); > > > > > > + > > > > > > -+ crash_target_init(); > > > > > > -+ > > > > > > + /* Back to crash. */ > > > > > > + main_loop(); > > > > > > +#endif > > > > > > diff --git a/gdb_interface.c b/gdb_interface.c > > > > > > index b14319c..03178f5 100644 > > > > > > --- a/gdb_interface.c > > > > > > +++ b/gdb_interface.c > > > > > > @@ -3,6 +3,8 @@ > > > > > > * Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > > > > > * Copyright (C) 2002-2015,2018-2019 David Anderson > > > > > > * Copyright (C) 2002-2015,2018-2019 Red Hat, Inc. All rights reserved. > > > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" > > > > > > + * refers to Broadcom Inc. and/or its subsidiaries. > > > > > > * > > > > > > * This program is free software; you can redistribute it and/or modify > > > > > > * it under the terms of the GNU General Public License as published by > > > > > > @@ -711,7 +713,7 @@ static char *prohibited_list[] = { > > > > > > "watch", "rwatch", "awatch", "attach", "continue", "c", "fg", > > > > > "detach", > > > > > > "finish", "handle", "interrupt", "jump", "kill", "next", "nexti", > > > > > > "signal", "step", "s", "stepi", "target", "until", "delete", > > > > > > - "clear", "disable", "enable", "condition", "ignore", "frame", > > > > > "catch", > > > > > > + "clear", "disable", "enable", "condition", "ignore", "gcore", > > > > > "catch", > > > > > > "tcatch", "return", "file", "exec-file", "core-file", > > > > > "symbol-file", > > > > > > "load", "si", "ni", "shell", "sy", > > > > > > NULL /* must be last */ > > > > > > @@ -877,6 +879,7 @@ gdb_readmem_callback(ulong addr, void *buf, int len, > > > > > int write) > > > > > > switch (len) > > > > > > { > > > > > > case SIZEOF_8BIT: > > > > > > + fprintf(fp, "%s\n", pc->curcmd); > > > > > > if (STREQ(pc->curcmd, "bt")) { > > > > > > if (readmem(addr, memtype, buf, SIZEOF_8BIT, > > > > > > "gdb_readmem_callback", readflags)) > > > > > > @@ -1063,34 +1066,52 @@ get_frame_offset(ulong pc) > > > > > > unsigned long crash_get_kaslr_offset(void); > > > > > > unsigned long crash_get_kaslr_offset(void) > > > > > > { > > > > > > - return kt->relocate * -1; > > > > > > + return kt->relocate * -1; > > > > > > } > > > > > > > > > > > > /* Callbacks for crash_target */ > > > > > > -int crash_get_nr_cpus(void); > > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > > > > +int crash_get_nr_tasks(void); > > > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm); > > > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname, > > > > > > int regsize, void *val); > > > > > > > > > > > > -int crash_get_nr_cpus(void) > > > > > > +int crash_get_nr_tasks(void) > > > > > > { > > > > > > - if (SADUMP_DUMPFILE()) > > > > > > - return sadump_get_nr_cpus(); > > > > > > - else if (DISKDUMP_DUMPFILE()) > > > > > > - return diskdump_get_nr_cpus(); > > > > > > - else if (KDUMP_DUMPFILE()) > > > > > > - return kdump_get_nr_cpus(); > > > > > > - else if (VMSS_DUMPFILE()) > > > > > > - return vmware_vmss_get_nr_cpus(); > > > > > > - > > > > > > - /* Just CPU #0 */ > > > > > ()> - return 1; > > > > > > + return RUNNING_TASKS(); > > > > > > } > > > > > > > > > > The crash_get_nr_tasks() will return a fixed number of running tasks, > > > > > which is the task status when crash is loading. However for live debug > > > > > mode, there will always be tasks exit and create, so we need to sync > > > > > the tasks number with the "info threads". I guess it is not an easy > > > > > work because that will involve: > > > > > > > > > > when crash loading: > > > > > for (int i = 0; i < nr_tasks; i++) > > > > > { > > > > > thread_info *thread = add_thread_silent (target, ... > > > > > } > > > > > > > > > > when new task create: > > > > > thread_info *thread = add_thread_silent (target, ... > > > > > > > > > > when task exit: > > > > > delete_thread(...) > > > > > > > > > > We may monitor those task create/exit events and do thread add/delete > > > > > in a callback function or we just always do all tasks delete/add when > > > > > "info thread" is invoked. But I guess none of these are as simple as > > > > > using crash "ps" cmd and "set <pid>", there won't be any problem. > > > > > Because crash "ps" can always show the current tasks status, and set > > > > > <pid> can switch the task context to it. > > > > > > > > > > What do you think? > > > > > > > > > > Thanks, > > > > > Tao Liu > > > > > > > > > > > > > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > > > > - int regsize, void *value) > > > > > > +/* Get task information by its index number in TT */ > > > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm) > > > > > > { > > > > > > - if (!machdep->get_cpu_reg) > > > > > > - return FALSE; > > > > > > - return machdep->get_cpu_reg(cpu, regno, regname, regsize, > > > > > value); > > > > > > + int i; > > > > > > + struct task_context *tc; > > > > > > + > > > > > > + tc = FIRST_CONTEXT(); > > > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > > > > + if (i == task_nr) { > > > > > > + *pid = tc->pid; > > > > > > + *comm = tc->comm; > > > > > > + return; > > > > > > + } > > > > > > + *pid = 0; > > > > > > + *comm = NULL; > > > > > > + return; > > > > > > +} > > > > > > + > > > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname, > > > > > > + int regsize, void *value) > > > > > > +{ > > > > > > + int i; > > > > > > + struct task_context *tc; > > > > > > + > > > > > > + if (!machdep->get_task_reg) > > > > > > + return FALSE; > > > > > > + > > > > > > + tc = FIRST_CONTEXT(); > > > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > > > > + if (i == task_nr) { > > > > > > + return machdep->get_task_reg(tc, regno, regname, > > > > > regsize, value); > > > > > > + } > > > > > > + return FALSE; > > > > > > } > > > > > > > > > > > > diff --git a/help.c b/help.c > > > > > > index a9c4d30..85dbda5 100644 > > > > > > --- a/help.c > > > > > > +++ b/help.c > > > > > > @@ -8520,6 +8520,7 @@ char *version_info[] = { > > > > > > "Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.", > > > > > > "Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.", > > > > > > "Copyright (C) 2015, 2021 VMware, Inc.", > > > > > > +"Copyright (C) 2024 Broadcom, Inc.", > > > > > > "This program is free software, covered by the GNU General Public > > > > > License,", > > > > > > "and you are welcome to change it and/or distribute copies of it under", > > > > > > "certain conditions. Enter \"help copying\" to see the conditions.", > > > > > > diff --git a/main.c b/main.c > > > > > > index 0b6b927..13acd2d 100644 > > > > > > --- a/main.c > > > > > > +++ b/main.c > > > > > > @@ -794,6 +794,7 @@ main_loop(void) > > > > > > } else > > > > > > SIGACTION(SIGINT, restart, &pc->sigaction, NULL); > > > > > > > > > > > > + crash_target_init(); > > > > > > /* > > > > > > * Display system statistics and current context. > > > > > > */ > > > > > > diff --git a/task.c b/task.c > > > > > > index ebdb5be..5d26c52 100644 > > > > > > --- a/task.c > > > > > > +++ b/task.c > > > > > > @@ -298,6 +298,7 @@ task_init(void) > > > > > > tt->flags |= THREAD_INFO; > > > > > > } > > > > > > > > > > > > + STRUCT_SIZE_INIT(inactive_task_frame, "inactive_task_frame"); > > > > > > MEMBER_OFFSET_INIT(task_struct_state, "task_struct", "state"); > > > > > > MEMBER_SIZE_INIT(task_struct_state, "task_struct", "state"); > > > > > > if (INVALID_MEMBER(task_struct_state)) { > > > > > > diff --git a/x86_64.c b/x86_64.c > > > > > > index 502817d..b6e36a5 100644 > > > > > > --- a/x86_64.c > > > > > > +++ b/x86_64.c > > > > > > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *, > > > > > ulong, ulong, char *); > > > > > > static void x86_64_framesize_debug(struct bt_info *); > > > > > > static void x86_64_get_active_set(void); > > > > > > static int x86_64_get_kvaddr_ranges(struct vaddr_range *); > > > > > > -static int x86_64_get_cpu_reg(int, int, const char *, int, void *); > > > > > > +static int x86_64_get_task_reg(struct task_context *, int, const char > > > > > *, int, void *); > > > > > > static int x86_64_verify_paddr(uint64_t); > > > > > > static void GART_init(void); > > > > > > static void x86_64_exception_stacks_init(void); > > > > > > @@ -195,7 +195,7 @@ x86_64_init(int when) > > > > > > machdep->machspec->irq_eframe_link = UNINITIALIZED; > > > > > > machdep->machspec->irq_stack_gap = UNINITIALIZED; > > > > > > machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges; > > > > > > - machdep->get_cpu_reg = x86_64_get_cpu_reg; > > > > > > + machdep->get_task_reg = x86_64_get_task_reg; > > > > > > if (machdep->cmdline_args[0]) > > > > > > parse_cmdline_args(); > > > > > > if ((string = pc->read_vmcoreinfo("relocate"))) { > > > > > > @@ -891,7 +891,7 @@ x86_64_dump_machdep_table(ulong arg) > > > > > > fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n"); > > > > > > fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n"); > > > > > > fprintf(fp, " get_kvaddr_ranges: > > > > > x86_64_get_kvaddr_ranges()\n"); > > > > > > - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n"); > > > > > > + fprintf(fp, " get_task_reg: x86_64_get_task_reg()\n"); > > > > > > fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n"); > > > > > > fprintf(fp, "clear_machdep_cache: > > > > > x86_64_clear_machdep_cache()\n"); > > > > > > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? > > > > > > @@ -6398,6 +6398,9 @@ x86_64_ORC_init(void) > > > > > > }; > > > > > > struct ORC_data *orc; > > > > > > > > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_bp, > > > > > "inactive_task_frame", "bp"); > > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, > > > > > "inactive_task_frame", "ret_addr"); > > > > > > + > > > > > > if (machdep->flags & FRAMEPOINTER) > > > > > > return; > > > > > > > > > > > > @@ -6455,9 +6458,6 @@ x86_64_ORC_init(void) > > > > > > orc->__stop_orc_unwind = symbol_value("__stop_orc_unwind"); > > > > > > orc->orc_lookup = symbol_value("orc_lookup"); > > > > > > > > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_bp, > > > > > "inactive_task_frame", "bp"); > > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, > > > > > "inactive_task_frame", "ret_addr"); > > > > > > - > > > > > > orc->has_signal = MEMBER_EXISTS("orc_entry", "signal"); /* added > > > > > at 6.3 */ > > > > > > orc->has_end = MEMBER_EXISTS("orc_entry", "end"); /* > > > > > removed at 6.4 */ > > > > > > > > > > > > @@ -9070,14 +9070,64 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp) > > > > > > } > > > > > > > > > > > > static int > > > > > > -x86_64_get_cpu_reg(int cpu, int regno, const char *name, > > > > > > +x86_64_get_task_reg(struct task_context *tc, int regno, const char > > > > > *name, > > > > > > int size, void *value) > > > > > > { > > > > > > if (regno >= LAST_REGNUM) > > > > > > return FALSE; > > > > > > > > > > > > + /* > > > > > > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15 > > > > > from > > > > > > + * inactive_task_frame (see __switch_to_asm). Other regs saved on > > > > > > + * regular frame. > > > > > > + */ > > > > > > + if (!is_task_active(tc->task)) { > > > > > > + int frame_size = STRUCT_SIZE("inactive_task_frame"); > > > > > > + > > > > > > + /* Only modern kernels supported. */ > > > > > > + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) { > > > > > > + ulong rsp; > > > > > > + int offset = 0; > > > > > > + switch (regno) { > > > > > > + case RSP_REGNUM: > > > > > > + readmem(tc->task + > > > > > OFFSET(task_struct_thread) + > > > > > > + > > > > > OFFSET(thread_struct_rsp), KVADDR, > > > > > > + &rsp, sizeof(void *), > > > > > > + "thread_struct rsp", > > > > > FAULT_ON_ERROR); > > > > > > + rsp += frame_size; > > > > > > + memcpy(value, &rsp, size); > > > > > > + return TRUE; > > > > > > + case RIP_REGNUM: > > > > > > + offset += 8; > > > > > > + case RBP_REGNUM: > > > > > > + offset += 8; > > > > > > + case RBX_REGNUM: > > > > > > + offset += 8; > > > > > > + case R12_REGNUM: > > > > > > + offset += 8; > > > > > > + case R13_REGNUM: > > > > > > + offset += 8; > > > > > > + case R14_REGNUM: > > > > > > + offset += 8; > > > > > > + case R15_REGNUM: > > > > > > + readmem(tc->task + > > > > > OFFSET(task_struct_thread) + > > > > > > + > > > > > OFFSET(thread_struct_rsp), KVADDR, > > > > > > + &rsp, sizeof(void *), > > > > > > + "thread_struct rsp", > > > > > FAULT_ON_ERROR); > > > > > > + readmem(rsp + offset, KVADDR, > > > > > value, sizeof(void *), > > > > > > + > > > > > "inactive_thread_frame saved regs", FAULT_ON_ERROR); > > > > > > + return TRUE; > > > > > > + } > > > > > > + } > > > > > > + /* TBD: older kernels support. */ > > > > > > + return FALSE; > > > > > > + } > > > > > > + > > > > > > + /* > > > > > > + * Task is active, grab CPU's registers > > > > > > + */ > > > > > > if (VMSS_DUMPFILE()) > > > > > > - return vmware_vmss_get_cpu_reg(cpu, regno, name, size, > > > > > value); > > > > > > + return vmware_vmss_get_cpu_reg(tc->processor, regno, > > > > > name, size, value); > > > > > > > > > > > > return FALSE; > > > > > > } > > > > > > -- > > > > > > 2.39.0 > > > > > > > > > > > > > > > > > > > >

Hi folks, please take a look on v2 of this patch and give it a try. IMHO, Only remaining piece is to clean up x86_64_get_current_task_reg around getting pt_regs. Things done: 1. Rename machdep->get_cpu_reg to machdep->get_current_task_reg and rename x86_64_get_cpu_reg to x86_64_get_current_task_reg, plus corresponding changes for arm64 and ppc64. 2. Untied gdb's TPID from any tasks or CPUs ID. Always use CURRENT_CONTEXT for regs and properties fetching. 3. When CURRENT switched, gdb_change_thread_context() must be called to invalidate caches. 4. Update x86_64_get_current_task_reg to support inactive tasks for offline debugging. 5. Fixed frame selection issue. It was caused by multiple calls of set_context() which calls gdb_change_thread_context() unnecessarily dropping gdb caches. TODO: 1. x86_64_get_current_task_reg is questionable about fetching pt_regs for inactive tasks. How do you gyus get pt_regs structure on modern linux kernels?

PS: I added my approach. More testing required. Known issues 1. After switching from all tasks to one thread: b. unwinding started to work differently around top (garbaged) frames - Not quite an issue for me. Signed-off-by: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; --- arm64.c | 4 ++-- crash_target.c | 48 ++++++++++++++++++------------------- defs.h | 10 +++----- gdb_interface.c | 8 +++---- kernel.c | 23 ------------------ ppc64.c | 8 +++---- task.c | 16 ++++--------- x86_64.c | 64 ++++++++++++++++++++++++++++++++++++++++++++----- 8 files changed, 99 insertions(+), 82 deletions(-) diff --git a/arm64.c b/arm64.c index 8ff10bf..340db2d 100644 --- a/arm64.c +++ b/arm64.c @@ -153,7 +153,7 @@ static void arm64_calc_kernel_start(void) } static int -arm64_get_cpu_reg(int cpu, int regno, const char *name, +arm64_get_current_task_reg(int regno, const char *name, int size, void *value) { struct bt_info bt_info, bt_setup; @@ -511,7 +511,7 @@ arm64_init(int when) machdep->dumpfile_init = NULL; machdep->verify_line_number = NULL; machdep->init_kernel_pgd = arm64_init_kernel_pgd; - machdep->get_cpu_reg = arm64_get_cpu_reg; + machdep->get_current_task_reg = arm64_get_current_task_reg; /* use machdep parameters */ arm64_calc_phys_offset(); diff --git a/crash_target.c b/crash_target.c index fb3b634..6e96506 100644 --- a/crash_target.c +++ b/crash_target.c @@ -26,11 +26,8 @@ void crash_target_init (void); extern "C" int gdb_readmem_callback(unsigned long, void *, int, int); -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char *regname, +extern "C" int crash_get_current_task_reg (int regno, const char *regname, int regsize, void *val); -extern "C" void gdb_refresh_regcache(unsigned int cpu); -extern "C" int set_cpu(int cpu, int print_context); -extern "C" int crash_set_thread(ulong); extern "C" int gdb_change_thread_context (ulong); extern "C" void crash_get_current_task_info(unsigned long *pid, char **comm); @@ -72,22 +69,27 @@ public: }; -/* We just get all the registers, so we don't use regno. */ void crash_target::fetch_registers (struct regcache *regcache, int regno) { + int r; gdb_byte regval[16]; - int cpu = inferior_ptid.tid(); struct gdbarch *arch = regcache->arch (); - for (int r = 0; r < gdbarch_num_regs (arch); r++) + if (regno >= 0) { + r = regno; + goto onetime; + } + + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) { +onetime: const char *regname = gdbarch_register_name(arch, r); int regsize = register_size (arch, r); if (regsize > sizeof (regval)) error (_("fatal error: buffer size is not enough to fit register value")); - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)&regval)) + if (crash_get_current_task_reg (r, regname, regsize, (void *)&regval)) regcache->raw_supply (r, regval); else regcache->raw_supply (r, NULL); @@ -142,19 +144,17 @@ crash_target_init (void) extern "C" int gdb_change_thread_context (ulong task) { - inferior* inf = current_inferior(); - int cpu = crash_set_thread(task); - if (cpu < 0) - return FALSE; - - ptid_t ptid = ptid_t(CRASH_INFERIOR_PID, 0, cpu); - thread_info *tp = find_thread_ptid (inf, ptid); - - if (tp == nullptr) - return FALSE; - - target_fetch_registers(get_thread_regcache(tp), -1); - switch_to_thread(tp); - reinit_frame_cache (); - return TRUE; -} \ No newline at end of file + static ulong prev_task = 0; + + /* + * Crash calls this method even if CURRENT task is not updated. + * Ignore it and keep gdb caches active. + */ + if (task == prev_task) + return TRUE; + prev_task = task; + + registers_changed(); + reinit_frame_cache(); + return TRUE; +} diff --git a/defs.h b/defs.h index d7e3b18..7596179 100644 --- a/defs.h +++ b/defs.h @@ -1081,7 +1081,7 @@ struct machdep_table { void (*get_irq_affinity)(int); void (*show_interrupts)(int, ulong *); int (*is_page_ptr)(ulong, physaddr_t *); - int (*get_cpu_reg)(int, int, const char *, int, void *); + int (*get_current_task_reg)(int, const char *, int, void *); int (*is_cpu_prstatus_valid)(int cpu); }; @@ -6124,7 +6124,6 @@ void dump_kernel_table(int); void dump_bt_info(struct bt_info *, char *where); void dump_log(int); void parse_kernel_version(char *); -int crash_set_thread(ulong); #define LOG_LEVEL(v) ((v) & 0x07) #define SHOW_LOG_LEVEL (0x1) @@ -8023,7 +8022,7 @@ extern int have_full_symbols(void); /* * Register numbers must be in sync with gdb/features/i386/64bit-core.c - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg() + * to make crash_target->fetch_registers() ---> machdep->get_current_task_reg() * working properly. */ enum x86_64_regnum { @@ -8081,7 +8080,7 @@ enum arm64_regnum { /* * Register numbers to make crash_target->fetch_registers() - * ---> machdep->get_cpu_reg() work properly. + * ---> machdep->get_current_task_reg() work properly. * * These register numbers and names are given according to output of * `rs6000_register_name`, because that is what was being used by @@ -8199,7 +8198,4 @@ enum ppc64_regnum { PPC64_VRSAVE_REGNU = 139 }; -/* crash_target.c */ -extern void gdb_refresh_regcache (unsigned int cpu); - #endif /* !GDB_COMMON */ diff --git a/gdb_interface.c b/gdb_interface.c index 28264c2..2e9117a 100644 --- a/gdb_interface.c +++ b/gdb_interface.c @@ -1073,14 +1073,14 @@ unsigned long crash_get_kaslr_offset(void) } /* Callbacks for crash_target */ -int crash_get_cpu_reg (int cpu, int regno, const char *regname, +int crash_get_current_task_reg (int regno, const char *regname, int regsize, void *val); -int crash_get_cpu_reg (int cpu, int regno, const char *regname, +int crash_get_current_task_reg (int regno, const char *regname, int regsize, void *value) { - if (!machdep->get_cpu_reg) + if (!machdep->get_current_task_reg) return FALSE; - return machdep->get_cpu_reg(cpu, regno, regname, regsize, value); + return machdep->get_current_task_reg(regno, regname, regsize, value); } diff --git a/kernel.c b/kernel.c index 446fd78..15b00dc 100644 --- a/kernel.c +++ b/kernel.c @@ -6543,29 +6543,6 @@ set_cpu(int cpu, int print_context) show_context(CURRENT_CONTEXT()); } -int -crash_set_thread(ulong task) -{ - bool found = FALSE; - struct task_context *tc = FIRST_CONTEXT(); - - for (int i = 0; i < RUNNING_TASKS(); i++, tc++) { - if (tc->task == task) { - found = TRUE; - break; - } - } - - if (!found) - return -1; - - if (CURRENT_TASK() == tc->task) - return 0; - - set_context(tc->task, NO_PID, TRUE); - return 0; -} - /* * Collect the irq_desc[] entry along with its associated handler and * action structures. diff --git a/ppc64.c b/ppc64.c index a87e621..aeccb70 100644 --- a/ppc64.c +++ b/ppc64.c @@ -55,7 +55,7 @@ static void ppc64_set_bt_emergency_stack(enum emergency_stack_type type, static char * ppc64_check_eframe(struct ppc64_pt_regs *); static void ppc64_print_eframe(char *, struct ppc64_pt_regs *, struct bt_info *); -static int ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, +static int ppc64_get_current_task_reg(int regno, const char *name, int size, void *value); static void parse_cmdline_args(void); static int ppc64_paca_percpu_offset_init(int); @@ -706,7 +706,7 @@ ppc64_init(int when) error(FATAL, "cannot malloc hwirqstack buffer space."); } - machdep->get_cpu_reg = ppc64_get_cpu_reg; + machdep->get_current_task_reg = ppc64_get_current_task_reg; ppc64_init_paca_info(); @@ -2506,7 +2506,7 @@ ppc64_print_eframe(char *efrm_str, struct ppc64_pt_regs *regs, } static int -ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, +ppc64_get_current_task_reg(int regno, const char *name, int size, void *value) { struct bt_info bt_info, bt_setup; @@ -2555,7 +2555,7 @@ ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, pt_regs = (struct ppc64_pt_regs *)bt_info.machdep; if (!pt_regs) { - error(WARNING, "pt_regs not available for cpu %d\n", cpu); + error(WARNING, "pt_regs not available for cpu %d\n", tc->processor); return FALSE; } diff --git a/task.c b/task.c index f7e5b05..5f1ddad 100644 --- a/task.c +++ b/task.c @@ -11286,19 +11286,11 @@ check_stack_end_magic: fprintf(fp, "No stack overflows detected\n"); } +void crash_get_current_task_info(unsigned long *pid, char **comm); void crash_get_current_task_info(unsigned long *pid, char **comm) { - int i; - struct task_context *tc; + struct task_context *tc = CURRENT_CONTEXT(); - tc = FIRST_CONTEXT(); - for (i = 0; i < RUNNING_TASKS(); i++, tc++) - if (tc->task == CURRENT_TASK()) { - *pid = tc->pid; - *comm = tc->comm; - return; - } - *pid = 0; - *comm = NULL; - return; + *pid = tc->pid; + *comm = tc->comm; } diff --git a/x86_64.c b/x86_64.c index 41e67c6..73bb353 100644 --- a/x86_64.c +++ b/x86_64.c @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *, ulong, ulong, char *); static void x86_64_framesize_debug(struct bt_info *); static void x86_64_get_active_set(void); static int x86_64_get_kvaddr_ranges(struct vaddr_range *); -static int x86_64_get_cpu_reg(int, int, const char *, int, void *); +static int x86_64_get_current_task_reg(int, const char *, int, void *); static int x86_64_verify_paddr(uint64_t); static void GART_init(void); static void x86_64_exception_stacks_init(void); @@ -207,7 +207,7 @@ x86_64_init(int when) machdep->machspec->irq_eframe_link = UNINITIALIZED; machdep->machspec->irq_stack_gap = UNINITIALIZED; machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges; - machdep->get_cpu_reg = x86_64_get_cpu_reg; + machdep->get_current_task_reg = x86_64_get_current_task_reg; if (machdep->cmdline_args[0]) parse_cmdline_args(); if ((string = pc->read_vmcoreinfo("relocate"))) { @@ -898,7 +898,7 @@ x86_64_dump_machdep_table(ulong arg) fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n"); fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n"); fprintf(fp, " get_kvaddr_ranges: x86_64_get_kvaddr_ranges()\n"); - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n"); + fprintf(fp, "get_current_task_reg: x86_64_get_current_task_reg()\n"); fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n"); fprintf(fp, "clear_machdep_cache: x86_64_clear_machdep_cache()\n"); fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? @@ -9186,7 +9186,7 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp) } static int -x86_64_get_cpu_reg(int cpu, int regno, const char *name, +x86_64_get_current_task_reg(int regno, const char *name, int size, void *value) { struct bt_info bt_info, bt_setup; @@ -9195,8 +9195,6 @@ x86_64_get_cpu_reg(int cpu, int regno, const char *name, ulong ip, sp; bool ret = FALSE; - if (VMSS_DUMPFILE()) - return vmware_vmss_get_cpu_reg(cpu, regno, name, size, value); switch (regno) { case RAX_REGNUM ... GS_REGNUM: case FS_BASE_REGNUM ... ORIG_RAX_REGNUM: @@ -9208,6 +9206,60 @@ x86_64_get_cpu_reg(int cpu, int regno, const char *name, tc = CURRENT_CONTEXT(); if (!tc) return FALSE; + + /* + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15 from + * inactive_task_frame (see __switch_to_asm). Other regs saved on + * regular frame. + */ + if (!is_task_active(tc->task)) { + int frame_size = STRUCT_SIZE("inactive_task_frame"); + + /* Only modern kernels supported. */ + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) { + ulong rsp; + int offset = 0; + switch (regno) { + case RSP_REGNUM: + readmem(tc->task + OFFSET(task_struct_thread) + + OFFSET(thread_struct_rsp), KVADDR, + &rsp, sizeof(void *), + "thread_struct rsp", FAULT_ON_ERROR); + rsp += frame_size; + memcpy(value, &rsp, size); + return TRUE; + case RIP_REGNUM: + offset += 8; + case RBP_REGNUM: + offset += 8; + case RBX_REGNUM: + offset += 8; + case R12_REGNUM: + offset += 8; + case R13_REGNUM: + offset += 8; + case R14_REGNUM: + offset += 8; + case R15_REGNUM: + readmem(tc->task + OFFSET(task_struct_thread) + + OFFSET(thread_struct_rsp), KVADDR, + &rsp, sizeof(void *), + "thread_struct rsp", FAULT_ON_ERROR); + readmem(rsp + offset, KVADDR, value, sizeof(void *), + "inactive_thread_frame saved regs", FAULT_ON_ERROR); + return TRUE; + } + } + /* TBD: older kernels support. */ + return FALSE; + } + + /* + * Task is active, grab CPU's registers + */ + if (VMSS_DUMPFILE()) + return vmware_vmss_get_cpu_reg(tc->processor, regno, name, size, value); + BZERO(&bt_setup, sizeof(struct bt_info)); clone_bt_info(&bt_setup, &bt_info, tc); fill_stackbuf(&bt_info); -- 2.39.0

Hi Alexey, On Fri, Mar 22, 2024 at 7:20 AM Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; wrote: > > Hi folks, please take a look on v2 of this patch and give it a try. > IMHO, Only remaining piece is to clean up x86_64_get_current_task_reg > around getting pt_regs. > > Things done: > 1. Rename machdep->get_cpu_reg to machdep->get_current_task_reg and rename > x86_64_get_cpu_reg to x86_64_get_current_task_reg, plus corresponding changes > for arm64 and ppc64. > 2. Untied gdb's TPID from any tasks or CPUs ID. Always use CURRENT_CONTEXT > for regs and properties fetching. > 3. When CURRENT switched, gdb_change_thread_context() must be called to > invalidate caches. > 4. Update x86_64_get_current_task_reg to support inactive tasks for offline > debugging. > 5. Fixed frame selection issue. It was caused by multiple calls of set_context() > which calls gdb_change_thread_context() unnecessarily dropping gdb caches. > > TODO: > 1. x86_64_get_current_task_reg is questionable about fetching pt_regs for > inactive tasks. How do you gyus get pt_regs structure on modern linux kernels? > PS: I added my approach. More testing required. > > Known issues > 1. After switching from all tasks to one thread: > b. unwinding started to work differently around top (garbaged) frames - Not quite > an issue for me. > > Signed-off-by: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > --- > arm64.c | 4 ++-- > crash_target.c | 48 ++++++++++++++++++------------------- > defs.h | 10 +++----- > gdb_interface.c | 8 +++---- > kernel.c | 23 ------------------ > ppc64.c | 8 +++---- > task.c | 16 ++++--------- > x86_64.c | 64 ++++++++++++++++++++++++++++++++++++++++++++----- > 8 files changed, 99 insertions(+), 82 deletions(-) > > diff --git a/arm64.c b/arm64.c > index 8ff10bf..340db2d 100644 > --- a/arm64.c > +++ b/arm64.c > @@ -153,7 +153,7 @@ static void arm64_calc_kernel_start(void) > } > > static int > -arm64_get_cpu_reg(int cpu, int regno, const char *name, > +arm64_get_current_task_reg(int regno, const char *name, > int size, void *value) > { > struct bt_info bt_info, bt_setup; > @@ -511,7 +511,7 @@ arm64_init(int when) > machdep->dumpfile_init = NULL; > machdep->verify_line_number = NULL; > machdep->init_kernel_pgd = arm64_init_kernel_pgd; > - machdep->get_cpu_reg = arm64_get_cpu_reg; > + machdep->get_current_task_reg = arm64_get_current_task_reg; > > /* use machdep parameters */ > arm64_calc_phys_offset(); > diff --git a/crash_target.c b/crash_target.c > index fb3b634..6e96506 100644 > --- a/crash_target.c > +++ b/crash_target.c > @@ -26,11 +26,8 @@ > void crash_target_init (void); > > extern "C" int gdb_readmem_callback(unsigned long, void *, int, int); > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char *regname, > +extern "C" int crash_get_current_task_reg (int regno, const char *regname, > int regsize, void *val); > -extern "C" void gdb_refresh_regcache(unsigned int cpu); > -extern "C" int set_cpu(int cpu, int print_context); > -extern "C" int crash_set_thread(ulong); > extern "C" int gdb_change_thread_context (ulong); > extern "C" void crash_get_current_task_info(unsigned long *pid, char **comm); > > @@ -72,22 +69,27 @@ public: > > }; > > -/* We just get all the registers, so we don't use regno. */ > void > crash_target::fetch_registers (struct regcache *regcache, int regno) > { > + int r; > gdb_byte regval[16]; > - int cpu = inferior_ptid.tid(); > struct gdbarch *arch = regcache->arch (); > > - for (int r = 0; r < gdbarch_num_regs (arch); r++) > + if (regno >= 0) { > + r = regno; > + goto onetime; > + } > + > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) > { > +onetime: > const char *regname = gdbarch_register_name(arch, r); > int regsize = register_size (arch, r); > if (regsize > sizeof (regval)) > error (_("fatal error: buffer size is not enough to fit register value")); > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)&regval)) > + if (crash_get_current_task_reg (r, regname, regsize, (void *)&regval)) > regcache->raw_supply (r, regval); > else > regcache->raw_supply (r, NULL); > @@ -142,19 +144,17 @@ crash_target_init (void) > extern "C" int > gdb_change_thread_context (ulong task) > { > - inferior* inf = current_inferior(); > - int cpu = crash_set_thread(task); > - if (cpu < 0) > - return FALSE; > - > - ptid_t ptid = ptid_t(CRASH_INFERIOR_PID, 0, cpu); > - thread_info *tp = find_thread_ptid (inf, ptid); > - > - if (tp == nullptr) > - return FALSE; > - > - target_fetch_registers(get_thread_regcache(tp), -1); > - switch_to_thread(tp); > - reinit_frame_cache (); > - return TRUE; > -} > \ No newline at end of file > + static ulong prev_task = 0; > + > + /* > + * Crash calls this method even if CURRENT task is not updated. > + * Ignore it and keep gdb caches active. > + */ > + if (task == prev_task) > + return TRUE; > + prev_task = task; > + > + registers_changed(); > + reinit_frame_cache(); > + return TRUE; > +} > diff --git a/defs.h b/defs.h > index d7e3b18..7596179 100644 > --- a/defs.h > +++ b/defs.h > @@ -1081,7 +1081,7 @@ struct machdep_table { > void (*get_irq_affinity)(int); > void (*show_interrupts)(int, ulong *); > int (*is_page_ptr)(ulong, physaddr_t *); > - int (*get_cpu_reg)(int, int, const char *, int, void *); > + int (*get_current_task_reg)(int, const char *, int, void *); > int (*is_cpu_prstatus_valid)(int cpu); > }; > > @@ -6124,7 +6124,6 @@ void dump_kernel_table(int); > void dump_bt_info(struct bt_info *, char *where); > void dump_log(int); > void parse_kernel_version(char *); > -int crash_set_thread(ulong); > > #define LOG_LEVEL(v) ((v) & 0x07) > #define SHOW_LOG_LEVEL (0x1) > @@ -8023,7 +8022,7 @@ extern int have_full_symbols(void); > > /* > * Register numbers must be in sync with gdb/features/i386/64bit-core.c > - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg() > + * to make crash_target->fetch_registers() ---> machdep->get_current_task_reg() > * working properly. > */ > enum x86_64_regnum { > @@ -8081,7 +8080,7 @@ enum arm64_regnum { > > /* > * Register numbers to make crash_target->fetch_registers() > - * ---> machdep->get_cpu_reg() work properly. > + * ---> machdep->get_current_task_reg() work properly. > * > * These register numbers and names are given according to output of > * `rs6000_register_name`, because that is what was being used by > @@ -8199,7 +8198,4 @@ enum ppc64_regnum { > PPC64_VRSAVE_REGNU = 139 > }; > > -/* crash_target.c */ > -extern void gdb_refresh_regcache (unsigned int cpu); > - > #endif /* !GDB_COMMON */ > diff --git a/gdb_interface.c b/gdb_interface.c > index 28264c2..2e9117a 100644 > --- a/gdb_interface.c > +++ b/gdb_interface.c > @@ -1073,14 +1073,14 @@ unsigned long crash_get_kaslr_offset(void) > } > > /* Callbacks for crash_target */ > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > +int crash_get_current_task_reg (int regno, const char *regname, > int regsize, void *val); > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > +int crash_get_current_task_reg (int regno, const char *regname, > int regsize, void *value) > { > - if (!machdep->get_cpu_reg) > + if (!machdep->get_current_task_reg) > return FALSE; > - return machdep->get_cpu_reg(cpu, regno, regname, regsize, value); > + return machdep->get_current_task_reg(regno, regname, regsize, value); > } > > diff --git a/kernel.c b/kernel.c > index 446fd78..15b00dc 100644 > --- a/kernel.c > +++ b/kernel.c > @@ -6543,29 +6543,6 @@ set_cpu(int cpu, int print_context) > show_context(CURRENT_CONTEXT()); > } > > -int > -crash_set_thread(ulong task) > -{ > - bool found = FALSE; > - struct task_context *tc = FIRST_CONTEXT(); > - > - for (int i = 0; i < RUNNING_TASKS(); i++, tc++) { > - if (tc->task == task) { > - found = TRUE; > - break; > - } > - } > - > - if (!found) > - return -1; > - > - if (CURRENT_TASK() == tc->task) > - return 0; > - > - set_context(tc->task, NO_PID, TRUE); > - return 0; > -} > - > /* > * Collect the irq_desc[] entry along with its associated handler and > * action structures. > diff --git a/ppc64.c b/ppc64.c > index a87e621..aeccb70 100644 > --- a/ppc64.c > +++ b/ppc64.c > @@ -55,7 +55,7 @@ static void ppc64_set_bt_emergency_stack(enum emergency_stack_type type, > static char * ppc64_check_eframe(struct ppc64_pt_regs *); > static void ppc64_print_eframe(char *, struct ppc64_pt_regs *, > struct bt_info *); > -static int ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, > +static int ppc64_get_current_task_reg(int regno, const char *name, int size, > void *value); > static void parse_cmdline_args(void); > static int ppc64_paca_percpu_offset_init(int); > @@ -706,7 +706,7 @@ ppc64_init(int when) > error(FATAL, "cannot malloc hwirqstack buffer space."); > } > > - machdep->get_cpu_reg = ppc64_get_cpu_reg; > + machdep->get_current_task_reg = ppc64_get_current_task_reg; > > ppc64_init_paca_info(); > > @@ -2506,7 +2506,7 @@ ppc64_print_eframe(char *efrm_str, struct ppc64_pt_regs *regs, > } > > static int > -ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, > +ppc64_get_current_task_reg(int regno, const char *name, int size, > void *value) > { > struct bt_info bt_info, bt_setup; > @@ -2555,7 +2555,7 @@ ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, > pt_regs = (struct ppc64_pt_regs *)bt_info.machdep; > > if (!pt_regs) { > - error(WARNING, "pt_regs not available for cpu %d\n", cpu); > + error(WARNING, "pt_regs not available for cpu %d\n", tc->processor); > return FALSE; > } > > diff --git a/task.c b/task.c > index f7e5b05..5f1ddad 100644 > --- a/task.c > +++ b/task.c > @@ -11286,19 +11286,11 @@ check_stack_end_magic: > fprintf(fp, "No stack overflows detected\n"); > } > > +void crash_get_current_task_info(unsigned long *pid, char **comm); > void crash_get_current_task_info(unsigned long *pid, char **comm) > { > - int i; > - struct task_context *tc; > + struct task_context *tc = CURRENT_CONTEXT(); > > - tc = FIRST_CONTEXT(); > - for (i = 0; i < RUNNING_TASKS(); i++, tc++) > - if (tc->task == CURRENT_TASK()) { > - *pid = tc->pid; > - *comm = tc->comm; > - return; > - } > - *pid = 0; > - *comm = NULL; > - return; > + *pid = tc->pid; > + *comm = tc->comm; > } > diff --git a/x86_64.c b/x86_64.c > index 41e67c6..73bb353 100644 > --- a/x86_64.c > +++ b/x86_64.c > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *, ulong, ulong, char *); > static void x86_64_framesize_debug(struct bt_info *); > static void x86_64_get_active_set(void); > static int x86_64_get_kvaddr_ranges(struct vaddr_range *); > -static int x86_64_get_cpu_reg(int, int, const char *, int, void *); > +static int x86_64_get_current_task_reg(int, const char *, int, void *); > static int x86_64_verify_paddr(uint64_t); > static void GART_init(void); > static void x86_64_exception_stacks_init(void); > @@ -207,7 +207,7 @@ x86_64_init(int when) > machdep->machspec->irq_eframe_link = UNINITIALIZED; > machdep->machspec->irq_stack_gap = UNINITIALIZED; > machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges; > - machdep->get_cpu_reg = x86_64_get_cpu_reg; > + machdep->get_current_task_reg = x86_64_get_current_task_reg; > if (machdep->cmdline_args[0]) > parse_cmdline_args(); > if ((string = pc->read_vmcoreinfo("relocate"))) { > @@ -898,7 +898,7 @@ x86_64_dump_machdep_table(ulong arg) > fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n"); > fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n"); > fprintf(fp, " get_kvaddr_ranges: x86_64_get_kvaddr_ranges()\n"); > - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n"); > + fprintf(fp, "get_current_task_reg: x86_64_get_current_task_reg()\n"); > fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n"); > fprintf(fp, "clear_machdep_cache: x86_64_clear_machdep_cache()\n"); > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? > @@ -9186,7 +9186,7 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp) > } > > static int > -x86_64_get_cpu_reg(int cpu, int regno, const char *name, > +x86_64_get_current_task_reg(int regno, const char *name, > int size, void *value) > { > struct bt_info bt_info, bt_setup; > @@ -9195,8 +9195,6 @@ x86_64_get_cpu_reg(int cpu, int regno, const char *name, > ulong ip, sp; > bool ret = FALSE; > > - if (VMSS_DUMPFILE()) > - return vmware_vmss_get_cpu_reg(cpu, regno, name, size, value); > switch (regno) { > case RAX_REGNUM ... GS_REGNUM: > case FS_BASE_REGNUM ... ORIG_RAX_REGNUM: > @@ -9208,6 +9206,60 @@ x86_64_get_cpu_reg(int cpu, int regno, const char *name, > tc = CURRENT_CONTEXT(); > if (!tc) > return FALSE; > + > + /* > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15 from > + * inactive_task_frame (see __switch_to_asm). Other regs saved on > + * regular frame. > + */ > + if (!is_task_active(tc->task)) { > + int frame_size = STRUCT_SIZE("inactive_task_frame"); > + > + /* Only modern kernels supported. */ > + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) { > + ulong rsp; > + int offset = 0; > + switch (regno) { > + case RSP_REGNUM: > + readmem(tc->task + OFFSET(task_struct_thread) + > + OFFSET(thread_struct_rsp), KVADDR, > + &rsp, sizeof(void *), > + "thread_struct rsp", FAULT_ON_ERROR); > + rsp += frame_size; > + memcpy(value, &rsp, size); > + return TRUE; > + case RIP_REGNUM: > + offset += 8; > + case RBP_REGNUM: > + offset += 8; > + case RBX_REGNUM: > + offset += 8; > + case R12_REGNUM: > + offset += 8; > + case R13_REGNUM: > + offset += 8; > + case R14_REGNUM: > + offset += 8; > + case R15_REGNUM: > + readmem(tc->task + OFFSET(task_struct_thread) + > + OFFSET(thread_struct_rsp), KVADDR, > + &rsp, sizeof(void *), > + "thread_struct rsp", FAULT_ON_ERROR); > + readmem(rsp + offset, KVADDR, value, sizeof(void *), > + "inactive_thread_frame saved regs", FAULT_ON_ERROR); > + return TRUE; > + } For this part, it looks similar to what x86_64.c:x86_64_get_stack_frame():5016 is trying to do, which is extracting reg values from stack. Is there a special consideration to put it in here? IMHO, the x86_64_get_stack_frame() is responsible for extracting reg values from stack frame or elf notes(aka the regs value reproducer). And it will give a universal data structure, aka pt_regs, to x86_64_get_current_task_reg() (aka the regs value consumer). In this design, the x86_64_get_current_task_reg() is decoupled from stack, no matter inactive_task_frame or traditional linked list stack frame or active tasks regs coming from elf notes, the consumer won't care, all it need is a pt_regs structure and get values from it. Personally I prefer this design. Any comments? For other parts of the patch, I'm still reviewing and testing. Thanks, Tao Liu > + } > + /* TBD: older kernels support. */ > + return FALSE; > + } > + > + /* > + * Task is active, grab CPU's registers > + */ > + if (VMSS_DUMPFILE()) > + return vmware_vmss_get_cpu_reg(tc->processor, regno, name, size, value); > + > BZERO(&bt_setup, sizeof(struct bt_info)); > clone_bt_info(&bt_setup, &bt_info, tc); > fill_stackbuf(&bt_info); > -- > 2.39.0 >

Hi Tao, I need to see if there is any existing code in crash which is doing it. I would like to avoid wheel reinvention. For just stack unwinding, IP/SP is enough. But to see function arguments and local variables especially lower frames, gdb needs to know other GPRs. inactive_task_frame does not use pt_regs layout for saved registers and from the latest Linux sources __switch_to_asm() does not provide dwarf/debug annotations regarding where the registers were saved. That's why I ended up with an inactive_task_frame parsing implementation. About other kernel versions, I need to see how it has evolved over time. AFAIR, at some point in the past it was pt_regs based. Thanks, --Alexey On Tue, Mar 26, 2024 at 2:14 AM Tao Liu <ltao(a)redhat.com&gt; wrote: > > Hi Alexey, > > For the rest of the patch, I didn't see any problem. But since it only > supports modern kernels, could you please implement the rest, so I can > have a testing for variety versions of kernels(ranging from 2.6.x to > 5.x), in order to find if there are further problems. > > Thanks, > Tao Liu > > On Mon, Mar 25, 2024 at 11:34 AM Tao Liu <ltao(a)redhat.com&gt; wrote: > > > > Hi Alexey, > > > > On Fri, Mar 22, 2024 at 7:20 AM Alexey Makhalov > > <alexey.makhalov(a)broadcom.com&gt; wrote: > > > > > > Hi folks, please take a look on v2 of this patch and give it a try. > > > IMHO, Only remaining piece is to clean up x86_64_get_current_task_reg > > > around getting pt_regs. > > > > > > Things done: > > > 1. Rename machdep->get_cpu_reg to machdep->get_current_task_reg and rename > > > x86_64_get_cpu_reg to x86_64_get_current_task_reg, plus corresponding changes > > > for arm64 and ppc64. > > > 2. Untied gdb's TPID from any tasks or CPUs ID. Always use CURRENT_CONTEXT > > > for regs and properties fetching. > > > 3. When CURRENT switched, gdb_change_thread_context() must be called to > > > invalidate caches. > > > 4. Update x86_64_get_current_task_reg to support inactive tasks for offline > > > debugging. > > > 5. Fixed frame selection issue. It was caused by multiple calls of set_context() > > > which calls gdb_change_thread_context() unnecessarily dropping gdb caches. > > > > > > TODO: > > > 1. x86_64_get_current_task_reg is questionable about fetching pt_regs for > > > inactive tasks. How do you gyus get pt_regs structure on modern linux kernels? > > > PS: I added my approach. More testing required. > > > > > > Known issues > > > 1. After switching from all tasks to one thread: > > > b. unwinding started to work differently around top (garbaged) frames - Not quite > > > an issue for me. > > > > > > Signed-off-by: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > > > --- > > > arm64.c | 4 ++-- > > > crash_target.c | 48 ++++++++++++++++++------------------- > > > defs.h | 10 +++----- > > > gdb_interface.c | 8 +++---- > > > kernel.c | 23 ------------------ > > > ppc64.c | 8 +++---- > > > task.c | 16 ++++--------- > > > x86_64.c | 64 ++++++++++++++++++++++++++++++++++++++++++++----- > > > 8 files changed, 99 insertions(+), 82 deletions(-) > > > > > > diff --git a/arm64.c b/arm64.c > > > index 8ff10bf..340db2d 100644 > > > --- a/arm64.c > > > +++ b/arm64.c > > > @@ -153,7 +153,7 @@ static void arm64_calc_kernel_start(void) > > > } > > > > > > static int > > > -arm64_get_cpu_reg(int cpu, int regno, const char *name, > > > +arm64_get_current_task_reg(int regno, const char *name, > > > int size, void *value) > > > { > > > struct bt_info bt_info, bt_setup; > > > @@ -511,7 +511,7 @@ arm64_init(int when) > > > machdep->dumpfile_init = NULL; > > > machdep->verify_line_number = NULL; > > > machdep->init_kernel_pgd = arm64_init_kernel_pgd; > > > - machdep->get_cpu_reg = arm64_get_cpu_reg; > > > + machdep->get_current_task_reg = arm64_get_current_task_reg; > > > > > > /* use machdep parameters */ > > > arm64_calc_phys_offset(); > > > diff --git a/crash_target.c b/crash_target.c > > > index fb3b634..6e96506 100644 > > > --- a/crash_target.c > > > +++ b/crash_target.c > > > @@ -26,11 +26,8 @@ > > > void crash_target_init (void); > > > > > > extern "C" int gdb_readmem_callback(unsigned long, void *, int, int); > > > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > +extern "C" int crash_get_current_task_reg (int regno, const char *regname, > > > int regsize, void *val); > > > -extern "C" void gdb_refresh_regcache(unsigned int cpu); > > > -extern "C" int set_cpu(int cpu, int print_context); > > > -extern "C" int crash_set_thread(ulong); > > > extern "C" int gdb_change_thread_context (ulong); > > > extern "C" void crash_get_current_task_info(unsigned long *pid, char **comm); > > > > > > @@ -72,22 +69,27 @@ public: > > > > > > }; > > > > > > -/* We just get all the registers, so we don't use regno. */ > > > void > > > crash_target::fetch_registers (struct regcache *regcache, int regno) > > > { > > > + int r; > > > gdb_byte regval[16]; > > > - int cpu = inferior_ptid.tid(); > > > struct gdbarch *arch = regcache->arch (); > > > > > > - for (int r = 0; r < gdbarch_num_regs (arch); r++) > > > + if (regno >= 0) { > > > + r = regno; > > > + goto onetime; > > > + } > > > + > > > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) > > > { > > > +onetime: > > > const char *regname = gdbarch_register_name(arch, r); > > > int regsize = register_size (arch, r); > > > if (regsize > sizeof (regval)) > > > error (_("fatal error: buffer size is not enough to fit register value")); > > > > > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)&regval)) > > > + if (crash_get_current_task_reg (r, regname, regsize, (void *)&regval)) > > > regcache->raw_supply (r, regval); > > > else > > > regcache->raw_supply (r, NULL); > > > @@ -142,19 +144,17 @@ crash_target_init (void) > > > extern "C" int > > > gdb_change_thread_context (ulong task) > > > { > > > - inferior* inf = current_inferior(); > > > - int cpu = crash_set_thread(task); > > > - if (cpu < 0) > > > - return FALSE; > > > - > > > - ptid_t ptid = ptid_t(CRASH_INFERIOR_PID, 0, cpu); > > > - thread_info *tp = find_thread_ptid (inf, ptid); > > > - > > > - if (tp == nullptr) > > > - return FALSE; > > > - > > > - target_fetch_registers(get_thread_regcache(tp), -1); > > > - switch_to_thread(tp); > > > - reinit_frame_cache (); > > > - return TRUE; > > > -} > > > \ No newline at end of file > > > + static ulong prev_task = 0; > > > + > > > + /* > > > + * Crash calls this method even if CURRENT task is not updated. > > > + * Ignore it and keep gdb caches active. > > > + */ > > > + if (task == prev_task) > > > + return TRUE; > > > + prev_task = task; > > > + > > > + registers_changed(); > > > + reinit_frame_cache(); > > > + return TRUE; > > > +} > > > diff --git a/defs.h b/defs.h > > > index d7e3b18..7596179 100644 > > > --- a/defs.h > > > +++ b/defs.h > > > @@ -1081,7 +1081,7 @@ struct machdep_table { > > > void (*get_irq_affinity)(int); > > > void (*show_interrupts)(int, ulong *); > > > int (*is_page_ptr)(ulong, physaddr_t *); > > > - int (*get_cpu_reg)(int, int, const char *, int, void *); > > > + int (*get_current_task_reg)(int, const char *, int, void *); > > > int (*is_cpu_prstatus_valid)(int cpu); > > > }; > > > > > > @@ -6124,7 +6124,6 @@ void dump_kernel_table(int); > > > void dump_bt_info(struct bt_info *, char *where); > > > void dump_log(int); > > > void parse_kernel_version(char *); > > > -int crash_set_thread(ulong); > > > > > > #define LOG_LEVEL(v) ((v) & 0x07) > > > #define SHOW_LOG_LEVEL (0x1) > > > @@ -8023,7 +8022,7 @@ extern int have_full_symbols(void); > > > > > > /* > > > * Register numbers must be in sync with gdb/features/i386/64bit-core.c > > > - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg() > > > + * to make crash_target->fetch_registers() ---> machdep->get_current_task_reg() > > > * working properly. > > > */ > > > enum x86_64_regnum { > > > @@ -8081,7 +8080,7 @@ enum arm64_regnum { > > > > > > /* > > > * Register numbers to make crash_target->fetch_registers() > > > - * ---> machdep->get_cpu_reg() work properly. > > > + * ---> machdep->get_current_task_reg() work properly. > > > * > > > * These register numbers and names are given according to output of > > > * `rs6000_register_name`, because that is what was being used by > > > @@ -8199,7 +8198,4 @@ enum ppc64_regnum { > > > PPC64_VRSAVE_REGNU = 139 > > > }; > > > > > > -/* crash_target.c */ > > > -extern void gdb_refresh_regcache (unsigned int cpu); > > > - > > > #endif /* !GDB_COMMON */ > > > diff --git a/gdb_interface.c b/gdb_interface.c > > > index 28264c2..2e9117a 100644 > > > --- a/gdb_interface.c > > > +++ b/gdb_interface.c > > > @@ -1073,14 +1073,14 @@ unsigned long crash_get_kaslr_offset(void) > > > } > > > > > > /* Callbacks for crash_target */ > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > +int crash_get_current_task_reg (int regno, const char *regname, > > > int regsize, void *val); > > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > +int crash_get_current_task_reg (int regno, const char *regname, > > > int regsize, void *value) > > > { > > > - if (!machdep->get_cpu_reg) > > > + if (!machdep->get_current_task_reg) > > > return FALSE; > > > - return machdep->get_cpu_reg(cpu, regno, regname, regsize, value); > > > + return machdep->get_current_task_reg(regno, regname, regsize, value); > > > } > > > > > > diff --git a/kernel.c b/kernel.c > > > index 446fd78..15b00dc 100644 > > > --- a/kernel.c > > > +++ b/kernel.c > > > @@ -6543,29 +6543,6 @@ set_cpu(int cpu, int print_context) > > > show_context(CURRENT_CONTEXT()); > > > } > > > > > > -int > > > -crash_set_thread(ulong task) > > > -{ > > > - bool found = FALSE; > > > - struct task_context *tc = FIRST_CONTEXT(); > > > - > > > - for (int i = 0; i < RUNNING_TASKS(); i++, tc++) { > > > - if (tc->task == task) { > > > - found = TRUE; > > > - break; > > > - } > > > - } > > > - > > > - if (!found) > > > - return -1; > > > - > > > - if (CURRENT_TASK() == tc->task) > > > - return 0; > > > - > > > - set_context(tc->task, NO_PID, TRUE); > > > - return 0; > > > -} > > > - > > > /* > > > * Collect the irq_desc[] entry along with its associated handler and > > > * action structures. > > > diff --git a/ppc64.c b/ppc64.c > > > index a87e621..aeccb70 100644 > > > --- a/ppc64.c > > > +++ b/ppc64.c > > > @@ -55,7 +55,7 @@ static void ppc64_set_bt_emergency_stack(enum emergency_stack_type type, > > > static char * ppc64_check_eframe(struct ppc64_pt_regs *); > > > static void ppc64_print_eframe(char *, struct ppc64_pt_regs *, > > > struct bt_info *); > > > -static int ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, > > > +static int ppc64_get_current_task_reg(int regno, const char *name, int size, > > > void *value); > > > static void parse_cmdline_args(void); > > > static int ppc64_paca_percpu_offset_init(int); > > > @@ -706,7 +706,7 @@ ppc64_init(int when) > > > error(FATAL, "cannot malloc hwirqstack buffer space."); > > > } > > > > > > - machdep->get_cpu_reg = ppc64_get_cpu_reg; > > > + machdep->get_current_task_reg = ppc64_get_current_task_reg; > > > > > > ppc64_init_paca_info(); > > > > > > @@ -2506,7 +2506,7 @@ ppc64_print_eframe(char *efrm_str, struct ppc64_pt_regs *regs, > > > } > > > > > > static int > > > -ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, > > > +ppc64_get_current_task_reg(int regno, const char *name, int size, > > > void *value) > > > { > > > struct bt_info bt_info, bt_setup; > > > @@ -2555,7 +2555,7 @@ ppc64_get_cpu_reg(int cpu, int regno, const char *name, int size, > > > pt_regs = (struct ppc64_pt_regs *)bt_info.machdep; > > > > > > if (!pt_regs) { > > > - error(WARNING, "pt_regs not available for cpu %d\n", cpu); > > > + error(WARNING, "pt_regs not available for cpu %d\n", tc->processor); > > > return FALSE; > > > } > > > > > > diff --git a/task.c b/task.c > > > index f7e5b05..5f1ddad 100644 > > > --- a/task.c > > > +++ b/task.c > > > @@ -11286,19 +11286,11 @@ check_stack_end_magic: > > > fprintf(fp, "No stack overflows detected\n"); > > > } > > > > > > +void crash_get_current_task_info(unsigned long *pid, char **comm); > > > void crash_get_current_task_info(unsigned long *pid, char **comm) > > > { > > > - int i; > > > - struct task_context *tc; > > > + struct task_context *tc = CURRENT_CONTEXT(); > > > > > > - tc = FIRST_CONTEXT(); > > > - for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > - if (tc->task == CURRENT_TASK()) { > > > - *pid = tc->pid; > > > - *comm = tc->comm; > > > - return; > > > - } > > > - *pid = 0; > > > - *comm = NULL; > > > - return; > > > + *pid = tc->pid; > > > + *comm = tc->comm; > > > } > > > diff --git a/x86_64.c b/x86_64.c > > > index 41e67c6..73bb353 100644 > > > --- a/x86_64.c > > > +++ b/x86_64.c > > > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *, ulong, ulong, char *); > > > static void x86_64_framesize_debug(struct bt_info *); > > > static void x86_64_get_active_set(void); > > > static int x86_64_get_kvaddr_ranges(struct vaddr_range *); > > > -static int x86_64_get_cpu_reg(int, int, const char *, int, void *); > > > +static int x86_64_get_current_task_reg(int, const char *, int, void *); > > > static int x86_64_verify_paddr(uint64_t); > > > static void GART_init(void); > > > static void x86_64_exception_stacks_init(void); > > > @@ -207,7 +207,7 @@ x86_64_init(int when) > > > machdep->machspec->irq_eframe_link = UNINITIALIZED; > > > machdep->machspec->irq_stack_gap = UNINITIALIZED; > > > machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges; > > > - machdep->get_cpu_reg = x86_64_get_cpu_reg; > > > + machdep->get_current_task_reg = x86_64_get_current_task_reg; > > > if (machdep->cmdline_args[0]) > > > parse_cmdline_args(); > > > if ((string = pc->read_vmcoreinfo("relocate"))) { > > > @@ -898,7 +898,7 @@ x86_64_dump_machdep_table(ulong arg) > > > fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n"); > > > fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n"); > > > fprintf(fp, " get_kvaddr_ranges: x86_64_get_kvaddr_ranges()\n"); > > > - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n"); > > > + fprintf(fp, "get_current_task_reg: x86_64_get_current_task_reg()\n"); > > > fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n"); > > > fprintf(fp, "clear_machdep_cache: x86_64_clear_machdep_cache()\n"); > > > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? > > > @@ -9186,7 +9186,7 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp) > > > } > > > > > > static int > > > -x86_64_get_cpu_reg(int cpu, int regno, const char *name, > > > +x86_64_get_current_task_reg(int regno, const char *name, > > > int size, void *value) > > > { > > > struct bt_info bt_info, bt_setup; > > > @@ -9195,8 +9195,6 @@ x86_64_get_cpu_reg(int cpu, int regno, const char *name, > > > ulong ip, sp; > > > bool ret = FALSE; > > > > > > - if (VMSS_DUMPFILE()) > > > - return vmware_vmss_get_cpu_reg(cpu, regno, name, size, value); > > > switch (regno) { > > > case RAX_REGNUM ... GS_REGNUM: > > > case FS_BASE_REGNUM ... ORIG_RAX_REGNUM: > > > @@ -9208,6 +9206,60 @@ x86_64_get_cpu_reg(int cpu, int regno, const char *name, > > > tc = CURRENT_CONTEXT(); > > > if (!tc) > > > return FALSE; > > > + > > > + /* > > > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15 from > > > + * inactive_task_frame (see __switch_to_asm). Other regs saved on > > > + * regular frame. > > > + */ > > > + if (!is_task_active(tc->task)) { > > > + int frame_size = STRUCT_SIZE("inactive_task_frame"); > > > + > > > + /* Only modern kernels supported. */ > > > + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) { > > > + ulong rsp; > > > + int offset = 0; > > > + switch (regno) { > > > + case RSP_REGNUM: > > > + readmem(tc->task + OFFSET(task_struct_thread) + > > > + OFFSET(thread_struct_rsp), KVADDR, > > > + &rsp, sizeof(void *), > > > + "thread_struct rsp", FAULT_ON_ERROR); > > > + rsp += frame_size; > > > + memcpy(value, &rsp, size); > > > + return TRUE; > > > + case RIP_REGNUM: > > > + offset += 8; > > > + case RBP_REGNUM: > > > + offset += 8; > > > + case RBX_REGNUM: > > > + offset += 8; > > > + case R12_REGNUM: > > > + offset += 8; > > > + case R13_REGNUM: > > > + offset += 8; > > > + case R14_REGNUM: > > > + offset += 8; > > > + case R15_REGNUM: > > > + readmem(tc->task + OFFSET(task_struct_thread) + > > > + OFFSET(thread_struct_rsp), KVADDR, > > > + &rsp, sizeof(void *), > > > + "thread_struct rsp", FAULT_ON_ERROR); > > > + readmem(rsp + offset, KVADDR, value, sizeof(void *), > > > + "inactive_thread_frame saved regs", FAULT_ON_ERROR); > > > + return TRUE; > > > + } > > > > For this part, it looks similar to what > > x86_64.c:x86_64_get_stack_frame():5016 is trying to do, which is > > extracting reg values from stack. Is there a special consideration to > > put it in here? > > > > IMHO, the x86_64_get_stack_frame() is responsible for extracting reg > > values from stack frame or elf notes(aka the regs value reproducer). > > And it will give a universal data structure, aka pt_regs, to > > x86_64_get_current_task_reg() (aka the regs value consumer). > > > > In this design, the x86_64_get_current_task_reg() is decoupled from > > stack, no matter inactive_task_frame or traditional linked list stack > > frame or active tasks regs coming from elf notes, the consumer won't > > care, all it need is a pt_regs structure and get values from it. > > Personally I prefer this design. Any comments? > > > > For other parts of the patch, I'm still reviewing and testing. > > > > Thanks, > > Tao Liu > > > > > + } > > > + /* TBD: older kernels support. */ > > > + return FALSE; > > > + } > > > + > > > + /* > > > + * Task is active, grab CPU's registers > > > + */ > > > + if (VMSS_DUMPFILE()) > > > + return vmware_vmss_get_cpu_reg(tc->processor, regno, name, size, value); > > > + > > > BZERO(&bt_setup, sizeof(struct bt_info)); > > > clone_bt_info(&bt_setup, &bt_info, tc); > > > fill_stackbuf(&bt_info); > > > -- > > > 2.39.0 > > > >

Please have a test or review, any comments would be nice. [1]: https://github.com/liutgnu/crash-dev/tree/tao-rebase-v2 [2]: https://github.com/adi-g15-ibm/crash/tree/stack-unwind-one-thread-merged

Hi Aditya & Alexey, On Mon, Apr 1, 2024 at 4:39 PM Aditya Gupta <adityag(a)linux.ibm.com&gt; wrote: > > Hi Tao & Alexey, > > > Please have a test or review, any comments would be nice. > > > > [1]: https://github.com/liutgnu/crash-dev/tree/tao-rebase-v2 > > [2]: https://github.com/adi-g15-ibm/crash/tree/stack-unwind-one-thread-merged > > Works nicely for me. nitpick in commit 'Let crash change gdb context': > Thanks for the comment, I get it updated. Please see v3 at: https://github.com/liutgnu/crash-dev/tree/tao-rebase-v3

Hi Tao, > From my view, I think we can keep the current "gdb thread/CPUs = nr_cpus", or > be more ambitious, leaving "gdb thread/CPUs = 1", in case there are > systems with thousands of cpus, and we only need this in task.c: Nice idea, it might break some expectations that 'info threads' list active CPUs, but I guess that was anyways not of much help earlier. Also simplifies many things, and removes much of the synchronisations between gdb <-> crash. If this happens, I guess we should just make 'info threads' behave as 'thread', ie. just showing current frame, as "CPU 0" might not make sense.

The idea seems nice to me too. Amazing. :)

> > + gdb_refresh_regcache(0); > - for (int i = 0; i < get_cpus_online(); i++) > - gdb_refresh_regcache(i); Also, with Alexey's change of moving 'crash_target_init' to a later time, even 'gdb_refresh_regcache(0)' might not be needed. It was needed earlier due to crash_target_init being called before any architecture gets a chance to set 'machdep->get_cpu_reg', and hence initialising registers for CPU 0 had failed, 'gdb_refresh_regcache(0)' just caused it to fetch registers again, after architectures have set 'machdep->get_cpu_reg'.

Haven't tested this yet. Thanks, Aditya Gupta > > Do you have any thoughts? > > [1]: https://github.com/liutgnu/crash-dev/commits/one-thread > > Thanks, > Tao Liu > > > Thanks, > > Tao Liu > > > > > > > > > > I can think of solving it in two ways: > > > 1. Having a array in crash_target, which tells what CPU in gdb maps to > > > which task in crash, this can be done without requiring changes to 'info > > > threads', while we do the 'add_thread_silent' > > > 2. Having all tasks in gdb itself, so it's a one-to-one, so `set_cpu` > > > will then refer to the task ID, I will have to explore more on this. > > > > > > Any comments ? > > > > > > Thanks, > > > Aditya Gupta > > > > > > > > > > > Thanks, > > > > --Alexey > > > > > > > > > > > > On Thu, Mar 14, 2024 at 9:22 PM Tao Liu <ltao(a)redhat.com&gt; wrote: > > > > > > > > > Hi Alexey, > > > > > > > > > > On Thu, Mar 14, 2024 at 6:29 PM Alexey Makhalov > > > > > <alexey.makhalov(a)broadcom.com&gt; wrote: > > > > > > > > > > > > Support for GDB debugging of all tasks active and inactive. > > > > > > Before this commit only active tasks were listed by "info threads" > > > > > > with "CPU #" as a Target Id. > > > > > > > > > > > > "info threads" will now show all tasks, similar to "ps", example: > > > > > > crash> info threads > > > > > > Id Target Id Frame > > > > > > * 1 0 swapper/0 0xffffffffadba19d4 in default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 2 0 swapper/1 0xffffffffadba19d4 in default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 3 0 swapper/2 0xffffffffadba19d4 in default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 4 0 swapper/3 0xffffffffadba19d4 in default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 5 0 swapper/4 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0000f3e88, next=0xffff9ecb04908000, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > ... > > > > > > 730 970325 taskset 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf006a0fd18, next=0xffff9ecb0aec0000, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 731 975217 sleep 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf005743c20, next=0xffff9ecac0692880, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 732 975228 sleep 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf00696fb58, next=0xffff9ecac0690000, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > ... > > > > > > 876 976084 docker 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0153dbd10, next=0xffff9ecac0645100, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 877 976085 systemd-userwor 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0153cbc58, next=0xffff9ecac0645100, prev=<optimized out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 878 976086 systemd-userwor 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0153e3c58, next=0xffffffffaec15a40 <init_task>, > > > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 879 976087 systemd-userwor 0xffffffffadb97292 in context_switch > > > > > (rf=0xffffbaf0153ebc58, next=0xffffffffaec15a40 <init_task>, > > > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > Where "Target ID" contains "PID COMM" of the task > > > > > > > > > > > > > > > > I see here, we chose a different tech path of viewing arbitrary tasks. > > > > > You improved the "info threads" as an alternative to crash "ps" cmd, > > > > > and use "thread X" to switch to task X and view its stacktrace. > > > > > However mine is using crash "ps" cmd first, getting specific tasks pid > > > > > or task struct, then using crash cmd "set <pid>" or "set > > > > > <task_struct>" to switch to task X and view its stacktrace. > > > > > > > > > > I notice there is one problem of using the "info threads as ps" tech > > > > > path, it is difficult to handle the live debug, see the comment below: > > > > > > > > > > > Example of "731 975217 sleep" debugging, real case, trying to > > > > > > figure out why sleep was stuck in uninterruptable sleep. > > > > > > Backtrace using crash: > > > > > > crash> ps | grep 975217 > > > > > > 975217 969797 3 ffff9ecb3956a880 UN 0.0 0 0 > > > > > sleep > > > > > > crash> bt 975217 > > > > > > PID: 975217 TASK: ffff9ecb3956a880 CPU: 3 COMMAND: "sleep" > > > > > > #0 [ffffbaf005743ba0] __schedule at ffffffffadb97292 > > > > > > #1 [ffffbaf005743c60] schedule at ffffffffadb982b8 > > > > > > #2 [ffffbaf005743c80] rwbase_write_lock at ffffffffadb9aed7 > > > > > > #3 [ffffbaf005743cc0] down_write at ffffffffadb9b133 > > > > > > #4 [ffffbaf005743cd0] unlink_file_vma at ffffffffad2b0e2e > > > > > > #5 [ffffbaf005743cf8] free_pgtables at ffffffffad2a47b0 > > > > > > #6 [ffffbaf005743d88] exit_mmap at ffffffffad2b3b8d > > > > > > #7 [ffffbaf005743e80] mmput at ffffffffad08c81f > > > > > > #8 [ffffbaf005743e98] do_exit at ffffffffad09636c > > > > > > #9 [ffffbaf005743ef8] do_group_exit at ffffffffad096c78 > > > > > > RIP: 00007f111c70ddf9 RSP: 00007fff451817e8 RFLAGS: 00000246 > > > > > > RAX: ffffffffffffffda RBX: 00007f111c8089e0 RCX: 00007f111c70ddf9 > > > > > > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 > > > > > > RBP: 0000000000000000 R8: ffffffffffffff80 R9: 0000000000000000 > > > > > > R10: 00007fff451817b0 R11: 0000000000000246 R12: 00007f111c8089e0 > > > > > > R13: 00007f111c80e2e0 R14: 0000000000000002 R15: 00007f111c80e2c8 > > > > > > ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b > > > > > > > > > > > > Backtrace using gdb (pay attention, task must be selected by thread Id): > > > > > > crash> thread 731 > > > > > > [Switching to thread 731 ( 975217 sleep)] > > > > > > 5372 switch_to(prev, next, prev); > > > > > > crash> gdb bt > > > > > > #0 0xffffffffadb97292 in context_switch (rf=0xffffbaf005743c20, > > > > > next=0xffff9ecac0692880, prev=<optimized out>, rq=<optimized out>) at > > > > > kernel/sched/core.c:5372 > > > > > > #1 __schedule (sched_mode=sched_mode@entry=0) at > > > > > kernel/sched/core.c:6696 > > > > > > #2 0xffffffffadb982b8 in schedule () at kernel/sched/core.c:6772 > > > > > > #3 0xffffffffadb9aed7 in rwbase_write_lock (rwb=rwb@entry=0xffff9ecaf1831430, > > > > > state=state@entry=2) at kernel/locking/rwbase_rt.c:259 > > > > > > #4 0xffffffffadb9b133 in __down_write (sem=sem@entry=0xffff9ecaf1831430) > > > > > at kernel/locking/rwsem.c:1474 > > > > > > #5 down_write (sem=sem@entry=0xffff9ecaf1831430) at > > > > > kernel/locking/rwsem.c:1574 > > > > > > #6 0xffffffffad2b0e2e in i_mmap_lock_write (mapping=<optimized out>) > > > > > at ./include/linux/fs.h:466 > > > > > > #7 unlink_file_vma (vma=vma@entry=0xffff9ecadd566090) at mm/mmap.c:127 > > > > > > #8 0xffffffffad2a47b0 in free_pgtables (tlb=tlb@entry=0xffffbaf005743dd0, > > > > > mt=mt@entry=0xffff9ecb28e3b180, vma=0xffff9ecadd566090, vma@entry=0xffff9ecadd566000, > > > > > floor=floor@entry=0, ceiling=ceiling@entry=0) at mm/memory.c:431 > > > > > > #9 0xffffffffad2b3b8d in exit_mmap (mm=mm@entry=0xffff9ecb28e3b180) > > > > > at mm/mmap.c:3237 > > > > > > #10 0xffffffffad08c81f in __mmput (mm=0xffff9ecb28e3b180) at > > > > > kernel/fork.c:1204 > > > > > > #11 mmput (mm=mm@entry=0xffff9ecb28e3b180) at kernel/fork.c:1226 > > > > > > #12 0xffffffffad09636c in exit_mm () at kernel/exit.c:563 > > > > > > #13 do_exit (code=code@entry=0) at kernel/exit.c:856 > > > > > > #14 0xffffffffad096c78 in do_group_exit (exit_code=0) at > > > > > kernel/exit.c:1019 > > > > > > #15 0xffffffffad096cf8 in __do_sys_exit_group (error_code=<optimized > > > > > out>) at kernel/exit.c:1030 > > > > > > #16 __se_sys_exit_group (error_code=<optimized out>) at > > > > > kernel/exit.c:1028 > > > > > > #17 __x64_sys_exit_group (regs=<optimized out>) at kernel/exit.c:1028 > > > > > > #18 0xffffffffadb8a327 in do_syscall_x64 (nr=<optimized out>, > > > > > regs=0xffffbaf005743f58) at arch/x86/entry/common.c:51 > > > > > > #19 do_syscall_64 (regs=0xffffbaf005743f58, nr=<optimized out>) at > > > > > arch/x86/entry/common.c:81 > > > > > > #20 0xffffffffadc000dc in entry_SYSCALL_64 () at > > > > > arch/x86/entry/entry_64.S:120 > > > > > > #21 0x00007f111c80e2c8 in ?? () > > > > > > #22 0x0000000000000002 in ?? () > > > > > > #23 0x00007f111c80e2e0 in ?? () > > > > > > #24 0x00007f111c8089e0 in ?? () > > > > > > #25 0x0000000000000000 in ?? () > > > > > > crash> f 3 > > > > > > 259 rwbase_schedule(); > > > > > > crash> p *rwb > > > > > > $1 = { > > > > > > readers = { > > > > > > counter = 1 > > > > > > }, > > > > > > rtmutex = { > > > > > > wait_lock = { > > > > > > raw_lock = { > > > > > > { > > > > > > val = { > > > > > > counter = 0 > > > > > > }, > > > > > > { > > > > > > locked = 0 '\000', > > > > > > pending = 0 '\000' > > > > > > }, > > > > > > { > > > > > > locked_pending = 0, > > > > > > tail = 0 > > > > > > } > > > > > > } > > > > > > } > > > > > > }, > > > > > > waiters = { > > > > > > rb_root = { > > > > > > rb_node = 0xffffbaf006977be0 > > > > > > }, > > > > > > rb_leftmost = 0xffffbaf00696fbe0 > > > > > > }, > > > > > > owner = 0xffff9ecb3956a881 > > > > > > } > > > > > > } > > > > > > > > > > > > Additional changes: > > > > > > 1. Allow gdb "frame" command. > > > > > > 2. Blacklist useless gdb "gcore" command. Use gcore plugin instead. > > > > > > 3. Move crash_target_init() to later time as crash target requires a > > > > > list of > > > > > > tasks to be initialized. > > > > > > > > > > > > Known issues and TBD items: > > > > > > 1. "info threads" may bail out first time throwing errors trying to > > > > > access > > > > > > userspace address during unwind process. Following "info threads" > > > > > > invokations run without issues. > > > > > > 2. To unwind a stack of inactive task, only modern Linux versions, which > > > > > use > > > > > > inactive_task_frame, are supported and only x86_64 architecture. > > > > > > 3. gdb bt unwinder does not stop properly and may show invalid frames > > > > > (21-25 > > > > > > on example above). Not a regression, existed before. > > > > > > 4. gdb bt unwinder does not work on active tasks in userspace. Not a > > > > > regression, > > > > > > existed before. > > > > > > 5. Only x86_64 architecture supported. machdep->get_task_reg() must be > > > > > > implemented for others. Not a regression, existed before. > > > > > > 6. Active tasks registers fetching imlemented only for VMware dumps, see > > > > > > x86_64_get_task_reg() for more details. Not a regression, existed > > > > > before. > > > > > > > > > > > > Signed-off-by: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > > > > > > --- > > > > > > crash_target.c | 39 ++++++++++++++++++++--------- > > > > > > defs.h | 10 ++++++-- > > > > > > gdb-10.2.patch | 7 ++---- > > > > > > gdb_interface.c | 63 ++++++++++++++++++++++++++++++---------------- > > > > > > help.c | 1 + > > > > > > main.c | 1 + > > > > > > task.c | 1 + > > > > > > x86_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++------ > > > > > > 8 files changed, 140 insertions(+), 48 deletions(-) > > > > > > > > > > > > diff --git a/crash_target.c b/crash_target.c > > > > > > index 4554806..2fdf203 100644 > > > > > > --- a/crash_target.c > > > > > > +++ b/crash_target.c > > > > > > @@ -2,6 +2,8 @@ > > > > > > * crash_target.c > > > > > > * > > > > > > * Copyright (c) 2021 VMware, Inc. > > > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" > > > > > > + * refers to Broadcom Inc. and/or its subsidiaries. > > > > > > * > > > > > > * This program is free software; you can redistribute it and/or modify > > > > > > * it under the terms of the GNU General Public License as published by > > > > > > @@ -13,7 +15,7 @@ > > > > > > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > > > > * GNU General Public License for more details. > > > > > > * > > > > > > - * Author: Alexey Makhalov <amakhalov(a)vmware.com&gt; > > > > > > + * Author: Alexey Makhalov <alexey.makhalov(a)broadcom.com&gt; > > > > > > */ > > > > > > > > > > > > #include <defs.h> > > > > > > @@ -23,11 +25,11 @@ > > > > > > #include "regcache.h" > > > > > > #include "gdbarch.h" > > > > > > > > > > > > -void crash_target_init (void); > > > > > > - > > > > > > +extern "C" void crash_target_init (void); > > > > > > extern "C" int gdb_readmem_callback(unsigned long, void *, int, int); > > > > > > -extern "C" int crash_get_nr_cpus(void); > > > > > > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char > > > > > *regname, > > > > > > +extern "C" int crash_get_nr_tasks(void); > > > > > > +extern "C" void crash_get_task_info(int task_nr, unsigned long *pid, > > > > > char **comm); > > > > > > +extern "C" int crash_get_task_reg (int task_nr, int regno, const char > > > > > *regname, > > > > > > int regsize, void *val); > > > > > > > > > > > > > > > > > > @@ -60,7 +62,13 @@ public: > > > > > > bool has_registers () override { return true; } > > > > > > bool thread_alive (ptid_t ptid) override { return true; } > > > > > > std::string pid_to_str (ptid_t ptid) override > > > > > > - { return string_printf ("CPU %ld", ptid.tid ()); } > > > > > > + { > > > > > > + unsigned long pid; > > > > > > + char *comm; > > > > > > + > > > > > > + crash_get_task_info(ptid.tid(), &pid, &comm); > > > > > > + return string_printf ("%7ld %s", pid, comm); > > > > > > + } > > > > > > > > > > > > }; > > > > > > > > > > > > @@ -68,18 +76,25 @@ public: > > > > > > void > > > > > > crash_target::fetch_registers (struct regcache *regcache, int regno) > > > > > > { > > > > > > + int r; > > > > > > gdb_byte regval[16]; > > > > > > - int cpu = inferior_ptid.tid(); > > > > > > + int task_nr = inferior_ptid.tid(); > > > > > > struct gdbarch *arch = regcache->arch (); > > > > > > > > > > > > - for (int r = 0; r < gdbarch_num_regs (arch); r++) > > > > > > + if (regno >= 0) { > > > > > > + r = regno; > > > > > > + goto onetime; > > > > > > + } > > > > > > + > > > > > > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) > > > > > > { > > > > > > +onetime: > > > > > > const char *regname = gdbarch_register_name(arch, r); > > > > > > int regsize = register_size (arch, r); > > > > > > if (regsize > sizeof (regval)) > > > > > > error (_("fatal error: buffer size is not enough to fit > > > > > register value")); > > > > > > > > > > > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)&regval)) > > > > > > + if (crash_get_task_reg (task_nr, r, regname, regsize, (void > > > > > *)&regval)) > > > > > > regcache->raw_supply (r, regval); > > > > > > else > > > > > > regcache->raw_supply (r, NULL); > > > > > > @@ -107,10 +122,10 @@ crash_target::xfer_partial (enum target_object > > > > > object, const char *annex, > > > > > > > > > > > > #define CRASH_INFERIOR_PID 1 > > > > > > > > > > > > -void > > > > > > +extern "C" void > > > > > > crash_target_init (void) > > > > > > { > > > > > > - int nr_cpus = crash_get_nr_cpus(); > > > > > > + int nr_tasks = crash_get_nr_tasks(); > > > > > > crash_target *target = new crash_target (); > > > > > > > > > > > > /* Own the target until it is successfully pushed. */ > > > > > > @@ -119,7 +134,7 @@ crash_target_init (void) > > > > > > push_target (std::move (target_holder)); > > > > > > > > > > > > inferior_appeared (current_inferior (), CRASH_INFERIOR_PID); > > > > > > - for (int i = 0; i < nr_cpus; i++) > > > > > > + for (int i = 0; i < nr_tasks; i++) > > > > > > { > > > > > > thread_info *thread = add_thread_silent (target, > > > > > > ptid_t(CRASH_INFERIOR_PID, 0, > > > > > i)); > > > > > > diff --git a/defs.h b/defs.h > > > > > > index 98650e8..2b3f247 100644 > > > > > > --- a/defs.h > > > > > > +++ b/defs.h > > > > > > @@ -1080,7 +1080,7 @@ struct machdep_table { > > > > > > void (*get_irq_affinity)(int); > > > > > > void (*show_interrupts)(int, ulong *); > > > > > > int (*is_page_ptr)(ulong, physaddr_t *); > > > > > > - int (*get_cpu_reg)(int, int, const char *, int, void *); > > > > > > + int (*get_task_reg)(struct task_context *, int, const char *, > > > > > int, void *); > > > > > > int (*is_cpu_prstatus_valid)(int cpu); > > > > > > }; > > > > > > > > > > > > @@ -2263,6 +2263,7 @@ struct size_table { /* stash of > > > > > commonly-used sizes */ > > > > > > long pt_regs; > > > > > > long task_struct; > > > > > > long thread_info; > > > > > > + long inactive_task_frame; > > > > > > long softirq_state; > > > > > > long desc_struct; > > > > > > long umode_t; > > > > > > @@ -8001,9 +8002,14 @@ extern int have_full_symbols(void); > > > > > > #define XEN_HYPERVISOR_ARCH > > > > > > #endif > > > > > > > > > > > > +/* > > > > > > + * crash_target.c > > > > > > + */ > > > > > > +extern void crash_target_init (void); > > > > > > + > > > > > > /* > > > > > > * Register numbers must be in sync with gdb/features/i386/64bit-core.c > > > > > > - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg() > > > > > > + * to make crash_target->fetch_registers() ---> machdep->get_task_reg() > > > > > > * working properly. > > > > > > */ > > > > > > enum x86_64_regnum { > > > > > > diff --git a/gdb-10.2.patch b/gdb-10.2.patch > > > > > > index a7018a2..ecf673d 100644 > > > > > > --- a/gdb-10.2.patch > > > > > > +++ b/gdb-10.2.patch > > > > > > @@ -221,7 +221,7 @@ exit 0 > > > > > > warning (_("\ > > > > > > --- gdb-10.2/gdb/main.c.orig > > > > > > +++ gdb-10.2/gdb/main.c > > > > > > -@@ -392,6 +392,14 @@ start_event_loop () > > > > > > +@@ -392,6 +392,13 @@ start_event_loop () > > > > > > return; > > > > > > } > > > > > > > > > > > > @@ -230,7 +230,6 @@ exit 0 > > > > > > +extern "C" void main_loop(void); > > > > > > +extern "C" unsigned long crash_get_kaslr_offset(void); > > > > > > +extern "C" int console(const char *, ...); > > > > > > -+void crash_target_init (void); > > > > > > +#endif > > > > > > + > > > > > > /* Call command_loop. */ > > > > > > @@ -316,7 +315,7 @@ exit 0 > > > > > > } > > > > > > } > > > > > > > > > > > > -@@ -1242,6 +1274,16 @@ captured_main (void *data) > > > > > > +@@ -1242,6 +1274,14 @@ captured_main (void *data) > > > > > > > > > > > > captured_main_1 (context); > > > > > > > > > > > > @@ -324,8 +323,6 @@ exit 0 > > > > > > + /* Relocate the vmlinux. */ > > > > > > + objfile_rebase (symfile_objfile, crash_get_kaslr_offset()); > > > > > > + > > > > > > -+ crash_target_init(); > > > > > > -+ > > > > > > + /* Back to crash. */ > > > > > > + main_loop(); > > > > > > +#endif > > > > > > diff --git a/gdb_interface.c b/gdb_interface.c > > > > > > index b14319c..03178f5 100644 > > > > > > --- a/gdb_interface.c > > > > > > +++ b/gdb_interface.c > > > > > > @@ -3,6 +3,8 @@ > > > > > > * Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > > > > > * Copyright (C) 2002-2015,2018-2019 David Anderson > > > > > > * Copyright (C) 2002-2015,2018-2019 Red Hat, Inc. All rights reserved. > > > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" > > > > > > + * refers to Broadcom Inc. and/or its subsidiaries. > > > > > > * > > > > > > * This program is free software; you can redistribute it and/or modify > > > > > > * it under the terms of the GNU General Public License as published by > > > > > > @@ -711,7 +713,7 @@ static char *prohibited_list[] = { > > > > > > "watch", "rwatch", "awatch", "attach", "continue", "c", "fg", > > > > > "detach", > > > > > > "finish", "handle", "interrupt", "jump", "kill", "next", "nexti", > > > > > > "signal", "step", "s", "stepi", "target", "until", "delete", > > > > > > - "clear", "disable", "enable", "condition", "ignore", "frame", > > > > > "catch", > > > > > > + "clear", "disable", "enable", "condition", "ignore", "gcore", > > > > > "catch", > > > > > > "tcatch", "return", "file", "exec-file", "core-file", > > > > > "symbol-file", > > > > > > "load", "si", "ni", "shell", "sy", > > > > > > NULL /* must be last */ > > > > > > @@ -877,6 +879,7 @@ gdb_readmem_callback(ulong addr, void *buf, int len, > > > > > int write) > > > > > > switch (len) > > > > > > { > > > > > > case SIZEOF_8BIT: > > > > > > + fprintf(fp, "%s\n", pc->curcmd); > > > > > > if (STREQ(pc->curcmd, "bt")) { > > > > > > if (readmem(addr, memtype, buf, SIZEOF_8BIT, > > > > > > "gdb_readmem_callback", readflags)) > > > > > > @@ -1063,34 +1066,52 @@ get_frame_offset(ulong pc) > > > > > > unsigned long crash_get_kaslr_offset(void); > > > > > > unsigned long crash_get_kaslr_offset(void) > > > > > > { > > > > > > - return kt->relocate * -1; > > > > > > + return kt->relocate * -1; > > > > > > } > > > > > > > > > > > > /* Callbacks for crash_target */ > > > > > > -int crash_get_nr_cpus(void); > > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > > > > +int crash_get_nr_tasks(void); > > > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm); > > > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname, > > > > > > int regsize, void *val); > > > > > > > > > > > > -int crash_get_nr_cpus(void) > > > > > > +int crash_get_nr_tasks(void) > > > > > > { > > > > > > - if (SADUMP_DUMPFILE()) > > > > > > - return sadump_get_nr_cpus(); > > > > > > - else if (DISKDUMP_DUMPFILE()) > > > > > > - return diskdump_get_nr_cpus(); > > > > > > - else if (KDUMP_DUMPFILE()) > > > > > > - return kdump_get_nr_cpus(); > > > > > > - else if (VMSS_DUMPFILE()) > > > > > > - return vmware_vmss_get_nr_cpus(); > > > > > > - > > > > > > - /* Just CPU #0 */ > > > > > ()> - return 1; > > > > > > + return RUNNING_TASKS(); > > > > > > } > > > > > > > > > > The crash_get_nr_tasks() will return a fixed number of running tasks, > > > > > which is the task status when crash is loading. However for live debug > > > > > mode, there will always be tasks exit and create, so we need to sync > > > > > the tasks number with the "info threads". I guess it is not an easy > > > > > work because that will involve: > > > > > > > > > > when crash loading: > > > > > for (int i = 0; i < nr_tasks; i++) > > > > > { > > > > > thread_info *thread = add_thread_silent (target, ... > > > > > } > > > > > > > > > > when new task create: > > > > > thread_info *thread = add_thread_silent (target, ... > > > > > > > > > > when task exit: > > > > > delete_thread(...) > > > > > > > > > > We may monitor those task create/exit events and do thread add/delete > > > > > in a callback function or we just always do all tasks delete/add when > > > > > "info thread" is invoked. But I guess none of these are as simple as > > > > > using crash "ps" cmd and "set <pid>", there won't be any problem. > > > > > Because crash "ps" can always show the current tasks status, and set > > > > > <pid> can switch the task context to it. > > > > > > > > > > What do you think? > > > > > > > > > > Thanks, > > > > > Tao Liu > > > > > > > > > > > > > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > > > > - int regsize, void *value) > > > > > > +/* Get task information by its index number in TT */ > > > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm) > > > > > > { > > > > > > - if (!machdep->get_cpu_reg) > > > > > > - return FALSE; > > > > > > - return machdep->get_cpu_reg(cpu, regno, regname, regsize, > > > > > value); > > > > > > + int i; > > > > > > + struct task_context *tc; > > > > > > + > > > > > > + tc = FIRST_CONTEXT(); > > > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > > > > + if (i == task_nr) { > > > > > > + *pid = tc->pid; > > > > > > + *comm = tc->comm; > > > > > > + return; > > > > > > + } > > > > > > + *pid = 0; > > > > > > + *comm = NULL; > > > > > > + return; > > > > > > +} > > > > > > + > > > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname, > > > > > > + int regsize, void *value) > > > > > > +{ > > > > > > + int i; > > > > > > + struct task_context *tc; > > > > > > + > > > > > > + if (!machdep->get_task_reg) > > > > > > + return FALSE; > > > > > > + > > > > > > + tc = FIRST_CONTEXT(); > > > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > > > > + if (i == task_nr) { > > > > > > + return machdep->get_task_reg(tc, regno, regname, > > > > > regsize, value); > > > > > > + } > > > > > > + return FALSE; > > > > > > } > > > > > > > > > > > > diff --git a/help.c b/help.c > > > > > > index a9c4d30..85dbda5 100644 > > > > > > --- a/help.c > > > > > > +++ b/help.c > > > > > > @@ -8520,6 +8520,7 @@ char *version_info[] = { > > > > > > "Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.", > > > > > > "Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.", > > > > > > "Copyright (C) 2015, 2021 VMware, Inc.", > > > > > > +"Copyright (C) 2024 Broadcom, Inc.", > > > > > > "This program is free software, covered by the GNU General Public > > > > > License,", > > > > > > "and you are welcome to change it and/or distribute copies of it under", > > > > > > "certain conditions. Enter \"help copying\" to see the conditions.", > > > > > > diff --git a/main.c b/main.c > > > > > > index 0b6b927..13acd2d 100644 > > > > > > --- a/main.c > > > > > > +++ b/main.c > > > > > > @@ -794,6 +794,7 @@ main_loop(void) > > > > > > } else > > > > > > SIGACTION(SIGINT, restart, &pc->sigaction, NULL); > > > > > > > > > > > > + crash_target_init(); > > > > > > /* > > > > > > * Display system statistics and current context. > > > > > > */ > > > > > > diff --git a/task.c b/task.c > > > > > > index ebdb5be..5d26c52 100644 > > > > > > --- a/task.c > > > > > > +++ b/task.c > > > > > > @@ -298,6 +298,7 @@ task_init(void) > > > > > > tt->flags |= THREAD_INFO; > > > > > > } > > > > > > > > > > > > + STRUCT_SIZE_INIT(inactive_task_frame, "inactive_task_frame"); > > > > > > MEMBER_OFFSET_INIT(task_struct_state, "task_struct", "state"); > > > > > > MEMBER_SIZE_INIT(task_struct_state, "task_struct", "state"); > > > > > > if (INVALID_MEMBER(task_struct_state)) { > > > > > > diff --git a/x86_64.c b/x86_64.c > > > > > > index 502817d..b6e36a5 100644 > > > > > > --- a/x86_64.c > > > > > > +++ b/x86_64.c > > > > > > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *, > > > > > ulong, ulong, char *); > > > > > > static void x86_64_framesize_debug(struct bt_info *); > > > > > > static void x86_64_get_active_set(void); > > > > > > static int x86_64_get_kvaddr_ranges(struct vaddr_range *); > > > > > > -static int x86_64_get_cpu_reg(int, int, const char *, int, void *); > > > > > > +static int x86_64_get_task_reg(struct task_context *, int, const char > > > > > *, int, void *); > > > > > > static int x86_64_verify_paddr(uint64_t); > > > > > > static void GART_init(void); > > > > > > static void x86_64_exception_stacks_init(void); > > > > > > @@ -195,7 +195,7 @@ x86_64_init(int when) > > > > > > machdep->machspec->irq_eframe_link = UNINITIALIZED; > > > > > > machdep->machspec->irq_stack_gap = UNINITIALIZED; > > > > > > machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges; > > > > > > - machdep->get_cpu_reg = x86_64_get_cpu_reg; > > > > > > + machdep->get_task_reg = x86_64_get_task_reg; > > > > > > if (machdep->cmdline_args[0]) > > > > > > parse_cmdline_args(); > > > > > > if ((string = pc->read_vmcoreinfo("relocate"))) { > > > > > > @@ -891,7 +891,7 @@ x86_64_dump_machdep_table(ulong arg) > > > > > > fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n"); > > > > > > fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n"); > > > > > > fprintf(fp, " get_kvaddr_ranges: > > > > > x86_64_get_kvaddr_ranges()\n"); > > > > > > - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n"); > > > > > > + fprintf(fp, " get_task_reg: x86_64_get_task_reg()\n"); > > > > > > fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n"); > > > > > > fprintf(fp, "clear_machdep_cache: > > > > > x86_64_clear_machdep_cache()\n"); > > > > > > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? > > > > > > @@ -6398,6 +6398,9 @@ x86_64_ORC_init(void) > > > > > > }; > > > > > > struct ORC_data *orc; > > > > > > > > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_bp, > > > > > "inactive_task_frame", "bp"); > > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, > > > > > "inactive_task_frame", "ret_addr"); > > > > > > + > > > > > > if (machdep->flags & FRAMEPOINTER) > > > > > > return; > > > > > > > > > > > > @@ -6455,9 +6458,6 @@ x86_64_ORC_init(void) > > > > > > orc->__stop_orc_unwind = symbol_value("__stop_orc_unwind"); > > > > > > orc->orc_lookup = symbol_value("orc_lookup"); > > > > > > > > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_bp, > > > > > "inactive_task_frame", "bp"); > > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, > > > > > "inactive_task_frame", "ret_addr"); > > > > > > - > > > > > > orc->has_signal = MEMBER_EXISTS("orc_entry", "signal"); /* added > > > > > at 6.3 */ > > > > > > orc->has_end = MEMBER_EXISTS("orc_entry", "end"); /* > > > > > removed at 6.4 */ > > > > > > > > > > > > @@ -9070,14 +9070,64 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp) > > > > > > } > > > > > > > > > > > > static int > > > > > > -x86_64_get_cpu_reg(int cpu, int regno, const char *name, > > > > > > +x86_64_get_task_reg(struct task_context *tc, int regno, const char > > > > > *name, > > > > > > int size, void *value) > > > > > > { > > > > > > if (regno >= LAST_REGNUM) > > > > > > return FALSE; > > > > > > > > > > > > + /* > > > > > > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15 > > > > > from > > > > > > + * inactive_task_frame (see __switch_to_asm). Other regs saved on > > > > > > + * regular frame. > > > > > > + */ > > > > > > + if (!is_task_active(tc->task)) { > > > > > > + int frame_size = STRUCT_SIZE("inactive_task_frame"); > > > > > > + > > > > > > + /* Only modern kernels supported. */ > > > > > > + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) { > > > > > > + ulong rsp; > > > > > > + int offset = 0; > > > > > > + switch (regno) { > > > > > > + case RSP_REGNUM: > > > > > > + readmem(tc->task + > > > > > OFFSET(task_struct_thread) + > > > > > > + > > > > > OFFSET(thread_struct_rsp), KVADDR, > > > > > > + &rsp, sizeof(void *), > > > > > > + "thread_struct rsp", > > > > > FAULT_ON_ERROR); > > > > > > + rsp += frame_size; > > > > > > + memcpy(value, &rsp, size); > > > > > > + return TRUE; > > > > > > + case RIP_REGNUM: > > > > > > + offset += 8; > > > > > > + case RBP_REGNUM: > > > > > > + offset += 8; > > > > > > + case RBX_REGNUM: > > > > > > + offset += 8; > > > > > > + case R12_REGNUM: > > > > > > + offset += 8; > > > > > > + case R13_REGNUM: > > > > > > + offset += 8; > > > > > > + case R14_REGNUM: > > > > > > + offset += 8; > > > > > > + case R15_REGNUM: > > > > > > + readmem(tc->task + > > > > > OFFSET(task_struct_thread) + > > > > > > + > > > > > OFFSET(thread_struct_rsp), KVADDR, > > > > > > + &rsp, sizeof(void *), > > > > > > + "thread_struct rsp", > > > > > FAULT_ON_ERROR); > > > > > > + readmem(rsp + offset, KVADDR, > > > > > value, sizeof(void *), > > > > > > + > > > > > "inactive_thread_frame saved regs", FAULT_ON_ERROR); > > > > > > + return TRUE; > > > > > > + } > > > > > > + } > > > > > > + /* TBD: older kernels support. */ > > > > > > + return FALSE; > > > > > > + } > > > > > > + > > > > > > + /* > > > > > > + * Task is active, grab CPU's registers > > > > > > + */ > > > > > > if (VMSS_DUMPFILE()) > > > > > > - return vmware_vmss_get_cpu_reg(cpu, regno, name, size, > > > > > value); > > > > > > + return vmware_vmss_get_cpu_reg(tc->processor, regno, > > > > > name, size, value); > > > > > > > > > > > > return FALSE; > > > > > > } > > > > > > -- > > > > > > 2.39.0 > > > > > > > > > > > > > > > > > > > >