On Mon, Mar 18, 2024 at 2:56 AM Tao Liu <ltao@redhat.com> wrote:

Hi Aditya and Alexey,

On Mon, Mar 18, 2024 at 11:35:41AM +0800, Tao Liu wrote:
> On Sun, Mar 17, 2024 at 03:40:52PM +0530, Aditya Gupta wrote:
> > Hi Tao and Alexey,
> >
> > On Fri, Mar 15, 2024 at 03:15:19PM -0700, Alexey Makhalov wrote:
> > > Hi Tao,
> > >
> > > Thanks for the details. Agree, my approach is targeted primarily for
> > > offline debugging.
> > > Create/exit monitoring sounds like an overkill. And from the other side,
> > > maintaining a static list of threads (CPUs or Tasks) does not work for live
> > > debugging.
> > > "ps" followed by "set" where GDB will receive just one inferior/thread at a
> > > time is probably the right universal approach.
> > > Let me experiment with a combination of my and your patches and come back
> > > to you.
> >
> > Sorry I noticed a bug late while trying the combination of tao's and my
> > patches.
> >
> > I agree with Tao that current approach might not be good for live
> > debugging, but I think the approach of having all tasks in 'info
> > threads' is a good advantage for offline debugging, since then gdb knows
> > about all tasks (similar behaviour as usual gdb), and makes it easier
> > to synchronise crash <-> gdb context.
> >
> > I had refrained from changing the format of 'info threads' from active
> > tasks to all tasks, but there are some issues with crash <-> gdb sync,
> > when doing arbitrary task unwinding.
> >
> > 'set 1' (let's assume it's an INACTIVE task on CPU 5)
> >
> > 'info threads'
> > -> when it switches to CPU 5, it asks crash to sync, ie. `set_cpu(5)`
> > -> crash will return CPU 5's active tasks registers
> > -> WRONG registers for CPU 5, since we wanted registers for PID 1 (which
> > is an inactive task on CPU 5)
>
> What do you mean of "WRONG registers"? Isn't the CPU5's regcache be pid
> 1's context after 'set 1'? If it isn't then it's a bug.
>
> After crash finished loading and we see crash> prompt, the cpu regcaches
> will be all active tasks', which are running on each cpu. That is what we saw
> using "info threads", like:
>
> crash> info threads
> Id Target Id Frame
> ....
> * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640
> ....
> 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54
> ....
>
> Then we use "set <pid>" to switch to task <pid>, its cpu regcache will be
> flushed, and if we use command "info threads" again, it will show the status
> after the flushing. E.g in the following, task pid 1's cpu is #13, so
> cpu #13 is flushed after "set 1":
>
> crash> ps
> PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd
>
> crash> set 1
> crash> info threads
> Id Target Id Frame
> ....
> 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640
> ....
> * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
> ....
>
> When design, I assume the CPUs/threads are like power slots, its number is
> constant, but the contents can be changed. We don't need to prepare the
> "slots" as many as our tasks quantity (I don't know the maxinum threads that
> a gdb can accept, e.g. on a system which have 65K tasks, but for most
> tasks we'd never switch to it in order to view its stack trace, I guess
> it would be a waste to call add_thread_silent (target, ...) 65K times).
>
> Currently the gdb CPUs/threads(slots) equals to the system's cpu number. But
> if a system which have thousands of cpus, which we think is unnecessary to
> call add_thread_silent 1K times, we can even shrink the slots to be less.
>

I made a trial patch of leaving only 1 gdb CPUs/threads(slots) in [1].
And it can also view arbitrary tasks stack unwinding. In this case,
"info threads" and "thread XX" will have no meaning to be kept.

Please see my test results:

KERNEL: /home/ltao/kernel/linux-next-next-20220914/vmlinux
DUMPFILE: /proc/kcore
CPUS: 4
^^^^^^^ there are 4 cpus in the system
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
...
1 0 1 ffff9908002a6500 IN 0.1 253024 19444 systemd
...
5 2 0 ffff9908002a0000 ID 0.0 0 0 [netns]
...
^^^^^^^^^^^^^ we use pid 1 and pid 5, which belongs to 2 different cpus
crash> set 1
crash> gdb bt
#0 context_switch (rf=0xffffbf750001bd68, next=0xffff99080032cbc0, prev=0x0, rq=0xffff990b6fcafe80) at kernel/sched/core.c:5201
#1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6510
#2 0xffffffffb3cf0603 in schedule () at kernel/sched/core.c:6586
#3 0xffffffffb3cf6ab9 in schedule_hrtimeout_range_clock (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS, clock_id=clock_id@entry=1) at kernel/time/hrtimer.c:2296
...
crash> bt
PID: 1 TASK: ffff9908002a6500 CPU: 1 COMMAND: "systemd"
#0 [ffffbf750001bdb0] schedule at ffffffffb3cf0603
#1 [ffffbf750001bdc0] schedule_hrtimeout_range_clock at ffffffffb3cf6ab9
...
^^^ we can view pid 1's stack trace with no problem
crash> info threads
Id Target Id Frame
* 1 CPU 0 context_switch (rf=0xffffbf750001bd68, next=0xffffffffb4c18a00 <init_task>, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201

crash> set 5
crash> gdb bt
#0 context_switch (rf=0xffffbf750003be58, next=0xffff9908002d6500, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201
#1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6510
#2 0xffffffffb3cf0603 in schedule () at kernel/sched/core.c:6586
#3 0xffffffffb3308752 in rescuer_thread (__rescuer=0xffff9908001f7480) at kernel/workqueue.c:2599
...
crash> bt
PID: 5 TASK: ffff9908002a0000 CPU: 0 COMMAND: "netns"
#0 [ffffbf750003bea0] schedule at ffffffffb3cf0603
#1 [ffffbf750003beb0] rescuer_thread at ffffffffb3308752
...
^^^ we can view pid 5's stack trace with no problem
crash> info threads
Id Target Id Frame
* 1 CPU 0 context_switch (rf=0xffffbf750003be58, next=0xffff9908002d6500, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201
^^^^^^^^^^^^^^^^^^^^^ now its pid 5's context

From my view, I think we can keep the current "gdb thread/CPUs = nr_cpus", or
be more ambitious, leaving "gdb thread/CPUs = 1", in case there are
systems with thousands of cpus, and we only need this in task.c:

+ gdb_refresh_regcache(0);
- for (int i = 0; i < get_cpus_online(); i++)
- gdb_refresh_regcache(i);

Do you have any thoughts?

[1]: https://github.com/liutgnu/crash-dev/commits/one-thread

Thanks,
Tao Liu

> Thanks,
> Tao Liu
>
>
> >
> > I can think of solving it in two ways:
> > 1. Having a array in crash_target, which tells what CPU in gdb maps to
> > which task in crash, this can be done without requiring changes to 'info
> > threads', while we do the 'add_thread_silent'
> > 2. Having all tasks in gdb itself, so it's a one-to-one, so `set_cpu`
> > will then refer to the task ID, I will have to explore more on this.
> >
> > Any comments ?
> >
> > Thanks,
> > Aditya Gupta
> >
> > >
> > > Thanks,
> > > --Alexey
> > >
> > >
> > > On Thu, Mar 14, 2024 at 9:22 PM Tao Liu <ltao@redhat.com> wrote:
> > >
> > > > Hi Alexey,
> > > >
> > > > On Thu, Mar 14, 2024 at 6:29 PM Alexey Makhalov
> > > > <alexey.makhalov@broadcom.com> wrote:
> > > > >
> > > > > Support for GDB debugging of all tasks active and inactive.
> > > > > Before this commit only active tasks were listed by "info threads"
> > > > > with "CPU #" as a Target Id.
> > > > >
> > > > > "info threads" will now show all tasks, similar to "ps", example:
> > > > > crash> info threads
> > > > > Id Target Id Frame
> > > > > * 1 0 swapper/0 0xffffffffadba19d4 in default_idle () at
> > > > arch/x86/kernel/process.c:731
> > > > > 2 0 swapper/1 0xffffffffadba19d4 in default_idle () at
> > > > arch/x86/kernel/process.c:731
> > > > > 3 0 swapper/2 0xffffffffadba19d4 in default_idle () at
> > > > arch/x86/kernel/process.c:731
> > > > > 4 0 swapper/3 0xffffffffadba19d4 in default_idle () at
> > > > arch/x86/kernel/process.c:731
> > > > > 5 0 swapper/4 0xffffffffadb97292 in context_switch
> > > > (rf=0xffffbaf0000f3e88, next=0xffff9ecb04908000, prev=<optimized out>,
> > > > rq=<optimized out>) at kernel/sched/core.c:5372
> > > > > ...
> > > > > 730 970325 taskset 0xffffffffadb97292 in context_switch
> > > > (rf=0xffffbaf006a0fd18, next=0xffff9ecb0aec0000, prev=<optimized out>,
> > > > rq=<optimized out>) at kernel/sched/core.c:5372
> > > > > 731 975217 sleep 0xffffffffadb97292 in context_switch
> > > > (rf=0xffffbaf005743c20, next=0xffff9ecac0692880, prev=<optimized out>,
> > > > rq=<optimized out>) at kernel/sched/core.c:5372
> > > > > 732 975228 sleep 0xffffffffadb97292 in context_switch
> > > > (rf=0xffffbaf00696fb58, next=0xffff9ecac0690000, prev=<optimized out>,
> > > > rq=<optimized out>) at kernel/sched/core.c:5372
> > > > > ...
> > > > > 876 976084 docker 0xffffffffadb97292 in context_switch
> > > > (rf=0xffffbaf0153dbd10, next=0xffff9ecac0645100, prev=<optimized out>,
> > > > rq=<optimized out>) at kernel/sched/core.c:5372
> > > > > 877 976085 systemd-userwor 0xffffffffadb97292 in context_switch
> > > > (rf=0xffffbaf0153cbc58, next=0xffff9ecac0645100, prev=<optimized out>,
> > > > rq=<optimized out>) at kernel/sched/core.c:5372
> > > > > 878 976086 systemd-userwor 0xffffffffadb97292 in context_switch
> > > > (rf=0xffffbaf0153e3c58, next=0xffffffffaec15a40 <init_task>,
> > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372
> > > > > 879 976087 systemd-userwor 0xffffffffadb97292 in context_switch
> > > > (rf=0xffffbaf0153ebc58, next=0xffffffffaec15a40 <init_task>,
> > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372
> > > > > Where "Target ID" contains "PID COMM" of the task
> > > > >
> > > >
> > > > I see here, we chose a different tech path of viewing arbitrary tasks.
> > > > You improved the "info threads" as an alternative to crash "ps" cmd,
> > > > and use "thread X" to switch to task X and view its stacktrace.
> > > > However mine is using crash "ps" cmd first, getting specific tasks pid
> > > > or task struct, then using crash cmd "set <pid>" or "set
> > > > <task_struct>" to switch to task X and view its stacktrace.
> > > >
> > > > I notice there is one problem of using the "info threads as ps" tech
> > > > path, it is difficult to handle the live debug, see the comment below:
> > > >
> > > > > Example of "731 975217 sleep" debugging, real case, trying to
> > > > > figure out why sleep was stuck in uninterruptable sleep.
> > > > > Backtrace using crash:
> > > > > crash> ps | grep 975217
> > > > > 975217 969797 3 ffff9ecb3956a880 UN 0.0 0 0
> > > > sleep
> > > > > crash> bt 975217
> > > > > PID: 975217 TASK: ffff9ecb3956a880 CPU: 3 COMMAND: "sleep"
> > > > > #0 [ffffbaf005743ba0] __schedule at ffffffffadb97292
> > > > > #1 [ffffbaf005743c60] schedule at ffffffffadb982b8
> > > > > #2 [ffffbaf005743c80] rwbase_write_lock at ffffffffadb9aed7
> > > > > #3 [ffffbaf005743cc0] down_write at ffffffffadb9b133
> > > > > #4 [ffffbaf005743cd0] unlink_file_vma at ffffffffad2b0e2e
> > > > > #5 [ffffbaf005743cf8] free_pgtables at ffffffffad2a47b0
> > > > > #6 [ffffbaf005743d88] exit_mmap at ffffffffad2b3b8d
> > > > > #7 [ffffbaf005743e80] mmput at ffffffffad08c81f
> > > > > #8 [ffffbaf005743e98] do_exit at ffffffffad09636c
> > > > > #9 [ffffbaf005743ef8] do_group_exit at ffffffffad096c78
> > > > > RIP: 00007f111c70ddf9 RSP: 00007fff451817e8 RFLAGS: 00000246
> > > > > RAX: ffffffffffffffda RBX: 00007f111c8089e0 RCX: 00007f111c70ddf9
> > > > > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
> > > > > RBP: 0000000000000000 R8: ffffffffffffff80 R9: 0000000000000000
> > > > > R10: 00007fff451817b0 R11: 0000000000000246 R12: 00007f111c8089e0
> > > > > R13: 00007f111c80e2e0 R14: 0000000000000002 R15: 00007f111c80e2c8
> > > > > ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b
> > > > >
> > > > > Backtrace using gdb (pay attention, task must be selected by thread Id):
> > > > > crash> thread 731
> > > > > [Switching to thread 731 ( 975217 sleep)]
> > > > > 5372 switch_to(prev, next, prev);
> > > > > crash> gdb bt
> > > > > #0 0xffffffffadb97292 in context_switch (rf=0xffffbaf005743c20,
> > > > next=0xffff9ecac0692880, prev=<optimized out>, rq=<optimized out>) at
> > > > kernel/sched/core.c:5372
> > > > > #1 __schedule (sched_mode=sched_mode@entry=0) at
> > > > kernel/sched/core.c:6696
> > > > > #2 0xffffffffadb982b8 in schedule () at kernel/sched/core.c:6772
> > > > > #3 0xffffffffadb9aed7 in rwbase_write_lock (rwb=rwb@entry=0xffff9ecaf1831430,
> > > > state=state@entry=2) at kernel/locking/rwbase_rt.c:259
> > > > > #4 0xffffffffadb9b133 in __down_write (sem=sem@entry=0xffff9ecaf1831430)
> > > > at kernel/locking/rwsem.c:1474
> > > > > #5 down_write (sem=sem@entry=0xffff9ecaf1831430) at
> > > > kernel/locking/rwsem.c:1574
> > > > > #6 0xffffffffad2b0e2e in i_mmap_lock_write (mapping=<optimized out>)
> > > > at ./include/linux/fs.h:466
> > > > > #7 unlink_file_vma (vma=vma@entry=0xffff9ecadd566090) at mm/mmap.c:127
> > > > > #8 0xffffffffad2a47b0 in free_pgtables (tlb=tlb@entry=0xffffbaf005743dd0,
> > > > mt=mt@entry=0xffff9ecb28e3b180, vma=0xffff9ecadd566090, vma@entry=0xffff9ecadd566000,
> > > > floor=floor@entry=0, ceiling=ceiling@entry=0) at mm/memory.c:431
> > > > > #9 0xffffffffad2b3b8d in exit_mmap (mm=mm@entry=0xffff9ecb28e3b180)
> > > > at mm/mmap.c:3237
> > > > > #10 0xffffffffad08c81f in __mmput (mm=0xffff9ecb28e3b180) at
> > > > kernel/fork.c:1204
> > > > > #11 mmput (mm=mm@entry=0xffff9ecb28e3b180) at kernel/fork.c:1226
> > > > > #12 0xffffffffad09636c in exit_mm () at kernel/exit.c:563
> > > > > #13 do_exit (code=code@entry=0) at kernel/exit.c:856
> > > > > #14 0xffffffffad096c78 in do_group_exit (exit_code=0) at
> > > > kernel/exit.c:1019
> > > > > #15 0xffffffffad096cf8 in __do_sys_exit_group (error_code=<optimized
> > > > out>) at kernel/exit.c:1030
> > > > > #16 __se_sys_exit_group (error_code=<optimized out>) at
> > > > kernel/exit.c:1028
> > > > > #17 __x64_sys_exit_group (regs=<optimized out>) at kernel/exit.c:1028
> > > > > #18 0xffffffffadb8a327 in do_syscall_x64 (nr=<optimized out>,
> > > > regs=0xffffbaf005743f58) at arch/x86/entry/common.c:51
> > > > > #19 do_syscall_64 (regs=0xffffbaf005743f58, nr=<optimized out>) at
> > > > arch/x86/entry/common.c:81
> > > > > #20 0xffffffffadc000dc in entry_SYSCALL_64 () at
> > > > arch/x86/entry/entry_64.S:120
> > > > > #21 0x00007f111c80e2c8 in ?? ()
> > > > > #22 0x0000000000000002 in ?? ()
> > > > > #23 0x00007f111c80e2e0 in ?? ()
> > > > > #24 0x00007f111c8089e0 in ?? ()
> > > > > #25 0x0000000000000000 in ?? ()
> > > > > crash> f 3
> > > > > 259 rwbase_schedule();
> > > > > crash> p *rwb
> > > > > $1 = {
> > > > > readers = {
> > > > > counter = 1
> > > > > },
> > > > > rtmutex = {
> > > > > wait_lock = {
> > > > > raw_lock = {
> > > > > {
> > > > > val = {
> > > > > counter = 0
> > > > > },
> > > > > {
> > > > > locked = 0 '\000',
> > > > > pending = 0 '\000'
> > > > > },
> > > > > {
> > > > > locked_pending = 0,
> > > > > tail = 0
> > > > > }
> > > > > }
> > > > > }
> > > > > },
> > > > > waiters = {
> > > > > rb_root = {
> > > > > rb_node = 0xffffbaf006977be0
> > > > > },
> > > > > rb_leftmost = 0xffffbaf00696fbe0
> > > > > },
> > > > > owner = 0xffff9ecb3956a881
> > > > > }
> > > > > }
> > > > >
> > > > > Additional changes:
> > > > > 1. Allow gdb "frame" command.
> > > > > 2. Blacklist useless gdb "gcore" command. Use gcore plugin instead.
> > > > > 3. Move crash_target_init() to later time as crash target requires a
> > > > list of
> > > > > tasks to be initialized.
> > > > >
> > > > > Known issues and TBD items:
> > > > > 1. "info threads" may bail out first time throwing errors trying to
> > > > access
> > > > > userspace address during unwind process. Following "info threads"
> > > > > invokations run without issues.
> > > > > 2. To unwind a stack of inactive task, only modern Linux versions, which
> > > > use
> > > > > inactive_task_frame, are supported and only x86_64 architecture.
> > > > > 3. gdb bt unwinder does not stop properly and may show invalid frames
> > > > (21-25
> > > > > on example above). Not a regression, existed before.
> > > > > 4. gdb bt unwinder does not work on active tasks in userspace. Not a
> > > > regression,
> > > > > existed before.
> > > > > 5. Only x86_64 architecture supported. machdep->get_task_reg() must be
> > > > > implemented for others. Not a regression, existed before.
> > > > > 6. Active tasks registers fetching imlemented only for VMware dumps, see
> > > > > x86_64_get_task_reg() for more details. Not a regression, existed
> > > > before.
> > > > >
> > > > > Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
> > > > > ---
> > > > > crash_target.c | 39 ++++++++++++++++++++---------
> > > > > defs.h | 10 ++++++--
> > > > > gdb-10.2.patch | 7 ++----
> > > > > gdb_interface.c | 63 ++++++++++++++++++++++++++++++----------------
> > > > > help.c | 1 +
> > > > > main.c | 1 +
> > > > > task.c | 1 +
> > > > > x86_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++------
> > > > > 8 files changed, 140 insertions(+), 48 deletions(-)
> > > > >
> > > > > diff --git a/crash_target.c b/crash_target.c
> > > > > index 4554806..2fdf203 100644
> > > > > --- a/crash_target.c
> > > > > +++ b/crash_target.c
> > > > > @@ -2,6 +2,8 @@
> > > > > * crash_target.c
> > > > > *
> > > > > * Copyright (c) 2021 VMware, Inc.
> > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom"
> > > > > + * refers to Broadcom Inc. and/or its subsidiaries.
> > > > > *
> > > > > * This program is free software; you can redistribute it and/or modify
> > > > > * it under the terms of the GNU General Public License as published by
> > > > > @@ -13,7 +15,7 @@
> > > > > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > * GNU General Public License for more details.
> > > > > *
> > > > > - * Author: Alexey Makhalov <amakhalov@vmware.com>
> > > > > + * Author: Alexey Makhalov <alexey.makhalov@broadcom.com>
> > > > > */
> > > > >
> > > > > #include <defs.h>
> > > > > @@ -23,11 +25,11 @@
> > > > > #include "regcache.h"
> > > > > #include "gdbarch.h"
> > > > >
> > > > > -void crash_target_init (void);
> > > > > -
> > > > > +extern "C" void crash_target_init (void);
> > > > > extern "C" int gdb_readmem_callback(unsigned long, void *, int, int);
> > > > > -extern "C" int crash_get_nr_cpus(void);
> > > > > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char
> > > > *regname,
> > > > > +extern "C" int crash_get_nr_tasks(void);
> > > > > +extern "C" void crash_get_task_info(int task_nr, unsigned long *pid,
> > > > char **comm);
> > > > > +extern "C" int crash_get_task_reg (int task_nr, int regno, const char
> > > > *regname,
> > > > > int regsize, void *val);
> > > > >
> > > > >
> > > > > @@ -60,7 +62,13 @@ public:
> > > > > bool has_registers () override { return true; }
> > > > > bool thread_alive (ptid_t ptid) override { return true; }
> > > > > std::string pid_to_str (ptid_t ptid) override
> > > > > - { return string_printf ("CPU %ld", ptid.tid ()); }
> > > > > + {
> > > > > + unsigned long pid;
> > > > > + char *comm;
> > > > > +
> > > > > + crash_get_task_info(ptid.tid(), &pid, &comm);
> > > > > + return string_printf ("%7ld %s", pid, comm);
> > > > > + }
> > > > >
> > > > > };
> > > > >
> > > > > @@ -68,18 +76,25 @@ public:
> > > > > void
> > > > > crash_target::fetch_registers (struct regcache *regcache, int regno)
> > > > > {
> > > > > + int r;
> > > > > gdb_byte regval[16];
> > > > > - int cpu = inferior_ptid.tid();
> > > > > + int task_nr = inferior_ptid.tid();
> > > > > struct gdbarch *arch = regcache->arch ();
> > > > >
> > > > > - for (int r = 0; r < gdbarch_num_regs (arch); r++)
> > > > > + if (regno >= 0) {
> > > > > + r = regno;
> > > > > + goto onetime;
> > > > > + }
> > > > > +
> > > > > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++)
> > > > > {
> > > > > +onetime:
> > > > > const char *regname = gdbarch_register_name(arch, r);
> > > > > int regsize = register_size (arch, r);
> > > > > if (regsize > sizeof (regval))
> > > > > error (_("fatal error: buffer size is not enough to fit
> > > > register value"));
> > > > >
> > > > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)&regval))
> > > > > + if (crash_get_task_reg (task_nr, r, regname, regsize, (void
> > > > *)&regval))
> > > > > regcache->raw_supply (r, regval);
> > > > > else
> > > > > regcache->raw_supply (r, NULL);
> > > > > @@ -107,10 +122,10 @@ crash_target::xfer_partial (enum target_object
> > > > object, const char *annex,
> > > > >
> > > > > #define CRASH_INFERIOR_PID 1
> > > > >
> > > > > -void
> > > > > +extern "C" void
> > > > > crash_target_init (void)
> > > > > {
> > > > > - int nr_cpus = crash_get_nr_cpus();
> > > > > + int nr_tasks = crash_get_nr_tasks();
> > > > > crash_target *target = new crash_target ();
> > > > >
> > > > > /* Own the target until it is successfully pushed. */
> > > > > @@ -119,7 +134,7 @@ crash_target_init (void)
> > > > > push_target (std::move (target_holder));
> > > > >
> > > > > inferior_appeared (current_inferior (), CRASH_INFERIOR_PID);
> > > > > - for (int i = 0; i < nr_cpus; i++)
> > > > > + for (int i = 0; i < nr_tasks; i++)
> > > > > {
> > > > > thread_info *thread = add_thread_silent (target,
> > > > > ptid_t(CRASH_INFERIOR_PID, 0,
> > > > i));
> > > > > diff --git a/defs.h b/defs.h
> > > > > index 98650e8..2b3f247 100644
> > > > > --- a/defs.h
> > > > > +++ b/defs.h
> > > > > @@ -1080,7 +1080,7 @@ struct machdep_table {
> > > > > void (*get_irq_affinity)(int);
> > > > > void (*show_interrupts)(int, ulong *);
> > > > > int (*is_page_ptr)(ulong, physaddr_t *);
> > > > > - int (*get_cpu_reg)(int, int, const char *, int, void *);
> > > > > + int (*get_task_reg)(struct task_context *, int, const char *,
> > > > int, void *);
> > > > > int (*is_cpu_prstatus_valid)(int cpu);
> > > > > };
> > > > >
> > > > > @@ -2263,6 +2263,7 @@ struct size_table { /* stash of
> > > > commonly-used sizes */
> > > > > long pt_regs;
> > > > > long task_struct;
> > > > > long thread_info;
> > > > > + long inactive_task_frame;
> > > > > long softirq_state;
> > > > > long desc_struct;
> > > > > long umode_t;
> > > > > @@ -8001,9 +8002,14 @@ extern int have_full_symbols(void);
> > > > > #define XEN_HYPERVISOR_ARCH
> > > > > #endif
> > > > >
> > > > > +/*
> > > > > + * crash_target.c
> > > > > + */
> > > > > +extern void crash_target_init (void);
> > > > > +
> > > > > /*
> > > > > * Register numbers must be in sync with gdb/features/i386/64bit-core.c
> > > > > - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg()
> > > > > + * to make crash_target->fetch_registers() ---> machdep->get_task_reg()
> > > > > * working properly.
> > > > > */
> > > > > enum x86_64_regnum {
> > > > > diff --git a/gdb-10.2.patch b/gdb-10.2.patch
> > > > > index a7018a2..ecf673d 100644
> > > > > --- a/gdb-10.2.patch
> > > > > +++ b/gdb-10.2.patch
> > > > > @@ -221,7 +221,7 @@ exit 0
> > > > > warning (_("\
> > > > > --- gdb-10.2/gdb/main.c.orig
> > > > > +++ gdb-10.2/gdb/main.c
> > > > > -@@ -392,6 +392,14 @@ start_event_loop ()
> > > > > +@@ -392,6 +392,13 @@ start_event_loop ()
> > > > > return;
> > > > > }
> > > > >
> > > > > @@ -230,7 +230,6 @@ exit 0
> > > > > +extern "C" void main_loop(void);
> > > > > +extern "C" unsigned long crash_get_kaslr_offset(void);
> > > > > +extern "C" int console(const char *, ...);
> > > > > -+void crash_target_init (void);
> > > > > +#endif
> > > > > +
> > > > > /* Call command_loop. */
> > > > > @@ -316,7 +315,7 @@ exit 0
> > > > > }
> > > > > }
> > > > >
> > > > > -@@ -1242,6 +1274,16 @@ captured_main (void *data)
> > > > > +@@ -1242,6 +1274,14 @@ captured_main (void *data)
> > > > >
> > > > > captured_main_1 (context);
> > > > >
> > > > > @@ -324,8 +323,6 @@ exit 0
> > > > > + /* Relocate the vmlinux. */
> > > > > + objfile_rebase (symfile_objfile, crash_get_kaslr_offset());
> > > > > +
> > > > > -+ crash_target_init();
> > > > > -+
> > > > > + /* Back to crash. */
> > > > > + main_loop();
> > > > > +#endif
> > > > > diff --git a/gdb_interface.c b/gdb_interface.c
> > > > > index b14319c..03178f5 100644
> > > > > --- a/gdb_interface.c
> > > > > +++ b/gdb_interface.c
> > > > > @@ -3,6 +3,8 @@
> > > > > * Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > > > > * Copyright (C) 2002-2015,2018-2019 David Anderson
> > > > > * Copyright (C) 2002-2015,2018-2019 Red Hat, Inc. All rights reserved.
> > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom"
> > > > > + * refers to Broadcom Inc. and/or its subsidiaries.
> > > > > *
> > > > > * This program is free software; you can redistribute it and/or modify
> > > > > * it under the terms of the GNU General Public License as published by
> > > > > @@ -711,7 +713,7 @@ static char *prohibited_list[] = {
> > > > > "watch", "rwatch", "awatch", "attach", "continue", "c", "fg",
> > > > "detach",
> > > > > "finish", "handle", "interrupt", "jump", "kill", "next", "nexti",
> > > > > "signal", "step", "s", "stepi", "target", "until", "delete",
> > > > > - "clear", "disable", "enable", "condition", "ignore", "frame",
> > > > "catch",
> > > > > + "clear", "disable", "enable", "condition", "ignore", "gcore",
> > > > "catch",
> > > > > "tcatch", "return", "file", "exec-file", "core-file",
> > > > "symbol-file",
> > > > > "load", "si", "ni", "shell", "sy",
> > > > > NULL /* must be last */
> > > > > @@ -877,6 +879,7 @@ gdb_readmem_callback(ulong addr, void *buf, int len,
> > > > int write)
> > > > > switch (len)
> > > > > {
> > > > > case SIZEOF_8BIT:
> > > > > + fprintf(fp, "%s\n", pc->curcmd);
> > > > > if (STREQ(pc->curcmd, "bt")) {
> > > > > if (readmem(addr, memtype, buf, SIZEOF_8BIT,
> > > > > "gdb_readmem_callback", readflags))
> > > > > @@ -1063,34 +1066,52 @@ get_frame_offset(ulong pc)
> > > > > unsigned long crash_get_kaslr_offset(void);
> > > > > unsigned long crash_get_kaslr_offset(void)
> > > > > {
> > > > > - return kt->relocate * -1;
> > > > > + return kt->relocate * -1;
> > > > > }
> > > > >
> > > > > /* Callbacks for crash_target */
> > > > > -int crash_get_nr_cpus(void);
> > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname,
> > > > > +int crash_get_nr_tasks(void);
> > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm);
> > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname,
> > > > > int regsize, void *val);
> > > > >
> > > > > -int crash_get_nr_cpus(void)
> > > > > +int crash_get_nr_tasks(void)
> > > > > {
> > > > > - if (SADUMP_DUMPFILE())
> > > > > - return sadump_get_nr_cpus();
> > > > > - else if (DISKDUMP_DUMPFILE())
> > > > > - return diskdump_get_nr_cpus();
> > > > > - else if (KDUMP_DUMPFILE())
> > > > > - return kdump_get_nr_cpus();
> > > > > - else if (VMSS_DUMPFILE())
> > > > > - return vmware_vmss_get_nr_cpus();
> > > > > -
> > > > > - /* Just CPU #0 */
> > > > ()> - return 1;
> > > > > + return RUNNING_TASKS();
> > > > > }
> > > >
> > > > The crash_get_nr_tasks() will return a fixed number of running tasks,
> > > > which is the task status when crash is loading. However for live debug
> > > > mode, there will always be tasks exit and create, so we need to sync
> > > > the tasks number with the "info threads". I guess it is not an easy
> > > > work because that will involve:
> > > >
> > > > when crash loading:
> > > > for (int i = 0; i < nr_tasks; i++)
> > > > {
> > > > thread_info *thread = add_thread_silent (target, ...
> > > > }
> > > >
> > > > when new task create:
> > > > thread_info *thread = add_thread_silent (target, ...
> > > >
> > > > when task exit:
> > > > delete_thread(...)
> > > >
> > > > We may monitor those task create/exit events and do thread add/delete
> > > > in a callback function or we just always do all tasks delete/add when
> > > > "info thread" is invoked. But I guess none of these are as simple as
> > > > using crash "ps" cmd and "set <pid>", there won't be any problem.
> > > > Because crash "ps" can always show the current tasks status, and set
> > > > <pid> can switch the task context to it.
> > > >
> > > > What do you think?
> > > >
> > > > Thanks,
> > > > Tao Liu
> > > >
> > > > >
> > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname,
> > > > > - int regsize, void *value)
> > > > > +/* Get task information by its index number in TT */
> > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm)
> > > > > {
> > > > > - if (!machdep->get_cpu_reg)
> > > > > - return FALSE;
> > > > > - return machdep->get_cpu_reg(cpu, regno, regname, regsize,
> > > > value);
> > > > > + int i;
> > > > > + struct task_context *tc;
> > > > > +
> > > > > + tc = FIRST_CONTEXT();
> > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++)
> > > > > + if (i == task_nr) {
> > > > > + *pid = tc->pid;
> > > > > + *comm = tc->comm;
> > > > > + return;
> > > > > + }
> > > > > + *pid = 0;
> > > > > + *comm = NULL;
> > > > > + return;
> > > > > +}
> > > > > +
> > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname,
> > > > > + int regsize, void *value)
> > > > > +{
> > > > > + int i;
> > > > > + struct task_context *tc;
> > > > > +
> > > > > + if (!machdep->get_task_reg)
> > > > > + return FALSE;
> > > > > +
> > > > > + tc = FIRST_CONTEXT();
> > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++)
> > > > > + if (i == task_nr) {
> > > > > + return machdep->get_task_reg(tc, regno, regname,
> > > > regsize, value);
> > > > > + }
> > > > > + return FALSE;
> > > > > }
> > > > >
> > > > > diff --git a/help.c b/help.c
> > > > > index a9c4d30..85dbda5 100644
> > > > > --- a/help.c
> > > > > +++ b/help.c
> > > > > @@ -8520,6 +8520,7 @@ char *version_info[] = {
> > > > > "Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.",
> > > > > "Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.",
> > > > > "Copyright (C) 2015, 2021 VMware, Inc.",
> > > > > +"Copyright (C) 2024 Broadcom, Inc.",
> > > > > "This program is free software, covered by the GNU General Public
> > > > License,",
> > > > > "and you are welcome to change it and/or distribute copies of it under",
> > > > > "certain conditions. Enter \"help copying\" to see the conditions.",
> > > > > diff --git a/main.c b/main.c
> > > > > index 0b6b927..13acd2d 100644
> > > > > --- a/main.c
> > > > > +++ b/main.c
> > > > > @@ -794,6 +794,7 @@ main_loop(void)
> > > > > } else
> > > > > SIGACTION(SIGINT, restart, &pc->sigaction, NULL);
> > > > >
> > > > > + crash_target_init();
> > > > > /*
> > > > > * Display system statistics and current context.
> > > > > */
> > > > > diff --git a/task.c b/task.c
> > > > > index ebdb5be..5d26c52 100644
> > > > > --- a/task.c
> > > > > +++ b/task.c
> > > > > @@ -298,6 +298,7 @@ task_init(void)
> > > > > tt->flags |= THREAD_INFO;
> > > > > }
> > > > >
> > > > > + STRUCT_SIZE_INIT(inactive_task_frame, "inactive_task_frame");
> > > > > MEMBER_OFFSET_INIT(task_struct_state, "task_struct", "state");
> > > > > MEMBER_SIZE_INIT(task_struct_state, "task_struct", "state");
> > > > > if (INVALID_MEMBER(task_struct_state)) {
> > > > > diff --git a/x86_64.c b/x86_64.c
> > > > > index 502817d..b6e36a5 100644
> > > > > --- a/x86_64.c
> > > > > +++ b/x86_64.c
> > > > > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *,
> > > > ulong, ulong, char *);
> > > > > static void x86_64_framesize_debug(struct bt_info *);
> > > > > static void x86_64_get_active_set(void);
> > > > > static int x86_64_get_kvaddr_ranges(struct vaddr_range *);
> > > > > -static int x86_64_get_cpu_reg(int, int, const char *, int, void *);
> > > > > +static int x86_64_get_task_reg(struct task_context *, int, const char
> > > > *, int, void *);
> > > > > static int x86_64_verify_paddr(uint64_t);
> > > > > static void GART_init(void);
> > > > > static void x86_64_exception_stacks_init(void);
> > > > > @@ -195,7 +195,7 @@ x86_64_init(int when)
> > > > > machdep->machspec->irq_eframe_link = UNINITIALIZED;
> > > > > machdep->machspec->irq_stack_gap = UNINITIALIZED;
> > > > > machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges;
> > > > > - machdep->get_cpu_reg = x86_64_get_cpu_reg;
> > > > > + machdep->get_task_reg = x86_64_get_task_reg;
> > > > > if (machdep->cmdline_args[0])
> > > > > parse_cmdline_args();
> > > > > if ((string = pc->read_vmcoreinfo("relocate"))) {
> > > > > @@ -891,7 +891,7 @@ x86_64_dump_machdep_table(ulong arg)
> > > > > fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n");
> > > > > fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n");
> > > > > fprintf(fp, " get_kvaddr_ranges:
> > > > x86_64_get_kvaddr_ranges()\n");
> > > > > - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n");
> > > > > + fprintf(fp, " get_task_reg: x86_64_get_task_reg()\n");
> > > > > fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n");
> > > > > fprintf(fp, "clear_machdep_cache:
> > > > x86_64_clear_machdep_cache()\n");
> > > > > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ?
> > > > > @@ -6398,6 +6398,9 @@ x86_64_ORC_init(void)
> > > > > };
> > > > > struct ORC_data *orc;
> > > > >
> > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_bp,
> > > > "inactive_task_frame", "bp");
> > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr,
> > > > "inactive_task_frame", "ret_addr");
> > > > > +
> > > > > if (machdep->flags & FRAMEPOINTER)
> > > > > return;
> > > > >
> > > > > @@ -6455,9 +6458,6 @@ x86_64_ORC_init(void)
> > > > > orc->__stop_orc_unwind = symbol_value("__stop_orc_unwind");
> > > > > orc->orc_lookup = symbol_value("orc_lookup");
> > > > >
> > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_bp,
> > > > "inactive_task_frame", "bp");
> > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr,
> > > > "inactive_task_frame", "ret_addr");
> > > > > -
> > > > > orc->has_signal = MEMBER_EXISTS("orc_entry", "signal"); /* added
> > > > at 6.3 */
> > > > > orc->has_end = MEMBER_EXISTS("orc_entry", "end"); /*
> > > > removed at 6.4 */
> > > > >
> > > > > @@ -9070,14 +9070,64 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp)
> > > > > }
> > > > >
> > > > > static int
> > > > > -x86_64_get_cpu_reg(int cpu, int regno, const char *name,
> > > > > +x86_64_get_task_reg(struct task_context *tc, int regno, const char
> > > > *name,
> > > > > int size, void *value)
> > > > > {
> > > > > if (regno >= LAST_REGNUM)
> > > > > return FALSE;
> > > > >
> > > > > + /*
> > > > > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15
> > > > from
> > > > > + * inactive_task_frame (see __switch_to_asm). Other regs saved on
> > > > > + * regular frame.
> > > > > + */
> > > > > + if (!is_task_active(tc->task)) {
> > > > > + int frame_size = STRUCT_SIZE("inactive_task_frame");
> > > > > +
> > > > > + /* Only modern kernels supported. */
> > > > > + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) {
> > > > > + ulong rsp;
> > > > > + int offset = 0;
> > > > > + switch (regno) {
> > > > > + case RSP_REGNUM:
> > > > > + readmem(tc->task +
> > > > OFFSET(task_struct_thread) +
> > > > > +
> > > > OFFSET(thread_struct_rsp), KVADDR,
> > > > > + &rsp, sizeof(void *),
> > > > > + "thread_struct rsp",
> > > > FAULT_ON_ERROR);
> > > > > + rsp += frame_size;
> > > > > + memcpy(value, &rsp, size);
> > > > > + return TRUE;
> > > > > + case RIP_REGNUM:
> > > > > + offset += 8;
> > > > > + case RBP_REGNUM:
> > > > > + offset += 8;
> > > > > + case RBX_REGNUM:
> > > > > + offset += 8;
> > > > > + case R12_REGNUM:
> > > > > + offset += 8;
> > > > > + case R13_REGNUM:
> > > > > + offset += 8;
> > > > > + case R14_REGNUM:
> > > > > + offset += 8;
> > > > > + case R15_REGNUM:
> > > > > + readmem(tc->task +
> > > > OFFSET(task_struct_thread) +
> > > > > +
> > > > OFFSET(thread_struct_rsp), KVADDR,
> > > > > + &rsp, sizeof(void *),
> > > > > + "thread_struct rsp",
> > > > FAULT_ON_ERROR);
> > > > > + readmem(rsp + offset, KVADDR,
> > > > value, sizeof(void *),
> > > > > +
> > > > "inactive_thread_frame saved regs", FAULT_ON_ERROR);
> > > > > + return TRUE;
> > > > > + }
> > > > > + }
> > > > > + /* TBD: older kernels support. */
> > > > > + return FALSE;
> > > > > + }
> > > > > +
> > > > > + /*
> > > > > + * Task is active, grab CPU's registers
> > > > > + */
> > > > > if (VMSS_DUMPFILE())
> > > > > - return vmware_vmss_get_cpu_reg(cpu, regno, name, size,
> > > > value);
> > > > > + return vmware_vmss_get_cpu_reg(tc->processor, regno,
> > > > name, size, value);
> > > > >
> > > > > return FALSE;
> > > > > }
> > > > > --
> > > > > 2.39.0
> > > > >
> > > >
> > > >
> >