[Crash-utility][PATCH] Fix crash initialization failure on LoongArch with recent GDB versions
by Ming Wang
The crash tool failed to initialize on LoongArch64 when using
GDB 16.2 (and likely other recent GDB versions that have enhanced
LoongArch support) due to the error:
"fatal error: buffer size is not enough to fit register value".
This occurs in supply_registers() because GDB now correctly
reports the size of LoongArch LASX (256-bit) vector registers
(xr0-xr31) as 32 bytes. The `regval` buffer in `crash_target.c`
was previously fixed at 16 bytes.
This patch increases the `regval` buffer size to 32 bytes to
accommodate the largest LoongArch registers reported by GDB.
This allows crash to initialize successfully.
Signed-off-by: Ming Wang <wangming01(a)loongson.cn>
---
crash_target.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/crash_target.c b/crash_target.c
index 5966b7b..d93d58c 100644
--- a/crash_target.c
+++ b/crash_target.c
@@ -71,7 +71,7 @@ public:
static void supply_registers(struct regcache *regcache, int regno)
{
- gdb_byte regval[16];
+ gdb_byte regval[32];
struct gdbarch *arch = regcache->arch ();
const char *regname = gdbarch_register_name(arch, regno);
int regsize = register_size(arch, regno);
--
2.41.3
1 week
[PATCH] Fix the issue of "page excluded" messages flooding
by Lianbo Jiang
The current issue is only observed on PPC64le machine when loading crash,
E.g:
...
crash: page excluded: kernel virtual address: c0000000022d6098 type: "gdb_readmem_callback"
crash: page excluded: kernel virtual address: c0000000022d6098 type: "gdb_readmem_callback"
...
crash>
And this issue can not be reproduced on crash 8, which only occurred
after the gdb-16.2 upgrade(see commit dfb2bb55e530).
So far I haven't found out why it always reads the same address(excluded
page) many times, anyway, crash tool should avoid flooding messages firstly,
similarly let's use the same debug level(8) such as the read_diskdump()(see
diskdump.c).
Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
---
memory.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/memory.c b/memory.c
index 0d8d89862383..58624bb5f44c 100644
--- a/memory.c
+++ b/memory.c
@@ -2504,7 +2504,7 @@ readmem(ulonglong addr, int memtype, void *buffer, long size,
case PAGE_EXCLUDED:
RETURN_ON_PARTIAL_READ();
- if (PRINT_ERROR_MESSAGE)
+ if (CRASHDEBUG(8))
error(INFO, PAGE_EXCLUDED_ERRMSG, memtype_string(memtype, 0), addr, type);
goto readmem_error;
--
2.47.1
2 weeks, 4 days
[PATCH] Fix "kmem -p" option on Linux 6.16-rc1 and later kernels
by HAGIO KAZUHITO(萩尾 一仁)
Kernel commit acc53a0b4c156 ("mm: rename page->index to
page->__folio_index"), which is contained in Linux 6.16-rc1 and later
kernels, renamed the member. Without the patch, the "kmem -p" option
fails with the following error:
kmem: invalid structure member offset: page_index
FILE: memory.c LINE: 6016 FUNCTION: dump_mem_map_SPARSEMEM()
Signed-off-by: Kazuhito Hagio <k-hagio-ab(a)nec.com>
---
memory.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/memory.c b/memory.c
index 0d8d89862383..5cb8b58e2181 100644
--- a/memory.c
+++ b/memory.c
@@ -531,6 +531,8 @@ vm_init(void)
ASSIGN_OFFSET(page_mapping) = MEMBER_OFFSET("page", "_mapcount") +
STRUCT_SIZE("atomic_t") + sizeof(ulong);
MEMBER_OFFSET_INIT(page_index, "page", "index");
+ if (INVALID_MEMBER(page_index)) /* 6.16 and later */
+ MEMBER_OFFSET_INIT(page_index, "page", "__folio_index");
if (INVALID_MEMBER(page_index))
ANON_MEMBER_OFFSET_INIT(page_index, "page", "index");
MEMBER_OFFSET_INIT(page_buffers, "page", "buffers");
--
2.31.1
2 weeks, 4 days
Re: [PATCH v4 0/5] gdb multi-stack unwinding support
by lijiang
On Wed, Jun 25, 2025 at 12:04 PM <devel-request(a)lists.crash-utility.osci.io>
wrote:
> Date: Wed, 25 Jun 2025 16:01:58 +1200
> From: Tao Liu <ltao(a)redhat.com>
> Subject: [Crash-utility] [PATCH v4 0/5] gdb multi-stack unwinding
> support
> To: devel(a)lists.crash-utility.osci.io
> Cc: Tao Liu <ltao(a)redhat.com>
> Message-ID: <20250625040203.60334-1-ltao(a)redhat.com>
> Content-Type: text/plain; charset="US-ASCII"; x-default=true
>
> This patchset is based on Alexy's work [1], and is the follow-up of the
> previous "gdb stack unwinding support for crash utility" patchset.
>
> Currently gdb target analyzes only one task at a time and it backtraces
> only straight stack until end of the stack. If stacks were concatenated
> during exceptions or interrupts, gdb bt will show only the topmost one.
>
> This patchset will introduce multiple stacks support for gdb stack
> unwinding,
> which can be observed as a different threads from gdb perspective. A
> short usage is as follows:
>
> 'set <PID>' - to switch to a specific task
> 'gdb info threads' - to see list of in-kernel stacks of this task.
> 'gdb thread <ID>' - to switch to the stack.
> 'gdb bt' - to unwind it.
>
> E.g, with the patchset:
>
> crash> bt
> PID: 17636 TASK: ffff88032e0742c0 CPU: 11 COMMAND: "kworker/11:4"
> #0 [ffff88037fca6b58] machine_kexec at ffffffff8103cef2
> #1 [ffff88037fca6ba8] crash_kexec at ffffffff810c9aa3
> #2 [ffff88037fca6c70] panic at ffffffff815f0444
> ...
> #9 [ffff88037fca6ec8] do_nmi at ffffffff815fd980
> #10 [ffff88037fca6ef0] end_repeat_nmi at ffffffff815fcec1
> [exception RIP: memcpy+13]
> RIP: ffffffff812f5b1d RSP: ffff88034f2a9728 RFLAGS: 00010046
> RAX: ffffc900139fe000 RBX: ffff880374b7a1b0 RCX: 0000000000000030
> RBP: ffff88034f2a9778 R8: 000000007fffffff R9: 00000000ffffffff
> ...
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> --- <NMI exception stack> ---
> #11 [ffff88034f2a9728] memcpy at ffffffff812f5b1d
> #12 [ffff88034f2a9728] mga_dirty_update at ffffffffa024ad2b [mgag200]
> #13 [ffff88034f2a9780] mga_imageblit at ffffffffa024ae3f [mgag200]
> #14 [ffff88034f2a97a0] bit_putcs at ffffffff813424ef
> ...
>
> crash> info threads
> Id Target Id Frame
> * 1 17636 kworker/11:4 (stack 0) crash_setup_regs (oldregs=0x0,
> newregs=0xffff88037fca6bb0)
> 2 17636 kworker/11:4 (stack 1) 0xffffffff812f5b1d in memcpy ()
>
> crash> thread 2
> crash> gdb bt
> #0 0xffffffff812f5b1d in memcpy () at arch/x86/lib/memcpy_64.S:69
> ...
>
> There are 2 stacks of the current task, and we can list/switch-to/unwind
> each stack.
>
> [1]:
> https://www.mail-archive.com/devel@lists.crash-utility.osci.io/msg01204.html
>
> v2 -> v1: 1) Rebase this patchset onto gdb-16.2 [2].
> 2) Improved the silent_call_bt() to catch the error FATAL.
>
> [2]:
> https://www.mail-archive.com/devel@lists.crash-utility.osci.io/msg01354.html
>
> v3 -> v2: 1) Rebase this patchset to crash v9.0.0.
> 2) Fix v2's segfault in cmd "bt -E".
> 3) Elimit repeat stacks by adding constraints before
> gdb_add_substack().
>
> v4 -> v3: 1) Fix compiling warning of silent_call_bt()
> 2) Add known issue link found in ppc arch.
>
>
Thank you for the update, Tao.
For the v4: Ack
Thanks
Lianbo
Tao Liu (5):
> Add multi-threads support in crash target
> Call cmd_bt silently after "set pid"
> x86_64: Add gdb multi-stack unwind support
> arm64: Add gdb multi-stack unwind support
> ppc64: Add gdb multi-stack unwind support
>
> arm64.c | 102 +++++++++++++++++++++++++++++++++--
> crash_target.c | 49 +++++++++++++++--
> defs.h | 3 +-
> gdb_interface.c | 6 +--
> kernel.c | 44 +++++++++++++++
> ppc64.c | 78 ++++++++++++++++++++++++---
> task.c | 4 +-
> x86_64.c | 138 +++++++++++++++++++++++++++++++++++++++++++++---
> 8 files changed, 394 insertions(+), 30 deletions(-)
>
> --
> 2.47.0
>
2 weeks, 4 days
[PATCH] Filter repeated rq for cmd dev -d/-D
by Tao Liu
CEE reported an issue that "dev -d/-D" reports incorrect value
of read/write:
crash> dev -d
MAJOR GENDISK NAME REQUEST_QUEUE TOTAL ASYNC SYNC
8 ffff90528df86000 sda ffff9052a3d61800 144 144 0
8 ffff905280718c00 sdb ffff9052a3d63c00 48 48 0
crash> epython rqlist
ffff90528e94a5c0 sda is unknown, deadline: 89.992 (90) rq_alloc: 0.196
ffff90528e92f700 sda is unknown, deadline: 89.998 (90) rq_alloc: 0.202
ffff90528e95ccc0 sda is unknown, deadline: 89.999 (90) rq_alloc: 0.203
ffff90528e968bc0 sdb is unknown, deadline: 89.997 (90) rq_alloc: 0.201
The value of 144 ASYNC is incorrect and epython rqlist only show 3 items for sda.
The reason is, mq_check_inflight() may get the same rq multiple times during
iteration, so they are counted repeatly.
This patch will add a rq repetition check. After apply the patch:
crash> dev -d
MAJOR GENDISK NAME REQUEST_QUEUE TOTAL READ WRITE
8 ffff90528df86000 sda ffff9052a3d61800 3 3 0
8 ffff905280718c00 sdb ffff9052a3d63c00 1 1 0
Signed-off-by: Tao Liu <ltao(a)redhat.com>
---
dev.c | 43 ++++++++++++++++++++++++++++++++++++++++---
1 file changed, 40 insertions(+), 3 deletions(-)
diff --git a/dev.c b/dev.c
index 9d38aef..b0434cf 100644
--- a/dev.c
+++ b/dev.c
@@ -4316,6 +4316,9 @@ struct bt_iter_data {
ulong tags;
uint reserved;
uint nr_reserved_tags;
+ ulong **rq_list;
+ int *rq_list_len;
+ int *rq_list_cap;
busy_tag_iter_fn *fn;
void *data;
};
@@ -4381,10 +4384,30 @@ static bool bt_iter(uint bitnr, void *data)
if (!readmem(addr, KVADDR, &rq, sizeof(ulong), "blk_mq_tags.rqs[]", RETURN_ON_ERROR))
return FALSE;
+ for (int i = 0; i < *iter_data->rq_list_len; i++) {
+ /* Skip the handled rq */
+ if ((*iter_data->rq_list)[i] == rq)
+ return TRUE;
+ }
+ /* Mark the rq is handled */
+ (*iter_data->rq_list)[(*iter_data->rq_list_len)++] = rq;
+ if (*iter_data->rq_list_len > *iter_data->rq_list_cap / 2) {
+ *iter_data->rq_list_cap <<= 1;
+ ulong *tmp = reallocarray(*iter_data->rq_list,
+ *iter_data->rq_list_cap, sizeof(ulong));
+ if (!tmp) {
+ free(*iter_data->rq_list);
+ error(FATAL, "cannot reallocarray rq_list array");
+ }
+ *iter_data->rq_list = tmp;
+ }
+
return iter_data->fn(rq, iter_data->data);
}
-static void bt_for_each(ulong q, ulong tags, ulong sbq, uint reserved, uint nr_resvd_tags, struct diskio *dio)
+static void bt_for_each(ulong q, ulong tags, ulong sbq, uint reserved,
+ uint nr_resvd_tags, ulong **rq_list, int *rq_list_len,
+ int *rq_list_cap, struct diskio *dio)
{
struct sbitmap_context sc = {0};
struct mq_inflight mi = {
@@ -4395,6 +4418,9 @@ static void bt_for_each(ulong q, ulong tags, ulong sbq, uint reserved, uint nr_r
.tags = tags,
.reserved = reserved,
.nr_reserved_tags = nr_resvd_tags,
+ .rq_list = rq_list,
+ .rq_list_len = rq_list_len,
+ .rq_list_cap = rq_list_cap,
.fn = mq_check_inflight,
.data = &mi,
};
@@ -4407,10 +4433,18 @@ static void queue_for_each_hw_ctx(ulong q, ulong *hctx, uint cnt, struct diskio
{
uint i;
int bitmap_tags_is_ptr = 0;
+ ulong *rq_list;
+ int rq_list_len = 0;
+ int rq_list_cap = 1;
if (MEMBER_TYPE("blk_mq_tags", "bitmap_tags") == TYPE_CODE_PTR)
bitmap_tags_is_ptr = 1;
+ rq_list = calloc(rq_list_cap, sizeof(ulong));
+ if (!rq_list) {
+ error(FATAL, "cannot malloc rq_list array");
+ }
+
for (i = 0; i < cnt; i++) {
ulong addr = 0, tags = 0;
uint nr_reserved_tags = 0;
@@ -4432,15 +4466,18 @@ static void queue_for_each_hw_ctx(ulong q, ulong *hctx, uint cnt, struct diskio
!readmem(addr, KVADDR, &addr, sizeof(ulong),
"blk_mq_tags.bitmap_tags", RETURN_ON_ERROR))
break;
- bt_for_each(q, tags, addr, 1, nr_reserved_tags, dio);
+ bt_for_each(q, tags, addr, 1, nr_reserved_tags, &rq_list,
+ &rq_list_len, &rq_list_cap, dio);
}
addr = tags + OFFSET(blk_mq_tags_bitmap_tags);
if (bitmap_tags_is_ptr &&
!readmem(addr, KVADDR, &addr, sizeof(ulong),
"blk_mq_tags.bitmap_tags", RETURN_ON_ERROR))
break;
- bt_for_each(q, tags, addr, 0, nr_reserved_tags, dio);
+ bt_for_each(q, tags, addr, 1, nr_reserved_tags, &rq_list,
+ &rq_list_len, &rq_list_cap, dio);
}
+ free(rq_list);
}
static void get_mq_diskio_from_hw_queues(ulong q, struct diskio *dio)
--
2.47.0
2 weeks, 5 days
[PATCH] Add blk_mq shared tags support for dev -d/-D
by Tao Liu
When blk_mq shared tags enabled for devices like scsi, the IO status is
incorrect, e.g:
crash> dev -d
MAJOR GENDISK NAME REQUEST_QUEUE TOTAL ASYNC SYNC
8 ffff90528df86000 sda ffff9052a3d61800 144 144 0
8 ffff905280718c00 sdb ffff9052a3d63c00 48 48 0
crash> epython rqlist
ffff90528e94a5c0 sda is unknown, deadline: 89.992 (90) rq_alloc: 0.196
ffff90528e92f700 sda is unknown, deadline: 89.998 (90) rq_alloc: 0.202
ffff90528e95ccc0 sda is unknown, deadline: 89.999 (90) rq_alloc: 0.203
ffff90528e968bc0 sdb is unknown, deadline: 89.997 (90) rq_alloc: 0.201
The root cause is: for shared tags case, only the shared tags are put
into count. Without this patch, tags of all the hw_ctx are counted,
which is incorrect.
After apply the patch:
crash> dev -d
MAJOR GENDISK NAME REQUEST_QUEUE TOTAL READ WRITE
8 ffff90528df86000 sda ffff9052a3d61800 3 3 0
8 ffff905280718c00 sdb ffff9052a3d63c00 1 1 0
This patch makes the following modification:
1) blk_mq shared tag support.
2) Function renaming: queue_for_each_hw_ctx -> blk_mq_queue_tag_busy_iter,
because the latter is more close to the corresponding kernel function.
3) Extract a new queue_for_each_hw_ctx() function to be called for both
shared-tags case and the hw_ctx case.
Note:
The patch is safe for earlier kernels which have no blk_mq shared tags
implemented, because the blk_mq_is_shared_tags() check will exit safely.
Signed-off-by: Tao Liu <ltao(a)redhat.com>
---
Please discard the previous patch "Filter repeated rq for cmd dev -d/-D",
because filtering is an incorrect fix.
---
defs.h | 3 ++
dev.c | 96 ++++++++++++++++++++++++++++++++++++++-----------------
symbols.c | 6 ++++
3 files changed, 76 insertions(+), 29 deletions(-)
diff --git a/defs.h b/defs.h
index bbd6d4b..4fecb83 100644
--- a/defs.h
+++ b/defs.h
@@ -2271,6 +2271,9 @@ struct offset_table { /* stash of commonly-used offsets */
long task_struct_thread_context_x28;
long neigh_table_hash_heads;
long neighbour_hash;
+ long request_queue_tag_set;
+ long blk_mq_tag_set_flags;
+ long blk_mq_tag_set_shared_tags;
};
struct size_table { /* stash of commonly-used sizes */
diff --git a/dev.c b/dev.c
index 9d38aef..0a4d5c9 100644
--- a/dev.c
+++ b/dev.c
@@ -4326,6 +4326,12 @@ struct bt_iter_data {
#define MQ_RQ_IN_FLIGHT 1
#define REQ_OP_BITS 8
#define REQ_OP_MASK ((1 << REQ_OP_BITS) - 1)
+#define BLK_MQ_F_TAG_HCTX_SHARED (1 << 3)
+
+static bool blk_mq_is_shared_tags(unsigned int flags)
+{
+ return flags & BLK_MQ_F_TAG_HCTX_SHARED;
+}
static uint op_is_write(uint op)
{
@@ -4403,43 +4409,72 @@ static void bt_for_each(ulong q, ulong tags, ulong sbq, uint reserved, uint nr_r
sbitmap_for_each_set(&sc, bt_iter, &iter_data);
}
-static void queue_for_each_hw_ctx(ulong q, ulong *hctx, uint cnt, struct diskio *dio)
+static bool queue_for_each_hw_ctx(ulong q, ulong blk_mq_tags_ptr,
+ bool bitmap_tags_is_ptr, struct diskio *dio)
{
- uint i;
+ uint i, nr_reserved_tags = 0;
+ ulong tags = 0, addr = 0;
+ bool ret = FALSE;
+
+ if (!readmem(blk_mq_tags_ptr, KVADDR, &tags, sizeof(ulong),
+ "blk_mq_hw_ctx.tags", RETURN_ON_ERROR))
+ goto out;
+
+ addr = tags + OFFSET(blk_mq_tags_nr_reserved_tags);
+ if (!readmem(addr, KVADDR, &nr_reserved_tags, sizeof(uint),
+ "blk_mq_tags_nr_reserved_tags", RETURN_ON_ERROR))
+ goto out;
+
+ if (nr_reserved_tags) {
+ addr = tags + OFFSET(blk_mq_tags_breserved_tags);
+ if (bitmap_tags_is_ptr &&
+ !readmem(addr, KVADDR, &addr, sizeof(ulong),
+ "blk_mq_tags.bitmap_tags", RETURN_ON_ERROR))
+ goto out;
+ bt_for_each(q, tags, addr, 1, nr_reserved_tags, dio);
+ }
+ addr = tags + OFFSET(blk_mq_tags_bitmap_tags);
+ if (bitmap_tags_is_ptr &&
+ !readmem(addr, KVADDR, &addr, sizeof(ulong),
+ "blk_mq_tags.bitmap_tags", RETURN_ON_ERROR))
+ goto out;
+ bt_for_each(q, tags, addr, 0, nr_reserved_tags, dio);
+
+ ret = TRUE;
+out:
+ return ret;
+}
+
+/*
+ * Replica of kernel block/blk-mq-tag.c:blk_mq_queue_tag_busy_iter()
+*/
+static void blk_mq_queue_tag_busy_iter(ulong q, ulong *hctx, uint cnt,
+ struct diskio *dio)
+{
+ uint i, flags;
int bitmap_tags_is_ptr = 0;
+ ulong addr = 0;
if (MEMBER_TYPE("blk_mq_tags", "bitmap_tags") == TYPE_CODE_PTR)
bitmap_tags_is_ptr = 1;
- for (i = 0; i < cnt; i++) {
- ulong addr = 0, tags = 0;
- uint nr_reserved_tags = 0;
+ readmem(q + OFFSET(request_queue_tag_set), KVADDR, &addr,
+ sizeof(ulong), "request_queue.tag_set", RETURN_ON_ERROR);
- /* Tags owned by the block driver */
- addr = hctx[i] + OFFSET(blk_mq_hw_ctx_tags);
- if (!readmem(addr, KVADDR, &tags, sizeof(ulong),
- "blk_mq_hw_ctx.tags", RETURN_ON_ERROR))
- break;
+ readmem(addr + OFFSET(blk_mq_tag_set_flags), KVADDR,
+ &flags, sizeof(uint), "blk_mq_tag_set.flags", RETURN_ON_ERROR);
- addr = tags + OFFSET(blk_mq_tags_nr_reserved_tags);
- if (!readmem(addr, KVADDR, &nr_reserved_tags, sizeof(uint),
- "blk_mq_tags_nr_reserved_tags", RETURN_ON_ERROR))
- break;
+ if (blk_mq_is_shared_tags(flags)) {
+ addr = addr + OFFSET(blk_mq_tag_set_shared_tags);
+ queue_for_each_hw_ctx(q, addr, bitmap_tags_is_ptr, dio);
+ return;
+ }
- if (nr_reserved_tags) {
- addr = tags + OFFSET(blk_mq_tags_breserved_tags);
- if (bitmap_tags_is_ptr &&
- !readmem(addr, KVADDR, &addr, sizeof(ulong),
- "blk_mq_tags.bitmap_tags", RETURN_ON_ERROR))
- break;
- bt_for_each(q, tags, addr, 1, nr_reserved_tags, dio);
- }
- addr = tags + OFFSET(blk_mq_tags_bitmap_tags);
- if (bitmap_tags_is_ptr &&
- !readmem(addr, KVADDR, &addr, sizeof(ulong),
- "blk_mq_tags.bitmap_tags", RETURN_ON_ERROR))
- break;
- bt_for_each(q, tags, addr, 0, nr_reserved_tags, dio);
+ for (i = 0; i < cnt; i++) {
+ /* Tags owned by the block driver */
+ addr = hctx[i] + OFFSET(blk_mq_hw_ctx_tags);
+ if (queue_for_each_hw_ctx(q, addr, bitmap_tags_is_ptr, dio) == FALSE)
+ return;
}
}
@@ -4489,7 +4524,7 @@ static void get_mq_diskio_from_hw_queues(ulong q, struct diskio *dio)
return;
}
- queue_for_each_hw_ctx(q, hctx_array, cnt, dio);
+ blk_mq_queue_tag_busy_iter(q, hctx_array, cnt, dio);
FREEBUF(hctx_array);
}
@@ -4914,6 +4949,9 @@ void diskio_init(void)
MEMBER_SIZE_INIT(class_private_devices, "class_private",
"class_devices");
MEMBER_OFFSET_INIT(disk_stats_in_flight, "disk_stats", "in_flight");
+ MEMBER_OFFSET_INIT(request_queue_tag_set, "request_queue", "tag_set");
+ MEMBER_OFFSET_INIT(blk_mq_tag_set_flags, "blk_mq_tag_set", "flags");
+ MEMBER_OFFSET_INIT(blk_mq_tag_set_shared_tags, "blk_mq_tag_set", "shared_tags");
dt->flags |= DISKIO_INIT;
}
diff --git a/symbols.c b/symbols.c
index e30fafe..794519a 100644
--- a/symbols.c
+++ b/symbols.c
@@ -11487,6 +11487,12 @@ dump_offset_table(char *spec, ulong makestruct)
OFFSET(blk_mq_tags_nr_reserved_tags));
fprintf(fp, " blk_mq_tags_rqs: %ld\n",
OFFSET(blk_mq_tags_rqs));
+ fprintf(fp, " request_queue_tag_set: %ld\n",
+ OFFSET(request_queue_tag_set));
+ fprintf(fp, " blk_mq_tag_set_flags: %ld\n",
+ OFFSET(blk_mq_tag_set_flags));
+ fprintf(fp, " blk_mq_tag_set_shared_tags: %ld\n",
+ OFFSET(blk_mq_tag_set_shared_tags));
fprintf(fp, " subsys_private_subsys: %ld\n", OFFSET(subsys_private_subsys));
fprintf(fp, " subsys_private_klist_devices: %ld\n",
--
2.47.0
2 weeks, 5 days
Crash tool failed to parse vmcore from Linux v6.15 on RISCV
by Pnina Feder
Hi,
When parsing a vmcore generated by Linux 6.15 on RISC-V, the crash tool fails.
We identified the root cause: the tool is unable to read memory addresses that are marked as reserved in the /proc/iomem map.
These addresses are missing from the vmcore, yet certain kernel structures (e.g., IRQ pointers) reference them.
This issue did not occur with Linux 6.14, where the same addresses were not marked as reserved in /proc/iomem and were correctly included in the vmcore.
Did anybody see something like that?
Thanks,
Pnina
2 weeks, 6 days
Crash tool failes to boot with vmcore from linux 6.15 on riscv64 (the same works on linux 6.14)
by pnina.feder＠mobileye.com
crash: CONFIG_NR_CPUS: 32
crash: CONFIG_HZ: 1000
crash: # CONFIG_DEBUG_INFO_REDUCED is not set
cpu_possible_mask: cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
cpu_present_mask: cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
cpu_online_mask: cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
cpu_active_mask: cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
xtime timespec.tv_sec: 0: Thu Jan 1 02:00:00 IST 1970
utsname:
sysname: Linux
nodename: buildroot
release: 6.15.0
version: #1 SMP PREEMPT_RT Mon Jun 23 15:09:03 IDT 2025
machine: riscv64
domainname: (none)
base kernel version: 6.15.0
verify_namelist:
dumpfile /proc/version:
Linux version 6.15.0 (pfeder@epgd034) (riscv64-mti-linux-gnu-gcc (MIPS GNU Tools v1.13 for RISC-V Linux) 11.2.0, GNU ld (MIPS GNU Tools v1.13 for RISC-V Linux) 2.42) #1 SMP PREEMPT_RT Mon Jun 23 15:09:03 IDT 2025
/localdrive/users/pfeder/EQ7_22_06/open_src/eyeq7/buildroot/out-eq7-qemu/build/linux-custom/vmlinux:
Linux version 6.15.0 (pfeder@epgd034) (riscv64-mti-linux-gnu-gcc (MIPS GNU Tools v1.13 for RISC-V Linux) 11.2.0, GNU ld (MIPS GNU Tools v1.13 for RISC-V Linux) 2.42) #1 SMP PREEMPT_RT Mon Jun 23 15:09:03 IDT 2025
crash: get_cpus_present: present: 24
crash: get_cpus_present: present: 24
hypervisor: (undetermined)
irq_stack_ptr:
type: 1, TYPE_CODE_PTR
target_typecode: 17, other
target_length: 8
length: 8
IRQ stack pointer[0] is ffffffd6fbdce068
crash: read error: kernel virtual address: ffffffd6fbdce068 type: "IRQ stack pointer"
IRQ stack pointer[1] is ffffffd6fbde3068
crash: read error: kernel virtual address: ffffffd6fbde3068 type: "IRQ stack pointer"
IRQ stack pointer[2] is ffffffd6fbdf8068
crash: read error: kernel virtual address: ffffffd6fbdf8068 type: "IRQ stack pointer"
IRQ stack pointer[3] is ffffffd6fbe0d068
crash: read error: kernel virtual address: ffffffd6fbe0d068 type: "IRQ stack pointer"
IRQ stack pointer[4] is ffffffd6fbe22068
crash: read error: kernel virtual address: ffffffd6fbe22068 type: "IRQ stack pointer"
IRQ stack pointer[5] is ffffffd6fbe37068
crash: read error: kernel virtual address: ffffffd6fbe37068 type: "IRQ stack pointer"
IRQ stack pointer[6] is ffffffd6fbe4c068
crash: read error: kernel virtual address: ffffffd6fbe4c068 type: "IRQ stack pointer"
IRQ stack pointer[7] is ffffffd6fbe61068
crash: read error: kernel virtual address: ffffffd6fbe61068 type: "IRQ stack pointer"
IRQ stack pointer[8] is ffffffd6fbe76068
crash: read error: kernel virtual address: ffffffd6fbe76068 type: "IRQ stack pointer"
IRQ stack pointer[9] is ffffffd6fbe8b068
crash: read error: kernel virtual address: ffffffd6fbe8b068 type: "IRQ stack pointer"
IRQ stack pointer[10] is ffffffd6fbea0068
crash: read error: kernel virtual address: ffffffd6fbea0068 type: "IRQ stack pointer"
IRQ stack pointer[11] is ffffffd6fbeb5068
crash: read error: kernel virtual address: ffffffd6fbeb5068 type: "IRQ stack pointer"
IRQ stack pointer[12] is ffffffd6fbeca068
crash: read error: kernel virtual address: ffffffd6fbeca068 type: "IRQ stack pointer"
IRQ stack pointer[13] is ffffffd6fbedf068
crash: read error: kernel virtual address: ffffffd6fbedf068 type: "IRQ stack pointer"
IRQ stack pointer[14] is ffffffd6fbef4068
crash: read error: kernel virtual address: ffffffd6fbef4068 type: "IRQ stack pointer"
IRQ stack pointer[15] is ffffffd6fbf09068
crash: read error: kernel virtual address: ffffffd6fbf09068 type: "IRQ stack pointer"
IRQ stack pointer[16] is ffffffd6fbf1e068
crash: read error: kernel virtual address: ffffffd6fbf1e068 type: "IRQ stack pointer"
IRQ stack pointer[17] is ffffffd6fbf33068
crash: read error: kernel virtual address: ffffffd6fbf33068 type: "IRQ stack pointer"
IRQ stack pointer[18] is ffffffd6fbf48068
crash: read error: kernel virtual address: ffffffd6fbf48068 type: "IRQ stack pointer"
IRQ stack pointer[19] is ffffffd6fbf5d068
crash: read error: kernel virtual address: ffffffd6fbf5d068 type: "IRQ stack pointer"
IRQ stack pointer[20] is ffffffd6fbf72068
crash: read error: kernel virtual address: ffffffd6fbf72068 type: "IRQ stack pointer"
IRQ stack pointer[21] is ffffffd6fbf87068
crash: read error: kernel virtual address: ffffffd6fbf87068 type: "IRQ stack pointer"
IRQ stack pointer[22] is ffffffd6fbf9c068
crash: read error: kernel virtual address: ffffffd6fbf9c068 type: "IRQ stack pointer"
IRQ stack pointer[23] is ffffffd6fbfb1068
crash: read error: kernel virtual address: ffffffd6fbfb1068 type: "IRQ stack pointer"
overflow_stack:
type: 2, TYPE_CODE_ARRAY
target_typecode: 8, TYPE_CODE_INT
target_length: 8
length: 4096
kernel NR_CPUS: 32
node_online_map: [1] -> nodes online: 1
node_table[0]:
id: 0
pgdat: ffffffff80ebb980
size: 1048576
present: 1048576
mem_map: ffffffd6fbfc9200
start_paddr: 800000000
start_mapnr: 8388608
NOTE: page_hash_table does not exist in this kernel
please wait... (gathering kmem slab cache data)
kmem_cache_downsize: 192 to 192
pageflags from pageflag_names:
00000001 locked
00000080 waiters
00000004 referenced
00000008 uptodate
00000010 dirty
00000020 lru
00000100 active
00000200 workingset
00000400 owner_priv_1
00000800 owner_2
00001000 arch_1
00002000 reserved
00004000 private
00008000 private_2
00000002 writeback
00000040 head
00010000 reclaim
00020000 swapbacked
00040000 unevictable
00080000 dropbehind
00100000 mlocked
crash: read error: kernel virtual address: ffffffd6fbddee00 type: "note_buf_t"
WARNING: cannot find NT_PRSTATUS note for cpu: 0
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: get_cpus_online: online: 24
crash: struct module_memory detected.
crash: read error: kernel virtual address: ffffffd6fbdd8880 type: "runqueues entry (per_cpu)"
3 weeks
[PATCH v4 0/5] gdb multi-stack unwinding support
by Tao Liu
This patchset is based on Alexy's work [1], and is the follow-up of the
previous "gdb stack unwinding support for crash utility" patchset.
Currently gdb target analyzes only one task at a time and it backtraces
only straight stack until end of the stack. If stacks were concatenated
during exceptions or interrupts, gdb bt will show only the topmost one.
This patchset will introduce multiple stacks support for gdb stack unwinding,
which can be observed as a different threads from gdb perspective. A
short usage is as follows:
'set <PID>' - to switch to a specific task
'gdb info threads' - to see list of in-kernel stacks of this task.
'gdb thread <ID>' - to switch to the stack.
'gdb bt' - to unwind it.
E.g, with the patchset:
crash> bt
PID: 17636 TASK: ffff88032e0742c0 CPU: 11 COMMAND: "kworker/11:4"
#0 [ffff88037fca6b58] machine_kexec at ffffffff8103cef2
#1 [ffff88037fca6ba8] crash_kexec at ffffffff810c9aa3
#2 [ffff88037fca6c70] panic at ffffffff815f0444
...
#9 [ffff88037fca6ec8] do_nmi at ffffffff815fd980
#10 [ffff88037fca6ef0] end_repeat_nmi at ffffffff815fcec1
[exception RIP: memcpy+13]
RIP: ffffffff812f5b1d RSP: ffff88034f2a9728 RFLAGS: 00010046
RAX: ffffc900139fe000 RBX: ffff880374b7a1b0 RCX: 0000000000000030
RBP: ffff88034f2a9778 R8: 000000007fffffff R9: 00000000ffffffff
...
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#11 [ffff88034f2a9728] memcpy at ffffffff812f5b1d
#12 [ffff88034f2a9728] mga_dirty_update at ffffffffa024ad2b [mgag200]
#13 [ffff88034f2a9780] mga_imageblit at ffffffffa024ae3f [mgag200]
#14 [ffff88034f2a97a0] bit_putcs at ffffffff813424ef
...
crash> info threads
Id Target Id Frame
* 1 17636 kworker/11:4 (stack 0) crash_setup_regs (oldregs=0x0, newregs=0xffff88037fca6bb0)
2 17636 kworker/11:4 (stack 1) 0xffffffff812f5b1d in memcpy ()
crash> thread 2
crash> gdb bt
#0 0xffffffff812f5b1d in memcpy () at arch/x86/lib/memcpy_64.S:69
...
There are 2 stacks of the current task, and we can list/switch-to/unwind
each stack.
[1]: https://www.mail-archive.com/devel@lists.crash-utility.osci.io/msg01204.html
v2 -> v1: 1) Rebase this patchset onto gdb-16.2 [2].
2) Improved the silent_call_bt() to catch the error FATAL.
[2]: https://www.mail-archive.com/devel@lists.crash-utility.osci.io/msg01354.html
v3 -> v2: 1) Rebase this patchset to crash v9.0.0.
2) Fix v2's segfault in cmd "bt -E".
3) Elimit repeat stacks by adding constraints before
gdb_add_substack().
v4 -> v3: 1) Fix compiling warning of silent_call_bt()
2) Add known issue link found in ppc arch.
Tao Liu (5):
Add multi-threads support in crash target
Call cmd_bt silently after "set pid"
x86_64: Add gdb multi-stack unwind support
arm64: Add gdb multi-stack unwind support
ppc64: Add gdb multi-stack unwind support
arm64.c | 102 +++++++++++++++++++++++++++++++++--
crash_target.c | 49 +++++++++++++++--
defs.h | 3 +-
gdb_interface.c | 6 +--
kernel.c | 44 +++++++++++++++
ppc64.c | 78 ++++++++++++++++++++++++---
task.c | 4 +-
x86_64.c | 138 +++++++++++++++++++++++++++++++++++++++++++++---
8 files changed, 394 insertions(+), 30 deletions(-)
--
2.47.0
3 weeks
"make target=ARM" fails on Ubuntu24 LTS
by Naveen Kumar Chaudhary
I am trying to build crash-9.0.0 on Ubuntu 24 LTS, but it always fails with
below error :
configure: error: Building GDB requires GMP 4.2+, and MPFR 3.1.0+.
Try the --with-gmp and/or --with-mpfr options to specify
$ pkg-config --modversion gmp
6.3.0
$ pkg-config --modversion mpfr
4.2.1
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
The packages are properly installed and the header files are also present :
$ ls /usr/include/x86_64-linux-gnu/gmp.h
/usr/include/x86_64-linux-gnu/gmp.h
$ ls /usr/include/mpfr.h
/usr/include/mpfr.h
I even tried this :
$ CPPFLAGS="-I/usr/include/x86_64-linux-gnu -I/usr/include"
LDFLAGS="-L/usr/lib/x86_64-linux-gnu" make target=ARM
but still same error. Can someone please give any pointers.
Regards,
Naveen
3 weeks, 4 days
