Crash-utility April 2019

devel@lists.crash-utility.osci.io

6 participants
16 discussions

by Pierguido Lambri

Hello, Today while I was looking into a vmcore, I got suddenly the message in $SUBJECT. It started after I did a search into the process stack pages (search -t) and for each command I run afterwards I kept getting that message. For example: $ retrace-server-interact 603967269 crash ... crash> search -t ffff88040a0d5280 search: invalid list entry: 0 search: invalid list entry: 0 search: invalid list entry: 0 PID: 606 TASK: ffff88082d226eb0 CPU: 5 COMMAND: "xfsaild/dm-0" ffff88083ff5b948: ffff88040a0d5280 ffff88083ff5b990: ffff88040a0d5280 ffff88083ff5baa8: ffff88040a0d5280 ffff88083ff5baf0: ffff88040a0d5280 ffff88083ff5bcf0: ffff88040a0d5280 ffff88083ff5bd38: ffff88040a0d5280 ffff88083ff5bd98: ffff88040a0d5280 WARNING: malloc/free mismatch (29/32) crash> ps -m | grep UN [ 0 00:00:00.146] [UN] PID: 1811 TASK: ffff880c17bd1fa0 CPU: 1 COMMAND: "cp" WARNING: malloc/free mismatch (29/32) I guess this comes from a possible corrupted vmcore (I just got it from this vmcore), but I wonder why every new command keeps returning the same message. Thanks, Pier

5 years, 2 months

3
10
0 / 0

[PATCH 0/4] Add support to extract hardware device dumps from vmcore

by Surendra Mobiya

When kernel panic happens and kdump crash kernel is loaded, device drivers enabled in the kdump crash kernel collect device specific snapshot of the hardware/firmware state of their underlying devices. These snapshots are exported as ELF notes with note type NT_VMCOREDD (i.e., 0x700) in vmcore [1]. This series of patches enhance crash utility to analyze and extract these hardware specific device dumps from vmcore using "dev" command. Patches 1 and 2 enhance help -D to parse NT_VMCOREDD ELF notes in ELF vmcore and KDUMP vmcore, respectively. Patches 3 and 4 enhance dev command to analyze and extract hardware specific device dumps from ELF vmcore and KDUMP vmcore, respectively. Suggestions and feedback will be much appreciated. Thanks, Surendra [1] https://lkml.org/lkml/2018/5/2/190 --- rfc: - Moved logic to extract device dumps from "devdump" to "dev" command, in patches 3 and 4. - By default, device dumps are output to screen as hexdump, if output file is not provided. Surendra Mobiya (4): parse NT_VMCOREDD ELF notes in ELF vmcore parse NT_VMCOREDD ELF notes in KDUMP vmcore enhance dev command to extract NT_VMCOREDD from ELF vmcore enhance dev command to extract NT_VMCOREDD from KDUMP vmcore defs.h | 10 +++++ dev.c | 80 +++++++++++++++++++++++++++++++++++- diskdump.c | 84 ++++++++++++++++++++++++++++++++++++++ help.c | 23 ++++++++++- memory.c | 10 +++++ netdump.c | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++++- netdump.h | 3 ++ vmcore.h | 36 ++++++++++++++++ 8 files changed, 359 insertions(+), 5 deletions(-) create mode 100644 vmcore.h -- 2.21.0

5 years, 2 months

2
6
0 / 0

[RFC 0/4] Add support to extract hardware device dumps from vmcore

by Surendra Mobiya

When kernel panic happens and kdump crash kernel is loaded, device drivers enabled in the kdump crash kernel collect device specific snapshot of the hardware/firmware state of their underlying devices. These snapshots are exported as ELF notes with note type NT_VMCOREDD (i.e., 0x700) in vmcore [1]. This series of patches enhance crash utility to analyze and extract these hardware specific device dumps from vmcore using a new "devdump" command. Patches 1 and 2 enhance help -D to parse NT_VMCOREDD ELF notes in ELF vmcore and KDUMP vmcore, respectively. Patches 3 and 4 implement devdump command to analyze and extract hardware specific device dumps from ELF vmcore and KDUMP vmcore, respectively. Suggestions and feedback will be much appreciated. Thanks, Surendra [1] https://lkml.org/lkml/2018/5/2/190 Surendra Mobiya (4): parse NT_VMCOREDD ELF notes in ELF vmcore parse NT_VMCOREDD ELF notes in KDUMP vmcore add devdump command to extract NT_VMCOREDD from ELF vmcore enhance devdump command to extract NT_VMCOREDD from KDUMP vmcore Makefile | 4 +-- defs.h | 16 +++++++++ devdump.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ diskdump.c | 71 +++++++++++++++++++++++++++++++++++- global_data.c | 1 + help.c | 23 ++++++++++++ memory.c | 7 ++++ netdump.c | 89 +++++++++++++++++++++++++++++++++++++++++++-- netdump.h | 3 ++ vmcore.h | 36 +++++++++++++++++++ 10 files changed, 359 insertions(+), 5 deletions(-) create mode 100644 devdump.c create mode 100644 vmcore.h -- 1.8.3.1

5 years, 2 months

2
10
0 / 0

[PATCH] Fix for the "dev -[dD]" options on Linux 5.1-rc1 and later

by Kazuhito Hagio

Fix for the "dev -[dD]" options on Linux 5.1-rc1 and later kernels that contain commit 570d0200123fb4f809aa2f6226e93a458d664d70, titled "driver core: move device->knode_class to device_private". Without the patch, the commands fail with the following error message: dev: invalid structure member offset: device_knode_class FILE: dev.c LINE: 4046 FUNCTION: match_klist() Signed-off-by: Kazuhito Hagio <k-hagio(a)ab.jp.nec.com> --- defs.h | 1 + dev.c | 34 ++++++++++++++++++++++++++++++++-- symbols.c | 2 ++ 3 files changed, 35 insertions(+), 2 deletions(-) diff --git a/defs.h b/defs.h index 2466681..db192f3 100644 --- a/defs.h +++ b/defs.h @@ -2068,6 +2068,7 @@ struct offset_table { /* stash of commonly-used offsets */ long disk_stats_in_flight; long cpu_context_save_r7; long dentry_d_sb; + long device_private_knode_class; }; struct size_table { /* stash of commonly-used sizes */ diff --git a/dev.c b/dev.c index 24efea2..91b81f3 100644 --- a/dev.c +++ b/dev.c @@ -4020,6 +4020,21 @@ get_gendisk_4(unsigned long entry) OFFSET(gendisk_part0); } +/* kernel version >= 5.1 */ +static unsigned long +get_gendisk_5(unsigned long entry) +{ + unsigned long device_address; + unsigned long device_private_address; + + device_private_address = entry - OFFSET(device_private_knode_class); + readmem(device_private_address + OFFSET(device_private_device), + KVADDR, &device_address, sizeof(device_address), + "device_private.device", FAULT_ON_ERROR); + + return device_address - OFFSET(hd_struct_dev) - OFFSET(gendisk_part0); +} + /* 2.6.24 < kernel version <= 2.6.27 */ static int match_list(struct iter *i, unsigned long entry) @@ -4042,8 +4057,18 @@ match_klist(struct iter *i, unsigned long entry) { unsigned long device_address; unsigned long device_type; + unsigned long device_private_address; - device_address = entry - OFFSET(device_knode_class); + if (VALID_MEMBER(device_knode_class)) + device_address = entry - OFFSET(device_knode_class); + else { + /* kernel version >= 5.1 */ + device_private_address = entry - + OFFSET(device_private_knode_class); + readmem(device_private_address + OFFSET(device_private_device), + KVADDR, &device_address, sizeof(device_address), + "device_private.device", FAULT_ON_ERROR); + } readmem(device_address + OFFSET(device_type), KVADDR, &device_type, sizeof(device_type), "device.type", FAULT_ON_ERROR); if (device_type != i->type_address) @@ -4348,8 +4373,10 @@ init_iter(struct iter *i) i->match = match_klist; if (VALID_MEMBER(gendisk_dev)) i->get_gendisk = get_gendisk_3; - else + else if (VALID_MEMBER(device_knode_class)) i->get_gendisk = get_gendisk_4; + else + i->get_gendisk = get_gendisk_5; } } else { option_not_supported('d'); @@ -4470,6 +4497,9 @@ void diskio_init(void) MEMBER_OFFSET_INIT(device_knode_class, "device", "knode_class"); MEMBER_OFFSET_INIT(device_node, "device", "node"); MEMBER_OFFSET_INIT(device_type, "device", "type"); + MEMBER_OFFSET_INIT(device_private_device, "device_private", "device"); + MEMBER_OFFSET_INIT(device_private_knode_class, "device_private", + "knode_class"); MEMBER_OFFSET_INIT(gendisk_dev, "gendisk", "dev"); if (INVALID_MEMBER(gendisk_dev)) MEMBER_OFFSET_INIT(gendisk_dev, "gendisk", "__dev"); diff --git a/symbols.c b/symbols.c index 1ed75fe..da68eda 100644 --- a/symbols.c +++ b/symbols.c @@ -10113,6 +10113,8 @@ dump_offset_table(char *spec, ulong makestruct) OFFSET(device_private_device)); fprintf(fp, " device_private_knode_bus: %ld\n", OFFSET(device_private_knode_bus)); + fprintf(fp, " device_private_knode_class: %ld\n", + OFFSET(device_private_knode_class)); fprintf(fp, " gendisk_dev: %ld\n", OFFSET(gendisk_dev)); fprintf(fp, " gendisk_kobj: %ld\n", -- 2.18.1

5 years, 2 months

2
1
0 / 0

[PATCH 0/1] Add --kaslr support for s390x

by Mikhail Zaslonko

Since we plan to support kernel address space layout randomization (KASLR) for s390, we should adjust crash-utility accordingly. This is the minimal crash patch required to process the s390 dumps for the kernels configured with CONFIG_RANDOMIZE_BASE. Mikhail Zaslonko (1): Add --kaslr support for s390x help.c | 2 +- main.c | 3 ++- symbols.c | 9 +++++---- 3 files changed, 8 insertions(+), 6 deletions(-) -- 2.16.4

5 years, 2 months

2
2
0 / 0

Make 'files -d' return path when dentry.d_inode is null

by Moore, Martin (CSC Americas (Linux RTCC))

Hi Dave, On a recurring series of crashes on kernel 3.10.0-693.17.1.el7.x86_64 (RHEL 7.4), a problem was triggered by deletion of a file by a process with this stack trace: #0 [ffff88115b0ef798] __schedule at ffffffff816ab2ac #1 [ffff88115b0ef828] schedule at ffffffff816ab8a9 #2 [ffff88115b0ef838] jbd2_log_wait_commit at ffffffffc0177455 [jbd2] #3 [ffff88115b0ef8b0] jbd2_log_do_checkpoint at ffffffffc017405f [jbd2] #4 [ffff88115b0ef918] __jbd2_log_wait_for_space at ffffffffc017445f [jbd2] #5 [ffff88115b0ef960] add_transaction_credits at ffffffffc016e3d3 [jbd2] #6 [ffff88115b0ef9c0] start_this_handle at ffffffffc016e5e1 [jbd2] #7 [ffff88115b0efa58] jbd2__journal_restart at ffffffffc016ec9e [jbd2] #8 [ffff88115b0efa98] jbd2_journal_restart at ffffffffc016ed13 [jbd2] #9 [ffff88115b0efaa8] ext4_truncate_restart_trans at ffffffffc027be7e [ext4]: #10 [ffff88115b0efad8] ext4_free_branches at ffffffffc02bc9d7 [ext4] #11 [ffff88115b0efb38] ext4_free_branches at ffffffffc02bc887 [ext4] #12 [ffff88115b0efb98] ext4_free_branches at ffffffffc02bc887 [ext4] #13 [ffff88115b0efbf8] ext4_ind_truncate at ffffffffc02bda5e [ext4] #14 [ffff88115b0efcb8] ext4_truncate at ffffffffc0280ca8 [ext4] #15 [ffff88115b0efcf0] ext4_evict_inode at ffffffffc02818e0 [ext4] #16 [ffff88115b0efd10] evict at ffffffff8121f879 #17 [ffff88115b0efd38] iput at ffffffff81220189 #18 [ffff88115b0efd68] rfs_d_iput at ffffffffc03f7d10 [redirfs] #19 [ffff88115b0efe00] dentry_kill at ffffffff8121a90c #20 [ffff88115b0efe30] dput at ffffffff8121a9ce #21 [ffff88115b0efe50] path_put at ffffffff8120d576 #22 [ffff88115b0efe68] gsch_nd_release at ffffffffc043733e [gsch] #23 [ffff88115b0efe78] gsch_unlink_hook_fn at ffffffffc043 It was useful for our troubleshooting to identify the file being deleted in each of these crashes, and dentry_kill() takes a dentry pointer as its only argument: struct dentry *dentry_kill(struct dentry *); Finding the dentry pointer (using 'fregs' command in PyKdump, or digging it off the stack): #19 dentry_kill called from 0xffffffff8121a9ce <dput+94> +R12: 0xffff880328399858 +R13: 0x0 +R14: 0xffff880a0fb975b8 +RBP: 0xffff88115b0efe48 +RBX: 0xffff880328399800 1 RDI: 0xffff880328399800 arg0 struct dentry * 'files -d' on this dentry doesn't return the path: crash64> files -d 0xffff880328399800 DENTRY INODE SUPERBLK TYPE PATH ffff880328399800 0 0 N/A This is because dentry.d_inode is null; at this point in the removal process, dentry_iput() has cleared it. crash64> dentry.d_inode 0xffff880328399800 d_inode = 0x0 And display_dentry_info() in crash gives up if this is the case: if (!inode || !superblock) goto nopath; But the dentry still contains all the information needed to find the path: crash64> dentry.d_sb,d_name ffff880328399800 d_sb = 0xffff8817daaf6800 d_name = { { { hash = 3169988838, len = 30 }, hash_len = 132019007718 }, name = 0xffff880328399838 "MRAQ0431_1_10357_982129979.arc" So I modified the following: (defs.h) 2069d2068 < long dentry_d_sb; (filesys.c) 1698,1702c1698,1702 < } else { < inode_buf = NULL; < } < < superblock = ULONG(dentry_buf + OFFSET(dentry_d_sb)); --- > superblock = ULONG(inode_buf + OFFSET(inode_i_sb)); > } else { > inode_buf = NULL; > superblock = 0; > } 1704c1704 < if (!superblock) --- > if (!inode || !superblock) 2018d2017 < MEMBER_OFFSET_INIT(dentry_d_sb, "dentry", "d_sb"); With this patch, 'files -d' correctly returns the path: crash> files -d ffff880328399800 DENTRY INODE SUPERBLK TYPE PATH ffff880328399800 0 ffff8817daaf6800 N/A /u02/oraarch/MRAQ0431/MRAQ0431_1_10357_982129979.arc Can this be included as a patch to crash? Thanks, Martin Martin Moore Linux/Tru64 RTCC Engineer CSC Americas HPE Technology Services Hewlett Packard Enterprise

5 years, 3 months

2
3
0 / 0

← Newer
1
2
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Crash-utility April 2019