9:16 p.m.
Recently the following failure has been observed on some vmcores when
using the mount command:
crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff97a4818a3480 ffff979500013800 rootfs none /
ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
...
ffff97b484753420 0 mount: invalid kernel virtual address: 0 type:
"super_block buffer"
The kernel virtual address of the super_block is zero when the mount
command fails at the address 0xffff97b484753420. And the remaining
dumping information will be discarded. That is not expected.
Check the address and skip it, if this is an invalid kernel virtual
address, that can avoid truncating the remaining mount dumps.
Reported-by: Dave Wysochanski <dwysocha(a)redhat.com>
Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
---
filesys.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/filesys.c b/filesys.c
index c2ea78de821d..8c2d4e316208 100644
--- a/filesys.c
+++ b/filesys.c
@@ -1491,6 +1491,8 @@ show_mounts(ulong one_vfsmount, int flags, struct task_context
*namespace_contex
}
sbp = ULONG(vfsmount_buf + OFFSET(vfsmount_mnt_sb));
+ if (!IS_KVADDR(sbp))
+ continue;
if (flags)
fprintf(fp, "%s", mount_hdr);
--
2.37.1
11:22 p.m.
On 2022/12/20 12:16, Lianbo Jiang wrote:
Recently the following failure has been observed on some vmcores
when
using the mount command:
crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff97a4818a3480 ffff979500013800 rootfs none /
ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
...
ffff97b484753420 0 mount: invalid kernel virtual address: 0 type:
"super_block buffer"
Thank you for the fix.
Just out of curiosity, do you see why the vfsmount.mnt_sb is zero?
That panic occurred during mounting or unmounting?
Thanks,
Kazu
>
> The kernel virtual address of the super_block is zero when the mount
> command fails at the address 0xffff97b484753420. And the remaining
> dumping information will be discarded. That is not expected.
>
> Check the address and skip it, if this is an invalid kernel virtual
> address, that can avoid truncating the remaining mount dumps.
>
> Reported-by: Dave Wysochanski <dwysocha(a)redhat.com>
> Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
> ---
> filesys.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/filesys.c b/filesys.c
> index c2ea78de821d..8c2d4e316208 100644
> --- a/filesys.c
> +++ b/filesys.c
> @@ -1491,6 +1491,8 @@ show_mounts(ulong one_vfsmount, int flags, struct task_context
*namespace_contex
> }
>
> sbp = ULONG(vfsmount_buf + OFFSET(vfsmount_mnt_sb));
> + if (!IS_KVADDR(sbp))
> + continue;
>
> if (flags)
> fprintf(fp, "%s", mount_hdr);
11:48 p.m.
On 2022/12/21 14:22, HAGIO KAZUHITO(萩尾 一仁) wrote:
On 2022/12/20 12:16, Lianbo Jiang wrote:
> Recently the following failure has been observed on some vmcores when
> using the mount command:
>
> crash> mount
> MOUNT SUPERBLK TYPE DEVNAME DIRNAME
> ffff97a4818a3480 ffff979500013800 rootfs none /
> ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
> ...
> ffff97b484753420 0 mount: invalid kernel virtual address: 0 type:
"super_block buffer"
Thank you for the fix.
Just out of curiosity, do you see why the vfsmount.mnt_sb is zero?
That panic occurred during mounting or unmounting?
I think, it might be more helpful to print the MOUNT and SUPERBLK values
in the list even if it's zero or invalid, rather than skip it completely.
What do you think? e.g. continue here:
@@ -1500,6 +1500,11 @@ show_mounts(ulong one_vfsmount, int flags, struct task_context
*namespace_contex
mkstring(buf2, VADDR_PRLEN, RJUST|LONG_HEX,
MKSTR(sbp)));
+ if (!IS_KVADDR(sbp)) {
+ error(INFO, "cannot get super_block\n");
+ continue;
+ }
+
readmem(sbp, KVADDR, super_block_buf, SIZE(super_block),
"super_block buffer", FAULT_ON_ERROR);
type = ULONG(super_block_buf + OFFSET(super_block_s_type));
Thanks,
Kazu
>
> Thanks,
> Kazu
>
>>
>> The kernel virtual address of the super_block is zero when the mount
>> command fails at the address 0xffff97b484753420. And the remaining
>> dumping information will be discarded. That is not expected.
>>
>> Check the address and skip it, if this is an invalid kernel virtual
>> address, that can avoid truncating the remaining mount dumps.
>>
>> Reported-by: Dave Wysochanski <dwysocha(a)redhat.com>
>> Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
>> ---
>> filesys.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/filesys.c b/filesys.c
>> index c2ea78de821d..8c2d4e316208 100644
>> --- a/filesys.c
>> +++ b/filesys.c
>> @@ -1491,6 +1491,8 @@ show_mounts(ulong one_vfsmount, int flags, struct
task_context *namespace_contex
>> }
>>
>> sbp = ULONG(vfsmount_buf + OFFSET(vfsmount_mnt_sb));
>> + if (!IS_KVADDR(sbp))
>> + continue;
>>
>> if (flags)
>> fprintf(fp, "%s", mount_hdr);
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://listman.redhat.com/mailman/listinfo/crash-utility
> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
12:14 a.m.
Thank you for the comment, Kazu.
On Wed, Dec 21, 2022 at 1:48 PM HAGIO KAZUHITO(萩尾 一仁)
<k-hagio-ab(a)nec.com> wrote:
On 2022/12/21 14:22, HAGIO KAZUHITO(萩尾 一仁) wrote:
> On 2022/12/20 12:16, Lianbo Jiang wrote:
>> Recently the following failure has been observed on some vmcores when
>> using the mount command:
>>
>> crash> mount
>> MOUNT SUPERBLK TYPE DEVNAME DIRNAME
>> ffff97a4818a3480 ffff979500013800 rootfs none /
>> ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
>> ...
>> ffff97b484753420 0 mount: invalid kernel virtual address: 0
type: "super_block buffer"
>
> Thank you for the fix.
>
> Just out of curiosity, do you see why the vfsmount.mnt_sb is zero?
> That panic occurred during mounting or unmounting?
I didn't reproduce this issue, but I got several vmcores from
customers and did the debugging.
Maybe the super block has been released, but the data has not been
completely synced. I guess.
I think, it might be more helpful to print the MOUNT and SUPERBLK
values
in the list even if it's zero or invalid, rather than skip it completely.
What do you think? e.g. continue here:
Sounds good.
@@ -1500,6 +1500,11 @@ show_mounts(ulong one_vfsmount, int flags,
struct task_context *namespace_contex
mkstring(buf2, VADDR_PRLEN, RJUST|LONG_HEX,
MKSTR(sbp)));
+ if (!IS_KVADDR(sbp)) {
+ error(INFO, "cannot get super_block\n");
+ error(INFO, "cannot get super_block from
vfsmnt: 0x%lx\n", *vfsmnt);
If we would like to print the information, also try to output the vfs
mount address, looks better?
Thanks.
Lianbo
+ continue;
+ }
+
readmem(sbp, KVADDR, super_block_buf, SIZE(super_block),
"super_block buffer", FAULT_ON_ERROR);
type = ULONG(super_block_buf + OFFSET(super_block_s_type));
Thanks,
Kazu
>
> Thanks,
> Kazu
>
>>
>> The kernel virtual address of the super_block is zero when the mount
>> command fails at the address 0xffff97b484753420. And the remaining
>> dumping information will be discarded. That is not expected.
>>
>> Check the address and skip it, if this is an invalid kernel virtual
>> address, that can avoid truncating the remaining mount dumps.
>>
>> Reported-by: Dave Wysochanski <dwysocha(a)redhat.com>
>> Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
>> ---
>> filesys.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/filesys.c b/filesys.c
>> index c2ea78de821d..8c2d4e316208 100644
>> --- a/filesys.c
>> +++ b/filesys.c
>> @@ -1491,6 +1491,8 @@ show_mounts(ulong one_vfsmount, int flags, struct
task_context *namespace_contex
>> }
>>
>> sbp = ULONG(vfsmount_buf + OFFSET(vfsmount_mnt_sb));
>> + if (!IS_KVADDR(sbp))
>> + continue;
>>
>> if (flags)
>> fprintf(fp, "%s", mount_hdr);
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://listman.redhat.com/mailman/listinfo/crash-utility
> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
1:06 a.m.
On 2022/12/21 15:14, lijiang wrote:
Thank you for the comment, Kazu.
On Wed, Dec 21, 2022 at 1:48 PM HAGIO KAZUHITO(萩尾 一仁)
<k-hagio-ab(a)nec.com> wrote:
>
> On 2022/12/21 14:22, HAGIO KAZUHITO(萩尾 一仁) wrote:
>> On 2022/12/20 12:16, Lianbo Jiang wrote:
>>> Recently the following failure has been observed on some vmcores when
>>> using the mount command:
>>>
>>> crash> mount
>>> MOUNT SUPERBLK TYPE DEVNAME DIRNAME
>>> ffff97a4818a3480 ffff979500013800 rootfs none /
>>> ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
>>> ...
>>> ffff97b484753420 0 mount: invalid kernel virtual address:
0 type: "super_block buffer"
>>
>> Thank you for the fix.
>>
>> Just out of curiosity, do you see why the vfsmount.mnt_sb is zero?
>> That panic occurred during mounting or unmounting?
>
I didn't reproduce this issue, but I got several vmcores from
customers and did the debugging.
Maybe the super block has been released, but the data has not been
completely synced. I guess.
OK.
> I think, it might be more helpful to print the MOUNT and SUPERBLK values
> in the list even if it's zero or invalid, rather than skip it completely.
> What do you think? e.g. continue here:
>
Sounds good.
> @@ -1500,6 +1500,11 @@ show_mounts(ulong one_vfsmount, int flags, struct task_context
*namespace_contex
> mkstring(buf2, VADDR_PRLEN, RJUST|LONG_HEX,
> MKSTR(sbp)));
>
> + if (!IS_KVADDR(sbp)) {
> + error(INFO, "cannot get super_block\n");
+ error(INFO, "cannot get super_block from
vfsmnt: 0x%lx\n", *vfsmnt);
If we would like to print the information, also try to output the vfs
mount address, looks better?
ah, your example already has it and mnt_sb, so I didn't put them in the
error message.
>>> crash> mount
>>> MOUNT SUPERBLK TYPE DEVNAME DIRNAME
>>> ffff97a4818a3480 ffff979500013800 rootfs none /
>>> ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
>>> ...
>>> ffff97b484753420 0 mount: invalid kernel virtual address:
0 type: "super_block buffer"
^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
but printing it again may be better, and WARNING may be better
in this case. Please play around for good output and can I have
an example output?
Thanks,
Kazu
1:46 a.m.
On Wed, Dec 21, 2022 at 3:06 PM HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab(a)nec.com>
wrote:
On 2022/12/21 15:14, lijiang wrote:
> Thank you for the comment, Kazu.
>
> On Wed, Dec 21, 2022 at 1:48 PM HAGIO KAZUHITO(萩尾 一仁)
> <k-hagio-ab(a)nec.com> wrote:
>>
>> On 2022/12/21 14:22, HAGIO KAZUHITO(萩尾 一仁) wrote:
>>> On 2022/12/20 12:16, Lianbo Jiang wrote:
>>>> Recently the following failure has been observed on some vmcores when
>>>> using the mount command:
>>>>
>>>> crash> mount
>>>> MOUNT SUPERBLK TYPE DEVNAME DIRNAME
>>>> ffff97a4818a3480 ffff979500013800 rootfs none /
>>>> ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
>>>> ...
>>>> ffff97b484753420 0 mount: invalid kernel virtual
address: 0 type: "super_block buffer"
>>>
>>> Thank you for the fix.
>>>
>>> Just out of curiosity, do you see why the vfsmount.mnt_sb is zero?
>>> That panic occurred during mounting or unmounting?
>>
>
> I didn't reproduce this issue, but I got several vmcores from
> customers and did the debugging.
>
> Maybe the super block has been released, but the data has not been
> completely synced. I guess.
OK.
>
>> I think, it might be more helpful to print the MOUNT and SUPERBLK
values
>> in the list even if it's zero or invalid, rather than
skip it
completely.
>> What do you think? e.g. continue here:
>>
>
> Sounds good.
>
>> @@ -1500,6 +1500,11 @@ show_mounts(ulong one_vfsmount, int flags,
struct
task_context *namespace_contex
>> mkstring(buf2, VADDR_PRLEN,
RJUST|LONG_HEX,
>> MKSTR(sbp)));
>>
>> + if (!IS_KVADDR(sbp)) {
>> + error(INFO, "cannot get super_block\n");
>
> + error(INFO, "cannot get super_block from
> vfsmnt: 0x%lx\n", *vfsmnt);
>
> If we would like to print the information, also try to output the vfs
> mount address, looks better?
ah, your example already has it and mnt_sb, so I didn't put them in the
error message.
>>>> crash> mount
>>>> MOUNT SUPERBLK TYPE DEVNAME DIRNAME
>>>> ffff97a4818a3480 ffff979500013800 rootfs none /
>>>> ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
>>>> ...
>>>> ffff97b484753420 0 mount: invalid kernel virtual
address: 0 type: "super_block buffer"
^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
but printing it again may be better, and WARNING may be better
in this case. Please play around for good output and can I have
an example output?
With the warning as below:
+ error(WARNING, "cannot get super_block from vfsmnt:
0x%lx\n", *vfsmnt);
The output looks like this:
crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff97a4818a3480 ffff979500013800 rootfs none /
ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
ffff97e4846cab80 ffff979500017000 proc proc /proc
...
ffff97f482966e80 ffff97f4824e7800 ext4 /dev/sda1 /boot
...
ffff9804839e4480 ffff9804842b5800 autofs auto.hwlnx.ah /tnet/agoura/hwnet
ffff9804839e5680 ffff98048343c000 autofs auto.hwlnx.odc /tnet/odc/hwnet
ffff9804839e6700 ffff9804f55a9800 autofs auto.hwlnx.mun /tnet/mun/hwnet
ffff97a501d57a80 ffff97b484c2a800 autofs auto.hwlnx.mdc /tnet/mdc/hwnet
mount: WARNING: cannot get super_block from vfsmnt: 0xffff97b484753420
...
ffff97f514bec600 ffff9806d74ab800 mvfs /dev/mvfs /vobs/matrix_fpga
crash>
Thanks.
Lianbo
Thanks,
Kazu
2:03 a.m.
On 2022/12/21 16:46, lijiang wrote:
With the warning as below:
+ error(WARNING, "cannot get super_block from vfsmnt:
0x%lx\n", *vfsmnt);
The output looks like this:
crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff97a4818a3480 ffff979500013800 rootfs none /
ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
ffff97e4846cab80 ffff979500017000 proc proc /proc
...
ffff97f482966e80 ffff97f4824e7800 ext4 /dev/sda1 /boot
...
ffff9804839e4480 ffff9804842b5800 autofs auto.hwlnx.ah /tnet/agoura/hwnet
ffff9804839e5680 ffff98048343c000 autofs auto.hwlnx.odc /tnet/odc/hwnet
ffff9804839e6700 ffff9804f55a9800 autofs auto.hwlnx.mun /tnet/mun/hwnet
ffff97a501d57a80 ffff97b484c2a800 autofs auto.hwlnx.mdc /tnet/mdc/hwnet
mount: WARNING: cannot get super_block from vfsmnt: 0xffff97b484753420
...
ffff97f514bec600 ffff9806d74ab800 mvfs /dev/mvfs /vobs/matrix_fpga
crash>
Thanks, looks good. Let's go with this.
Thanks,
Kazu
2:42 a.m.
On Wed, Dec 21, 2022 at 4:03 PM HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab(a)nec.com>
wrote:
On 2022/12/21 16:46, lijiang wrote:
> With the warning as below:
>
> + error(WARNING, "cannot get super_block from
vfsmnt:
> 0x%lx\n", *vfsmnt);
>
> The output looks like this:
>
> crash> mount
> MOUNT SUPERBLK TYPE DEVNAME DIRNAME
> ffff97a4818a3480 ffff979500013800 rootfs none /
> ffff97e4846ca700 ffff97e484653000 sysfs sysfs /sys
> ffff97e4846cab80 ffff979500017000 proc proc /proc
> ...
> ffff97f482966e80 ffff97f4824e7800 ext4 /dev/sda1 /boot
> ...
> ffff9804839e4480 ffff9804842b5800 autofs auto.hwlnx.ah /tnet/agoura/hwnet
> ffff9804839e5680 ffff98048343c000 autofs auto.hwlnx.odc /tnet/odc/hwnet
> ffff9804839e6700 ffff9804f55a9800 autofs auto.hwlnx.mun /tnet/mun/hwnet
> ffff97a501d57a80 ffff97b484c2a800 autofs auto.hwlnx.mdc /tnet/mdc/hwnet
> mount: WARNING: cannot get super_block from vfsmnt: 0xffff97b484753420
> ...
> ffff97f514bec600 ffff9806d74ab800 mvfs /dev/mvfs /vobs/matrix_fpga
> crash>
Thanks, looks good. Let's go with this.
Thanks for the review, Kazu.
I will post v2 later with the above change.
Thanks.
Lianbo
Thanks,
Kazu
737
days inactive
738
days old
devel@lists.crash-utility.osci.io
7 comments
3 participants
participants (3)
-
HAGIO KAZUHITO(萩尾 一仁) -
Lianbo Jiang -
lijiang