4:52 a.m.
Hi, Jiri
在 2021年01月22日 01:00, crash-utility-request(a)redhat.com 写道:
Date: Thu, 21 Jan 2021 17:18:29 +0100
From: Jiri Bohac <jbohac(a)suse.cz>
To: crash-utility(a)redhat.com
Cc: anderson(a)redhat.com
Subject: [Crash-utility] increase __PHYSICAL_MASK_SHIFT_XEN?
Message-ID: <20210121161829.vizm27sb6pfrgmrl(a)dwarf.suse.cz>
Content-Type: text/plain; charset=us-ascii
Hi,
I've just come across a situation where crash failed to open a
dump generated by a 4.12 XEN PV dom0 kernel, terminating with
this message:
crash: read error: physical address: ffffffffffffffff type: "pud page"
The problem is a failed machine-to-physical translation.
xen_m2p() returns an error (-1UL) and x86_64_pud_offset() than
uses that value as a physical address.
I debugged the problem by running crash inside gdb. The backtrace was:
#0 xen_m2p (machine=973135872) at kernel.c:9714
#1 0x000000000053ed4a in x86_64_pud_offset (pgd_pte=3299508019303,
vaddr=18446744072642223552, verbose=0, IS_XEN=1)
at x86_64.c:1889
#2 0x0000000000540c8a in x86_64_kvtop_xen_wpt (tc=0x0, kvaddr=18446744072642223552,
paddr=0x7fffffffdb30, verbose=0)
at x86_64.c:2523
#3 0x00000000005407d0 in x86_64_kvtop (tc=0x0, kvaddr=18446744072642223552,
paddr=0x7fffffffdb30, verbose=0)
at x86_64.c:2413
#4 0x0000000000491a97 in kvtop (tc=0x0, kvaddr=18446744072642223552,
paddr=0x7fffffffdb30, verbose=0) at memory.c:3062
#5 0x000000000048f3f0 in readmem (addr=18446744072642223552, memtype=1, buffer=0xba63a0
<shared_bufs>, size=832,
type=0x92e1f2 "module struct", error_handle=6) at memory.c:2314
#6 0x00000000005071e2 in module_init () at kernel.c:3699
....
I tracked the problem to a wrong value of
__PHYSICAL_MASK_SHIFT_XEN. The current value of 40 does not
correspond to the current kernel value of 52 since kernel commit
6f0e8bf16730a36ff6773802d8c8df56d10e60cd (xen: support 52 bit
physical addresses in pv guests).
The result is visible in the above backtrace:
x86_64_pud_offset() is called with pgd_pte=0x3003a00e067 and that
value is wrongly masked by "pud_paddr = pgd_pte &
PHYSICAL_PAGE_MASK" and passed to xen_m2p() as 0x3a00e000 instead
of 0x3003a00e000, causing the m2p translation to fail.
Thank you for sharing the details.
Setting __PHYSICAL_MASK_SHIFT_XEN to 52 fixes the problem with
this dump.
But I am not confident it's a safe change. My understanding is
that it should be safe, as all the unused bits of the physical
address inside the PTEs should be 0 and thus having the mask wider
than necessary should not hurt. But I am suspicious if my
reasoning is correct. Why does crash go into such trouble
differentiating between different kernels and sets
machdep->machspec->physical_mask_shift dynamically to one of
__PHYSICAL_MASK_SHIFT_XEN (40), __PHYSICAL_MASK_SHIFT_2_6 (46),
The reason is that the supported kernel virtual address allows to
be configured on other architectures, and that still relies on the
hardware capability.
and __PHYSICAL_MASK_SHIFT_5LEVEL (52)? Would something break if
it were always set to 52? The commit adding the logic is
307e7f35.
It's time to update the physical mask for xen, but can you help to
test the address translation via some command such as vtop, ptov,
etc? Or follow up the 5-level page changes to check if it has the
capability(cr4 & CR4_LA57)?
Thanks.
Lianbo
Thanks,
-- Jiri Bohac <jbohac(a)suse.cz> SUSE Labs, Prague, Czechia
6:32 a.m.
New subject: increase __PHYSICAL_MASK_SHIFT_XEN?
Hi Lianbo,
It's time to update the physical mask for xen, but can you help
to
test the address translation via some command such as vtop, ptov,
etc?
you mean with that dump where crash previously failed? Sure. This is the patch
I applied on top of the current git:
--- a/defs.h
+++ b/defs.h
@@ -3583,7 +3583,7 @@ struct arm64_stackframe {
* PHYSICAL_PAGE_MASK changed (enlarged) between 2.4 and 2.6, so
* for safety, use the 2.6 values to generate it.
*/
-#define __PHYSICAL_MASK_SHIFT_XEN 40
+#define __PHYSICAL_MASK_SHIFT_XEN 52
#define __PHYSICAL_MASK_SHIFT_2_6 46
#define __PHYSICAL_MASK_SHIFT_5LEVEL 52
#define __PHYSICAL_MASK_SHIFT (machdep->machspec->physical_mask_shift)
I can open the dump now:
$./crash vmcore vmlinux-4.12.14-122.32-default vmlinux-4.12.14-122.32-default.debug
crash 7.2.9++
Copyright (C) 2002-2020 Red Hat, Inc.
...
KERNEL: vmlinux-4.12.14-122.32-default
DEBUG KERNEL: vmlinux-4.12.14-122.32-default.debug
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 8 [OFFLINE: 7]
DATE: Wed Sep 9 10:16:38 CEST 2020
UPTIME: 00:28:41
LOAD AVERAGE: 0.03, 0.05, 0.07
TASKS: 228
NODENAME: rsa1419
RELEASE: 4.12.14-122.32-default
VERSION: #1 SMP Wed Aug 5 12:59:08 UTC 2020 (477c426)
MACHINE: x86_64 (2194 Mhz)
MEMORY: 16 GB
PANIC: "sysrq: SysRq : Trigger a crash"
PID: 7401
COMMAND: "bash"
TASK: ffff8804590bc200 [THREAD_INFO: ffff8804590bc200]
CPU: 4
STATE: TASK_RUNNING (SYSRQ)
crash> p &jiffies
$2 = (volatile unsigned long *) 0xffffffff82005000 <jiffies_64>
crash> vtop 0xffffffff82005000
VIRTUAL PHYSICAL
ffffffff82005000 2005000
PGD DIRECTORY: ffffffff8200a000
PAGE DIRECTORY: 3003a00e067 [machine]
PGD: 200e000
PUD: 200eff0 => 3003a00f067 [machine]
PUD: 200f000
PMD: 200f080 => 3003b2d5067 [machine]
PMD: 32d5000
PTE: 32d5028 => 801003003a005067 [machine]
PTE: 2005067
PAGE: 3003a005000 [machine]
PAGE: 2005000
PTE PHYSICAL FLAGS
2005067 2005000 (PRESENT|RW|USER|ACCESSED|DIRTY)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea0000080140 2005000 0 0 1 fffffc0000800 reserved
crash> ptov 2005000
VIRTUAL PHYSICAL
ffff880002005000 2005000
Or follow up the 5-level page changes to check if it has the
capability(cr4 & CR4_LA57)?
sorry, I don't understand what you mean; the crashed linux dump contains an oops that
shows the CR4 value as
0000000000040660, i.e. the LA57 bit is not set.
Let me know if you want me to try something else.
Thanks,
--
Jiri Bohac <jbohac(a)suse.cz>
SUSE Labs, Prague, Czechia
4:33 a.m.
New subject: increase __PHYSICAL_MASK_SHIFT_XEN?
Hi, Jiri
在 2021年01月22日 20:32, Jiri Bohac 写道:
Hi Lianbo,
> It's time to update the physical mask for xen, but can you help to
> test the address translation via some command such as vtop, ptov,
> etc?
you mean with that dump where crash previously failed? Sure. This is the patch
Sorry about this.
I would suggest to do some tests with your vmcore via these commands: vtop and ptov,
maybe that can help to check if the the address translation can work as we expected
after applied this patch.
I applied on top of the current git:
--- a/defs.h
+++ b/defs.h
@@ -3583,7 +3583,7 @@ struct arm64_stackframe {
* PHYSICAL_PAGE_MASK changed (enlarged) between 2.4 and 2.6, so
* for safety, use the 2.6 values to generate it.
*/
-#define __PHYSICAL_MASK_SHIFT_XEN 40
+#define __PHYSICAL_MASK_SHIFT_XEN 52
#define __PHYSICAL_MASK_SHIFT_2_6 46
#define __PHYSICAL_MASK_SHIFT_5LEVEL 52
#define __PHYSICAL_MASK_SHIFT (machdep->machspec->physical_mask_shift)
I can open the dump now:
$./crash vmcore vmlinux-4.12.14-122.32-default vmlinux-4.12.14-122.32-default.debug
crash 7.2.9++
Copyright (C) 2002-2020 Red Hat, Inc.
...
KERNEL: vmlinux-4.12.14-122.32-default
DEBUG KERNEL: vmlinux-4.12.14-122.32-default.debug
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 8 [OFFLINE: 7]
DATE: Wed Sep 9 10:16:38 CEST 2020
UPTIME: 00:28:41
LOAD AVERAGE: 0.03, 0.05, 0.07
TASKS: 228
NODENAME: rsa1419
RELEASE: 4.12.14-122.32-default
VERSION: #1 SMP Wed Aug 5 12:59:08 UTC 2020 (477c426)
MACHINE: x86_64 (2194 Mhz)
MEMORY: 16 GB
PANIC: "sysrq: SysRq : Trigger a crash"
PID: 7401
COMMAND: "bash"
TASK: ffff8804590bc200 [THREAD_INFO: ffff8804590bc200]
CPU: 4
STATE: TASK_RUNNING (SYSRQ)
crash> p &jiffies
$2 = (volatile unsigned long *) 0xffffffff82005000 <jiffies_64>
crash> vtop 0xffffffff82005000
VIRTUAL PHYSICAL
ffffffff82005000 2005000
PGD DIRECTORY: ffffffff8200a000
PAGE DIRECTORY: 3003a00e067 [machine]
PGD: 200e000
PUD: 200eff0 => 3003a00f067 [machine]
PUD: 200f000
PMD: 200f080 => 3003b2d5067 [machine]
PMD: 32d5000
PTE: 32d5028 => 801003003a005067 [machine]
PTE: 2005067
PAGE: 3003a005000 [machine]
PAGE: 2005000
PTE PHYSICAL FLAGS
2005067 2005000 (PRESENT|RW|USER|ACCESSED|DIRTY)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea0000080140 2005000 0 0 1 fffffc0000800 reserved
crash> ptov 2005000
VIRTUAL PHYSICAL
ffff880002005000 2005000
> Or follow up the 5-level page changes to check if it has the
> capability(cr4 & CR4_LA57)?
sorry, I don't understand what you mean; the crashed linux dump contains an oops that
shows the CR4 value as
0000000000040660, i.e. the LA57 bit is not set.
Thanks for the feedback. The guest itself may not support the 5-level paging for your
situation according to this
commit <6f0e8bf16730>.
Let me know if you want me to try something else.
No other issue for me, only several commands tests mentioned above.
Thank you for the help.
Lianbo
Thanks,
4:49 a.m.
New subject: increase __PHYSICAL_MASK_SHIFT_XEN?
在 2021年01月25日 18:33, lijiang 写道:
Hi, Jiri
在 2021年01月22日 20:32, Jiri Bohac 写道:
> Hi Lianbo,
>
>> It's time to update the physical mask for xen, but can you help to
>> test the address translation via some command such as vtop, ptov,
>> etc?
>
> you mean with that dump where crash previously failed? Sure. This is the patch
Sorry about this.
I would suggest to do some tests with your vmcore via these commands: vtop and ptov,
maybe that can help to check if the the address translation can work as we expected
after applied this patch.
> I applied on top of the current git:
>
> --- a/defs.h
> +++ b/defs.h
> @@ -3583,7 +3583,7 @@ struct arm64_stackframe {
> * PHYSICAL_PAGE_MASK changed (enlarged) between 2.4 and 2.6, so
> * for safety, use the 2.6 values to generate it.
> */
> -#define __PHYSICAL_MASK_SHIFT_XEN 40
> +#define __PHYSICAL_MASK_SHIFT_XEN 52
> #define __PHYSICAL_MASK_SHIFT_2_6 46
> #define __PHYSICAL_MASK_SHIFT_5LEVEL 52
> #define __PHYSICAL_MASK_SHIFT (machdep->machspec->physical_mask_shift)
>
>
> I can open the dump now:
>
> $./crash vmcore vmlinux-4.12.14-122.32-default vmlinux-4.12.14-122.32-default.debug
> crash 7.2.9++
> Copyright (C) 2002-2020 Red Hat, Inc.
> ...
> KERNEL: vmlinux-4.12.14-122.32-default
> DEBUG KERNEL: vmlinux-4.12.14-122.32-default.debug
> DUMPFILE: vmcore [PARTIAL DUMP]
> CPUS: 8 [OFFLINE: 7]
> DATE: Wed Sep 9 10:16:38 CEST 2020
> UPTIME: 00:28:41
> LOAD AVERAGE: 0.03, 0.05, 0.07
> TASKS: 228
> NODENAME: rsa1419
> RELEASE: 4.12.14-122.32-default
> VERSION: #1 SMP Wed Aug 5 12:59:08 UTC 2020 (477c426)
> MACHINE: x86_64 (2194 Mhz)
> MEMORY: 16 GB
> PANIC: "sysrq: SysRq : Trigger a crash"
> PID: 7401
> COMMAND: "bash"
> TASK: ffff8804590bc200 [THREAD_INFO: ffff8804590bc200]
> CPU: 4
> STATE: TASK_RUNNING (SYSRQ)
>
> crash> p &jiffies
> $2 = (volatile unsigned long *) 0xffffffff82005000 <jiffies_64>
> crash> vtop 0xffffffff82005000
> VIRTUAL PHYSICAL
> ffffffff82005000 2005000
>
> PGD DIRECTORY: ffffffff8200a000
> PAGE DIRECTORY: 3003a00e067 [machine]
> PGD: 200e000
> PUD: 200eff0 => 3003a00f067 [machine]
> PUD: 200f000
> PMD: 200f080 => 3003b2d5067 [machine]
> PMD: 32d5000
> PTE: 32d5028 => 801003003a005067 [machine]
> PTE: 2005067
> PAGE: 3003a005000 [machine]
> PAGE: 2005000
>
> PTE PHYSICAL FLAGS
> 2005067 2005000 (PRESENT|RW|USER|ACCESSED|DIRTY)
>
> PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> ffffea0000080140 2005000 0 0 1 fffffc0000800 reserved
> crash> ptov 2005000
> VIRTUAL PHYSICAL
> ffff880002005000 2005000
>
OK, I just saw the above vtop and ptov commands tests, please ignore the last reply.
Thank you for the feedback, Jiri. Can you post a formal patch?
Lianbo.
>
>> Or follow up the 5-level page changes to check if it has the
>> capability(cr4 & CR4_LA57)?
>
> sorry, I don't understand what you mean; the crashed linux dump contains an oops
that shows the CR4 value as
> 0000000000040660, i.e. the LA57 bit is not set.
>
Thanks for the feedback. The guest itself may not support the 5-level paging for your
situation according to this
commit <6f0e8bf16730>.
> Let me know if you want me to try something else.
>
No other issue for me, only several commands tests mentioned above.
Thank you for the help.
Lianbo
> Thanks,
>
3:44 p.m.
New subject: [PATCH] increase __PHYSICAL_MASK_SHIFT_XEN
The current value of __PHYSICAL_MASK_SHIFT_XEN in crash (40) is
smaller than the kernel (52) since kernel commit 6f0e8bf167 (xen:
support 52 bit physical addresses in pv guests).
This can cause x86_64_pud_offset() to lose the most significant
bits of pgd_pte, leading to a failed xen_m2p() translation,
resulting in crash failing with an error message like this:
crash: read error: physical address: ffffffffffffffff type: "pud page"
Both Intel and AMD documentation mandate that unused physical
address bits must be 0, so there is no need to explicitly mask them
out with a mask narrower than the architecture limit of 52. This
is also confirmed by this kernel commit: b83ce5ee91.
Increase the value of __PHYSICAL_MASK_SHIFT_XEN to 52.
Signed-off-by: Jiri Bohac <jbohac(a)suse.cz>
diff --git a/defs.h b/defs.h
index e468b1d..848c1f6 100644
--- a/defs.h
+++ b/defs.h
@@ -3583,7 +3583,7 @@ struct arm64_stackframe {
* PHYSICAL_PAGE_MASK changed (enlarged) between 2.4 and 2.6, so
* for safety, use the 2.6 values to generate it.
*/
-#define __PHYSICAL_MASK_SHIFT_XEN 40
+#define __PHYSICAL_MASK_SHIFT_XEN 52
#define __PHYSICAL_MASK_SHIFT_2_6 46
#define __PHYSICAL_MASK_SHIFT_5LEVEL 52
#define __PHYSICAL_MASK_SHIFT (machdep->machspec->physical_mask_shift)
--
Jiri Bohac <jbohac(a)suse.cz>
SUSE Labs, Prague, Czechia
8:23 p.m.
New subject: [PATCH] increase __PHYSICAL_MASK_SHIFT_XEN
-----Original Message-----
The current value of __PHYSICAL_MASK_SHIFT_XEN in crash (40) is
smaller than the kernel (52) since kernel commit 6f0e8bf167 (xen:
support 52 bit physical addresses in pv guests).
This can cause x86_64_pud_offset() to lose the most significant
bits of pgd_pte, leading to a failed xen_m2p() translation,
resulting in crash failing with an error message like this:
crash: read error: physical address: ffffffffffffffff type: "pud page"
Both Intel and AMD documentation mandate that unused physical
address bits must be 0, so there is no need to explicitly mask them
out with a mask narrower than the architecture limit of 52. This
is also confirmed by this kernel commit: b83ce5ee91.
Increase the value of __PHYSICAL_MASK_SHIFT_XEN to 52.
Signed-off-by: Jiri Bohac <jbohac(a)suse.cz>
Looks good and tested ok with several old xen dumpfiles on hand.
Acked-by: Kazuhito Hagio <k-hagio-ab(a)nec.com>
Thanks,
Kazu
diff --git a/defs.h b/defs.h
index e468b1d..848c1f6 100644
--- a/defs.h
+++ b/defs.h
@@ -3583,7 +3583,7 @@ struct arm64_stackframe {
* PHYSICAL_PAGE_MASK changed (enlarged) between 2.4 and 2.6, so
* for safety, use the 2.6 values to generate it.
*/
-#define __PHYSICAL_MASK_SHIFT_XEN 40
+#define __PHYSICAL_MASK_SHIFT_XEN 52
#define __PHYSICAL_MASK_SHIFT_2_6 46
#define __PHYSICAL_MASK_SHIFT_5LEVEL 52
#define __PHYSICAL_MASK_SHIFT (machdep->machspec->physical_mask_shift)
--
Jiri Bohac <jbohac(a)suse.cz>
SUSE Labs, Prague, Czechia
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
1388
days inactive
1392
days old
devel@lists.crash-utility.osci.io
5 comments
3 participants
participants (3)
-
HAGIO KAZUHITO(萩尾 一仁) -
Jiri Bohac -
lijiang