4:12 a.m.
Changelog:
V1-->V2:
-Fix the backwards compatibility issues suggested by Dave
-Test with "mod","kmem -f","vm -p" command again.
I found Dave had alread done the first phase of future support for x86_64
5-level page tables(commit 307e7f35f510). when I asked him about the
state of this work, he gave me a more detailed answer and suggestion.
I follow his advice, and do the following job.
1. Refine the original logical:
1) Create some new common function for getting the offset of page table
2) Repace the PML4 and UPML with the common PGD:
machdep->machspec->pml4/upml ==> machdep->pdg
3) Using the PUD in x86_64
2. Add 5-level page tables support for x86_64_k/uvtop()
This patchset is the second phase of the work, As Dave said, we need to be
a manner of determining very early on whether the kernel page tables are
using 5-level and whether each user-space task is using 4- or 5-level page
tables. These will be done after this phase.
About test work:
I have tested this patchset with 4-level and 5-level paging table.
sadump are not be tested.
Dou Liyang (6):
defs.h: Fix the PHYSICAL_PAGE_MASK macro
x86_64: Extract public page table mapping code
x86_64: Unify the page table parsing for 4-level mode
x86_64.c: Add the VMEMMAP support for 5 level page table
x86_64: Add 5-level paging support for x86_64_k/uvtop()
x86_64: Fix the misusage of PGDIR_SHIFT
defs.h | 58 ++---
sadump.c | 7 +-
x86_64.c | 732 ++++++++++++++++++++++++++++++++-------------------------------
3 files changed, 399 insertions(+), 398 deletions(-)
--
2.14.3
4:12 a.m.
New subject: [PATCH v2 1/6] defs.h: Fix the PHYSICAL_PAGE_MASK macro
Original crash defines the PHYSICAL_PAGE_MASK as
(~(PAGE_SIZE-1) & (__PHYSICAL_MASK << PAGE_SHIFT))
It moves left PAGE_SHIFT bits for safety, But As crash expands the physical
bits to 52, this will cause the conversion of phythcal address to virtual
address failed.
Remove the PAGE_SHIFT to fix this problem.
Signed-off-by: Dou Liyang <douly.fnst(a)cn.fujitsu.com>
---
defs.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/defs.h b/defs.h
index dcd6c26..8082d6c 100644
--- a/defs.h
+++ b/defs.h
@@ -3403,7 +3403,7 @@ struct arm64_stackframe {
#define __VIRTUAL_MASK ((1UL << __VIRTUAL_MASK_SHIFT) - 1)
#define PAGE_SHIFT 12
#define PAGE_SIZE (1UL << PAGE_SHIFT)
-#define PHYSICAL_PAGE_MASK (~(PAGE_SIZE-1) & (__PHYSICAL_MASK <<
PAGE_SHIFT))
+#define PHYSICAL_PAGE_MASK (~(PAGE_SIZE-1) & __PHYSICAL_MASK )
#define _PAGE_BIT_NX 63
#define _PAGE_PRESENT 0x001
--
2.14.3
4:12 a.m.
New subject: [PATCH v2 2/6] x86_64: Extract public page table mapping code
Currently, page table mapping code is scattered everywhere. So Extract public
page table mapping code, Prepare for 5-level page
No functionality change
Signed-off-by: Dou Liyang <douly.fnst(a)cn.fujitsu.com>
---
x86_64.c | 410 ++++++++++++++++++++++++++++++---------------------------------
1 file changed, 193 insertions(+), 217 deletions(-)
diff --git a/x86_64.c b/x86_64.c
index d8fade4..7134b4b 100644
--- a/x86_64.c
+++ b/x86_64.c
@@ -78,6 +78,11 @@ static void x86_64_calc_phys_base(void);
static int x86_64_is_module_addr(ulong);
static int x86_64_is_kvaddr(ulong);
static int x86_64_is_uvaddr(ulong, struct task_context *);
+static ulong *x86_64_kpgd_offset(ulong, int, int);
+static ulong x86_64_upgd_offset(struct task_context *, ulong, int, int);
+static ulong x86_64_pud_offset(ulong, ulong, int, int);
+static ulong x86_64_pmd_offset(ulong, ulong, int, int);
+static ulong x86_64_pte_offset(ulong, ulong, int, int);
void x86_64_compiler_warning_stub(void);
static void x86_64_init_kernel_pgd(void);
static void x86_64_cpu_pda_init(void);
@@ -1557,6 +1562,153 @@ x86_64_is_uvaddr(ulong addr, struct task_context *tc)
return (addr < USERSPACE_TOP);
}
+/*
+ * Find the kernel pgd entry..
+ * pgd = pgd_offset_k(addr);
+ */
+static ulong *
+x86_64_kpgd_offset(ulong kvaddr, int verbose , int IS_XEN)
+{
+ ulong *pgd;
+
+ FILL_PML4();
+ pgd = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
+ if (verbose) {
+ fprintf(fp, "PGD DIRECTORY: %lx\n", vt->kernel_pgd[0]);
+ if(IS_XEN)
+ fprintf(fp, "PAGE DIRECTORY: %lx [machine]\n", *pgd);
+ else
+ fprintf(fp, "PAGE DIRECTORY: %lx\n", *pgd);
+ }
+
+ return pgd;
+}
+
+/*
+ * Find the user pgd entry..
+ * pgd = pgd_offset(mm, address);
+ */
+static ulong
+x86_64_upgd_offset(struct task_context *tc, ulong uvaddr, int verbose , int IS_XEN)
+{
+ ulong *pgd;
+ ulong pgd_paddr;
+ ulong pgd_pte;
+
+ if (task_mm(tc->task, TRUE))
+ pgd = ULONG_PTR(tt->mm_struct + OFFSET(mm_struct_pgd));
+ else
+ readmem(tc->mm_struct + OFFSET(mm_struct_pgd), KVADDR, &pgd,
+ sizeof(long), "mm_struct pgd", FAULT_ON_ERROR);
+
+ pgd_paddr = x86_64_VTOP((ulong)pgd);
+ FILL_UPML(pgd_paddr, PHYSADDR, PAGESIZE());
+ pgd = ((ulong *)pgd_paddr) + pml4_index(uvaddr);
+ pgd_pte = ULONG(machdep->machspec->pml4 + PAGEOFFSET(pgd));
+ if (verbose) {
+ if(IS_XEN)
+ fprintf(fp, " PGD: %lx => %lx [machine]\n", (ulong)pgd, pgd_pte);
+ else
+ fprintf(fp, " PGD: %lx => %lx\n", (ulong)pgd, pgd_pte);
+ }
+
+ return pgd_pte;
+}
+
+/*
+ * Find an entry in the third-level page table..
+ * pud = pud_offset(pgd, address);
+ */
+static ulong
+x86_64_pud_offset(ulong pgd_pte, ulong vaddr, int verbose , int IS_XEN)
+{
+ ulong *pud;
+ ulong pud_paddr;
+ ulong pud_pte;
+
+ pud_paddr = pgd_pte & PHYSICAL_PAGE_MASK;
+
+ if(IS_XEN) {
+ pud_paddr = xen_m2p(pud_paddr);
+ if (verbose)
+ fprintf(fp, " PGD: %lx\n", pud_paddr);
+ }
+
+ FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
+ pud = ((ulong *)pud_paddr) + pgd_index(vaddr);
+ pud_pte = ULONG(machdep->pud + PAGEOFFSET(pud));
+ if (verbose) {
+ if(IS_XEN)
+ fprintf(fp, " PUD: %lx => %lx [machine]\n", (ulong)pud, pud_pte);
+ else
+ fprintf(fp, " PUD: %lx => %lx\n", (ulong)pud, pud_pte);
+ }
+
+ return pud_pte;
+}
+
+/*
+ * Find an entry in the middle page table..
+ * pmd = pmd_offset(pud, address);
+ */
+static ulong
+x86_64_pmd_offset(ulong pud_pte, ulong vaddr, int verbose , int IS_XEN)
+{
+ ulong *pmd;
+ ulong pmd_paddr;
+ ulong pmd_pte;
+
+ pmd_paddr = pud_pte & PHYSICAL_PAGE_MASK;
+
+ if(IS_XEN) {
+ pmd_paddr = xen_m2p(pmd_paddr);
+ if (verbose)
+ fprintf(fp, " PUD: %lx\n", pmd_paddr);
+ }
+
+ FILL_PMD(pmd_paddr, PHYSADDR, PAGESIZE());
+ pmd = ((ulong *)pmd_paddr) + pmd_index(vaddr);
+ pmd_pte = ULONG(machdep->pmd + PAGEOFFSET(pmd));
+ if (verbose) {
+ if(IS_XEN)
+ fprintf(fp, " PMD: %lx => %lx [machine]\n", (ulong)pmd, pmd_pte);
+ else
+ fprintf(fp, " PMD: %lx => %lx\n", (ulong)pmd, pmd_pte);
+ }
+ return pmd_pte;
+}
+
+/*
+ * Find an entry in the pet page table..
+ * pmd = pmd_offset(pud, address);
+ */
+static ulong
+x86_64_pte_offset(ulong pmd_pte, ulong vaddr, int verbose , int IS_XEN)
+{
+ ulong *ptep;
+ ulong pte_paddr;
+ ulong pte;
+
+ pte_paddr = pmd_pte & PHYSICAL_PAGE_MASK;
+
+ if(IS_XEN) {
+ pte_paddr = xen_m2p(pte_paddr);
+ if (verbose)
+ fprintf(fp, " PMD: %lx\n", pte_paddr);
+ }
+
+ FILL_PTBL(pte_paddr, PHYSADDR, PAGESIZE());
+ ptep = ((ulong *)pte_paddr) + pte_index(vaddr);
+ pte = ULONG(machdep->ptbl + PAGEOFFSET(ptep));
+ if (verbose) {
+ if(IS_XEN)
+ fprintf(fp, " PTE: %lx => %lx [machine]\n", (ulong)ptep, pte);
+ else
+ fprintf(fp, " PTE: %lx => %lx\n", (ulong)ptep, pte);
+ }
+
+ return pte;
+}
/*
* Translates a user virtual address to its physical address. cmd_vtop()
@@ -1570,18 +1722,9 @@ x86_64_is_uvaddr(ulong addr, struct task_context *tc)
static int
x86_64_uvtop_level4(struct task_context *tc, ulong uvaddr, physaddr_t *paddr, int
verbose)
{
- ulong mm;
- ulong *pml;
- ulong pml_paddr;
- ulong pml_pte;
- ulong *pgd;
- ulong pgd_paddr;
ulong pgd_pte;
- ulong *pmd;
- ulong pmd_paddr;
+ ulong pud_pte;
ulong pmd_pte;
- ulong *ptep;
- ulong pte_paddr;
ulong pte;
physaddr_t physpage;
@@ -1596,43 +1739,25 @@ x86_64_uvtop_level4(struct task_context *tc, ulong uvaddr,
physaddr_t *paddr, in
if ((machdep->flags & VM_5LEVEL) && x86_64_task_uses_5level(tc))
return x86_64_uvtop_5level(tc, uvaddr, paddr, verbose);
- if ((mm = task_mm(tc->task, TRUE)))
- pml = ULONG_PTR(tt->mm_struct + OFFSET(mm_struct_pgd));
- else
- readmem(tc->mm_struct + OFFSET(mm_struct_pgd), KVADDR, &pml,
- sizeof(long), "mm_struct pgd", FAULT_ON_ERROR);
-
- pml_paddr = x86_64_VTOP((ulong)pml);
- FILL_UPML(pml_paddr, PHYSADDR, PAGESIZE());
- pml = ((ulong *)pml_paddr) + pml4_index(uvaddr);
- pml_pte = ULONG(machdep->machspec->upml + PAGEOFFSET(pml));
- if (verbose)
- fprintf(fp, " PML: %lx => %lx\n", (ulong)pml, pml_pte);
- if (!(pml_pte & _PAGE_PRESENT))
+ pgd_pte = x86_64_upgd_offset(tc, uvaddr, verbose, FALSE);
+ if (!(pgd_pte & _PAGE_PRESENT))
goto no_upage;
- pgd_paddr = pml_pte & PHYSICAL_PAGE_MASK;
- FILL_PGD(pgd_paddr, PHYSADDR, PAGESIZE());
- pgd = ((ulong *)pgd_paddr) + pgd_index(uvaddr);
- pgd_pte = ULONG(machdep->pgd + PAGEOFFSET(pgd));
- if (verbose)
- fprintf(fp, " PUD: %lx => %lx\n", (ulong)pgd, pgd_pte);
- if (!(pgd_pte & _PAGE_PRESENT))
+ /*
+ * pud = pud_offset(pgd, address);
+ */
+ pud_pte = x86_64_pud_offset(pgd_pte, uvaddr, verbose, FALSE);
+ if (!(pud_pte & _PAGE_PRESENT))
goto no_upage;
/*
- * pmd = pmd_offset(pgd, address);
+ * pmd = pmd_offset(pud, address);
*/
- pmd_paddr = pgd_pte & PHYSICAL_PAGE_MASK;
- FILL_PMD(pmd_paddr, PHYSADDR, PAGESIZE());
- pmd = ((ulong *)pmd_paddr) + pmd_index(uvaddr);
- pmd_pte = ULONG(machdep->pmd + PAGEOFFSET(pmd));
- if (verbose)
- fprintf(fp, " PMD: %lx => %lx\n", (ulong)pmd, pmd_pte);
+ pmd_pte = x86_64_pmd_offset(pud_pte, uvaddr, verbose, FALSE);
if (!(pmd_pte & (_PAGE_PRESENT | _PAGE_PROTNONE)))
goto no_upage;
if (pmd_pte & _PAGE_PSE) {
- if (verbose) {
+ if (verbose) {
fprintf(fp, " PAGE: %lx (2MB)\n\n",
PAGEBASE(pmd_pte) & PHYSICAL_PAGE_MASK);
x86_64_translate_pte(pmd_pte, 0, 0);
@@ -1648,13 +1773,7 @@ x86_64_uvtop_level4(struct task_context *tc, ulong uvaddr,
physaddr_t *paddr, in
* ptep = pte_offset_map(pmd, address);
* pte = *ptep;
*/
- pte_paddr = pmd_pte & PHYSICAL_PAGE_MASK;
- FILL_PTBL(pte_paddr, PHYSADDR, PAGESIZE());
- ptep = ((ulong *)pte_paddr) + pte_index(uvaddr);
- pte = ULONG(machdep->ptbl + PAGEOFFSET(ptep));
- if (verbose)
- fprintf(fp, " PTE: %lx => %lx\n", (ulong)ptep, pte);
-
+ pte = x86_64_pte_offset(pmd_pte, uvaddr, verbose, FALSE);
if (!(pte & (_PAGE_PRESENT | _PAGE_PROTNONE))) {
*paddr = pte;
@@ -1696,19 +1815,10 @@ x86_64_uvtop_5level(struct task_context *tc, ulong uvaddr,
physaddr_t *paddr, in
static int
x86_64_uvtop_level4_xen_wpt(struct task_context *tc, ulong uvaddr, physaddr_t *paddr, int
verbose)
{
- ulong mm;
- ulong *pml;
- ulong pml_paddr;
- ulong pml_pte;
- ulong *pgd;
- ulong pgd_paddr;
ulong pgd_pte;
- ulong *pmd;
- ulong pmd_paddr;
+ ulong pud_pte;
ulong pmd_pte;
ulong pseudo_pmd_pte;
- ulong *ptep;
- ulong pte_paddr;
ulong pte;
ulong pseudo_pte;
physaddr_t physpage;
@@ -1722,45 +1832,18 @@ x86_64_uvtop_level4_xen_wpt(struct task_context *tc, ulong uvaddr,
physaddr_t *p
if (IS_KVADDR(uvaddr))
return x86_64_kvtop(tc, uvaddr, paddr, verbose);
- if ((mm = task_mm(tc->task, TRUE)))
- pml = ULONG_PTR(tt->mm_struct + OFFSET(mm_struct_pgd));
- else
- readmem(tc->mm_struct + OFFSET(mm_struct_pgd), KVADDR, &pml,
- sizeof(long), "mm_struct pgd", FAULT_ON_ERROR);
-
- pml_paddr = x86_64_VTOP((ulong)pml);
- FILL_UPML(pml_paddr, PHYSADDR, PAGESIZE());
- pml = ((ulong *)pml_paddr) + pml4_index(uvaddr);
- pml_pte = ULONG(machdep->machspec->upml + PAGEOFFSET(pml));
- if (verbose)
- fprintf(fp, " PML: %lx => %lx [machine]\n", (ulong)pml, pml_pte);
- if (!(pml_pte & _PAGE_PRESENT))
+ pgd_pte = x86_64_upgd_offset(tc, uvaddr, verbose, TRUE);
+ if (!(pgd_pte & _PAGE_PRESENT))
goto no_upage;
- pgd_paddr = pml_pte & PHYSICAL_PAGE_MASK;
- pgd_paddr = xen_m2p(pgd_paddr);
- if (verbose)
- fprintf(fp, " PML: %lx\n", pgd_paddr);
- FILL_PGD(pgd_paddr, PHYSADDR, PAGESIZE());
- pgd = ((ulong *)pgd_paddr) + pgd_index(uvaddr);
- pgd_pte = ULONG(machdep->pgd + PAGEOFFSET(pgd));
- if (verbose)
- fprintf(fp, " PUD: %lx => %lx [machine]\n", (ulong)pgd,
pgd_pte);
- if (!(pgd_pte & _PAGE_PRESENT))
+ pud_pte = x86_64_pud_offset(pgd_pte, uvaddr, verbose, TRUE);
+ if (!(pud_pte & _PAGE_PRESENT))
goto no_upage;
/*
- * pmd = pmd_offset(pgd, address);
+ * pmd = pmd_offset(pud, address);
*/
- pmd_paddr = pgd_pte & PHYSICAL_PAGE_MASK;
- pmd_paddr = xen_m2p(pmd_paddr);
- if (verbose)
- fprintf(fp, " PUD: %lx\n", pmd_paddr);
- FILL_PMD(pmd_paddr, PHYSADDR, PAGESIZE());
- pmd = ((ulong *)pmd_paddr) + pmd_index(uvaddr);
- pmd_pte = ULONG(machdep->pmd + PAGEOFFSET(pmd));
- if (verbose)
- fprintf(fp, " PMD: %lx => %lx [machine]\n", (ulong)pmd,
pmd_pte);
+ pmd_pte = x86_64_pmd_offset(pud_pte, uvaddr, verbose, TRUE);
if (!(pmd_pte & _PAGE_PRESENT))
goto no_upage;
if (pmd_pte & _PAGE_PSE) {
@@ -1799,15 +1882,7 @@ x86_64_uvtop_level4_xen_wpt(struct task_context *tc, ulong uvaddr,
physaddr_t *p
* ptep = pte_offset_map(pmd, address);
* pte = *ptep;
*/
- pte_paddr = pmd_pte & PHYSICAL_PAGE_MASK;
- pte_paddr = xen_m2p(pte_paddr);
- if (verbose)
- fprintf(fp, " PMD: %lx\n", pte_paddr);
- FILL_PTBL(pte_paddr, PHYSADDR, PAGESIZE());
- ptep = ((ulong *)pte_paddr) + pte_index(uvaddr);
- pte = ULONG(machdep->ptbl + PAGEOFFSET(ptep));
- if (verbose)
- fprintf(fp, " PTE: %lx => %lx [machine]\n", (ulong)ptep, pte);
+ pte = x86_64_pte_offset(pmd_pte, uvaddr, verbose, TRUE);
if (!(pte & (_PAGE_PRESENT))) {
*paddr = pte;
@@ -1817,7 +1892,7 @@ x86_64_uvtop_level4_xen_wpt(struct task_context *tc, ulong uvaddr,
physaddr_t *p
}
goto no_upage;
}
-
+
pseudo_pte = xen_m2p(pte & PHYSICAL_PAGE_MASK);
if (verbose)
fprintf(fp, " PTE: %lx\n", pseudo_pte + PAGEOFFSET(pte));
@@ -1846,12 +1921,8 @@ x86_64_uvtop_level4_rhel4_xen_wpt(struct task_context *tc, ulong
uvaddr, physadd
ulong *pgd;
ulong pgd_paddr;
ulong pgd_pte;
- ulong *pmd;
- ulong pmd_paddr;
ulong pmd_pte;
ulong pseudo_pmd_pte;
- ulong *ptep;
- ulong pte_paddr;
ulong pte;
ulong pseudo_pte;
physaddr_t physpage;
@@ -1883,15 +1954,7 @@ x86_64_uvtop_level4_rhel4_xen_wpt(struct task_context *tc, ulong
uvaddr, physadd
/*
* pmd = pmd_offset(pgd, address);
*/
- pmd_paddr = pgd_pte & PHYSICAL_PAGE_MASK;
- pmd_paddr = xen_m2p(pmd_paddr);
- if (verbose)
- fprintf(fp, " PGD: %lx\n", pmd_paddr);
- FILL_PMD(pmd_paddr, PHYSADDR, PAGESIZE());
- pmd = ((ulong *)pmd_paddr) + pmd_index(uvaddr);
- pmd_pte = ULONG(machdep->pmd + PAGEOFFSET(pmd));
- if (verbose)
- fprintf(fp, " PMD: %lx => %lx [machine]\n", (ulong)pmd,
pmd_pte);
+ pmd_pte = x86_64_pmd_offset(pgd_pte, uvaddr, verbose, TRUE);
if (!(pmd_pte & _PAGE_PRESENT))
goto no_upage;
if (pmd_pte & _PAGE_PSE) {
@@ -1930,15 +1993,7 @@ x86_64_uvtop_level4_rhel4_xen_wpt(struct task_context *tc, ulong
uvaddr, physadd
* ptep = pte_offset_map(pmd, address);
* pte = *ptep;
*/
- pte_paddr = pmd_pte & PHYSICAL_PAGE_MASK;
- pte_paddr = xen_m2p(pte_paddr);
- if (verbose)
- fprintf(fp, " PMD: %lx\n", pte_paddr);
- FILL_PTBL(pte_paddr, PHYSADDR, PAGESIZE());
- ptep = ((ulong *)pte_paddr) + pte_index(uvaddr);
- pte = ULONG(machdep->ptbl + PAGEOFFSET(ptep));
- if (verbose)
- fprintf(fp, " PTE: %lx => %lx [machine]\n", (ulong)ptep, pte);
+ pte = x86_64_pte_offset(pmd_pte, uvaddr, verbose, TRUE);
if (!(pte & (_PAGE_PRESENT))) {
*paddr = pte;
@@ -1977,11 +2032,7 @@ x86_64_uvtop(struct task_context *tc, ulong uvaddr, physaddr_t
*paddr, int verbo
ulong *pgd;
ulong pgd_paddr;
ulong pgd_pte;
- ulong *pmd;
- ulong pmd_paddr;
ulong pmd_pte;
- ulong *ptep;
- ulong pte_paddr;
ulong pte;
physaddr_t physpage;
@@ -2014,12 +2065,7 @@ x86_64_uvtop(struct task_context *tc, ulong uvaddr, physaddr_t
*paddr, int verbo
/*
* pmd = pmd_offset(pgd, address);
*/
- pmd_paddr = pgd_pte & PHYSICAL_PAGE_MASK;
- FILL_PMD(pmd_paddr, PHYSADDR, PAGESIZE());
- pmd = ((ulong *)pmd_paddr) + pmd_index(uvaddr);
- pmd_pte = ULONG(machdep->pmd + PAGEOFFSET(pmd));
- if (verbose)
- fprintf(fp, " PMD: %lx => %lx\n", (ulong)pmd, pmd_pte);
+ pmd_pte = x86_64_pmd_offset(pgd_pte, uvaddr, verbose, FALSE);
if (!(pmd_pte & _PAGE_PRESENT))
goto no_upage;
if (pmd_pte & _PAGE_PSE) {
@@ -2039,12 +2085,7 @@ x86_64_uvtop(struct task_context *tc, ulong uvaddr, physaddr_t
*paddr, int verbo
* ptep = pte_offset_map(pmd, address);
* pte = *ptep;
*/
- pte_paddr = pmd_pte & PHYSICAL_PAGE_MASK;
- FILL_PTBL(pte_paddr, PHYSADDR, PAGESIZE());
- ptep = ((ulong *)pte_paddr) + pte_index(uvaddr);
- pte = ULONG(machdep->ptbl + PAGEOFFSET(ptep));
- if (verbose)
- fprintf(fp, " PTE: %lx => %lx\n", (ulong)ptep, pte);
+ pte = x86_64_pte_offset(pmd_pte, uvaddr, verbose, FALSE);
if (!(pte & (_PAGE_PRESENT))) {
*paddr = pte;
@@ -2070,7 +2111,6 @@ no_upage:
return FALSE;
}
-
/*
* Translates a kernel virtual address to its physical address. cmd_vtop()
* sets the verbose flag so that the pte translation gets displayed; all
@@ -2080,14 +2120,8 @@ static int
x86_64_kvtop(struct task_context *tc, ulong kvaddr, physaddr_t *paddr, int verbose)
{
ulong *pml4;
- ulong *pgd;
- ulong pgd_paddr;
- ulong pgd_pte;
- ulong *pmd;
- ulong pmd_paddr;
+ ulong pud_pte;
ulong pmd_pte;
- ulong *ptep;
- ulong pte_paddr;
ulong pte;
physaddr_t physpage;
@@ -2098,12 +2132,7 @@ x86_64_kvtop(struct task_context *tc, ulong kvaddr, physaddr_t
*paddr, int verbo
* stage phys_base is not initialized yet and x86_64_VTOP()
* does not work. Jump to the code of pagetable translation.
*/
- FILL_PML4();
- pml4 = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
- if (verbose) {
- fprintf(fp, "PML4 DIRECTORY: %lx\n", vt->kernel_pgd[0]);
- fprintf(fp, "PAGE DIRECTORY: %lx\n", *pml4);
- }
+ pgd = x86_64_kpgd_offset(kvaddr, verbose, FALSE);
goto start_vtop_with_pagetable;
}
@@ -2146,35 +2175,21 @@ x86_64_kvtop(struct task_context *tc, ulong kvaddr, physaddr_t
*paddr, int verbo
/*
* pgd = pgd_offset_k(addr);
*/
- FILL_PML4();
- pml4 = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
- if (verbose) {
- fprintf(fp, "PML4 DIRECTORY: %lx\n", vt->kernel_pgd[0]);
- fprintf(fp, "PAGE DIRECTORY: %lx\n", *pml4);
- }
+ pgd = x86_64_kpgd_offset(kvaddr, verbose, FALSE);
}
start_vtop_with_pagetable:
if (!(*pml4 & _PAGE_PRESENT))
goto no_kpage;
- pgd_paddr = (*pml4) & PHYSICAL_PAGE_MASK;
- FILL_PGD(pgd_paddr, PHYSADDR, PAGESIZE());
- pgd = ((ulong *)pgd_paddr) + pgd_index(kvaddr);
- pgd_pte = ULONG(machdep->pgd + PAGEOFFSET(pgd));
- if (verbose)
- fprintf(fp, " PUD: %lx => %lx\n", (ulong)pgd, pgd_pte);
- if (!(pgd_pte & _PAGE_PRESENT))
+
+ pud_pte = x86_64_pud_offset(*pml4, kvaddr, verbose, FALSE);
+ if (!(pud_pte & _PAGE_PRESENT))
goto no_kpage;
/*
- * pmd = pmd_offset(pgd, addr);
+ * pmd = pmd_offset(pud, address);
*/
- pmd_paddr = pgd_pte & PHYSICAL_PAGE_MASK;
- FILL_PMD(pmd_paddr, PHYSADDR, PAGESIZE());
- pmd = ((ulong *)pmd_paddr) + pmd_index(kvaddr);
- pmd_pte = ULONG(machdep->pmd + PAGEOFFSET(pmd));
- if (verbose)
- fprintf(fp, " PMD: %lx => %lx\n", (ulong)pmd, pmd_pte);
+ pmd_pte = x86_64_pmd_offset(pud_pte, kvaddr, verbose, FALSE);
if (!(pmd_pte & _PAGE_PRESENT))
goto no_kpage;
if (pmd_pte & _PAGE_PSE) {
@@ -2194,12 +2209,7 @@ start_vtop_with_pagetable:
* ptep = pte_offset_map(pmd, addr);
* pte = *ptep;
*/
- pte_paddr = pmd_pte & PHYSICAL_PAGE_MASK;
- FILL_PTBL(pte_paddr, PHYSADDR, PAGESIZE());
- ptep = ((ulong *)pte_paddr) + pte_index(kvaddr);
- pte = ULONG(machdep->ptbl + PAGEOFFSET(ptep));
- if (verbose)
- fprintf(fp, " PTE: %lx => %lx\n", (ulong)ptep, pte);
+ pte = x86_64_pte_offset(pmd_pte, kvaddr, verbose, FALSE);
if (!(pte & (_PAGE_PRESENT))) {
if (pte && verbose) {
fprintf(fp, "\n");
@@ -2233,15 +2243,9 @@ static int
x86_64_kvtop_xen_wpt(struct task_context *tc, ulong kvaddr, physaddr_t *paddr, int
verbose)
{
ulong *pml4;
- ulong *pgd;
- ulong pgd_paddr;
- ulong pgd_pte;
- ulong *pmd;
- ulong pmd_paddr;
+ ulong pud_pte;
ulong pmd_pte;
ulong pseudo_pmd_pte;
- ulong *ptep;
- ulong pte_paddr;
ulong pte;
ulong pseudo_pte;
physaddr_t physpage;
@@ -2250,38 +2254,18 @@ x86_64_kvtop_xen_wpt(struct task_context *tc, ulong kvaddr,
physaddr_t *paddr, i
/*
* pgd = pgd_offset_k(addr);
*/
- FILL_PML4();
- pml4 = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
- if (verbose) {
- fprintf(fp, "PML4 DIRECTORY: %lx\n", vt->kernel_pgd[0]);
- fprintf(fp, "PAGE DIRECTORY: %lx [machine]\n", *pml4);
- }
+ pml4 = x86_64_kpgd_offset(kvaddr, verbose, TRUE);
if (!(*pml4 & _PAGE_PRESENT))
goto no_kpage;
- pgd_paddr = (*pml4) & PHYSICAL_PAGE_MASK;
- pgd_paddr = xen_m2p(pgd_paddr);
- if (verbose)
- fprintf(fp, "PAGE DIRECTORY: %lx\n", pgd_paddr);
- FILL_PGD(pgd_paddr, PHYSADDR, PAGESIZE());
- pgd = ((ulong *)pgd_paddr) + pgd_index(kvaddr);
- pgd_pte = ULONG(machdep->pgd + PAGEOFFSET(pgd));
- if (verbose)
- fprintf(fp, " PUD: %lx => %lx [machine]\n", (ulong)pgd,
pgd_pte);
- if (!(pgd_pte & _PAGE_PRESENT))
+
+ pud_pte = x86_64_pud_offset(*pml4, kvaddr, verbose, TRUE);
+ if (!(pud_pte & _PAGE_PRESENT))
goto no_kpage;
/*
* pmd = pmd_offset(pgd, addr);
*/
- pmd_paddr = pgd_pte & PHYSICAL_PAGE_MASK;
- pmd_paddr = xen_m2p(pmd_paddr);
- if (verbose)
- fprintf(fp, " PUD: %lx\n", pmd_paddr);
- FILL_PMD(pmd_paddr, PHYSADDR, PAGESIZE());
- pmd = ((ulong *)pmd_paddr) + pmd_index(kvaddr);
- pmd_pte = ULONG(machdep->pmd + PAGEOFFSET(pmd));
- if (verbose)
- fprintf(fp, " PMD: %lx => %lx [machine]\n", (ulong)pmd,
pmd_pte);
+ pmd_pte = x86_64_pmd_offset(pud_pte, kvaddr, verbose, TRUE);
if (!(pmd_pte & _PAGE_PRESENT))
goto no_kpage;
if (pmd_pte & _PAGE_PSE) {
@@ -2320,15 +2304,7 @@ x86_64_kvtop_xen_wpt(struct task_context *tc, ulong kvaddr,
physaddr_t *paddr, i
* ptep = pte_offset_map(pmd, addr);
* pte = *ptep;
*/
- pte_paddr = pmd_pte & PHYSICAL_PAGE_MASK;
- pte_paddr = xen_m2p(pte_paddr);
- if (verbose)
- fprintf(fp, " PMD: %lx\n", pte_paddr);
- FILL_PTBL(pte_paddr, PHYSADDR, PAGESIZE());
- ptep = ((ulong *)pte_paddr) + pte_index(kvaddr);
- pte = ULONG(machdep->ptbl + PAGEOFFSET(ptep));
- if (verbose)
- fprintf(fp, " PTE: %lx => %lx [machine]\n", (ulong)ptep,
pte);
+ pte = x86_64_pte_offset(pmd_pte, kvaddr, verbose, TRUE);
if (!(pte & (_PAGE_PRESENT))) {
if (pte && verbose) {
fprintf(fp, "\n");
--
2.14.3
4:12 a.m.
New subject: [PATCH v2 3/6] x86_64: Unify the page table parsing for 4-level mode
Repace the PML4 and UPML with the common PGD to make page table uniform
for supporting 5 level mode.
Signed-off-by: Dou Liyang <douly.fnst(a)cn.fujitsu.com>
---
defs.h | 55 ++++++---------
sadump.c | 9 +--
x86_64.c | 230 +++++++++++++++++++++++++++------------------------------------
3 files changed, 123 insertions(+), 171 deletions(-)
diff --git a/defs.h b/defs.h
index 8082d6c..c055e48 100644
--- a/defs.h
+++ b/defs.h
@@ -3330,56 +3330,44 @@ struct arm64_stackframe {
#define VTOP(X) x86_64_VTOP((ulong)(X))
#define IS_VMALLOC_ADDR(X) x86_64_IS_VMALLOC_ADDR((ulong)(X))
-#define PML4_SHIFT 39
-#define PTRS_PER_PML4 512
+/* origin level page */
#define PGDIR_SHIFT 30
#define PTRS_PER_PGD 512
#define PMD_SHIFT 21
#define PTRS_PER_PMD 512
#define PTRS_PER_PTE 512
+/* 4 level page */
+#define PGDIR_SHIFT_4LEVEL 39
+#define PTRS_PER_PGD_4LEVEL 512
+#define PUD_SHIFT 30
+#define PTRS_PER_PUD 512
+
+/* 5 level page */
#define PGDIR_SHIFT_5LEVEL 48
#define PTRS_PER_PGD_5LEVEL 512
#define P4D_SHIFT 39
#define PTRS_PER_P4D 512
#define __PGDIR_SHIFT (machdep->machspec->pgdir_shift)
+#define __PTRS_PER_PGD (machdep->machspec->ptrs_per_pgd)
-#define pml4_index(address) (((address) >> PML4_SHIFT) & (PTRS_PER_PML4-1))
+#define pgd_index(address) (((address) >> __PGDIR_SHIFT) &
(__PTRS_PER_PGD-1))
#define p4d_index(address) (((address) >> P4D_SHIFT) & (PTRS_PER_P4D - 1))
-#define pgd_index(address) (((address) >> __PGDIR_SHIFT) & (PTRS_PER_PGD-1))
+#define pud_index(address) (((address) >> PUD_SHIFT) & (PTRS_PER_PUD - 1))
#define pmd_index(address) (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1))
#define pte_index(address) (((address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1))
-#define IS_LAST_PML4_READ(pml4) ((ulong)(pml4) ==
machdep->machspec->last_pml4_read)
-
-#define FILL_PML4() \
- if (!(pc->flags & RUNTIME) || ACTIVE()) { \
- if (!IS_LAST_PML4_READ(vt->kernel_pgd[0])) { \
- readmem(vt->kernel_pgd[0], KVADDR, machdep->machspec->pml4, \
- PAGESIZE(), "init_level4_pgt", FAULT_ON_ERROR); \
- machdep->machspec->last_pml4_read = (ulong)(vt->kernel_pgd[0]); \
- } \
+#define FILL_TOP_PGD() \
+ if (!(pc->flags & RUNTIME) || ACTIVE()) { \
+ FILL_PGD(vt->kernel_pgd[0], KVADDR, PAGESIZE()); \
}
-#define FILL_PML4_HYPER() \
- if (!machdep->machspec->last_pml4_read) { \
- unsigned long idle_pg_table = symbol_exists("idle_pg_table_4") ? \
- symbol_value("idle_pg_table_4") : \
- symbol_value("idle_pg_table"); \
- readmem(idle_pg_table, KVADDR, machdep->machspec->pml4, PAGESIZE(), \
- "idle_pg_table", FAULT_ON_ERROR); \
- machdep->machspec->last_pml4_read = idle_pg_table; \
- }
-
-#define IS_LAST_UPML_READ(pml) ((ulong)(pml) == machdep->machspec->last_upml_read)
-
-#define FILL_UPML(PML, TYPE, SIZE) \
- if (!IS_LAST_UPML_READ(PML)) { \
- readmem((ulonglong)((ulong)(PML)), TYPE, machdep->machspec->upml, \
- SIZE, "pml page", FAULT_ON_ERROR);
\
- machdep->machspec->last_upml_read = (ulong)(PML); \
- }
+#define FILL_TOP_PGD_HYPER() \
+ unsigned long idle_pg_table = symbol_exists("idle_pg_table_4") ? \
+ symbol_value("idle_pg_table_4") : \
+ symbol_value("idle_pg_table"); \
+ FILL_PGD(idle_pg_table, KVADDR, PAGESIZE());
#define IS_LAST_P4D_READ(p4d) ((ulong)(p4d) == machdep->machspec->last_p4d_read)
@@ -5747,10 +5735,6 @@ struct machine_specific {
ulong modules_vaddr;
ulong modules_end;
ulong phys_base;
- char *pml4;
- char *upml;
- ulong last_upml_read;
- ulong last_pml4_read;
char *irqstack;
ulong irq_eframe_link;
struct x86_64_pt_regs_offsets pto;
@@ -5765,6 +5749,7 @@ struct machine_specific {
ulong kernel_image_size;
ulong physical_mask_shift;
ulong pgdir_shift;
+ ulong ptrs_per_pgd;
char *p4d;
ulong last_p4d_read;
struct ORC_data orc;
diff --git a/sadump.c b/sadump.c
index 6b912d4..a189c2d 100644
--- a/sadump.c
+++ b/sadump.c
@@ -2051,10 +2051,11 @@ sadump_calc_kaslr_offset(ulong *kaslr_offset)
* TODO: XEN and 5-level is not supported
*/
vt->kernel_pgd[0] = cr3;
- machdep->machspec->last_pml4_read = vt->kernel_pgd[0];
+ machdep->last_pgd_read = vt->kernel_pgd[0];
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_2_6;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT;
- if (!readmem(cr3, PHYSADDR, machdep->machspec->pml4, PAGESIZE(),
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT_4LEVEL;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_4LEVEL;
+ if (!readmem(cr3, PHYSADDR, machdep->pgd, PAGESIZE(),
"cr3", RETURN_ON_ERROR))
goto quit;
@@ -2099,7 +2100,7 @@ sadump_calc_kaslr_offset(ulong *kaslr_offset)
ret = TRUE;
quit:
vt->kernel_pgd[0] = 0;
- machdep->machspec->last_pml4_read = 0;
+ machdep->last_pgd_read = 0;
return ret;
}
#else
diff --git a/x86_64.c b/x86_64.c
index 7134b4b..0f2e24a 100644
--- a/x86_64.c
+++ b/x86_64.c
@@ -166,20 +166,17 @@ x86_64_init(int when)
machdep->pageoffset = machdep->pagesize - 1;
machdep->pagemask = ~((ulonglong)machdep->pageoffset);
machdep->stacksize = machdep->pagesize * 2;
- if ((machdep->machspec->upml = (char *)malloc(PAGESIZE())) ==
NULL)
- error(FATAL, "cannot malloc upml space.");
- if ((machdep->pgd = (char *)malloc(PAGESIZE())) == NULL)
+ if ((machdep->pgd = (char *)malloc(PAGESIZE())) == NULL)
error(FATAL, "cannot malloc pgd space.");
+ if ((machdep->pud = (char *)malloc(PAGESIZE())) == NULL)
+ error(FATAL, "cannot malloc pud space.");
if ((machdep->pmd = (char *)malloc(PAGESIZE())) == NULL)
error(FATAL, "cannot malloc pmd space.");
if ((machdep->ptbl = (char *)malloc(PAGESIZE())) == NULL)
error(FATAL, "cannot malloc ptbl space.");
- if ((machdep->machspec->pml4 =
- (char *)malloc(PAGESIZE()*2)) == NULL)
- error(FATAL, "cannot malloc pml4 space.");
- machdep->machspec->last_upml_read = 0;
- machdep->machspec->last_pml4_read = 0;
+
machdep->last_pgd_read = 0;
+ machdep->last_pud_read = 0;
machdep->last_pmd_read = 0;
machdep->last_ptbl_read = 0;
machdep->verify_paddr = x86_64_verify_paddr;
@@ -233,12 +230,10 @@ x86_64_init(int when)
machdep->machspec->modules_vaddr = MODULES_VADDR_ORIG;
machdep->machspec->modules_end = MODULES_END_ORIG;
- free(machdep->machspec->upml);
- machdep->machspec->upml = NULL;
-
machdep->uvtop = x86_64_uvtop;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_2_6;
machdep->machspec->pgdir_shift = PGDIR_SHIFT;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD;
break;
case VM_2_6_11:
@@ -264,7 +259,8 @@ x86_64_init(int when)
machdep->uvtop = x86_64_uvtop_level4;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_2_6;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT;
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT_4LEVEL;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_4LEVEL;
break;
case VM_XEN:
@@ -276,7 +272,6 @@ x86_64_init(int when)
machdep->machspec->modules_vaddr = MODULES_VADDR_XEN;
machdep->machspec->modules_end = MODULES_END_XEN;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_XEN;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT;
break;
case VM_XEN_RHEL4:
@@ -288,7 +283,6 @@ x86_64_init(int when)
machdep->machspec->modules_vaddr =
MODULES_VADDR_XEN_RHEL4;
machdep->machspec->modules_end = MODULES_END_XEN_RHEL4;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_XEN;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT;
break;
case VM_5LEVEL:
@@ -302,6 +296,7 @@ x86_64_init(int when)
machdep->machspec->vmemmap_end = VMEMMAP_END_5LEVEL;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_5LEVEL;
machdep->machspec->pgdir_shift = PGDIR_SHIFT_5LEVEL;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_5LEVEL;
if ((machdep->machspec->p4d = (char *)malloc(PAGESIZE())) == NULL)
error(FATAL, "cannot malloc p4d space.");
machdep->machspec->last_p4d_read = 0;
@@ -629,14 +624,21 @@ x86_64_init(int when)
case VM_XEN:
case VM_2_6_11:
machdep->uvtop = x86_64_uvtop_level4_xen_wpt;
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT_4LEVEL;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_4LEVEL;
break;
case VM_XEN_RHEL4:
machdep->uvtop = x86_64_uvtop_level4_rhel4_xen_wpt;
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD;
break;
}
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_XEN;
- } else
- machdep->uvtop = x86_64_uvtop_level4;
+ } else {
+ machdep->uvtop = x86_64_uvtop_level4;
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT_4LEVEL;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_4LEVEL;
+ }
MEMBER_OFFSET_INIT(vcpu_guest_context_user_regs,
"vcpu_guest_context", "user_regs");
ASSIGN_OFFSET(cpu_user_regs_rsp) =
@@ -811,9 +813,11 @@ x86_64_dump_machdep_table(ulong arg)
fprintf(fp, " value_to_symbol: x86_64_value_to_symbol()\n");
fprintf(fp, " in_alternate_stack: x86_64_in_alternate_stack()\n");
fprintf(fp, " last_pgd_read: %lx\n", machdep->last_pgd_read);
+ fprintf(fp, " last_pud_read: %lx\n", machdep->last_pud_read);
fprintf(fp, " last_pmd_read: %lx\n", machdep->last_pmd_read);
fprintf(fp, " last_ptbl_read: %lx\n", machdep->last_ptbl_read);
fprintf(fp, " pgd: %lx\n", (ulong)machdep->pgd);
+ fprintf(fp, " pud: %lx\n", (ulong)machdep->pud);
fprintf(fp, " pmd: %lx\n", (ulong)machdep->pmd);
fprintf(fp, " ptbl: %lx\n", (ulong)machdep->ptbl);
fprintf(fp, " ptrs_per_pgd: %d\n", machdep->ptrs_per_pgd);
@@ -848,15 +852,7 @@ x86_64_dump_machdep_table(ulong arg)
fprintf(fp, " pgdir_shift: %ld\n", ms->pgdir_shift);
fprintf(fp, " GART_start: %lx\n", ms->GART_start);
fprintf(fp, " GART_end: %lx\n", ms->GART_end);
- fprintf(fp, " pml4: %lx\n", (ulong)ms->pml4);
- fprintf(fp, " last_pml4_read: %lx\n",
(ulong)ms->last_pml4_read);
- if (ms->upml) {
- fprintf(fp, " upml: %lx\n", (ulong)ms->upml);
- fprintf(fp, " last_upml_read: %lx\n",
(ulong)ms->last_upml_read);
- } else {
- fprintf(fp, " upml: (unused)\n");
- fprintf(fp, " last_upml_read: (unused)\n");
- }
+
if (ms->p4d) {
fprintf(fp, " p4d: %lx\n", (ulong)ms->p4d);
fprintf(fp, " last_p4d_read: %lx\n",
(ulong)ms->last_p4d_read);
@@ -1511,7 +1507,7 @@ x86_64_init_kernel_pgd(void)
for (i = 0; i < NR_CPUS; i++)
vt->kernel_pgd[i] = kernel_pgt;
- FILL_PML4();
+ FILL_TOP_PGD();
}
/*
@@ -1571,8 +1567,8 @@ x86_64_kpgd_offset(ulong kvaddr, int verbose , int IS_XEN)
{
ulong *pgd;
- FILL_PML4();
- pgd = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
+ FILL_TOP_PGD();
+ pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
if (verbose) {
fprintf(fp, "PGD DIRECTORY: %lx\n", vt->kernel_pgd[0]);
if(IS_XEN)
@@ -1602,9 +1598,9 @@ x86_64_upgd_offset(struct task_context *tc, ulong uvaddr, int
verbose , int IS_X
sizeof(long), "mm_struct pgd", FAULT_ON_ERROR);
pgd_paddr = x86_64_VTOP((ulong)pgd);
- FILL_UPML(pgd_paddr, PHYSADDR, PAGESIZE());
- pgd = ((ulong *)pgd_paddr) + pml4_index(uvaddr);
- pgd_pte = ULONG(machdep->machspec->pml4 + PAGEOFFSET(pgd));
+ FILL_PGD(pgd_paddr, PHYSADDR, PAGESIZE());
+ pgd = ((ulong *)pgd_paddr) + pgd_index(uvaddr);
+ pgd_pte = ULONG(machdep->pgd + PAGEOFFSET(pgd));
if (verbose) {
if(IS_XEN)
fprintf(fp, " PGD: %lx => %lx [machine]\n", (ulong)pgd, pgd_pte);
@@ -1635,7 +1631,7 @@ x86_64_pud_offset(ulong pgd_pte, ulong vaddr, int verbose , int
IS_XEN)
}
FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
- pud = ((ulong *)pud_paddr) + pgd_index(vaddr);
+ pud = ((ulong *)pud_paddr) + pud_index(vaddr);
pud_pte = ULONG(machdep->pud + PAGEOFFSET(pud));
if (verbose) {
if(IS_XEN)
@@ -1917,9 +1913,6 @@ no_upage:
static int
x86_64_uvtop_level4_rhel4_xen_wpt(struct task_context *tc, ulong uvaddr, physaddr_t
*paddr, int verbose)
{
- ulong mm;
- ulong *pgd;
- ulong pgd_paddr;
ulong pgd_pte;
ulong pmd_pte;
ulong pseudo_pmd_pte;
@@ -1936,18 +1929,7 @@ x86_64_uvtop_level4_rhel4_xen_wpt(struct task_context *tc, ulong
uvaddr, physadd
if (IS_KVADDR(uvaddr))
return x86_64_kvtop(tc, uvaddr, paddr, verbose);
- if ((mm = task_mm(tc->task, TRUE)))
- pgd = ULONG_PTR(tt->mm_struct + OFFSET(mm_struct_pgd));
- else
- readmem(tc->mm_struct + OFFSET(mm_struct_pgd), KVADDR, &pgd,
- sizeof(long), "mm_struct pgd", FAULT_ON_ERROR);
-
- pgd_paddr = x86_64_VTOP((ulong)pgd);
- FILL_PGD(pgd_paddr, PHYSADDR, PAGESIZE());
- pgd = ((ulong *)pgd_paddr) + pgd_index(uvaddr);
- pgd_pte = ULONG(machdep->pgd + PAGEOFFSET(pgd));
- if (verbose)
- fprintf(fp, " PGD: %lx => %lx [machine]\n", (ulong)pgd,
pgd_pte);
+ pgd_pte = x86_64_upgd_offset(tc, uvaddr, verbose, TRUE);
if (!(pgd_pte & _PAGE_PRESENT))
goto no_upage;
@@ -2028,9 +2010,6 @@ no_upage:
static int
x86_64_uvtop(struct task_context *tc, ulong uvaddr, physaddr_t *paddr, int verbose)
{
- ulong mm;
- ulong *pgd;
- ulong pgd_paddr;
ulong pgd_pte;
ulong pmd_pte;
ulong pte;
@@ -2047,18 +2026,7 @@ x86_64_uvtop(struct task_context *tc, ulong uvaddr, physaddr_t
*paddr, int verbo
/*
* pgd = pgd_offset(mm, address);
*/
- if ((mm = task_mm(tc->task, TRUE)))
- pgd = ULONG_PTR(tt->mm_struct + OFFSET(mm_struct_pgd));
- else
- readmem(tc->mm_struct + OFFSET(mm_struct_pgd), KVADDR, &pgd,
- sizeof(long), "mm_struct pgd", FAULT_ON_ERROR);
-
- pgd_paddr = x86_64_VTOP((ulong)pgd);
- FILL_PGD(pgd_paddr, PHYSADDR, PAGESIZE());
- pgd = ((ulong *)pgd_paddr) + pgd_index(uvaddr);
- pgd_pte = ULONG(machdep->pgd + PAGEOFFSET(pgd));
- if (verbose)
- fprintf(fp, " PGD: %lx => %lx\n", (ulong)pgd, pgd_pte);
+ pgd_pte = x86_64_upgd_offset(tc, uvaddr, verbose, FALSE);
if (!(pgd_pte & _PAGE_PRESENT))
goto no_upage;
@@ -2119,7 +2087,7 @@ no_upage:
static int
x86_64_kvtop(struct task_context *tc, ulong kvaddr, physaddr_t *paddr, int verbose)
{
- ulong *pml4;
+ ulong *pgd;
ulong pud_pte;
ulong pmd_pte;
ulong pte;
@@ -2148,11 +2116,11 @@ x86_64_kvtop(struct task_context *tc, ulong kvaddr, physaddr_t
*paddr, int verbo
*paddr = kvaddr - DIRECTMAP_VIRT_START;
return TRUE;
}
- FILL_PML4_HYPER();
- pml4 = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
+ FILL_TOP_PGD_HYPER();
+ pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
if (verbose) {
- fprintf(fp, "PML4 DIRECTORY: %lx\n", vt->kernel_pgd[0]);
- fprintf(fp, "PAGE DIRECTORY: %lx\n", *pml4);
+ fprintf(fp, "PGD DIRECTORY: %lx\n", vt->kernel_pgd[0]);
+ fprintf(fp, "PAGE DIRECTORY: %lx\n", *pgd);
}
} else {
if (!vt->vmalloc_start) {
@@ -2179,10 +2147,10 @@ x86_64_kvtop(struct task_context *tc, ulong kvaddr, physaddr_t
*paddr, int verbo
}
start_vtop_with_pagetable:
- if (!(*pml4 & _PAGE_PRESENT))
+ if (!(*pgd & _PAGE_PRESENT))
goto no_kpage;
- pud_pte = x86_64_pud_offset(*pml4, kvaddr, verbose, FALSE);
+ pud_pte = x86_64_pud_offset(*pgd, kvaddr, verbose, FALSE);
if (!(pud_pte & _PAGE_PRESENT))
goto no_kpage;
@@ -2242,7 +2210,7 @@ x86_64_kvtop_5level(struct task_context *tc, ulong kvaddr,
physaddr_t *paddr, in
static int
x86_64_kvtop_xen_wpt(struct task_context *tc, ulong kvaddr, physaddr_t *paddr, int
verbose)
{
- ulong *pml4;
+ ulong *pgd;
ulong pud_pte;
ulong pmd_pte;
ulong pseudo_pmd_pte;
@@ -2254,11 +2222,11 @@ x86_64_kvtop_xen_wpt(struct task_context *tc, ulong kvaddr,
physaddr_t *paddr, i
/*
* pgd = pgd_offset_k(addr);
*/
- pml4 = x86_64_kpgd_offset(kvaddr, verbose, TRUE);
- if (!(*pml4 & _PAGE_PRESENT))
+ pgd = x86_64_kpgd_offset(kvaddr, verbose, TRUE);
+ if (!(*pgd & _PAGE_PRESENT))
goto no_kpage;
- pud_pte = x86_64_pud_offset(*pml4, kvaddr, verbose, TRUE);
+ pud_pte = x86_64_pud_offset(*pgd, kvaddr, verbose, TRUE);
if (!(pud_pte & _PAGE_PRESENT))
goto no_kpage;
@@ -5827,7 +5795,8 @@ parse_cmdline_args(void)
void
x86_64_clear_machdep_cache(void)
{
- machdep->machspec->last_upml_read = 0;
+ if (machdep->last_pgd_read != vt->kernel_pgd[0])
+ machdep->last_pgd_read = 0;
}
#define PUSH_RBP_MOV_RSP_RBP 0xe5894855
@@ -6206,12 +6175,12 @@ use_cr3:
if (CRASHDEBUG(1))
fprintf(fp, "x86_64_xen_kdump_p2m_create: cr3: %lx\n",
xkd->cr3);
- if (!readmem(PTOB(xkd->cr3), PHYSADDR, machdep->machspec->pml4,
+ if (!readmem(PTOB(xkd->cr3), PHYSADDR, machdep->pgd,
PAGESIZE(), "xen kdump cr3 page", RETURN_ON_ERROR))
error(FATAL, "cannot read xen kdump cr3 page\n");
if (CRASHDEBUG(7))
- x86_64_debug_dump_page(fp, machdep->machspec->pml4,
+ x86_64_debug_dump_page(fp, machdep->pgd,
"contents of PML4 page:");
/*
@@ -6246,7 +6215,7 @@ use_cr3:
if (CRASHDEBUG(1))
fprintf(fp, "phys_to_machine_mapping: %lx\n", kvaddr);
- machdep->last_pgd_read = BADADDR;
+ machdep->last_pud_read = BADADDR;
machdep->last_pmd_read = BADADDR;
machdep->last_ptbl_read = BADADDR;
@@ -6261,7 +6230,7 @@ use_cr3:
fprintf(fp, "\n");
}
- machdep->last_pgd_read = 0;
+ machdep->last_pud_read = 0;
machdep->last_ptbl_read = 0;
machdep->last_pmd_read = 0;
pc->curcmd_flags &= ~XEN_MACHINE_ADDR;
@@ -6275,33 +6244,33 @@ static char *
x86_64_xen_kdump_load_page(ulong kvaddr, char *pgbuf)
{
ulong mfn;
- ulong *pml4, *pgd, *pmd, *ptep;
+ ulong *pgd, *pud, *pmd, *ptep;
- pml4 = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
- mfn = ((*pml4) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
+ pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
+ mfn = ((*pgd) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
if (CRASHDEBUG(3))
fprintf(fp,
- "[%lx] pml4: %lx mfn: %lx pml4_index: %lx\n",
- kvaddr, *pml4, mfn, pml4_index(kvaddr));
+ "[%lx] pgd: %lx mfn: %lx pgd_index: %lx\n",
+ kvaddr, *pgd, mfn, pgd_index(kvaddr));
- if (!readmem(PTOB(mfn), PHYSADDR, machdep->pgd, PAGESIZE(),
+ if (!readmem(PTOB(mfn), PHYSADDR, machdep->pud, PAGESIZE(),
"xen kdump pud page", RETURN_ON_ERROR))
error(FATAL, "cannot read/find pud page\n");
- machdep->last_pgd_read = mfn;
+ machdep->last_pud_read = mfn;
if (CRASHDEBUG(7))
- x86_64_debug_dump_page(fp, machdep->pgd,
+ x86_64_debug_dump_page(fp, machdep->pud,
"contents of page upper directory page:");
- pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
- mfn = ((*pgd) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
+ pud = ((ulong *)machdep->pud) + pud_index(kvaddr);
+ mfn = ((*pud) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
if (CRASHDEBUG(3))
fprintf(fp,
- "[%lx] pgd: %lx mfn: %lx pgd_index: %lx\n",
- kvaddr, *pgd, mfn, pgd_index(kvaddr));
+ "[%lx] pud: %lx mfn: %lx pud_index: %lx\n",
+ kvaddr, *pgd, mfn, pud_index(kvaddr));
if (!readmem(PTOB(mfn), PHYSADDR, machdep->pmd, PAGESIZE(),
"xen kdump pmd page", RETURN_ON_ERROR))
@@ -6354,21 +6323,21 @@ static ulong
x86_64_xen_kdump_page_mfn(ulong kvaddr)
{
ulong mfn;
- ulong *pml4, *pgd, *pmd, *ptep;
+ ulong *pgd, *pud, *pmd, *ptep;
- pml4 = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
- mfn = ((*pml4) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
+ pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
+ mfn = ((*pgd) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
- if ((mfn != machdep->last_pgd_read) &&
- !readmem(PTOB(mfn), PHYSADDR, machdep->pgd, PAGESIZE(),
+ if ((mfn != machdep->last_pud_read) &&
+ !readmem(PTOB(mfn), PHYSADDR, machdep->pud, PAGESIZE(),
"xen kdump pud entry", RETURN_ON_ERROR))
error(FATAL, "cannot read/find pud page\n");
- machdep->last_pgd_read = mfn;
+ machdep->last_pud_read = mfn;
- pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
- mfn = ((*pgd) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
+ pud = ((ulong *)machdep->pud) + pud_index(kvaddr);
+ mfn = ((*pud) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
- if ((mfn != machdep->last_pmd_read) &&
+ if ((mfn != machdep->last_pmd_read) &&
!readmem(PTOB(mfn), PHYSADDR, machdep->pmd, PAGESIZE(),
"xen kdump pmd entry", RETURN_ON_ERROR))
error(FATAL, "cannot read/find pmd page\n");
@@ -6723,12 +6692,12 @@ x86_64_xendump_p2m_create(struct xendump_data *xd)
mfn = ctrlreg[3] >> PAGESHIFT();
- if (!xc_core_mfn_to_page(mfn, machdep->machspec->pml4))
+ if (!xc_core_mfn_to_page(mfn, machdep->pgd))
error(FATAL, "cannot read/find cr3 page\n");
if (CRASHDEBUG(7))
- x86_64_debug_dump_page(xd->ofp, machdep->machspec->pml4,
- "contents of PML4 page:");
+ x86_64_debug_dump_page(xd->ofp, machdep->pgd,
+ "contents of PGD page:");
/*
* kernel version < 2.6.27 => end_pfn
@@ -6809,12 +6778,12 @@ x86_64_pvops_xendump_p2m_create(struct xendump_data *xd)
mfn = ctrlreg[3] >> PAGESHIFT();
- if (!xc_core_mfn_to_page(mfn, machdep->machspec->pml4))
+ if (!xc_core_mfn_to_page(mfn, machdep->pgd))
error(FATAL, "cannot read/find cr3 page\n");
if (CRASHDEBUG(7))
- x86_64_debug_dump_page(xd->ofp, machdep->machspec->pml4,
- "contents of PML4 page:");
+ x86_64_debug_dump_page(xd->ofp, machdep->pgd,
+ "contents of PGD page:");
/*
* kernel version < 2.6.27 => end_pfn
@@ -6990,32 +6959,32 @@ static char *
x86_64_xendump_load_page(ulong kvaddr, struct xendump_data *xd)
{
ulong mfn;
- ulong *pml4, *pgd, *pmd, *ptep;
+ ulong *pgd, *pud, *pmd, *ptep;
- pml4 = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
- mfn = ((*pml4) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
+ pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
+ mfn = ((*pgd) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
if (CRASHDEBUG(3))
fprintf(xd->ofp,
- "[%lx] pml4: %lx mfn: %lx pml4_index: %lx\n",
- kvaddr, *pml4, mfn, pml4_index(kvaddr));
+ "[%lx] pgd: %lx mfn: %lx pgd_index: %lx\n",
+ kvaddr, *pgd, mfn, pgd_index(kvaddr));
- if (!xc_core_mfn_to_page(mfn, machdep->pgd))
+ if (!xc_core_mfn_to_page(mfn, machdep->pud))
error(FATAL, "cannot read/find pud page\n");
- machdep->last_pgd_read = mfn;
+ machdep->last_pud_read = mfn;
if (CRASHDEBUG(7))
- x86_64_debug_dump_page(xd->ofp, machdep->pgd,
+ x86_64_debug_dump_page(xd->ofp, machdep->pud,
"contents of page upper directory page:");
- pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
- mfn = ((*pgd) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
+ pud = ((ulong *)machdep->pud) + pud_index(kvaddr);
+ mfn = ((*pud) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
if (CRASHDEBUG(3))
fprintf(xd->ofp,
- "[%lx] pgd: %lx mfn: %lx pgd_index: %lx\n",
- kvaddr, *pgd, mfn, pgd_index(kvaddr));
+ "[%lx] pud: %lx mfn: %lx pud_index: %lx\n",
+ kvaddr, *pud, mfn, pud_index(kvaddr));
if (!xc_core_mfn_to_page(mfn, machdep->pmd))
error(FATAL, "cannot read/find pmd page\n");
@@ -7069,18 +7038,18 @@ x86_64_xendump_page_index(ulong kvaddr, struct xendump_data *xd)
{
int idx;
ulong mfn;
- ulong *pml4, *pgd, *pmd, *ptep;
+ ulong *pgd, *pud, *pmd, *ptep;
- pml4 = ((ulong *)machdep->machspec->pml4) + pml4_index(kvaddr);
- mfn = ((*pml4) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
+ pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
+ mfn = ((*pgd) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
- if ((mfn != machdep->last_pgd_read) &&
- !xc_core_mfn_to_page(mfn, machdep->pgd))
+ if ((mfn != machdep->last_pud_read) &&
+ !xc_core_mfn_to_page(mfn, machdep->pud))
error(FATAL, "cannot read/find pud page\n");
- machdep->last_pgd_read = mfn;
+ machdep->last_pud_read = mfn;
- pgd = ((ulong *)machdep->pgd) + pgd_index(kvaddr);
- mfn = ((*pgd) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
+ pud = ((ulong *)machdep->pud) + pud_index(kvaddr);
+ mfn = ((*pud) & PHYSICAL_PAGE_MASK) >> PAGESHIFT();
if ((mfn != machdep->last_pmd_read) &&
!xc_core_mfn_to_page(mfn, machdep->pmd))
@@ -7571,20 +7540,17 @@ x86_64_init_hyper(int when)
machdep->pageoffset = machdep->pagesize - 1;
machdep->pagemask = ~((ulonglong)machdep->pageoffset);
machdep->stacksize = machdep->pagesize * 2;
- if ((machdep->machspec->upml = (char *)malloc(PAGESIZE())) ==
NULL)
- error(FATAL, "cannot malloc upml space.");
if ((machdep->pgd = (char *)malloc(PAGESIZE())) == NULL)
error(FATAL, "cannot malloc pgd space.");
+ if ((machdep->pud = (char *)malloc(PAGESIZE())) == NULL)
+ error(FATAL, "cannot malloc pud space.");
if ((machdep->pmd = (char *)malloc(PAGESIZE())) == NULL)
error(FATAL, "cannot malloc pmd space.");
if ((machdep->ptbl = (char *)malloc(PAGESIZE())) == NULL)
error(FATAL, "cannot malloc ptbl space.");
- if ((machdep->machspec->pml4 =
- (char *)malloc(PAGESIZE()*2)) == NULL)
- error(FATAL, "cannot malloc pml4 space.");
- machdep->machspec->last_upml_read = 0;
- machdep->machspec->last_pml4_read = 0;
+
machdep->last_pgd_read = 0;
+ machdep->last_pud_read = 0;
machdep->last_pmd_read = 0;
machdep->last_ptbl_read = 0;
machdep->verify_paddr = generic_verify_paddr;
--
2.14.3
4:12 a.m.
New subject: [PATCH v2 4/6] x86_64.c: Add the VMEMMAP support for 5 level page table
Append the "VMEMMAP" flag to machdep's flags too.
Signed-off-by: Dou Liyang <douly.fnst(a)cn.fujitsu.com>
---
x86_64.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/x86_64.c b/x86_64.c
index 0f2e24a..8914830 100644
--- a/x86_64.c
+++ b/x86_64.c
@@ -294,6 +294,8 @@ x86_64_init(int when)
machdep->machspec->modules_end = MODULES_END_5LEVEL;
machdep->machspec->vmemmap_vaddr = VMEMMAP_VADDR_5LEVEL;
machdep->machspec->vmemmap_end = VMEMMAP_END_5LEVEL;
+ if (symbol_exists("vmemmap_populate"))
+ machdep->flags |= VMEMMAP;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_5LEVEL;
machdep->machspec->pgdir_shift = PGDIR_SHIFT_5LEVEL;
machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_5LEVEL;
--
2.14.3
4:12 a.m.
New subject: [PATCH v2 5/6] x86_64: Add 5-level paging support for x86_64_k/uvtop()
Add the support for x86_64 5-level page tables. Squeeze the 5-level support
into the currently existing x86_64_kvtop() and x86_64_uvtop() functions instead
of using the new xxx_5level() function placeholders.
Signed-off-by: Dou Liyang <douly.fnst(a)cn.fujitsu.com>
---
x86_64.c | 90 +++++++++++++++++++++++++++++++++++++++++++++-------------------
1 file changed, 63 insertions(+), 27 deletions(-)
diff --git a/x86_64.c b/x86_64.c
index 8914830..944c7ee 100644
--- a/x86_64.c
+++ b/x86_64.c
@@ -19,13 +19,11 @@
#ifdef X86_64
static int x86_64_kvtop(struct task_context *, ulong, physaddr_t *, int);
-static int x86_64_kvtop_5level(struct task_context *, ulong, physaddr_t *, int);
static int x86_64_kvtop_xen_wpt(struct task_context *, ulong, physaddr_t *, int);
static int x86_64_uvtop(struct task_context *, ulong, physaddr_t *, int);
static int x86_64_uvtop_level4(struct task_context *, ulong, physaddr_t *, int);
static int x86_64_uvtop_level4_xen_wpt(struct task_context *, ulong, physaddr_t *, int);
static int x86_64_uvtop_level4_rhel4_xen_wpt(struct task_context *, ulong, physaddr_t *,
int);
-static int x86_64_uvtop_5level(struct task_context *, ulong, physaddr_t *, int);
static int x86_64_task_uses_5level(struct task_context *);
static ulong x86_64_vmalloc_start(void);
static int x86_64_is_task_addr(ulong);
@@ -80,6 +78,7 @@ static int x86_64_is_kvaddr(ulong);
static int x86_64_is_uvaddr(ulong, struct task_context *);
static ulong *x86_64_kpgd_offset(ulong, int, int);
static ulong x86_64_upgd_offset(struct task_context *, ulong, int, int);
+static ulong x86_64_p4d_offset(ulong, ulong, int, int);
static ulong x86_64_pud_offset(ulong, ulong, int, int);
static ulong x86_64_pmd_offset(ulong, ulong, int, int);
static ulong x86_64_pte_offset(ulong, ulong, int, int);
@@ -1613,6 +1612,31 @@ x86_64_upgd_offset(struct task_context *tc, ulong uvaddr, int
verbose , int IS_X
return pgd_pte;
}
+/*
+ * Find an entry in the fourth-level page table..
+ * p4d = p4d_offset(pgd, address);
+ */
+static ulong
+x86_64_p4d_offset(ulong pgd_pte, ulong vaddr, int verbose ,int IS_XEN)
+{
+ ulong *p4d;
+ ulong p4d_paddr;
+ ulong p4d_pte;
+
+ p4d_paddr = pgd_pte & PHYSICAL_PAGE_MASK;
+ FILL_P4D(p4d_paddr, PHYSADDR, PAGESIZE());
+ p4d = ((ulong *)p4d_paddr) + p4d_index(vaddr);
+ p4d_pte = ULONG(machdep->machspec->p4d + PAGEOFFSET(p4d));
+ if (verbose) {
+ if(IS_XEN)
+ fprintf(fp, " P4D: %lx => %lx [machine]\n", (ulong)p4d, p4d_pte);
+ else
+ fprintf(fp, " P4D: %lx => %lx\n", (ulong)p4d, p4d_pte);
+ }
+
+ return p4d_pte;
+}
+
/*
* Find an entry in the third-level page table..
* pud = pud_offset(pgd, address);
@@ -1734,17 +1758,30 @@ x86_64_uvtop_level4(struct task_context *tc, ulong uvaddr,
physaddr_t *paddr, in
if (IS_KVADDR(uvaddr))
return x86_64_kvtop(tc, uvaddr, paddr, verbose);
- if ((machdep->flags & VM_5LEVEL) && x86_64_task_uses_5level(tc))
- return x86_64_uvtop_5level(tc, uvaddr, paddr, verbose);
-
pgd_pte = x86_64_upgd_offset(tc, uvaddr, verbose, FALSE);
if (!(pgd_pte & _PAGE_PRESENT))
goto no_upage;
- /*
- * pud = pud_offset(pgd, address);
- */
- pud_pte = x86_64_pud_offset(pgd_pte, uvaddr, verbose, FALSE);
+ /* If the VM is in 5-level page table*/
+ if (machdep->flags & VM_5LEVEL && x86_64_task_uses_5level(tc)) {
+ ulong p4d_pte;
+ /*
+ * p4d = p4d_offset(pgd, address);
+ */
+ p4d_pte = x86_64_p4d_offset(pgd_pte, uvaddr, verbose, FALSE);
+ if (!(p4d_pte & _PAGE_PRESENT))
+ goto no_upage;
+ /*
+ * pud = pud_offset(p4d, address);
+ */
+ pud_pte = x86_64_pud_offset(p4d_pte, uvaddr, verbose, FALSE);
+ } else {
+ /*
+ * pud = pud_offset(pgd, address);
+ */
+ pud_pte = x86_64_pud_offset(pgd_pte, uvaddr, verbose, FALSE);
+ }
+
if (!(pud_pte & _PAGE_PRESENT))
goto no_upage;
@@ -1803,13 +1840,6 @@ x86_64_task_uses_5level(struct task_context *tc)
return FALSE;
}
-static int
-x86_64_uvtop_5level(struct task_context *tc, ulong uvaddr, physaddr_t *paddr, int
verbose)
-{
- error(INFO, "support for 5-level page tables TBD\n");
- return FALSE;
-}
-
static int
x86_64_uvtop_level4_xen_wpt(struct task_context *tc, ulong uvaddr, physaddr_t *paddr, int
verbose)
{
@@ -2139,9 +2169,6 @@ x86_64_kvtop(struct task_context *tc, ulong kvaddr, physaddr_t
*paddr, int verbo
if (XEN() && (kt->xen_flags & WRITABLE_PAGE_TABLES))
return (x86_64_kvtop_xen_wpt(tc, kvaddr, paddr, verbose));
- if (machdep->flags & VM_5LEVEL)
- return (x86_64_kvtop_5level(tc, kvaddr, paddr, verbose));
-
/*
* pgd = pgd_offset_k(addr);
*/
@@ -2152,7 +2179,23 @@ start_vtop_with_pagetable:
if (!(*pgd & _PAGE_PRESENT))
goto no_kpage;
- pud_pte = x86_64_pud_offset(*pgd, kvaddr, verbose, FALSE);
+ /* If the VM is in 5-level page table*/
+ if (machdep->flags & VM_5LEVEL) {
+ ulong p4d_pte;
+ /*
+ * p4d = p4d_offset(pgd, address);
+ */
+ p4d_pte = x86_64_p4d_offset(*pgd, kvaddr, verbose, FALSE);
+ if (!(p4d_pte & _PAGE_PRESENT))
+ goto no_kpage;
+ /*
+ * pud = pud_offset(p4d, address);
+ */
+ pud_pte = x86_64_pud_offset(p4d_pte, kvaddr, verbose, FALSE);
+ } else {
+ pud_pte = x86_64_pud_offset(*pgd, kvaddr, verbose, FALSE);
+ }
+
if (!(pud_pte & _PAGE_PRESENT))
goto no_kpage;
@@ -2202,13 +2245,6 @@ no_kpage:
return FALSE;
}
-static int
-x86_64_kvtop_5level(struct task_context *tc, ulong kvaddr, physaddr_t *paddr, int
verbose)
-{
- error(INFO, "support for 5-level page tables TBD\n");
- return FALSE;
-}
-
static int
x86_64_kvtop_xen_wpt(struct task_context *tc, ulong kvaddr, physaddr_t *paddr, int
verbose)
{
--
2.14.3
4:12 a.m.
New subject: [PATCH v2 6/6] x86_64: Fix the misusage of PGDIR_SHIFT
Due to the backwards compatibility, we should setup machdep->machspec->pgdir_shift
correctly in x86_64_init() for all versions of linux. this patch set the default
4-level for each versions, and add a new function named x86_64_upgd_offset_legacy()
for old linuxes to get the user page global directory entry.
Signed-off-by: Dou Liyang <douly.fnst(a)cn.fujitsu.com>
---
defs.h | 15 +++++++--------
sadump.c | 4 ++--
x86_64.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++----------
3 files changed, 55 insertions(+), 20 deletions(-)
diff --git a/defs.h b/defs.h
index c055e48..ee24acb 100644
--- a/defs.h
+++ b/defs.h
@@ -3330,19 +3330,18 @@ struct arm64_stackframe {
#define VTOP(X) x86_64_VTOP((ulong)(X))
#define IS_VMALLOC_ADDR(X) x86_64_IS_VMALLOC_ADDR((ulong)(X))
-/* origin level page */
-#define PGDIR_SHIFT 30
+/*
+ * the default page table level for x86_64:
+ * 4 level page tables
+ */
+#define PGDIR_SHIFT 39
#define PTRS_PER_PGD 512
+#define PUD_SHIFT 30
+#define PTRS_PER_PUD 512
#define PMD_SHIFT 21
#define PTRS_PER_PMD 512
#define PTRS_PER_PTE 512
-/* 4 level page */
-#define PGDIR_SHIFT_4LEVEL 39
-#define PTRS_PER_PGD_4LEVEL 512
-#define PUD_SHIFT 30
-#define PTRS_PER_PUD 512
-
/* 5 level page */
#define PGDIR_SHIFT_5LEVEL 48
#define PTRS_PER_PGD_5LEVEL 512
diff --git a/sadump.c b/sadump.c
index a189c2d..1743e63 100644
--- a/sadump.c
+++ b/sadump.c
@@ -2053,8 +2053,8 @@ sadump_calc_kaslr_offset(ulong *kaslr_offset)
vt->kernel_pgd[0] = cr3;
machdep->last_pgd_read = vt->kernel_pgd[0];
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_2_6;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT_4LEVEL;
- machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_4LEVEL;
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD;
if (!readmem(cr3, PHYSADDR, machdep->pgd, PAGESIZE(),
"cr3", RETURN_ON_ERROR))
goto quit;
diff --git a/x86_64.c b/x86_64.c
index 944c7ee..95a1791 100644
--- a/x86_64.c
+++ b/x86_64.c
@@ -78,6 +78,7 @@ static int x86_64_is_kvaddr(ulong);
static int x86_64_is_uvaddr(ulong, struct task_context *);
static ulong *x86_64_kpgd_offset(ulong, int, int);
static ulong x86_64_upgd_offset(struct task_context *, ulong, int, int);
+static ulong x86_64_upgd_offset_legacy(struct task_context *, ulong, int, int);
static ulong x86_64_p4d_offset(ulong, ulong, int, int);
static ulong x86_64_pud_offset(ulong, ulong, int, int);
static ulong x86_64_pmd_offset(ulong, ulong, int, int);
@@ -258,8 +259,8 @@ x86_64_init(int when)
machdep->uvtop = x86_64_uvtop_level4;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_2_6;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT_4LEVEL;
- machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_4LEVEL;
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD;
break;
case VM_XEN:
@@ -271,6 +272,8 @@ x86_64_init(int when)
machdep->machspec->modules_vaddr = MODULES_VADDR_XEN;
machdep->machspec->modules_end = MODULES_END_XEN;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_XEN;
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD;
break;
case VM_XEN_RHEL4:
@@ -282,6 +285,8 @@ x86_64_init(int when)
machdep->machspec->modules_vaddr =
MODULES_VADDR_XEN_RHEL4;
machdep->machspec->modules_end = MODULES_END_XEN_RHEL4;
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_XEN;
+ machdep->machspec->pgdir_shift = PGDIR_SHIFT;
+ machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD;
break;
case VM_5LEVEL:
@@ -625,20 +630,14 @@ x86_64_init(int when)
case VM_XEN:
case VM_2_6_11:
machdep->uvtop = x86_64_uvtop_level4_xen_wpt;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT_4LEVEL;
- machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_4LEVEL;
break;
case VM_XEN_RHEL4:
machdep->uvtop = x86_64_uvtop_level4_rhel4_xen_wpt;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT;
- machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD;
break;
}
machdep->machspec->physical_mask_shift = __PHYSICAL_MASK_SHIFT_XEN;
} else {
machdep->uvtop = x86_64_uvtop_level4;
- machdep->machspec->pgdir_shift = PGDIR_SHIFT_4LEVEL;
- machdep->machspec->ptrs_per_pgd = PTRS_PER_PGD_4LEVEL;
}
MEMBER_OFFSET_INIT(vcpu_guest_context_user_regs,
"vcpu_guest_context", "user_regs");
@@ -1581,6 +1580,43 @@ x86_64_kpgd_offset(ulong kvaddr, int verbose , int IS_XEN)
return pgd;
}
+/*
+ * In x86 64 bit system, Linux uses the 4-level page table as the default both
+ * in Kernel page tables and user page tables.
+ *
+ * But in some old versions(pre-2.6.11), the 3-level page table is used for
+ * user page tables.
+ *
+ * So reuse the PUD and find the user pgd entry for this older version Linux..
+ * pgd = pgd_offset(mm, address);
+ */
+static ulong
+x86_64_upgd_offset_legacy(struct task_context *tc, ulong uvaddr, int verbose , int
IS_XEN)
+{
+ ulong *pud;
+ ulong pud_paddr;
+ ulong pud_pte;
+
+ if (task_mm(tc->task, TRUE))
+ pud = ULONG_PTR(tt->mm_struct + OFFSET(mm_struct_pgd));
+ else
+ readmem(tc->mm_struct + OFFSET(mm_struct_pgd), KVADDR, &pud,
+ sizeof(long), "mm_struct pgd", FAULT_ON_ERROR);
+
+ pud_paddr = x86_64_VTOP((ulong)pud);
+ FILL_PUD(pud_paddr, PHYSADDR, PAGESIZE());
+ pud = ((ulong *)pud_paddr) + pud_index(uvaddr);
+ pud_pte = ULONG(machdep->pud + PAGEOFFSET(pud));
+ if (verbose) {
+ if(IS_XEN)
+ fprintf(fp, " PGD: %lx => %lx [machine]\n", (ulong)pud, pud_pte);
+ else
+ fprintf(fp, " PGD: %lx => %lx\n", (ulong)pud, pud_pte);
+ }
+
+ return pud_pte;
+}
+
/*
* Find the user pgd entry..
* pgd = pgd_offset(mm, address);
@@ -1961,7 +1997,7 @@ x86_64_uvtop_level4_rhel4_xen_wpt(struct task_context *tc, ulong
uvaddr, physadd
if (IS_KVADDR(uvaddr))
return x86_64_kvtop(tc, uvaddr, paddr, verbose);
- pgd_pte = x86_64_upgd_offset(tc, uvaddr, verbose, TRUE);
+ pgd_pte = x86_64_upgd_offset_legacy(tc, uvaddr, verbose, TRUE);
if (!(pgd_pte & _PAGE_PRESENT))
goto no_upage;
@@ -2058,7 +2094,7 @@ x86_64_uvtop(struct task_context *tc, ulong uvaddr, physaddr_t
*paddr, int verbo
/*
* pgd = pgd_offset(mm, address);
*/
- pgd_pte = x86_64_upgd_offset(tc, uvaddr, verbose, FALSE);
+ pgd_pte = x86_64_upgd_offset_legacy(tc, uvaddr, verbose, FALSE);
if (!(pgd_pte & _PAGE_PRESENT))
goto no_upage;
--
2.14.3
2:56 p.m.
Hi Dou,
This v2 version tests OK on my sample set of dumpfiles. I didn't
get a chance to fully review the patchset, and I will be out of the
office until next Tuesday. I'll try to review the patches then.
Thanks very much,
Dave
----- Original Message -----
Changelog:
V1-->V2:
-Fix the backwards compatibility issues suggested by Dave
-Test with "mod","kmem -f","vm -p" command again.
I found Dave had alread done the first phase of future support for x86_64
5-level page tables(commit 307e7f35f510). when I asked him about the
state of this work, he gave me a more detailed answer and suggestion.
I follow his advice, and do the following job.
1. Refine the original logical:
1) Create some new common function for getting the offset of page table
2) Repace the PML4 and UPML with the common PGD:
machdep->machspec->pml4/upml ==> machdep->pdg
3) Using the PUD in x86_64
2. Add 5-level page tables support for x86_64_k/uvtop()
This patchset is the second phase of the work, As Dave said, we need to be
a manner of determining very early on whether the kernel page tables are
using 5-level and whether each user-space task is using 4- or 5-level page
tables. These will be done after this phase.
About test work:
I have tested this patchset with 4-level and 5-level paging table.
sadump are not be tested.
Dou Liyang (6):
defs.h: Fix the PHYSICAL_PAGE_MASK macro
x86_64: Extract public page table mapping code
x86_64: Unify the page table parsing for 4-level mode
x86_64.c: Add the VMEMMAP support for 5 level page table
x86_64: Add 5-level paging support for x86_64_k/uvtop()
x86_64: Fix the misusage of PGDIR_SHIFT
defs.h | 58 ++---
sadump.c | 7 +-
x86_64.c | 732
++++++++++++++++++++++++++++++++-------------------------------
3 files changed, 399 insertions(+), 398 deletions(-)
--
2.14.3
1:35 a.m.
Hi Dave,
At 01/13/2018 04:56 AM, Dave Anderson wrote:
Hi Dou,
This v2 version tests OK on my sample set of dumpfiles. I didn't
get a chance to fully review the patchset, and I will be out of the
office until next Tuesday. I'll try to review the patches then.
Thank you for telling me that.
I also test this patchset with *sadump* file with my colleagues. it is
OK too.
And I am trying to make the 5-level page tables support for Xen.
Thanks,
dou.
Thanks very much,
Dave
----- Original Message -----
> Changelog:
> V1-->V2:
> -Fix the backwards compatibility issues suggested by Dave
> -Test with "mod","kmem -f","vm -p" command again.
>
> I found Dave had alread done the first phase of future support for x86_64
> 5-level page tables(commit 307e7f35f510). when I asked him about the
> state of this work, he gave me a more detailed answer and suggestion.
> I follow his advice, and do the following job.
>
> 1. Refine the original logical:
> 1) Create some new common function for getting the offset of page table
> 2) Repace the PML4 and UPML with the common PGD:
> machdep->machspec->pml4/upml ==> machdep->pdg
> 3) Using the PUD in x86_64
>
> 2. Add 5-level page tables support for x86_64_k/uvtop()
>
> This patchset is the second phase of the work, As Dave said, we need to be
> a manner of determining very early on whether the kernel page tables are
> using 5-level and whether each user-space task is using 4- or 5-level page
> tables. These will be done after this phase.
>
> About test work:
>
> I have tested this patchset with 4-level and 5-level paging table.
>
> sadump are not be tested.
>
> Dou Liyang (6):
> defs.h: Fix the PHYSICAL_PAGE_MASK macro
> x86_64: Extract public page table mapping code
> x86_64: Unify the page table parsing for 4-level mode
> x86_64.c: Add the VMEMMAP support for 5 level page table
> x86_64: Add 5-level paging support for x86_64_k/uvtop()
> x86_64: Fix the misusage of PGDIR_SHIFT
>
> defs.h | 58 ++---
> sadump.c | 7 +-
> x86_64.c | 732
> ++++++++++++++++++++++++++++++++-------------------------------
> 3 files changed, 399 insertions(+), 398 deletions(-)
>
> --
> 2.14.3
>
>
>
>
2:37 p.m.
----- Original Message -----
Hi Dave,
At 01/13/2018 04:56 AM, Dave Anderson wrote:
>
> Hi Dou,
>
> This v2 version tests OK on my sample set of dumpfiles. I didn't
> get a chance to fully review the patchset, and I will be out of the
> office until next Tuesday. I'll try to review the patches then.
Thank you for telling me that.
I also test this patchset with *sadump* file with my colleagues. it is
OK too.
And I am trying to make the 5-level page tables support for Xen.
Thanks,
dou.
Hello Dou,
Completing my review may take me longer than expected due to other tasks
I have to take care of first. Please excuse my delay.
However, I did notice one problem w/respect to your changes to defs.h.
When modifying any exported data structure in defs.h, be aware that the
change may break extension modules that have been previously compiled
against older versions of defs.h.
For example, the following changes to the x86_64 machine_specific data
structure would break extension modules that have been previously compiled
against earlier versions of defs.h -- if the extension module uses fields in
the machine_specific data structure that come after the members that you
have removed:
@@ -5747,10 +5735,6 @@ struct machine_specific {
ulong modules_vaddr;
ulong modules_end;
ulong phys_base;
- char *pml4;
- char *upml;
- ulong last_upml_read;
- ulong last_pml4_read;
char *irqstack;
ulong irq_eframe_link;
struct x86_64_pt_regs_offsets pto;
@@ -5765,6 +5749,7 @@ struct machine_specific {
ulong kernel_image_size;
ulong physical_mask_shift;
ulong pgdir_shift;
+ ulong ptrs_per_pgd;
char *p4d;
ulong last_p4d_read;
struct ORC_data orc;
So in the situation above:
(1) Please do not remove the pml4, upml, last_upml_read and last_pml4_read members.
(You could add an "/* obsolete */" comment after them if you'd like.)
(2) When adding a new member such as ptrs_per_pgd above, please place it at the
end of the machine_specific data structure.
That way, the member offsets of the original structure members would not change
and previously compiled extension modules would be unaffected.
Thanks,
Dave
3:02 a.m.
Hi Dave,
[...]
Hello Dou,
Completing my review may take me longer than expected due to other tasks
I have to take care of first. Please excuse my delay.
Aha, never mind! As you said, this patchset also has some problem,
I will do more test and resend a new version.
However, I did notice one problem w/respect to your changes to
defs.h.
When modifying any exported data structure in defs.h, be aware that the
change may break extension modules that have been previously compiled
against older versions of defs.h.
Oops! I did not consider with this, really thank
you for your remind.
For example, the following changes to the x86_64 machine_specific
data
structure would break extension modules that have been previously compiled
against earlier versions of defs.h -- if the extension module uses fields in
the machine_specific data structure that come after the members that you
have removed:
@@ -5747,10 +5735,6 @@ struct machine_specific {
ulong modules_vaddr;
ulong modules_end;
ulong phys_base;
- char *pml4;
- char *upml;
- ulong last_upml_read;
- ulong last_pml4_read;
char *irqstack;
ulong irq_eframe_link;
struct x86_64_pt_regs_offsets pto;
@@ -5765,6 +5749,7 @@ struct machine_specific {
ulong kernel_image_size;
ulong physical_mask_shift;
ulong pgdir_shift;
+ ulong ptrs_per_pgd;
char *p4d;
ulong last_p4d_read;
struct ORC_data orc;
So in the situation above:
(1) Please do not remove the pml4, upml, last_upml_read and last_pml4_read members.
(You could add an "/* obsolete */" comment after them if you'd
like.)
(2) When adding a new member such as ptrs_per_pgd above, please place it at the
end of the machine_specific data structure.
That way, the member offsets of the original structure members would not change
and previously compiled extension modules would be unaffected.
Yes, Understood! I will check that.
Thanks,
dou
1:41 p.m.
----- Original Message -----
Hi Dave,
At 01/13/2018 04:56 AM, Dave Anderson wrote:
>
> Hi Dou,
>
> This v2 version tests OK on my sample set of dumpfiles. I didn't
> get a chance to fully review the patchset, and I will be out of the
> office until next Tuesday. I'll try to review the patches then.
Thank you for telling me that.
I also test this patchset with *sadump* file with my colleagues. it is
OK too.
And I am trying to make the 5-level page tables support for Xen.
Thanks,
dou.
Hello Dou,
Would it be possible for you to post your latest patchset against
the current github repository? There have been a number of recent
changes, although I don't believe they should affect your patchset.
Thanks,
Dave
7:23 p.m.
At 01/27/2018 03:41 AM, Dave Anderson wrote:
----- Original Message -----
> Hi Dave,
>
> At 01/13/2018 04:56 AM, Dave Anderson wrote:
>>
>> Hi Dou,
>>
>> This v2 version tests OK on my sample set of dumpfiles. I didn't
>> get a chance to fully review the patchset, and I will be out of the
>> office until next Tuesday. I'll try to review the patches then.
>
> Thank you for telling me that.
>
> I also test this patchset with *sadump* file with my colleagues. it is
> OK too.
>
> And I am trying to make the 5-level page tables support for Xen.
>
> Thanks,
> dou.
Hello Dou,
Hi Dave,
Would it be possible for you to post your latest patchset against
the current github repository? There have been a number of recent
Sorry for my delay!
Recently, I just used the vmcore file which was generated by "virsh
dump" command. So I tried to use kdump to generate the file in 5-level
supported GUEST for testing, But I am failed. Seems Kexec can't support
5-level now.
Anyway, I will post the patchset against the current github repository
right now.
Thanks,
dou
changes, although I don't believe they should affect your
patchset.
Thanks,
Dave
2350
days inactive
2367
days old
devel@lists.crash-utility.osci.io
12 comments
2 participants
participants (2)
-
Dave Anderson -
Dou Liyang