Hi folks, I've just discovered that the crash utility fails to initialize the vm subsystem properly on our latest SLES 32-bit kernels. It turns out that our kernels are compiled with CONFIG_DISCONTIGMEM=y, which causes pgdat structs to be allocated by the remap allocator (cf. arch/x86/mm/numa_32.c and also the code in setup_node_data).

Dne Út 10. ledna 2012 19:23:24 Petr Tesarik napsal(a): > Dne Út 10. ledna 2012 19:14:32 Petr Tesarik napsal(a):

> crash> vtop f2800080 > VIRTUAL PHYSICAL > f2800080 1fde00080 > > PAGE DIRECTORY: c08ed000 > PGD: c08ed018 => 8ea001 > PMD: 8eaca0 => 80000001fde001e3 > PAGE: 1fde00000 (2MB) > > PTE PHYSICAL FLAGS > 80000001fde001e3 1fde00000 (PRESENT|RW|ACCESSED|DIRTY|PSE|GLOBAL|NX) > > PAGE PHYSICAL MAPPING INDEX CNT FLAGS BTW the data from struct page is really missing here. I traced this down to an integer overflow in dump_memory_nodes():

David (Mair), could you address this, as already discussed in private mails, please?

----- Original Message ----- > On 01/11/2012 06:53 AM, Petr Tesarik wrote: > > Dne Út 10. ledna 2012 19:23:24 Petr Tesarik napsal(a): > >> Dne Út 10. ledna 2012 19:14:32 Petr Tesarik napsal(a): > ... [ cut ] ... > >> crash> vtop f2800080 > >> VIRTUAL PHYSICAL > >> f2800080 1fde00080 > >> > >> PAGE DIRECTORY: c08ed000 > >> PGD: c08ed018 => 8ea001 > >> PMD: 8eaca0 => 80000001fde001e3 > >> PAGE: 1fde00000 (2MB) > >> > >> PTE PHYSICAL FLAGS > >> 80000001fde001e3 1fde00000 > >> (PRESENT|RW|ACCESSED|DIRTY|PSE|GLOBAL|NX) > >> > >> PAGE PHYSICAL MAPPING INDEX CNT FLAGS > > > > BTW the data from struct page is really missing here. I traced > > this down to an > > integer overflow in dump_memory_nodes(): > ... [ cut ] ... > > David (Mair), could you address this, as already discussed in > > private mails, > > please? > > The attached patch fixes this for me. Hmmm, not so much for me... When I test the patch on RHEL5, RHEL6 and Fedora x86 kernels, the command always fails like this: crash> kmem -n NODE SIZE PGLIST_DATA BOOTMEM_DATA NODE_ZONES 0 262075 c0a3a680 c0aa5ce8 c0a3a680 c0a3b1c0 c0a3bd00 c0a3c840 MEM_MAP START_PADDR START_MAPNR Segmentation fault $ I haven't look into it, and this is probably not related: cc -c -g -DX86 -m32 -D_FILE_OFFSET_BITS=64 -DGDB_7_3_1 memory.c -Wall -O2 -Wstrict-prototypes -Wmissing-prototypes -fstack-protector memory.c: In function 'dump_memory_nodes': memory.c:13410: warning: cast to pointer from integer of different size memory.c:13199: warning: 'node_start_paddr' may be used uninitialized in this function But from under gdb: crash> kmem -n [Detaching after fork from child process 16870.] Program received signal SIGSEGV, Segmentation fault. 0x0809cd94 in mkstring (s=0xffe8a8bc " c0a3a680 ", size=16, flags=133, opt=0x1000 <Address 0x1000 out of bounds>) at tools.c:1620 1620 sprintf(s, "%llx", *((ulonglong *)opt)); (gdb) bt #0 0x0809cd94 in mkstring (s=0xffe8a8bc " c0a3a680 ", size=16, flags=133, opt=0x1000 <Address 0x1000 out of bounds>) at tools.c:1620 #1 0x080b1ad7 in dump_memory_nodes (initialize=0) at memory.c:13406 #2 0x080d46cc in cmd_kmem () at memory.c:4221 #3 0x080941b8 in exec_command () at main.c:751 #4 0x08093fe6 in main_loop () at main.c:699 #5 0x081db622 in current_interp_command_loop () #6 0x081dbf01 in captured_command_loop () #7 0x081db0a4 in catch_errors () #8 0x081dce07 in captured_main () #9 0x081db0a4 in catch_errors () #10 0x081dce49 in gdb_main () #11 0x081dce99 in gdb_main_entry () #12 0x08116668 in gdb_main_loop (argc=2, argv=0xffe8d494) at gdb_interface.c:75 #13 0x08093ce0 in main (argc=3, argv=0xffe8d494) at main.c:603 (gdb) p opt $1 = 0x1000 <Address 0x1000 out of bounds> (gdb) I thought it might be just an x86 issue, but it also fails the same way on RHEL5, RHEL6 and Fedora x86_64 kernels. Dave

Hi Dave, On 01/12/2012 12:58 PM, Dave Anderson wrote: > > > ----- Original Message ----- >> On 01/11/2012 06:53 AM, Petr Tesarik wrote: >>> Dne Út 10. ledna 2012 19:23:24 Petr Tesarik napsal(a): >>>> Dne Út 10. ledna 2012 19:14:32 Petr Tesarik napsal(a): >> ... [ cut ] ... >>>> crash> vtop f2800080 >>>> VIRTUAL PHYSICAL >>>> f2800080 1fde00080 >>>> >>>> PAGE DIRECTORY: c08ed000 >>>> PGD: c08ed018 => 8ea001 >>>> PMD: 8eaca0 => 80000001fde001e3 >>>> PAGE: 1fde00000 (2MB) >>>> >>>> PTE PHYSICAL FLAGS >>>> 80000001fde001e3 1fde00000 >>>> (PRESENT|RW|ACCESSED|DIRTY|PSE|GLOBAL|NX) >>>> >>>> PAGE PHYSICAL MAPPING INDEX CNT FLAGS >>> >>> BTW the data from struct page is really missing here. I traced >>> this down to an >>> integer overflow in dump_memory_nodes(): >> ... [ cut ] ... >>> David (Mair), could you address this, as already discussed in >>> private mails, >>> please? >> >> The attached patch fixes this for me. > > Hmmm, not so much for me... > > When I test the patch on RHEL5, RHEL6 and Fedora x86 kernels, the > command always fails like this: > > crash> kmem -n > NODE SIZE PGLIST_DATA BOOTMEM_DATA NODE_ZONES > 0 262075 c0a3a680 c0aa5ce8 c0a3a680 > c0a3b1c0 > c0a3bd00 > c0a3c840 > MEM_MAP START_PADDR START_MAPNR > Segmentation fault > $ > > I haven't look into it, and this is probably not related: > > cc -c -g -DX86 -m32 -D_FILE_OFFSET_BITS=64 -DGDB_7_3_1 memory.c > -Wall -O2 -Wstrict-prototypes -Wmissing-prototypes > -fstack-protector > memory.c: In function 'dump_memory_nodes': > memory.c:13410: warning: cast to pointer from integer of different > size > memory.c:13199: warning: 'node_start_paddr' may be used > uninitialized in this function > > But from under gdb: > > crash> kmem -n > [Detaching after fork from child process 16870.] > > Program received signal SIGSEGV, Segmentation fault. > 0x0809cd94 in mkstring (s=0xffe8a8bc " c0a3a680 ", size=16, > flags=133, opt=0x1000<Address 0x1000 out of bounds>) > at tools.c:1620 > 1620 sprintf(s, "%llx", *((ulonglong *)opt)); > (gdb) bt > #0 0x0809cd94 in mkstring (s=0xffe8a8bc " c0a3a680 ", size=16, > flags=133, opt=0x1000<Address 0x1000 out of bounds>) > at tools.c:1620 > #1 0x080b1ad7 in dump_memory_nodes (initialize=0) at > memory.c:13406 > #2 0x080d46cc in cmd_kmem () at memory.c:4221 > #3 0x080941b8 in exec_command () at main.c:751 > #4 0x08093fe6 in main_loop () at main.c:699 > #5 0x081db622 in current_interp_command_loop () > #6 0x081dbf01 in captured_command_loop () > #7 0x081db0a4 in catch_errors () > #8 0x081dce07 in captured_main () > #9 0x081db0a4 in catch_errors () > #10 0x081dce49 in gdb_main () > #11 0x081dce99 in gdb_main_entry () > #12 0x08116668 in gdb_main_loop (argc=2, argv=0xffe8d494) at > gdb_interface.c:75 > #13 0x08093ce0 in main (argc=3, argv=0xffe8d494) at main.c:603 > (gdb) p opt > $1 = 0x1000<Address 0x1000 out of bounds> > (gdb) > > I thought it might be just an x86 issue, but it also fails > the same way on RHEL5, RHEL6 and Fedora x86_64 kernels. I'm looking into it, that's the change I made to the fprintf() that uses node_start_paddr. The patch would be a fix for the problem Petr reported if it did not include the expansion to 64-bits in the fprintf() that's the whole last piece of the patch. The fix is pretty obvious now to the original patch: LONGLONG_HEX takes a ulonglong * LONG_HEX takes a ulong Changing the last modified MKSTR() to take &node_start_paddr should resolve it. Patch attached. -- David.

----- Original Message ----- > Hi Dave, > > On 01/12/2012 12:58 PM, Dave Anderson wrote: >> >> >> ----- Original Message ----- >>> On 01/11/2012 06:53 AM, Petr Tesarik wrote: >>>> Dne Út 10. ledna 2012 19:23:24 Petr Tesarik napsal(a): >>>>> Dne Út 10. ledna 2012 19:14:32 Petr Tesarik napsal(a): >>> ... [ cut ] ... >>>>> crash> vtop f2800080 >>>>> VIRTUAL PHYSICAL >>>>> f2800080 1fde00080 >>>>> >>>>> PAGE DIRECTORY: c08ed000 >>>>> PGD: c08ed018 => 8ea001 >>>>> PMD: 8eaca0 => 80000001fde001e3 >>>>> PAGE: 1fde00000 (2MB) >>>>> >>>>> PTE PHYSICAL FLAGS >>>>> 80000001fde001e3 1fde00000 >>>>> (PRESENT|RW|ACCESSED|DIRTY|PSE|GLOBAL|NX) >>>>> >>>>> PAGE PHYSICAL MAPPING INDEX CNT FLAGS >>>> >>>> BTW the data from struct page is really missing here. I traced >>>> this down to an >>>> integer overflow in dump_memory_nodes(): >>> ... [ cut ] ... >>>> David (Mair), could you address this, as already discussed in >>>> private mails, >>>> please? >>> >>> The attached patch fixes this for me. >> >> Hmmm, not so much for me... >> >> When I test the patch on RHEL5, RHEL6 and Fedora x86 kernels, the >> command always fails like this: >> >> crash> kmem -n >> NODE SIZE PGLIST_DATA BOOTMEM_DATA NODE_ZONES >> 0 262075 c0a3a680 c0aa5ce8 c0a3a680 >> c0a3b1c0 >> c0a3bd00 >> c0a3c840 >> MEM_MAP START_PADDR START_MAPNR >> Segmentation fault >> $ >> >> I haven't look into it, and this is probably not related: >> >> cc -c -g -DX86 -m32 -D_FILE_OFFSET_BITS=64 -DGDB_7_3_1 memory.c >> -Wall -O2 -Wstrict-prototypes -Wmissing-prototypes >> -fstack-protector >> memory.c: In function 'dump_memory_nodes': >> memory.c:13410: warning: cast to pointer from integer of different >> size >> memory.c:13199: warning: 'node_start_paddr' may be used >> uninitialized in this function >> >> But from under gdb: >> >> crash> kmem -n >> [Detaching after fork from child process 16870.] >> >> Program received signal SIGSEGV, Segmentation fault. >> 0x0809cd94 in mkstring (s=0xffe8a8bc " c0a3a680 ", size=16, >> flags=133, opt=0x1000<Address 0x1000 out of bounds>) >> at tools.c:1620 >> 1620 sprintf(s, "%llx", *((ulonglong *)opt)); >> (gdb) bt >> #0 0x0809cd94 in mkstring (s=0xffe8a8bc " c0a3a680 ", size=16, >> flags=133, opt=0x1000<Address 0x1000 out of bounds>) >> at tools.c:1620 >> #1 0x080b1ad7 in dump_memory_nodes (initialize=0) at >> memory.c:13406 >> #2 0x080d46cc in cmd_kmem () at memory.c:4221 >> #3 0x080941b8 in exec_command () at main.c:751 >> #4 0x08093fe6 in main_loop () at main.c:699 >> #5 0x081db622 in current_interp_command_loop () >> #6 0x081dbf01 in captured_command_loop () >> #7 0x081db0a4 in catch_errors () >> #8 0x081dce07 in captured_main () >> #9 0x081db0a4 in catch_errors () >> #10 0x081dce49 in gdb_main () >> #11 0x081dce99 in gdb_main_entry () >> #12 0x08116668 in gdb_main_loop (argc=2, argv=0xffe8d494) at >> gdb_interface.c:75 >> #13 0x08093ce0 in main (argc=3, argv=0xffe8d494) at main.c:603 >> (gdb) p opt >> $1 = 0x1000<Address 0x1000 out of bounds> >> (gdb) >> >> I thought it might be just an x86 issue, but it also fails >> the same way on RHEL5, RHEL6 and Fedora x86_64 kernels. > > I'm looking into it, that's the change I made to the fprintf() that uses > node_start_paddr. The patch would be a fix for the problem Petr reported > if it did not include the expansion to 64-bits in the fprintf() that's > the whole last piece of the patch. The fix is pretty obvious now to the > original patch: > > LONGLONG_HEX takes a ulonglong * > LONG_HEX takes a ulong > > Changing the last modified MKSTR() to take&node_start_paddr should > resolve it. Patch attached. > > -- > David. Right -- our mails crossed... But I don't want the output format to change, especially on 64-bit machines. For example on x86_64, it currently looks like these examples, where the 3 fields are centered: MEM_MAP START_PADDR START_MAPNR ffff8100006e6000 0 0 MEM_MAP START_PADDR START_MAPNR ffffea0004280000 130000000 1245184 MEM_MAP START_PADDR START_MAPNR ffffea0000000038 1000 1 With your patch they looks like this: crash> kmem -n ... MEM_MAP START_PADDR START_MAPNR ffff8100006e6000 0 0 MEM_MAP START_PADDR START_MAPNR ffffea0004280000 130000000 1245184 MEM_MAP START_PADDR START_MAPNR ffffea0000000038 1000 1 And even on 32-bit, it seems to vary. Here's 3 without the patch: MEM_MAP START_PADDR START_MAPNR c9000000 0 0 MEM_MAP START_PADDR START_MAPNR c188a020 1000 1 MEM_MAP START_PADDR START_MAPNR f5d2c200 10000 16 And with it applied: MEM_MAP START_PADDR START_MAPNR c9000000 0 0 MEM_MAP START_PADDR START_MAPNR c188a020 1000 1 MEM_MAP START_PADDR START_MAPNR f5d2c200 10000 16 It should be possible to incorporate the variable-size adjustment without changing the display.

Hi Dave, I'm sorry, v4 is now missing the expansion of the strlen() argument. v5 (attached) should be the charm. --- David Mair SUSE Linux

----- Original Message ----- > Hi folks, > > I've just discovered that the crash utility fails to initialize the vm > subsystem properly on our latest SLES 32-bit kernels. It turns out that > our kernels are compiled with CONFIG_DISCONTIGMEM=y, which causes pgdat > structs to be allocated by the remap allocator (cf. > arch/x86/mm/numa_32.c and also the code in setup_node_data). > > If you don't know what the remap allocator is (like I didn't before I hit > the bug), it's a very special early-boot allocator which remaps physical > pages from low memory to high memory, giving them virtual addresses from > the > > identity mapping. Looks a bit like this: > physical addr > > +------------+ > > +------------+ > > +--> | KVA RAM | > > | +------------+ > | > | \/\/\/\/\/\/\/ > | /\/\/\/\/\/\/\ > > virtual addr | | highmem | > > +------------+ | |------------| > > | | -----> | | > > +------------+ | +------------+ > > | remap va | --+ | KVA PG | (unused) > > +------------+ +------------+ > > | | -----> | RAM bottom | > > +------------+ +------------+ > > This breaks a very basic assumption that crash makes about low-memory > virtual addresses. Hmmm, yeah, I am also unaware of this, and I'm not entirely clear based upon your explanation. What do "KVA PG" and "KVA RAM" mean exactly? And do just the pgdat structures (which I know can be huge) get moved from low to high physical memory (per-node perhaps), and then remapped with mapped virtual addresses?

Anyway, I trust you know what you're doing...

> The attached patch fixes the issue for me, but may not be the cleanest > method to handle these mappings. Anyway, what I can't wrap my head around is that the initialization sequence is being done by the first call to x86_ktop_PAE(), which calls x86_kvtop_remap(), which calls initialize_remap(), which calls readmem(), which calls x86_kvtop_PAE(), starting the whole thing over again. How does that recursion work? Would it be possible to call initialize_remap() earlier on instead of doing it upon the first kvtop() call?

> Ken'ichi Ohmichi, please note that makedumpfile is also affected by this > deficiency. On my test system, it will fail to produce any output if I > set dump level to anything greater than zero: > > makedumpfile -c -d 31 -x vmlinux-3.0.13-0.5-pae.debug vmcore kdump.31 > readmem: Can't convert a physical address(34a012b4) to offset. > readmem: type_addr: 0, addr:f4a012b4, size:4 > get_mm_discontigmem: Can't get node_start_pfn. > > makedumpfile Failed. > > However, fixing this for makedumpfile is harder, and it will most likely > require a few more lines in VMCOREINFO, because debug symbols may not be > available at dump time, and I can't see any alternative method to locate > the remapped regions. > > Regards, > Petr Tesarik > SUSE Linux -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Dne St 11. ledna 2012 00:37:50 Petr Tesarik napsal(a): > [...] > I can see now that this is unnecessarily complicated, because the > node_remap_* variables are static arrays of MAX_NUMNODES elements, so I > can get their size from the debuginfo at POST_GDB init and initialize a > machine-specific data type with it. I'll post another patch tomorrow. And here we go. Tested on my system and seems to work just fine. Petr Tesarik SUSE Linux

Dne St 11. ledna 2012 15:37:50 Dave Anderson napsal(a): > ----- Original Message ----- > > > Dne St 11. ledna 2012 00:37:50 Petr Tesarik napsal(a): > > > [...] > > > I can see now that this is unnecessarily complicated, because the > > > node_remap_* variables are static arrays of MAX_NUMNODES elements, so I > > > can get their size from the debuginfo at POST_GDB init and initialize a > > > machine-specific data type with it. I'll post another patch tomorrow. > > > > And here we go. Tested on my system and seems to work just fine. > > > > Petr Tesarik > > SUSE Linux > > Hi Petr, > > This looks pretty good to me. However, just to clarify the > chicken-and-egg situation here... > > When remap_init() does these 3 readmem() calls, they will pass > through x86_kvtop_remap() -- which I guess would fail because > the arrays would still be at least partially uninitialized?: Hi Dave, first of all, it's not a real issue, because kernel static data is always below the remapped addresses, but I agree, it's ugly and making non-obvious assumptions about memory layout. To make things clean, I can simply set machdep->machspec->max_numnodes as the last thing, so the loop in x86_kvtop_remap() cannot be executed until everything is initialized.

Patch attached. Petr Tesarik SUSE Linux

Hello Petr, On Tue, 10 Jan 2012 19:14:32 +0100 Petr Tesarik <ptesarik(a)suse.cz&gt; wrote: > Ken'ichi Ohmichi, please note that makedumpfile is also affected by this > deficiency. On my test system, it will fail to produce any output if I > set dump level to anything greater than zero: > > makedumpfile -c -d 31 -x vmlinux-3.0.13-0.5-pae.debug vmcore kdump.31 > readmem: Can't convert a physical address(34a012b4) to offset. > readmem: type_addr: 0, addr:f4a012b4, size:4 > get_mm_discontigmem: Can't get node_start_pfn. > > makedumpfile Failed. > > However, fixing this for makedumpfile is harder, and it will most likely > require a few more lines in VMCOREINFO, because debug symbols may not be > available at dump time, and I can't see any alternative method to locate > the remapped regions. Thank you for your indication. Could you send me your kernel configuration so that I can reproduce the issue ?