Hello List, I've a two ELF coredumps from two different HyperV VMs generated by this tool (https://github.com/Azure/azure-linux-utils/tree/master/vm2core). Crash works with one of these coredumps but do not work with other. I've placed the output generated by crash tool here: Not ok with crash: ./crash/crash /usr/lib/debug/boot/vmlinux-4.15.0-88-generic vm1_numa_4gb_5cpu.coredump --kaslr 600000 -m phys_base=4355784704 -d8 https://raw.githubusercontent.com/santoshx/temp/master/notok_with_crash.txt Ok with crash: ./crash/crash /usr/lib/debug/boot/vmlinux-4.15.0-88-generic vm1_nonuma_4gb_5cpu.coredump --kaslr 3c00000 -m phys_base=2344615936 -d8 https://raw.githubusercontent.com/santoshx/temp/master/ok_with_crash.txt The problem I see that in non-working case crash fails to detect correct cpu_possible_mask: Relevant part of $ diff ok_with_crash.txt notok_with_crash.txt: < cpu_active_mask: cpus: 0 1 2 3 4 < FREEBUF(0) < <readmem: ffffffff86039f40, KVADDR, "pv_init_ops", 8, (ROE), 7ffe01722870> < <read_kdump: addr: ffffffff86039f40 paddr: 91c39f40 cnt: 8> < read_netdump: addr: ffffffff86039f40 paddr: 91c39f40 cnt: 8 offset: 91c3a760 --- > <readmem: ffffffff826f2b60, KVADDR, "possible", 1024, (ROE), > 5638a35a2280> > <read_kdump: addr: ffffffff826f2b60 paddr: 1060f2b60 cnt: 1024> > read_netdump: addr: ffffffff826f2b60 paddr: 1060f2b60 cnt: 1024 offset: > fe0f3380 > cpu_possible_mask: cpus: 3 4 5 6 8 13 14 18 20 21 22 26 28 29 30 33 36 > 37 38 48 49 52 53 54 56 59 60 61 62 64 65 68 69 70 72 73 74 75 76 78 82 > 83 85 86 90 91 93 94 96 99 101 102 104 105 108 109 110 114 116 117 118 > 123 124 125 126 128 133 134 138 140 141 142 146 148 149 150 153 156 157 > 158 168 169 172 173 174 176 179 180 181 182 184 185 188 189 190 192 193 > 194 195 196 198 200 202 205 206 211 212 213 214 216 219 221 222 226 228 > 229 230 232 233 234 235 236 238 242 243 245 246 248 251 253 254 256 257 > 260 261 262 266 268 269 270 275 276 277 278 280 285 286 290 292 293 294 > 298 300 301 302 305 308 309 310 320 321 324 325 326 328 331 332 333 334 > 336 337 340 341 342 344 345 346 347 348 350 352 354 357 358 361 362 363 > 365 366 370 372 373 374 376 378 381 382 385 388 389 390 392 393 394 395 > 396 398 402 403 405 406 408 411 413 414 416 417 420 421 422 426 428 429 > 430 435 436 437 438 440 445 446 450 452 453 454 458 460 461 462 465 468 > 469 470 480 481 484 485 486 488 491 492 493 494 496 497 500 50 1 502 504 505 506 507 508 510 514 515 517 518 520 523 525 526 528 529 532 533 534 538 540 541 542 547 548 549 I'm trying to find where the problem is? in the crash too or the tool that generated the ELF coredumps?

On Wed, Mar 4, 2020 at 2:49 PM Dave Anderson <anderson(a)prospeed.net&gt; wrote: > > > Hello List, > > > > I've a two ELF coredumps from two different HyperV VMs generated by this > > tool (https://github.com/Azure/azure-linux-utils/tree/master/vm2core). > > > > Crash works with one of these coredumps but do not work with other. > > > > I've placed the output generated by crash tool here: > > > > Not ok with crash: > > ./crash/crash /usr/lib/debug/boot/vmlinux-4.15.0-88-generic > > vm1_numa_4gb_5cpu.coredump --kaslr 600000 -m phys_base=4355784704 -d8 > > https://raw.githubusercontent.com/santoshx/temp/master/notok_with_crash.txt > > > > Ok with crash: > > ./crash/crash /usr/lib/debug/boot/vmlinux-4.15.0-88-generic > > vm1_nonuma_4gb_5cpu.coredump --kaslr 3c00000 -m phys_base=2344615936 -d8 > > https://raw.githubusercontent.com/santoshx/temp/master/ok_with_crash.txt > > > > > > The problem I see that in non-working case crash fails to detect correct > > cpu_possible_mask: > > > > Relevant part of $ diff ok_with_crash.txt notok_with_crash.txt: > > > > < cpu_active_mask: cpus: 0 1 2 3 4 > > < FREEBUF(0) > > < <readmem: ffffffff86039f40, KVADDR, "pv_init_ops", 8, (ROE), > > 7ffe01722870> > > < <read_kdump: addr: ffffffff86039f40 paddr: 91c39f40 cnt: 8> > > < read_netdump: addr: ffffffff86039f40 paddr: 91c39f40 cnt: 8 offset: > > 91c3a760 > > --- > >> <readmem: ffffffff826f2b60, KVADDR, "possible", 1024, (ROE), > >> 5638a35a2280> > >> <read_kdump: addr: ffffffff826f2b60 paddr: 1060f2b60 cnt: 1024> > >> read_netdump: addr: ffffffff826f2b60 paddr: 1060f2b60 cnt: 1024 offset: > >> fe0f3380 > >> cpu_possible_mask: cpus: 3 4 5 6 8 13 14 18 20 21 22 26 28 29 30 33 36 > >> 37 38 48 49 52 53 54 56 59 60 61 62 64 65 68 69 70 72 73 74 75 76 78 82 > >> 83 85 86 90 91 93 94 96 99 101 102 104 105 108 109 110 114 116 117 118 > >> 123 124 125 126 128 133 134 138 140 141 142 146 148 149 150 153 156 157 > >> 158 168 169 172 173 174 176 179 180 181 182 184 185 188 189 190 192 193 > >> 194 195 196 198 200 202 205 206 211 212 213 214 216 219 221 222 226 228 > >> 229 230 232 233 234 235 236 238 242 243 245 246 248 251 253 254 256 257 > >> 260 261 262 266 268 269 270 275 276 277 278 280 285 286 290 292 293 294 > >> 298 300 301 302 305 308 309 310 320 321 324 325 326 328 331 332 333 334 > >> 336 337 340 341 342 344 345 346 347 348 350 352 354 357 358 361 362 363 > >> 365 366 370 372 373 374 376 378 381 382 385 388 389 390 392 393 394 395 > >> 396 398 402 403 405 406 408 411 413 414 416 417 420 421 422 426 428 429 > >> 430 435 436 437 438 440 445 446 450 452 453 454 458 460 461 462 465 468 > >> 469 470 480 481 484 485 486 488 491 492 493 494 496 497 500 50 > > 1 502 504 505 506 507 508 510 514 515 517 518 520 523 525 526 528 529 532 > > 533 534 538 540 541 542 547 548 549 > > > > I'm trying to find where the problem is? in the crash too or the tool that > > generated the ELF coredumps? > > I suspect that it's a problem with either the --kaslr offset and/or > the phys_base value that you have used. Is there method to know or print kaslr & phy_base in a running Linux system?

> > It appears that the read of the cpu_possible mask is not using the > correct virtual address, or perhaps the wrong physical address, and > as a result it is trying to translate bogus data. In fact, the full > output txt file shows that every thing that it reads is garbage, e.g., > the cpu masks, the utsname data structure, the linux_banner string, etc. > > Dave > > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility >

> I suspect that it's a problem with either the --kaslr offset and/or > the phys_base value that you have used. Is there method to know or print kaslr & phy_base in a running Linux system?

On Thu, Mar 5, 2020 at 12:54 PM Dave Anderson <anderson(a)redhat.com&gt; wrote: > > > > I suspect that it's a problem with either the --kaslr offset and/or > > > the phys_base value that you have used. > > > > Is there method to know or print kaslr & phy_base in a running Linux system? > > They are normally passed in the VMCOREINFO data that is contained in an ELF PT_NOTE > in the dumpfile header. For example, here's a dump of the normal VMCOREINFO data, > where the phys_base and KASLR offsets are down near the bottom: > > OSRELEASE=4.18.0-185.el8.x86_64 > PAGESIZE=4096 > SYMBOL(init_uts_ns)=ffffffffbd812540 > SYMBOL(node_online_map)=ffffffffbda0f520 > SYMBOL(swapper_pg_dir)=ffffffffbd80a000 > SYMBOL(_stext)=ffffffffbc600000 > SYMBOL(vmap_area_list)=ffffffffbd8d78b0 > SYMBOL(mem_section)=ffff956a3ffd2000 > LENGTH(mem_section)=2048 > SIZE(mem_section)=16 > OFFSET(mem_section.section_mem_map)=0 > SIZE(page)=64 > SIZE(pglist_data)=171968 > SIZE(zone)=1472 > SIZE(free_area)=88 > SIZE(list_head)=16 > SIZE(nodemask_t)=128 > OFFSET(page.flags)=0 > OFFSET(page._refcount)=52 > OFFSET(page.mapping)=24 > OFFSET(page.lru)=8 > OFFSET(page._mapcount)=48 > OFFSET(page.private)=40 > OFFSET(page.compound_dtor)=16 > OFFSET(page.compound_order)=17 > OFFSET(page.compound_head)=8 > OFFSET(pglist_data.node_zones)=0 > OFFSET(pglist_data.nr_zones)=171232 > OFFSET(pglist_data.node_start_pfn)=171240 > OFFSET(pglist_data.node_spanned_pages)=171256 > OFFSET(pglist_data.node_id)=171264 > OFFSET(zone.free_area)=192 > OFFSET(zone.vm_stat)=1296 > OFFSET(zone.spanned_pages)=112 > OFFSET(free_area.free_list)=0 > OFFSET(list_head.next)=0 > OFFSET(list_head.prev)=8 > OFFSET(vmap_area.va_start)=0 > OFFSET(vmap_area.list)=48 > LENGTH(zone.free_area)=11 > SYMBOL(log_buf)=ffffffffbd85b140 > SYMBOL(log_buf_len)=ffffffffbd85b13c > SYMBOL(log_first_idx)=ffffffffbe319778 > SYMBOL(clear_idx)=ffffffffbe319744 > SYMBOL(log_next_idx)=ffffffffbe319768 > SIZE(printk_log)=16 > OFFSET(printk_log.ts_nsec)=0 > OFFSET(printk_log.len)=8 > OFFSET(printk_log.text_len)=10 > OFFSET(printk_log.dict_len)=12 > LENGTH(free_area.free_list)=5 > NUMBER(NR_FREE_PAGES)=0 > NUMBER(PG_lru)=5 > NUMBER(PG_private)=12 > NUMBER(PG_swapcache)=9 > NUMBER(PG_swapbacked)=18 > NUMBER(PG_slab)=8 > NUMBER(PG_hwpoison)=22 > NUMBER(PG_head_mask)=32768 > NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129 > NUMBER(HUGETLB_PAGE_DTOR)=2 > NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257 > ===============> NUMBER(phys_base)=16437477376 > SYMBOL(init_top_pgt)=ffffffffbd80a000 > NUMBER(pgtable_l5_enabled)=0 > SYMBOL(node_data)=ffffffffbda0ad20 > LENGTH(node_data)=1024 > ===============> KERNELOFFSET=3b600000 > NUMBER(KERNEL_IMAGE_SIZE)=1073741824 > NUMBER(sme_mask)=0 > CRASHTIME=1583350919 > > But in your Azure-generated dumpfile, I note that each cpu's NT_PRSTATUS note > contains junk data, and while does have a VMCOREINFO note, it contains this: > > Elf64_Nhdr: > n_namesz: 11 ("VMCOREINFO") > n_descsz: 42 > n_type: 0 (unused) > FAKE1=IGNORE1 > FAKE2=IGNORE2 > FAKE3=IGNORE3 > > So that's why you need to pass in the two arguments. > > Now, the crash utility should be able to be brought up successfully > on a live system without passing the arguments. And once you've done > that, you could get the values like this: > > crash> help -m | grep phys_base > phys_base: 3d3c00000 > crash> help -k | grep relocate > relocate: ffffffffc4a00000 (KASLR offset: 3b600000 / 950MB) > crash> > > But since they change with each reboot, you would have to capture them > while running on the live system, and save them somewhere for a subsequent > crash. So that goes back to my question -- how did you get the numbers > that you used? The number I had got by simply grepping through coredump strings. $ strings vm1_numa_4gb_5cpu.coredump | grep -v strings | grep 'KERNELOFFSET=\|NUMBER(phys_base)=' Machine is still running and I cross verified those numbers with crash and those were correct. crash> p vmcoreinfo_data+1600 $1 = (unsigned char *) 0xffff917d3cde1640 "poison)=22\nNUMBER(PG_head_mask)=32768\nNUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128\nNUMBER(HUGETLB_PAGE_DTOR)=2\nNUMBER(phys_base)=4355784704\nSYMBOL(init_top_pgt)=ffffffff82a0a000\nSYMBOL(node_data)=ffffffff82c5d780\nLENGTH(node_data)=1024\nKERNELOFFSET=600000\nNUMBER"... Now it appears to me that something wrong in Azure generated dump file.

> > Dave > > > > > > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility >

----- Original Message ----- > On Thu, Mar 5, 2020 at 1:07 PM Santosh <ysan99(a)gmail.com&gt; wrote: > > > > On Thu, Mar 5, 2020 at 12:54 PM Dave Anderson <anderson(a)redhat.com&gt; wrote: > > > > > > > > I suspect that it's a problem with either the --kaslr offset and/or > > > > > the phys_base value that you have used. > > > > > > > > Is there method to know or print kaslr & phy_base in a running Linux > > > > system? > > > > > > They are normally passed in the VMCOREINFO data that is contained in an > > > ELF PT_NOTE > > > in the dumpfile header. For example, here's a dump of the normal > > > VMCOREINFO data, > > > where the phys_base and KASLR offsets are down near the bottom: > > > > > > OSRELEASE=4.18.0-185.el8.x86_64 > > > PAGESIZE=4096 > > > SYMBOL(init_uts_ns)=ffffffffbd812540 > > > SYMBOL(node_online_map)=ffffffffbda0f520 > > > SYMBOL(swapper_pg_dir)=ffffffffbd80a000 > > > SYMBOL(_stext)=ffffffffbc600000 > > > SYMBOL(vmap_area_list)=ffffffffbd8d78b0 > > > SYMBOL(mem_section)=ffff956a3ffd2000 > > > LENGTH(mem_section)=2048 > > > SIZE(mem_section)=16 > > > OFFSET(mem_section.section_mem_map)=0 > > > SIZE(page)=64 > > > SIZE(pglist_data)=171968 > > > SIZE(zone)=1472 > > > SIZE(free_area)=88 > > > SIZE(list_head)=16 > > > SIZE(nodemask_t)=128 > > > OFFSET(page.flags)=0 > > > OFFSET(page._refcount)=52 > > > OFFSET(page.mapping)=24 > > > OFFSET(page.lru)=8 > > > OFFSET(page._mapcount)=48 > > > OFFSET(page.private)=40 > > > OFFSET(page.compound_dtor)=16 > > > OFFSET(page.compound_order)=17 > > > OFFSET(page.compound_head)=8 > > > OFFSET(pglist_data.node_zones)=0 > > > OFFSET(pglist_data.nr_zones)=171232 > > > OFFSET(pglist_data.node_start_pfn)=171240 > > > OFFSET(pglist_data.node_spanned_pages)=171256 > > > OFFSET(pglist_data.node_id)=171264 > > > OFFSET(zone.free_area)=192 > > > OFFSET(zone.vm_stat)=1296 > > > OFFSET(zone.spanned_pages)=112 > > > OFFSET(free_area.free_list)=0 > > > OFFSET(list_head.next)=0 > > > OFFSET(list_head.prev)=8 > > > OFFSET(vmap_area.va_start)=0 > > > OFFSET(vmap_area.list)=48 > > > LENGTH(zone.free_area)=11 > > > SYMBOL(log_buf)=ffffffffbd85b140 > > > SYMBOL(log_buf_len)=ffffffffbd85b13c > > > SYMBOL(log_first_idx)=ffffffffbe319778 > > > SYMBOL(clear_idx)=ffffffffbe319744 > > > SYMBOL(log_next_idx)=ffffffffbe319768 > > > SIZE(printk_log)=16 > > > OFFSET(printk_log.ts_nsec)=0 > > > OFFSET(printk_log.len)=8 > > > OFFSET(printk_log.text_len)=10 > > > OFFSET(printk_log.dict_len)=12 > > > LENGTH(free_area.free_list)=5 > > > NUMBER(NR_FREE_PAGES)=0 > > > NUMBER(PG_lru)=5 > > > NUMBER(PG_private)=12 > > > NUMBER(PG_swapcache)=9 > > > NUMBER(PG_swapbacked)=18 > > > NUMBER(PG_slab)=8 > > > NUMBER(PG_hwpoison)=22 > > > NUMBER(PG_head_mask)=32768 > > > NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129 > > > NUMBER(HUGETLB_PAGE_DTOR)=2 > > > NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257 > > > ===============> NUMBER(phys_base)=16437477376 > > > SYMBOL(init_top_pgt)=ffffffffbd80a000 > > > NUMBER(pgtable_l5_enabled)=0 > > > SYMBOL(node_data)=ffffffffbda0ad20 > > > LENGTH(node_data)=1024 > > > ===============> KERNELOFFSET=3b600000 > > > NUMBER(KERNEL_IMAGE_SIZE)=1073741824 > > > NUMBER(sme_mask)=0 > > > CRASHTIME=1583350919 > > > > > > But in your Azure-generated dumpfile, I note that each cpu's NT_PRSTATUS > > > note > > > contains junk data, and while does have a VMCOREINFO note, it contains > > > this: > > > > > > Elf64_Nhdr: > > > n_namesz: 11 ("VMCOREINFO") > > > n_descsz: 42 > > > n_type: 0 (unused) > > > FAKE1=IGNORE1 > > > FAKE2=IGNORE2 > > > FAKE3=IGNORE3 > > > > > > So that's why you need to pass in the two arguments. > > > > > > Now, the crash utility should be able to be brought up successfully > > > on a live system without passing the arguments. And once you've done > > > that, you could get the values like this: > > > > > > crash> help -m | grep phys_base > > > phys_base: 3d3c00000 > > > crash> help -k | grep relocate > > > relocate: ffffffffc4a00000 (KASLR offset: 3b600000 / 950MB) > > > crash> > > > > > > But since they change with each reboot, you would have to capture them > > > while running on the live system, and save them somewhere for a > > > subsequent > > > crash. So that goes back to my question -- how did you get the numbers > > > that you used? > > > > The number I had got by simply grepping through coredump strings. > > $ strings vm1_numa_4gb_5cpu.coredump | grep -v strings | grep > > 'KERNELOFFSET=\|NUMBER(phys_base)=' > > > > Machine is still running and I cross verified those numbers with crash > > and those were correct. > > > > crash> p vmcoreinfo_data+1600 > > $1 = (unsigned char *) 0xffff917d3cde1640 > > "poison)=22\nNUMBER(PG_head_mask)=32768\nNUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128\nNUMBER(HUGETLB_PAGE_DTOR)=2\nNUMBER(phys_base)=4355784704\nSYMBOL(init_top_pgt)=ffffffff82a0a000\nSYMBOL(node_data)=ffffffff82c5d780\nLENGTH(node_data)=1024\nKERNELOFFSET=600000\nNUMBER"... > > > > Now it appears to me that something wrong in Azure generated dump file. > > Something to do with numa: > > santosh@u1804lts:~$ cat /proc/sys/kernel/numa_balancing > 1 > > HyperV VM with 1 numa node (numa_balancing = 0) -- Linux with nokaslr > -- vm2core -- ELF coredump -- crash tool -- Ok > HyperV VM with 1 numa node (numa_balancing = 0) -- LInux with kaslr -- > vm2core -- ELF coredump -- crash tool -- Ok > HyperV VM with 2 numa nodes (numa_balancing = 1) -- Linux with nokaslr > -- vm2core -- ELF coredump -- crash tool -- Ok > HyperV VM with 2 numa nodes (numa_balancing = 1) -- LInux with kaslr > -- vm2core -- ELF coredump -- crash tool -- Not ok > > Do we have to specify the numa topology somehow to crash tool or it > should already be handled in coredump file? Definitely not. The crash utility is only interested in: 1. kernel virtual address values -- which KASLR modifies from the values compiled into the vmlinux file, 2. translating those kernel virtual addresses into physical addresses, and 3. accessing those physical addresses from the memory source. As I understand it, numa_balancing is concerned with user-space virtual address mapping, where the kernel may re-map an underlying physical address from one NUMA node to another. User-space memory is never accessed by the crash utility unless requested by a run-time command that specifically specifies it. Dave

-- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility