Re: [Crash-utility] Problem with "xm save" x86-64 cores - crash.4.0-3.5

Tejasvi Aswathanarayana wrote: Interesting -- I got a private message yesterday from another 2.6.16-era xen user with the same problem. Anyway, over time the manner of pid table handling has changed several times, and so to handle the various manners, the crash utility's task_init() function assigns an appropriate function to gather all pids/tasks running on the system. You're referring to this entry in the changelog: 4.0-2.24 - Fix for 2.6.17 kernels that do not use "pgdat_list" memory node list header, which would cause crash to fail during initialization with a "crash: cannot resolve: pgdat_list" error message. (anderson(a)redhat.com) - Fix for 2.6.17 kernels that have re-worked the kernel pid_hash handling, which would cause crash to fail during initialization with a "crash: cannot determine pid_hash array dimensions" error message. (anderson(a)redhat.com) ... The change above implemented yet another pid hash handler called refresh_hlist_task_table_v2(), which as the comments above indicate, was required for 2.6.17 kernels. In crash version 4.0-2.24, a new refresh_hlist_task_table_v2() function was added to replace refresh_hlist_task_table(), to account for this change: /* * 2.6.17 replaced: * static struct hlist_head *pid_hash[PIDTYPE_MAX]; * with * static struct hlist_head *pid_hash; */ So, for starters, can you display how "pid_hash" is declared in your kernel? Unfortunately the only 2.6.16-era x86_64 kernel dumpfile that I have on-hand as a reference is an early kdump dumpfile, (non-xen) which selects the older refresh_hlist_task_table(): crash> sys KERNEL: /usr/dumps/kdump/vmlinux (2.6.16-20-smp) DUMPFILE: /usr/dumps/kdump/vmcore CPUS: 2 DATE: Mon Apr 24 11:02:03 2006 UPTIME: 00:31:04 LOAD AVERAGE: 0.00, 0.00, 0.00 TASKS: 63 NODENAME: elm3a242 RELEASE: 2.6.16-20-smp VERSION: #1 SMP Mon Apr 10 04:51:13 UTC 2006 MACHINE: x86_64 (3000 Mhz) MEMORY: 4.6 GB PANIC: "SysRq : Trigger a crashdump" crash> help -t current: 1678f60 [62] .pid: 3235 .comm: "bash" .task: ffff810121a7d810 .thread_info: ffff81011d826000 .processor: 1 .ptask: ffff810121d91790 .mm_struct: ffff810122f9eb00 .tc_next: 0 context_array: 1677df0 refresh_task_table: refresh_hlist_task_table() ... This is the code sequence in task_init() that selects refresh_hlist_task_table() or refresh_hlist_task_table_v2(): } else { tt->pidhash_addr = symbol_value("pid_hash"); if (!get_array_length("pid_hash", NULL, sizeof(void *)) && VALID_STRUCT(pid_link)) tt->refresh_task_table = refresh_hlist_task_table_v2; else tt->refresh_task_table = refresh_hlist_task_table; } Since you are breaking down on initialization, can you put some debug code in place that displays?: 1. the output of the get_array_length("pid_hash",...) call, and 2. the output of VALID_STRUCT(pid_link) Something must be slightly different between the mainline and xen 2.6.16-era kernels. What appears to be happening is that refresh_hlist_task_table() is being selected, but in that function, this subsequent call is failing: if (!(plen = get_array_length("pid_hash", NULL, sizeof(void *)))) error(FATAL, "cannot determine pid_hash array dimensions\n"); Alternatively, if you want to make the vmlinux/dumpfile pair available to me, I can take a look at it. Thanks, Dave