Re: [Crash-utility] [PATCH] Add -T option to configure task table via a file

> -----Original Message----- > ----- Original Message ----- > > I faced an incomplete vmcore previous week which was generated because > > system running in kdump kernel was somehow rebooted in the middle of > > copying vmcore. > > > > Unfortunately, in the incomplete vmcore, most of the tasks failed to > > be detected via PID hash table because the objects relevant to PID > > hash including ptes needed to refer to the objects were lost. > > > > Although I successfully found many of objects of task_struct from > > another data structure such as via a circular list of task_struct::tasks > > and via run queue, crash sub-commands never work with the following > > message if a given object is not contained in the task table: > > > > crash> bt 0xffffffff00000000 > > bt: invalid task or pid value: 0xffffffff00000000 > > > > To address this issue, I made a patch to add a command-line option > > to pass a list of addresses of task_struct objects to make crash > > try to detect them in task table. > > > > I made this in very short time and there may be better interface > > than command-line option. > > > > Tested on top of crash-7.2.5. > > Yeah, what bothers me about this patch is that even though it worked for > your > particular half-baked vmcore, it may never be of any help to anybody else > in the future. > > It's similar in nature to patches that have posted that address a > particular > unique kernel bug that was seen in one vmcore, but it would be highly > unlikely > that the circumstances would ever be seen again. I'm posting this patch because I think this could be useful for everyone... It was unfortunate that incomplete vmcore was generated and sent to us but there's case where engineers have to investigate issues based on the incomplete vmcore. I also think there would other cases where restoring task_struct could fail due to pure software issues, for example, memory corruption bugs, and think it natural that crash doesn't behave as expected when kernel data structures are abnormal state. I think options such as --minimal and --no_kmem_cache are to deal with such cases and this feature is similar in this sense.

By the way, I feel like I saw vmcores where some error messages were output during "(gathering task table data)" in the past and I guess some tasks were missing there but this was the first case I actually needed to try to restore them. > > But in this case, it's even more unlikely given that it's dealing with > an incomplete vmcore. You were lucky that you were able to even > bring up a crash session at all -- and then were able to generate > a task list after that. It was incomplete but was complete about 98%. The detection from PID hash was affected by loss of the remaining 2%. > > Following the task_struct.tasks list doesn't gather all of the > tasks in a task group, so it doesn't create a fully populated task > list, correct? Yes, I needed to repeat iterating successfully detected task_struct objects in order until all the target tasks were covered, and as your guess, there's no guarantee that I found all the task_struct objects, so I said 'many of'.

> > Plus it doesn't make sense to add it unless it's documented *how* to > create the task list to begin with. How about writing use case in help message or/and manual page? -T file Make crash detect task_struct objects listed in file as in task table. This is useful when your interesting tasks are missing in task table. You may find your interesting task_struct objects from various kernel data structures: From task_struct::tasks: crash> list task_struct.tasks -s task_struct.pid,comm -h ffff88003daa8000 ffff88003daa8000 pid = 1 comm = "systemd\000\060\000\000\000\000\000\000" ffff88003daa8fd0 pid = 2 comm = "kthreadd\000\000\000\000\000\000\000" ffff88003daa9fa0 pid = 3 comm = "ksoftirqd/0\000\000\000\000" ...<snip>... From runqueue: crash> runq CPU 0 RUNQUEUE: ffff88003fc16cc0 CURRENT: PID: 2188 TASK: ffff8800360f8000 COMMAND: "foobar.sh" RT PRIO_ARRAY: ffff88003fc16e50 [no tasks queued] CFS RB_ROOT: ffff88003fc16d68 [no tasks queued] CPU 1 RUNQUEUE: ffff88003fd16cc0 CURRENT: PID: 1 TASK: ffff88003daa8000 COMMAND: "systemd" RT PRIO_ARRAY: ffff88003fd16e50 [no tasks queued] CFS RB_ROOT: ffff88003fd16d68 [120] PID: 19054 TASK: ffff88000b684f10 COMMAND: "kworker/1:0" [120] PID: 3863 TASK: ffff88003bd02f70 COMMAND: "emacs" This might be lengthy under option section.

> > I don't know, let me think about this... I don't think the current design is best. For example, it might be better to be able to update task table at runtime by some crash sub-command. Looking at source code, it appears that task table is updated at command execution when needed, so it's not so difficult? void exec_command(void) { ...<snip>... if ((ct = get_command_table_entry(args[0]))) { if (ct->flags & REFRESH_TASK_TABLE) { if (XEN_HYPER_MODE()) { #ifdef XEN_HYPERVISOR_ARCH xen_hyper_refresh_domain_context_space(); xen_hyper_refresh_vcpu_context_space(); #else error(FATAL, XEN_HYPERVISOR_NOT_SUPPORTED); #endif } else if (!(pc->flags & MINIMAL_MODE)) { tt->refresh_task_table() sort_context_array(); sort_tgid_array(); } }