6:31 a.m.
Hello Dave,
I'd like to discuss about the following feature with you and get some advise.
vm command is used to display virtual memory information of a task. But if the
task is exiting(according to crash, 'tsk->flags & PF_EXITING' is true), vm
will
set mm to 0 in get_task_mem_usage(). But the mm may be not freed yet, the mm and
its related virtual information is helpful when debuging a exiting task.
I was considering to ignore the IS_EXITING(task) in get_task_mem_usage() and
if tsk->mm is set to NULL but the mm is not freed(see the following case), then we
can specify the mm manually.
CASE(the code is from kernel):
<cut>
exit_mm()
{
...
tsk->mm = NULL; --> dump after this, and before mmput() freeing mm
...
mmput(mm);
}
<cut>
But I guess it is not a good design to you. So I reconsidered it. What about specifying
mm to vm just like task's pid or address. Then vm can retrieve virtual memory
information
from specified mm directly. And get the owner task from mm->owner.
--
Regards
Qiao Nuohan
8 a.m.
----- Original Message -----
Hello Dave,
I'd like to discuss about the following feature with you and get some advise.
vm command is used to display virtual memory information of a task. But if the
task is exiting(according to crash, 'tsk->flags & PF_EXITING' is true), vm
will
set mm to 0 in get_task_mem_usage(). But the mm may be not freed yet, the mm and
its related virtual information is helpful when debuging a exiting task.
But it may have been freed, and in the case of CONFIG_SLUB, the mm_struct.mmap member
would be overwritten as a free slab object link pointer, making it useless. Or it
could have been freed-and-reused.
I was considering to ignore the IS_EXITING(task) in get_task_mem_usage() and
if tsk->mm is set to NULL but the mm is not freed(see the following case), then we
can specify the mm manually.
CASE(the code is from kernel):
<cut>
exit_mm()
{
...
tsk->mm = NULL; --> dump after this, and before mmput() freeing mm
...
mmput(mm);
}
<cut>
But I guess it is not a good design to you. So I reconsidered it. What about specifying
mm to vm just like task's pid or address. Then vm can retrieve virtual memory
information
from specified mm directly. And get the owner task from mm->owner.
That might work, at least if:
(1) the mm_struct has not been freed (SLUB),
(2) the mm_struct has not been freed-and-reused, and
(3) the kernel is configured with CONFIG_MEMCG and mm->owner points to the exiting
task.
But how would a typical user of this option know what the mm_struct address is?
Dave
8:53 a.m.
On 12/02/2014 10:00 PM, Dave Anderson wrote:
----- Original Message -----
> Hello Dave,
>
> I'd like to discuss about the following feature with you and get some advise.
>
> vm command is used to display virtual memory information of a task. But if the
> task is exiting(according to crash, 'tsk->flags& PF_EXITING' is
true), vm will
> set mm to 0 in get_task_mem_usage(). But the mm may be not freed yet, the mm and
> its related virtual information is helpful when debuging a exiting task.
But it may have been freed, and in the case of CONFIG_SLUB, the mm_struct.mmap member
would be overwritten as a free slab object link pointer, making it useless. Or it
could have been freed-and-reused.
>
> I was considering to ignore the IS_EXITING(task) in get_task_mem_usage() and
> if tsk->mm is set to NULL but the mm is not freed(see the following case), then
we
> can specify the mm manually.
>
> CASE(the code is from kernel):
> <cut>
> exit_mm()
> {
> ...
> tsk->mm = NULL; --> dump after this, and before mmput() freeing mm
> ...
> mmput(mm);
> }
> <cut>
>
> But I guess it is not a good design to you. So I reconsidered it. What about
specifying
> mm to vm just like task's pid or address. Then vm can retrieve virtual memory
information
> from specified mm directly. And get the owner task from mm->owner.
The thought above is to make specifying mm to be a more common use, just like specifying
pid or taskp. Users may get mm from back trace, then he doesn't need to find its
owner(task)
first. So it is just another way to display virtual memory information of a task.
For the special cases, the task->mm is exiting but mm is not freed (or we can judge it
by
checking whether mm->mm_users is 0), the original vm command is not available to
display
virtual memory information. But I think there are people care about virtual memory even
when the task is exiting. Since the mm is still there, why doesn't crash show them?
That might work, at least if:
(1) the mm_struct has not been freed (SLUB),
(2) the mm_struct has not been freed-and-reused, and
(3) the kernel is configured with CONFIG_MEMCG and mm->owner points to the exiting
task.
But how would a typical user of this option know what the mm_struct address is?
It is true there are people who don't know what mm_struct is. But there are still some
people
who know what mm is, especially for those who tries to debug exiting tasks.
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Regards
Qiao Nuohan
9:12 a.m.
----- Original Message -----
On 12/02/2014 10:00 PM, Dave Anderson wrote:
>
>
> ----- Original Message -----
>> Hello Dave,
>>
>> I'd like to discuss about the following feature with you and get some
>> advise.
>>
>> vm command is used to display virtual memory information of a task. But if the
>> task is exiting(according to crash, 'tsk->flags& PF_EXITING' is
true), vm will
>> set mm to 0 in get_task_mem_usage(). But the mm may be not freed yet, the mm
and
>> its related virtual information is helpful when debuging a exiting task.
>
> But it may have been freed, and in the case of CONFIG_SLUB, the mm_struct.mmap
member
> would be overwritten as a free slab object link pointer, making it useless. Or it
> could have been freed-and-reused.
>
>>
>> I was considering to ignore the IS_EXITING(task) in get_task_mem_usage() and
>> if tsk->mm is set to NULL but the mm is not freed(see the following case),
then we
>> can specify the mm manually.
>>
>> CASE(the code is from kernel):
>> <cut>
>> exit_mm()
>> {
>> ...
>> tsk->mm = NULL; --> dump after this, and before mmput() freeing mm
>> ...
>> mmput(mm);
>> }
>> <cut>
>>
>> But I guess it is not a good design to you. So I reconsidered it. What about
specifying
>> mm to vm just like task's pid or address. Then vm can retrieve virtual
memory information
>> from specified mm directly. And get the owner task from mm->owner.
>
The thought above is to make specifying mm to be a more common use, just like specifying
pid or taskp. Users may get mm from back trace, then he doesn't need to find its
owner(task)
first. So it is just another way to display virtual memory information of a task.
For the special cases, the task->mm is exiting but mm is not freed (or we can judge it
by
checking whether mm->mm_users is 0), the original vm command is not available to
display
virtual memory information. But I think there are people care about virtual memory even
when the task is exiting. Since the mm is still there, why doesn't crash show them?
> That might work, at least if:
>
> (1) the mm_struct has not been freed (SLUB),
> (2) the mm_struct has not been freed-and-reused, and
> (3) the kernel is configured with CONFIG_MEMCG and mm->owner points to
> the exiting task.
>
> But how would a typical user of this option know what the mm_struct address is?
It is true there are people who don't know what mm_struct is. But there are still
some people
who know what mm is, especially for those who tries to debug exiting tasks.
You misunderstood my question -- how would a typical user determine the *address* of the
mm_struct?
Have you actually tried this as an experiment? At a minimum, the option would have
to reject an mm_struct that has a mm->mm_users of 0.
In any case, feel free to post a patch and we'll see what it looks like.
Dave
2:11 p.m.
----- Original Message -----
----- Original Message -----
> On 12/02/2014 10:00 PM, Dave Anderson wrote:
> >
> >
> > ----- Original Message -----
> >> Hello Dave,
> >>
> >> I'd like to discuss about the following feature with you and get some
advise.
> >>
> >> vm command is used to display virtual memory information of a task. But if
the
> >> task is exiting(according to crash, 'tsk->flags&
PF_EXITING' is true), vm will
> >> set mm to 0 in get_task_mem_usage(). But the mm may be not freed yet, the
mm and
> >> its related virtual information is helpful when debuging a exiting task.
> >
> > But it may have been freed, and in the case of CONFIG_SLUB, the mm_struct.mmap
member
> > would be overwritten as a free slab object link pointer, making it useless. Or
it
> > could have been freed-and-reused.
> >
> >>
> >> I was considering to ignore the IS_EXITING(task) in get_task_mem_usage()
and
> >> if tsk->mm is set to NULL but the mm is not freed(see the following
case), then we
> >> can specify the mm manually.
> >>
> >> CASE(the code is from kernel):
> >> <cut>
> >> exit_mm()
> >> {
> >> ...
> >> tsk->mm = NULL; --> dump after this, and before mmput() freeing mm
> >> ...
> >> mmput(mm);
> >> }
> >> <cut>
> >>
> >> But I guess it is not a good design to you. So I reconsidered it. What
about specifying
> >> mm to vm just like task's pid or address. Then vm can retrieve virtual
memory information
> >> from specified mm directly. And get the owner task from mm->owner.
> >
>
> The thought above is to make specifying mm to be a more common use, just like
specifying
> pid or taskp. Users may get mm from back trace, then he doesn't need to find its
owner(task)
> first. So it is just another way to display virtual memory information of a task.
>
> For the special cases, the task->mm is exiting but mm is not freed (or we can
judge it by
> checking whether mm->mm_users is 0), the original vm command is not available to
display
> virtual memory information. But I think there are people care about virtual memory
even
> when the task is exiting. Since the mm is still there, why doesn't crash show
them?
>
> > That might work, at least if:
> >
> > (1) the mm_struct has not been freed (SLUB),
> > (2) the mm_struct has not been freed-and-reused, and
> > (3) the kernel is configured with CONFIG_MEMCG and mm->owner points to
> > the exiting task.
> >
> > But how would a typical user of this option know what the mm_struct address
is?
>
> It is true there are people who don't know what mm_struct is. But there are
still some people
> who know what mm is, especially for those who tries to debug exiting tasks.
You misunderstood my question -- how would a typical user determine the *address* of the
mm_struct?
Have you actually tried this as an experiment? At a minimum, the option would have
to reject an mm_struct that has a mm->mm_users of 0.
In any case, feel free to post a patch and we'll see what it looks like.
Dave
Hi Qiao,
Interestingly enough, today I was asked to look at a vmcore in which an oops
occurred during task exit after tsk->mm had been NULL'd out in exit_mm():
crash> bt
PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres"
#0 [ffff8804a5c31960] machine_kexec at ffffffff81038f3b
#1 [ffff8804a5c319c0] crash_kexec at ffffffff810c5d92
#2 [ffff8804a5c31a90] oops_end at ffffffff8152b510
#3 [ffff8804a5c31ac0] no_context at ffffffff8104a00b
#4 [ffff8804a5c31b10] __bad_area_nosemaphore at ffffffff8104a295
#5 [ffff8804a5c31b60] bad_area_nosemaphore at ffffffff8104a363
#6 [ffff8804a5c31b70] __do_page_fault at ffffffff8104aabf
#7 [ffff8804a5c31c90] do_page_fault at ffffffff8152d45e
#8 [ffff8804a5c31cc0] page_fault at ffffffff8152a815
[exception RIP: _spin_lock+14]
RIP: ffffffff8152a2fe RSP: ffff8804a5c31d78 RFLAGS: 00010282
RAX: 0000000000010000 RBX: ffff8804ac6368d0 RCX: ffff8804a219cf20
RDX: 00007ff0bd10a000 RSI: ffff88046d2b8200 RDI: 0000000000000000
RBP: ffff8804a5c31d78 R8: 00007ff0bd10a000 R9: 0000000000000005
R10: ffff8804800b6080 R11: 0000000000000000 R12: ffff880491b90b28
R13: ffff880491b90af8 R14: ffff880491b90b08 R15: ffff880810280be0
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff8804a5c31d80] unlink_anon_vmas at ffffffff81154144
#10 [ffff8804a5c31dc0] free_pgtables at ffffffff81146f61
#11 [ffff8804a5c31e10] exit_mmap at ffffffff8114e4a0
#12 [ffff8804a5c31e60] mmput at ffffffff8106f22c
#13 [ffff8804a5c31e80] exit_mm at ffffffff81076b6b
#14 [ffff8804a5c31ec0] do_exit at ffffffff81076f1f
#15 [ffff8804a5c31f40] do_group_exit at ffffffff81077688
#16 [ffff8804a5c31f70] sys_exit_group at ffffffff81077717
#17 [ffff8804a5c31f80] system_call_fastpath at ffffffff8100b072
RIP: 00000037e98ad028 RSP: 00007fff8b9bd750 RFLAGS: 00010246
RAX: 00000000000000e7 RBX: ffffffff8100b072 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
RBP: 0000000000000000 R8: 00000000000000e7 R9: ffffffffffffffa8
R10: 00000037e9b91808 R11: 0000000000000246 R12: ffffffff81077717
R13: ffff8804a5c31f78 R14: 0000000000000001 R15: 0000000000000000
ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b
crash>
And as expected, "vm" doesn't show much:
crash> vm
PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres"
MM PGD RSS TOTAL_VM
0 0 0k 0k
crash>
The mm_struct address was easy enough to find on stack, where both mm_struct
slab references were from the same object:
crash> bt -FF | grep mm_struct
ffff8804a5c31de8: [ffff880495120dc0:mm_struct] [ffff880495120dc0:mm_struct]
ffff8804a5c31df8: [ffff880495120e28:mm_struct] 00000000ffffffff
ffff8804a5c31e18: ffff880400000001 [ffff880495120dc0:mm_struct]
ffff8804a5c31e38: ffff8804a5c31e58 [ffff880495120dc0:mm_struct]
ffff8804a5c31e48: [ffff880495120e50:mm_struct] [ffff880495120dc0:mm_struct]
crash>
The mm_struct looks usable given the counter and the owner fields:
crash> mm_struct.mm_count,owner,mm_users ffff880495120dc0
mm_count = {
counter = 2
}
owner = 0xffff88049863f500
mm_users = {
counter = 0
}
crash>
And with the relatively simple attached patch, where "vm -M mm_struct"
overrides
the NULL pointer found in the task_struct, the basic command works in this case:
crash> vm -M ffff880495120dc0
PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres"
MM PGD RSS TOTAL_VM
0 0 0k 0k
VMA START END FLAGS FILE
ffff8804a085ce90 400000 f56000 8001875
/usr/local/greenplum-db-4.3.3.1/bin/postgres
ffff8804a085cdc8 1155000 1171000 8101873
/usr/local/greenplum-db-4.3.3.1/bin/postgres
ffff8804a085cd00 1171000 13b8000 100073
ffff8804a085cc38 1a68000 1b15000 100073
ffff8804a085cb70 1b15000 1b2f000 100073
ffff8802a965fdc8 1b2f000 1d81000 100073
ffff8804a085caa8 37e9000000 37e9020000 8000875 /lib64/ld-2.12.so
ffff8804a085c9e0 37e921f000 37e9220000 8100871 /lib64/ld-2.12.so
ffff8804a085c918 37e9220000 37e9221000 8100873 /lib64/ld-2.12.so
ffff8804a085c850 37e9221000 37e9222000 100073
ffff8804a085c788 37e9400000 37e9483000 8000075 /lib64/libm-2.12.so
ffff8804a085c6c0 37e9483000 37e9682000 8000070 /lib64/libm-2.12.so
ffff8804a085c5f8 37e9682000 37e9683000 8100071 /lib64/libm-2.12.so
ffff8804a085c530 37e9683000 37e9684000 8100073 /lib64/libm-2.12.so
ffff8804a085c468 37e9800000 37e998b000 8000075 /lib64/libc-2.12.so
ffff8804a085c3a0 37e998b000 37e9b8a000 8000070 /lib64/libc-2.12.so
ffff8804a085c2d8 37e9b8a000 37e9b8e000 8100071 /lib64/libc-2.12.so
ffff8804a085c210 37e9b8e000 37e9b8f000 8100073 /lib64/libc-2.12.so
ffff8804a085c148 37e9b8f000 37e9b94000 100073
ffff8804a085c080 37e9c00000 37e9c17000 8000075 /lib64/libpthread-2.12.so
ffff8804a91e9ed0 37e9c17000 37e9e17000 8000070 /lib64/libpthread-2.12.so
ffff8804a91e9e08 37e9e17000 37e9e18000 8100071 /lib64/libpthread-2.12.so
ffff8804a91e9d40 37e9e18000 37e9e19000 8100073 /lib64/libpthread-2.12.so
ffff8804a91e9c78 37e9e19000 37e9e1d000 100073
ffff8804a91e9bb0 37ea000000 37ea002000 8000075 /lib64/libdl-2.12.so
ffff8804a91e9ae8 37ea002000 37ea202000 8000070 /lib64/libdl-2.12.so
ffff8804a91e9a20 37ea202000 37ea203000 8100071 /lib64/libdl-2.12.so
ffff8804a91e9958 37ea203000 37ea204000 8100073 /lib64/libdl-2.12.so
ffff8804a91e9890 37ea800000 37ea807000 8000075 /lib64/librt-2.12.so
ffff8804a91e97c8 37ea807000 37eaa06000 8000070 /lib64/librt-2.12.so
ffff8804a91e9700 37eaa06000 37eaa07000 8100071 /lib64/librt-2.12.so
ffff8804a91e9638 37eaa07000 37eaa08000 8100073 /lib64/librt-2.12.so
ffff8804a91e9570 37eac00000 37eac16000 8000075 /lib64/libresolv-2.12.so
ffff8804a91e94a8 37eac16000 37eae16000 8000070 /lib64/libresolv-2.12.so
ffff8804a91e93e0 37eae16000 37eae17000 8100071 /lib64/libresolv-2.12.so
ffff8804a91e9318 37eae17000 37eae18000 8100073 /lib64/libresolv-2.12.so
ffff8804a91e9250 37eae18000 37eae1a000 100073
ffff8804a91e9188 37efc00000 37efc0c000 8000075 /lib64/libpam.so.0.82.2
ffff8804a91e90c0 37efc0c000 37efe0c000 8000070 /lib64/libpam.so.0.82.2
ffff880495cb0e90 37efe0c000 37efe0d000 8100071 /lib64/libpam.so.0.82.2
ffff880495cb0dc8 37efe0d000 37efe0e000 8100073 /lib64/libpam.so.0.82.2
ffff880495cb0d00 37f7400000 37f7432000 8000075 /lib64/libidn.so.11.6.1
ffff880495cb0c38 37f7432000 37f7631000 8000070 /lib64/libidn.so.11.6.1
ffff880495cb0b70 37f7631000 37f7632000 8100073 /lib64/libidn.so.11.6.1
ffff880495cb0aa8 37f7800000 37f7807000 8000075 /lib64/libcrypt-2.12.so
ffff880495cb09e0 37f7807000 37f7a07000 8000070 /lib64/libcrypt-2.12.so
ffff880495cb0918 37f7a07000 37f7a08000 8100071 /lib64/libcrypt-2.12.so
ffff880495cb0850 37f7a08000 37f7a09000 8100073 /lib64/libcrypt-2.12.so
ffff880495cb0788 37f7a09000 37f7a37000 100073
ffff880495cb06c0 37f7c00000 37f7c17000 8000075 /lib64/libaudit.so.1.0.0
ffff880495cb05f8 37f7c17000 37f7e16000 8000070 /lib64/libaudit.so.1.0.0
ffff880495cb0530 37f7e16000 37f7e17000 8100071 /lib64/libaudit.so.1.0.0
ffff880495cb0468 37f7e17000 37f7e1c000 8100073 /lib64/libaudit.so.1.0.0
ffff880495cb03a0 37f8000000 37f8071000 8000075 /lib64/libfreebl3.so
ffff880495cb02d8 37f8071000 37f8270000 8000070 /lib64/libfreebl3.so
ffff880495cb0210 37f8270000 37f8272000 8100071 /lib64/libfreebl3.so
ffff880495cb0148 37f8272000 37f8273000 8100073 /lib64/libfreebl3.so
ffff880495cb0080 37f8273000 37f8277000 100073
ffff88049a8ffed0 37fbe00000 37fbe16000 8000075 /lib64/libnsl-2.12.so
ffff88049a8ffe08 37fbe16000 37fc015000 8000070 /lib64/libnsl-2.12.so
ffff88049a8ffd40 37fc015000 37fc016000 8100071 /lib64/libnsl-2.12.so
ffff88049a8ffc78 37fc016000 37fc017000 8100073 /lib64/libnsl-2.12.so
ffff88049a8ffbb0 37fc017000 37fc019000 100073
ffff8804a5c32918 7ff0a0000000 7ff0a0028000 100073
ffff8804a5c32850 7ff0a0028000 7ff0a4000000 70
ffff8804977d1468 7ff0a48d5000 7ff0a4916000 100073
ffff88011028e148 7ff0a4937000 7ff0a4938000 100070
ffff8802a965fe90 7ff0a4938000 7ff0a4958000 100077
ffff88011028e080 7ff0a4958000 7ff0a4997000 100073
ffff88049a8ffae8 7ff0a4997000 7ff0a49a3000 8000075 /lib64/libnss_files-2.12.so
ffff88049a8ffa20 7ff0a49a3000 7ff0a4ba3000 8000070 /lib64/libnss_files-2.12.so
ffff88049a8ff958 7ff0a4ba3000 7ff0a4ba4000 8100071 /lib64/libnss_files-2.12.so
ffff88049a8ff890 7ff0a4ba4000 7ff0a4ba5000 8100073 /lib64/libnss_files-2.12.so
ffff88049a8ff7c8 7ff0a4ba5000 7ff0b4b2d000 80000fb /SYSV02faf469
ffff88049a8ff700 7ff0b4b2d000 7ff0ba9be000 8000071 /usr/lib/locale/locale-archive
ffff88049a8ff638 7ff0ba9be000 7ff0baa14000 100073
ffff88049a8ff570 7ff0baa14000 7ff0bace9000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libgpopt.so
ffff88049a8ff4a8 7ff0bace9000 7ff0baee9000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libgpopt.so
ffff88049a8ff3e0 7ff0baee9000 7ff0baf08000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libgpopt.so
ffff88049a8ff318 7ff0baf08000 7ff0baf0a000 100073
ffff88049a8ff250 7ff0baf0a000 7ff0baf19000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libgpdbcost.so
ffff88049a8ff188 7ff0baf19000 7ff0bb119000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libgpdbcost.so
ffff88049a8ff0c0 7ff0bb119000 7ff0bb11a000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libgpdbcost.so
ffff8804a5c2fe90 7ff0bb11a000 7ff0bb331000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libnaucrates.so
ffff8804a5c2fdc8 7ff0bb331000 7ff0bb531000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libnaucrates.so
ffff8804a5c2fd00 7ff0bb531000 7ff0bb55c000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libnaucrates.so
ffff8804a5c2fc38 7ff0bb55c000 7ff0bb55e000 100073
ffff8804a5c2fb70 7ff0bb55e000 7ff0bb9b6000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libxerces-c-3.1.so
ffff8804a5c2faa8 7ff0bb9b6000 7ff0bbbb6000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libxerces-c-3.1.so
ffff8804a5c2f9e0 7ff0bbbb6000 7ff0bbbfa000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libxerces-c-3.1.so
ffff8804a5c2f918 7ff0bbbfa000 7ff0bbbfb000 100073
ffff8804a5c2f850 7ff0bbbfb000 7ff0bbc11000 8000075 /lib64/libgcc_s-4.4.7-20120601.so.1
ffff8804a5c2f788 7ff0bbc11000 7ff0bbe10000 8000070 /lib64/libgcc_s-4.4.7-20120601.so.1
ffff8804a5c2f6c0 7ff0bbe10000 7ff0bbe11000 8100073 /lib64/libgcc_s-4.4.7-20120601.so.1
ffff8804a5c2f5f8 7ff0bbe11000 7ff0bbeff000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libstdc++.so.6
ffff8804a5c2f530 7ff0bbeff000 7ff0bc0ff000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libstdc++.so.6
ffff8804a5c2f468 7ff0bc0ff000 7ff0bc106000 8100071
/usr/local/greenplum-db-4.3.3.1/lib/libstdc++.so.6
ffff8804a5c2f3a0 7ff0bc106000 7ff0bc108000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libstdc++.so.6
ffff8804a5c2f2d8 7ff0bc108000 7ff0bc11d000 100073
ffff8804a5c2f210 7ff0bc11d000 7ff0bc12d000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/liblber-2.3.so.0
ffff8804a5c2f148 7ff0bc12d000 7ff0bc32c000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/liblber-2.3.so.0
ffff8804a5c2f080 7ff0bc32c000 7ff0bc32d000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/liblber-2.3.so.0
ffff8804a46dded0 7ff0bc32d000 7ff0bc32f000 100073
ffff8804a46dde08 7ff0bc32f000 7ff0bc336000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libkrb5support.so.0
ffff8804a46ddd40 7ff0bc336000 7ff0bc535000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libkrb5support.so.0
ffff8804a46ddc78 7ff0bc535000 7ff0bc536000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libkrb5support.so.0
ffff8804a46ddbb0 7ff0bc536000 7ff0bc559000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libk5crypto.so.3
ffff8804a46ddae8 7ff0bc559000 7ff0bc758000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libk5crypto.so.3
ffff8804a46dda20 7ff0bc758000 7ff0bc75a000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libk5crypto.so.3
ffff8804a46dd958 7ff0bc75a000 7ff0bc75b000 100073
ffff8804a46dd890 7ff0bc75b000 7ff0bc75f000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libcom_err.so.3
ffff8804a46dd7c8 7ff0bc75f000 7ff0bc95f000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libcom_err.so.3
ffff8804a46dd700 7ff0bc95f000 7ff0bc960000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libcom_err.so.3
ffff8804a46dd638 7ff0bc960000 7ff0bc961000 100073
ffff8804a46dd570 7ff0bc961000 7ff0bc9e8000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libkrb5.so.3
ffff8804a46dd4a8 7ff0bc9e8000 7ff0bcbe7000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libkrb5.so.3
ffff8804a46dd3e0 7ff0bcbe7000 7ff0bcbeb000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libkrb5.so.3
ffff8804a46dd318 7ff0bcbeb000 7ff0bcbec000 100073
ffff8804a46dd250 7ff0bcbec000 7ff0bccaa000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libdxltranslators.so
ffff8804a46dd188 7ff0bccaa000 7ff0bceaa000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libdxltranslators.so
ffff8804a46dd0c0 7ff0bceaa000 7ff0bceb0000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libdxltranslators.so
ffff880810280e90 7ff0bceb0000 7ff0bceb1000 100073
ffff880810280dc8 7ff0bceb1000 7ff0bcf06000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libgpos.so
ffff880810280d00 7ff0bcf06000 7ff0bd106000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libgpos.so
ffff880810280c38 7ff0bd106000 7ff0bd10a000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libgpos.so
ffff880810280b70 7ff0bd10a000 7ff0bd46c000 100073
ffff880810280aa8 7ff0bd46c000 7ff0bd482000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libbsafe_wrapper.so.1
ffff8808102809e0 7ff0bd482000 7ff0bd682000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libbsafe_wrapper.so.1
ffff880810280918 7ff0bd682000 7ff0bd685000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libbsafe_wrapper.so.1
ffff880810280850 7ff0bd685000 7ff0bd6c7000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libldap-2.3.so.0
ffff880810280788 7ff0bd6c7000 7ff0bd8c7000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libldap-2.3.so.0
ffff8808102806c0 7ff0bd8c7000 7ff0bd8c9000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libldap-2.3.so.0
ffff8808102805f8 7ff0bd8c9000 7ff0bd8ca000 100073
ffff880810280530 7ff0bd8ca000 7ff0bd91d000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libcurl.so.4
ffff880810280468 7ff0bd91d000 7ff0bdb1c000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libcurl.so.4
ffff8808102803a0 7ff0bdb1c000 7ff0bdb1f000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libcurl.so.4
ffff8808102802d8 7ff0bdb1f000 7ff0bdcce000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libxml2.so.2
ffff880810280210 7ff0bdcce000 7ff0bdecd000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libxml2.so.2
ffff880810280148 7ff0bdecd000 7ff0bded7000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libxml2.so.2
ffff880810280080 7ff0bded7000 7ff0bded8000 100073
ffff880862b72ed0 7ff0bded8000 7ff0bdeee000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libz.so.1
ffff880862b72e08 7ff0bdeee000 7ff0be0ed000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libz.so.1
ffff880862b72d40 7ff0be0ed000 7ff0be0ee000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libz.so.1
ffff880862b72c78 7ff0be0ee000 7ff0be0ef000 100073
ffff880862b72bb0 7ff0be0ef000 7ff0be118000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libgssapi_krb5.so.2
ffff880862b72ae8 7ff0be118000 7ff0be317000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libgssapi_krb5.so.2
ffff880862b72a20 7ff0be317000 7ff0be319000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libgssapi_krb5.so.2
ffff880862b72958 7ff0be319000 7ff0be483000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libcrypto.so.0.9.8
ffff880862b72890 7ff0be483000 7ff0be683000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libcrypto.so.0.9.8
ffff880862b727c8 7ff0be683000 7ff0be6a9000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libcrypto.so.0.9.8
ffff880862b72700 7ff0be6a9000 7ff0be6ad000 100073
ffff880862b72638 7ff0be6ad000 7ff0be6fb000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libssl.so.0.9.8
ffff880862b72570 7ff0be6fb000 7ff0be8fb000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libssl.so.0.9.8
ffff880862b724a8 7ff0be8fb000 7ff0be902000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libssl.so.0.9.8
ffff880862b723e0 7ff0be902000 7ff0be903000 100073
ffff88011028e2d8 7ff0be903000 7ff0be924000 100073
ffff880862b72318 7ff0be925000 7ff0be9d0000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libnetsnmp.so.20
ffff880862b72250 7ff0be9d0000 7ff0bebd0000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libnetsnmp.so.20
ffff880862b72188 7ff0bebd0000 7ff0bebd4000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libnetsnmp.so.20
ffff880862b720c0 7ff0bebd4000 7ff0bec09000 100073
ffff8804a5c32e90 7ff0bec09000 7ff0bec1f000 8000075
/usr/local/greenplum-db-4.3.3.1/lib/libesmtp.so.5
ffff8804a5c32dc8 7ff0bec1f000 7ff0bee1f000 8000070
/usr/local/greenplum-db-4.3.3.1/lib/libesmtp.so.5
ffff8804a5c32d00 7ff0bee1f000 7ff0bee20000 8100073
/usr/local/greenplum-db-4.3.3.1/lib/libesmtp.so.5
ffff8804a5c32c38 7ff0bee20000 7ff0bee21000 100073
ffff8804a5c32b70 7fff8b990000 7fff8b9bf000 100177
ffff8804a5c32aa8 7fff8b9bf000 7fff8b9c1000 100173
ffff8804a5c329e0 7fff8b9d7000 7fff8b9d8000 40075
crash>
Of course it has its limitations. Since the page tables are being broken down in this
case,
"vm -p" fails:
crash> vm -M ffff880495120dc0 -p
PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres"
MM PGD RSS TOTAL_VM
0 0 0k 0k
VMA START END FLAGS FILE
ffff8804a085ce90 400000 f56000 8001875
/usr/local/greenplum-db-4.3.3.1/bin/postgres
VIRTUAL PHYSICAL
vm: invalid kernel virtual address: 50 type: "mm_struct pgd"
crash>
But it does seems like a worthwhile addition.
The patch doesn't check whether mm->owner or mm->mm_count are legitimate, but
I'm not
sure whether it's even worth it? If it fails, it fails, and the help page should
just
indicate that the command option is not guaranteed to work. Does the attached patch work
for you?
Thanks,
Dave
7:11 a.m.
On 12/06/2014 04:11 AM, Dave Anderson wrote:
Interestingly enough, today I was asked to look at a vmcore in which
an oops
occurred during task exit after tsk->mm had been NULL'd out in exit_mm():
It almost matches what I am facing, when tsk->mm is set to NULL and memory
mapping is supposed to be displayed. This is a more simple implementation.
I have tried to command like vm [taskp | pid | [-M mm_struct]]. But it have
to modify a lot of thing.
By the way, I feel the code is becoming more and more complicated, maybe a
reconstruction is needed.
Of course it has its limitations. Since the page tables are being broken down in this
case,
"vm -p" fails:
crash> vm -M ffff880495120dc0 -p
PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres"
MM PGD RSS TOTAL_VM
0 0 0k 0k
VMA START END FLAGS FILE
ffff8804a085ce90 400000 f56000 8001875
/usr/local/greenplum-db-4.3.3.1/bin/postgres
VIRTUAL PHYSICAL
vm: invalid kernel virtual address: 50 type: "mm_struct pgd"
crash>
After a glance, the pgd comes from the mm of task_struct. We need a lot of work to make
it
replaced by argument of -M, I don't think it worse it right now.
But it does seems like a worthwhile addition.
The patch doesn't check whether mm->owner or mm->mm_count are legitimate, but
I'm not
sure whether it's even worth it? If it fails, it fails, and the help page should
just
indicate that the command option is not guaranteed to work. Does the attached patch
work
for you?
Similar to the core I got. And I modified the patch to add some check. At least I think
we need to make sure the address still belongs to a mm_struct object.
--
Regards
Qiao Nuohan
8:28 a.m.
----- Original Message -----
On 12/06/2014 04:11 AM, Dave Anderson wrote:
> Interestingly enough, today I was asked to look at a vmcore in which an oops
> occurred during task exit after tsk->mm had been NULL'd out in exit_mm():
It almost matches what I am facing, when tsk->mm is set to NULL and memory
mapping is supposed to be displayed. This is a more simple implementation.
I have tried to command like vm [taskp | pid | [-M mm_struct]]. But it have
to modify a lot of thing.
By the way, I feel the code is becoming more and more complicated, maybe a
reconstruction is needed.
Well, the vm_area_dump() function is relatively stable, so let's not go crazy
here for what's essentially an "experimental" option.
>
> Of course it has its limitations. Since the page tables are being broken down in
this case,
> "vm -p" fails:
>
> crash> vm -M ffff880495120dc0 -p
> PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres"
> MM PGD RSS TOTAL_VM
> 0 0 0k 0k
> VMA START END FLAGS FILE
> ffff8804a085ce90 400000 f56000 8001875
> /usr/local/greenplum-db-4.3.3.1/bin/postgres
> VIRTUAL PHYSICAL
> vm: invalid kernel virtual address: 50 type: "mm_struct pgd"
> crash>
After a glance, the pgd comes from the mm of task_struct. We need a lot of work to make
it
replaced by argument of -M, I don't think it worse it right now.
Actually it doesn't take much work at all. If both tc->mm_struct and
tm->mm_struct_addr
are replaced with the supplied address:
tc->mm_struct = tm->mm_struct_addr = pc->curcmd_private;
then "vm -M ffff880495120dc0 -p" also works OK with my sample vmcore.
>
> But it does seems like a worthwhile addition.
>
> The patch doesn't check whether mm->owner or mm->mm_count are legitimate,
but I'm not
> sure whether it's even worth it? If it fails, it fails, and the help page
should just
> indicate that the command option is not guaranteed to work. Does the attached patch
work
> for you?
Similar to the core I got. And I modified the patch to add some check. At least I think
we need to make sure the address still belongs to a mm_struct object.
I suppose you could, although in all probability it's going to be stay in the
mm_struct
slab cache, and worst case, have been re-used by another task.
Dave
2:58 a.m.
On 12/08/2014 10:28 PM, Dave Anderson wrote:
----- Original Message -----
> On 12/06/2014 04:11 AM, Dave Anderson wrote:
>> Interestingly enough, today I was asked to look at a vmcore in which an oops
>> occurred during task exit after tsk->mm had been NULL'd out in exit_mm():
>
> It almost matches what I am facing, when tsk->mm is set to NULL and memory
> mapping is supposed to be displayed. This is a more simple implementation.
> I have tried to command like vm [taskp | pid | [-M mm_struct]]. But it have
> to modify a lot of thing.
>
> By the way, I feel the code is becoming more and more complicated, maybe a
> reconstruction is needed.
Well, the vm_area_dump() function is relatively stable, so let's not go crazy
here for what's essentially an "experimental" option.
I see.
>
>>
>> Of course it has its limitations. Since the page tables are being broken down in
this case,
>> "vm -p" fails:
>>
>> crash> vm -M ffff880495120dc0 -p
>> PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres"
>> MM PGD RSS TOTAL_VM
>> 0 0 0k 0k
>> VMA START END FLAGS FILE
>> ffff8804a085ce90 400000 f56000 8001875
>> /usr/local/greenplum-db-4.3.3.1/bin/postgres
>> VIRTUAL PHYSICAL
>> vm: invalid kernel virtual address: 50 type: "mm_struct pgd"
>> crash>
>
> After a glance, the pgd comes from the mm of task_struct. We need a lot of work to
make it
> replaced by argument of -M, I don't think it worse it right now.
Actually it doesn't take much work at all. If both tc->mm_struct and
tm->mm_struct_addr
are replaced with the supplied address:
tc->mm_struct = tm->mm_struct_addr = pc->curcmd_private;
then "vm -M ffff880495120dc0 -p" also works OK with my sample vmcore.
Yes, vm -p will tc->mm_struct to get pgd. But I was afraid permanently changing
tc->mm_struct
is not good.
Take my core into consideration, the case is:
<cut>
crash> help -t | grep mm_struct
.mm_struct: 0
mm_struct: 354cc80
crash> vm -M 0xffff88003ae98ac0
PID: 4860 TASK: ffff88003ae7eaf0 CPU: 0 COMMAND: "bash"
MM PGD RSS TOTAL_VM
0 0 0k 0k
VMA START END FLAGS FILE
ffff88003acc3ad8 400000 4d4000 8001875 /bin/bash
...
crash> help -t | grep mm_struct
.mm_struct: ffff88003ae98ac0
mm_struct: 354cc80
crash>
<cut>
Is it OK to have ".mm_struct" changed here?
>>
>> But it does seems like a worthwhile addition.
>>
>> The patch doesn't check whether mm->owner or mm->mm_count are
legitimate, but I'm not
>> sure whether it's even worth it? If it fails, it fails, and the help page
should just
>> indicate that the command option is not guaranteed to work. Does the attached
patch work
>> for you?
>
> Similar to the core I got. And I modified the patch to add some check. At least I
think
> we need to make sure the address still belongs to a mm_struct object.
I suppose you could, although in all probability it's going to be stay in the
mm_struct
slab cache, and worst case, have been re-used by another task.
So you like the modification.
Dave
.
--
Regards
Qiao Nuohan
8:30 a.m.
----- Original Message -----
On 12/08/2014 10:28 PM, Dave Anderson wrote:
>
>
> ----- Original Message -----
>> On 12/06/2014 04:11 AM, Dave Anderson wrote:
>>> Interestingly enough, today I was asked to look at a vmcore in which an
>>> oops
>>> occurred during task exit after tsk->mm had been NULL'd out in
exit_mm():
>>
>> It almost matches what I am facing, when tsk->mm is set to NULL and memory
>> mapping is supposed to be displayed. This is a more simple implementation.
>> I have tried to command like vm [taskp | pid | [-M mm_struct]]. But it have
>> to modify a lot of thing.
>>
>> By the way, I feel the code is becoming more and more complicated, maybe a
>> reconstruction is needed.
>
> Well, the vm_area_dump() function is relatively stable, so let's not go crazy
> here for what's essentially an "experimental" option.
I see.
>
>>
>>>
>>> Of course it has its limitations. Since the page tables are being broken
down in this case,
>>> "vm -p" fails:
>>>
>>> crash> vm -M ffff880495120dc0 -p
>>> PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND:
"postgres"
>>> MM PGD RSS TOTAL_VM
>>> 0 0 0k 0k
>>> VMA START END FLAGS FILE
>>> ffff8804a085ce90 400000 f56000 8001875
/usr/local/greenplum-db-4.3.3.1/bin/postgres
>>> VIRTUAL PHYSICAL
>>> vm: invalid kernel virtual address: 50 type: "mm_struct pgd"
>>> crash>
>>
>> After a glance, the pgd comes from the mm of task_struct. We need a lot of work
to make it
>> replaced by argument of -M, I don't think it worse it right now.
>
> Actually it doesn't take much work at all. If both tc->mm_struct and
tm->mm_struct_addr
> are replaced with the supplied address:
>
> tc->mm_struct = tm->mm_struct_addr = pc->curcmd_private;
>
> then "vm -M ffff880495120dc0 -p" also works OK with my sample vmcore.
Yes, vm -p will tc->mm_struct to get pgd. But I was afraid permanently changing
tc->mm_struct
is not good.
Take my core into consideration, the case is:
<cut>
crash> help -t | grep mm_struct
.mm_struct: 0
mm_struct: 354cc80
crash> vm -M 0xffff88003ae98ac0
PID: 4860 TASK: ffff88003ae7eaf0 CPU: 0 COMMAND: "bash"
MM PGD RSS TOTAL_VM
0 0 0k 0k
VMA START END FLAGS FILE
ffff88003acc3ad8 400000 4d4000 8001875 /bin/bash
...
crash> help -t | grep mm_struct
.mm_struct: ffff88003ae98ac0
mm_struct: 354cc80
crash>
<cut>
Is it OK to have ".mm_struct" changed here?
Probably not...
I did take a quick look at the other usages of tc->mm_struct, and I think
the only major difference is that the "search -u" command would search
the user-space memory of the changed task. But there's also an oddball
check for an i386 hypervisor callback from user-space, and it looks like
get_task_mem_usage() would use it from that point on, so I agree with
you that it should be restored to NULL.
Since there's a strong possibility of an error(FATAL, ...) call while
executing the command, it's not safe to simply restore it to NULL at the
end of the command, but rather the pc->cmd_cleanup() facility should
be used with the task address as the pc->cmd_cleanup_arg.
>
>>>
>>> But it does seems like a worthwhile addition.
>>>
>>> The patch doesn't check whether mm->owner or mm->mm_count are
legitimate, but I'm not
>>> sure whether it's even worth it? If it fails, it fails, and the help
page should just
>>> indicate that the command option is not guaranteed to work. Does the
attached patch work
>>> for you?
>>
>> Similar to the core I got. And I modified the patch to add some check. At least
I think
>> we need to make sure the address still belongs to a mm_struct object.
>
> I suppose you could, although in all probability it's going to be stay in the
mm_struct
> slab cache, and worst case, have been re-used by another task.
So you like the modification.
Sure -- with the pc->cmd_cleanup in place, I don't see how it can hurt.
Dave
3:53 a.m.
Hello Dave,
The patch has been modified, using pc->cmd_cleanup to clean tc->mm_struct.
Please check the attached patch.
On 12/09/2014 10:30 PM, Dave Anderson wrote:
----- Original Message -----
> On 12/08/2014 10:28 PM, Dave Anderson wrote:
>>
>>
>> ----- Original Message -----
>>> On 12/06/2014 04:11 AM, Dave Anderson wrote:
>>>> Interestingly enough, today I was asked to look at a vmcore in which an
>>>> oops
>>>> occurred during task exit after tsk->mm had been NULL'd out in
exit_mm():
>>>
>>> It almost matches what I am facing, when tsk->mm is set to NULL and
memory
>>> mapping is supposed to be displayed. This is a more simple implementation.
>>> I have tried to command like vm [taskp | pid | [-M mm_struct]]. But it have
>>> to modify a lot of thing.
>>>
>>> By the way, I feel the code is becoming more and more complicated, maybe a
>>> reconstruction is needed.
>>
>> Well, the vm_area_dump() function is relatively stable, so let's not go
crazy
>> here for what's essentially an "experimental" option.
>
> I see.
>
>>
>>>
>>>>
>>>> Of course it has its limitations. Since the page tables are being broken
down in this case,
>>>> "vm -p" fails:
>>>>
>>>> crash> vm -M ffff880495120dc0 -p
>>>> PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND:
"postgres"
>>>> MM PGD RSS TOTAL_VM
>>>> 0 0 0k 0k
>>>> VMA START END FLAGS FILE
>>>> ffff8804a085ce90 400000 f56000 8001875
/usr/local/greenplum-db-4.3.3.1/bin/postgres
>>>> VIRTUAL PHYSICAL
>>>> vm: invalid kernel virtual address: 50 type: "mm_struct
pgd"
>>>> crash>
>>>
>>> After a glance, the pgd comes from the mm of task_struct. We need a lot of
work to make it
>>> replaced by argument of -M, I don't think it worse it right now.
>>
>> Actually it doesn't take much work at all. If both tc->mm_struct and
tm->mm_struct_addr
>> are replaced with the supplied address:
>>
>> tc->mm_struct = tm->mm_struct_addr = pc->curcmd_private;
>>
>> then "vm -M ffff880495120dc0 -p" also works OK with my sample vmcore.
>
> Yes, vm -p will tc->mm_struct to get pgd. But I was afraid permanently changing
tc->mm_struct
> is not good.
>
> Take my core into consideration, the case is:
>
> <cut>
> crash> help -t | grep mm_struct
> .mm_struct: 0
> mm_struct: 354cc80
> crash> vm -M 0xffff88003ae98ac0
> PID: 4860 TASK: ffff88003ae7eaf0 CPU: 0 COMMAND: "bash"
> MM PGD RSS TOTAL_VM
> 0 0 0k 0k
> VMA START END FLAGS FILE
> ffff88003acc3ad8 400000 4d4000 8001875 /bin/bash
> ...
> crash> help -t | grep mm_struct
> .mm_struct: ffff88003ae98ac0
> mm_struct: 354cc80
> crash>
> <cut>
>
> Is it OK to have ".mm_struct" changed here?
Probably not...
I did take a quick look at the other usages of tc->mm_struct, and I think
the only major difference is that the "search -u" command would search
the user-space memory of the changed task. But there's also an oddball
check for an i386 hypervisor callback from user-space, and it looks like
get_task_mem_usage() would use it from that point on, so I agree with
you that it should be restored to NULL.
Since there's a strong possibility of an error(FATAL, ...) call while
executing the command, it's not safe to simply restore it to NULL at the
end of the command, but rather the pc->cmd_cleanup() facility should
be used with the task address as the pc->cmd_cleanup_arg.
>>
>>>>
>>>> But it does seems like a worthwhile addition.
>>>>
>>>> The patch doesn't check whether mm->owner or mm->mm_count are
legitimate, but I'm not
>>>> sure whether it's even worth it? If it fails, it fails, and the help
page should just
>>>> indicate that the command option is not guaranteed to work. Does the
attached patch work
>>>> for you?
>>>
>>> Similar to the core I got. And I modified the patch to add some check. At
least I think
>>> we need to make sure the address still belongs to a mm_struct object.
>>
>> I suppose you could, although in all probability it's going to be stay in the
mm_struct
>> slab cache, and worst case, have been re-used by another task.
>
> So you like the modification.
Sure -- with the pc->cmd_cleanup in place, I don't see how it can hurt.
Dave
.
--
Regards
Qiao Nuohan
10:34 a.m.
----- Original Message -----
Hello Dave,
The patch has been modified, using pc->cmd_cleanup to clean tc->mm_struct.
Please check the attached patch.
Hello Qiao,
One major point, and a few minor ones...
First, this won't work with the sample vmcore I showed you:
+static int
+is_valid_mm(ulong mm)
+{
+ char kbuf[BUFSIZE];
+ char *p;
+ int mm_users;
+
+ if (!(p = vaddr_to_kmem_cache(mm, kbuf, VERBOSE)))
+ goto bailout;
+
+ if (!STRNEQ(p, "mm_struct"))
+ goto bailout;
+
+ readmem(mm + OFFSET(mm_struct_mm_users), KVADDR, &mm_users, sizeof(int),
+ "mm_struct mm_users", FAULT_ON_ERROR);
+
+ return mm_users;
+
+bailout:
+ error(FATAL, "invalid mm_struct address\n");
+ return 0;
+}
If you recall my sample vmcore, the mm->users was zero, but because
the mm->count was non-zero, mmdrop() had not called __mmdrop() to
free the mm_struct:
crash> mm_struct.mm_count,owner,mm_users ffff880495120dc0
mm_count = {
counter = 2
}
owner = 0xffff88049863f500
mm_users = {
counter = 0
}
crash>
So it should it test mm->count instead, correct?
And maybe the error message above should indicate "invalid or stale", given
that the user-supplied address may have been valid earlier?
Secondly, wouldn't it make more sense to just pass pc->curcmd_private to
is_valid_mm() below, and if it fails, return NULL? If it's an invalid
argument, it doesn't make much sense to make the switch and set-up the
call to pc->cmd_cleanup():
- if (!tm->mm_struct_addr)
- return (ulong)NULL;
+ if (!tm->mm_struct_addr) {
+ if (pc->curcmd_flags & MM_STRUCT_FORCE) {
+ tc->mm_struct = tm->mm_struct_addr =
pc->curcmd_private;
+
+ /*
+ * tc->mm_struct may be changed, use vm_cleanup to
+ * restore it.
+ */
+ pc->cmd_cleanup_arg = (void *)task;
+ pc->cmd_cleanup = vm_cleanup;
+
+ if (!is_valid_mm(tm->mm_struct_addr))
+ return (ulong)NULL;
+ } else
+ return (ulong)NULL;
+ }
And lastly, this works, but is somewhat of an overkill:
+static void
+vm_cleanup(void *arg)
+{
+ ulong task;
+ struct task_context *tc;
+ ulong mm_struct;
+
+ pc->cmd_cleanup = NULL;
+ pc->cmd_cleanup_arg = NULL;
+
+ task = (ulong)arg;
+
+ fill_task_struct(task);
+ mm_struct = ULONG(tt->task_struct + OFFSET(task_struct_mm));
+
+ tc = task_to_context(task);
+ tc->mm_struct = mm_struct;
+}
The only way the tc->mm_struct swap is made is if tc->mm_struct was
originally set to NULL. So it could simply be:
tc = task_to_context(task);
tc->mm_struct = NULL;
Right?
Thanks,
Dave
8:18 p.m.
On 12/11/2014 12:34 AM, Dave Anderson wrote:
----- Original Message -----
> Hello Dave,
>
> The patch has been modified, using pc->cmd_cleanup to clean tc->mm_struct.
> Please check the attached patch.
Hello Qiao,
One major point, and a few minor ones...
First, this won't work with the sample vmcore I showed you:
+static int
+is_valid_mm(ulong mm)
+{
+ char kbuf[BUFSIZE];
+ char *p;
+ int mm_users;
+
+ if (!(p = vaddr_to_kmem_cache(mm, kbuf, VERBOSE)))
+ goto bailout;
+
+ if (!STRNEQ(p, "mm_struct"))
+ goto bailout;
+
+ readmem(mm + OFFSET(mm_struct_mm_users), KVADDR,&mm_users, sizeof(int),
+ "mm_struct mm_users", FAULT_ON_ERROR);
+
+ return mm_users;
+
+bailout:
+ error(FATAL, "invalid mm_struct address\n");
+ return 0;
+}
If you recall my sample vmcore, the mm->users was zero, but because
the mm->count was non-zero, mmdrop() had not called __mmdrop() to
free the mm_struct:
crash> mm_struct.mm_count,owner,mm_users ffff880495120dc0
mm_count = {
counter = 2
}
owner = 0xffff88049863f500
mm_users = {
counter = 0
}
crash>
So it should it test mm->count instead, correct?
Fixed.
And maybe the error message above should indicate "invalid or stale", given
that the user-supplied address may have been valid earlier?
Add an error message("stale mm_struct address") when mm_count equals to 0.
Secondly, wouldn't it make more sense to just pass pc->curcmd_private to
is_valid_mm() below, and if it fails, return NULL? If it's an invalid
argument, it doesn't make much sense to make the switch and set-up the
call to pc->cmd_cleanup():
Yes. Valid check should go first.
- if (!tm->mm_struct_addr)
- return (ulong)NULL;
+ if (!tm->mm_struct_addr) {
+ if (pc->curcmd_flags& MM_STRUCT_FORCE) {
+ tc->mm_struct = tm->mm_struct_addr =
pc->curcmd_private;
+
+ /*
+ * tc->mm_struct may be changed, use vm_cleanup to
+ * restore it.
+ */
+ pc->cmd_cleanup_arg = (void *)task;
+ pc->cmd_cleanup = vm_cleanup;
+
+ if (!is_valid_mm(tm->mm_struct_addr))
+ return (ulong)NULL;
+ } else
+ return (ulong)NULL;
+ }
And lastly, this works, but is somewhat of an overkill:
+static void
+vm_cleanup(void *arg)
+{
+ ulong task;
+ struct task_context *tc;
+ ulong mm_struct;
+
+ pc->cmd_cleanup = NULL;
+ pc->cmd_cleanup_arg = NULL;
+
+ task = (ulong)arg;
+
+ fill_task_struct(task);
+ mm_struct = ULONG(tt->task_struct + OFFSET(task_struct_mm));
+
+ tc = task_to_context(task);
+ tc->mm_struct = mm_struct;
+}
The only way the tc->mm_struct swap is made is if tc->mm_struct was
originally set to NULL. So it could simply be:
tc = task_to_context(task);
tc->mm_struct = NULL;
Right?
I failed to realize tm->mm_struct_addr comes from tc->mm_struct firstly, and
if tm->mm_struct_addr == 0, then tc->mm_struct must equal to 0. Thanks for
pointing it out.
I made a more change, passing tc to vm_cleanup, then task_to_context() will also
be omitted.
Thanks,
Dave
.
--
Regards
Qiao Nuohan
4:09 p.m.
----- Original Message -----
On 12/11/2014 12:34 AM, Dave Anderson wrote:
... [ cut ] ...
I made a more change, passing tc to vm_cleanup, then task_to_context() will also
be omitted.
Thanks Qiao, it looks good. I updated the help page and dump_offset_table().
Queued for crash-7.1.0:
https://github.com/crash-utility/crash/commit/361bdc2fc43a18cc3caf9c6927f...
Dave
3567
days inactive
3576
days old
devel@lists.crash-utility.osci.io
12 comments
2 participants
participants (2)
-
Dave Anderson -
qiaonuohan