Hi Dave,     Recently I was trying crash tool with kdump dumpfile & structure layout randomized kernel[*](), and it fails without any surprise. After looking into the different errors crash reports, I can confirm it is a result from randomized structure layout. So my questions is, do you ever consider supporting this feature[*] in crash?   If yes, do you have any plan & technique evaluation about it?   If no, what's the reason? [*]https://lwn.net/Articles/722293/ -- Sincerely, Cao jin

... The 1st error is from reading "init_uts_ns"(randomized) value in kernel_init, so kt->kernel_version is wrong, then results invalid virtual addresses in "case POST_GDB:" of x86_64_init, the symptom of this error is segment fault results from infinite function call:

On 01/23/2018 11:19 PM, Dave Anderson wrote: > > > ----- Original Message ----- >> Hi Dave, >> >>     Recently I was trying crash tool with kdump dumpfile & structure >> layout randomized kernel[*](), and it fails without any surprise. After >> looking into the different errors crash reports, I can confirm it is a >> result from randomized structure layout. >> >> So my questions is, do you ever consider supporting this feature[*] in >> crash? >>   If yes, do you have any plan & technique evaluation about it? >>   If no, what's the reason? >> >> [*]https://lwn.net/Articles/722293/ >> -- >> Sincerely, >> Cao jin > > I was under the impression that the structure layout was done at > compile-time, and that the vmlinux file's debuginfo data would > represent the randomized layout. And that being the case, the > inconvenience would be that the crash session would show the > randomized layout, while the associated source code would show > the original layout. > BTW, I don't have any compiler knowledge before, just from these two days learning, I feel you are right at "vmlinux file's debuginfo data would represent the randomized layout". But when I debug, it seem not like what it should be. I have two file pairs, randomized and non-randomized one. I print some member offset of structure tagged with __randomize_layout after MEMBER_OFFSET_INIT, like this one: (gdb) p offset_table->task_struct_state $1 = 8 (gdb) p offset_table->task_struct_exit_state $2 = 2164 (gdb) p offset_table->task_struct_pid $3 = 2264 (gdb) p offset_table->task_struct_comm $4 = 2744 (gdb) p offset_table->task_struct_next_task $5 = -1 (gdb) p offset_table->task_struct_processor $6 = -1 (gdb) p offset_table->task_struct_p_pptr $7 = -1 (gdb) p offset_table->task_struct_parent $8 = 2288 Under both file pairs, these offset value are the same, so, I think that is why I have the impression that debuginfo has the original structure layout. I guess this is one kind of "MEMBER_OFFSET() no longer work"?

On 01/24/2018 11:23 PM, Dave Anderson wrote: > > > ----- Original Message ----- >> >> >> On 01/23/2018 11:19 PM, Dave Anderson wrote: >>> >>> >>> ----- Original Message ----- >>>> Hi Dave, >>>> >>>>     Recently I was trying crash tool with kdump dumpfile & structure >>>> layout randomized kernel[*](), and it fails without any surprise. After >>>> looking into the different errors crash reports, I can confirm it is a >>>> result from randomized structure layout. >>>> >>>> So my questions is, do you ever consider supporting this feature[*] in >>>> crash? >>>>   If yes, do you have any plan & technique evaluation about it? >>>>   If no, what's the reason? >>>> >>>> [*]https://lwn.net/Articles/722293/ >>>> -- >>>> Sincerely, >>>> Cao jin >>> >>> I was under the impression that the structure layout was done at >>> compile-time, and that the vmlinux file's debuginfo data would >>> represent the randomized layout. And that being the case, the >>> inconvenience would be that the crash session would show the >>> randomized layout, while the associated source code would show >>> the original layout. >>> >> >> BTW, I don't have any compiler knowledge before, just from these two >> days learning, I feel you are right at "vmlinux file's debuginfo data >> would represent the randomized layout". >> >> But when I debug, it seem not like what it should be. I have two file >> pairs, randomized and non-randomized one. I print some member offset of >> structure tagged with __randomize_layout after MEMBER_OFFSET_INIT, like >> this one: >> >> (gdb) p offset_table->task_struct_state >> $1 = 8 >> (gdb) p offset_table->task_struct_exit_state >> $2 = 2164 >> (gdb) p offset_table->task_struct_pid >> $3 = 2264 >> (gdb) p offset_table->task_struct_comm >> $4 = 2744 >> (gdb) p offset_table->task_struct_next_task >> $5 = -1 >> (gdb) p offset_table->task_struct_processor >> $6 = -1 >> (gdb) p offset_table->task_struct_p_pptr >> $7 = -1 >> (gdb) p offset_table->task_struct_parent >> $8 = 2288 >> >> Under both file pairs, these offset value are the same, so, I think that >> is why I have the impression that debuginfo has the original structure >> layout. I guess this is one kind of "MEMBER_OFFSET() no longer work"? > > > It appears that way. A better manner would be to simply run > gdb on the 2 vmlinux files, i.e., run "gdb vmlinux", and then > print out one of the required data structures that has the > "__randomize_layout" compiler annotation, such as the mm_struct: > > struct mm_struct { > struct vm_area_struct *mmap; /* list of VMAs */ > struct rb_root mm_rb; > u32 vmacache_seqnum; /* per-thread vmacache */ > > ... [ cut ] ... > > } __randomize_layout; > > In each gdb session, run the command "ptype struct mm_struct", > and compare the two outputs. If they are the identical, then there > is a major problem. > I compared the output of "ptype struct uts_namespace"(and also mm_struct), it seems there does is a major problem, the two outputs are the same:( > A fundamental underpinning of the crash utility is the ability > to gather correct structure member offsets from the embedded gdb > module, which gets them from the debuginfo data of the vmlinux file. > If the structure layout is identical in both the randomized > and non-randomized vmlinux files, then the crash utility will > be pretty much useless. > Seems horrible thing happens. > I agree with Linus Torvalds with respect to this feature, where in > the article that you referenced, he says: > > Nevertheless, Torvalds is unimpressed by structure randomization, > calling it "security theater". The fact that distributions need to > publish the randomization seeds for module-building meant it did > not provide as big of a security feature as advertised. Torvalds > however did add: "So it's imnsho a pretty questionable security > thing. It's likely most useful for one-off 'special secure > installations' than mass productions. > > I have long believed that the constant onslaught of new "security" > features pushed into the kernel by Kees Cook will eventually kill > the capability of using the crash utility to debug the kernel. So > far we've been able to work around or adapt to previous security > related changes, but this particular feature looks to be a crash > killer. > Yes, also feels that it is not easy to adapt or work around since the randomized structure layout info is not reflect in the vmlinux's debuginfo. I don't know gcc internals, so a blind guess is that it is done in this way on purpose for security, so attackers won't bother to seek the random seed, but just using gdb to find out the structure layout.

On Thu, 25 Jan 2018 16:37:01 +0800 Cao jin <caoj.fnst(a)cn.fujitsu.com&gt; wrote: > On 01/24/2018 11:23 PM, Dave Anderson wrote: >> >> >> ----- Original Message ----- >>> >>> >>> On 01/23/2018 11:19 PM, Dave Anderson wrote: >>>> >>>> >>>> ----- Original Message ----- >>>>> Hi Dave, >>>>> >>>>>     Recently I was trying crash tool with kdump dumpfile & structure >>>>> layout randomized kernel[*](), and it fails without any surprise. After >>>>> looking into the different errors crash reports, I can confirm it is a >>>>> result from randomized structure layout. >>>>> >>>>> So my questions is, do you ever consider supporting this feature[*] in >>>>> crash? >>>>>   If yes, do you have any plan & technique evaluation about it? >>>>>   If no, what's the reason? >>>>> >>>>> [*]https://lwn.net/Articles/722293/ >>>>> -- >>>>> Sincerely, >>>>> Cao jin >>>> >>>> I was under the impression that the structure layout was done at >>>> compile-time, and that the vmlinux file's debuginfo data would >>>> represent the randomized layout. And that being the case, the >>>> inconvenience would be that the crash session would show the >>>> randomized layout, while the associated source code would show >>>> the original layout. >>>> >>> >>> BTW, I don't have any compiler knowledge before, just from these two >>> days learning, I feel you are right at "vmlinux file's debuginfo data >>> would represent the randomized layout". >>> >>> But when I debug, it seem not like what it should be. I have two file >>> pairs, randomized and non-randomized one. I print some member offset of >>> structure tagged with __randomize_layout after MEMBER_OFFSET_INIT, like >>> this one: >>> >>> (gdb) p offset_table->task_struct_state >>> $1 = 8 >>> (gdb) p offset_table->task_struct_exit_state >>> $2 = 2164 >>> (gdb) p offset_table->task_struct_pid >>> $3 = 2264 >>> (gdb) p offset_table->task_struct_comm >>> $4 = 2744 >>> (gdb) p offset_table->task_struct_next_task >>> $5 = -1 >>> (gdb) p offset_table->task_struct_processor >>> $6 = -1 >>> (gdb) p offset_table->task_struct_p_pptr >>> $7 = -1 >>> (gdb) p offset_table->task_struct_parent >>> $8 = 2288 >>> >>> Under both file pairs, these offset value are the same, so, I think that >>> is why I have the impression that debuginfo has the original structure >>> layout. I guess this is one kind of "MEMBER_OFFSET() no longer work"? >> >> >> It appears that way. A better manner would be to simply run >> gdb on the 2 vmlinux files, i.e., run "gdb vmlinux", and then >> print out one of the required data structures that has the >> "__randomize_layout" compiler annotation, such as the mm_struct: >> >> struct mm_struct { >> struct vm_area_struct *mmap; /* list of VMAs */ >> struct rb_root mm_rb; >> u32 vmacache_seqnum; /* per-thread vmacache */ >> >> ... [ cut ] ... >> >> } __randomize_layout; >> >> In each gdb session, run the command "ptype struct mm_struct", >> and compare the two outputs. If they are the identical, then there >> is a major problem. >> > > I compared the output of "ptype struct uts_namespace"(and also > mm_struct), it seems there does is a major problem, the two outputs are > the same:( > >> A fundamental underpinning of the crash utility is the ability >> to gather correct structure member offsets from the embedded gdb >> module, which gets them from the debuginfo data of the vmlinux file. >> If the structure layout is identical in both the randomized >> and non-randomized vmlinux files, then the crash utility will >> be pretty much useless. >> > > Seems horrible thing happens. > >> I agree with Linus Torvalds with respect to this feature, where in >> the article that you referenced, he says: >> >> Nevertheless, Torvalds is unimpressed by structure randomization, >> calling it "security theater". The fact that distributions need to >> publish the randomization seeds for module-building meant it did >> not provide as big of a security feature as advertised. Torvalds >> however did add: "So it's imnsho a pretty questionable security >> thing. It's likely most useful for one-off 'special secure >> installations' than mass productions. >> >> I have long believed that the constant onslaught of new "security" >> features pushed into the kernel by Kees Cook will eventually kill >> the capability of using the crash utility to debug the kernel. So >> far we've been able to work around or adapt to previous security >> related changes, but this particular feature looks to be a crash >> killer. >> > > Yes, also feels that it is not easy to adapt or work around since the > randomized structure layout info is not reflect in the vmlinux's > debuginfo. I don't know gcc internals, so a blind guess is that it is > done in this way on purpose for security, so attackers won't bother to > seek the random seed, but just using gdb to find out the structure layout. I'm not an expert either, but I feel that this is a gcc bug. If anyone is concerned about security, they needn't publish any debuginfo at all, rather than giving useless debuginfo. Have you tried reporting this behaviour upstream as abug? Petr T .

-----Original Message----- From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson Sent: Thursday, January 25, 2018 12:23 AM To: Cao, Jin/曹 晋 <caoj.fnst(a)cn.fujitsu.com&gt; Cc: crash-utility(a)redhat.com Subject: Re: [Crash-utility] Using crash with structure layout randomized kernel ----- Original Message ----- > > > On 01/23/2018 11:19 PM, Dave Anderson wrote: > > > > > > ----- Original Message ----- > >> Hi Dave, > >> > >>     Recently I was trying crash tool with kdump dumpfile & structure > >> layout randomized kernel[*](), and it fails without any surprise. After > >> looking into the different errors crash reports, I can confirm it is a > >> result from randomized structure layout. > >> > >> So my questions is, do you ever consider supporting this feature[*] in > >> crash? > >>   If yes, do you have any plan & technique evaluation about it? > >>   If no, what's the reason? > >> > >> [*]https://lwn.net/Articles/722293/ > >> -- > >> Sincerely, > >> Cao jin > > > > I was under the impression that the structure layout was done at > > compile-time, and that the vmlinux file's debuginfo data would > > represent the randomized layout. And that being the case, the > > inconvenience would be that the crash session would show the > > randomized layout, while the associated source code would show > > the original layout. > > > > BTW, I don't have any compiler knowledge before, just from these two > days learning, I feel you are right at "vmlinux file's debuginfo data > would represent the randomized layout". > > But when I debug, it seem not like what it should be. I have two file > pairs, randomized and non-randomized one. I print some member offset of > structure tagged with __randomize_layout after MEMBER_OFFSET_INIT, like > this one: > > (gdb) p offset_table->task_struct_state > $1 = 8 > (gdb) p offset_table->task_struct_exit_state > $2 = 2164 > (gdb) p offset_table->task_struct_pid > $3 = 2264 > (gdb) p offset_table->task_struct_comm > $4 = 2744 > (gdb) p offset_table->task_struct_next_task > $5 = -1 > (gdb) p offset_table->task_struct_processor > $6 = -1 > (gdb) p offset_table->task_struct_p_pptr > $7 = -1 > (gdb) p offset_table->task_struct_parent > $8 = 2288 > > Under both file pairs, these offset value are the same, so, I think that > is why I have the impression that debuginfo has the original structure > layout. I guess this is one kind of "MEMBER_OFFSET() no longer work"? It appears that way. A better manner would be to simply run gdb on the 2 vmlinux files, i.e., run "gdb vmlinux", and then print out one of the required data structures that has the "__randomize_layout" compiler annotation, such as the mm_struct: struct mm_struct { struct vm_area_struct *mmap; /* list of VMAs */ struct rb_root mm_rb; u32 vmacache_seqnum; /* per-thread vmacache */ ... [ cut ] ... } __randomize_layout; In each gdb session, run the command "ptype struct mm_struct", and compare the two outputs. If they are the identical, then there is a major problem. A fundamental underpinning of the crash utility is the ability to gather correct structure member offsets from the embedded gdb module, which gets them from the debuginfo data of the vmlinux file. If the structure layout is identical in both the randomized and non-randomized vmlinux files, then the crash utility will be pretty much useless. I agree with Linus Torvalds with respect to this feature, where in the article that you referenced, he says: Nevertheless, Torvalds is unimpressed by structure randomization, calling it "security theater". The fact that distributions need to publish the randomization seeds for module-building meant it did not provide as big of a security feature as advertised. Torvalds however did add: "So it's imnsho a pretty questionable security thing. It's likely most useful for one-off 'special secure installations' than mass productions. I have long believed that the constant onslaught of new "security" features pushed into the kernel by Kees Cook will eventually kill the capability of using the crash utility to debug the kernel. So far we've been able to work around or adapt to previous security related changes, but this particular feature looks to be a crash killer.