Re: [Crash-utility] Crash utility with RT patchset

Hi Kazu, I worked on this for a couple days. I had asked below question in earlier mail: > "Can anyone suggest whether any change is required in either crash utility or RT Linux to make use of crash utility to analyze ramdump with RT Linux?" I was referring to the Linux 5.4 RT patchset (I had mentioned 5.4.61-rt37), I realized that crash does not (need to) support RT Linux. It should just support the mainline kernel(s).

> If it still fails, what do you see with -d8 option? I see kernel configuration(.config) being dumped, correct linux banner and lot of readmem/read_ramdump prints until failure is seen.

Sharing the last few lines from the log, if it helps, can share full if needed:

[--Start of Log--] <readmem: ffffff8019e52100, KVADDR, "fill_task_struct", 4224, (ROE), 464bd90> <read_ramdump: addr: ffffff8019e52100 paddr: 99e52100 cnt: 3840> read_ramdump: addr: ffffff8019e52100 paddr: 99e52100 cnt: 3840 offset: 19e52100 <read_ramdump: addr: ffffff8019e53000 paddr: 99e53000 cnt: 384> read_ramdump: addr: ffffff8019e53000 paddr: 99e53000 cnt: 384 offset: 19e53000 <readmem: ffffff8019e52100, KVADDR, "fill_thread_info", 40, (ROE), 4c92290> <read_ramdump: addr: ffffff8019e52100 paddr: 99e52100 cnt: 40> read_ramdump: addr: ffffff8019e52100 paddr: 99e52100 cnt: 40 offset: 19e52100 FREEBUF(1) GETBUF(8 -> 1) <readmem: fffffff00af339f8, KVADDR, "cpu_online_map", 8, (ROE), d21fc0> <read_ramdump: addr: fffffff00af339f8 paddr: 819339f8 cnt: 8> read_ramdump: addr: fffffff00af339f8 paddr: 819339f8 cnt: 8 offset: 19339f8 crash: get_cpus_online: online: 8 FREEBUF(1) please wait... (determining panic task) crash: get_netdump_panic_task: failed crash: invalid kernel virtual address: fffffff791ff5a5f type: "64-bit KVADDR" [--End of Log--]

I had an additional question, just out of curiosity: After the latest printk ring buffer implementation was added to crash 7.3.0; does crash still work with the old printk implementation? For ex., is the latest crash 7.3.0 expected to work for dmesg with mainline kernel 5.4?

Regards, Abhishek On Fri, Jul 16, 2021 at 12:04 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com&gt; wrote: > > Hi Abhishek, > > -----Original Message----- > > I see the same issue with crash-7.3.0 as well. > > > > crash: invalid kernel virtual address: fffffff791ff5a5f type: "64-bit KVADDR" > > Do you see the same issue with the latest commit f53b73e8380bc? > If it still fails, what do you see with -d8 option? > > Thanks, > Kazu > > > > > > > Regards, > > Abhishek > > > > On Fri, Jul 9, 2021 at 2:20 PM Abhishek Shah <abrs612(a)gmail.com&gt; wrote: > > > > > > Hi Kazu, > > > > > > I am trying crash from master branch, top commit: d6b4f36d6b22 > > > ("Handle task_struct state member changes for kernels >= 5.14-rc1") > > > I will give it a try with the released crash-7.3.0 and update. > > > > > > > > > Regards, > > > Abhishek > > > > > > On Fri, Jul 9, 2021 at 1:26 PM HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com&gt; wrote: > > > > > > > > Hi Abhishek, > > > > > > > > Does it happens with the released crash-7.3.0 ? > > > > Recently we had four patches for arm64 after the release, > > > > I'd like to know whether it's a regression or not to narrow down. > > > > > > > > Thanks, > > > > Kazu > > > > > > > > -----Original Message----- > > > > > Hi Ron, > > > > > > > > > > > I guess the address convention seems x8664 when looking at 0xffffffxxxxxxxx type sort of 64 bit > > virtual > > > > > address. > > > > > > > > > > arm64 also has a similar range of virtual addresses (see > > > > > "Documentation/arm64/memory.rst" in Linux), > > > > > currently my target is using 39-bits (as specified in crash command > > > > > used below) and 4 kB page table. > > > > > > > > > > I see below address range being used, when giving "-d 1" to enable debug prints: > > > > > > > > > > crash: get_cpus_present: present: 8 > > > > > hypervisor: (undetermined) > > > > > ikconfig: 1616 valid configs. > > > > > SECTION_SIZE_BITS: 30 > > > > > CONFIG_ARM64_VA_BITS: 39 > > > > > VA_BITS_ACTUAL: 39 > > > > > (calculated) VA_BITS: 39 > > > > > PAGE_OFFSET: ffffff8000000000 > > > > > VA_START: ffffffc000000000 > > > > > modules: ffffffc008000000 - ffffffc00fffffff > > > > > vmalloc: ffffffc010000000 - fffffffebffeffff > > > > > kernel image: fffffff009680000 - fffffff00b194000 > > > > > vmemmap: fffffffeffe00000 - ffffffffffffffff > > > > > > > > > > > > > > > Also ,There is no issue with crash (compiled for arm64) or vmlinux (arm64) type. > > > > > > > > > > > Either that or possibly the offset at which you are trying to load. > > > > > Are you referring to kaslr offset? Without that in place, I get a > > > > > vmlinux/ramdump mismatch error. > > > > > So I am confident that it is correct. > > > > > > > > > > Can anyone suggest whether any change is required in either crash utility or > > > > > RT Linux to make use of crash utility to analyze ramdump with RT Linux? > > > > > > > > > > > > > > > Regards, > > > > > Abhishek > > > > > > > > > > On Thu, Jul 8, 2021 at 7:49 AM Ron HM <gpl.positron(a)gmail.com&gt; wrote: > > > > > > > > > > > > Either that or possibly the offset at which you are trying to load. > > > > > > > > > > > > On Thu, 8 Jul 2021, 9:28 am Ron HM, <gpl.positron(a)gmail.com&gt; wrote: > > > > > >> > > > > > >> Hi Abhishek, > > > > > >> I guess the address convention seems x8664 when looking at 0xffffffxxxxxxxx type sort of 64 bit > > virtual > > > > > address. > > > > > >> > > > > > >> ------ > > > > > >> crash: invalid kernel virtual address: fffffff791ff5a5f type: "64-bit KVADDR" > > > > > >> ----- > > > > > >> > > > > > >> For arm64 bit the addressing would more be sort of 0xCCAbxxxxxxx sorts. Due to the convention of > > memory > > > > > addressing start addresses. > > > > > >> > > > > > >> As a first check confirm if your crash and vmlinuz file is x8664 type > > > > > >> # file vmlinux > > > > > >> # file crash > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Wed, 7 Jul 2021, 3:53 pm Abhishek Shah, <abrs612(a)gmail.com&gt; wrote: > > > > > >>> > > > > > >>> Hi All, > > > > > >>> > > > > > >>> I was wondering if there is any change required in either crash utility or > > > > > >>> RT Linux to make use of crash utility to analyze ramdump with RT Linux. > > > > > >>> > > > > > >>> I am using crash 7.3.0++ and Linux - 5.4.61-rt37 on arm64 target. > > > > > >>> I see the below error: > > > > > >>> ../crash DDR0.BIN@0x80000000,DDR1.BIN(a)0x100000000 vmlinux > > > > > >>> --machdep vabits_actual=39 --kaslr 0x2ff9600000 > > > > > >>> ...... > > > > > >>> crash: invalid kernel virtual address: fffffff791ff5a5f type: "64-bit KVADDR" > > > > > >>> > > > > > >>> > > > > > >>> Regards, > > > > > >>> Abhishek > > > > > >>> > > > > > >>> -- > > > > > >>> Crash-utility mailing list > > > > > >>> Crash-utility(a)redhat.com > > > > > >>> https://listman.redhat.com/mailman/listinfo/crash-utility > > > > > >>> > > > > > > -- > > > > > > Crash-utility mailing list > > > > > > Crash-utility(a)redhat.com > > > > > > https://listman.redhat.com/mailman/listinfo/crash-utility > > > > > > > > > > -- > > > > > Crash-utility mailing list > > > > > Crash-utility(a)redhat.com > > > > > https://listman.redhat.com/mailman/listinfo/crash-utility > > > > > > > > > > > > -- > > > > Crash-utility mailing list > > > > Crash-utility(a)redhat.com > > > > https://listman.redhat.com/mailman/listinfo/crash-utility > > > > > > > > > > -- > > Crash-utility mailing list > > Crash-utility(a)redhat.com > > https://listman.redhat.com/mailman/listinfo/crash-utility > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://listman.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://listman.redhat.com/mailman/listinfo/crash-utility