Hi lianbo: I have a bug fixed need to merge to crash-utility which patch in attachment. It can obviously see that kmem -i show memory info of system. But CACHED have wrong output for add up size of cached memory. The reason of this bug is: newest kernel version use "struct address_space *swapper_spaces[MAX_SWAPFILES]" instead of "struct address_space swapper_spaces[MAX_SWAPFILES]" in old kernel. this change bring into kernel after kernel 4.11.0

So newest version need to readmem twice, first is address of one struct address_space second is value of struct address_space. This bug fix patch add twice readmem when read swapper_spaces struct, it can accurately read nrpage in struct address_space. And i found that definition of struct address_space swapper_spaces change in kernel 4.11.0, so i make kernel version compatible for this change. fix before: crash> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 2854115 10.9 GB ---- FREE 169699 662.9 MB 5% of TOTAL MEM USED 2684416 10.2 GB 94% of TOTAL MEM SHARED 891094 3.4 GB 31% of TOTAL MEM BUFFERS 329 1.3 MB 0% of TOTAL MEM CACHED 873327085626 3331478.4 GB 30598875% of TOTAL MEM SLAB 230128 898.9 MB 8% of TOTAL MEM fix after: crash> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 2854115 10.9 GB ---- FREE 169699 662.9 MB 5% of TOTAL MEM USED 2684416 10.2 GB 94% of TOTAL MEM SHARED 891094 3.4 GB 31% of TOTAL MEM BUFFERS 329 1.3 MB 0% of TOTAL MEM CACHED 1729018 6.6 GB 60% of TOTAL MEM SLAB 230128 898.9 MB 8% of TOTAL MEM Thanks Best Regards

> RELEASE: 6.1.25-android14-11-maybe-dirty-qki-consolidate > crash> nr_swapper_spaces > crash: command not found: nr_swapper_spaces

Hi kazu: 1. I found that vmlinux of kernel-6.1 did not have symbol nr_swapper_spaces but only have swapper_spaces Also check in GKI vmlinux, can not find nr_swapper_spaces

It may have some problem in symbol of nr_swapper_spaces 2. I think that crash tools will check nr_swapper_spaces first and then check *swapper_spaces, swaper_space[] last. so first check nr_swapper_spaces not found, crash tool can use *swapper_spaces to enhance compatibility. The patch i send can resolve this situation which nr_swapper_spaces symbols can not found.

Hi kazu: I found that vmlinux of GKI kernel-6.1 which is newest version in links below, it can make sure that nr_swapper_spaces not found. https://ci.android.com/builds/submitted/11087000/kernel_aarch64/latest?hl... <https://ci.android.com/builds/submitted/11087000/kernel_aarch64/latest?hl... Download vmlinux in links and run: readelf -a vmlinux | grep nr_swapper_spaces

can not find this symbol. Thanks！ Best Regards ------------------------------------------------------------------------ *From:* 薛国伦 *Sent:* Friday, December 1, 2023 4:07:08 PM *To:* HAGIO KAZUHITO(萩尾　一仁); devel(a)lists.crash-utility.osci.io *Cc:* Lianbo Jiang *Subject:* Re: [External Mail]Re: [Crash-utility] [BUG FIXED]fix bug of CACHED in kmem -i show memory Hi kazu: 1. I know that kernel have nr_swapper_spaces since kernel 4.11 which brought into kernel with struct address_space *swapper_spaces[] my team think GKI vmlinux did not have nr_swapper_spaces this symbol, may due to compile rule changed in AOSP code. we will try to find the root cause of vmlinux without symbol name nr_swappera_space.      >> RELEASE: 6.1.25-android14-11-maybe-dirty-qki-consolidate      >> crash> nr_swapper_spaces      >> crash: command not found: nr_swapper_spaces 2. I printf log of crash when execute kmem -i, it run into branch of "symbol_exists("swapper_spaces")", also can not find nr_swapper_spaces if (symbol_exists("nr_swapper_spaces") {     ...... } else if (symbol_exists("swapper_spaces") {     ...... } else if (symbol_exists("swapper_space") {     ...... } 3. using my patch, it can fix output log of kmem -i which CACHED show not right. Fix before: crash> kmem -i                  PAGES        TOTAL      PERCENTAGE     TOTAL MEM  2854115      10.9 GB         ----          FREE   169699     662.9 MB    5% of TOTAL MEM          USED  2684416      10.2 GB   94% of TOTAL MEM        SHARED   891094       3.4 GB   31% of TOTAL MEM       BUFFERS      329       1.3 MB    0% of TOTAL MEM *       CACHED  873327085626  3331478.4 GB  30598875% of TOTAL MEM*          SLAB   230128     898.9 MB    8% of TOTAL MEM Fix after: crash> kmem -i                  PAGES        TOTAL      PERCENTAGE     TOTAL MEM  2854115      10.9 GB         ----          FREE   169699     662.9 MB    5% of TOTAL MEM          USED  2684416      10.2 GB   94% of TOTAL MEM        SHARED   891094       3.4 GB   31% of TOTAL MEM       BUFFERS      329       1.3 MB    0% of TOTAL MEM *       CACHED  1729018       6.6 GB   60% of TOTAL MEM*          SLAB   230128     898.9 MB    8% of TOTAL MEM I think crash tools without my patch, maybe miss some situation like this which symbol miss in kenrel vmlinux. I think that using both branch of symbol_exist("nr_swapper_spaces") and symbol_exists("swapper_spaces") means to double check the situation of all kinds of kernel and vmlinux. It seen that bring my patch can deal with more unusual situation and make crash tool more compatible. Thanks！ Best Regards ------------------------------------------------------------------------ *From:* HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com&gt; *Sent:* Friday, December 1, 2023 3:08:03 PM *To:* 薛国伦; devel(a)lists.crash-utility.osci.io *Cc:* Lianbo Jiang *Subject:* Re: [External Mail]Re: [Crash-utility] [BUG FIXED]fix bug of CACHED in kmem -i show memory [外部邮件] 此邮件来源于小米公司外部，请谨慎处理。若对邮件安全性存疑，请将邮件转发给misec(a)xiaomi.xn--com-iw3ew31vyqjqpq On 2023/11/30 22:05, 薛国伦 wrote: > Hi kazu: > > > 1. I found that vmlinux of kernel-6.1 did not have symbol nr_swapper_spaces but only have swapper_spaces > > Also check in GKI vmlinux, can not find nr_swapper_spaces Hmm, upstream kernel 6.1 has it. $ git show v6.1:mm/swap_state.c struct address_space *swapper_spaces[MAX_SWAPFILES] __read_mostly; static unsigned int nr_swapper_spaces[MAX_SWAPFILES] __read_mostly; > > It may have some problem in symbol of nr_swapper_spaces > > > 2. I think that crash tools will check nr_swapper_spaces first and then check *swapper_spaces, swaper_space[] last. > > so first check nr_swapper_spaces not found, crash tool can use *swapper_spaces to enhance compatibility. > > The patch i send can resolve this situation which nr_swapper_spaces symbols can not found. But I cannot determine whether your patch is correct and should be applied, without kernel patches or the cause of no nr_swapper_spaces. Could you please find the information (e.g. links) of the related kernel patches? Thanks, Kazu #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

Hi Lianbo: the vmlinux i download from links: https://ci.android.com/builds/submitted/11087000/kernel_aarch64/latest?hl... <https://ci.android.com/builds/submitted/11087000/kernel_aarch64/latest?hl... md5sum ~/Downloads/vmlinux 5927b52ad342e6e34136b04b9d16ab46 vmlinux

can not get nr_swapper_spaces symbols Thanks! Best regards ------------------------------------------------------------------------ *From:* Lianbo Jiang <lijiang(a)redhat.com&gt; *Sent:* Wednesday, December 6, 2023 11:33:16 AM *To:* 薛国伦; HAGIO KAZUHITO(萩尾 一仁); devel(a)lists.crash-utility.osci.io *Subject:* Re: [External Mail]Re: [Crash-utility] [BUG FIXED]fix bug of CACHED in kmem -i show memory [外部邮件] 此邮件来源于小米公司外部，请谨慎处理。若对邮件安全性存疑，请将邮件转发给misec(a)xiaomi.xn--com-iw3ew31vyqjqpq On 12/1/23 16:30, 薛国伦 wrote: > Hi kazu: > > > I found that vmlinux of GKI kernel-6.1 which is newest version > in links below, it can make sure that nr_swapper_spaces not found. > > > https://ci.android.com/builds/submitted/11087000/kernel_aarch64/latest?hl... > <https://ci.android.com/builds/submitted/11087000/kernel_aarch64/latest?hl... > > > Download vmlinux in links and run: readelf -a vmlinux | grep > nr_swapper_spaces > It's weird, I downloaded the vmlinux from the above link, got the nr_swapper_spaces: $ md5sum vmlinux 7d47aa96b3812e6ca607de7e297307a1  vmlinux $ strings vmlinux |grep nr_swapper_spaces nr_swapper_spaces nr_swapper_spaces nr_swapper_spaces $ readelf -a vmlinux |grep nr_swapper_spaces   17111: c000000001856810   108 OBJECT  LOCAL  DEFAULT   47 nr_swapper_spaces Has the vmlinux been updated again? Thanks Lianbo > > can not find this symbol. > > > Thanks！ > Best Regards > ------------------------------------------------------------------------ > *From:* 薛国伦 > *Sent:* Friday, December 1, 2023 4:07:08 PM > *To:* HAGIO KAZUHITO(萩尾　一仁); devel(a)lists.crash-utility.osci.io > *Cc:* Lianbo Jiang > *Subject:* Re: [External Mail]Re: [Crash-utility] [BUG FIXED]fix bug > of CACHED in kmem -i show memory > > Hi kazu: > > > 1. I know that kernel have nr_swapper_spaces since kernel 4.11 which > brought into kernel with struct address_space *swapper_spaces[] > > my team think GKI vmlinux did not have nr_swapper_spaces this symbol, > may due to compile rule changed in AOSP code. > > we will try to find the root cause of vmlinux without symbol name > nr_swappera_space. > > >      >> RELEASE: 6.1.25-android14-11-maybe-dirty-qki-consolidate >      >> crash> nr_swapper_spaces >      >> crash: command not found: nr_swapper_spaces > > 2. I printf log of crash when execute kmem -i, it run into branch of > "symbol_exists("swapper_spaces")", also can not find nr_swapper_spaces > > > if (symbol_exists("nr_swapper_spaces") { > >     ...... > > } else if (symbol_exists("swapper_spaces") { > >     ...... > > } else if (symbol_exists("swapper_space") { > >     ...... > > } > > > 3. using my patch, it can fix output log of kmem -i which CACHED show > not right. > > > Fix before: > crash> kmem -i >                  PAGES        TOTAL      PERCENTAGE >     TOTAL MEM  2854115      10.9 GB         ---- >          FREE   169699     662.9 MB    5% of TOTAL MEM >          USED  2684416      10.2 GB   94% of TOTAL MEM >        SHARED   891094       3.4 GB   31% of TOTAL MEM >       BUFFERS      329       1.3 MB    0% of TOTAL MEM > *       CACHED  873327085626  3331478.4 GB  30598875% of TOTAL MEM* >          SLAB   230128     898.9 MB    8% of TOTAL MEM > > Fix after: > crash> kmem -i >                  PAGES        TOTAL      PERCENTAGE >     TOTAL MEM  2854115      10.9 GB         ---- >          FREE   169699     662.9 MB    5% of TOTAL MEM >          USED  2684416      10.2 GB   94% of TOTAL MEM >        SHARED   891094       3.4 GB   31% of TOTAL MEM >       BUFFERS      329       1.3 MB    0% of TOTAL MEM > *       CACHED  1729018       6.6 GB   60% of TOTAL MEM* >          SLAB   230128     898.9 MB    8% of TOTAL MEM > > > I think crash tools without my patch, maybe miss some situation like > this which symbol miss in kenrel vmlinux. > I think that using both branch of symbol_exist("nr_swapper_spaces") > and symbol_exists("swapper_spaces") means to double check > the situation of all kinds of kernel and vmlinux. > It seen that bring my patch can deal with more unusual situation and > make crash tool more compatible. > > > Thanks！ > Best Regards > > ------------------------------------------------------------------------ > *From:* HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab(a)nec.com&gt; > *Sent:* Friday, December 1, 2023 3:08:03 PM > *To:* 薛国伦; devel(a)lists.crash-utility.osci.io > *Cc:* Lianbo Jiang > *Subject:* Re: [External Mail]Re: [Crash-utility] [BUG FIXED]fix bug > of CACHED in kmem -i show memory > [外部邮件] > 此邮件来源于小米公司外部，请谨慎处理。若对邮件安全性存疑，请将邮件转发给misec(a)xiaomi.xn--com-iw3ew31vyqjqpq > > On 2023/11/30 22:05, 薛国伦 wrote: > > Hi kazu: > > > > > > 1. I found that vmlinux of kernel-6.1 did not have symbol > nr_swapper_spaces but only have swapper_spaces > > > > Also check in GKI vmlinux, can not find nr_swapper_spaces > > Hmm, upstream kernel 6.1 has it. > > $ git show v6.1:mm/swap_state.c > > struct address_space *swapper_spaces[MAX_SWAPFILES] __read_mostly; > static unsigned int nr_swapper_spaces[MAX_SWAPFILES] __read_mostly; > > > > > It may have some problem in symbol of nr_swapper_spaces > > > > > > 2. I think that crash tools will check nr_swapper_spaces first and > then check *swapper_spaces, swaper_space[] last. > > > > so first check nr_swapper_spaces not found, crash tool can use > *swapper_spaces to enhance compatibility. > > > > The patch i send can resolve this situation which nr_swapper_spaces > symbols can not found. > > But I cannot determine whether your patch is correct and should be > applied, without kernel patches or the cause of no nr_swapper_spaces. > Could you please find the information (e.g. links) of the related kernel > patches? > > Thanks, > Kazu > #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ > This e-mail and its attachments contain confidential information from > XIAOMI, which is intended only for the person or entity whose address > is listed above. Any use of the information contained herein in any > way (including, but not limited to, total or partial disclosure, > reproduction, or dissemination) by persons other than the intended > recipient(s) is prohibited. If you receive this e-mail in error, > please notify the sender by phone or email immediately and delete > it!******/# #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

On 2023/12/07 10:07, Lianbo Jiang wrote: > On 12/6/23 11:45, 薛国伦 wrote: > >> Hi Lianbo: >> >> >> the vmlinux i download from links: >> https://ci.android.com/builds/submitted/11087000/kernel_aarch64/latest?hl... <https://ci.android.com/builds/submitted/11087000/kernel_aarch64/latest?hl... >> >> >> md5sum ~/Downloads/vmlinux >> >> 5927b52ad342e6e34136b04b9d16ab46 vmlinux >> > It's true,  it looks like the variable nr_swapper_spaces is optimized away. > > > (gdb) p linux_banner > $3 = "Linux version 6.1.43-android14-11-gebd072ea164a-ab11087000 > (build-user@build-host) (Android (10087095, +pgo, +bolt, +lto, -mlgo, > based on r487747c) clang version 17.0.2 (https://android.googlesource.c"... > > > (gdb)  p nr_swapper_spaces > $4 = <optimized out> Thank you folks, for the various information. Apparently the data of nr_swapper_spaces exists in the kernel, but its symbol does not exist. I think this means that we cannot calculate the number of pages from swapper_spaces[] with such a debuginfo, because we cannot get the array size of a swapper_spaces entry.

So I'd like to go with Johan's patch. cc -c -g -DX86_64 -DLZO -DSNAPPY -DGDB_10_2 memory.c -Wall -O2 -Wstrict-prototypes -Wmissing-prototypes -fstack-protector -Wformat-security memory.c: In function dump_kmeminfo: memory.c:8612:50: warning: pointer targets in passing argument 2 of dump_vm_stat differ in signedness [-Wpointer-sign] 8612 | if (dump_vm_stat("NR_SWAPCACHE", &swapper_space_nrpages, 0)) { | ^~~~~~~~~~~~~~~~~~~~~~ | | | ulong * {aka long unsigned int *} memory.c:298:33: note: expected long int * but argument is of type ulong * {aka long unsigned int *} 298 | static int dump_vm_stat(char *, long *, ulong); | ^~~~~~ So with the following, - ulong swapper_space_nrpages; + long swapper_space_nrpages;

Acked-by: Kazuhito Hagio <k-hagio-ab(a)nec.com&gt; Thanks, Kazu

>>> (gdb) p nr_swapper_spaces >>> $4 = <optimized out> >> Thank you folks, for the various information. >> >> Apparently the data of nr_swapper_spaces exists in the kernel, but its >> symbol does not exist. I think this means that we cannot calculate the >> number of pages from swapper_spaces[] with such a debuginfo, because we >> cannot get the array size of a swapper_spaces entry. > > That's correct, Kazu. > > >> So I'd like to go with Johan's patch. >> >> cc -c -g -DX86_64 -DLZO -DSNAPPY -DGDB_10_2 memory.c -Wall -O2 >> -Wstrict-prototypes -Wmissing-prototypes -fstack-protector >> -Wformat-security >> memory.c: In function dump_kmeminfo: >> memory.c:8612:50: warning: pointer targets in passing argument 2 of >> dump_vm_stat differ in signedness [-Wpointer-sign] >> 8612 | if (dump_vm_stat("NR_SWAPCACHE", >> &swapper_space_nrpages, 0)) { >> | >> ^~~~~~~~~~~~~~~~~~~~~~ >> | | >> | ulong * {aka >> long unsigned int *} >> memory.c:298:33: note: expected long int * but argument is of type ulong >> * {aka long unsigned int *} >> 298 | static int dump_vm_stat(char *, long *, ulong); >> | ^~~~~~ >> >> So with the following, >> >> - ulong swapper_space_nrpages; >> + long swapper_space_nrpages; > > Looks good. Maybe the patch log also needs to be improved a little bit. > > Although the checking of kernel version looks not very good, seems no > better way. hmm, this sounds strange. My ack is for Johan's patch (attached), which does not have kernel version check, is this ok? sorry for confusing. Johan, your patch does not have your Signed-off-by tag, could I have it?

It indicates that the attached patch may not work well between kernel 4.11 and 5.12, is it possible to encounter the similar issue(the nr_swapper_spaces may be optimized away)?

On 2023/12/07 16:57, Lianbo Jiang wrote: >>> (gdb) p nr_swapper_spaces >>> $4 = <optimized out> >> Thank you folks, for the various information. >> >> Apparently the data of nr_swapper_spaces exists in the kernel, but its >> symbol does not exist. I think this means that we cannot calculate the >> number of pages from swapper_spaces[] with such a debuginfo, because we >> cannot get the array size of a swapper_spaces entry. > That's correct, Kazu. > > >> So I'd like to go with Johan's patch. >> >> cc -c -g -DX86_64 -DLZO -DSNAPPY -DGDB_10_2 memory.c -Wall -O2 >> -Wstrict-prototypes -Wmissing-prototypes -fstack-protector >> -Wformat-security >> memory.c: In function dump_kmeminfo: >> memory.c:8612:50: warning: pointer targets in passing argument 2 of >> dump_vm_stat differ in signedness [-Wpointer-sign] >> 8612 | if (dump_vm_stat("NR_SWAPCACHE", >> &swapper_space_nrpages, 0)) { >> | >> ^~~~~~~~~~~~~~~~~~~~~~ >> | | >> | ulong * {aka >> long unsigned int *} >> memory.c:298:33: note: expected long int * but argument is of type ulong >> * {aka long unsigned int *} >> 298 | static int dump_vm_stat(char *, long *, ulong); >> | ^~~~~~ >> >> So with the following, >> >> - ulong swapper_space_nrpages; >> + long swapper_space_nrpages; > Looks good. Maybe the patch log also needs to be improved a little bit. > > Although the checking of kernel version looks not very good, seems no > better way. hmm, this sounds strange. My ack is for Johan's patch (attached), which

does not have kernel version check, is this ok? sorry for confusing.

Johan, your patch does not have your Signed-off-by tag, could I have it? Thanks, Kazu

> RELEASE: 6.1.25-android14-11-maybe-dirty-qki-consolidate > crash> nr_swapper_spaces > crash: command not found: nr_swapper_spaces

Hi kazu: 1. I found that vmlinux of kernel-6.1 did not have symbol nr_swapper_spaces but only have swapper_spaces Also check in GKI vmlinux, can not find nr_swapper_spaces

It may have some problem in symbol of nr_swapper_spaces 2. I think that crash tools will check nr_swapper_spaces first and then check *swapper_spaces, swaper_space[] last. so first check nr_swapper_spaces not found, crash tool can use *swapper_spaces to enhance compatibility. The patch i send can resolve this situation which nr_swapper_spaces symbols can not found.