Re: [Crash-utility] [PATCH] files: support dump file page cache

----- Original Message ----- Hello Oliver, Before getting into the patch specifics, please make it apply to the current git tree contents: $ git clone git://github.com/crash-utility/crash.git Cloning into 'crash'... remote: Counting objects: 954, done. remote: Compressing objects: 100% (3/3), done. remote: Total 954 (delta 0), reused 0 (delta 0), pack-reused 951 Receiving objects: 100% (954/954), 2.08 MiB, done. Resolving deltas: 100% (634/634), done. $ cd crash $ patch -p1 < ../0001-files-support-dump-file-page-caches.patch patching file defs.h Hunk #3 succeeded at 4755 (offset -16 lines). patching file filesys.c patching file memory.c Hunk #1 succeeded at 134 with fuzz 1 (offset 2 lines). Hunk #2 succeeded at 6467 (offset 305 lines). patching file task.c Hunk #1 succeeded at 5612 (offset 11 lines). Hunk #2 succeeded at 5636 (offset 11 lines). Hunk #3 succeeded at 6144 (offset 11 lines). Hunk #4 succeeded at 6377 (offset 11 lines). $ And make sure it compiles cleanly with "make warn": $ make warn ... [ cut ] ... cc -c -g -DX86_64 -DLZO -DSNAPPY -DGDB_7_6 memory.c -Wall -O2 -Wstrict-prototypes -Wmissing-prototypes -fstack-protector -Wformat-security memory.c: In function 'dump_file_address_mappings': memory.c:6477:22: warning: unused variable 'ret' [-Wunused-variable] memory.c:6475:8: warning: unused variable 'radix_tree_rnode' [-Wunused-variable] memory.c: At top level: memory.c:6505:1: warning: no previous prototype for 'get_page_tree_count' [-Wmissing-prototypes] memory.c: In function 'get_page_tree_count': memory.c:6507:8: warning: unused variable 'radix_tree_rnode' [-Wunused-variable] cc -c -g -DX86_64 -DLZO -DSNAPPY -DGDB_7_6 filesys.c -Wall -O2 -Wstrict-prototypes -Wmissing-prototypes -fstack-protector -Wformat-security filesys.c: In function 'cmd_files': filesys.c:2215:4: warning: implicit declaration of function 'dump_file_address_mappings' [-Wimplicit-function-declaration] filesys.c: In function 'file_dump': filesys.c:2894:4: warning: implicit declaration of function 'get_page_tree_count' [-Wimplicit-function-declaration] ... I've only done some quick testing, but for starters, The PATH translation for /dev files is not working the same way as the regular "files" command: crash> files PID: 19772 TASK: ffff810278593820 CPU: 7 COMMAND: "sshd" ROOT: / CWD: / FD FILE DENTRY INODE TYPE PATH 0 ffff8102777021c0 ffff81027cc06078 ffff81027f3efa18 CHR /dev/null 1 ffff8102777021c0 ffff81027cc06078 ffff81027f3efa18 CHR /dev/null 2 ffff81027760abc0 ffff81027cc06078 ffff81027f3efa18 CHR /dev/null 3 ffff810213616dc0 ffff8102130de738 ffff81026d575610 SOCK socket:/[72770] 4 ffff81027e5b5cc0 ffff8102130de8e8 ffff810274152ad0 SOCK socket:/[72980] 5 ffff810216a63e80 ffff810229a28228 ffff81021006a910 PIPE 6 ffff810276ba8c80 ffff810229a28228 ffff81021006a910 PIPE 7 ffff81027e0757c0 ffff8102130dec48 ffff81027f2eb110 SOCK socket:/[72991] 8 ffff8102136160c0 ffff810213079b70 ffff8102741525d0 SOCK socket:/[72992] 9 ffff81027e0755c0 ffff81027f3f0588 ffff81027f3ef418 CHR /dev/ptmx 10 ffff81027e0755c0 ffff81027f3f0588 ffff81027f3ef418 CHR /dev/ptmx 11 ffff81027e0755c0 ffff81027f3f0588 ffff81027f3ef418 CHR /dev/ptmx crash> files -M PID: 19772 TASK: ffff810278593820 CPU: 7 COMMAND: "sshd" ROOT: / CWD: / FD ADDR-SPACE PGCACHE-PGS INODE TYPE PATH 0 ffff81027f3efb28 0 ffff81027f3efa18 CHR /null 1 ffff81027f3efb28 0 ffff81027f3efa18 CHR /null 2 ffff81027f3efb28 0 ffff81027f3efa18 CHR /null 3 ffff81026d575720 0 ffff81026d575610 SOCK socket:/[72770] 4 ffff810274152be0 0 ffff810274152ad0 SOCK socket:/[72980] 5 ffff81021006aa20 0 ffff81021006a910 PIPE 6 ffff81021006aa20 0 ffff81021006a910 PIPE 7 ffff81027f2eb220 0 ffff81027f2eb110 SOCK socket:/[72991] 8 ffff8102741526e0 0 ffff8102741525d0 SOCK socket:/[72992] 9 ffff81027f3ef528 0 ffff81027f3ef418 CHR /ptmx 10 ffff81027f3ef528 0 ffff81027f3ef418 CHR /ptmx 11 ffff81027f3ef528 0 ffff81027f3ef418 CHR /ptmx crash> But more importantly, for "files -M", it's not clear to me what the PGCACHE-PGS count should or does mean. One might expect to pass any ADDR-SPACE address shown by "files -M" to the "files -m <address-space>" option, and see PGCACHE-PGS worth of pages dumped. But that's not always true. For example: crash> files -M PID: 30700 TASK: ffff810876c8d7a0 CPU: 0 COMMAND: "_progres" ROOT: / CWD: /home/TCusa FD ADDR-SPACE PGCACHE-PGS INODE TYPE PATH 0 ffff81080e7095c0 0 ffff81080e7094b0 CHR /45 1 ffff81080e7095c0 0 ffff81080e7094b0 CHR /45 2 ffff81080e7095c0 0 ffff81080e7094b0 CHR /45 3 ffff810fe6da85c0 0 ffff810fe6da84b0 PIPE 4 ffff81021b28eb88 35 ffff81021b28ea78 REG /dlc/101c/convmap.cp 5 ffff811024532b88 6 ffff811024532a78 REG /ProgTemp/lbiVQS7cd 6 ffff810224fbd850 78 ffff810224fbd740 REG /usa/mfg/usa.lg 7 ffff810218e31220 12 ffff810218e31110 REG /usa/mfg/usa.db 8 ffff810218e31220 12 ffff810218e31110 REG /usa/mfg/usa.db 9 ffff810e371fe260 42937 ffff810e371fe150 REG /usa/mfg/usa.b1 10 ffff81102f12ee40 0 ffff81102f12ed30 REG /usa/mfg/usa.b2 11 ffff810224fbde40 490 ffff810224fbdd30 REG /usa/mfg/usa.d1 12 ffff810224fbde40 490 ffff810224fbdd30 REG /usa/mfg/usa.d1 ... Taking FD 7's address space structure, the 12 page cache pages can be dumped: crash> files -m ffff810218e31220 Address Space ffff810218e31220 : 12 pages in page cache PAGE PHYSICAL MAPPING INDEX CNT FLAGS ffff81010774d1f8 221609000 ffff810218e31220 0 1 22010000001006c ffff81010485b378 14ac59000 ffff810218e31220 1 1 14810000001006c ffff810107993a78 22bc79000 ffff810218e31220 2 1 228100000010028 ffff8101049d3660 1517d4000 ffff810218e31220 3 1 150100000010028 ffff810103b29670 10e742000 ffff810218e31220 4 1 108100000010028 ffff810106d51ba0 1f3bec000 ffff810218e31220 5 1 1f0100000010028 ffff810103ac95f0 10cbd2000 ffff810218e31220 6 1 108100000010028 ffff810106c8cc18 1f03a5000 ffff810218e31220 7 1 1f0100000010028 ffff8101077b1028 223293000 ffff810218e31220 8 1 220100000010028 ffff810106cc03b0 1f125a000 ffff810218e31220 9 1 1f0100000010028 ffff810107a04cd8 22dccd000 ffff810218e31220 a 1 228100000010028 ffff8101078741b8 226a51000 ffff810218e31220 b 1 220100000010028 crash> So taking FD 11's ffff810224fbde40, you would expect all 490 pages plus the 3 line header to be dumped: crash> files -m ffff810224fbde40 | wc -l 46 crash> Or taking FD 9's ffff810e371fe260, you would expect 42937 pages plus the header: crash> files -m ffff810e371fe260 | wc -l 2444 crash> So what does that mean exactly? Should the PGCACHE-PGS display show "x of y", where "x" is the number of a file's "y" cached pages that are mapped into the specified address space? I haven't looked too deeply at the patch-set yet, but in my quick test, I ran into this in your new dump_file_address_mappings() function: + /* Now walk the tree, counting all the pages in the tree */ + for (index = 0; index <= count; index++) { + rtp.index = index; + if (do_radix_tree(root_rnode, RADIX_TREE_SEARCH, &rtp)) { + meminfo.spec_addr = (ulong)rtp.value; + meminfo.memtype = KVADDR; + meminfo.flags = ADDRESS_SPECIFIED; + dump_mem_map_SPARSEMEM(&meminfo); + } + } Also, if the kernel is not configured with CONFIG_SPARSEMEM, the "files -m" option fails like this, on a 2.6.9 RHEL4 x86_64 kernel (yes the address space virtual address is correct): crash> files -m 1016fcaa668 Address Space 1016fcaa668 : 20 pages in page cache files: cannot resolve "mem_section" crash> For backwards-compatibility, I did a quick check on a couple older 32-bit x86 kernels, and on a RHEL4 2.6.9-based x86 kernel, "files -M" fails every time: crash> files -M PID: 4846 TASK: c09de0b0 CPU: 0 COMMAND: "dmach7" ROOT: / CWD: /home/m7istp.4.6.18.b4/m7istp/dmach7/bin FD ADDR-SPACE PGCACHE-PGS INODE TYPE PATH radix_tree_root at cf53a3ac: struct radix_tree_root { height = 0x220, gfp_mask = 0x0, rnode = 0x1d244b3c } files: height 544 is greater than height_to_maxindex[] index 7 crash> I thought it might be a problem with really old kernels, but it happens on a 32-bit RHEL5 2.6.18-128.2.1.el5 kernel: crash> files -M PID: 22328 TASK: dbdeb000 CPU: 0 COMMAND: "sushiremote" ROOT: / CWD: /afs/cs.wisc.edu/u/s/u/sushi FD ADDR-SPACE PGCACHE-PGS INODE TYPE PATH radix_tree_root at de8cb798: struct radix_tree_root { height = 0x220, gfp_mask = 0x0, rnode = 0x1000000 } files: height 544 is greater than height_to_maxindex[] index 7 crash> And the same thing on the most recent 32-bit x86 kernel I have on hand, which is 2.6.40.4-5.fc15.i686.PAE: crash> files -M PID: 3804 TASK: f466a5e0 CPU: 0 COMMAND: "crash" ROOT: / CWD: /root/crash-5.1.8 FD ADDR-SPACE PGCACHE-PGS INODE TYPE PATH radix_tree_root at e6d6f644: struct radix_tree_root { height = 0x20, gfp_mask = 0x0, rnode = 0x101 } files: height 32 is greater than height_to_maxindex[] index 7 crash> So then I tried it on a 32-bit ARM 3.10.17 kernel, which also fails: crash> files -M PID: 13429 TASK: db944580 CPU: 1 COMMAND: "AudioIn_5F8" ROOT: / CWD: / FD ADDR-SPACE PGCACHE-PGS INODE TYPE PATH radix_tree_root at db6b1a7c: struct radix_tree_root { height = 0x20, gfp_mask = 0x0, rnode = 0x0 } files: height 32 is greater than height_to_maxindex[] index 7 crash> So I'm guessing that the patch fails on all 32-bit kernels. Dave