5:16 p.m.
Hi Dave, et al.,
Having hacked my way around the self-sabotage on the temp file:
#define mWARN( _m, _a...) do { \
FILE * _sv_fp = fp; \
fp = stdout; \
error(WARNING, _m, ##_a); \
fp = _sv_fp; \
} while (0)
That also removed the mysterious problem of having a duplicate error
message show up on the console.
My next problem is that I seem to be getting inconsistencies between
the data I can print and the data I can find mapping information about:
crash> gdb set $tp = (struct cfs_trace_page *)0xffff8807fb590740
crash> p $tp->page
$7 = (cfs_page_t *) 0xffffea001bb1d1e8
crash> p *$tp->page
$8 = {
flags = 144115188075855872,
_count = {
counter = 1
},
[...]
lru = {
next = 0xdead000000100100,
prev = 0xdead000000200200
}
}
crash> kmem 0xffffea001bb1d1e8
kmem: WARNING: cannot find mem_map page for address: ffffea001bb1d1e8
879b1d1e8: kernel virtual address not found in mem map
So I can print out the page_t structure (renamed as cfs_page_t in Lustre)
at address 0xffff8807fb590740, but when I try to get kmem information about
it, it cannot find the page. What am I missing?
Thanks for hints/pointers! Regards, Bruce
11:03 a.m.
----- Original Message -----
Hi Dave, et al.,
Having hacked my way around the self-sabotage on the temp file:
> #define mWARN( _m, _a...) do { \
> FILE * _sv_fp = fp; \
> fp = stdout; \
> error(WARNING, _m, ##_a); \
> fp = _sv_fp; \
> } while (0)
That also removed the mysterious problem of having a duplicate error
message show up on the console.
OK, if that works for you...
As I mentioned before, the mysterious duplicate to stdout comes from
the else part of the if-else above the code segment you showed, and
is intended.
My next problem is that I seem to be getting inconsistencies between
the data I can print and the data I can find mapping information
about:
> crash> gdb set $tp = (struct cfs_trace_page *)0xffff8807fb590740
> crash> p $tp->page
> $7 = (cfs_page_t *) 0xffffea001bb1d1e8
> crash> p *$tp->page
> $8 = {
> flags = 144115188075855872,
> _count = {
> counter = 1
> },
> [...]
> lru = {
> next = 0xdead000000100100,
> prev = 0xdead000000200200
> }
> }
> crash> kmem 0xffffea001bb1d1e8
> kmem: WARNING: cannot find mem_map page for address: ffffea001bb1d1e8
> 879b1d1e8: kernel virtual address not found in mem map
So I can print out the page_t structure (renamed as cfs_page_t in Lustre)
at address 0xffff8807fb590740, but when I try to get kmem information about
it, it cannot find the page. What am I missing?
Thanks for hints/pointers! Regards, Bruce
I'm not sure, other than it doesn't seem to be able to find ffffea001bb1d1e8
in the ranges of virtual memmap regions used to map page structures. I tried
to reproduce it using a vmemmap address known to be beyond the end of the mapped
pages, but I get a different message:
crash> kmem -p | tail
ffffea00087ffd80 21fff6000 0 0 0 0
ffffea00087ffdc0 21fff7000 0 0 0 0
ffffea00087ffe00 21fff8000 0 0 0 0
ffffea00087ffe40 21fff9000 0 0 0 0
ffffea00087ffe80 21fffa000 0 0 0 0
ffffea00087ffec0 21fffb000 0 0 0 0
ffffea00087fff00 21fffc000 0 0 0 0
ffffea00087fff40 21fffd000 0 0 0 0
ffffea00087fff80 21fffe000 0 0 0 0
ffffea00087fffc0 21ffff000 0 0 0 0
crash> kmem ffffea00087fffc0
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea00087fffc0 21ffff000 0 0 0 0
crash> kmem ffffea0008800000
kmem: WARNING: cannot make virtual-to-physical translation: ffffea0008800000
ffffea0008800000: kernel virtual address not found in mem map
crash>
Is ffffea001bb1d1e8 beyond the end of your "kmem -p" output?
If not, the "kmem -n" command shows each (virtually contiguous) memmap region
starting address under the MEM_MAP column:
crash> kmem -n
NODE SIZE PGLIST_DATA BOOTMEM_DATA NODE_ZONES
0 2221552 ffff88021e5ec000 ---- ffff88021e5ec000
ffff88021e5ec6c0
ffff88021e5ecd80
ffff88021e5ed440
MEM_MAP START_PADDR START_MAPNR
ffffea0000000400 10000 16
ZONE NAME SIZE MEM_MAP START_PADDR START_MAPNR
0 DMA 4080 ffffea0000000400 10000 16
1 DMA32 1044480 ffffea0000040000 1000000 4096
2 Normal 1172992 ffffea0004000000 100000000 1048576
3 Movable 0 0 0 0
-------------------------------------------------------------------
NR SECTION CODED_MEM_MAP MEM_MAP PFN
0 ffff88021e5eb000 ffffea0000000000 ffffea0000000000 0
1 ffff88021e5eb020 ffffea0000000000 ffffea0000200000 32768
2 ffff88021e5eb040 ffffea0000000000 ffffea0000400000 65536
3 ffff88021e5eb060 ffffea0000000000 ffffea0000600000 98304
4 ffff88021e5eb080 ffffea0000000000 ffffea0000800000 131072
5 ffff88021e5eb0a0 ffffea0000000000 ffffea0000a00000 163840
6 ffff88021e5eb0c0 ffffea0000000000 ffffea0000c00000 196608
7 ffff88021e5eb0e0 ffffea0000000000 ffffea0000e00000 229376
8 ffff88021e5eb100 ffffea0000000000 ffffea0001000000 262144
9 ffff88021e5eb120 ffffea0000000000 ffffea0001200000 294912
10 ffff88021e5eb140 ffffea0000000000 ffffea0001400000 327680
11 ffff88021e5eb160 ffffea0000000000 ffffea0001600000 360448
12 ffff88021e5eb180 ffffea0000000000 ffffea0001800000 393216
13 ffff88021e5eb1a0 ffffea0000000000 ffffea0001a00000 425984
14 ffff88021e5eb1c0 ffffea0000000000 ffffea0001c00000 458752
15 ffff88021e5eb1e0 ffffea0000000000 ffffea0001e00000 491520
16 ffff88021e5eb200 ffffea0000000000 ffffea0002000000 524288
17 ffff88021e5eb220 ffffea0000000000 ffffea0002200000 557056
18 ffff88021e5eb240 ffffea0000000000 ffffea0002400000 589824
19 ffff88021e5eb260 ffffea0000000000 ffffea0002600000 622592
20 ffff88021e5eb280 ffffea0000000000 ffffea0002800000 655360
21 ffff88021e5eb2a0 ffffea0000000000 ffffea0002a00000 688128
22 ffff88021e5eb2c0 ffffea0000000000 ffffea0002c00000 720896
23 ffff88021e5eb2e0 ffffea0000000000 ffffea0002e00000 753664
24 ffff88021e5eb300 ffffea0000000000 ffffea0003000000 786432
25 ffff88021e5eb320 ffffea0000000000 ffffea0003200000 819200
26 ffff88021e5eb340 ffffea0000000000 ffffea0003400000 851968
32 ffff88021e5eb400 ffffea0000000000 ffffea0004000000 1048576
33 ffff88021e5eb420 ffffea0000000000 ffffea0004200000 1081344
34 ffff88021e5eb440 ffffea0000000000 ffffea0004400000 1114112
35 ffff88021e5eb460 ffffea0000000000 ffffea0004600000 1146880
36 ffff88021e5eb480 ffffea0000000000 ffffea0004800000 1179648
37 ffff88021e5eb4a0 ffffea0000000000 ffffea0004a00000 1212416
38 ffff88021e5eb4c0 ffffea0000000000 ffffea0004c00000 1245184
39 ffff88021e5eb4e0 ffffea0000000000 ffffea0004e00000 1277952
40 ffff88021e5eb500 ffffea0000000000 ffffea0005000000 1310720
41 ffff88021e5eb520 ffffea0000000000 ffffea0005200000 1343488
42 ffff88021e5eb540 ffffea0000000000 ffffea0005400000 1376256
43 ffff88021e5eb560 ffffea0000000000 ffffea0005600000 1409024
44 ffff88021e5eb580 ffffea0000000000 ffffea0005800000 1441792
45 ffff88021e5eb5a0 ffffea0000000000 ffffea0005a00000 1474560
46 ffff88021e5eb5c0 ffffea0000000000 ffffea0005c00000 1507328
47 ffff88021e5eb5e0 ffffea0000000000 ffffea0005e00000 1540096
48 ffff88021e5eb600 ffffea0000000000 ffffea0006000000 1572864
49 ffff88021e5eb620 ffffea0000000000 ffffea0006200000 1605632
50 ffff88021e5eb640 ffffea0000000000 ffffea0006400000 1638400
51 ffff88021e5eb660 ffffea0000000000 ffffea0006600000 1671168
52 ffff88021e5eb680 ffffea0000000000 ffffea0006800000 1703936
53 ffff88021e5eb6a0 ffffea0000000000 ffffea0006a00000 1736704
54 ffff88021e5eb6c0 ffffea0000000000 ffffea0006c00000 1769472
55 ffff88021e5eb6e0 ffffea0000000000 ffffea0006e00000 1802240
56 ffff88021e5eb700 ffffea0000000000 ffffea0007000000 1835008
57 ffff88021e5eb720 ffffea0000000000 ffffea0007200000 1867776
58 ffff88021e5eb740 ffffea0000000000 ffffea0007400000 1900544
59 ffff88021e5eb760 ffffea0000000000 ffffea0007600000 1933312
60 ffff88021e5eb780 ffffea0000000000 ffffea0007800000 1966080
61 ffff88021e5eb7a0 ffffea0000000000 ffffea0007a00000 1998848
62 ffff88021e5eb7c0 ffffea0000000000 ffffea0007c00000 2031616
63 ffff88021e5eb7e0 ffffea0000000000 ffffea0007e00000 2064384
64 ffff88021e5eb800 ffffea0000000000 ffffea0008000000 2097152
65 ffff88021e5eb820 ffffea0000000000 ffffea0008200000 2129920
66 ffff88021e5eb840 ffffea0000000000 ffffea0008400000 2162688
67 ffff88021e5eb860 ffffea0000000000 ffffea0008600000 2195456
crash>
Is your address contained with one of your regions?
Dave
1:02 p.m.
On 12/17/12 09:03, Dave Anderson wrote:
> That also removed the mysterious problem of having a duplicate
error
> message show up on the console.
OK, if that works for you...
In the sense of being able to do my day job, yes.
In the sense of aesthetics, not so much.
Right -- I would never expect error() to be called while inside
an open_tmpfile() operation. Normally the behind-the-scenes data
is parsed, and if anything is to be displayed while open_tmpfile()
is still in play, it would be fprint()'ed using pc->saved_fp.
I think the aesthetically pleasing solution is an "i_am_playing_with_tmpfile()"
call that says it isn't closed and crash functions shouldn't be using it.
Plus a parallel "i_am_done_with_tmpfile()" that gets implied by
"close_tmpfile()".
I can supply a patch, if you like. Probably with less verbose
function names.
>> crash> gdb set $tp = (struct cfs_trace_page
*)0xffff8807fb590740
>> crash> p $tp->page
>> $7 = (cfs_page_t *) 0xffffea001bb1d1e8
>> crash> p *$tp->page
>> $8 = {
>> flags = 144115188075855872,
>> _count = {
>> counter = 1
>> },
>> [...]
>> lru = {
>> next = 0xdead000000100100,
>> prev = 0xdead000000200200
>> }
>> }
>> crash> kmem 0xffffea001bb1d1e8
>> kmem: WARNING: cannot find mem_map page for address: ffffea001bb1d1e8
>> 879b1d1e8: kernel virtual address not found in mem map
>
> So I can print out the page_t structure (renamed as cfs_page_t in Lustre)
> at address 0xffff8807fb590740, but when I try to get kmem information about
> it, it cannot find the page. What am I missing?
>
> Thanks for hints/pointers! Regards, Bruce
I'm not sure, other than it doesn't seem to be able to find ffffea001bb1d1e8
I was able to figure that out. I also printed out the "kmem -v" table and
sorted the result. The result with "kmem -n"
[...]
66 ffff88087fffa420 ffffea0000000000 ffffea0007380000 2162688
67 ffff88087fffa430 ffffea0000000000 ffffea0007540000 2195456
132608 ffff88083c9bdb98 ffff88083c9bdd98 ffff8840e49bdd98 4345298944
132609 ffff88083c9bdba8 ffff88083c9796c0 ffff8840e4b396c0 4345331712
;...]
viz. it ain't there. Which is quite interesting, because if the lustre
cluster file system structure "cfs_trace_data" actually pointed off into
unmapped memory, it would have fallen over long, long before the point
where it did fall over.
>>> ffff8807fb590740
>>> struct cfs_trace_page {
>>> page = 0xffffea001bb1d1e8, <<<<==== address in question
>>> linkage = {
>>> next = 0xffff8807fb590ee8,
>>> prev = 0xffff880824e3d810
>>> },
It seems like it is both there and not there, so I am misunderstanding something.
For sure.
crash> whatis $tp->page
cfs_page_t *
crash> p $tp->page
$8 = (cfs_page_t *) 0xffffea001bb1d1e8
crash> p *$tp->page
$9 = {
flags = 0x200000000000000,
_count = {
counter = 1
},
{
_mapcount = {
counter = -1
},
{
inuse = 65535,
objects = 65535
}
},
{
{
private = 0,
mapping = 0x0
},
ptl = {
{
rlock = {
raw_lock = {
slock = 0
}
}
}
},
slab = 0x0,
first_page = 0x0
},
{
index = 0,
freelist = 0x0,
pfmemalloc = false
},
lru = {
next = 0xdead000000100100,
prev = 0xdead000000200200
}
}
So clearly, I am able to read cfs_page_t data at that address.
But I cannot get the mappings for it, and neither can my lustre
extensions. (We are trying to extract trace data from an in-kernel
circular buffer that is 5,100 pages in size (20 Meg).)
Thank you any help at all! Regards, Bruce
1:23 p.m.
----- Original Message -----
On 12/17/12 09:03, Dave Anderson wrote:
>> That also removed the mysterious problem of having a duplicate
>> error
>> message show up on the console.
>
> OK, if that works for you...
In the sense of being able to do my day job, yes.
In the sense of aesthetics, not so much.
> Right -- I would never expect error() to be called while inside
> an open_tmpfile() operation. Normally the behind-the-scenes data
> is parsed, and if anything is to be displayed while open_tmpfile()
> is still in play, it would be fprint()'ed using pc->saved_fp.
I think the aesthetically pleasing solution is an
"i_am_playing_with_tmpfile()"
call that says it isn't closed and crash functions shouldn't be using it.
Plus a parallel "i_am_done_with_tmpfile()" that gets implied by
"close_tmpfile()".
I can supply a patch, if you like. Probably with less verbose function names.
If pc->tmpfile is non-NULL, then open_tmpfile() is in use. What would be
the purpose of the extra functions?
>>> crash> gdb set $tp = (struct cfs_trace_page *)0xffff8807fb590740
>>> crash> p $tp->page
>>> $7 = (cfs_page_t *) 0xffffea001bb1d1e8
>>> crash> p *$tp->page
>>> $8 = {
>>> flags = 144115188075855872,
>>> _count = {
>>> counter = 1
>>> },
>>> [...]
>>> lru = {
>>> next = 0xdead000000100100,
>>> prev = 0xdead000000200200
>>> }
>>> }
>>> crash> kmem 0xffffea001bb1d1e8
>>> kmem: WARNING: cannot find mem_map page for address:
>>> ffffea001bb1d1e8
>>> 879b1d1e8: kernel virtual address not found in mem map
>>
>> So I can print out the page_t structure (renamed as cfs_page_t in
>> Lustre)
>> at address 0xffff8807fb590740, but when I try to get kmem
>> information about
>> it, it cannot find the page. What am I missing?
>>
>> Thanks for hints/pointers! Regards, Bruce
>
> I'm not sure, other than it doesn't seem to be able to find
> ffffea001bb1d1e8
I was able to figure that out. I also printed out the "kmem -v" table and
sorted the result. The result with "kmem -n"
[...]
66 ffff88087fffa420 ffffea0000000000 ffffea0007380000 2162688
67 ffff88087fffa430 ffffea0000000000 ffffea0007540000 2195456
132608 ffff88083c9bdb98 ffff88083c9bdd98 ffff8840e49bdd98 4345298944
132609 ffff88083c9bdba8 ffff88083c9796c0 ffff8840e4b396c0 4345331712
;...]
viz. it ain't there. Which is quite interesting, because if the lustre
cluster file system structure "cfs_trace_data" actually pointed off into
unmapped memory, it would have fallen over long, long before the point
where it did fall over.
I don't see the vmemmap range in the "kmem -v" output. It is mapped
kernel memory, but AFAIK it's not kept in the kernel's "vmlist" list.
Do you see that range in your "kmem -v" output?
>>>> ffff8807fb590740
>>>> struct cfs_trace_page {
>>>> page = 0xffffea001bb1d1e8, <<<<==== address in question
>>>> linkage = {
>>>> next = 0xffff8807fb590ee8,
>>>> prev = 0xffff880824e3d810
>>>> },
It seems like it is both there and not there, so I am misunderstanding something.
For sure.
> crash> whatis $tp->page
> cfs_page_t *
> crash> p $tp->page
> $8 = (cfs_page_t *) 0xffffea001bb1d1e8
> crash> p *$tp->page
> $9 = {
> flags = 0x200000000000000,
> _count = {
> counter = 1
> },
> {
> _mapcount = {
> counter = -1
> },
> {
> inuse = 65535,
> objects = 65535
> }
> },
> {
> {
> private = 0,
> mapping = 0x0
> },
> ptl = {
> {
> rlock = {
> raw_lock = {
> slock = 0
> }
> }
> }
> },
> slab = 0x0,
> first_page = 0x0
> },
> {
> index = 0,
> freelist = 0x0,
> pfmemalloc = false
> },
> lru = {
> next = 0xdead000000100100,
> prev = 0xdead000000200200
> }
> }
So clearly, I am able to read cfs_page_t data at that address.
But I cannot get the mappings for it, and neither can my lustre
extensions. (We are trying to extract trace data from an in-kernel
circular buffer that is 5,100 pages in size (20 Meg).)
Thank you any help at all! Regards, Bruce
OK so you say you cannot get the mappings for it, but what
does "vtop 0xffffea001bb1d1e8" show?
Dave
1:44 p.m.
----- Original Message -----
OK so you say you cannot get the mappings for it, but what
does "vtop 0xffffea001bb1d1e8" show?
Sorry -- that's irrelevant. You want to access the physical
memory that the odd vmemmap page address references (not the
physical page behind the page structure itself).
Dave
5:30 p.m.
Hi Dave,
On 12/17/12 11:23, Dave Anderson wrote:
>> Right -- I would never expect error() to be called while
inside
>> an open_tmpfile() operation. Normally the behind-the-scenes data
>> is parsed, and if anything is to be displayed while open_tmpfile()
>> is still in play, it would be fprint()'ed using pc->saved_fp.
>
> I think the aesthetically pleasing solution is an
"i_am_playing_with_tmpfile()"
> call that says it isn't closed and crash functions shouldn't be using it.
> Plus a parallel "i_am_done_with_tmpfile()" that gets implied by
"close_tmpfile()".
> I can supply a patch, if you like. Probably with less verbose function names.
If pc->tmpfile is non-NULL, then open_tmpfile() is in use. What would be
the purpose of the extra functions?
It would be to allow the client code that is processing that temp file to emit
warning/info messages without disrupting the reading of that file pointer.
To me, that doesn't seem unreasonable. You run some code that emits output
to a temp file and you reprocess those data. You surely do not want such
messages showing up in the file you are re-processing. And you cannot
call close_tmpfile() because it calls ftruncate().
So, what is your recommendation for how to reprocess diverted output
wherein you might occasionally want to say something during that reprocessing?
Three solutions come to mind:
1. Juggle file pointers before and after the __error() function call (please say,
"No.")
2. Create my own temporary file and fiddle the global "fp" and "pc"
state so it
gets used while I am gathering data and crash code doesn't know about it later.
(I insist the answer must be, "No." because there is too much fiddling with
intricate crash state.)
3. These two functions that I am suggesting:
void
resume_tmpfile(void)
{
int ret ATTRIBUTE_UNUSED;
if (pc->tmpfile)
error(FATAL, "recursive temporary file usage\n");
if (!pc->tmp_fp)
error(FATAL, "temporary file not ready\n");
rewind(pc->tmp_fp);
pc->tmpfile = pc->tmp_fp;
pc->saved_fp = fp;
fp = pc->tmpfile;
}
void
sequester_tmpfile(void)
{
int ret ATTRIBUTE_UNUSED;
if (pc->tmpfile) {
fflush(pc->tmpfile);
rewind(pc->tmpfile);
pc->tmpfile = NULL;
fp = pc->saved_fp;
} else
error(FATAL, "trying to sequester an unopened temporary file\n");
}
I sequester the file after doing the data gathering and resume it
after I am done reprocessing it. It might be worth putting in a little jig
to ensure that open/close_tmpfile work reasonably, too. (I would guess
that either would cancel the sequestration.)
>> I'm not sure, other than it doesn't seem to be able
to find
>> ffffea001bb1d1e8
>
> I was able to figure that out. I also printed out the "kmem -v" table and
> sorted the result. The result with "kmem -n"
>
> [...]
> 66 ffff88087fffa420 ffffea0000000000 ffffea0007380000 2162688
> 67 ffff88087fffa430 ffffea0000000000 ffffea0007540000 2195456
> 132608 ffff88083c9bdb98 ffff88083c9bdd98 ffff8840e49bdd98 4345298944
> 132609 ffff88083c9bdba8 ffff88083c9796c0 ffff8840e4b396c0 4345331712
> ;...]
>
> viz. it ain't there. Which is quite interesting, because if the lustre
> cluster file system structure "cfs_trace_data" actually pointed off into
> unmapped memory, it would have fallen over long, long before the point
> where it did fall over.
I don't see the vmemmap range in the "kmem -v" output. It is mapped
kernel memory, but AFAIK it's not kept in the kernel's "vmlist" list.
Do you see that range in your "kmem -v" output?
Also no. "kmem -v" and "kmem -n" both show the same memory mappings
(as best as _my_ memory serves, that is. For certain, neither has a mapping
for 0xffffea001bb1d1e8.)
OK so you say you cannot get the mappings for it, but what
does "vtop 0xffffea001bb1d1e8" show?
This:
crash> vtop 0xffffea001bb1d1e8
VIRTUAL PHYSICAL
ffffea001bb1d1e8 879b1d1e8
PML4 DIRECTORY: ffffffff817e7000
PAGE DIRECTORY: 87fdf7067
PUD: 87fdf7000 => 87fdf6067
PMD: 87fdf66e8 => 8000000879a001e3
PAGE: 879a00000 (2MB)
PTE PHYSICAL FLAGS
8000000879a001e3 879a00000 (PRESENT|RW|ACCESSED|DIRTY|PSE|GLOBAL|NX)
But given:
Sorry -- that's irrelevant. You want to access the physical
memory that the odd vmemmap page address references (not the
physical page behind the page structure itself).
Exactly right. I need to be able to see the binary bits for that page so I can
pull them in and write them back out to a file of just those bits. From there,
we'll be formatting a text file showing the lustre trace log.
Thank you so much! Regards, Bruce
8:52 a.m.
----- Original Message -----
Hi Dave,
On 12/17/12 11:23, Dave Anderson wrote:
>>> Right -- I would never expect error() to be called while inside
>>> an open_tmpfile() operation. Normally the behind-the-scenes data
>>> is parsed, and if anything is to be displayed while open_tmpfile()
>>> is still in play, it would be fprint()'ed using pc->saved_fp.
>>
>> I think the aesthetically pleasing solution is an
"i_am_playing_with_tmpfile()"
>> call that says it isn't closed and crash functions shouldn't be using
it.
>> Plus a parallel "i_am_done_with_tmpfile()" that gets implied by
"close_tmpfile()".
>> I can supply a patch, if you like. Probably with less verbose function names.
>
> If pc->tmpfile is non-NULL, then open_tmpfile() is in use. What would be
> the purpose of the extra functions?
It would be to allow the client code that is processing that temp file to emit
warning/info messages without disrupting the reading of that file pointer.
To me, that doesn't seem unreasonable. You run some code that emits output
to a temp file and you reprocess those data. You surely do not want such
messages showing up in the file you are re-processing. And you cannot
call close_tmpfile() because it calls ftruncate().
So, what is your recommendation for how to reprocess diverted output
wherein you might occasionally want to say something during that
reprocessing?
Three solutions come to mind:
1. Juggle file pointers before and after the __error() function call
(please say, "No.")
No.
2. Create my own temporary file and fiddle the global "fp"
and "pc" state so it
gets used while I am gathering data and crash code doesn't know about it later.
(I insist the answer must be, "No." because there is too much fiddling
with
intricate crash state.)
No.
3. These two functions that I am suggesting:
void
resume_tmpfile(void)
{
int ret ATTRIBUTE_UNUSED;
if (pc->tmpfile)
error(FATAL, "recursive temporary file usage\n");
if (!pc->tmp_fp)
error(FATAL, "temporary file not ready\n");
rewind(pc->tmp_fp);
pc->tmpfile = pc->tmp_fp;
pc->saved_fp = fp;
fp = pc->tmpfile;
}
void
sequester_tmpfile(void)
{
int ret ATTRIBUTE_UNUSED;
if (pc->tmpfile) {
fflush(pc->tmpfile);
rewind(pc->tmpfile);
pc->tmpfile = NULL;
fp = pc->saved_fp;
} else
error(FATAL, "trying to sequester an unopened temporary file\n");
}
And no...
When open_tmpfile() is in play and you want to print something, you can
always use fprintf(pc->saved_fp, ...) as is done everywhere now.
That being said, if you truly desire to use error() during an open_tmpfile()
operation, then that anomoly should be handled in the error() function.
So, if error() is called during open_tmpfile(), i.e., then the message should
be displayed as it is done now, which is to pc->stdpipe (i.e., the current
more/less scroller if it is in effect), or to stdout if not:
if (pc->stdpipe) {
fprintf(pc->stdpipe, "%s%s%s %s%s",
new_line ? "\n" : "",
type == CONT ? spacebuf : pc->curcmd,
type == CONT ? " " : ":",
type == WARNING ? "WARNING: " :
type == NOTE ? "NOTE: " : "",
buf);
fflush(pc->stdpipe);
} else {
fprintf(stdout, "%s%s%s %s%s",
new_line || end_of_line ? "\n" : "",
type == WARNING ? "WARNING" :
type == NOTE ? "NOTE" :
type == CONT ? spacebuf : pc->curcmd,
type == CONT ? " " : ":",
buf, end_of_line ? "\n" : "");
fflush(stdout);
}
and if the output is currently being redirected to a file or to a pipe,
then it is also issued to those end-points here:
if ((fp != stdout) && (fp != pc->stdpipe)) {
fprintf(fp, "%s%s%s %s", new_line ? "\n" :
"",
type == WARNING ? "WARNING" :
type == NOTE ? "NOTE" :
type == CONT ? spacebuf : pc->curcmd,
type == CONT ? " " : ":",
buf);
fflush(fp);
}
It's that "duplication" above that you're seeing.
And I am simply suggesting that the if statement above should be:
if ((fp != stdout) && (fp != pc->stdpipe) && (fp !=
pc->tmpfile)) {
because you obviously don't want the message intermingled with your open_tmpfile()
output.
I sequester the file after doing the data gathering and resume it
after I am done reprocessing it. It might be worth putting in a little jig
to ensure that open/close_tmpfile work reasonably, too. (I would guess
that either would cancel the sequestration.)
>>> I'm not sure, other than it doesn't seem to be able to find
ffffea001bb1d1e8
>>
>> I was able to figure that out. I also printed out the "kmem -v" table
and
>> sorted the result. The result with "kmem -n"
>>
>> [...]
>> 66 ffff88087fffa420 ffffea0000000000 ffffea0007380000 2162688
>> 67 ffff88087fffa430 ffffea0000000000 ffffea0007540000 2195456
>> 132608 ffff88083c9bdb98 ffff88083c9bdd98 ffff8840e49bdd98 4345298944
>> 132609 ffff88083c9bdba8 ffff88083c9796c0 ffff8840e4b396c0 4345331712
>> ;...]
>>
>> viz. it ain't there. Which is quite interesting, because if the lustre
>> cluster file system structure "cfs_trace_data" actually pointed off
into
>> unmapped memory, it would have fallen over long, long before the point
>> where it did fall over.
>
> I don't see the vmemmap range in the "kmem -v" output. It is mapped
> kernel memory, but AFAIK it's not kept in the kernel's "vmlist"
list.
> Do you see that range in your "kmem -v" output?
Also no. "kmem -v" and "kmem -n" both show the same memory mappings
(as best as _my_ memory serves, that is. For certain, neither has a mapping
for 0xffffea001bb1d1e8.)
> OK so you say you cannot get the mappings for it, but what
> does "vtop 0xffffea001bb1d1e8" show?
This:
> crash> vtop 0xffffea001bb1d1e8
> VIRTUAL PHYSICAL
> ffffea001bb1d1e8 879b1d1e8
>
> PML4 DIRECTORY: ffffffff817e7000
> PAGE DIRECTORY: 87fdf7067
> PUD: 87fdf7000 => 87fdf6067
> PMD: 87fdf66e8 => 8000000879a001e3
> PAGE: 879a00000 (2MB)
>
> PTE PHYSICAL FLAGS
> 8000000879a001e3 879a00000 (PRESENT|RW|ACCESSED|DIRTY|PSE|GLOBAL|NX)
But given:
> Sorry -- that's irrelevant. You want to access the physical
> memory that the odd vmemmap page address references (not the
> physical page behind the page structure itself).
Exactly right. I need to be able to see the binary bits for that page so I can
pull them in and write them back out to a file of just those bits. From there,
we'll be formatting a text file showing the lustre trace log.
Thank you so much! Regards, Bruce
Right... seems like it should be such a simple thing to do... :-(
I don't understand what's going on, but I'm presuming that even if the
vmemmap-type address doesn't fit into the "advertised" vmemmap range,
that the kernel's __page_to_pfn() macro should still work to get the
pfn represented by the page:
#elif defined(CONFIG_SPARSEMEM)
/*
* Note: section's mem_map is encorded to reflect its start_pfn.
* section[i].section_mem_map == mem_map's address - start_pfn;
*/
#define __page_to_pfn(pg) \
({ const struct page *__pg = (pg); \
int __sec = page_to_section(__pg); \
(unsigned long)(__pg - __section_mem_map_addr(__nr_to_section(__sec))); \
})
Maybe you could play around with emulating that macro w/crash, and see what
comes up?
Dave
9:01 a.m.
Good morning, Dave,
On Tue, Dec 18, 2012 at 6:52 AM, Dave Anderson <anderson(a)redhat.com> wrote:
then it is also issued to those end-points here:
if ((fp != stdout) && (fp != pc->stdpipe)) {
fprintf(fp, "%s%s%s %s", new_line ? "\n" :
"",
[...]
}
It's that "duplication" above that you're seeing.
And I am simply suggesting that the if statement above should be:
if ((fp != stdout) && (fp != pc->stdpipe) && (fp !=
pc->tmpfile)) {
That works best. I had dismissed that because it was a behavioral change.
Thank you.
> Also no. "kmem -v" and "kmem -n" both show
the same memory mappings
> (as best as _my_ memory serves, that is. For certain, neither has a mapping
> for 0xffffea001bb1d1e8.)
>
> > OK so you say you cannot get the mappings for it, but what
> > does "vtop 0xffffea001bb1d1e8" show?
>
> This:
>
> > crash> vtop 0xffffea001bb1d1e8
> > VIRTUAL PHYSICAL
> > ffffea001bb1d1e8 879b1d1e8
> >
> > PML4 DIRECTORY: ffffffff817e7000
> > PAGE DIRECTORY: 87fdf7067
> > PUD: 87fdf7000 => 87fdf6067
> > PMD: 87fdf66e8 => 8000000879a001e3
> > PAGE: 879a00000 (2MB)
> >
> > PTE PHYSICAL FLAGS
> > 8000000879a001e3 879a00000 (PRESENT|RW|ACCESSED|DIRTY|PSE|GLOBAL|NX)
>
> But given:
>
> > Sorry -- that's irrelevant. You want to access the physical
> > memory that the odd vmemmap page address references (not the
> > physical page behind the page structure itself).
>
> Exactly right. I need to be able to see the binary bits for that page so I can
> pull them in and write them back out to a file of just those bits. From there,
> we'll be formatting a text file showing the lustre trace log.
>
> Thank you so much! Regards, Bruce
Right... seems like it should be such a simple thing to do... :-(
I don't understand what's going on, but I'm presuming that even if the
vmemmap-type address doesn't fit into the "advertised" vmemmap range,
that the kernel's __page_to_pfn() macro should still work to get the
pfn represented by the page:
#elif defined(CONFIG_SPARSEMEM)
/*
* Note: section's mem_map is encorded to reflect its start_pfn.
* section[i].section_mem_map == mem_map's address - start_pfn;
*/
#define __page_to_pfn(pg) \
({ const struct page *__pg = (pg); \
int __sec = page_to_section(__pg); \
(unsigned long)(__pg - __section_mem_map_addr(__nr_to_section(__sec))); \
})
Maybe you could play around with emulating that macro w/crash, and see what
comes up?
Will do, after a couple of con calls. Thank you! Regards, Bruce
3:34 p.m.
Hi Dave,
On Tue, Dec 18, 2012 at 6:52 AM, Dave Anderson <anderson(a)redhat.com> wrote:
#define __page_to_pfn(pg) \
({ const struct page *__pg = (pg); \
int __sec = page_to_section(__pg); \
(unsigned long)(__pg - __section_mem_map_addr(__nr_to_section(__sec))); \
})
Maybe you could play around with emulating that macro w/crash, and see what
comes up?
Hey, that was sure fun. :(
Since "struct page" is another name for "cfs_page_t":
crash> p $tp->page->flags
$9 = 0x200000000000000
"page to section" is a 43 bit right shift:
crash> p ($tp->page->flags >> 43)
$10 = 0x4000
static inline struct mem_section *__nr_to_section(unsigned long nr)
{
if (!mem_section[SECTION_NR_TO_ROOT(nr)])
return NULL;
return &mem_section[SECTION_NR_TO_ROOT(nr)][nr & SECTION_ROOT_MASK];
}
There are 16 maps per mem section, so a 4 bit shift and mask yield:
return &mem_section[0x400][0]
crash> p mem_section[0x400]
$13 = (struct mem_section *) 0xffffffff81a26658
crash> p mem_section[0x400][0x0]
$14 = {
section_mem_map = 0xffffffff81a26630,
pageblock_flags = 0xffffffff81a26680
}
So the result of __page_to_pfn is the difference between
crash> p $pg
$24 = (cfs_page_t *) 0xffffea001bb1d1e8
and the return result of
__section_mem_map_addr((struct mem_section *) 0xffffffff81a26658)
static inline struct page *__section_mem_map_addr(struct mem_section
*section)
{
unsigned long map = section->section_mem_map;
map &= SECTION_MAP_MASK;
return (struct page *)map;
}
SECTION_MAP_MASK maps off the low order 3 bits, yielding:
(cfs_page_t *) 0xffffffff81a26630
crash> gdb set $mpg = (cfs_page_t *)0xffffffff81a26630
crash> p ($pg - $mpg)
$32 = 0xffffff9b707721ed
Now what do I do with that number? :) It is the difference between
the numeric values of $pg and $mpg, divided by the size of "struct page"
(which is 0x38 bytes) and sign extended.
4:12 p.m.
----- Original Message -----
Hi Dave,
On Tue, Dec 18, 2012 at 6:52 AM, Dave Anderson <anderson(a)redhat.com>
wrote:
> #define __page_to_pfn(pg) \
> ({ const struct page *__pg = (pg); \
> int __sec = page_to_section(__pg); \
> (unsigned long)(__pg -
> __section_mem_map_addr(__nr_to_section(__sec))); \
> })
>
> Maybe you could play around with emulating that macro w/crash, and
> see what comes up?
Hey, that was sure fun. :(
It's a nightmare, isn't it?
Since "struct page" is another name for
"cfs_page_t":
> crash> p $tp->page->flags
> $9 = 0x200000000000000
"page to section" is a 43 bit right shift:
> crash> p ($tp->page->flags >> 43)
> $10 = 0x4000
> static inline struct mem_section *__nr_to_section(unsigned long nr)
> {
> if (!mem_section[SECTION_NR_TO_ROOT(nr)])
> return NULL;
> return &mem_section[SECTION_NR_TO_ROOT(nr)][nr &
> SECTION_ROOT_MASK];
> }
There are 16 maps per mem section, so a 4 bit shift and mask yield:
return &mem_section[0x400][0]
> crash> p mem_section[0x400]
> $13 = (struct mem_section *) 0xffffffff81a26658
> crash> p mem_section[0x400][0x0]
> $14 = {
> section_mem_map = 0xffffffff81a26630,
> pageblock_flags = 0xffffffff81a26680
> }
So the result of __page_to_pfn is the difference between
> crash> p $pg
> $24 = (cfs_page_t *) 0xffffea001bb1d1e8
and the return result of
__section_mem_map_addr((struct mem_section *) 0xffffffff81a26658)
> static inline struct page *__section_mem_map_addr(struct
> mem_section *section)
> {
> unsigned long map = section->section_mem_map;
> map &= SECTION_MAP_MASK;
> return (struct page *)map;
> }
SECTION_MAP_MASK maps off the low order 3 bits, yielding:
(cfs_page_t *) 0xffffffff81a26630
> crash> gdb set $mpg = (cfs_page_t *)0xffffffff81a26630
> crash> p ($pg - $mpg)
> $32 = 0xffffff9b707721ed
Now what do I do with that number? :) It is the difference between
the numeric values of $pg and $mpg, divided by the size of "struct page"
(which is 0x38 bytes) and sign extended.
The $mpg value would also have to resolve to a vmemmap address (ffffea00...),
which would have to be smaller in value than your $pg address of 0xffffea001bb1d1e8
in order for it to make sense.
So, you show that you stripped off 3 bits from something (not shown) in order
to get the $mpg of 0xffffffff81a26630 -- which should be derived from the
section_mem_map member of the mem_section at address 0xffffffff81a26658.
What did the mem_section structure look like? In other words:
crash> struct mem_section 0xffffffff81a26658
struct mem_section {
section_mem_map = <vmemmap-addr-you-need-to-strip-3-bits-from>
...
Here's an example mem_section:
crash> struct mem_section ffff88021e5eb000 -x
struct mem_section {
section_mem_map = 0xffffea0000000003,
pageblock_flags = 0xffff88021e1eaa00,
page_cgroup = 0xffff880215880000,
pad = 0x0
}
where the section_mem_map field contains a vmemmap address plus a few
flag bits in the lower byte.
Dave
4:52 p.m.
Hi Dave,
On Tue, Dec 18, 2012 at 2:12 PM, Dave Anderson <anderson(a)redhat.com> wrote:
> > crash> p mem_section[0x400]
> > $13 = (struct mem_section *) 0xffffffff81a26658
> > crash> p mem_section[0x400][0x0]
> > $14 = {
> > section_mem_map = 0xffffffff81a26630,
> > pageblock_flags = 0xffffffff81a26680
> > }
I think I did show the memory section. But it is understandable if
you missed it.
I plodded my way through the macros to figure out that the flag value of
crash> p $tp->page->flags
$9 = 0x200000000000000
ultimately maps to indexes of 0x400 and 0.
(47 bit shift for first index)
There are only two fields because of kernel build options.
crash> struct mem_section ffff88021e5eb000 -x
struct mem_section {
section_mem_map = 0xffffea0000000003,
pageblock_flags = 0xffff88021e1eaa00,
Since your example has upper 4 bytes of 0xffffea00, similar to
the (cfs_page_t *) 0xffffea001bb1d1e8 page address I am trying to understand,
maybe the 0xffffffff81a26630 address is bogus?
page_cgroup = 0xffff880215880000,
pad = 0x0
7:59 a.m.
----- Original Message -----
Hi Dave,
On Tue, Dec 18, 2012 at 2:12 PM, Dave Anderson <anderson(a)redhat.com>
wrote:
>> > crash> p mem_section[0x400]
>> > $13 = (struct mem_section *) 0xffffffff81a26658
>> > crash> p mem_section[0x400][0x0]
>> > $14 = {
>> > section_mem_map = 0xffffffff81a26630,
>> > pageblock_flags = 0xffffffff81a26680
>> > }
I think I did show the memory section. But it is understandable if you missed it.
I plodded my way through the macros to figure out that the flag value
of
> crash> p $tp->page->flags
> $9 = 0x200000000000000
ultimately maps to indexes of 0x400 and 0.
(47 bit shift for first index)
There are only two fields because of kernel build options.
> crash> struct mem_section ffff88021e5eb000 -x
> struct mem_section {
> section_mem_map = 0xffffea0000000003,
> pageblock_flags = 0xffff88021e1eaa00,
Since your example has upper 4 bytes of 0xffffea00, similar to
the (cfs_page_t *) 0xffffea001bb1d1e8 page address I am trying to
understand, maybe the 0xffffffff81a26630 address is bogus?
> page_cgroup = 0xffff880215880000,
> pad = 0x0
Right, mem_section->section_mem_map has to contain a vmemmap address for
this to make any sense.
You might try the obvious, which is to do what is done for the advertised
vmemmap pages.
Get the page structure offset from the beginning of the vmemmap range:
crash> eval 0xffffea001bb1d1e8 - ffffea0000000000
hexadecimal: 1bb1d1e8
decimal: 464638440
octal: 3354350750
binary: 0000000000000000000000000000000000011011101100011101000111101000
crash>
Get the page number (pfn) by dividing that by the sizeof a page (64 on my system):
crash> eval 1bb1d1e8 / 64
hexadecimal: 6ec747
decimal: 7259975
octal: 33543507
binary: 0000000000000000000000000000000000000000011011101100011101000111
crash>
Multiply the resultant pfn times the page size to get the physical address:
crash> eval 6ec747 * 4k
hexadecimal: 6ec747000 (29039900KB)
decimal: 29736857600
octal: 335435070000
binary: 0000000000000000000000000000011011101100011101000111000000000000
crash>
And then "rd -p" the physical address to see if (1) it exists, and (2) you can
somehow verify that the data is correct -- if that's even possible.
Dave
4212
days inactive
4217
days old
devel@lists.crash-utility.osci.io
11 comments
2 participants
participants (2)
-
Bruce Korb -
Dave Anderson