Thanks for the info Dave - maybe this belongs in the appropriate Xen list, but this code seems fundamentally broken to me for a couple of reasons:

1. I'm pretty sure that the address passed into read_mem is a physical address NOT a machine-address and yet the code in is_local_lowmem seems to treat this as an mfn (at least, I would assume that mfn_to_local_pfn would be doing so)
2. That aside, it just seems wrong to me to break the operation of /dev/mem so that the one thing it _cant_ return is low memory for the guest!

Time for a Xen bug I think!

Simon

You're preaching to the choir, man...

And I believe you're right -- it does seem to be a bug rather than
a feature, unless whatever user processes that utilize /dev/mem are
supposed to have some kind of pre-knowledge of the machine addresses
behind the pseudo-physical address?  How can that be?
 

/*
 * Does @address reside within a non-highmem page that is local to this virtual
 * machine (i.e., not an I/O page, nor a memory page belonging to another VM).
 * See the comment that accompanies mfn_to_local_pfn() in page.h to understand
 * why this works.
 */
static inline int is_local_lowmem(unsigned long address)
{
        extern unsigned long max_low_pfn;
        return (mfn_to_local_pfn(address >> PAGE_SHIFT) < max_low_pfn);
}

I've always been under the impression that addresses passed into
/dev/mem are by definition be pseudo-physical addresses; and that the
PTE entry used to map them would be subsequently over-written by the
hypervisor for kernels with writable page tables.

But in any case, early on I moved on from /dev/mem to using /dev/crash
like for RHEL4, and never really noticed what you've discovered.

Dave