Hi
MAP_LOCK flag to mmap() on LMDB readers should populate page cache from the file being mapped and preclude those pages from eviction. I was wondering if that was ever tested, especially with frequent writes to the underlying file. And is there a standard way (a compile option) to include that flag without modifying source code directly? I see quite a few major faults with perf stat, and I am looking if they could be avoided.
I am new to LMDB and I guess this discussion may have taken place before, if you could point me out to the right place..
Thank you.
Vitaly
Vitaly Zuevsky wrote:
Hi
MAP_LOCK flag to mmap() on LMDB readers should populate page cache from the file being mapped and preclude those pages from eviction. I was wondering if that was ever tested, especially with frequent writes to the underlying file. And is there a standard way (a compile option) to include that flag without modifying source code directly? I see quite a few major faults with perf stat, and I am looking if they could be avoided.
I am new to LMDB and I guess this discussion may have taken place before, if you could point me out to the right place..
No. In general it's a bad idea to interfere with the OS's paging decisions. In particular, on a machine with plenty of RAM, pageouts aren't an issue anyway. On a system where memory is tight, page faults will be unavoidable, so the question is what other memory are you going to lose if you force the LMDB pages to stay locked in?
Thank you.
Vitaly
Vitaly Zuevsky wrote:
Hi
MAP_LOCK flag to mmap() on LMDB readers should populate page cache from the file being mapped and preclude those pages from eviction. I was wondering if that was ever tested, especially with frequent writes to the underlying file. And is there a standard way (a compile option) to include that flag without modifying source code directly? I see quite a few major faults with perf stat, and I am looking if they could be
avoided.
I am new to LMDB and I guess this discussion may have taken place before, if you could point me out to the right place..
No. In general it's a bad idea to interfere with the OS's paging decisions. In particular, on a machine with plenty of RAM, pageouts aren't an issue anyway. On a system where memory is tight, page faults will be unavoidable, so the question is what other memory are you going to lose if you force the LMDB pages to stay locked in?
Memory pressure isn't my problem. I see 2-15 major faults a second in the event loop of the reader thread, which can effectively stall the loop for anything between 1-400ms (I measure iowait jiffies of the pid). I understand the faults result from the writer thread, actively modifying the file mmaped by the reader. I assume writes to the file invalidate corresponding pages of the reader, and those invalid pages only get updated (via major faults) on first access by the reader. If not locking, perhaps madvise/willneed combination could help, what you think?
Vitaly Zuevsky wrote:
Memory pressure isn't my problem. I see 2-15 major faults a second in the event loop of the reader thread, which can effectively stall the loop for anything between 1-400ms (I measure iowait jiffies of the pid). I understand the faults result from the writer thread, actively modifying the file mmaped by the reader. I assume writes to the file invalidate corresponding pages of the reader, and those invalid pages only get updated (via major faults) on first access by the reader. If not locking, perhaps madvise/willneed combination could help, what you think?
I think if this is your concern you should just use WRITEMAP and forget about it.
Vitaly Zuevsky wrote: Memory pressure isn't my problem. I see 2-15 major faults a second in the event loop of the reader thread, which can effectively stall the loop for anything between 1-400ms (I measure iowait jiffies of the pid). I understand the faults result from the writer thread, actively modifying the file mmaped by the reader. I assume writes to the file invalidate corresponding pages of the reader, and those invalid pages only get updated (via major faults) on first access by the reader. If not locking, perhaps madvise/willneed combination could help, what you think?
I think if this is your concern you should just use WRITEMAP and forget about it.
-- -- Howard Chu
Thanks for pointing me out to writemap option https://lmdb.readthedocs.io/en/release/#lmdb.Environment Re ACID implications it will have - metasync=True, sync=True, map_async=False, mode=493, create=True, readahead=True, writemap=True will this ensure integrity on disk after every write? What sequence of events (contrasted with writemap=False) could lead to data corruption here?
Vitaly Zuevsky wrote:
Vitaly Zuevsky wrote: Memory pressure isn't my problem. I see 2-15 major faults a second in the event loop of the reader thread, which can effectively stall the loop for anything between 1-400ms (I measure iowait jiffies of the pid). I understand the faults result from the writer thread, actively modifying the file mmaped by the reader. I assume writes to the file invalidate corresponding pages of the reader, and those invalid pages only get updated (via major faults) on first access by the reader. If not locking, perhaps madvise/willneed combination could help, what you think?
I think if this is your concern you should just use WRITEMAP and forget about it.
-- -- Howard Chu
Thanks for pointing me out to writemap option https://lmdb.readthedocs.io/en/release/#lmdb.Environment Re ACID implications it will have - metasync=True, sync=True, map_async=False, mode=493, create=True, readahead=True, writemap=True will this ensure integrity on disk after every write? What sequence of events (contrasted with writemap=False) could lead to data corruption here?
With sync=true it has the same integrity guarantees as writemap=false. There is no possibility of data corruption.
metasync flag is irrelevant when sync is set. map_async flag is irrelevant when sync is set.
Vitaly Zuevsky wrote: Memory pressure isn't my problem. I see 2-15 major faults a second in the event loop of the reader thread, which can effectively stall the loop for anything between 1-400ms (I measure iowait jiffies of the pid). I understand the faults result from the writer thread, actively modifying the file mmaped by the reader. I assume writes to the file invalidate corresponding pages of the reader, and those invalid pages only get updated (via major faults) on first access by the reader. If not locking, perhaps madvise/willneed combination could
help, what you think?
I think if this is your concern you should just use WRITEMAP and forget about it.
-- -- Howard Chu
Thanks for pointing me out to writemap option https://lmdb.readthedocs.io/en/release/#lmdb.Environment Re ACID implications it will have - metasync=True, sync=True, map_async=False, mode=493, create=True, readahead=True, writemap=True will this ensure integrity on disk after
every write? What sequence of events (contrasted with writemap=False) could lead to data corruption here?
With sync=true it has the same integrity guarantees as writemap=false. There is no possibility of data corruption.
metasync flag is irrelevant when sync is set. map_async flag is irrelevant when sync is set.
-- -- Howard Chu
You mentioned I need writemap=True to solve my original problem. So I am looking to not jeopardize integrity by adding that. You say I can use sync=true and writemap=True combination to achieve it? Or those flags are mutually exclusive?
"Vitaly Zuevsky" vitaly.zuevsky@gmail.com schrieb am 24.05.2021 um 22:06 in
Nachricht 068e01d750d8$400a64e0$c01f2ea0$@gmail.com:
Vitaly Zuevsky wrote:
Hi
MAP_LOCK flag to mmap() on LMDB readers should populate page cache from the file being mapped and preclude those pages from eviction. I was wondering if that was ever tested, especially with frequent writes to the underlying file. And is there a standard way (a compile option) to include that flag without modifying source code directly? I see quite a few major faults with perf stat, and I am looking if they could be
avoided.
I am new to LMDB and I guess this discussion may have taken place before, if you could point me out to the right place..
No. In general it's a bad idea to interfere with the OS's paging decisions.
In
particular, on a machine with plenty of RAM, pageouts aren't an issue anyway. On a system where memory is tight, page faults will be unavoidable, so the question is what other memory are you going to lose if you force the LMDB pages to stay locked in?
Memory pressure isn't my problem. I see 2-15 major faults a second in the event loop of the reader thread, which can effectively stall the loop for anything between 1-400ms (I measure iowait jiffies of the pid). I understand the faults result from the writer thread, actively modifying the file mmaped by the reader. I assume writes to the file invalidate corresponding pages of the reader, and those invalid pages only get updated (via major faults) on first access by the reader. If not locking, perhaps madvise/willneed combination could help, what you think?
I don't know the internals of LMDB, but writing to a file that has mapped region seems to be a poor idea. Why not write the region? AFAIR HP-UX really didn't like that.
Regards, Ulrich
Ulrich Windl wrote:
"Vitaly Zuevsky" vitaly.zuevsky@gmail.com schrieb am 24.05.2021 um 22:06 in
Nachricht 068e01d750d8$400a64e0$c01f2ea0$@gmail.com:
Vitaly Zuevsky wrote:
Hi
MAP_LOCK flag to mmap() on LMDB readers should populate page cache from the file being mapped and preclude those pages from eviction. I was wondering if that was ever tested, especially with frequent writes to the underlying file. And is there a standard way (a compile option) to include that flag without modifying source code directly? I see quite a few major faults with perf stat, and I am looking if they could be
avoided.
I am new to LMDB and I guess this discussion may have taken place before, if you could point me out to the right place..
No. In general it's a bad idea to interfere with the OS's paging decisions.
In
particular, on a machine with plenty of RAM, pageouts aren't an issue anyway. On a system where memory is tight, page faults will be unavoidable, so the question is what other memory are you going to lose if you force the LMDB pages to stay locked in?
Memory pressure isn't my problem. I see 2-15 major faults a second in the event loop of the reader thread, which can effectively stall the loop for anything between 1-400ms (I measure iowait jiffies of the pid). I understand the faults result from the writer thread, actively modifying the file mmaped by the reader. I assume writes to the file invalidate corresponding pages of the reader, and those invalid pages only get updated (via major faults) on first access by the reader. If not locking, perhaps madvise/willneed combination could help, what you think?
I don't know the internals of LMDB, but writing to a file that has mapped region seems to be a poor idea. Why not write the region?
On any modern OS with unified buffer cache, there is no difference. (OpenBSD is notable in its lack of a unified buffer cache, can't think of any other such deficient OSs these days.)
AFAIR HP-UX really didn't like that.
Ulrich Windl wrote:
"Vitaly Zuevsky" vitaly.zuevsky@gmail.com schrieb am 24.05.2021 um 22:06 in
Nachricht 068e01d750d8$400a64e0$c01f2ea0$@gmail.com:
Vitaly Zuevsky wrote:
Hi
MAP_LOCK flag to mmap() on LMDB readers should populate page
cache
from the file being mapped and preclude those pages from eviction. I was wondering if that was ever tested, especially with frequent writes to the underlying file. And is there a standard way (a compile option) to include that flag without modifying source code directly? I see quite a few major faults with perf stat, and I am looking if they could be
avoided.
I am new to LMDB and I guess this discussion may have taken place before, if you could point me out to the right place..
No. In general it's a bad idea to interfere with the OS's paging decisions.
In
particular, on a machine with plenty of RAM, pageouts aren't an issue anyway. On a system where memory is tight, page faults will be unavoidable, so the question is what other memory are you going to lose if you force the LMDB pages to stay locked in?
Memory pressure isn't my problem. I see 2-15 major faults a second in the event loop of the reader thread, which can effectively stall the loop for anything between 1-400ms (I measure iowait jiffies of the pid). I understand the faults result from the writer thread, actively modifying the file mmaped by the reader. I assume writes to the file invalidate corresponding pages of the reader, and those invalid pages only get updated (via major faults) on first access by the reader. If not locking, perhaps madvise/willneed combination could help, what you
think?
I don't know the internals of LMDB, but writing to a file that has mapped
region seems to be a poor idea.
Why not write the region?
On any modern OS with unified buffer cache, there is no difference. (OpenBSD is notable in its lack of a unified buffer cache, can't think of any other such deficient OSs these days.)
I was originally wrong assuming that memory pressure isn't my problem. The writers are not as per further testing (and in line with Howard's statement above). There are situations (such as my real-time loop), where user should have ability to lock populated pages even at the expanse of potential oom scenaria. I have now found a workaround outside LMDB code, but I would suggest to provide that option, especially in light of "softer" MAP_LOCK flag semantics compared to mlock().
On Mon, May 31, 2021 at 9:09 PM Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de wrote:
[...]
I don't know the internals of LMDB, but writing to a file that has mapped region seems to be a poor idea. Why not write the region? AFAIR HP-UX really didn't like that.
Regards, Ulrich
AFAIK, this is not a problem, since ~2003 HP-UX has "unified file cache" (aka unified page cache).
Regards, Leonid.
openldap-technical@openldap.org
-
Howard Chu -
Ulrich Windl -
Vitaly Zuevsky -
Леонид Юрьев (Leonid Yuriev)