Howard Chu wrote:
Mark Zealey wrote:
I'm not doing *any* commits just one big txn for all the data...
The below C works fine up until i=4m (ie 500mb of residential memory shown in top), then has massive slowdown, shared memory (again, as seen in top) increases, waits about 20-30 seconds and then disks get hammered writing 10mb/sec (200txns) when they are capable of 100-200mb/sec streaming writes... Does it do the same for you? ...
Kyoto writes async by default. You should do the same here, use MDB_NOSYNC on the env_open.
MDB_NOSYNC makes no difference in my test case above - seeing exactly the same memory, speed and disk patterns. Are you able to reproduce it?
Yes, I see it here, and I see the problem. LMDB was not originally designed to handle transactions of unlimited size. It originally had a txn sizelimit of about 512MB. In 0.9.7 we added some code to raise this limit, and it's performing quite poorly here. I've tweaked my copy of the code to alleviate that problem but your test program still fails here because the volume of data being written also exceeds the map size. You were able to run this to completion?
Two things... I've committed a patch to mdb.master to help this case out. It sped up my run of your program, using only 10M records, from 19min to 7min.
Additionally, if you change your test program to commit every 2M records, and avoid running into the large txn situation, then the 10M records are stored in only 1m51s.
Running it now with the original 100M count. Will see how it goes.