2014-10-03 3:13 GMT+04:00 Howard Chu hyc@symas.com:
commit fc409d89e0d9dde20f612e34c2a463c8a81ea000 Author: Leo Yuriev leo@yuriev.ru Date: 2014-09-20 06:51:04 +0400
EXTENSION - lmdb: more usefull info from mdb_stat tool.A bit ambiguous. me_tail_txnid is actually the ID of the oldest reader, n=
ot
the "last" reader. I'm not convinced of the value of this patch, since yo=
u
can already view the readers list.
I am agree that "tail" is NOT a best choice. But the main value of this patch is not to show a txn of oldest reader, but to show an info about pages usage. Especially the amount of pages which are "blocked" by oldest (laggard) reader, and how much pages are actually available.
2014-10-04 0:04 GMT+04:00 =D0=9B=D0=B5=D0=BE=D0=BD=D0=B8=D0=B4 =D0=AE=D1=80= =D1=8C=D0=B5=D0=B2 leo@yuriev.ru:
Fwd: (ITS#7841) high disk utilization
2014-10-03 3:13 GMT+04:00 Howard Chu hyc@symas.com:
commit 841059330fd44769e93eb4b937c3ce42654fad6f Author: Leo Yuriev leo@yuriev.ru Date: 2014-09-20 07:16:15 +0400
BUGFIX - lmdb: lock meta-pages in writemap-mode to avoid unexpect=
ed
write, before the data pages would be synchronized.
Without locking the meta-pages may be writen by OS before otherdata, in this case database would be inconsistent.
Seems unnecessary. Won't happen by default; could happen with MDB_NOSYNC=
but
that risk is already documented.
We are using the combination: envflags writemap nosync lifo checkpoint 0 1
If the checkpoint is set in seconds, it gives us the assurance consistent state database on disk. However, without this patch meta-pages can be written by the kernel before the data.
In fact, for a full guarantee in case of death slapd process, meta-page should be written explicitly. But it requires a lot of changes and I do not do that.
commit 0c168d0e63ed78d13df3fc8a42f3667335678639 Author: Leo Yuriev leo@yuriev.ru Date: 2014-09-20 10:13:28 +0400
FEATURE - lmdb: MDB_LIFORECLAIM & MDB_COALESCE modes. Reclaim FreeDB in LIFO order - this is a main feature. Also aim to coalesce small FreeDFB records.Will spend more time looking at this closer.
I would be suggested, but do not insist, review this patch on github.
commit 8ddd63161aeb2689822d1a8d27385d62e4e341ae Author: Leo Yuriev leo@yuriev.ru Date: 2014-09-19 22:47:19 +0400
BUGFIX - lmdb: properly sync meta-pages in mdb_sync_env(). Meta-pages may be updated during data-syncing in mdb_sync_env(), in this case database would be inconsistent. Check-and-retry if lead txn-id changed during flushing data inmdb_sync_env().
Probably could simplify this, just obtain the write mutex unconditionall=
y,
then there's no need to loop or retry. But also, this depends on MDB_NOL=
OCK
- if that's set, then do no locking at all.
I did so for reasons of performance and less a lock retention time.
Retries will be if there an intensive flow of changes. In this case it will be a lot of updated pages, the record which will take some time.
However, in subsequent iterations (if a transactions had committed while there was a record), the modified pages will be much fewer, and the sync will be quick.
Thus (and it was seen in tests) even when a substantial amount of the transactions, usually only two iterations of the cycle, without locking and flow of changes are not suspended.
commit 147f41a8110f28456bc32123bde86d47183f9c0a Author: Leo Yuriev leo@yuriev.ru Date: 2014-09-04 16:01:15 +0400
FEATURE - lmdb: implementation of "checkpoint kbytes". Force flush when volume of the changes reached a configurablethreshold.
Probably OK. Needs some typographical cleanup. Not sure "syncbytes" is a good name.
Agree. I just took the first choice and try to retaining the style. Ideas?
commit fb82a0b688f4c31313d0790415feda8aaa18651c Author: Leo Yuriev leo@yuriev.ru Date: 2014-09-04 15:18:16 +0400
CHANGE - lmdb-backend: checkpoint-interval in seconds instead ofminutes.
Gratuitous change. We used minutes since the BDB backend uses minutes, a=
nd
the intention was to maintain parallel functionality. What's the justification for this change?
As I had wrote above, we are using the combination: envflags writemap nosync lifo checkpoint 0 1
If the interval is specified in minutes, then it can not be set less than one minute. But it's too big amount of time to allow lost the updates.
However, setting the synchronization interval of one second, we reduce the amount of losses in the event of an accident to an acceptable level, while the load on the storage system is acceptable even for a large flow of updates.
As a result, I have not found a better solution than simply replace the minutes by the seconds.
commit fc409d89e0d9dde20f612e34c2a463c8a81ea000 Author: Leo Yuriev leo@yuriev.ru Date: 2014-09-20 06:51:04 +0400
EXTENSION - lmdb: more usefull info from mdb_stat tool.A bit ambiguous. me_tail_txnid is actually the ID of the oldest reader, =
not
the "last" reader. I'm not convinced of the value of this patch, since y=
ou
can already view the readers list.
I am agree then "tail" is a best choice. But the main value of this patch is not to show a txn of oldest reader, but to show an info about pages usage. Especially the amount of pages which are "blocked" by oldest (laggard) reader, and how much pages are actually available.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Thank you in advance. BR. Leonid Yuriev.
-
leo＠yuriev.ru