One thing I just noticed, while testing replication with 3 servers on my
laptop - during a refresh, the provider gets blocked waiting to write to
the consumers after writing about 4000 entries. I.e., the consumers
aren't processing fast enough to keep up with the search running on the
(That's actually not too surprising since reads are usually faster than
The consumer code has lots of problems as it is, just adding this note
to the pile.
I'm considering adding an option to the consumer to write its entries
with dbnosync during the refresh phase. The rationale being, there's
nothing to lose anyway if the refresh is interrupted. I.e., the consumer
can't update its contextCSN until the very end of the refresh, so any
partial refresh that gets interrupted is wasted effort - the consumer
will always have to start over from the beginning on its next refresh
attempt. As such, there's no point in safely/synchronously writing any
of the received entries - they're useless until the final contextCSN update.
The implementation approach would be to define a new control e.g. "fast
write" for the consumer to pass to the underlying backend on any write
op. We would also have to e.g. add an MDB_TXN_NOSYNC flag to
mdb_txn_begin() (BDB already has the equivalent flag).
This would only be used for writes that are part of a refresh phase. In
persist mode the provider and consumers' write speeds should be more
closely matched so it wouldn't be necessary or useful.
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Apologies if this arrives twice, I sent it once two days ago before
subscribing to the list and it hasn't appeared yet so I'm resending after
Memory mapping files on Windows requires the file size to match the size of
the memory mapped region however in the current implementation this
requires creating a file the full size of the environment which may be much
larger than the actual amount of data in the database.
This patch creates the database as a sparse file on Windows. Windows
Explorer will report the file size as the size of the entire memory mapped
region but the size on disk will be proportional to the size of committed
data in the database. This can be checked by checking the "size on disk"
property of the file.
Let me know if this is posted incorrectly, otherwise feel free to modify
the patch in whatever way is necessary for approval.
I am working on a proof of concept of a simple key value store service
using lmdb as the back end.
I am using sendfile to avoid an extra copy.
I accomplished this by creating function to extract get the mmap ptr from
the env so I can calculate the offset of data.
I figured that function might be useful to the bigger community.
If it does not fit in the bigger vision of lmdb it is fine not to include
You will find the diff attached.
I'd like a slap tool which verifies an LDIF before I try to ldapadd/slapadd it.
"slapadd -u -o value-check=yes" is fairly close. What does it fail to catch?
I can think of:
- Duplicate entries.
- Missing entries (if the initial DB is expected to be empty).
- Child entries before parents (OK for slapadd to at least back-<bdb,hdb,mdb>).
- Issues which the tool can only catch if it opens the database, like attempts
to add already-existing entries. I probably don't want to do that.
- Issues which overlays like slapo-unique would reject. Can't do that,
since the overlay won't have a non-empty DB to check against and slap
tools do not use overlays anyway. Might special-case "unique" though,
since the "duplicate entries" check will need uniqueness code anyway.
Hi list ,
Recently I read docs about lmdb , there are two sentences
1) readers do not block writers
2) writers do not block readers
I can understand 'readers do not block writers' , but cannot understand the second one , can someone help explain , how lmdb achieve 'writers do not block readers', below is my understandings , please correct me if anything wrong.
if the access pattern is write - read, since lmdb only support two version of data , when the write has been started but not committed , the concurrent read may read stale data since write has not been committed.
Sometimes many interesting changes are solely in RE24 and releasing the next
true release takes some time. During this period I'd love to see pre-release
version numbers assigned to RE24.
Especially in case Quanah sends out an inquiry for another RE24 test round
something like currently 2.4.41a1 or 2.4.41b5 or 2.4.41rc42 could be assigned
This would make it possible to package RPMs etc. of pre-releases with
non-conflicting version numbers which get replace by the real release package
later. And for test rounds everybody would test the same snapshot.
What do you think?
I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations. The function signatures of the JSON-RPC calls
are modeled to be similar to the ones used in json2ldap (which does just the opposite
direction to talk LDAP via JSON-RPC) . The previous hand-crafted format passed on the
socket was harder to parse and needed a manually built parser whereas now a standard
library can be used. However, handling the JSON data structures now imposes an additional
dependency to Jansson (a JSON access library in C) . Jansson itself is leightweight and
has no dependencies itself. Due to the limited use of back-sock and the enhanced ease of
use I think it would be acceptable to add this dependency.
I would be glad if the modification would be possible to be applied to OpenLDAP and I
happily submit a patch.
 JSON2LDAP interface from JSON-RPC to LDAP
 Jansson, a C library for reading and writing JSON data structures
"You don't become great by trying to be great, you become great by wanting to do something,
and then doing it so hard that you become great in the process." - xkcd #896