LMDB overflow pages - MDB_RESERVE
by Christian Sell
I forgot to mention: I am using MDB_RESERVE to avoid extra memcpy. Could it be
that this is the cause for the extreme database bloat?
> Hello,
>
> something wrong with the question below?
>
> I am trying to use LMDB to store large (huge) amounts of binary data which,
> for
> the reason of limiting memory footprint, are split into chunks. Each chunk ist
> stored under a separate key, made up of [collectionId, chunkId], so that I can
> later iterate the chunks using a LMDB cursor. Chunk size is configurable.
>
> During my tests, I encountered a strange scenario where, after inserting some
> 2000 chunks consisting of 512KB each, the database size had grown to a value
> that was roughly 135 times the calculated size of the data. I ran the stat
> utility over the db and saw that there were > 12000 overflow pages vs. approx.
> 2000 data pages. When I reduced the chunk size to 4060 bytes, the number of
> overflow pages went down to 1000, and the database size went down to the
> expected number (I experimented with different sizes, this was the best
> result).
> I did not find any documentation to explain this behaviour, or how to deal
> with
> it. Of course it makes me worry about database bloat and the consequences. Can
> anyone shed light on this?
>
> thanks,
> Christian
>
Christian Sell
8 years
Getting around the single-threaded syncrepl model?
by Bannister, Mark
I was speaking with Howard in person last week at LDAPCon about the problems I might hit if I wanted to run hundreds of replicas off a single master. If I heard right, Howard told me that OpenLDAP uses a single thread for replication, and therefore processes each replica in a serial fashion, one at a time. If any replica is going slowly (high network latency for example), it would have a knock-on effect to the time it takes to get changes replicated across the whole estate.
We have an aggressive target to meet - updates out in 5 minutes. Updates are small, but alarm bells are now ringing in my head about OpenLDAP's ability to deliver this.
This leads to a couple of questions:
1 - How easy would it be to patch OpenLDAP 2.4 to get the master to use multiple threads for replication? Is that a reasonably straightforward fix, or is this quite a sizeable architectural change?
2 - If my master server has 40 cores, would there be mileage in setting up a number of slapd processes (say 10) on the same host, bound to different sockets, all acting as first-level replicas, and then all the replicas fanning out across the WAN synchronise from these processes rather than from the master process? That would be a way, I suppose, of getting the master server to make better use of the available cores to get updates out quicker?
Thanks,
Mark.
________________________________
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
8 years
Sync replication halted
by espeake＠oreillyauto.com
We are run a 3 node MMR with Openldap 2.4.39 on Ubuntu 14.04 (trusty)
Everything runs great with an occasional hiccup like this. I see the
following in the logs that appear to be searches on the RID numbers that
are used for config replicationL
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: syncprov_matchops: skipping
relayed sid 003
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: syncprov_matchops: skipping
original sid 001
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: syncprov_matchops: skipping
relayed sid 003
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: syncprov_matchops: skipping
original sid 001
Imeadiately after this all changes show csn_too_old errors:
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004
cookie=rid=004,sid=001,csn=20151125103121.701210Z#000000#001#000000
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old,
ignoring 20151125103121.701210Z#000000#001#000000
(uid=JRIVAS21,ou=Users,dc=oreillyauto,dc=com)
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004
cookie=rid=004,sid=001,csn=20151125103121.705110Z#000000#001#000000
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old,
ignoring 20151125103121.705110Z#000000#001#000000
(uid=JRIVAS21,ou=Users,dc=oreillyauto,dc=com)
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004
cookie=rid=004,sid=001,csn=20151125103121.713760Z#000000#001#000000
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old,
ignoring 20151125103121.713760Z#000000#001#000000
(uid=ACERVANT4,ou=Users,dc=oreillyauto,dc=com)
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004
cookie=rid=004,sid=001,csn=20151125103121.718018Z#000000#001#000000
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old,
ignoring 20151125103121.718018Z#000000#001#000000
(uid=ACERVANT4,ou=Users,dc=oreillyauto,dc=com)
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004
cookie=rid=004,sid=001,csn=20151125103121.726847Z#000000#001#000000
Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old,
ignoring 20151125103121.726847Z#000000#001#000000
(uid=KCOOPER10,ou=Users,dc=oreillyauto,dc=com)
This happened on one node. The third node had the same replication error
for a few miliseconds and then recovered.
I am attaching the database config file as well.
Thank you,
Eric Speake
Senior Systems Administrator
Information Systems
O'Reilly Auto Parts
(417) 862-2674 Ext. 1975
(See attached file: olcDatabase={1}mdb.ldif)This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
8 years
LMDB: successive fillups and drops break the DB after a while
by Dominik Taborsky
Hello,
I've tried stress-testing LMDB in our use case and I've discovered something I consider a bug.
In our use case we need to keep inserting to the DB until it's full, flush it and repeat. We've designed the insert function to fail if there are not enough headroom pages for safe delete / DB drop. During the tests the DB works correctly for some amount of repeats, but then it suddenly refuses to insert anything. It seems LMDB doesn't clear or reuse all of its pages. I would like to get anyone's opinion on this.
I have attached the source file containing the test. The DB size is set to 20MB, the inserted values have by default 25kB. In the first run the DB accepts 365 inserts, which decreases over time and stabilizes at 280. After 18 repeats the DB does not accept any inserts at all:
ok 1 - pass #1 fillup run (365 inserts)
ok 2 - pass #2 fillup run (300 inserts)
ok 3 - pass #3 fillup run (286 inserts)
ok 4 - pass #4 fillup run (283 inserts)
ok 5 - pass #5 fillup run (281 inserts)
ok 6 - pass #6 fillup run (280 inserts)
ok 7 - pass #7 fillup run (280 inserts)
ok 8 - pass #8 fillup run (280 inserts)
ok 9 - pass #9 fillup run (280 inserts)
ok 10 - pass #10 fillup run (280 inserts)
ok 11 - pass #11 fillup run (280 inserts)
ok 12 - pass #12 fillup run (280 inserts)
ok 13 - pass #13 fillup run (280 inserts)
ok 14 - pass #14 fillup run (280 inserts)
ok 15 - pass #15 fillup run (280 inserts)
ok 16 - pass #16 fillup run (280 inserts)
ok 17 - pass #17 fillup run (280 inserts)
ok 18 - pass #18 fillup run (108 inserts)
not ok 19 - pass #19 fillup run (0 inserts)
not ok 20 - pass #20 fillup run (0 inserts)
The test is run 4 times with different approaches that we thought could have had some impact. These approaches combine opening the DB per insert or per single fillup, and reopening the DB just for the DB drop. In this case these options have no effect and the results are the same, but originally a unit test that this originates from had some differences so I kept them in.
Note that I also tried calling mdb_drop with "1" as the last parameter, but it had no effect on the result.
I am linking against the latest LMDB from sources.
I also tried adding a constant number of pages to the reserve before the insert function fails. This only changes the amount of inserted items but doesn't change the functionality.
If you compile the program you can supply it with an argument to change the default size of the item value size. Lower values break the DB sooner (20kB requires only 12 repeated fillups and drops).
If anyone has any idea what might be the issue, please let me know. I know deleting the DB from filesystem and creating a new one would work, but that's a hack, not a fix.
Thanks.
Dominik
8 years
rwm overlay
by BÖSCH Christian
Hi,
I configured rwm overlay like the example in the man page to allow
binds with the email address.
dn: olcOverlay={0}rwm,olcDatabase={-1}frontend,cn=config
objectClass: olcOverlayConfig
objectClass: olcRwmConfig
olcOverlay: {0}rwm
olcRwmRewrite: {0}rwm-rewriteEngine "on"
olcRwmRewrite: {1}rwm-rewriteMap "ldap" "attr2dn" "ldap://localhost/dc=abcd,dc
=net?dn?sub"
olcRwmRewrite: {2}rwm-rewriteContext "bindDN"
olcRwmRewrite: {3}rwm-rewriteRule "^mail=[^,]+@[^,]+$" "${attr2dn($0)}" ":@I"
olcRwmTFSupport: false
olcRwmNormalizeMapped: FALSE
But I get the error message the the DN is invalid:
ldapsearch -x -D "cb(a)abcd.net" -W -b 'dc=abcd,dc=net' -H ldap://openldap1.abcd.net/ 'uid=cb'
Enter LDAP Password:
ldap_bind: Invalid DN syntax (34)
additional info: invalid DN
Is there something missing or wrong?
Thanks,
Chris
8 years
LMDB overflow pages
by Christian Sell
Hello,
something wrong with the question below?
I am trying to use LMDB to store large (huge) amounts of binary data which, for
the reason of limiting memory footprint, are split into chunks. Each chunk ist
stored under a separate key, made up of [collectionId, chunkId], so that I can
later iterate the chunks using a LMDB cursor. Chunk size is configurable.
During my tests, I encountered a strange scenario where, after inserting some
2000 chunks consisting of 512KB each, the database size had grown to a value
that was roughly 135 times the calculated size of the data. I ran the stat
utility over the db and saw that there were > 12000 overflow pages vs. approx.
2000 data pages. When I reduced the chunk size to 4060 bytes, the number of
overflow pages went down to 1000, and the database size went down to the
expected number (I experimented with different sizes, this was the best result).
I did not find any documentation to explain this behaviour, or how to deal with
it. Of course it makes me worry about database bloat and the consequences. Can
anyone shed light on this?
thanks,
Christian
8 years
Re: ERR_employeeadd {'info': 'modifications require authentication', 'desc': 'Strong(er) authentication required'}
by Andrei Valoshyn
Le 19/11/2015 19:43, Andrei Valoshyn a écrit :
Hello!
I have slapd 2.4.39 and python 2.6
I tried to create an user via python when I tried do that with
rootpermission - it's OK. But when I did this with config in
slapd.conf"access to *
bygroup.exact="cn=LDAP_admins,ou=Roles,ou=Groups,dc=exadel,dc=com"
write"I have an error " ERR_employeeadd {'info': 'modifications
requireauthentication', 'desc': 'Strong(er) authentication required'} "
I tried to use " l.protocol_version = ldap.VERSION{2,3} " via 389 port
My function for adding ldif is :
l = ldap.initialize(server)
l.simple_bind(username, ldapsrvpassword)
def employeeadd():
ldif = modlist.addModlist(attrs)
l.add_s(dn,ldif)
Will be very appreciate for any help
Hello Andrei,
I suppose that the username you use is a member
ofcn=LDAP_admins,ou=Roles,ou=Groups,dc=exadel,dc=com, but
whichobjectClass did you use in your group? By default, the OpenLDAP
ACLsystem will use groupOfNames, searching user in the member attribute.
Ifyou have for example a groupOfUnixNames, you need to set your ACL
likethis:access to *
bygroup/groupOfUniqueNames/uniqueMember.exact="cn=LDAP_admins,ou=Roles,ou=Groups,dc=exadel,dc=com"write
Hello *Clément*,
Thank you for reply!
I have groupOfNames and posixGroup objectClass. But at that moment this
problem was fixed. The matter of fact Python had some problem. It was
reinstalled.
Thank you!
--
CONFIDENTIALITY NOTICE: This email and files attached to it are
confidential. If you are not the intended recipient you are hereby notified
that using, copying, distributing or taking any action in reliance on the
contents of this information is strictly prohibited. If you have received
this email in error please notify the sender and delete this email.
8 years
Multiple certificates in slapd
by Olivier Nicole
Hi,
I am planing a transition of the certificate I use in OpenLDAP for LDAP
over SSL (port 636).
My selft signed certificate is quite old and has become obsolete/not
recognized on some systems (for example Mac OS 10.11) so it is time to
update.
But I have many systems that use LDAP and updating all of them cannot be
done at once.
So I was wondering if it is possible to have one single slapd process
running with several posut open over SSL, keeping port 636 with the old
certificate and opening port 637 with the new certificate.
That way, i can transition the clients at my onw pace, not needing to do
all at same time.
I know that I could set-up a slave server, but that would be not as
transparent s0 I'd prefer my idea of havingslapd -h
ldaps://192.168.10.1:636/ ldaps:/192.168.10.1:637/ each using a
different certificate.
Thanks in advance,
Olivier
--
8 years
Re: Trying to set up multimaster syncrepl, error attribute 'olcTLSCertificateFile' not allowed , why?
by Betsy Schwartz
On Sat, Nov 21, 2015 at 2:00 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>
wrote:
>
> I would suggest using slapcat to export the config database and clean up
> the invalid attribute values that were incorrectly added to the bdb
> database.
>
> Thank you very much! I have made progress but am now in an entertaining
new place (having *overwritten* my database with my overlay files). But I'm
going to go RTFM and see if I can't dig myself out of that one.
> a) Upgrading to a current openldap release
> b) Switching to back-mdb, assuming a 64-bit OS.
>
Thank you! Sigh, 2.4.40 seems to be the latest in the Oracle repo (we are
using Oracle's version of RHEL6). It looks like there have been some good
bug fixes and I'll make the case for going outside the repo. Sometime in
the next couple months these two new servers will become the primary
production servers and then we'll be able to do what we want with them.
The two old servers, current production, are running 2.4.39 and 2.4.23
(and not syncing with each other!) . I've been hoping that I could sync
data between at least these two new servers and the 2.4.39 server, is that
possible or a foolish hope? Again, once these new ones become primary I'll
be able to keep them identical to each other, but I don't really 'own' the
old ones and don't want to break them. I'm still working through the manual
and the configuration settings.
One more question - still trying to understand what was done on these old
servers - on these servers the config database is 0, the monitor database
is 1 and the bdb database is 2. I can't slapcat the monitor database, is
that normal? I get
slapcat -n 1
slapcat: database doesn't support necessary operations.
I had a little trouble with this particular section of the manual which
reads simply:
20.1. Monitor configuration via cn=config(5)
This section has yet to be written.
thanks again
Betsy
8 years
questions about memberof-refint option
by M. P.
Hi,
I'm playing with memberof overlay. For my tests, I use the default
database (numbered 1) from slapd installation with suffix dc=nodomain.
The tests are running on debian jessie 8.2 and slapd version
2.4.40+dfsg-1
Activating the module in cn=module entry and activating the overlay for
the database, I have something that works like (I think) it should. I
mean adding a user (attribute member) in a group creates an attribute
memberOf for the user and deleting a user from the group deletes the
user's memberOf attribute. That's great.
There is nothing special configured.
# Entrée 1: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config
dn: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config
objectclass: olcConfig
objectclass: olcOverlayConfig
objectclass: olcMemberOf
objectclass: top
olcoverlay: {0}memberof
Reading the man page, I saw memberof-refint option. From what I
understand, when set to true, you can alter the user's "is member of"
attribute and that would be reflected in the group's "member" attribute.
Right ?
But, the member attribute is an operational attribute and can't be
modified. So I started to search for an alternative and found the
eduMember schema from here
https://spaces.internet2.edu/display/macedir/OpenLDAP+eduMember. Once
added to the installation I could use it for objects. It adds isMemberOf
and hasMember attributes that can be setable for users and groups. But
can't make it work with memberof overlay. When trying to add isMemberOf
as memberof-memberof-ad it was rejected with
member attribute=”isMemberOf” must either have DN
(1.3.6.1.4.1.1466.115.121.1.12) or nameUID
(1.3.6.1.4.1.1466.115.121.1.34) syntax
And the same error was reported with hasMember as memberof-member-ad.
To make it work together I modified the attribute's definitions and
reimported them to openldap. So I can now set isMemberOf as
memberof-memberof-ad and the same for hasMember as memberof-member-ad.
The configuration then was like this
# Entrée 1: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config
dn: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config
objectclass: olcConfig
objectclass: olcOverlayConfig
objectclass: olcMemberOf
objectclass: top
olcmemberofmemberofad: isMemberOf
olcoverlay: {0}memberof
Now that works like (I think) it should. I mean adding a user (attribute
member) in a group creates an attribute isMemberOf for the user and
deleting a user from the group deletes the user's isMemberOf attribute.
That's great.
isMemberOf is a modifiable attribute so it's time to test the
memberof-refint and set it to TRUE
# Entrée 1: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config
dn: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config
objectclass: olcConfig
objectclass: olcOverlayConfig
objectclass: olcMemberOf
objectclass: top
olcmemberofmemberofad: isMemberOf
olcmemberofrefint: TRUE
olcoverlay: {0}memberof
And this is where things do not work. I mean what was working before is
still working. If I add a member in a group an atttribute isMemberOf is
created for the user. But if I add a second attribute isMemberOf with a
second group, no new member is created on the second group. And if I
delete the attribute isMemberOf from the user's entry, it is still
visible on the group.
Does anybody have any idea why the modifications made on the user (with
the deletion of isMemberOf) are not applied to the group ? Is there
something I'm doing wrong ?
Thanks.
--
------------
M. P.
8 years
