openldap-technical

openldap-technical@openldap.org

14 participants
9796 discussions

LMDB error
by Shu, Xinxin 09 Apr '15

09 Apr '15

Hi list , I am working on intergrating LMDB with ceph, I met follow error "Invalid argument" when I commit transaction to LMDB, details are descripted as following. Two threads operate on LMDB, each thread is responsible for create write transaction and read transaction , the write transaction will update LMDB, and read transaction is for iterator that read values from LMDB, the two threads share same MDB_env. I check mdb_txn_commit source code , there are two places to return "EINVAL", 1. if (txn == NULL || txn->mt_env == NULL) return EINVAL; 2. if (txn != env->me_txn) { DPUTS("attempt to commit unknown transaction"); rc = EINVAL; goto fail; } I check txn and txn->mt_env and rule out the first case , so only the second case cause "EINVAL", and variable env->me_txn is only initialized when you start a write transaction with mdb_txn_open, and there is a lock to prevent several writer from operating on LMDB. Btw , I tried to add lock to synchronize the two threads , everything seems ok. My question is in what situation can cause "txn != env-> me_txn",. I'm very new to LMDB, if there is anything wrong, please correct me. Cheers, xinxin

2 1

Re: Very slow ldapserach
by Saša-Stjepan Bakša 09 Apr '15

09 Apr '15

On 8 April 2015 at 18:05, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Wednesday, April 08, 2015 11:58 AM +0200 Saša-Stjepan Bakša < > ssbaksa(a)gmail.com> wrote: > > >> >> I have tested search with your suggestion. No crash - slapd crashes only >> when deref is on or in best case it is slow (10 to 20 seconds for >> answer). As I have said, when database is small (relative size) it is >> quick as we expected. >> >> >> So, which alternatives do I have? Back to hdb? Uh, I don't like it to >> much. >> >> Is this long standing issue a big problem to solve? I am not developer so >> maybe this was a rude question (sorry!). >> > > Please keep replies on the list if you want help. > > My guess is this is ITS#7657. > > > --Quanah > > I am sorry for this mistake with answering to you and not to the group. It was unintentional mistake. Ok, I will check ITS#7657. Sasa

2 1

two namingContexts problem
by Wang, Hui 09 Apr '15

09 Apr '15

Hi, All, I got two nameContexts, but I only set up one in slapd.conf which is o=csun. Is there any way to find out where the other nameContexts coming from? Is this normal? I also searched all the files on Linux, I didn’t find any file contains “cn=changelog” in the file content. [root@oldaptest01 init.d]# ldapsearch -x -b '' -s base '(objectclass=*)' namingContexts # extended LDIF # # LDAPv3 # base <> with scope baseObject # filter: (objectclass=*) # requesting: namingContexts # # dn: namingContexts: cn=changelog namingContexts: o=csun # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 # find -name "*.conf" -print | xargs grep -i "cn=change" No result Thank you Hui

1 0

[lmdb] MDB_BAD_DBI error upon mdb_txn_commit (newbie question)
by HLaw 08 Apr '15

08 Apr '15

Dear list, I am new to lmdb and my apologies if this is on something obvious. I encountered the MDB_BAD_DBI error in a simple single process, multi-threaded program using lmdb. The scenario is : * One thread doing a relatively long write transaction mdb_txn_begin -> mdb_dbi_open on several database -> (write) -> mdb_txn_commit * Another thread (or other threads) of the same process doing short read-only transactions mdb_txn_begin -> mdb_dbi_open on some of the database above -> (read) -> mdb_txn_commit As per the advice in the document I do not close the database directly. Also the environment created / opened is shared by all threads, being in the same process. All is fine when these threads execute separately, but if the read transaction thread(s) open the database and commit within the time span of the write transaction in another thread, the above error occurs upon attempt to commit the write transaction. I have done a search on the MDB_BAD_DBI error and noticed the commit below. https://gitorious.org/mdb/mdb/commit/0401f2deed75a83d2de790b8a1313e1792e5a0… Upon a brief look into this and the source of lmdb, my understanding is that the error is due to the increment of lmdb's internal sequence numbers of the database kept by the environment (by the read thread when the read transaction commits). When the write thread subsequently tries to commit, the database sequence number kept privately by the write transaction (copied from the environment when the transaction began) does not match that of the environment, causing the error. To avoid this in a single process multi-threaded environment, it seems that no read transactions could commit on databases while they are involved in a write transaction, i.e. I need to complete all read transactions before a write transaction on the same database could begin. Alternatively, to have concurrent read transactions with a write transaction, the read transactions have to take place in a separate process (such that they would not share the same environment and caught by the above error). Grateful if anyone could kindly enlighten me on whether the understanding is correct, or if I am doing something contrary to the how lmdb should be used. Thank you very much. - H Law

4 5

Re: Very slow ldapserach
by Quanah Gibson-Mount 08 Apr '15

08 Apr '15

--On Wednesday, April 08, 2015 11:58 AM +0200 Saša-Stjepan Bakša <ssbaksa(a)gmail.com> wrote: > > > > > > On 7 April 2015 at 21:14, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > > --On Tuesday, April 07, 2015 1:10 PM -0700 Quanah Gibson-Mount > <quanah(a)zimbra.com> wrote: > > > > ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w password -s sub > -a always -b dc=SPR objectClass=* > > > Sounds like there's something wrong with your openldap build or server. > Zimbra has clients with DBs of various sizes and entry counts, often in > the multi-GBs, and we're not seeing any such issues. > > > Actually, you are doing alias deref. I believe there's a known issue > with back-mdb and alias deref. Do you see the same behavior if you > select -a never instead? > > > > --Quanah > > > > > Yes, we are depending on aliases. Without them we will need at least 2 > searches to get data we need. In new version even more because we have > data which is similar to many users and we use another aliases inside > each user to fetch that data for particular user. Let we say that each > user have 3 aliases to 3 groups of data which can have at leas 30 > different possibilities for each user. Without aliases data size for each > user will be to big and in summary database will be huge. > > > I have tested search with your suggestion. No crash - slapd crashes only > when deref is on or in best case it is slow (10 to 20 seconds for > answer). As I have said, when database is small (relative size) it is > quick as we expected. > > > So, which alternatives do I have? Back to hdb? Uh, I don't like it to > much. > > Is this long standing issue a big problem to solve? I am not developer so > maybe this was a rude question (sorry!). Please keep replies on the list if you want help. My guess is this is ITS#7657. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Openldap NON responsive frequently
by MOHAMED Akbar 08 Apr '15

08 Apr '15

Hello, Current openldap setup is REPLICATION, with MirrorMode on. Host is solaris11. Frequently the clients(mostly solaris10), complain of : libsldap: Status: 91 Mesg: openConnection: simple bind failed - Can't connect to the LDAP server Restarting the opeldap service fixes this How can I fix this issue? ******************************* This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

2 1

Definition of an object.
by dE 08 Apr '15

08 Apr '15

Hi! I was reading RFC 4512, here there is a mention of 'object' for the first time in "Object identifiers (OIDs) [X.680] are represented in....". Question is what is an object? Is it an entry (aka directory) in the server?

2 2

Very slow ldapserach
by Saša-Stjepan Bakša 07 Apr '15

07 Apr '15

Hi, Year ago we have tested openldap with back_mdb and it was fantastic. Search worked as a charm. Database was filled with 20 mil. users and serach returned some 20 k results per sec (my colegue did the test). Now we need that setup for some tests and we encountered very slow response - 1 search for user data with some aliased data need 8 to 20 seconds to be retrieved. ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w test -s sub -a always -b num=1234563123,dc=num,dc=SPR ObjectClass=* num=1234563123,dc=num,dc=SPR is alias to uid aliasedObjectName: uid=1234563123,ds=USERS,o=STANDARD,dc=spr We build our openldap from git source. We have tried new as older versions as well and no change is seen. Hardware: SuperMicro, 2xQuad core, 32 GB RAM, RAID 10 storage. HP blade 2xQuad core, 64 GB RAM sorage 2 disks in mirror. Results are the same and not depending on hardware. Openldap ver: root@centdevel openldap# git log commit 68d9aa207f51b4d1ef29bb9876e7da8c7eaf0eee Author: Quanah Gibson-Mount <quanah(a)openldap.org> Date: Tue Apr 8 21:16:52 2014 -0500 ITS#7430, ITS#6359 OS is Centos 6.4 (also tryed on Centos 6.6)numx mdb config part is: [root@spr2 cn=config]# cat olcDatabase\=\{1\}mdb.ldif # AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify. # CRC32 2c245069 dn: olcDatabase={1}mdb objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {1}mdb olcDbDirectory: /opt/openldap/var/openldap-data olcSuffix: dc=spr olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by anonymou s auth by dn="cn=admin,dc=spr" write by * none olcAccess: {1}to attrs=shadowLastChange by self write by * read olcAccess: {2}to dn.base="" by * read olcAccess: {3}to * by self write by dn="cn=admin,dc=spr" write by * read olcLastMod: TRUE olcRootDN: cn=admin,dc=spr olcRootPW:: xyzdgsdsadeew olcDbCheckpoint: 4096 10 olcDbNoSync: TRUE olcDbIndex: objectClass eq olcDbIndex: uid eq olcDbIndex: num eq olcDbIndex: numx eq olcDbIndex: Username eq olcDbIndex: entryCSN eq olcDbIndex: entryUUID eq olcDbIndex: contextCSN eq olcDbMaxSize: 16106127360 structuralObjectClass: olcMdbConfig entryUUID: 21ac150c-6b30-1034-9009-81396a683c5e creatorsName: cn=admin,cn=config createTimestamp: 20150330135513Z entryCSN: 20150330135513.544218Z#000000#000#000000 modifiersName: cn=admin,cn=config modifyTimestamp: 20150330135513Z MDB database stat: [root@spr2 openldap]# /opt/openldap/sbin/mdb_stat /opt/openldap/var/openldap-data/ -e -rr -a Environment Info Map address: (nil) Map size: 16106127360 Page size: 4096 Max pages: 3932160 Number of pages used: 1523336 Last transaction ID: 16058165 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 0 stale readers cleared. (no active readers) Status of Main DB Tree depth: 1 Branch pages: 0 Leaf pages: 1 Overflow pages: 0 Entries: 11 Status of ad2i Tree depth: 1 Branch pages: 0 Leaf pages: 1 Overflow pages: 0 Entries: 38 Status of contextCSN Tree depth: 0 Branch pages: 0 Leaf pages: 0 Overflow pages: 0 Entries: 0 Status of dn2i Tree depth: 4 Branch pages: 2937 Leaf pages: 333338 Overflow pages: 0 Entries: 16000069 Status of entryCSN Tree depth: 3 Branch pages: 3 Leaf pages: 307 Overflow pages: 0 Entries: 8000034 Status of entryUUID Tree depth: 3 Branch pages: 259 Leaf pages: 62932 Overflow pages: 0 Entries: 8000034 Status of id2e Tree depth: 4 Branch pages: 4446 Leaf pages: 1000005 Overflow pages: 0 Entries: 8000034 Status of numx Tree depth: 3 Branch pages: 128 Leaf pages: 22295 Overflow pages: 0 Entries: 2000004 Status of num Tree depth: 3 Branch pages: 129 Leaf pages: 22325 Overflow pages: 0 Entries: 2000004 Status of objectClass Tree depth: 1 Branch pages: 0 Leaf pages: 1 Overflow pages: 0 Entries: 29 Status of Username Tree depth: 0 Branch pages: 0 Leaf pages: 0 Overflow pages: 0 Entries: 0 Status of uid Tree depth: 3 Branch pages: 34 Leaf pages: 7883 Overflow pages: 0 Entries: 1000004 Build config: make clean ./configure --enable-hdb=no \ --enable-bdb=no \ --enable-monitor=yes \ --prefix=/opt/openldap \ --enable-local=yes \ --enable-accesslog=yes \ --enable-syncprov=yes \ --enable-debug=yes make depend make #STRIP='' rm -r /opt/openldap/etc/openldap/schema make install #STRIP='' removing debug has no efect Do you have any hint for us? Br Sasa

4 8

Multiple programs not able to read LMDB concurrently
by Sravan Kumar Reddy Javaji 02 Apr '15

02 Apr '15

Hello Everyone, I am trying to access same LMDB source using multiple programs at the same time. I set max_readers to 2 at the time of creating environment by the first program, but still second program is not able to read the LMDB. Could some one please let me know how could I implement this feature? - Thanks and Regards, Sravan

2 2

how to enable debug message
by Shu, Xinxin 02 Apr '15

02 Apr '15

Hi list , I want to enable debug message for lmdb, I have noticed MDB_DEBUG Macro, and I define MDB_DEBUG when compiling lmdb, but did not get any debug message, how can I enable debug message ?

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical