Error apperas when number of users is higher than 21800 (total users 21900) and difference between successful query and bad one is:

Good

517a35fd => access_allowed: search access to "uid=1000,ds=USERS,o=STANDARD,dc=SPR" "entry" requested 517a35fd <= root access granted 517a35fd => access_allowed: search access granted by manage(=mwrscxd) 517a35fd search_candidates: base="uid=1000,ds=users,o=standard,dc=spr" (0x00002afa) scope=2 517a35fd => mdb_filter_candidates 517a35fd EQUALITY 517a35fd => mdb_equality_candidates (objectClass) 517a35fd => key_read 517a35fd mdb_idl_fetch_key: [01872a84] 517a35fd <= mdb_index_read 30000 candidates 517a35fd <= mdb_equality_candidates: id=30000, first=21, last=110012 517a35fd <= mdb_filter_candidates: id=30000 first=21 last=110012 517a35fd => mdb_filter_candidates 517a35fd PRESENT 517a35fd => mdb_presence_candidates (objectClass) 517a35fd <= mdb_filter_candidates: id=-1 first=1 last=-1 517a35fd mdb_search_candidates: id=-1 first=1 last=-1 517a35fd => test_filter 517a35fd PRESENT 517a35fd => access_allowed: search access to "uid=1000,ds=USERS,o=STANDARD,dc=SPR" "objectClass" requested 517a35fd <= root access granted 517a35fd => access_allowed: search access granted by manage(=mwrscxd) 517a35fd <= test_filter 6 517a35fd => send_search_entry: conn 1000 dn="uid=1000,ds=USERS,o=STANDARD,dc=SPR"

Bad:

517a4879 => access_allowed: search access to "uid=1000,ds=USERS,o=STANDARD,dc=SPR" "entry" requested 517a4879 <= root access granted 517a4879 => access_allowed: search access granted by manage(=mwrscxd) 517a4879 search_candidates: base="uid=1000,ds=users,o=standard,dc=spr" (0x00002afa) scope=2 517a4879 => mdb_filter_candidates 517a4879 EQUALITY 517a4879 => mdb_equality_candidates (objectClass) 517a4879 => key_read 517a4879 mdb_idl_fetch_key: [01872a84] 517a4879 <= mdb_index_read 240892 candidates 517a4879 <= mdb_equality_candidates: id=-1, first=21, last=240912 517a4879 <= mdb_filter_candidates: id=-1 first=21 last=240912 517a4879 mdb_search_candidates: failed (rc=-30798) 517a4879 mdb_search: no candidates 517a4879 send_ldap_result: conn=1021 op=1 p=3 517a4879 send_ldap_result: err=0 matched="" text="" 517a4879 send_ldap_response: msgid=2 tag=101 err=0

On Fri, Apr 26, 2013 at 8:51 AM, Saša-Stjepan Bakša ssbaksa@gmail.comwrote:

If I may add some of my observations,

Problem (for me) is starting to show when I have more than 20000 users inserted in LMDB database.

I am testing with latest (almost) OpenLDAP compiled from source (git pull 24.04.2013).

Test was done by adding 10k users first then adding 5k at a time. Between 20k and 25k ldapsearch started to return only this:

root@test7kde:~# ldapsearch -x -h 172.17.103.200 -D cn=admin,dc=spr -s sub -a always -w secret -b pfUsername=user20000,dc=USERNAME,dc=spr # extended LDIF # # LDAPv3 # base <pfUsername=user20000,dc=USERNAME,dc=spr> with scope subtree # filter: (objectclass=*) # requesting: ALL #

# search result search: 2 result: 0 Success

# numResponses: 1

I have attached ldapsearch results for reference.

You can see data structure from ldapsearch.

LMDB is configured like this:

dn: olcDatabase={1}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {1}mdb olcDbDirectory: /opt/openldap/var/openldap-data olcSuffix: dc=spr olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by anonymou s auth by dn="cn=admin,dc=spr" write by * none olcAccess: {1}to attrs=shadowLastChange by self write by * read olcAccess: {2}to dn.base="" by * read olcAccess: {3}to * by self write by dn="cn=admin,dc=spr" write by * read olcLastMod: TRUE olcRootDN: cn=admin,dc=spr olcRootPW:: e1NTSEF9eDRVRTBiV2Z4YnFSNnZDVDdKRWEwSWRhWFRhMDN2M3I= olcDbCheckpoint: 4096 10 olcDbNoSync: TRUE olcDbMaxSize: 38654705664 structuralObjectClass: olcMdbConfig entryUUID: 804a8ede-2cc3-1032-9b7a-c7b2eb845e03 creatorsName: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth createTimestamp: 20130329135129Z olcDbIndex: objectClass eq olcDbIndex: uid eq olcDbIndex: MSISDN eq olcDbIndex: IMSI eq olcDbIndex: pfUsername eq olcDbIndex: entryCSN eq olcDbIndex: entryUUID eq olcDbIndex: contextCSN eq olcMirrorMode: TRUE olcSyncrepl: {0}rid=003 provider=ldap://spr1.lab.os binddn="cn=admin,dc=spr" b indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly int erval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncrepl: {1}rid=004 provider=ldap://spr2.lab.os binddn="cn=admin,dc=spr" b indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly int erval=00:00:00:10 retry="5 5 300 5" timeout=1 entryCSN: 20130329141637.442296Z#000000#002#000000 modifiersName: cn=admin,cn=config modifyTimestamp: 20130329141637Z

If you need more data I wiil be more than happy to provide.

On Wed, Apr 24, 2013 at 11:06 AM, Howard Chu hyc@symas.com wrote:

...

Juergen.Sprenger@swisscom.com wrote:

...

Hi Michael,

NSS results must not be dependent on the backend database a directory service uses.

I activated connection logging and here's the proof that NSS is not the culprit.

Searches initiated by NSS are identical and exactly this behavior can also be seen when using ldapsearch from command line with parameters from the log:

# running 'getent passwd' with hdb backend: Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 BIND dn="cn=itsAgent,ou=**customerAgent,dc=scom" method=128 Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 BIND dn="cn=itsAgent,ou=**customerAgent,dc=scom" mech=SIMPLE ssf=0 Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 RESULT tag=97 err=0 text= Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SRCH base="ou=account,dc=its,dc=**scom" scope=1 deref=3 filter="(objectClass= **posixAccount)" Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SRCH attr=uid userPassword uidNumber gidNumber cn homeDirectory x-LinuxLoginShell gecos description objectClass Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SEARCH RESULT tag=101 err=0 nentries=656 text= Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 fd=13 closed (connection lost)

# running 'getent passwd' with mdb backend: Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 BIND dn="cn=itsAgent,ou=**customerAgent,dc=scom" method=128 Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 BIND dn="cn=itsAgent,ou=**customerAgent,dc=scom" mech=SIMPLE ssf=0 Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 RESULT tag=97 err=0 text= Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SRCH base="ou=account,dc=its,dc=**scom" scope=1 deref=3 filter="(objectClass= **posixAccount)" Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SRCH attr=uid userPassword uidNumber gidNumber cn homeDirectory x-LinuxLoginShell gecos description objectClass Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SEARCH RESULT tag=101 err=0 text= Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 fd=13 closed (connection lost)

I suspect that aliases are not handled the same way in hdb and mdb as I am using aliases here and deref=3 in both searches, example:

dn: uid=joe,ou=Account,dc=its,dc=**scom objectClass: alias objectClass: extensibleObject uid: joe aliasedObjectName: uid=joe,ou=Person,dc=its,dc=**scom structuralObjectClass: alias

When using hdb, the alias is dereferenced correctly (nentries=656), when using mdb it seems not to be dereferenced at all (nentries=0).

Maybe there's a parameter around for mdb which I couldn't find in the docs

to prevent this, but if not I consider this as a bug.

There is no parameter. Seems like you've found a bug, please submit the info to the ITS.

...

Regards

Juergen

michael@stroeder.com wrote:

...

It would certainly help if you could examine the issue with pure LDAP search operations preferably with OpenLDAP's ldapsearch command-line tool.

When looking at NSS results too many things can go wrong with other components' configuration.

Ciao, Michael.

Juergen.Sprenger@swisscom.com wrote:

...

Hi,

I have running OpenDLAP 2.4.35 on Gentoo Linux and wanted to make some tests with mdb.

Slapd was running fine with hdb, no problems so far. Then I exported contents via slapcat and switched config to mdb. When slapd started using mdb no users from directory were shown by 'getent passwd':

### hdb part #### # using hdb parameters database hdb dirtyread cachesize 150000 cachefree 100 idlcachesize 450000 dncachesize 100000

# slapadd from backup and run slapd with hdb backend /etc/init.d/unscd stop /etc/init.d/slapd stop rm /var/lib/openldap-data/* rm -rf /etc/openldap/slapd.d/* cp -p /etc/openldap/DB_CONFIG /var/lib/openldap-data/ cp -p /etc/openldap/slapd.conf.hdb /etc/openldap/slapd.conf su ldap -c '/usr/sbin/slapadd -f /etc/openldap/slapd.conf -l odsldap-dev.ldif' /etc/init.d/slapd start /etc/init.d/unscd start slapcat -f /etc/openldap/slapd.conf -b dc=scom | md5sum # 73850f9a3f7ff9d3d1ddb7663cd046**a6 -

getent passwd # all users shown, everything ok

### mdb part #### # using mdb paramters database mdb dbnosync maxsize 2094967296 searchstack 64

# slapadd from backup and run slapd with mdb backend /etc/init.d/unscd stop /etc/init.d/slapd stop rm /var/lib/openldap-data/* rm -rf /etc/openldap/slapd.d/* cp -p /etc/openldap/slapd.conf.mdb /etc/openldap/slapd.conf su ldap -c '/usr/sbin/slapadd -f /etc/openldap/slapd.conf -l odsldap-dev.ldif' /etc/init.d/slapd start /etc/init.d/unscd start slapcat -f /etc/openldap/slapd.conf -b dc=scom | md5sum # 73850f9a3f7ff9d3d1ddb7663cd046**a6 -

<>

...

getent passwd

...

# no users from ldap shown

Am I missing something when setting up and using mdb? Both backends have exactly the same content, and so the results for searches should also be identical.

Regards

Jürgen Sprenger

-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/**project/http://www.openldap.org/project/