- openldap-bugs - openldap.org

Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by hyc＠symas.com 30 Sep '10

30 Sep '10

masarati(a)aero.polimi.it wrote: >> --On Wednesday, September 29, 2010 8:34 PM +0200 masarati(a)aero.polimi.it >> wrote: >> >>>> --On Wednesday, September 29, 2010 12:38 AM +0000 quanah(a)zimbra.com >>>> wrote: >>>> >>>>> It appears to be a problem with the entry cache, which is set to >>>>> 25,000: >>>> >>>> Running with a fix from Howard, the entry cache behaves correctly. >>>> However, slapd still grows at the same rate. >>>> >>>> If I limit to only 10 paged results searches, slapd grows at a rate of >>>> 300MB Virtual and 300MB Resident for every set of 10 paged results >>>> searches >>>> I do concurrently, up until I run slapd out of memory. There's >>>> something >>>> very wrong with paged results searches. >>> >>> Could it be configuration-specific? I tested with a plain configuration >>> resulting from test003; maybe some player in the middle, say, is causing >>> entries to be duplicated and leaked, or read-locks on originals are not >>> released correctly? Can you post a configuration that shows the issue? >> >> Hi Pierangelo, >> >> My testing shows the issue is only really visible with large databases >> that >> return giant result sets. I don't expect you to see it with a small >> database and test003, because the amount of "lost" memory will be a few >> bytes at best. > > If it's something related to bdb's cache size I agree; if it's related to > paged results I'd expect to notice it anyway using valgrind or so. > There's no actual leak, so valgrind won't point anything out. It's simply an issue with the entry cache not running its purge in some cases where it needs to. cn=monitor shows that the entry cache grows far beyond its configured size. Still looking into a proper fix. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by masarati＠aero.polimi.it 30 Sep '10

30 Sep '10

> --On Wednesday, September 29, 2010 8:34 PM +0200 masarati(a)aero.polimi.it > wrote: > >>> --On Wednesday, September 29, 2010 12:38 AM +0000 quanah(a)zimbra.com >>> wrote: >>> >>>> It appears to be a problem with the entry cache, which is set to >>>> 25,000: >>> >>> Running with a fix from Howard, the entry cache behaves correctly. >>> However, slapd still grows at the same rate. >>> >>> If I limit to only 10 paged results searches, slapd grows at a rate of >>> 300MB Virtual and 300MB Resident for every set of 10 paged results >>> searches >>> I do concurrently, up until I run slapd out of memory. There's >>> something >>> very wrong with paged results searches. >> >> Could it be configuration-specific? I tested with a plain configuration >> resulting from test003; maybe some player in the middle, say, is causing >> entries to be duplicated and leaked, or read-locks on originals are not >> released correctly? Can you post a configuration that shows the issue? > > Hi Pierangelo, > > My testing shows the issue is only really visible with large databases > that > return giant result sets. I don't expect you to see it with a small > database and test003, because the amount of "lost" memory will be a few > bytes at best. If it's something related to bdb's cache size I agree; if it's related to paged results I'd expect to notice it anyway using valgrind or so. > My configuration itself is very minimal, but it is in cn=config format. ;) > > dn: olcDatabase={2}hdb > objectClass: olcDatabaseConfig > objectClass: olcHdbConfig > olcDatabase: {2}hdb > olcSuffix: > olcAccess: {0}to attrs=userPassword by anonymous auth by > dn.children="cn=adm > ins,cn=zimbra" write > olcAccess: {1}to dn.subtree="cn=zimbra" by > dn.children="cn=admins,cn=zimbra" > write > olcAccess: {2}to > attrs=zimbraZimletUserProperties,zimbraGalLdapBindPassword,zi > mbraGalLdapBindDn,zimbraAuthTokenKey,zimbraPreAuthKey,zimbraPasswordHistory, > z > imbraIsAdminAccount,zimbraAuthLdapSearchBindPassword by > dn.children="cn=admi > ns,cn=zimbra" write by * none > olcAccess: {3}to attrs=objectclass by dn.children="cn=admins,cn=zimbra" > write > by dn.base="uid=zmpostfix,cn=appaccts,cn=zimbra" read by > dn.base="uid=zmam > avis,cn=appaccts,cn=zimbra" read by users read by * none > olcAccess: {4}to attrs=@amavisAccount by > dn.children="cn=admins,cn=zimbra" > wr > ite by dn.base="uid=zmamavis,cn=appaccts,cn=zimbra" read by * +0 break > olcAccess: {5}to attrs=mail by dn.children="cn=admins,cn=zimbra" write > by > dn > .base="uid=zmamavis,cn=appaccts,cn=zimbra" read by * +0 break > olcAccess: {6}to attrs=zimbraAllowFromAddress by > dn.children="cn=admins,cn=zi > mbra" write by dn.base="uid=zmpostfix,cn=appaccts,cn=zimbra" read by * > none > olcAccess: {7}to filter="(!(zimbraHideInGal=TRUE))" > attrs=cn,co,company,dc,di > splayName,givenName,gn,initials,l,mail,o,ou,physicalDeliveryOfficeName,posta > l > Code,sn,st,street,streetAddress,telephoneNumber,title,uid,homePhone,mobile,p > a > ger by dn.children="cn=admins,cn=zimbra" write by > dn.base="uid=zmpostfix,cn > =appaccts,cn=zimbra" read by users read by * none > olcAccess: {8}to > attrs=zimbraId,zimbraMailAddress,zimbraMailAlias,zimbraMailCa > nonicalAddress,zimbraMailCatchAllAddress,zimbraMailCatchAllCanonicalAddress, > z > imbraMailCatchAllForwardingAddress,zimbraMailDeliveryAddress,zimbraMailForwa > r > dingAddress,zimbraPrefMailForwardingAddress,zimbraMailHost,zimbraMailStatus, > z > imbraMailTransport,zimbraDomainName,zimbraDomainType,zimbraPrefMailLocalDeli > v > eryDisabled by dn.children="cn=admins,cn=zimbra" write by > dn.base="uid=zmpo > stfix,cn=appaccts,cn=zimbra" read by * none > olcAccess: {9}to attrs=entry by dn.children="cn=admins,cn=zimbra" write > by * > read > olcLastMod: TRUE > olcMaxDerefDepth: 15 > olcReadOnly: FALSE > olcRootDN: cn=config > olcSizeLimit: unlimited > olcTimeLimit: unlimited > olcMonitoring: TRUE > olcDbDirectory: /opt/zimbra/data/ldap/hdb/db > olcDbCacheSize: 25000 > olcDbCheckpoint: 64 5 > olcDbConfig: {0}# > olcDbConfig: {1}# Set the database in memory cache size. > olcDbConfig: {2}# > olcDbConfig: {3}set_cachesize 0 52428800 0 > olcDbConfig: {4} > olcDbConfig: {5}# > olcDbConfig: {6}# Set database flags. > olcDbConfig: {7}# Automatically remove log files that are no longer > needed. > olcDbConfig: {8}set_log_config DB_LOG_AUTO_REMOVE > olcDbConfig: {9} > olcDbConfig: {10}# > olcDbConfig: {11}# Set log values. > olcDbConfig: {12}# > olcDbConfig: {13}set_lg_regionmax 262144 > olcDbConfig: {14}set_lg_max 10485760 > olcDbConfig: {15}set_lg_bsize 2097152 > olcDbConfig: {16}set_lg_dir /opt/zimbra/data/ldap/hdb/logs > olcDbConfig: {17}# Increase locks > olcDbConfig:: ezE4fXNldF9sa19tYXhfbG9ja3MJMzAwMA== > olcDbConfig:: ezE5fXNldF9sa19tYXhfb2JqZWN0cwkxNTAw > olcDbConfig:: ezIwfXNldF9sa19tYXhfbG9ja2VycwkxNTAw > olcDbNoSync: FALSE > olcDbDirtyRead: FALSE > olcDbIDLcacheSize: 25000 > olcDbIndex: objectClass eq > olcDbIndex: entryUUID eq > olcDbIndex: entryCSN eq > olcDbIndex: cn pres,eq,sub > olcDbIndex: uid pres,eq > olcDbIndex: zimbraForeignPrincipal eq > olcDbIndex: zimbraYahooId eq > olcDbIndex: zimbraId eq > olcDbIndex: zimbraVirtualHostname eq > olcDbIndex: zimbraVirtualIPAddress eq > olcDbIndex: zimbraMailDeliveryAddress eq,sub > olcDbIndex: zimbraAuthKerberos5Realm eq > olcDbIndex: zimbraMailForwardingAddress eq > olcDbIndex: zimbraMailCatchAllAddress eq,sub > olcDbIndex: zimbraShareInfo sub > olcDbIndex: zimbraMailTransport eq > olcDbIndex: zimbraMailAlias eq,sub > olcDbIndex: zimbraACE sub > olcDbIndex: zimbraDomainName eq,sub > olcDbIndex: mail pres,eq,sub > olcDbIndex: zimbraCalResSite eq,sub > olcDbIndex: givenName pres,eq,sub > olcDbIndex: displayName pres,eq,sub > olcDbIndex: sn pres,eq,sub > olcDbIndex: zimbraCalResRoom eq,sub > olcDbIndex: zimbraCalResCapacity eq > olcDbIndex: zimbraCalResBuilding eq,sub > olcDbIndex: zimbraCalResFloor eq,sub > olcDbLinearIndex: FALSE > olcDbMode: 0600 > olcDbSearchStack: 16 > olcDbShmKey: 0 > olcDbCacheFree: 1000 > olcDbDNcacheSize: 0 > structuralObjectClass: olcHdbConfig > entryUUID: 152ab0a8-333e-102d-8700-d562901af228 > creatorsName: cn=config > createTimestamp: 20081020215916Z > entryCSN: 20081020215916.275992Z#000000#000#000000 > modifiersName: cn=config > modifyTimestamp: 20081020215916Z There's no mention of overlays; I was wondering whether or not it could be related to some adverse interaction, e.g. with sssvlv, which has some paged-results specific code. Can you also post on overlays either global or specific to "olcDatabase={2}hdb"? p.

1 0

ITS#6661 (Was: FW: (6661))
by masarati＠aero.polimi.it 30 Sep '10

30 Sep '10

> Hi Pierangelo, > > I replied to the ticket's list but I forgot to include your address. > > Here is my reply if you care to read it, > > Regards, > > George > > > > -----Original Message----- > From: George Tzanetis > Sent: Thursday, September 30, 2010 10:37 AM > To: 'openldap-its(a)openldap.org' > Subject: (ITS#6661) > > Hi, > > I built openldap using the new code. The rootpw now works, but if a wrong > password in an ldap query, then the ldap query process locks. > > e.g.: > with rootdn: 'cn=root,dc=example,dc=gr' > and rootpw: secret > > -when rootdn and rootpw are correct: > ldapwhoami -h 192.168.6.10 -D 'cn=root,dc=example,dc=gr' -w 'secret' >>dn:cn=root,dc=example,dc=gr > > -when rootdn is wrong: > Ldapwhoami -h 192.168.6.10 -D 'cn=root,dc=example,dc=com' -w 'secret' >>ldap_bind: Invalid credentials (49) > > -when rootdn is correct and rootpw is wrong > Ldapwhoami -h 192.168.6.10 -D 'cn=root,dc=example,dc=com' -w 'secret1' > "NO RESULT, the ldapwhoami locks" > > > Here are the logs of the slapd process: > > > ################################### > #with correct rootdn & rootpw # > ################################### > daemon: activity on 1 descriptor > daemon: activity on: > slap_listener_activate(8): > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 busy >>>> slap_listener(ldap:///) > daemon: activity on 1 descriptor > daemon: activity on: > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > daemon: listen=8, new connection on 23 > daemon: activity on 1 descriptor > daemon: activity on: 23r > daemon: read active on 23 > daemon: added 23r (active) listener=(nil) > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > daemon: activity on 1 descriptor > daemon: activity on: > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > conn=1000 fd=23 ACCEPT from IP=192.168.6.10:47722 (IP=0.0.0.0:389) > connection_get(23) > connection_get(23): got connid=1000 > connection_read(23): checking for input on id=1000 > ber_get_next > ldap_read: want=8, got=8 > ldap_read: want=36, got=36 > ber_get_next: tag 0x30 len 42 contents: > ber_dump: buf=0x1d047ee0 ptr=0x1d047ee0 end=0x1d047f0a len=42 > op tag 0x60, time 1285831215 > ber_get_next > ldap_read: want=8 error=Resource temporarily unavailable > daemon: activity on 1 descriptor > daemon: activity on: > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > conn=1000 op=0 do_bind > ber_scanf fmt ({imt) ber: > ber_dump: buf=0x1d047ee0 ptr=0x1d047ee3 end=0x1d047f0a len=39 > ber_scanf fmt (m}) ber: > ber_dump: buf=0x1d047ee0 ptr=0x1d047f01 end=0x1d047f0a len=9 >>>> dnPrettyNormal: <cn=root,dc=example,dc=gr> > => ldap_bv2dn(cn=root,dc=example,dc=gr,0) > <= ldap_bv2dn(cn=root,dc=example,dc=gr)=0 > => ldap_dn2bv(272) > <= ldap_dn2bv(cn=root,dc=example,dc=gr)=0 > => ldap_dn2bv(272) > <= ldap_dn2bv(cn=root,dc=example,dc=gr)=0 > <<< dnPrettyNormal: <cn=root,dc=example,dc=gr>, <cn=root,dc=example,dc=gr> > conn=1000 op=0 BIND dn="cn=root,dc=example,dc=gr" method=128 > do_bind: version=3 dn="cn=root,dc=example,dc=gr" method=128 > ==> ndb_back_bind: dn: cn=root,dc=example,dc=gr > conn=1000 op=0 BIND dn="cn=root,dc=example,dc=gr" mech=SIMPLE ssf=0 > do_bind: v3 bind: "cn=root,dc=example,dc=gr" to "cn=root,dc=example,dc=gr" > send_ldap_result: conn=1000 op=0 p=3 > send_ldap_result: err=0 matched="" text="" > send_ldap_response: msgid=1 tag=97 err=0 > ber_flush2: 14 bytes to sd 23 > ldap_write: want=14, written=14 > conn=1000 op=0 RESULT tag=97 err=0 text= > daemon: activity on 1 descriptor > daemon: activity on: 23r > daemon: read active on 23 > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > connection_get(23) > connection_get(23): got connid=1000 > connection_read(23): checking for input on id=1000 > ber_get_next > ldap_read: want=8, got=8 > ldap_read: want=24, got=24 > ber_get_next: tag 0x30 len 30 contents: > ber_dump: buf=0x1d045c10 ptr=0x1d045c10 end=0x1d045c2e len=30 > op tag 0x77, time 1285831215 > ber_get_next > ldap_read: want=8 error=Resource temporarily unavailable > daemon: activity on 1 descriptor > daemon: activity on: > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > conn=1000 op=1 do_extended > ber_scanf fmt ({m) ber: > ber_dump: buf=0x1d045c10 ptr=0x1d045c13 end=0x1d045c2e len=27 > conn=1000 op=1 EXT oid=1.3.6.1.4.1.4203.1.11.3 > do_extended: oid=1.3.6.1.4.1.4203.1.11.3 > conn=1000 op=1 WHOAMI > send_ldap_extended: err=0 oid= len=26 > send_ldap_response: msgid=2 tag=120 err=0 > ber_flush2: 42 bytes to sd 23 > ldap_write: want=42, written=42 > conn=1000 op=1 RESULT oid= err=0 text= > daemon: activity on 1 descriptor > daemon: activity on: 23r > daemon: read active on 23 > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > connection_get(23) > connection_get(23): got connid=1000 > connection_read(23): checking for input on id=1000 > ber_get_next > ldap_read: want=8, got=7 > ber_get_next: tag 0x30 len 5 contents: > ber_dump: buf=0x1d045c10 ptr=0x1d045c10 end=0x1d045c15 len=5 > op tag 0x42, time 1285831215 > ber_get_next > ldap_read: want=8, got=0 > > ber_get_next on fd 23 failed errno=0 (Success) > connection_read(23): input error=-2 id=1000, closing. > connection_closing: readying conn=1000 sd=23 for close > daemon: activity on 1 descriptor > daemon: activity on: > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > connection_close: deferring conn=1000 sd=23 > conn=1000 op=2 do_unbind > conn=1000 op=2 UNBIND > connection_resched: attempting closing conn=1000 sd=23 > connection_close: conn=1000 sd=23 > daemon: removing 23 > conn=1000 fd=23 closed > > > ########################################## > #with correct rootdn & incorrect rootpw # > ########################################## > daemon: activity on 1 descriptor > daemon: activity on: > slap_listener_activate(8): > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 busy >>>> slap_listener(ldap:///) > daemon: listen=8, new connection on 23 > daemon: added 23r (active) listener=(nil) > conn=1001 fd=23 ACCEPT from IP=192.168.6.10:47723 (IP=0.0.0.0:389) > daemon: activity on 2 descriptors > daemon: activity on: 23r > daemon: read active on 23 > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL > connection_get(23) > connection_get(23): got connid=1001 > connection_read(23): checking for input on id=1001 > ber_get_next > ldap_read: want=8, got=8 > ldap_read: want=37, got=37 > ber_get_next: tag 0x30 len 43 contents: > ber_dump: buf=0x1d0460b0 ptr=0x1d0460b0 end=0x1d0460db len=43 > op tag 0x60, time 1285831240 > ber_get_next > ldap_read: want=8 error=Resource temporarily unavailable > conn=1001 op=0 do_bind > ber_scanf fmt ({imt) ber: > ber_dump: buf=0x1d0460b0 ptr=0x1d0460b3 end=0x1d0460db len=40 > ber_scanf fmt (m}) ber: > ber_dump: buf=0x1d0460b0 ptr=0x1d0460d1 end=0x1d0460db len=10 >>>> dnPrettyNormal: <cn=root,dc=example,dc=gr> > => ldap_bv2dn(cn=root,dc=example,dc=gr,0) > <= ldap_bv2dn(cn=root,dc=example,dc=gr)=0 > => ldap_dn2bv(272) > <= ldap_dn2bv(cn=root,dc=example,dc=gr)=0 > => ldap_dn2bv(272) > <= ldap_dn2bv(cn=root,dc=example,dc=gr)=0 > <<< dnPrettyNormal: <cn=root,dc=example,dc=gr>, <cn=root,dc=example,dc=gr> > conn=1001 op=0 BIND dn="cn=root,dc=example,dc=gr" method=128 > do_bind: version=3 dn="cn=root,dc=example,dc=gr" method=128 > ==> ndb_back_bind: dn: cn=root,dc=example,dc=gr > daemon: activity on 1 descriptor > daemon: activity on: > daemon: epoll: listen=7 active_threads=0 tvp=NULL > daemon: epoll: listen=8 active_threads=0 tvp=NULL Should be re-fixed now, sorry. Thanks for the report. p.

1 0

(ITS#6662) ldapsearch with slapd-ndb only works when filter is a substring
by gtzanetis＠pylones.gr 30 Sep '10

30 Sep '10

Full_Name: George Tzanetis Version: 2.4.23 stable OS: Red Hat Enterprise 5.5 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (62.169.213.126) It seems that when using slapd-ndb the filters in ldapsearches only work if they are substrings. i.e *text or text* or te*xt for attributes that are not defined as indices. If the attribute is defined as an index then the substring filter does not work, as indicate in the manual. The slapd.conf is as follows: pidfile /usr/local/openldap/var/run/slapd.pid argsfile /usr/local/openldap/var/run/slapd.args ####################################################################### # NDB database definitions ####################################################################### #NDB database defintions database ndb suffix "dc=example,dc=gr" rootdn "cn=root,dc=example,dc=gr" rootpw secret dbconnect 192.168.6.11 dbhost 192.168.6.12 dbport 3306 dbname openldap dbuser ldapUser dbpass "1234" dbconnections 3 dbsocket /tmp/mysql.sock attrblob description index uid ####################################################################### # Monitor Database definitions ####################################################################### database monitor loglevel 5 The ldif of an ou: version: 1 dn: ou=test,dc=example,dc=gr objectClass: top objectClass: organizationalUnit ou: test dn: uid=user1,ou=test,dc=example,dc=gr objectClass: top objectClass: inetOrgPerson objectClass: posixAccount cn: user1 gidNumber: -1 givenName: user1 homeDirectory: * sn: user1 uid: user1 uidNumber: -1 userPassword:: 1234 dn: uid=user2,ou=test,dc=example,dc=gr objectClass: top objectClass: inetOrgPerson objectClass: posixAccount cn: user2 gidNumber: -1 givenName: user2 homeDirectory: * sn: user2 uid: user2 uidNumber: -1 userPassword:: 1234 dn: uid=user3,ou=test,dc=example,dc=gr objectClass: top objectClass: inetOrgPerson objectClass: posixAccount cn: user3 gidNumber: -1 givenName: user3 homeDirectory: * sn: user3 uid: user3 uidNumber: -1 userPassword:: 1234 dn: uid=user4,ou=test,dc=example,dc=gr objectClass: top objectClass: inetOrgPerson objectClass: posixAccount cn: user4 gidNumber: -1 givenName: user4 homeDirectory: * sn: user4 uid: user4 uidNumber: -1 userPassword:: 1234 the ldapsearch queries: -search with specific cn inside the ou: --------------------------------------------------------------------- ldapsearch -h 192.168.132.177 -b 'ou=test,dc=example,dc=gr' -D "cn=root,dc=example,dc=gr" -L -w 'secret' "cn=user1" version: 1 # # LDAPv3 # base <ou=test,dc=example,dc=gr> with scope subtree # filter: cn=user1 # requesting: ALL # # search result # numResponses: 1 --------------------------------------------------------------------- No result but if we search the cn as a substring: --------------------------------------------------------------------- ldapsearch -h 192.168.132.177 -b 'ou=test,dc=example,dc=gr' -D "cn=root,dc=example,dc=gr" -L -w 'secret1' "cn=user1*" version: 1 # # LDAPv3 # base <ou=test,dc=example,dc=gr> with scope subtree # filter: cn=user1* # requesting: ALL # # user1@test, test, example.gr dn: uid=user1@test,ou=test,dc=example,dc=gr objectClass: top objectClass: inetOrgPerson objectClass: posixAccount userPassword:: 1234 sn: user1 cn: user1 uid: user1@test givenName: user1 uidNumber: -1 gidNumber: -1 homeDirectory: * # search result # numResponses: 2 # numEntries: 1 --------------------------------------------------------------------- any substring will give a result i.e. cn=*user1, cn=user1*, cn=us*er1 etc. if we search for cn=user* it will display all entries of the ou as expected. The same behavior exists if we filter using any other attribute with the exception of the objectClass attribute, or with the uid attribute which is indexed. Is this normal? Thank you, George

1 0

(ITS#6626)
by gtzanetis＠pylones.gr 30 Sep '10

30 Sep '10

I wanted to add that I do not find the contextCSN attribute anywhere inside= the database schema that the slapd-ndb backend creates. Maybe that is the reason I get the segmentation fault every time the slapd = process tries to update this attribute?

1 0

(ITS#6661)
by gtzanetis＠pylones.gr 30 Sep '10

30 Sep '10

Hi, I built openldap using the new code. The rootpw now works, but if a wrong p= assword in an ldap query, then the ldap query process locks. e.g.: with rootdn: 'cn=3Droot,dc=3Dexample,dc=3Dgr' and rootpw: secret -when rootdn and rootpw are correct: ldapwhoami -h 192.168.6.10 -D 'cn=3Droot,dc=3Dexample,dc=3Dgr' -w 'secret' >dn:cn=3Droot,dc=3Dexample,dc=3Dgr -when rootdn is wrong: Ldapwhoami -h 192.168.6.10 -D 'cn=3Droot,dc=3Dexample,dc=3Dcom' -w 'secret' >ldap_bind: Invalid credentials (49) -when rootdn is correct and rootpw is wrong Ldapwhoami -h 192.168.6.10 -D 'cn=3Droot,dc=3Dexample,dc=3Dcom' -w 'secret1= ' "NO RESULT, the ldapwhoami locks" Here are the logs of the slapd process: ################################### #with correct rootdn & rootpw # ################################### daemon: activity on 1 descriptor daemon: activity on: slap_listener_activate(8): daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 busy >>> slap_listener(ldap:///) daemon: activity on 1 descriptor daemon: activity on: daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL daemon: listen=3D8, new connection on 23 daemon: activity on 1 descriptor daemon: activity on: 23r daemon: read active on 23 daemon: added 23r (active) listener=3D(nil) daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL daemon: activity on 1 descriptor daemon: activity on: daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL conn=3D1000 fd=3D23 ACCEPT from IP=3D192.168.6.10:47722 (IP=3D0.0.0.0:389) connection_get(23) connection_get(23): got connid=3D1000 connection_read(23): checking for input on id=3D1000 ber_get_next ldap_read: want=3D8, got=3D8 ldap_read: want=3D36, got=3D36 ber_get_next: tag 0x30 len 42 contents: ber_dump: buf=3D0x1d047ee0 ptr=3D0x1d047ee0 end=3D0x1d047f0a len=3D42 op tag 0x60, time 1285831215 ber_get_next ldap_read: want=3D8 error=3DResource temporarily unavailable daemon: activity on 1 descriptor daemon: activity on: daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL conn=3D1000 op=3D0 do_bind ber_scanf fmt ({imt) ber: ber_dump: buf=3D0x1d047ee0 ptr=3D0x1d047ee3 end=3D0x1d047f0a len=3D39 ber_scanf fmt (m}) ber: ber_dump: buf=3D0x1d047ee0 ptr=3D0x1d047f01 end=3D0x1d047f0a len=3D9 >>> dnPrettyNormal: <cn=3Droot,dc=3Dexample,dc=3Dgr> =3D> ldap_bv2dn(cn=3Droot,dc=3Dexample,dc=3Dgr,0) <=3D ldap_bv2dn(cn=3Droot,dc=3Dexample,dc=3Dgr)=3D0 =3D> ldap_dn2bv(272) <=3D ldap_dn2bv(cn=3Droot,dc=3Dexample,dc=3Dgr)=3D0 =3D> ldap_dn2bv(272) <=3D ldap_dn2bv(cn=3Droot,dc=3Dexample,dc=3Dgr)=3D0 <<< dnPrettyNormal: <cn=3Droot,dc=3Dexample,dc=3Dgr>, <cn=3Droot,dc=3Dexamp= le,dc=3Dgr> conn=3D1000 op=3D0 BIND dn=3D"cn=3Droot,dc=3Dexample,dc=3Dgr" method=3D128 do_bind: version=3D3 dn=3D"cn=3Droot,dc=3Dexample,dc=3Dgr" method=3D128 =3D=3D> ndb_back_bind: dn: cn=3Droot,dc=3Dexample,dc=3Dgr conn=3D1000 op=3D0 BIND dn=3D"cn=3Droot,dc=3Dexample,dc=3Dgr" mech=3DSIMPLE= ssf=3D0 do_bind: v3 bind: "cn=3Droot,dc=3Dexample,dc=3Dgr" to "cn=3Droot,dc=3Dexamp= le,dc=3Dgr" send_ldap_result: conn=3D1000 op=3D0 p=3D3 send_ldap_result: err=3D0 matched=3D"" text=3D"" send_ldap_response: msgid=3D1 tag=3D97 err=3D0 ber_flush2: 14 bytes to sd 23 ldap_write: want=3D14, written=3D14 conn=3D1000 op=3D0 RESULT tag=3D97 err=3D0 text=3D daemon: activity on 1 descriptor daemon: activity on: 23r daemon: read active on 23 daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL connection_get(23) connection_get(23): got connid=3D1000 connection_read(23): checking for input on id=3D1000 ber_get_next ldap_read: want=3D8, got=3D8 ldap_read: want=3D24, got=3D24 ber_get_next: tag 0x30 len 30 contents: ber_dump: buf=3D0x1d045c10 ptr=3D0x1d045c10 end=3D0x1d045c2e len=3D30 op tag 0x77, time 1285831215 ber_get_next ldap_read: want=3D8 error=3DResource temporarily unavailable daemon: activity on 1 descriptor daemon: activity on: daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL conn=3D1000 op=3D1 do_extended ber_scanf fmt ({m) ber: ber_dump: buf=3D0x1d045c10 ptr=3D0x1d045c13 end=3D0x1d045c2e len=3D27 conn=3D1000 op=3D1 EXT oid=3D1.3.6.1.4.1.4203.1.11.3 do_extended: oid=3D1.3.6.1.4.1.4203.1.11.3 conn=3D1000 op=3D1 WHOAMI send_ldap_extended: err=3D0 oid=3D len=3D26 send_ldap_response: msgid=3D2 tag=3D120 err=3D0 ber_flush2: 42 bytes to sd 23 ldap_write: want=3D42, written=3D42 conn=3D1000 op=3D1 RESULT oid=3D err=3D0 text=3D daemon: activity on 1 descriptor daemon: activity on: 23r daemon: read active on 23 daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL connection_get(23) connection_get(23): got connid=3D1000 connection_read(23): checking for input on id=3D1000 ber_get_next ldap_read: want=3D8, got=3D7 ber_get_next: tag 0x30 len 5 contents: ber_dump: buf=3D0x1d045c10 ptr=3D0x1d045c10 end=3D0x1d045c15 len=3D5 op tag 0x42, time 1285831215 ber_get_next ldap_read: want=3D8, got=3D0 ber_get_next on fd 23 failed errno=3D0 (Success) connection_read(23): input error=3D-2 id=3D1000, closing. connection_closing: readying conn=3D1000 sd=3D23 for close daemon: activity on 1 descriptor daemon: activity on: daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL connection_close: deferring conn=3D1000 sd=3D23 conn=3D1000 op=3D2 do_unbind conn=3D1000 op=3D2 UNBIND connection_resched: attempting closing conn=3D1000 sd=3D23 connection_close: conn=3D1000 sd=3D23 daemon: removing 23 conn=3D1000 fd=3D23 closed ########################################## #with correct rootdn & incorrect rootpw # ########################################## daemon: activity on 1 descriptor daemon: activity on: slap_listener_activate(8): daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 busy >>> slap_listener(ldap:///) daemon: listen=3D8, new connection on 23 daemon: added 23r (active) listener=3D(nil) conn=3D1001 fd=3D23 ACCEPT from IP=3D192.168.6.10:47723 (IP=3D0.0.0.0:389) daemon: activity on 2 descriptors daemon: activity on: 23r daemon: read active on 23 daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL connection_get(23) connection_get(23): got connid=3D1001 connection_read(23): checking for input on id=3D1001 ber_get_next ldap_read: want=3D8, got=3D8 ldap_read: want=3D37, got=3D37 ber_get_next: tag 0x30 len 43 contents: ber_dump: buf=3D0x1d0460b0 ptr=3D0x1d0460b0 end=3D0x1d0460db len=3D43 op tag 0x60, time 1285831240 ber_get_next ldap_read: want=3D8 error=3DResource temporarily unavailable conn=3D1001 op=3D0 do_bind ber_scanf fmt ({imt) ber: ber_dump: buf=3D0x1d0460b0 ptr=3D0x1d0460b3 end=3D0x1d0460db len=3D40 ber_scanf fmt (m}) ber: ber_dump: buf=3D0x1d0460b0 ptr=3D0x1d0460d1 end=3D0x1d0460db len=3D10 >>> dnPrettyNormal: <cn=3Droot,dc=3Dexample,dc=3Dgr> =3D> ldap_bv2dn(cn=3Droot,dc=3Dexample,dc=3Dgr,0) <=3D ldap_bv2dn(cn=3Droot,dc=3Dexample,dc=3Dgr)=3D0 =3D> ldap_dn2bv(272) <=3D ldap_dn2bv(cn=3Droot,dc=3Dexample,dc=3Dgr)=3D0 =3D> ldap_dn2bv(272) <=3D ldap_dn2bv(cn=3Droot,dc=3Dexample,dc=3Dgr)=3D0 <<< dnPrettyNormal: <cn=3Droot,dc=3Dexample,dc=3Dgr>, <cn=3Droot,dc=3Dexamp= le,dc=3Dgr> conn=3D1001 op=3D0 BIND dn=3D"cn=3Droot,dc=3Dexample,dc=3Dgr" method=3D128 do_bind: version=3D3 dn=3D"cn=3Droot,dc=3Dexample,dc=3Dgr" method=3D128 =3D=3D> ndb_back_bind: dn: cn=3Droot,dc=3Dexample,dc=3Dgr daemon: activity on 1 descriptor daemon: activity on: daemon: epoll: listen=3D7 active_threads=3D0 tvp=3DNULL daemon: epoll: listen=3D8 active_threads=3D0 tvp=3DNULL thanks, George

1 0

Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by quanah＠zimbra.com 29 Sep '10

29 Sep '10

--On Wednesday, September 29, 2010 8:34 PM +0200 masarati(a)aero.polimi.it wrote: >> --On Wednesday, September 29, 2010 12:38 AM +0000 quanah(a)zimbra.com >> wrote: >> >>> It appears to be a problem with the entry cache, which is set to 25,000: >> >> Running with a fix from Howard, the entry cache behaves correctly. >> However, slapd still grows at the same rate. >> >> If I limit to only 10 paged results searches, slapd grows at a rate of >> 300MB Virtual and 300MB Resident for every set of 10 paged results >> searches >> I do concurrently, up until I run slapd out of memory. There's something >> very wrong with paged results searches. > > Could it be configuration-specific? I tested with a plain configuration > resulting from test003; maybe some player in the middle, say, is causing > entries to be duplicated and leaked, or read-locks on originals are not > released correctly? Can you post a configuration that shows the issue? Hi Pierangelo, My testing shows the issue is only really visible with large databases that return giant result sets. I don't expect you to see it with a small database and test003, because the amount of "lost" memory will be a few bytes at best. My configuration itself is very minimal, but it is in cn=config format. ;) dn: olcDatabase={2}hdb objectClass: olcDatabaseConfig objectClass: olcHdbConfig olcDatabase: {2}hdb olcSuffix: olcAccess: {0}to attrs=userPassword by anonymous auth by dn.children="cn=adm ins,cn=zimbra" write olcAccess: {1}to dn.subtree="cn=zimbra" by dn.children="cn=admins,cn=zimbra" write olcAccess: {2}to attrs=zimbraZimletUserProperties,zimbraGalLdapBindPassword,zi mbraGalLdapBindDn,zimbraAuthTokenKey,zimbraPreAuthKey,zimbraPasswordHistory, z imbraIsAdminAccount,zimbraAuthLdapSearchBindPassword by dn.children="cn=admi ns,cn=zimbra" write by * none olcAccess: {3}to attrs=objectclass by dn.children="cn=admins,cn=zimbra" write by dn.base="uid=zmpostfix,cn=appaccts,cn=zimbra" read by dn.base="uid=zmam avis,cn=appaccts,cn=zimbra" read by users read by * none olcAccess: {4}to attrs=@amavisAccount by dn.children="cn=admins,cn=zimbra" wr ite by dn.base="uid=zmamavis,cn=appaccts,cn=zimbra" read by * +0 break olcAccess: {5}to attrs=mail by dn.children="cn=admins,cn=zimbra" write by dn .base="uid=zmamavis,cn=appaccts,cn=zimbra" read by * +0 break olcAccess: {6}to attrs=zimbraAllowFromAddress by dn.children="cn=admins,cn=zi mbra" write by dn.base="uid=zmpostfix,cn=appaccts,cn=zimbra" read by * none olcAccess: {7}to filter="(!(zimbraHideInGal=TRUE))" attrs=cn,co,company,dc,di splayName,givenName,gn,initials,l,mail,o,ou,physicalDeliveryOfficeName,posta l Code,sn,st,street,streetAddress,telephoneNumber,title,uid,homePhone,mobile,p a ger by dn.children="cn=admins,cn=zimbra" write by dn.base="uid=zmpostfix,cn =appaccts,cn=zimbra" read by users read by * none olcAccess: {8}to attrs=zimbraId,zimbraMailAddress,zimbraMailAlias,zimbraMailCa nonicalAddress,zimbraMailCatchAllAddress,zimbraMailCatchAllCanonicalAddress, z imbraMailCatchAllForwardingAddress,zimbraMailDeliveryAddress,zimbraMailForwa r dingAddress,zimbraPrefMailForwardingAddress,zimbraMailHost,zimbraMailStatus, z imbraMailTransport,zimbraDomainName,zimbraDomainType,zimbraPrefMailLocalDeli v eryDisabled by dn.children="cn=admins,cn=zimbra" write by dn.base="uid=zmpo stfix,cn=appaccts,cn=zimbra" read by * none olcAccess: {9}to attrs=entry by dn.children="cn=admins,cn=zimbra" write by * read olcLastMod: TRUE olcMaxDerefDepth: 15 olcReadOnly: FALSE olcRootDN: cn=config olcSizeLimit: unlimited olcTimeLimit: unlimited olcMonitoring: TRUE olcDbDirectory: /opt/zimbra/data/ldap/hdb/db olcDbCacheSize: 25000 olcDbCheckpoint: 64 5 olcDbConfig: {0}# olcDbConfig: {1}# Set the database in memory cache size. olcDbConfig: {2}# olcDbConfig: {3}set_cachesize 0 52428800 0 olcDbConfig: {4} olcDbConfig: {5}# olcDbConfig: {6}# Set database flags. olcDbConfig: {7}# Automatically remove log files that are no longer needed. olcDbConfig: {8}set_log_config DB_LOG_AUTO_REMOVE olcDbConfig: {9} olcDbConfig: {10}# olcDbConfig: {11}# Set log values. olcDbConfig: {12}# olcDbConfig: {13}set_lg_regionmax 262144 olcDbConfig: {14}set_lg_max 10485760 olcDbConfig: {15}set_lg_bsize 2097152 olcDbConfig: {16}set_lg_dir /opt/zimbra/data/ldap/hdb/logs olcDbConfig: {17}# Increase locks olcDbConfig:: ezE4fXNldF9sa19tYXhfbG9ja3MJMzAwMA== olcDbConfig:: ezE5fXNldF9sa19tYXhfb2JqZWN0cwkxNTAw olcDbConfig:: ezIwfXNldF9sa19tYXhfbG9ja2VycwkxNTAw olcDbNoSync: FALSE olcDbDirtyRead: FALSE olcDbIDLcacheSize: 25000 olcDbIndex: objectClass eq olcDbIndex: entryUUID eq olcDbIndex: entryCSN eq olcDbIndex: cn pres,eq,sub olcDbIndex: uid pres,eq olcDbIndex: zimbraForeignPrincipal eq olcDbIndex: zimbraYahooId eq olcDbIndex: zimbraId eq olcDbIndex: zimbraVirtualHostname eq olcDbIndex: zimbraVirtualIPAddress eq olcDbIndex: zimbraMailDeliveryAddress eq,sub olcDbIndex: zimbraAuthKerberos5Realm eq olcDbIndex: zimbraMailForwardingAddress eq olcDbIndex: zimbraMailCatchAllAddress eq,sub olcDbIndex: zimbraShareInfo sub olcDbIndex: zimbraMailTransport eq olcDbIndex: zimbraMailAlias eq,sub olcDbIndex: zimbraACE sub olcDbIndex: zimbraDomainName eq,sub olcDbIndex: mail pres,eq,sub olcDbIndex: zimbraCalResSite eq,sub olcDbIndex: givenName pres,eq,sub olcDbIndex: displayName pres,eq,sub olcDbIndex: sn pres,eq,sub olcDbIndex: zimbraCalResRoom eq,sub olcDbIndex: zimbraCalResCapacity eq olcDbIndex: zimbraCalResBuilding eq,sub olcDbIndex: zimbraCalResFloor eq,sub olcDbLinearIndex: FALSE olcDbMode: 0600 olcDbSearchStack: 16 olcDbShmKey: 0 olcDbCacheFree: 1000 olcDbDNcacheSize: 0 structuralObjectClass: olcHdbConfig entryUUID: 152ab0a8-333e-102d-8700-d562901af228 creatorsName: cn=config createTimestamp: 20081020215916Z entryCSN: 20081020215916.275992Z#000000#000#000000 modifiersName: cn=config modifyTimestamp: 20081020215916Z The actual DB_CONFIG is: zimbra@zre-ldap001:~/data/ldap/hdb/db$ cat DB_CONFIG # # Set the database in memory cache size. # set_cachesize 10 0 0 # # Set database flags. # Automatically remove log files that are no longer needed. set_log_config DB_LOG_AUTO_REMOVE # # Set log values. # set_lg_regionmax 262144 set_lg_max 10485760 set_lg_bsize 2097152 set_lg_dir /opt/zimbra/data/ldap/hdb/logs # Increase locks set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500 -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by masarati＠aero.polimi.it 29 Sep '10

29 Sep '10

> --On Wednesday, September 29, 2010 12:38 AM +0000 quanah(a)zimbra.com wrote: > >> It appears to be a problem with the entry cache, which is set to 25,000: > > Running with a fix from Howard, the entry cache behaves correctly. > However, slapd still grows at the same rate. > > If I limit to only 10 paged results searches, slapd grows at a rate of > 300MB Virtual and 300MB Resident for every set of 10 paged results > searches > I do concurrently, up until I run slapd out of memory. There's something > very wrong with paged results searches. Could it be configuration-specific? I tested with a plain configuration resulting from test003; maybe some player in the middle, say, is causing entries to be duplicated and leaked, or read-locks on originals are not released correctly? Can you post a configuration that shows the issue? p.

1 0

Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by quanah＠zimbra.com 29 Sep '10

29 Sep '10

--On Wednesday, September 29, 2010 12:38 AM +0000 quanah(a)zimbra.com wrote: > It appears to be a problem with the entry cache, which is set to 25,000: Running with a fix from Howard, the entry cache behaves correctly. However, slapd still grows at the same rate. If I limit to only 10 paged results searches, slapd grows at a rate of 300MB Virtual and 300MB Resident for every set of 10 paged results searches I do concurrently, up until I run slapd out of memory. There's something very wrong with paged results searches. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

ITS#6661
by masarati＠aero.polimi.it 29 Sep '10

29 Sep '10

Please try back-ndb/bind.cpp 1.5->1.6 from HEAD's CVS. Thanks for the report. p.

1 0

(ITS#6661) rootpw is not verified with slapd.conf
by gtzanetis＠pylones.gr 29 Sep '10

29 Sep '10

Full_Name: George Tzanetis Version: 2.4.23 stable OS: Red Hat Enterprise 5.5 URL: Submission from: (NULL) (62.169.213.126) I have built openldap 2.4.23 with the back-ndb in 4 machines. I created the slapd.conf as follows: pidfile /usr/local/openldap/var/run/slapd.pid argsfile /usr/local/openldap/var/run/slapd.args ####################################################################### # NDB database definitions ####################################################################### #NDB database defintions database ndb suffix "dc=example,dc=gr" rootdn "cn=root,dc=example,dc=gr" rootpw secret dbconnect 192.168.6.11 dbhost 192.168.6.12 dbport 3306 dbname openldap dbuser ldapUser dbpass "1234" dbconnections 3 dbsocket /tmp/mysql.sock attrblob description index uid ####################################################################### # Monitor Database definitions ####################################################################### database monitor loglevel 5 My problem is that I can authenticate to the ldap with any password for the cn=root,dc=example,dc=gr (rootdn) user, as long as I specify a password. To make it clearer, all the following ldapsearches work: ldapsearch -h 192.168.6.10 -b 'dc=example,dc=gr' -w secret1 -D "cn=root,dc=example,dc=gr" ldapsearch -h 192.168.6.10 -b 'dc=example,dc=gr' -w secret -D "cn=root,dc=example,dc=gr" ldapsearch -h 192.168.6.10 -b 'dc=example,dc=gr' -w sec -D "cn=root,dc=example,dc=gr" ldapsearch -h 192.168.6.10 -b 'dc=example,dc=gr' -w " " -D "cn=root,dc=example,dc=gr" If I do not specify a password, (i.e. -w flag is omitted) I get the message: ldap_bind: Server is unwilling to perform (53) additional info: unauthenticated bind (DN with no password) disallowed In addition if I don input the correct rootdn user, I get the message: ldap_bind: Invalid credentials (49). This behavior exists in all instances of openldap with ndb as back-end. I did some more testing, and I built openldap with the bdb and ndb backends. The issue appears only to the suffix that is stored in the ndb back-end and not to the bdb back-end, so there must be something wrong with the bind operation of the slapd-ndb. Finally, I would like to state that with the slapd-ndb, all the ldapsearches / modifications / deletions are performed correctly, even if the rootpw password is wrong.

1 0

Re: (ITS#6659) logfile directive not logging any log
by masarati＠aero.polimi.it 29 Sep '10

29 Sep '10

hayashis(a)indiana.edu wrote: > --001636d34cf42a20380491547f9e > Content-Type: text/plain; charset=ISO-8859-1 > > Hi. I will use the correct mailing list. > > The place that I found the *wrong* information for logfile directive > (according to Google) was this doc >> http://www.zytrax.com/books/ldap/ch6/#logfile > > Here, it states, > > OpenLDAP logs via syslogd (using LOCAL4) in all cases (see loglevel for > configuration information on streaming syslogd LDAP messages to a separate > file). In addition the logfile directive may be used to create a separate > file containing just LDAP log information. > > Based on this information, I have *perceived* the behavior of slapd as a > BUG.. my apologies. Your opinion is valuable, just like that of anyone. The code is perfectible. However, to discuss opinions and improvements of code, a technical discussion forum is more appropriate that a bug tracking system. I believe that slapd already has provisions to address your needs, and pointed you to what I think are the appropriate sources of information. Changing the way software works (and has been working for decades) requires a real need and consensus. Fixing bugs is a completely different business. p.

1 0

Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by quanah＠zimbra.com 28 Sep '10

28 Sep '10

--On Tuesday, September 28, 2010 11:10 PM +0000 quanah(a)zimbra.com wrote: > --On Tuesday, September 28, 2010 8:04 PM +0200 masarati(a)aero.polimi.it > wrote: > >> Each paged results control request mallocs a small amount of memory on >> the slab which is not explicitly freed. It should not be an issue, >> since it occurs so early in the life of an operation that the slab >> should never be exhausted. In any case, I've committed a fix. Can you >> test and report, please? Thanks, p. > > Hi Pierangelo, > > I'm testing a build with your patch now. In the meantime, you can see a > callgraph generated from tcmalloc's heap profiling tool at > <http://www.stanford.edu/~quanah/pprof.pdf>. I baselined the graph to > start after the first initial searches have been to load the cache, etc. > So you can see that slapd grew by 14.3GB once I started doing the paged > results searches. It appears to be a problem with the entry cache, which is set to 25,000: olcDatabase={2}hdb.ldif:olcDbCacheSize: 25000 yet: olmBDBEntryCache: 209759 from cn=monitor. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by quanah＠zimbra.com 28 Sep '10

28 Sep '10

--On Tuesday, September 28, 2010 8:04 PM +0200 masarati(a)aero.polimi.it wrote: > Each paged results control request mallocs a small amount of memory on the > slab which is not explicitly freed. It should not be an issue, since it > occurs so early in the life of an operation that the slab should never be > exhausted. In any case, I've committed a fix. Can you test and report, > please? Thanks, p. Hi Pierangelo, I'm testing a build with your patch now. In the meantime, you can see a callgraph generated from tcmalloc's heap profiling tool at <http://www.stanford.edu/~quanah/pprof.pdf>. I baselined the graph to start after the first initial searches have been to load the cache, etc. So you can see that slapd grew by 14.3GB once I started doing the paged results searches. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by masarati＠aero.polimi.it 28 Sep '10

28 Sep '10

Each paged results control request mallocs a small amount of memory on the slab which is not explicitly freed. It should not be an issue, since it occurs so early in the life of an operation that the slab should never be exhausted. In any case, I've committed a fix. Can you test and report, please? Thanks, p.

1 0

(ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
by quanah＠zimbra.com 28 Sep '10

28 Sep '10

Full_Name: Quanah Gibson-Mount Version: 2.4.23 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.111.45.108) When using paged result searches with large databases, it quickly becomes apparent that slapd is failing to deallocate memory allocated during those searches once they complete. The example database I am using has 6,326,513 entries, and is some 8.5GB on disk (id2entry.bdb + dn2id.bdb). What has been found is that over time when using multiple paged results searches, slapd's memory continually grows until it eventually consumes all memory on the system. Once slapd is stopped, the allocated memory is freed. This behavior can be seen in a more pronounced fashion by running multiple paged results searches and searches of the full database simultaneously. For example, I start slapd, and preload all the caches by doing 4 full searches of the database. At this point, top reports (V/R/S): 12.2G/10G/8.4G I then start 10 ldap searches dumping the full database (using root dn), and 10 paged result searches that only return some 2.6 million entries. After these all complete, I run this sequence again. Instead of slapd's memory remaining stable (i.e., re-using all the memory it had previously allocated), it instead continues to grow. At the end of just 2 of these runs, we end up at: 22.5G/20G/8.4G Or a 10.3GB growth in virtual memory and 10GB growth in resident memory. As soon as slapd is halted, all of this memory is freed (Making it impossible to track this via valgrind). I'm currently using the heap profiler shipped with tcmalloc to gather further information and will follow up when that is complete. --Quanah

1 0

Re: (ITS#6659) logfile directive not logging any log
by quanah＠zimbra.com 28 Sep '10

28 Sep '10

--On Tuesday, September 28, 2010 12:40 PM -0400 Soichi Hayashi <hayashis(a)indiana.edu> wrote: > Hi. I will use the correct mailing list. > > The place that I found the *wrong* information for logfile directive > (according to Google) was this doc >> http://www.zytrax.com/books/ldap/ch6/#logfile This is why we always suggest you read the project supplied documentation. Zytrax is known to have numerous issues with what they have written. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6659) logfile directive not logging any log
by hayashis＠indiana.edu 28 Sep '10

28 Sep '10

--001636d34cf42a20380491547f9e Content-Type: text/plain; charset=ISO-8859-1 Hi. I will use the correct mailing list. The place that I found the *wrong* information for logfile directive (according to Google) was this doc > http://www.zytrax.com/books/ldap/ch6/#logfile Here, it states, OpenLDAP logs via syslogd (using LOCAL4) in all cases (see loglevel for configuration information on streaming syslogd LDAP messages to a separate file). In addition the logfile directive may be used to create a separate file containing just LDAP log information. Based on this information, I have *perceived* the behavior of slapd as a BUG.. my apologies. On Tue, Sep 28, 2010 at 12:16 PM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Tuesday, September 28, 2010 2:07 PM +0000 hayashis(a)indiana.edu wrote: > >> Thanks. >> >>> Please direct further sofware usage questions to the most appropriate >>> mai= >> >> ling lists >> >> Sorry to keep bugging, but I believe this *is* the most appropriate >> mailing list. > > You are wrong. This list is about bugs in the software. It has already > been noted there is no bug here. Please use the > openldap-technical(a)openldap.org if you have further questions. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > --001636d34cf42a20380491547f9e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi. I will use the correct mailing list. The place that I found the = *wrong* information for logfile directive (according to Google) was this do= c > <a href=3D"http://www.zytrax.com/books/ldap/ch6/#logfile">http://= www.zytrax.com/books/ldap/ch6/#logfile</a> Here, it states, OpenLDAP logs via syslogd (using LOCAL4) in al= l cases (see loglevel for configuration information on streaming syslogd LD= AP messages to a separate file). In addition the logfile directive may be u= sed to create a separate file containing just LDAP log information. <div>Based on this information, I have *perceived* the behavior of slap= d as a BUG.. my apologies.</div><div>=A0</div><div>On Tue, Sep 28, 2010 at = 12:16 PM, Quanah Gibson-Mount <<a href=3D"mailto:quanah@zimbra.com">quan= ah(a)zimbra.com</a>> wrote: > --On Tuesday, September 28, 2010 2:07 PM +0000 <a href=3D"mailto:hayas= his(a)indiana.edu">hayashis(a)indiana.edu</a> wrote: > >> Thanks= . >> >>> Please direct further sofware usage questions= to the most appropriate >>> mai=3D >> >> ling lists >> >&= gt; Sorry to keep bugging, but I believe this *is* the most appropriate = >> mailing list. > > You are wrong. =A0This list is about= bugs in the software. =A0It has already > been noted there is no bug here. =A0Please use the > <a href=3D"= mailto:openldap-technical@openldap.org">openldap-technical(a)openldap.org</a>= if you have further questions. > > --Quanah > > --= > > Quanah Gibson-Mount > Principal Software Engineer &gt= ; Zimbra, Inc > -------------------- > Zimbra :: =A0the leader = in open source messaging and collaboration > </div> --001636d34cf42a20380491547f9e--

1 0

Re: (ITS#6659) logfile directive not logging any log
by quanah＠zimbra.com 28 Sep '10

28 Sep '10

--On Tuesday, September 28, 2010 2:07 PM +0000 hayashis(a)indiana.edu wrote: > Thanks. > >> Please direct further sofware usage questions to the most appropriate >> mai= > ling lists > > Sorry to keep bugging, but I believe this *is* the most appropriate > mailing list. You are wrong. This list is about bugs in the software. It has already been noted there is no bug here. Please use the openldap-technical(a)openldap.org if you have further questions. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6659) logfile directive not logging any log
by masarati＠aero.polimi.it 28 Sep '10

28 Sep '10

> Thanks. > >> Please direct further sofware usage questions to the most appropriate >> mailing lists > > Sorry to keep bugging, but I believe this *is* the most appropriate > mailing list. > > Having to create a different user just so that I can log to a > different files doesn't seem like an optimal solution. Clearly, this > is a lack of necessary feature for slapd where user can not override > the location of the log file. Also, I've re-read the documentation > many times but it seems to suggest that I am supposed to be able to > forward the content of the log (sent to syslog) to a separate file. > Perhaps the documentation needs to be updated? "user" here means "facility" in syslog's nomenclature (don't know why it was called like that originally in slapd). So you don't need to create different users, you need to run slapd with "-l LOCAL0", "-l LOCAL1" etc. to be able to discriminate logs as "local0", "local1" and so in syslog.conf. The documentation of "logfile" I'm reading says logfile <filename> Specify a file for recording debug log messages. By default these messages only go to stderr and are not recorded anywhere else. Specifying a logfile copies messages to both stderr and the logfile. so it explicitly mentions "stderr" and never mentions "syslog". So it clearly refers to logging that goes to "stderr", which is controlled by the "-d" switch, which has nothing to do with syslog as well. I don't think it needs any amendment or clarification, but feel free to suggest any improvement (possibly in the form of a specific ITS with a patch to current documentation sources). p.

1 0

Re: (ITS#6659) logfile directive not logging any log
by hayashis＠indiana.edu 28 Sep '10

28 Sep '10

Thanks. > Please direct further sofware usage questions to the most appropriate mai= ling lists Sorry to keep bugging, but I believe this *is* the most appropriate mailing list. Having to create a different user just so that I can log to a different files doesn't seem like an optimal solution. Clearly, this is a lack of necessary feature for slapd where user can not override the location of the log file. Also, I've re-read the documentation many times but it seems to suggest that I am supposed to be able to forward the content of the log (sent to syslog) to a separate file. Perhaps the documentation needs to be updated? Thanks, Soichi On Tue, Sep 28, 2010 at 9:33 AM, <masarati(a)aero.polimi.it> wrote: > >> I am running multiple BDII servers (listening on different port), and >> I need them to log to different location. If I understand correctly, >> "logfile" directive only works to capture stderr messages (not stats, >> or other logs currently sent to syslog). Since I can't tell syslog to >> log to a different file based on which instance of LDAP server, logs >> from both server goes to a single file in /var/log. >> >> Is there anyway that I can redirect logs from each LDAP server to a >> different locations? > > You can use '-l' to have different instances of slapd log as different > syslog users. =A0See slapd(8) and syslog(8) for further details. > > Please direct further sofware usage questions to the most appropriate > mailing lists (openldap-technical for OpenLDAP software; Syslog-specific > lists for syslog(8) detaild usage). =A0This ITS will be closed. > > p. > >

1 0

Re: (ITS#6659) logfile directive not logging any log
by masarati＠aero.polimi.it 28 Sep '10

28 Sep '10

> I am running multiple BDII servers (listening on different port), and > I need them to log to different location. If I understand correctly, > "logfile" directive only works to capture stderr messages (not stats, > or other logs currently sent to syslog). Since I can't tell syslog to > log to a different file based on which instance of LDAP server, logs > from both server goes to a single file in /var/log. > > Is there anyway that I can redirect logs from each LDAP server to a > different locations? You can use '-l' to have different instances of slapd log as different syslog users. See slapd(8) and syslog(8) for further details. Please direct further sofware usage questions to the most appropriate mailing lists (openldap-technical for OpenLDAP software; Syslog-specific lists for syslog(8) detaild usage). This ITS will be closed. p.

1 0

Re: (ITS#6659) logfile directive not logging any log
by hayashis＠indiana.edu 28 Sep '10

28 Sep '10

Hi. Thank you for your reply! I am running multiple BDII servers (listening on different port), and I need them to log to different location. If I understand correctly, "logfile" directive only works to capture stderr messages (not stats, or other logs currently sent to syslog). Since I can't tell syslog to log to a different file based on which instance of LDAP server, logs from both server goes to a single file in /var/log. Is there anyway that I can redirect logs from each LDAP server to a different locations? On Tue, Sep 28, 2010 at 8:26 AM, <masarati(a)aero.polimi.it> wrote: >> Full_Name: Soichi Hayashi >> Version: 2.3.43-12.el5_5.2 >> OS: RHEL5 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (99.50.225.74) >> >> >> I have following in my slapd.conf >> >> logfile /var/log/ldap.log >> >> When I start slapd, it creates the file specified but the filesize remai= ns >> 0 >> although I see a lot of log sent to syslog. >> >> It looks like the logging through "logfile" directive is not working. > > According to the documentation, this directive collects messages that are > sent to stderr when "-d" is used, so it is not related to syslog. =A0It i= s > equivalent to > > $ slapd -d <level> >>/var/log/ldap.log 2>&1 > > I've checked recent code and it works as expected. =A0Unless this is what > you're expecting, and it does not happen, this ITS will be closed. > > p. > >

1 0

Re: (ITS#6659) logfile directive not logging any log
by masarati＠aero.polimi.it 28 Sep '10

28 Sep '10

> Full_Name: Soichi Hayashi > Version: 2.3.43-12.el5_5.2 > OS: RHEL5 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (99.50.225.74) > > > I have following in my slapd.conf > > logfile /var/log/ldap.log > > When I start slapd, it creates the file specified but the filesize remains > 0 > although I see a lot of log sent to syslog. > > It looks like the logging through "logfile" directive is not working. According to the documentation, this directive collects messages that are sent to stderr when "-d" is used, so it is not related to syslog. It is equivalent to $ slapd -d <level> >>/var/log/ldap.log 2>&1 I've checked recent code and it works as expected. Unless this is what you're expecting, and it does not happen, this ITS will be closed. p.

1 0

modifyTimestamp does not always update
by Jani Salonen 27 Sep '10

27 Sep '10

Seems that modifyTimestamp of groupOfUniqueNames does not always update as it should. We have one application that updates uniqueMembers of groupOfUniqueNames groups (we have about 30000 groups, from 5 to 1000 members each). Another application syncs its sql database based on groupOfUniqueNames member's once an hour, and for speeding things up, that another application first looks from modifyTimeStamp if group has changed within last hour, and if so, it will be synced. But here is the problem: another application's database does not always sync because modifyTimestamp does not always update for some group. Unfortunately this happens quite rarely and I cannot provide exact instructions how to reproduce this, I was not able to find any pattern either. If anyone has any ideas/experiments what is wrong/how to fix this, they are most welcome. Thanks, Jani Salonen

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs