openldap-technical September 2009

openldap-technical@openldap.org

70 participants
72 discussions

translucent ldap, local search timeouts
by theod＠physics.auth.gr 15 Sep '09

15 Sep '09

Hi to all, I'm a newbie with openldap, but I've spent some time reading the online docs/mailing list archives and testing various configurations with no success. I'm trying to configure a translucent ldap server (v2.4.12) in a Fedora core 10 vhost with the following configuration: (local-server, URI:local.xxx.yyy.zzz) -------------- moduleload translucent.la overlay translucent translucent_local uid #also tried _remote uri ldap://remote.xxx.yyy.zzz:389 lastmod off acl-bind bindmethod=simple #idassert-bind bindmethod=simple mode=none #(also tried this) loglevel 4095 access to dn.base="" by * read access to * by * read database bdb suffix "dc=xxx,dc=yyy,dc=zzz" checkpoint 1024 15 rootdn "uid=me,ou=people,dc=xxx,dc=yyy,dc=zzz" rootpw secret directory /var/lib/ldap-proxy index objectClass eq,pres index ou,cn,mail,surname,givenname eq,pres,sub index uidNumber,gidNumber,loginShell eq,pres index uid,memberUid eq,pres,sub index nisMapName,nisMapEntry eq,pres,sub *I have full permissions in my local server but only read permission in the remote ldap (hence the anonymous bind)*. I only want to override the already existing remote entries (but not add new ones). Both servers have very similar structure, and the SAME suffix, as suggested in the manuals. I've created a test user (uid=test,ou=people,dc=xxx,dc=yyy,dc=zzz) in the local server that exists also in the remote one. What happens is: when translucent is disabled i get proper results in both servers with the same ldapsearch command. When translucent is enabled, when i search the local server, the SEARCH HANGS (timeouts)! (remote still works, of course) The LOCAL logs are: (I've included only the suspicious entries) [...] Sep 15 00:43:46 ldap slapd[5029]: slapd startup: initiated. Sep 15 00:43:46 ldap slapd[5029]: ==> translucent_db_open Sep 15 00:43:46 ldap slapd[5029]: backend_startup_one: starting "cn=config" Sep 15 00:43:46 ldap slapd[5029]: config_back_db_open Sep 15 00:43:46 ldap slapd[5029]: config_back_db_open: line 0: warning: cannot assess the validity of the ACL scope within backend naming context [...] Sep 15 00:43:46 ldap slapd[5029]: backend_startup_one: starting "dc=xxx,dc=yyy,dc=zzz" Sep 15 00:43:46 ldap slapd[5029]: bdb_db_open: "dc=xxx,dc=yyy,dc=zzz" Sep 15 00:43:46 ldap slapd[5029]: bdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/ldap-proxy: (2).#012Expect poor performance for suffix "dc=xxx,dc=yyy,dc=zzz". Sep 15 00:43:46 ldap slapd[5029]: bdb_db_open: database "dc=xxx,dc=yyy,dc=zzz": dbenv_open(/var/lib/ldap-proxy). [...] Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 1 descriptor Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on: Sep 15 00:43:54 ldap slapd[5029]: Sep 15 00:43:54 ldap slapd[5029]: slap_listener_activate(7): Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7 busy Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: >>> slap_listener(ldap:///) Sep 15 00:43:54 ldap slapd[5029]: daemon: listen=7, new connection on 14 Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 1 descriptor Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on: Sep 15 00:43:54 ldap slapd[5029]: Sep 15 00:43:54 ldap slapd[5029]: daemon: added 14r (active) listener=(nil) Sep 15 00:43:54 ldap slapd[5029]: conn=0 fd=14 ACCEPT from IP=127.0.0.1:48453 (IP=0.0.0.0:389) Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 2 descriptors Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on: Sep 15 00:43:54 ldap slapd[5029]: 14r Sep 15 00:43:54 ldap slapd[5029]: Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 1 descriptor Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on: Sep 15 00:43:54 ldap slapd[5029]: 14r Sep 15 00:43:54 ldap slapd[5029]: Sep 15 00:43:54 ldap slapd[5029]: daemon: read active on 14 Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: connection_get(14) Sep 15 00:43:54 ldap slapd[5029]: connection_get(14): got connid=0 Sep 15 00:43:54 ldap slapd[5029]: connection_read(14): checking for input on id=0 Sep 15 00:43:54 ldap slapd[5029]: conn=0 op=0 do_bind Sep 15 00:43:54 ldap slapd[5029]: >>> dnPrettyNormal: <> Sep 15 00:43:54 ldap slapd[5029]: <<< dnPrettyNormal: <>, <> Sep 15 00:43:54 ldap slapd[5029]: conn=0 op=0 BIND dn="" method=128 Sep 15 00:43:54 ldap slapd[5029]: do_bind: version=3 dn="" method=128 Sep 15 00:43:54 ldap slapd[5029]: translucent_bind: <> method 128 Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 1 descriptor Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on: Sep 15 00:43:54 ldap slapd[5029]: Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9 active_threads=0 tvp=zero Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10 active_threads=0 tvp=zero [...here i pressed control-c on the ldapsearch...] Sep 15 00:44:00 ldap slapd[5029]: daemon: activity on 1 descriptor Sep 15 00:44:00 ldap slapd[5029]: daemon: activity on: Sep 15 00:44:00 ldap slapd[5029]: 14r Sep 15 00:44:00 ldap slapd[5029]: Sep 15 00:44:00 ldap slapd[5029]: daemon: read active on 14 Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=7 active_threads=0 tvp=zero Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=8 active_threads=0 tvp=zero Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=9 active_threads=0 tvp=zero Sep 15 00:44:00 ldap slapd[5029]: connection_get(14) Sep 15 00:44:00 ldap slapd[5029]: connection_get(14): got connid=0 Sep 15 00:44:00 ldap slapd[5029]: connection_read(14): checking for input on id=0 Sep 15 00:44:00 ldap slapd[5029]: ber_get_next on fd 14 failed errno=0 (Success) Sep 15 00:44:00 ldap slapd[5029]: connection_read(14): input error=-2 id=0, closing. Sep 15 00:44:00 ldap slapd[5029]: connection_closing: readying conn=0 sd=14 for close Sep 15 00:44:00 ldap slapd[5029]: connection_close: conn=0 sd=14 Sep 15 00:44:00 ldap slapd[5029]: daemon: removing 14 Sep 15 00:44:00 ldap slapd[5029]: conn=0 fd=14 closed (connection lost) Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=10 active_threads=0 tvp=zero Sep 15 00:44:00 ldap slapd[5029]: daemon: activity on 1 descriptor Sep 15 00:44:00 ldap slapd[5029]: daemon: activity on: Sep 15 00:44:00 ldap slapd[5029]: Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=7 active_threads=0 tvp=zero Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=8 active_threads=0 tvp=zero Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=9 active_threads=0 tvp=zero Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=10 active_threads=0 tvp=zero The REMOTE logs (from a yesterday's search though...) are the following: Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 fd=59 ACCEPT from IP=local.xxx.yyy.zzz:45311 (IP=0.0.0.0:389) Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 op=0 BIND dn="" method=128 Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 op=0 RESULT tag=97 err=0 text= Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 op=1 UNBIND Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 fd=59 closed [why there is NO SRCH command?? this is why it timeouts...] Please help! I've spent several hours on this problem, and no matter what options I've tried, it is always the same behavior... Best regards, Theodoros

1 0

Re: connections falling short
by Amol Kulkarni 14 Sep '09

14 Sep '09

1 0

connections falling short
by Amol Kulkarni 14 Sep '09

14 Sep '09

2 1

Re: ldapsearch hangs
by Tihomir Culjaga 11 Sep '09

11 Sep '09

What do you mean by: "And setting no cache or idlcachesize" you want me to comment this out ? #cachesize 2500000 #idlcachesize 7500000 cachefree 1000 dncachesize 0 T. On Fri, Sep 11, 2009 at 6:12 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > --On Friday, September 11, 2009 5:25 PM +0200 Tihomir Culjaga < > tculjaga(a)gmail.com> wrote: > > >> I just sow what is going on... >> > > > Please stop top-posting. > > Since you didn't change slapd.conf, you never implemented the dncachesize 0 > setting, which I noted was important to do. Go do that. > > > --Quanah > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration >

2 1

Re: ldapsearch hangs
by Tihomir Culjaga 11 Sep '09

11 Sep '09

Hi Quanah, I moved to OpenLDAP 2.4.18 and patched B DB 4.7.25 with all 4 patches from oracle. I DIDN't change slapd.config at all i reduced the number of entries to a total of 3437278. [root@l01lnp2 ~]# du -c -h /var/lib/ldap/*.bdb 200K /var/lib/ldap/bestMatchPrefix.bdb 982M /var/lib/ldap/dn2id.bdb 2.4G /var/lib/ldap/id2entry.bdb 1.8M /var/lib/ldap/objectClass.bdb 1.2M /var/lib/ldap/originatorPrefixID.bdb 48M /var/lib/ldap/uniqueID.bdb 3.4G total <= interesting ... almost the same as number of entries :) changed DB_CONFIG to cache 7 GB: set_cachesize 7 0 1 set_lg_regionmax 262144 set_lg_bsize 2097152 my system has 10 GB of RAM and the situation now is: [root@l01lnp2 ~]# free total used free shared buffers cached Mem: 10234924 10176544 58380 0 2144 3786596 -/+ buffers/cache: 6387804 3847120 Swap: 4096564 753572 3342992 [root@l01lnp2 ~]# When i'm doing ldapsearch (time ldapsearch -h localhost -x -b ou=bestMatchPrefixList,ou=sipDirektor,dc=ot,dc=hr -D cn=admin,dc=ot,dc=hr -w pero99) before i actuall add anything with ldapadd, the search completes within 40 seconds. slapd process takes 24 - 26% memory. After I add new entries (just 2 more) and perform the same search, it hangs after a while. When it ldapsearch finishes returning entries, i see slapd process memory starts growing .... it is taking almost everything.... reaching 97% ?!?! It is always like this.... the search throws all entries and then waits for some time .. it is almost random 60 seconds - 6 minutes to actually exit. Please can you take a loot to strace logs i've attached in my previous e-mail... as asoon as the ldapsearch stops returning entries i see a lot of jubrish there... Here is slapd process memory growth: *top - 16:42:22 up* 4 days, 1:02, 2 users, load average: 2.13, 0.67, 0.23 Tasks: 119 total, 1 running, 118 sleeping, 0 stopped, 0 zombie Cpu(s): 0.8%us, 0.2%sy, 0.0%ni, 70.0%id, 28.8%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 10234924k total, 10177568k used, 57356k free, 6676k buffers Swap: 4096564k total, 36516k used, 4060048k free, 3603688k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9404 ldap 25 0 13.3g 8.8g 2.8g S 4.0 *89.7 * 1:13.49 slapd * * 1 root 15 0 10344 372 344 S 0.0 0.0 0:01.69 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/0 Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie Cpu(s): 7.2%us, 0.7%sy, 0.0%ni, 67.5%id, 24.3%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 10234924k total, 10177968k used, 56956k free, 6656k buffers Swap: 4096564k total, 36516k used, 4060048k free, 3580356k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9404 ldap 25 0 13.3g 8.9g 2.9g S 30.3 *90.9* 1:16.76 slapd 325 root 10 -5 0 0 0 S 0.7 0.0 5:37.11 kswapd0 8458 root 15 0 0 0 0 D 0.3 0.0 0:02.02 pdflush Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie Cpu(s): 1.0%us, 0.3%sy, 0.0%ni, 72.3%id, 26.1%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 10234924k total, 10180560k used, 54364k free, 6140k buffers Swap: 4096564k total, 36516k used, 4060048k free, 3488164k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9404 ldap 25 0 13.4g 9.3g 3.2g S 4.7 *95.5* 1:28.86 slapd 8458 root 15 0 0 0 0 D 0.7 0.0 0:02.20 pdflush Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie Cpu(s): 0.9%us, 0.4%sy, 0.0%ni, 70.5%id, 28.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 10234924k total, 10177812k used, 57112k free, 3492k buffers Swap: 4096564k total, 36516k used, 4060048k free, 3481476k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9404 ldap 25 0 13.4g 9.4g 3.2g S 4.3* 95.9* 1:30.39 slapd * * 325 root 10 -5 0 0 0 S 0.7 0.0 5:38.08 kswapd0 *top - 16:45:01 up *4 days, 1:05, 2 users, load average: 1.91, 1.40, 0.59 Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie Cpu(s): 3.2%us, 0.2%sy, 0.0%ni, 75.0%id, 21.4%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 10234924k total, 10179744k used, 55180k free, 396k buffers Swap: 4096564k total, 42328k used, 4054236k free, 3473624k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9404 ldap 25 0 13.5g 9.4g 3.3g S 13.6 *96.7* 1:33.44 slapd 9490 root 15 0 0 0 0 S 0.3 0.0 0:00.31 pdflush *top - 16:45:33 up *4 days, 1:05, 2 users, load average: 1.55, 1.36, 0.60 Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie Cpu(s): 2.7%us, 0.2%sy, 0.0%ni, 74.7%id, 22.3%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 10234924k total, 10180100k used, 54824k free, 652k buffers Swap: 4096564k total, 118616k used, 3977948k free, 3521232k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9404 ldap 25 0 13.5g 9.4g 3.3g S 10.6 *96.6* 1:37.36 slapd 325 root 10 -5 0 0 0 S 0.3 0.0 5:38.63 kswapd0 This looks to me as a memory leak bug to me. Tihomir. On Thu, Sep 10, 2009 at 9:37 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > --On Thursday, September 10, 2009 8:56 PM +0200 Tihomir Culjaga < > tculjaga(a)gmail.com> wrote: > > So, the situation is that i have 2 ldif files i'm recreating the database >> from. >> >> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f >> /usr/local/etc/openldap/slapd.conf >> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f >> /usr/local/etc/openldap/slapd.conf >> > > I would suggest you just make these a single file, so all the work can be > done at one time. > > I tried to re-index with /usr/local/libexec/slapindex -f >> /usr/local/etc/openldap/slapd.conf -v >> restart slapd process, restart the machine ... it is always the same >> issue. >> > > Nothing here indicates a problem with your indices. Running slapindex > repeatedly is a waste of your time. > > [root@l01lnp2 traces]# /usr/local/libexec/slapd -V >> @(#) $OpenLDAP: slapd 2.4.16 (Sep 9 2009 14:39:44) $ >> root@l01lnp2:/home/tculjaga/openldap-2.4.16/servers/slapd >> > > I would strongly urge you to upgrade to 2.4.18 (for reasons I will note > further down) > > > [root@l01lnp2 traces]# /usr/local/BerkeleyDB.4.7/bin/db_stat -V >> Berkeley DB 4.7.25: (May 15, 2008) - unpached! >> > > You need to rebuild BDB 4.7.25 with the 4 patches from Oracle. There are > known issues when running BDB 4.7 without them. > > [root@l01lnp2 traces]# du -c -h /var/lib/ldap/*.bdb >> 200K /var/lib/ldap/bestMatchPrefix.bdb >> 3.8G /var/lib/ldap/dn2id.bdb >> 6.2G /var/lib/ldap/id2entry.bdb >> 1.8M /var/lib/ldap/objectClass.bdb >> 1.2M /var/lib/ldap/originatorPrefixID.bdb >> 48M /var/lib/ldap/uniqueID.bdb >> 10G total >> > > Since your database is a total of 10 GB in size, for slapadd to work at > optimum efficiency, you need at least 10GB of cache for your DB_CONFIG file. > Unfortunately, you only have 10GB of RAM. Essentially, your system is > under powered for your database size. > > > > [tculjaga@l01lnp2 ~]$ cat ot.ldif | grep -c "dn: " >> 101588 >> [tculjaga@l01lnp2 ~]$ cat l01sipdir1.ldif | grep -c "dn: " >> 9994864 >> [tculjaga@l01lnp2 ~]$ >> > > So you have 10,096,452 entries total. > > [root@l01lnp2 traces]# cat /var/lib/ldap/DB_CONFIG | grep -v "#" >> >> set_cachesize 0 3221225472 1 >> set_lg_regionmax 262144 >> set_lg_bsize 2097152 >> > > You only have a 3GB DB cachesize configured here. Expect things to perform > sub optimally. It would have been easier to set this by going > > set_cachesize 3 0 1 > > Which would have the same effect, since the first number is the number of > gigabytes to allocate. > > Please find attached slapd.conf >> > > Ok, so the relevant bits from here are: > > cachesize 2500000 > idlcachesize 7500000 > cachefree 1000 > > Which means you have a cachesize of 2.5 million, an idlcachesize of 7.5 > million, and (with OL 2.4.16) a dncachesize of 5 million. > > I would highly advise you upgrade to OpenLDAP 2.4.18, and change the > slapd.conf settings to: > > dncachesize 0 (which means unlimited). > > And setting no cache or idlcachesize, and fixing your DB_CONFIG. But you > also need to buy a substantial amount of RAM for a DB of this size. :P I > would advise you upgrade to at least 32GB total. Then you can more > optimally tune the system. > > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration >

2 2

What could cause a ber_flush errno=11?
by Oliver Henriot 11 Sep '09

11 Sep '09

Dear list users, I am getting the following errors on my Openldap servers : ber_flush failed errno=11 reason="Resource temporarily unavailable" The same error occurs indifferently on master and replica servers. I'm trying to work out what might be causing this. I presume it's a network issue. What I'm not sure of is if it is when then servers try to communicate or if it is something else. One possible other cause is when the server tries to write the data to its database, as the filesystem is hosted on an iscsi NAS, but then the error would presumably be with iscsi and not openldap. My servers are all Openldap 2.3.43-3.el5 on CentOS 5.3, using bdb for the database and syncrepl for replication. I tried to find info about ber_flush and its various error codes but I haven't been very successful so far. Do you have any clues? Thanks. Oliver

3 4

Re: Reg Adding New Attributes
by Asimananda Mohanty 11 Sep '09

11 Sep '09

Hi, Thanks for the reply. I have to define my own schema so as to use the new attribute and it should be *similar* to the schema which is used to create new user like the one I mentioned in my first mail. Also the methods that I found through google suggest to do some modification to the file slapd.conf where as in my case, I don't have any slapd.conf file at all. Please comment. Regards Asimananda On Thu, Sep 10, 2009 at 10:10 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > --On Thursday, September 10, 2009 9:33 AM -0700 Quanah Gibson-Mount < > quanah(a)zimbra.com> wrote: > > --On Thursday, September 10, 2009 3:20 PM +0530 Asimananda Mohanty >> <asimananda.mohanty(a)gmail.com> wrote: >> >> Hi All, >>> >>> >>> I have installed OpenLdap and PHPLdapAdmin. >>> >>> >>> I have added a user which objectclass is : >>> >>> >>> objectClass inetOrgPerson posixAccount shadowAccount >>> >>> >>> I need to add a new attribute which will be application specific >>> (application which will need LDAP authentication and based on the >>> attribute value, it will provide service to the user). >>> >>> >>> When I tried to do the same by adding attribute and value to ldif files >>> and then adding the ldif file, it showed error: >>> >>> >>> ldap_add: Undefined attribute type (17) additional info: AppAttribute1: >>> attribute type undefined. >>> >> >> >> What is AppAttribute1? It's not a part of any of those schema. >> > > Or more to the point -- Have you defined your own custom schema, with that > attribute and whatever other custom attributes you need, with an objectClass > holding them? > > I suggest reading over <http://www.openldap.org/faq/data/cache/219.html> > > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration >

1 2

Problem with bdb_dn2id
by koitoer 10 Sep '09

10 Sep '09

Hello everybody, I have the next problem. When I tried to add new records to my ldap server , this take a long time, and never do it. First of all I install openldap from sources with those commands env CPPFLAGS="-I/usr/local/include -I/usr/local/BerkeleyDB.4.7/include -I/usr/local/ssl/include/openssl" LDFLAGS="-L/usr/local/lib -L/usr/local/BerkeleyDB.4.7/lib -L/usr/local/ssl/lib -R/usr/local/lib -R/usr/local/BerkeleyDB.4.7/lib -R/usr/local/ssl/lib" LD_LIBRARY_PATH="/usr/local/BerkeleyDB.4.7/lib" ./configure --with-tls --enable-slurpd --enable-crypt --enable-syslog --enable-ldap --enable-ppolicy --enable-sql --enable-dynamic --enable-modules --enable-backends=mod --enable-overlays=mod --prefix=/etc make depend make make test make install And no error appears. So then when I run this command, openldap takes long time, and never response or do something koitoerlp:/etc/etc/openldap# ldapadd -a -x -W -D "cn=manager,dc=koitoerldap,dc=com" -f koitoerldap.ldif Enter LDAP Password: < = LDAP dont send any message or error only wait for the answer but it never comes. The koitoerldap.ldif is : dn: ou=Mounts,dc=koitoerldap,dc=com ou: Mounts objectClass: top objectClass: organizationalUnit dn: ou=Networks,dc=koitoerldap,dc=com ou: Networks objectClass: top objectClass: organizationalUnit dn: ou=People,dc=koitoerldap,dc=com ou: People objectClass: top objectClass: organizationalUnit In the debug mode of ldap I see this before the ldapadd command when i run this command /etc/libexec/slapd -d 1 start @(#) $OpenLDAP: slapd 2.4.16 (Sep 7 2009 16:27:31) $ root@koitoerlp:/usr/src/openldap-2.4.16/servers/slapd ldap_pvt_gethostbyname_a: host=koitoerlp, r=0 daemon_init: listen on ldap:/// daemon_init: 1 listeners to open... ldap_url_parse_ext(ldap:///) daemon: listener initialized ldap:/// daemon_init: 2 listeners opened ldap_create slapd init: initiated server. slap_sasl_init: initialized! bdb_back_initialize: initialize BDB backend bdb_back_initialize: Berkeley DB 4.7.25: (May 15, 2008) hdb_back_initialize: initialize HDB backend hdb_back_initialize: Berkeley DB 4.7.25: (May 15, 2008) ==>sql_back_initialize() <==sql_back_initialize() bdb_db_init: Initializing BDB database >>> dnPrettyNormal: <dc=koitoerldap,dc=com> <<< dnPrettyNormal: <dc=koitoerldap,dc=com>, <dc=koitoerldap,dc=com> >>> dnPrettyNormal: <cn=Manager,dc=koitoerldap,dc=com> <<< dnPrettyNormal: <cn=Manager,dc=koitoerldap,dc=com>, <cn=manager,dc=koitoerldap,dc=com> >>> dnNormalize: <cn=Subschema> <<< dnNormalize: <cn=subschema> matching_rule_use_init ??????????????? Ommited some lines slapd startup: initiated. backend_startup_one: starting "cn=config" config_back_db_open config_build_entry: "cn=config" config_build_entry: "cn=schema" config_build_entry: "cn={0}core" config_build_entry: "cn={1}cosine" config_build_entry: "cn={2}nis" config_build_entry: "cn={3}inetorgperson" config_build_entry: "olcDatabase={-1}frontend" config_build_entry: "olcDatabase={0}config" config_build_entry: "olcDatabase={1}bdb" backend_startup_one: starting "dc=koitoerldap,dc=com" bdb_db_open: database "dc=koitoerldap,dc=com": unclean shutdown detected; attempting recovery. bdb_db_open: database "dc=koitoerldap,dc=com": dbenv_open(/etc/var/openldap-data). bdb_monitor_db_open: monitoring disabled; configure monitor database to enable slapd starting When I launch the ldapadd command my debug mode ldap shows slapd starting slap_listener_activate(8): >>> slap_listener(ldap:///) connection_get(12): got connid=0 connection_read(12): checking for input on id=0 ber_get_next ber_get_next: tag 0x30 len 51 contents: ber_get_next conn=0 op=0 do_bind ber_scanf fmt ({imt) ber: ber_scanf fmt (m}) ber: >>> dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com> <<< dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>, <cn=manager,dc=koitoerldap,dc=com> do_bind: version=3 dn="cn=manager,dc=koitoerldap,dc=com" method=128 do_bind: v3 bind: "cn=manager,dc=koitoerldap,dc=com" to "cn=manager,dc=koitoerldap,dc=com" send_ldap_result: conn=0 op=0 p=3 send_ldap_response: msgid=1 tag=97 err=0 ber_flush2: 14 bytes to sd 12 connection_get(12): got connid=0 connection_read(12): checking for input on id=0 ber_get_next ber_get_next: tag 0x30 len 98 contents: ber_get_next conn=0 op=1 do_add ber_scanf fmt ({m) ber: ber_scanf fmt ({m{W}}) ber: ber_scanf fmt ({m{W}}) ber: ber_scanf fmt (}) ber: >>> dnPrettyNormal: <ou=Mounts,dc=koitoerldap,dc=com> <<< dnPrettyNormal: <ou=Mounts,dc=koitoerldap,dc=com>, <ou=mounts,dc=koitoerldap,dc=com> bdb_dn2entry("ou=mounts,dc=koitoerldap,dc=com") => bdb_dn2id("dc=koitoerldap,dc=com") <= bdb_dn2id: got id=0x1 => bdb_dn2id("ou=mounts,dc=koitoerldap,dc=com") And never past this command, to finish I have to kill openldap process, I think is a problem maybe in db but im not sure. Maybe in my installation, I tried to change some parameters in the slapd.conf but it doesnt work. In some case openldap insert the first record in the ldif file, but in the second this actions comes again, and no more record will be insert. My ldap.conf , slapd.conf and DB.CONFIG HOST 127.0.0.1 BASE dc=koitoerldap,dc=com koitoerlp:/etc/etc/openldap# cat slapd.conf # # See slapd.conf(5) for details on configuration options. # This file should NOT be world readable. # include /etc/etc/openldap/schema/core.schema include /etc/etc/openldap/schema/cosine.schema include /etc/etc/openldap/schema/nis.schema include /etc/etc/openldap/schema/inetorgperson.schema pidfile /etc/var/run/slapd.pid argsfile /etc/var/run/slapd.args # Load dynamic backend modules: # modulepath /etc/libexec/openldap moduleload back_bdb.la # moduleload back_hdb.la # moduleload back_ldap.la ####################################################################### # BDB database definitions ####################################################################### database bdb suffix "dc=koitoerldap,dc=com" rootdn "cn=Manager,dc=koitoerldap,dc=com" # Cleartext passwords, especially for the rootdn, should # be avoid. See slappasswd(8) and slapd.conf(5) for details. # Use of strong authentication encouraged. rootpw {SSHA}HO3g6J/KgbIUQGsanP8ld9hrEyPNhfKs # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /etc/var/openldap-data # Indices to maintain index objectClass eq cat /etc/var/openldap-data/DB_CONFIG # one 0.25 GB cache set_cachesize 0 268435456 1 # Data Directory #set_data_dir db # Transaction Log settings set_lg_regionmax 262144 set_lg_bsize 2097152 #set_lg_dir logs finally when I tried to login in phpopenldapadmin, I have the same trouble when I click in the login button, this message appears in the openldap debug mode slapd starting slap_listener_activate(8): >>> slap_listener(ldap:///) connection_get(12): got connid=0 connection_read(12): checking for input on id=0 ber_get_next ber_get_next: tag 0x30 len 51 contents: ber_get_next conn=0 op=0 do_bind ber_scanf fmt ({imt) ber: ber_scanf fmt (m}) ber: >>> dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com> <<< dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>, <cn=manager,dc=koitoerldap,dc=com> do_bind: version=3 dn="cn=manager,dc=koitoerldap,dc=com" method=128 do_bind: v3 bind: "cn=manager,dc=koitoerldap,dc=com" to "cn=manager,dc=koitoerldap,dc=com" send_ldap_result: conn=0 op=0 p=3 send_ldap_response: msgid=1 tag=97 err=0 ber_flush2: 14 bytes to sd 12 connection_get(12): got connid=0 connection_read(12): checking for input on id=0 ber_get_next ber_get_next: tag 0x30 len 5 contents: ber_get_next conn=0 op=1 do_unbind connection_close: conn=0 sd=12 slap_listener_activate(8): >>> slap_listener(ldap:///) connection_get(12): got connid=1 connection_read(12): checking for input on id=1 ber_get_next ber_get_next: tag 0x30 len 51 contents: ber_get_next conn=1 op=0 do_bind ber_scanf fmt ({imt) ber: ber_scanf fmt (m}) ber: >>> dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com> <<< dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>, <cn=manager,dc=koitoerldap,dc=com> do_bind: version=3 dn="cn=manager,dc=koitoerldap,dc=com" method=128 do_bind: v3 bind: "cn=manager,dc=koitoerldap,dc=com" to "cn=manager,dc=koitoerldap,dc=com" send_ldap_result: conn=1 op=0 p=3 send_ldap_response: msgid=1 tag=97 err=0 ber_flush2: 14 bytes to sd 12 connection_get(12): got connid=1 connection_read(12): checking for input on id=1 ber_get_next ber_get_next: tag 0x30 len 67 contents: ber_get_next conn=1 op=1 do_search ber_scanf fmt ({miiiib) ber: >>> dnPrettyNormal: <dc=koitoerldap,dc=com> <<< dnPrettyNormal: <dc=koitoerldap,dc=com>, <dc=koitoerldap,dc=com> ber_scanf fmt (m) ber: ber_scanf fmt ({M}}) ber: => bdb_search bdb_dn2entry("dc=koitoerldap,dc=com") => bdb_dn2id("dc=koitoerldap,dc=com") <= bdb_dn2id: got id=0x1 entry_decode: "dc=koitoerldap,dc=com" <= entry_decode(dc=koitoerldap,dc=com) => bdb_dn2id_children("dc=koitoerldap,dc=com") <= bdb_dn2id_children("dc=koitoerldap,dc=com"): (0) => send_search_entry: conn 1 dn="dc=koitoerldap,dc=com" ber_flush2: 514 bytes to sd 12 <= send_search_entry: conn 1 exit. send_ldap_result: conn=1 op=1 p=3 send_ldap_response: msgid=2 tag=101 err=0 ber_flush2: 14 bytes to sd 12 connection_get(12): got connid=1 connection_read(12): checking for input on id=1 ber_get_next ber_get_next: tag 0x30 len 538 contents: ber_get_next conn=1 op=2 do_search ber_scanf fmt ({miiiib) ber: >>> dnPrettyNormal: <> <<< dnPrettyNormal: <>, <> ber_scanf fmt (m) ber: ber_scanf fmt ({M}}) ber: => send_search_entry: conn 1 dn="" ber_flush2: 778 bytes to sd 12 <= send_search_entry: conn 1 exit. send_ldap_result: conn=1 op=2 p=3 send_ldap_response: msgid=3 tag=101 err=0 ber_flush2: 14 bytes to sd 12 connection_get(12): got connid=1 connection_read(12): checking for input on id=1 ber_get_next ber_get_next: tag 0x30 len 78 contents: ber_get_next conn=1 op=3 do_search ber_scanf fmt ({miiiib) ber: >>> dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com> <<< dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>, <cn=manager,dc=koitoerldap,dc=com> ber_scanf fmt (m) ber: ber_scanf fmt ({M}}) ber: => bdb_search bdb_dn2entry("cn=manager,dc=koitoerldap,dc=com") => bdb_dn2id("cn=manager,dc=koitoerldap,dc=com") And again neves past this point, I dont know why, maybe I install bad openldap and need more parameters. Please dont say me install with apt-get , I want to make this from source, but a few time ago I install with apt and this error not appear, but in this case I want to make without apt. Any suggestion, opinion or help will be useful and graceful. In advance thanks a lot.

2 4

Re: ldapsearch hangs
by Tihomir Culjaga 10 Sep '09

10 Sep '09

nice link... thnaks! Will let you know the results. T. On Thu, Sep 10, 2009 at 10:10 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > --On Thursday, September 10, 2009 10:07 PM +0200 Tihomir Culjaga < > tculjaga(a)gmail.com> wrote: > > wow, Quanah, >> >> that's the best answer i ever got... >> >> I'm gong to move to 2.4.18 and try to tune up the machine... regarding >> more RAM ... i can get it but not tha fast. >> >> what can i expect with a total of 10 GB ?... >> >> I plan to set set_cachesize 7 0 1 >> >> what is the relation between cachesize (in slapd.conf) and set_cachesize >> (in DB_CONFIG)? >> > > They aren't related. cachesize (in slapd.conf) is for the entry cache. > set_cachesize is for the BDB backing cache. > > set_cachesize =>how much memory i wiill cache >> cachesize => number of entries i will keep within the cache >> >> how much one entry take ? >> >> Lets say if i use 7 GB of cache, what cachesize should i put? >> > > Hard to say. ;) A bit of the BDB cache will likely be swapped out. > > You may also want to read: > > <http://wiki.zimbra.com/index.php?title=OpenLDAP_Performance_Tuning_6.0> > > It has some zimbra specific commands in it, but it should be fairly trivial > to see how it maps to your openldap install. I'd also particularly read > over the bit about using a shared memory key for the BDB cache once you go > beyond 8GB cachesize (which you will do after you get more RAM. ;) ). > > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration >

1 0

Re: ldapsearch hangs
by Tihomir Culjaga 10 Sep '09

10 Sep '09

wow, Quanah, thats the best answer i ever got... I'm gong to move to 2.4.18 and try to tune up the machine... regarding more RAM ... i can get it but not tha fast. what can i expect with a total of 10 GB ?... I plan to set set_cachesize 7 0 1 what is the relation between cachesize (in slapd.conf) and set_cachesize (in DB_CONFIG)? set_cachesize =>how much memory i wiill cache cachesize => number of entries i will keep within the cache how much one entry take ? Lets say if i use 7 GB of cache, what cachesize should i put? T. On Thu, Sep 10, 2009 at 9:37 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > --On Thursday, September 10, 2009 8:56 PM +0200 Tihomir Culjaga < > tculjaga(a)gmail.com> wrote: > > So, the situation is that i have 2 ldif files i'm recreating the database >> from. >> >> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f >> /usr/local/etc/openldap/slapd.conf >> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f >> /usr/local/etc/openldap/slapd.conf >> > > I would suggest you just make these a single file, so all the work can be > done at one time. > > I tried to re-index with /usr/local/libexec/slapindex -f >> /usr/local/etc/openldap/slapd.conf -v >> restart slapd process, restart the machine ... it is always the same >> issue. >> > > Nothing here indicates a problem with your indices. Running slapindex > repeatedly is a waste of your time. > > [root@l01lnp2 traces]# /usr/local/libexec/slapd -V >> @(#) $OpenLDAP: slapd 2.4.16 (Sep 9 2009 14:39:44) $ >> root@l01lnp2:/home/tculjaga/openldap-2.4.16/servers/slapd >> > > I would strongly urge you to upgrade to 2.4.18 (for reasons I will note > further down) > > > [root@l01lnp2 traces]# /usr/local/BerkeleyDB.4.7/bin/db_stat -V >> Berkeley DB 4.7.25: (May 15, 2008) - unpached! >> > > You need to rebuild BDB 4.7.25 with the 4 patches from Oracle. There are > known issues when running BDB 4.7 without them. > > [root@l01lnp2 traces]# du -c -h /var/lib/ldap/*.bdb >> 200K /var/lib/ldap/bestMatchPrefix.bdb >> 3.8G /var/lib/ldap/dn2id.bdb >> 6.2G /var/lib/ldap/id2entry.bdb >> 1.8M /var/lib/ldap/objectClass.bdb >> 1.2M /var/lib/ldap/originatorPrefixID.bdb >> 48M /var/lib/ldap/uniqueID.bdb >> 10G total >> > > Since your database is a total of 10 GB in size, for slapadd to work at > optimum efficiency, you need at least 10GB of cache for your DB_CONFIG file. > Unfortunately, you only have 10GB of RAM. Essentially, your system is > under powered for your database size. > > > > [tculjaga@l01lnp2 ~]$ cat ot.ldif | grep -c "dn: " >> 101588 >> [tculjaga@l01lnp2 ~]$ cat l01sipdir1.ldif | grep -c "dn: " >> 9994864 >> [tculjaga@l01lnp2 ~]$ >> > > So you have 10,096,452 entries total. > > [root@l01lnp2 traces]# cat /var/lib/ldap/DB_CONFIG | grep -v "#" >> >> set_cachesize 0 3221225472 1 >> set_lg_regionmax 262144 >> set_lg_bsize 2097152 >> > > You only have a 3GB DB cachesize configured here. Expect things to perform > sub optimally. It would have been easier to set this by going > > set_cachesize 3 0 1 > > Which would have the same effect, since the first number is the number of > gigabytes to allocate. > > Please find attached slapd.conf >> > > Ok, so the relevant bits from here are: > > cachesize 2500000 > idlcachesize 7500000 > cachefree 1000 > > Which means you have a cachesize of 2.5 million, an idlcachesize of 7.5 > million, and (with OL 2.4.16) a dncachesize of 5 million. > > I would highly advise you upgrade to OpenLDAP 2.4.18, and change the > slapd.conf settings to: > > dncachesize 0 (which means unlimited). > > And setting no cache or idlcachesize, and fixing your DB_CONFIG. But you > also need to buy a substantial amount of RAM for a DB of this size. :P I > would advise you upgrade to at least 32GB total. Then you can more > optimally tune the system. > > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration >

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical September 2009