translucent ldap, local search timeouts
by theod＠physics.auth.gr
Hi to all,
I'm a newbie with openldap, but I've spent some time reading the
online docs/mailing list archives and testing various configurations
with no success.
I'm trying to configure a translucent ldap server (v2.4.12) in a
Fedora core 10 vhost with the following configuration:
(local-server, URI:local.xxx.yyy.zzz)
--------------
moduleload translucent.la
overlay translucent
translucent_local uid #also tried _remote
uri ldap://remote.xxx.yyy.zzz:389
lastmod off
acl-bind bindmethod=simple
#idassert-bind bindmethod=simple mode=none #(also tried this)
loglevel 4095
access to dn.base="" by * read
access to * by * read
database bdb
suffix "dc=xxx,dc=yyy,dc=zzz"
checkpoint 1024 15
rootdn "uid=me,ou=people,dc=xxx,dc=yyy,dc=zzz"
rootpw secret
directory /var/lib/ldap-proxy
index objectClass eq,pres
index ou,cn,mail,surname,givenname eq,pres,sub
index uidNumber,gidNumber,loginShell eq,pres
index uid,memberUid eq,pres,sub
index nisMapName,nisMapEntry eq,pres,sub
*I have full permissions in my local server but only read permission
in the remote ldap (hence the anonymous bind)*. I only want to
override the already existing remote entries (but not add new ones).
Both servers have very similar structure, and the SAME suffix, as
suggested in the manuals. I've created a test user
(uid=test,ou=people,dc=xxx,dc=yyy,dc=zzz) in the local server that
exists also in the remote one.
What happens is: when translucent is disabled i get proper results in
both servers with the same ldapsearch command.
When translucent is enabled, when i search the local server, the
SEARCH HANGS (timeouts)! (remote still works, of course)
The LOCAL logs are: (I've included only the suspicious entries)
[...]
Sep 15 00:43:46 ldap slapd[5029]: slapd startup: initiated.
Sep 15 00:43:46 ldap slapd[5029]: ==> translucent_db_open
Sep 15 00:43:46 ldap slapd[5029]: backend_startup_one: starting "cn=config"
Sep 15 00:43:46 ldap slapd[5029]: config_back_db_open
Sep 15 00:43:46 ldap slapd[5029]: config_back_db_open: line 0:
warning: cannot assess the validity of the ACL scope within backend
naming context
[...]
Sep 15 00:43:46 ldap slapd[5029]: backend_startup_one: starting
"dc=xxx,dc=yyy,dc=zzz"
Sep 15 00:43:46 ldap slapd[5029]: bdb_db_open: "dc=xxx,dc=yyy,dc=zzz"
Sep 15 00:43:46 ldap slapd[5029]: bdb_db_open: warning - no DB_CONFIG
file found in directory /var/lib/ldap-proxy: (2).#012Expect poor
performance for suffix "dc=xxx,dc=yyy,dc=zzz".
Sep 15 00:43:46 ldap slapd[5029]: bdb_db_open: database
"dc=xxx,dc=yyy,dc=zzz": dbenv_open(/var/lib/ldap-proxy).
[...]
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 1 descriptor
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on:
Sep 15 00:43:54 ldap slapd[5029]:
Sep 15 00:43:54 ldap slapd[5029]: slap_listener_activate(7):
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7 busy
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: >>> slap_listener(ldap:///)
Sep 15 00:43:54 ldap slapd[5029]: daemon: listen=7, new connection on 14
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 1 descriptor
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on:
Sep 15 00:43:54 ldap slapd[5029]:
Sep 15 00:43:54 ldap slapd[5029]: daemon: added 14r (active) listener=(nil)
Sep 15 00:43:54 ldap slapd[5029]: conn=0 fd=14 ACCEPT from
IP=127.0.0.1:48453 (IP=0.0.0.0:389)
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 2 descriptors
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on:
Sep 15 00:43:54 ldap slapd[5029]: 14r
Sep 15 00:43:54 ldap slapd[5029]:
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 1 descriptor
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on:
Sep 15 00:43:54 ldap slapd[5029]: 14r
Sep 15 00:43:54 ldap slapd[5029]:
Sep 15 00:43:54 ldap slapd[5029]: daemon: read active on 14
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: connection_get(14)
Sep 15 00:43:54 ldap slapd[5029]: connection_get(14): got connid=0
Sep 15 00:43:54 ldap slapd[5029]: connection_read(14): checking for
input on id=0
Sep 15 00:43:54 ldap slapd[5029]: conn=0 op=0 do_bind
Sep 15 00:43:54 ldap slapd[5029]: >>> dnPrettyNormal: <>
Sep 15 00:43:54 ldap slapd[5029]: <<< dnPrettyNormal: <>, <>
Sep 15 00:43:54 ldap slapd[5029]: conn=0 op=0 BIND dn="" method=128
Sep 15 00:43:54 ldap slapd[5029]: do_bind: version=3 dn="" method=128
Sep 15 00:43:54 ldap slapd[5029]: translucent_bind: <> method 128
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on 1 descriptor
Sep 15 00:43:54 ldap slapd[5029]: daemon: activity on:
Sep 15 00:43:54 ldap slapd[5029]:
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=7
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=8
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=9
active_threads=0 tvp=zero
Sep 15 00:43:54 ldap slapd[5029]: daemon: epoll: listen=10
active_threads=0 tvp=zero
[...here i pressed control-c on the ldapsearch...]
Sep 15 00:44:00 ldap slapd[5029]: daemon: activity on 1 descriptor
Sep 15 00:44:00 ldap slapd[5029]: daemon: activity on:
Sep 15 00:44:00 ldap slapd[5029]: 14r
Sep 15 00:44:00 ldap slapd[5029]:
Sep 15 00:44:00 ldap slapd[5029]: daemon: read active on 14
Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=7
active_threads=0 tvp=zero
Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=8
active_threads=0 tvp=zero
Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=9
active_threads=0 tvp=zero
Sep 15 00:44:00 ldap slapd[5029]: connection_get(14)
Sep 15 00:44:00 ldap slapd[5029]: connection_get(14): got connid=0
Sep 15 00:44:00 ldap slapd[5029]: connection_read(14): checking for
input on id=0
Sep 15 00:44:00 ldap slapd[5029]: ber_get_next on fd 14 failed errno=0
(Success)
Sep 15 00:44:00 ldap slapd[5029]: connection_read(14): input error=-2
id=0, closing.
Sep 15 00:44:00 ldap slapd[5029]: connection_closing: readying conn=0
sd=14 for close
Sep 15 00:44:00 ldap slapd[5029]: connection_close: conn=0 sd=14
Sep 15 00:44:00 ldap slapd[5029]: daemon: removing 14
Sep 15 00:44:00 ldap slapd[5029]: conn=0 fd=14 closed (connection lost)
Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=10
active_threads=0 tvp=zero
Sep 15 00:44:00 ldap slapd[5029]: daemon: activity on 1 descriptor
Sep 15 00:44:00 ldap slapd[5029]: daemon: activity on:
Sep 15 00:44:00 ldap slapd[5029]:
Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=7
active_threads=0 tvp=zero
Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=8
active_threads=0 tvp=zero
Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=9
active_threads=0 tvp=zero
Sep 15 00:44:00 ldap slapd[5029]: daemon: epoll: listen=10
active_threads=0 tvp=zero
The REMOTE logs (from a yesterday's search though...) are the following:
Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 fd=59 ACCEPT from
IP=local.xxx.yyy.zzz:45311 (IP=0.0.0.0:389)
Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 op=0 BIND dn="" method=128
Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 op=0 RESULT tag=97 err=0 text=
Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 op=1 UNBIND
Sep 14 12:18:32 srv004 slapd[2080]: conn=46075 fd=59 closed
[why there is NO SRCH command?? this is why it timeouts...]
Please help! I've spent several hours on this problem, and no matter
what options I've tried, it is always the same behavior...
Best regards,
Theodoros
12 years, 4 months
Re: ldapsearch hangs
by Tihomir Culjaga
What do you mean by: "And setting no cache or idlcachesize"
you want me to comment this out ?
#cachesize 2500000
#idlcachesize 7500000
cachefree 1000
dncachesize 0
T.
On Fri, Sep 11, 2009 at 6:12 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> --On Friday, September 11, 2009 5:25 PM +0200 Tihomir Culjaga <
> tculjaga(a)gmail.com> wrote:
>
>
>> I just sow what is going on...
>>
>
>
> Please stop top-posting.
>
> Since you didn't change slapd.conf, you never implemented the dncachesize 0
> setting, which I noted was important to do. Go do that.
>
>
> --Quanah
>
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
12 years, 4 months
Re: ldapsearch hangs
by Tihomir Culjaga
Hi Quanah,
I moved to OpenLDAP 2.4.18 and patched B DB 4.7.25 with all 4 patches from
oracle.
I DIDN't change slapd.config at all
i reduced the number of entries to a total of 3437278.
[root@l01lnp2 ~]# du -c -h /var/lib/ldap/*.bdb
200K /var/lib/ldap/bestMatchPrefix.bdb
982M /var/lib/ldap/dn2id.bdb
2.4G /var/lib/ldap/id2entry.bdb
1.8M /var/lib/ldap/objectClass.bdb
1.2M /var/lib/ldap/originatorPrefixID.bdb
48M /var/lib/ldap/uniqueID.bdb
3.4G total <= interesting ... almost the same as number of entries :)
changed DB_CONFIG to cache 7 GB:
set_cachesize 7 0 1
set_lg_regionmax 262144
set_lg_bsize 2097152
my system has 10 GB of RAM and the situation now is:
[root@l01lnp2 ~]# free
total used free shared buffers cached
Mem: 10234924 10176544 58380 0 2144 3786596
-/+ buffers/cache: 6387804 3847120
Swap: 4096564 753572 3342992
[root@l01lnp2 ~]#
When i'm doing ldapsearch (time ldapsearch -h localhost -x -b
ou=bestMatchPrefixList,ou=sipDirektor,dc=ot,dc=hr -D cn=admin,dc=ot,dc=hr
-w pero99) before i actuall add anything with ldapadd, the search completes
within 40 seconds. slapd process takes 24 - 26% memory.
After I add new entries (just 2 more) and perform the same search, it hangs
after a while. When it ldapsearch finishes returning entries, i see slapd
process memory starts growing .... it is taking almost everything....
reaching 97% ?!?!
It is always like this.... the search throws all entries and then waits for
some time .. it is almost random 60 seconds - 6 minutes to actually exit.
Please can you take a loot to strace logs i've attached in my previous
e-mail... as asoon as the ldapsearch stops returning entries i see a lot of
jubrish there...
Here is slapd process memory growth:
*top - 16:42:22 up* 4 days, 1:02, 2 users, load average: 2.13, 0.67, 0.23
Tasks: 119 total, 1 running, 118 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.8%us, 0.2%sy, 0.0%ni, 70.0%id, 28.8%wa, 0.0%hi, 0.2%si,
0.0%st
Mem: 10234924k total, 10177568k used, 57356k free, 6676k buffers
Swap: 4096564k total, 36516k used, 4060048k free, 3603688k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9404 ldap 25 0 13.3g 8.8g 2.8g S 4.0 *89.7 * 1:13.49 slapd * *
1 root 15 0 10344 372 344 S 0.0 0.0 0:01.69 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/0
Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.2%us, 0.7%sy, 0.0%ni, 67.5%id, 24.3%wa, 0.0%hi, 0.3%si,
0.0%st
Mem: 10234924k total, 10177968k used, 56956k free, 6656k buffers
Swap: 4096564k total, 36516k used, 4060048k free, 3580356k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9404 ldap 25 0 13.3g 8.9g 2.9g S 30.3 *90.9* 1:16.76 slapd
325 root 10 -5 0 0 0 S 0.7 0.0 5:37.11 kswapd0
8458 root 15 0 0 0 0 D 0.3 0.0 0:02.02 pdflush
Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.0%us, 0.3%sy, 0.0%ni, 72.3%id, 26.1%wa, 0.0%hi, 0.3%si,
0.0%st
Mem: 10234924k total, 10180560k used, 54364k free, 6140k buffers
Swap: 4096564k total, 36516k used, 4060048k free, 3488164k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9404 ldap 25 0 13.4g 9.3g 3.2g S 4.7 *95.5* 1:28.86 slapd
8458 root 15 0 0 0 0 D 0.7 0.0 0:02.20 pdflush
Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.9%us, 0.4%sy, 0.0%ni, 70.5%id, 28.0%wa, 0.0%hi, 0.2%si,
0.0%st
Mem: 10234924k total, 10177812k used, 57112k free, 3492k buffers
Swap: 4096564k total, 36516k used, 4060048k free, 3481476k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9404 ldap 25 0 13.4g 9.4g 3.2g S 4.3* 95.9* 1:30.39 slapd * *
325 root 10 -5 0 0 0 S 0.7 0.0 5:38.08 kswapd0
*top - 16:45:01 up *4 days, 1:05, 2 users, load average: 1.91, 1.40, 0.59
Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.2%us, 0.2%sy, 0.0%ni, 75.0%id, 21.4%wa, 0.0%hi, 0.1%si,
0.0%st
Mem: 10234924k total, 10179744k used, 55180k free, 396k buffers
Swap: 4096564k total, 42328k used, 4054236k free, 3473624k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9404 ldap 25 0 13.5g 9.4g 3.3g S 13.6 *96.7* 1:33.44 slapd
9490 root 15 0 0 0 0 S 0.3 0.0 0:00.31 pdflush
*top - 16:45:33 up *4 days, 1:05, 2 users, load average: 1.55, 1.36, 0.60
Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.7%us, 0.2%sy, 0.0%ni, 74.7%id, 22.3%wa, 0.0%hi, 0.1%si,
0.0%st
Mem: 10234924k total, 10180100k used, 54824k free, 652k buffers
Swap: 4096564k total, 118616k used, 3977948k free, 3521232k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9404 ldap 25 0 13.5g 9.4g 3.3g S 10.6 *96.6* 1:37.36 slapd
325 root 10 -5 0 0 0 S 0.3 0.0 5:38.63 kswapd0
This looks to me as a memory leak bug to me.
Tihomir.
On Thu, Sep 10, 2009 at 9:37 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> --On Thursday, September 10, 2009 8:56 PM +0200 Tihomir Culjaga <
> tculjaga(a)gmail.com> wrote:
>
> So, the situation is that i have 2 ldif files i'm recreating the database
>> from.
>>
>> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f
>> /usr/local/etc/openldap/slapd.conf
>> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f
>> /usr/local/etc/openldap/slapd.conf
>>
>
> I would suggest you just make these a single file, so all the work can be
> done at one time.
>
> I tried to re-index with /usr/local/libexec/slapindex -f
>> /usr/local/etc/openldap/slapd.conf -v
>> restart slapd process, restart the machine ... it is always the same
>> issue.
>>
>
> Nothing here indicates a problem with your indices. Running slapindex
> repeatedly is a waste of your time.
>
> [root@l01lnp2 traces]# /usr/local/libexec/slapd -V
>> @(#) $OpenLDAP: slapd 2.4.16 (Sep 9 2009 14:39:44) $
>> root@l01lnp2:/home/tculjaga/openldap-2.4.16/servers/slapd
>>
>
> I would strongly urge you to upgrade to 2.4.18 (for reasons I will note
> further down)
>
>
> [root@l01lnp2 traces]# /usr/local/BerkeleyDB.4.7/bin/db_stat -V
>> Berkeley DB 4.7.25: (May 15, 2008) - unpached!
>>
>
> You need to rebuild BDB 4.7.25 with the 4 patches from Oracle. There are
> known issues when running BDB 4.7 without them.
>
> [root@l01lnp2 traces]# du -c -h /var/lib/ldap/*.bdb
>> 200K /var/lib/ldap/bestMatchPrefix.bdb
>> 3.8G /var/lib/ldap/dn2id.bdb
>> 6.2G /var/lib/ldap/id2entry.bdb
>> 1.8M /var/lib/ldap/objectClass.bdb
>> 1.2M /var/lib/ldap/originatorPrefixID.bdb
>> 48M /var/lib/ldap/uniqueID.bdb
>> 10G total
>>
>
> Since your database is a total of 10 GB in size, for slapadd to work at
> optimum efficiency, you need at least 10GB of cache for your DB_CONFIG file.
> Unfortunately, you only have 10GB of RAM. Essentially, your system is
> under powered for your database size.
>
>
>
> [tculjaga@l01lnp2 ~]$ cat ot.ldif | grep -c "dn: "
>> 101588
>> [tculjaga@l01lnp2 ~]$ cat l01sipdir1.ldif | grep -c "dn: "
>> 9994864
>> [tculjaga@l01lnp2 ~]$
>>
>
> So you have 10,096,452 entries total.
>
> [root@l01lnp2 traces]# cat /var/lib/ldap/DB_CONFIG | grep -v "#"
>>
>> set_cachesize 0 3221225472 1
>> set_lg_regionmax 262144
>> set_lg_bsize 2097152
>>
>
> You only have a 3GB DB cachesize configured here. Expect things to perform
> sub optimally. It would have been easier to set this by going
>
> set_cachesize 3 0 1
>
> Which would have the same effect, since the first number is the number of
> gigabytes to allocate.
>
> Please find attached slapd.conf
>>
>
> Ok, so the relevant bits from here are:
>
> cachesize 2500000
> idlcachesize 7500000
> cachefree 1000
>
> Which means you have a cachesize of 2.5 million, an idlcachesize of 7.5
> million, and (with OL 2.4.16) a dncachesize of 5 million.
>
> I would highly advise you upgrade to OpenLDAP 2.4.18, and change the
> slapd.conf settings to:
>
> dncachesize 0 (which means unlimited).
>
> And setting no cache or idlcachesize, and fixing your DB_CONFIG. But you
> also need to buy a substantial amount of RAM for a DB of this size. :P I
> would advise you upgrade to at least 32GB total. Then you can more
> optimally tune the system.
>
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
12 years, 4 months
What could cause a ber_flush errno=11?
by Oliver Henriot
Dear list users,
I am getting the following errors on my Openldap servers :
ber_flush failed errno=11 reason="Resource temporarily unavailable"
The same error occurs indifferently on master and replica servers.
I'm trying to work out what might be causing this. I presume it's a
network issue. What I'm not sure of is if it is when then servers try to
communicate or if it is something else. One possible other cause is when
the server tries to write the data to its database, as the filesystem is
hosted on an iscsi NAS, but then the error would presumably be with
iscsi and not openldap.
My servers are all Openldap 2.3.43-3.el5 on CentOS 5.3, using bdb for
the database and syncrepl for replication.
I tried to find info about ber_flush and its various error codes but I
haven't been very successful so far.
Do you have any clues?
Thanks.
Oliver
12 years, 4 months
Re: Reg Adding New Attributes
by Asimananda Mohanty
Hi,
Thanks for the reply.
I have to define my own schema so as to use the new attribute and it should
be *similar* to the schema which is used to create new user like the one I
mentioned in my first mail.
Also the methods that I found through google suggest to do some modification
to the file slapd.conf where as in my case, I don't have any slapd.conf file
at all.
Please comment.
Regards
Asimananda
On Thu, Sep 10, 2009 at 10:10 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> --On Thursday, September 10, 2009 9:33 AM -0700 Quanah Gibson-Mount <
> quanah(a)zimbra.com> wrote:
>
> --On Thursday, September 10, 2009 3:20 PM +0530 Asimananda Mohanty
>> <asimananda.mohanty(a)gmail.com> wrote:
>>
>> Hi All,
>>>
>>>
>>> I have installed OpenLdap and PHPLdapAdmin.
>>>
>>>
>>> I have added a user which objectclass is :
>>>
>>>
>>> objectClass inetOrgPerson posixAccount shadowAccount
>>>
>>>
>>> I need to add a new attribute which will be application specific
>>> (application which will need LDAP authentication and based on the
>>> attribute value, it will provide service to the user).
>>>
>>>
>>> When I tried to do the same by adding attribute and value to ldif files
>>> and then adding the ldif file, it showed error:
>>>
>>>
>>> ldap_add: Undefined attribute type (17) additional info: AppAttribute1:
>>> attribute type undefined.
>>>
>>
>>
>> What is AppAttribute1? It's not a part of any of those schema.
>>
>
> Or more to the point -- Have you defined your own custom schema, with that
> attribute and whatever other custom attributes you need, with an objectClass
> holding them?
>
> I suggest reading over <http://www.openldap.org/faq/data/cache/219.html>
>
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
12 years, 4 months
Problem with bdb_dn2id
by koitoer
Hello everybody,
I have the next problem. When I tried to add new records to my ldap server ,
this take a long time, and never do it.
First of all I install openldap from sources with those commands
env CPPFLAGS="-I/usr/local/include -I/usr/local/BerkeleyDB.4.7/include
-I/usr/local/ssl/include/openssl" LDFLAGS="-L/usr/local/lib
-L/usr/local/BerkeleyDB.4.7/lib -L/usr/local/ssl/lib -R/usr/local/lib
-R/usr/local/BerkeleyDB.4.7/lib -R/usr/local/ssl/lib"
LD_LIBRARY_PATH="/usr/local/BerkeleyDB.4.7/lib"
./configure --with-tls --enable-slurpd --enable-crypt --enable-syslog
--enable-ldap --enable-ppolicy --enable-sql --enable-dynamic
--enable-modules --enable-backends=mod --enable-overlays=mod --prefix=/etc
make depend
make
make test
make install
And no error appears.
So then when I run this command, openldap takes long time, and never
response or do something
koitoerlp:/etc/etc/openldap# ldapadd -a -x -W -D
"cn=manager,dc=koitoerldap,dc=com" -f koitoerldap.ldif
Enter LDAP Password: < = LDAP dont send any message or error only wait
for the answer but it never comes.
The koitoerldap.ldif is :
dn: ou=Mounts,dc=koitoerldap,dc=com
ou: Mounts
objectClass: top
objectClass: organizationalUnit
dn: ou=Networks,dc=koitoerldap,dc=com
ou: Networks
objectClass: top
objectClass: organizationalUnit
dn: ou=People,dc=koitoerldap,dc=com
ou: People
objectClass: top
objectClass: organizationalUnit
In the debug mode of ldap I see this before the ldapadd command when i run
this command
/etc/libexec/slapd -d 1 start
@(#) $OpenLDAP: slapd 2.4.16 (Sep 7 2009 16:27:31) $
root@koitoerlp:/usr/src/openldap-2.4.16/servers/slapd
ldap_pvt_gethostbyname_a: host=koitoerlp, r=0
daemon_init: listen on ldap:///
daemon_init: 1 listeners to open...
ldap_url_parse_ext(ldap:///)
daemon: listener initialized ldap:///
daemon_init: 2 listeners opened
ldap_create
slapd init: initiated server.
slap_sasl_init: initialized!
bdb_back_initialize: initialize BDB backend
bdb_back_initialize: Berkeley DB 4.7.25: (May 15, 2008)
hdb_back_initialize: initialize HDB backend
hdb_back_initialize: Berkeley DB 4.7.25: (May 15, 2008)
==>sql_back_initialize()
<==sql_back_initialize()
bdb_db_init: Initializing BDB database
>>> dnPrettyNormal: <dc=koitoerldap,dc=com>
<<< dnPrettyNormal: <dc=koitoerldap,dc=com>, <dc=koitoerldap,dc=com>
>>> dnPrettyNormal: <cn=Manager,dc=koitoerldap,dc=com>
<<< dnPrettyNormal: <cn=Manager,dc=koitoerldap,dc=com>,
<cn=manager,dc=koitoerldap,dc=com>
>>> dnNormalize: <cn=Subschema>
<<< dnNormalize: <cn=subschema>
matching_rule_use_init
??????????????? Ommited some lines
slapd startup: initiated.
backend_startup_one: starting "cn=config"
config_back_db_open
config_build_entry: "cn=config"
config_build_entry: "cn=schema"
config_build_entry: "cn={0}core"
config_build_entry: "cn={1}cosine"
config_build_entry: "cn={2}nis"
config_build_entry: "cn={3}inetorgperson"
config_build_entry: "olcDatabase={-1}frontend"
config_build_entry: "olcDatabase={0}config"
config_build_entry: "olcDatabase={1}bdb"
backend_startup_one: starting "dc=koitoerldap,dc=com"
bdb_db_open: database "dc=koitoerldap,dc=com": unclean shutdown detected;
attempting recovery.
bdb_db_open: database "dc=koitoerldap,dc=com":
dbenv_open(/etc/var/openldap-data).
bdb_monitor_db_open: monitoring disabled; configure monitor database to
enable
slapd starting
When I launch the ldapadd command my debug mode ldap shows
slapd starting
slap_listener_activate(8):
>>> slap_listener(ldap:///)
connection_get(12): got connid=0
connection_read(12): checking for input on id=0
ber_get_next
ber_get_next: tag 0x30 len 51 contents:
ber_get_next
conn=0 op=0 do_bind
ber_scanf fmt ({imt) ber:
ber_scanf fmt (m}) ber:
>>> dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>
<<< dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>,
<cn=manager,dc=koitoerldap,dc=com>
do_bind: version=3 dn="cn=manager,dc=koitoerldap,dc=com" method=128
do_bind: v3 bind: "cn=manager,dc=koitoerldap,dc=com" to
"cn=manager,dc=koitoerldap,dc=com"
send_ldap_result: conn=0 op=0 p=3
send_ldap_response: msgid=1 tag=97 err=0
ber_flush2: 14 bytes to sd 12
connection_get(12): got connid=0
connection_read(12): checking for input on id=0
ber_get_next
ber_get_next: tag 0x30 len 98 contents:
ber_get_next
conn=0 op=1 do_add
ber_scanf fmt ({m) ber:
ber_scanf fmt ({m{W}}) ber:
ber_scanf fmt ({m{W}}) ber:
ber_scanf fmt (}) ber:
>>> dnPrettyNormal: <ou=Mounts,dc=koitoerldap,dc=com>
<<< dnPrettyNormal: <ou=Mounts,dc=koitoerldap,dc=com>,
<ou=mounts,dc=koitoerldap,dc=com>
bdb_dn2entry("ou=mounts,dc=koitoerldap,dc=com")
=> bdb_dn2id("dc=koitoerldap,dc=com")
<= bdb_dn2id: got id=0x1
=> bdb_dn2id("ou=mounts,dc=koitoerldap,dc=com")
And never past this command, to finish I have to kill openldap process, I
think is a problem maybe in db but im not sure.
Maybe in my installation, I tried to change some parameters in the
slapd.conf but it doesnt work.
In some case openldap insert the first record in the ldif file, but in the
second this actions comes again, and no more record will be insert.
My ldap.conf , slapd.conf and DB.CONFIG
HOST 127.0.0.1
BASE dc=koitoerldap,dc=com
koitoerlp:/etc/etc/openldap# cat slapd.conf
#
# See slapd.conf(5) for details on configuration options.
# This file should NOT be world readable.
#
include /etc/etc/openldap/schema/core.schema
include /etc/etc/openldap/schema/cosine.schema
include /etc/etc/openldap/schema/nis.schema
include /etc/etc/openldap/schema/inetorgperson.schema
pidfile /etc/var/run/slapd.pid
argsfile /etc/var/run/slapd.args
# Load dynamic backend modules:
# modulepath /etc/libexec/openldap
moduleload back_bdb.la
# moduleload back_hdb.la
# moduleload back_ldap.la
#######################################################################
# BDB database definitions
#######################################################################
database bdb
suffix "dc=koitoerldap,dc=com"
rootdn "cn=Manager,dc=koitoerldap,dc=com"
# Cleartext passwords, especially for the rootdn, should
# be avoid. See slappasswd(8) and slapd.conf(5) for details.
# Use of strong authentication encouraged.
rootpw {SSHA}HO3g6J/KgbIUQGsanP8ld9hrEyPNhfKs
# The database directory MUST exist prior to running slapd AND
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory /etc/var/openldap-data
# Indices to maintain
index objectClass eq
cat /etc/var/openldap-data/DB_CONFIG
# one 0.25 GB cache
set_cachesize 0 268435456 1
# Data Directory
#set_data_dir db
# Transaction Log settings
set_lg_regionmax 262144
set_lg_bsize 2097152
#set_lg_dir logs
finally when I tried to login in phpopenldapadmin, I have the same trouble
when I click in the login button, this message appears in the openldap debug
mode
slapd starting
slap_listener_activate(8):
>>> slap_listener(ldap:///)
connection_get(12): got connid=0
connection_read(12): checking for input on id=0
ber_get_next
ber_get_next: tag 0x30 len 51 contents:
ber_get_next
conn=0 op=0 do_bind
ber_scanf fmt ({imt) ber:
ber_scanf fmt (m}) ber:
>>> dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>
<<< dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>,
<cn=manager,dc=koitoerldap,dc=com>
do_bind: version=3 dn="cn=manager,dc=koitoerldap,dc=com" method=128
do_bind: v3 bind: "cn=manager,dc=koitoerldap,dc=com" to
"cn=manager,dc=koitoerldap,dc=com"
send_ldap_result: conn=0 op=0 p=3
send_ldap_response: msgid=1 tag=97 err=0
ber_flush2: 14 bytes to sd 12
connection_get(12): got connid=0
connection_read(12): checking for input on id=0
ber_get_next
ber_get_next: tag 0x30 len 5 contents:
ber_get_next
conn=0 op=1 do_unbind
connection_close: conn=0 sd=12
slap_listener_activate(8):
>>> slap_listener(ldap:///)
connection_get(12): got connid=1
connection_read(12): checking for input on id=1
ber_get_next
ber_get_next: tag 0x30 len 51 contents:
ber_get_next
conn=1 op=0 do_bind
ber_scanf fmt ({imt) ber:
ber_scanf fmt (m}) ber:
>>> dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>
<<< dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>,
<cn=manager,dc=koitoerldap,dc=com>
do_bind: version=3 dn="cn=manager,dc=koitoerldap,dc=com" method=128
do_bind: v3 bind: "cn=manager,dc=koitoerldap,dc=com" to
"cn=manager,dc=koitoerldap,dc=com"
send_ldap_result: conn=1 op=0 p=3
send_ldap_response: msgid=1 tag=97 err=0
ber_flush2: 14 bytes to sd 12
connection_get(12): got connid=1
connection_read(12): checking for input on id=1
ber_get_next
ber_get_next: tag 0x30 len 67 contents:
ber_get_next
conn=1 op=1 do_search
ber_scanf fmt ({miiiib) ber:
>>> dnPrettyNormal: <dc=koitoerldap,dc=com>
<<< dnPrettyNormal: <dc=koitoerldap,dc=com>, <dc=koitoerldap,dc=com>
ber_scanf fmt (m) ber:
ber_scanf fmt ({M}}) ber:
=> bdb_search
bdb_dn2entry("dc=koitoerldap,dc=com")
=> bdb_dn2id("dc=koitoerldap,dc=com")
<= bdb_dn2id: got id=0x1
entry_decode: "dc=koitoerldap,dc=com"
<= entry_decode(dc=koitoerldap,dc=com)
=> bdb_dn2id_children("dc=koitoerldap,dc=com")
<= bdb_dn2id_children("dc=koitoerldap,dc=com"): (0)
=> send_search_entry: conn 1 dn="dc=koitoerldap,dc=com"
ber_flush2: 514 bytes to sd 12
<= send_search_entry: conn 1 exit.
send_ldap_result: conn=1 op=1 p=3
send_ldap_response: msgid=2 tag=101 err=0
ber_flush2: 14 bytes to sd 12
connection_get(12): got connid=1
connection_read(12): checking for input on id=1
ber_get_next
ber_get_next: tag 0x30 len 538 contents:
ber_get_next
conn=1 op=2 do_search
ber_scanf fmt ({miiiib) ber:
>>> dnPrettyNormal: <>
<<< dnPrettyNormal: <>, <>
ber_scanf fmt (m) ber:
ber_scanf fmt ({M}}) ber:
=> send_search_entry: conn 1 dn=""
ber_flush2: 778 bytes to sd 12
<= send_search_entry: conn 1 exit.
send_ldap_result: conn=1 op=2 p=3
send_ldap_response: msgid=3 tag=101 err=0
ber_flush2: 14 bytes to sd 12
connection_get(12): got connid=1
connection_read(12): checking for input on id=1
ber_get_next
ber_get_next: tag 0x30 len 78 contents:
ber_get_next
conn=1 op=3 do_search
ber_scanf fmt ({miiiib) ber:
>>> dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>
<<< dnPrettyNormal: <cn=manager,dc=koitoerldap,dc=com>,
<cn=manager,dc=koitoerldap,dc=com>
ber_scanf fmt (m) ber:
ber_scanf fmt ({M}}) ber:
=> bdb_search
bdb_dn2entry("cn=manager,dc=koitoerldap,dc=com")
=> bdb_dn2id("cn=manager,dc=koitoerldap,dc=com")
And again neves past this point, I dont know why, maybe I install bad
openldap and need more parameters.
Please dont say me install with apt-get , I want to make this from source,
but a few time ago I install with apt and this error not appear, but in this
case I want to make without apt.
Any suggestion, opinion or help will be useful and graceful.
In advance thanks a lot.
12 years, 4 months
Re: ldapsearch hangs
by Tihomir Culjaga
nice link... thnaks!
Will let you know the results.
T.
On Thu, Sep 10, 2009 at 10:10 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> --On Thursday, September 10, 2009 10:07 PM +0200 Tihomir Culjaga <
> tculjaga(a)gmail.com> wrote:
>
> wow, Quanah,
>>
>> that's the best answer i ever got...
>>
>> I'm gong to move to 2.4.18 and try to tune up the machine... regarding
>> more RAM ... i can get it but not tha fast.
>>
>> what can i expect with a total of 10 GB ?...
>>
>> I plan to set set_cachesize 7 0 1
>>
>> what is the relation between cachesize (in slapd.conf) and set_cachesize
>> (in DB_CONFIG)?
>>
>
> They aren't related. cachesize (in slapd.conf) is for the entry cache.
> set_cachesize is for the BDB backing cache.
>
> set_cachesize =>how much memory i wiill cache
>> cachesize => number of entries i will keep within the cache
>>
>> how much one entry take ?
>>
>> Lets say if i use 7 GB of cache, what cachesize should i put?
>>
>
> Hard to say. ;) A bit of the BDB cache will likely be swapped out.
>
> You may also want to read:
>
> <http://wiki.zimbra.com/index.php?title=OpenLDAP_Performance_Tuning_6.0>
>
> It has some zimbra specific commands in it, but it should be fairly trivial
> to see how it maps to your openldap install. I'd also particularly read
> over the bit about using a shared memory key for the BDB cache once you go
> beyond 8GB cachesize (which you will do after you get more RAM. ;) ).
>
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
12 years, 4 months
Re: ldapsearch hangs
by Tihomir Culjaga
wow, Quanah,
thats the best answer i ever got...
I'm gong to move to 2.4.18 and try to tune up the machine... regarding more
RAM ... i can get it but not tha fast.
what can i expect with a total of 10 GB ?...
I plan to set set_cachesize 7 0 1
what is the relation between cachesize (in slapd.conf) and set_cachesize (in
DB_CONFIG)?
set_cachesize =>how much memory i wiill cache
cachesize => number of entries i will keep within the cache
how much one entry take ?
Lets say if i use 7 GB of cache, what cachesize should i put?
T.
On Thu, Sep 10, 2009 at 9:37 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> --On Thursday, September 10, 2009 8:56 PM +0200 Tihomir Culjaga <
> tculjaga(a)gmail.com> wrote:
>
> So, the situation is that i have 2 ldif files i'm recreating the database
>> from.
>>
>> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f
>> /usr/local/etc/openldap/slapd.conf
>> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f
>> /usr/local/etc/openldap/slapd.conf
>>
>
> I would suggest you just make these a single file, so all the work can be
> done at one time.
>
> I tried to re-index with /usr/local/libexec/slapindex -f
>> /usr/local/etc/openldap/slapd.conf -v
>> restart slapd process, restart the machine ... it is always the same
>> issue.
>>
>
> Nothing here indicates a problem with your indices. Running slapindex
> repeatedly is a waste of your time.
>
> [root@l01lnp2 traces]# /usr/local/libexec/slapd -V
>> @(#) $OpenLDAP: slapd 2.4.16 (Sep 9 2009 14:39:44) $
>> root@l01lnp2:/home/tculjaga/openldap-2.4.16/servers/slapd
>>
>
> I would strongly urge you to upgrade to 2.4.18 (for reasons I will note
> further down)
>
>
> [root@l01lnp2 traces]# /usr/local/BerkeleyDB.4.7/bin/db_stat -V
>> Berkeley DB 4.7.25: (May 15, 2008) - unpached!
>>
>
> You need to rebuild BDB 4.7.25 with the 4 patches from Oracle. There are
> known issues when running BDB 4.7 without them.
>
> [root@l01lnp2 traces]# du -c -h /var/lib/ldap/*.bdb
>> 200K /var/lib/ldap/bestMatchPrefix.bdb
>> 3.8G /var/lib/ldap/dn2id.bdb
>> 6.2G /var/lib/ldap/id2entry.bdb
>> 1.8M /var/lib/ldap/objectClass.bdb
>> 1.2M /var/lib/ldap/originatorPrefixID.bdb
>> 48M /var/lib/ldap/uniqueID.bdb
>> 10G total
>>
>
> Since your database is a total of 10 GB in size, for slapadd to work at
> optimum efficiency, you need at least 10GB of cache for your DB_CONFIG file.
> Unfortunately, you only have 10GB of RAM. Essentially, your system is
> under powered for your database size.
>
>
>
> [tculjaga@l01lnp2 ~]$ cat ot.ldif | grep -c "dn: "
>> 101588
>> [tculjaga@l01lnp2 ~]$ cat l01sipdir1.ldif | grep -c "dn: "
>> 9994864
>> [tculjaga@l01lnp2 ~]$
>>
>
> So you have 10,096,452 entries total.
>
> [root@l01lnp2 traces]# cat /var/lib/ldap/DB_CONFIG | grep -v "#"
>>
>> set_cachesize 0 3221225472 1
>> set_lg_regionmax 262144
>> set_lg_bsize 2097152
>>
>
> You only have a 3GB DB cachesize configured here. Expect things to perform
> sub optimally. It would have been easier to set this by going
>
> set_cachesize 3 0 1
>
> Which would have the same effect, since the first number is the number of
> gigabytes to allocate.
>
> Please find attached slapd.conf
>>
>
> Ok, so the relevant bits from here are:
>
> cachesize 2500000
> idlcachesize 7500000
> cachefree 1000
>
> Which means you have a cachesize of 2.5 million, an idlcachesize of 7.5
> million, and (with OL 2.4.16) a dncachesize of 5 million.
>
> I would highly advise you upgrade to OpenLDAP 2.4.18, and change the
> slapd.conf settings to:
>
> dncachesize 0 (which means unlimited).
>
> And setting no cache or idlcachesize, and fixing your DB_CONFIG. But you
> also need to buy a substantial amount of RAM for a DB of this size. :P I
> would advise you upgrade to at least 32GB total. Then you can more
> optimally tune the system.
>
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
12 years, 4 months
