I have a large number of mail aliases stored in LDAP used by sendmail. They're stored off by themselves as opposed to hung off the user objects, so that the mail servers can have their own LDAP replicas containing only a portion of the tree rather than all the user objects as well. An example entry would be thus:
dn: cn=broken2,cn=dal,cn=Mailmaps,cn=Services,dc=DAL,dc=CA objectClass: top objectClass: applicationProcess objectClass: inetLocalMailRecipient cn: broken2 mailRoutingAddress: broken2@dal.ca mailLocalAddress: broken2@imap.dal.ca
However, this entry will not show up in searches, depending on what I use as a search base:
A successful search looks like:
$ /opt/csw/bin/ldapsearch -x -W -D cn=noc,dc=dal,dc=ca -h kil-ds-3.its.dal.ca -b dc=dal,dc=ca -LLL -s sub '(cn=broken2)' dn Enter LDAP Password: dn: cn=broken2,cn=dal,cn=Mailmaps,cn=Services,dc=DAL,dc=CA $
However, if I change *only* the search base:
$ /opt/csw/bin/ldapsearch -x -W -D cn=noc,dc=dal,dc=ca -h kil-ds-3.its.dal.ca -b cn=services,dc=dal,dc=ca -LLL -s sub '(cn=broken2)' dn Enter LDAP Password: $
A base of cn=Mailmaps,cn=Services,dc=DAL,dc=CA likewise doesn't work, but the entry appears again if I use cn=dal,cn=Mailmaps,cn=Services,dc=DAL,dc=CA.
This server is running 2.4.30 built with BerkeleyDB 5.3.15. The backend I'm using is HDB. I've *completely* removed any ACLs I had defined and it made no difference... not that I expected it to, since cn=noc,dc=dal,dc=ca is the rootDN.
I thought I might have been running into a limit of number of objects inside a container (there's about 123k entries under cn=dal) so I tried dumping the DIT, and moving everything into subcontainers based on first character of the alias (cn=a,cn=dal ; cn=b,cn=dal and so on) which made sure there was no container with more than 10k objects... it still made no difference.
I've run the server with olcLogLevel=Any, and I see the following difference in the syslog:
A "good" search:
Apr 4 16:12:43 kil-ds-3 slapd[29544]: conn=1035 op=1 SRCH base="dc=dal,dc=ca" scope=2 deref=0 filter="(cn=broken2)" Apr 4 16:12:43 kil-ds-3 slapd[29544]: conn=1035 op=1 SRCH attr=dn Apr 4 16:12:43 kil-ds-3 slapd[29544]: => hdb_search Apr 4 16:12:43 kil-ds-3 slapd[29544]: bdb_dn2entry("dc=dal,dc=ca") Apr 4 16:12:43 kil-ds-3 slapd[29544]: => access_allowed: search access to "dc=DAL,dc=CA" "entry" requested Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= root access granted Apr 4 16:12:43 kil-ds-3 slapd[29544]: => access_allowed: search access granted by manage(=mwrscxd) Apr 4 16:12:43 kil-ds-3 slapd[29544]: search_candidates: base="dc=dal,dc=ca" (0x00000001) scope=2 Apr 4 16:12:43 kil-ds-3 slapd[29544]: daemon: activity on 1 descriptor Apr 4 16:12:43 kil-ds-3 slapd[29544]: daemon: activity on: Apr 4 16:12:43 kil-ds-3 slapd[29544]: Apr 4 16:12:43 kil-ds-3 slapd[29544]: daemon: epoll: listen=7 active_threads=0 tvp=zero Apr 4 16:12:43 kil-ds-3 slapd[29544]: daemon: epoll: listen=8 active_threads=0 tvp=zero Apr 4 16:12:43 kil-ds-3 slapd[29544]: daemon: epoll: listen=9 active_threads=0 tvp=zero Apr 4 16:12:43 kil-ds-3 slapd[29544]: => hdb_dn2idl("dc=dal,dc=ca") Apr 4 16:12:43 kil-ds-3 slapd[29544]: => bdb_filter_candidates Apr 4 16:12:43 kil-ds-3 slapd[29544]: #011AND Apr 4 16:12:43 kil-ds-3 slapd[29544]: => bdb_list_candidates 0xa0 Apr 4 16:12:43 kil-ds-3 slapd[29544]: => bdb_filter_candidates Apr 4 16:12:43 kil-ds-3 slapd[29544]: #011OR Apr 4 16:12:43 kil-ds-3 slapd[29544]: => bdb_list_candidates 0xa1 Apr 4 16:12:43 kil-ds-3 slapd[29544]: => bdb_filter_candidates Apr 4 16:12:43 kil-ds-3 slapd[29544]: #011EQUALITY Apr 4 16:12:43 kil-ds-3 slapd[29544]: => bdb_equality_candidates (objectClass) Apr 4 16:12:43 kil-ds-3 slapd[29544]: => key_read Apr 4 16:12:43 kil-ds-3 slapd[29544]: bdb_idl_fetch_key: [b49d1940] Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_index_read: failed (-30988) Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_equality_candidates: id=0, first=0, last=0 Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_filter_candidates: id=0 first=0 last=0 Apr 4 16:12:43 kil-ds-3 slapd[29544]: => bdb_filter_candidates Apr 4 16:12:43 kil-ds-3 slapd[29544]: #011EQUALITY Apr 4 16:12:43 kil-ds-3 slapd[29544]: => bdb_equality_candidates (cn) Apr 4 16:12:43 kil-ds-3 slapd[29544]: => key_read Apr 4 16:12:43 kil-ds-3 slapd[29544]: bdb_idl_fetch_key: [2714190a] Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_index_read 1 candidates Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_equality_candidates: id=1, first=338261, last=338261 Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_filter_candidates: id=1 first=338261 last=338261 Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_list_candidates: id=1 first=338261 last=338261 Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_filter_candidates: id=1 first=338261 last=338261 Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_list_candidates: id=1 first=338261 last=338261 Apr 4 16:12:43 kil-ds-3 slapd[29544]: <= bdb_filter_candidates: id=1 first=338261 last=338261 Apr 4 16:12:43 kil-ds-3 slapd[29544]: bdb_search_candidates: id=1 first=338261 last=338261 Apr 4 16:12:43 kil-ds-3 slapd[29544]: => test_filter Apr 4 16:12:43 kil-ds-3 slapd[29544]: EQUALITY Apr 4 16:12:43 kil-ds-3 slapd[29544]: => access_allowed: search access to "cn=broken2,cn=b,cn=dal,cn=Mailmaps,cn=Services,dc=DAL,dc=CA" "cn" requested
And a "bad" search:
Apr 4 16:13:43 kil-ds-3 slapd[29544]: conn=1036 op=1 SRCH base="cn=services,dc=dal,dc=ca" scope=2 deref=0 filter="(cn=broken2)" Apr 4 16:13:43 kil-ds-3 slapd[29544]: conn=1036 op=1 SRCH attr=dn Apr 4 16:13:43 kil-ds-3 slapd[29544]: => hdb_search Apr 4 16:13:43 kil-ds-3 slapd[29544]: bdb_dn2entry("cn=services,dc=dal,dc=ca") Apr 4 16:13:43 kil-ds-3 slapd[29544]: => access_allowed: search access to "cn=Services,dc=DAL,dc=CA" "entry" requested Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= root access granted Apr 4 16:13:43 kil-ds-3 slapd[29544]: => access_allowed: search access granted by manage(=mwrscxd) Apr 4 16:13:43 kil-ds-3 slapd[29544]: search_candidates: base="cn=services,dc=dal,dc=ca" (0x00000003) scope=2 Apr 4 16:13:43 kil-ds-3 slapd[29544]: => hdb_dn2idl("cn=services,dc=dal,dc=ca") Apr 4 16:13:43 kil-ds-3 slapd[29544]: daemon: activity on 1 descriptor Apr 4 16:13:43 kil-ds-3 slapd[29544]: daemon: activity on: Apr 4 16:13:43 kil-ds-3 slapd[29544]: Apr 4 16:13:43 kil-ds-3 slapd[29544]: daemon: epoll: listen=7 active_threads=0 tvp=zero Apr 4 16:13:43 kil-ds-3 slapd[29544]: daemon: epoll: listen=8 active_threads=0 tvp=zero Apr 4 16:13:43 kil-ds-3 slapd[29544]: daemon: epoll: listen=9 active_threads=0 tvp=zero Apr 4 16:13:43 kil-ds-3 slapd[29544]: => bdb_filter_candidates Apr 4 16:13:43 kil-ds-3 slapd[29544]: #011AND Apr 4 16:13:43 kil-ds-3 slapd[29544]: => bdb_list_candidates 0xa0 Apr 4 16:13:43 kil-ds-3 slapd[29544]: => bdb_filter_candidates Apr 4 16:13:43 kil-ds-3 slapd[29544]: #011OR Apr 4 16:13:43 kil-ds-3 slapd[29544]: => bdb_list_candidates 0xa1 Apr 4 16:13:43 kil-ds-3 slapd[29544]: => bdb_filter_candidates Apr 4 16:13:43 kil-ds-3 slapd[29544]: #011EQUALITY Apr 4 16:13:43 kil-ds-3 slapd[29544]: => bdb_equality_candidates (objectClass) Apr 4 16:13:43 kil-ds-3 slapd[29544]: => key_read Apr 4 16:13:43 kil-ds-3 slapd[29544]: bdb_idl_fetch_key: [b49d1940] Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_index_read: failed (-30988) Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_equality_candidates: id=0, first=0, last=0 Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_filter_candidates: id=0 first=0 last=0 Apr 4 16:13:43 kil-ds-3 slapd[29544]: => bdb_filter_candidates Apr 4 16:13:43 kil-ds-3 slapd[29544]: #011EQUALITY Apr 4 16:13:43 kil-ds-3 slapd[29544]: => bdb_equality_candidates (cn) Apr 4 16:13:43 kil-ds-3 slapd[29544]: => key_read Apr 4 16:13:43 kil-ds-3 slapd[29544]: bdb_idl_fetch_key: [2714190a] Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_index_read 1 candidates Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_equality_candidates: id=1, first=338261, last=338261 Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_filter_candidates: id=1 first=338261 last=338261 Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_list_candidates: id=1 first=338261 last=338261 Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_filter_candidates: id=1 first=338261 last=338261 Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_list_candidates: id=0 first=3 last=0 Apr 4 16:13:43 kil-ds-3 slapd[29544]: <= bdb_filter_candidates: id=0 first=3 last=0 Apr 4 16:13:43 kil-ds-3 slapd[29544]: bdb_search_candidates: id=0 first=3 last=0 Apr 4 16:13:43 kil-ds-3 slapd[29544]: hdb_search: no candidates Apr 4 16:13:43 kil-ds-3 slapd[29544]: send_ldap_result: conn=1036 op=1 p=3 Apr 4 16:13:43 kil-ds-3 slapd[29544]: send_ldap_result: err=0 matched="" text="" Apr 4 16:13:43 kil-ds-3 slapd[29544]: send_ldap_response: msgid=2 tag=101 err=0 Apr 4 16:13:43 kil-ds-3 slapd[29544]: conn=1036 op=1 SEARCH RESULT tag=101 err=0 nentries=0 text=
Notable is hdb_search finding no candidates.
Is there anything obviously wrong, here?
Brandon Hume wrote:
I have a large number of mail aliases stored in LDAP used by sendmail. They're stored off by themselves as opposed to hung off the user objects, so that the mail servers can have their own LDAP replicas containing only a portion of the tree rather than all the user objects as well. An example entry would be thus:
dn: cn=broken2,cn=dal,cn=Mailmaps,cn=Services,dc=DAL,dc=CA objectClass: top objectClass: applicationProcess objectClass: inetLocalMailRecipient cn: broken2 mailRoutingAddress: broken2@dal.ca mailLocalAddress: broken2@imap.dal.ca
However, this entry will not show up in searches, depending on what I use as a search base:
I can confirm that I also saw such a strange effect on a customer's server running OpenLDAP 2.4.28 with back-hdb based on BDB 5.2.x (not sure about x). All compiled as 64-bit software on HP-UX 10.x (not sure about the exact version). I don't have easy access to this production system so it's quite hard to examine this. But I'm somewhat glad I'm not alone.
In my case I had to use search base ou=xxx,dc=example,dc=org instead of also valid dc=example,dc=org to find the entries. I also examined that no ACLs were standing in the way.
Ciao, Michael.
--On Wednesday, April 04, 2012 4:15 PM -0300 Brandon Hume hume-ol@bofh.ca wrote:
I have a large number of mail aliases stored in LDAP used by sendmail. They're stored off by themselves as opposed to hung off the user objects, so that the mail servers can have their own LDAP replicas containing only a portion of the tree rather than all the user objects as well. An example entry would be thus:
Notable is hdb_search finding no candidates.
Is there anything obviously wrong, here?
I suggest you file an ITS. In the meantime, download RE24 and see if back-mdb does better for you.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
In the meantime, download RE24 and see if back-mdb does better for you.
Are you serious? The bug Brandon observed and which I can confirm is a serious threat to LDAP functional model. I could definitely reproduce it for a while.
The reason I did not file an ITS for this yet is that I'm not allowed to debug it on this particular system and I was often blamed by you and Howard for writing "content-free" ITS in cases where I provided way enough information to reproduce. For this reason I'm somewhat happy that someone else reports this.
IMO back-mdb is still rather experimental though I definitely appreciate all the work done recently.
Ciao, Michael.
Hi,
Am 5. April 2012 09:44 schrieb Michael Ströder michael@stroeder.com:
Quanah Gibson-Mount wrote:
In the meantime, download RE24 and see if back-mdb does better for you.
Are you serious? The bug Brandon observed and which I can confirm is a serious threat to LDAP functional model. I could definitely reproduce it for a while.
Me too: http://www.openldap.org/lists/openldap-technical/201201/msg00234.html
Yours, Karsten
Karsten Heymann wrote:
Hi,
Am 5. April 2012 09:44 schrieb Michael Ströder michael@stroeder.com:
Quanah Gibson-Mount wrote:
In the meantime, download RE24 and see if back-mdb does better for you.
Are you serious? The bug Brandon observed and which I can confirm is a serious threat to LDAP functional model. I could definitely reproduce it for a while.
Me too: http://www.openldap.org/lists/openldap-technical/201201/msg00234.html
In opposite what Brandon and me experienced in your case a shorter search base successfully returns results. :-/
Ciao, Michael.
Hi Michael,
Am 5. April 2012 10:34 schrieb Michael Ströder michael@stroeder.com:
Karsten Heymann wrote:
Am 5. April 2012 09:44 schrieb Michael Ströder michael@stroeder.com:
Are you serious? The bug Brandon observed and which I can confirm is a serious threat to LDAP functional model. I could definitely reproduce it for a while.
Me too: http://www.openldap.org/lists/openldap-technical/201201/msg00234.html
In opposite what Brandon and me experienced in your case a shorter search base successfully returns results. :-/
Yes, but nevertheless widening or narrowing the search base should not change the result. But you are right, it's not exactly the same problem.
Yours Karsten
Karsten Heymann wrote:
Hi Michael,
Am 5. April 2012 10:34 schrieb Michael Ströder michael@stroeder.com:
Karsten Heymann wrote:
Am 5. April 2012 09:44 schrieb Michael Ströder michael@stroeder.com:
Are you serious? The bug Brandon observed and which I can confirm is a serious threat to LDAP functional model. I could definitely reproduce it for a while.
Me too: http://www.openldap.org/lists/openldap-technical/201201/msg00234.html
In opposite what Brandon and me experienced in your case a shorter search base successfully returns results. :-/
Yes, but nevertheless widening or narrowing the search base should not change the result. But you are right, it's not exactly the same problem.
Do you still have this problem? Could you try to reproduce this with your data with a build of recent RE24 ?
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/he...
I can't reproduce this here at home myself.
Ciao, Michael.
Hi,
Am 5. April 2012 12:37 schrieb Michael Ströder michael@stroeder.com:
Karsten Heymann wrote:
Yes, but nevertheless widening or narrowing the search base should not change the result. But you are right, it's not exactly the same problem.
Do you still have this problem? Could you try to reproduce this with your data with a build of recent RE24 ?
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/he...
I can't reproduce this here at home myself.
No, I reinstalled the server completely and this time took care to only modify cn=config via ldap modifications and not on the filesystem. Now I have a stock debian 2.4.23 Installation running quite well. I learned a lot since then by just following this list. Packaging a newer version of slapd is still on my todo.
Yours Karsten
Karsten Heymann wrote:
Hi,
Am 5. April 2012 12:37 schrieb Michael Ströder michael@stroeder.com:
Karsten Heymann wrote:
Yes, but nevertheless widening or narrowing the search base should not change the result. But you are right, it's not exactly the same problem.
Do you still have this problem? Could you try to reproduce this with your data with a build of recent RE24 ?
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/he...
I can't reproduce this here at home myself.
No, I reinstalled the server completely and this time took care to only modify cn=config via ldap modifications and not on the filesystem. Now I have a stock debian 2.4.23 Installation running quite well. I learned a lot since then by just following this list. Packaging a newer version of slapd is still on my todo.
So in your particular case it could have also been a configuration error regarding ACLs or similar?
Ciao, Michael.
--On Thursday, April 05, 2012 9:44 AM +0200 Michael Ströder michael@stroeder.com wrote:
Quanah Gibson-Mount wrote:
In the meantime, download RE24 and see if back-mdb does better for you.
Are you serious? The bug Brandon observed and which I can confirm is a serious threat to LDAP functional model. I could definitely reproduce it for a while.
Yes, I'm quite serious. It would be worthwhile to confirm whether or not back-mdb suffers the same issue. The issues I've seen with back-hdb that are similar to this (ITS#6983, for example), is due to the IDL cache code in back-hdb. back-mdb does not have an IDL cache. So if you can't reproduce it with back-mdb, it is likely the issue is once again in the IDL cache. Regardless, an ITS should most certainly be filed by Brandon so this can be tracked and fixed.
IMO back-mdb is still rather experimental though I definitely appreciate all the work done recently.
I think it is at least beta quality as of yesterday. I'm running it in production now on a few servers, and except for the issues I had with accesslog cleanup (which is what got fixed yesterday), it has been quite stable for a few weeks.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Thursday, April 05, 2012 9:44 AM +0200 Michael Ströder michael@stroeder.com wrote:
Quanah Gibson-Mount wrote:
In the meantime, download RE24 and see if back-mdb does better for you.
Are you serious? The bug Brandon observed and which I can confirm is a serious threat to LDAP functional model. I could definitely reproduce it for a while.
Yes, I'm quite serious. It would be worthwhile to confirm whether or not back-mdb suffers the same issue.
Hard to tell since this bug isn't easily reproducable. As said: I can't easily test on the system in question because of strict operational and privacy regulations.
The issues I've seen with back-hdb that are similar to this (ITS#6983, for example), is due to the IDL cache code in back-hdb.
Hmm, this might be a hint into the right direction. Could be that this system was running 2.4.26 at that time and fix for ITS#6983 appears in CHANGES as of 2.4.27. Definitely there were subtree renames triggered at that time.
If it's the IDL cache would it be helpful to throw away the shared mem files?
back-mdb does not have an IDL cache.
Yes, I listened to Howard's presentation on back-mdb and of course I highly appreciate this work.
IMO back-mdb is still rather experimental though I definitely appreciate all the work done recently.
I think it is at least beta quality as of yesterday. I'm running it in production now on a few servers, and except for the issues I had with accesslog cleanup (which is what got fixed yesterday), it has been quite stable for a few weeks.
I'm not in the position to propose changing the backend soon on this customer system.
Ciao, Michael.
--On Thursday, April 05, 2012 8:07 PM +0200 Michael Ströder michael@stroeder.com wrote:
I think it is at least beta quality as of yesterday. I'm running it in production now on a few servers, and except for the issues I had with accesslog cleanup (which is what got fixed yesterday), it has been quite stable for a few weeks.
I'm not in the position to propose changing the backend soon on this customer system.
Understood. ;) The proposal was for Brandon, who can play around with this. ;)
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On 04/ 4/12 07:47 PM, Quanah Gibson-Mount wrote:
I suggest you file an ITS. In the meantime, download RE24 and see if back-mdb does better for you.
Will do. I can confirm that it does not happen in RE24 with back-mdb, and it *does* happen with back-hdb.
I'm currently trying to cook an LDIF I can provide as a sample. I've successfully been able to reproduce the issue with everything except dc=dal,dc=ca and the cn=services,dc=dal,dc=ca tree pared out. If I can trim out or anonymize the remaining user data I'll include a link in the ITS.
On 05/04/2012 3:06 PM, Brandon Hume wrote:
Will do. I can confirm that it does not happen in RE24 with back-mdb, and it *does* happen with back-hdb.
I've filed Incoming/7231, and dumped the broken config+dit at http://den.bofh.ca/~hume/ldap_missing_results.tar.gz.
In a completely unrelated note, Incoming/6136 was filed in 2009, but I've seen identical behaviour with 2.4.29. I haven't had a chance to try to reproduce it with RE24.
openldap-technical@openldap.org
-
Brandon Hume -
Karsten Heymann -
Michael Ströder -
Quanah Gibson-Mount