These OpenLDAP servers are in an Internet facing DMZ, so they are using external DNS servers.

I pointed them to internal DNS servers and created firewall rules to allow this traffic, and they can now resolve the internal 'ForestDnsZones.domain.com', which is were it appears to have been hanging previously (I missed this with my tcpdump filter).  The Internal DNS servers are returning a list of 4 internal AD servers (I only have OpenLDAP pointed to two of them, and firewall rules for the two, not all four).

After getting that DNS response, I see OpenLDAP is hung trying to send LDAP queries to the other two internal AD servers (that the firewall is blocking) for some reason.

When I create firewall rules to allow OpenLDAP to hit the other 2 internal AD boxes via ldap/ldaps/dns, the search works quickly and as expected.

Pointing OpenLDAP back to external DNS servers, and creating local host entries for ForestDnsZones.domain.com, DomainDnsZones.domain.com, and domain.com pointing to the two internal AD boxes I am using for LDAP, everything works quickly and as expected.

> Date: Wed, 22 Oct 2014 10:08:46 +0100
> From: hyc@symas.com
> To: jeflebo@outlook.com; quanah@zimbra.com; openldap-technical@openldap.org
> Subject: Re: LDAP searches hang after returning results...
>
> Jeff Lebo wrote:
> > I was able to get 2.4.40 compiled and installed. Having the same issue.. here
> > is the syslog output... you can see the timestamp difference between where the
> > hang happens, and where slapd finally disconnects.
>
> Use slapd -d7, not syslog.
>
> The null search references look suspicious, most likely AD is returning
> malformed references and back-ldap is hitting a DNS timeout trying to resolve
> them.
>
> > Oct 21 20:51:24 LDAP02 slapd[28955]: => send_search_entry: conn 1001
> > dn="cn=TEST,ou=Support,ou=Users,ou=Staff,dc=domain,dc=com"
> > Oct 21 20:51:24 LDAP02 slapd[28955]: <= send_search_entry: conn 1001 exit.
> > Oct 21 20:51:24 LDAP02 slapd[28955]: => send_search_reference: dn="(null)"
> > Oct 21 20:51:24 LDAP02 slapd[28955]: <= send_search_reference
> > Oct 21 20:51:25 LDAP02 slapd[28955]: => send_search_reference: dn="(null)"
> > Oct 21 20:51:25 LDAP02 slapd[28955]: <= send_search_reference
> > Oct 21 20:53:32 LDAP02 slapd[28955]: => send_search_reference: dn="(null)"
> > Oct 21 20:53:32 LDAP02 slapd[28955]: <= send_search_reference
> > Oct 21 20:53:32 LDAP02 slapd[28955]: send_ldap_result: conn=1001 op=1 p=3
> > Oct 21 20:53:32 LDAP02 slapd[28955]: send_ldap_response: msgid=2 tag=101 err=0
> > Oct 21 20:53:32 LDAP02 slapd[28955]: connection_get(12): got connid=1001
> > Oct 21 20:53:32 LDAP02 slapd[28955]: connection_read(12): checking for input
> > on id=1001
> > Oct 21 20:53:32 LDAP02 slapd[28955]: op tag 0x42, time 1413950012
> > Oct 21 20:53:32 LDAP02 slapd[28955]: ber_get_next on fd 12 failed errno=0
> > (Success)
> > Oct 21 20:53:32 LDAP02 slapd[28955]: conn=1001 op=2 do_unbind
> > Oct 21 20:53:32 LDAP02 slapd[28955]: connection_close: conn=1001 sd=12
> > Oct 21 20:53:32 LDAP02 slapd[28955]: =>ldap_back_conn_destroy: fetching conn 1001
> > Oct 21 20:53:32 LDAP02 slapd[28955]: =>ldap_back_conn_destroy: destroying conn
> > 1001 refcnt=0 flags=0x00000101
> >
> > Any ideas?
> >
> > ------------------------------------------------------------------------------
> > From: jeflebo@outlook.com
> > To: quanah@zimbra.com; openldap-technical@openldap.org
> > Subject: RE: LDAP searches hang after returning results...
> > Date: Tue, 21 Oct 2014 19:39:45 -0700
> >
> > Unfortunately just dealing with what was handed to me. I'll try and get a new
> > version compiled and see if that resolves this issue.
> >
> > > Date: Tue, 21 Oct 2014 18:21:56 -0700
> > > From: quanah@zimbra.com
> > > To: jeflebo@outlook.com; openldap-technical@openldap.org
> > > Subject: RE: LDAP searches hang after returning results...
> > >
> > > --On Tuesday, October 21, 2014 6:27 PM -0700 Jeff Lebo
> > > <jeflebo@outlook.com> wrote:
> > >
> > > >
> > > > 2.4.31
> > >
> > > Why are you wasting everyone's time, but most importantly your own time,
> > > using something so ancient?
> > >
> > > <http://www.openldap.org/software/release/changes.html>
> > >
> > > Install a current release (2.4.40), and then see what issues you hit.
>
>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>