Thanks John for the useful response! Yes we are using ldap at /opt/ldap because we are building it with our SSL lib which is at /opt/ssl and it is OpenSSL 1.1.1. For the process, /opt/ldap/lib and /opt/ssl/lib are set as preferred LD_LIBRARY_PATH.

This is the code of getRebindCredentials(). Let me know if you find anything suspicious here.

int getRebindCredentials( LDAP *ld, LDAP_CONST char *url, ber_tag_t request, ber_int_t msgid, void *params ) { int rc; const sec::LdapProvider* lLdapProvider = (sec::LdapProvider*)params; //LdapProvider is the ldap config object if(lLdapProvider->rootDn.getSize() == 0) { LOG(LOG_DEBUG3, "getRebindCredentials called - NULL DN"); rc = ldap_simple_bind_s(ld, NULL, NULL); } else { LOG(LOG_DEBUG3, "getRebindCredentials called - DN"); rc = ldap_simple_bind_s(ld, lLdapProvider->rootDn.getBuffer(), lLdapProvider->password.getBuffer()); LOG(LOG_DEBUG3, "getRebindCredentials called - DN:%s pass: %s", lLdapProvider->rootDn.getBuffer(), lLdapProvider->password.getBuffer()); } return rc; }

Sample compilation logs (not putting everything here): /bin/sh ../../libtool --mode=compile clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c bind.c libtool: compile: clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c bind.c -fPIC -DPIC -o .libs/bind.o /bin/sh ../../libtool --mode=compile clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c open.c libtool: compile: clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c open.c -fPIC -DPIC -o .libs/open.o /bin/sh ../../libtool --mode=compile clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c result.c libtool: compile: clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c result.c -fPIC -DPIC -o .libs/result.o /bin/sh ../../libtool --mode=compile clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c error.c libtool: compile: clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c error.c -fPIC -DPIC -o .libs/error.o /bin/sh ../../libtool --mode=compile clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c compare.c libtool: compile: clang-17 -Wno-implicit-int -Wno-int-conversion -I../../include -I../../include -I/opt/ciscossl/include -DLDAP_LIBRARY -c compare.c -fPIC -DPIC -o .libs/compare.o

//more below like /bin/sh ../../libtool --mode=link clang-17 -Wno-implicit-int -Wno-int-conversion -L/opt/ciscossl/lib -version-info 2:200:0 -rpath /opt/ldap/lib -Wl,--version-script=./ldap.map -o libldap.la bind.lo open.lo result.lo error.lo compare.lo search.lo controls.lo messages.lo references.lo extended.lo cyrus.lo modify.lo add.lo modrdn.lo delete.lo abandon.lo sasl.lo sbind.lo unbind.lo cancel.lo filter.lo free.lo sort.lo passwd.lo whoami.lo vc.lo getdn.lo getentry.lo getattr.lo getvalues.lo addentry.lo request.lo os-ip.lo url.lo pagectrl.lo sortctrl.lo vlvctrl.lo init.lo options.lo print.lo string.lo util-int.lo schema.lo charray.lo os-local.lo dnssrv.lo utf-8.lo utf-8-conv.lo tls2.lo tls_o.lo tls_g.lo turn.lo ppolicy.lo dds.lo txn.lo ldap_sync.lo stctrl.lo assertion.lo deref.lo ldifutil.lo ldif.lo fetch.lo lbase64.lo msctrl.lo psearchctrl.lo threads.lo rdwr.lo tpool.lo rq.lo thr_posix.lo thr_thr.lo thr_nt.lo thr_pth.lo thr_debug.lo account_usability.lo avl.lo tavl.lo version.lo ../../libraries/liblber/liblber.la -lsasl2 -lssl -lcrypto

On Sat, Jun 28, 2025 at 5:19 PM John Scott jscott@posteo.net wrote:

radiatejava wrote:

...

Hello experts

Hi! I'm not an expert but I have a couple thoughts that others will probably correct me on 🙃

...

We are using OpenLDAP lib ver 2.6.7 on Ubuntu 24.04. When doing LDAP search using ldap_search_ext_s() we are getting assert failure. Any idea what could be going wrong?

The official OpenLDAP packages for Ubuntu 24.04 are indeed version 2.6.7 which you can see at https://packages.ubuntu.com/source/noble/openldap

However, your application is loading the library from /opt/ldap/ at /opt/ldap/lib/libldap.so.2 which is not where Ubuntu ships it. This isn't itself a problem, but I want to make sure you're aware of that. If this was manually installed and you're sure it really is 2.6.7, you should try using the official build shipped with Ubuntu and see if this continues. If you have a reason that's not appropriate for your application then so be it, but otherwise it has all sensible features enabled and is in good shape. (A reason not to use Ubuntu's build is if your application relies on OpenLDAP using OpenSSL in particular instead of GnuTLS, like if it uses the LDAP_OPT_X_TLS_CTX option). You should also make sure when compiling your application that it uses header files from this custom installation in preference to any provided by Ubuntu's packages.

What features is your custom OpenLDAP built with? Knowing if it has Cyrus SASL and OpenSSL or GnuTLS would be helpful. Also the backtrace is lacking details that would be helpful because debugging symbols for OpenLDAP are either not available or not very good. If—by some magic—this version installed in /opt/ldap/ actually *is* the same as what ships in Ubuntu's package, you can use these two GDB commands before starting your program to fetch the debugging symbols on-the-fly: set debuginfod enabled on set debuginfod urls https://debuginfod.ubuntu.com https://debuginfod.elfutils.org The backtrace after that will likely be more helpful.

Lastly, your backtrace hints that several things are being done recursively. For example it seems like your rebind credentials callback myapp::getRebindCredentials() *itself* calls ldap_simple_bind_s() and tries to do further operations, and when it gets another referral it does this even one more time. I'm suspicious if the code is really reentrant like that, and the end of the backtrace alludes that the recursive operations could be causing a double-free or other memory corruption, which would explain everything.

...

#5 0x000071e8b9684409 in ldap_do_free_request () from /opt/ldap/lib/libldap.so.2 #6 0x000071e8b96b4bf9 in ldap_tavl_free () from /opt/ldap/lib/libldap.so.2 #7 0x000071e8b968359a in ldap_free_connection () from /opt/ldap/lib/libldap.so.2 #8 0x000071e8b9683e9c in ldap_new_connection () from /opt/ldap/lib/libldap.so.2 #9 0x000071e8b9682aa7 in ldap_send_server_request () from /opt/ldap/lib/libldap.so.2 #10 0x000071e8b9685619 in ldap_chase_v3referrals () from /opt/ldap/lib/libldap.so.2 #11 0x000071e8b9667640 in try_read1msg () from /opt/ldap/lib/libldap.so.2 #12 0x000071e8b9666515 in wait4msg () from /opt/ldap/lib/libldap.so.2 #13 0x000071e8b9665f09 in ldap_result () from /opt/ldap/lib/libldap.so.2 #14 0x000071e8b9672a95 in ldap_sasl_bind_s () from /opt/ldap/lib/libldap.so.2 #15 0x000071e8b96740b2 in ldap_simple_bind_s () from /opt/ldap/lib/libldap.so.2 #16 0x000071e8bb737e2a in myapp::getRebindCredentials(ldap*, char const*, unsigned long, int, void*) () from /mgmt/usr/lib/x86_64-linux-gnu/libsvc_myapp.so #17 0x000071e8b9683e0b in ldap_new_connection () from /opt/ldap/lib/libldap.so.2 #18 0x000071e8b9682aa7 in ldap_send_server_request () from /opt/ldap/lib/libldap.so.2 #19 0x000071e8b9685619 in ldap_chase_v3referrals () from /opt/ldap/lib/libldap.so.2 #20 0x000071e8b9667640 in try_read1msg () from /opt/ldap/lib/libldap.so.2 #21 0x000071e8b9666515 in wait4msg () from /opt/ldap/lib/libldap.so.2 #22 0x000071e8b9665f09 in ldap_result () from /opt/ldap/lib/libldap.so.2 #23 0x000071e8b966acf7 in ldap_pvt_search_s () from /opt/ldap/lib/libldap.so.2 #24 0x000071e8b966ac25 in ldap_search_ext_s () from /opt/ldap/lib/libldap.so.2 #25 0x000071e8bb732c15 in myapp::LdapAuthenticationBroker::doLdapAuthenticate() () from /mgmt/usr/lib/x86_64-linux-gnu/libsvc_myapp.so #26 0x000071e8bb732484 in myapp::LdapAuthenticationBroker::start(bool&, void (*)(myapp::ExternalAuthenticationBroker*)) () from /mgmt/usr/lib/x86_64-linux-gnu/libsvc_myapp.so

I hope any of those ideas gives you a lead. If possible, I'd like to know what's going in inside myapp::getRebindCredentials() in particular.

You may hinting in the right direction, thanks.

$ ldd /opt/ldap/lib/libldap.so linux-vdso.so.1 (0x00007fffc04e6000) liblber.so.2 => /opt/ldap/lib/liblber.so.2 (0x000074c880976000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x000074c88095c000) libssl.so.1.1 => /opt/cssl/lib/libssl.so.1.1 (0x000074c8808c1000) libcrypto.so.1.1 => /opt/cssl/lib/libcrypto.so.1.1 (0x000074c8805c5000) libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x000074c8803b3000) libcrypto.so.3 => /usr/lib/x86_64-linux-gnu/libcrypto.so.3 (0x000074c87fe9e000) /lib64/ld-linux-x86-64.so.2 (0x000074c880a05000)

So libcrypto and libssl it is linking to ours.

However I see that it is also linking to libcrypto.so.3 and that is because of libsasl2.so.2 that is from the platform. Do you think having libsasl2.so.2 also built with our customer SSL would help here?

On Sat, Jun 28, 2025 at 7:13 PM Jeffrey Walton noloader@gmail.com wrote:

On Sat, Jun 28, 2025 at 3:53 PM radiatejava radiatejava@gmail.com wrote:

...

Hello experts We are using OpenLDAP lib ver 2.6.7 on Ubuntu 24.04. When doing LDAP search using ldap_search_ext_s() we are getting assert failure. Any idea what could be going wrong? Appreciate your reply. Here is the stacktrace:

#0 0x000071e8baceeb2c in pthread_kill () from /usr/lib/x86_64-linux-gnu/libc.so.6 #1 0x000071e8bac9527e in raise () from /usr/lib/x86_64-linux-gnu/libc.so.6 #2 0x000071e8bac788ff in abort () from /usr/lib/x86_64-linux-gnu/libc.so.6 #3 0x000071e8bac7881b in ?? () from /usr/lib/x86_64-linux-gnu/libc.so.6 #4 0x000071e8bac8b517 in __assert_fail () from /usr/lib/x86_64-linux-gnu/libc.so.6 #5 0x000071e8b9684409 in ldap_do_free_request () from /opt/ldap/lib/libldap.so.2 #6 0x000071e8b96b4bf9 in ldap_tavl_free () from /opt/ldap/lib/libldap.so.2 #7 0x000071e8b968359a in ldap_free_connection () from /opt/ldap/lib/libldap.so.2 #8 0x000071e8b9683e9c in ldap_new_connection () from /opt/ldap/lib/libldap.so.2 #9 0x000071e8b9682aa7 in ldap_send_server_request () from /opt/ldap/lib/libldap.so.2 #10 0x000071e8b9685619 in ldap_chase_v3referrals () from /opt/ldap/lib/libldap.so.2 #11 0x000071e8b9667640 in try_read1msg () from /opt/ldap/lib/libldap.so.2 #12 0x000071e8b9666515 in wait4msg () from /opt/ldap/lib/libldap.so.2 #13 0x000071e8b9665f09 in ldap_result () from /opt/ldap/lib/libldap.so.2 #14 0x000071e8b9672a95 in ldap_sasl_bind_s () from /opt/ldap/lib/libldap.so.2 #15 0x000071e8b96740b2 in ldap_simple_bind_s () from /opt/ldap/lib/libldap.so.2 #16 0x000071e8bb737e2a in myapp::getRebindCredentials(ldap*, char const*, unsigned long, int, void*) () from /mgmt/usr/lib/x86_64-linux-gnu/libsvc_myapp.so #17 0x000071e8b9683e0b in ldap_new_connection () from /opt/ldap/lib/libldap.so.2 #18 0x000071e8b9682aa7 in ldap_send_server_request () from /opt/ldap/lib/libldap.so.2 #19 0x000071e8b9685619 in ldap_chase_v3referrals () from /opt/ldap/lib/libldap.so.2 #20 0x000071e8b9667640 in try_read1msg () from /opt/ldap/lib/libldap.so.2 #21 0x000071e8b9666515 in wait4msg () from /opt/ldap/lib/libldap.so.2 #22 0x000071e8b9665f09 in ldap_result () from /opt/ldap/lib/libldap.so.2 #23 0x000071e8b966acf7 in ldap_pvt_search_s () from /opt/ldap/lib/libldap.so.2 #24 0x000071e8b966ac25 in ldap_search_ext_s () from /opt/ldap/lib/libldap.so.2 #25 0x000071e8bb732c15 in myapp::LdapAuthenticationBroker::doLdapAuthenticate() () from /mgmt/usr/lib/x86_64-linux-gnu/libsvc_myapp.so #26 0x000071e8bb732484 in myapp::LdapAuthenticationBroker::start(bool&, void (*)(myapp::ExternalAuthenticationBroker*)) () from /mgmt/usr/lib/x86_64-linux-gnu/libsvc_myapp.so

My uneducated guess is, the problem lies around fram #8, where the connection is established. It looks like the connection fails, and things go to pot when freeing the connection. I assume the connection happens using TLS, but I don't see libcrypto or libssl called out in the stack trace.

What does `ldd /opt/ldap/lib/libldap.so.2` tell you? Are you getting your custom libcrypto and libssl? or are you getting the system one?

Or maybe, is the connection object uninitialized when calling free due to the failure? Should you even call free when the connection fails in frame #8?

Jeff