https://bugs.openldap.org/show_bug.cgi?id=9287
Issue ID: 9287 Summary: ldap_create cause 10s delay in some scenario Product: OpenLDAP Version: 2.4.44 Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: libraries Assignee: bugs@openldap.org Reporter: happy.mu@nokia-sbell.com Target Milestone: ---
hi LDAP guys
there has some issue in our product when we integrate ipv6.
scenario: ipv4 internal + ipv6 external after ldap search finish, we need build ldap message by ldap_create function. some times the ldap_create message will cause 10s delay.
for example: the ldap search has 4 ocs, and only the first oc's ldap_create will cause 10s delay, the fellowing ocs is within 0.0001s. we just think it may because the mutext is global and locked by other thread.
could you help on this ?
LOGs:
there has 10s delay, and only execute ldap_create between them.
2020/07/08-16:32:11.942114-[INFO] 13 (DBL) [DBL_LlbSearchResponse:76]LLB replied successfully with [4] entries 2020/07/08-16:32:11.942181- [INFO] 13 (DBL) [DBL_LlbSearchResponse:125]Entry Buffer[0] : l_entryBerBufferPtr="0x64820136xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" 2020/07/08-16:32:21.957488 [INFO] 13 (DBL) [DBL_LlbSearchResponse:138]Successfully created LDAP dummy handle
https://bugs.openldap.org/show_bug.cgi?id=9287
--- Comment #1 from Quanah Gibson-Mount quanah@openldap.org --- This sounds like a DNS issue rather than a mutex issue.
If it is a mutex issue, you should be able to provide
https://bugs.openldap.org/show_bug.cgi?id=9287
--- Comment #2 from Quanah Gibson-Mount quanah@openldap.org --- This sounds like a DNS issue rather than a mutex issue.
If it is a mutex issue, you should be able to provide a stack trace demonstrating that case.
https://bugs.openldap.org/show_bug.cgi?id=9287
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Resolution|--- |FEEDBACK
https://bugs.openldap.org/show_bug.cgi?id=9287
happy mu happy.mu@nokia-sbell.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|FEEDBACK |--- Status|RESOLVED |UNCONFIRMED
--- Comment #3 from happy mu happy.mu@nokia-sbell.com --- hi Quanah
yes, there is an issue in ldap DNS search.
currently we use *result=gethostbyname_r in function ldap_pvt_gethostbyname_a. but gethostbyname_r is not support ipv6 and deprecated , should use getaddrinfo() instead.
will openldap fix this issue ? if yes, which version can be use.
/*********************************************************************/
ref : http://refspecs.linux-foundation.org/LSB_4.0.0/LSB-Core-generic/LSB-Core-gen...
pstack : 1412 #0 0x00007f5e92426c3d in poll () from /lib64/libc.so.6 1413 #1 0x00007f5e8ffa4f62 in __res_context_send () from /lib64/libresolv.so.2 1414 #2 0x00007f5e8ffa2394 in __res_context_query () from /lib64/libresolv.so.2 1415 #3 0x00007f5e8ffa35ca in __res_context_search () from /lib64/libresolv.so.2 1416 #4 0x00007f5e80a13208 in gethostbyname3_context () from /lib64/libnss_dns.so.2 1417 #5 0x00007f5e80a13cdb in _nss_dns_gethostbyname_r () from /lib64/libnss_dns.so.2 1418 #6 0x00007f5e9244ce6f in gethostbyname_r@@GLIBC_2.2.5 () from /lib64/libc.so.6 1419 #7 0x00007f5e90e2667e in ldap_pvt_gethostbyname_a (name=name@entry=0x7fff152052c0 "mix-udmee-1-67784ddf44-5krt4", resbuf=resbuf@entry=0x7ff f152052a0, buf=buf@entry=0x7fff15205290, result=result@entry=0x7fff15205298, herrno_ptr=herrno_ptr@entry=0x7fff1520528c) at util-int.c:453 1420 #8 0x00007f5e90e268cd in ldap_pvt_get_fqdn (name=0x7fff152052c0 "mix-udmee-1-67784ddf44-5krt4", name@entry=0x0) at util-int.c:851 1421 #9 0x00007f5e90e24afb in ldap_int_initialize (gopts=gopts@entry=0x7f5e91049120 <ldap_int_global_options>, dbglvl=dbglvl@entry=0x7fff1520549 0) at init.c:682 1422 #10 0x00007f5e90e25841 in ldap_set_option (ld=0x0, option=20481, invalue=0x7fff15205490) at options.c:463
https://bugs.openldap.org/show_bug.cgi?id=9287
--- Comment #4 from happy mu happy.mu@nokia-sbell.com --- (In reply to Quanah Gibson-Mount from comment #2)
This sounds like a DNS issue rather than a mutex issue.
If it is a mutex issue, you should be able to provide a stack trace demonstrating that case.
root cause: The gethostbyname_r() function is deprecated; applications should use getaddrinfo() instead.
we test that it works if use getaddrinfo.
https://bugs.openldap.org/show_bug.cgi?id=9287
--- Comment #5 from happy mu happy.mu@nokia-sbell.com --- hi Quanah
yes, there is an issue in ldap DNS search.
currently we use *result=gethostbyname_r in function ldap_pvt_gethostbyname_a. but gethostbyname_r is not support 'ipv6' and deprecated , should use getaddrinfo() instead.
will openldap fix this issue ? if yes, which version can be use.
/*********************************************************************/
ref : http://refspecs.linux-foundation.org/LSB_4.0.0/LSB-Core-generic/LSB-Core-gen...
pstack : 1412 #0 0x00007f5e92426c3d in poll () from /lib64/libc.so.6 1413 #1 0x00007f5e8ffa4f62 in __res_context_send () from /lib64/libresolv.so.2 1414 #2 0x00007f5e8ffa2394 in __res_context_query () from /lib64/libresolv.so.2 1415 #3 0x00007f5e8ffa35ca in __res_context_search () from /lib64/libresolv.so.2 1416 #4 0x00007f5e80a13208 in gethostbyname3_context () from /lib64/libnss_dns.so.2 1417 #5 0x00007f5e80a13cdb in _nss_dns_gethostbyname_r () from /lib64/libnss_dns.so.2 1418 #6 0x00007f5e9244ce6f in gethostbyname_r@@GLIBC_2.2.5 () from /lib64/libc.so.6 1419 #7 0x00007f5e90e2667e in ldap_pvt_gethostbyname_a (name=name@entry=0x7fff152052c0 "mix-udmee-1-67784ddf44-5krt4", resbuf=resbuf@entry=0x7ff f152052a0, buf=buf@entry=0x7fff15205290, result=result@entry=0x7fff15205298, herrno_ptr=herrno_ptr@entry=0x7fff1520528c) at util-int.c:453 1420 #8 0x00007f5e90e268cd in ldap_pvt_get_fqdn (name=0x7fff152052c0 "mix-udmee-1-67784ddf44-5krt4", name@entry=0x0) at util-int.c:851 1421 #9 0x00007f5e90e24afb in ldap_int_initialize (gopts=gopts@entry=0x7f5e91049120 <ldap_int_global_options>, dbglvl=dbglvl@entry=0x7fff1520549 0) at init.c:682 1422 #10 0x00007f5e90e25841 in ldap_set_option (ld=0x0, option=20481, invalue=0x7fff15205490) at options.c:463
Thanks & Regard Happy -----Original Message----- From: openldap-its@openldap.org openldap-its@openldap.org Sent: 2020年7月14日 23:44 To: Mu, Happy (NSB - CN/Qingdao) happy.mu@nokia-sbell.com Subject: [Issue 9287] ldap_create cause 10s delay in some scenario
https://bugs.openldap.org/show_bug.cgi?id=9287
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Resolution|--- |FEEDBACK
https://bugs.openldap.org/show_bug.cgi?id=9287
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Resolution|--- |INVALID
--- Comment #6 from Quanah Gibson-Mount quanah@openldap.org --- (In reply to happy mu from comment #5)
hi Quanah
yes, there is an issue in ldap DNS search.
currently we use *result=gethostbyname_r in function ldap_pvt_gethostbyname_a. but gethostbyname_r is not support 'ipv6' and deprecated , should use getaddrinfo() instead.
OpenLDAP already uses getaddrinfo if it is available.
The only function that uses gethostbyname_r is the private internal function ldap_pvt_gethostbyname_a in libldap/util-int.c
That function is only called in one location, in libldap/os-ip.c function ldap_connect_to_host, and that is if and only if getaddrinfo was not found.
I.e., there does not appear that there is anything here for OpenLDAP to do, it would appear your library is compiled in correctly.
Regards, Quanah
https://bugs.openldap.org/show_bug.cgi?id=9287
Ryan Tandy ryan@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.openldap.org/s | |how_bug.cgi?id=8376
--- Comment #7 from Ryan Tandy ryan@openldap.org --- (In reply to Quanah Gibson-Mount from comment #6)
The only function that uses gethostbyname_r is the private internal function ldap_pvt_gethostbyname_a in libldap/util-int.c
That function is only called in one location, in libldap/os-ip.c function ldap_connect_to_host, and that is if and only if getaddrinfo was not found.
The stack trace that was posted shows where it is used:
1418 #6 0x00007f5e9244ce6f in gethostbyname_r@@GLIBC_2.2.5 () from /lib64/libc.so.6 1419 #7 0x00007f5e90e2667e in ldap_pvt_gethostbyname_a (name=name@entry=0x7fff152052c0 "mix-udmee-1-67784ddf44-5krt4", resbuf=resbuf@entry=0x7ff f152052a0, buf=buf@entry=0x7fff15205290, result=result@entry=0x7fff15205298, herrno_ptr=herrno_ptr@entry=0x7fff1520528c) at util-int.c:453 1420 #8 0x00007f5e90e268cd in ldap_pvt_get_fqdn (name=0x7fff152052c0 "mix-udmee-1-67784ddf44-5krt4", name@entry=0x0) at util-int.c:851
That is valid and true in RE24 (and I think still in master too), getaddrinfo is not used in getting our own FQDN, even if it's available.
This issue might actually be a better test case for answering my concern from 8376 since nss_ldap is not involved this time.
Where exactly is the 10 sec delay coming from, though? Even if gethostbyname doesn't support ipv6, I'd expect things to fail quickly rather than delay...
https://bugs.openldap.org/show_bug.cgi?id=9287
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CONFIRMED Resolution|INVALID |--- Ever confirmed|0 |1
https://bugs.openldap.org/show_bug.cgi?id=9287
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |2.4.51
--- Comment #8 from Quanah Gibson-Mount quanah@openldap.org --- Ugh, I missed that call, thanks.
Definitely a problem then, marking for 2.4.51.
https://bugs.openldap.org/show_bug.cgi?id=9287
--- Comment #9 from jessica.1.chen@nokia-sbell.com --- Per our configuration, NSS would firstly check /etc/hosts and then do DNS query when gethostbyname_r is invokded from ldap_set_option or ldap_create.
In our IPv4 test bed, gethostbyname_r returned successfully with the info found in /etc/hosts. ==================================================== NSS/HOST configuration in IPv4 test bed ==================================================== bash-4.2$ hostname udm5g-udmee-1-788bbbb649-f667m bash-4.2$ bash-4.2$ cat /etc/hosts |grep udm5g-udmee-1-788bbbb649-f667m 192.168.1.11 udm5g-udmee-1-788bbbb649-f667m bash-4.2$ bash-4.2$ cat /etc/nsswitch.conf |grep host hosts: files dns myhostname bash-4.2$
However in our IPv6 test bed, gethostbyname_r somehow didn't recognize the entry in /etc/hosts so it started DNS query which timed out after about 10 seconds which was expected behavior. The timer duration is controlled by the following configuration in /usr/include/resolv.h.
# define RES_TIMEOUT 5 /* min. seconds between retries */ # define RES_DFLRETRY 2 /* Default #/tries. */
With exactly the same configuration, getaddrinfo() would return successfully with info in /etc/hosts.
==================================================== NSS/HOST Configuration in IPv6 TB ==================================================== bash-4.2$ hostname mix-udmee-1-6f869bc959-px7gx bash-4.2$ bash-4.2$ cat /etc/hosts |grep mix-udmee fd00::7bd1:e96f:ccda:8aa0 mix-udmee-1-6f869bc959-px7gx bash-4.2$ bash-4.2$ cat nsswitch.conf |grep hosts hosts: files dns myhostname
https://bugs.openldap.org/show_bug.cgi?id=9287
--- Comment #10 from Quanah Gibson-Mount quanah@openldap.org --- master:
• 24b45f57 by Howard Chu at 2020-07-16T21:08:36+01:00 ITS#9287 use getaddrinfo for ldap_pvt_get_fqdn
If getaddrinfo is available, should use it here
https://bugs.openldap.org/show_bug.cgi?id=9287
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|CONFIRMED |RESOLVED
--- Comment #11 from Quanah Gibson-Mount quanah@openldap.org --- RE24:
commit c91cafcf10b9e4591ea666beda571b9a3d4d55bc Author: Howard Chu hyc@openldap.org Date: Thu Jul 16 21:08:36 2020 +0100
ITS#9287 use getaddrinfo for ldap_pvt_get_fqdn
If getaddrinfo is available, should use it here
https://bugs.openldap.org/show_bug.cgi?id=9287
--- Comment #12 from happy mu happy.mu@nokia-sbell.com --- so, will you fix this issue to use getaddrinfo instead of gethostbyname_r in 2.4.51 ?
(In reply to Quanah Gibson-Mount from comment #11)
RE24:
commit c91cafcf10b9e4591ea666beda571b9a3d4d55bc Author: Howard Chu hyc@openldap.org Date: Thu Jul 16 21:08:36 2020 +0100
ITS#9287 use getaddrinfo for ldap_pvt_get_fqdn If getaddrinfo is available, should use it here
https://bugs.openldap.org/show_bug.cgi?id=9287
--- Comment #13 from Quanah Gibson-Mount quanah@openldap.org --- (In reply to happy mu from comment #12)
so, will you fix this issue to use getaddrinfo instead of gethostbyname_r in 2.4.51 ?
Correct, that's why the target release on the bug is 2.4.51 and why it's been added to RE24, and why it's in the 2.4.51 Changelog.
https://bugs.openldap.org/show_bug.cgi?id=9287
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |VERIFIED
-
openldap-its＠openldap.org