openldap-technical

openldap-technical@openldap.org

18 participants
9785 discussions

Slapd process randomly hangs on consumers and comes back to life after a while with no intervention
by Pierre-Jean CROUZET 20 Sep '24

20 Sep '24

Hello, To start with, this is my very first post on a mailing list of this kind. I've read much about how to do things right, like here: https://www.openldap.org/doc/admin24/troubleshooting.html or here: https://www.openldap.org/faq/data/cache/59.html Nonetheless, please do tell me if I'm not following the standards. I'm facing an issue with my OpenLDAP servers that I struggle to fix or even find info about. To summarize: We have an architecture of 2 providers with mirror mode replication for their configuration. They are LXC's hosted in a datacenter. We have 2 consumers with mirror mode replication for their config also, and they get the DB (mdb backend) from the above providers. They are VMs hosted in a different hypervisor and datacenter than the providers. We have a loadbalancing architecture in front that sends requests evenly to the 4 servers. The writes requests directed to the consumers are forwarded to the providers using the chain overlay. The slapd process of those 2 consumers randomly hangs. When it happens, the slapd service is still running and listening, but not a single query can go through, even the logging stops. Like a complete freeze. The service comes back to life after some time (several minutes, different every time) without any manual intervention. Trying to manually restart the process during such even obviously fails, unless I kill -9, which is not something I want to do. My observations so far : - The hangs appear only during business hours (nothing during nights or week-end). - Only the consumers hang. The providers are always fine. - CPU / RAM / Disks are fine before / during the crashes. - The hangs seem random and I could not find a way to trigger them at will. - strace shows the following lines in an infinite loop during hangs. ``` [pid 1082] gettimeofday({tv_sec=1726669191, tv_usec=694473}, NULL) = 0 [pid 1082] poll([{fd=18, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout) [pid 1082] sched_yield() ``` It seems that the process is waiting for some data that never comes. The fd=18 is referencing to a TCP socket connected to provider1. So it would suggests that the provider is not sending what the consumer wants. This has led me to several suppositions: 1. Network issues Unlikely, since only those 2 servers in the DC are experiencing misbehavior, and I can telnet / ldapsearch from the faulty consumer to the provider with no problem during hangs. 2. Syncrepl / slapo-chain misconfiguration I might have a configuration problem, but no matter how much I review it, I don't see what's wrong. I've provided the syncrepl / slapo-chain conf in the config.ldif attachment. I've also provided several other attachments that I thought would be helpful: - log.txt => The slapd logs in trace level that show the last bind before the service hangs. - ldaps_request.log => An ldapsearch performed against the faulty consumer during hang. - starttls_request.log => Same as above, but using StartTLS => the connection seems successful and stops at the server hello TLS exchange. Versions : - slapd: 2.5.18 LTS - Server: Ubuntu 22.04 LTS I am a bit out of ideas on things I could try to fix the issues. Hence this email. I thank you for your time and I am very grateful for any help you could give me. Regards, Pierre-Jean

1 0

"realdn" in auditlog
by Christopher Paul 17 Sep '24

17 Sep '24

Hi OpenLDAP-Technical, What does a comment beginning with "realdn" mean in an auditlog LDIF? I see it after the first line of COMMENT FIELD INFORMATION on some modify operations. Running version 2.5.18 on Ubuntu. At first glance, it looks like it's due to overlays (refint, ppolicy) making modifications as a result of an operation by "realdn"(?)b Many thanks for any insight on this. -- Chris Paul | Rex Consulting |https://www.rexconsulting.net ParaphraseBasicClose

2 1

Double free in ldap_set_option
by radiatejava 16 Sep '24

16 Sep '24

I am getting double free core from ldap library. We are running this version of the code (2.4.49 with 2 patches). git clone https://github.com/openldap/openldap.git -b OPENLDAP_REL_ENG_2_4_49 git cherry-pick 32e965c27134d3d83adbf471186330e361150491 git cherry-pick 5452fb154eaabc46992402a4f1bb1081bc367157 Any idea what could be going on? I am putting the stack trace below: #2 0x00007a1e506d7590 in DoLiveCoreDump (cause=0x7a1e506df7e2 "DoubleFree") at src/heap-profiler.cc:732 #3 0x00007a1e506bed31 in (anonymous namespace)::do_free_with_callback (ptr=ptr@entry=0x5af3d4216dc8, invalid_free_fn=0x7a1e506bd270 <(anonymous namespace)::InvalidFree(void*)>) at src/tcmalloc.cc:1299 #4 0x00007a1e506dd467 in (anonymous namespace)::do_free (ptr=0x5af3d4216dc8) at src/tcmalloc.cc:1376 #5 tc_free (ptr=0x5af3d4216dc8) at src/tcmalloc.cc:1761 #6 0x00007a1e3f19106b in ldap_pvt_tls_set_option (ld=ld@entry=0x0, option=option@entry=24579, arg=arg@entry=0x7a1e416e2220) at tls2.c:766 #7 0x00007a1e3f186386 in ldap_set_option (ld=0x0, option=24579, invalue=0x7a1e416e2220) at options.c:791 #8 0x00007a1e415e03fd in auth::LdapAuthenticationBroker::connectAndBindLdap(sec::LdapProvider const&) () from /mgmt/usr/lib/x86_64-linux-gnu/libauth.so

2 1

Search callbacks for refreshAndPersist sync searches
by Hawes, David 10 Sep '24

10 Sep '24

I have a custom overlay that registers a callback to rewrite attribute values for searches. I noticed that it was not working for refreshAndPersist sync searches for queued responses, and the reason for this is that there are no callbacks registered in syncprov_qtask() [1]. I'd see the initial responses come through with rewritten attributes, but later responses would not be rewritten. I had some success adding the callbacks back, but it was a bit of a minefield. Before investigating further, I was hoping to understand why the decision was made to not set callbacks in syncprov_qtask(). For now I'm operating under the assumption that this is simply not supported. So, why aren't callbacks supported for queued sync responses? I noticed that valsort doesn't handle syncrepl responses [2], and I wonder if the reason is because the results may be inconsistent, as I am experiencing. [1] https://git.openldap.org/openldap/openldap/-/blob/OPENLDAP_REL_ENG_2_5/serv… [2] https://git.openldap.org/openldap/openldap/-/blob/OPENLDAP_REL_ENG_2_5/serv…

2 3

certutil - nsldap + nss trustargs
by venugopal chinnakotla 04 Sep '24

04 Sep '24

Hi Team, We are migrating our source code to use openldap client libraries instead of nsldap libraries. We have the below query for the openldap team. Please share your thoughts on this feature availability in the openldap library. nsldap has the below feature while validating TLS Connections. please refer to the nss tool, and refer to the section "Options and Arguments" https://www-archive.mozilla.org/projects/security/pki/nss/tools/certutil -t trustargs Specify the trust attributes to modify in an existing certificate or to apply to a certificate when creating it or adding it to a database. There are three available trust categories for each certificate, expressed in this order: "SSL, email, object signing". In each category position use zero or more of the following attribute codes: p Valid peer P Trusted peer (implies p) c Valid CA T Trusted CA to issue client certificates (implies c) C Trusted CA to issue server certificates (SSL only) (implies c) u Certificate can be used for authentication or signing w Send warning (use with other attributes to include a warning when the certificate is used in that context) The attribute codes for the categories are separated by commas, and the entire set of attributes enclosed by quotation marks. For example: -t "TCu,Cu,Tuw" Use the -L option to see a list of the current certificates and trust attributes in a certificate database. our query: Does openldap provide trustargs support to verify TLS connection? Please share details on this. Thanks In advance. -- Thanks, *c.venugopal*

4 3

Syncrepl getting stuck on partial sync
by jose Pereira 04 Sep '24

04 Sep '24

Hello All! I have multiple setups with a Provider(2.6.4) and consumer(2.6.2) with RefreshAndPersist. From time to time I see data missing in the consumer e.g only groups are present and all users are missing, even though users were never removed from the Provider, and were present in the Consumer beforehand. On the Provider all data is present and correct. I can still see the incoming connection, from the Consumers Sync, in the Provider without errors. No errors on the Comsumer logs from syncrepl (with -256 logs). Attempted to look in debug (-1 logs), but nothing caught my eye. When this happened I attempted restart services on both the Provider and the Consumer, disabling and re-enabling the sync on the Consumer. Only clearing the Consumers database seemed to fix the issue. There maybe be sporatic failures in the network connectivity between both systems. This is the config on the Consumer olcSyncrepl: {0}rid=000 provider=ldaps://Provider:636 type=refreshAndPersist retry="30 5 300 +" searchbase="ou=subtree,dc=test,dc=net" scope=sub schemachecking=on binddn="cn=Manager,dc=test,dc=net" credentials=credentials keepalive=240:10:30 Any advice on how to troubleshoot this or fix it?

4 18

FW: [External] RE: Help Openldap objectclass schema add
by Khan, Afaque 04 Sep '24

04 Sep '24

Hi Openldap- Version 2.6.7 I am using below command to change the current running port to 2636 but I am getting below error message Error: name '_pretty_exception' is not defined dsconf -D "cn=admin..... " ldap://(hostname) config replace nsslapd-secureport=2636 Please let me know the command or steps to change the port and also please let me know where to add below local objectclass created file to add new objectclass in schema I am not able to find the slapd.conf(5) or slapd.conf(8) file to include below schema objectclass file '/usr/local/etc/openldap/schema/local.schema' Thanks Afaque khan From: Chris Paul <chris.paul(a)rexconsulting.net<mailto:chris.paul@rexconsulting.net>> Sent: Tuesday, September 3, 2024 1:10 AM Subject: [External] RE: Help Openldap objectclass schema add You don't often get email from chris.paul(a)rexconsulting.net<mailto:chris.paul@rexconsulting.net>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> WARNING: This message has originated from an External Source. This may be a phishing email that can result in unauthorized access to Honeywell systems. Please use proper judgment and caution when opening attachments, clicking links, scanning QR codes, or responding. Hi Afaque, You have not indicated what you have tried, given any error, your configuration, or useful information except that everyone now knows that you work for..... Have you read the OpenLDAP admin guide, and the man pages? When I post to openldap-technical(a)openldap.org<mailto:openldap-technical@openldap.org>, I never reveal my clients' identity, for their protection. Yes, obscurity is a layer of security. Good luck to you and....., -- Chris Paul | https://www.rexconsulting.net Sent: Monday, September 2, 2024 3:13 AM To: openldap-technical-request(a)openldap.org<mailto:openldap-technical-request@openldap.org>; openldap-technical(a)openldap.org<mailto:openldap-technical@openldap.org> Subject: Help Openldap objectclass schema add Hi Openldap Team, I have installed the openldap openldap-2.6.7 version on RHEL 8 but I am not able to add the new objectclass hworgperson and attribute manager and hwtype. Please let me know how I can add the new objectclass and attributes. Help -----Original Message----- From: openldap-technical-request(a)openldap.org<mailto:openldap-technical-request@openldap.org> <openldap-technical-request(a)openldap.org<mailto:openldap-technical-request@openldap.org>> Sent: Monday, September 2, 2024 3:37 PM Subject: [External] Welcome to the "openldap-technical" mailing list WARNING: This message has originated from an External Source. This may be a phishing email that can result in unauthorized access to Honeywell systems. Please use proper judgment and caution when opening attachments, clicking links, scanning QR codes, or responding. Welcome to the "openldap-technical" mailing list! To post to this list, send your message to: openldap-technical(a)openldap.org<mailto:openldap-technical@openldap.org> You can unsubscribe or make adjustments to your options via email by sending a message to: openldap-technical-request(a)openldap.org<mailto:openldap-technical-request@openldap.org> with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI.

2 1

Help Openldap objectclass schema add
by Khan, Afaque 04 Sep '24

04 Sep '24

Hi Openldap Team, I have installed the openldap openldap-2.6.7 version on RHEL 8 but I am not able to add the new objectclass hworgperson and attribute manager and hwtype. Please let me know how I can add the new objectclass and attributes. Help Thanks & Regards, Afaque khan Advanced Cyber Sec Archt/Engr Honeywell | Honeywell Global Security Mobile: +91 9599800986 md.khan(a)honeywell.com<mailto:md.khan@honeywell.com> Thanks & Regards, Afaque khan Advanced Cyber Sec Archt/Engr Honeywell | Honeywell Global Security Mobile: +91 9599800986 md.khan(a)honeywell.com<mailto:md.khan@honeywell.com> -----Original Message----- From: openldap-technical-request(a)openldap.org <openldap-technical-request(a)openldap.org> Sent: Monday, September 2, 2024 3:37 PM To: Khan, Afaque <md.khan(a)honeywell.com> Subject: [External] Welcome to the "openldap-technical" mailing list WARNING: This message has originated from an External Source. This may be a phishing email that can result in unauthorized access to Honeywell systems. Please use proper judgment and caution when opening attachments, clicking links, scanning QR codes, or responding. Welcome to the "openldap-technical" mailing list! To post to this list, send your message to: openldap-technical(a)openldap.org<mailto:openldap-technical@openldap.org> You can unsubscribe or make adjustments to your options via email by sending a message to: openldap-technical-request(a)openldap.org<mailto:openldap-technical-request@openldap.org> with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI.

2 1

long restart time of openldap
by Norbert 29 Aug '24

29 Aug '24

Hi, we have ldap servers with many ldap entries (3 million and more) these are replicating with other instances (Master-Master + Read slaves). When we need to restart and ldap due to maintenance or other problems those are taking a lot of time before they serving a requests again. It looks like that in most cases it does a full refresh of the whole tree (printed to the log) but there are is also a timespan where nothing is printed to the log but the slapd process is really busy and consuming more than one core. We are currently using 2.5.16.1 of openldap but have experienced this with older version too. Any hint about how to improve this? Thanks, Norbert

2 5

LMDB write commit time hockey-sticks [corrected]
by Alec Matusis 22 Aug '24

22 Aug '24

Problem: Our application first writes data into LMDB. After the write cursor is closed, the LMDB is used with a read-only cursor. Our problem is that writing commit time hockey-sticks. Here is the graph on the machine with 128GB of RAM: https://i.imgur.com/py83dRV.jpeg (happens at 3200) We used a virtual machine with 16 GB, to wait less for this hockey-stick: https://i.imgur.com/CYohM3O.jpeg (red curve is the commit time, happens at 3080; the lmdb env stats are on the bottom graph. Size on the top graph is total program size from /proc/[pid]/statm) . We tried various combinations of environment flags: MDB_WRITEMAP, MDB_MAPASYNC, MDB_NOSYNC and MDB_NOMETASYNC. We also tried increasing mapsize. Nothing seems to matter (e.g. https://i.imgur.com/MkZJkxp.jpeg with 64G mapsize and MDB_NOSYNC), the best combination seems to be without any flags at all. Smaller mapsizes are worse, but increasing mapsize above 4GB in a 16GB RAM machine doesn't seem to gain anything. Can you recommend a way to prevent or delay this hockey-stick when we are in the write regime and the reads are very minimal? We just want to write the data to disk fast, we don’t even need mmap() for read performance, we will need it in the read-only mode later, after the write cursor finishes and a read-only cursor is used. Details: The environment contains two databases, one with no flags (keys are 32 bytes, values are 8 byte integers), second one with MDB_INTEGERKEY (keys are 8 byte integers, values are 32 bytes, uses MDB_APPEND for writing). All commits are similar sized, 6000-8000 pages (page size is 4096). Each commit therefore is ~400k keys total for boths dbs, (7000*4096/((32+8)*2). The VM has no disk swap, from perf output here https://i.imgur.com/zuTGawz.jpeg we can see that the machine is under page swap load. On a VM with 16GB RAM, hockey-stick happens when the LMDB size is only 1.5GB, 12% of the available RAM. Detailed stats: Below is the excerpt from the application log on each commit, the same data is plotted in the linked graphs above. Memory stats are from. /proc/[pid]/statm , lmdb stats are from the environment.“Difference” the time the commit takes, it degrades from 45 seconds to 3+ minutes. Final LMDB size in this log is pgs_in_use 448993*4096 = 1.84GB. Start of log before write degradation: 2024-08-09 02:26:19 [INFO][src/importer.rs:198] File 3054: 132932384 bytes. Memory stats: Size: 27540910 pages, Resident: 286072 pages, Shared: 99484 pages, Text: 981 pages, Data: 191849 pages, lmdb: dict_env: branch_pgs 0 leaf_pgs 1 of_pgs 0 last_pgno 108959 last_txnid 18 max_readers 126 num_readers 1 free_pgs 56171 pgs_in_use 52788 Difference: 0 min 44 sec 2024-08-09 02:27:03 [INFO][src/importer.rs:198] File 3055: 133951354 bytes. Memory stats: Size: 27541730 pages, Resident: 298463 pages, Shared: 111020 pages, Text: 981 pages, Data: 192669 pages, lmdb: dict_env: branch_pgs 0 leaf_pgs 1 of_pg s 0 last_pgno 121928 last_txnid 19 max_readers 126 num_readers 1 free_pgs 62881 pgs_in_use 59047 Difference: 0 min 48 sec 2024-08-09 02:27:51 [INFO][src/importer.rs:198] File 3056: 133218473 bytes. Memory stats: Size: 27543735 pages, Resident: 313441 pages, Shared: 123980 pages, Text: 981 pages, Data: 194674 pages, lmdb: dict_env: branch_pgs 0 leaf_pgs 1 of_pg s 0 last_pgno 133243 last_txnid 20 max_readers 126 num_readers 1 free_pgs 69672 pgs_in_use 63571 Difference: 0 min 40 sec …… End of log after write degradation: 2024-08-09 04:39:33 [INFO][src/importer.rs:198] File 3127: 132694310 bytes. Memory stats: Size: 27654397 pages, Resident: 1028693 pages, Shared: 727992 pages, Text: 981 pages, Data: 305336 pages, lmdb: dict_env: branch_pgs 0 leaf_pgs 1 of_pgs 0 last_pgno 745440 last_txnid 91 max_readers 126 num_readers 1 free_pgs 308699 pgs_in_use 436741 Difference: 2 min 36 sec 2024-08-09 04:42:09 [INFO][src/importer.rs:198] File 3128: 133418687 bytes. Memory stats: Size: 27654397 pages, Resident: 1033516 pages, Shared: 732815 pages, Text: 981 pages, Data: 305336 pages, lmdb: dict_env: branch_pgs 0 leaf_pgs 1 of_pgs 0 last_pgno 745440 last_txnid 92 max_readers 126 num_readers 1 free_pgs 304043 pgs_in_use 441397 Difference: 3 min 18 sec 2024-08-09 04:45:27 [INFO][src/importer.rs:198] File 3129: 133118782 bytes. Memory stats: Size: 27654397 pages, Resident: 1043876 pages, Shared: 743175 pages, Text: 981 pages, Data: 305336 pages, lmdb: dict_env: branch_pgs 0 leaf_pgs 1 of_pgs 0 last_pgno 762922 last_txnid 93 max_readers 126 num_readers 1 free_pgs 313929 pgs_in_use 448993

1 1

← Newer
1
2
3
4
5
6
7
8
...
979
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical