All, I have been struggling with upgrading OpenLDAP from 2.4 to 2.5/2.6 for some time. We have finally found that we needed to rebuild the schema from scratch and re-add our customizations. The database is now running much better with one lingering problem. Our Established connections just continues to grow until we run out of resources. Below is our cn=config (minus some unrelated info). This is on the same server as where the previous version was running, so changes are openldap and openssl versions. Any insights as to what might be causing the ESTABLISHED connections to continually grow would be very appreciated.
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify. # CRC32 422b88f4 dn: cn=config objectClass: olcGlobal cn: config olcAttributeOptions: lang- olcConcurrency: 0 olcConnMaxPending: 100 olcConnMaxPendingAuth: 1000 olcGentleHUP: FALSE olcIdleTimeout: 0 olcIndexSubstrIfMaxLen: 4 olcIndexSubstrIfMinLen: 2 olcIndexSubstrAnyLen: 4 olcIndexSubstrAnyStep: 2 olcIndexHash64: FALSE olcIndexIntLen: 4 olcListenerThreads: 1 olcLocalSSF: 71 olcLogLevel: 256 olcLogFileOnly: FALSE olcMaxFilterDepth: 1000 olcReadOnly: FALSE olcSaslAuxpropsDontUseCopyIgnore: FALSE olcSaslSecProps: noplain,noanonymous olcSockbufMaxIncoming: 262143 olcSockbufMaxIncomingAuth: 16777215 olcThreads: 16 olcThreadQueues: 1 olcTLSCRLCheck: none olcTLSVerifyClient: never olcTLSProtocolMin: 0.0 olcToolThreads: 1 structuralObjectClass: olcGlobal creatorsName: cn=config createTimestamp: 20220726200129Z olcAuthzPolicy: any olcWriteTimeout: 30 olcSizeLimit: size.soft=unlimited size.hard=unlimited size.unchecked=unlimited size.pr=1000 size.prtotal=unlimited
Bradley T Gill bgill@aep.com schrieb am 27.07.2022 um 15:59 in Nachricht
84030354e2e44d13b5463c6c070e36cc@aep.com:
All, I have been struggling with upgrading OpenLDAP from 2.4 to 2.5/2.6
for some time. We have finally found that we needed to rebuild the schema from scratch and re‑add our customizations. The database is now running
much
better with one lingering problem. Our Established connections just continues to grow until we run out of resources. Below is our cn=config (minus some unrelated info). This is on the same server as where the previous version was running, so changes are openldap and openssl versions.
Any insights as to what might be causing the ESTABLISHED connections to continually grow would be very appreciated.
# AUTO‑GENERATED FILE ‑ DO NOT EDIT!! Use ldapmodify. # CRC32 422b88f4 dn: cn=config objectClass: olcGlobal cn: config olcAttributeOptions: lang‑ olcConcurrency: 0 olcConnMaxPending: 100 olcConnMaxPendingAuth: 1000 olcGentleHUP: FALSE olcIdleTimeout: 0
What about "olcIdleTimeout: 86400" (or even more aggressive)? In the past we had cases where applications opened new LDAP connections, never reused or closed old ones.
olcIndexSubstrIfMaxLen: 4 olcIndexSubstrIfMinLen: 2 olcIndexSubstrAnyLen: 4 olcIndexSubstrAnyStep: 2 olcIndexHash64: FALSE olcIndexIntLen: 4 olcListenerThreads: 1 olcLocalSSF: 71 olcLogLevel: 256 olcLogFileOnly: FALSE olcMaxFilterDepth: 1000 olcReadOnly: FALSE olcSaslAuxpropsDontUseCopyIgnore: FALSE olcSaslSecProps: noplain,noanonymous olcSockbufMaxIncoming: 262143 olcSockbufMaxIncomingAuth: 16777215 olcThreads: 16 olcThreadQueues: 1 olcTLSCRLCheck: none olcTLSVerifyClient: never olcTLSProtocolMin: 0.0 olcToolThreads: 1 structuralObjectClass: olcGlobal creatorsName: cn=config createTimestamp: 20220726200129Z olcAuthzPolicy: any olcWriteTimeout: 30 olcSizeLimit: size.soft=unlimited size.hard=unlimited size.unchecked=unlimited size.pr=1000 size.prtotal=unlimited
Ulrich, Thanks for the response. I have been considering that setting, but it is unchanged from the previous version. I am concerned that it might break applications that are expecting a persistent connection. It also seems to be growing so fast that there is a fundamental change somewhere. I will add that I found that some overlay configs were missing including a relay for Oracle searches. I am hoping that not knowing how to handle the requests created the problem. Now that it is fixed, I am monitoring the ports.
Thanks, Bradley Gill, CISSP, CCSP
-----Original Message----- From: Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de Sent: Thursday, July 28, 2022 5:35 AM To: Bradley T Gill bgill@aep.com; openldap-technical@openldap.org Subject: [EXTERNAL] Antw: [EXT] OpenLDAP 2.6 is holding connections open
This is an EXTERNAL email. STOP. THINK before you CLICK links or OPEN attachments. If suspicious please click the 'Report to Incidents' button in Outlook or forward to incidents@aep.com from a mobile device.
Bradley T Gill bgill@aep.com schrieb am 27.07.2022 um 15:59 in Nachricht
84030354e2e44d13b5463c6c070e36cc@aep.com:
All, I have been struggling with upgrading OpenLDAP from 2.4 to 2.5/2.6
for some time. We have finally found that we needed to rebuild the schema from scratch and re‑add our customizations. The database is now running
much
better with one lingering problem. Our Established connections just continues to grow until we run out of resources. Below is our cn=config (minus some unrelated info). This is on the same server as where the previous version was running, so changes are openldap and openssl versions.
Any insights as to what might be causing the ESTABLISHED connections to continually grow would be very appreciated.
# AUTO‑GENERATED FILE ‑ DO NOT EDIT!! Use ldapmodify. # CRC32 422b88f4 dn: cn=config objectClass: olcGlobal cn: config olcAttributeOptions: lang‑ olcConcurrency: 0 olcConnMaxPending: 100 olcConnMaxPendingAuth: 1000 olcGentleHUP: FALSE olcIdleTimeout: 0
What about "olcIdleTimeout: 86400" (or even more aggressive)? In the past we had cases where applications opened new LDAP connections, never reused or closed old ones.
olcIndexSubstrIfMaxLen: 4 olcIndexSubstrIfMinLen: 2 olcIndexSubstrAnyLen: 4 olcIndexSubstrAnyStep: 2 olcIndexHash64: FALSE olcIndexIntLen: 4 olcListenerThreads: 1 olcLocalSSF: 71 olcLogLevel: 256 olcLogFileOnly: FALSE olcMaxFilterDepth: 1000 olcReadOnly: FALSE olcSaslAuxpropsDontUseCopyIgnore: FALSE olcSaslSecProps: noplain,noanonymous olcSockbufMaxIncoming: 262143 olcSockbufMaxIncomingAuth: 16777215 olcThreads: 16 olcThreadQueues: 1 olcTLSCRLCheck: none olcTLSVerifyClient: never olcTLSProtocolMin: 0.0 olcToolThreads: 1 structuralObjectClass: olcGlobal creatorsName: cn=config createTimestamp: 20220726200129Z olcAuthzPolicy: any olcWriteTimeout: 30 olcSizeLimit: size.soft=unlimited size.hard=unlimited size.unchecked=unlimited size.pr=1000 size.prtotal=unlimited
On Thu, 28 Jul 2022, Bradley T Gill wrote:
I will add that I found that some overlay configs were missing including a relay for Oracle searches. I am hoping t hat not knowing how to handle the requests created the problem. Now that it is fixed, I am monitoring the ports.
ISRT that the ldap backend woiuld simply never reply to queries if configured with no uris. No error, just never reply, leaving connections open.
Philip Guenther
Bradley T Gill bgill@aep.com schrieb am 28.07.2022 um 14:17 in Nachricht
6777136244004f3ba0fd72253ee0ed7d@aep.com:
Ulrich, Thanks for the response. I have been considering that setting, but it
is
unchanged from the previous version. I am concerned that it might break applications that are expecting a persistent connection. It also seems to
be growing so fast that there is a fundamental change somewhere.
Hi!
I had similar concerns; that's why I used the large value. But as an LDAP server can go down, clients should be able to handle if a connection goes down. Also (as I understand it), one request per day would keep the connection alive. OTOH: Does a connection that is not transferring a single packet within 24 hours have a justification to exist (consuming system resources on the server)?
I will add that I found that some overlay configs were missing
including a
relay for Oracle searches. I am hoping that not knowing how to handle the requests created the problem. Now that it is fixed, I am monitoring the ports.
Maybe try step 1: Who is opening those connections (on the client)? "netstat -tnp" on Linux if you still have netstat.
Regards, Ulrich
Thanks, Bradley Gill, CISSP, CCSP
-----Original Message----- From: Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de Sent: Thursday, July 28, 2022 5:35 AM To: Bradley T Gill bgill@aep.com; openldap-technical@openldap.org Subject: [EXTERNAL] Antw: [EXT] OpenLDAP 2.6 is holding connections open
This is an EXTERNAL email. STOP. THINK before you CLICK links or OPEN attachments. If suspicious please click the 'Report to Incidents' button in
Outlook or forward to incidents@aep.com from a mobile device.
Bradley T Gill bgill@aep.com schrieb am 27.07.2022 um 15:59 in Nachricht
84030354e2e44d13b5463c6c070e36cc@aep.com:
All, I have been struggling with upgrading OpenLDAP from 2.4 to 2.5/2.6
for some time. We have finally found that we needed to rebuild the schema
from scratch and re‑add our customizations. The database is now running
much
better with one lingering problem. Our Established connections just continues to grow until we run out of resources. Below is our cn=config (minus some unrelated info). This is on the same server as where the previous version was running, so changes are openldap and openssl
versions.
Any insights as to what might be causing the ESTABLISHED connections to continually grow would be very appreciated.
# AUTO‑GENERATED FILE ‑ DO NOT EDIT!! Use ldapmodify. # CRC32 422b88f4 dn: cn=config objectClass: olcGlobal cn: config olcAttributeOptions: lang‑ olcConcurrency: 0 olcConnMaxPending: 100 olcConnMaxPendingAuth: 1000 olcGentleHUP: FALSE olcIdleTimeout: 0
What about "olcIdleTimeout: 86400" (or even more aggressive)? In the past we had cases where applications opened new LDAP connections, never reused or closed old ones.
olcIndexSubstrIfMaxLen: 4 olcIndexSubstrIfMinLen: 2 olcIndexSubstrAnyLen: 4 olcIndexSubstrAnyStep: 2 olcIndexHash64: FALSE olcIndexIntLen: 4 olcListenerThreads: 1 olcLocalSSF: 71 olcLogLevel: 256 olcLogFileOnly: FALSE olcMaxFilterDepth: 1000 olcReadOnly: FALSE olcSaslAuxpropsDontUseCopyIgnore: FALSE olcSaslSecProps: noplain,noanonymous olcSockbufMaxIncoming: 262143 olcSockbufMaxIncomingAuth: 16777215 olcThreads: 16 olcThreadQueues: 1 olcTLSCRLCheck: none olcTLSVerifyClient: never olcTLSProtocolMin: 0.0 olcToolThreads: 1 structuralObjectClass: olcGlobal creatorsName: cn=config createTimestamp: 20220726200129Z olcAuthzPolicy: any olcWriteTimeout: 30 olcSizeLimit: size.soft=unlimited size.hard=unlimited size.unchecked=unlimited size.pr=1000 size.prtotal=unlimited
It appears that the problem was created by missing overlays. When I rebuild the schema, I think I missed part of the mdb. When I added back my overlays, it fixed the port issue, but brought back the slowness (looks like a problem with dynlist that I am going to create a new thread for). For the record, I think that the overlay that caused the port issues was a relay that created a dc=Oracle,dc=com overlay for legacy Oracle clients to do onames lookups.
Thanks,
Bradley Gill CISSP, CCSP
-----Original Message----- From: Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de Sent: Friday, July 29, 2022 1:52 AM To: Bradley T Gill bgill@aep.com; openldap-technical@openldap.org Subject: Antw: RE: [EXTERNAL] Antw: [EXT] OpenLDAP 2.6 is holding connections open
This is an EXTERNAL email. STOP. THINK before you CLICK links or OPEN attachments. If suspicious please click the 'Report to Incidents' button in Outlook or forward to incidents@aep.com from a mobile device.
Bradley T Gill bgill@aep.com schrieb am 28.07.2022 um 14:17 in Nachricht
6777136244004f3ba0fd72253ee0ed7d@aep.com:
Ulrich, Thanks for the response. I have been considering that setting, but it
is
unchanged from the previous version. I am concerned that it might break applications that are expecting a persistent connection. It also seems to
be growing so fast that there is a fundamental change somewhere.
Hi!
I had similar concerns; that's why I used the large value. But as an LDAP server can go down, clients should be able to handle if a connection goes down. Also (as I understand it), one request per day would keep the connection alive. OTOH: Does a connection that is not transferring a single packet within 24 hours have a justification to exist (consuming system resources on the server)?
I will add that I found that some overlay configs were missing
including a
relay for Oracle searches. I am hoping that not knowing how to handle the requests created the problem. Now that it is fixed, I am monitoring the ports.
Maybe try step 1: Who is opening those connections (on the client)? "netstat -tnp" on Linux if you still have netstat.
Regards, Ulrich
Thanks, Bradley Gill, CISSP, CCSP
-----Original Message----- From: Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de Sent: Thursday, July 28, 2022 5:35 AM To: Bradley T Gill bgill@aep.com; openldap-technical@openldap.org Subject: [EXTERNAL] Antw: [EXT] OpenLDAP 2.6 is holding connections open
This is an EXTERNAL email. STOP. THINK before you CLICK links or OPEN attachments. If suspicious please click the 'Report to Incidents' button in
Outlook or forward to incidents@aep.com from a mobile device.
Bradley T Gill bgill@aep.com schrieb am 27.07.2022 um 15:59 in Nachricht
84030354e2e44d13b5463c6c070e36cc@aep.com:
All, I have been struggling with upgrading OpenLDAP from 2.4 to 2.5/2.6
for some time. We have finally found that we needed to rebuild the schema
from scratch and re‑add our customizations. The database is now running
much
better with one lingering problem. Our Established connections just continues to grow until we run out of resources. Below is our cn=config (minus some unrelated info). This is on the same server as where the previous version was running, so changes are openldap and openssl
versions.
Any insights as to what might be causing the ESTABLISHED connections to continually grow would be very appreciated.
# AUTO‑GENERATED FILE ‑ DO NOT EDIT!! Use ldapmodify. # CRC32 422b88f4 dn: cn=config objectClass: olcGlobal cn: config olcAttributeOptions: lang‑ olcConcurrency: 0 olcConnMaxPending: 100 olcConnMaxPendingAuth: 1000 olcGentleHUP: FALSE olcIdleTimeout: 0
What about "olcIdleTimeout: 86400" (or even more aggressive)? In the past we had cases where applications opened new LDAP connections, never reused or closed old ones.
olcIndexSubstrIfMaxLen: 4 olcIndexSubstrIfMinLen: 2 olcIndexSubstrAnyLen: 4 olcIndexSubstrAnyStep: 2 olcIndexHash64: FALSE olcIndexIntLen: 4 olcListenerThreads: 1 olcLocalSSF: 71 olcLogLevel: 256 olcLogFileOnly: FALSE olcMaxFilterDepth: 1000 olcReadOnly: FALSE olcSaslAuxpropsDontUseCopyIgnore: FALSE olcSaslSecProps: noplain,noanonymous olcSockbufMaxIncoming: 262143 olcSockbufMaxIncomingAuth: 16777215 olcThreads: 16 olcThreadQueues: 1 olcTLSCRLCheck: none olcTLSVerifyClient: never olcTLSProtocolMin: 0.0 olcToolThreads: 1 structuralObjectClass: olcGlobal creatorsName: cn=config createTimestamp: 20220726200129Z olcAuthzPolicy: any olcWriteTimeout: 30 olcSizeLimit: size.soft=unlimited size.hard=unlimited size.unchecked=unlimited size.pr=1000 size.prtotal=unlimited
openldap-technical@openldap.org
-
Bradley T Gill -
Philip Guenther -
Ulrich Windl