Hi,
After upgrading to the latest Release (Solaris 11.3 SRU35, OpenLDAP 2.4.45) we are experiencing massive workloads caused by single clients consuming all available threads and CPU-resources. Service does not longer respond to requests, even cn=monitor on loopback interface stops to respond properly.
OS: # pkg info entire Name: entire Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.35.6.0). Description: This package constrains system package versions to the same build. WARNING: Proper system update and correct package selection depend on the presence of this incorporation. Removing this package will result in an unsupported system. For more information see: https://support.oracle.com/rs?type=doc&id=2045311.1 Category: Meta Packages/Incorporations State: Installed Publisher: solaris Version: 0.5.11 (Oracle Solaris 11.3.35.6.0) Build Release: 5.11 Branch: 0.175.3.35.0.6.0 Packaging Date: August 10, 2018 03:22:59 PM Size: 5.46 kB FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.35.0.6.0:20180810T152259Z
OpenSSL: # pkg info entire Name: entire Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.35.6.0). Description: This package constrains system package versions to the same build. WARNING: Proper system update and correct package selection depend on the presence of this incorporation. Removing this package will result in an unsupported system. For more information see: https://support.oracle.com/rs?type=doc&id=2045311.1 Category: Meta Packages/Incorporations State: Installed Publisher: solaris Version: 0.5.11 (Oracle Solaris 11.3.35.6.0) Build Release: 5.11 Branch: 0.175.3.35.0.6.0 Packaging Date: August 10, 2018 03:22:59 PM Size: 5.46 kB FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.35.0.6.0:20180810T152259Z
OpenLDAP: # pkg info entire Name: entire Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.35.6.0). Description: This package constrains system package versions to the same build. WARNING: Proper system update and correct package selection depend on the presence of this incorporation. Removing this package will result in an unsupported system. For more information see: https://support.oracle.com/rs?type=doc&id=2045311.1 Category: Meta Packages/Incorporations State: Installed Publisher: solaris Version: 0.5.11 (Oracle Solaris 11.3.35.6.0) Build Release: 5.11 Branch: 0.175.3.35.0.6.0 Packaging Date: August 10, 2018 03:22:59 PM Size: 5.46 kB FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.35.0.6.0:20180810T152259Z
Part of slapd.conf: loglevel none stats sync sizelimit 15000 timelimit 30 threads 64 tool-threads 8 idletimeout 0 writetimeout 0
security tls=0
conn_max_pending 100 conn_max_pending_auth 1000
database mdb suffix "dc=scom" rootdn "cn=*****" rootpw {SSHA}*****
maxsize 17179869184 maxreaders 4096 searchstack 64 checkpoint 0 1 dbnosync
Machine is a X6-2, 44 cores, 88 threads, 256GB RAM: # prtdiag System Configuration: Oracle Corporation ORACLE SERVER X6-2 BIOS Configuration: American Megatrends Inc. 38070000 12/16/2016 BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style)
==== Processor Sockets ====================================
Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz P0 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz P1
Even monitoring (cn=monitor) is no longer accessible when this occurs.
So far we experienced this behavior with clients of Oracle Enterprise Linux 6.x, Redhat Enterprise Linux 6.x and AIX. Service requests are opened at vendors support, but I'd prefer to have an installation which is less vulnerable and more resilient to issues of this kind.
No problems or issues with Solaris and HPUX clients.
Has anyone experienced similar problems or suggestions for configuration?
To avoid performance issues loglevel is now "none stats sync" but can be changed for some time to track down the cause.
Best regards
Jürgen Sprenger
Hi,
Additional information about OpenLDAP relase used.
OpenLDAP:
PROD:root@spodsscm104:~# /usr/lib/amd64/slapd -VVV @(#) $OpenLDAP: slapd 2.4.45 (Jun 7 2018 02:20:06) $ @sig-ul-sru11-3-x01:/builds/ul11u3sru-gate/components/openldap/build/amd64/servers/slapd
Included static overlays: accesslog auditlog collect constraint dds deref dyngroup dynlist memberof ppolicy pcache refint retcode rwm seqmod sssvlv syncprov translucent unique valsort Included static backends: config ldif monitor ldap mdb meta null passwd relay shell
Best regards
Jürgen Sprenger
--On Wednesday, January 30, 2019 9:43 AM +0000 Juergen.Sprenger@swisscom.com wrote:
Hi,
After upgrading to the latest Release (Solaris 11.3 SRU35, OpenLDAP 2.4.45) we are experiencing massive workloads caused by single clients consuming all available threads and CPU-resources.
Solaris 11.4 is out (just to note). And the latest OpenLDAP release is 2.4.47 (just to note).
tool-threads 8
A tool-threads setting > 2 is ignored with back-mdb.
maxreaders 4096
This is an extremely large value (default is 126). Are you sure you need this bumped so high?
searchstack 64
Similarly here.
So far we experienced this behavior with clients of Oracle Enterprise Linux 6.x, Redhat Enterprise Linux 6.x and AIX. Service requests are opened at vendors support, but I'd prefer to have an installation which is less vulnerable and more resilient to issues of this kind.
I'd estimate that your ability to get a useful response from either is near zero. If you have an enterprise operation that needs support for OpenLDAP you may want to look at Symas' enterprise OpenLDAP builds (Symas Gold). We provide builds and support for Solaris 11.
To avoid performance issues loglevel is now "none stats sync" but can be changed for some time to track down the cause.
You could just do "stats sync", not really a reason to leave "none" in the list.
Generally if you hit an issue such as this, you would want to provde a full backtrace of the slapd process from a utility such as GDB.
Warm regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Hi,
Anyway, I consider this issue a denial of service vulnerability and OpenLDAP 2.4.45 will not be used for production here.
The only major change in configuration was switching from hdb to mdb, so I will have to check whether mdb may be the cause of this issue.
Regards Juergen
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@symas.com] Sent: Mittwoch, 30. Januar 2019 15:53 To: Sprenger Jürgen, INI-ONE-CIS-SDI-HES Juergen.Sprenger@swisscom.com; openldap-technical@openldap.org Subject: Re: OpenLDAP 2.4.45 possible denial of service vulnerability?
--On Wednesday, January 30, 2019 9:43 AM +0000 Juergen.Sprenger@swisscom.com wrote:
Hi,
After upgrading to the latest Release (Solaris 11.3 SRU35, OpenLDAP 2.4.45) we are experiencing massive workloads caused by single clients consuming all available threads and CPU-resources.
Solaris 11.4 is out (just to note). And the latest OpenLDAP release is 2.4.47 (just to note).
tool-threads 8
A tool-threads setting > 2 is ignored with back-mdb.
maxreaders 4096
This is an extremely large value (default is 126). Are you sure you need this bumped so high?
searchstack 64
Similarly here.
So far we experienced this behavior with clients of Oracle Enterprise Linux 6.x, Redhat Enterprise Linux 6.x and AIX. Service requests are opened at vendors support, but I'd prefer to have an installation which is less vulnerable and more resilient to issues of this kind.
I'd estimate that your ability to get a useful response from either is near zero. If you have an enterprise operation that needs support for OpenLDAP you may want to look at Symas' enterprise OpenLDAP builds (Symas Gold). We provide builds and support for Solaris 11.
To avoid performance issues loglevel is now "none stats sync" but can be changed for some time to track down the cause.
You could just do "stats sync", not really a reason to leave "none" in the list.
Generally if you hit an issue such as this, you would want to provde a full backtrace of the slapd process from a utility such as GDB.
Warm regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Juergen.Sprenger@swisscom.com wrote:
Hi,
Anyway, I consider this issue a denial of service vulnerability and OpenLDAP 2.4.45 will not be used for production here.
The only major change in configuration was switching from hdb to mdb, so I will have to check whether mdb may be the cause of this issue.
You never answered Quanah's questions about your config. Are you actually running on a server with 4096 CPUs? If not, why is your maxreaders set to 4096?
Regards Juergen
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@symas.com] Sent: Mittwoch, 30. Januar 2019 15:53 To: Sprenger Jürgen, INI-ONE-CIS-SDI-HES Juergen.Sprenger@swisscom.com; openldap-technical@openldap.org Subject: Re: OpenLDAP 2.4.45 possible denial of service vulnerability?
--On Wednesday, January 30, 2019 9:43 AM +0000 Juergen.Sprenger@swisscom.com wrote:
Hi,
After upgrading to the latest Release (Solaris 11.3 SRU35, OpenLDAP 2.4.45) we are experiencing massive workloads caused by single clients consuming all available threads and CPU-resources.
Solaris 11.4 is out (just to note). And the latest OpenLDAP release is 2.4.47 (just to note).
tool-threads 8
A tool-threads setting > 2 is ignored with back-mdb.
maxreaders 4096
This is an extremely large value (default is 126). Are you sure you need this bumped so high?
searchstack 64
Similarly here.
So far we experienced this behavior with clients of Oracle Enterprise Linux 6.x, Redhat Enterprise Linux 6.x and AIX. Service requests are opened at vendors support, but I'd prefer to have an installation which is less vulnerable and more resilient to issues of this kind.
I'd estimate that your ability to get a useful response from either is near zero. If you have an enterprise operation that needs support for OpenLDAP you may want to look at Symas' enterprise OpenLDAP builds (Symas Gold). We provide builds and support for Solaris 11.
To avoid performance issues loglevel is now "none stats sync" but can be changed for some time to track down the cause.
You could just do "stats sync", not really a reason to leave "none" in the list.
Generally if you hit an issue such as this, you would want to provde a full backtrace of the slapd process from a utility such as GDB.
Warm regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
We started with the default of 126 and increased the value to see whether it has an impact on the issue.
The issue occurs also with maxreaders=126. Same with searchstack and toolthreads with default values.
-----Original Message----- From: Howard Chu [mailto:hyc@symas.com] Sent: Mittwoch, 6. Februar 2019 14:32 To: Sprenger Jürgen, INI-ONE-CIS-SDI-HES Juergen.Sprenger@swisscom.com; quanah@symas.com; openldap-technical@openldap.org Subject: Re: OpenLDAP 2.4.45 possible denial of service vulnerability?
Juergen.Sprenger@swisscom.com wrote:
Hi,
Anyway, I consider this issue a denial of service vulnerability and OpenLDAP 2.4.45 will not be used for production here.
The only major change in configuration was switching from hdb to mdb, so I will have to check whether mdb may be the cause of this issue.
You never answered Quanah's questions about your config. Are you actually running on a server with 4096 CPUs? If not, why is your maxreaders set to 4096?
Regards Juergen
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@symas.com] Sent: Mittwoch, 30. Januar 2019 15:53 To: Sprenger Jürgen, INI-ONE-CIS-SDI-HES Juergen.Sprenger@swisscom.com; openldap-technical@openldap.org Subject: Re: OpenLDAP 2.4.45 possible denial of service vulnerability?
--On Wednesday, January 30, 2019 9:43 AM +0000 Juergen.Sprenger@swisscom.com wrote:
Hi,
After upgrading to the latest Release (Solaris 11.3 SRU35, OpenLDAP 2.4.45) we are experiencing massive workloads caused by single clients consuming all available threads and CPU-resources.
Solaris 11.4 is out (just to note). And the latest OpenLDAP release is 2.4.47 (just to note).
tool-threads 8
A tool-threads setting > 2 is ignored with back-mdb.
maxreaders 4096
This is an extremely large value (default is 126). Are you sure you need this bumped so high?
searchstack 64
Similarly here.
So far we experienced this behavior with clients of Oracle Enterprise Linux 6.x, Redhat Enterprise Linux 6.x and AIX. Service requests are opened at vendors support, but I'd prefer to have an installation which is less vulnerable and more resilient to issues of this kind.
I'd estimate that your ability to get a useful response from either is near zero. If you have an enterprise operation that needs support for OpenLDAP you may want to look at Symas' enterprise OpenLDAP builds (Symas Gold). We provide builds and support for Solaris 11.
To avoid performance issues loglevel is now "none stats sync" but can be changed for some time to track down the cause.
You could just do "stats sync", not really a reason to leave "none" in the list.
Generally if you hit an issue such as this, you would want to provde a full backtrace of the slapd process from a utility such as GDB.
Warm regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
On Wed, Jan 30, 2019 at 06:53:02 -0800, Quanah Gibson-Mount wrote:
A tool-threads setting > 2 is ignored with back-mdb.
Interesting, it seems this is not docmented?
We have it set to the number of CPU's on the system.
Geert
--On Wednesday, February 06, 2019 2:42 PM +0100 Geert Hendrickx geert@hendrickx.be wrote:
On Wed, Jan 30, 2019 at 06:53:02 -0800, Quanah Gibson-Mount wrote:
A tool-threads setting > 2 is ignored with back-mdb.
Interesting, it seems this is not docmented?
I documented it for Zimbra at https://wiki.zimbra.com/wiki/OpenLDAP_Tuning_Keys_8.0
It probably should be noted in slapd.conf(5)/slapd-config(5) man pages. With back-mdb, any setting above 2 is simply converted to 2. This was due to the fact that extensive testing I did several years back found that the greater the number of tools threads (past 2), the slower the import.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
openldap-technical@openldap.org
-
Geert Hendrickx -
Howard Chu -
Juergen.Sprenger＠swisscom.com -
Quanah Gibson-Mount