Dear all,
we're currently testing performance of OpenLDAP on Oracle/RedHat Linux and quite unexpected actually hit systemd-journald to be a bottleneck. While OpenLDAP happily makes use of all available CPUs, that one is single threaded, braking everything. The only way around I have found is to set olcLoglevel to 0, speeding up my test run by a factor of 6(!). That now of course is not an option to use in production. I'd happily directly write to a file as I did in the old days but I cannot get olcLogfile to work. And even if I was able to get there, how do I stop OpenLDAP from logging to syslogd (which is inevitably forwarding everything to system-journald ....) ? Can anyone give advice how to handle this ? Any hint appreciated (short of "get a decent OS" - that is not an option).
Thanks, Markus
--On Wednesday, March 18, 2020 6:16 PM +0000 Markus.Storm@t-systems.com wrote:
Dear all,
we're currently testing performance of OpenLDAP on Oracle/RedHat Linux and quite unexpected actually hit systemd-journald to be a bottleneck. While OpenLDAP happily makes use of all available CPUs, that one is single threaded, braking everything. The only way around I have found is to set olcLoglevel to 0, speeding up my test run by a factor of 6(!). That now of course is not an option to use in production. I'd happily directly write to a file as I did in the old days but I cannot get olcLogfile to work. And even if I was able to get there, how do I stop OpenLDAP from logging to syslogd (which is inevitably forwarding everything to system-journald ….) ? Can anyone give advice how to handle this ? Any hint appreciated (short of "get a decent OS" – that is not an option).
Hi Markus,
We always advise our customers to disable the abominable syslog<->journald integration that RedHat has pushed onto their customers with complete disregard to the consequences of this short-sighted action. Not only does it destroy performance, it can also cause the system to completely deadlock.
If you need instructions on how to do this, let me know.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Hi all,
removing the rsyslogd <-> systemd-journald integration in RedHat as Quanah suggested did the trick. OpenLDAP now makes use of all available CPU cores. Syslogd takes the rest for standard level request logging to files, consuming ~180% of (1)CPU on a 24 core machine. Systemd-journald ain't noticeable any longer.
Thanks Quanah for your excellent support!
Regards Markus
-----Original Message----- From: Quanah Gibson-Mount quanah@symas.com Sent: Wednesday, March 18, 2020 6:53 PM To: Storm, Markus Markus.Storm@t-systems.com; openldap- technical@openldap.org Subject: Re: logging through systemd-journald is bottleneck
--On Wednesday, March 18, 2020 6:16 PM +0000 Markus.Storm@t- systems.com wrote:
Dear all,
we're currently testing performance of OpenLDAP on Oracle/RedHat Linux and quite unexpected actually hit systemd-journald to be a bottleneck. While OpenLDAP happily makes use of all available CPUs, that one is single threaded, braking everything. The only way around I have found is to set olcLoglevel to 0, speeding up my test run by a factor of 6(!). That now of course is not an option to use in production. I'd happily directly write to a file as I did in the old days but I cannot get olcLogfile to work. And even if I was able to get there, how do I stop OpenLDAP from logging to syslogd (which is inevitably forwarding everything to system-journald ….) ? Can anyone give advice how to handle this ? Any hint appreciated (short of "get a decent OS" – that is not an option).
Hi Markus,
We always advise our customers to disable the abominable syslog<->journald integration that RedHat has pushed onto their customers with complete disregard to the consequences of this short-sighted action. Not only does it destroy performance, it can also cause the system to completely deadlock.
If you need instructions on how to do this, let me know.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Am Wed, 18 Mar 2020 17:16:53 +0000 schrieb Markus.Storm@t-systems.com:
Dear all,
we're currently testing performance of OpenLDAP on Oracle/RedHat Linux and quite unexpected actually hit systemd-journald to be a bottleneck. While OpenLDAP happily makes use of all available CPUs, that one is single threaded, braking everything. The only way around I have found is to set olcLoglevel to 0, speeding up my test run by a factor of 6(!). That now of course is not an option to use in production. I'd happily directly write to a file as I did in the old days but I cannot get olcLogfile to work. And even if I was able to get there, how do I stop OpenLDAP from logging to syslogd (which is inevitably forwarding everything to system-journald ....) ? Can anyone give advice how to handle this ? Any hint appreciated (short of "get a decent OS" - that is not an option).
I support Qanah's advice! Beside this, consider a logging strategy based on required information and neglected information, as well as min. and max. server load.
Based on my experience I would disable logging as default, but enable logging for a short given time, just a modify operation on atribute olcLogLevel. With regard to journald I advice to define filters, see man journalctl(1). If syslog is a requirement, change to rsyslog. Don't make use of logstash!
-Dieter
openldap-technical@openldap.org