Dieter Klünter dieter@dkluenter.de schrieb am 18.03.2020 um 19:57 in
Nachricht 30206_1584557842_5E726F12_30206_1570_1_20200318195706.1d992aa0@pink.fritz.box:
Am Wed, 18 Mar 2020 17:16:53 +0000 schrieb Markus.Storm@t-systems.com:
Dear all,
we're currently testing performance of OpenLDAP on Oracle/RedHat Linux and quite unexpected actually hit systemd-journald to be a bottleneck. While OpenLDAP happily makes use of all available CPUs, that one is single threaded, braking everything. The only way around I have found is to set olcLoglevel to 0, speeding up my test run by a factor of 6(!). That now of course is not an option to use in production. I'd happily directly write to a file as I did in the old days but I cannot get olcLogfile to work. And even if I was able to get there, how do I stop OpenLDAP from logging to syslogd (which is inevitably forwarding everything to system-journald ....) ? Can anyone give advice how to handle this ? Any hint appreciated (short of "get a decent OS" - that is not an option).
I support Qanah's advice! Beside this, consider a logging strategy based on required information and neglected information, as well as min. and max. server load.
Based on my experience I would disable logging as default, but enable logging for a short given time, just a modify operation on atribute olcLogLevel. With regard to journald I advice to define filters, see man journalctl(1). If syslog is a requirement, change to rsyslog. Don't make use of logstash!
Can't openLDAP simply log to a different port than the default syslog port? If so, just set up some alternate local syslog server. Or, in case an external syslog server is supported, just use one that isn't using systemd-journald.
Sorry, I neved had the need to use a different syslog mechanism...
-Dieter
-- Dieter Klünter | Systemberatung http://sys4.de GPG Key ID: E9ED159B 53°37'09,95"N 10°08'02,42"E
Hi!
Sorry about slightly offtopic question, but I'm curious: how much log do you guys typically generate when it starts to act as a bottleneck?
I have systemd-journald that receives the log and propagates it via imjournal to rsyslog which dumps it on disk and forwards it to one central logging server using the udp syslog protocol and to another using the tcp-plain syslog protocol and I've experienced no logging related performance issues (although I had to tune the rate limit settings of both systemd-journald and rsyslogd's imjournal module so that they wouldn't start discarding log; I actually had some trouble with that because the ratelimitburst option in rsyslog is an unsigned short but that isn't documented anywhare and rsyslog doesn't complain if you set it higher than fits into the data type). Typical loads here are something like 500 slapd log lines per second, peaks something like 5000 slapd log lines per second at slapd's default logging settings. Block devices are local (we also have some vmware nodes with vmware virtual devices, the vmware system's data storage is on a SAN) The OS is RHEL7. The highest %CPU I see on the two logging processes is around 20, typically in the low single digits.
Best,
Janne Peltonen IT Dept University of Helsinki
--On Thursday, March 19, 2020 6:46 PM +0200 Janne Peltonen janne.peltonen@helsinki.fi wrote:
Hi!
Sorry about slightly offtopic question, but I'm curious: how much log do you guys typically generate when it starts to act as a bottleneck?
In the environment where it deadlocked the server, there were some 2 million active users, the site was MMR, and the log level was stats+sync. There was also a rogue mobile device that was sending thousands of failing authentications per second on top of the normal load.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Am Thu, 19 Mar 2020 08:57:11 +0100 schrieb "Ulrich Windl" Ulrich.Windl@rz.uni-regensburg.de:
Dieter Klünter dieter@dkluenter.de schrieb am 18.03.2020 um 19:57 in
Nachricht 30206_1584557842_5E726F12_30206_1570_1_20200318195706.1d992aa0@pink.fritz.box:
Am Wed, 18 Mar 2020 17:16:53 +0000 schrieb Markus.Storm@t-systems.com:
Dear all,
we're currently testing performance of OpenLDAP on Oracle/RedHat Linux and quite unexpected actually hit systemd-journald to be a bottleneck. While OpenLDAP happily makes use of all available CPUs, that one is single threaded, braking everything. The only way around I have found is to set olcLoglevel to 0, speeding up my test run by a factor of 6(!). That now of course is not an option to use in production. I'd happily directly write to a file as I did in the old days but I cannot get olcLogfile to work. And even if I was able to get there, how do I stop OpenLDAP from logging to syslogd (which is inevitably forwarding everything to system-journald ....) ? Can anyone give advice how to handle this ? Any hint appreciated (short of "get a decent OS" - that is not an option).
I support Qanah's advice! Beside this, consider a logging strategy based on required information and neglected information, as well as min. and max. server load.
Based on my experience I would disable logging as default, but enable logging for a short given time, just a modify operation on atribute olcLogLevel. With regard to journald I advice to define filters, see man journalctl(1). If syslog is a requirement, change to rsyslog. Don't make use of logstash!
Can't openLDAP simply log to a different port than the default syslog port? If so, just set up some alternate local syslog server. Or, in case an external syslog server is supported, just use one that isn't using systemd-journald.
Sorry, I neved had the need to use a different syslog mechanism...
mann slapd(8), syslog-local-user, slapd logs, as default, to local4
-Dieter
openldap-technical@openldap.org
-
Dieter Klünter -
Janne Peltonen -
Quanah Gibson-Mount -
Ulrich Windl