Pallissard, Matthew wrote:
On 2020-03-06T17:02:14, Howard Chu wrote:
Howard Chu wrote:
Just some initial thoughts on what a new logging daemon should do for us:
The primary goal - we want to use a binary message format with as few format conversions as possible between log sender and log processor.
One other concern - what do we do about debug output to stderr? If we keep that, then we still have message formatting overhead inside slapd itself, and we're potentially managing two redundant sets of message formats.
I'm not sure if you're toying with the idea of removing output to stdout/stderr but I figure I'll share my 2cents anyway.
We make *heavy* use of openldap in containers and having the output go to stdout and stderr is *super* helpful when debugging issues in vanilla docker or kubernetes.
Not only does stdout allow you to use native tools such as `journalctl` or `kubectl` out of the box, log aggregation is a completely solved problem in this workflow and is trivial to implement.
That being said, persisting logs to disk in a binary format has its merits. Personally, I don't think plain text logs scale all that well.
Plaintext logs scale horribly. When you have servers processing hundreds of thousands to millions of queries/sec, you have to rotate log files pretty frequently otherwise your disks get filled. The majority of the text associated with each message is constant and redundant as well. (Which also means the logfiles are highly compressible, but that still requires CPU to accomplish.)
We may be able to tolerate keeping the stderr messaging intact. When it's turned on it probably won't be worse than a 10% performance hit. https://www.openldap.org/lists/openldap-devel/201511/msg00000.html And writing to stderr is still several times faster than syslog() at the moment.