openldap-commit2devel@OpenLDAP.org wrote:
A ref change was pushed to the OpenLDAP (openldap.git) repository. It will be available in the public mirror shortly.
The branch, master has been updated via b7a291a488dff3902931559670cf94c7abe2655b (commit) from 1ab08d2f8e4c9baf71e3c146e95d013d7b8bd656 (commit)
Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below.
- Log -----------------------------------------------------------------
commit b7a291a488dff3902931559670cf94c7abe2655b Author: Howard Chu hyc@openldap.org Date: Wed Jul 8 14:22:29 2015 +0100
Experimental syslog() replacement 2-3x faster than libc. Add it to the Makefile yourself if you want to test it.
Summary of changes: servers/slapd/overlays/syncprov.c | 3 +-
Oops. Ignore the above, reverted already.
servers/slapd/syslog.c | 329 +++++++++++++++++++++++++++++++++++++
The most obvious braindeadedness in glibc syslog() is that it acquires a mutex before writing to the log socket. This is utterly inane; the log socket is datagram-based and as such all writes to it are already atomic and need no mutex protection.
The other big problem is that the function that generates the log timestamp also acquires 2 mutexes. I avoid all that crap too by writing my own implementation of localtime(). This overhead can be reduced further still by simply omitting the timestamp - it turns out that modern syslogds (rsyslog, syslog-ng) actually ignore the timestamp in the message and generate their own anyway. (Note that this localtime implementation is broken wrt DST.)
There are a few other areas to explore here, including a log socket per thread, which would improve throughput even more in heavy workloads. These things can be explored without major overhauls of existing code.
For greater throughput we need a logging mechanism that doesn't rely on sprintf. I.e., we should consider a binary log format where we simply pass around short message IDs (and relevant message parameters) and use a compiled message catalog that maps IDs to text. This would require a post-processor to read the binary log and generate human-readable messages somewhere else. (It's obviously not a new concept, IBM mainframes have done this for decades; the Microsoft system logger does this as well.)
* Howard Chu:
For greater throughput we need a logging mechanism that doesn't rely on sprintf. I.e., we should consider a binary log format where we simply pass around short message IDs (and relevant message parameters) and use a compiled message catalog that maps IDs to text. This would require a post-processor to read the binary log and generate human-readable messages somewhere else. (It's obviously not a new concept, IBM mainframes have done this for decades; the Microsoft system logger does this as well.)
Current systems have journald, which does something like that.
Florian Weimer wrote:
- Howard Chu:
For greater throughput we need a logging mechanism that doesn't rely on sprintf. I.e., we should consider a binary log format where we simply pass around short message IDs (and relevant message parameters) and use a compiled message catalog that maps IDs to text. This would require a post-processor to read the binary log and generate human-readable messages somewhere else. (It's obviously not a new concept, IBM mainframes have done this for decades; the Microsoft system logger does this as well.)
Current systems have journald, which does something like that.
I had thought of journald, but looking at its API, it seems to still primarily work with fully-formatted messages being sent from the logging clients.
http://www.freedesktop.org/software/systemd/man/sd_journal_print.html
We want to be able to dump unformatted data items, and incur all the formatting cost in the post-processor. Otherwise, we haven't gained anything in terms of performance.
-
Florian Weimer -
Howard Chu