Just some initial thoughts on what a new logging daemon should do for us:
The primary goal - we want to use a binary message format with as few format conversions as possible between log sender and log processor.
I'm thinking that we use message catalogs; we will need a tool to preprocess every logging invocation in the source tree and replace them with a integer messageID. So at runtime only the messageID and the message parameters need to be sent, not any plaintext.
The message catalog will be compiled into the binary. When it performs its "openlog" to talk to the logging server, it will send the UUID of its catalog. If the logging server doesn't know this UUID, it will transmit the message catalog to the logging server, before doing anything else. (It may make more sense just to use a SHA hash here instead of a UUID.)
This way the logging server will work with any version of the binaries, and we don't need to do special coordination to update message catalogs between revisions. The logging server will just know that a specific catalog is to be used with a particular logging session.
The message protocol will be length-prefixed. We may even just use DER, since that would allow us to encode arrays of parameters, and other such stuff.
The logging server will write the log messages to disk/network verbatim, doing no parsing at all. It may prefix the records with a log session ID, so that a postprocessor can lookup the catalog that belongs to the session, for dumping out as text.
The logging server can store its received catalogs in an LMDB database. The postprocessor can then lookup individual messageIDs in this database, interpolate the parameters, and dump out in text.
... that's what I have so far. It's a bit worrisome because of the additional moving parts: message catalog creator, log server, log postprocessor. There's definitely more complexity here, but most of it is moved out of the runtime hot path, which is the main goal. Suggestions?