On Fri, 2017-06-02 at 13:19 -0400, Prentice Bisbal wrote:
Fair enough answer. How would you store and retrieve your results? Are you creating/using a schema specifically for this? If so, what does it look like.
I'm still having trouble seeing the performance benefits, though. The data is still stored in a backend DB, right? Let's say that backend is some sort of SQL DB. You write your ldapsearch query, and then LDAP converts that to SQL, and then the DB returns the results to LDAP, which then translated the results to LDIF and returns them to you. Isn't there performance hits every time you traverse the LDAP layer?
I suppose this might still be faster than reading a plain-text log file. Is that the point I'm missing?
Prentice
There is no point in using an SQL DB because log data is not relational. The SQL database would have a lot of overhead for features I wouldn't use.
I envision that I can use rsyslog to forward all of the logs to a log server and write them out to a journal using omjournal and then present the logs over the network via LDAP which can be injected into many different tools for filtering and displaying the information.
We won't have to worry about getting compatibility for NoSQL implantation of the month with its own special network protocol, on disk format, pile of dependencies, and assemble of startup companies from the San Francisco Bay Area, with their binding arbitration agreements there to pump and dump the whole stack every 2-5 years. We can use LDAP instead. We can do all of this crap or we can just use LDAP.
In envision that we will be able to start ignoring all of the hype because we will all of the relevant features, and maximum interoperability.
As for implementation, the jounral's file format is documented here https://www.freedesktop.org/wiki/Software/systemd/journal-files/, and there is a C API https://www.freedesktop.org/software/systemd/man/sd-journal.html that is recommended to be used.
A new LDAP schema for the file format may need to be written, but it would not surprise me if there is a schema that already exists that perfectly maps to every attribute that was written 15 years ago and it is already installed by default on every instance of OpenLDAP.
You could ask why to bother with the journal when the logs could be written directly to lmdb. The answer is, I don't know if lmdb would be better for logs yet, but I know this format was designed specially for logs. When I get the logs into ldap, it would be trivial to copy the data to a lmdb backend and find out.