Prentice Bisbal wrote:
On 05/31/2017 12:37 AM, John Lewis wrote:
What if I wanted to write a OpenLDAP backend for a systemd journal file or Elasticsearch so I can present my logs as an LDAP subtree so I can use my LDAP tools to filter my logs? Should I use back-shell for prototyping? If so, what is the usual work flow?
This sounds like the wrong tool for this job.
Really? Can you give a point by point comparison against what you think is the "right" tool for the job?
On 05/31/2017 02:55 PM, Howard Chu wrote:
Prentice Bisbal wrote:
On 05/31/2017 12:37 AM, John Lewis wrote:
What if I wanted to write a OpenLDAP backend for a systemd journal file or Elasticsearch so I can present my logs as an LDAP subtree so I can use my LDAP tools to filter my logs? Should I use back-shell for prototyping? If so, what is the usual work flow?
This sounds like the wrong tool for this job.
Really? Can you give a point by point comparison against what you think is the "right" tool for the job?
That's the wrong question to ask, since I never made any claim as to what the right tool is. A better question to ask would be why I think this is the wrong tool. So here's my reply to that: It doesn't seem like this is a job LDAP was designed for. To me, LDAP is really just a standardized database interface that make certain type of DB looks up easy, and most importantly, standardized.I would think a more general purpose database interface would be better for this.
Unfortunately, I'm not a database expert, so I can't give you the "point-by-point argument you requested.
Now your turn - why would OpenLDAP be a good fit for this over other databases?
-- Prentice
On Wed, 2017-05-31 at 15:07 -0400, Prentice Bisbal wrote:
On 05/31/2017 02:55 PM, Howard Chu wrote:
Prentice Bisbal wrote:
On 05/31/2017 12:37 AM, John Lewis wrote:
This sounds like the wrong tool for this job.
Really? Can you give a point by point comparison against what you think is the "right" tool for the job?
That's the wrong question to ask, since I never made any claim as to what the right tool is. A better question to ask would be why I think this is the wrong tool. So here's my reply to that: It doesn't seem like this is a job LDAP was designed for. To me, LDAP is really just a standardized database interface that make certain type of DB looks up easy, and most importantly, standardized.I would think a more general purpose database interface would be better for this.
Unfortunately, I'm not a database expert, so I can't give you the "point-by-point argument you requested.
Now your turn - why would OpenLDAP be a good fit for this over other databases?
-- Prentice
Yes, LDAP is really just a standardized database interface. That is the point. I want my logs available through an IETF standardized protocol so I can reuse tools so I don't have to maintain as many tools.
OpenLDAP would be a good fit over other databases because Michael, and Howard, and the other contributers to OpenLDAP did most of the work for me.
On 05/31/2017 04:39 PM, John Lewis wrote:
On Wed, 2017-05-31 at 15:07 -0400, Prentice Bisbal wrote:
On 05/31/2017 02:55 PM, Howard Chu wrote:
Prentice Bisbal wrote:
On 05/31/2017 12:37 AM, John Lewis wrote:
This sounds like the wrong tool for this job.
Really? Can you give a point by point comparison against what you think is the "right" tool for the job?
That's the wrong question to ask, since I never made any claim as to what the right tool is. A better question to ask would be why I think this is the wrong tool. So here's my reply to that: It doesn't seem like this is a job LDAP was designed for. To me, LDAP is really just a standardized database interface that make certain type of DB looks up easy, and most importantly, standardized.I would think a more general purpose database interface would be better for this.
Unfortunately, I'm not a database expert, so I can't give you the "point-by-point argument you requested.
Now your turn - why would OpenLDAP be a good fit for this over other databases?
-- Prentice
Yes, LDAP is really just a standardized database interface. That is the point. I want my logs available through an IETF standardized protocol so I can reuse tools so I don't have to maintain as many tools.
OpenLDAP would be a good fit over other databases because Michael, and Howard, and the other contributers to OpenLDAP did most of the work for me.
Fair enough answer. How would you store and retrieve your results? Are you creating/using a schema specifically for this? If so, what does it look like.
I'm still having trouble seeing the performance benefits, though. The data is still stored in a backend DB, right? Let's say that backend is some sort of SQL DB. You write your ldapsearch query, and then LDAP converts that to SQL, and then the DB returns the results to LDAP, which then translated the results to LDIF and returns them to you. Isn't there performance hits every time you traverse the LDAP layer?
I suppose this might still be faster than reading a plain-text log file. Is that the point I'm missing?
Prentice
On Fri, 2017-06-02 at 13:19 -0400, Prentice Bisbal wrote:
Fair enough answer. How would you store and retrieve your results? Are you creating/using a schema specifically for this? If so, what does it look like.
I'm still having trouble seeing the performance benefits, though. The data is still stored in a backend DB, right? Let's say that backend is some sort of SQL DB. You write your ldapsearch query, and then LDAP converts that to SQL, and then the DB returns the results to LDAP, which then translated the results to LDIF and returns them to you. Isn't there performance hits every time you traverse the LDAP layer?
I suppose this might still be faster than reading a plain-text log file. Is that the point I'm missing?
Prentice
There is no point in using an SQL DB because log data is not relational. The SQL database would have a lot of overhead for features I wouldn't use.
I envision that I can use rsyslog to forward all of the logs to a log server and write them out to a journal using omjournal and then present the logs over the network via LDAP which can be injected into many different tools for filtering and displaying the information.
We won't have to worry about getting compatibility for NoSQL implantation of the month with its own special network protocol, on disk format, pile of dependencies, and assemble of startup companies from the San Francisco Bay Area, with their binding arbitration agreements there to pump and dump the whole stack every 2-5 years. We can use LDAP instead. We can do all of this crap or we can just use LDAP.
In envision that we will be able to start ignoring all of the hype because we will all of the relevant features, and maximum interoperability.
As for implementation, the jounral's file format is documented here https://www.freedesktop.org/wiki/Software/systemd/journal-files/, and there is a C API https://www.freedesktop.org/software/systemd/man/sd-journal.html that is recommended to be used.
A new LDAP schema for the file format may need to be written, but it would not surprise me if there is a schema that already exists that perfectly maps to every attribute that was written 15 years ago and it is already installed by default on every instance of OpenLDAP.
You could ask why to bother with the journal when the logs could be written directly to lmdb. The answer is, I don't know if lmdb would be better for logs yet, but I know this format was designed specially for logs. When I get the logs into ldap, it would be trivial to copy the data to a lmdb backend and find out.
openldap-technical@openldap.org
-
Howard Chu -
John Lewis -
Prentice Bisbal