11:55 a.m.
Prentice Bisbal wrote:
On 05/31/2017 12:37 AM, John Lewis wrote:
> What if I wanted to write a OpenLDAP backend for a systemd journal file
> or Elasticsearch so I can present my logs as an LDAP subtree so I can
> use my LDAP tools to filter my logs? Should I use back-shell for
> prototyping? If so, what is the usual work flow?
This sounds like the wrong tool for this job.
Really? Can you give a point by point comparison against what you think is the
"right" tool for the job?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
12:07 p.m.
New subject: How wold you go about writing a new OpenLDAP backend?
On 05/31/2017 02:55 PM, Howard Chu wrote:
Prentice Bisbal wrote:
> On 05/31/2017 12:37 AM, John Lewis wrote:
>
>> What if I wanted to write a OpenLDAP backend for a systemd journal file
>> or Elasticsearch so I can present my logs as an LDAP subtree so I can
>> use my LDAP tools to filter my logs? Should I use back-shell for
>> prototyping? If so, what is the usual work flow?
> This sounds like the wrong tool for this job.
Really? Can you give a point by point comparison against what you
think is the "right" tool for the job?
That's the wrong question to ask, since I never made any claim as to
what the right tool is. A better question to ask would be why I think
this is the wrong tool. So here's my reply to that: It doesn't seem like
this is a job LDAP was designed for. To me, LDAP is really just a
standardized database interface that make certain type of DB looks up
easy, and most importantly, standardized.I would think a more general
purpose database interface would be better for this.
Unfortunately, I'm not a database expert, so I can't give you the
"point-by-point argument you requested.
Now your turn - why would OpenLDAP be a good fit for this over other
databases?
--
Prentice
1:39 p.m.
New subject: How wold you go about writing a new OpenLDAP backend?
On Wed, 2017-05-31 at 15:07 -0400, Prentice Bisbal wrote:
On 05/31/2017 02:55 PM, Howard Chu wrote:
> Prentice Bisbal wrote:
>> On 05/31/2017 12:37 AM, John Lewis wrote:
>>
>
>> This sounds like the wrong tool for this job.
>
> Really? Can you give a point by point comparison against what you
> think is the "right" tool for the job?
That's the wrong question to ask, since I never made any claim as to
what the right tool is. A better question to ask would be why I think
this is the wrong tool. So here's my reply to that: It doesn't seem like
this is a job LDAP was designed for. To me, LDAP is really just a
standardized database interface that make certain type of DB looks up
easy, and most importantly, standardized.I would think a more general
purpose database interface would be better for this.
Unfortunately, I'm not a database expert, so I can't give you the
"point-by-point argument you requested.
Now your turn - why would OpenLDAP be a good fit for this over other
databases?
--
Prentice
Yes, LDAP is really just a standardized database interface. That is the
point. I want my logs available through an IETF standardized protocol so
I can reuse tools so I don't have to maintain as many tools.
OpenLDAP would be a good fit over other databases because Michael, and
Howard, and the other contributers to OpenLDAP did most of the work for
me.
10:19 a.m.
New subject: How wold you go about writing a new OpenLDAP backend?
On 05/31/2017 04:39 PM, John Lewis wrote:
On Wed, 2017-05-31 at 15:07 -0400, Prentice Bisbal wrote:
> On 05/31/2017 02:55 PM, Howard Chu wrote:
>> Prentice Bisbal wrote:
>>> On 05/31/2017 12:37 AM, John Lewis wrote:
>>>
>>> This sounds like the wrong tool for this job.
>> Really? Can you give a point by point comparison against what you
>> think is the "right" tool for the job?
> That's the wrong question to ask, since I never made any claim as to
> what the right tool is. A better question to ask would be why I think
> this is the wrong tool. So here's my reply to that: It doesn't seem like
> this is a job LDAP was designed for. To me, LDAP is really just a
> standardized database interface that make certain type of DB looks up
> easy, and most importantly, standardized.I would think a more general
> purpose database interface would be better for this.
>
> Unfortunately, I'm not a database expert, so I can't give you the
> "point-by-point argument you requested.
>
> Now your turn - why would OpenLDAP be a good fit for this over other
> databases?
>
> --
> Prentice
>
Yes, LDAP is really just a standardized database interface. That is the
point. I want my logs available through an IETF standardized protocol so
I can reuse tools so I don't have to maintain as many tools.
OpenLDAP would be a good fit over other databases because Michael, and
Howard, and the other contributers to OpenLDAP did most of the work for
me.
Fair enough answer. How would you store and retrieve your results? Are
you creating/using a schema specifically for this? If so, what does it
look like.
I'm still having trouble seeing the performance benefits, though. The
data is still stored in a backend DB, right? Let's say that backend is
some sort of SQL DB. You write your ldapsearch query, and then LDAP
converts that to SQL, and then the DB returns the results to LDAP, which
then translated the results to LDIF and returns them to you. Isn't there
performance hits every time you traverse the LDAP layer?
I suppose this might still be faster than reading a plain-text log file.
Is that the point I'm missing?
Prentice
8:34 a.m.
New subject: How wold you go about writing a new OpenLDAP backend?
On Fri, 2017-06-02 at 13:19 -0400, Prentice Bisbal wrote:
Fair enough answer. How would you store and retrieve your results?
Are
you creating/using a schema specifically for this? If so, what does it
look like.
I'm still having trouble seeing the performance benefits, though. The
data is still stored in a backend DB, right? Let's say that backend is
some sort of SQL DB. You write your ldapsearch query, and then LDAP
converts that to SQL, and then the DB returns the results to LDAP, which
then translated the results to LDIF and returns them to you. Isn't there
performance hits every time you traverse the LDAP layer?
I suppose this might still be faster than reading a plain-text log file.
Is that the point I'm missing?
Prentice
There is no point in using an SQL DB because log data is not relational.
The SQL database would have a lot of overhead for features I wouldn't
use.
I envision that I can use rsyslog to forward all of the logs to a log
server and write them out to a journal using omjournal and then present
the logs over the network via LDAP which can be injected into many
different tools for filtering and displaying the information.
We won't have to worry about getting compatibility for NoSQL
implantation of the month with its own special network protocol, on disk
format, pile of dependencies, and assemble of startup companies from the
San Francisco Bay Area, with their binding arbitration agreements there
to pump and dump the whole stack every 2-5 years. We can use LDAP
instead. We can do all of this crap or we can just use LDAP.
In envision that we will be able to start ignoring all of the hype
because we will all of the relevant features, and maximum
interoperability.
As for implementation, the jounral's file format is documented here
https://www.freedesktop.org/wiki/Software/systemd/journal-files/, and
there is a C API
https://www.freedesktop.org/software/systemd/man/sd-journal.html that is
recommended to be used.
A new LDAP schema for the file format may need to be written, but it
would not surprise me if there is a schema that already exists that
perfectly maps to every attribute that was written 15 years ago and it
is already installed by default on every instance of OpenLDAP.
You could ask why to bother with the journal when the logs could be
written directly to lmdb. The answer is, I don't know if lmdb would be
better for logs yet, but I know this format was designed specially for
logs. When I get the logs into ldap, it would be trivial to copy the
data to a lmdb backend and find out.
2191
days inactive
2196
days old
openldap-technical@openldap.org
4 comments
3 participants
participants (3)
-
Howard Chu -
John Lewis -
Prentice Bisbal