New logging system ideas
by Howard Chu
Just some initial thoughts on what a new logging daemon should do for us:
The primary goal - we want to use a binary message format with as few format conversions as possible between log
sender and log processor.
I'm thinking that we use message catalogs; we will need a tool to preprocess every logging
invocation in the source tree and replace them with a integer messageID. So at runtime only
the messageID and the message parameters need to be sent, not any plaintext.
The message catalog will be compiled into the binary. When it performs its "openlog" to talk
to the logging server, it will send the UUID of its catalog. If the logging server doesn't
know this UUID, it will transmit the message catalog to the logging server, before doing
anything else. (It may make more sense just to use a SHA hash here instead of a UUID.)
This way the logging server will work with any version of the binaries, and we don't need
to do special coordination to update message catalogs between revisions. The logging server
will just know that a specific catalog is to be used with a particular logging session.
The message protocol will be length-prefixed. We may even just use DER, since that would
allow us to encode arrays of parameters, and other such stuff.
The logging server will write the log messages to disk/network verbatim, doing no
parsing at all. It may prefix the records with a log session ID, so that a postprocessor
can lookup the catalog that belongs to the session, for dumping out as text.
The logging server can store its received catalogs in an LMDB database. The postprocessor
can then lookup individual messageIDs in this database, interpolate the parameters, and
dump out in text.
... that's what I have so far. It's a bit worrisome because of the additional moving parts:
message catalog creator, log server, log postprocessor. There's definitely more complexity
here, but most of it is moved out of the runtime hot path, which is the main goal. Suggestions?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
3 months
- 5
- 12
Re: OpenLDAP 2.4.51 available, LMDB 0.9.26 available
by Clément OUDOT
Hello,
changelog seems not well updated on the website:
https://www.openldap.org/software/release/changes.html
Clément.
Le mer. 12 août 2020 à 07:02, <project(a)openldap.org> a écrit :
>
> OpenLDAP 2.4.51 is now available for download as detailed on our download page:
>
> https://www.openldap.org/software/download/
>
> and should soon be available on all official mirrors:
>
> ftp://ftp.openldap.org/pub/OpenLDAP/MIRRORS
>
> This is a maintenance release and is made available for general use. Users of OpenLDAP Software are encouraged to upgrade.
>
> Significant contributors are:
>
> Howard Chu (Symas Corp)
> Quanah Gibson-Mount (Symas Corp)
> Ondřej Kuzník (Symas Corp)
> Ryan Tandy
>
> OpenLDAP 2.4.51 Release (2020/08/11)
> Added slapo-ppolicy implement Netscape password policy controls (ITS#9279)
> Fixed libldap retry loop in ldap_int_tls_connect (ITS#8650)
> Fixed libldap to use getaddrinfo in ldap_pvt_get_fqdn (ITS#9287)
> Fixed slapd to enforce singular existence of some overlays (ITS#9309)
> Fixed slapd syncrepl to not delete non-replicated attrs (ITS#9227)
> Fixed slapd syncrepl to correctly delete entries on resync (ITS#9282)
> Fixed slapd syncrepl to use replace on single valued attrs (ITS#9294, ITS#9295)
> Fixed slapd-perl dynamic config with threaded slapd (ITS#7573)
> Fixed slapo-ppolicy to expose the ppolicy control (ITS#9285)
> Fixed slapo-ppolicy race condition for pwdFailureTime (ITS#9302)
> Fixed slapo-ppolicy so it can only exist once per DB (ITS#9309)
> Fixed slapo-chain to check referral (ITS#9262)
> Build Environment
> Fix test064 so it no longer uses bashisms (ITS#9263)
> Contrib
> Fix default prefix value for pw-argon2, pw-pbkdf2 modules (ITS#9248)
> slapo-allowed - Fix usage of unitialized variable (ITS#9308)
> Documentation
> ldap_parse_result(3) - Document ldap_parse_intermediate (ITS#9271)
>
>
> MD5(openldap-2.4.51.tgz)= 0d2025896cf1c17af7304ecc57ec9531
> SHA1(openldap-2.4.51.tgz)= 4fe7f0e5766d0d9a5431871b581938c05b4eb873
>
> LMDB 0.9.26 Release (2020/08/11)
> ITS#9278 fix robust mutex cleanup for FreeBSD
1 year, 2 months
- 2
- 1
Two new repos
by Ondřej Kuzník
This is an announcement that two new repositories have just been added
on Gitlab:
https://git.openldap.org/openldap/syncmonitor
This is a project to support real-time monitoring of OpenLDAP's
replication. It comes as a Python module using the asyncio library to
provide a fully reactive interface. An asyncio wrapper around Michael's
python-ldap0 is included for that reason.
A sample terminal application is also provided to showcase what is
possible.
https://git.openldap.org/ondra/pldap (in personal namespace for now)
The above mentioned asyncio wrapper is a bit of a break with how
traditional python-ldap wrappers are used. This comes out of attempts to
use and extend the existing Python LDAP modules to do what I needed them
to do and discussions with Michael Ströder, maintainer of python-ldap0,
how that might be achieved.
In the end I've drafted a new module that has two goals:
- explore a new API that can stay mostly uniform across plain Python and
async code, where result and intermediate responses are extracted in a
more natural way
- move towards run-time wrapping of libldap so that the module is
immediately usable in environments without the need to compile a
native wrapper
At some point, this code should either be extended to cover the scope
that the other two wrappers (python-ldap and python-ldap0) provide or
the approach should be adopted in either. It is my understanding that
the OpenLDAP project might be interested in adopting either of the
modules at some point (subject to that module adopting the OpenLDAP
Public License).
In view of the above, I've kept it in my personal namespace for now so
as not to give any impression that this the official Python module to
use for all things LDAP.
At some point, these (just like the mod-harness module published last
year) should be the building blocks for the test suite that I hope to
build at some point. As such, there are many smaller tools of general
utility bundled in there, especially in the syncmonitor repository.
Should you find them useful, let's turn them into something more
complete. Please review and come back with suggestions or, if you feel
so inclined, patches.
Regards,
--
Ondřej Kuzník
Senior Software Engineer
Symas Corporation http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP
1 year, 2 months
- 1
- 0
- ← Newer
- 1
- Older →