Re: back-llog
by Howard Chu
Michael Ströder wrote:
> Howard Chu wrote:
>> Michael Ströder wrote:
>>> Howard Chu wrote:
>>>> Was just thinking we could do a quite simple backend for use with the accesslog
>>>> overlay and delta-syncrepl. It would write into flat files and do typical
>>>> logfile rotation on its own. The backing store would have a minimum of two files
>>>> - one for the suffix entry, one for the current chunk of logs. There would be a
>>>> configurable number of logfiles, with the oldest simply being deleted when it's
>>>> time to purge.
>>>
>>> Two things I'd consider helpful for long-time archiving/auditing:
>>>
>>> 1. filenames to allow of "merging" different log flat files into one big file
>>> store on a different slapd instance.
>>
>> Not sure what you mean by this. As sequential logs you could just cat them if
>> you want to combine files into one, what does the filename have to do with it?
>
> Sorry for unclear wording.
> I'd like to move archived files of all MMR replicas into another single
> filesystem directory and let a special auditing slapd with back-llog search in
> them without having to muck with file names.
Then you would have to merge them entry-by-entry to preserve chronological order.
>> If we just use datestamps for filenames YYYYMMDD it should be no problem for you
>> to cat them in proper order.
>
> There could be conflicting timestamps from different MMR replicas. Maybe
> appending the serverID could be a solution?
Just keep them all in their own separate filesystem directories.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
4 years, 3 months
Re: back-llog
by Howard Chu
Michael Ströder wrote:
> Howard Chu wrote:
>> Was just thinking we could do a quite simple backend for use with the accesslog
>> overlay and delta-syncrepl. It would write into flat files and do typical
>> logfile rotation on its own. The backing store would have a minimum of two files
>> - one for the suffix entry, one for the current chunk of logs. There would be a
>> configurable number of logfiles, with the oldest simply being deleted when it's
>> time to purge.
>
> Two things I'd consider helpful for long-time archiving/auditing:
>
> 1. filenames to allow of "merging" different log flat files into one big file
> store on a different slapd instance.
Not sure what you mean by this. As sequential logs you could just cat them if
you want to combine files into one, what does the filename have to do with it?
If we just use datestamps for filenames YYYYMMDD it should be no problem for
you to cat them in proper order.
> 2. instead of deleting moving files to an archive directory.
Sounds fine as an option.
> I know it's a different use-case but quite handy.
>
> Ciao, Michael.
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
4 years, 3 months
back-llog
by Howard Chu
Was just thinking we could do a quite simple backend for use with the
accesslog overlay and delta-syncrepl. It would write into flat files and do
typical logfile rotation on its own. The backing store would have a minimum of
two files - one for the suffix entry, one for the current chunk of logs. There
would be a configurable number of logfiles, with the oldest simply being
deleted when it's time to purge. It would only support Add/Search in general,
and Modify on the suffix entry.
A rolling append-only format like this would allow much higher throughput for
accesslog recording. Using a binary entry format would of course also help.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
4 years, 3 months
