Hello,
Here is our setup: 2 master in mirrormode with accesslog replication and several slaves with lastbind and chain overlay. Everything works, but the accesslog db is quite large and filled with authtimestamp updates from client hosts running sssd. We also have a disk performance issue when the cleanup process is launched every 4h.
We plan to reduce the accesslog history by setting 12h retention and increase cleanup . It should reduce the number of entries and therefore lighten the cleanup process but:
I wonder if increasing the frequency will also have a positive effects (as the though here is that there will be less entries to delete). I think that allocated pages in the mdb database won't change. Is there a safe way to reduce the db size ?
Finally, we considered not logging the sssd auth timestamp updates, but lastbind doesn't have any option to filter which branch is allowed to update, the only option is to set the precision to a higher value. Is there a way to achieve this ?
Any suggestion will be warmly appreciated.
Thank you Jerome
--On Friday, May 24, 2024 3:24 PM +0000 BECOT Jérôme jbecot@itsgroup.com wrote:
I wonder if increasing the frequency will also have a positive effects (as the though here is that there will be less entries to delete). I think that allocated pages in the mdb database won't change. Is there a safe way to reduce the db size ?
Increasing the frequency has helped for me in the past.
Finally, we considered not logging the sssd auth timestamp updates, but lastbind doesn't have any option to filter which branch is allowed to update, the only option is to set the precision to a higher value. Is there a way to achieve this ?
Increasing the precision is the only option at this time.
As a side note, in OpenLDAP 2.6, this functionality is part of the core slapd process and uses pwdLastSucces instead of authTimestamp to comply with the current ppolicy drafts.
--Quanah
--On Friday, May 24, 2024 12:30 PM -0700 Quanah Gibson-Mount quanah@fast-mail.org wrote:
--On Friday, May 24, 2024 3:24 PM +0000 BECOT Jérôme jbecot@itsgroup.com wrote:
I wonder if increasing the frequency will also have a positive effects (as the though here is that there will be less entries to delete). I think that allocated pages in the mdb database won't change. Is there a safe way to reduce the db size ?
Increasing the frequency has helped for me in the past.
On the first question - you can stop slapd, export the accesslog to ldif, delete the accesslog mdb files, and then re-import the accesslog. This will reset the database size. I would do this after you've made your other adjustments so that the actual LDIF export is in line with what the updated expectations are.
I would also note, that in OpenLDAP 2.6+, "standard" syncrepl is the safer replication mechanism for multi-provider environments. While in the past, I always went with delta-syncrepl, for my last roles, I've used OpenLDAP 2.6 + standard syncrepl.
--Quanah
On 5/24/2024 12:06 PM, Quanah Gibson-Mount wrote:
I would also note, that in OpenLDAP 2.6+, "standard" syncrepl is the safer replication mechanism for multi-provider environments. While in the past, I always went with delta-syncrepl, for my last roles, I've used OpenLDAP 2.6 + standard syncrepl.
I find standard syncrepl is often best, since it's so much simpler to set up. And many sites only have single digit megabytes or less amount of write traffic per day. Of course not all (like a telco I know well), but if the amount of write traffic is low, then delta-syncrepl is an unnecessary complication. In a recent client I worked for who has servers in three continents, we were able to maintain total consistency within 1 second, for any given change.
--On Friday, May 24, 2024 1:14 PM -0700 Christopher Paul chris.paul@rexconsulting.net wrote:
On 5/24/2024 12:06 PM, Quanah Gibson-Mount wrote:
I would also note, that in OpenLDAP 2.6+, "standard" syncrepl is the safer replication mechanism for multi-provider environments. While in the past, I always went with delta-syncrepl, for my last roles, I've used OpenLDAP 2.6 + standard syncrepl.
I find standard syncrepl is often best, since it's so much simpler to set up. And many sites only have single digit megabytes or less amount of write traffic per day. Of course not all (like a telco I know well), but if the amount of write traffic is low, then delta-syncrepl is an unnecessary complication. In a recent client I worked for who has servers in three continents, we were able to maintain total consistency within 1 second, for any given change.
In OpenLDAP 2.3 and OpenLDAP 2.4, it was not safe to use syncrepl. A lot of work went into improving both delta-sync and syncrepl in OpenLDAP 2.5, but some innate issues with delta-syncrepl leave standard syncrepl the best option at this time. A lot of work was also done on the syncrepl side to reduce its traffic overhead compared to prior releases. So what is best to use varies significantly depending on the release series. At this point, I generally recommend 2.6 for deployments, and using syncrepl for replication.
--Quanah
--On Friday, May 24, 2024 1:21 PM -0700 Quanah Gibson-Mount quanah@fast-mail.org wrote:
In OpenLDAP 2.3 and OpenLDAP 2.4, it was not safe to use syncrepl. A lot of work went into improving both delta-sync and syncrepl in OpenLDAP 2.5, but some innate issues with delta-syncrepl leave standard syncrepl the best option at this time. A lot of work was also done on the syncrepl side to reduce its traffic overhead compared to prior releases. So what is best to use varies significantly depending on the release series. At this point, I generally recommend 2.6 for deployments, and using syncrepl for replication.
For completeness: One general issue with syncrepl (whether using delta-sync or syncrepl, since it affects fallback) is the sessionlog. In 2.6, you can have it use the accesslog db for the sessionlog instead of an in-memory sessionlog. This can help in cert REFRESH situations, but necessitates maintaing the accesslog db even if it's not being used for delta-syncrepl.
--Quanah
Quanah Gibson-Mount wrote:
--On Friday, May 24, 2024 12:30 PM -0700 Quanah Gibson-Mount quanah@fast-mail.org wrote:
--On Friday, May 24, 2024 3:24 PM +0000 BECOT Jérôme jbecot@itsgroup.com wrote:
I wonder if increasing the frequency will also have a positive effects (as the though here is that there will be less entries to delete). I think that allocated pages in the mdb database won't change. Is there a safe way to reduce the db size ?
Increasing the frequency has helped for me in the past.
On the first question - you can stop slapd, export the accesslog to ldif, delete the accesslog mdb files, and then re-import the accesslog. This will reset the database size. I would do this after you've made your other adjustments so that the actual LDIF export is in line with what the updated expectations are.
There's no point to doing this, since the accesslog DB will just reuse the old pages anyway, and since it is always being written to, every page will be reused sooner or later.
--On Sunday, May 26, 2024 10:12 PM +0100 Howard Chu hyc@symas.com wrote:
On the first question - you can stop slapd, export the accesslog to ldif, delete the accesslog mdb files, and then re-import the accesslog. This will reset the database size. I would do this after you've made your other adjustments so that the actual LDIF export is in line with what the updated expectations are.
There's no point to doing this, since the accesslog DB will just reuse the old pages anyway, and since it is always being written to, every page will be reused sooner or later.
If increasing the frequency of purges was the only change that had been discussed in this thread, then I would agree. But it wasn't, that's why I noted explicitly above "after you've made the other adjustments". One of those adjustments was decreasing how far back the accesslog records changes. If you have an accesslog that was storing 7 days worth of data, and you change it to only store 3 days worth to data, your database is generally going to end up at a much smaller total database size than before. But you won't be able to reclaim the disk space being consumed w/o reloading the accesslog db. And it will generally have a much smaller pool of free pages it needs to recycle through.
--Quanah
openldap-technical@openldap.org
-
BECOT Jérôme -
Christopher Paul -
Howard Chu -
Quanah Gibson-Mount