Thanks for the reply, Howard.
Thanks for pointing me in the right direction. From what I have read there are two options.
1) Copy /usr/share/openldap-servers/DB_CONFIG.example to /var/lib/domain then rebuild the database. 2) Enable checkpointing in slapd.conf
Does enabling checkpointing in slapd.conf require rebuilding the database or can I simply restart slapd.conf? We are not using online configuration.
Best Doug
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 F: 212-746-8690
On Fri, Aug 25, 2017 at 8:55 AM, Howard Chu hyc@symas.com wrote:
Douglas Duckworth wrote:
Hi
I am running openldap-servers-2.4.40-16.el6.x86_64 cluster on Centos
6.9. My
/var/lib/ldap directory contains many 10MB log files. /var partition
rather
small...
I've read they can be removed either by running "sudo db_archive -d -h /var/lib/ldap/domain" or by defining "DB_LOG_AUTOREMOVE" within the file "DB_CONFIG." That file does not presently exist whereas the db_archive command does not actually remove any of the log files.
If the db_archive command doesn't remove anything, that means it thinks all of the log files are still in active use.
Read the docs more carefully. https://urldefense.proofpoint.com/v2/url?u=http-3A__docs. oracle.com_cd_E17076-5F05_html_programmer-5Freference_ transapp-5Flogfile.html&d=DwICaQ&c=lb62iw4YL4RFalcE2hQUQealT9- RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= WP95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yEEDA&s= Kfi27b4v7vABZjPQYMkeo4xBqUyDGZeyB8pHAFin8xY&e=
Can I remove the old log files manually using rm?
Not if the above is true, you will corrupt the logs and the DB will fail to open on a subsequent restart.
If not should I create /var/lib/ldap/DB_CONFIG then restart slapd to make this removal
automatic?
Do you have any idea why db_archive does not work or produce any helpful
error
to stdout?
There's no error message because there's no error, everything is working as designed.
You need to do periodic checkpoints to allow log files to be closed, and then db_archive will be able to remove some of them.
-- -- Howard Chu CTO, Symas Corp. https://urldefense.proofpoint. com/v2/url?u=http-3A__www.symas.com&d=DwICaQ&c=lb62iw4YL4RFalcE2hQUQealT9- RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= WP95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yEEDA&s=IT7tNF72SCugdO8WpRd- oNsk4nPNpdjE2aUFL4R4X_M&e= Director, Highland Sun https://urldefense.proofpoint. com/v2/url?u=http-3A__highlandsun.com_hyc_&d=DwICaQ& c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yEEDA&s= XqfYCnjG9ibPbeW05QZOlWdl9u0ZH-7IXkxx0gh238k&e= Chief Architect, OpenLDAP https://urldefense.proofpoint. com/v2/url?u=http-3A__www.openldap.org_project_&d=DwICaQ&c= lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yEEDA&s=- tGdeTJRpeaRbljBBUq49XgfNWzVElqiGEgv0LeqspU&e=
Adding
# checkpointing - added 8/29/2017 checkpoint 128 10
To slapd.conf then running
sudo db_archive -d -h /var/lib/ldap/domain
Removed the old log files. /var now using under 1GB.
Thanks Howard!
Our LDAP server contains about 4000 entries. At what point would adding DB_CONFIG be needed for performance reasons? How would I even ascertain that there's performance issues?
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 F: 212-746-8690
On Mon, Aug 28, 2017 at 10:19 AM, Douglas Duckworth <dod2014@med.cornell.edu
wrote:
Thanks for the reply, Howard.
Thanks for pointing me in the right direction. From what I have read there are two options.
- Copy /usr/share/openldap-servers/DB_CONFIG.example to /var/lib/domain
then rebuild the database. 2) Enable checkpointing in slapd.conf
Does enabling checkpointing in slapd.conf require rebuilding the database or can I simply restart slapd.conf? We are not using online configuration.
Best Doug
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Fri, Aug 25, 2017 at 8:55 AM, Howard Chu hyc@symas.com wrote:
Douglas Duckworth wrote:
Hi
I am running openldap-servers-2.4.40-16.el6.x86_64 cluster on Centos
6.9. My
/var/lib/ldap directory contains many 10MB log files. /var partition
rather
small...
I've read they can be removed either by running "sudo db_archive -d -h /var/lib/ldap/domain" or by defining "DB_LOG_AUTOREMOVE" within the file "DB_CONFIG." That file does not presently exist whereas the db_archive command does not actually remove any of the log files.
If the db_archive command doesn't remove anything, that means it thinks all of the log files are still in active use.
Read the docs more carefully. https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ora cle.com_cd_E17076-5F05_html_programmer-5Freference_transap p-5Flogfile.html&d=DwICaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrry qt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= WP95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yEEDA&s=Kfi27b4v7vABZj PQYMkeo4xBqUyDGZeyB8pHAFin8xY&e=
Can I remove the old log files manually using rm?
Not if the above is true, you will corrupt the logs and the DB will fail to open on a subsequent restart.
If not should I create /var/lib/ldap/DB_CONFIG then restart slapd to make this removal
automatic?
Do you have any idea why db_archive does not work or produce any
helpful error
to stdout?
There's no error message because there's no error, everything is working as designed.
You need to do periodic checkpoints to allow log files to be closed, and then db_archive will be able to remove some of them.
-- -- Howard Chu CTO, Symas Corp. https://urldefense. proofpoint.com/v2/url?u=http-3A__www.symas.com&d=DwICaQ&c=l b62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQp l_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTt JDAUAKN756yEEDA&s=IT7tNF72SCugdO8WpRd-oNsk4nPNpdjE2aUFL4R4X_M&e= Director, Highland Sun https://urldefense. proofpoint.com/v2/url?u=http-3A__highlandsun.com_hyc_&d=DwICaQ&c= lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQ pl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTt JDAUAKN756yEEDA&s=XqfYCnjG9ibPbeW05QZOlWdl9u0ZH-7IXkxx0gh238k&e= Chief Architect, OpenLDAP https://urldefense.proofpoint. com/v2/url?u=http-3A__www.openldap.org_project_&d=DwICaQ&c=l b62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQp l_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTt JDAUAKN756yEEDA&s=-tGdeTJRpeaRbljBBUq49XgfNWzVElqiGEgv0LeqspU&e=
This seems to help
user@ldap[~]$ sudo -u ldap /usr/bin/db_stat -h /var/lib/ldap/domain -c 566 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 1000 Maximum number of locks possible 1000 Maximum number of lockers possible 1000 Maximum number of lock objects possible 80 Number of lock object partitions 16 Number of current locks 986 Maximum number of locks at any one time 14 Maximum number of locks in any one bucket 303 Maximum number of locks stolen by for an empty partition 18 Maximum number of locks stolen for any one partition 90 Number of current lockers 130 Maximum number of lockers at any one time 16 Number of current lock objects 519 Maximum number of lock objects at any one time 8 Maximum number of lock objects in any one bucket 0 Maximum number of objects stolen by for an empty partition 0 Maximum number of objects stolen for any one partition 348M Total number of locks requested (348174715) 348M Total number of locks released (348174394) 0 Total number of locks upgraded 112 Total number of locks downgraded 10622 Lock requests not available due to conflicts, for which we waited <------ sounds bad 0 Lock requests not available due to conflicts, for which we did not wait 2 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 744KB The size of the lock region 221341 The number of partition locks that required waiting (0%) 5041 The maximum number of times any partition lock was waited for (0%) 1 The number of object queue operations that required waiting (0%) 40577 The number of locker allocations that required waiting (0%) 0 The number of region locks that required waiting (0%) 8 Maximum hash bucket length
Only four clients are currently using this cluster so perhaps I should actually use DB_CONFIG before putting it into production.
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 F: 212-746-8690
On Tue, Aug 29, 2017 at 2:13 PM, Douglas Duckworth dod2014@med.cornell.edu wrote:
Adding
# checkpointing - added 8/29/2017 checkpoint 128 10
To slapd.conf then running
sudo db_archive -d -h /var/lib/ldap/domain
Removed the old log files. /var now using under 1GB.
Thanks Howard!
Our LDAP server contains about 4000 entries. At what point would adding DB_CONFIG be needed for performance reasons? How would I even ascertain that there's performance issues?
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Mon, Aug 28, 2017 at 10:19 AM, Douglas Duckworth < dod2014@med.cornell.edu> wrote:
Thanks for the reply, Howard.
Thanks for pointing me in the right direction. From what I have read there are two options.
- Copy /usr/share/openldap-servers/DB_CONFIG.example to /var/lib/domain
then rebuild the database. 2) Enable checkpointing in slapd.conf
Does enabling checkpointing in slapd.conf require rebuilding the database or can I simply restart slapd.conf? We are not using online configuration.
Best Doug
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Fri, Aug 25, 2017 at 8:55 AM, Howard Chu hyc@symas.com wrote:
Douglas Duckworth wrote:
Hi
I am running openldap-servers-2.4.40-16.el6.x86_64 cluster on Centos
6.9. My
/var/lib/ldap directory contains many 10MB log files. /var partition
rather
small...
I've read they can be removed either by running "sudo db_archive -d -h /var/lib/ldap/domain" or by defining "DB_LOG_AUTOREMOVE" within the
file
"DB_CONFIG." That file does not presently exist whereas the db_archive command does not actually remove any of the log files.
If the db_archive command doesn't remove anything, that means it thinks all of the log files are still in active use.
Read the docs more carefully. https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ora cle.com_cd_E17076-5F05_html_programmer-5Freference_transapp- 5Flogfile.html&d=DwICaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt 9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=W P95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yEEDA&s=Kfi27b4v7vABZjP QYMkeo4xBqUyDGZeyB8pHAFin8xY&e=
Can I remove the old log files manually using rm?
Not if the above is true, you will corrupt the logs and the DB will fail to open on a subsequent restart.
If not should I create /var/lib/ldap/DB_CONFIG then restart slapd to make this removal
automatic?
Do you have any idea why db_archive does not work or produce any
helpful error
to stdout?
There's no error message because there's no error, everything is working as designed.
You need to do periodic checkpoints to allow log files to be closed, and then db_archive will be able to remove some of them.
-- -- Howard Chu CTO, Symas Corp. https://urldefense.proofpoint .com/v2/url?u=http-3A__www.symas.com&d=DwICaQ&c=lb62iw4YL 4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJD AUAKN756yEEDA&s=IT7tNF72SCugdO8WpRd-oNsk4nPNpdjE2aUFL4R4X_M&e= Director, Highland Sun https://urldefense.proofpoint .com/v2/url?u=http-3A__highlandsun.com_hyc_&d=DwICaQ& c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJD AUAKN756yEEDA&s=XqfYCnjG9ibPbeW05QZOlWdl9u0ZH-7IXkxx0gh238k&e= Chief Architect, OpenLDAP https://urldefense.proofpoint. com/v2/url?u=http-3A__www.openldap.org_project_&d=DwICaQ&c=l b62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQp l_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJ DAUAKN756yEEDA&s=-tGdeTJRpeaRbljBBUq49XgfNWzVElqiGEgv0LeqspU&e=
Once I put in place DB_CONFIG I should stop slapd then run this to reinitialize the database with the new runtime configuration:
sudo -u ldap /usr/bin/db_recover -h /var/lib/ldap/domain -v
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 F: 212-746-8690
On Wed, Aug 30, 2017 at 10:06 AM, Douglas Duckworth <dod2014@med.cornell.edu
wrote:
This seems to help
user@ldap[~]$ sudo -u ldap /usr/bin/db_stat -h /var/lib/ldap/domain -c 566 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 1000 Maximum number of locks possible 1000 Maximum number of lockers possible 1000 Maximum number of lock objects possible 80 Number of lock object partitions 16 Number of current locks 986 Maximum number of locks at any one time 14 Maximum number of locks in any one bucket 303 Maximum number of locks stolen by for an empty partition 18 Maximum number of locks stolen for any one partition 90 Number of current lockers 130 Maximum number of lockers at any one time 16 Number of current lock objects 519 Maximum number of lock objects at any one time 8 Maximum number of lock objects in any one bucket 0 Maximum number of objects stolen by for an empty partition 0 Maximum number of objects stolen for any one partition 348M Total number of locks requested (348174715) 348M Total number of locks released (348174394) 0 Total number of locks upgraded 112 Total number of locks downgraded 10622 Lock requests not available due to conflicts, for which we waited <------ sounds bad 0 Lock requests not available due to conflicts, for which we did not wait 2 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 744KB The size of the lock region 221341 The number of partition locks that required waiting (0%) 5041 The maximum number of times any partition lock was waited for (0%) 1 The number of object queue operations that required waiting (0%) 40577 The number of locker allocations that required waiting (0%) 0 The number of region locks that required waiting (0%) 8 Maximum hash bucket length
Only four clients are currently using this cluster so perhaps I should actually use DB_CONFIG before putting it into production.
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Tue, Aug 29, 2017 at 2:13 PM, Douglas Duckworth < dod2014@med.cornell.edu> wrote:
Adding
# checkpointing - added 8/29/2017 checkpoint 128 10
To slapd.conf then running
sudo db_archive -d -h /var/lib/ldap/domain
Removed the old log files. /var now using under 1GB.
Thanks Howard!
Our LDAP server contains about 4000 entries. At what point would adding DB_CONFIG be needed for performance reasons? How would I even ascertain that there's performance issues?
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Mon, Aug 28, 2017 at 10:19 AM, Douglas Duckworth < dod2014@med.cornell.edu> wrote:
Thanks for the reply, Howard.
Thanks for pointing me in the right direction. From what I have read there are two options.
- Copy /usr/share/openldap-servers/DB_CONFIG.example to
/var/lib/domain then rebuild the database. 2) Enable checkpointing in slapd.conf
Does enabling checkpointing in slapd.conf require rebuilding the database or can I simply restart slapd.conf? We are not using online configuration.
Best Doug
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Fri, Aug 25, 2017 at 8:55 AM, Howard Chu hyc@symas.com wrote:
Douglas Duckworth wrote:
Hi
I am running openldap-servers-2.4.40-16.el6.x86_64 cluster on Centos
6.9. My
/var/lib/ldap directory contains many 10MB log files. /var partition
rather
small...
I've read they can be removed either by running "sudo db_archive -d -h /var/lib/ldap/domain" or by defining "DB_LOG_AUTOREMOVE" within the
file
"DB_CONFIG." That file does not presently exist whereas the
db_archive
command does not actually remove any of the log files.
If the db_archive command doesn't remove anything, that means it thinks all of the log files are still in active use.
Read the docs more carefully. https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ora cle.com_cd_E17076-5F05_html_programmer-5Freference_transapp- 5Flogfile.html&d=DwICaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt 9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP 95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yEEDA&s=Kfi27b4v7vABZjPQ YMkeo4xBqUyDGZeyB8pHAFin8xY&e=
Can I remove the old log files manually using rm?
Not if the above is true, you will corrupt the logs and the DB will fail to open on a subsequent restart.
If not should I create /var/lib/ldap/DB_CONFIG then restart slapd to make this removal
automatic?
Do you have any idea why db_archive does not work or produce any
helpful error
to stdout?
There's no error message because there's no error, everything is working as designed.
You need to do periodic checkpoints to allow log files to be closed, and then db_archive will be able to remove some of them.
-- -- Howard Chu CTO, Symas Corp. https://urldefense.proofpoint .com/v2/url?u=http-3A__www.symas.com&d=DwICaQ&c=lb62iw4YL4RF alcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xU jnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yE EDA&s=IT7tNF72SCugdO8WpRd-oNsk4nPNpdjE2aUFL4R4X_M&e= Director, Highland Sun https://urldefense.proofpoint .com/v2/url?u=http-3A__highlandsun.com_hyc_&d=DwICaQ&c= lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQ pl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTt JDAUAKN756yEEDA&s=XqfYCnjG9ibPbeW05QZOlWdl9u0ZH-7IXkxx0gh238k&e= Chief Architect, OpenLDAP https://urldefense.proofpoint. com/v2/url?u=http-3A__www.openldap.org_project_&d=DwICaQ&c=l b62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQp l_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJ DAUAKN756yEEDA&s=-tGdeTJRpeaRbljBBUq49XgfNWzVElqiGEgv0LeqspU&e=
Yeah, that worked.
DB_CONFIG now contains:
# one 1 GB cache set_cachesize 0 1073741824 1
# Data Directory set_data_dir /var/lib/ldap/domain
# Transaction Log settings set_lg_regionmax 1048576 set_lg_max 10485760 set_lg_bsize 2097152 set_lg_dir /var/lib/ldap/domain
# autoremove log files set_flags DB_LOG_AUTOREMOVE
Stats now show:
80 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 1000 Maximum number of locks possible 1000 Maximum number of lockers possible 1000 Maximum number of lock objects possible 80 Number of lock object partitions 16 Number of current locks 724 Maximum number of locks at any one time 12 Maximum number of locks in any one bucket 9 Maximum number of locks stolen by for an empty partition 2 Maximum number of locks stolen for any one partition 80 Number of current lockers 82 Maximum number of lockers at any one time 16 Number of current lock objects 391 Maximum number of lock objects at any one time 6 Maximum number of lock objects in any one bucket 0 Maximum number of objects stolen by for an empty partition 0 Maximum number of objects stolen for any one partition 1630324 Total number of locks requested 1630299 Total number of locks released 0 Total number of locks upgraded 18 Total number of locks downgraded 1 Lock requests not available due to conflicts, for which we waited 0 Lock requests not available due to conflicts, for which we did not wait 0 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 744KB The size of the lock region 1 The number of partition locks that required waiting (0%) 1 The maximum number of times any partition lock was waited for (0%) 0 The number of object queue operations that required waiting (0%) 0 The number of locker allocations that required waiting (0%) 0 The number of region locks that required waiting (0%) 6 Maximum hash bucket length
Our entire database dumped to ldif happens to be only 1MB in size while we only do writes maybe once or twice a week. So the 1GB cache seems rather wasteful. From reading http://www.openldap.org/faq/data/cache/1072.html the cache should be in memory not on disk? I see the 1GB file exists on disk while memory usage for slapd has not actually increased. Does it preallocate the entire cache to disk then only flush from memory to disk when "set_lg_bsize" value has been reached?
Moreover, after removing "checkpoint" from slapd.conf, and using the above DB_CONFIG, I now see several more 10MB *.log files.
-rw------- 1 ldap ldap 10M Aug 30 11:45 log.0000000455 -rw------- 1 ldap ldap 10M Aug 30 11:45 log.0000000456 -rw------- 1 ldap ldap 10M Aug 30 11:45 log.0000000457 -rw------- 1 ldap ldap 10M Aug 30 11:45 log.0000000458 -rw------- 1 ldap ldap 24K Aug 30 11:46 __db.001 -rw------- 1 ldap ldap 32K Aug 30 11:55 __db.006 -rw------- 1 ldap ldap 3.0M Aug 30 11:55 __db.004 -rw------- 1 ldap ldap 1.0G Aug 30 11:55 __db.003 -rw------- 1 ldap ldap 744K Aug 30 11:56 __db.005 -rw------- 1 ldap ldap 35M Aug 30 11:56 __db.002
Why do so many transaction log files exist if my entire database happens to be only 1MB?
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 F: 212-746-8690
On Wed, Aug 30, 2017 at 10:11 AM, Douglas Duckworth <dod2014@med.cornell.edu
wrote:
Once I put in place DB_CONFIG I should stop slapd then run this to reinitialize the database with the new runtime configuration:
sudo -u ldap /usr/bin/db_recover -h /var/lib/ldap/domain -v
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Wed, Aug 30, 2017 at 10:06 AM, Douglas Duckworth < dod2014@med.cornell.edu> wrote:
This seems to help
user@ldap[~]$ sudo -u ldap /usr/bin/db_stat -h /var/lib/ldap/domain -c 566 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 1000 Maximum number of locks possible 1000 Maximum number of lockers possible 1000 Maximum number of lock objects possible 80 Number of lock object partitions 16 Number of current locks 986 Maximum number of locks at any one time 14 Maximum number of locks in any one bucket 303 Maximum number of locks stolen by for an empty partition 18 Maximum number of locks stolen for any one partition 90 Number of current lockers 130 Maximum number of lockers at any one time 16 Number of current lock objects 519 Maximum number of lock objects at any one time 8 Maximum number of lock objects in any one bucket 0 Maximum number of objects stolen by for an empty partition 0 Maximum number of objects stolen for any one partition 348M Total number of locks requested (348174715) 348M Total number of locks released (348174394) 0 Total number of locks upgraded 112 Total number of locks downgraded 10622 Lock requests not available due to conflicts, for which we waited <------ sounds bad 0 Lock requests not available due to conflicts, for which we did not wait 2 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 744KB The size of the lock region 221341 The number of partition locks that required waiting (0%) 5041 The maximum number of times any partition lock was waited for (0%) 1 The number of object queue operations that required waiting (0%) 40577 The number of locker allocations that required waiting (0%) 0 The number of region locks that required waiting (0%) 8 Maximum hash bucket length
Only four clients are currently using this cluster so perhaps I should actually use DB_CONFIG before putting it into production.
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Tue, Aug 29, 2017 at 2:13 PM, Douglas Duckworth < dod2014@med.cornell.edu> wrote:
Adding
# checkpointing - added 8/29/2017 checkpoint 128 10
To slapd.conf then running
sudo db_archive -d -h /var/lib/ldap/domain
Removed the old log files. /var now using under 1GB.
Thanks Howard!
Our LDAP server contains about 4000 entries. At what point would adding DB_CONFIG be needed for performance reasons? How would I even ascertain that there's performance issues?
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Mon, Aug 28, 2017 at 10:19 AM, Douglas Duckworth < dod2014@med.cornell.edu> wrote:
Thanks for the reply, Howard.
Thanks for pointing me in the right direction. From what I have read there are two options.
- Copy /usr/share/openldap-servers/DB_CONFIG.example to
/var/lib/domain then rebuild the database. 2) Enable checkpointing in slapd.conf
Does enabling checkpointing in slapd.conf require rebuilding the database or can I simply restart slapd.conf? We are not using online configuration.
Best Doug
Thanks,
Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Physiology and Biophysics Weill Cornell Medicine E: doug@med.cornell.edu O: 212-746-6305 <(212)%20746-6305> F: 212-746-8690 <(212)%20746-8690>
On Fri, Aug 25, 2017 at 8:55 AM, Howard Chu hyc@symas.com wrote:
Douglas Duckworth wrote:
Hi
I am running openldap-servers-2.4.40-16.el6.x86_64 cluster on
Centos 6.9. My
/var/lib/ldap directory contains many 10MB log files. /var
partition rather
small...
I've read they can be removed either by running "sudo db_archive -d
-h
/var/lib/ldap/domain" or by defining "DB_LOG_AUTOREMOVE" within the
file
"DB_CONFIG." That file does not presently exist whereas the
db_archive
command does not actually remove any of the log files.
If the db_archive command doesn't remove anything, that means it thinks all of the log files are still in active use.
Read the docs more carefully. https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ora cle.com_cd_E17076-5F05_html_programmer-5Freference_transapp- 5Flogfile.html&d=DwICaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt 9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP 95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yEEDA&s=Kfi27b4v7vABZjPQ YMkeo4xBqUyDGZeyB8pHAFin8xY&e=
Can I remove the old log files manually using rm?
Not if the above is true, you will corrupt the logs and the DB will fail to open on a subsequent restart.
If not should I create /var/lib/ldap/DB_CONFIG then restart slapd to make this removal
automatic?
Do you have any idea why db_archive does not work or produce any
helpful error
to stdout?
There's no error message because there's no error, everything is working as designed.
You need to do periodic checkpoints to allow log files to be closed, and then db_archive will be able to remove some of them.
-- -- Howard Chu CTO, Symas Corp. https://urldefense.proofpoint .com/v2/url?u=http-3A__www.symas.com&d=DwICaQ&c=lb62iw4YL4RF alcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xU jnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJDAUAKN756yE EDA&s=IT7tNF72SCugdO8WpRd-oNsk4nPNpdjE2aUFL4R4X_M&e= Director, Highland Sun https://urldefense.proofpoint .com/v2/url?u=http-3A__highlandsun.com_hyc_&d=DwICaQ&c=lb62i w4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJD AUAKN756yEEDA&s=XqfYCnjG9ibPbeW05QZOlWdl9u0ZH-7IXkxx0gh238k&e= Chief Architect, OpenLDAP https://urldefense.proofpoint. com/v2/url?u=http-3A__www.openldap.org_project_&d=DwICaQ&c=l b62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQp l_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=WP95x8mwSiEHHqUWRqJv6WdpfcTtJ DAUAKN756yEEDA&s=-tGdeTJRpeaRbljBBUq49XgfNWzVElqiGEgv0LeqspU&e=
Douglas Duckworth dod2014@med.cornell.edu schrieb am 30.08.2017 um 16:06 in
Nachricht CAAKHBKm0FKBR4BLNpy6jrszFp35Eid_J2F19MKmfS+G47EKCJA@mail.gmail.com: [...]
Only four clients are currently using this cluster so perhaps I should actually use DB_CONFIG before putting it into production.
Maybe you could start considering this from a moderately loaded and sized server: set_cachesize 0 15000000 1 set_lg_regionmax 262144 set_lg_bsize 2097152 set_flags DB_LOG_AUTOREMOVE set_lk_max_locks 30000 set_lk_max_objects 30000 ---
Also note "olcDbConfig: <DB_CONFIG setting>" is probably the modern way to do it.
Regards, Ulrich
openldap-technical@openldap.org
-
Douglas Duckworth -
Ulrich Windl