Hi,
I'm observing an issue where a large number of searches against an openldap server results in a large amount of disk writes occurring.
I have 10 hosts performing the same workload, the hosts are running slapd 2.4.21 under Ubuntu Lucid. If I stop searching against one of the hosts I see disk write activity from that host drop dramatically. I initially noticed the high write activity in the VMware stats for the hosts and have used iostat on the hosts to gather further info. The hosts being searched are slaves in a slurpd replication environment; they do not have replication logging enabled, the directory is using a hdb backend.
Over a 5 minute period the average disk write activity is 8517KB/s when being searched, and 41KB/s when not being searched; over the same time period the average disk read activity is 8KB/s.
When performing searches against a host an average workload is 300 searches/second, average entry size is 450bytes, and average entries returned is 30.
The hosts have plenty of free RAM and are not using any swap. I have disabled the monitor backend but haven't seen much of an improvement by doing this, given the monitor database is instantiated at startup is this stored in memory? (if not, why not make that an option?)
Does anyone have any ideas as to why reading from the directory would be causing disk writes at such a high rate?
Thanks, Adam
Adam Wale writes:
I'm observing an issue where a large number of searches against an openldap server results in a large amount of disk writes occurring.
Maybe you have set a high loglevel in slapd.conf, or you are using the slapd '-d' argument.
Loglevel is what gets logged to syslog. Default logevel is 'stats' (256), which gives a few lines per LDAP request. Some years ago our site had to set loglevel=0 because it could not handle all the syslogging, but a hardware upgrade fixed that. Default syslog user.level=local4.debug, see man slapd.
The hosts have plenty of free RAM and are not using any swap. I have disabled the monitor backend but haven't seen much of an improvement by doing this, given the monitor database is instantiated at startup is this stored in memory? (if not, why not make that an option?)
Yes, monitor is memory-only.
Hi,
Thanks for the response, unfortunately we are already using loglevel 0, and are not using slapd -d.
________________________________________ From: Hallvard Breien Furuseth [h.b.furuseth@usit.uio.no] Sent: Saturday, 29 October 2011 8:47 AM To: Adam Wale Cc: openldap-technical@openldap.org Subject: Re: Searches causing disk writes
Adam Wale writes:
I'm observing an issue where a large number of searches against an openldap server results in a large amount of disk writes occurring.
Maybe you have set a high loglevel in slapd.conf, or you are using the slapd '-d' argument.
Loglevel is what gets logged to syslog. Default logevel is 'stats' (256), which gives a few lines per LDAP request. Some years ago our site had to set loglevel=0 because it could not handle all the syslogging, but a hardware upgrade fixed that. Default syslog user.level=local4.debug, see man slapd.
The hosts have plenty of free RAM and are not using any swap. I have disabled the monitor backend but haven't seen much of an improvement by doing this, given the monitor database is instantiated at startup is this stored in memory? (if not, why not make that an option?)
Yes, monitor is memory-only.
-- Hallvard
Perhaps use iotop while you do a big search?
On Sat, Oct 29, 2011 at 11:11:44AM +0800, Adam Wale wrote:
Hi,
Thanks for the response, unfortunately we are already using loglevel 0, and are not using slapd -d.
From: Hallvard Breien Furuseth [h.b.furuseth@usit.uio.no] Sent: Saturday, 29 October 2011 8:47 AM To: Adam Wale Cc: openldap-technical@openldap.org Subject: Re: Searches causing disk writes
Adam Wale writes:
I'm observing an issue where a large number of searches against an openldap server results in a large amount of disk writes occurring.
Maybe you have set a high loglevel in slapd.conf, or you are using the slapd '-d' argument.
Loglevel is what gets logged to syslog. Default logevel is 'stats' (256), which gives a few lines per LDAP request. Some years ago our site had to set loglevel=0 because it could not handle all the syslogging, but a hardware upgrade fixed that. Default syslog user.level=local4.debug, see man slapd.
The hosts have plenty of free RAM and are not using any swap. I have disabled the monitor backend but haven't seen much of an improvement by doing this, given the monitor database is instantiated at startup is this stored in memory? (if not, why not make that an option?)
Yes, monitor is memory-only.
-- Hallvard
iotop shows me that the slapd process is doing the writing, I'm trying to identify what exactly it's writing, and if there's a way I can prevent it
________________________________________ From: openldap-technical-bounces@OpenLDAP.org [openldap-technical-bounces@OpenLDAP.org] On Behalf Of Christopher Wood [christopher_wood@pobox.com] Sent: Saturday, 29 October 2011 11:30 AM To: openldap-technical@openldap.org Subject: Re: Searches causing disk writes
Perhaps use iotop while you do a big search?
On Sat, Oct 29, 2011 at 11:11:44AM +0800, Adam Wale wrote:
Hi,
Thanks for the response, unfortunately we are already using loglevel 0, and are not using slapd -d.
From: Hallvard Breien Furuseth [h.b.furuseth@usit.uio.no] Sent: Saturday, 29 October 2011 8:47 AM To: Adam Wale Cc: openldap-technical@openldap.org Subject: Re: Searches causing disk writes
Adam Wale writes:
I'm observing an issue where a large number of searches against an openldap server results in a large amount of disk writes occurring.
Maybe you have set a high loglevel in slapd.conf, or you are using the slapd '-d' argument.
Loglevel is what gets logged to syslog. Default logevel is 'stats' (256), which gives a few lines per LDAP request. Some years ago our site had to set loglevel=0 because it could not handle all the syslogging, but a hardware upgrade fixed that. Default syslog user.level=local4.debug, see man slapd.
The hosts have plenty of free RAM and are not using any swap. I have disabled the monitor backend but haven't seen much of an improvement by doing this, given the monitor database is instantiated at startup is this stored in memory? (if not, why not make that an option?)
Yes, monitor is memory-only.
-- Hallvard
In that case, strace/ltrace/dtrace/etc. on the slapd process will show you what the process itself is doing.
On Sat, Oct 29, 2011 at 08:22:38PM +0800, Adam Wale wrote:
iotop shows me that the slapd process is doing the writing, I'm trying to identify what exactly it's writing, and if there's a way I can prevent it
From: openldap-technical-bounces@OpenLDAP.org [openldap-technical-bounces@OpenLDAP.org] On Behalf Of Christopher Wood [christopher_wood@pobox.com] Sent: Saturday, 29 October 2011 11:30 AM To: openldap-technical@openldap.org Subject: Re: Searches causing disk writes
Perhaps use iotop while you do a big search?
On Sat, Oct 29, 2011 at 11:11:44AM +0800, Adam Wale wrote:
Hi,
Thanks for the response, unfortunately we are already using loglevel 0, and are not using slapd -d.
From: Hallvard Breien Furuseth [h.b.furuseth@usit.uio.no] Sent: Saturday, 29 October 2011 8:47 AM To: Adam Wale Cc: openldap-technical@openldap.org Subject: Re: Searches causing disk writes
Adam Wale writes:
I'm observing an issue where a large number of searches against an openldap server results in a large amount of disk writes occurring.
Maybe you have set a high loglevel in slapd.conf, or you are using the slapd '-d' argument.
Loglevel is what gets logged to syslog. Default logevel is 'stats' (256), which gives a few lines per LDAP request. Some years ago our site had to set loglevel=0 because it could not handle all the syslogging, but a hardware upgrade fixed that. Default syslog user.level=local4.debug, see man slapd.
The hosts have plenty of free RAM and are not using any swap. I have disabled the monitor backend but haven't seen much of an improvement by doing this, given the monitor database is instantiated at startup is this stored in memory? (if not, why not make that an option?)
Yes, monitor is memory-only.
-- Hallvard
On Sat, 29 Oct 2011, Christopher Wood wrote:
On Sat, Oct 29, 2011 at 08:22:38PM +0800, Adam Wale wrote:
iotop shows me that the slapd process is doing the writing, I'm trying to identify what exactly it's writing, and if there's a way I can prevent it
How do memory writes to mmap'ed file show up in iotop? The BDB library, by default, uses mmap'ed filed for its shared-memory regions for its mpool page cache, holding locks and the transaction data structures, etc. If a search requires lots of pages to be pulled into BDB's page cache in those mmap'ed files, will those show as writes to iotop.
If so, and those really are getting written to disk, then switching from mmap'ed files to SysV shared memory may be the solution. The shm_key option (olcDbShmKey in back-config) lets you enable that. Be aware that on many OS's, the the default system limits for shared memory size were fairly low, so you may need to bump that up. On Linux take a look at /proc/sys/kernel/shmmax and/or the kernel.shmmax sysctl in /etc/sysctl.conf
In that case, strace/ltrace/dtrace/etc. on the slapd process will show you what the process itself is doing.
strace may confirm the above to be the problem by the *absence* of syscalls writing data. dtrace might be able to show it directly...
Philip Guenther
On Thu, Oct 27, 2011 at 5:27 AM, Adam Wale adamw@staff.iinet.net.au wrote:
Hi,****
I’m observing an issue where a large number of searches against an openldap server results in a large amount of disk writes occurring.****
I have 10 hosts performing the same workload, the hosts are running slapd 2.4.21 under Ubuntu Lucid. If I stop searching against one of the hosts I see disk write activity from that host drop dramatically. I initially noticed the high write activity in the VMware stats for the hosts and have used iostat on the hosts to gather further info. The hosts being searched are slaves in a slurpd replication environment; they do not have replication logging enabled, the directory is using a hdb backend.****
Over a 5 minute period the average disk write activity is 8517KB/s when being searched, and 41KB/s when not being searched; over the same time period the average disk read activity is 8KB/s.****
When performing searches against a host an average workload is 300 searches/second, average entry size is 450bytes, and average entries returned is 30.****
The hosts have plenty of free RAM and are not using any swap. I have disabled the monitor backend but haven’t seen much of an improvement by doing this, given the monitor database is instantiated at startup is this stored in memory? (if not, why not make that an option?)****
Does anyone have any ideas as to why reading from the directory would be causing disk writes at such a high rate?****
**
Are you sure that slapd aren't logging event to the syslogd regular file ? If so and you need this logging you may prefix each entry with the minus ''-'' sign to omit syncing the file after every logging.
Best Regards
**
Thanks,****
Adam****
For anyone that was interested in the fix for this, moving to shared memory resolved the issue.
-----Original Message----- From: Philip Guenther [mailto:guenther+ldaptech@sendmail.com] Sent: Sunday, 30 October 2011 4:06 AM To: Christopher Wood; Adam Wale Cc: openldap-technical@openldap.org Subject: Re: Searches causing disk writes
On Sat, 29 Oct 2011, Christopher Wood wrote:
On Sat, Oct 29, 2011 at 08:22:38PM +0800, Adam Wale wrote:
iotop shows me that the slapd process is doing the writing, I'm trying to identify what exactly it's writing, and if there's a way I can prevent it
How do memory writes to mmap'ed file show up in iotop? The BDB library, by default, uses mmap'ed filed for its shared-memory regions for its mpool page cache, holding locks and the transaction data structures, etc. If a search requires lots of pages to be pulled into BDB's page cache in those mmap'ed files, will those show as writes to iotop.
If so, and those really are getting written to disk, then switching from mmap'ed files to SysV shared memory may be the solution. The shm_key option (olcDbShmKey in back-config) lets you enable that. Be aware that on many OS's, the the default system limits for shared memory size were fairly low, so you may need to bump that up. On Linux take a look at /proc/sys/kernel/shmmax and/or the kernel.shmmax sysctl in /etc/sysctl.conf
In that case, strace/ltrace/dtrace/etc. on the slapd process will show you what the process itself is doing.
strace may confirm the above to be the problem by the *absence* of syscalls writing data. dtrace might be able to show it directly...
Philip Guenther
2011/11/10 Adam Wale adamw@staff.iinet.net.au:
For anyone that was interested in the fix for this, moving to shared memory resolved the issue.
Hello Adam,
we had the same problem and could it solve the same way. Sorry, I haven't seen this thread..
Do you have tried to mount your partition where the data directory is located with "noatime"? This (atime) causes a lot of very expensive write operations on every read access of the memory mapped files. It would be very interesting if mounting the data partition with "noatime" can solve the problem too? But be careful, some other programs (e.g. mail, backup,..) may rely on atime, it depends what programs using this partition.
kindly regards Meike
Meike Stone wrote:
2011/11/10 Adam Waleadamw@staff.iinet.net.au:
For anyone that was interested in the fix for this, moving to shared memory resolved the issue.
Hello Adam,
we had the same problem and could it solve the same way. Sorry, I haven't seen this thread..
Do you have tried to mount your partition where the data directory is located with "noatime"? This (atime) causes a lot of very expensive write operations on every read access of the memory mapped files. It would be very interesting if mounting the data partition with "noatime" can solve the problem too? But be careful, some other programs (e.g. mail, backup,..) may rely on atime, it depends what programs using this partition.
Use relatime, that will take care of any other dependencies.
openldap-technical@openldap.org
-
Adam Wale -
Christopher Wood -
devzero2000 -
Hallvard B Furuseth -
Howard Chu -
Meike Stone -
Philip Guenther