Hi Quanah,
Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012 i686 GNU/Linux The latest OpenLDAP 2.4.39 All of those tests done with the mdb backend, of course, and the actual file system is ext4
It's a fairly stock 10.04 system, no special config/kernel changes.
Cheers, Andrew
Andrew Eross CTO Locatrix Communications Office: +61 7 3123 1469 Mobile: +55 37 9858 9815 eross@locatrix.com
On Wed, Feb 5, 2014 at 1:26 PM, Quanah Gibson-Mount quanah@zimbra.comwrote:
--On Tuesday, February 04, 2014 6:52 PM -0200 Andrew Eross < eross@locatrix.com> wrote:
Thanks, Dieter, Quanah.
I've been doing some experimenting with those mdb options.
I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time:
Base-line, no extra options: 4m8sWith "writemap" enabled: 8m55s
With "writemap+mapasync" enabled: 5m12s With "dbnosync+checkpoint 0kb/1min": 0m14s
I know you answered some of this before, but please update with:
What kernel? What OpenLDAP version? What Ubuntu release? What filesystem for the LDAP DB?
Thanks,
Quanah
--
Quanah Gibson-Mount Architect - Server Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration
--On Wednesday, February 05, 2014 1:30 PM -0200 Andrew Eross eross@locatrix.com wrote:
Hi Quanah,
Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012 i686 GNU/Linux
The latest OpenLDAP 2.4.39
All of those tests done with the mdb backend, of course, and the actual file system is ext4
It's a fairly stock 10.04 system, no special config/kernel changes.
Ok. I had some wierd perf issues with Ubuntu10 and Ubuntu12. I was able to resolve them on Ubuntu12 by upgrading to a newer 3.x kernel (currently 3.10.11-031011-generic on my Ubuntu12 box).
Also, it isn't necessary to use ext4 (I use ext2 for the ldap data with mdb). ext4 in the 2.x kernel has some serious perf problems as well (when using writemap).
--Quanah
--
Quanah Gibson-Mount Architect - Server Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On 02/05/2014 11:45 AM, Quanah Gibson-Mount wrote:
Ok. I had some wierd perf issues with Ubuntu10 and Ubuntu12. I was able to resolve them on Ubuntu12 by upgrading to a newer 3.x kernel (currently 3.10.11-031011-generic on my Ubuntu12 box).
Also, it isn't necessary to use ext4 (I use ext2 for the ldap data with mdb). ext4 in the 2.x kernel has some serious perf problems as well (when using writemap).
Yeah, what he said. :-) It's also worth compiling from source rather than just using whatever version of whatever binaries some distro provides. It's not hard and that way you have control over exactly what version of OpenLDAP you run regardless of distro.
BTW, I was using xfs for the OpenLDAP instance I sorta optimized for writes that I benchmarked recently. I think Howard and/or Quanah did a bunch of benchmarking of the MDB back-end on a whole bunch of filesystems and ext2 won. I use XFS simply cuz a modern version of XFS is decently fast, yields more usable space than ext4, and I really wanted a journaled filesystem. OpenLDAP on XFS was good 'nuff for my purposes, but it's prob'ly significantly faster on ext2 (from what I read).
Brent
Brent Bice wrote:
On 02/05/2014 11:45 AM, Quanah Gibson-Mount wrote:
Ok. I had some wierd perf issues with Ubuntu10 and Ubuntu12. I was able to resolve them on Ubuntu12 by upgrading to a newer 3.x kernel (currently 3.10.11-031011-generic on my Ubuntu12 box).
Also, it isn't necessary to use ext4 (I use ext2 for the ldap data with mdb). ext4 in the 2.x kernel has some serious perf problems as well (when using writemap).
Yeah, what he said. :-) It's also worth compiling from sourcerather than just using whatever version of whatever binaries some distro provides. It's not hard and that way you have control over exactly what version of OpenLDAP you run regardless of distro.
BTW, I was using xfs for the OpenLDAP instance I sorta optimized forwrites that I benchmarked recently. I think Howard and/or Quanah did a bunch of benchmarking of the MDB back-end on a whole bunch of filesystems and ext2 won. I use XFS simply cuz a modern version of XFS is decently fast, yields more usable space than ext4, and I really wanted a journaled filesystem. OpenLDAP on XFS was good 'nuff for my purposes, but it's prob'ly significantly faster on ext2 (from what I read).
Journaling may be good for other uses, but it's completely redundant for LMDB. If you have an entire partition to dedicate to the purpose, you're still better off just using ext2. xfs and some of these other filesystems can be faster when allocating new space (extent-based allocation) but if you manually preallocate the space for the DB file that's also moot.
With an ext2 filesystem, mounted with noatime, and preallocated space, LMDB can run with zero metadata update overhead. That makes all writes faster and makes it immune to filesystem corruption, thus rendering fsck irrelevant.
Andrew Eross eross@locatrix.com schrieb am 05.02.2014 um 16:30 in Nachricht
CAL_tfFf2qW5BcT=Xs4uFOSUO=wL0AN=9CyfS+D-XYPLitMZ_aw@mail.gmail.com:
Hi Quanah,
Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012 i686 GNU/Linux The latest OpenLDAP 2.4.39 All of those tests done with the mdb backend, of course, and the actual file system is ext4
Did you try btrfs? I'd guess it could be faster for massive random writes.
It's a fairly stock 10.04 system, no special config/kernel changes.
Cheers, Andrew
Andrew Eross CTO Locatrix Communications Office: +61 7 3123 1469 Mobile: +55 37 9858 9815 eross@locatrix.com
On Wed, Feb 5, 2014 at 1:26 PM, Quanah Gibson-Mount quanah@zimbra.comwrote:
--On Tuesday, February 04, 2014 6:52 PM -0200 Andrew Eross < eross@locatrix.com> wrote:
Thanks, Dieter, Quanah.
I've been doing some experimenting with those mdb options.
I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time:
Base-line, no extra options: 4m8sWith "writemap" enabled: 8m55s
With "writemap+mapasync" enabled: 5m12s With "dbnosync+checkpoint 0kb/1min": 0m14s
I know you answered some of this before, but please update with:
What kernel? What OpenLDAP version? What Ubuntu release? What filesystem for the LDAP DB?
Thanks,
Quanah
--
Quanah Gibson-Mount Architect - Server Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration
Hi guys,
Ulrich - thanks for the suggestions - btrfs in particular is certainly worth a shot.
Quanah - very cool to hear about the 12.04 kernel and ext2 suggestions. thanks!
I've just run some new tests on a similar machine with 12.04.4 LTS and a newly installed 3.11.0-15-generic x86_64 kernel.
Exact same testing method as before, 10K records, etc:
Running on an ext4 partition:
Base-line, no extra options: 5m14s With "writemap" enabled: 9m40s With "writemap+mapasync" enabled: 4m35s
Overall, about the same as 10.04 for me.
I created a new ext2 partition to give that a shot on the 12.04 box.
Base-line, no extra options: 1m31s With "writemap" enabled: 1m33s With "writemap+mapasync" enabled: 1m41s
Ahah! I'd say that's the killer answer.
Summary for future generations who may see this thread:
1) Using ext2 for your db directory (on Ubuntu at least) is waay faster than ext4 (~2-3x as fast according to my tests). This is the secret as far as I'm concerned since you can use this while still using the most conservative DB options that don't risk your data. 2) Using "dbnosync+checkpoint" with mdb is the absolute fastest method, but at the cost of risking data loss
Cheers, Andrew
On Thu, Feb 6, 2014 at 5:08 AM, Ulrich Windl < Ulrich.Windl@rz.uni-regensburg.de> wrote:
Andrew Eross eross@locatrix.com schrieb am 05.02.2014 um 16:30 in
Nachricht CAL_tfFf2qW5BcT=Xs4uFOSUO=wL0AN=9CyfS+D-XYPLitMZ_aw@mail.gmail.com:
Hi Quanah,
Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012
i686
GNU/Linux The latest OpenLDAP 2.4.39 All of those tests done with the mdb backend, of course, and the actual file system is ext4
Did you try btrfs? I'd guess it could be faster for massive random writes.
It's a fairly stock 10.04 system, no special config/kernel changes.
Cheers, Andrew
On Wed, Feb 5, 2014 at 1:26 PM, Quanah Gibson-Mount <quanah@zimbra.com wrote:
--On Tuesday, February 04, 2014 6:52 PM -0200 Andrew Eross < eross@locatrix.com> wrote:
Thanks, Dieter, Quanah.
I've been doing some experimenting with those mdb options.
I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time:
Base-line, no extra options: 4m8sWith "writemap" enabled: 8m55s
With "writemap+mapasync" enabled: 5m12s With "dbnosync+checkpoint 0kb/1min": 0m14s
I know you answered some of this before, but please update with:
What kernel? What OpenLDAP version? What Ubuntu release? What filesystem for the LDAP DB?
Thanks,
Quanah
--
Quanah Gibson-Mount Architect - Server Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration
Hi!
Another suggestion: Dependingon your secondary storage you could experiment with mount option barrier=0: If you have a RAID controller with battery backed-up cache (that guarantees that any data confirmed "written" to the host will acually be written in the end) it makes no sense to force the controller to flush the cache before the host may continue. Especially if you have big shared SAN storage system...
For plain local disks you are putting data integrity at risc, especially for journaled filesystems in you skip barrieres. Still if you plan a repeatable initial database load, you could disable barriers temporarily. In case of a system crash, you probably have to re-load all your data...
Ulrich
Andrew Eross eross@locatrix.com schrieb am 06.02.2014 um 16:59 in Nachricht
CAL_tfFdS4suKqn7ZFSkFGQ1rE7qEhsyyGOs0RqV_Ow71J_Ri8Q@mail.gmail.com:
Hi guys,
Ulrich - thanks for the suggestions - btrfs in particular is certainly worth a shot.
Quanah - very cool to hear about the 12.04 kernel and ext2 suggestions. thanks!
I've just run some new tests on a similar machine with 12.04.4 LTS and a newly installed 3.11.0-15-generic x86_64 kernel.
Exact same testing method as before, 10K records, etc:
Running on an ext4 partition:
Base-line, no extra options: 5m14s With "writemap" enabled: 9m40s With "writemap+mapasync" enabled: 4m35s
Overall, about the same as 10.04 for me.
I created a new ext2 partition to give that a shot on the 12.04 box.
Base-line, no extra options: 1m31s With "writemap" enabled: 1m33s With "writemap+mapasync" enabled: 1m41s
Ahah! I'd say that's the killer answer.
Summary for future generations who may see this thread:
- Using ext2 for your db directory (on Ubuntu at least) is waay faster
than ext4 (~2-3x as fast according to my tests). This is the secret as far as I'm concerned since you can use this while still using the most conservative DB options that don't risk your data. 2) Using "dbnosync+checkpoint" with mdb is the absolute fastest method, but at the cost of risking data loss
Cheers, Andrew
On Thu, Feb 6, 2014 at 5:08 AM, Ulrich Windl < Ulrich.Windl@rz.uni-regensburg.de> wrote:
Andrew Eross eross@locatrix.com schrieb am 05.02.2014 um 16:30 in
Nachricht CAL_tfFf2qW5BcT=Xs4uFOSUO=wL0AN=9CyfS+D-XYPLitMZ_aw@mail.gmail.com:
Hi Quanah,
Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012
i686
GNU/Linux The latest OpenLDAP 2.4.39 All of those tests done with the mdb backend, of course, and the actual file system is ext4
Did you try btrfs? I'd guess it could be faster for massive random writes.
It's a fairly stock 10.04 system, no special config/kernel changes.
Cheers, Andrew
On Wed, Feb 5, 2014 at 1:26 PM, Quanah Gibson-Mount <quanah@zimbra.com wrote:
--On Tuesday, February 04, 2014 6:52 PM -0200 Andrew Eross < eross@locatrix.com> wrote:
Thanks, Dieter, Quanah.
I've been doing some experimenting with those mdb options.
I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time:
Base-line, no extra options: 4m8sWith "writemap" enabled: 8m55s
With "writemap+mapasync" enabled: 5m12s With "dbnosync+checkpoint 0kb/1min": 0m14s
I know you answered some of this before, but please update with:
What kernel? What OpenLDAP version? What Ubuntu release? What filesystem for the LDAP DB?
Thanks,
Quanah
--
Quanah Gibson-Mount Architect - Server Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration
Thanks, Ulrich.
That made me do a bit searching, there's a nice summary of some of the ext4 performance options here: http://blog.smartlogicsolutions.com/2009/06/04/mount-options-to-improve-ext4...
Definitely cool ideas (but more academic for us specifically) - I'd be a bit nervous to use those options in our XenServer (with local disks) VM environment, when looking to only gain some %s in ext4 performance.
Andrew Eross CTO Locatrix Communications Office: +61 7 3123 1469 Mobile: +55 37 9858 9815 eross@locatrix.com
On Fri, Feb 7, 2014 at 5:21 AM, Ulrich Windl < Ulrich.Windl@rz.uni-regensburg.de> wrote:
Hi!
Another suggestion: Dependingon your secondary storage you could experiment with mount option barrier=0: If you have a RAID controller with battery backed-up cache (that guarantees that any data confirmed "written" to the host will acually be written in the end) it makes no sense to force the controller to flush the cache before the host may continue. Especially if you have big shared SAN storage system...
For plain local disks you are putting data integrity at risc, especially for journaled filesystems in you skip barrieres. Still if you plan a repeatable initial database load, you could disable barriers temporarily. In case of a system crash, you probably have to re-load all your data...
Ulrich
Andrew Eross eross@locatrix.com schrieb am 06.02.2014 um 16:59 in
Nachricht CAL_tfFdS4suKqn7ZFSkFGQ1rE7qEhsyyGOs0RqV_Ow71J_Ri8Q@mail.gmail.com:
Hi guys,
Ulrich - thanks for the suggestions - btrfs in particular is certainly worth a shot.
Quanah - very cool to hear about the 12.04 kernel and ext2 suggestions. thanks!
I've just run some new tests on a similar machine with 12.04.4 LTS and a newly installed 3.11.0-15-generic x86_64 kernel.
Exact same testing method as before, 10K records, etc:
Running on an ext4 partition:
Base-line, no extra options: 5m14s With "writemap" enabled: 9m40s With "writemap+mapasync" enabled: 4m35s
Overall, about the same as 10.04 for me.
I created a new ext2 partition to give that a shot on the 12.04 box.
Base-line, no extra options: 1m31s With "writemap" enabled: 1m33s With "writemap+mapasync" enabled: 1m41s
Ahah! I'd say that's the killer answer.
Summary for future generations who may see this thread:
- Using ext2 for your db directory (on Ubuntu at least) is waay faster
than ext4 (~2-3x as fast according to my tests). This is the secret as
far
as I'm concerned since you can use this while still using the most conservative DB options that don't risk your data. 2) Using "dbnosync+checkpoint" with mdb is the absolute fastest method,
but
at the cost of risking data loss
Cheers, Andrew
On Thu, Feb 6, 2014 at 5:08 AM, Ulrich Windl < Ulrich.Windl@rz.uni-regensburg.de> wrote:
Andrew Eross eross@locatrix.com schrieb am 05.02.2014 um 16:30 in
Nachricht CAL_tfFf2qW5BcT=Xs4uFOSUO=wL0AN=9CyfS+D-XYPLitMZ_aw@mail.gmail.com:
Hi Quanah,
Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012
i686
GNU/Linux The latest OpenLDAP 2.4.39 All of those tests done with the mdb backend, of course, and the
actual
file system is ext4
Did you try btrfs? I'd guess it could be faster for massive random
writes.
It's a fairly stock 10.04 system, no special config/kernel changes.
Cheers, Andrew
On Wed, Feb 5, 2014 at 1:26 PM, Quanah Gibson-Mount <
quanah@zimbra.com
wrote:
--On Tuesday, February 04, 2014 6:52 PM -0200 Andrew Eross < eross@locatrix.com> wrote:
Thanks, Dieter, Quanah.
I've been doing some experimenting with those mdb options.
I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time:
Base-line, no extra options: 4m8sWith "writemap" enabled: 8m55s
With "writemap+mapasync" enabled: 5m12s With "dbnosync+checkpoint 0kb/1min": 0m14s
I know you answered some of this before, but please update with:
What kernel? What OpenLDAP version? What Ubuntu release? What filesystem for the LDAP DB?
Thanks,
Quanah
--
Quanah Gibson-Mount Architect - Server Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration
openldap-technical@openldap.org
-
Andrew Eross -
Brent Bice -
Howard Chu -
Quanah Gibson-Mount -
Ulrich Windl