Folks,
I'm going through the documentation at http://www.openldap.org/doc/admin24/, the OpenLDAP FAQ-o-Matic at http://www.openldap.org/faq/data/cache/1.html, and the archives of the various Open-LDAP mailing lists, but I have not yet found anything that discusses how one might want to architect a large-scale OpenLDAP system with multiple masters, multiple slaves, etc... for best performance and low latency.
In our case, we have OpenLDAP 2.3.something (a few versions behind the official latest stable release), and we've recently hit our four millionth object (at a large University with something like 48,000 students, 2700 faculty, and 19,000 employees), and we're running into some performance issues that are going to keep us from rolling out some other large projects, at least until we can get the problems resolved.
I do not yet understand a great deal about how our existing OpenLDAP systems are designed, but I am curious to learn what kinds of recommendations you folks would have for a large scale system like this.
In the far, dark, distant past, I know that OpenLDAP did not handle situations well when you had both updates and reads occurring on the same system, so the recommendation at the time was to make all updates on the master server, then replicate that out to the slaves where all the read operations would occur. You could even go so far as to set up slaves on pretty much every single major client machine, for maximum distribution and replication of the data, and maximum scalability of the overall LDAP system.
I know that modern versions of OpenLDAP are able to handle a mix of both updates and reads much better, so that the old style architecture is not so necessary. But for a large-scale system like we have, would it not be wise to use the old-style architecture for maximum performance and scalability?
If you did use a multi-master cluster pair environment that handled all the updates and all the LDAP queries that were generated, what kind of performance do you think you should reasonably be able to get with the latest version of 2.4.whatever on high-end hardware, and what kind of hardware would you consider to be "high-end" for that environment? Is CPU more important, or RAM, or disk space/latency?
Alternatively, if you went to a three-level master(s)->proxies->slaves architecture [0], what kind of performance would you expect to be able to get, and how many machines would you expect that to be able to scale to? Are there any other major issues to be concerned about with that kind of architecture, like latency of updates getting pushed out to the leaf-node slaves?
How about the ultimate maximum distribution scenario, where you put an LDAP slave on virtually every major LDAP client machine?
Any and all advice you can provide would be appreciated, and in particular I would greatly appreciate it if you can provide any references to documentation, FAQs, mailing list archives where I can read more.
Thanks!
[0] Is this test045? As I believe is mentioned at http://www.openldap.org/lists/openldap-software/200707/msg00320.html?
--On Thursday, January 24, 2008 12:12 PM -0600 Brad Knowles b.knowles@its.utexas.edu wrote:
I do not yet understand a great deal about how our existing OpenLDAP systems are designed, but I am curious to learn what kinds of recommendations you folks would have for a large scale system like this.
This is generally good information to know...
But basically, have you read over the information on understanding your system requirements? I.e., how to properly tune DB_CONFIG and slapd.conf?
In the far, dark, distant past, I know that OpenLDAP did not handle situations well when you had both updates and reads occurring on the same system, so the recommendation at the time was to make all updates on the master server, then replicate that out to the slaves where all the read operations would occur. You could even go so far as to set up slaves on pretty much every single major client machine, for maximum distribution and replication of the data, and maximum scalability of the overall LDAP system.
Updates -> master is always recommended. You can set up multi-master with 2.4, but it will be slower than a single master scenario. The general best practice for fail over is to have a primary master that receives writes, and a secondary master that is getting the updates, and will take over via fail-over mechanisms if the primary goes down, becoming the new primary.
If you did use a multi-master cluster pair environment that handled all the updates and all the LDAP queries that were generated, what kind of performance do you think you should reasonably be able to get with the latest version of 2.4.whatever on high-end hardware, and what kind of hardware would you consider to be "high-end" for that environment? Is CPU more important, or RAM, or disk space/latency?
RAM is probably the most important, but you also will want fast disks, proper partitioning of the logs separate from the database and logs, and I recommend a non-journaling filesystem. 2 or more cores is also useful. Unfortunately I don't really see enough information from your end (yet) to really say much beyond that.
Alternatively, if you went to a three-level master(s)->proxies->slaves architecture [0], what kind of performance would you expect to be able to get, and how many machines would you expect that to be able to scale to? Are there any other major issues to be concerned about with that kind of architecture, like latency of updates getting pushed out to the leaf-node slaves?
On the SunFire x4100 servers I used to have, I could easily obtain some 23,000+ reads/second with OpenLDAP 2.3 on a single server.
How about the ultimate maximum distribution scenario, where you put an LDAP slave on virtually every major LDAP client machine?
Seems like major overkill to me, unless you are getting hundreds of thousands of reads/second.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Brad Knowles wrote:
Folks,
I'm going through the documentation at http://www.openldap.org/doc/admin24/, the OpenLDAP FAQ-o-Matic at http://www.openldap.org/faq/data/cache/1.html, and the archives of the various Open-LDAP mailing lists, but I have not yet found anything that discusses how one might want to architect a large-scale OpenLDAP system with multiple masters, multiple slaves, etc... for best performance and low latency.
In our case, we have OpenLDAP 2.3.something (a few versions behind the official latest stable release), and we've recently hit our four millionth object (at a large University with something like 48,000 students, 2700 faculty, and 19,000 employees), and we're running into some performance issues that are going to keep us from rolling out some other large projects, at least until we can get the problems resolved.
In my experience, 4 million objects (at around 3KB per entry) is near the limit of what will fit into 16GB of RAM. Sounds like you need a server with more than 16GB if you want to keep growing and not be waiting on disks.
I do not yet understand a great deal about how our existing OpenLDAP systems are designed, but I am curious to learn what kinds of recommendations you folks would have for a large scale system like this.
In the far, dark, distant past, I know that OpenLDAP did not handle situations well when you had both updates and reads occurring on the same system, so the recommendation at the time was to make all updates on the master server, then replicate that out to the slaves where all the read operations would occur. You could even go so far as to set up slaves on pretty much every single major client machine, for maximum distribution and replication of the data, and maximum scalability of the overall LDAP system.
The single-master constraints on OpenLDAP were never about performance. Even with OpenLDAP 2.2 the concurrent read/write rates for back-bdb are faster than any other directory server. It's always been about data consistency, and the fact that it's so easy to lose it in a multi-master setup.
I know that modern versions of OpenLDAP are able to handle a mix of both updates and reads much better, so that the old style architecture is not so necessary. But for a large-scale system like we have, would it not be wise to use the old-style architecture for maximum performance and scalability?
If you did use a multi-master cluster pair environment that handled all the updates and all the LDAP queries that were generated, what kind of performance do you think you should reasonably be able to get with the latest version of 2.4.whatever on high-end hardware,
You've been brainwashed by all the marketing lies other LDAP vendors tell about multi-master replication. Multi-master has no relation to performance. It's only about fault tolerance and high availability. No matter whether you choose a single-master or a multi-master setup, with the same number of machines, the same number of writes must be propagated to all servers, so the overall performance will be the same.
and what kind of hardware would you consider to be "high-end" for that environment?
That's a pointless question. The right question is - how fast do you need it to be? What load are you experiencing now, what constitutes a noticeable delay, and how often do you see those?
Is CPU more important, or RAM, or disk space/latency?
If you have enough RAM, disk latency shouldn't be a problem. Disk space is so cheap today that it should never be a problem. CPU, well, that depends on your performance target.
Alternatively, if you went to a three-level master(s)->proxies->slaves architecture [0], what kind of performance would you expect to be able to get, and how many machines would you expect that to be able to scale to? Are there any other major issues to be concerned about with that kind of architecture, like latency of updates getting pushed out to the leaf-node slaves?
Yes.
How about the ultimate maximum distribution scenario, where you put an LDAP slave on virtually every major LDAP client machine?
Generally I like the idea of having compact/simple slapd configs spread all over. With the old slapd.conf that would have been rather painful to administer though. Also in general, more moving parts means more things that can break.
Any and all advice you can provide would be appreciated, and in particular I would greatly appreciate it if you can provide any references to documentation, FAQs, mailing list archives where I can read more.
On Thu, 2008-01-24 at 14:50 -0800, Quanah Gibson-Mount wrote:
But basically, have you read over the information on understanding your system requirements? I.e., how to properly tune DB_CONFIG and slapd.conf?
I've read the OpenLDAP performance tuning stuff at http://www.openldap.org/doc/admin24/tuning.html#Performance%20Factors, but I do not yet have access to the boxes in question so I can't say anything about the specifics of the configuration, etc....
Updates -> master is always recommended. You can set up multi-master with 2.4, but it will be slower than a single master scenario. The general best practice for fail over is to have a primary master that receives writes, and a secondary master that is getting the updates, and will take over via fail-over mechanisms if the primary goes down, becoming the new primary.
Good to know. Do you have any sense for the kinds of performance differences you'd typically see in a multi-master versus master/backup master scenario? If it's just a typical 10% performance hit, we might prefer to go with a multi-master configuration anyway (well, once we upgrade to 2.4.something), but if it's considerably larger then we might want to think again.
RAM is probably the most important, but you also will want fast disks, proper partitioning of the logs separate from the database and logs, and I recommend a non-journaling filesystem. 2 or more cores is also useful. Unfortunately I don't really see enough information from your end (yet) to really say much beyond that.
Also good to know. I'm assuming they've already done at least most of this stuff, but I'll have to wait until I can get on the boxes and start looking around to be sure.
On the SunFire x4100 servers I used to have, I could easily obtain some 23,000+ reads/second with OpenLDAP 2.3 on a single server.
But that's reads only, right? Do you have any sense for what kind of performance you might see in a balanced 50/50 or even a write-heavy environment?
How about the ultimate maximum distribution scenario, where you put an LDAP slave on virtually every major LDAP client machine?
Seems like major overkill to me, unless you are getting hundreds of thousands of reads/second.
At this stage, I'm not making any assumptions. ;-)
--On Thursday, January 24, 2008 5:10 PM -0600 Brad Knowles b.knowles@its.utexas.edu wrote:
On the SunFire x4100 servers I used to have, I could easily obtain some 23,000+ reads/second with OpenLDAP 2.3 on a single server.
But that's reads only, right? Do you have any sense for what kind of performance you might see in a balanced 50/50 or even a write-heavy environment?
I'd use back-hdb for a case like that, but I haven't spent much time tracking that type of performance hit, as LDAP applications are usually READ intensive.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
<quote who="Quanah Gibson-Mount">
--On Thursday, January 24, 2008 12:12 PM -0600 Brad Knowles b.knowles@its.utexas.edu wrote:
I do not yet understand a great deal about how our existing OpenLDAP systems are designed, but I am curious to learn what kinds of recommendations you folks would have for a large scale system like this.
This is generally good information to know...
But basically, have you read over the information on understanding your system requirements? I.e., how to properly tune DB_CONFIG and slapd.conf?
In the far, dark, distant past, I know that OpenLDAP did not handle situations well when you had both updates and reads occurring on the same system, so the recommendation at the time was to make all updates on the master server, then replicate that out to the slaves where all the read operations would occur. You could even go so far as to set up slaves on pretty much every single major client machine, for maximum distribution and replication of the data, and maximum scalability of the overall LDAP system.
Updates -> master is always recommended. You can set up multi-master with 2.4, but it will be slower than a single master scenario. The general best practice for fail over is to have a primary master that receives writes, and a secondary master that is getting the updates, and will take over via fail-over mechanisms if the primary goes down, becoming the new primary.
If you did use a multi-master cluster pair environment that handled all the updates and all the LDAP queries that were generated, what kind of performance do you think you should reasonably be able to get with the latest version of 2.4.whatever on high-end hardware, and what kind of hardware would you consider to be "high-end" for that environment? Is CPU more important, or RAM, or disk space/latency?
RAM is probably the most important, but you also will want fast disks, proper partitioning of the logs separate from the database and logs, and I recommend a non-journaling filesystem. 2 or more cores is also useful. Unfortunately I don't really see enough information from your end (yet) to really say much beyond that.
Alternatively, if you went to a three-level master(s)->proxies->slaves architecture [0], what kind of performance would you expect to be able to get, and how many machines would you expect that to be able to scale to? Are there any other major issues to be concerned about with that kind of architecture, like latency of updates getting pushed out to the leaf-node slaves?
On the SunFire x4100 servers I used to have, I could easily obtain some 23,000+ reads/second with OpenLDAP 2.3 on a single server.
See:
http://www.connexitor.com/blog/pivot/entry.php?id=191#body http://www.openldap.org/doc/admin24/replication.html#MirrorMode
How about the ultimate maximum distribution scenario, where you put an LDAP slave on virtually every major LDAP client machine?
Seems like major overkill to me, unless you are getting hundreds of thousands of reads/second.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
On Thu, 2008-01-24 at 15:08 -0800, Howard Chu wrote:
In my experience, 4 million objects (at around 3KB per entry) is near the limit of what will fit into 16GB of RAM. Sounds like you need a server with more than 16GB if you want to keep growing and not be waiting on disks.
I was going through the caching discussion in section 19.4 at http://www.openldap.org/doc/admin24/tuning.html#Performance%20Factors, which talks about how much RAM to devote to what type of cache, but it had not occurred to me to try to just throw the whole thing (of this size) into RAM and be done with it.
The single-master constraints on OpenLDAP were never about performance. Even with OpenLDAP 2.2 the concurrent read/write rates for back-bdb are faster than any other directory server. It's always been about data consistency, and the fact that it's so easy to lose it in a multi-master setup.
I wasn't concerned so much about single-master versus multi-master. I was thinking more about the issue where a very read-intensive environment did not mix well with an environment where a lot of writes were also occurring directly to the system, but where bulk writes from slurpd (at the time) would be handled relatively efficiently while also handling high read intensity.
Thus the reason for ensuring that all updates went directly and only to the master, the master was replicated out to the slaves via slurpd, and the slaves only handled reads.
Of course, the modern architecture doesn't use slurpd, I was just wondering if it might make more sense from a scalability perspective to have a similar data flow architecture.
You've been brainwashed by all the marketing lies other LDAP vendors tell about multi-master replication. Multi-master has no relation to performance.
Again, I wasn't looking at single-master versus multi-master. If I gave that impression, I'm sorry.
It's only about fault tolerance and high availability. No matter whether you choose a single-master or a multi-master setup, with the same number of machines, the same number of writes must be propagated to all servers, so the overall performance will be the same.
I'm confused.
So there's no performance benefit to doing bulk writes via syncrepl to the slaves as opposed to individual writes to the master(s) via ldapadd? Then why have syncrepl at all and instead just have everything handled by ldapadd?
I understand the consistency argument for single-master versus multi-master, I'm just trying to find a way to partition the problem space for performance reasons, in addition to any consistency reasons.
That's a pointless question. The right question is - how fast do you need it to be? What load are you experiencing now, what constitutes a noticeable delay, and how often do you see those?
Good questions, but I'm not sure I've got the answers. I know that our OpenLDAP directory system is going to be used as a critical component of a campus-wide authentication system, and the target for the authentication system is to handle at least hundreds of authentications per second. Problem is, I don't know what that translates to in terms of numbers of OpenLDAP operations per second, and what the mix of reads are relative to the writes.
And the authentication system is just one of the many consumers of data from the OpenLDAP system.
So, at the very least, I would be surprised if the OpenLDAP system didn't have to handle at least thousands of read operations per second, and at peak it may also have to handle thousands of write operations per second -- a single student might potentially have dozens or a hundred or more data elements to be written or updated, and then they may be registering for a half-dozen classes or more at once, and each class might have hundreds of data elements that might also need to be updated. The domino effect of a single high-level entity being added or modified could potentially result in hundreds or thousands of smaller operations that need to be done as a result.
But right now, I'm just guessing.
I haven't actually seen the systems yet, and I don't know what the schemas look like, so I can only speak from my limited experience with OpenLDAP in the past and places where relatively simple uses could result in dozens of data elements for a single entity, and I don't know how OpenLDAP handles those kinds of things internally.
Is CPU more important, or RAM, or disk space/latency?
If you have enough RAM, disk latency shouldn't be a problem. Disk space is so cheap today that it should never be a problem. CPU, well, that depends on your performance target.
I'm not so worried about disk space per se. I would be more concerned about disk latency and throughput being potential bottlenecks.
Generally I like the idea of having compact/simple slapd configs spread all over. With the old slapd.conf that would have been rather painful to administer though. Also in general, more moving parts means more things that can break.
Much appreciated. Thanks!
Quanah Gibson-Mount wrote:
RAM is probably the most important, but you also will want fast disks, proper partitioning of the logs separate from the database and logs, and I recommend a non-journaling filesystem. 2 or more cores is also useful. Unfortunately I don't really see enough information from your end (yet) to really say much beyond that.
A non-journaling filesystem? Really? Can you explain why you would make that performance trade-off?
Keep in mind that there are two journaling methodologies that are prevalent: 1: meta-data only journaling (most common) 2: full-block journaling, or data journaling (available in several filesystems, but usually defaulted OFF)
I can understand being reticent about #2 if (and only if) you are in a high-write scenario, which doesn't seem to be clear from the original post.. if the LDAP server is a high-read, low-write server (as most are) then either type of journaling should be of negligible performance impact.
Even in a high-write scenario, method #1 (meta-data only) journaling should be of negligible impact.
Since the original poster mentions high end hardware, I'm going to assume Solaris is the platform.. If so, he'll likely be choosing between either UFS or ZFS..
In the case of UFS, logging (meta-data only journaling) is on by default in Solaris 10. Performance impact here should be negligible, even in a high-write scenario.
In the case of ZFS, logging is mandatory and cannot be disabled or circumvented. This is due to the nature of the filesystem -- it employs a copy-on-write strategy for ALL blocks, all of the time. This is the equivelant to full-block journaling, and I'm not sure how much it would impact the performance of a high-write scenario. I haven't used ZFS much in the real world yet.
If we're on Linux, then our choice of filesystem is quite a bit more varied, but most of the logging filesystem choices on Linux have both meta-data only and full-block journaling options, with meta-data only being the default.
Let me know if I'm missing something here -- I'm not intimately familiar with bdb's storage mechanisms.. I suppose if it created and deleted a small file per transaction or something similar, then you MAY see a degredation from meta-data only journaling. As a system administrator though, I'd be inclined to find performance elsewhere... pry the journaling filesystem out of my cold dead fingers ;-)
Cheers,
Joseph Dickson -- Sr. UNIX Admin joseph.dickson@meritain.com | 800.748.0003 ext 2151
--On Thursday, January 24, 2008 12:12 PM -0600 Brad Knowles b.knowles@its.utexas.edu wrote:
I do not yet understand a great deal about how our existing OpenLDAP systems are designed, but I am curious to learn what kinds of recommendations you folks would have for a large scale system like this.
This is generally good information to know...
But basically, have you read over the information on understanding your system requirements? I.e., how to properly tune DB_CONFIG and slapd.conf?
In the far, dark, distant past, I know that OpenLDAP did not handle situations well when you had both updates and reads occurring on the same system, so the recommendation at the time was to make all updates on the master server, then replicate that out to the slaves where all the read operations would occur. You could even go so far as to set up slaves on pretty much every single major client machine, for maximum distribution and replication of the data, and maximum scalability of the overall LDAP system.
Updates -> master is always recommended. You can set up multi-master with 2.4, but it will be slower than a single master scenario. The general best practice for fail over is to have a primary master that receives writes, and a secondary master that is getting the updates, and will take over via fail-over mechanisms if the primary goes down, becoming the new primary.
If you did use a multi-master cluster pair environment that handled all the updates and all the LDAP queries that were generated, what kind of performance do you think you should reasonably be able to get with the latest version of 2.4.whatever on high-end hardware, and what kind of hardware would you consider to be "high-end" for that environment? Is CPU more important, or RAM, or disk space/latency?
--On Thursday, January 24, 2008 5:35 PM -0600 Brad Knowles b.knowles@its.utexas.edu wrote:
I'm confused.
So there's no performance benefit to doing bulk writes via syncrepl to the slaves as opposed to individual writes to the master(s) via ldapadd? Then why have syncrepl at all and instead just have everything handled by ldapadd?
Yes, you are. :)
Syncrepl is a replication mechanism to get changes made on the master to the slaves. Howard's point is, that it doesn't matter *how* you replicate to the slaves, every slave will be having the same amount of write load as the master, as all writes that succeed at the master will also be propagated to the slave. There is no "differing" write load on the master vs the slave (except that syncrepl is actually somewhat less efficient than say delta-syncrepl, so it's write load is actually higher than what the master gets in reality, which is why I use delta-syncrepl rather than plain syncrepl).
Now, for *bulk loading* the master on the initial data load, "slapadd" is significantly faster than "ldapadd", but that has nothing to do with replication.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Brad Knowles wrote:
On Thu, 2008-01-24 at 15:08 -0800, Howard Chu wrote:
The single-master constraints on OpenLDAP were never about performance. Even with OpenLDAP 2.2 the concurrent read/write rates for back-bdb are faster than any other directory server. It's always been about data consistency, and the fact that it's so easy to lose it in a multi-master setup.
I wasn't concerned so much about single-master versus multi-master. I was thinking more about the issue where a very read-intensive environment did not mix well with an environment where a lot of writes were also occurring directly to the system, but where bulk writes from slurpd (at the time) would be handled relatively efficiently while also handling high read intensity.
You seem to believe there was something magic about how slurpd propagated updates to slaves. There wasn't; slurpd was just a fairly stupid LDAP client that issued the same types of operations as any other client.
Thus the reason for ensuring that all updates went directly and only to the master, the master was replicated out to the slaves via slurpd, and the slaves only handled reads.
No, that wasn't the reason.
Of course, the modern architecture doesn't use slurpd, I was just wondering if it might make more sense from a scalability perspective to have a similar data flow architecture.
Not really. If you have a dedicated master that never has to answer any clients' read requests, you can get a slight performance boost for writes by disabling most indexing. The slaves will still have to index any writes they receive though, to provide good search performance, so on equal hardware the slaves will generally accept updates more slowly than they can arrive on the master.
You've been brainwashed by all the marketing lies other LDAP vendors tell about multi-master replication. Multi-master has no relation to performance.
Again, I wasn't looking at single-master versus multi-master. If I gave that impression, I'm sorry.
It's only about fault tolerance and high availability. No matter whether you choose a single-master or a multi-master setup, with the same number of machines, the same number of writes must be propagated to all servers, so the overall performance will be the same.
I'm confused.
So there's no performance benefit to doing bulk writes via syncrepl to the slaves as opposed to individual writes to the master(s) via ldapadd?
There's no such thing as "bulk writes" in LDAP. Some vendors may have implemented their own proprietary bulk update mechanisms, but that's not standard LDAP.
Then why have syncrepl at all and instead just have everything handled by ldapadd?
Huh?
I understand the consistency argument for single-master versus multi-master, I'm just trying to find a way to partition the problem space for performance reasons, in addition to any consistency reasons.
That's a pointless question. The right question is - how fast do you need it to be? What load are you experiencing now, what constitutes a noticeable delay, and how often do you see those?
Good questions, but I'm not sure I've got the answers. I know that our OpenLDAP directory system is going to be used as a critical component of a campus-wide authentication system, and the target for the authentication system is to handle at least hundreds of authentications per second. Problem is, I don't know what that translates to in terms of numbers of OpenLDAP operations per second, and what the mix of reads are relative to the writes.
Hundreds = trivial. When you get to *hundreds of thousands per second*, then you'll have a problem, because that will require multiples of gigabit network bandwidth. Even so, OpenLDAP can handle it.
--On Thursday, January 24, 2008 6:55 PM -0500 Joseph Dickson jdd@weyco.com wrote:
Quanah Gibson-Mount wrote:
RAM is probably the most important, but you also will want fast disks, proper partitioning of the logs separate from the database and logs, and I recommend a non-journaling filesystem. 2 or more cores is also useful. Unfortunately I don't really see enough information from your end (yet) to really say much beyond that.
A non-journaling filesystem? Really? Can you explain why you would make that performance trade-off?
One bit of data here:
http://www.openldap.org/lists/openldap-devel/200801/msg00019.html
I was specifically thinking of linux, where I found that ext3 filesystems were substantially slower than ext2fs when you are dealing with data being written out (checkpoints, etc). BDB is doing its own checkpointing via its log.* files, there's no need for the extra overhead the journaling filesystem adds.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Thu, 2008-01-24 at 18:55 -0500, Joseph Dickson wrote:
Since the original poster mentions high end hardware, I'm going to assume Solaris is the platform.. If so, he'll likely be choosing between either UFS or ZFS..
I believe that the current machines are Solaris, but there has previously been discussion of throwing those boxes out the door and going with Linux instead. One factor would be the synchronous versus asynchronous syslog factor, although an alternative would be to install a syslog replacement on Solaris that can handle asynchronous mode as well as synchronous. I'm not sure what the other factors are.
In the case of UFS, logging (meta-data only journaling) is on by default in Solaris 10. Performance impact here should be negligible, even in a high-write scenario.
In the case of ZFS, logging is mandatory and cannot be disabled or circumvented. This is due to the nature of the filesystem -- it employs a copy-on-write strategy for ALL blocks, all of the time. This is the equivelant to full-block journaling, and I'm not sure how much it would impact the performance of a high-write scenario. I haven't used ZFS much in the real world yet.
I don't know what kind of filesystem is currently being used, but I'm guessing it's probably UFS, although it could also be mounting the data via NFS. I'd seriously consider ZFS, if I thought there was a good chance that there might be a performance or reliability win.
Thanks!
--On Thursday, January 24, 2008 6:13 PM -0600 Brad Knowles b.knowles@its.utexas.edu wrote:
I don't know what kind of filesystem is currently being used, but I'm guessing it's probably UFS, although it could also be mounting the data via NFS. I'd seriously consider ZFS, if I thought there was a good chance that there might be a performance or reliability win.
Using NFS with OpenLDAP is not supported. So I hope not.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Thu, 2008-01-24 at 15:08 -0800, Howard Chu wrote:
In my experience, 4 million objects (at around 3KB per entry) is near the limit of what will fit into 16GB of RAM. Sounds like you need a server with more than 16GB if you want to keep growing and not be waiting on disks.
I've been out sick the last couple of workdays, but I did discover a new piece of information today -- the machines in question have 32GB of RAM and are using 20GB of cache for the databases, although I do not yet know how that's split between the BDB, the HDB, the IDL, and the slapd entry caches. Finding that out will be one of my next tasks.
I also don't know if the processes are individually limited to 2GB of RAM each (as mentioned at http://www.openldap.org/faq/data/cache/1076.html).
Okay, so performance-wise, there doesn't seem to be much advantage to the master/slave or master/proxy/slave system architecture -- that seems to be more of a directory integrity and synchronization thing.
I would imagine that partitioning the OpenLDAP database into multiple slices and putting different schemas onto different servers should help with performance, although you'd have to make sure that the appropriate clients are querying the appropriate servers for that slice of the data. Are there any other system architecture things I could/should be looking at with regards to performance?
Thanks!
--On Tuesday, January 29, 2008 10:55 AM -0600 Brad Knowles b.knowles@its.utexas.edu wrote:
Are there any other system architecture things I could/should be looking at with regards to performance?
I think the first thing you really need to do is get access to the systems and get a real look at their configurations. Then come asking what to see about performance. Everything else right now is like throwing darts in the dark at an unknown target somewhere in the distance.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-software@openldap.org
-
Brad Knowles -
Gavin Henry -
Howard Chu -
Joseph Dickson -
Quanah Gibson-Mount