Hey folks,
As it stands, I've got an environment with six slapd servers (2.4.18) in an N-Way Multi-Master configuration. At any given moment, only one of these six nodes is receiving client requests (reads, writes, etc.). We use Pacemaker to provide some basic high-availability services, such that if the current 'master' (a term I'll use in this thread to refer to the one node receiving all reads/writes from clients) croaks, a shared IP/hostname resource is migrated to the next available server.
I would like to modify this setup such that ALL of these six nodes can receive client requests, but in the interest of preserving data consistency, I'd like to refer writes to just one of them (that one being the current 'master'), such that the rest receive the update(s) via syncrepl. From my own research and reading, it seems as though I'll need a combination of referrals (specifically, the 'updateref' directive) and chaining, with the latter being introduced so that the job of chasing the referrals is handled server-side instead of being offloaded to the clients. I have two questions about this setup, though:
1. In order to prevent an endless loop of referrals, I'm assuming that the terminal node (the 'master') should NOT have an updateref option that points to the 'master' shared IP/hostname (i.e., itself), like the rest of the nodes do. Is this correct, or will slapd realize that the referral target is itself, and answer authoritatively?
2. If the assumption in #1 is correct, what is the most advisable way to handle referrals in the situation where the identity of the terminal node is subject to change? My thought was to trigger an ldapmodify operation which effectively removes the olcUpdateRef option during the failover, but I'm open to better ideas if they exist.
Thanks!
-Ryan
I actually realized that my logic was slightly faulty - since each of the six masters is capable of performing the write operation, the referral will never be chased. Given that, is it impossible to have an N-Way MultiMaster configuration in which writes are possible on any node, but the writes at any given time are performed on only one of them to ensure consistency? That is, short of doing what I'm doing now (sending ALL traffic to only one master), which doesn't scale very well.
Ryan Steele wrote:
Hey folks,
As it stands, I've got an environment with six slapd servers (2.4.18) in an N-Way Multi-Master configuration. At any given moment, only one of these six nodes is receiving client requests (reads, writes, etc.). We use Pacemaker to provide some basic high-availability services, such that if the current 'master' (a term I'll use in this thread to refer to the one node receiving all reads/writes from clients) croaks, a shared IP/hostname resource is migrated to the next available server.
I would like to modify this setup such that ALL of these six nodes can receive client requests, but in the interest of preserving data consistency, I'd like to refer writes to just one of them (that one being the current 'master'), such that the rest receive the update(s) via syncrepl. From my own research and reading, it seems as though I'll need a combination of referrals (specifically, the 'updateref' directive) and chaining, with the latter being introduced so that the job of chasing the referrals is handled server-side instead of being offloaded to the clients. I have two questions about this setup, though:
- In order to prevent an endless loop of referrals, I'm assuming that the terminal node (the 'master') should NOT have
an updateref option that points to the 'master' shared IP/hostname (i.e., itself), like the rest of the nodes do. Is this correct, or will slapd realize that the referral target is itself, and answer authoritatively?
- If the assumption in #1 is correct, what is the most advisable way to handle referrals in the situation where the
identity of the terminal node is subject to change? My thought was to trigger an ldapmodify operation which effectively removes the olcUpdateRef option during the failover, but I'm open to better ideas if they exist.
Thanks!
-Ryan
On Wed, 24 Mar 2010, Ryan Steele wrote:
I actually realized that my logic was slightly faulty - since each of the six masters is capable of performing the write operation, the referral will never be chased. Given that, is it impossible to have an N-Way MultiMaster configuration in which writes are possible on any node, but the writes at any given time are performed on only one of them to ensure consistency? That is, short of doing what I'm doing now (sending ALL traffic to only one master), which doesn't scale very well.
Have you considered mirrormode (possibly combined with some back-ldap on the consumers so that you can present as only one always-able-to-write IP)?
With N-way multimaster you lose write performance as you add additional servers, as the write needs to be committed across an increasing N. More commits means more complexity means less performance. mirrormode keeps N=1, for good write performance, while still allowing for redundancy.
I'd probably set it up a little differently. Something like this (very general/generic):
2 of your 6 servers as suppliers/masters, accepting writes. If you are using a load balancer, set it up so that you have a vip that always goes to one of these servers (the "primary" supplier), and directs traffic to the other ("secondary" supplier) only when the primary in unavailable. This gives you redundancy for high availability, while minimizing risk of replication conflicts.
4 of your 6 servers as consumers. If you are using a load balancer, create a vip for your "read-only" ldap cluster, such that traffic is balanced across these servers. Referrals on these should go to the supplier vip (not directly to either "real" supplier). If your client apps chase referrals, you are fine. If not, you may need to set up chaining to pass writes to the supplier vip.
To scale read access, add more servers to the consumer cluster (scale horizontally). To scale your suppliers, get bigger boxes (Scale vertically - distributing writes across more supplier servers doesn't help with scaling, since replication kicks in and they all see all the write load anyway, but details of that have been discussed ad nauseum here).
Caveats: If you are using SSL/TLS, there are some complications with naming and making certs match up. If you don't use load balancers, you'll have to do some config/dns/etc tricks, maybe some manual or scripted failover, etc.
If you are wed to your current setup - you'll probably need to do something like: - Make only one server the supplier - Make the rest consumers - Monitor the supplier, and write a script that makes all the appropriate config changes to let you quickly promote a consumer to supplier and redirect the remaining consumers, but this can get tricky, I'm thinking.
-----Original Message----- From: openldap-software-bounces+jeff_clowser=fanniemae.com@OpenLDAP. org [mailto:openldap-software-bounces+jeff_clowser=fanniemae.com@O
penLDAP.org] On Behalf Of Ryan Steele
Sent: Wednesday, March 24, 2010 12:38 PM To: openldap-software@openldap.org Subject: Re: Multi-Master, referrals, and chaining
I actually realized that my logic was slightly faulty - since each of the six masters is capable of performing the write operation, the referral will never be chased. Given that, is it impossible to have an N-Way MultiMaster configuration in which writes are possible on any node, but the writes at any given time are performed on only one of them to ensure consistency? That is, short of doing what I'm doing now (sending ALL traffic to only one master), which doesn't scale very well.
Ryan Steele wrote:
Hey folks,
As it stands, I've got an environment with six slapd
servers (2.4.18) in an N-Way Multi-Master configuration. At any
given moment, only one of these six nodes is receiving
client requests (reads, writes, etc.). We use Pacemaker to
provide some basic high-availability services, such that if
the current 'master' (a term I'll use in this thread to
refer to the one node receiving all reads/writes from
clients) croaks, a shared IP/hostname resource is migrated to the
next available server.
I would like to modify this setup such that ALL of these
six nodes can receive client requests, but in the interest of
preserving data consistency, I'd like to refer writes to
just one of them (that one being the current 'master'), such
that the rest receive the update(s) via syncrepl. From my
own research and reading, it seems as though I'll need a
combination of referrals (specifically, the 'updateref'
directive) and chaining, with the latter being introduced so
that the job of chasing the referrals is handled
server-side instead of being offloaded to the clients. I have two
questions about this setup, though:
- In order to prevent an endless loop of referrals, I'm
assuming that the terminal node (the 'master') should NOT have
an updateref option that points to the 'master' shared
IP/hostname (i.e., itself), like the rest of the nodes do. Is
this correct, or will slapd realize that the referral
target is itself, and answer authoritatively?
- If the assumption in #1 is correct, what is the most
advisable way to handle referrals in the situation where the
identity of the terminal node is subject to change? My
thought was to trigger an ldapmodify operation which effectively
removes the olcUpdateRef option during the failover, but
I'm open to better ideas if they exist.
Thanks!
-Ryan
openldap-software@openldap.org
-
Aaron Richton -
Clowser, Jeff -
Ryan Steele