Hello Everyone.
I was trying to implement uidNumber Attribute Auto-Incrementing Method and I read http://www.rexconsulting.net/ldap-protocol-uidNumber.html .
I specifically want to point to this line here.
Create a “uidNext” entry (objectClass: uidNext) at an specific location in the directory to store the incrementing value. Publish this location in your application programming guides as the well- known location for obtaining the next UID. Also publish this method as the required method to retrieve a next UID.
But I already know from http://www.openldap.org/doc/admin23/schema.html & http://www.zytrax.com/books/ldap/ch3/ that object classes are defined only. So the writer left out that they defined a schema and what name the called the schema. Maybe it isn't important. What is important is that they used object class "objectclass ( 1.3.6.1.4.1.19173.2.2.2.8" to define it, but I can't find the registration of the object identifier on https://www.ldap.com/ldap-oid- reference or https://www.iana.org/assignments/ldap-parameters/ldap-para meters.xhtml#ldap-parameters-3.
It makes perfect scene because it is a PRIVATE ENTERPRISE NUMBER. It would mean that anyone outside of Rex Consulting, Inc. https://www.iana .org/assignments/enterprise-numbers/enterprise-numbers would be using the wrong OID and that the specific object wouldn't be listed.
Under no circumstances should you hijack OID namespace!
- OpenLDAP Software 2.4 Administrator's Guide
That is a lot of data from a lot of different websites to string together that information. I have a good idea how to implement uidNumber, but I haven't seen it done and I can't do it CORRECT today because I would have to register for a Private Enterprise Number so I won't hijack an OID namespace and that would take up to 30 days and there is no documented contingency plan anywhere.
We are all familiar with the the LDAP call out articles that come out every year. All of the articles seem to come from a place of frustration. To be fair I think call out articles are a trend with databases.
Do you think existing documentation is kind of vague?
Hi,
You don't need a special object class or schema, you can use this:
dn: cn=user,ou=increment,dc=foo,dc=bar objectClass: top objectClass: account objectClass: posixAccount cn: user gidNumber: 99999 homeDirectory: /no/such/location uid: user uidNumber: 1000 description: Modify-increment user
And here is a python method to do the job for you:
def increment_uidnumber(l, base): """ Perform LDAP modify-increment operation on uidNumber tracking object """ modlist = [(ldap.MOD_INCREMENT, "uidNumber", "1")] l.modify_s("cn=user,{0}".format(base), modlist) r = l.search_s(base, ldap.SCOPE_SUBTREE, "(objectClass=posixAccount)", ['uidNumber']) next_number = r[0][1]['uidNumber'][0]
return next_number
Hope that helps, Mike
On Tue, Nov 14, 2017 at 9:13 PM, John Lewis jl@hyperbolicinnovation.com wrote:
Hello Everyone.
I was trying to implement uidNumber Attribute Auto-Incrementing Method and I read http://www.rexconsulting.net/ldap-protocol-uidNumber.html .
I specifically want to point to this line here.
Create a “uidNext” entry (objectClass: uidNext) at an specific location in the directory to store the incrementing value. Publish this location in your application programming guides as the well- known location for obtaining the next UID. Also publish this method as the required method to retrieve a next UID.
But I already know from http://www.openldap.org/doc/admin23/schema.html & http://www.zytrax.com/books/ldap/ch3/ that object classes are defined only. So the writer left out that they defined a schema and what name the called the schema. Maybe it isn't important. What is important is that they used object class "objectclass ( 1.3.6.1.4.1.19173.2.2.2.8" to define it, but I can't find the registration of the object identifier on https://www.ldap.com/ldap-oid- reference or https://www.iana.org/assignments/ldap-parameters/ldap-para meters.xhtml#ldap-parameters-3.
It makes perfect scene because it is a PRIVATE ENTERPRISE NUMBER. It would mean that anyone outside of Rex Consulting, Inc. https://www.iana .org/assignments/enterprise-numbers/enterprise-numbers would be using the wrong OID and that the specific object wouldn't be listed.
Under no circumstances should you hijack OID namespace!
- OpenLDAP Software 2.4 Administrator's Guide
That is a lot of data from a lot of different websites to string together that information. I have a good idea how to implement uidNumber, but I haven't seen it done and I can't do it CORRECT today because I would have to register for a Private Enterprise Number so I won't hijack an OID namespace and that would take up to 30 days and there is no documented contingency plan anywhere.
We are all familiar with the the LDAP call out articles that come out every year. All of the articles seem to come from a place of frustration. To be fair I think call out articles are a trend with databases.
Do you think existing documentation is kind of vague?
MJ J wrote:
You don't need a special object class or schema, you can use this: dn: cn=user,ou=increment,dc=foo,dc=bar objectClass: top objectClass: account objectClass: posixAccount
Ouch! Depending on the config of your LDAP server and client systems this is a visible user account with username 'user' and *changing* and *conflicting* uidNumber. This is bad design. No wonder if you're running into problems with nss-pam-ldapd (nslcd) or sssd later.
You have been warned.
And here is a python method to do the job for you:
def increment_uidnumber(l, base): """ Perform LDAP modify-increment operation on uidNumber tracking object """ modlist = [(ldap.MOD_INCREMENT, "uidNumber", "1")] l.modify_s("cn=user,{0}".format(base), modlist) r = l.search_s(base, ldap.SCOPE_SUBTREE, "(objectClass=posixAccount)", ['uidNumber']) next_number = r[0][1]['uidNumber'][0]
return next_number
Using MOD_INCREMENT is good but the above read-after-write is prone to race conditions. Since you're using Python you can look at file Demo/pyasn1/readentrycontrol.py to learn how to use the PreReadControl or PostReadControl class (see also RFC 4527) for incrementing *and* reading the new ID within a single atomic modify operation.
Also note that you should only do this on a single provider or "primary" MMR-enabled provider (ITS#8757 would be nice for that).
BTW: That's how it's done in Æ-DIR but with a special object class:
https://demo.ae-dir.com/web2ldap?ldapi://%2Fopt%2Fae-dir%2Frun%2Fslapd%2Flda...
One of the unsolved problems is that IDs consumed from the pool might not really be used by the client application possibly resulting in gaps in your ID space.
Ciao, Michael.
Client apps are not scoped to do subtree searches from the root of the directory where the autoincrement objects live, nor do the ACLs permit it, but you knew that already.
Duplicate a race condition using the above code and you shouldn't be using LDAP in the first place.
On Tue, Nov 14, 2017 at 10:26 PM, Michael Ströder michael@stroeder.com wrote:
MJ J wrote:
You don't need a special object class or schema, you can use this: dn: cn=user,ou=increment,dc=foo,dc=bar objectClass: top objectClass: account objectClass: posixAccount
Ouch! Depending on the config of your LDAP server and client systems this is a visible user account with username 'user' and *changing* and *conflicting* uidNumber. This is bad design. No wonder if you're running into problems with nss-pam-ldapd (nslcd) or sssd later.
You have been warned.
And here is a python method to do the job for you:
def increment_uidnumber(l, base): """ Perform LDAP modify-increment operation on uidNumber tracking object """ modlist = [(ldap.MOD_INCREMENT, "uidNumber", "1")] l.modify_s("cn=user,{0}".format(base), modlist) r = l.search_s(base, ldap.SCOPE_SUBTREE, "(objectClass=posixAccount)", ['uidNumber']) next_number = r[0][1]['uidNumber'][0]
return next_numberUsing MOD_INCREMENT is good but the above read-after-write is prone to race conditions. Since you're using Python you can look at file Demo/pyasn1/readentrycontrol.py to learn how to use the PreReadControl or PostReadControl class (see also RFC 4527) for incrementing *and* reading the new ID within a single atomic modify operation.
Also note that you should only do this on a single provider or "primary" MMR-enabled provider (ITS#8757 would be nice for that).
BTW: That's how it's done in Æ-DIR but with a special object class:
https://demo.ae-dir.com/web2ldap?ldapi://%2Fopt%2Fae-dir%2Frun%2Fslapd%2Flda...
One of the unsolved problems is that IDs consumed from the pool might not really be used by the client application possibly resulting in gaps in your ID space.
Ciao, Michael.
John Lewis wrote:
I was trying to implement uidNumber Attribute Auto-Incrementing Method and I read http://www.rexconsulting.net/ldap-protocol-uidNumber.html
This is 3rd-party documentation. Just a blog article, but not bad. => Take it with a grain of salt.
what name the called the schema. Maybe it isn't important. What is important is that they used object class "objectclass ( 1.3.6.1.4.1.19173.2.2.2.8" to define it, but I can't find the registration of the object identifier on https://www.ldap.com/ldap-oid- reference or https://www.iana.org/assignments/ldap-parameters/ldap-para meters.xhtml#ldap-parameters-3.
It makes perfect scene because it is a PRIVATE ENTERPRISE NUMBER. It would mean that anyone outside of Rex Consulting, Inc. https://www.iana .org/assignments/enterprise-numbers/enterprise-numbers would be using the wrong OID and that the specific object wouldn't be listed.
I have several branches in my OID arc (my private enterprise number). One of the branches is dedicated for public use e.g. in I-Ds.
So simply ask the author of the blog article whether he's ever reusing the above mentioned OID.
Another option is to use this object class from the Samba schema:
( 1.3.6.1.4.1.7165.2.2.7 NAME 'sambaUnixIdPool' DESC 'Pool for allocating UNIX uids/gids' SUP top AUXILIARY MUST ( uidNumber $ gidNumber ) )
Ciao, Michael.
MJ J wrote:
Client apps are not scoped to do subtree searches from the root of the directory where the autoincrement objects live, nor do the ACLs permit it, but you knew that already.
Good to hear it's alright in your deployment. But please add this extra note next time you give general advice to others.
Duplicate a race condition using the above code and you shouldn't be using LDAP in the first place.
Did you really read what I wrote? Did you look at RFC 4527?
Ciao, Michael.
On Tue, Nov 14, 2017 at 10:26 PM, Michael Ströder michael@stroeder.com wrote:
MJ J wrote:
You don't need a special object class or schema, you can use this: dn: cn=user,ou=increment,dc=foo,dc=bar objectClass: top objectClass: account objectClass: posixAccount
Ouch! Depending on the config of your LDAP server and client systems this is a visible user account with username 'user' and *changing* and *conflicting* uidNumber. This is bad design. No wonder if you're running into problems with nss-pam-ldapd (nslcd) or sssd later.
You have been warned.
And here is a python method to do the job for you:
def increment_uidnumber(l, base): """ Perform LDAP modify-increment operation on uidNumber tracking object """ modlist = [(ldap.MOD_INCREMENT, "uidNumber", "1")] l.modify_s("cn=user,{0}".format(base), modlist) r = l.search_s(base, ldap.SCOPE_SUBTREE, "(objectClass=posixAccount)", ['uidNumber']) next_number = r[0][1]['uidNumber'][0]
return next_numberUsing MOD_INCREMENT is good but the above read-after-write is prone to race conditions. Since you're using Python you can look at file Demo/pyasn1/readentrycontrol.py to learn how to use the PreReadControl or PostReadControl class (see also RFC 4527) for incrementing *and* reading the new ID within a single atomic modify operation.
Also note that you should only do this on a single provider or "primary" MMR-enabled provider (ITS#8757 would be nice for that).
BTW: That's how it's done in Æ-DIR but with a special object class:
https://demo.ae-dir.com/web2ldap?ldapi://%2Fopt%2Fae-dir%2Frun%2Fslapd%2Flda...
One of the unsolved problems is that IDs consumed from the pool might not really be used by the client application possibly resulting in gaps in your ID space.
Ciao, Michael.
MJ J mikedotjackson@gmail.com schrieb am 14.11.2017 um 20:36 in
Nachricht CANCEyfNNs0CC-KxBV4kS3v3QRLFROKsp4BueMQrhOzxgZ1drwQ@mail.gmail.com:
Hi,
You don't need a special object class or schema, you can use this:
dn: cn=user,ou=increment,dc=foo,dc=bar objectClass: top objectClass: account objectClass: posixAccount cn: user gidNumber: 99999 homeDirectory: /no/such/location uid: user uidNumber: 1000 description: Modify-increment user
And here is a python method to do the job for you:
def increment_uidnumber(l, base): """ Perform LDAP modify-increment operation on uidNumber tracking object
""" modlist = [(ldap.MOD_INCREMENT, "uidNumber", "1")] l.modify_s("cn=user,{0}".format(base), modlist) r = l.search_s(base, ldap.SCOPE_SUBTREE, "(objectClass=posixAccount)", ['uidNumber']) next_number = r[0][1]['uidNumber'][0]
return next_number
Out of curiosity: How does it play with concurrency on different multi-master nodes? Is there a chance to assign the same number multiple times?
Regards, Ulrich
Hope that helps, Mike
On Tue, Nov 14, 2017 at 9:13 PM, John Lewis jl@hyperbolicinnovation.com wrote:
Hello Everyone.
I was trying to implement uidNumber Attribute Auto-Incrementing Method and I read http://www.rexconsulting.net/ldap-protocol-uidNumber.html .
I specifically want to point to this line here.
Create a “uidNext” entry (objectClass: uidNext) at an specific location in the directory to store the incrementing value. Publish this location in your application programming guides as the well- known location for obtaining the next UID. Also publish this method as the required method to retrieve a next UID.
But I already know from http://www.openldap.org/doc/admin23/schema.html & http://www.zytrax.com/books/ldap/ch3/ that object classes are defined only. So the writer left out that they defined a schema and what name the called the schema. Maybe it isn't important. What is important is that they used object class "objectclass ( 1.3.6.1.4.1.19173.2.2.2.8" to define it, but I can't find the registration of the object identifier on https://www.ldap.com/ldap-oid- reference or https://www.iana.org/assignments/ldap-parameters/ldap-para meters.xhtml#ldap-parameters-3.
It makes perfect scene because it is a PRIVATE ENTERPRISE NUMBER. It would mean that anyone outside of Rex Consulting, Inc. https://www.iana .org/assignments/enterprise-numbers/enterprise-numbers would be using the wrong OID and that the specific object wouldn't be listed.
Under no circumstances should you hijack OID namespace!
- OpenLDAP Software 2.4 Administrator's Guide
That is a lot of data from a lot of different websites to string together that information. I have a good idea how to implement uidNumber, but I haven't seen it done and I can't do it CORRECT today because I would have to register for a Private Enterprise Number so I won't hijack an OID namespace and that would take up to 30 days and there is no documented contingency plan anywhere.
We are all familiar with the the LDAP call out articles that come out every year. All of the articles seem to come from a place of frustration. To be fair I think call out articles are a trend with databases.
Do you think existing documentation is kind of vague?
TLDR; in a split-brain situation, you could run into trouble. But this isn't the only place. Efffective systems monitoring is the key here.
Long answer;
If you are loading entries hard and fast with multi-threaded code, they aren't going to be posixAccount entries - they are going to be simpleSecurityObject or inetOrgPerson entries for application users. Small lists of posixAccount entries (up to 100 thousand or so) are added using a single-threaded bulk user loader, and large lists will be done using slapadd. No organisation on this planet is going to have more than 100 thousand posixAccount entries in production on the same database, with the possible exception of the US or Chinese governments - and they will be so slow at adding accounts that they never experience a collision ;-)
In the case of universities, probably the only situation where there is a possibility to encounter a race condition with regards to auto-incrementing UID numbers, the accounts will likely be numbered using the student ID number and batch-loaded anyway.
So, yes, you CAN screw it up if you really go out of your way to prove it is possible but the operational use cases are just not very likely going to put you into that situation. There is just not a use case that I have ever seen where you would need to do hard and fast multi-threaded addition of numbered accounts. You can do serial loading over-the-wire into an mdb backend about 1 million users in 5 minutes if you are just building up a db with different scenarios for load-testing purposes.
The solution I posted has been in production in a large, dynamic company for several years and never encountered a problem.
On Wed, Nov 15, 2017 at 9:09 AM, Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de wrote:
MJ J mikedotjackson@gmail.com schrieb am 14.11.2017 um 20:36 in
Nachricht CANCEyfNNs0CC-KxBV4kS3v3QRLFROKsp4BueMQrhOzxgZ1drwQ@mail.gmail.com:
Hi,
You don't need a special object class or schema, you can use this:
dn: cn=user,ou=increment,dc=foo,dc=bar objectClass: top objectClass: account objectClass: posixAccount cn: user gidNumber: 99999 homeDirectory: /no/such/location uid: user uidNumber: 1000 description: Modify-increment user
And here is a python method to do the job for you:
def increment_uidnumber(l, base): """ Perform LDAP modify-increment operation on uidNumber tracking object
""" modlist = [(ldap.MOD_INCREMENT, "uidNumber", "1")] l.modify_s("cn=user,{0}".format(base), modlist) r = l.search_s(base, ldap.SCOPE_SUBTREE, "(objectClass=posixAccount)", ['uidNumber']) next_number = r[0][1]['uidNumber'][0]
return next_numberOut of curiosity: How does it play with concurrency on different multi-master nodes? Is there a chance to assign the same number multiple times?
Regards, Ulrich
Hope that helps, Mike
On Tue, Nov 14, 2017 at 9:13 PM, John Lewis jl@hyperbolicinnovation.com wrote:
Hello Everyone.
I was trying to implement uidNumber Attribute Auto-Incrementing Method and I read http://www.rexconsulting.net/ldap-protocol-uidNumber.html .
I specifically want to point to this line here.
Create a “uidNext” entry (objectClass: uidNext) at an specific location in the directory to store the incrementing value. Publish this location in your application programming guides as the well- known location for obtaining the next UID. Also publish this method as the required method to retrieve a next UID.
But I already know from http://www.openldap.org/doc/admin23/schema.html & http://www.zytrax.com/books/ldap/ch3/ that object classes are defined only. So the writer left out that they defined a schema and what name the called the schema. Maybe it isn't important. What is important is that they used object class "objectclass ( 1.3.6.1.4.1.19173.2.2.2.8" to define it, but I can't find the registration of the object identifier on https://www.ldap.com/ldap-oid- reference or https://www.iana.org/assignments/ldap-parameters/ldap-para meters.xhtml#ldap-parameters-3.
It makes perfect scene because it is a PRIVATE ENTERPRISE NUMBER. It would mean that anyone outside of Rex Consulting, Inc. https://www.iana .org/assignments/enterprise-numbers/enterprise-numbers would be using the wrong OID and that the specific object wouldn't be listed.
Under no circumstances should you hijack OID namespace!
- OpenLDAP Software 2.4 Administrator's Guide
That is a lot of data from a lot of different websites to string together that information. I have a good idea how to implement uidNumber, but I haven't seen it done and I can't do it CORRECT today because I would have to register for a Private Enterprise Number so I won't hijack an OID namespace and that would take up to 30 days and there is no documented contingency plan anywhere.
We are all familiar with the the LDAP call out articles that come out every year. All of the articles seem to come from a place of frustration. To be fair I think call out articles are a trend with databases.
Do you think existing documentation is kind of vague?
MJ J wrote:
TLDR; in a split-brain situation, you could run into trouble. But this isn't the only place. Efffective systems monitoring is the key here.
Long answer; [..] The solution I posted has been in production in a large, dynamic company for several years and never encountered a problem.
Maybe it works for you. But I still don't understand why you post such a lengthy justification insisting on your MOD_INCREMENT / read-after-write approach with possible race condition even in a single master deployment while there are two proper solutions with just a few lines code more:
1. delete-by-value to provoke a conflict like the original poster mentioned by pointing to http://www.rexconsulting.net/ldap-protocol-uidNumber.html
2. MOD_INCREMENT with pre-read control
Of course none of the solutions work when hitting multiple providers hard in a MMR setup or in a split-brain situation. One has to choose a "primary" provider then. BTW: AFAIK with FreeIPA each provider has its own ID range to prevent that.
Ciao, Michael.
Sure, it can be improved to become invulnerable to the academically imaginative race conditions that are not going to happen in real life. That will go to the very bottom of my list of things to do now, thanks.
FreeIPA is a cool concept, too bad it's not scalable or multi-tenant capable.
On Wed, Nov 15, 2017 at 11:09 PM, Michael Ströder michael@stroeder.com wrote:
MJ J wrote:
TLDR; in a split-brain situation, you could run into trouble. But this isn't the only place. Efffective systems monitoring is the key here.
Long answer; [..] The solution I posted has been in production in a large, dynamic company for several years and never encountered a problem.
Maybe it works for you. But I still don't understand why you post such a lengthy justification insisting on your MOD_INCREMENT / read-after-write approach with possible race condition even in a single master deployment while there are two proper solutions with just a few lines code more:
- delete-by-value to provoke a conflict like the original poster
mentioned by pointing to http://www.rexconsulting.net/ldap-protocol-uidNumber.html
- MOD_INCREMENT with pre-read control
Of course none of the solutions work when hitting multiple providers hard in a MMR setup or in a split-brain situation. One has to choose a "primary" provider then. BTW: AFAIK with FreeIPA each provider has its own ID range to prevent that.
Ciao, Michael.
MJ J wrote:
Sure, it can be improved to become invulnerable to the academically imaginative race conditions that are not going to happen in real life. That will go to the very bottom of my list of things to do now, thanks.
Adding a couple of lines of Python code is such a low-hanging fruit, especially since there free examples out there. Would have been less time than writing the list postings.
FreeIPA is a cool concept, too bad it's not scalable or multi-tenant capable.
Æ-DIR aims to provide similar features while providing a much better level of isolation (need-to-know, least privilege). I'd not call it multi-tenant approach though because I'm super-cautious with that term. Full multi-tenancy can always be achieved by setting up new instance in a separate name space anyway. But that's not what people usually want.
Ciao, Michael.
On Thu, 2017-11-16 at 05:54 +0200, MJ J wrote:
Sure, it can be improved to become invulnerable to the academically imaginative race conditions that are not going to happen in real life. That will go to the very bottom of my list of things to do now, thanks.
FreeIPA is a cool concept, too bad it's not scalable or multi-tenant capable.
It's a lot more scalable depending on which features you enable/disable. It won't even be multi-tenant due to the design with gssapi/krb.
At the end of the day, the atomic UID/GID alloc in FreeIPA is from the DNA plugin from 389-ds-base (which you can multi-instance on a server or multi-tentant with many backends). We use a similar method to AD in that each master has a pool of ids to alloc from, and they can atomically request pools. This prevents the race issues you are describing here with openldap.
So that's an option for you, because those race conditions *do* and *will* happen, and it will be a bad day for you when they do.
Another option is an external IDM system that allocs the uid's and feeds them to your LDAP environment instead,
Full disclosure: I'm a core dev of 389 directory server, so that's why I'm speaking in this context. Not here to say bad about openldap or try to poach you, they are a great project, just want to offer objective insight from "the other (dark?) side". :)
On Wed, Nov 15, 2017 at 11:09 PM, Michael Ströder <michael@stroeder.c om> wrote:
MJ J wrote:
TLDR; in a split-brain situation, you could run into trouble. But this isn't the only place. Efffective systems monitoring is the key here.
Long answer; [..] The solution I posted has been in production in a large, dynamic company for several years and never encountered a problem.
Maybe it works for you. But I still don't understand why you post such a lengthy justification insisting on your MOD_INCREMENT / read-after- write approach with possible race condition even in a single master deployment while there are two proper solutions with just a few lines code more:
- delete-by-value to provoke a conflict like the original poster
mentioned by pointing to http://www.rexconsulting.net/ldap-protocol-uidNumber.html
- MOD_INCREMENT with pre-read control
Of course none of the solutions work when hitting multiple providers hard in a MMR setup or in a split-brain situation. One has to choose a "primary" provider then. BTW: AFAIK with FreeIPA each provider has its own ID range to prevent that.
Ciao, Michael.
You're right, except for the fact that deploying 2 lines of new code into production can still be a long process ;-) The phrase comes to mind: If it ain't broken, don't fix it.
Why is this mailing list constantly used as a vehicle for people pushing their own consulting services and products? That supports the OP's question - keep the documentation terse, the examples weak and focus on the consulting-ware business model.
On Thu, Nov 16, 2017 at 12:08 PM, Michael Ströder michael@stroeder.com wrote:
MJ J wrote:
Sure, it can be improved to become invulnerable to the academically imaginative race conditions that are not going to happen in real life. That will go to the very bottom of my list of things to do now, thanks.
Adding a couple of lines of Python code is such a low-hanging fruit, especially since there free examples out there. Would have been less time than writing the list postings.
FreeIPA is a cool concept, too bad it's not scalable or multi-tenant capable.
Æ-DIR aims to provide similar features while providing a much better level of isolation (need-to-know, least privilege). I'd not call it multi-tenant approach though because I'm super-cautious with that term. Full multi-tenancy can always be achieved by setting up new instance in a separate name space anyway. But that's not what people usually want.
Ciao, Michael.
No matter how you wrap poll() and select(), they will always be poll() and select() - you will always run loops around an ever increasing stack of file descriptors while doing I/O. BDB is always going to have the same old problems... That's what I'm talking about - sacrificing performance for platform portability (NSPR).
FreeIPA could be multi-tenant i.e.support top-level and subordinate kerberos realms if it supported a more sensible DIT layout. I know because I have built such a system (based on OpenLDAP) and deployed it internationally. Probably the best piece of code to come out of the project is bind-dyndb-ldap.
On Fri, Nov 17, 2017 at 4:49 AM, William Brown wibrown@redhat.com wrote:
On Thu, 2017-11-16 at 05:54 +0200, MJ J wrote:
Sure, it can be improved to become invulnerable to the academically imaginative race conditions that are not going to happen in real life. That will go to the very bottom of my list of things to do now, thanks.
FreeIPA is a cool concept, too bad it's not scalable or multi-tenant capable.
It's a lot more scalable depending on which features you enable/disable. It won't even be multi-tenant due to the design with gssapi/krb.
At the end of the day, the atomic UID/GID alloc in FreeIPA is from the DNA plugin from 389-ds-base (which you can multi-instance on a server or multi-tentant with many backends). We use a similar method to AD in that each master has a pool of ids to alloc from, and they can atomically request pools. This prevents the race issues you are describing here with openldap.
So that's an option for you, because those race conditions *do* and *will* happen, and it will be a bad day for you when they do.
Another option is an external IDM system that allocs the uid's and feeds them to your LDAP environment instead,
Full disclosure: I'm a core dev of 389 directory server, so that's why I'm speaking in this context. Not here to say bad about openldap or try to poach you, they are a great project, just want to offer objective insight from "the other (dark?) side". :)
On Wed, Nov 15, 2017 at 11:09 PM, Michael Ströder <michael@stroeder.c om> wrote:
MJ J wrote:
TLDR; in a split-brain situation, you could run into trouble. But this isn't the only place. Efffective systems monitoring is the key here.
Long answer; [..] The solution I posted has been in production in a large, dynamic company for several years and never encountered a problem.
Maybe it works for you. But I still don't understand why you post such a lengthy justification insisting on your MOD_INCREMENT / read-after- write approach with possible race condition even in a single master deployment while there are two proper solutions with just a few lines code more:
- delete-by-value to provoke a conflict like the original poster
mentioned by pointing to http://www.rexconsulting.net/ldap-protocol-uidNumber.html
- MOD_INCREMENT with pre-read control
Of course none of the solutions work when hitting multiple providers hard in a MMR setup or in a split-brain situation. One has to choose a "primary" provider then. BTW: AFAIK with FreeIPA each provider has its own ID range to prevent that.
Ciao, Michael.
-- Sincerely,
William Brown Software Engineer Red Hat, Australia/Brisbane
On Fri, 2017-11-17 at 08:27 +0200, MJ J wrote:
No matter how you wrap poll() and select(), they will always be poll() and select() - you will always run loops around an ever increasing stack of file descriptors while doing I/O. BDB is always going to have the same old problems... That's what I'm talking about - sacrificing performance for platform portability (NSPR).
FreeIPA could be multi-tenant i.e.support top-level and subordinate kerberos realms if it supported a more sensible DIT layout. I know because I have built such a system (based on OpenLDAP) and deployed it internationally. Probably the best piece of code to come out of the project is bind-dyndb-ldap.
Whoa mate - I'm not here to claim that 389 is a better ldap server - we just do some different things. We acknowledge our limitations and are really working on them and paying down our tech debt. We want to remove parts of nspr, replace bdb and more. :)
I'm here to follow the progress of the openldap project, who have a team of people I respect greatly and want to learn from, and here to help discussions and provide input from a different perspective.
There are things that today openldap does much better than us for certain - and there are also some things that we do differently too like DNA plugin uid allocation, replication etc,
There are also project focusses and decisions made to improve supportability in systems like FreeIPA - we can discuss them forever, but reality is today, FreeIPA is not targeting multi-tennant environments because the majority of our consumers don't want that functionality. We made a design decision and have to live with it. I'm providing this information to help give the ability for people to construct an informed opinion.
As mentioned, I'm not here to throw insults and criticisms, I'm here to have positive, respectful discussions about technology, to provide different ideas, and to learn from others :)
Thanks,
On Fri, Nov 17, 2017 at 4:49 AM, William Brown wibrown@redhat.com wrote:
On Thu, 2017-11-16 at 05:54 +0200, MJ J wrote:
Sure, it can be improved to become invulnerable to the academically imaginative race conditions that are not going to happen in real life. That will go to the very bottom of my list of things to do now, thanks.
FreeIPA is a cool concept, too bad it's not scalable or multi- tenant capable.
It's a lot more scalable depending on which features you enable/disable. It won't even be multi-tenant due to the design with gssapi/krb.
At the end of the day, the atomic UID/GID alloc in FreeIPA is from the DNA plugin from 389-ds-base (which you can multi-instance on a server or multi-tentant with many backends). We use a similar method to AD in that each master has a pool of ids to alloc from, and they can atomically request pools. This prevents the race issues you are describing here with openldap.
So that's an option for you, because those race conditions *do* and *will* happen, and it will be a bad day for you when they do.
Another option is an external IDM system that allocs the uid's and feeds them to your LDAP environment instead,
Full disclosure: I'm a core dev of 389 directory server, so that's why I'm speaking in this context. Not here to say bad about openldap or try to poach you, they are a great project, just want to offer objective insight from "the other (dark?) side". :)
On Wed, Nov 15, 2017 at 11:09 PM, Michael Ströder <michael@stroed er.c om> wrote:
MJ J wrote:
TLDR; in a split-brain situation, you could run into trouble. But this isn't the only place. Efffective systems monitoring is the key here.
Long answer; [..] The solution I posted has been in production in a large, dynamic company for several years and never encountered a problem.
Maybe it works for you. But I still don't understand why you post such a lengthy justification insisting on your MOD_INCREMENT / read- after- write approach with possible race condition even in a single master deployment while there are two proper solutions with just a few lines code more:
- delete-by-value to provoke a conflict like the original
poster mentioned by pointing to http://www.rexconsulting.net/ldap-protocol-uidNumber.html
- MOD_INCREMENT with pre-read control
Of course none of the solutions work when hitting multiple providers hard in a MMR setup or in a split-brain situation. One has to choose a "primary" provider then. BTW: AFAIK with FreeIPA each provider has its own ID range to prevent that.
Ciao, Michael.
-- Sincerely,
William Brown Software Engineer Red Hat, Australia/Brisbane
MJ J wrote:
You're right, except for the fact that deploying 2 lines of new code into production can still be a long process ;-) The phrase comes to mind: If it ain't broken, don't fix it.
You're free to decide to ignore good advice.
But you have to accept that someone might point out flaws in your solution to prevent other list readers falling into the same trap. Your emotional reactions on-list and off-list are completely inappropriate.
With security hat on: The sum of such loose ends make out the attack surface.
Personally even after almost 20 years with LDAP I'm a dwarf standing on the shoulder of giants. And I'm still learning.
In particular in the context of this discussion I'm happy that others wrote down protocol specs for improving robustness, e.g. RFC 4527 etc. Especially since I already had to track down read-after-write issues in replicated deployments with so-called enterprise software - which can take days.
Ciao, Michael.
I actually like 389 a lot and I have used Netscape DS extensively in managing international telecom networks about 15 years ago. There are quite many management features that are superior to OpenLDAP still to this day, but I simply cannot use it anymore because of the lack of scalability. I know the original Netscape DS devs quite extensively...
-mike
On Fri, Nov 17, 2017 at 8:34 AM, William Brown wibrown@redhat.com wrote:
On Fri, 2017-11-17 at 08:27 +0200, MJ J wrote:
No matter how you wrap poll() and select(), they will always be poll() and select() - you will always run loops around an ever increasing stack of file descriptors while doing I/O. BDB is always going to have the same old problems... That's what I'm talking about - sacrificing performance for platform portability (NSPR).
FreeIPA could be multi-tenant i.e.support top-level and subordinate kerberos realms if it supported a more sensible DIT layout. I know because I have built such a system (based on OpenLDAP) and deployed it internationally. Probably the best piece of code to come out of the project is bind-dyndb-ldap.
Whoa mate - I'm not here to claim that 389 is a better ldap server - we just do some different things. We acknowledge our limitations and are really working on them and paying down our tech debt. We want to remove parts of nspr, replace bdb and more. :)
I'm here to follow the progress of the openldap project, who have a team of people I respect greatly and want to learn from, and here to help discussions and provide input from a different perspective.
There are things that today openldap does much better than us for certain - and there are also some things that we do differently too like DNA plugin uid allocation, replication etc,
There are also project focusses and decisions made to improve supportability in systems like FreeIPA - we can discuss them forever, but reality is today, FreeIPA is not targeting multi-tennant environments because the majority of our consumers don't want that functionality. We made a design decision and have to live with it. I'm providing this information to help give the ability for people to construct an informed opinion.
As mentioned, I'm not here to throw insults and criticisms, I'm here to have positive, respectful discussions about technology, to provide different ideas, and to learn from others :)
Thanks,
On Fri, Nov 17, 2017 at 4:49 AM, William Brown wibrown@redhat.com wrote:
On Thu, 2017-11-16 at 05:54 +0200, MJ J wrote:
Sure, it can be improved to become invulnerable to the academically imaginative race conditions that are not going to happen in real life. That will go to the very bottom of my list of things to do now, thanks.
FreeIPA is a cool concept, too bad it's not scalable or multi- tenant capable.
It's a lot more scalable depending on which features you enable/disable. It won't even be multi-tenant due to the design with gssapi/krb.
At the end of the day, the atomic UID/GID alloc in FreeIPA is from the DNA plugin from 389-ds-base (which you can multi-instance on a server or multi-tentant with many backends). We use a similar method to AD in that each master has a pool of ids to alloc from, and they can atomically request pools. This prevents the race issues you are describing here with openldap.
So that's an option for you, because those race conditions *do* and *will* happen, and it will be a bad day for you when they do.
Another option is an external IDM system that allocs the uid's and feeds them to your LDAP environment instead,
Full disclosure: I'm a core dev of 389 directory server, so that's why I'm speaking in this context. Not here to say bad about openldap or try to poach you, they are a great project, just want to offer objective insight from "the other (dark?) side". :)
On Wed, Nov 15, 2017 at 11:09 PM, Michael Ströder <michael@stroed er.c om> wrote:
MJ J wrote:
TLDR; in a split-brain situation, you could run into trouble. But this isn't the only place. Efffective systems monitoring is the key here.
Long answer; [..] The solution I posted has been in production in a large, dynamic company for several years and never encountered a problem.
Maybe it works for you. But I still don't understand why you post such a lengthy justification insisting on your MOD_INCREMENT / read- after- write approach with possible race condition even in a single master deployment while there are two proper solutions with just a few lines code more:
- delete-by-value to provoke a conflict like the original
poster mentioned by pointing to http://www.rexconsulting.net/ldap-protocol-uidNumber.html
- MOD_INCREMENT with pre-read control
Of course none of the solutions work when hitting multiple providers hard in a MMR setup or in a split-brain situation. One has to choose a "primary" provider then. BTW: AFAIK with FreeIPA each provider has its own ID range to prevent that.
Ciao, Michael.
-- Sincerely,
William Brown Software Engineer Red Hat, Australia/Brisbane
-- Sincerely,
William Brown Software Engineer Red Hat, Australia/Brisbane
I had a requirement to build a centrally managed SSO system that replicated subordinate subtrees (kerberos, identities, roles, permissions, resources, dns, etc) to the respective sites and handle tens of thousands of concurrent requests per second. I determined that FreeIPA was unable to perform this mission due to 1) inflexible DIT, 2) inflexible management tooling, and 3) lack of scalability. So, I built a system to achieve those goals. And it really wasn't rocket science. Perhaps I will write a book about it and ask for competent reviewers from the OL community.
On Fri, Nov 17, 2017 at 4:21 PM, Michael Ströder michael@stroeder.com wrote:
MJ J wrote:
I know because I have built such a system (based on OpenLDAP) and deployed it internationally.
So what makes your system special, which goals does it reach and how?
Ciao, Michael.
There are no emotional reactions - there are simply statements that I won't be submitting to your condescending attitude. I have also been working in this same arena for 20 years and I have long ago found what is need to make large systems function. Perhaps if you would drop the zero-sum-game attitude wrt to your own consulting business and agree to see other points of view, we would advance the state of the art.
But I won't be asking you for approval as to which advice I can give. And that is basically the sum of what I told you personally as well.
-mike
On Fri, Nov 17, 2017 at 9:20 AM, Michael Ströder michael@stroeder.com wrote:
MJ J wrote:
You're right, except for the fact that deploying 2 lines of new code into production can still be a long process ;-) The phrase comes to mind: If it ain't broken, don't fix it.
You're free to decide to ignore good advice.
But you have to accept that someone might point out flaws in your solution to prevent other list readers falling into the same trap. Your emotional reactions on-list and off-list are completely inappropriate.
With security hat on: The sum of such loose ends make out the attack surface.
Personally even after almost 20 years with LDAP I'm a dwarf standing on the shoulder of giants. And I'm still learning.
In particular in the context of this discussion I'm happy that others wrote down protocol specs for improving robustness, e.g. RFC 4527 etc. Especially since I already had to track down read-after-write issues in replicated deployments with so-called enterprise software - which can take days.
Ciao, Michael.
openldap-technical@openldap.org
-
John Lewis -
Michael Ströder -
MJ J -
Ulrich Windl -
William Brown