Hi,
I work in a company that has 140,000 registered users in OpenLDAP. This OpenLDAP is used for authentication of our internal systems. In our tree of groups we have the systems and below the each system there are the groups' authorization (systems profiles). The user is bound in each group according to position, function and department in the company. When a user replaces another user hierarchically higher, this user is taken from the respective group (that he belonged) and registered in user_group with the highest hierarchy. This movement in the company is very common, and this is the cause of our problems. We have a group with 50,000 registered users, and when we need to delete a user of that group or add a new one, OpenLADP takes up to 6 minute to effect the transaction. We have a tool (BMC Identity Management (formerly Control-SA)) that automates the transactions, but due to delay in the transactions are with a row of 100,000 operations of insert / delete to perform. I wonder if you have any way to improve the performance of OpenLDAP for these write operations. The OpenLDAP version is 2.4.40.
Thanks,
Alessandro Lasmar Mourão
Below is our slapd.conf:
############################################## serverID 2 idletimeout 0 include /etc/ldap/schema/core.schema include /etc/ldap/schema/cosine.schema include /etc/ldap/schema/nis.schema include /etc/ldap/schema/inetorgperson.schema include /etc/ldap/schema/ldap.schema include /etc/ldap/schema/ppolicy.schema pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args loglevel 256 modulepath /usr/lib/ldap/ moduleload back_mdb moduleload back_monitor moduleload memberof moduleload ppolicy moduleload syncprov moduleload refint moduleload accesslog sizelimit 250 tool-threads 16 password-hash {SSHA} monitoring true TLSCACertificateFile /etc/ssl/certs/cacert.pem TLSCertificateFile /etc/ssl/certs/servercrt.pem TLSCertificateKeyFile /etc/ssl/certs/serverkey.pem backend mdb database config rootdn "cn=admin,cn=config" rootpw secret monitoring true database monitor rootdn "cn=admin,cn=monitor" rootpw secret monitoring true database mdb suffix "cn=accesslog" rootdn "cn=admin,cn=accesslog" rootpw secret maxsize 1073741824 monitoring true directory "/var/lib/ldap/intranet/log" index default eq,pres,sub index entryCSN eq,pres index objectClass,reqEnd eq,pres index reqResult,reqStart eq,pres limits dn.exact="uid=replication,ou=Users,o=company" size.soft=unlimited size.hard=unlimited time.soft=unlimited time.hard=unlimited overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE access to * by dn.base="uid=replication,ou=Users,o=company" read by * break database mdb suffix "o=company" rootdn "cn=admin,o=company" rootpw secret maxsize 4294967296 monitoring true overlay ppolicy ppolicy_use_lockout ppolicy_hash_cleartext ppolicy_default "cn=default,ou=policy,o=company" overlay memberof memberof-group-oc groupOfUniqueNames memberof-member-ad uniqueMember memberof-refint true overlay refint refint_attributes uniqueMember overlay accesslog logdb "cn=accesslog" logops writes logsuccess TRUE logpurge 07+00:00 01+00:00 limits dn.exact="uid=replication01,ou=Users,o=company" size.soft=unlimited size.hard=unlimited time.soft=unlimited time.hard=unlimited limits dn.exact="uid=replication02,ou=Users,o=company" size.soft=unlimited size.hard=unlimited time.soft=unlimited time.hard=unlimited limits dn.exact="uid=replication03,ou=Users,o=company" size.soft=unlimited size.hard=unlimited time.soft=unlimited time.hard=unlimited limits dn.exact="uid=replication04,ou=Users,o=company" size.soft=unlimited size.hard=unlimited time.soft=unlimited time.hard=unlimited overlay syncprov syncprov-checkpoint 1000 20 syncprov-sessionlog 10000 syncrepl rid=100 provider=ldap://10.192.184.195:389 searchbase="o=company" logbase="cn=accesslog" logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" type=refreshAndPersist retry="60 +" scope=sub schemachecking=on bindmethod=simple binddn="uid=replication01,ou=Users,o=company" credentials=secret mirrormode true directory "/var/lib/ldap/intranet" directory "/var/lib/ldap/intranet" index objectClass eq,pres index uniqueMember,memberof eq,pres index nu-cpf,nu-cnpj eq,pres index dt-nascimento pres index entryUUID,entryCSN eq,pres index uid,ou,cn,sn,mail eq,pres,sub index default,givenname eq,pres,sub lastmod on checkpoint 1024 10 access to attrs=userPassword,shadowLastChange by dn="cn=admin,o=company" write by dn.exact="uid=replica01,ou=Users,o=company" read by dn.exact="uid=replica02,ou=Users,o=company" read by dn.exact="uid=replica03,ou=Users,o=company" read by dn.exact="uid=replica04,ou=Users,o=company" read by anonymous auth by self write by * none access to * by dn="cn=admin,o=company" write by * read
Alessandro Lasmar Mourao wrote:
I work in a company that has 140,000 registered users in OpenLDAP. This OpenLDAP is used for authentication of our internal systems. In our tree of groups we have the systems and below the each system there are the groups' authorization (systems profiles). The user is bound in each group according to position, function and department in the company. When a user replaces another user hierarchically higher, this user is taken from the respective group (that he belonged) and registered in user_group with the highest hierarchy. This movement in the company is very common, and this is the cause of our problems. We have a group with 50,000 registered users, and when we need to delete a user of that group or add a new one, OpenLADP takes up to 6 minute to effect the transaction. We have a tool (BMC Identity Management (formerly Control-SA)) that automates the transactions, but due to delay in the transactions are with a row of 100,000 operations of insert / delete to perform. I wonder if you have any way to improve the performance of OpenLDAP for these write operations. The OpenLDAP version is 2.4.40.
Do you use the term "group" actually for a node in the tree? If yes, this sounds like a broken DIT design.
Also it seems your management client application is not able to leverage renaming whole trees with a single modrdn request (like support in back-hdb and back-mdb). Instead it moves user entries one by one. This is also waste of resources.
You should seriously consider a partial re-design and another management application.
Ciao, Michael.
On Thu, Jan 29, 2015 at 07:49:29PM +0000, Alessandro Lasmar Mourao wrote:
I work in a company that has 140,000 registered users in OpenLDAP. This OpenLDAP is used for authentication of our internal systems. In our tree of groups we have the systems and below the each system there are the groups' authorization (systems profiles). The user is bound in each group according to position, function and department in the company. When a user replaces another user hierarchically higher, this user is taken from the respective group (that he belonged) and registered in user_group with the highest hierarchy. This movement in the company is very common, and this is the cause of our problems. We have a group with 50,000 registered users, and when we need to delete a user of that group or add a new one, OpenLADP takes up to 6 minute to effect the transaction.
Assuming that these groups are the conventional 'single entry with many values of the member attribute' type, you should turn on the 'sortvals' global option:
sortvals <attr> [...] Specify a list of multi-valued attributes whose values will always be maintained in sorted order. Using this option will allow Modify, Compare, and filter evaluations on these attributes to be performed more efficiently. The resulting sort order depends on the attributes' syntax and matching rules and may not correspond to lexical order or any other recognizable order.
I cannot offhand remember whether you need to re-load the data for that to take effect.
We have a tool (BMC Identity Management (formerly Control-SA)) that automates the transactions, but due to delay in the transactions are with a row of 100,000 operations of insert / delete to perform.
Check how it modifies the groups. If it rewrites the whole set of 'member' attributes each time then you are forcing the server to re-index every value. The efficient way to do this is to specify the value that you want to remove or the one you want to add. It should be very quick.
Also look at your replication setup. With this sort of data you really do need delta mode.
Andrew
Andrew Findlay wrote:
We have a tool (BMC Identity Management (formerly Control-SA))
Check how it modifies the groups.
First of all it's not clear to me whether the original poster really meant group entries.
If it rewrites the whole set of 'member' attributes each time then you are forcing the server to re-index every value.
Not only that it's horribly inefficient. With this approach you run into the famous security issue back in those Windows 2000 days where removed group members were accidently re-added because of concurrent write access.
The efficient way to do this is to specify the value that you want to remove or the one you want to add. It should be very quick.
Additionally LDAP PDUs manipulating many values at once grow very big.
Also look at your replication setup. With this sort of data you really do need delta mode.
delta-syncrepl would only help if group membership is changed only for a few members in one modify request.
Ciao, Michael.
It's really like Andrew understood! When I used the term group was to specify a node in the tree (objectclass type groupOfUniqueNames). Group members are uniquemember attributes. We have over 50000 uniquemembers in a single groupOfUniqueNames. We have not set the sortvals, but I will provide. (When configured is necessary to make a new load of data?)
As the management tool, it makes the inclusion or deletion of a single uniquemember attribute at a time. But is not fast these operations thus need some tuning in OpenLDAP and so I'm using you!
As for replication, I'm using delta, because as I read, and get it right, it only replicates the operations on the attributes of the node, and not all its value as is done in other types of replication. I like it to be faster.
Alessandro
-----Mensagem original----- De: openldap-technical [mailto:openldap-technical-bounces@openldap.org] Em nome de Michael Ströder Enviada em: sexta-feira, 30 de janeiro de 2015 09:48 Para: openldap-technical@openldap.org Assunto: Re: Large Number of Transactions x Low performance
Andrew Findlay wrote:
We have a tool (BMC Identity Management (formerly Control-SA))
Check how it modifies the groups.
First of all it's not clear to me whether the original poster really meant group entries.
If it rewrites the whole set of 'member' attributes each time then you are forcing the server to re-index every value.
Not only that it's horribly inefficient. With this approach you run into the famous security issue back in those Windows 2000 days where removed group members were accidently re-added because of concurrent write access.
The efficient way to do this is to specify the value that you want to remove or the one you want to add. It should be very quick.
Additionally LDAP PDUs manipulating many values at once grow very big.
Also look at your replication setup. With this sort of data you really do need delta mode.
delta-syncrepl would only help if group membership is changed only for a few members in one modify request.
Ciao, Michael.
openldap-technical@openldap.org
-
Alessandro Lasmar Mourao -
Andrew Findlay -
Michael Ströder