data import very slow for data migration
by anil beniwal
Hi
We are having 4 million users to migrate, all data exported from oracle to
multiple ldif files.
Imported 1 million till now, took almost 28 hours. and openldap-data dir of
about 28G.
openldap version 2.4.33 bdb version 5.1.29 RHEL 6.3 RAM 8G 4 cpu , system
is a VM.
Currently running slapadd output
+ /apps/openldap/sbin/slapadd -q -c -w -f
/apps/openldap/etc/openldap/slapd.conf -l /root/User9.ldif
bdb_monitor_db_open: monitoring disabled; configure monitor database to
enable
. 2.27% eta 21h31m elapsed 29m57s spd 1.6
k/s str2entry: invalid value for attributeType postalAddress #0 (syntax
1.3.6.1.4.1.1466.115.121.1.41)
slapadd: could not parse entry (line=394416)
* 2.81% eta 19h59m elapsed 34m40s spd 10.1
k/s
Its seems to be taking weeks go import whole data.
is there any tool or any other approach which we can use to make it
fast,Or we are going with wrong configuration.
Or we have to switch to ODS or RHDS
Top output
top - 10:26:04 up 21 days, 6:51, 3 users, load average: 2.13, 2.06, 1.79
Tasks: 153 total, 2 running, 151 sleeping, 0 stopped, 0 zombie
Cpu0 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu3 : 3.0%us, 0.3%sy, 0.0%ni, 0.0%id, 96.6%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 9095980k total, 8956852k used, 139128k free, 31452k buffers
Swap: 6291448k total, 21300k used, 6270148k free, 7431012k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
27877 ldap 20 0 4855m 539m 284m S 99.8 6.1 1807:28
slapd
21130 root 20 0 5267m 3.8g 3.0g R 4.3 43.6 0:59.27 slapadd
DB_CONFIG
set_cachesize 0 4294967295 0
set_lg_regionmax 2048576
set_lg_max 20485760
set_lg_bsize 2097152
set_lk_max_locks 10000
set_lk_max_objects 5000
set_lk_max_lockers 5000
slapd.conf
include /apps/openldap/etc/openldap/schema/core.schema
include /apps/openldap/etc/openldap/schema/cosine.schema
include /apps/openldap/etc/openldap/schema/nis.schema
include /apps/openldap/etc/openldap/schema/inetorgperson.schema
include /apps/openldap/etc/openldap/schema/openldap.schema
include /apps/openldap/etc/openldap/schema/dyngroup.schema
include /apps/openldap/etc/openldap/schema/ppolicy.schema
include /apps/openldap/etc/openldap/schema/channelIdentifier.schema
include /apps/openldap/etc/openldap/schema/platform.schema
include /apps/openldap/etc/openldap/schema/extendedProfileKey.schema
include
/apps/openldap/etc/openldap/schema/extendedProfileValue.schema
include /apps/openldap/etc/openldap/schema/behaviorKey.schema
include /apps/openldap/etc/openldap/schema/behaviorValue.schema
include /apps/openldap/etc/openldap/schema/questionAnswer.schema
include /apps/openldap/etc/openldap/schema/extendedTop.schema
include /apps/openldap/etc/openldap/schema/counter.schema
pidfile /apps/openldap/var/run/slapd.pid
argsfile /apps/openldap/var/run/slapd.args
logfile /apps/logs/ldap
loglevel 16640
database bdb
suffix "dc=example,dc=com"
access to attrs=userPassword
by self write
by anonymous auth
by * break
access to *
by
group/groupOfUniqueNames/uniqueMember.exact="cn=VWrite,ou=businessUsersGroup,dc=example,dc=com"
manage
by
group/groupOfUniqueNames/uniqueMember.exact="cn=VRead,ou=businessUsersGroup,dc=example,dc=com"
read
by * break
access to *
by self write
by anonymous auth
by * read
rootdn "cn=Manager,dc=example,dc=com"
rootpw {SSHA}dXDFSQeFjSofJ3TAzYf8DrDSYWY
################## SSL ##########################################
#
TLSCipherSuite HIGH:MEDIUM:+SSLv3
TLSCACertificateFile /apps/openldap/etc/openldap/cacerts/cacert.pem
TLSCertificateFile /apps/openldap/etc/openldap/cacerts/dam01.crt
TLSCertificateKeyFile /apps/openldap/etc/openldap/cacerts/dam01.key
#
####################################################################
####ache Entries #####
cachesize 900000
#idlcachesize 900000
lastmod on
checkpoint 128 15
concurrency 100
index entryCSN eq
index entryUUID eq
index
mail,uid,postalCode,smail,channelType,channelValue,answer,behavName,objectclass,tokenID,type
eq
index givenName,sn,city,question,behavValue,cn,extName sub
index displayName approx
# Replication Configuration
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
serverid 1
syncrepl rid=111
provider=ldap://s01.com
binddn="cn=Manager,dc=example,dc=com"
bindmethod=simple
starttls=yes
tls_reqcert=allow
schemachecking=off
credentials=G00gle#
searchbase="dc=example,dc=com"
type=refreshAndPersist
retry="5 5 300 +"
interval=00:00:00:10
syncrepl rid=222
provider=ldap://m04.com
binddn="cn=Manager,dc=example,dc=com"
bindmethod=simple
starttls=yes
tls_reqcert=allow
schemachecking=off
credentials=G00gle#
searchbase="dc=example,dc=com"
type=refreshAndPersist
retry="5 5 300 +"
interval=00:00:00:10
######
mirrormode TRUE
directory /apps/openldap/var/openldap-data
overlay unique
unique_attributes mail
overlay ppolicy
ppolicy_default "cn=default,ou=pwdPolicy,dc=example,dc=com"
ppolicy_use_lockout
--
Please let me know in case you need further details.
Thanks&Regards
Anil Beniwal
+919891695048
8 years, 11 months
How to force password change upon account creation
by Kyle Harris
Hello All,
I have a perl script that allows for the creation of new accounts in
OpenLDAP. I am attempting to find a way to force the newly created user to
change his or her password upon first login. I tried setting the attribute
pwdMustChange to TRUE but that attribute must not be definable upon user
creation. So, how can this be accomplished so that a new user is forced to
change passwords after they first log on?
Thank you.
8 years, 11 months
Migration from openldap 2.4.20 to 2.4.33
by anil beniwal
Hi
We are planing migration from openldap 2.4.20 (with bdb 4.8) to openldap
2.4.33 (bdb 5.1.29)
No of users are 4 million and about to go live within next 10 days.
We are using flat file for configuration in use.
Below is my slapd.conf and DB_CONFIG files
include /apps/openldap/etc/openldap/schema/core.schema
include /apps/openldap/etc/openldap/schema/cosine.schema
include /apps/openldap/etc/openldap/schema/nis.schema
include /apps/openldap/etc/openldap/schema/inetorgperson.schema
include /apps/openldap/etc/openldap/schema/openldap.schema
include /apps/openldap/etc/openldap/schema/dyngroup.schema
include /apps/openldap/etc/openldap/schema/ppolicy.schema
include /apps/openldap/etc/openldap/schema/channelIdentifier.schema
include /apps/openldap/etc/openldap/schema/platform.schema
include /apps/openldap/etc/openldap/schema/extendedProfileKey.schema
include
/apps/openldap/etc/openldap/schema/extendedProfileValue.schema
include /apps/openldap/etc/openldap/schema/behaviorKey.schema
include /apps/openldap/etc/openldap/schema/behaviorValue.schema
include /apps/openldap/etc/openldap/schema/questionAnswer.schema
include /apps/openldap/etc/openldap/schema/extendedTop.schema
include /apps/openldap/etc/openldap/schema/counter.schema
pidfile /apps/openldap/var/run/slapd.pid
argsfile /apps/openldap/var/run/slapd.args
logfile /apps/logs/ldap
loglevel 16640
database bdb
suffix "dc=ibm,dc=com"
access to attrs=userPassword
by self write
by anonymous auth
by * break
access to *
by
group/groupOfUniqueNames/uniqueMember.exact="cn=VWrite,ou=businessUsersGroup,dc=ibm,dc=com"
manage
by
group/groupOfUniqueNames/uniqueMember.exact="cn=VRead,ou=businessUsersGroup,dc=ibm,dc=com"
read
by * break
access to *
by self write
by anonymous auth
by * read
rootdn "cn=Manager,dc=ibm,dc=com"
rootpw {SSHA}dXDFSQeFjSoa/A1HfJ3TAzYf8
################## SSL ##########################################
#
#TLSVerifyClient allow
TLSCipherSuite HIGH:MEDIUM:+SSLv3
TLSCACertificateFile /apps/openldap/etc/openldap/cacerts/nascarcacert.pem
TLSCertificateFile /apps/openldap/etc/openldap/cacerts/sj.crt
TLSCertificateKeyFile /apps/openldap/etc/openldap/cacerts/sj.key
#
index entryCSN eq
index entryUUID eq
index
mail,uid,postalCode,smail,channelType,channelValue,answer,behavName,objectclass,tokenID,type
eq
index givenName,sn,city,question,behavValue,cn,extName sub
index displayName approx
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
serverid 3
syncrepl rid=111
provider=ldap://mmprod04
binddn="cn=Manager,dc=ibm,dc=com"
bindmethod=simple
starttls=yes
tls_reqcert=allow
credentials=G00gle#
searchbase="dc=ibm,dc=com"
type=refreshAndPersist
retry="5 5 300 +"
interval=00:00:00:10
syncrepl rid=222
provider=ldap://mmprod05
binddn="cn=Manager,dc=ibm,dc=com"
bindmethod=simple
starttls=yes
tls_reqcert=allow
credentials=G00gle#
searchbase="dc=idm,dc=com"
type=refreshAndPersist
retry="5 5 300 +"
interval=00:00:00:10
mirrormode TRUE
cachesize 100000
idlcachesize 300000
lastmod on
checkpoint 128 15
concurrency 100
directory /apps/openldap/var/openldap-data
overlay unique
unique_attributes mail
overlay ppolicy
ppolicy_default "cn=default,ou=pwdPolicy,dc=idm,dc=com"
ppolicy_use_lockout
DB_CONFIG
set_cachesize 0 4294967295 0
set_lg_regionmax 2048576
set_lg_max 20485760
set_lg_bsize 2097152
set_lk_max_locks 10000
set_lk_max_objects 5000
set_lk_max_lockers 5000
My querries are:-
1. What should be taken care(Best Practices).
2. Data migration can be db_hotbackup will work?
3. Can same flat file method be used, if not what could be the way should
work out.
4. any thing else i should be aware and is critical.
--
Thanks&Regards
Anil Beniwal
8 years, 11 months
Why ldapsearch is not working with anonymous bind after upgrading OpenLDAP to v2.4?
by Sachin Divekar
Dear all,
I have a setup of **OpenLDAP v2.3** which I am using for last few years.
Following are the lines in `slapd.conf` for access control.
access to dn.one="o=abc, c=IN"
by * read
access to dn.base="o=abc, c=IN"
by * none
When I do ldapsearch using anonymous bind gives me result.
For example following command gives result.
ldapsearch -x -h localhost -b "o=abc,c=IN"
Now I upgraded the OS, CentOS from 5.5 to 6.3 so the version of OpenLDAP is
**OpenLDAP v2.4**. We have not changed the schema.
But now the same `ldapsearch` gives me `result: 32 No such object` error.
But it works when I added following line in access control configuration.
access to dn.one="o=abc, c=IN"
by * read
access to dn.base="o=abc, c=IN"
by anonymous read
by * none
What can be the reason? Is there any security risk in doing so?
Thank you.
--
Regards,
Sachin Divekar
8 years, 11 months
Why I can't get back attribute tokenGroups from AD with ldapsearch command?
by ctosgh
Hi, World
I have one question about my recent work on LDAP.
Why I can't get tokenGroups back but can get other attributes back with following search against an AD server?
[root@fc11-lab ~]# ldapsearch -x -D "cn=admin,cn=Users,dc=jacky,dc=org,dc=cn" -b "cn=Users,dc=jacky,dc=org,dc=cn" -w 11111111 -H "ldap://x.x.x.x:389" "sAMAccountName=user1" cn whenChanged userPrincipalName tokenGroups
# extended LDIF
#
# LDAPv3
# base <cn=Users,dc=jacky,dc=org,dc=cn> with scope subtree
# filter: sAMAccountName=user1
# requesting: cn whenChanged userPrincipalName tokenGroups
#
# search result
search: 2
result: 1 Operations error
text: 00002120: SvcErr: DSID-03140293, problem 5012 (DIR_ERROR), data 0
# numResponses: 1
However, if I do NOT request tokenGroups attribute I get a successful response.
[root@fc11-lab ~]# ldapsearch -x -D "cn=admin,cn=Users,dc=jacky,dc=org,dc=cn" -b "cn=Users,dc=jacky,dc=org,dc=cn" -w 11111111 -H "ldap://x.x.x.x:389" "sAMAccountName=user1" cn whenChanged userPrincipalName
# extended LDIF
#
# LDAPv3
# base <cn=Users,dc=jacky,dc=org,dc=cn> with scope subtree
# filter: sAMAccountName=user1
# requesting: cn whenChanged userPrincipalName
#
# user1, Users, jacky.org.cn
dn: CN=user1,CN=Users,DC=jacky,DC=org,DC=cn
cn: user1
whenChanged: 20121221012448.0Z
userPrincipalName: user1(a)jacky.org.cn
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
I do see entry "CN=user1,CN=Users,DC=jacky,DC=org,DC=cn" has the attribute tokenGroups on AD.
Any thoughs? TIA
Thanks,
Jacky
8 years, 11 months
nssov works fine, unable to set nssov-pam-session
by Василий Молостов
Before enabling nssov I have got nslcd working, so the transition from
one to another was clear (ubuntu 12.04.1, openldap
2.4.28-1.1ubuntu4.2).
according to slapo-nssov man page I have added to my slapd.conf:
moduleload nssov
overlay nssov
nssov-pam userhost
nssov-pam-session login
nssov-pam-session sshd
but olcOverlay=nssov,olcDatabase=hdb,cn=config has no olcNssPamSession
and its related values after creation of db.
When I add olcNssPamSession into cn=config with above values by hand -
loginstatus is working well, but when nssov-pam-session into
slapd.conf - none.
Does it mean that I am missed something to correctly set
nssov-pam-session in slapd.conf?
8 years, 11 months
RE: META database root DN : no such object
by Bryce Powell
Aaron, thanks very much for your suggestion. I understand the key concept here is "subordinate", in order to glue the meta database into another superior naming context.
I was able to create a LDIF database, containing a baseDN, which was superior to the subordinate META database. The vendor application appears to accept this LDIF database now, as it can verify the existence of the baseDN. My next hurdle will be to get it authorize users based on group membership within the underlying Active Directories. But I think that may be beyond the scope of this forum.
Thanks again for your help.
Bryce
-----Original Message-----
From: Aaron Richton [mailto:richton@nbcs.rutgers.edu]
Sent: December 20, 2012 03:20 PM
To: Bryce Powell
Cc: openldap-technical(a)openldap.org
Subject: Re: META database root DN : no such object
On Thu, 20 Dec 2012, Bryce Powell wrote:
> ?When a search with base "dc=foo,dc=com" is attempted, if the scope is
> "base" it fails with "no such object"; in fact, the common root
> of the two targets (prior to massaging) does not exist.? The vendor
> won?t change their code to skip the verification, and recommended I
> use Microsoft?s ADAM instead of OpenLDAP. I would prefer to leverage
> OpenLDAP, so does anyone have any recommendations as to what I could do?
> Thanks, Bryce
You're quoting from "scenario 2a" from the man page, which envisions dc=a,dc=foo,dc=com and dc=b,dc=foo,dc=com; your desire is to serve some data at dc=foo,dc=com. So you have to make that exist (obviously). You'll need a data store to place your "dc=foo,dc=com" data, and you'll need to "attach" dc=a,dc=foo,dc=com and dc=b,dc=foo,dc=com. So basically...
database meta # maybe ldap or even relay in some installations
subordinate
suffix "dc=a,dc=foo,dc=com"
uri "ldap://a.foo.com/dc=a,dc=foo,dc=com"
database meta
subordinate
suffix "dc=b,dc=foo,dc=com"
uri "ldap://b.foo.com/dc=a,dc=foo,dc=com"
database mdb # or hdb or bdb or even ldif or.....
suffix "dc=foo,dc=com"
So then dc=a and dc=b live over the wire, and dc=foo,dc=com can be filled with Whatever You Want. Like, say, your base-scope data at dc=foo,dc=com.
You'll almost certainly want to set up some careful ACLs and make sure, in particular, that nobody writes any dc=a/dc=b data to the on-disk database.
Without trying it, I don't think it would cause a failure per se, but it would cause a very confused LDAP admin (quite undesirable)! (As for "dc=c"
data on-disk, that's up to you and your site.)
8 years, 11 months
Can we configure the way by which slapd return search response?
by ctosgh
Hi, All
I do the following similiar search against AD and openldap respectively with ldapsearch command.
For AD:
ldapsearch -x -D "cn=admin,cn=Users,dc=jacky,dc=org,dc=cn" -b "cn=Users,dc=jacky,dc=org,dc=cn" -w 11111111 -H "ldap://x.x.x.x:389" "sAMAccountName=user1"
For openldap:
ldapsearch -x -D "cn=admin,dc=jacky,dc=com" -b "ou=users,dc=jacky,dc=com" -w 11111111 -H "ldap://x.x.x.x:389" "uid=user"
I capture all the packets for above two searchs with tcpdump and I find that AD server will send search response(including search result entry & search result) in ONE packet while openldap will send search result entry and search result independently in two packets.
My question:
Is how to send search response back(in one or more packets) implementation specific?
Thanks,
Jacky
8 years, 11 months
META database root DN : no such object
by Bryce Powell
Hi,
I've configured a META database to proxy two LDAP directories. Each of those LDAP directories is in turn a proxy for an Active Directory. My intention was to use the meta directory as a single point for user authentication, however, the vendor application does not allow one to use the OpenLDAP meta directory. Their LDAP authentication setup wizard performs a verification of the specified baseDN, i.e. the root DN (suffix) of the meta database, and this does not exist. As per the slapd-meta man page:
"When a search with base "dc=foo,dc=com" is attempted, if the scope is "base" it fails with "no such object"; in fact, the common root of the two targets (prior to massaging) does not exist."
The vendor won't change their code to skip the verification, and recommended I use Microsoft's ADAM instead of OpenLDAP. I would prefer to leverage OpenLDAP, so does anyone have any recommendations as to what I could do?
Thanks,
Bryce
8 years, 11 months
