Dooh, sorry, this was supposed to go to the list instead of you directly.
---------------------------- Original Message ---------------------------- Subject: Re: Bug(?) With OpenLDAP 2.3.32 From: daniel@ncsu.edu Date: Wed, January 24, 2007 1:51 pm To: "Gavin Henry" ghenry@suretecsystems.com --------------------------------------------------------------------------
I'm not supplying the old attribute. This particular individual has a double major and hence the way we are using it, he has two ou's associated with him.
Daniel
<quote who="daniel@ncsu.edu"> > Hi folk! > > We upgraded to OpenLDAP 2.3.32 recently and I ran into something that, > unless I have completely lost my mind, should not be occuring: > > /local/ldap/data # /local/ldap/bin/ldapmodify -x -h localhost -D > "cn=ldapadmin,dc=ncsu,dc=edu" -w LDAPADMINPASSWORD > dn: uid=STUDENTUSERNAME,ou=students,ou=people,dc=ncsu,dc=edu > changetype: modify > replace: ou > ou: B A - Physics > ou: B S - Philosophy > - > replace: ncsucurriculumcode > ncsucurriculumcode: PYA > ncsucurriculumcode: LSL > > modifying entry > "uid=STUDENTUSERNAME,ou=students,ou=people,dc=ncsu,dc=edu" > ldap_modify: Type or value exists (20) > additional info: modify/replace: ou: value #1 already exists > > > Obviously I replaced the user's username and my ldap admin password. ;D > A replace should literally be replacing the ou and ignoring what it's > currently set to, correct? And since those two ou's are not the same, > it > should be fine? What's even more bizarre is that I didn't run into this > while populating the database in the first place. Is this, perchance, > fixed in 2.3.33? Thanks!
You don't supply the old attribute value, just the new one.
man ldapmodify
Daniel
--On Wednesday, January 24, 2007 1:53 PM -0500 daniel@ncsu.edu wrote:
Dooh, sorry, this was supposed to go to the list instead of you directly.
---------------------------- Original Message ---------------------------- Subject: Re: Bug(?) With OpenLDAP 2.3.32 From: daniel@ncsu.edu Date: Wed, January 24, 2007 1:51 pm To: "Gavin Henry" ghenry@suretecsystems.com
I'm not supplying the old attribute. This particular individual has a double major and hence the way we are using it, he has two ou's associated with him.
Works for me?
ldap-dev0:~> ldapmodify -Q -h ldap-dev0 dn: suRegID=85e49978f61311d2ae662436000baa77,cn=people,dc=stanford,dc=edu changetype: modify replace: ou ou: B A - Physics ou: B S - Philosophy
modifying entry "suRegID=85e49978f61311d2ae662436000baa77,cn=people,dc=stanford,dc=edu"
Then a second time:
ldap-dev0:~> ldapmodify -Q -h ldap-dev0 dn: suRegID=85e49978f61311d2ae662436000baa77,cn=people,dc=stanford,dc=edu changetype: modify replace: ou ou: B A - Physics ou: B S - Philosophy
modifying entry "suRegID=85e49978f61311d2ae662436000baa77,cn=people,dc=stanford,dc=edu"
so I'm not seeing it. This is with OL 2.3.32.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Hrm. Well I don't know what caused it in the first place, but get this... performing this set of operations fixed it altogether:
dn: uid=USERNAME,ou=students,ou=people,dc=ncsu,dc=edu changetype: modify delete: ou - replace: ou ou: B A - Physics ou: B S - Philosophy - delete: ncsucurriculumcode - replace: ncsucurriculumcode ncsucurriculumcode: PYA ncsucurriculumcode: LSL
After that, even the original method works just fine:
dn: uid=USERNAME,ou=students,ou=people,dc=ncsu,dc=edu changetype: modify replace: ou ou: B A - Physics ou: B S - Philosophy - replace: ncsucurriculumcode ncsucurriculumcode: PYA ncsucurriculumcode: LSL
I guess I can just keep an eye on it and see if this happens again, but it's not exactly confidence inspiring. =)
Daniel
--On Wednesday, January 24, 2007 1:53 PM -0500 daniel@ncsu.edu wrote:
Dooh, sorry, this was supposed to go to the list instead of you directly.
---------------------------- Original Message
Subject: Re: Bug(?) With OpenLDAP 2.3.32 From: daniel@ncsu.edu Date: Wed, January 24, 2007 1:51 pm To: "Gavin Henry" ghenry@suretecsystems.com
I'm not supplying the old attribute. This particular individual has a double major and hence the way we are using it, he has two ou's associated with him.
Works for me?
ldap-dev0:~> ldapmodify -Q -h ldap-dev0 dn: suRegID=85e49978f61311d2ae662436000baa77,cn=people,dc=stanford,dc=edu changetype: modify replace: ou ou: B A - Physics ou: B S - Philosophy
modifying entry "suRegID=85e49978f61311d2ae662436000baa77,cn=people,dc=stanford,dc=edu"
Then a second time:
ldap-dev0:~> ldapmodify -Q -h ldap-dev0 dn: suRegID=85e49978f61311d2ae662436000baa77,cn=people,dc=stanford,dc=edu changetype: modify replace: ou ou: B A - Physics ou: B S - Philosophy
modifying entry "suRegID=85e49978f61311d2ae662436000baa77,cn=people,dc=stanford,dc=edu"
so I'm not seeing it. This is with OL 2.3.32.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
daniel@ncsu.edu wrote:
Hrm. Well I don't know what caused it in the first place, but get this... performing this set of operations fixed it altogether:
dn: uid=USERNAME,ou=students,ou=people,dc=ncsu,dc=edu changetype: modify delete: ou
replace: ou ou: B A - Physics ou: B S - Philosophy
delete: ncsucurriculumcode
replace: ncsucurriculumcode ncsucurriculumcode: PYA ncsucurriculumcode: LSL
After that, even the original method works just fine:
dn: uid=USERNAME,ou=students,ou=people,dc=ncsu,dc=edu changetype: modify replace: ou ou: B A - Physics ou: B S - Philosophy
replace: ncsucurriculumcode ncsucurriculumcode: PYA ncsucurriculumcode: LSL
I guess I can just keep an eye on it and see if this happens again, but it's not exactly confidence inspiring. =)
It's too bad you didn't save a copy of the original data for us to dissect, but what you've described so far sounds like these entries actually had two separate instances of the ou attribute. Ordinarily that should never happen. What commands did you use to initially populate the database? If you repeat that (on a separate, fresh database) does the same problem recur?
daniel@ncsu.edu wrote:
Hrm. Well I don't know what caused it in the first place, but get this... performing this set of operations fixed it altogether:
dn: uid=USERNAME,ou=students,ou=people,dc=ncsu,dc=edu changetype: modify delete: ou
replace: ou ou: B A - Physics ou: B S - Philosophy
delete: ncsucurriculumcode
replace: ncsucurriculumcode ncsucurriculumcode: PYA ncsucurriculumcode: LSL
After that, even the original method works just fine:
dn: uid=USERNAME,ou=students,ou=people,dc=ncsu,dc=edu changetype: modify replace: ou ou: B A - Physics ou: B S - Philosophy
replace: ncsucurriculumcode ncsucurriculumcode: PYA ncsucurriculumcode: LSL
I guess I can just keep an eye on it and see if this happens again, but it's not exactly confidence inspiring. =)
It's too bad you didn't save a copy of the original data for us to dissect, but what you've described so far sounds like these entries actually had two separate instances of the ou attribute. Ordinarily that should never
Ya know what! Now that you said, that, I actually -do- have a copy of the data in ldif dump format. (pulled via slapcat) Is that what would be useful to see?
happen. What commands did you use to initially populate the database? If you
I used slapadd to populate the database initially.
repeat that (on a separate, fresh database) does the same problem recur?
I've test that theory if I can find some resources to try it with. (kinda 'strapped' at the moment so to speak)
Little background, our LDAP service is 100% pulled from other sources. Wiping it and rebuilding it from scratch is always possible. (in other words, no end users make changes directly, the only thing that makes any modifications what-so-ever is the update script we run on the master ldap server which generates ldif files of what -should- be in the database, then via some scripts that come with ... Net::LDAP? some perl module, not remembering off the top of my head... ldifsort, ldifdiff, etc ... I compare the last build to todays build and push the modifications via ldapmodify.
If this occurs again, what all would be useful for me to pull aside before fixing it? I'll make a point of doing so.
Daniel
-- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/
daniel@ncsu.edu wrote:
daniel@ncsu.edu wrote:
dn: uid=USERNAME,ou=students,ou=people,dc=ncsu,dc=edu
Ya know what! Now that you said, that, I actually -do- have a copy of the data in ldif dump format. (pulled via slapcat) Is that what would be useful to see?
Yes, the LDIF for the uid=USERNAME entry that first showed the problem would be a good start.
happen. What commands did you use to initially populate the database? If you
I used slapadd to populate the database initially.
No other mods were done to the entry in the meantime?
repeat that (on a separate, fresh database) does the same problem recur?
I've test that theory if I can find some resources to try it with. (kinda 'strapped' at the moment so to speak)
OK.
Little background, our LDAP service is 100% pulled from other sources. Wiping it and rebuilding it from scratch is always possible. (in other words, no end users make changes directly, the only thing that makes any modifications what-so-ever is the update script we run on the master ldap server which generates ldif files of what -should- be in the database, then via some scripts that come with ... Net::LDAP? some perl module, not remembering off the top of my head... ldifsort, ldifdiff, etc ... I compare the last build to todays build and push the modifications via ldapmodify.
If this occurs again, what all would be useful for me to pull aside before fixing it? I'll make a point of doing so.
Probably a copy of the id2entry database for starters. That's assuming the entry got stored incorrectly on disk. If it only got corrupted in the in-memory cache, we'd have to see the actual Entry structure in memory. That would mean using gdb to break into the flow of things while the bad entry is being accessed. (So you need a debug build of slapd with the full symbol table intact.)
As a complete aside, you might consider using something like the addpartial overlay in ITS#3593 http://www.openldap.org/its/index.cgi/Contrib?id=3593 instead of sort/diff. Just ldapadd everything and let the overlay figure out the changes. That's essentially what the syncrepl consumer does with the entries it receives from the provider.
On Wednesday 24 January 2007 19:31, Howard Chu wrote:
As a complete aside, you might consider using something like the addpartial overlay in ITS#3593 http://www.openldap.org/its/index.cgi/Contrib?id=3593 instead of sort/diff. Just ldapadd everything and let the overlay figure out the changes. That's essentially what the syncrepl consumer does with the entries it receives from the provider.
If you happen to be interested in this overlay, feel free to contact me about it. I have never versions that aren't available at that URL (that ensure syncrepl works, among other things).
dave
On Wednesday 24 January 2007 19:31, Howard Chu wrote:
As a complete aside, you might consider using something like the addpartial overlay in ITS#3593 http://www.openldap.org/its/index.cgi/Contrib?id=3593 instead of sort/diff. Just ldapadd everything and let the overlay figure out the changes. That's essentially what the syncrepl consumer does with the entries it receives from the provider.
If you happen to be interested in this overlay, feel free to contact me about it. I have never versions that aren't available at that URL (that ensure syncrepl works, among other things).
Oh good =D I have a lot of different responses for pieces of Howard's message, but this makes it easy to split off this conversation. ;D For one thing, I think that functionality is awesome! And at some level I would love to be able to make use of it! The only reason I don't think I would end up using it right now is that I do a lot of sanity checks on what I have. Basically I create the ldifdiff output, parse it and see how many mods, deletes, adds, etc are being done, and cap the max number that can be performed without someone looking over what's going on first. So in effect, I'm already doing the work to do the diffs, might as well use what I have. That said, I adore the concept and is there any chance it'll ever end up in openldap's dist?
Oh also, I would actually love to 'play with it' if you wouldn't mind hooking me up with a copy. =)
Daniel
On Thursday 25 January 2007 19:06, daniel@ncsu.edu wrote:
On Wednesday 24 January 2007 19:31, Howard Chu wrote:
As a complete aside, you might consider using something like the addpartial overlay in ITS#3593 http://www.openldap.org/its/index.cgi/Contrib?id=3593 instead of sort/diff. Just ldapadd everything and let the overlay figure out the changes. That's essentially what the syncrepl consumer does with the entries it receives from the provider.
If you happen to be interested in this overlay, feel free to contact me about it. I have never versions that aren't available at that URL (that ensure syncrepl works, among other things).
Oh good =D I have a lot of different responses for pieces of Howard's message, but this makes it easy to split off this conversation. ;D For one thing, I think that functionality is awesome! And at some level I would love to be able to make use of it! The only reason I don't think I would end up using it right now is that I do a lot of sanity checks on what I have. Basically I create the ldifdiff output, parse it and see how many mods, deletes, adds, etc are being done, and cap the max number that can be performed without someone looking over what's going on first. So in effect, I'm already doing the work to do the diffs, might as well use what I have. That said, I adore the concept and is there any chance it'll ever end up in openldap's dist?
Oh also, I would actually love to 'play with it' if you wouldn't mind hooking me up with a copy. =)
An updated version of addpartial is available at http://www.openldap.org/its/index.cgi/Contrib?id=3593. Enjoy playing with it. This overlay has served us well since we put it in production in 2004.
dave
daniel@ncsu.edu wrote:
daniel@ncsu.edu wrote:
dn: uid=USERNAME,ou=students,ou=people,dc=ncsu,dc=edu
Ya know what! Now that you said, that, I actually -do- have a copy of the data in ldif dump format. (pulled via slapcat) Is that what would be useful to see?
Yes, the LDIF for the uid=USERNAME entry that first showed the problem would be a good start.
I XXX'd out everything that indicated who it was. ;)
dn: uid=XXX,ou=students,ou=people,dc=ncsu,dc=edu objectClass: person objectClass: inetOrgPerson objectClass: ncsuPerson uid: XXX cn: XXX sn: XXX title: Senior ncsuTwoPartName: XXX organizationalStatus: registered o: NC State University gn: XXX initials: XXX displayName: XXX ncsuAltDisplayName: XXX ncsuCampusID: XXX ncsuClassCode: SR ou: Physics ncsuCurriculumCode: PY ou: B S - Philosophy ncsuCurriculumCode: LSL mail: XXX@unity.ncsu.edu ncsuPrimaryEMail: XXX@unity.ncsu.edu registeredAddress: XXX postalAddress: XXX telephoneNumber: XXX l: Raleigh st: NC postalCode: 27603 ncsuPrimaryRole: staff
happen. What commands did you use to initially populate the database? If you
I used slapadd to populate the database initially.
No other mods were done to the entry in the meantime?
Nope, not a one. =(
repeat that (on a separate, fresh database) does the same problem recur?
I've test that theory if I can find some resources to try it with. (kinda 'strapped' at the moment so to speak)
OK.
Little background, our LDAP service is 100% pulled from other sources. Wiping it and rebuilding it from scratch is always possible. (in other words, no end users make changes directly, the only thing that makes any modifications what-so-ever is the update script we run on the master ldap server which generates ldif files of what -should- be in the database, then via some scripts that come with ... Net::LDAP? some perl module, not remembering off the top of my head... ldifsort, ldifdiff, etc ... I compare the last build to todays build and push the modifications via ldapmodify.
If this occurs again, what all would be useful for me to pull aside before fixing it? I'll make a point of doing so.
Probably a copy of the id2entry database for starters. That's assuming the entry got stored incorrectly on disk. If it only got corrupted in the in-memory cache, we'd have to see the actual Entry structure in memory. That would mean using gdb to break into the flow of things while the bad entry is being accessed. (So you need a debug build of slapd with the full symbol table intact.)
It happened again this morning with another record. I am making a tarball of the entire database at this moment. One thing you brought up there made me think, what if I just restarted slapd.. would that not free up the structure in memory and, if that -is- the problem, resolve it and at least point to that being the problem?
Just tried that and it didn't solve anything. Unfortunately I can't leave it in this state so I'm going to repair the issue for now. But I do have a dump of the busted db.
As a complete aside, you might consider using something like the addpartial overlay in ITS#3593 http://www.openldap.org/its/index.cgi/Contrib?id=3593 instead of sort/diff. Just ldapadd everything and let the overlay figure out the changes. That's essentially what the syncrepl consumer does with the entries it receives from the provider.
I posted separately about this. I really do like that concept, a lot, and hope it gets pulled into openldap 'proper' at some point! That said, right now the sanity checks I'm doing are based off the diffs and since I'm already doing the work . . . =) Might as well use it.
Daniel
Well, in this LDIF, the two ou: values are not contiguous. slapadd is supposed to reorder its input to make the values contiguous, but apparently it's not doing a good job here. In which case the entry would indeed get loaded with two separate ou attributes.
Pretty sure this is the culprit, no need to dig any further in the database.
daniel@ncsu.edu wrote:
dn: uid=XXX,ou=students,ou=people,dc=ncsu,dc=edu objectClass: person objectClass: inetOrgPerson objectClass: ncsuPerson uid: XXX cn: XXX sn: XXX title: Senior ncsuTwoPartName: XXX organizationalStatus: registered o: NC State University gn: XXX initials: XXX displayName: XXX ncsuAltDisplayName: XXX ncsuCampusID: XXX ncsuClassCode: SR ou: Physics ncsuCurriculumCode: PY ou: B S - Philosophy ncsuCurriculumCode: LSL mail: XXX@unity.ncsu.edu ncsuPrimaryEMail: XXX@unity.ncsu.edu registeredAddress: XXX postalAddress: XXX telephoneNumber: XXX l: Raleigh st: NC postalCode: 27603 ncsuPrimaryRole: staff
Howard Chu wrote:
Well, in this LDIF, the two ou: values are not contiguous. slapadd is supposed to reorder its input to make the values contiguous, but apparently it's not doing a good job here. In which case the entry would indeed get loaded with two separate ou attributes.
Pretty sure this is the culprit, no need to dig any further in the database.
Except that feeding this entry to slapadd shows that it is processed correctly. So much for that idea.
daniel@ncsu.edu wrote:
dn: uid=XXX,ou=students,ou=people,dc=ncsu,dc=edu objectClass: person objectClass: inetOrgPerson objectClass: ncsuPerson uid: XXX cn: XXX sn: XXX title: Senior ncsuTwoPartName: XXX organizationalStatus: registered o: NC State University gn: XXX initials: XXX displayName: XXX ncsuAltDisplayName: XXX ncsuCampusID: XXX ncsuClassCode: SR ou: Physics ncsuCurriculumCode: PY ou: B S - Philosophy ncsuCurriculumCode: LSL mail: XXX@unity.ncsu.edu ncsuPrimaryEMail: XXX@unity.ncsu.edu registeredAddress: XXX postalAddress: XXX telephoneNumber: XXX l: Raleigh st: NC postalCode: 27603 ncsuPrimaryRole: staff
On Jan 25, 2007, at 4:53 PM, Howard Chu wrote:
Howard Chu wrote:
Well, in this LDIF, the two ou: values are not contiguous. slapadd is supposed to reorder its input to make the values contiguous, but apparently it's not doing a good job here. In which case the entry would indeed get loaded with two separate ou attributes. Pretty sure this is the culprit, no need to dig any further in the database.
Except that feeding this entry to slapadd shows that it is processed correctly. So much for that idea.
Out-of-order attributes didn't work for us, when we tried it by accident. We had some LDIF entries with attributes like a: a1 b: bx a: a2
The way I remember it, this structure was preserved when we read the LDIF and wrote it back out, but when we processed a search result we would miss one of the "a" attribute values, so our reconciler would attempt to restore it. The modify could see the value we couldn't see, though, so it would refuse, "value #1 already exists." 2.3.24. I expect I could reproduce that.
Donn Cave, donn@u.washington.edu
<quote who="daniel@ncsu.edu">
Dooh, sorry, this was supposed to go to the list instead of you directly.
---------------------------- Original Message ---------------------------- Subject: Re: Bug(?) With OpenLDAP 2.3.32 From: daniel@ncsu.edu Date: Wed, January 24, 2007 1:51 pm To: "Gavin Henry" ghenry@suretecsystems.com
I'm not supplying the old attribute. This particular individual has a double major and hence the way we are using it, he has two ou's associated with him.
Hi Daniel,
Sorry about this misunderstanding. Glad you got this sorted out in the end.
Gavin.
Daniel
<quote who="daniel@ncsu.edu"> > Hi folk! > > We upgraded to OpenLDAP 2.3.32 recently and I ran into something that, > unless I have completely lost my mind, should not be occuring: > > /local/ldap/data # /local/ldap/bin/ldapmodify -x -h localhost -D > "cn=ldapadmin,dc=ncsu,dc=edu" -w LDAPADMINPASSWORD > dn: uid=STUDENTUSERNAME,ou=students,ou=people,dc=ncsu,dc=edu > changetype: modify > replace: ou > ou: B A - Physics > ou: B S - Philosophy > - > replace: ncsucurriculumcode > ncsucurriculumcode: PYA > ncsucurriculumcode: LSL > > modifying entry > "uid=STUDENTUSERNAME,ou=students,ou=people,dc=ncsu,dc=edu" > ldap_modify: Type or value exists (20) > additional info: modify/replace: ou: value #1 already exists > > > Obviously I replaced the user's username and my ldap admin password. > ;D > A replace should literally be replacing the ou and ignoring what it's > currently set to, correct? And since those two ou's are not the same, > it > should be fine? What's even more bizarre is that I didn't run into > this > while populating the database in the first place. Is this, perchance, > fixed in 2.3.33? Thanks!
You don't supply the old attribute value, just the new one.
man ldapmodify
Daniel
openldap-software@openldap.org
-
daniel＠ncsu.edu -
David Hawes -
Donn Cave -
Gavin Henry -
Howard Chu -
Quanah Gibson-Mount