Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
Thanks!
--On Tuesday, October 03, 2006 3:11 PM -0400 Robert Petkus rpetkus@bnl.gov wrote:
Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
Hard to say without knowing how your configuration is, for example, if you have multiple databases configured. You also don't say how the database melted down, which make give some helpful clues.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Quanah Gibson-Mount wrote:
--On Tuesday, October 03, 2006 3:11 PM -0400 Robert Petkus rpetkus@bnl.gov wrote:
Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
Hard to say without knowing how your configuration is, for example, if you have multiple databases configured. You also don't say how the database melted down, which make give some helpful clues.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
I hate making these things long because folks lose interest and stop reading but my environment is complex so:
Along with the main database, I am also using monitor and accesslog. Recently I began storing ssh public keys in LDAP for use with ssh-lpk. This past weekend ~15k accounts were added to LDAP and maybe 700 ssh keys (I manage LDAP not account management..). Replication failed on 2 nodes. I noticed on these nodes incoherency because I was using an outdated custom schema file (my fault) so I decided to wipe the database and reload it from backup. Not a big deal but I notice that my nightly slapcat ldifs (slapcat -n 2 -l ldap.ldif) are polluted with accesslog entries that *replace* the original entries. For example, my account dn won't include, say, sshPublicKey, but I'd see a reqMod entry with this attribute.
I can see every dn with a ldapsearch but am missing many dns using slapcat. Obversely, when I do slapcat, I get dn attributes from accesslog that I can't see with ldapsearch. It looks like some weird cross-pollination of the 2 databases.
Maybe there is something I am missing in my config. Here is a snippet -- the full config is available upon request.
Thanks!
--On Tuesday, October 03, 2006 8:49 PM -0400 Robert Petkus rpetkus@bnl.gov wrote:
slapcat ldifs (slapcat -n 2 -l ldap.ldif) are polluted with accesslog entries that *replace* the original entries. For example, my account dn won't include, say, sshPublicKey, but I'd see a reqMod entry with this attribute.
First, I'd make life simpler by listing the monitoring database last.
Second, your slapcat by definition dumps the accesslog database, not your main database, since your databases are:
1: monitor 2: cn=changelog 3: dc=bnl,dc=gov
Or at least, that's my guess, and it seems to go with what you note. Or, you could change your slapcat to use "-b dc=bnl,dc=gov" which would be more explicit. That is, of course, assuming that you want to dump your main DB and not the accesslog DB. ;)
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Quanah Gibson-Mount wrote:
--On Tuesday, October 03, 2006 8:49 PM -0400 Robert Petkus rpetkus@bnl.gov wrote:
slapcat ldifs (slapcat -n 2 -l ldap.ldif) are polluted with accesslog entries that *replace* the original entries. For example, my account dn won't include, say, sshPublicKey, but I'd see a reqMod entry with this attribute.
First, I'd make life simpler by listing the monitoring database last.
Second, your slapcat by definition dumps the accesslog database, not your main database, since your databases are:
1: monitor 2: cn=changelog 3: dc=bnl,dc=gov
Or at least, that's my guess, and it seems to go with what you note. Or, you could change your slapcat to use "-b dc=bnl,dc=gov" which would be more explicit. That is, of course, assuming that you want to dump your main DB and not the accesslog DB. ;)
Yeah it would be convenient if I was that dumb ;) , but I had tried "-b", -n3, removing the accesslog db entries in slapd.conf and rerunning slapcat. All with the same results -- most of the main DB with a bunch of accesslog DB garbage. What is dogging me *so* much here is that these are 2 distinct physical databases.
This is an example of the garbage I got yesterday from a slapcat for my user (an illustration that some attributes are not attached to the main DB but instead the accesslog DB, yet ldapsearchable to the main DB):
Cheers, Robert
***********ldapsearch results*****************
# rpetkus, People, racf.bnl.gov dn: uid=rpetkus,ou=People,dc=stuff,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf objectClass: ldapPublicKey uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS sn: rapetkus employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XasdfasdftDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48=
******Here is the slapcat for my user**************
dn: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS structuralObjectClass: inetOrgPerson entryUUID: 689ce5e4-010f-102a-8eef-9882d4436e05 creatorsName: cn=account,dc=bnl,dc=gov createTimestamp: 20051214170418Z sn: rapetkus userPassword:: employeeNumber: number loginShellGateway: /bin/rbash sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XZELCHtDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKReqWx5hc9Id5q6oStWrNuNmpV48= rpetkus@r sec00 employeeStatus: Active gecos: Robert Petkus 1 entryCSN: 20060906145341Z#000000#00#000000 modifiersName: cn=Manager,dc=bnl,dc=gov modifyTimestamp: 20060906145341Z
dn: reqStart=20060920134512.000000Z,cn=changelog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20060920134512.000000Z reqEnd: 20060920134512.000001Z reqType: modify reqSession: 423 reqAuthzID: cn=Manager,dc=bnl,dc=gov reqDN: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov reqResult: 0 reqMod: sshPublicKey:= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDXasdfasdftDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48= reqMod: entryCSN:= 20060920134512Z#000000#00#000000 reqMod: modifiersName:= cn=account,dc=bnl,dc=gov reqMod: modifyTimestamp:= 20060920134512Z entryUUID: fb865d9c-dcf9-102a-8a91-e5d2e62e4f1a creatorsName: cn=changelog createTimestamp: 20060920134512Z entryCSN: 20060920134512Z#000000#00#000000 modifiersName: cn=changelog modifyTimestamp: 20060920134512Z
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
I provided an example but it was the wrong one. You can see that the sshPublicKey attribute is shown in ldapsearch but isn't attached to the main DB entry produced from a slapcat.
***********ldapsearch results*****************
# rpetkus, People, racf.bnl.gov dn: uid=rpetkus,ou=People,dc=stuff,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf objectClass: ldapPublicKey uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS sn: rapetkus employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XasdfasdftDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48=
******Here is the slapcat for my user**************
dn: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS structuralObjectClass: inetOrgPerson entryUUID: 689ce5e4-010f-102a-8eef-9882d4436e05 creatorsName: cn=account,dc=bnl,dc=gov createTimestamp: 20051214170418Z sn: rapetkus userPassword:: employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus 1 entryCSN: 20060906145341Z#000000#00#000000 modifiersName: cn=Manager,dc=bnl,dc=gov modifyTimestamp: 20060906145341Z
dn: reqStart=20060920134512.000000Z,cn=changelog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20060920134512.000000Z reqEnd: 20060920134512.000001Z reqType: modify reqSession: 423 reqAuthzID: cn=Manager,dc=bnl,dc=gov reqDN: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov reqResult: 0 reqMod: sshPublicKey:= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDXasdfasdftDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48= reqMod: entryCSN:= 20060920134512Z#000000#00#000000 reqMod: modifiersName:= cn=account,dc=bnl,dc=gov reqMod: modifyTimestamp:= 20060920134512Z entryUUID: fb865d9c-dcf9-102a-8a91-e5d2e62e4f1a creatorsName: cn=changelog createTimestamp: 20060920134512Z entryCSN: 20060920134512Z#000000#00#000000 modifiersName: cn=changelog modifyTimestamp: 20060920134512Z
Robert Petkus wrote:
Quanah Gibson-Mount wrote:
--On Tuesday, October 03, 2006 8:49 PM -0400 Robert Petkus rpetkus@bnl.gov wrote:
slapcat ldifs (slapcat -n 2 -l ldap.ldif) are polluted with accesslog entries that *replace* the original entries. For example, my account dn won't include, say, sshPublicKey, but I'd see a reqMod entry with this attribute.
First, I'd make life simpler by listing the monitoring database last.
Second, your slapcat by definition dumps the accesslog database, not your main database, since your databases are:
1: monitor 2: cn=changelog 3: dc=bnl,dc=gov
Or at least, that's my guess, and it seems to go with what you note. Or, you could change your slapcat to use "-b dc=bnl,dc=gov" which would be more explicit. That is, of course, assuming that you want to dump your main DB and not the accesslog DB. ;)
Yeah it would be convenient if I was that dumb ;) , but I had tried "-b", -n3, removing the accesslog db entries in slapd.conf and rerunning slapcat. All with the same results -- most of the main DB with a bunch of accesslog DB garbage. What is dogging me *so* much here is that these are 2 distinct physical databases.
This is an example of the garbage I got yesterday from a slapcat for my user (an illustration that some attributes are not attached to the main DB but instead the accesslog DB, yet ldapsearchable to the main DB):
Cheers, Robert
***********ldapsearch results*****************
# rpetkus, People, racf.bnl.gov dn: uid=rpetkus,ou=People,dc=stuff,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf objectClass: ldapPublicKey uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS sn: rapetkus employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XasdfasdftDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48=
******Here is the slapcat for my user**************
dn: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS structuralObjectClass: inetOrgPerson entryUUID: 689ce5e4-010f-102a-8eef-9882d4436e05 creatorsName: cn=account,dc=bnl,dc=gov createTimestamp: 20051214170418Z sn: rapetkus userPassword:: employeeNumber: number loginShellGateway: /bin/rbash sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XZELCHtDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKReqWx5hc9Id5q6oStWrNuNmpV48= rpetkus@r sec00 employeeStatus: Active gecos: Robert Petkus 1 entryCSN: 20060906145341Z#000000#00#000000 modifiersName: cn=Manager,dc=bnl,dc=gov modifyTimestamp: 20060906145341Z
dn: reqStart=20060920134512.000000Z,cn=changelog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20060920134512.000000Z reqEnd: 20060920134512.000001Z reqType: modify reqSession: 423 reqAuthzID: cn=Manager,dc=bnl,dc=gov reqDN: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov reqResult: 0 reqMod: sshPublicKey:= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDXasdfasdftDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48= reqMod: entryCSN:= 20060920134512Z#000000#00#000000 reqMod: modifiersName:= cn=account,dc=bnl,dc=gov reqMod: modifyTimestamp:= 20060920134512Z entryUUID: fb865d9c-dcf9-102a-8a91-e5d2e62e4f1a creatorsName: cn=changelog createTimestamp: 20060920134512Z entryCSN: 20060920134512Z#000000#00#000000 modifiersName: cn=changelog modifyTimestamp: 20060920134512Z
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
On Wednesday 04 October 2006 11:28, Robert Petkus wrote:
I provided an example but it was the wrong one. You can see that the sshPublicKey attribute is shown in ldapsearch but isn't attached to the main DB entry produced from a slapcat.
***********ldapsearch results*****************
# rpetkus, People, racf.bnl.gov dn: uid=rpetkus,ou=People,dc=stuff,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf objectClass: ldapPublicKey uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS sn: rapetkus employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg
8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XasdfasdftDvN xbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48=
******Here is the slapcat for my user**************
dn: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS structuralObjectClass: inetOrgPerson entryUUID: 689ce5e4-010f-102a-8eef-9882d4436e05 creatorsName: cn=account,dc=bnl,dc=gov createTimestamp: 20051214170418Z sn: rapetkus userPassword:: employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus 1 entryCSN: 20060906145341Z#000000#00#000000 modifiersName: cn=Manager,dc=bnl,dc=gov modifyTimestamp: 20060906145341Z
The DN's are different, the first one is "...,dc=stuff,...", the second one is "...,dc=racf,...". Also the gecos attributes are different ("Robert Petkus" versus "Robert Petkus 1"). Are you sure you're looking at the same entries? Maybe you should also get the operations attributes via ldapsearch and compare the entryUUID's?
dn: reqStart=20060920134512.000000Z,cn=changelog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20060920134512.000000Z reqEnd: 20060920134512.000001Z reqType: modify reqSession: 423 reqAuthzID: cn=Manager,dc=bnl,dc=gov reqDN: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov reqResult: 0 reqMod: sshPublicKey:= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDXasdfasdftDvNxb z3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48= reqMod: entryCSN:= 20060920134512Z#000000#00#000000 reqMod: modifiersName:= cn=account,dc=bnl,dc=gov reqMod: modifyTimestamp:= 20060920134512Z entryUUID: fb865d9c-dcf9-102a-8a91-e5d2e62e4f1a creatorsName: cn=changelog createTimestamp: 20060920134512Z entryCSN: 20060920134512Z#000000#00#000000 modifiersName: cn=changelog modifyTimestamp: 20060920134512Z
That was a modification to the "racf" entry, not the "stuff" entry.
Karsten.
I don't want to sabotage this thread with my inaccuracies -- I do appreciate folk's interest in helping out -- this is indeed the same entry, I just stupidly didn't edit out all the *real* info -- too many e-mails and things to do and I got careless.
The main point is that an attribute is absent from the main DB entry in a slapcat, and instead appears attached to an accesslog DB entry. ldapsearch shows *all* the attributes.
Since yanking out the accesslog overlay directives, slapcat behaves as expected. I'm disappointed because accesslog was useful but my confidence in reinstantiating the overlay has been disrupted. I am not eager to be in a mess like yesterday.
I'm interested to know what kind of environments others are using accesslog in -- how many changes daily, etc.
Robert
Karsten Künne wrote:
On Wednesday 04 October 2006 11:28, Robert Petkus wrote:
I provided an example but it was the wrong one. You can see that the sshPublicKey attribute is shown in ldapsearch but isn't attached to the main DB entry produced from a slapcat.
***********ldapsearch results*****************
# rpetkus, People, racf.bnl.gov dn: uid=rpetkus,ou=People,dc=stuff,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf objectClass: ldapPublicKey uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS sn: rapetkus employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg
8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XasdfasdftDvN xbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48=
******Here is the slapcat for my user**************
dn: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS structuralObjectClass: inetOrgPerson entryUUID: 689ce5e4-010f-102a-8eef-9882d4436e05 creatorsName: cn=account,dc=bnl,dc=gov createTimestamp: 20051214170418Z sn: rapetkus userPassword:: employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus 1 entryCSN: 20060906145341Z#000000#00#000000 modifiersName: cn=Manager,dc=bnl,dc=gov modifyTimestamp: 20060906145341Z
The DN's are different, the first one is "...,dc=stuff,...", the second one is "...,dc=racf,...". Also the gecos attributes are different ("Robert Petkus" versus "Robert Petkus 1"). Are you sure you're looking at the same entries? Maybe you should also get the operations attributes via ldapsearch and compare the entryUUID's?
dn: reqStart=20060920134512.000000Z,cn=changelog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20060920134512.000000Z reqEnd: 20060920134512.000001Z reqType: modify reqSession: 423 reqAuthzID: cn=Manager,dc=bnl,dc=gov reqDN: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov reqResult: 0 reqMod: sshPublicKey:= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDXasdfasdftDvNxb z3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48= reqMod: entryCSN:= 20060920134512Z#000000#00#000000 reqMod: modifiersName:= cn=account,dc=bnl,dc=gov reqMod: modifyTimestamp:= 20060920134512Z entryUUID: fb865d9c-dcf9-102a-8a91-e5d2e62e4f1a creatorsName: cn=changelog createTimestamp: 20060920134512Z entryCSN: 20060920134512Z#000000#00#000000 modifiersName: cn=changelog modifyTimestamp: 20060920134512Z
That was a modification to the "racf" entry, not the "stuff" entry.
Karsten.
--On Wednesday, October 04, 2006 2:15 PM -0400 Robert Petkus rpetkus@bnl.gov wrote:
I don't want to sabotage this thread with my inaccuracies -- I do appreciate folk's interest in helping out -- this is indeed the same entry, I just stupidly didn't edit out all the *real* info -- too many e-mails and things to do and I got careless.
The main point is that an attribute is absent from the main DB entry in a slapcat, and instead appears attached to an accesslog DB entry. ldapsearch shows *all* the attributes.
Since yanking out the accesslog overlay directives, slapcat behaves as expected. I'm disappointed because accesslog was useful but my confidence in reinstantiating the overlay has been disrupted. I am not eager to be in a mess like yesterday.
I'm interested to know what kind of environments others are using accesslog in -- how many changes daily, etc.
I've been using accesslog for delta-syncrepl replication for several months now. It receives some 2000+ MOD/ADD ops a day. I've not seen any issues with it.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
On 10/4/06, Robert Petkus rpetkus@bnl.gov wrote:
Quanah Gibson-Mount wrote:
--On Tuesday, October 03, 2006 8:49 PM -0400 Robert Petkus rpetkus@bnl.gov wrote:
slapcat ldifs (slapcat -n 2 -l ldap.ldif) are polluted with accesslog entries that *replace* the original entries. For example, my account dn won't include, say, sshPublicKey, but I'd see a reqMod entry with this attribute.
First, I'd make life simpler by listing the monitoring database last.
Second, your slapcat by definition dumps the accesslog database, not your main database, since your databases are:
1: monitor 2: cn=changelog 3: dc=bnl,dc=gov
Or at least, that's my guess, and it seems to go with what you note. Or, you could change your slapcat to use "-b dc=bnl,dc=gov" which would be more explicit. That is, of course, assuming that you want to dump your main DB and not the accesslog DB. ;)
Yeah it would be convenient if I was that dumb ;) , but I had tried "-b", -n3, removing the accesslog db entries in slapd.conf and rerunning slapcat. All with the same results -- most of the main DB with a bunch of accesslog DB garbage. What is dogging me *so* much here is that these are 2 distinct physical databases.
This is an example of the garbage I got yesterday from a slapcat for my user (an illustration that some attributes are not attached to the main DB but instead the accesslog DB, yet ldapsearchable to the main DB):
Cheers, Robert
***********ldapsearch results*****************
# rpetkus, People, racf.bnl.gov dn: uid=rpetkus,ou=People,dc=stuff,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf objectClass: ldapPublicKey uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS sn: rapetkus employeeNumber: number loginShellGateway: /bin/rbash employeeStatus: Active gecos: Robert Petkus sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XasdfasdftDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48=
******Here is the slapcat for my user**************
dn: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov uid: rpetkus cn: Robert Petkus objectClass: inetOrgPerson objectClass: posixAccount objectClass: top objectClass: racf uidNumber: number gidNumber: number homeDirectory: /somewhere/rpetkus loginShell: /bin/bash gidNumberAtlas: number homeDirectoryAtlas: /somewhere/rpetkus experiment: RHIC/USATLAS structuralObjectClass: inetOrgPerson entryUUID: 689ce5e4-010f-102a-8eef-9882d4436e05 creatorsName: cn=account,dc=bnl,dc=gov createTimestamp: 20051214170418Z sn: rapetkus userPassword:: employeeNumber: number loginShellGateway: /bin/rbash sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDX1XZELCHtDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKReqWx5hc9Id5q6oStWrNuNmpV48= rpetkus@r sec00 employeeStatus: Active gecos: Robert Petkus 1 entryCSN: 20060906145341Z#000000#00#000000 modifiersName: cn=Manager,dc=bnl,dc=gov modifyTimestamp: 20060906145341Z
dn: reqStart=20060920134512.000000Z,cn=changelog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20060920134512.000000Z reqEnd: 20060920134512.000001Z reqType: modify reqSession: 423 reqAuthzID: cn=Manager,dc=bnl,dc=gov reqDN: uid=rpetkus,ou=People,dc=racf,dc=bnl,dc=gov reqResult: 0 reqMod: sshPublicKey:= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA36Y8jfKTKJgphUO30oaI9W5QVRUg 8+fM0FFYIkaiZUuaXBYpKaIiguUcQsy+3P+KjBTI0g1Qr3gewO20S0T4i8pDXasdfasdftDvNxbz3w se4V+PPGQ/Bm4DXTjGRoMVNBABIoqWo3vYOVCvKasdfasdfId5q6oStWrNuNmpV48= reqMod: entryCSN:= 20060920134512Z#000000#00#000000 reqMod: modifiersName:= cn=account,dc=bnl,dc=gov reqMod: modifyTimestamp:= 20060920134512Z entryUUID: fb865d9c-dcf9-102a-8a91-e5d2e62e4f1a creatorsName: cn=changelog createTimestamp: 20060920134512Z entryCSN: 20060920134512Z#000000#00#000000 modifiersName: cn=changelog modifyTimestamp: 20060920134512Z
What does a slapcat with the -f <config file> produce?
(To Karsten Kunne's point) Also, the slapo-accesslog man page states: These slapd.conf options apply to the Access Logging overlay. They should appear after the overlay directive and before any subsequent database directive.
Maybe your database is, in fact, getting corrupted by poor ordering.
matthew sporleder wrote:
What does a slapcat with the -f <config file> produce?
(To Karsten Kunne's point) Also, the slapo-accesslog man page states: These slapd.conf options apply to the Access Logging overlay. They should appear after the overlay directive and before any subsequent database directive.
Maybe your database is, in fact, getting corrupted by poor ordering.
Interesting. I read that as the accesslog overlay options "logdb", "logops", "logpurge" should be listed <under> the overlay directive but <before> a database directive. That would seem to contradict the example in the man page which I based my configuration on:
database bdb suffix cn=log ... index reqStart eq
database bdb suffix dc=example,dc=com ... overlay accesslog logdb cn=log logops writes reads
-
Robert Petkus Brookhaven National Laboratory Physics Dept. - Bldg. 510A Upton, New York 11973 http://www.bnl.gov/RHIC http://www.acf.bnl.gov
--On Wednesday, October 04, 2006 11:58 AM -0400 Robert Petkus rpetkus@bnl.gov wrote:
matthew sporleder wrote:
What does a slapcat with the -f <config file> produce?
(To Karsten Kunne's point) Also, the slapo-accesslog man page states: These slapd.conf options apply to the Access Logging overlay. They should appear after the overlay directive and before any subsequent database directive.
Maybe your database is, in fact, getting corrupted by poor ordering.
Interesting. I read that as the accesslog overlay options "logdb", "logops", "logpurge" should be listed <under> the overlay directive but <before> a database directive. That would seem to contradict the example in the man page which I based my configuration on:
database bdb suffix cn=log ... index reqStart eq database bdb suffix dc=example,dc=com ... overlay accesslog logdb cn=log logops writes reads
http://www.connexitor.com/forums/viewtopic.php?t=3
and I still recommend listing the monitor backend last. ;) Your configuration snippets look right to me.
My master has:
####################################################################### # accesslog database definitions ####################################################################### database hdb suffix cn=accesslog directory /var/lib/ldap/accesslog rootdn cn=accesslog index default eq index entryCSN index objectClass index reqEnd index reqResult index reqStart
####################################################################### # stanford.edu database definitions #######################################################################
database hdb suffix "dc=stanford,dc=edu" rootdn "cn=manager,dc=stanford,dc=edu"
...
overlay accesslog logdb cn=accesslog logops writes logsuccess TRUE logpurge 07+00:00 01+00:00
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
On Tuesday 03 October 2006 20:49, Robert Petkus wrote: ...
I hate making these things long because folks lose interest and stop reading but my environment is complex so:
Along with the main database, I am also using monitor and accesslog. Recently I began storing ssh public keys in LDAP for use with ssh-lpk. This past weekend ~15k accounts were added to LDAP and maybe 700 ssh keys (I manage LDAP not account management..). Replication failed on 2 nodes. I noticed on these nodes incoherency because I was using an outdated custom schema file (my fault) so I decided to wipe the database and reload it from backup. Not a big deal but I notice that my nightly slapcat ldifs (slapcat -n 2 -l ldap.ldif) are polluted with accesslog entries that *replace* the original entries. For example, my account dn won't include, say, sshPublicKey, but I'd see a reqMod entry with this attribute.
I can see every dn with a ldapsearch but am missing many dns using slapcat. Obversely, when I do slapcat, I get dn attributes from accesslog that I can't see with ldapsearch. It looks like some weird cross-pollination of the 2 databases.
Maybe there is something I am missing in my config. Here is a snippet -- the full config is available upon request.
Thanks!
...
database monitor
database bdb suffix cn=changelog rootdn cn=changelog rootpw secret directory /var/lib/accesslog index reqStart eq index reqAuthzID eq index reqDN eq index reqMod eq overlay accesslog logdb cn=changelog logops writes
database bdb suffix "dc=bnl,dc=gov" rootdn "cn=admin,dc=bnl,dc=gov" rootpw {SSHA}secret
directory /var/lib/ldap
sizelimit unlimited cachesize 500000 idlcachesize 500000
Maybe I'm confused but doesn't the accesslog overlay belong in the main database definition? At least that's what I have:
database hdb suffix "cn=log" .....
database bdb suffix "dc=rentec,dc=com" .... overlay accesslog .... logdb cn=log logops writes logpurge 30+00:00:00 01+00:00:00 ....
Karsten.
Karsten Künne wrote:
Maybe I'm confused but doesn't the accesslog overlay belong in the main database definition? At least that's what I have:
database hdb suffix "cn=log" .....
database bdb suffix "dc=rentec,dc=com" .... overlay accesslog .... logdb cn=log logops writes logpurge 30+00:00:00 01+00:00:00 ....
Karsten.
Yes, actually that's what I have, too. I must have cut and pasted wrong. Still, they're 2 distinct databases.
database monitor .................... database bdb suffix cn=changelog rootdn cn=changelog rootpw secret directory /var/lib/accesslog index reqStart eq index reqAuthzID eq index reqDN eq index reqMod eq
database bdb suffix "dc=bnl,dc=gov" rootdn "cn=admin,dc=bnl,dc=gov" rootpw {SSHA}secret
directory /var/lib/ldap .............. overlay accesslog logdb cn=changelog logops writes
On 10/3/06, Robert Petkus rpetkus@bnl.gov wrote:
Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
Was it stopping somewhere, dumping core, giving errors? Try running it with debugging on, or at least give some more details about your specific failure.
On 10/3/06, Robert Petkus rpetkus@bnl.gov wrote:
Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
tried db4_recover yet?
--On Tuesday, October 03, 2006 3:35 PM -0700 Bryan Irvine sparctacus@gmail.com wrote:
On 10/3/06, Robert Petkus rpetkus@bnl.gov wrote:
Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
tried db4_recover yet?
Starting slapd would have run recovery. I thought running slapcat would too. But that's why I wanted to know what type of meltdown happened.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Quanah Gibson-Mount wrote:
--On Tuesday, October 03, 2006 3:35 PM -0700 Bryan Irvine sparctacus@gmail.com wrote:
On 10/3/06, Robert Petkus rpetkus@bnl.gov wrote:
Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
tried db4_recover yet?
Starting slapd would have run recovery. I thought running slapcat would too. But that's why I wanted to know what type of meltdown happened.
Not any more. Due to other complaints about slapcat unexpectedly leaving DB environment files owned by root or some other unintended userID, slapcat no longer performs recovery, it is strictly a read-only command. But yes, slapd would have run recovery if needed.
--On Tuesday, October 03, 2006 4:38 PM -0700 Howard Chu hyc@symas.com wrote:
Quanah Gibson-Mount wrote:
--On Tuesday, October 03, 2006 3:35 PM -0700 Bryan Irvine sparctacus@gmail.com wrote:
On 10/3/06, Robert Petkus rpetkus@bnl.gov wrote:
Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
tried db4_recover yet?
Starting slapd would have run recovery. I thought running slapcat would too. But that's why I wanted to know what type of meltdown happened.
Not any more. Due to other complaints about slapcat unexpectedly leaving DB environment files owned by root or some other unintended userID, slapcat no longer performs recovery, it is strictly a read-only command. But yes, slapd would have run recovery if needed.
Okay. Then that is likely why slapcat failed and ldapsearch worked. ;) I'm guessing if slapcat were done now, it'd work just fine.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Quanah Gibson-Mount wrote:
--On Tuesday, October 03, 2006 4:38 PM -0700 Howard Chu hyc@symas.com wrote:
Quanah Gibson-Mount wrote:
--On Tuesday, October 03, 2006 3:35 PM -0700 Bryan Irvine sparctacus@gmail.com wrote:
On 10/3/06, Robert Petkus rpetkus@bnl.gov wrote:
Folks, We had a major meltdown of LDAP this morning. I understood why, but the problem was restoring the database. slapcat, no matter how invoked, would simply not dump the full contents of the database. I needed to do a ldapsearch -L. This one doesn't make sense to me -- any ideas?? openldap-2.3.24 db4-4.2.52
tried db4_recover yet?
Starting slapd would have run recovery. I thought running slapcat would too. But that's why I wanted to know what type of meltdown happened.
Not any more. Due to other complaints about slapcat unexpectedly leaving DB environment files owned by root or some other unintended userID, slapcat no longer performs recovery, it is strictly a read-only command. But yes, slapd would have run recovery if needed.
Okay. Then that is likely why slapcat failed and ldapsearch worked. ;) I'm guessing if slapcat were done now, it'd work just fine.
But, I guess, then slapcat should have complained and not run at all because the database couldn't be used without recover.
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati@sys-net.it ------------------------------------------
openldap-software@openldap.org
-
Bryan Irvine -
Howard Chu -
Karsten Künne -
matthew sporleder -
Pierangelo Masarati -
Quanah Gibson-Mount -
Robert Petkus