- openldap-bugs - openldap.org

Re: (ITS#7378) Slapd hangs on bdb write lock
by michael＠stroeder.com 02 Sep '12

02 Sep '12

This is a cryptographically signed message in MIME format. --------------ms080100030105010600070605 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable A couple of days ago I had a hang with OpenLDAP 2.4.32 / back-hdb running= on Debian Squeeze, self-compiled against BDB 4.8.30. It seemed Database was locked as restarting slapd of even rebooting OS did not help. Unfortunate= ly I had to bring up the system as fast as possible and could not examine the = problem. The system has only 200 entries and not much load yet. I had renamed entr= ies with web2ldap when all 4 masters (4-way MMR) locked up one after the othe= r. So there seem to be lockup problems in 2.4.32. --------------ms080100030105010600070605 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIILHzCC BT8wggQnoAMCAQICDwCmSwABAAIAivjZQ8SBvzANBgkqhkiG9w0BAQUFADB8MQswCQYDVQQG EwJERTEcMBoGA1UEChMTVEMgVHJ1c3RDZW50ZXIgR21iSDElMCMGA1UECxMcVEMgVHJ1c3RD ZW50ZXIgQ2xhc3MgMSBMMSBDQTEoMCYGA1UEAxMfVEMgVHJ1c3RDZW50ZXIgQ2xhc3MgMSBM MSBDQSBJWDAeFw0xMjA2MDYxOTAyMTZaFw0xMzA2MDcxOTAyMTZaMCgxCzAJBgNVBAYTAkRF MRkwFwYDVQQDDBBNaWNoYWVsIFN0csO2ZGVyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEAxXZGav40rnGNLxEggBW94MILWHlfC8a23Jew5U1gPlfRTXOjjzmoaZ1uCyGdgF6M VvuO9T1aTQNGH+OdeGe3P7Tfc/NsLJFJ2wtd8blvhmodUgse2eypiWjNOd4gZuhalBhgsQ0K b5D6/1foghII4E264iZlJ7AJ+UYcO+GxvFWT0YMTbLckgDkZk7c3qwTozdhYvXarvqx+8Ou/ kuxpQQhac/ebzxpu0N+RHSf2KIUS0g0tEGnPtGv6iL+9QNHc4JKo9Y9KKVw3tQy+Re+FQLxB 1fPE5F+qxuD3AUENpOwkMsqWLM94ohtx3CFqLpxfUPrnKFLAHOhHEbByYGvFPwIDAQABo4IC EDCCAgwwgaUGCCsGAQUFBwEBBIGYMIGVMFEGCCsGAQUFBzAChkVodHRwOi8vd3d3LnRydXN0 Y2VudGVyLmRlL2NlcnRzZXJ2aWNlcy9jYWNlcnRzL3RjX2NsYXNzMV9MMV9DQV9JWC5jcnQw QAYIKwYBBQUHMAGGNGh0dHA6Ly9vY3NwLml4LnRjY2xhc3MxLnRjdW5pdmVyc2FsLWkudHJ1 c3RjZW50ZXIuZGUwHwYDVR0jBBgwFoAU6bgoHUbP/M34TpvF7ktg69g7P9EwDAYDVR0TAQH/ BAIwADBKBgNVHSAEQzBBMD8GCSqCFAAsAQEBATAyMDAGCCsGAQUFBwIBFiRodHRwOi8vd3d3 LnRydXN0Y2VudGVyLmRlL2d1aWRlbGluZXMwDgYDVR0PAQH/BAQDAgTwMB0GA1UdDgQWBBS2 KAWfTfgJ/JQ63qLGwTXYLnI+LzBiBgNVHR8EWzBZMFegVaBThlFodHRwOi8vY3JsLml4LnRj Y2xhc3MxLnRjdW5pdmVyc2FsLWkudHJ1c3RjZW50ZXIuZGUvY3JsL3YyL3RjX0NsYXNzMV9M MV9DQV9JWC5jcmwwMwYDVR0lBCwwKgYIKwYBBQUHAwIGCCsGAQUFBwMEBggrBgEFBQcDBwYK KwYBBAGCNxQCAjAfBgNVHREEGDAWgRRtaWNoYWVsQHN0cm9lZGVyLmNvbTANBgkqhkiG9w0B AQUFAAOCAQEAQ3bvVUpEq+cQrLpcogyt5BJNk/WvUvOHqhzyj28M9pg9hcDl1+MYl5qqj6tR GSTLPQZyf287pcmbMwbcTGZO/gbW9v7RYcut6RauWdwKMCUmKC3J4fVfDq9ZETA2WOV68ef4 B3Gzdhghsbp3Rhp5dDmrCVKAHlafm6ZwJrEQ9P76fxnQZzRLgeKpZep5ePH5YHUB3+YaOQvJ FG0bOXvfHhRiRG7/HW2G+yDgjHSxDz8AFzMWL/RFePqZ4pn6T/SM/qU6WEpW39MWyJNoH/Kx QDYK8gGYuesn1ciMCTnjrvZQj0fonGTO4SfWekJRkuGrJ7dYSZRjYbDcWBBkdFLWzzCCBdgw ggTAoAMCAQICDgboAAEAAkqWLSQM/sXJMA0GCSqGSIb3DQEBBQUAMHkxCzAJBgNVBAYTAkRF MRwwGgYDVQQKExNUQyBUcnVzdENlbnRlciBHbWJIMSQwIgYDVQQLExtUQyBUcnVzdENlbnRl ciBVbml2ZXJzYWwgQ0ExJjAkBgNVBAMTHVRDIFRydXN0Q2VudGVyIFVuaXZlcnNhbCBDQSBJ MB4XDTA5MTEwMzE0MDgxOVoXDTI1MTIzMTIxNTk1OVowfDELMAkGA1UEBhMCREUxHDAaBgNV BAoTE1RDIFRydXN0Q2VudGVyIEdtYkgxJTAjBgNVBAsTHFRDIFRydXN0Q2VudGVyIENsYXNz IDEgTDEgQ0ExKDAmBgNVBAMTH1RDIFRydXN0Q2VudGVyIENsYXNzIDEgTDEgQ0EgSVgwggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC75pBuz2Lp6QuqthDVR+V8XSsncZpozVVt 5KLv5P7yemMRwleKyH3PjmYfZUVL64Biab1GjovFblqVGCrep/EfdRonq20yU+P7TVhiLP8Z 5cegDZotIYhZhM0d8cPIij6w5d4IJM/8QCy6QSOUu4ASiTVItoYE4AFPjLqpmPwcie0fiqHH hpgmHnJla/7PZdkMZEsaCfVDEWBmJuMzVprJPT40anjG5VBLyM2I5DlsUCaeQCy2O3w3sqf1 3dyzUcv03IICuNc63towXA31Qt0TaVNU6YAmQjMepdfMbspmCZ+G8D2+xophEPPR/1vkstst smUMqX0XrLonTUJczglPAgMBAAGjggJZMIICVTCBmgYIKwYBBQUHAQEEgY0wgYowUgYIKwYB BQUHMAKGRmh0dHA6Ly93d3cudHJ1c3RjZW50ZXIuZGUvY2VydHNlcnZpY2VzL2NhY2VydHMv dGNfdW5pdmVyc2FsX3Jvb3RfSS5jcnQwNAYIKwYBBQUHMAGGKGh0dHA6Ly9vY3NwLnRjdW5p dmVyc2FsLUkudHJ1c3RjZW50ZXIuZGUwHwYDVR0jBBgwFoAUkqR1LKSevoFE63n8isWVpesQ dXMwEgYDVR0TAQH/BAgwBgEB/wIBADBSBgNVHSAESzBJMAYGBFUdIAAwPwYJKoIUACwBAQEB MDIwMAYIKwYBBQUHAgEWJGh0dHA6Ly93d3cudHJ1c3RjZW50ZXIuZGUvZ3VpZGVsaW5lczAO BgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFOm4KB1Gz/zN+E6bxe5LYOvYOz/RMIH9BgNVHR8E gfUwgfIwge+ggeyggemGRmh0dHA6Ly9jcmwudGN1bml2ZXJzYWwtSS50cnVzdGNlbnRlci5k ZS9jcmwvdjIvdGNfdW5pdmVyc2FsX3Jvb3RfSS5jcmyGgZ5sZGFwOi8vd3d3LnRydXN0Y2Vu dGVyLmRlL0NOPVRDJTIwVHJ1c3RDZW50ZXIlMjBVbml2ZXJzYWwlMjBDQSUyMEksTz1UQyUy MFRydXN0Q2VudGVyJTIwR21iSCxPVT1yb290Y2VydHMsREM9dHJ1c3RjZW50ZXIsREM9ZGU/ Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlPzANBgkqhkiG9w0BAQUFAAOCAQEAOcjE m+6+mO5Icm+N53G2DpCM07LBFSGoRpBoX0oE8TrJaIQh2KXmBHVdn9LU8kt3QzLclctgvwJV 0KwcsMUUl5tlCsMPpR3s2Ek5lbWpvvr0HqtW56blAQiINV9nBd1EJFASIkRjefGbV2nOq9Yz UU+N8HA7jq1ROhd/NZZraGhjthwKyfjfHV7PKxGlY+3M0MbTIG+q/GhIfm0euDpFqhKG88e9 ALXr/uoSn3MzeOcoOWjTpW3adtFO4VWVgKbgG7jNrFbvRVlHmFLbOm4msjE5aXWxLiTwpJ2X iF4zKca1vAdAOgw9us90jEtOeiH6GzjNxEMvb7TfeO6Zkuc6HDGCA84wggPKAgEBMIGPMHwx CzAJBgNVBAYTAkRFMRwwGgYDVQQKExNUQyBUcnVzdENlbnRlciBHbWJIMSUwIwYDVQQLExxU QyBUcnVzdENlbnRlciBDbGFzcyAxIEwxIENBMSgwJgYDVQQDEx9UQyBUcnVzdENlbnRlciBD bGFzcyAxIEwxIENBIElYAg8ApksAAQACAIr42UPEgb8wCQYFKw4DAhoFAKCCAhMwGAYJKoZI hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTIwOTAyMTMxMTU0WjAjBgkq hkiG9w0BCQQxFgQUCNHgSamIXCRCd4ANj7PKCgRv3BwwbAYJKoZIhvcNAQkPMV8wXTALBglg hkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggq hkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBoAYJKwYBBAGCNxAEMYGSMIGP MHwxCzAJBgNVBAYTAkRFMRwwGgYDVQQKExNUQyBUcnVzdENlbnRlciBHbWJIMSUwIwYDVQQL ExxUQyBUcnVzdENlbnRlciBDbGFzcyAxIEwxIENBMSgwJgYDVQQDEx9UQyBUcnVzdENlbnRl ciBDbGFzcyAxIEwxIENBIElYAg8ApksAAQACAIr42UPEgb8wgaIGCyqGSIb3DQEJEAILMYGS oIGPMHwxCzAJBgNVBAYTAkRFMRwwGgYDVQQKExNUQyBUcnVzdENlbnRlciBHbWJIMSUwIwYD VQQLExxUQyBUcnVzdENlbnRlciBDbGFzcyAxIEwxIENBMSgwJgYDVQQDEx9UQyBUcnVzdENl bnRlciBDbGFzcyAxIEwxIENBIElYAg8ApksAAQACAIr42UPEgb8wDQYJKoZIhvcNAQEBBQAE ggEAn/eQg7NrUAmmHsFX+FOpynezJ9ocm/9InAnbWvoHFvdbqKYZIRRmu+aZ1a6q3irVg1FX YrzH+OivltVXyvoQvt7WCMeUjiQRjM0PuRd0YlUGu/8qvEVPtcv8i3K/v74MIibRUpItxNes 47NssmY88640oWdfBqkI/KQS44c5rjBADFwPMMXAbWptp4AXoit1MaMTrsPt+2+O2iXAVREB IJkffN1lb6zeizUzWc/H3iPvVMszLZXdTOh/hknsHJHZmwsEqNzNHYhRgYkPoYiiHyi+faa/ mqPSc+22TqsnWL1wYE0HhG3uVWg1fUf3UyhUJ8/VXwmNe9ucfiQI9Kk5rwAAAAAAAA== --------------ms080100030105010600070605--

1 0

Re: (ITS#7378) Slapd hangs on bdb write lock
by nikolai＠net24.co.nz 02 Sep '12

02 Sep '12

I haven't yet - I wanted to collect information before making any changes. I did look at that fix and wasn't confident it would solve our problem. You're right though - I need to test it to rule it out. I will upgrade all the servers to 2.4.32 and report back. On 2/09/2012, at 7:07 AM, Quanah Gibson-Mount wrote: > --On Saturday, September 01, 2012 1:46 PM +0000 nikolai(a)net24.co.nz wrote: > >> Full_Name: Nikolai Schupbach >> Version: 2.4.31 >> OS: FreeBSD >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (202.78.158.60) > > Have you confirmed this isn't the same thing ITS#7222, fixed in OpenLDAP 2.4.32? > > --Quanah > > > > -- > > Quanah Gibson-Mount > Sr. Member of Technical Staff > Zimbra, Inc > A Division of VMware, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7378) Slapd hangs on bdb write lock
by quanah＠zimbra.com 01 Sep '12

01 Sep '12

--On Saturday, September 01, 2012 1:46 PM +0000 nikolai(a)net24.co.nz wrote: > Full_Name: Nikolai Schupbach > Version: 2.4.31 > OS: FreeBSD > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (202.78.158.60) Have you confirmed this isn't the same thing ITS#7222, fixed in OpenLDAP 2.4.32? --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7378) Slapd hangs on bdb write lock
by nikolai＠net24.co.nz 01 Sep '12

01 Sep '12

Full_Name: Nikolai Schupbach Version: 2.4.31 OS: FreeBSD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (202.78.158.60) We are experiencing frequent hangs in slapd. Once hung we can continue to connect, but all searches will just hang indefinitely until we kill -9 the slapd process and restart it. The directory is used for mail routing and we have been migrating to it from an existing directory server over the last 3 weeks - we have noted the busier the directory becomes the more often it hangs (now once every 2 days). We have one master and 10 syncrepl read only replicas - the master is used mainly for writes and has not hung yet, but most of the replicas have hung at least once. The replicas receive anywhere between 50 to 300 searches/sec, while the master would only get 1/sec. There are 45k entries in the directory. We are running: FreeBSD 8.3/9.0 x64 OpenLDAP 2.4.31 Berkeley DB 4.6.21 The old directory we are migrating from has the same load and is also running OpenLDAP, but has been rock solid for 5 years. It is running Berkeley DB 4.3.29 and OpenLDAP 2.3.27. We have managed to collect db_stat lock information, which indicates the same issue each time - a write lock on dn2id.bdb. Locks grouped by object: Locker Mode Count Status ----------------- Object --------------- 8000a85e READ 1 HELD 0xb26c8 len: 9 data: 60xa800000000000000 8a READ 1 HELD id2entry.bdb handle 0 8c READ 1 HELD dn2id.bdb handle 0 96 READ 1 HELD objectClass.bdb handle 0 93 READ 1 HELD entryCSN.bdb handle 0 90 READ 1 HELD entryUUID.bdb handle 0 8000a85f WRITE 4 HELD dn2id.bdb page 219 80000782 READ 1 HELD dn2id.bdb page 768 80000a45 READ 1 HELD dn2id.bdb page 768 80000b9e READ 1 HELD dn2id.bdb page 768 800006a0 READ 1 HELD dn2id.bdb page 768 80000771 READ 1 HELD dn2id.bdb page 768 80000534 READ 1 HELD dn2id.bdb page 768 80000a44 READ 1 HELD dn2id.bdb page 768 80000641 READ 1 HELD dn2id.bdb page 768 80001049 READ 1 HELD dn2id.bdb page 768 8000104a READ 1 HELD dn2id.bdb page 768 80001048 READ 1 HELD dn2id.bdb page 768 80000783 READ 1 HELD dn2id.bdb page 768 80000535 READ 1 HELD dn2id.bdb page 768 8000066e READ 1 HELD dn2id.bdb page 768 80000697 READ 1 HELD dn2id.bdb page 768 8000a85f READ 1 HELD dn2id.bdb page 768 8000a85e READ 1 HELD 0xb19a8 len: 9 data: 40xa800000000000000 8000a85f READ 1 HELD dn2id.bdb page 933 8000a85f WRITE 2 HELD dn2id.bdb page 933 80001047 WRITE 1 HELD dn2id.bdb page 559 80000782 READ 1 WAIT dn2id.bdb page 559 80000a45 READ 1 WAIT dn2id.bdb page 559 80000b9e READ 1 WAIT dn2id.bdb page 559 800006a0 READ 1 WAIT dn2id.bdb page 559 80000771 READ 1 WAIT dn2id.bdb page 559 80000534 READ 1 WAIT dn2id.bdb page 559 80000a44 READ 1 WAIT dn2id.bdb page 559 80000641 READ 1 WAIT dn2id.bdb page 559 80001049 READ 1 WAIT dn2id.bdb page 559 8000104a READ 1 WAIT dn2id.bdb page 559 80001048 READ 1 WAIT dn2id.bdb page 559 80000783 READ 1 WAIT dn2id.bdb page 559 80000535 READ 1 WAIT dn2id.bdb page 559 8000066e READ 1 WAIT dn2id.bdb page 559 80000697 READ 1 WAIT dn2id.bdb page 559 8000a85f READ 1 WAIT dn2id.bdb page 559 8000a85f READ 2 HELD dn2id.bdb page 1362 8000a85f WRITE 2 HELD dn2id.bdb page 1362 8000a85f READ 2 HELD dn2id.bdb page 1353 8000a85f WRITE 2 HELD dn2id.bdb page 1353 b6 READ 1 HELD uid.bdb handle 0 a5 READ 1 HELD mail.bdb handle 0 af READ 1 HELD mailLocalAddress.bdb handle 0 9b READ 1 HELD miLoginid.bdb handle 0 aa READ 1 HELD mailHost.bdb handle 0 bb READ 1 HELD miDomainName.bdb handle 0 c0 READ 1 HELD mpMailHost.bdb handle 0 a0 READ 1 HELD mpMailUserType.bdb handle 0 We have also collected the backtrace for all the threads which I have uploaded to: ftp://ftp.openldap.org/incoming/nikolai-gdb-120902.txt The full db_stat output is located at: ftp://ftp.openldap.org/incoming/nikolai-dbstat-120902.txt Our DB_CONFIG: # One 512MB cache set_cachesize 0 536870912 1 # Transaction Log settings set_lg_regionmax 1048576 set_lg_max 10485760 set_lg_bsize 2097152 set_flags DB_LOG_AUTOREMOVE # Increase lock maximums set_lk_max_locks 2000 set_lk_max_lockers 2000 set_lk_max_objects 2000 Our slapd.conf on our replicas: # Load the following schema files include /usr/local/etc/openldap/schema/core.schema include /usr/local/etc/openldap/schema/cosine.schema include /usr/local/etc/openldap/schema/nis.schema include /usr/local/etc/openldap/schema/inetorgperson.schema include /usr/local/etc/openldap/schema/misc.schema include /usr/local/etc/openldap/schema/mirapoint.schema include /usr/local/etc/openldap/schema/smp.schema # Runtime settings for slapd pidfile /var/run/openldap/slapd.pid argsfile /var/run/openldap/slapd.args loglevel none # TLS security options for slapd. TLSCipherSuite HIGH TLSCACertificateFile /usr/local/etc/openldap/tls/ca-cert.pem TLSCertificateFile /usr/local/etc/openldap/tls/server-cert.pem TLSCertificateKeyFile /usr/local/etc/openldap/tls/server-key.pem # This option configures one or more hashes to be used in generation # of user passwords stored in the userPassword attribute during # processing of LDAP Password Modify Extended Operations (RFC 3062). password-hash {SSHA} # Load dynamic backend modules: modulepath /usr/local/libexec/openldap moduleload back_bdb moduleload back_monitor # Do not limit size or time of requests. sizelimit unlimited timelimit unlimited # Require authentication prior to directory operations require authc ############################################################################### # BDB Database Definitions # # The following configuration directives relate to bdb database definitions ############################################################################### # The remaining configuration directives relate to bdb database definitions database bdb suffix "o=top" rootdn "cn=root,o=top" # Cleartext passwords, especially for the rootdn, should # be avoid. See slappasswd(8) and slapd.conf(5) for details. rootpw {SSHA}********** # The database directory must exist prior to running slapd and # should only be accessible by the slapd and slap tools. directory /var/db/openldap-data # Indices to maintain index cn eq,sub,pres index entryUUID eq index entryCSN eq index mail eq,sub,pres index mailHost eq index mailLocalAddress eq,sub,pres index miDomainName eq,sub index miLoginId eq,pres index mpMailHost eq index mpMailUserType eq index mpSystemRole eq index objectClass eq,pres index uid eq,pres # Specify the number of entries which should be held in memory cachesize 200000 # Set transactional checkpoint checkpoint 512 60 ############################################################################### # LDAP Sync Replication # # A unique replica id number is required for each replication client ############################################################################### # LDAP sync replication settings syncrepl rid=36 provider=ldaps://ldapmaster/ type=refreshAndPersist retry=30,+ searchbase="o=top" filter="(objectClass=*)" scope=sub attrs="*" sizelimit=unlimited timelimit=unlimited schemachecking=off bindmethod=simple binddn="cn=replica,ou=users,ou=directory,o=top" credentials=********** # Where to refer ldap updates to updateref ldaps://ldapmaster/ ############################################################################### # LDAP Statistics # # The OpenLDAP server can be configured to provide real time performance # statistics through the monitor branch. ############################################################################### # Enable the statistics monitoring database database monitor # Allow access to monitoring user only access to dn.subtree="cn=monitor" by dn.exact="cn=monitor,ou=users,ou=directory,o=top" read by * none Sincerely, Nikolai Schupbach

1 0

Re: (ITS#7364) mdb: clean up POSIX semaphores on environment close.
by h.b.furuseth＠usit.uio.no 01 Sep '12

01 Sep '12

Reopening this. This is worse with a database with is intended to be used by different users (A and B): A opens the DB and creates the semaphores with e.g. mode 0660. B opens it, A closes, B closes - and fails sem_unlink() which only A and root can do. Next, B (or C) fails mdb_env_open() because sem_unlink() fails again. The work-around I can think of is a "multi-uid" mode which instead resets the semaphore with sem_post() if sem_getvalue() returns 0. I don't know how ugly that is considered to be. Could ask comp.programming.unix, or check what Berkeley DB does. This mode should use mode 0666 for the semaphores (temporarily setting umask 0, yuck), or it should not sem_unlink() since next user may create the semaphores with a group which gives the wrong users access. Or root may give it root-only access, as in the original report. If not unlinking, we need a special "remove database + semaphores" API call. Or use some other sync primitive, like file locks. The Linux manual says these work over NFS though, which sounds like they must be rather slow. flock() does not, but this time mdb would need to seek first. I don't know which sync variants BSD offers. Other matters with the current implementation - I'll patch these: mdb_env_excl_lock() need not retry getting a non-exclusive lock when closing. mdb_env_close() can pass *excl = -1 to tell it not to. mdb_env_setup_locks() can sem_unlink both semaphores before doing anything else, so that reopening a database as root will clean up. Drop the error checks of sem_unlink (so both get called), instead use O_EXCL in sem_open(,O_CREAT,,). Unless I'm missing something, the error checks just work like an emulation of O_EXCL anyway.

1 0

Re: (ITS#7363) libmdb should use POSIX semaphores on non-apple BSD systems too.
by h.b.furuseth＠usit.uio.no 01 Sep '12

01 Sep '12

Chris Mikkelson writes: > The *BSD systems define PTHREAD_PROCESS_SHARED but do > not implement sharable pthread mutexes. The flag name > is part of the API, but implementation of the behavior > it requests is optional. Yes, sorry about that. How about something like a "__BSD" preprocessor symbol to replace "BSD"? There surely is one. It'll start with an underscore followed by either another underscore or an uppercase letter. -- Hallvard

1 0

Re: (ITS#7377) Poor libmdb error handling
by Hallvard B Furuseth 01 Sep '12

01 Sep '12

Howard Chu wrote: >h.b.furuseth(a)usit.uio.no wrote: >> Maybe add a flag to request error paranoia, like setting >> PTHREAD_MUTEX_ERRORCHECK. Trying to catch if a current mdb >> process has lost is locks. (The fcntl F_SETLK is lost if the >> process closes another file descriptor to the same file, >> e.g. after some other module in the process opened and closed the >> mdb lock file.) Catching a change to the pid from getpid(). > > That's really not interesting to me. No other module has any business > opening and reading our lock file. I skipped a step there. The other module could also be using mdb. "other module" = "a program component which the module using this MDB_env does not know about". They may even be linking different libmdb.so's, so the 'libmdb's can't notify each other via static vars. There are a few other use cases like a monolith program which also can backup a directory - which the user could point at the mdb dir. Anyway, a CAVEATS section in the doc is the obvious fix. > We haven't bothered with this for alock either, and > it hasn't been a problem. Note that this ITS is about libmdb, not back-mdb. >> The fork() restriction needs to be documented, as does "don't open >> or close the same database twice in the same process". > > Go ahead. Hmm. I suggested a too pessimistic caveats section once, I can dig that up and try again.

1 0

Re: (ITS#7377) Poor libmdb error handling
by hyc＠symas.com 01 Sep '12

01 Sep '12

h.b.furuseth(a)usit.uio.no wrote: > Full_Name: Hallvard B Furuseth > Version: 2.4.32 > OS: Linux x86_64 > URL: > Submission from: (NULL) (193.69.163.163) > Submitted by: hallvard > > > libmdb omits necessary checks for system call errors, and does > not always clean up after the errors it does detect. > > Also mdb could catch more user errors - including "cannot happen" > mutex errors from mutex lock/unlock, since some threading user > errors are be non-obvious from the doc. > > At least sem_wait() needs an EINTR loop. POSIX says "The > sem_wait() function is interruptible by the delivery of a signal." > The Linux manpage adds that a signal handler always interrupts it > regardless of sigaction SA_RESTART. I don't know about BSD. > > Some error checks first need updates to the Windows emulation, or > will look cleaner that way. E.g. pthread_key_create(). > > I'll fix mdb_env_open()'s error cleanup, at least. > (Hyc, that moves cleanup out of mdb_env_setup_locks(). Any > preference for using ~25 exit points 'return <ErrCode() or rc>' > vs. ~25 'goto <failure label>' which does the same?) In general I prefer that functions have only one exit point, so multiple gotos is preferable. > Maybe add a flag to request error paranoia, like setting > PTHREAD_MUTEX_ERRORCHECK. Trying to catch if a current mdb > process has lost is locks. (The fcntl F_SETLK is lost if the > process closes another file descriptor to the same file, > e.g. after some other module in the process opened and closed the > mdb lock file.) Catching a change to the pid from getpid(). That's really not interesting to me. No other module has any business opening and reading our lock file. We haven't bothered with this for alock either, and it hasn't been a problem. Change of pid... not worth bothering. Just add a note to the docs. > Regarding the pid: Repeating getpid() - in mdb_txn_renew0() and > mdb_env_close() - is useless since mdb cannot survive fork() > anyway. fork drops fcntl locks. Might as well store the pid (or > just some unique "environment id") in MDB_env and use that. Agreed. I had a patch here that does just that, seems it got lost in some other edits. > OTOH > some strategically placed getpid()s can then catch the user error > of having forked, and set e.g. MDB_FATAL_ERROR. Not really interesting. > The fork() restriction needs to be documented, as does "don't open > or close the same database twice in the same process". Go ahead. > mdb_env_open() should catch being called twice. Except, should > it try to allow a 2nd call if 1st mdb_env_open() failed? It can > easily do that if 1st attempt failed before reading params from > the DB files, but fail if the 1st open got that far. Called twice on the same env handle? Again, not interesting. As a rule, we should of course check for failure return codes from normal operation. We should not add extensive checks for programmer error/misuse of API. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7377) Poor libmdb error handling
by openldap-its＠OpenLDAP.org 01 Sep '12

01 Sep '12

I messed up last reply headers somehow. Trying again. I wrote: > Also mdb could catch more user errors - including "cannot happen" > mutex errors from mutex lock/unlock, since some threading user > errors are be non-obvious from the doc. That is, "cannot happen" when there is no user error. > The fork() restriction needs to be documented, as does "don't > open or close the same database twice in the same process". Correction, "don't have the same database open twice at the same time in the same process".

1 0

Re: (ITS#7377) Poor libmdb error handling
by openldap-its＠OpenLDAP.org 01 Sep '12

01 Sep '12

I wrote: > Also mdb could catch more user errors - including "cannot happen" > mutex errors from mutex lock/unlock, since some threading user > errors are be non-obvious from the doc. That is, "cannot happen" when there is no user error. > The fork() restriction needs to be documented, as does "don't > open or close the same database twice in the same process". Correction, "don't have the same database open twice at the same time in the same process".

1 0

(ITS#7377) Poor libmdb error handling
by h.b.furuseth＠usit.uio.no 01 Sep '12

01 Sep '12

Full_Name: Hallvard B Furuseth Version: 2.4.32 OS: Linux x86_64 URL: Submission from: (NULL) (193.69.163.163) Submitted by: hallvard libmdb omits necessary checks for system call errors, and does not always clean up after the errors it does detect. Also mdb could catch more user errors - including "cannot happen" mutex errors from mutex lock/unlock, since some threading user errors are be non-obvious from the doc. At least sem_wait() needs an EINTR loop. POSIX says "The sem_wait() function is interruptible by the delivery of a signal." The Linux manpage adds that a signal handler always interrupts it regardless of sigaction SA_RESTART. I don't know about BSD. Some error checks first need updates to the Windows emulation, or will look cleaner that way. E.g. pthread_key_create(). I'll fix mdb_env_open()'s error cleanup, at least. (Hyc, that moves cleanup out of mdb_env_setup_locks(). Any preference for using ~25 exit points 'return <ErrCode() or rc>' vs. ~25 'goto <failure label>' which does the same?) Maybe add a flag to request error paranoia, like setting PTHREAD_MUTEX_ERRORCHECK. Trying to catch if a current mdb process has lost is locks. (The fcntl F_SETLK is lost if the process closes another file descriptor to the same file, e.g. after some other module in the process opened and closed the mdb lock file.) Catching a change to the pid from getpid(). Regarding the pid: Repeating getpid() - in mdb_txn_renew0() and mdb_env_close() - is useless since mdb cannot survive fork() anyway. fork drops fcntl locks. Might as well store the pid (or just some unique "environment id") in MDB_env and use that. OTOH some strategically placed getpid()s can then catch the user error of having forked, and set e.g. MDB_FATAL_ERROR. The fork() restriction needs to be documented, as does "don't open or close the same database twice in the same process". mdb_env_open() should catch being called twice. Except, should it try to allow a 2nd call if 1st mdb_env_open() failed? It can easily do that if 1st attempt failed before reading params from the DB files, but fail if the 1st open got that far.

1 0

Re: (ITS#7375) slapo-dynlist is very slow
by hyc＠symas.com 31 Aug '12

31 Aug '12

rouazana(a)linagora.com wrote: > Note that using "sortvals uniqueMember" configuration allows dynlist to > work in a correct time (about 3s for a base search on the group). > > So I don't know I you want this issue to be fixed or not. Closing this ITS. > > Regards, > Raphaël Ouazana. > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7375) slapo-dynlist is very slow
by rouazana＠linagora.com 31 Aug '12

31 Aug '12

Note that using "sortvals uniqueMember" configuration allows dynlist to work in a correct time (about 3s for a base search on the group). So I don't know I you want this issue to be fixed or not. Regards, Raphaël Ouazana.

1 0

(ITS#7249) slapd segfault with memberof overlay on frontend db
by c.phan＠active-system.com 31 Aug '12

31 Aug '12

Full_Name: Chinh Phan Version: 2.4.32 OS: Fedora 16 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (84.96.74.38) Hello, I'm still having a segfault error even with the fix suggested by Howard Chu (http://www.openldap.org/lists/openldap-bugs/201204/msg00171.html): slapd[2812]: segfault at 7f8d0b7fffd8 ip 00000000004377b0 sp 00007f8d0b7fffe0 error 6 in slapd[400000+1c6000] Does anyone know if this issue has been addressed or is planned to be fixed in the next release? Appreciate your feedback. chinh

1 0

(ITS#7375) slapo-dynlist is very slow
by raphael.ouazana＠linagora.com 31 Aug '12

31 Aug '12

Full_Name: Raphael Ouazana Version: 2.4.31 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (88.173.78.196) Hi, I'm using dynlist to generate a dynamic group with ~50k users. The LDAP request written in the memberURL attribute takes about 8 seconds to execute. But a base search request on the group takes more than 2 minutes to execute. So I think dynlist should be optimized. Regards, Raphaël Ouazana.

1 0

(ITS#7374) [PATCH] MozNSS: better file name matching for hashed CA certificate directory
by jvcelak＠redhat.com 29 Aug '12

29 Aug '12

Full_Name: Jan Vcelak Version: git master OS: Linux URL: ftp://ftp.openldap.org/incoming/jvcelak-20120829-moznss-beter-file-name-mat… Submission from: (NULL) (209.132.186.34) This change concerns Mozilla NSS crypto backend: CA certificate files in OpenSSL compatible CACERTDIR were loaded if the file extension was '.0'. However the file name should be 8 letters long certificate hash of the certificate subject name, followed by a numeric suffix which is used to differentiate between two certificates with the same subject name. With this patch, certificate file names are matched correctly (using regular expressions). The attached file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Red Hat. Red Hat has not assigned rights and/or interest in this work to any party. I, Jan Vcelak am authorized by Red Hat, my employer, to release this work under the following terms. Red Hat hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.

1 0

Re: (ITS#7363) libmdb should use POSIX semaphores on non-apple BSD systems too.
by cmikk＠qwest.net 29 Aug '12

29 Aug '12

On Wed, Aug 29, 2012 at 08:07:31AM +0200, Hallvard Breien Furuseth wrote: > We can #ifndef PTHREAD_PROCESS_SHARED instead of BSD, unless someone > #define it without supporting it. Not for MDB_FDATASYNC though. The *BSD systems define PTHREAD_PROCESS_SHARED but do not implement sharable pthread mutexes. The flag name is part of the API, but implementation of the behavior it requests is optional. -- Chris Mikkelson | Quidquid latine dictum sit, altum viditur cmikk(a)qwest.net |

1 0

Re: (ITS#7363) libmdb should use POSIX semaphores on non-apple BSD systems too.
by h.b.furuseth＠usit.uio.no 29 Aug '12

29 Aug '12

Howard Chu writes: > PTHREAD_PROCESS_SHARED is defined just about everywhere, that's not a useful > test. (You only discover at runtime that the mutex creation/init failed...) Argh. Oh well, we should only do the namespace cleanup, then. __BSD__ symbols etc. BTW, I think the lock file should contain a field indicating the sync primitive type, so different compilations of MDB won't use different sync primitives. Which maybe also means we might as well pick the primitive at runtime (try PTHREAD_PROCESS_SHARED first, then sem_open). -- Hallvard

1 0

Re: (ITS#7363) libmdb should use POSIX semaphores on non-apple BSD systems too.
by hyc＠symas.com 29 Aug '12

29 Aug '12

h.b.furuseth(a)usit.uio.no wrote: > We can #ifndef PTHREAD_PROCESS_SHARED instead of BSD, unless someone > #define it without supporting it. Not for MDB_FDATASYNC though. > PTHREAD_PROCESS_SHARED is defined just about everywhere, that's not a useful test. (You only discover at runtime that the mutex creation/init failed...) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7363) libmdb should use POSIX semaphores on non-apple BSD systems too.
by h.b.furuseth＠usit.uio.no 29 Aug '12

29 Aug '12

Please try this patch, I can't. It's split up for readability. Replace defined(BSD) in 1st patch with defined(__BSD__) or whatever, see below. http://folk.uio.no/hbf/its7363-cleanup.txt Also, maybe we can drop __APPLE__ - it may #define __BSD__ too. ANDROID should be something like __ANDROID__. I can't test either. Explanation: defined(BSD) breaks in strict ISO C mode. Is there a symbol like __BSD__ or __BSD to use instead? Namespace-wise, "MDB_USE_POSIX_SEM" would be better. MDB_FDATASYNC should not depend on USE_POSIX_SEM. We can #ifndef PTHREAD_PROCESS_SHARED instead of BSD, unless someone #define it without supporting it. Not for MDB_FDATASYNC though. -- Hallvard

1 0

Re: (ITS#7364) mdb: clean up POSIX semaphores on environment close.
by h.b.furuseth＠usit.uio.no 28 Aug '12

28 Aug '12

Howard Chu writes: > h.b.furuseth(a)usit.uio.no wrote: >> That won't help if the mdb proccess crashes. > > If which process crashes? Sorry, I meant the process which would have removed the semaphore. -- Hallvard

1 0

Re: (ITS#7364) mdb: clean up POSIX semaphores on environment close.
by hyc＠symas.com 28 Aug '12

28 Aug '12

h.b.furuseth(a)usit.uio.no wrote: > cmikk(a)qwest.net writes: >> The patch at: >> >> http://mikk.net/~chris/patches/0002-Remove-POSIX-semaphores-when-the-last-u… >> >> Attempts to upgrade the lockfile lock to exclusive when closing the environment. >> If that upgrade succeeds, it removes the semaphores. > > That won't help if the mdb proccess crashes. If which process crashes? If some other process crashes then this lock will succeed and remove the semaphores. > Sounds like mdb needs some > cleanup API calls. Maybe one which does the same as your patch after > opening the environment - except it does not create any files, and fails > if the exclusive lock fails. Sounds OK. Could add an invocation of it as an option for mdb_stat. mdb_stat needs to be extended anyway. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7364) mdb: clean up POSIX semaphores on environment close.
by h.b.furuseth＠usit.uio.no 28 Aug '12

28 Aug '12

cmikk(a)qwest.net writes: > The patch at: > > http://mikk.net/~chris/patches/0002-Remove-POSIX-semaphores-when-the-last-u… > > Attempts to upgrade the lockfile lock to exclusive when closing the environment. > If that upgrade succeeds, it removes the semaphores. That won't help if the mdb proccess crashes. Sounds like mdb needs some cleanup API calls. Maybe one which does the same as your patch after opening the environment - except it does not create any files, and fails if the exclusive lock fails. -- Hallvard

1 0

Re: (ITS#7373) [PATCH] TLS: do not reuse tls_session if hostname check fails
by jvcelak＠redhat.com 28 Aug '12

28 Aug '12

> Original report: https://bugzilla.redhat.com/show_bug.cgi?id=843056 Sorry, correct one: https://bugzilla.redhat.com/show_bug.cgi?id=852476

1 0

(ITS#7373) [PATCH] TLS: do not reuse tls_session if hostname check fails
by jvcelak＠redhat.com 28 Aug '12

28 Aug '12

Full_Name: Jan Vcelak Version: git master OS: Linux URL: ftp://ftp.openldap.org/incoming/jvcelak-20120828-tls-do-not-reuse-tls_sessi… Submission from: (NULL) (209.132.186.34) If multiple URIs are specified when creating LDAP connection, the connection to the first one succeeds but the hostname verification fails, *tls_session is not dropped, but reused when connecting to the second server. This is a problem with Mozilla NSS backend because another handshake cannot be performed on the same file descriptor and SSL_ForceHandshake() will hang. From this reason, hostname checking was moved into ldap_int_tls_connect() before connection error handling. I have verified for all backends (OpenSSL, GnuTLS, MoznSSS) that this fix does not break hostname verification and that this bug does not present. Original report: https://bugzilla.redhat.com/show_bug.cgi?id=843056 The attached file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Red Hat. Red Hat has not assigned rights and/or interest in this work to any party. I, Jan Vcelak am authorized by Red Hat, my employer, to release this work under the following terms. Red Hat hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs