- openldap-bugs - openldap.org

(ITS#7781) constraint_attribute <attr> <size|count> cannot take 0
by fumiyas＠osstech.co.jp 09 Jan '14

09 Jan '14

Full_Name: SATOH Fumiyasu Version: master OS: URL: ftp://ftp.openldap.org/incoming/0001-constraint_attribute-attr-size-count-c… Submission from: (NULL) (240b:10:2640:bf0:2268:9dff:fec7:5444) The uploaded patch modifies slapo-constraint for constraint_attribute <attr> <size|count> can take 0. This patch depends on a patch attached in ITS#7780. http://www.openldap.org/its/index.cgi?findid=7780

1 0

(ITS#7780) constraint_attribute <attr> <size|count> 0 cause segfault
by fumiyas＠osstech.co.jp 09 Jan '14

09 Jan '14

Full_Name: SATOH Fumiyasu Version: master OS: URL: ftp://ftp.openldap.org/incoming/0001-Fix-segfault-by-constraint_attribute-a… Submission from: (NULL) (240b:10:2640:bf0:2268:9dff:fec7:5444) If slapd.conf has the following config, slapd segfault at startup. overlay constraint constraint_attribute memberUid size 0 # or constraint_attribute memberUid count 0

1 0

Re: (ITS#7771) liblmdb cursor issues
by h.b.furuseth＠usit.uio.no 09 Jan '14

09 Jan '14

Another problem when an xcursor's mc_pg[0] refers to a sub-page but has not been kept updated: The comparison 'mc_pg[i] == mp' below fails and mc_ki[i] does not get modified: if (m3->mc_pg[i] == mp && m3->mc_ki[i] >= mc->mc_ki[i]) { m3->mc_ki[i]++; } Then it's too late to fixup the sub-page pointer later, the xcursor refers to the wrong node in the sub-page. The above code is from mdb_cursor_put(), but there are other similar cases elsewhere. -- Hallvard

1 0

Re: (ITS#7777) Filtering on modifyTimestamp is slower with index
by openldap-technical＠OpenLDAP.org 09 Jan '14

09 Jan '14

This is a cryptographically signed message in MIME format. --------------ms030505010104020400000608 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable The ITS is not for support requests. You should post things like this on = the openldap-technical mailing list. raphael.ouazana(a)linagora.com wrote: > I have a case where using modifyTimestamp index is far slower than with= no > index. > [..] > '(&(modifyTimestamp>=3D20131023163804Z)(modifyTimestamp<=3D201311231638= 04Z))' You might wanna read this old thread: http://www.openldap.org/lists/openldap-technical/201204/msg00116.html Try to use this filter instead: (&(modifyTimestamp>=3D20131023163804Z)(!(modifyTimestamp>=3D2013112316380= 4Z))) Ciao, Michael. --------------ms030505010104020400000608 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFfzCC BXswggNjoAMCAQICAwxOfTANBgkqhkiG9w0BAQUFADB5MRAwDgYDVQQKEwdSb290IENBMR4w HAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmlu ZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAeFw0xMjEw MDIyMDE3MDlaFw0xNDEwMDIyMDE3MDlaMD8xGDAWBgNVBAMUD01pY2hhZWwgU3Ry9mRlcjEj MCEGCSqGSIb3DQEJARYUbWljaGFlbEBzdHJvZWRlci5jb20wggEiMA0GCSqGSIb3DQEBAQUA A4IBDwAwggEKAoIBAQDo2SKth5GhtaDrCyfGtyUG+/hAAa/J52L0NFN4SSRvTtdGf9HfWwwd NCtgae0TVGWk2lKDbXA9d5vmyIiRhuwxd90H6FLErhRBeB9G67qtw87E8WUoXt2DwPQEUTWV hqHpPadlmgFw3+i3TGQQTe3O3W9MMMd4GJNhObem2VGRuCD37OXnzBksTcq0FPJgcWAhe3d/ 0ItOkNWBqgq8Mf3p7WFBhaQ0a27BC/mKtH8fI3kPcS305imPRja69Msq3EwUZBc9ToVp6FRQ NYKjfOBybDUzVkmRZl3H8xutQP2w8Zxb8m5f7Q1BfLLrIFScfYvIDgOERxTCd4lab8+/09XH AgMBAAGjggFEMIIBQDAMBgNVHRMBAf8EAjAAMFYGCWCGSAGG+EIBDQRJFkdUbyBnZXQgeW91 ciBvd24gY2VydGlmaWNhdGUgZm9yIEZSRUUgaGVhZCBvdmVyIHRvIGh0dHA6Ly93d3cuQ0Fj ZXJ0Lm9yZzAOBgNVHQ8BAf8EBAMCA6gwQAYDVR0lBDkwNwYIKwYBBQUHAwQGCCsGAQUFBwMC BgorBgEEAYI3CgMEBgorBgEEAYI3CgMDBglghkgBhvhCBAEwMgYIKwYBBQUHAQEEJjAkMCIG CCsGAQUFBzABhhZodHRwOi8vb2NzcC5jYWNlcnQub3JnMDEGA1UdHwQqMCgwJqAkoCKGIGh0 dHA6Ly9jcmwuY2FjZXJ0Lm9yZy9yZXZva2UuY3JsMB8GA1UdEQQYMBaBFG1pY2hhZWxAc3Ry b2VkZXIuY29tMA0GCSqGSIb3DQEBBQUAA4ICAQC9ouXq3p/bDWMM4tBKgD3tl4HY5H0eECl8 q9/nqk0UL6YeWkrCiQdrDtNPW7DcGqNYtzdgtzmyTr1GhiAX+igrOjdk/ge5NRcQOpONK/4b zrmpQEcIUyxSSDKLWh211/kcFfxxLEiJ5teF4GL8Fc1qbrLP4+DCvJXWfYaaR5NLjZMqm2VP yKTv3qpXWnGohiRkGTwS/11QM2XCfIGdRsQT9a8mO4m2fn2tGPp2TEIoCLrDDrbGVeDWaOWB OIeTrp4wa3Q4OI6yCptJhEqKvjhV96IBRYgM76nTBqsqnDzwxExAyhhWiUS5DunRHOr/+NyF pUpD4883RBLO0g9kUEGOhtZNF1u+8zEL0YgMGvifAom9JEklLOXZuqj0MThypKs/3d/OyOQb 4gURnu6oZwcKZ7LskytWnlRKUxF6o0A8grtmyKkqe14TS7cQbg0NTaIYXPkHR+dfFmb3uEqn BBjvpJXFcEtWI2lQXC/ET+au991pK797ExBOmpQwjIn3SjiW80vw/UoL6DMvqY/6JhVhyNTP MJ2W5AX5kc27DIbVtVGZs8J4AYhuNALJUq9N9Ka7rPRj3RcYDrfehDLOkM5iMnarpmtuOpLK d1SvZhqj/0N/JWGIDpPSTkTFOPP6ZN9I9Rqyf+9NGqb2sjo4DkIiZcHxt735/GJLwus5KLBl 2DGCA6EwggOdAgEBMIGAMHkxEDAOBgNVBAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93 d3cuY2FjZXJ0Lm9yZzEiMCAGA1UEAxMZQ0EgQ2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8G CSqGSIb3DQEJARYSc3VwcG9ydEBjYWNlcnQub3JnAgMMTn0wCQYFKw4DAhoFAKCCAfUwGAYJ KoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwMTA5MTUyNjEwWjAj BgkqhkiG9w0BCQQxFgQUhK88KTaaE97cUDowyV4g0ADaD78wbAYJKoZIhvcNAQkPMV8wXTAL BglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDAN BggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBkQYJKwYBBAGCNxAEMYGD MIGAMHkxEDAOBgNVBAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9y ZzEiMCAGA1UEAxMZQ0EgQ2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJARYS c3VwcG9ydEBjYWNlcnQub3JnAgMMTn0wgZMGCyqGSIb3DQEJEAILMYGDoIGAMHkxEDAOBgNV BAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9yZzEiMCAGA1UEAxMZ Q0EgQ2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJARYSc3VwcG9ydEBjYWNl cnQub3JnAgMMTn0wDQYJKoZIhvcNAQEBBQAEggEAsCtltB7mwSwYu91VjcKwVRXjP+SKJBXo e2uODXEsak3cHvHzRqvMSGAuj5LYyiLB6zCqJ2ra8MY/JtSI5+o1OHHrNO8kWwlgjaPzwyQo YTMeU9zVFXYiij247Ku1jepUT7WAxxiIjwNLJVt4OjpTmsDeOLG3z9f36bEIZLeaA9Dl5jUv 9m5oAqCZFnDNqVISv9MYAZGU8bB8EkxDb1ArSKt3ANFHL8e2ULYOnwExKgoqMyMPhpoepDhc AlPLktZvyq5Sxp4Y35J6Pw0NgJxExOC+iv03w6bSiAUNbSRBrMPXi2Tew7iCA4+Q87DeY5Nh X2fCUrJoStCDcoTm4Qi4OQAAAAAAAA== --------------ms030505010104020400000608--

1 0

Re: (ITS#7779) Index and derived attribute bug
by yann.cezard＠univ-pau.fr 09 Jan '14

09 Jan '14

Sorry, it's a duplicate (#7778) due to a reload in web browser...

1 0

(ITS#7779) Index and derived attribute bug
by yann.cezard＠univ-pau.fr 09 Jan '14

09 Jan '14

Full_Name: Yann Cezard Version: 2.4.38 OS: Debian GNU/Linux Wheezy / amd64 URL: ftp://ftp.openldap.org/incoming/Yann-Cezard-130109.tgz Submission from: (NULL) (2001:660:6701:14:225:64ff:fe92:685b) Hello, I hit a bug with the latest version of OpenLDAP (2.4.38) that was not present, at least in 2.4.17 (made a test on an older server that we were using before I discovered this). This happens on both lmdb and bdb backend (as we were using BDB before upgrading to 2.4.38, I tested with both on 2.4.38 to ensure that the problem was not LMDB related). The problem shows up when you delete the last attribute of en entry, which is derived from another attribute, which has an index on it (not easy to explain...). As an example (complete LDIF files - config/schema + data - provided on the incoming FTP), considering an attribute that is derived from the 'mail' attribute (which is defined in core.schema, and used in InetOrgPerson) : attributetype ( 1.3.6.1.4.1.7135.1.2.1.34 NAME 'supannAutreMail' DESC 'adresses mail non institutionnelles' SUP mail ) and an associated objectClass that use it (called supannPerson in the next example). Now place an index on mail : olcDbIndex: mail pres,eq,sub Now with an entry like this : dn: uid=test,ou=users,dc=example,dc=com uid: test sn: Test cn: Test objectClass: inetOrgPerson objectClass: organizationalPerson objectClass: person objectClass: top objectClass: supannPerson structuralObjectClass: inetOrgPerson entryUUID: 2699b5b4-0d7c-1033-9546-613643720ad1 creatorsName: cn=root,dc=example,dc=com createTimestamp: 20140109131736Z mail: test(a)example.com supannAutreMail: root(a)example.com supannAutreMail: user.test(a)example.com If you make a search with '(mail=*)', the entry matches : ############# $ ldapsearch -x -h ldap-test -b ou=users,dc=example,dc=com -D cn=root,dc=example,dc=com -w test -s one '(mail=*)' dn # extended LDIF # # LDAPv3 # base <ou=users,dc=example,dc=com> with scope oneLevel # filter: (mail=*) # requesting: dn # # test, users, example.com dn: uid=test,ou=users,dc=example,dc=com # search result search: 2 result: 0 Success ############## That's pretty normal. Now, remove both the supannAutreMail attributes from the entry. Run the same ldapsearch : ############## $ ldapsearch -x -h ldap-test -b ou=users,dc=example,dc=com -D cn=root,dc=example,dc=com -w test -s one '(mail=*)' dn # extended LDIF # # LDAPv3 # base <ou=users,dc=example,dc=com> with scope oneLevel # filter: (mail=*) # requesting: dn # # search result search: 2 result: 0 Success # numResponses: 1 ############## The entry doesn't match anymore... But still it has a mail attribute : ############## $ ldapsearch -x -h ldap-test -b ou=users,dc=example,dc=com -D cn=root,dc=example,dc=com -w test -s one '(uid=test)' mail # extended LDIF # # LDAPv3 # base <ou=users,dc=example,dc=com> with scope oneLevel # filter: (uid=test) # requesting: mail # # test, users, example.com dn: uid=test,ou=users,dc=example,dc=com mail: test(a)example.com # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 ############## The only ways to make the '(mail=*)' filter matches again the modified entry, is to modify the mail attribute, or to run slapindex. If you remove the index on the mail attribute, then there is no such problem. It only happens when deleting all/the last derived attributes (supannAutreMail here) from the entry. Sounds to me like a bug, isn't it ? Regards.

1 0

(ITS#7778) Index and derived attribute bug
by yann.cezard＠univ-pau.fr 09 Jan '14

09 Jan '14

Full_Name: Yann Cezard Version: 2.4.38 OS: Debian GNU/Linux Wheezy / amd64 URL: ftp://ftp.openldap.org/incoming/Yann-Cezard-130109.tgz Submission from: (NULL) (2001:660:6701:14:225:64ff:fe92:685b) Hello, I hit a bug with the latest version of OpenLDAP (2.4.38) that was not present, at least in 2.4.17 (made a test on an older server that we were using before I discovered this). This happens on both lmdb and bdb backend (as we were using BDB before upgrading to 2.4.38, I tested with both on 2.4.38 to ensure that the problem was not LMDB related). The problem shows up when you delete the last attribute of en entry, which is derived from another attribute, which has an index on it (not easy to explain...). As an example (complete LDIF files - config/schema + data - provided on the incoming FTP), considering an attribute that is derived from the 'mail' attribute (which is defined in core.schema, and used in InetOrgPerson) : attributetype ( 1.3.6.1.4.1.7135.1.2.1.34 NAME 'supannAutreMail' DESC 'adresses mail non institutionnelles' SUP mail ) and an associated objectClass that use it (called supannPerson in the next example). Now place an index on mail : olcDbIndex: mail pres,eq,sub Now with an entry like this : dn: uid=test,ou=users,dc=example,dc=com uid: test sn: Test cn: Test objectClass: inetOrgPerson objectClass: organizationalPerson objectClass: person objectClass: top objectClass: supannPerson structuralObjectClass: inetOrgPerson entryUUID: 2699b5b4-0d7c-1033-9546-613643720ad1 creatorsName: cn=root,dc=example,dc=com createTimestamp: 20140109131736Z mail: test(a)example.com supannAutreMail: root(a)example.com supannAutreMail: user.test(a)example.com If you make a search with '(mail=*)', the entry matches : ############# $ ldapsearch -x -h ldap-test -b ou=users,dc=example,dc=com -D cn=root,dc=example,dc=com -w test -s one '(mail=*)' dn # extended LDIF # # LDAPv3 # base <ou=users,dc=example,dc=com> with scope oneLevel # filter: (mail=*) # requesting: dn # # test, users, example.com dn: uid=test,ou=users,dc=example,dc=com # search result search: 2 result: 0 Success ############## That's pretty normal. Now, remove both the supannAutreMail attributes from the entry. Run the same ldapsearch : ############## $ ldapsearch -x -h ldap-test -b ou=users,dc=example,dc=com -D cn=root,dc=example,dc=com -w test -s one '(mail=*)' dn # extended LDIF # # LDAPv3 # base <ou=users,dc=example,dc=com> with scope oneLevel # filter: (mail=*) # requesting: dn # # search result search: 2 result: 0 Success # numResponses: 1 ############## The entry doesn't match anymore... But still it has a mail attribute : ############## $ ldapsearch -x -h ldap-test -b ou=users,dc=example,dc=com -D cn=root,dc=example,dc=com -w test -s one '(uid=test)' mail # extended LDIF # # LDAPv3 # base <ou=users,dc=example,dc=com> with scope oneLevel # filter: (uid=test) # requesting: mail # # test, users, example.com dn: uid=test,ou=users,dc=example,dc=com mail: test(a)example.com # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 ############## The only ways to make the '(mail=*)' filter matches again the modified entry, is to modify the mail attribute, or to run slapindex. If you remove the index on the mail attribute, then there is no such problem. It only happens when deleting all/the last derived attributes (supannAutreMail here) from the entry. Sounds to me like a bug, isn't it ? Regards.

1 0

(ITS#7777) Filtering on modifyTimestamp is slower with index
by raphael.ouazana＠linagora.com 09 Jan '14

09 Jan '14

Full_Name: Raphael Ouazana Version: 2.4.38 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (88.173.78.196) Hi, I have a case where using modifyTimestamp index is far slower than with no index. First with indexing: ###Caching $ time ldapsearch -b <people> -D <manager> -W # search result search: 2 result: 0 Success # numResponses: 293886 # numEntries: 293885 real 0m12.153s user 0m7.805s sys 0m1.888s ### Request $ time ldapsearch -b <people> -D <manager> -W '(&(modifyTimestamp>=20131023163804Z)(modifyTimestamp<=20131123163804Z))' # search result search: 2 result: 0 Success # numResponses: 24277 # numEntries: 24276 real 0m23.441s user 0m0.554s sys 0m0.073s Now without modifyTimestamp index: ###Caching $ time ldapsearch -b <people> -D <manager> -W # search result search: 2 result: 0 Success # numResponses: 293886 # numEntries: 293885 real 0m11.627s user 0m7.354s sys 0m1.772s ### Request $ time ldapsearch -b <people> -D <manager> -W '(&(modifyTimestamp>=20131023163804Z)(modifyTimestamp<=20131123163804Z))' # search result search: 2 result: 0 Success # numResponses: 24277 # numEntries: 24276 real 0m0.701s user 0m0.569s sys 0m0.070s As you can see, without indexing the request takes less than 1s, while with index it takes more than 20s. I think it is not the expected result. Regards, Raphaël Ouazana.

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 08 Jan '14

08 Jan '14

Hallvard Breien Furuseth: > On 2014-01-08 18:23, Howard Chu wrote: > > Wietse Venema wrote: > >> Hallvard Breien Furuseth: > >>> Process A may see garbage data. > > > > This is wrong. The fcntl locks will definitely prevent two processes > > from interfering with each other. The reason this bug occurs is > > because fcntl locks cannot prevent two transactions within a single > > process from stomping on each other. > > Hmm. OK, after some discussion with Howard: > > This was so with Howard's original code, it wrapped each mdb_get() in > a transaction. postfix-2.11-20131228/src/util/slmdb.c breaks this by > preserving the read transaction until an error happens which is bad > enough for the code to notice. The Postfix first/next iterator has always used a long-term read transaction, ever since the first LMDB patch for Postfix. This broke with the switch to MDB_NOLOCK. > Locks around cursors are pointless as well as insufficient from > lmdb's point of view. It is transactions which must be protected. With MDB_NOLOCK the cursor's read transaction must be protected, and it must be closed before returning the (key, value) to the caller. Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 08 Jan '14

08 Jan '14

On 2014-01-08 18:23, Howard Chu wrote: > Wietse Venema wrote: >> Hallvard Breien Furuseth: >>> Process A may see garbage data. > > This is wrong. The fcntl locks will definitely prevent two processes > from interfering with each other. The reason this bug occurs is > because fcntl locks cannot prevent two transactions within a single > process from stomping on each other. Hmm. OK, after some discussion with Howard: This was so with Howard's original code, it wrapped each mdb_get() in a transaction. postfix-2.11-20131228/src/util/slmdb.c breaks this by preserving the read transaction until an error happens which is bad enough for the code to notice. Locks around cursors are pointless as well as insufficient from lmdb's point of view. It is transactions which must be protected. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by hyc＠symas.com 08 Jan '14

08 Jan '14

Wietse Venema wrote: > Hallvard Breien Furuseth: >> Wietse Venema writes: >>> (...) what happens when the two above activities happen >>> in different processes? That is, process A (opens the database with >>> MDB_NOLOCK. opens a cursor, and walks the database without making >>> any changes to it), while process B opens the same database with >>> MDB_NOLOCK and performs the delete transactions. >>> >>> Will the process A cursor read transaction trigger an assertion due >>> to the process B delete transactions? If not, why not? >> >> Process A may see garbage data. This is wrong. The fcntl locks will definitely prevent two processes from interfering with each other. The reason this bug occurs is because fcntl locks cannot prevent two transactions within a single process from stomping on each other. This can result in an assert(), a >> segfault or other misbehavior. Just like if you had process A == B. >> With MDB_NOLOCK, after a write transaction commits you need to >> end existing read-only transactions before starting a new writer. >> (Well, it's a bit looser than that, but that's a simple rule.) > > Thanks for confirming my suspicion. I think that the correct option > then is to use only short-lived cursor read transactions within > fcntl()-protected blocks and to save/restore the previous cursor > key where applicable. > > Wietse > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 08 Jan '14

08 Jan '14

I write: > Maybe MDB_NOLOCK should have kept the lockfile and reader table, and > just omitted the mutexes/semaphores that synchronize transactions. Duh, nevermind. As Howard reminded me, that'd require readers to have write access to the lockfile, which was the problem which started all this. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 08 Jan '14

08 Jan '14

Hallvard Breien Furuseth: > Wietse Venema writes: > > (...) what happens when the two above activities happen > > in different processes? That is, process A (opens the database with > > MDB_NOLOCK. opens a cursor, and walks the database without making > > any changes to it), while process B opens the same database with > > MDB_NOLOCK and performs the delete transactions. > > > > Will the process A cursor read transaction trigger an assertion due > > to the process B delete transactions? If not, why not? > > Process A may see garbage data. This can result in an assert(), a > segfault or other misbehavior. Just like if you had process A == B. > With MDB_NOLOCK, after a write transaction commits you need to > end existing read-only transactions before starting a new writer. > (Well, it's a bit looser than that, but that's a simple rule.) Thanks for confirming my suspicion. I think that the correct option then is to use only short-lived cursor read transactions within fcntl()-protected blocks and to save/restore the previous cursor key where applicable. Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 08 Jan '14

08 Jan '14

Wietse Venema writes: > (...) what happens when the two above activities happen > in different processes? That is, process A (opens the database with > MDB_NOLOCK. opens a cursor, and walks the database without making > any changes to it), while process B opens the same database with > MDB_NOLOCK and performs the delete transactions. > > Will the process A cursor read transaction trigger an assertion due > to the process B delete transactions? If not, why not? Process A may see garbage data. This can result in an assert(), a segfault or other misbehavior. Just like if you had process A == B. With MDB_NOLOCK, after a write transaction commits you need to end existing read-only transactions before starting a new writer. (Well, it's a bit looser than that, but that's a simple rule.) Because: A read-only transaction needs a frozen, valid snapshot of the DB. Normally it registers itself in the lockfile, so writers know to not touch the pages that make up that snapshot. MDB_NOLOCK disables the lockfile, leaving those pages unprotected from writers. Maybe MDB_NOLOCK should have kept the lockfile and reader table, and just omitted the mutexes/semaphores that synchronize transactions. OTOH mapsize changes in B are not a problem for _existing_ transactions in A, since it was actually the accompanying map _address_ change which broke existing transactions. But if A tries to begin new transactions, mdb_txn_begin() may soon fail with MDB_MAP_RESIZED until A also resizes. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 08 Jan '14

08 Jan '14

This is a follow-up question, not a request to reopen the issue. It is about what happens when different processes open the same database with MDB_NOLOCK, and then perform the exact same operations that were the subject of ITS#7774. Brief summary of ITS#7774: when an LMDB database is opened with MDB_NOLOCK, a long-lived cursor's read transaction triggers an assertion when the same process performs a delete transaction on a different database element. This happened when walking an LMDB-based cache to remove "expired" entries. The application uses fcntl() shared locks around cursor_get() and mdb_get() including all related MDB operations, and it uses fcntl() exclusive locks around mdb_put() and mdb_del() including all related MDB operations. The solution was to close the long-lived cursor's read transaction before the delete operation. Now my question: what happens when the two above activities happen in different processes? That is, process A (opens the database with MDB_NOLOCK. opens a cursor, and walks the database without making any changes to it), while process B opens the same database with MDB_NOLOCK and performs the delete transactions. Will the process A cursor read transaction trigger an assertion due to the process B delete transactions? If not, why not? Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 07 Jan '14

07 Jan '14

Quanah Gibson-Mount: > --On Tuesday, January 07, 2014 10:03 PM +0000 wietse(a)porcupine.org wrote: > > > Having obtained the patch off-list, I confirm that this call-back > > feature works satisfactorily, and that this ITS can be closed. > > > > I am looking forward to the next LMDB release. > > Hi Wietse, > > I'm currently working with the dev team to get a new release out in the > next week or so. Please do not speed up the process at the cost of code quality. I'm ready to turn on support for LMDB >= 0.9.11 in Postfix 2.11.0, as long as I know that this ITS will be addressed in 0.9.11. Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by quanah＠zimbra.com 07 Jan '14

07 Jan '14

--On Tuesday, January 07, 2014 10:03 PM +0000 wietse(a)porcupine.org wrote: > Having obtained the patch off-list, I confirm that this call-back > feature works satisfactorily, and that this ITS can be closed. > > I am looking forward to the next LMDB release. Hi Wietse, I'm currently working with the dev team to get a new release out in the next week or so. --Quanah -- Quanah Gibson-Mount Architect - Server Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 07 Jan '14

07 Jan '14

Wietse Venema: > Hallvard Breien Furuseth: > > Pushed to mdb.master. > > > > void assert_callback(MDB_env *env, const char *msg) { > > void *ctx = mdb_env_get_userctx(env); > > ...; > > } > > > > mdb_env_create(&env); > > mdb_env_set_userctx(env, ctx); > > mdb_env_set_assert(env, assert_callback); Having obtained the patch off-list, I confirm that this call-back feature works satisfactorily, and that this ITS can be closed. I am looking forward to the next LMDB release. Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 07 Jan '14

07 Jan '14

Hallvard Breien Furuseth: > Pushed to mdb.master. > > void assert_callback(MDB_env *env, const char *msg) { > void *ctx = mdb_env_get_userctx(env); > ...; > } > > mdb_env_create(&env); > mdb_env_set_userctx(env, ctx); > mdb_env_set_assert(env, assert_callback); 1) Where can I find this patch? I would like to confirm that the code works without surprises. 2) Will this code be part of LMDB 0.9.11 (or some other number)? That allows me to use the correct #ifdef in my code. 3) What is the expected release time frame for this code? If it's close enough to the Postfix 2.11 stable release late January, then it can include LMDB support. Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by h.b.furuseth＠usit.uio.no 06 Jan '14

06 Jan '14

Pushed to mdb.master. void assert_callback(MDB_env *env, const char *msg) { void *ctx = mdb_env_get_userctx(env); ...; } mdb_env_create(&env); mdb_env_set_userctx(env, ctx); mdb_env_set_assert(env, assert_callback); -- Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by h.b.furuseth＠usit.uio.no 05 Jan '14

05 Jan '14

On 2014-01-05 20:03, hyc(a)symas.com wrote: > I made this comment to Hallvard as well. It should be: > > MDB_assert_func * > mdb_env_set_assert(MDB_env *env, MDB_assert_func *func, void *info); Yes, I missed that. Hadn't intended to take this ITS into such a detour. Maybe such a context argument can be useful outside asserts too though, like a Python wrapper which wants the Python element wrapping the MDB_env of a given txn. If so, instead of passing the context to mdb_env_set_assert(): int mdb_env_set_userctx(env, void *ctx); void *mdb_env_get_userctx(env); assert() could still pass it out to the assert fallback. -- Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by hyc＠symas.com 05 Jan '14

05 Jan '14

wietse(a)porcupine.org wrote: > Wietse Venema: >>> env can be NULL. mdb_env_set_assert(env,cb) also sets a static >>> variable with the callback for the asserts which do not know of an >>> env. Could throw away that part later. Currently it only applies >>> to one assert(). >>> >>> The branch also invalidates a transaction on any page allocation >>> error, since the callers do not always clean up afterwards. Too >>> aggressive, can be reverted after updating the callers. But for >>> now it means MDB_MAP_FULL & co won't let the user commit an >>> inconsistent transaction. >> >> Thanks, this is more improvement than I asked for. I don't want to >> slip my schedule, and to avoid last-minute surprises, I will postpone >> the review and impact test of these improvements after the stable >> Postfix release 2-3 weeks from today. > > It looks like you are switching from a stop-gap fix (bare-bones > call-back) to a more permanent solution. > > In that case I must mention again that I need to register not only > a call-back function pointer but also a void* pointer that is > returned to me at call-back time. > > Getting the MDB_env at call-back time does not help me at all, > because my programs open many databases (including non-LMDB). I made this comment to Hallvard as well. It should be: MDB_assert_func * mdb_env_set_assert(MDB_env *env, MDB_assert_func *func, void *info); with the info being passed thru to the callback. > To find out which database is affected I need my own pointer to my > own (DICT) object that encapsulates the LMDB database. Then I can > decide if the problem is with a disposable cache. in which case the > program can continue with degraded performance. > > Wietse > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 05 Jan '14

05 Jan '14

Wietse Venema: > > env can be NULL. mdb_env_set_assert(env,cb) also sets a static > > variable with the callback for the asserts which do not know of an > > env. Could throw away that part later. Currently it only applies > > to one assert(). > > > > The branch also invalidates a transaction on any page allocation > > error, since the callers do not always clean up afterwards. Too > > aggressive, can be reverted after updating the callers. But for > > now it means MDB_MAP_FULL & co won't let the user commit an > > inconsistent transaction. > > Thanks, this is more improvement than I asked for. I don't want to > slip my schedule, and to avoid last-minute surprises, I will postpone > the review and impact test of these improvements after the stable > Postfix release 2-3 weeks from today. It looks like you are switching from a stop-gap fix (bare-bones call-back) to a more permanent solution. In that case I must mention again that I need to register not only a call-back function pointer but also a void* pointer that is returned to me at call-back time. Getting the MDB_env at call-back time does not help me at all, because my programs open many databases (including non-LMDB). To find out which database is affected I need my own pointer to my own (DICT) object that encapsulates the LMDB database. Then I can decide if the problem is with a disposable cache. in which case the program can continue with degraded performance. Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 05 Jan '14

05 Jan '14

Hallvard Breien Furuseth: > Drafted a branch "mdb/assert" in > <http://folk.uio.no/hbf/OpenLDAP/mdb.git/> and on Ada, based > roughly on Howard's original message to you. I didn't know about > that at first, which has caused some confusion. > > So the callback usually receives an env after all, and a NULL > context since I don't know what Howard intended there. > > Howard, do I push this? Feel free to grab it and tweak a bit:) > > env can be NULL. mdb_env_set_assert(env,cb) also sets a static > variable with the callback for the asserts which do not know of an > env. Could throw away that part later. Currently it only applies > to one assert(). > > The branch also invalidates a transaction on any page allocation > error, since the callers do not always clean up afterwards. Too > aggressive, can be reverted after updating the callers. But for > now it means MDB_MAP_FULL & co won't let the user commit an > inconsistent transaction. Thanks, this is more improvement than I asked for. I don't want to slip my schedule, and to avoid last-minute surprises, I will postpone the review and impact test of these improvements after the stable Postfix release 2-3 weeks from today. Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by h.b.furuseth＠usit.uio.no 05 Jan '14

05 Jan '14

Drafted a branch "mdb/assert" in <http://folk.uio.no/hbf/OpenLDAP/mdb.git/> and on Ada, based roughly on Howard's original message to you. I didn't know about that at first, which has caused some confusion. So the callback usually receives an env after all, and a NULL context since I don't know what Howard intended there. Howard, do I push this? Feel free to grab it and tweak a bit:) env can be NULL. mdb_env_set_assert(env,cb) also sets a static variable with the callback for the asserts which do not know of an env. Could throw away that part later. Currently it only applies to one assert(). The branch also invalidates a transaction on any page allocation error, since the callers do not always clean up afterwards. Too aggressive, can be reverted after updating the callers. But for now it means MDB_MAP_FULL & co won't let the user commit an inconsistent transaction. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 03 Jan '14

03 Jan '14

Hallvard Breien Furuseth: > I wrote: > >Wietse Venema writes: > >> I can confirm that with LMDB-0.9.10 + ITS#7756 patch, the assertion > >> failure goes away when I close the cursor (and its read transaction) > >> before changing the map or the database, and restore the cursor > >> afterwards. I guess this got broken when things were changed to use > >> external locks in order to avoid world-writable lockfiles. > > Does this mean we can close this ITS? You can close this ITS. Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 03 Jan '14

03 Jan '14

Hallvard Breien Furuseth: > wietse(a)porcupine.org writes: > > Below is my "least painful" patch for both parties. What do you think? > > That's roughly what I wrote was thinking, yes. > > We haven't heard if it will go in at all, though:-) If this ITS > is a reaction to ITS#7774, then the motivation is not catching > an lmdb bug, but letting the user band-aid a user error. Still > makes sense to me, lmdb lets the user screw up in many ways. I agree that we're talking about a stop-gap API in the absence of better error reporting. And I am not talking about users violating the C language model with invalid pointers and such; that would rightfully deserve a core dump. > --- a/libraries/liblmdb/mdb.c > +++ b/libraries/liblmdb/mdb.c Based on visual inspection, that code would work for me. Now the question for me is whether this will be available by the time Postfix 2.11 is released near the end of this month. Code freeze begins in days. I can't distribute LMDB support when daemons fall out of the sky without logfile message, whether it's your mistake, mine, or some middleperson distro maintainer. Prior art: Postfix does not support GnuTLS for the same reason. That would be a support problem. Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 03 Jan '14

03 Jan '14

I wrote: >Wietse Venema writes: >> I can confirm that with LMDB-0.9.10 + ITS#7756 patch, the assertion >> failure goes away when I close the cursor (and its read transaction) >> before changing the map or the database, and restore the cursor >> afterwards. I guess this got broken when things were changed to use >> external locks in order to avoid world-writable lockfiles. Does this mean we can close this ITS? I made a major typo here: > Yes. > > Ending the read-only transaction and starting a new one means you are > no longer seeing the same snapshot, though. So I hope the reason you > couldn't use single transaction which I suggested previously, was **not** > that > the reader transaction needed to keep seeing the same snapshot. -- Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by h.b.furuseth＠usit.uio.no 03 Jan '14

03 Jan '14

wietse(a)porcupine.org writes: > Below is my "least painful" patch for both parties. What do you think? That's roughly what I wrote was thinking, yes. We haven't heard if it will go in at all, though:-) If this ITS is a reaction to ITS#7774, then the motivation is not catching an lmdb bug, but letting the user band-aid a user error. Still makes sense to me, lmdb lets the user screw up in many ways. Anyway: Why drop the info parameter? If you don't need it after all, that's fine. But then it won't be coming back later. It doesn't fit in the varargs, they're reserved for the message. I ended up declaring a type for it in ldmb.h, to make future expansion a bit friendlier: /** Dummy definition for future extension of #mdb_assert_cb(). */ typedef struct MDB_assertion MDB_assertion; I dislike when a feature makes me #include <stdarg.h>, so I'd not make one myself. I'd build the message in a buffer. Also the code needs fleshing out a bit: Obey NDEBUG, and omit all this if something already #defined assert. See below. Finally, __func__ needs a wrapper. It can go in a renamed "compat" section: --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -151,3 +150,3 @@ */ -/** @defgroup compat Windows Compatibility Macros +/** @defgroup compat Compatibility Macros * A bunch of macros to minimize the amount of platform-specific ifdefs @@ -158,2 +157,13 @@ */ + + /** Wrapper around __func__, which is a C99 feature */ +#if __STDC_VERSION__ >= 199901L +# define mdb_func_ __func__ +#elif __GNUC__ >= 2 || _MSC_VER >= 1300 +# define mdb_func_ __FUNCTION__ +#else +/* If a debug message says <mdb_unknown>(), update the #if statements above */ +# define mdb_func_ "<mdb_unknown>" +#endif + #ifdef _WIN32 @@ -329,3 +339,3 @@ static txnid_t mdb_debug_start; # define DPRINTF0(fmt, ...) \ - fprintf(stderr, "%s:%d " fmt "\n", __func__, __LINE__, __VA_ARGS__) + fprintf(stderr, "%s:%d " fmt "\n", mdb_func_, __LINE__, __VA_ARGS__) #else Google found more compiler #ifdefs, but I'm not sure which of these are still relevant as non-C99 compilers. Could add them later at need: __func__: (defined(__SUNPRO_C) && defined(__C99FEATURES__)) __FUNC__: __BORLANDC__ >= 0x550 __FUNCTION__: __MWERKS__ >= 0x3000 || defined(__ghs__) || __DMC__ >= 0x810 || __IBMCPP__ >= 500 And the assert() itself: /** assert(3) replacement. * Omitted in programs that #include mdb.c and already define assert(). */ #ifndef assert # ifdef NDEBUG # define assert(expr) ((void) 0) # else # define assert(expr) ((expr) ? (void)0 : \ mdb_assert_fail(#expr, mdb_func_, __FILE__, __LINE__)) static void mdb_assert_fail(const char *expr, const char *func, const char *file, int line) { char buf[400]; sprintf(buf, "%.100s:%d: Assertion '%.200s' failed in %.40s()", file, line, expr, func); if (mdb_assert_cb) mdb_assert_cb(buf, NULL); fprintf(stderr, "%s\n", buf); abort(); } # endif /* NDEBUG */ #endif /* assert */ -- Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 02 Jan '14

02 Jan '14

Wietse Venema: > Hallvard Breien Furuseth: > > wietse(a)porcupine.org writes: > > >Hallvard Breien Furuseth: > > >> I suppose we could s/assert(foo)/mdb_assert(mc, foo)/ when that > > >> compiles so it can report env, txn, dbi. Howard, should I do that? > > > > > > That was my idea. Allow me to code up an example, it is simpler > > > than arguing in the abstract. > > > > By all means, but I expect we understand each other. I just needed to > > know how muchimpact this needs to have on the lmdb's code. The answer > > is it must walk all over it. And now that I think of it, it'll likely > > introduce merge conflicts in every outstanding branch of any size. > > So my preference is not to do this just yet. > > I am settling for less, and come with a few-line change, so that > LMDB might still end up in this month's stable Postfix release. > > Hang on for a few more minutes. Below is my "least painful" patch for both parties. What do you think? Wietse *** ./work/mdb-mdb/libraries/liblmdb/mdb.c- Tue Nov 12 11:10:33 2013 --- ./work/mdb-mdb/libraries/liblmdb/mdb.c Thu Jan 2 16:30:47 2014 *************** *** 65,71 **** #include <fcntl.h> #endif - #include <assert.h> #include <errno.h> #include <limits.h> #include <stddef.h> --- 65,70 ---- *************** *** 146,151 **** --- 145,155 ---- # error "Two's complement, reasonably sized integer types, please" #endif + /* assert(3) clone. Move this to the project-correct location. */ + static void mdb_assert(const char *, const char *, int, const char *); + #define assert(e) ((e) ? (void) 0 : mdb_assert(__func__, __FILE__, \ + __LINE__, #e)) + /** @defgroup internal MDB Internals * @{ */ *************** *** 3288,3293 **** --- 3292,3311 ---- return MDB_SUCCESS; } + void + (*mdb_assert_cb)(const char *,...); + + static void + mdb_assert(const char *func, const char *file, int line, const char *text) + { + if (mdb_assert_cb != 0) + mdb_assert_cb("Assertion failed: file %s, line %d, function %s: %s", + file, line, func, text); + fprintf(stderr, "Assertion failed: file %s, line %d, function %s: %s\n", + file, line, func, text); + abort(); + } + static int mdb_env_map(MDB_env *env, void *addr, int newsize) { *** ./work/mdb-mdb/libraries/liblmdb/lmdb.h- Tue Nov 12 11:10:33 2013 --- ./work/mdb-mdb/libraries/liblmdb/lmdb.h Thu Jan 2 16:01:37 2014 *************** *** 1429,1434 **** --- 1429,1441 ---- * @return 0 on success, non-zero on failure. */ int mdb_reader_check(MDB_env *env, int *dead); + + /** @brief Notify application of assertion failure. + * This should become obsolete as lmdb's error handling matures. + * @param[in] fmt The assertion message, not including newline. + */ + extern void (*mdb_assert_cb)(const char *fmt, ...); + /** @} */ #ifdef __cplusplus

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 02 Jan '14

02 Jan '14

Hallvard Breien Furuseth: > wietse(a)porcupine.org writes: > >Hallvard Breien Furuseth: > >> I suppose we could s/assert(foo)/mdb_assert(mc, foo)/ when that > >> compiles so it can report env, txn, dbi. Howard, should I do that? > > > > That was my idea. Allow me to code up an example, it is simpler > > than arguing in the abstract. > > By all means, but I expect we understand each other. I just needed to > know how muchimpact this needs to have on the lmdb's code. The answer > is it must walk all over it. And now that I think of it, it'll likely > introduce merge conflicts in every outstanding branch of any size. > So my preference is not to do this just yet. I am settling for less, and come with a few-line change, so that LMDB might still end up in this month's stable Postfix release. Hang on for a few more minutes. Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by h.b.furuseth＠usit.uio.no 02 Jan '14

02 Jan '14

wietse(a)porcupine.org writes: >Hallvard Breien Furuseth: >> I suppose we could s/assert(foo)/mdb_assert(mc, foo)/ when that >> compiles so it can report env, txn, dbi. Howard, should I do that? > > That was my idea. Allow me to code up an example, it is simpler > than arguing in the abstract. By all means, but I expect we understand each other. I just needed to know how muchimpact this needs to have on the lmdb's code. The answer is it must walk all over it. And now that I think of it, it'll likely introduce merge conflicts in every outstanding branch of any size. So my preference is not to do this just yet. -- Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 02 Jan '14

02 Jan '14

Hallvard Breien Furuseth: > wietse(a)porcupine.org writes: > >Hallvard Breien Furuseth: > >> /** User-settable callback for assert(), called before abort() > >> * instead of printing the message. > >> * > >> * assert(some user errors) might be unaffected. This hack > >> * should become obsolete as lmdb's error handling matures. > >> * @param[in] msg The assertion message, not including newline. > >> * @parem[in] info Currently NULL. Might change in the future. > >> */ > >> extern void (*mdb_assert_cb)(char *msg, void *info); > >> > >> (Or maybe call it -before- printing the message, you can abort or > >> longjmp yourself if you don't want the message.) > > > > Looks good. I'll code up an example implementation to set and use > > the callback. The "info" argument should be sufficient to figure > > out which database is failing. > > To repeat, 'info' would be NULL initially. Does that make it useless to > you? Various pieces of code don't know any context of interest to the > user anyway. That is unhelpful, as my opens lots of databases. Instead I propose that for each open database I give you my function pointer AND my context pointer. Then, you call my function with my context pointer when there is a problem. My program opens multiple databases. It is useless if you pass me your LMDB handle, as I now have to find out a) which of all my open databases is LMDB and b) which of those LMDB databases is messed up. That totally breaks all abstractions. > I suppose we could s/assert(foo)/mdb_assert(mc, foo)/ when that > compiles so it can report env, txn, dbi. Howard, should I do that? That was my idea. Allow me to code up an example, it is simpler than arguing in the abstract. Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 02 Jan '14

02 Jan '14

Wietse Venema writes: > As written in the bug report the assertion also failed with > mdb_map_size set to an insanely large value (half a billion) in > which case the map size never changes. Yes... > But, it is good to know that the cursor is invalidated by both map > size changes and transactions that change the database. No, that's the wrong idea. This is a case of a user error breaking snapshot isolation: Ending the transaction invalidates its cursors, and you have to end the transaction in these two cases (modifying when using MDB_NOLOCK, or resizing). Those are "user errors" because that made it simple to code those features. We could add code to make them non-errors on in common cases and on common hosts, but I'm not sure if that's worth the trouble. > I can confirm that with LMDB-0.9.10 + ITS#7756 patch, the assertion > failure goes away when I close the cursor (and its read transaction) > before changing the map or the database, and restore the cursor > afterwards. I guess this got broken when things were changed to use > external locks in order to avoid world-writable lockfiles. Yes. Ending the read-only transaction and starting a new one means you are no longer seeing the same snapshot, though. So I hope the reason you couldn't use single transaction which I suggested previously, was that the reader transaction needed to keep seeing the same snapshot. -- Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by h.b.furuseth＠usit.uio.no 02 Jan '14

02 Jan '14

wietse(a)porcupine.org writes: >Hallvard Breien Furuseth: >> /** User-settable callback for assert(), called before abort() >> * instead of printing the message. >> * >> * assert(some user errors) might be unaffected. This hack >> * should become obsolete as lmdb's error handling matures. >> * @param[in] msg The assertion message, not including newline. >> * @parem[in] info Currently NULL. Might change in the future. >> */ >> extern void (*mdb_assert_cb)(char *msg, void *info); >> >> (Or maybe call it -before- printing the message, you can abort or >> longjmp yourself if you don't want the message.) > > Looks good. I'll code up an example implementation to set and use > the callback. The "info" argument should be sufficient to figure > out which database is failing. To repeat, 'info' would be NULL initially. Does that make it useless to you? Various pieces of code don't know any context of interest to the user anyway. I suppose we could s/assert(foo)/mdb_assert(mc, foo)/ when that compiles so it can report env, txn, dbi. Howard, should I do that? -- Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 02 Jan '14

02 Jan '14

Hallvard Breien Furuseth: > > - Invoke an application call-back function with the error code and > > problem description text. If the error code is MDB_PANIC then > > the application knows that it needs to make final arrangements. > > Sounds good to me for now, since it doesn't mean walking all over > the code. mdb.c can just #define its own assert(). No error > code, the context is just "we're about to abort" and I don't know > of other planned callbacks which the code would help. > > /** User-settable callback for assert(), called before abort() > * instead of printing the message. > * > * assert(some user errors) might be unaffected. This hack > * should become obsolete as lmdb's error handling matures. > * @param[in] msg The assertion message, not including newline. > * @parem[in] info Currently NULL. Might change in the future. > */ > extern void (*mdb_assert_cb)(char *msg, void *info); > > (Or maybe call it -before- printing the message, you can abort or > longjmp yourself if you don't want the message.) Looks good. I'll code up an example implementation to set and use the callback. The "info" argument should be sufficient to figure out which database is failing. Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 02 Jan '14

02 Jan '14

Hallvard Breien Furuseth: > Wietse Venema writes: > > Why do you talk about map size changes when I delete a database entry?? > > That's how Postfix uses lmdb. It calls mdb_env_set_mapsize(). Looking As written in the bug report the assertion also failed with mdb_map_size set to an insanely large value (half a billion) in which case the map size never changes. But, it is good to know that the cursor is invalidated by both map size changes and transactions that change the database. I can confirm that with LMDB-0.9.10 + ITS#7756 patch, the assertion failure goes away when I close the cursor (and its read transaction) before changing the map or the database, and restore the cursor afterwards. I guess this got broken when things were changed to use external locks in order to avoid world-writable lockfiles. Wietse

1 0

Re: (ITS#7776) Bulk user add on ldap server
by quanah＠zimbra.com 02 Jan '14

02 Jan '14

--On Thursday, January 02, 2014 9:26 AM +0000 s.puspharaj(a)gmail.com wrote: > Full_Name: Puspharaj > Version: $OpenLDAP: slapd 2.4.23 (Apr 22 2013 05:03:41) > OS: Red Hat Enterprise Linux Server release 6.5 (Santiago) > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (103.4.125.30) The ITS system is for reporting bugs, not asking usage questions. I would advise you to take your question to the openldap-technical(a)openldap.org list. I would note a few things: a) The version of OpenLDAP shipped by RHEL is extremely broken. If you insist on using it, you must contact Red Hat for support. b) You can get up to date current OpenLDAP builds from Symas (http://www.symas.com) or the LTB project (<http://ltb-project.org/wiki/download#openldap>) c) I'm going to guess you are using back-bdb or back-hdb. Since you have groups with large membership, it is entirely possible you are running out of locks, lockers, or lock objects. I would advise setting an appropriate debugging level and looking for errors originating from Berkeley DB. --Quanah -- Quanah Gibson-Mount Architect - Server Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7776) Bulk user add on ldap server
by s.puspharaj＠gmail.com 02 Jan '14

02 Jan '14

Full_Name: Puspharaj Version: $OpenLDAP: slapd 2.4.23 (Apr 22 2013 05:03:41) OS: Red Hat Enterprise Linux Server release 6.5 (Santiago) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (103.4.125.30) Hi There, I have around 3000 user in people section were they hold multiple attributes and other things. The problem is, I have couple of groups that holds no user in it, When i tried to add users using ldapmodify command it sometimes not responding or it says the user exist even if the users won't exist in that group. Groupadd.ldif file contains : ################################################################# # dn: cn=typo3_frontend_global_salespartner,ou=Groups,o=company # # changetype: modify # # add: member # # member: uid=alesgian,ou=internal,c=it,ou=People,o=company # # member: uid=alesspin,ou=internal,c=it,ou=People,o=company # ################################################################# Somtimes the uid contains some special characters inbetween like . # , . When the user has the above characters it won't respond or it says the user exist. But when i added manually or with single line of user instead of 1000 user in that file. It accepts the ldapmodify command with that file. The command is ldapmodify -D "cn=Directory Manager,o=company" -f Groupadd.ldif -W This openldap version has any bug related to ldapmodify when adding 2000 users to a group through a single file? Thanks, Puspharaj

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by hyc＠symas.com 01 Jan '14

01 Jan '14

Wietse Venema wrote: > Hallvard Breien Furuseth: >>> Sofar the abstraction layer already hides the LMDB-specific MAP_FULL >>> and MAP_RESIZED error conditions. If this abstraction layer needs >>> additional code in order to maintain MDB cursor sanity, then please >>> educate me. >> >> ldmb.h says -- >> >> mdb_env_set_mapsize(): >> "It may be called at later times if no transactions are active in >> this process. Note that the library does not check for this condition, >> the caller must ensure it explicitly." >> MDB_NOLOCK: >> "[caller] must ensure that no readers are using old transactions >> while a writer is active. The simplest approach is to use an >> exclusive lock so that no readers may be active at all when a writer >> begins." >> >> That's why I talked about saving the cursor position and restoring it - >> cursors are per-transaction and you need a new transaction. > > I can remember the last key from mdb_cursor_get() and set the cursor > to that key. There does not appear to be a cursor "save" operation > in the API documentation http://symas.com/mdb/doc/group__mdb.html. Hallvard is suggesting exactly that - remember the key. There is no cursor "save" operation, any saved state would be meaningless after one write transaction. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 01 Jan '14

01 Jan '14

Hallvard Breien Furuseth: > > Sofar the abstraction layer already hides the LMDB-specific MAP_FULL > > and MAP_RESIZED error conditions. If this abstraction layer needs > > additional code in order to maintain MDB cursor sanity, then please > > educate me. > > ldmb.h says -- > > mdb_env_set_mapsize(): > "It may be called at later times if no transactions are active in > this process. Note that the library does not check for this condition, > the caller must ensure it explicitly." > MDB_NOLOCK: > "[caller] must ensure that no readers are using old transactions > while a writer is active. The simplest approach is to use an > exclusive lock so that no readers may be active at all when a writer > begins." > > That's why I talked about saving the cursor position and restoring it - > cursors are per-transaction and you need a new transaction. I can remember the last key from mdb_cursor_get() and set the cursor to that key. There does not appear to be a cursor "save" operation in the API documentation http://symas.com/mdb/doc/group__mdb.html. > Maybe you should have a single write-transaction with several cursors, > instead of several transactions. That also cures the "long-lived reader" > problem. That would result in delayed delete operations, and break correctness. If a database entry is updated after it is scheduled for delete, then the entry must not be deleted. Also, I am concerned that would pollute my database-neutral API with LMDB-specific details. Wietse > Howard, maybe NOLOCK should keep the reader table and just drop the > locks? More work though. > > -- > Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 31 Dec '13

31 Dec '13

Wietse Venema writes: > Why do you talk about map size changes when I delete a database entry?? That's how Postfix uses lmdb. It calls mdb_env_set_mapsize(). Looking closer though, this is in src/util/slmdb.c which is by Howard and you. Sounds like we should wait for whatever Howard has to say. > I need a software abstraction layer between Postfix and LMDB that > provides the following generic interface that is independent of > LMDB internals: > (...snip...) It does, normally. NOLOCK and map resizing interfere, and impose notable limitations. > Sofar the abstraction layer already hides the LMDB-specific MAP_FULL > and MAP_RESIZED error conditions. If this abstraction layer needs > additional code in order to maintain MDB cursor sanity, then please > educate me. ldmb.h says -- mdb_env_set_mapsize(): "It may be called at later times if no transactions are active in this process. Note that the library does not check for this condition, the caller must ensure it explicitly." MDB_NOLOCK: "[caller] must ensure that no readers are using old transactions while a writer is active. The simplest approach is to use an exclusive lock so that no readers may be active at all when a writer begins." That's why I talked about saving the cursor position and restoring it - cursors are per-transaction and you need a new transaction. Maybe you should have a single write-transaction with several cursors, instead of several transactions. That also cures the "long-lived reader" problem. Howard, maybe NOLOCK should keep the reader table and just drop the locks? More work though. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 31 Dec '13

31 Dec '13

Wietse Venema: > > You're using an old read-only transaction which cannot coexist with: > > - mdb_env_set_mapsize() which moves the map which a cursor in > > the reader is using. > > - several write-transactions + MDB_NOLOCK. The flag means the writers do > > not know about the reader, so they reuse pages from the snapshot the > > reader is using. The reader can survive while the metapages hold > > on to its snapshot, i.e. 1 or 2 write commits (I think). I am willing to be educated. Where in the LMDB API documentation would I have learned that I have to consider these details? Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 31 Dec '13

31 Dec '13

Hallvard Breien Furuseth: > [Due to a typo in your e-mail address, the ITS system did not mail out > this message anywhere. So I'm CC'ing Howard directly, just in case.] Sorry about that. Seems everyone screws up sometimes. > Wietse Venema writes: > > I wrote a test driver that reliably causes LMDB to abort during a > > simulated cache cleanup. This "exploit" produces the same result > > on Linux and FreeBSD, 32-bit and 64-bit systems. > > You're using an old read-only transaction which cannot coexist with: > - mdb_env_set_mapsize() which moves the map which a cursor in > the reader is using. > - several write-transactions + MDB_NOLOCK. The flag means the writers do > not know about the reader, so they reuse pages from the snapshot the > reader is using. The reader can survive while the metapages hold > on to its snapshot, i.e. 1 or 2 write commits (I think). > > I don't know if this is a thinko in your program or miscommunication > between you and Howard about MDB_NOLOCK and mapsize changes. With the > current liblmdb, a map change should involve: Remember the reader's > current position (key), resize the map, renew the txn and cursor, and > reposition the cursor. Why do you talk about map size changes when I delete a database entry?? I need a software abstraction layer between Postfix and LMDB that provides the following generic interface that is independent of LMDB internals: 1) Give me the first or the next element in the database. 2) Delete, look up, update, a specific database entry. 3) Operations under 1) and 2) must be possible to interleave. 4) Locking must be done outside LMDB, because world-writable lock files are not an option. (There is more, but the above is required to use LMDB as a cache with periodic cleanup. I am not makig up stuff here - the above interface has been used in Postfix for many years and it works with all supported databases that have an iterator. Sofar the abstraction layer already hides the LMDB-specific MAP_FULL and MAP_RESIZED error conditions. If this abstraction layer needs additional code in order to maintain MDB cursor sanity, then please educate me. I had expected that LMDB takes care of its cursors itself, since the side effects of other API calls are known inside LMDB only. Wietse > The test works if I (a) turn MDB_NOLOCK into MDB_NOTLS (I know that's > not what you want), and (b) detect map changes in mdb_cursor_get() and > update the cursor to match. > Old 'MDB_val's the reader fetched, are invalid after the mapsize change. > Also, remember that long-lived read-only transactions which write > transactions do know about, prevent them from reusing pages the reader > snapshot is using - resulting in further map growth. > > -- > Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by h.b.furuseth＠usit.uio.no 31 Dec '13

31 Dec '13

Wietse Venema writes: > (...) > A few alternatives: > > - When a request fails, return a distinct error code such as > MDB_PANIC, and allow the application to look up the text for the > error. For internal errors, yes. It'll take a fair amount of work to get there. And I don't expect ldmb to become enthusiastic at catching user errors, so those may cause crashes or other misbehavior anyway. After all you can disable asserts with NDEBUG, but then you may get e.g. NULL dereferences instead. > - Invoke an application call-back function with the error code and > problem description text. If the error code is MDB_PANIC then > the application knows that it needs to make final arrangements. Sounds good to me for now, since it doesn't mean walking all over the code. mdb.c can just #define its own assert(). No error code, the context is just "we're about to abort" and I don't know of other planned callbacks which the code would help. /** User-settable callback for assert(), called before abort() * instead of printing the message. * * assert(some user errors) might be unaffected. This hack * should become obsolete as lmdb's error handling matures. * @param[in] msg The assertion message, not including newline. * @parem[in] info Currently NULL. Might change in the future. */ extern void (*mdb_assert_cb)(char *msg, void *info); (Or maybe call it -before- printing the message, you can abort or longjmp yourself if you don't want the message.) > The second option preserves 100% backwards compatibility. The first > option may cause functions to return a result that they didn't > return previously. This is about situations where mdb didn't return at all, so compatibility with old behavior will rarely be a problem with new error codes. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 31 Dec '13

31 Dec '13

[Due to a typo in your e-mail address, the ITS system did not mail out this message anywhere. So I'm CC'ing Howard directly, just in case.] Wietse Venema writes: > I wrote a test driver that reliably causes LMDB to abort during a > simulated cache cleanup. This "exploit" produces the same result > on Linux and FreeBSD, 32-bit and 64-bit systems. You're using an old read-only transaction which cannot coexist with: - mdb_env_set_mapsize() which moves the map which a cursor in the reader is using. - several write-transactions + MDB_NOLOCK. The flag means the writers do not know about the reader, so they reuse pages from the snapshot the reader is using. The reader can survive while the metapages hold on to its snapshot, i.e. 1 or 2 write commits (I think). I don't know if this is a thinko in your program or miscommunication between you and Howard about MDB_NOLOCK and mapsize changes. With the current liblmdb, a map change should involve: Remember the reader's current position (key), resize the map, renew the txn and cursor, and reposition the cursor. The test works if I (a) turn MDB_NOLOCK into MDB_NOTLS (I know that's not what you want), and (b) detect map changes in mdb_cursor_get() and update the cursor to match. Old 'MDB_val's the reader fetched, are invalid after the mapsize change. Also, remember that long-lived read-only transactions which write transactions do know about, prevent them from reusing pages the reader snapshot is using - resulting in further map growth. -- Hallvard

1 0

(ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 29 Dec '13

29 Dec '13

Full_Name: Wietse Venema Version: All LMDB versions OS: Linux, BSD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (70.104.130.26) Subject: LMDB terminates Postfix daemon process without logfile record The LMDB library uses assert() which will abort() the program after writing a message to stderr. In the case of the Postfix MTA, the symptom is a that a daemon process falls out of the sky without logfile record. - This frustrates sysadmins (no logfile message). - This frustrates my efforts to make Postfix automatically recover from non-catastrophic error. In many cases the database is only a cache, a mere optimization. I want the option to just blow it away when LMDB has a problem with it. - Library modules should not abort programs after error. They should return an error result such as EINVAL, or use some other strategy that leaves control with the application. - Writing diagnostics to stderr is not useful for daemon programs in the general case. To fix this problem I suggest that LMDB provide way to notify the application that it was unable to complete a request, so that the application can make proper arrangements. A few alternatives: - When a request fails, return a distinct error code such as MDB_PANIC, and allow the application to look up the text for the error. - Invoke an application call-back function with the error code and problem description text. If the error code is MDB_PANIC then the application knows that it needs to make final arrangements. The second option preserves 100% backwards compatibility. The first option may cause functions to return a result that they didn't return previously. Wietse

1 0

Re: (ITS#7773) slapo-constraint falsely restricts modrdn with count 1
by pierangelo.masarati＠polimi.it 28 Dec '13

28 Dec '13

On 12/28/2013 01:52 PM, michael(a)stroeder.com wrote: > Full_Name: > Version: 2.4.38 > OS: > URL: > Submission from: (NULL) (212.227.35.93) > > > Sometimes it's useful to restrict an existing attribute type declared > multi-valued to a single value with slapo-constraint to avoid modifying the > standard schema. > > Example: > > constraint_attribute > uid > count 1 > restrict="ldap:///ou=example??sub?(objectClass=account)" > > Unfortunately when using uid in the RDN one cannot rename the entry anymore even > when deleteoldrdn is set to true. > > slapo-constraint should handle this case exactly like SINGLE-VALUE RDN > attributes are handled. Should be fixed in git master. Nothing specific to rename; rather, slapo-constraint was not handling internal modifications like softadd/softdel. p. -- Pierangelo Masarati Associate Professor Dipartimento di Scienze e Tecnologie Aerospaziali Politecnico di Milano

1 0

(ITS#7773) slapo-constraint falsely restricts modrdn with count 1
by michael＠stroeder.com 28 Dec '13

28 Dec '13

Full_Name: Version: 2.4.38 OS: URL: Submission from: (NULL) (212.227.35.93) Sometimes it's useful to restrict an existing attribute type declared multi-valued to a single value with slapo-constraint to avoid modifying the standard schema. Example: constraint_attribute uid count 1 restrict="ldap:///ou=example??sub?(objectClass=account)" Unfortunately when using uid in the RDN one cannot rename the entry anymore even when deleteoldrdn is set to true. slapo-constraint should handle this case exactly like SINGLE-VALUE RDN attributes are handled.

1 0

Re: (ITS#7665) LMDB: Documentation issue or mistake: MDB_NOMETASYNC vs MDB_CREATE
by h.b.furuseth＠usit.uio.no 28 Dec '13

28 Dec '13

Salvador Ortiz writes: > The documentation states that the Environment's flags should not overlap with > the Database's ones, but MDB_NOMETASYNC and MDB_CREATE share the same value. That doc is a current implementation detail for liblmdb developers adding new flags. I should not have put it in ldmb.h. I'll drop it and add an #if/#error check in mdb.c instead. -- Hallvard

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs