- openldap-bugs - openldap.org

(ITS#8961) Crash when deleting olcDbIndex (back-mdb)
by h.b.furuseth＠usit.uio.no 25 Jan '19

25 Jan '19

Full_Name: Hallvard Breien Furuseth Version: RE24 de23e57b13fa25ffc9b9c204fd22a50dbde6c0b2 OS: Linux: Fedora 28 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (193.90.49.117) ch_malloc(0) asserts when completely deleting olcDbIndex from an MDB database. $ mkdir db.foo $ servers/slapd/slapd -h ldapi://ldapi/ -f slapd.conf -d0 & $ ldapmodify -H ldapi://ldapi/ -QYEXTERNAL <<EOF dn: olcDatabase={1}mdb,cn=config changetype: modify delete: olcDbIndex - EOF modifying entry "olcDatabase={1}mdb,cn=config" slapd: ch_malloc.c:107: ch_calloc: Assertion `0' failed. ldap_result: Can't contact LDAP server (-1) $ cat slapd.conf include schema/core.schema authz-regexp ,cn=peercred,cn=external,cn=auth cn=admin database config rootdn cn=admin database mdb suffix o=foo directory db.foo index cn eq

1 0

Re: (ITS#8959) duplicate symbols
by ruga＠protonmail.com 24 Jan '19

24 Jan '19

This is a multi-part message in MIME format. --b1_2665a35367b9f29d29e4b0efc05c7981 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 TXkgYnVpbGQgZW52aXJvbm1lbnQgd29ya3Mgd2VsbCBmb3IgbWFueSBvcGVuIHNvdXJjZSBwcm9q ZWN0cy4KCk9uIFRodSwgSmFuIDI0LCAyMDE5IGF0IDE5OjU2LCBIb3dhcmQgQ2h1IDxoeWNAc3lt YXMuY29tPiB3cm90ZToKCj4gcnVnYUBwcm90b25tYWlsLmNvbSB3cm90ZToKPj4gRnVsbF9OYW1l OiBSdXBlcnQKPj4gVmVyc2lvbjogMi40LjQ3Cj4+IE9TOiBEYXJ3aW4gYWxpY2UgMTcuNy4wIERh cndpbiBLZXJuZWwgVmVyc2lvbiAxNy43LjA6IEZyaSBOb3YgMiAyMDo0MzoxNiBQRFQgMjAxODsg cm9vdDp4bnUtNDU3MC43MS4xN34xL1JFTEVBU0VfWDg2XzY0IHg4Nl82NAo+PiBVUkw6IGZ0cDov L2Z0cC5vcGVubGRhcC5vcmcvaW5jb21pbmcvCj4+IFN1Ym1pc3Npb24gZnJvbTogKE5VTEwpICg1 LjE3MC4xMjAuMTI2KQo+Pgo+Pgo+PiBOT1RFCj4+Cj4+PiBtYWtlCj4+Cj4+IFsuLi5dCj4+IGR1 cGxpY2F0ZSBzeW1ib2wgX2Jlcl9pbnRfb3B0aW9ucyBpbjoKPj4gL29wdC9zcmMvb3BlbmxkYXAt Mi40LjQ3L2xpYnJhcmllcy9saWJsYmVyLy5saWJzL2xpYmxiZXIuYShvcHRpb25zLm8pCj4+IC4u Ly4uL2xpYnJhcmllcy9saWJsYmVyLy5saWJzL2xpYmxiZXIuYShvcHRpb25zLm8pCj4+IGR1cGxp Y2F0ZSBzeW1ib2wgX2Jlcl9zZXRfb3B0aW9uIGluOgo+PiAvb3B0L3NyYy9vcGVubGRhcC0yLjQu NDcvbGlicmFyaWVzL2xpYmxiZXIvLmxpYnMvbGlibGJlci5hKG9wdGlvbnMubykKPj4gLi4vLi4v bGlicmFyaWVzL2xpYmxiZXIvLmxpYnMvbGlibGJlci5hKG9wdGlvbnMubykKPj4gZHVwbGljYXRl IHN5bWJvbCBfYmVyX2dldF9vcHRpb24gaW46Cj4+IC9vcHQvc3JjL29wZW5sZGFwLTIuNC40Ny9s aWJyYXJpZXMvbGlibGJlci8ubGlicy9saWJsYmVyLmEob3B0aW9ucy5vKQo+PiAuLi8uLi9saWJy YXJpZXMvbGlibGJlci8ubGlicy9saWJsYmVyLmEob3B0aW9ucy5vKQo+PiBsZDogMyBkdXBsaWNh dGUgc3ltYm9scyBmb3IgYXJjaGl0ZWN0dXJlIHg4Nl82NAo+PiBjbGFuZy03OiBlcnJvcjogbGlu a2VyIGNvbW1hbmQgZmFpbGVkIHdpdGggZXhpdCBjb2RlIDEgKHVzZSAtdiB0byBzZWUKPj4gaW52 b2NhdGlvbikKPj4gbWFrZVsyXTogKioqIFthcGl0ZXN0XSBFcnJvciAxCj4+IG1ha2VbMl06ICoq KiBXYWl0aW5nIGZvciB1bmZpbmlzaGVkIGpvYnMuLi4uCj4+IGR1cGxpY2F0ZSBzeW1ib2wgX2Jl cl9pbnRfb3B0aW9ucyBpbjoKPj4gL29wdC9zcmMvb3BlbmxkYXAtMi40LjQ3L2xpYnJhcmllcy9s aWJsYmVyLy5saWJzL2xpYmxiZXIuYShvcHRpb25zLm8pCj4+IC4uLy4uL2xpYnJhcmllcy9saWJs YmVyLy5saWJzL2xpYmxiZXIuYShvcHRpb25zLm8pCj4+IGR1cGxpY2F0ZSBzeW1ib2wgX2Jlcl9z ZXRfb3B0aW9uIGluOgo+PiAvb3B0L3NyYy9vcGVubGRhcC0yLjQuNDcvbGlicmFyaWVzL2xpYmxi ZXIvLmxpYnMvbGlibGJlci5hKG9wdGlvbnMubykKPj4gLi4vLi4vbGlicmFyaWVzL2xpYmxiZXIv LmxpYnMvbGlibGJlci5hKG9wdGlvbnMubykKPj4gZHVwbGljYXRlIHN5bWJvbCBfYmVyX2dldF9v cHRpb24gaW46Cj4+IC9vcHQvc3JjL29wZW5sZGFwLTIuNC40Ny9saWJyYXJpZXMvbGlibGJlci8u bGlicy9saWJsYmVyLmEob3B0aW9ucy5vKQo+PiAuLi8uLi9saWJyYXJpZXMvbGlibGJlci8ubGli cy9saWJsYmVyLmEob3B0aW9ucy5vKQo+PiBsZDogMyBkdXBsaWNhdGUgc3ltYm9scyBmb3IgYXJj aGl0ZWN0dXJlIHg4Nl82NAo+PiBjbGFuZy03OiBlcnJvcjogbGlua2VyIGNvbW1hbmQgZmFpbGVk IHdpdGggZXhpdCBjb2RlIDEgKHVzZSAtdiB0byBzZWUKPj4gaW52b2NhdGlvbikKPj4gbWFrZVsy XTogKioqIFtkbnRlc3RdIEVycm9yIDEKPj4gbWFrZVsxXTogKioqIFthbGwtY29tbW9uXSBFcnJv ciAxCj4+IG1ha2U6ICoqKiBbYWxsLWNvbW1vbl0gRXJyb3IgMQo+Pgo+Pgo+PiBDT01NRU5UCj4+ Cj4+IFNpbGx5IGJ1ZywgdHJpcHBpbmcgdGhlIGNvbXBpbGVyLgo+Cj4gU291bmRzIGxpa2UgYSBs aW5rZXIgYnVnIG9yIHNvbWUgb3RoZXIgcHJvYmxlbSBpbiB5b3VyIGJ1aWxkIGVudmlyb25tZW50 Lgo+IFRoZXJlIGFyZSBubyBkdXBsaWNhdGUgc3ltYm9scyBhbnl3aGVyZSBpbiB0aGlzIHNvdXJj ZSB0cmVlLgo+Cj4gLS0KPiAtLSBIb3dhcmQgQ2h1Cj4gQ1RPLCBTeW1hcyBDb3JwLiBodHRwOi8v d3d3LnN5bWFzLmNvbQo+IERpcmVjdG9yLCBIaWdobGFuZCBTdW4gaHR0cDovL2hpZ2hsYW5kc3Vu LmNvbS9oeWMvCj4gQ2hpZWYgQXJjaGl0ZWN0LCBPcGVuTERBUCBodHRwOi8vd3d3Lm9wZW5sZGFw Lm9yZy9wcm9qZWN0Lw== --b1_2665a35367b9f29d29e4b0efc05c7981 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: base64 ICAgPGRpdj5NeSBidWlsZCBlbnZpcm9ubWVudCB3b3JrcyB3ZWxsIGZvciBtYW55IG9wZW4gc291 cmNlIHByb2plY3RzLiZuYnNwOzwvZGl2PiA8ZGl2Pjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj5P biBUaHUsIEphbiAyNCwgMjAxOSBhdCAxOTo1NiwgSG93YXJkIENodSAmbHQ7PGEgaHJlZj0ibWFp bHRvOmh5Y0BzeW1hcy5jb20iIGNsYXNzPSIiPmh5Y0BzeW1hcy5jb208L2E+Jmd0OyB3cm90ZTo8 YmxvY2txdW90ZSBjbGFzcz0icHJvdG9ubWFpbF9xdW90ZSIgdHlwZT0iY2l0ZSI+ICBydWdhQHBy b3Rvbm1haWwuY29tIHdyb3RlOjxicj4mZ3Q7IEZ1bGxfTmFtZTogUnVwZXJ0PGJyPiZndDsgVmVy c2lvbjogMi40LjQ3PGJyPiZndDsgT1M6IERhcndpbiBhbGljZSAxNy43LjAgRGFyd2luIEtlcm5l bCBWZXJzaW9uIDE3LjcuMDogRnJpIE5vdiAgMiAyMDo0MzoxNiBQRFQgMjAxODsgcm9vdDp4bnUt NDU3MC43MS4xN34xL1JFTEVBU0VfWDg2XzY0IHg4Nl82NDxicj4mZ3Q7IFVSTDogZnRwOi8vZnRw Lm9wZW5sZGFwLm9yZy9pbmNvbWluZy88YnI+Jmd0OyBTdWJtaXNzaW9uIGZyb206IChOVUxMKSAo NS4xNzAuMTIwLjEyNik8YnI+Jmd0Ozxicj4mZ3Q7PGJyPiZndDsgTk9URTxicj4mZ3Q7PGJyPiZn dDsmZ3Q7IG1ha2U8YnI+Jmd0Ozxicj4mZ3Q7IFsuLi5dPGJyPiZndDsgZHVwbGljYXRlIHN5bWJv bCBfYmVyX2ludF9vcHRpb25zIGluOjxicj4mZ3Q7ICAgICAvb3B0L3NyYy9vcGVubGRhcC0yLjQu NDcvbGlicmFyaWVzL2xpYmxiZXIvLmxpYnMvbGlibGJlci5hKG9wdGlvbnMubyk8YnI+Jmd0OyAg ICAgLi4vLi4vbGlicmFyaWVzL2xpYmxiZXIvLmxpYnMvbGlibGJlci5hKG9wdGlvbnMubyk8YnI+ Jmd0OyBkdXBsaWNhdGUgc3ltYm9sIF9iZXJfc2V0X29wdGlvbiBpbjo8YnI+Jmd0OyAgICAgL29w dC9zcmMvb3BlbmxkYXAtMi40LjQ3L2xpYnJhcmllcy9saWJsYmVyLy5saWJzL2xpYmxiZXIuYShv cHRpb25zLm8pPGJyPiZndDsgICAgIC4uLy4uL2xpYnJhcmllcy9saWJsYmVyLy5saWJzL2xpYmxi ZXIuYShvcHRpb25zLm8pPGJyPiZndDsgZHVwbGljYXRlIHN5bWJvbCBfYmVyX2dldF9vcHRpb24g aW46PGJyPiZndDsgICAgIC9vcHQvc3JjL29wZW5sZGFwLTIuNC40Ny9saWJyYXJpZXMvbGlibGJl ci8ubGlicy9saWJsYmVyLmEob3B0aW9ucy5vKTxicj4mZ3Q7ICAgICAuLi8uLi9saWJyYXJpZXMv bGlibGJlci8ubGlicy9saWJsYmVyLmEob3B0aW9ucy5vKTxicj4mZ3Q7IGxkOiAzIGR1cGxpY2F0 ZSBzeW1ib2xzIGZvciBhcmNoaXRlY3R1cmUgeDg2XzY0PGJyPiZndDsgY2xhbmctNzogZXJyb3I6 IGxpbmtlciBjb21tYW5kIGZhaWxlZCB3aXRoIGV4aXQgY29kZSAxICh1c2UgLXYgdG8gc2VlPGJy PiZndDsgaW52b2NhdGlvbik8YnI+Jmd0OyBtYWtlWzJdOiAqKiogW2FwaXRlc3RdIEVycm9yIDE8 YnI+Jmd0OyBtYWtlWzJdOiAqKiogV2FpdGluZyBmb3IgdW5maW5pc2hlZCBqb2JzLi4uLjxicj4m Z3Q7IGR1cGxpY2F0ZSBzeW1ib2wgX2Jlcl9pbnRfb3B0aW9ucyBpbjo8YnI+Jmd0OyAgICAgL29w dC9zcmMvb3BlbmxkYXAtMi40LjQ3L2xpYnJhcmllcy9saWJsYmVyLy5saWJzL2xpYmxiZXIuYShv cHRpb25zLm8pPGJyPiZndDsgICAgIC4uLy4uL2xpYnJhcmllcy9saWJsYmVyLy5saWJzL2xpYmxi ZXIuYShvcHRpb25zLm8pPGJyPiZndDsgZHVwbGljYXRlIHN5bWJvbCBfYmVyX3NldF9vcHRpb24g aW46PGJyPiZndDsgICAgIC9vcHQvc3JjL29wZW5sZGFwLTIuNC40Ny9saWJyYXJpZXMvbGlibGJl ci8ubGlicy9saWJsYmVyLmEob3B0aW9ucy5vKTxicj4mZ3Q7ICAgICAuLi8uLi9saWJyYXJpZXMv bGlibGJlci8ubGlicy9saWJsYmVyLmEob3B0aW9ucy5vKTxicj4mZ3Q7IGR1cGxpY2F0ZSBzeW1i b2wgX2Jlcl9nZXRfb3B0aW9uIGluOjxicj4mZ3Q7ICAgICAvb3B0L3NyYy9vcGVubGRhcC0yLjQu NDcvbGlicmFyaWVzL2xpYmxiZXIvLmxpYnMvbGlibGJlci5hKG9wdGlvbnMubyk8YnI+Jmd0OyAg ICAgLi4vLi4vbGlicmFyaWVzL2xpYmxiZXIvLmxpYnMvbGlibGJlci5hKG9wdGlvbnMubyk8YnI+ Jmd0OyBsZDogMyBkdXBsaWNhdGUgc3ltYm9scyBmb3IgYXJjaGl0ZWN0dXJlIHg4Nl82NDxicj4m Z3Q7IGNsYW5nLTc6IGVycm9yOiBsaW5rZXIgY29tbWFuZCBmYWlsZWQgd2l0aCBleGl0IGNvZGUg MSAodXNlIC12IHRvIHNlZTxicj4mZ3Q7IGludm9jYXRpb24pPGJyPiZndDsgbWFrZVsyXTogKioq IFtkbnRlc3RdIEVycm9yIDE8YnI+Jmd0OyBtYWtlWzFdOiAqKiogW2FsbC1jb21tb25dIEVycm9y IDE8YnI+Jmd0OyBtYWtlOiAqKiogW2FsbC1jb21tb25dIEVycm9yIDE8YnI+Jmd0Ozxicj4mZ3Q7 PGJyPiZndDsgQ09NTUVOVDxicj4mZ3Q7PGJyPiZndDsgU2lsbHkgYnVnLCB0cmlwcGluZyB0aGUg Y29tcGlsZXIuPGJyPjxicj5Tb3VuZHMgbGlrZSBhIGxpbmtlciBidWcgb3Igc29tZSBvdGhlciBw cm9ibGVtIGluIHlvdXIgYnVpbGQgZW52aXJvbm1lbnQuPGJyPlRoZXJlIGFyZSBubyBkdXBsaWNh dGUgc3ltYm9scyBhbnl3aGVyZSBpbiB0aGlzIHNvdXJjZSB0cmVlLjxicj48YnI+LS08YnI+ICAt LSBIb3dhcmQgQ2h1PGJyPiAgQ1RPLCBTeW1hcyBDb3JwLiAgICAgICAgICAgaHR0cDovL3d3dy5z eW1hcy5jb208YnI+ICBEaXJlY3RvciwgSGlnaGxhbmQgU3VuICAgICBodHRwOi8vaGlnaGxhbmRz dW4uY29tL2h5Yy88YnI+ICBDaGllZiBBcmNoaXRlY3QsIE9wZW5MREFQICBodHRwOi8vd3d3Lm9w ZW5sZGFwLm9yZy9wcm9qZWN0Lzxicj48L2Jsb2NrcXVvdGU+PGRpdj48YnI+PC9kaXY+PGRpdj48 YnI+PC9kaXY+ --b1_2665a35367b9f29d29e4b0efc05c7981--

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 24 Jan '19

24 Jan '19

My bug report's bug.sh script is broken. It needs "mpid=$!" before "sleep 0.5", and "wait $mpid" before "kill $pid". -- Hallvard

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 24 Jan '19

24 Jan '19

On 22.01.2019 19:55, Ondřej Kuzník wrote: > The problem is not that it's still scheduled when the pause is > requested, scheduled tasks don't prevent a pause (and I think some > of them need to be a bit more aware of that fact), but that it's been > picked up by a worker thread and is running while cn=config waits for > them to pause. _And_ that a pause request is all-or-nothing. This demo code of graduated wait-for-pause works for me. For RE24, it doesn't combine easily with Git master's multiple tpool queues:-( ftp://ftp.openldap.org/incoming/Hallvard-Furuseth-190124.patch.gz But I'm not volunteering to write one for master anytime soon, unless someone has less complicated ideas for that than I do. A real version looks like more work than I expected anyway. > What I was suggesting was that it could check every so often whether it > might be holding up a pause and join it just like some other tasks do. > But that's only a good idea if it's able to handle the changes that > happen during the pause (indexes reconfigured again, the database going > away, ...) as well as being able to release any locks temporarily. Yes. I should have answered that, but I still cannot decide what I think of it. I see no locks to worry about in that code, but the rest does make me nervous. Possibly we can find some times where we can pause safely, but not a general case. For example, in the case of back-mdb we could register scheduled config operations the database itself, in a "(TODO)" LMDB DB. Doesn't help with schema changes, but does help renumbering slapd DBs. -- Hallvard

1 0

Re: (ITS#8959) duplicate symbols
by hyc＠symas.com 24 Jan '19

24 Jan '19

ruga(a)protonmail.com wrote: > Full_Name: Rupert > Version: 2.4.47 > OS: Darwin alice 17.7.0 Darwin Kernel Version 17.7.0: Fri Nov 2 20:43:16 PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64 x86_64 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (5.170.120.126) > > > NOTE > >> make > > [...] > duplicate symbol _ber_int_options in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > duplicate symbol _ber_set_option in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > duplicate symbol _ber_get_option in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > ld: 3 duplicate symbols for architecture x86_64 > clang-7: error: linker command failed with exit code 1 (use -v to see > invocation) > make[2]: *** [apitest] Error 1 > make[2]: *** Waiting for unfinished jobs.... > duplicate symbol _ber_int_options in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > duplicate symbol _ber_set_option in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > duplicate symbol _ber_get_option in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > ld: 3 duplicate symbols for architecture x86_64 > clang-7: error: linker command failed with exit code 1 (use -v to see > invocation) > make[2]: *** [dntest] Error 1 > make[1]: *** [all-common] Error 1 > make: *** [all-common] Error 1 > > > COMMENT > > Silly bug, tripping the compiler. Sounds like a linker bug or some other problem in your build environment. There are no duplicate symbols anywhere in this source tree. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8959) duplicate symbols
by quanah＠symas.com 24 Jan '19

24 Jan '19

--On Thursday, January 24, 2019 6:26 PM +0000 ruga(a)protonmail.com wrote: > Full_Name: Rupert > Version: 2.4.47 > OS: Darwin alice 17.7.0 Darwin Kernel Version 17.7.0: Fri Nov 2 20:43:16 > PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64 x86_64 URL: > ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (5.170.120.126) Builds fine for me using clang compiler on FreeBSD. Also appears to work fine for Apple, since it ships the OpenLDAP libraries. You'll need to provide more details as to exactly what you're doing to encounter this issue. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#8960) code quality
by quanah＠symas.com 24 Jan '19

24 Jan '19

--On Thursday, January 24, 2019 6:47 PM +0000 ruga(a)protonmail.com wrote: > Full_Name: Rupert > Version: - > OS: - > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (5.170.120.126) > > > The source code of openldap went through lumnify's meat grinder, > receiving a score of 1 (fail) out of 5. Score 1 means "A lot of patch Sounds like lumnify is a poor quality product. You may wish to use something better written? In any case, the ITS system is for bug reports, this is clearly not a bug report. It will be closed. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#8960) code quality
by ruga＠protonmail.com 24 Jan '19

24 Jan '19

Full_Name: Rupert Version: - OS: - URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (5.170.120.126) The source code of openldap went through lumnify's meat grinder, receiving a score of 1 (fail) out of 5. Score 1 means "A lot of patch work and bandages, project is most likely in initial stage of development. Most of source code is highly unmaintainable and feels a bit chaotic. It is good starting point, but plenty of work ahead. Don't give up yet!". I see old bugs pending. Is the project active? [1] https://sysadmin.libhunt.com/openldap-alternatives [2] https://lumnify.com/grades/?rel=libhunt

1 0

(ITS#8959) duplicate symbols
by ruga＠protonmail.com 24 Jan '19

24 Jan '19

Full_Name: Rupert Version: 2.4.47 OS: Darwin alice 17.7.0 Darwin Kernel Version 17.7.0: Fri Nov 2 20:43:16 PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64 x86_64 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (5.170.120.126) NOTE >make [...] duplicate symbol _ber_int_options in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) duplicate symbol _ber_set_option in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) duplicate symbol _ber_get_option in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) ld: 3 duplicate symbols for architecture x86_64 clang-7: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [apitest] Error 1 make[2]: *** Waiting for unfinished jobs.... duplicate symbol _ber_int_options in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) duplicate symbol _ber_set_option in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) duplicate symbol _ber_get_option in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) ld: 3 duplicate symbols for architecture x86_64 clang-7: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [dntest] Error 1 make[1]: *** [all-common] Error 1 make: *** [all-common] Error 1 COMMENT Silly bug, tripping the compiler.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by ondra＠mistotebe.net 22 Jan '19

22 Jan '19

On Tue, Jan 22, 2019 at 05:49:12PM +0000, h.b.furuseth(a)usit.uio.no wrote: > On 1/21/19 7:49 PM, Ondřej Kuzník wrote: >> Except there are no locks as you know being the author of parts of >> code that deals with what I'm about to outline anyway: >> >> Whenever a cn=config op is about to be processed, a pause is requested. >> That tells worker threads to stop picking up new work and waits until >> they're all quiet. The indexing task is run by one of these worker >> threads. > > Duuh, right. I got stuck looking for what's special about the > indexing task and couldn't find it:-( I need to make it special. > > So, let tasks declare their expected speed until finish or > between pausechecks. At FAST=1 (default) and SLOW=0. > A pause only stops tasks with speed < ltp_pause. > In thread_pool_pause(), replace the WANT_PAUSE stage with > > while (++ltp_pause <= max speed) { > wait until no more tasks with speed < ltp_pause; > } > > Then fast tasks should breeze past slow ones when preparing > to pause. Until all threads have slow tasks, anyway. > > To mitigate that, we'd need to predeclare the speed when > submitting a task, and limit the number of parallel slow > tasks. pool_submit() could stash the rest in a "slow queue" > instead of submitting. But I don't want to go there yet. The problem is not that it's still scheduled when the pause is requested, scheduled tasks don't prevent a pause (and I think some of them need to be a bit more aware of that fact), but that it's been picked up by a worker thread and is running while cn=config waits for them to pause. What I was suggesting was that it could check every so often whether it might be holding up a pause and join it just like some other tasks do. But that's only a good idea if it's able to handle the changes that happen during the pause (indexes reconfigured again, the database going away, ...) as well as being able to release any locks temporarily. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 22 Jan '19

22 Jan '19

On 1/21/19 7:49 PM, Ondřej Kuzník wrote: > Except there are no locks as you know being the author of parts of > code that deals with what I'm about to outline anyway: > > Whenever a cn=config op is about to be processed, a pause is requested. > That tells worker threads to stop picking up new work and waits until > they're all quiet. The indexing task is run by one of these worker > threads. Duuh, right. I got stuck looking for what's special about the indexing task and couldn't find it:-( I need to make it special. So, let tasks declare their expected speed until finish or between pausechecks. At FAST=1 (default) and SLOW=0. A pause only stops tasks with speed < ltp_pause. In thread_pool_pause(), replace the WANT_PAUSE stage with while (++ltp_pause <= max speed) { wait until no more tasks with speed < ltp_pause; } Then fast tasks should breeze past slow ones when preparing to pause. Until all threads have slow tasks, anyway. To mitigate that, we'd need to predeclare the speed when submitting a task, and limit the number of parallel slow tasks. pool_submit() could stash the rest in a "slow queue" instead of submitting. But I don't want to go there yet.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by ondra＠mistotebe.net 21 Jan '19

21 Jan '19

On Mon, Jan 21, 2019 at 03:23:55PM +0000, h.b.furuseth(a)usit.uio.no wrote: > On 20.01.2019 19:52, Hallvard Breien Furuseth wrote: > > Can cn=config have 2 lock levels? That fixes this ITS (my > > ldapwhoami hang) without introducing cn=config failures. > > > > The outer lock serializes cn=config ops including the indexer. > > Other ops do not lock it. The inner lock blocks slapd like > > cn=config does now, and should only be held briefly. Always > > hold the outer lock when taking the inner lock. > > Correction again: while taking and holding the inner lock. > Except non-config ops. Except there are no locks as you know being the author of parts of code that deals with what I'm about to outline anyway: Whenever a cn=config op is about to be processed, a pause is requested. That tells worker threads to stop picking up new work and waits until they're all quiet. The indexing task is run by one of these worker threads. Some long-running processes (syncrepl) will regularly check to see if they should pause as well: https://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/s… Handling things in a similar way might be an option if we could survive anything changing in cn=config while we yield. That might be a big "if" to implement, though. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 21 Jan '19

21 Jan '19

On 20.01.2019 19:52, Hallvard Breien Furuseth wrote: > Can cn=config have 2 lock levels? That fixes this ITS (my > ldapwhoami hang) without introducing cn=config failures. > > The outer lock serializes cn=config ops including the indexer. > Other ops do not lock it. The inner lock blocks slapd like > cn=config does now, and should only be held briefly. Always > hold the outer lock when taking the inner lock. Correction again: while taking and holding the inner lock. Except non-config ops. > Also, the inner lock could be taken and released several > times during a config op when feasible. That might improve > non-config op latency, for the price of slower config ops.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

I wrote: > Limiting the problems beyond that is a bonus. Or maybe not. > A cn=config which fails in some obscure cases is flaky. (...) Correction: Obscure _situations_, not obscure cases. As in, operation X normally succeds, but can fail if you just did the somewhat-related operation Y.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

Can cn=config have 2 lock levels? That fixes this ITS (my ldapwhoami hang) without introducing cn=config failures. The outer lock serializes cn=config ops including the indexer. Other ops do not lock it. The inner lock blocks slapd like cn=config does now, and should only be held briefly. Always hold the outer lock when taking the inner lock. Also, the inner lock could be taken and released several times during a config op when feasible. That might improve non-config op latency, for the price of slower config ops. On 1/20/19 4:42 PM, Howard Chu wrote: > Hallvard Breien Furuseth wrote: >> On 1/19/19 8:04 PM, Howard Chu wrote: >>> modifications to cn=config require no other threads to be active. Once the background indexer >>> starts running, its activity will of course block further cn=config modifications until it >>> completes. >> >> Nothing "of course" about it. I don't know why running >> the background indexer is considered a change to cn=config. > > It is not. But the indexing rules can be changed by further config mods, and we > don't want the config to change out from under it. OK. I was just parsing your worlds too literally. Also, I'm trying to think like an admin, not an OpenLDAP insider: >> The ldapmodify did the change to cn=config and completed. >> >> I can guess that it might block cn=config simply because that's >> the safe way to have both an indexer and dynamic config. Or >> I can read the code. But "read the code" != "of course". > > This was all discussed on the -devel mailing list when the feature was first implemented. > http://www.openldap.org/lists/openldap-devel/200504/msg00090.html Thanks. >> So... back to start, I guess I was suggesting that the >> indexing task should stop if another config mod comes in, >> if it can resume afterwards. And the config change need >> only fail if it might prevent the indexer from resuming. >> I suppose that gets complicated in a hurry though. > > Stopping and resuming is problematic, especially if other config changes to the DB occur > that would invalidate the in-progress indexing. OK. > We could arrange to return LDAP_BUSY to config mods if any background indexers are running. > Or as I already mentioned, we could try to only do this for relevant config mods, e.g. to > the active backend, or to global schema. I'm mostly interested in the worst expected case during normal cn=config use - whether hang or error. Limiting the problems beyond that is a bonus. Or maybe not. A cn=config which fails in some obscure cases is flaky. A cn=config which fails more frequently but in predictable ways can be annoying, but is easy to figure out how to operate. I suppose a simple test is how simple the documentation can be: If I want non-config operations to not hang, what do I do and what do I avoid? And same thing if want to write a config script which won't need to cope with expected failures.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 20 Jan '19

20 Jan '19

Hallvard Breien Furuseth wrote: > On 1/19/19 8:04 PM, Howard Chu wrote: >> modifications to cn=3Dconfig require no other threads to be active. On= ce the background indexer >> starts running, its activity will of course block further cn=3Dconfig = modifications until it >> completes. >=20 > Nothing "of course" about it.=C2=A0 I don't know why running > the background indexer is considered a change to cn=3Dconfig. It is not. But the indexing rules can be changed by further config mods, = and we don't want the config to change out from under it. > The ldapmodify did the change to cn=3Dconfig and completed. >=20 > I can guess that it might block cn=3Dconfig simply because that's > the safe way to have both an indexer and dynamic config.=C2=A0 Or > I can read the code.=C2=A0 But "read the code" !=3D "of course". This was all discussed on the -devel mailing list when the feature was fi= rst implemented. http://www.openldap.org/lists/openldap-devel/200504/msg00090.html > So... back to start, I guess I was suggesting that the > indexing task should stop if another config mod comes in, > if it can resume afterwards.=C2=A0 And the config change need > only fail if it might prevent the indexer from resuming. > I suppose that gets complicated in a hurry though. Stopping and resuming is problematic, especially if other config changes = to the DB occur that would invalidate the in-progress indexing. We could arrange to return LDAP_BUSY to config mods if any background ind= exers are running. Or as I already mentioned, we could try to only do this for relevant conf= ig mods, e.g. to the active backend, or to global schema. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

On 1/19/19 8:04 PM, Howard Chu wrote: > modifications to cn=config require no other threads to be active. Once the background indexer > starts running, its activity will of course block further cn=config modifications until it > completes. Nothing "of course" about it. I don't know why running the background indexer is considered a change to cn=config. The ldapmodify did the change to cn=config and completed. I can guess that it might block cn=config simply because that's the safe way to have both an indexer and dynamic config. Or I can read the code. But "read the code" != "of course". So... back to start, I guess I was suggesting that the indexing task should stop if another config mod comes in, if it can resume afterwards. And the config change need only fail if it might prevent the indexer from resuming. I suppose that gets complicated in a hurry though.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

On 1/20/19 5:26 AM, Howard Chu wrote: > Hallvard Breien Furuseth wrote: >> On 1/20/19 4:48 AM, Hallvard Breien Furuseth wrote: >>> On 1/20/19 2:14 AM, Howard Chu wrote: >>>> This was the safest approach, considering that a config change could alter/remove context out >>>> from under running threads otherwise. >>> >>> No, the safest approach is to fail the 2nd cn=config mod >>> unless the user says it's OK to block the server. > > How would the user say so? An LDAP control. Or the modify operation could modify an operational attribute which which doesn't exist in LDAP, it just changes the operation's behavior. Or only accept cn=config mods after the connection has added an ephemeral control entry which says how cn=config mods should behave in that connection. Maybe that's the least annoying way, since there won't be any special cases for the use to cope with. Either way, it sounds like there's no right answer for RE24's default behavior, and the only fix there would involve a config option which says which behavior to pick. (It'd be nice to be able to say things like "it's OK to hang if ldapi:// is the only listener", but I suppose then people have a hundred other conditions which would be nice too.) I don't like any of my own answers to this though. I just dislike server hangs more. >> Or fail the 1st cn=config mod too, if that's easier. > > Huh? That op has already completed. The indexer is started as a background task after > the mod op completes. I meant fail when the operation is issued, but never mind. That was silly, and cumbersome for the user to find out when he can safely issue the next operation. Either way, diagnosticMessage "RTFM: back-mdb(5) about olcDbIndex and slapd-config(5) about possible server hangs". And we should document when cn=config mods can be dangerous and how to use it safely without the users noticing. There should in any case be a method to ask the server "is cn=config very busy now, or is another cn=config mod safe?". Like failing the operation if it is not, like I said first.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 20 Jan '19

20 Jan '19

Hallvard Breien Furuseth wrote: > On 1/20/19 4:48 AM, Hallvard Breien Furuseth wrote: >> On 1/20/19 2:14 AM, Howard Chu wrote: >>> This was the safest approach, considering that a config change could alter/remove context out >>> from under running threads otherwise. >> >> No, the safest approach is to fail the 2nd cn=config mod >> unless the user says it's OK to block the server. How would the user say so? > Or fail the 1st cn=config mod too, if that's easier. Huh? That op has already completed. The indexer is started as a background task after the mod op completes. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

On 1/20/19 4:48 AM, Hallvard Breien Furuseth wrote: > On 1/20/19 2:14 AM, Howard Chu wrote: >> This was the safest approach, considering that a config change could >> alter/remove context out >> from under running threads otherwise. > > No, the safest approach is to fail the 2nd cn=config mod > unless the user says it's OK to block the server. Or fail the 1st cn=config mod too, if that's easier.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

On 1/20/19 2:14 AM, Howard Chu wrote: > Ah, the 2nd cn=config mod has requested a pause, which can't succeed until all > background threads complete. And no new threads will be run while a pause is being requested. Exactly. > This was the safest approach, considering that a config change could alter/remove context out > from under running threads otherwise. No, the safest approach is to fail the 2nd cn=config mod unless the user says it's OK to block the server.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 20 Jan '19

20 Jan '19

Hallvard Breien Furuseth wrote: > On 1/19/19 9:55 PM, h.b.furuseth(a)usit.uio.no wrote: >> On 1/19/19 8:04 PM, Howard Chu wrote: >>> The *server* is not hung. Only modifications to the cn=config DB are blocked. >> >> Yes it is. > > That is, the 2nd mod to cn=config blocks, and that's fine by me, except > _then_ the server is hung. Ah, the 2nd cn=config mod has requested a pause, which can't succeed until all background threads complete. And no new threads will be run while a pause is being requested. This was the safest approach, considering that a config change could alter/remove context out from under running threads otherwise. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/19/19 9:55 PM, h.b.furuseth(a)usit.uio.no wrote: > On 1/19/19 8:04 PM, Howard Chu wrote: >> The *server* is not hung. Only modifications to the cn=config DB are blocked. > > Yes it is. That is, the 2nd mod to cn=config blocks, and that's fine by me, except _then_ the server is hung. > See the ldapwhoami in my example. And the 'time ldapwhoami' result in output.txt. Sorry, unreadable scripting on my part. Notice the single '&' in ldapmodify & { sleep; ldapwhoami; } That is, ldapmodify in the background and then ldapwhoami. I was pasting to/from the command line, and didn't want the ldapmodify to start until I pressed Return for the ldapwhoami. Thus the weird concatenation.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/19/19 8:04 PM, Howard Chu wrote: > The *server* is not hung. Only modifications to the cn=config DB are blocked. Yes it is. See the ldapwhoami in my example.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 19 Jan '19

19 Jan '19

Hallvard Breien Furuseth wrote: > Testing again 'subordinate' looks irrelevant. I wonder what I > did in my original and bigger testcase to make it seem relevant. >=20 > On 1/19/19 6:30 PM, Howard Chu wrote: >> This is not really surprising. >=20 > I don't know these interactions well enough to be unsurprised. >=20 > It's not hard to guess that modify(olcDbIndex) might be heavy and > had better be tested.=C2=A0 A simple test says it returns quickly.=C2=A0= I > don't know why the ongoing indexing keeps cn=3Dconfig locked after > the operation succeeded, instead of keeping something in the > indexed DB locked.=C2=A0 I wouldn't expect others to guess that either. modifications to cn=3Dconfig require no other threads to be active. Once = the background indexer starts running, its activity will of course block further cn=3Dconfig mod= ifications until it completes. >> Are you suggesting the indexing task should stop if another config mod= comes in? >=20 > I'm suggesting to fail the 2nd operation if that's what it takes. > Or only hang if the user passes some LDAP control. >=20 > I'm suggesting that config pauses should be brief.=C2=A0 Server hangs > are bad.=C2=A0 Even a crash can be better when other load-balanced > servers can take over.=C2=A0 Not that I'm suggesting that "fix":-) The *server* is not hung. Only modifications to the cn=3Dconfig DB are bl= ocked. You might make a case that background indexing on a backend shouldn't blo= ck writes to unrelated parts of the cn=3Dconfig tree. The current pause desi= gn wouldn't easily support that, but it might be worth working on. Also it's not clea= r which parts of the tree are actually unrelated. E.g., changing schema elements = could affect all backends at once. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs