openldap-bugs

openldap-bugs@openldap.org

1 participants
21002 discussions

Re: (ITS#7566) ldapadd slower on Linux than BSD
by quanah＠zimbra.com 07 Apr '13

07 Apr '13

--On Saturday, April 06, 2013 1:07 PM -0600 - lidutu <lm5(a)ualberta.ca> wrote: > > > > > > > > So what is happening on the slapd server while things are paused? > > Also, if you use an ldapi:/// socket rather than a tcp/ip socket, do you > see the same issue? > > > > > A have changed to use the ldapi:/// socket and there is no improvement. > > > It seems that one thread of slapd is repeatedly waiting on epoll_wait. > Strace prints > epoll_wait(6, > > and then pauses for couple of seconds. > > > > Iotop shows to slapd threads writing to disk. Right after I start > ldapadd, they write to disk at approximately 60MB/s each. Over time, they > go down to 2-3 MB/s each. > > TID PRIO USER DISK READ DISK WRITE > SWAPIN IO COMMAND > 15264 be/4 ldap 0.00 B/s 2.81 M/s 0.00 % 99.99 % > slapd -u ldap -g ldap -f /etc/openldap/slapd.conf -h > ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock > 15255 be/4 ldap 0.00 B/s 3.00 M/s 0.00 % 19.48 % > slapd -u ldap -g ldap -f /etc/openldap/slapd.conf -h > ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock I noticed something similar on ubuntu10 as well. I upgraded my server to ubuntu12, and I also set /proc/sys/vm/dirty_ratio to 90 and /proc/sys/vm/dirty_expire_centisecs to 60000, and that helped significantly with back-mdb using current OpenLDAP master. An ldapadd of almost 3.5 million users takes approximately 4 hours now for my dataset vs 16. so 4x faster. --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7567) Alias dereferencing slowness
by luca.scamoni＠gruppopa.it 07 Apr '13

07 Apr '13

Full_Name: Luca Scamoni Version: 2.4.35 OS: Linux URL: ftp://www.sys-net.it/alias-deref-ITS2.tar.gz Submission from: (NULL) (78.134.117.13) Investigating alias dereferencing slowness on a customer production system I came upon a suspect behaviour with back-hdb/bdb. It seems that slapd, when dereferencing, instead of looking for the exact aliased DN, starts scanning more entries than needed (at least this is what I can tell looking to the logs set to log anything) On the customer system, with over 700,000 entries, the time taken to perform a simple search with alias dereferencing for a single entry, takes as much as performing a full ldapsearch of the entire ldap tree. To reproduce I created a small testcase (that I can't upload because there's no space left on ftp.openldap.org/incoming). The test case contains also the log and the operations performed on test data to show the issue. I tested with openldap 2.4.23, 2.4.31 and 2.4.35. The issue is present in all versions but seems to manifest itself only with back-bdb/hdb. Back-mdb seems not to show the issue (even looking at the logs) but right now, moving to back-mdb is not something the customer is ready to take. regards Luca

1 0

Re: (ITS#7566) ldapadd slower on Linux than BSD
by lm5＠ualberta.ca 06 Apr '13

06 Apr '13

--bcaec54eef8cfe673204d9b5eb1e Content-Type: text/plain; charset=ISO-8859-1 > So what is happening on the slapd server while things are paused? > > Also, if you use an ldapi:/// socket rather than a tcp/ip socket, do you > see the same issue? > A have changed to use the ldapi:/// socket and there is no improvement. It seems that one thread of slapd is repeatedly waiting on epoll_wait. Strace prints epoll_wait(6, and then pauses for couple of seconds. Iotop shows to slapd threads writing to disk. Right after I start ldapadd, they write to disk at approximately 60MB/s each. Over time, they go down to 2-3 MB/s each. TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 15264 be/4 ldap 0.00 B/s 2.81 M/s 0.00 % 99.99 % slapd -u ldap -g ldap -f /etc/openldap/slapd.conf -h ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock 15255 be/4 ldap 0.00 B/s 3.00 M/s 0.00 % 19.48 % slapd -u ldap -g ldap -f /etc/openldap/slapd.conf -h ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock I took an strace for all the process and threads involved over a 2 minute period. Process slapd - main thread: futex(0x7f2efa38e9d0, FUTEX_WAIT, 15251, NULL Process slapd - thread: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 57.08 1.639877 474 3457 268 futex 42.54 1.222208 1076 1136 fdatasync 0.17 0.004913 0 272650 writev 0.11 0.003194 3 1137 pwrite 0.06 0.001736 0 272649 lseek 0.02 0.000530 0 4544 1136 read 0.01 0.000282 0 2273 write 0.00 0.000132 0 2273 sendto 0.00 0.000065 0 1136 open 0.00 0.000061 0 1136 stat 0.00 0.000041 0 1136 epoll_ctl 0.00 0.000011 0 1136 close 0.00 0.000000 0 116 mprotect 0.00 0.000000 0 2272 fcntl 0.00 0.000000 0 1136 getuid 0.00 0.000000 0 1136 getppid 0.00 0.000000 0 1136 gettid ------ ----------- ----------- --------- --------- ---------------- 100.00 2.873050 570459 1404 total Process slapd - thread: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 50.67 1.128008 399 2828 213 futex 48.96 1.089974 1161 939 fdatasync 0.15 0.003311 0 226607 writev 0.13 0.002928 3 939 pwrite 0.07 0.001544 0 226607 lseek 0.01 0.000214 0 1879 write 0.01 0.000139 0 1879 sendto 0.00 0.000065 0 940 open 0.00 0.000046 0 3760 940 read 0.00 0.000029 0 94 mprotect 0.00 0.000021 0 940 close 0.00 0.000021 0 940 getppid 0.00 0.000019 0 940 getuid 0.00 0.000015 0 940 epoll_ctl 0.00 0.000014 0 940 stat 0.00 0.000000 0 1880 fcntl 0.00 0.000000 0 940 gettid ------ ----------- ----------- --------- --------- ---------------- 100.00 2.226348 473992 1153 total Process slapd - thread: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- -nan 0.000000 0 1386 read -nan 0.000000 0 13861 sendto -nan 0.000000 0 3866 294 futex -nan 0.000000 0 2772 epoll_wait -nan 0.000000 0 1386 epoll_ctl ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000000 23271 294 total Process ldapadd: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.19 0.056533 18 3194 poll 0.48 0.000276 0 4866 read 0.32 0.000183 0 6846 write 0.00 0.000000 0 1 restart_syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.056992 14907 total --bcaec54eef8cfe673204d9b5eb1e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48ZGl2IGNsYXNzPSJn bWFpbF9xdW90ZSI+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2lu OjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQp O3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgY2xhc3M9ImltIj48L2Rpdj48ZGl2IGNsYXNzPSJpbSI+ DQoNClNvIHdoYXQgaXMgaGFwcGVuaW5nIG9uIHRoZSBzbGFwZCBzZXJ2ZXIgd2hpbGUgdGhpbmdz IGFyZSBwYXVzZWQ/PGJyPg0KPGJyPg0KQWxzbywgaWYgeW91IHVzZSBhbiBsZGFwaTovLy8gc29j a2V0IHJhdGhlciB0aGFuIGEgdGNwL2lwIHNvY2tldCwgZG8geW91IHNlZSB0aGUgc2FtZSBpc3N1 ZT88YnI+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+oDxicj48L2Rpdj48ZGl2PkEgaGF2ZSBjaGFu Z2VkIHRvIHVzZSB0aGUgbGRhcGk6Ly8vIHNvY2tldCBhbmQgdGhlcmUgaXMgbm8gaW1wcm92ZW1l bnQuPGJyPjxicj48L2Rpdj4NCjxkaXY+SXQgc2VlbXMgdGhhdCBvbmUgdGhyZWFkIG9mIHNsYXBk IGlzIHJlcGVhdGVkbHkgd2FpdGluZyBvbiBlcG9sbF93YWl0LiBTdHJhY2UgcHJpbnRzPGJyPmVw b2xsX3dhaXQoNiw8YnI+PC9kaXY+PGRpdj5hbmQgdGhlbiBwYXVzZXMgZm9yIGNvdXBsZSBvZiBz ZWNvbmRzLjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PklvdG9wIHNob3dzIHRvIHNsYXBk IHRocmVhZHMgd3JpdGluZyB0byBkaXNrLiBSaWdodCBhZnRlciBJIHN0YXJ0IGxkYXBhZGQsIHRo ZXkgd3JpdGUgdG8gZGlzayBhdCBhcHByb3hpbWF0ZWx5IDYwTUIvcyBlYWNoLiBPdmVyIHRpbWUs IHRoZXkgZ28gZG93biB0byAyLTMgTUIvcyBlYWNoLjxicj4NCjxicj5USUSgoKAgUFJJT6CgoCBV U0VSoKCgIERJU0sgUkVBRKCgoCBESVNLIFdSSVRFoKCgIFNXQVBJTqCgoCBJT6CgoCBDT01NQU5E PGJyPjE1MjY0IGJlLzQgbGRhcKCgoKCgoKAgMC4wMCBCL3OgoKAgMi44MSBNL3OgIDAuMDAgJSA5 OS45OSAlIHNsYXBkIC11IGxkYXAgLWcgbGRhcCAtZiAvZXRjL29wZW5sZGFwL3NsYXBkLmNvbmYg LWggbGRhcGk6Ly8lMmZ2YXIlMmZydW4lMmZvcGVubGRhcCUyZnNsYXBkLnNvY2s8YnI+DQoxNTI1 NSBiZS80IGxkYXCgoKCgoKCgIDAuMDAgQi9zoKCgIDMuMDAgTS9zoCAwLjAwICUgMTkuNDggJSBz bGFwZCAtdSBsZGFwIC1nIGxkYXAgLWYgL2V0Yy9vcGVubGRhcC9zbGFwZC5jb25mIC1oIGxkYXBp Oi8vJTJmdmFyJTJmcnVuJTJmb3BlbmxkYXAlMmZzbGFwZC5zb2NrPGJyPjxicj48L2Rpdj48ZGl2 PkkgdG9vayBhbiBzdHJhY2UgZm9yIGFsbCB0aGUgcHJvY2VzcyBhbmQgdGhyZWFkcyBpbnZvbHZl ZCBvdmVyIGEgMiBtaW51dGUgcGVyaW9kLjxicj4NCjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+ UHJvY2VzcyBzbGFwZCAtIG1haW4gdGhyZWFkOjxicj5mdXRleCgweDdmMmVmYTM4ZTlkMCwgRlVU RVhfV0FJVCwgMTUyNTEsIE5VTEw8YnI+PGJyPjwvZGl2PjxkaXY+UHJvY2VzcyBzbGFwZCAtIHRo cmVhZDo8YnI+JSB0aW1loKCgoCBzZWNvbmRzoCB1c2Vjcy9jYWxsoKCgoCBjYWxsc6CgoCBlcnJv cnMgc3lzY2FsbDxicj4tLS0tLS0gLS0tLS0tLS0tLS0gLS0tLS0tLS0tLS0gLS0tLS0tLS0tIC0t LS0tLS0tLSAtLS0tLS0tLS0tLS0tLS0tPGJyPg0KoDU3LjA4oKCgIDEuNjM5ODc3oKCgoKCgoKAg NDc0oKCgoKAgMzQ1N6CgoKCgoCAyNjggZnV0ZXg8YnI+oDQyLjU0oKCgIDEuMjIyMjA4oKCgoKCg oCAxMDc2oKCgoKAgMTEzNqCgoKCgoKCgoKAgZmRhdGFzeW5jPGJyPqAgMC4xN6CgoCAwLjAwNDkx M6CgoKCgoKCgoKAgMKCgoCAyNzI2NTCgoKCgoKCgoKCgIHdyaXRldjxicj6gIDAuMTGgoKAgMC4w MDMxOTSgoKCgoKCgoKCgIDOgoKCgoCAxMTM3oKCgoKCgoKCgoCBwd3JpdGU8YnI+DQqgIDAuMDag oKAgMC4wMDE3MzagoKCgoKCgoKCgIDCgoKAgMjcyNjQ5oKCgoKCgoKCgoCBsc2Vlazxicj6gIDAu MDKgoKAgMC4wMDA1MzCgoKCgoKCgoKCgIDCgoKCgoCA0NTQ0oKCgoKAgMTEzNiByZWFkPGJyPqAg MC4wMaCgoCAwLjAwMDI4MqCgoKCgoKCgoKAgMKCgoKCgIDIyNzOgoKCgoKCgoKCgIHdyaXRlPGJy PqAgMC4wMKCgoCAwLjAwMDEzMqCgoKCgoKCgoKAgMKCgoKCgIDIyNzOgoKCgoKCgoKCgIHNlbmR0 bzxicj4NCqAgMC4wMKCgoCAwLjAwMDA2NaCgoKCgoKCgoKAgMKCgoKCgIDExMzagoKCgoKCgoKCg IG9wZW48YnI+oCAwLjAwoKCgIDAuMDAwMDYxoKCgoKCgoKCgoCAwoKCgoKAgMTEzNqCgoKCgoKCg oKAgc3RhdDxicj6gIDAuMDCgoKAgMC4wMDAwNDGgoKCgoKCgoKCgIDCgoKCgoCAxMTM2oKCgoKCg oKCgoCBlcG9sbF9jdGw8YnI+oCAwLjAwoKCgIDAuMDAwMDExoKCgoKCgoKCgoCAwoKCgoKAgMTEz NqCgoKCgoKCgoKAgY2xvc2U8YnI+DQqgIDAuMDCgoKAgMC4wMDAwMDCgoKCgoKCgoKCgIDCgoKCg oKAgMTE2oKCgoKCgoKCgoCBtcHJvdGVjdDxicj6gIDAuMDCgoKAgMC4wMDAwMDCgoKCgoKCgoKCg IDCgoKCgoCAyMjcyoKCgoKCgoKCgoCBmY250bDxicj6gIDAuMDCgoKAgMC4wMDAwMDCgoKCgoKCg oKCgIDCgoKCgoCAxMTM2oKCgoKCgoKCgoCBnZXR1aWQ8YnI+oCAwLjAwoKCgIDAuMDAwMDAwoKCg oKCgoKCgoCAwoKCgoKAgMTEzNqCgoKCgoKCgoKAgZ2V0cHBpZDxicj4NCqAgMC4wMKCgoCAwLjAw MDAwMKCgoKCgoKCgoKAgMKCgoKCgIDExMzagoKCgoKCgoKCgIGdldHRpZDxicj4tLS0tLS0gLS0t LS0tLS0tLS0gLS0tLS0tLS0tLS0gLS0tLS0tLS0tIC0tLS0tLS0tLSAtLS0tLS0tLS0tLS0tLS0t PGJyPjEwMC4wMKCgoCAyLjg3MzA1MKCgoKCgoKCgoKCgoKCgoCA1NzA0NTmgoKCgoCAxNDA0IHRv dGFsPGJyPjxicj5Qcm9jZXNzIHNsYXBkIC0gdGhyZWFkOjxicj4NCiUgdGltZaCgoKAgc2Vjb25k c6AgdXNlY3MvY2FsbKCgoKAgY2FsbHOgoKAgZXJyb3JzIHN5c2NhbGw8YnI+LS0tLS0tIC0tLS0t LS0tLS0tIC0tLS0tLS0tLS0tIC0tLS0tLS0tLSAtLS0tLS0tLS0gLS0tLS0tLS0tLS0tLS0tLTxi cj6gNTAuNjegoKAgMS4xMjgwMDigoKCgoKCgoCAzOTmgoKCgoCAyODI4oKCgoKCgIDIxMyBmdXRl eDxicj6gNDguOTagoKAgMS4wODk5NzSgoKCgoKCgIDExNjGgoKCgoKAgOTM5oKCgoKCgoKCgoCBm ZGF0YXN5bmM8YnI+DQqgIDAuMTWgoKAgMC4wMDMzMTGgoKCgoKCgoKCgIDCgoKAgMjI2NjA3oKCg oKCgoKCgoCB3cml0ZXY8YnI+oCAwLjEzoKCgIDAuMDAyOTI4oKCgoKCgoKCgoCAzoKCgoKCgIDkz OaCgoKCgoKCgoKAgcHdyaXRlPGJyPqAgMC4wN6CgoCAwLjAwMTU0NKCgoKCgoKCgoKAgMKCgoCAy MjY2MDegoKCgoKCgoKCgIGxzZWVrPGJyPqAgMC4wMaCgoCAwLjAwMDIxNKCgoKCgoKCgoKAgMKCg oKCgIDE4NzmgoKCgoKCgoKCgIHdyaXRlPGJyPg0KoCAwLjAxoKCgIDAuMDAwMTM5oKCgoKCgoKCg oCAwoKCgoKAgMTg3OaCgoKCgoKCgoKAgc2VuZHRvPGJyPqAgMC4wMKCgoCAwLjAwMDA2NaCgoKCg oKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCgoKCgIG9wZW48YnI+oCAwLjAwoKCgIDAuMDAwMDQ2oKCg oKCgoKCgoCAwoKCgoKAgMzc2MKCgoKCgoCA5NDAgcmVhZDxicj6gIDAuMDCgoKAgMC4wMDAwMjmg oKCgoKCgoKCgIDCgoKCgoKCgIDk0oKCgoKCgoKCgoCBtcHJvdGVjdDxicj4NCqAgMC4wMKCgoCAw LjAwMDAyMaCgoKCgoKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCgoKCgIGNsb3NlPGJyPqAgMC4wMKCg oCAwLjAwMDAyMaCgoKCgoKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCgoKCgIGdldHBwaWQ8YnI+oCAw LjAwoKCgIDAuMDAwMDE5oKCgoKCgoKCgoCAwoKCgoKCgIDk0MKCgoKCgoKCgoKAgZ2V0dWlkPGJy PqAgMC4wMKCgoCAwLjAwMDAxNaCgoKCgoKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCgoKCgIGVwb2xs X2N0bDxicj4NCqAgMC4wMKCgoCAwLjAwMDAxNKCgoKCgoKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCg oKCgIHN0YXQ8YnI+oCAwLjAwoKCgIDAuMDAwMDAwoKCgoKCgoKCgoCAwoKCgoKAgMTg4MKCgoKCg oKCgoKAgZmNudGw8YnI+oCAwLjAwoKCgIDAuMDAwMDAwoKCgoKCgoKCgoCAwoKCgoKCgIDk0MKCg oKCgoKCgoKAgZ2V0dGlkPGJyPi0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0t LS0gLS0tLS0tLS0tIC0tLS0tLS0tLS0tLS0tLS08YnI+DQoxMDAuMDCgoKAgMi4yMjYzNDigoKCg oKCgoKCgoKCgoKAgNDczOTkyoKCgoKAgMTE1MyB0b3RhbDxicj48YnI+PC9kaXY+PGRpdj5Qcm9j ZXNzIHNsYXBkIC0gdGhyZWFkOjxicj4lIHRpbWWgoKCgIHNlY29uZHOgIHVzZWNzL2NhbGygoKCg IGNhbGxzoKCgIGVycm9ycyBzeXNjYWxsPGJyPi0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0t LSAtLS0tLS0tLS0gLS0tLS0tLS0tIC0tLS0tLS0tLS0tLS0tLS08YnI+DQqgIC1uYW6goKAgMC4w MDAwMDCgoKCgoKCgoKCgIDCgoKCgoCAxMzg2oKCgoKCgoKCgoCByZWFkPGJyPqAgLW5hbqCgoCAw LjAwMDAwMKCgoKCgoKCgoKAgMKCgoKAgMTM4NjGgoKCgoKCgoKCgIHNlbmR0bzxicj6gIC1uYW6g oKAgMC4wMDAwMDCgoKCgoKCgoKCgIDCgoKCgoCAzODY2oKCgoKCgIDI5NCBmdXRleDxicj6gIC1u YW6goKAgMC4wMDAwMDCgoKCgoKCgoKCgIDCgoKCgoCAyNzcyoKCgoKCgoKCgoCBlcG9sbF93YWl0 PGJyPg0KoCAtbmFuoKCgIDAuMDAwMDAwoKCgoKCgoKCgoCAwoKCgoKAgMTM4NqCgoKCgoKCgoKAg ZXBvbGxfY3RsPGJyPi0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0gLS0t LS0tLS0tIC0tLS0tLS0tLS0tLS0tLS08YnI+MTAwLjAwoKCgIDAuMDAwMDAwoKCgoKCgoKCgoKCg oKCgoCAyMzI3MaCgoKCgoCAyOTQgdG90YWw8YnI+PGJyPlByb2Nlc3MgbGRhcGFkZDo8YnI+JSB0 aW1loKCgoCBzZWNvbmRzoCB1c2Vjcy9jYWxsoKCgoCBjYWxsc6CgoCBlcnJvcnMgc3lzY2FsbDxi cj4NCi0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0gLS0tLS0tLS0tIC0t LS0tLS0tLS0tLS0tLS08YnI+oDk5LjE5oKCgIDAuMDU2NTMzoKCgoKCgoKCgIDE4oKCgoKAgMzE5 NKCgoKCgoKCgoKAgcG9sbDxicj6gIDAuNDigoKAgMC4wMDAyNzagoKCgoKCgoKCgIDCgoKCgoCA0 ODY2oKCgoKCgoKCgoCByZWFkPGJyPqAgMC4zMqCgoCAwLjAwMDE4M6CgoKCgoKCgoKAgMKCgoKCg IDY4NDagoKCgoKCgoKCgIHdyaXRlPGJyPg0KoCAwLjAwoKCgIDAuMDAwMDAwoKCgoKCgoKCgoCAw oKCgoKCgoKAgMaCgoKCgoKCgoKAgcmVzdGFydF9zeXNjYWxsPGJyPi0tLS0tLSAtLS0tLS0tLS0t LSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0gLS0tLS0tLS0tIC0tLS0tLS0tLS0tLS0tLS08YnI+MTAw LjAwoKCgIDAuMDU2OTkyoKCgoKCgoKCgoKCgoKCgoCAxNDkwN6CgoKCgoKCgoKAgdG90YWw8YnI+ PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+DQo= --bcaec54eef8cfe673204d9b5eb1e--

1 0

Re: (ITS#7566) ldapadd slower on Linux than BSD
by quanah＠zimbra.com 05 Apr '13

05 Apr '13

--On Friday, April 05, 2013 8:52 PM -0600 - lidutu <lm5(a)ualberta.ca> wrote: > > > > Hi Quanah, > > I have started doing tests with Gentoo Linux running openldap 2.4.35, > glibc 2.16 and linux kernel 3.4, 3.8 and 3.9-rc5. I have tried ext2, > ext4, reiserfs and jfs. Ldapadd is still very slow compared to BSD > regardless of the openldap backend. > > > Also a correction to the initial report: using a memory based file > system, ldapadd the whole database took 22 min. So what is happening on the slapd server while things are paused? Also, if you use an ldapi:/// socket rather than a tcp/ip socket, do you see the same issue? --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7566) ldapadd slower on Linux than BSD
by lm5＠ualberta.ca 05 Apr '13

05 Apr '13

--047d7b414c2ebdcec404d9a84c9d Content-Type: text/plain; charset=ISO-8859-1 Hi Quanah, I have started doing tests with Gentoo Linux running openldap 2.4.35, glibc 2.16 and linux kernel 3.4, 3.8 and 3.9-rc5. I have tried ext2, ext4, reiserfs and jfs. Ldapadd is still very slow compared to BSD regardless of the openldap backend. Also a correction to the initial report: using a memory based file system, ldapadd the whole database took 22 min. Leo On Fri, Apr 5, 2013 at 5:48 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > --On Friday, April 05, 2013 11:46 PM +0000 quanah(a)zimbra.com wrote: > > --On Friday, April 05, 2013 11:36 PM +0000 lm5(a)ualberta.ca wrote: >> >> Full_Name: Leo Mocofan >>> Version: 2.4.31 >>> OS: Debian Testing >>> URL: ftp://ftp.openldap.org/**incoming/<ftp://ftp.openldap.org/incoming/> >>> Submission from: (NULL) (129.128.11.107) >>> >> >> If you are going to test back-mdb, then you need to use a current >> OpenLDAP release (2.4.35). >> >> In any case, I don't see a bug report here? It sounds more like you're >> asking for tuning advice? I would ask what filesystem you are using >> (ext2? ext4?) etc. For ext4, I had to modify dirty_bytes to keep the >> Debian/Ubuntu flush process from killing load performance. >> > > Oh, one other note -- I advise avoiding the debian/ubuntu build versions > of BDB, if that is what you are using. They do not build BDB with the > correct configure options, seriously impacting performance. > > > --Quanah > > -- > > Quanah Gibson-Mount > Sr. Member of Technical Staff > Zimbra, Inc > A Division of VMware, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > > --047d7b414c2ebdcec404d9a84c9d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div><div>Hi Quanah,<br><br></div>I have started doing tes= ts with Gentoo Linux running openldap 2.4.35, glibc 2.16 and linux kernel 3= .4, 3.8 and 3.9-rc5. I have tried ext2, ext4, reiserfs and jfs. Ldapadd is = still very slow compared to BSD regardless of the openldap backend. <br> <br></div><div>Also a correction to the initial report: using a memory base= d file system, ldapadd the whole database took 22 min.<br></div><div><br></= div>Leo<br></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quo= te"> On Fri, Apr 5, 2013 at 5:48 PM, Quanah Gibson-Mount <span dir=3D"ltr"><<= a href=3D"mailto:quanah@zimbra.com" target=3D"_blank">quanah(a)zimbra.com</a>= ></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0= 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class=3D"im">--On Friday, April 05, 2013 11:46 PM +0000 <a href=3D"mai= lto:quanah@zimbra.com" target=3D"_blank">quanah(a)zimbra.com</a> wrote:<br> <br> </div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0= 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> --On Friday, April 05, 2013 11:36 PM +0000 <a href=3D"mailto:lm5@ualberta.c= a" target=3D"_blank">lm5(a)ualberta.ca</a> wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: Leo Mocofan<br> Version: 2.4.31<br> OS: Debian Testing<br> URL: <a href=3D"ftp://ftp.openldap.org/incoming/" target=3D"_blank">ftp://f= tp.openldap.org/<u></u>incoming/</a><br> Submission from: (NULL) (129.128.11.107)<br> </blockquote> <br> If you are going to test back-mdb, then you need to use a current<br> OpenLDAP =A0release (2.4.35).<br> <br> In any case, I don't see a bug report here? =A0It sounds more like you&= #39;re<br> asking for tuning advice? =A0I would ask what filesystem you are using<br> (ext2? =A0ext4?) etc. =A0For ext4, I had to modify dirty_bytes to keep the<= br> Debian/Ubuntu flush process from killing load performance.<br> </blockquote> <br></div> Oh, one other note -- I advise avoiding the debian/ubuntu build versions of= BDB, if that is what you are using. =A0They do not build BDB with the corr= ect configure options, seriously impacting performance.<div class=3D"HOEnZb= "> <div class=3D"h5"><br> <br> --Quanah<br> <br> --<br> <br> Quanah Gibson-Mount<br> Sr. Member of Technical Staff<br> Zimbra, Inc<br> A Division of VMware, Inc.<br> --------------------<br> Zimbra :: =A0the leader in open source messaging and collaboration<br> <br> </div></div></blockquote></div><br></div> --047d7b414c2ebdcec404d9a84c9d--

1 0

Re: (ITS#7566) ldapadd slower on Linux than BSD
by quanah＠zimbra.com 05 Apr '13

05 Apr '13

--On Friday, April 05, 2013 11:46 PM +0000 quanah(a)zimbra.com wrote: > --On Friday, April 05, 2013 11:36 PM +0000 lm5(a)ualberta.ca wrote: > >> Full_Name: Leo Mocofan >> Version: 2.4.31 >> OS: Debian Testing >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (129.128.11.107) > > If you are going to test back-mdb, then you need to use a current > OpenLDAP release (2.4.35). > > In any case, I don't see a bug report here? It sounds more like you're > asking for tuning advice? I would ask what filesystem you are using > (ext2? ext4?) etc. For ext4, I had to modify dirty_bytes to keep the > Debian/Ubuntu flush process from killing load performance. Oh, one other note -- I advise avoiding the debian/ubuntu build versions of BDB, if that is what you are using. They do not build BDB with the correct configure options, seriously impacting performance. --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7566) ldapadd slower on Linux than BSD
by quanah＠zimbra.com 05 Apr '13

05 Apr '13

--On Friday, April 05, 2013 11:36 PM +0000 lm5(a)ualberta.ca wrote: > Full_Name: Leo Mocofan > Version: 2.4.31 > OS: Debian Testing > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (129.128.11.107) If you are going to test back-mdb, then you need to use a current OpenLDAP release (2.4.35). In any case, I don't see a bug report here? It sounds more like you're asking for tuning advice? I would ask what filesystem you are using (ext2? ext4?) etc. For ext4, I had to modify dirty_bytes to keep the Debian/Ubuntu flush process from killing load performance. --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7566) ldapadd slower on Linux than BSD
by lm5＠ualberta.ca 05 Apr '13

05 Apr '13

Full_Name: Leo Mocofan Version: 2.4.31 OS: Debian Testing URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (129.128.11.107) We are currently testing Linux, OpenBSD and FreeBSD to run our LDAP service. Our tests showed that Linux has great performance numbers in most of the areas but we found one test that is very slow. The database has 1 million entries and is 4.5 GB in size. Importing the whole dababase using slapadd takes minutes on all platforms. When we tried to import the database using ldapadd we noticed a huge difference: FreeBSD - 42 min, OpenBSD - 59 min, Linux - 863 min. Changing file systems for backend in Linux had no effect. Using memory based file system for the backend in Linux the time for ldapadd dropped to 3 min. Strace for ldapadd shows: write(1, "adding new entry \"uid=noname,o"..., 60) = 60 time(NULL) = 1365202241 getrusage(RUSAGE_SELF, {ru_utime={1, 684105}, ru_stime={0, 212013}, ...}) = 0 time(NULL) = 1365202241 times({tms_utime=168, tms_stime=21, tms_cutime=0, tms_cstime=0}) = 1719044914 getrusage(RUSAGE_SELF, {ru_utime={1, 684105}, ru_stime={0, 212013}, ...}) = 0 time(NULL) = 1365202241 times({tms_utime=168, tms_stime=21, tms_cutime=0, tms_cstime=0}) = 1719044914 write(4, "\27\27\27\27\17uJ,\324\324\324\324"..., 1237) = 1237 poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout) poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout) poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout) poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 1 ([{fd=4, revents=POLLIN}]) Strace for one thread of slapd: write(12, "\27\27\27\27\17uJ,\324\324\324\324"..., 117) = 117 time([1365202241]) = 1365202241 sendto(4, "<167>Apr 5 16:50:41 slapd[4513]"..., 79, MSG_NOSIGNAL, NULL, 0) = 79 futex(0x7f36f8fd6724, FUTEX_WAIT_PRIVATE, 9166, NULL) = 0 futex(0x7f36f8fd66f8, FUTEX_WAKE_PRIVATE, 1) = 0 time(NULL) = 1365202242 read(12, "", 5) = 0 write(6, "0", 1) = 1 futex(0x7f36f8fd6724, FUTEX_WAIT_PRIVATE, 9169, NULL Strace for another thread of slapd: futex(0x7f36f8fd66f8, FUTEX_WAKE_PRIVATE, 1) = 1 time(NULL) = 1365202241 epoll_wait(7, {{EPOLLIN, {u32=4177193844, u64=139874082155380}}}, 1024, 150000) = 1 read(5, "0", 8192) = 1 time(NULL) = 1365202241 epoll_wait(7, {{EPOLLIN, {u32=4177193872, u64=139874082155408}}}, 1024, 150000) = 1 epoll_ctl(7, EPOLL_CTL_MOD, 12, {0, {u32=4177193872, u64=139874082155408}}) = 0 futex(0x7f36f8fd6724, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f36f8fd6720, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 time(NULL) = 1365202242 epoll_wait(7, {{EPOLLIN, {u32=4177193844, u64=139874082155380}}}, 1024, 150000) = 1 read(5, "0", 8192) = 1 time(NULL) = 1365202242 epoll_wait(7, {}, 1024, 150000) = 0 time(NULL) = 1365202392 epoll_wait(7, Ldapadd is stuck in poll system call, while one thread of slapd is waiting on epoll_wait(7, After couple of seconds when apparently nothing happens, then everything starts working again. Changing the database backend from to bdb to hdb and mdb doesn't seem to have an effect. We then tried to force SLAP_EVENT_WAIT to use select instead of epoll_wait on Linux and ldapadd is still much slower on Linux. We are runnin Debian testing as of Apr 5, 2013, with glibc-2.13, linux-3.2.41, libdb-5.1.29, openldap-2.4.31.

1 0

Re: (ITS#7563) slapd modifies attribute value of pwdAttribute
by michael＠stroeder.com 05 Apr '13

05 Apr '13

Michael Ströder wrote: > Howard, please try yourself first before answering. > Dieter is right here. > > Example: > > dn: cn=Default Password Policy,ou=Policies,dc=stroeder,dc=de > changetype: modify > delete: pwdAttribute > pwdAttribute: userPassword > - > add: pwdAttribute > pwdAttribute: 2.5.4.35 > - > > Reading it: > > dn: cn=Default Password Policy,ou=Policies,dc=stroeder,dc=de > cn: Default Password Policy > [..] > pwdAttribute: userPassword > [..] > > I vaguely remember I had to implement a work-around in web2ldap to deal with > that when generating delta modification data when the user edits such an entry. BTW: This has nothing to do with replication. Howard Chu wrote before: "Also as a general rule the X.500 data model requires that a server store and return exactly what the user provided." see http://www.openldap.org/lists/openldap-technical/201303/msg00189.html Ciao, Michael.

1 0

(ITS#7565) unacceptable growth on delete with back-mdb
by quanah＠OpenLDAP.org 05 Apr '13

05 Apr '13

Full_Name: Quanah Gibson-Mount Version: 2.4.35 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.111.39.181) With back-mdb, if you add a large number of users to the database (via ldapadd), and then delete them (via ldapmodify), the DB explodes in size. After the initial ldapadd of 30 million users, the DB is 17GB in size. After deleting the 30 million users, the DB is 52GB in size. This is a 3.05 times larger than the DBs initial size. With back-hdb, there is no change in size of the DB (it is static). Clearly this is not a desirable state of affairs. This appears to be a bug with the freelist. After the add is complete, we have: zimbra@zre-ldap001:~/data/ldap/mdb/db$ mdb_stat -f . Freelist Status Tree depth: 3 Branch pages: 19 Leaf pages: 2007 Overflow pages: 5155 Entries: 15879 Free pages: 2612121 My delete run is not yet complete, but it is now: zimbra@zre-ldap001:~/data/ldap/mdb/db$ mdb_stat -f . Freelist Status Tree depth: 3 Branch pages: 99 Leaf pages: 11096 Overflow pages: 20891 Entries: 82932 Free pages: 9381096 (and growing) --Quanah

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs