(ITS#7564)
by tamas.kenez＠kishonti.net
--_000_06BD065177CD164EB234102C529395EF6483AF2AAMSPRD0311MB436_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Please cancel this issue.
Reason: the mtest.c test program uses an experimental flag MDB_FIXEDMAP whi=
ch causes the failure.
Without the flag it works fine.
--_000_06BD065177CD164EB234102C529395EF6483AF2AAMSPRD0311MB436_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"HU" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span lang=3D"EN-US">Please cancel this issue.<o:p><=
/o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Reason: the mtest.c test progra=
m uses an experimental flag MDB_FIXEDMAP which causes the failure.<o:p></o:=
p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Without the flag it works fine.=
<o:p></o:p></span></p>
</div>
</body>
</html>
--_000_06BD065177CD164EB234102C529395EF6483AF2AAMSPRD0311MB436_--
10 years, 7 months
Re: (ITS#7566) ldapadd slower on Linux than BSD
by quanah＠zimbra.com
--On Saturday, April 06, 2013 1:07 PM -0600 - lidutu <lm5(a)ualberta.ca>
wrote:
>
>
>
>
>
>
>
> So what is happening on the slapd server while things are paused?
>
> Also, if you use an ldapi:/// socket rather than a tcp/ip socket, do you
> see the same issue?
>
>
>
>
> A have changed to use the ldapi:/// socket and there is no improvement.
>
>
> It seems that one thread of slapd is repeatedly waiting on epoll_wait.
> Strace prints
> epoll_wait(6,
>
> and then pauses for couple of seconds.
>
>
>
> Iotop shows to slapd threads writing to disk. Right after I start
> ldapadd, they write to disk at approximately 60MB/s each. Over time, they
> go down to 2-3 MB/s each.
>
> TID PRIO USER DISK READ DISK WRITE
> SWAPIN IO COMMAND
> 15264 be/4 ldap 0.00 B/s 2.81 M/s 0.00 % 99.99 %
> slapd -u ldap -g ldap -f /etc/openldap/slapd.conf -h
> ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock
> 15255 be/4 ldap 0.00 B/s 3.00 M/s 0.00 % 19.48 %
> slapd -u ldap -g ldap -f /etc/openldap/slapd.conf -h
> ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock
I noticed something similar on ubuntu10 as well. I upgraded my server to
ubuntu12, and I also set /proc/sys/vm/dirty_ratio to 90 and
/proc/sys/vm/dirty_expire_centisecs to 60000, and that helped significantly
with back-mdb using current OpenLDAP master. An ldapadd of almost 3.5
million users takes approximately 4 hours now for my dataset vs 16. so 4x
faster.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 7 months
(ITS#7567) Alias dereferencing slowness
by luca.scamoni＠gruppopa.it
Full_Name: Luca Scamoni
Version: 2.4.35
OS: Linux
URL: ftp://www.sys-net.it/alias-deref-ITS2.tar.gz
Submission from: (NULL) (78.134.117.13)
Investigating alias dereferencing slowness on a customer production system I
came upon a suspect behaviour with back-hdb/bdb.
It seems that slapd, when dereferencing, instead of looking for the exact
aliased DN, starts scanning more entries than needed (at least this is what I
can tell looking to the logs set to log anything)
On the customer system, with over 700,000 entries, the time taken to perform a
simple search with alias dereferencing for a single entry, takes as much as
performing a full ldapsearch of the entire ldap tree.
To reproduce I created a small testcase (that I can't upload because there's no
space left on ftp.openldap.org/incoming). The test case contains also the log
and the operations performed on test data to show the issue.
I tested with openldap 2.4.23, 2.4.31 and 2.4.35. The issue is present in all
versions but seems to manifest itself only with back-bdb/hdb. Back-mdb seems not
to show the issue (even looking at the logs) but right now, moving to back-mdb
is not something the customer is ready to take.
regards
Luca
10 years, 7 months
Re: (ITS#7566) ldapadd slower on Linux than BSD
by lm5＠ualberta.ca
--bcaec54eef8cfe673204d9b5eb1e
Content-Type: text/plain; charset=ISO-8859-1
> So what is happening on the slapd server while things are paused?
>
> Also, if you use an ldapi:/// socket rather than a tcp/ip socket, do you
> see the same issue?
>
A have changed to use the ldapi:/// socket and there is no improvement.
It seems that one thread of slapd is repeatedly waiting on epoll_wait.
Strace prints
epoll_wait(6,
and then pauses for couple of seconds.
Iotop shows to slapd threads writing to disk. Right after I start ldapadd,
they write to disk at approximately 60MB/s each. Over time, they go down to
2-3 MB/s each.
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
15264 be/4 ldap 0.00 B/s 2.81 M/s 0.00 % 99.99 % slapd -u ldap
-g ldap -f /etc/openldap/slapd.conf -h
ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock
15255 be/4 ldap 0.00 B/s 3.00 M/s 0.00 % 19.48 % slapd -u ldap
-g ldap -f /etc/openldap/slapd.conf -h
ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock
I took an strace for all the process and threads involved over a 2 minute
period.
Process slapd - main thread:
futex(0x7f2efa38e9d0, FUTEX_WAIT, 15251, NULL
Process slapd - thread:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
57.08 1.639877 474 3457 268 futex
42.54 1.222208 1076 1136 fdatasync
0.17 0.004913 0 272650 writev
0.11 0.003194 3 1137 pwrite
0.06 0.001736 0 272649 lseek
0.02 0.000530 0 4544 1136 read
0.01 0.000282 0 2273 write
0.00 0.000132 0 2273 sendto
0.00 0.000065 0 1136 open
0.00 0.000061 0 1136 stat
0.00 0.000041 0 1136 epoll_ctl
0.00 0.000011 0 1136 close
0.00 0.000000 0 116 mprotect
0.00 0.000000 0 2272 fcntl
0.00 0.000000 0 1136 getuid
0.00 0.000000 0 1136 getppid
0.00 0.000000 0 1136 gettid
------ ----------- ----------- --------- --------- ----------------
100.00 2.873050 570459 1404 total
Process slapd - thread:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
50.67 1.128008 399 2828 213 futex
48.96 1.089974 1161 939 fdatasync
0.15 0.003311 0 226607 writev
0.13 0.002928 3 939 pwrite
0.07 0.001544 0 226607 lseek
0.01 0.000214 0 1879 write
0.01 0.000139 0 1879 sendto
0.00 0.000065 0 940 open
0.00 0.000046 0 3760 940 read
0.00 0.000029 0 94 mprotect
0.00 0.000021 0 940 close
0.00 0.000021 0 940 getppid
0.00 0.000019 0 940 getuid
0.00 0.000015 0 940 epoll_ctl
0.00 0.000014 0 940 stat
0.00 0.000000 0 1880 fcntl
0.00 0.000000 0 940 gettid
------ ----------- ----------- --------- --------- ----------------
100.00 2.226348 473992 1153 total
Process slapd - thread:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
-nan 0.000000 0 1386 read
-nan 0.000000 0 13861 sendto
-nan 0.000000 0 3866 294 futex
-nan 0.000000 0 2772 epoll_wait
-nan 0.000000 0 1386 epoll_ctl
------ ----------- ----------- --------- --------- ----------------
100.00 0.000000 23271 294 total
Process ldapadd:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.19 0.056533 18 3194 poll
0.48 0.000276 0 4866 read
0.32 0.000183 0 6846 write
0.00 0.000000 0 1 restart_syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.056992 14907 total
--bcaec54eef8cfe673204d9b5eb1e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: base64
PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48ZGl2IGNsYXNzPSJn
bWFpbF9xdW90ZSI+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2lu
OjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQp
O3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgY2xhc3M9ImltIj48L2Rpdj48ZGl2IGNsYXNzPSJpbSI+
DQoNClNvIHdoYXQgaXMgaGFwcGVuaW5nIG9uIHRoZSBzbGFwZCBzZXJ2ZXIgd2hpbGUgdGhpbmdz
IGFyZSBwYXVzZWQ/PGJyPg0KPGJyPg0KQWxzbywgaWYgeW91IHVzZSBhbiBsZGFwaTovLy8gc29j
a2V0IHJhdGhlciB0aGFuIGEgdGNwL2lwIHNvY2tldCwgZG8geW91IHNlZSB0aGUgc2FtZSBpc3N1
ZT88YnI+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+oDxicj48L2Rpdj48ZGl2PkEgaGF2ZSBjaGFu
Z2VkIHRvIHVzZSB0aGUgbGRhcGk6Ly8vIHNvY2tldCBhbmQgdGhlcmUgaXMgbm8gaW1wcm92ZW1l
bnQuPGJyPjxicj48L2Rpdj4NCjxkaXY+SXQgc2VlbXMgdGhhdCBvbmUgdGhyZWFkIG9mIHNsYXBk
IGlzIHJlcGVhdGVkbHkgd2FpdGluZyBvbiBlcG9sbF93YWl0LiBTdHJhY2UgcHJpbnRzPGJyPmVw
b2xsX3dhaXQoNiw8YnI+PC9kaXY+PGRpdj5hbmQgdGhlbiBwYXVzZXMgZm9yIGNvdXBsZSBvZiBz
ZWNvbmRzLjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PklvdG9wIHNob3dzIHRvIHNsYXBk
IHRocmVhZHMgd3JpdGluZyB0byBkaXNrLiBSaWdodCBhZnRlciBJIHN0YXJ0IGxkYXBhZGQsIHRo
ZXkgd3JpdGUgdG8gZGlzayBhdCBhcHByb3hpbWF0ZWx5IDYwTUIvcyBlYWNoLiBPdmVyIHRpbWUs
IHRoZXkgZ28gZG93biB0byAyLTMgTUIvcyBlYWNoLjxicj4NCjxicj5USUSgoKAgUFJJT6CgoCBV
U0VSoKCgIERJU0sgUkVBRKCgoCBESVNLIFdSSVRFoKCgIFNXQVBJTqCgoCBJT6CgoCBDT01NQU5E
PGJyPjE1MjY0IGJlLzQgbGRhcKCgoKCgoKAgMC4wMCBCL3OgoKAgMi44MSBNL3OgIDAuMDAgJSA5
OS45OSAlIHNsYXBkIC11IGxkYXAgLWcgbGRhcCAtZiAvZXRjL29wZW5sZGFwL3NsYXBkLmNvbmYg
LWggbGRhcGk6Ly8lMmZ2YXIlMmZydW4lMmZvcGVubGRhcCUyZnNsYXBkLnNvY2s8YnI+DQoxNTI1
NSBiZS80IGxkYXCgoKCgoKCgIDAuMDAgQi9zoKCgIDMuMDAgTS9zoCAwLjAwICUgMTkuNDggJSBz
bGFwZCAtdSBsZGFwIC1nIGxkYXAgLWYgL2V0Yy9vcGVubGRhcC9zbGFwZC5jb25mIC1oIGxkYXBp
Oi8vJTJmdmFyJTJmcnVuJTJmb3BlbmxkYXAlMmZzbGFwZC5zb2NrPGJyPjxicj48L2Rpdj48ZGl2
PkkgdG9vayBhbiBzdHJhY2UgZm9yIGFsbCB0aGUgcHJvY2VzcyBhbmQgdGhyZWFkcyBpbnZvbHZl
ZCBvdmVyIGEgMiBtaW51dGUgcGVyaW9kLjxicj4NCjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+
UHJvY2VzcyBzbGFwZCAtIG1haW4gdGhyZWFkOjxicj5mdXRleCgweDdmMmVmYTM4ZTlkMCwgRlVU
RVhfV0FJVCwgMTUyNTEsIE5VTEw8YnI+PGJyPjwvZGl2PjxkaXY+UHJvY2VzcyBzbGFwZCAtIHRo
cmVhZDo8YnI+JSB0aW1loKCgoCBzZWNvbmRzoCB1c2Vjcy9jYWxsoKCgoCBjYWxsc6CgoCBlcnJv
cnMgc3lzY2FsbDxicj4tLS0tLS0gLS0tLS0tLS0tLS0gLS0tLS0tLS0tLS0gLS0tLS0tLS0tIC0t
LS0tLS0tLSAtLS0tLS0tLS0tLS0tLS0tPGJyPg0KoDU3LjA4oKCgIDEuNjM5ODc3oKCgoKCgoKAg
NDc0oKCgoKAgMzQ1N6CgoKCgoCAyNjggZnV0ZXg8YnI+oDQyLjU0oKCgIDEuMjIyMjA4oKCgoKCg
oCAxMDc2oKCgoKAgMTEzNqCgoKCgoKCgoKAgZmRhdGFzeW5jPGJyPqAgMC4xN6CgoCAwLjAwNDkx
M6CgoKCgoKCgoKAgMKCgoCAyNzI2NTCgoKCgoKCgoKCgIHdyaXRldjxicj6gIDAuMTGgoKAgMC4w
MDMxOTSgoKCgoKCgoKCgIDOgoKCgoCAxMTM3oKCgoKCgoKCgoCBwd3JpdGU8YnI+DQqgIDAuMDag
oKAgMC4wMDE3MzagoKCgoKCgoKCgIDCgoKAgMjcyNjQ5oKCgoKCgoKCgoCBsc2Vlazxicj6gIDAu
MDKgoKAgMC4wMDA1MzCgoKCgoKCgoKCgIDCgoKCgoCA0NTQ0oKCgoKAgMTEzNiByZWFkPGJyPqAg
MC4wMaCgoCAwLjAwMDI4MqCgoKCgoKCgoKAgMKCgoKCgIDIyNzOgoKCgoKCgoKCgIHdyaXRlPGJy
PqAgMC4wMKCgoCAwLjAwMDEzMqCgoKCgoKCgoKAgMKCgoKCgIDIyNzOgoKCgoKCgoKCgIHNlbmR0
bzxicj4NCqAgMC4wMKCgoCAwLjAwMDA2NaCgoKCgoKCgoKAgMKCgoKCgIDExMzagoKCgoKCgoKCg
IG9wZW48YnI+oCAwLjAwoKCgIDAuMDAwMDYxoKCgoKCgoKCgoCAwoKCgoKAgMTEzNqCgoKCgoKCg
oKAgc3RhdDxicj6gIDAuMDCgoKAgMC4wMDAwNDGgoKCgoKCgoKCgIDCgoKCgoCAxMTM2oKCgoKCg
oKCgoCBlcG9sbF9jdGw8YnI+oCAwLjAwoKCgIDAuMDAwMDExoKCgoKCgoKCgoCAwoKCgoKAgMTEz
NqCgoKCgoKCgoKAgY2xvc2U8YnI+DQqgIDAuMDCgoKAgMC4wMDAwMDCgoKCgoKCgoKCgIDCgoKCg
oKAgMTE2oKCgoKCgoKCgoCBtcHJvdGVjdDxicj6gIDAuMDCgoKAgMC4wMDAwMDCgoKCgoKCgoKCg
IDCgoKCgoCAyMjcyoKCgoKCgoKCgoCBmY250bDxicj6gIDAuMDCgoKAgMC4wMDAwMDCgoKCgoKCg
oKCgIDCgoKCgoCAxMTM2oKCgoKCgoKCgoCBnZXR1aWQ8YnI+oCAwLjAwoKCgIDAuMDAwMDAwoKCg
oKCgoKCgoCAwoKCgoKAgMTEzNqCgoKCgoKCgoKAgZ2V0cHBpZDxicj4NCqAgMC4wMKCgoCAwLjAw
MDAwMKCgoKCgoKCgoKAgMKCgoKCgIDExMzagoKCgoKCgoKCgIGdldHRpZDxicj4tLS0tLS0gLS0t
LS0tLS0tLS0gLS0tLS0tLS0tLS0gLS0tLS0tLS0tIC0tLS0tLS0tLSAtLS0tLS0tLS0tLS0tLS0t
PGJyPjEwMC4wMKCgoCAyLjg3MzA1MKCgoKCgoKCgoKCgoKCgoCA1NzA0NTmgoKCgoCAxNDA0IHRv
dGFsPGJyPjxicj5Qcm9jZXNzIHNsYXBkIC0gdGhyZWFkOjxicj4NCiUgdGltZaCgoKAgc2Vjb25k
c6AgdXNlY3MvY2FsbKCgoKAgY2FsbHOgoKAgZXJyb3JzIHN5c2NhbGw8YnI+LS0tLS0tIC0tLS0t
LS0tLS0tIC0tLS0tLS0tLS0tIC0tLS0tLS0tLSAtLS0tLS0tLS0gLS0tLS0tLS0tLS0tLS0tLTxi
cj6gNTAuNjegoKAgMS4xMjgwMDigoKCgoKCgoCAzOTmgoKCgoCAyODI4oKCgoKCgIDIxMyBmdXRl
eDxicj6gNDguOTagoKAgMS4wODk5NzSgoKCgoKCgIDExNjGgoKCgoKAgOTM5oKCgoKCgoKCgoCBm
ZGF0YXN5bmM8YnI+DQqgIDAuMTWgoKAgMC4wMDMzMTGgoKCgoKCgoKCgIDCgoKAgMjI2NjA3oKCg
oKCgoKCgoCB3cml0ZXY8YnI+oCAwLjEzoKCgIDAuMDAyOTI4oKCgoKCgoKCgoCAzoKCgoKCgIDkz
OaCgoKCgoKCgoKAgcHdyaXRlPGJyPqAgMC4wN6CgoCAwLjAwMTU0NKCgoKCgoKCgoKAgMKCgoCAy
MjY2MDegoKCgoKCgoKCgIGxzZWVrPGJyPqAgMC4wMaCgoCAwLjAwMDIxNKCgoKCgoKCgoKAgMKCg
oKCgIDE4NzmgoKCgoKCgoKCgIHdyaXRlPGJyPg0KoCAwLjAxoKCgIDAuMDAwMTM5oKCgoKCgoKCg
oCAwoKCgoKAgMTg3OaCgoKCgoKCgoKAgc2VuZHRvPGJyPqAgMC4wMKCgoCAwLjAwMDA2NaCgoKCg
oKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCgoKCgIG9wZW48YnI+oCAwLjAwoKCgIDAuMDAwMDQ2oKCg
oKCgoKCgoCAwoKCgoKAgMzc2MKCgoKCgoCA5NDAgcmVhZDxicj6gIDAuMDCgoKAgMC4wMDAwMjmg
oKCgoKCgoKCgIDCgoKCgoKCgIDk0oKCgoKCgoKCgoCBtcHJvdGVjdDxicj4NCqAgMC4wMKCgoCAw
LjAwMDAyMaCgoKCgoKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCgoKCgIGNsb3NlPGJyPqAgMC4wMKCg
oCAwLjAwMDAyMaCgoKCgoKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCgoKCgIGdldHBwaWQ8YnI+oCAw
LjAwoKCgIDAuMDAwMDE5oKCgoKCgoKCgoCAwoKCgoKCgIDk0MKCgoKCgoKCgoKAgZ2V0dWlkPGJy
PqAgMC4wMKCgoCAwLjAwMDAxNaCgoKCgoKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCgoKCgIGVwb2xs
X2N0bDxicj4NCqAgMC4wMKCgoCAwLjAwMDAxNKCgoKCgoKCgoKAgMKCgoKCgoCA5NDCgoKCgoKCg
oKCgIHN0YXQ8YnI+oCAwLjAwoKCgIDAuMDAwMDAwoKCgoKCgoKCgoCAwoKCgoKAgMTg4MKCgoKCg
oKCgoKAgZmNudGw8YnI+oCAwLjAwoKCgIDAuMDAwMDAwoKCgoKCgoKCgoCAwoKCgoKCgIDk0MKCg
oKCgoKCgoKAgZ2V0dGlkPGJyPi0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0t
LS0gLS0tLS0tLS0tIC0tLS0tLS0tLS0tLS0tLS08YnI+DQoxMDAuMDCgoKAgMi4yMjYzNDigoKCg
oKCgoKCgoKCgoKAgNDczOTkyoKCgoKAgMTE1MyB0b3RhbDxicj48YnI+PC9kaXY+PGRpdj5Qcm9j
ZXNzIHNsYXBkIC0gdGhyZWFkOjxicj4lIHRpbWWgoKCgIHNlY29uZHOgIHVzZWNzL2NhbGygoKCg
IGNhbGxzoKCgIGVycm9ycyBzeXNjYWxsPGJyPi0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0t
LSAtLS0tLS0tLS0gLS0tLS0tLS0tIC0tLS0tLS0tLS0tLS0tLS08YnI+DQqgIC1uYW6goKAgMC4w
MDAwMDCgoKCgoKCgoKCgIDCgoKCgoCAxMzg2oKCgoKCgoKCgoCByZWFkPGJyPqAgLW5hbqCgoCAw
LjAwMDAwMKCgoKCgoKCgoKAgMKCgoKAgMTM4NjGgoKCgoKCgoKCgIHNlbmR0bzxicj6gIC1uYW6g
oKAgMC4wMDAwMDCgoKCgoKCgoKCgIDCgoKCgoCAzODY2oKCgoKCgIDI5NCBmdXRleDxicj6gIC1u
YW6goKAgMC4wMDAwMDCgoKCgoKCgoKCgIDCgoKCgoCAyNzcyoKCgoKCgoKCgoCBlcG9sbF93YWl0
PGJyPg0KoCAtbmFuoKCgIDAuMDAwMDAwoKCgoKCgoKCgoCAwoKCgoKAgMTM4NqCgoKCgoKCgoKAg
ZXBvbGxfY3RsPGJyPi0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0gLS0t
LS0tLS0tIC0tLS0tLS0tLS0tLS0tLS08YnI+MTAwLjAwoKCgIDAuMDAwMDAwoKCgoKCgoKCgoKCg
oKCgoCAyMzI3MaCgoKCgoCAyOTQgdG90YWw8YnI+PGJyPlByb2Nlc3MgbGRhcGFkZDo8YnI+JSB0
aW1loKCgoCBzZWNvbmRzoCB1c2Vjcy9jYWxsoKCgoCBjYWxsc6CgoCBlcnJvcnMgc3lzY2FsbDxi
cj4NCi0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0gLS0tLS0tLS0tIC0t
LS0tLS0tLS0tLS0tLS08YnI+oDk5LjE5oKCgIDAuMDU2NTMzoKCgoKCgoKCgIDE4oKCgoKAgMzE5
NKCgoKCgoKCgoKAgcG9sbDxicj6gIDAuNDigoKAgMC4wMDAyNzagoKCgoKCgoKCgIDCgoKCgoCA0
ODY2oKCgoKCgoKCgoCByZWFkPGJyPqAgMC4zMqCgoCAwLjAwMDE4M6CgoKCgoKCgoKAgMKCgoKCg
IDY4NDagoKCgoKCgoKCgIHdyaXRlPGJyPg0KoCAwLjAwoKCgIDAuMDAwMDAwoKCgoKCgoKCgoCAw
oKCgoKCgoKAgMaCgoKCgoKCgoKAgcmVzdGFydF9zeXNjYWxsPGJyPi0tLS0tLSAtLS0tLS0tLS0t
LSAtLS0tLS0tLS0tLSAtLS0tLS0tLS0gLS0tLS0tLS0tIC0tLS0tLS0tLS0tLS0tLS08YnI+MTAw
LjAwoKCgIDAuMDU2OTkyoKCgoKCgoKCgoKCgoKCgoCAxNDkwN6CgoKCgoKCgoKAgdG90YWw8YnI+
PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+DQo=
--bcaec54eef8cfe673204d9b5eb1e--
10 years, 7 months
Re: (ITS#7566) ldapadd slower on Linux than BSD
by quanah＠zimbra.com
--On Friday, April 05, 2013 8:52 PM -0600 - lidutu <lm5(a)ualberta.ca> wrote:
>
>
>
> Hi Quanah,
>
> I have started doing tests with Gentoo Linux running openldap 2.4.35,
> glibc 2.16 and linux kernel 3.4, 3.8 and 3.9-rc5. I have tried ext2,
> ext4, reiserfs and jfs. Ldapadd is still very slow compared to BSD
> regardless of the openldap backend.
>
>
> Also a correction to the initial report: using a memory based file
> system, ldapadd the whole database took 22 min.
So what is happening on the slapd server while things are paused?
Also, if you use an ldapi:/// socket rather than a tcp/ip socket, do you
see the same issue?
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 7 months
Re: (ITS#7566) ldapadd slower on Linux than BSD
by lm5＠ualberta.ca
--047d7b414c2ebdcec404d9a84c9d
Content-Type: text/plain; charset=ISO-8859-1
Hi Quanah,
I have started doing tests with Gentoo Linux running openldap 2.4.35, glibc
2.16 and linux kernel 3.4, 3.8 and 3.9-rc5. I have tried ext2, ext4,
reiserfs and jfs. Ldapadd is still very slow compared to BSD regardless of
the openldap backend.
Also a correction to the initial report: using a memory based file system,
ldapadd the whole database took 22 min.
Leo
On Fri, Apr 5, 2013 at 5:48 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> --On Friday, April 05, 2013 11:46 PM +0000 quanah(a)zimbra.com wrote:
>
> --On Friday, April 05, 2013 11:36 PM +0000 lm5(a)ualberta.ca wrote:
>>
>> Full_Name: Leo Mocofan
>>> Version: 2.4.31
>>> OS: Debian Testing
>>> URL: ftp://ftp.openldap.org/**incoming/<ftp://ftp.openldap.org/incoming/>
>>> Submission from: (NULL) (129.128.11.107)
>>>
>>
>> If you are going to test back-mdb, then you need to use a current
>> OpenLDAP release (2.4.35).
>>
>> In any case, I don't see a bug report here? It sounds more like you're
>> asking for tuning advice? I would ask what filesystem you are using
>> (ext2? ext4?) etc. For ext4, I had to modify dirty_bytes to keep the
>> Debian/Ubuntu flush process from killing load performance.
>>
>
> Oh, one other note -- I advise avoiding the debian/ubuntu build versions
> of BDB, if that is what you are using. They do not build BDB with the
> correct configure options, seriously impacting performance.
>
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Sr. Member of Technical Staff
> Zimbra, Inc
> A Division of VMware, Inc.
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
>
--047d7b414c2ebdcec404d9a84c9d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div>Hi Quanah,<br><br></div>I have started doing tes=
ts with Gentoo Linux running openldap 2.4.35, glibc 2.16 and linux kernel 3=
.4, 3.8 and 3.9-rc5. I have tried ext2, ext4, reiserfs and jfs. Ldapadd is =
still very slow compared to BSD regardless of the openldap backend. <br>
<br></div><div>Also a correction to the initial report: using a memory base=
d file system, ldapadd the whole database took 22 min.<br></div><div><br></=
div>Leo<br></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quo=
te">
On Fri, Apr 5, 2013 at 5:48 PM, Quanah Gibson-Mount <span dir=3D"ltr"><<=
a href=3D"mailto:quanah@zimbra.com" target=3D"_blank">quanah(a)zimbra.com</a>=
></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0=
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class=3D"im">--On Friday, April 05, 2013 11:46 PM +0000 <a href=3D"mai=
lto:quanah@zimbra.com" target=3D"_blank">quanah(a)zimbra.com</a> wrote:<br>
<br>
</div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0=
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
--On Friday, April 05, 2013 11:36 PM +0000 <a href=3D"mailto:lm5@ualberta.c=
a" target=3D"_blank">lm5(a)ualberta.ca</a> wrote:<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Full_Name: Leo Mocofan<br>
Version: 2.4.31<br>
OS: Debian Testing<br>
URL: <a href=3D"ftp://ftp.openldap.org/incoming/" target=3D"_blank">ftp://f=
tp.openldap.org/<u></u>incoming/</a><br>
Submission from: (NULL) (129.128.11.107)<br>
</blockquote>
<br>
If you are going to test back-mdb, then you need to use a current<br>
OpenLDAP =A0release (2.4.35).<br>
<br>
In any case, I don't see a bug report here? =A0It sounds more like you&=
#39;re<br>
asking for tuning advice? =A0I would ask what filesystem you are using<br>
(ext2? =A0ext4?) etc. =A0For ext4, I had to modify dirty_bytes to keep the<=
br>
Debian/Ubuntu flush process from killing load performance.<br>
</blockquote>
<br></div>
Oh, one other note -- I advise avoiding the debian/ubuntu build versions of=
BDB, if that is what you are using. =A0They do not build BDB with the corr=
ect configure options, seriously impacting performance.<div class=3D"HOEnZb=
">
<div class=3D"h5"><br>
<br>
--Quanah<br>
<br>
--<br>
<br>
Quanah Gibson-Mount<br>
Sr. Member of Technical Staff<br>
Zimbra, Inc<br>
A Division of VMware, Inc.<br>
--------------------<br>
Zimbra :: =A0the leader in open source messaging and collaboration<br>
<br>
</div></div></blockquote></div><br></div>
--047d7b414c2ebdcec404d9a84c9d--
10 years, 7 months
Re: (ITS#7566) ldapadd slower on Linux than BSD
by quanah＠zimbra.com
--On Friday, April 05, 2013 11:46 PM +0000 quanah(a)zimbra.com wrote:
> --On Friday, April 05, 2013 11:36 PM +0000 lm5(a)ualberta.ca wrote:
>
>> Full_Name: Leo Mocofan
>> Version: 2.4.31
>> OS: Debian Testing
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (129.128.11.107)
>
> If you are going to test back-mdb, then you need to use a current
> OpenLDAP release (2.4.35).
>
> In any case, I don't see a bug report here? It sounds more like you're
> asking for tuning advice? I would ask what filesystem you are using
> (ext2? ext4?) etc. For ext4, I had to modify dirty_bytes to keep the
> Debian/Ubuntu flush process from killing load performance.
Oh, one other note -- I advise avoiding the debian/ubuntu build versions of
BDB, if that is what you are using. They do not build BDB with the correct
configure options, seriously impacting performance.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 7 months
Re: (ITS#7566) ldapadd slower on Linux than BSD
by quanah＠zimbra.com
--On Friday, April 05, 2013 11:36 PM +0000 lm5(a)ualberta.ca wrote:
> Full_Name: Leo Mocofan
> Version: 2.4.31
> OS: Debian Testing
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (129.128.11.107)
If you are going to test back-mdb, then you need to use a current OpenLDAP
release (2.4.35).
In any case, I don't see a bug report here? It sounds more like you're
asking for tuning advice? I would ask what filesystem you are using (ext2?
ext4?) etc. For ext4, I had to modify dirty_bytes to keep the
Debian/Ubuntu flush process from killing load performance.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 7 months
(ITS#7566) ldapadd slower on Linux than BSD
by lm5＠ualberta.ca
Full_Name: Leo Mocofan
Version: 2.4.31
OS: Debian Testing
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (129.128.11.107)
We are currently testing Linux, OpenBSD and FreeBSD to run our LDAP service. Our
tests showed that Linux has great performance numbers in most of the areas but
we found one test that is very slow.
The database has 1 million entries and is 4.5 GB in size. Importing the whole
dababase using slapadd takes minutes on all platforms. When we tried to import
the database using ldapadd we noticed a huge difference: FreeBSD - 42 min,
OpenBSD - 59 min, Linux - 863 min. Changing file systems for backend in Linux
had no effect. Using memory based file system for the backend in Linux the time
for ldapadd dropped to 3 min.
Strace for ldapadd shows:
write(1, "adding new entry \"uid=noname,o"..., 60) = 60
time(NULL) = 1365202241
getrusage(RUSAGE_SELF, {ru_utime={1, 684105}, ru_stime={0, 212013}, ...}) = 0
time(NULL) = 1365202241
times({tms_utime=168, tms_stime=21, tms_cutime=0, tms_cstime=0}) = 1719044914
getrusage(RUSAGE_SELF, {ru_utime={1, 684105}, ru_stime={0, 212013}, ...}) = 0
time(NULL) = 1365202241
times({tms_utime=168, tms_stime=21, tms_cutime=0, tms_cstime=0}) = 1719044914
write(4, "\27\27\27\27\17uJ,\324\324\324\324"..., 1237) = 1237
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout)
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout)
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout)
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 1 ([{fd=4, revents=POLLIN}])
Strace for one thread of slapd:
write(12, "\27\27\27\27\17uJ,\324\324\324\324"..., 117) = 117
time([1365202241]) = 1365202241
sendto(4, "<167>Apr 5 16:50:41 slapd[4513]"..., 79, MSG_NOSIGNAL, NULL, 0) =
79
futex(0x7f36f8fd6724, FUTEX_WAIT_PRIVATE, 9166, NULL) = 0
futex(0x7f36f8fd66f8, FUTEX_WAKE_PRIVATE, 1) = 0
time(NULL) = 1365202242
read(12, "", 5) = 0
write(6, "0", 1) = 1
futex(0x7f36f8fd6724, FUTEX_WAIT_PRIVATE, 9169, NULL
Strace for another thread of slapd:
futex(0x7f36f8fd66f8, FUTEX_WAKE_PRIVATE, 1) = 1
time(NULL) = 1365202241
epoll_wait(7, {{EPOLLIN, {u32=4177193844, u64=139874082155380}}}, 1024, 150000)
= 1
read(5, "0", 8192) = 1
time(NULL) = 1365202241
epoll_wait(7, {{EPOLLIN, {u32=4177193872, u64=139874082155408}}}, 1024, 150000)
= 1
epoll_ctl(7, EPOLL_CTL_MOD, 12, {0, {u32=4177193872, u64=139874082155408}}) = 0
futex(0x7f36f8fd6724, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f36f8fd6720,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
time(NULL) = 1365202242
epoll_wait(7, {{EPOLLIN, {u32=4177193844, u64=139874082155380}}}, 1024, 150000)
= 1
read(5, "0", 8192) = 1
time(NULL) = 1365202242
epoll_wait(7, {}, 1024, 150000) = 0
time(NULL) = 1365202392
epoll_wait(7,
Ldapadd is stuck in poll system call, while one thread of slapd is waiting on
epoll_wait(7,
After couple of seconds when apparently nothing happens, then everything starts
working again.
Changing the database backend from to bdb to hdb and mdb doesn't seem to have an
effect. We then tried to force SLAP_EVENT_WAIT to use select instead of
epoll_wait on Linux and ldapadd is still much slower on Linux.
We are runnin Debian testing as of Apr 5, 2013, with glibc-2.13, linux-3.2.41,
libdb-5.1.29, openldap-2.4.31.
10 years, 7 months
Re: (ITS#7563) slapd modifies attribute value of pwdAttribute
by michael＠stroeder.com
Michael Ströder wrote:
> Howard, please try yourself first before answering.
> Dieter is right here.
>
> Example:
>
> dn: cn=Default Password Policy,ou=Policies,dc=stroeder,dc=de
> changetype: modify
> delete: pwdAttribute
> pwdAttribute: userPassword
> -
> add: pwdAttribute
> pwdAttribute: 2.5.4.35
> -
>
> Reading it:
>
> dn: cn=Default Password Policy,ou=Policies,dc=stroeder,dc=de
> cn: Default Password Policy
> [..]
> pwdAttribute: userPassword
> [..]
>
> I vaguely remember I had to implement a work-around in web2ldap to deal with
> that when generating delta modification data when the user edits such an entry.
BTW: This has nothing to do with replication.
Howard Chu wrote before:
"Also as a general rule the X.500 data model requires that a server store and
return exactly what the user provided."
see http://www.openldap.org/lists/openldap-technical/201303/msg00189.html
Ciao, Michael.
10 years, 7 months
