Re: (ITS#9002) Add option to slapcat to honor rtxnsize setting
by hyc＠symas.com
michael(a)stroeder.com wrote:
> On 3/29/19 8:58 PM, quanah(a)openldap.org wrote:
>> To work around this, slapcat could be given an option to honor the rtxnsize
>> setting in slapd.conf/cn=config.
>> [..]
>> It should be noted in the man page section for this option that the value of
>> such a backup is of dubious quality, since it is no longer an exact "point in
>> time" representation of the database but rather a bunch of different "point in
>> times" stitched together into a single file.
>
> Even if slapcat now outputs an exact "point in time" representation of
> the database this does not say anything about consistency of related
> LDAP entries anyway.
>
> But I wonder how rtxnsize is supposed to interop with
> LDAP transactions in 2.5.
There is no interaction. LDAP transactions only affect write ops.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
4 years, 2 months
Re: (ITS#9002) Add option to slapcat to honor rtxnsize setting
by michael＠stroeder.com
On 3/29/19 8:58 PM, quanah(a)openldap.org wrote:
> To work around this, slapcat could be given an option to honor the rtxnsize
> setting in slapd.conf/cn=config.
> [..]
> It should be noted in the man page section for this option that the value of
> such a backup is of dubious quality, since it is no longer an exact "point in
> time" representation of the database but rather a bunch of different "point in
> times" stitched together into a single file.
Even if slapcat now outputs an exact "point in time" representation of
the database this does not say anything about consistency of related
LDAP entries anyway.
But I wonder how rtxnsize is supposed to interop with
LDAP transactions in 2.5.
4 years, 2 months
(ITS#9002) Add option to slapcat to honor rtxnsize setting
by quanah＠openldap.org
Full_Name: Quanah Gibson-Mount
Version: 2.4.47
OS: N/A
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (47.208.144.40)
There is currently an issue with back-mdb, where if the database is quite large
(10GB+) and the environment is experiencing a heavy write load, that if you
start a backup (slapcat or mdb_copy), the database will heavily fragment and
grow significantly in size due to the read lock from the slapcat/mdb_copy
process.
To work around this, slapcat could be given an option to honor the rtxnsize
setting in slapd.conf/cn=config. This would allow the read lock to be released
periodically and thus reduce the amount of fragmentation that occurs during the
backup process. There is not (at this time) a solution proposed for mdb_copy.
It should be noted in the man page section for this option that the value of
such a backup is of dubious quality, since it is no longer an exact "point in
time" representation of the database but rather a bunch of different "point in
times" stitched together into a single file.
4 years, 2 months
Re: (ITS#9001) try_read1msg() takes O(n) steps to lookup a matching request
by ondra＠mistotebe.net
On Fri, Mar 29, 2019 at 01:38:30PM +0000, ondra(a)openldap.org wrote:
> Full_Name: Ondrej Kuznik
> Version: re24/master
> OS:
> URL: https://github.com/mistotebe/openldap/tree/its9001
> Submission from: (NULL) (82.10.24.68)
>
>
> If a client has thousands or more requests in flight at the same time, it has to
> look up the right request every time it receives a response. As the requests are
> tracked in a doubly-linked list, lookup tends to take O(n) steps whichever
> direction we take and things generally get out of hand from there on.
>
> The linked branch puts them into a TAvl for quicker lookup, with another commit
> to let people test as (t)avl code is part of liblutil at the moment.
A trivial program to expose this behaviour is available at
ftp://ftp.openldap.org/incoming/searcher.c
The same program takes under a second to handle 100000 searches for
cn=monitor and children when linked to the patched libldap.
--
Ondřej Kuzník
Senior Software Engineer
Symas Corporation http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP
4 years, 2 months
(ITS#9001) try_read1msg() takes O(n) steps to lookup a matching request
by ondra＠openldap.org
Full_Name: Ondrej Kuznik
Version: re24/master
OS:
URL: https://github.com/mistotebe/openldap/tree/its9001
Submission from: (NULL) (82.10.24.68)
If a client has thousands or more requests in flight at the same time, it has to
look up the right request every time it receives a response. As the requests are
tracked in a doubly-linked list, lookup tends to take O(n) steps whichever
direction we take and things generally get out of hand from there on.
The linked branch puts them into a TAvl for quicker lookup, with another commit
to let people test as (t)avl code is part of liblutil at the moment.
4 years, 2 months
Re: (ITS#8997) openldap-nssov/back ldap segfault
by matt＠pallissard.net
--mcr7yvikhm7jn4cn
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
> Please try this patch.
Right on. Ran some initial tests, so far so good.
I'll bang on it for a few hours and follow up sometime in the late afternoon.
Thanks a bunch! You guys rock!
Matt Pallissard
--mcr7yvikhm7jn4cn
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iHUEABYIAB0WIQTvIUMPApUGn6YFkXl1uof+t048SQUCXJzldgAKCRB1uof+t048
SaMGAP9G5n22WPLEVK/7TJ3lSGjl+BNRaj5AC0U3Im+DE60JewD/TrMXYDYlwpu8
kdOwSqY3UvRKHMdI13cEg8MG6iH3yAo=
=chSM
-----END PGP SIGNATURE-----
--mcr7yvikhm7jn4cn--
4 years, 2 months
Re: (ITS#8997) openldap-nssov/back ldap segfault
by hyc＠symas.com
This is a multi-part message in MIME format.
--------------23850F02105CA62A388BCB06
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Pallissard, Matthew wrote:
>> Actually now that I'm looking more closely at this output; the last line below looks interesting.
>>
>> (I probably should have tacked this on the backtrace)
>>
>> Thread 4 "slapd" received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7ffff0c7c700 (LWP 341)]
>> nssov_dn2uid (op=0x7ffff0c7b730, ni=0x555555a24420, dn=0x7fffe811eab0, uid=0x7fffe8002450) at passwd.c:137
>> 137 passwd.c: No such file or directory.
>>
>
> Sorry, that was from the wrong output, see below for the correct snippet.
>
> Thread 3 "slapd" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff277f700 (LWP 170)]
> nssov_dn2uid (op=0x7ffff277e730, ni=0x555555a24420, dn=0x7fffe4128430, uid=0x7fffe4002430) at passwd.c:137
> 137 passwd.c: No such file or directory.
>
> Matt Pallissard
>
Thanks that was a crucial piece.
Please try this patch.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
--------------23850F02105CA62A388BCB06
Content-Type: text/plain; charset=UTF-8;
name="dif.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="dif.txt"
diff --git a/servers/slapd/back-ldap/search.c b/servers/slapd/back-ldap/search.c
index 63345aed22..82e72aa8a7 100644
--- a/servers/slapd/back-ldap/search.c
+++ b/servers/slapd/back-ldap/search.c
@@ -1006,6 +1006,7 @@ retry:
e = ldap_first_entry( lc->lc_ld, result );
if ( e == NULL ) {
/* the entry exists, but it doesn't match the filter? */
+ rc = LDAP_NO_RESULTS_RETURNED;
goto cleanup;
}
--------------23850F02105CA62A388BCB06--
4 years, 2 months
Re: (ITS#8997) openldap-nssov/back ldap segfault
by matt＠pallissard.net
--mb3a5envygzqum4d
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
> Actually now that I'm looking more closely at this output; the last line below looks interesting.
>
> (I probably should have tacked this on the backtrace)
>
> Thread 4 "slapd" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff0c7c700 (LWP 341)]
> nssov_dn2uid (op=0x7ffff0c7b730, ni=0x555555a24420, dn=0x7fffe811eab0, uid=0x7fffe8002450) at passwd.c:137
> 137 passwd.c: No such file or directory.
>
Sorry, that was from the wrong output, see below for the correct snippet.
Thread 3 "slapd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff277f700 (LWP 170)]
nssov_dn2uid (op=0x7ffff277e730, ni=0x555555a24420, dn=0x7fffe4128430, uid=0x7fffe4002430) at passwd.c:137
137 passwd.c: No such file or directory.
Matt Pallissard
--mb3a5envygzqum4d
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iHUEABYIAB0WIQTvIUMPApUGn6YFkXl1uof+t048SQUCXJzYtgAKCRB1uof+t048
ST8nAP9UWBuG0358vqbQnGRxp4y4YMol81iU7UJswPBjPaBzgwD9HNk+aOAJJ1rn
Czboi8ftlPsM0xnB34jmHtIYLz6EWww=
=Vmu0
-----END PGP SIGNATURE-----
--mb3a5envygzqum4d--
4 years, 2 months
Re: (ITS#8997) openldap-nssov/back ldap segfault
by matt＠pallissard.net
--vpit5odwafhd55ju
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 2019-03-28T06:54:35, Pallissard, Matthew wrote:
> On 2019-03-21T15:13:30, Pallissard, Matthew wrote:
> > On 2019-03-20T22:01:40, Howard Chu wrote:
> > > matt(a)pallissard.net wrote:
> > >
> > > > 2. Turning the log level to 0 /seems/ to make the issue go away. I'=
ll report
> > > > back once I can confirm that.
>=20
> Follow up, turning the log level to 0 does *not* prevent these segfaults.
>=20
> Matt Pallissard
Actually now that I'm looking more closely at this output; the last line be=
low looks interesting.
(I probably should have tacked this on the backtrace)
=20
Thread 4 "slapd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff0c7c700 (LWP 341)]
nssov_dn2uid (op=3D0x7ffff0c7b730, ni=3D0x555555a24420, dn=3D0x7fffe811eab0=
, uid=3D0x7fffe8002450) at passwd.c:137
137 passwd.c: No such file or directory.
Matt Pallissard
--vpit5odwafhd55ju
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iHUEABYIAB0WIQTvIUMPApUGn6YFkXl1uof+t048SQUCXJzXNQAKCRB1uof+t048
SXioAQDKoJWdiIpuasjmkaDnSI/bPGhCdBEXuYQfyZV5pqe++AD9GvYTSddxicKP
wD0noDfTRI7oS5heRZHOnYO/ZK83RQc=
=djIS
-----END PGP SIGNATURE-----
--vpit5odwafhd55ju--
4 years, 2 months
Re: (ITS#8997) openldap-nssov/back ldap segfault
by matt＠pallissard.net
--dtfibegrdp7hfoq4
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On 2019-03-21T15:13:30, Pallissard, Matthew wrote:
> On 2019-03-20T22:01:40, Howard Chu wrote:
> > matt(a)pallissard.net wrote:
> >
> > > 2. Turning the log level to 0 /seems/ to make the issue go away. I'll report
> > > back once I can confirm that.
Follow up, turning the log level to 0 does *not* prevent these segfaults.
Matt Pallissard
--dtfibegrdp7hfoq4
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iHUEABYIAB0WIQTvIUMPApUGn6YFkXl1uof+t048SQUCXJzSGQAKCRB1uof+t048
SWYnAQDjIXjPORzGof/8TpqzansV7PXn5pdMnRxU14gqMcXsrwD+LS2nEqUmslpE
T4PWgWNKb0AY0eWcSNAj+klqAkolcAs=
=8ONg
-----END PGP SIGNATURE-----
--dtfibegrdp7hfoq4--
4 years, 2 months
