- openldap-bugs - openldap.org

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by hyc＠symas.com 01 Sep '07

01 Sep '07

ando(a)sys-net.it wrote: > ando(a)sys-net.it wrote: > >> Please disregard; there's still a couple of issues that I wasn't aware >> and that need to be dealt with. Actually, caching those results may >> make further requests that look compatible with this to erroneously use >> that dataset. > > The issue here is that a search exceeding sizelimit, if not cached, > would destroy cacheability of all searches contained in it. Since a > search that could exceed sizelimit is likely to include substrings or > so, things like (mail=*domain.com) exceeding sizelimit would make all > searches for (mail=foo(a)domain.com) non cacheable. I fear there's little > to do about this, unless we want to heavily rework query containment > stuff (and I don't feel like, as I'm having hard times in understanding > what that code does). So I have now a bottomline fix that handles > sizelimit correctly, but makes all cacheable queries contained in the > offending one no longer answerable. Can you spell out again what behavior you're aiming for? I think what makes sense so far is: If the client provides a sizelimit, save that away. Forward the request with no sizelimit, so the cache can see everything. If the forwarded request hits a sizelimit, I think we can still use the result. While there's no guarantee that repeated attempts to search the remote server would return the exact same set of entries, there's also no harm done if the cache does so. But if the result exceeds the cache's sizelimit, the result set must be uncached, same as now. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by ando＠sys-net.it 01 Sep '07

01 Sep '07

ando(a)sys-net.it wrote: > Please disregard; there's still a couple of issues that I wasn't aware > and that need to be dealt with. Actually, caching those results may > make further requests that look compatible with this to erroneously use > that dataset. The issue here is that a search exceeding sizelimit, if not cached, would destroy cacheability of all searches contained in it. Since a search that could exceed sizelimit is likely to include substrings or so, things like (mail=*domain.com) exceeding sizelimit would make all searches for (mail=foo(a)domain.com) non cacheable. I fear there's little to do about this, unless we want to heavily rework query containment stuff (and I don't feel like, as I'm having hard times in understanding what that code does). So I have now a bottomline fix that handles sizelimit correctly, but makes all cacheable queries contained in the offending one no longer answerable. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by ando＠sys-net.it 01 Sep '07

01 Sep '07

ando(a)sys-net.it wrote: > here's a patch that implements the above. > > <http://www.sys-net.it/~ando/Download/pcache_sizelimit.patch> Please disregard; there's still a couple of issues that I wasn't aware and that need to be dealt with. Actually, caching those results may make further requests that look compatible with this to erroneously use that dataset. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by ando＠sys-net.it 01 Sep '07

01 Sep '07

ando(a)sys-net.it wrote: > I'm getting inclined towards handling this case like a regular search, > only replacing LDAP_SUCCESS with LDAP_SIZELIMIT_EXCEEDED. In fact, it > would be more appropriate to make this request not cacheable, because > there's no guarantee the server would return the same entries for > repeated searches, but for caching purposes this shouldn't really matter. Ralf, here's a patch that implements the above. <http://www.sys-net.it/~ando/Download/pcache_sizelimit.patch> Does it harmonize with your improvements? Cheers, p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by ando＠sys-net.it 01 Sep '07

01 Sep '07

Howard Chu wrote: >> Right. Well, I understand if no limit was hit the request would become >> cacheable. In this case, the approach you suggest would be fine. > > Of course the result size has to also fit within the cache's configured > size limit. We may want to add another type of negative caching record > here for this case "we already know the result set is too large, ignore > the cache DB"... I'm getting inclined towards handling this case like a regular search, only replacing LDAP_SUCCESS with LDAP_SIZELIMIT_EXCEEDED. In fact, it would be more appropriate to make this request not cacheable, because there's no guarantee the server would return the same entries for repeated searches, but for caching purposes this shouldn't really matter. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5102) back-perl build issues
by ando＠sys-net.it 31 Aug '07

31 Aug '07

hyc(a)symas.com wrote: > quanah(a)zimbra.com wrote: >> In the name of getting 2.4.5 released, I've updated to Kurt's suggestion >> until such a time as this can be acted upon otherwise. > > That's fine. > > But I think renaming config.h to slapconf.h is a good idea anyway. > > Another thing that has nagged me ever since we unified the slap tools into > slapd - should have renamed slapadd.c -> tooladd.c etc... Personally, not. Only, slap* makes them quite personal, while tool* sounds quite generic. It's like calling them db*: could be any type of db (no pun with Berkeley intended :). p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5102) back-perl build issues
by hyc＠symas.com 31 Aug '07

31 Aug '07

quanah(a)zimbra.com wrote: > In the name of getting 2.4.5 released, I've updated to Kurt's suggestion > until such a time as this can be acted upon otherwise. That's fine. But I think renaming config.h to slapconf.h is a good idea anyway. Another thing that has nagged me ever since we unified the slap tools into slapd - should have renamed slapadd.c -> tooladd.c etc... Any objections? -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5102) back-perl build issues
by quanah＠zimbra.com 31 Aug '07

31 Aug '07

--On August 25, 2007 7:34:22 AM +0000 ando(a)sys-net.it wrote: > Kurt Zeilenga wrote: > >>> ConfigReply is declared in slapd/config.h; could it be another, >>> system-related config.h is present in an earlier include path? >>> >>> Even though, I see in the above cc command-line -I .. is present... >> >> Personally, I think have -I., -I.., etc. is really bad. This causes >> # include "config.h" >> and >> # include <config.h> >> >> to be equivalent. However, there could be both a local config.h and a >> config.h somewhere in the system include directory. >> >> I would be better if slapd/config.h was included here by: >> # include "../config.h" >> >> and -I.. not be added to the include path. It possible -I.. is causing >> a system header to be eclipsed by the local header. > > -I.. is needed to include slap.h, proto-slap.h, ... > > I'd rather rename config.h, which is quite generic, into slapconf.h or so. In the name of getting 2.4.5 released, I've updated to Kurt's suggestion until such a time as this can be acted upon otherwise. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: ITS#5112 -> 4
by rhafer＠suse.de 30 Aug '07

30 Aug '07

On Thursday 30 August 2007 20:34, Pierangelo Masarati wrote: > Ralf, > > are you working at ITS#5112-4? Yes. I have a fix for #5112 (the memory leak) mostly ready. As you/Howard suggested I just moved the cache populating code into the cleanup handler, seems to work quite well and has the advantage that final result is send to the client a lot earlier when caching large results (as you already noted). I am currently looking into fixing #5113. Haven't worked on #5114 yet, but the basic fix should be fairly easy. Changing the code to handle the limits inside pcache might be a bit harder though. > I might work on them during the weekend, > if you don't. It is my intention to avoid interfering with the release > of 2.4.5, though. Yes, that's what I intend as well. That's why I didn't submit anything yet. -- Ralf

1 0

Re: (ITS#5116) registeredAddress cannot be deleted
by binoy＠cordys.com 30 Aug '07

30 Aug '07

Hi, Please ignore the earlier message. It was a mistake. >Quanah Gibson-Mount [quanah(a)zimbra.com] wrote >This isn't a bug. Fix the way you delete the attribute. I use LDAPModification mdfn = new LDAPModification(LDAPModification.DELETE, entry.getAttribute("registeredAddress")); connection.modify(entry.getDN(), mdfn); Any suggestion on how should I correct this? Thanks and Regards, Binoy Joseph *********************************************************************** The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. This email does not constitute any commitment from Cordys Holding BV or any of its subsidiaries except when expressly agreed in a written agreement between the intended recipient and Cordys Holding BV or its subsidiaries. ***********************************************************************

1 0

ITS#5112 -> 4
by ando＠sys-net.it 30 Aug '07

30 Aug '07

Ralf, are you working at ITS#5112-4? I might work on them during the weekend, if you don't. It is my intention to avoid interfering with the release of 2.4.5, though. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5116) registeredAddress cannot be deleted
by quanah＠zimbra.com 30 Aug '07

30 Aug '07

--On Thursday, August 30, 2007 9:28 AM +0000 binoy(a)cordys.com wrote: > Full_Name: Binoy > Version: 2.2.26 > OS: Windows > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (210.210.79.19) > > > The attribute registeredAddress of an authenticationUser, once entered; > cannot be deleted. The error says, no equality matching rule. This isn't a bug. Fix the way you delete the attribute. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#5116) registeredAddress cannot be deleted
by binoy＠cordys.com 30 Aug '07

30 Aug '07

Full_Name: Binoy Version: 2.2.26 OS: Windows URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (210.210.79.19) The attribute registeredAddress of an authenticationUser, once entered; cannot be deleted. The error says, no equality matching rule.

1 0

Re: (ITS#5115) SASL authentication for AD
by quanah＠zimbra.com 29 Aug '07

29 Aug '07

--On Wednesday, August 29, 2007 6:25 PM +0000 rbeckett(a)exelixis.com wrote: > Any help would be appreciated. The ITS system is for bugs, not for issues you have building OpenLDAP. I suggest reading the OpenLDAP FAQ entries on linking, and becoming generally familiar with how to build software in non-standard paths. The FAQ entry can be found at: <http://www.openldap.org/faq/data/cache/1113.html> My OpenSSL build looked like: %configure env CC=/usr/local/bin/gcc CFLAGS='-g -O2' CXXFLAGS='-g -O2' \ PERL=/usr/bin/perl LD_RUN_PATH=/usr/local/lib \ ./Configure --prefix=/usr/local --openssldir=/usr/local/openssl \ shared linux-x86_64 -g -O2 %build env LD_RUN_PATH="/usr/local/lib" make %install env LD_LIBRARY_PATH="/usr/local/lib" \ LD_RUN_PATH="/usr/local/lib" \ make install INSTALL_PREFIX=%package% Note the use of LD_RUN_PATH. My OpenLDAP build looked like: %configure LD_LIBRARY_PATH="/usr/local/lib" CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ CFLAGS='-g -O2 -DSLAP_LIGHTWEIGHT_DISPATCHER' CXXFLAGS='-g -O2' sh configure --datadir='${prefix}/lib' --libexecdir='${prefix}/lib' --sharedstatedir='${prefix}/lib' \ --prefix=/usr/local \ --disable-ipv6 \ --with-cyrus-sasl \ --with-tls \ --enable-dynamic \ --enable-slapd \ --enable-modules \ --enable-spasswd \ --enable-rewrite \ --enable-rlookups \ --enable-wrappers \ --enable-backends=mod \ --disable-shell \ --disable-sql \ --enable-overlays=mod \ --enable-slurpd=yes \ --enable-slapi=yes make depend %build make -j3 %check make test %install make install DESTDIR=%package% STRIP="" I'll also note that the error you are receiving from the SASL bind has nothing to do with SSL specifically. All it notes is that the AD server has no SASL mechanisms presented that match the SASL mechanisms available to OpenLDAP. If you are wanting to support SASL/EXTERNAL, you'll definitely need to fix your build with OpenSSL support. I've compiled OpenSSL into /usr/local on many occassions without problem, but I use the correct flags to gcc, too, which I don't see indicated in your build. You do not indicate whether or not you link OpenLDAP against cyrus-sasl, but that will be necessary for support of some of the other SASL mechanisms. If you plan on doing SASL/GSSAPI binds to AD, you'll also need to build a Kerberos distribution for its libraries (I suggest Heimdal), and link cyrus-sasl against that before building OpenLDAP linked against cyrus-sasl. You may also want to read: <http://www.stanford.edu/services/directory/openldap/configuration/> --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#5115) SASL authentication for AD
by rbeckett＠exelixis.com 29 Aug '07

29 Aug '07

Full_Name: Richard Beckett Version: 2.3.38 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (65.209.203.254) I have posted this before but still have not resolved the problem. I am building OpenLDAP 2.3.38. on a RHEL ES v4 host When I make it without having first built TLS/SSL (openssl-0.9.8e) it builds and runs fine. However I need the SSL and SASL to authenticate to an Active Directory server. When I build SSL as follows: ./configure shared --openssldir=/usr/local make make install and then try to make OpenLDAP I ge the following: cc -g -O2 -o apitest apitest.o ./.libs/libldap.a /usr/local/src/LDAP/openldap-2.3.38/libraries/liblber/.libs/liblber.a -L/usr/kerberos/lib -L/lib -L/usr/lib/mysql ../../libraries/liblber/.libs/liblber.a ../../libraries/liblutil/liblutil.a /usr/lib/libsasl2.so -ldl -lcrypt -lssl -lcrypto -lresolv ./.libs/libldap.a(os-ip.o)(.text+0x606): In function `ldap_connect_to_host': /usr/local/src/LDAP/openldap-2.3.38/libraries/libldap/os-ip.c:205: warning: `sys_errlist' is deprecated; use `strerror' or `strerror_r' instead ./.libs/libldap.a(os-ip.o)(.text+0x5fd):/usr/local/src/LDAP/openldap-2.3.38/libraries/libldap/os-ip.c:205: warning: `sys_nerr' is deprecated; use `strerror' or `strerror_r' instead ./.libs/libldap.a(tls.o)(.text+0x593): In function `sb_tls_bio_read': /usr/local/src/LDAP/openldap-2.3.38/libraries/libldap/tls.c:676: undefined reference to `BIO_clear_flags' ./.libs/libldap.a(tls.o)(.text+0x5b8):/usr/local/src/LDAP/openldap-2.3.38/libraries/libldap/tls.c:680: undefined reference to `BIO_set_flags' ./.libs/libldap.a(tls.o)(.text+0x60b): In function `sb_tls_bio_write': /usr/local/src/LDAP/openldap-2.3.38/libraries/libldap/tls.c:703: undefined reference to `BIO_clear_flags' ./.libs/libldap.a(tls.o)(.text+0x630):/usr/local/src/LDAP/openldap-2.3.38/libraries/libldap/tls.c:707: undefined reference to `BIO_set_flags' ./.libs/libldap.a(tls.o)(.text+0x19f0): In function `ldap_pvt_tls_init_def_ctx': /usr/local/src/LDAP/openldap-2.3.38/libraries/libldap/tls.c:374: undefined reference to `SSL_CTX_set_info_callback' collect2: ld returned 1 exit status make[2]: *** [apitest] Error 1 make[2]: Leaving directory `/usr/local/src/LDAP/openldap-2.3.38/libraries/libldap' make[1]: *** [all-common] Error 1 make[1]: Leaving directory `/usr/local/src/LDAP/openldap-2.3.38/libraries' make: *** [all-common] Error 1 My environment is: HOSTNAME=ldaplx01.exelixis.com TERM=xterm SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=::ffff:172.29.4.19 51159 22 OLDPWD=/usr/local/lib SSH_TTY=/dev/pts/1 USER=root LD_LIBRARY_PATH=/usr/local/lib LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: MAIL=/var/spool/mail/root PATH=/opt/quest/bin:/opt/quest/sbin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin:/root/bin INPUTRC=/etc/inputrc PWD=/usr/local/src/LDAP/openldap-2.3.38 LANG=en_US.UTF-8 SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass SHLVL=1 HOME=/root LOGNAME=root SSH_CONNECTION=::ffff:172.29.4.19 51159 ::ffff:172.29.24.108 22 LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/bin/env Without SSL I am unable to authenticate to the Active Directory server. I get the following message: SASL/EXTERNAL authentication started ldap_perror ldap_sasl_interactive_bind_s: Unknown authentication method (-6) additional info: SASL(-4): no mechanism available: I have ssl yes set in /etc/ldap.conf and I assume (always dangerous) that the problem lies with the fact that I am unable to build ldap with SSL. Any help would be appreciated. Richard-Beckett-070829

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by hyc＠symas.com 29 Aug '07

29 Aug '07

ando(a)sys-net.it wrote: > rhafer(a)suse.de wrote: >> On Wednesday 29 August 2007 18:48, ando(a)sys-net.it wrote: >>> rhafer(a)suse.de wrote: >>>> But a malicous client can then just send requests with sizelimit 1. Those >>>> query will get cached and the database is of no real use anymore (IMO). >>> Well, in this case, the proxycache should either change the sizelimit >>> (and the timelimit) to unlimited, and deal with client-requested limits >>> locally, >> This seems like the nicest approach to solve the problem. But seems to be a >> bit more effort. > > I'd go this way, yes. > >> My suggest for a quick fix for this issue would be to just not cache queries >> that return one off the _LIMIT_EXCEEDED error codes. Probably with checking >> if it was a client-requested limit or a limit of the server side (in which >> case we could probably cache the results). > > Then please make a clear statement in the code that it's a temporary > workaround, so it is easier to spot and remove, eventually. > >>> or consider uncacheable those requests that specify a time or a >>> size limit. >> Well that would cause even those queries being uncachable that would not hit >> the requested limit. If I understand you correctly. > > Right. Well, I understand if no limit was hit the request would become > cacheable. In this case, the approach you suggest would be fine. Of course the result size has to also fit within the cache's configured size limit. We may want to add another type of negative caching record here for this case "we already know the result set is too large, ignore the cache DB"... -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by ando＠sys-net.it 29 Aug '07

29 Aug '07

rhafer(a)suse.de wrote: > On Wednesday 29 August 2007 18:48, ando(a)sys-net.it wrote: >> rhafer(a)suse.de wrote: >>> But a malicous client can then just send requests with sizelimit 1. Those >>> query will get cached and the database is of no real use anymore (IMO). >> Well, in this case, the proxycache should either change the sizelimit >> (and the timelimit) to unlimited, and deal with client-requested limits >> locally, > This seems like the nicest approach to solve the problem. But seems to be a > bit more effort. I'd go this way, yes. > My suggest for a quick fix for this issue would be to just not cache queries > that return one off the _LIMIT_EXCEEDED error codes. Probably with checking > if it was a client-requested limit or a limit of the server side (in which > case we could probably cache the results). Then please make a clear statement in the code that it's a temporary workaround, so it is easier to spot and remove, eventually. >> or consider uncacheable those requests that specify a time or a >> size limit. > Well that would cause even those queries being uncachable that would not hit > the requested limit. If I understand you correctly. Right. Well, I understand if no limit was hit the request would become cacheable. In this case, the approach you suggest would be fine. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by rhafer＠suse.de 29 Aug '07

29 Aug '07

On Wednesday 29 August 2007 18:48, ando(a)sys-net.it wrote: > rhafer(a)suse.de wrote: > > But a malicous client can then just send requests with sizelimit 1. Those > > query will get cached and the database is of no real use anymore (IMO). > > Well, in this case, the proxycache should either change the sizelimit > (and the timelimit) to unlimited, and deal with client-requested limits > locally, This seems like the nicest approach to solve the problem. But seems to be a bit more effort. My suggest for a quick fix for this issue would be to just not cache queries that return one off the _LIMIT_EXCEEDED error codes. Probably with checking if it was a client-requested limit or a limit of the server side (in which case we could probably cache the results). > or consider uncacheable those requests that specify a time or a > size limit. Well that would cause even those queries being uncachable that would not hit the requested limit. If I understand you correctly. > On the contrary, they should be considered answerable if a > corresponding request is cached, and the limits should be checked locally. -- Ralf

1 0

Re: (ITS#5112) memory leak in pcache overlay
by hyc＠symas.com 29 Aug '07

29 Aug '07

Pierangelo Masarati wrote: > hyc(a)symas.com wrote: > >> Ah yes, that's correct. I misspoke - the DB is written when the cache has >> received an LDAP_RESULT from the remote server. The client gets this result >> when the cache has finished writing. We could move the DB write to a separate >> cleanup handler instead of running it in the response handler, which would free >> up the client sooner. > > I also noticed that > <http://www.openldap.org/lists/openldap-devel/200708/msg00033.html>, and > I wanted to do something similar, but considering Ralf's observations, > multiple clients concurrently running the same search request would > cause it to be cached multiple times, because until the first one is not > cached the others will not find it. OTOH, letting the others know it's > being cached, and making them wait until caching is done would imply > some delay, which could be long based on how many entries are being > cached. However, given the purpose of a proxy cache, and considering > that so many identical searches are not likely to occur simultaneously, > I'd prefer to queue requests that are being cached. Agreed. Even with a delay it ought to be faster than making the remote search again. But this is probably not worth changing at this moment. Let's get 2.4.5 out first. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5112) memory leak in pcache overlay
by ando＠sys-net.it 29 Aug '07

29 Aug '07

hyc(a)symas.com wrote: > Ah yes, that's correct. I misspoke - the DB is written when the cache has > received an LDAP_RESULT from the remote server. The client gets this result > when the cache has finished writing. We could move the DB write to a separate > cleanup handler instead of running it in the response handler, which would free > up the client sooner. I also noticed that <http://www.openldap.org/lists/openldap-devel/200708/msg00033.html>, and I wanted to do something similar, but considering Ralf's observations, multiple clients concurrently running the same search request would cause it to be cached multiple times, because until the first one is not cached the others will not find it. OTOH, letting the others know it's being cached, and making them wait until caching is done would imply some delay, which could be long based on how many entries are being cached. However, given the purpose of a proxy cache, and considering that so many identical searches are not likely to occur simultaneously, I'd prefer to queue requests that are being cached. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5112) memory leak in pcache overlay
by hyc＠symas.com 29 Aug '07

29 Aug '07

rhafer(a)suse.de wrote: > On Wednesday 29 August 2007 18:10, Howard Chu wrote: >> Ralf Haferkamp wrote: >>> On Wednesday 29 August 2007 17:36, Howard Chu wrote: >>>> rhafer(a)suse.de wrote: >>>>> Full_Name: Ralf Haferkamp >>>>> Version: RE23, HEAD >>>>> >>>>> Valgrind gives me the log pasted below when I abort an ldapsearch >>>>> command (CTRL-C) that is running against a back-ldap database that uses >>>>> the pcache overlay. >>>> Makes sense. We need to use the cleanup handler and free the saved query >>>> info if op->o_abandon is set. Do you want to code this up? >>> I am still working on that, but checking for op->o_abandon doesn't seem >>> to be enough. E.g. if I abort the ldapsearch while slapd is updating its >>> cache (all entries have been returned, but the final LDAP_RESULT to the >>> client is still missing) a filter_free in the cleanup handler will >>> invalidly free the filter. >>> >>> We somehow need to make sure that the query is only free'd if nothing has >>> been written to the cache database. >> I don't understand the problem. Nothing can get written to the DB until the >> LDAP_RESULT has been sent. I.e., rs->sr_result == REP_RESULT. > Hm, but the ldapsearch command only returns after the complete query result > has been written to the cache (at least here it behaves so), which can take > quite a while. So it seems to me that the client has not received the > LDAP_RESULT. Ah yes, that's correct. I misspoke - the DB is written when the cache has received an LDAP_RESULT from the remote server. The client gets this result when the cache has finished writing. We could move the DB write to a separate cleanup handler instead of running it in the response handler, which would free up the client sooner. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by ando＠sys-net.it 29 Aug '07

29 Aug '07

rhafer(a)suse.de wrote: > But a malicous client can then just send requests with sizelimit 1. Those > query will get cached and the database is of no real use anymore (IMO). Well, in this case, the proxycache should either change the sizelimit (and the timelimit) to unlimited, and deal with client-requested limits locally, or consider uncacheable those requests that specify a time or a size limit. On the contrary, they should be considered answerable if a corresponding request is cached, and the limits should be checked locally. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by rhafer＠suse.de 29 Aug '07

29 Aug '07

On Wednesday 29 August 2007 18:36, Pierangelo Masarati wrote: > hyc(a)symas.com wrote: > >> pcache currently cache the results of search that hit server- or > >> clientside size- or timelimits. That means that subsequent search will > >> get the (incomplete) results from the cache. I guess pcache should only > >> cache operations that returned LDAP_SUCCESS. > > > > Makes sense... > > Well, I agree for timelimit, but sizelimit might be questionable. In > fact, the only reason not to cache searches ending in sizelimit exceeded > is that the size limit may depend on the client's identity. But this is > true in general also for access to entries and to entry data, but we > don't cache based on the identity of the client, so data cached with one > identity (set A) might differ from data that would be returned by > another identity for the very same search (set B), and both the relative > complement of A in B and of B in A may be not empty. So, if we accept > this for ACLs (no differences between the results returned for requests > with different identities) I don't see why we should differentiate with > respect to the size limit. But a malicous client can then just send requests with sizelimit 1. Those query will get cached and the database is of no real use anymore (IMO). -- Ralf

1 0

Re: (ITS#5114) pcache cache results for searches that hit size/timelimit
by ando＠sys-net.it 29 Aug '07

29 Aug '07

hyc(a)symas.com wrote: >> pcache currently cache the results of search that hit server- or clientside >> size- or timelimits. That means that subsequent search will get the (incomplete) >> results from the cache. I guess pcache should only cache operations that >> returned LDAP_SUCCESS. > > Makes sense... Well, I agree for timelimit, but sizelimit might be questionable. In fact, the only reason not to cache searches ending in sizelimit exceeded is that the size limit may depend on the client's identity. But this is true in general also for access to entries and to entry data, but we don't cache based on the identity of the client, so data cached with one identity (set A) might differ from data that would be returned by another identity for the very same search (set B), and both the relative complement of A in B and of B in A may be not empty. So, if we accept this for ACLs (no differences between the results returned for requests with different identities) I don't see why we should differentiate with respect to the size limit. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5112) memory leak in pcache overlay
by rhafer＠suse.de 29 Aug '07

29 Aug '07

On Wednesday 29 August 2007 18:10, Howard Chu wrote: > Ralf Haferkamp wrote: > > On Wednesday 29 August 2007 17:36, Howard Chu wrote: > >> rhafer(a)suse.de wrote: > >>> Full_Name: Ralf Haferkamp > >>> Version: RE23, HEAD > >>> > >>> Valgrind gives me the log pasted below when I abort an ldapsearch > >>> command (CTRL-C) that is running against a back-ldap database that uses > >>> the pcache overlay. > >> > >> Makes sense. We need to use the cleanup handler and free the saved query > >> info if op->o_abandon is set. Do you want to code this up? > > > > I am still working on that, but checking for op->o_abandon doesn't seem > > to be enough. E.g. if I abort the ldapsearch while slapd is updating its > > cache (all entries have been returned, but the final LDAP_RESULT to the > > client is still missing) a filter_free in the cleanup handler will > > invalidly free the filter. > > > > We somehow need to make sure that the query is only free'd if nothing has > > been written to the cache database. > > I don't understand the problem. Nothing can get written to the DB until the > LDAP_RESULT has been sent. I.e., rs->sr_result == REP_RESULT. Hm, but the ldapsearch command only returns after the complete query result has been written to the cache (at least here it behaves so), which can take quite a while. So it seems to me that the client has not received the LDAP_RESULT. With the suggested fix I get this, if I abort the ldapsearch during that time, as the op->o_abandon is set in cleanup handler: ==26949== Invalid free() / delete / delete[] ==26949== at 0x4C226DB: free (in /usr/lib64/valgrind/amd64-linux/vgpreload_memcheck.so) ==26949== by 0x508A819: ber_memfree_x (memory.c:152) ==26949== by 0x4AC9D7: slap_sl_free (sl_malloc.c:455) ==26949== by 0x44797D: filter_free_x (filter.c:526) ==26949== by 0x4478C9: filter_free_x (filter.c:509) ==26949== by 0x4479D9: filter_free (filter.c:538) ==26949== by 0x5A7794: free_query (pcache.c:1126) ==26949== by 0x5AF9B0: pcache_db_close (pcache.c:3282) ==26949== by 0x4C2F87: over_db_close (backover.c:192) ==26949== by 0x452658: backend_shutdown (backend.c:365) ==26949== by 0x47C03E: slap_shutdown (init.c:253) ==26949== by 0x422D65: main (main.c:943) ==26949== Address 0xA533A68 is 0 bytes inside a block of size 24 free'd ==26949== at 0x4C226DB: free (in /usr/lib64/valgrind/amd64-linux/vgpreload_memcheck.so) ==26949== by 0x508A819: ber_memfree_x (memory.c:152) ==26949== by 0x4AC9D7: slap_sl_free (sl_malloc.c:455) ==26949== by 0x44797D: filter_free_x (filter.c:526) ==26949== by 0x4478C9: filter_free_x (filter.c:509) ==26949== by 0x4479D9: filter_free (filter.c:538) ==26949== by 0x5A9F7F: pcache_op_cleanup (pcache.c:1886) ==26949== by 0x456A69: slap_cleanup_play (result.c:344) ==26949== by 0x4571DC: send_ldap_response (result.c:526) ==26949== by 0x457935: slap_send_ldap_result (result.c:632) ==26949== by 0x4F7AE4: ldap_back_search (search.c:528) ==26949== by 0x4C3FD0: overlay_op_walk (backover.c:652) -- Ralf

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs