- openldap-bugs - openldap.org

Re: (ITS#4702) out-of-memory on huge DB ?
by Paolo.Rossi.con＠h3g.it 11 Oct '06

11 Oct '06

> No. As I already mentioned, there is no reason for slapcat to use > large amounts of RAM, it should use whatever size your BDB cache is > configured for and that's about it. It's possible there is a memory > leak, but since no such leak appears under Linux that would point > to either the Solaris C library or some other library specific to > your platform. > > Perhaps you should post your actual slapd.conf file (removing any > passwords or other sensitive info). shure, here we are: ( this one, is the 64bit test; I've use the consumer for this try) ------------------------------------------ >$ cat slapd.conf # # See slapd.conf(5) for details on configuration options. # This file should NOT be world readable. # schemacheck on include /app/ldap23/consumer/openldap/schema/core.schema include /app/ldap23/consumer/openldap/schema/ppscore.schema include /app/ldap23/consumer/openldap/schema/ppsattributes.schema include /app/ldap23/consumer/openldap/schema/capability.schema include /app/ldap23/consumer/openldap/schema/java.schema include /app/ldap23/consumer/openldap/schema/cosine.schema include /app/ldap23/consumer/openldap/schema/inetorgperson.schema #loglevel stats config acl sync loglevel stats config acl pidfile /app/ldap23/consumer/run/slapd.pid argsfile /app/ldap23/consumer/run/slapd.args threads 20 idletimeout 30 ####################################################################### # db database definitions ####################################################################### database hdb shm_key 10000 suffix "dc=foo,dc=bar" rootdn "cn=Admin,dc=foo,dc=bar" checkpoint 15360 5 cachesize 100 #cachesize 5000 idlcachesize 5000 # Cleartext passwords, especially for the rootdn, should # be avoid. See slappasswd(8) and slapd.conf(5) for details. # Use of strong authentication encouraged. rootpw XXX # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /app/ldap23/consumer/openldap-data # Indices to maintain index objectClass,contactCode,partyId,entryCSN,entryUUID eq ####################################################################### # CONSUMER # ####################################################################### syncrepl rid=102 provider=ldap://10.xxx.xxx.xxx:389 type=refreshAndPersist retry="120 +" searchbase="ou=profiles,dc=foo,dc=bar" filter="(objectClass=*)" scope=sub attrs="*" schemachecking=off binddn="cn=syncuserman,dc=foo,dc=bar" credentials=XXX bindmethod=simple ------------------------------------------ and DB_CONFIG also: ------------------------------------------ >$ cat DB_CONFIG set_lg_dir /app/ldap23/consumer/openldap-data/logs #set_cachesize 0 256000000 1 set_cachesize 0 64000000 1 set_lg_regionmax 262144 set_lg_bsize 2097152 # Automatically remove log files that are no longer needed. set_flags DB_LOG_AUTOREMOVE set_flags DB_TXN_NOSYNC #set_flags DB_TXN_NOT_DURABLE ------------------------------------------ In this run, I've set << set_cachesize >> to 64Mb due to speed up the test. Paolo

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by hyc＠symas.com 11 Oct '06

11 Oct '06

Paolo.Rossi.con(a)h3g.it wrote: > Now, the question: Huge LDAP only possible with big RAM ? :D > > > Other ideas? :/ No. As I already mentioned, there is no reason for slapcat to use large amounts of RAM, it should use whatever size your BDB cache is configured for and that's about it. It's possible there is a memory leak, but since no such leak appears under Linux that would point to either the Solaris C library or some other library specific to your platform. Perhaps you should post your actual slapd.conf file (removing any passwords or other sensitive info). -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by Paolo.Rossi.con＠h3g.it 11 Oct '06

11 Oct '06

Quanah Gibson-Mount wrote: > 4GB is the memory limit on 32-bit Solaris, IIRC. Just like it is > 2GB on 32-bit linux. Try building everything 64-bit. Hello again. Today I've rebuilt everything 64-bit To have a "fast shot", I've load a 1m dn, then I've ulimit -d 96304 the data memory for user. After a some minutes slapcat: slapcat -f openldap/slapd.conf |gzip > delete-me.ldif ch_malloc of 16396 bytes failed Assertion failed: 0, file ch_malloc.c, line 57 Abort (core dumped) and a second try: slapcat -f openldap/slapd.conf > delete-me.ldif bdb_db_open: unclean shutdown detected; attempting recovery. bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered. ch_malloc of 16396 bytes failed Assertion failed: 0, file ch_malloc.c, line 57 Abort (core dumped) at crash moment, with top: size=180M res=176M Now, the question: Huge LDAP only possible with big RAM ? :D Other ideas? :/ Paolo

1 0

(ITS#4707) patch: option to bind client socket to an address
by rtsai＠ironport.com 10 Oct '06

10 Oct '06

Full_Name: Robert Tsai Version: 2.3.27 OS: FreeBSD 6.1-RELEASE URL: ftp://ftp.openldap.org/incoming/openldap-2.3.27-bindaddr.patch.txt Submission from: (NULL) (63.251.108.100) This is a patch that provides a mechanism to bind the LDAP client connection to a desired address via ldap_set_option(..., LDAP_OPT_BINDADDR). This call saves some state which is then used by ldap_int_prepare_socket to bind the socket before opening a connection to the LDAP server.

1 0

(ITS#4706) regex.h on Solaris 10 sparc
by stew.redfield＠gmail.com 10 Oct '06

10 Oct '06

Full_Name: Stew Redfield Version: 2.3.27 OS: Solaris 10 sparc URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (65.219.72.58) /configure --prefix=/usr/local --snip-- checking regex.h presence... yes configure: WARNING: regex.h: present but cannot be compiled configure: WARNING: regex.h: check for missing prerequisite headers? configure: WARNING: regex.h: see the Autoconf documentation configure: WARNING: regex.h: section "Present But Cannot Be Compiled" configure: WARNING: regex.h: proceeding with the preprocessor's result configure: WARNING: regex.h: in the future, the compiler will take precedence configure: WARNING: ## --------------------------------------------- ## configure: WARNING: ## Report this to <http://www.openldap.org/its/> ## configure: WARNING: ## --------------------------------------------- ## checking for regex.h... yes checking for library containing regfree... none required checking for compatible POSIX regex... no configure: error: broken POSIX regex!

1 0

Re: (ITS#4704) accesslog database fails to store contextCSN at shutdown
by quanah＠stanford.edu 09 Oct '06

09 Oct '06

--On Sunday, October 08, 2006 11:36 AM +0000 openldap-its(a)OpenLDAP.org wrote: I did some testing, and found that the problem is that accesslog is creating a new contextCSN every time slapd is started. Here's my testing: (a) Search accesslog for its contextCSN: ldap-test0:/tmp# cat contextCSN.before dn: cn=accesslog contextCSN: 20061009111121Z#000000#00#000000 (b) Stop slapd, dump the accesslog DB to LDIF, look at the entry values: ldap-test0:/tmp# cat contextCSN.LDIF dn: cn=accesslog objectClass: auditContainer cn: accesslog entryCSN: 20060829205636Z#000000#00#000000 structuralObjectClass: auditContainer contextCSN: 20061009111121Z#000000#00#000000 (c) Start slapd, query accesslog for its contextCSN: ldap-test0:/tmp# cat contextCSN.startup dn: cn=accesslog contextCSN: 20061009183449Z#000000#00#000000 As you can see, a new version was generated at startup, rather than it keeping the one it had at shutdown. Why, I don't know. Then, I did the following: (a) stop slapd (b) remove the current accesslog db (since all slaves are in sync) (c) get the contextCSN: dn: cn=accesslog contextCSN: 20061009183830Z#000000#00#000000 (d) stopped slapd (e) started slapd (f) queried, the contextCSN is the same(?!!) ldap-test0:~> ldapsearch -LLL -Q -h localhost -b cn=accesslog -s base contextCSN dn: cn=accesslog contextCSN: 20061009183830Z#000000#00#000000 (g) Made a modification (h) got the current contectCSN: dn: cn=accesslog contextCSN: 20061009184042Z#000000#00#000000 (i) Stopped slapd (j) restarted slapd (k) got the contectCSN: dn: cn=accesslog contextCSN: 20061009184042Z#000000#00#000000 It is the same again. So at best guess, something "corrupted" the accesslog DB somehow? I'm not really sure. I will monitor to see if I get back into the same situation again. --Quanah -- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

1 0

Re: (ITS#4705) 11866 Segmentation fault (core dumped)
by michael＠stroeder.com 09 Oct '06

09 Oct '06

Pierangelo Masarati wrote: > michael(a)stroeder.com wrote: > > configured with BDB 4.2 (+ patches) as > > ./configure \ > --enable-modules \ > --enable-dynamic \ > --enable-backends=mod \ > --enable-overlays=mod \ > --enable-perl=no This minimalistic configure does not even compile on my system. > passes all tests I think this error has something to do with the solution for ITS#4671. > Note that I typically use "make distclean" to regenerate things. This also does not work. See my current build script below. Ciao, Michael. --------------------------------- snip --------------------------------- . /home/michael/src/build-env-openldap #export CFLAGS="-g -O4 -march=pentium4" export CFLAGS="-g -O2" export CPPFLAGS="-I/usr/include/heimdal -I/usr/include/sasl -I${BDB}/include -I${SASL}/include -I${HEIMDAL}/include" export LDFLAGS="-L${BDB}/lib -L${SASL}/lib -L${HEIMDAL}/lib -R${BDB}/lib -R${SASL}/lib -R${HEIMDAL}/lib" export LD_LIBRARY_PATH="${BDB}/lib:${SASL}/lib:${HEIMDAL}/lib" # Disable test036-meta-concurrency since it always hangs export TEST_META=no PWD=`pwd` DIRNAME=`dirname ${PWD}` PREFIX=/opt/`basename ${DIRNAME}` echo "PREFIX="${PREFIX} make distclean ./configure \ --enable-dynamic \ --enable-modules=yes \ --enable-backends=mod \ --enable-overlays=mod \ --enable-aci=yes \ --enable-cleartext \ --enable-crypt=yes \ --enable-debug=yes \ --enable-kpasswd=yes \ --enable-lmpasswd=yes \ --enable-rewrite=yes \ --enable-rlookups=yes \ --enable-slapi=yes \ --enable-slp=yes \ --enable-spasswd=yes \ --enable-wrappers=no \ --prefix=${PREFIX} \ --with-cyrus-sasl=yes \ --with-kerberos=yes \ --with-readline=yes \ --with-threads=yes \ --with-tls=yes make && make test

1 0

Re: (ITS#4705) 11866 Segmentation fault (core dumped)
by ando＠sys-net.it 08 Oct '06

08 Oct '06

michael(a)stroeder.com wrote: > Full_Name: Michael Ströder > Version: HEAD (synced now!) > OS: SuSE Linux 10.1 > URL: > Submission from: (NULL) (83.124.30.52) > > > Configured and compiled from scratch after 'make veryclean'. > > $ ./run test000 > Cleaning up test run directory leftover from previous run. > Running ./scripts/test000-rootdse... > running defines.sh > Starting slapd on TCP/IP port 9011... > ./scripts/test000-rootdse: line 31: 11951 Segmentation fault (core dumped) > $SLAPD -f $CONF1 -h $URI1 -d $LVL $TIMING >$LOG1 2>&1 > Using ldapsearch to retrieve the root DSE... > Waiting 5 seconds for slapd to start... > Waiting 5 seconds for slapd to start... > [..] > > ------------------------------------------------------------------------------------- > (gdb) bt > #0 monitor_subsys_backend_init (be=0x0, ms=0xb78f1840) at backend.c:69 > #1 0xb78e3bde in monitor_back_db_open (be=0x81df950) at init.c:2214 > #2 0x0807badf in backend_startup_one (be=0x81df950) at backend.c:212 > #3 0x0807bd73 in backend_startup (be=0x81df950) at backend.c:303 > #4 0x0805e983 in main (argc=8, argv=0xbfa39d14) at main.c:917 > > > configured with BDB 4.2 (+ patches) as ./configure \ --enable-modules \ --enable-dynamic \ --enable-backends=mod \ --enable-overlays=mod \ --enable-perl=no passes all tests (I had some problems because I seldom use modules, and I forgot --enable-dynamic). Can you point out any relevant difference? Note that I typically use "make distclean" to regenerate things. I don't know the detailed difference between distclean and veryclean, but distclean seems to depend on veryclean so it could do something more. In any case, distclean requires to re-configure before building. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

1 0

(ITS#4705) 11866 Segmentation fault (core dumped)
by michael＠stroeder.com 08 Oct '06

08 Oct '06

Full_Name: Michael Ströder Version: HEAD (synced now!) OS: SuSE Linux 10.1 URL: Submission from: (NULL) (83.124.30.52) Configured and compiled from scratch after 'make veryclean'. $ ./run test000 Cleaning up test run directory leftover from previous run. Running ./scripts/test000-rootdse... running defines.sh Starting slapd on TCP/IP port 9011... ./scripts/test000-rootdse: line 31: 11951 Segmentation fault (core dumped) $SLAPD -f $CONF1 -h $URI1 -d $LVL $TIMING >$LOG1 2>&1 Using ldapsearch to retrieve the root DSE... Waiting 5 seconds for slapd to start... Waiting 5 seconds for slapd to start... [..] ------------------------------------------------------------------------------------- (gdb) bt #0 monitor_subsys_backend_init (be=0x0, ms=0xb78f1840) at backend.c:69 #1 0xb78e3bde in monitor_back_db_open (be=0x81df950) at init.c:2214 #2 0x0807badf in backend_startup_one (be=0x81df950) at backend.c:212 #3 0x0807bd73 in backend_startup (be=0x81df950) at backend.c:303 #4 0x0805e983 in main (argc=8, argv=0xbfa39d14) at main.c:917

1 0

Re: authz-regexp matches twice on same dse (ITS#4698)
by ap＠d-dt.de 08 Oct '06

08 Oct '06

Pierangelo Masarati wrote: >> Pierangelo Masarati wrote: >> > What authz-regexp does is run an internal search. If the search returns > exactly one entry, then there's no way it can be, say, returned twice, > otherwise it would also when running aregular search. Moreover, I've > recrated you scenario in 2.3.27 and HEAD, and everything seems to work as > expected in all cases. I suspect something else is wrong, for example > data in your DB is not like it appears. The objects used actually were 'person's with auxiliary 'posixAccount' and 'organizationalRoles' on the other side. I also installed a clean directory and at least I must agree with you, that the problem is caused somewhere else: After I dumped and reloaded data with slapcat/slapadd into bdb it finally works as expected. Thanks for verifing and sorry for the inconvenience caused. A --

1 0

(ITS#4704) accesslog database fails to store contextCSN at shutdown
by quanah＠stanford.edu 08 Oct '06

08 Oct '06

Full_Name: Quanah Gibson-Mount Version: 2.3.27 OS: Linux 2.6 kernel URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (171.66.155.86) I recently set up a script to monitor whether or not my replicas are in sync with the master. I've found that since my master shuts down nightly, this triggers a bug in accesslog where it fails to store its current contextCSN at that time. Then, when the master restarts, accesslog creates a new contextCSN that the slaves pick up, but the master's main database (dc=stanford,dc=edu) doesn't. For example: nagios1:~> /usr/lib/nagios/plugins/check_syncrepl -H ldap-uat1 CRITICAL dc=stanford,dc=edu out of sync 0 weeks 0 days 2:46:12 ahead nagios1:~> ldapsearch -LLL -Q -h ldap-uat0 -s base -b cn=accesslog contextCSN dn: cn=accesslog contextCSN: 20061008111122Z#000000#00#000000 nagios1:~> ldapsearch -LLL -Q -h ldap-uat1 -s base -b dc=stanford,dc=edu contextCSN dn: dc=stanford,dc=edu contextCSN: 20061008111122Z#000000#00#000000 nagios1:~> ldapsearch -LLL -Q -h ldap-uat0 -s base -b dc=stanford,dc=edu contextCSN dn: dc=stanford,dc=edu contextCSN: 20061008082510Z#000000#00#000000 where ldap-uat0 is my master. You can see the accesslog DB has the new contextCSN, which its pushed out to the slaves, while dc=stanford,dc=edu on the master still has the correct contextCSN. This happens in my test, uat, and prod environments, and remains this way until the environments receive their first update, after which the CSN's all get sync'd. --Quanah

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by Paolo.Rossi.con＠h3g.it 08 Oct '06

08 Oct '06

> What is your entry cache size? I suggest you set it to something > small, like 100, during the slapcat test. It was 5000 I've tried 100 and ulimit mem at 1024000 after about 50 mins of slapcat: 1600 MB ca. MEM allocated coredump (1691966556 bytes) output ldif ca. 20% of full DB :( Paolo Rossi

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by hyc＠symas.com 08 Oct '06

08 Oct '06

Paolo Rossi wrote: > >> PS: 4GB of RAM is probably not enough for a 20GB database, you won't >> be able to configure caches large enough to get decent search >> performance. I think 8GB would be a practical minimum here. >> > > This is also my opinion, for good performances are not sufficient, but > now the question is: > 4GB are not enough to anyway, to run (slowly) similar huge server > without coredump? > > p.s. or maybe I missed something, somewhere... :P > What is your entry cache size? I suggest you set it to something small, like 100, during the slapcat test. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by Paolo.Rossi.con＠h3g.it 08 Oct '06

08 Oct '06

> 4GB is the memory limit on 32-bit Solaris, IIRC. Just like it is > 2GB on 32-bit linux. Try building everything 64-bit. I'll give a check. > Another thing to try, from when I ran my servers on Solaris, is to > fix the limits in /etc/system for the maximum amount of memory that > can be allocated. I vaguely recall having to do this for my > Solaris systems. uhm, maybe you referred "data seg size" ? now, is set to "unlimited" now I'm trying with ulimit -d 1024000, but now slapd have reached 1106MB > Oh, and with BDB on Solaris, you want to use the "shm_key" flag in > slapd.conf to use shared memory segments, when running on SPARC. > Here's the last portions of /etc/system on my old Solaris LDAP > servers: > > * turn off executable stacks > set noexec_user_stack = 1 > set noexec_user_stack_log = 1 > * increase the size of the kernel stack to 24k for Solaris 8/64-bit > set rpcmod:svc_default_stksize=0x6000 > set lwp_default_stksize=0x6000 > * force load the shared module kernel information. > forceload: sys/shmsys > * allow shared memory segments of up to 3 GB (default 1MB) > * See http://www.sun.com/sun-on-net/itworld/UIR960101perf.html > set shmsys:shminfo_shmmax=3221225472 > * Increase memory performance when filesystem is being heavily used. > * See http://www.sun.com/sun-on-net/performance/priority_paging.html > set priority_paging=1 > * Increase memory performance when filesystem is being heavily used. > * See http://www.samag.com/documents/s=1323/sam0110e/0110e.htm > set maxpgio=25468 > set slowscan=500 > * autoup influences how much RAM is checked by fsflush every 5 > seconds. > * Default = 30. Increase autoup to decrease mem management overhead. > * See http://docs.sun.com/db/doc/806-7009/6jftnqsin?a=view > set autoup = 60 > * Up tcp conn hash size. Defaults to 256. > * See http://www.deny-all.com/en/solsecu/tech/tuning.html > set tcp:tcp_conn_hash_size = 16384 > * Up file descriptor limit. Defaults to 1024. > set rlim_fd_max = 16384 > This is mine: set shmsys:shminfo_shmmax=2147483648 set shmsys:shminfo_shmmin=100 set shmsys:shminfo_shmmni=100 set shmsys:shminfo_shmseg=100 set semsys:seminfo_semmap=64 set semsys:seminfo_semmni=4096 set semsys:seminfo_semmns=4096 set semsys:seminfo_semmnu=4096 set semsys:seminfo_semume=64 set semsys:seminfo_semmsl=100 Thx for details, I will cross check if I miss somethings Paolo Rossi

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by Paolo.Rossi.con＠h3g.it 08 Oct '06

08 Oct '06

> PS: 4GB of RAM is probably not enough for a 20GB database, you > won't be able to configure caches large enough to get decent search > performance. I think 8GB would be a practical minimum here. > This is also my opinion, for good performances are not sufficient, but now the question is: 4GB are not enough to anyway, to run (slowly) similar huge server without coredump? p.s. or maybe I missed something, somewhere... :P Thx Paolo Rossi

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by Paolo.Rossi.con＠h3g.it 08 Oct '06

08 Oct '06

--Apple-Mail-1--548387425 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Il giorno 08/ott/06, alle ore 06:35, Howard Chu ha scritto: > No ideas, this problem doesn't occur on my Linux system. We've > slapadd'd and slapcat'd databases of over a terabyte in size, with > hundreds of millions of entries, and never seen slapcat grow beyond > the size of the BDB cache. cool :) > How large is your cache in DB_CONFIG? Here the entry my DB_CONFIG: set_cachesize 0 512000000 1 set_lg_regionmax 262144 set_lg_bsize 2097152 and, yes, I use shm_key in slapd.conf Thx for reply Paolo Rossi --Apple-Mail-1--548387425 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1 <HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; = -khtml-line-break: after-white-space; "><BR><DIV><DIV>Il giorno = 08/ott/06, alle ore 06:35, Howard Chu ha scritto:</DIV><BR = class=3D"Apple-interchange-newline"><BLOCKQUOTE type=3D"cite"><DIV = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">No ideas, this problem doesn't occur on my Linux = system. We've slapadd'd and slapcat'd databases of over a terabyte in = size, with hundreds of millions of entries, and never seen slapcat grow = beyond the size of the BDB cache. <BR></DIV></BLOCKQUOTE><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>cool = :)</DIV><BR><BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">How large is = your cache in DB_CONFIG?</DIV><DIV style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: = 14px; "></DIV></BLOCKQUOTE><BR></DIV><DIV>Here the entry my = DB_CONFIG:</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV><SPAN = class=3D"Apple-tab-span" style=3D"white-space:pre"> = </SPAN>set_cachesize 0 512000000 1 </DIV><DIV><SPAN = class=3D"Apple-tab-span" style=3D"white-space:pre"> = </SPAN>set_lg_regionmax 262144 </DIV><DIV><SPAN class=3D"Apple-tab-span" = style=3D"white-space:pre"> </SPAN>set_lg_bsize 2097152 = </DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV><SPAN = class=3D"Apple-style-span" style=3D"border-collapse: separate; = border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 15px; font-style: normal; font-variant: normal; font-weight: = normal; letter-spacing: normal; line-height: normal; text-align: auto; = -khtml-text-decorations-in-effect: none; text-indent: 0px; = -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = white-space: normal; widows: 2; word-spacing: 0px; ">and, yes, I use = shm_key in slapd.conf=A0<BR class=3D"Apple-interchange-newline"></SPAN> = </DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV>Thx for = reply</DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV>Paolo = Rossi<BR></BODY></HTML>= --Apple-Mail-1--548387425--

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by quanah＠stanford.edu 07 Oct '06

07 Oct '06

--On Sunday, October 08, 2006 4:35 AM +0000 hyc(a)symas.com wrote: >> Then I've tried to syncrepl the entire DB, turning on the empty >> consumers (crazy idea ;) i know ) but the provider memory allocated, >> again reached 4GB and... bum >> >> some ideas? 4GB is the memory limit on 32-bit Solaris, IIRC. Just like it is 2GB on 32-bit linux. Try building everything 64-bit. Another thing to try, from when I ran my servers on Solaris, is to fix the limits in /etc/system for the maximum amount of memory that can be allocated. I vaguely recall having to do this for my Solaris systems. Oh, and with BDB on Solaris, you want to use the "shm_key" flag in slapd.conf to use shared memory segments, when running on SPARC. Here's the last portions of /etc/system on my old Solaris LDAP servers: * turn off executable stacks set noexec_user_stack = 1 set noexec_user_stack_log = 1 * increase the size of the kernel stack to 24k for Solaris 8/64-bit set rpcmod:svc_default_stksize=0x6000 set lwp_default_stksize=0x6000 * force load the shared module kernel information. forceload: sys/shmsys * allow shared memory segments of up to 3 GB (default 1MB) * See http://www.sun.com/sun-on-net/itworld/UIR960101perf.html set shmsys:shminfo_shmmax=3221225472 * Increase memory performance when filesystem is being heavily used. * See http://www.sun.com/sun-on-net/performance/priority_paging.html set priority_paging=1 * Increase memory performance when filesystem is being heavily used. * See http://www.samag.com/documents/s=1323/sam0110e/0110e.htm set maxpgio=25468 set slowscan=500 * autoup influences how much RAM is checked by fsflush every 5 seconds. * Default = 30. Increase autoup to decrease mem management overhead. * See http://docs.sun.com/db/doc/806-7009/6jftnqsin?a=view set autoup = 60 * Up tcp conn hash size. Defaults to 256. * See http://www.deny-all.com/en/solsecu/tech/tuning.html set tcp:tcp_conn_hash_size = 16384 * Up file descriptor limit. Defaults to 1024. set rlim_fd_max = 16384 --Quanah -- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by hyc＠symas.com 07 Oct '06

07 Oct '06

PS: 4GB of RAM is probably not enough for a 20GB database, you won't be able to configure caches large enough to get decent search performance. I think 8GB would be a practical minimum here. Paolo.Rossi.con(a)h3g.it wrote: > Full_Name: Paolo Rossi > Version: 2.3.27 > OS: Solaris 8 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (88.149.168.114) > > > Hi, during some test on very huge DB, due to see how syncrepl works in this > scenario, I've found a strange behavior: > > Solaris 8 on 2xUSIII+ 4GB RAM > openLDAP 2.3.27 > BDB 4.2.52.4 > > backend hdb > > 1 provider, 1 consumer, 1 consumer with filter. > > on 1 million dn LDAP whit 2 sub-dn for each dn, all the systems works fine, > when I've tried to use a 10m dn with 3 sub-dn (very big ldap, openldap-data dir > about 20GB): -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

Re: (ITS#4702) out-of-memory on huge DB ?
by hyc＠symas.com 07 Oct '06

07 Oct '06

No ideas, this problem doesn't occur on my Linux system. We've slapadd'd and slapcat'd databases of over a terabyte in size, with hundreds of millions of entries, and never seen slapcat grow beyond the size of the BDB cache. How large is your cache in DB_CONFIG? Paolo.Rossi.con(a)h3g.it wrote: > Full_Name: Paolo Rossi > Version: 2.3.27 > OS: Solaris 8 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (88.149.168.114) > > > Hi, during some test on very huge DB, due to see how syncrepl works in this > scenario, I've found a strange behavior: > > Solaris 8 on 2xUSIII+ 4GB RAM > openLDAP 2.3.27 > BDB 4.2.52.4 > > backend hdb > > 1 provider, 1 consumer, 1 consumer with filter. > > on 1 million dn LDAP whit 2 sub-dn for each dn, all the systems works fine, > when I've tried to use a 10m dn with 3 sub-dn (very big ldap, openldap-data dir > about 20GB): > > slapadd with -w on producer. it works. > some ldapsearch, it works. > > stop producer, > slapcat the producer to obtain the ldif for consumer preload and... bum > > after about 150 minutes of slapcat, memory full (look the screenshot of top ) > > PID |USERNAME | SIZE |RES |TIME | CPU |COMMAND > 21495 | ldap |4072M |3591M |150:07 | 21.36% |slapd > > memory full, then coredump and a console messages: > > ch_malloc of 16392 bytes failed > ch_malloc.c:57: failed assertion `0' > > the out ldif was about 85% of the full LDAP > > I've tried again with same results. > > > Then I've tried to syncrepl the entire DB, turning on the empty consumers (crazy > idea ;) i know ) but the provider memory allocated, again reached 4GB and... > bum > > core dumped > > in the slapd.log > > ch_calloc of 1 elems of 80 bytes failed > > second try: > ch_malloc of 16 bytes failed > > seems to be a issue like ITS#4010 > > some ideas? > > Regards -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

Re: (ITS#4695) syncrepl modrdn failing with garbage returned by bdb_dn2entry
by richton＠nbcs.rutgers.edu 06 Oct '06

06 Oct '06

Rutgers just ran into this earlier today, fix (from RE23 CVS) verified on an x86_64 client, will try on sparc later (likely next week).

1 0

(ITS#4703) mirrormode hole
by hyc＠OpenLDAP.org 06 Oct '06

06 Oct '06

Full_Name: Howard Chu Version: HEAD/re24 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (24.126.120.178) Submitted by: hyc If the current master commits an update but crashes before the change is propagated to the mirror, and then new updates are subsequently committed on the new master, the non-propagated changes will never be picked up, because the new master's context is newer than everything on the crashed master. The hole can be plugged by keeping a per-consumer contextCSN in addition to the overall contextCSN, so that the old state isn't lost. While we're at it, we should add a consumer option for specifying additional locations in which to store contextCSNs. This would e.g. solve ITS#4626 by allowing a consumer in a glued subordinate to keep its parent context up to date.

1 0

Re: authz-regexp matches twice on same dse (ITS#4698)
by ando＠sys-net.it 06 Oct '06

06 Oct '06

> Pierangelo Masarati wrote: > >> I don't see an error in OpenLDAP software here. authz regexp matching >> is >> designed to succeed only if the identity is univoquely resolved to >> exactly one >> DN. I'm afraid but I cannot even imagine how slapd could decide to pick >> one out >> of many DNs when authenticating a user; I guess noone else can. >> >> p. >> > > Matched dn's are unique, as they describing the same Entry: > > dn: uid=works,dc=example,dc=org > objectClass: extensibleObject > uid: works > > dn: cn=worksalso,dc=example,dc=org > objectClass: extensibleObject > cn: worksalso > > dn: uid=fails,dc=example,dc=org > objectClass: extensibleObject > uid: fails > cn: fails > > "(|(cn=works)(uid=works))" and "(|(cn=worksalso)(uid=worksalso))" matching > either attribute, whereas "(|(cn=works)(uid=works))" matches twice, but > describes the same object. > > ldapsearching for "(|(cn=fails)(uid=fails))" will also return only the one > and unique entry "uid=fails,dc=example,dc=org" What authz-regexp does is run an internal search. If the search returns exactly one entry, then there's no way it can be, say, returned twice, otherwise it would also when running aregular search. Moreover, I've recrated you scenario in 2.3.27 and HEAD, and everything seems to work as expected in all cases. I suspect something else is wrong, for example data in your DB is not like it appears. Usually, guessing and expecting is a bad practice when debugging software. Please perform offending operations with full logs on; check that your data is not duplicated (for example, you might not see duplicates because they're hidden by ACLs) and so. Unless you can show a clear malfunction of the software (which I don't see here) I'm inclined towards closing this ITS. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

1 0

(ITS#4702) out-of-memory on huge DB ?
by Paolo.Rossi.con＠h3g.it 06 Oct '06

06 Oct '06

Full_Name: Paolo Rossi Version: 2.3.27 OS: Solaris 8 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (88.149.168.114) Hi, during some test on very huge DB, due to see how syncrepl works in this scenario, I've found a strange behavior: Solaris 8 on 2xUSIII+ 4GB RAM openLDAP 2.3.27 BDB 4.2.52.4 backend hdb 1 provider, 1 consumer, 1 consumer with filter. on 1 million dn LDAP whit 2 sub-dn for each dn, all the systems works fine, when I've tried to use a 10m dn with 3 sub-dn (very big ldap, openldap-data dir about 20GB): slapadd with -w on producer. it works. some ldapsearch, it works. stop producer, slapcat the producer to obtain the ldif for consumer preload and... bum after about 150 minutes of slapcat, memory full (look the screenshot of top ) PID |USERNAME | SIZE |RES |TIME | CPU |COMMAND 21495 | ldap |4072M |3591M |150:07 | 21.36% |slapd memory full, then coredump and a console messages: ch_malloc of 16392 bytes failed ch_malloc.c:57: failed assertion `0' the out ldif was about 85% of the full LDAP I've tried again with same results. Then I've tried to syncrepl the entire DB, turning on the empty consumers (crazy idea ;) i know ) but the provider memory allocated, again reached 4GB and... bum core dumped in the slapd.log ch_calloc of 1 elems of 80 bytes failed second try: ch_malloc of 16 bytes failed seems to be a issue like ITS#4010 some ideas? Regards

1 0

Re: (ITS#4701) slapd crash using slapmodrdn
by ando＠sys-net.it 06 Oct '06

06 Oct '06

> slapd crashed with an "assert failure" in lastmod.c line 593 when using > slapmodrn command. Fault is reproducable. Get around is to remove lastmod > overlay from config file. There is no slapmodrdn tool in OpenLDAP suite; I assume you mean ldapmodrdn. In any case, slapo-lastmod(5) is essentially unmaintained and basically useless. I suggest you use slapo-accesslog(5) instead. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

1 0

(ITS#4701) slapd crash using slapmodrdn
by andy.n.parker＠unilever.com 06 Oct '06

06 Oct '06

Full_Name: Andrew Neil Parker Version: 2.3.27 OS: RHEL 3.2 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (194.60.106.5) slapd crashed with an "assert failure" in lastmod.c line 593 when using slapmodrn command. Fault is reproducable. Get around is to remove lastmod overlay from config file. Trace output using -d 1 is provided on ftp site. 150 Opening BINARY mode data connection for 'andy.n.parker-061006.ext'. 226 Transfer complete (unique file name:andy.n.parker-061006.ext). 3036 bytes sent in 0.00025 seconds (1.2e+04 Kbytes/s)

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs