openldap-bugs

openldap-bugs@openldap.org

1 participants
20967 discussions

Re: ITS#5174 openldap.schema entry not valid per RFC 4512
by bhanafee＠gmail.com 08 Oct '07

08 Oct '07

p., Thanks for looking at it. I think the RFC 4234 parse of the elements in the RFC 4512 definition would be as a 'concatenation' (based on rule->elements->alternation->concatenation). To get "any order" behavior under RFC 4234, I think the elements in the definition would be separated by '/' characters, and all the elements would be enclosed in a group with a repetition element. There's an example in the "Generalized Time" definition (RFC 4517, section 3.3.13) that shows why the optional members must be in sequence by default. In that particular case, the [fraction] can't be allowed to appear before the [minute ...] part, or it would be ambiguous whether the final "3015" in "2007100807.53015" was part of the fraction or a minute and second that appears (for no good reason) after the fraction ".5" The reason I reported it was that I wrote a parser for LDAP schema based directly on the formal grammar downloaded all the schema from OpenLDAP as a test. That was the only schema that broke, and only on the one element. I did not have any trouble with the OpenLDAP software accepting the schema. If you use a grammar builder (I used ANTLR) rather than a hand-coded parser, being permissive makes it a bit more complex. Allowing the elements to appear in any order would be easy to parse for the RFC 4512 elements, because they happen to be defined with markers ("NAME", "DESC" and so on) that makes them easy to differentiate, but then enforcing at-most-once semantics on the elements requires either an additional pass or some hand coded predicates to check for duplicates in a data structure being built on-the-fly. -- Brian On 10/8/07, Pierangelo Masarati <ando(a)sys-net.it> wrote: > Not sure if ordering of optional sequence members is required by RFC > 4234, but the change you suggest sounds harmless. OpenLDAP software, in > this sense, is usually permissive in what is accepted and strict in what > is emitted. > > Thanks, p. > > > > Ing. Pierangelo Masarati > OpenLDAP Core Team > > SysNet s.r.l. > via Dossi, 8 - 27100 Pavia - ITALIA > http://www.sys-net.it > --------------------------------------- > Office: +39 02 23998309 > Mobile: +39 333 4963172 > Email: pierangelo.masarati(a)sys-net.it > --------------------------------------- > > >

1 0

Re: (ITS#4940) libldap doesn't wait for server's TLS close_notify
by hyc＠symas.com 08 Oct '07

08 Oct '07

guenther+ldapdev(a)sendmail.com wrote: > Full_Name: Philip Guenther > Version: 2.3.27 > OS: Linux and Solaris > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (64.58.1.252) > > > [I vaguely recall seeing a report of this issue in the archives of one of the > mailing lists, but I can no longer find the original.] > > If you trace the packets sent when you use, for example, ldapsearch against a > server on a different host, using either the -Z option to do TLS or using an > ldaps URI, you'll discover that the TCP connection is actually reset instead of > being closed cleanly: the client sends TCP RSTs in response to the server's > final packets. > > This is because libldap uses the following sequence when unbind a TLS or SSL > connection: > 1) send the unbind request (over the TLS or SSL layer) > 2) call SSL_shutdown(), sending the TLS close_notify alert > 3) call close() > > After receiving the close_notify alert from step (2), the server sends back its > own close_notify alert and then calls close(). However, because the client > didn't wait for the server's response before calling close() on its end, the > client's TCP stack considers the TCP connection to already be gone and responds > with the RST packets. This occurs with Linux and Solaris clients and probably > most other unices: the response to packets after a close() doesn't vary in my > experience. > > There are a number of ways this can be handled: > 1) change the client to wait until it sees the server's close_notify alert by > replacing "SSL_shutdown( p->ssl );" in tls.c with the two lines: > if (SSL_shutdown( p->ssl ) == 0) > SSL_shutdown( p->ssl ); > (I have confirmed that this works. As documented, the first call will return > 1 > if the server's close_notify has already been received, if not, the second > call > will block until it is received.) So if the server doesn't send one, the client will be stuck waiting forever? > 2) change the client to not bother to send a close_notify alert when it's just > going to close() the connection; change the server to not send a > close_notify > if it didn't get one. This probably violates the TLS spec, but the fact > that > TLS/1.1 permits resumption of sessions without close_notify having been sent > indicates that the violation is not a major issue, particularly given that > LDAP's > unbind request prevents truncation attacks. Close_notifies are, of course, > required if the client just wants to terminate the TLS layer and resume > unprotected LDAP operations. Sounds like a change in the SSL library, not something for us to worry about. > > 3) ignore the issue: it only causes one or two extra packets to be sent. While > it > also eliminates the TIME_WAIT state, LDAP's application-level close (the > unbind > request) means it doesn't need reliable full-duplex closure, so the only > concern > would be random connection issues from reincarnations of the TCP tuple, > which > is unlikely for an LDAP connection. > Personally, I like the simplicity and cleanliness of solution (1). (1) has the possibility of an indefinite hang. As such, I think it best to leave it with the current behavior. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5171) hdb txn_checkpoint failures
by richton＠nbcs.rutgers.edu 08 Oct '07

08 Oct '07

> One more thing to check is just using "ls -l" to see if the actual size of > the log files corresponds with the db_stat offsets. E.g. if slave6 base1's > log.0000001 is really 8MB but the LSN is only 233KB, then we have to look for > a weird in-memory corruption. If not, then somebody reset your logs. No, it looks like those sizes all match. Actually, the "reset logs" may well be the case (although I still can't imagine how, I'm willing to just chalk this whole thing up to user error...of course logs show that the user was me, which is a shame :) and is hard to disprove (with only one log file active) with the exception of base2. base2 has multiple log files going back: [slave4] -rw------- 1 root root 9999986 Sep 6 18:03 log.0000000001 -rw------- 1 root root 9999967 Sep 10 14:03 log.0000000002 -rw------- 1 root root 9999983 Sep 18 16:33 log.0000000003 -rw------- 1 root root 9429761 Oct 8 05:33 log.0000000004 [slave6] -rw------- 1 root root 9999986 Sep 6 18:03 log.0000000001 -rw------- 1 root root 9999967 Sep 10 14:03 log.0000000002 -rw------- 1 root root 9999983 Sep 18 16:33 log.0000000003 -rw------- 1 root root 9429761 Oct 8 05:33 log.0000000004 which of course match the db_stat -l, but also extend back prior to September 24 according to the filesystem timestamps. I guess the argument could be made that log 4 was truncated on September 24...would that be detected/come up sane/come up bad in the db_stat?

1 0

(ITS#5177) refreshAndPersist race condition
by hyc＠OpenLDAP.org 08 Oct '07

08 Oct '07

Full_Name: Howard Chu Version: 2.3/2.4 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (76.168.84.21) Submitted by: hyc If heavy Add traffic is occurring while a refreshAndPersist consumer is refreshing, newly added entries may be omitted from the refresh phase without being queued into the Persist phase. reported by stelios.xx.grigoriadis(a)ericsson.com http://www.openldap.org/lists/openldap-bugs/200709/msg00172.html

1 0

Sync replication failure during startup.
by Stelios Grigoriadis 08 Oct '07

08 Oct '07

OpenLDAP v. 2.3.32 Berkeley DB 4.6 gcc 4.1.0 Replication doesn't work if the master server is started after the replica servers and a large amount of simoultaneous updates are performed while the server is starting up. The entries that didn't get replicated to the replicas will not be replicated even after a restart of both master and replicas. The contextCSN is set to a value larger than the entryCSN of the "lost" entries. This is what I think happens during a master server startup with simoultaneous updates ongoing (and replicas trying to sync in the initial phase). Suppose that two clients (Client1 and Client2) are adding the entries a and b respectively. If that happens between t1 and t2 (one second between) they will get the same entryCSN (same timestamp). If entry a is committed at tc1 and b at tc2, any replica search inbetween will only get the entry a. The entry b will be lost. Client1 entry=a, csn=x Client2 entry=b, csn=x Timeline ------+----------+---------+----+------> | | t1 | | t2=t1+1 | | tc1=entry a tc2=entry b committed committed Replica search query between tc1 and tc2. I don't know if a higher granularity would prevent this, or even better, to have some kind of a counter so that every modification gets a unique csn. Can you please comment on our analyzis to let us know if the analyzis is correct or if we have missed something important? Any help or hints on how to avoid or fix this problem is greatly appreciated. If I receive useful information direcly in private email, I will post a summary. Regards Stelios Grigoriadis

4 23

Re: (ITS#5171) hdb txn_checkpoint failures
by hyc＠symas.com 08 Oct '07

08 Oct '07

Aaron Richton wrote: >> It's still rather suspicious that slave4 and slave6 both had identical log >> status for base1 (1/188113) but different requested locations (1/8730339 vs >> 1/8730401). If they're identically configured slaves then they ought to be in >> lock-step. Then again, obviously they're not identical since slave6 doesn't >> show base4 in your log. > > Identical is relative. They've got the same OpenLDAP and supporting > binaries running on the same patches of Solaris 9 running identical > turn-up scripts with identical configuration files. But this is > production, so we've got data changes over time. For instance, the slaves > bootstrap with a slapadd -q, and the underlying slapcat could easily be > different from slave4 vs. slave6 (the most recent one is automatically > used). I'd imagine this would look different at the db layer, even once > syncrepl eventually converged the logical data? > >> Do you have the db_stat output from an uncorrupted slave? What about the >> master? > > Sure... https://www.nbcs.rutgers.edu/~richton/its5171_dbstatl2 Judging from the LSNs in use on these other servers, it sure looks like somebody went in and zeroed out your logs on slave4 and slave6. I don't think the environment spontaneously corrupted itself and reset the log offsets... One more thing to check is just using "ls -l" to see if the actual size of the log files corresponds with the db_stat offsets. E.g. if slave6 base1's log.0000001 is really 8MB but the LSN is only 233KB, then we have to look for a weird in-memory corruption. If not, then somebody reset your logs. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5176) Segmentation Fault in test001-slapadd
by openldap＠consotec.de 08 Oct '07

08 Oct '07

Full_Name: Mark Version: 2.3.38 OS: Suse Linux 10.01 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (84.128.87.178) ** Problem test001-slapdadd fails with segmentation fault. ** Environment Suse Linux 10.1 Berkly DB 4.6.21 Openldap 2.3.38 ** Configuration of ldap configure --prefix=/usr --enable-debug ** Shared Library depency libdb-4.6.so => /usr/lib/libdb-4.6.so (0x40028000) libsasl2.so.2 => /usr/lib/libsasl2.so.2 (0x40169000) libdl.so.2 => /lib/libdl.so.2 (0x40182000) libresolv.so.2 => /lib/libresolv.so.2 (0x40185000) libpthread.so.0 => /lib/i686/libpthread.so.0 (0x40197000) libc.so.6 => /lib/i686/libc.so.6 (0x401e9000) ** StackTrace Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 32771 (LWP 28247)] 0x400bd3bd in __lock_get_internal (lt=0x8215ae8, sh_locker=0x7, flags=0, obj=0x82168d4, lock_mode=DB_LOCK_READ, timeout=0, lock=0x40d8e50c) at ../lock/lock.c:740 740 no_dd = sh_locker->master_locker == INVALID_ROFF && (gdb) bt #0 0x400bd3bd in __lock_get_internal (lt=0x8215ae8, sh_locker=0x7, flags=0, obj=0x82168d4, lock_mode=DB_LOCK_READ, timeout=0, lock=0x40d8e50c) at ../lock/lock.c:740 #1 0x400bcc85 in __lock_get (dbenv=0x82154e8, locker=0x7, flags=0, obj=0x82168d4, lock_mode=DB_LOCK_READ, lock=0x40d8e50c) at ../lock/lock.c:447 #2 0x400ec2f7 in __db_lget (dbc=0x8216858, action=0, pgno=1, mode=DB_LOCK_READ, lkflags=0, lockp=0x40d8e50c) at ../db/db_meta.c:1012 #3 0x40054d7d in __bam_get_root (dbc=0x8216858, pg=1, slevel=1, flags=1409, stack=0x40d8e614) at ../btree/bt_search.c:94 #4 0x400551a4 in __bam_search (dbc=0x8216858, root_pgno=1, key=0x40d8e97c, flags=1409, slevel=1, recnop=0x0, exactp=0x40d8e818) at ../btree/bt_search.c:200 #5 0x40045160 in __bamc_search (dbc=0x8216858, root_pgno=0, key=0x40d8e97c, flags=26, exactp=0x40d8e818) at ../btree/bt_cursor.c:2486 #6 0x400411bc in __bamc_get (dbc=0x8216858, key=0x40d8e97c, data=0x40d8e95c, flags=26, pgnop=0x40d8e8ac) at ../btree/bt_cursor.c:961 #7 0x400da81d in __dbc_get (dbc_arg=0x8217158, key=0x40d8e97c, data=0x40d8e95c, flags=26) at ../db/db_cam.c:697 #8 0x400e7dfb in __dbc_get_pp (dbc=0x8217158, key=0x40d8e97c, data=0x40d8e95c, flags=26) at ../db/db_iface.c:2022 #9 0x080d455f in bdb_id2entry (be=0x7, tid=0x0, locker=7, id=1, e=0x40d8e9f8) at id2entry.c:125 #10 0x080cdabb in bdb_cache_find_id (op=0x822a638, tid=0x0, id=1, eip=0x40d8ea84, islocked=0, locker=7, lock=0x40d8eb1c) at cache.c:760 #11 0x080d12cd in bdb_dn2entry (op=0x822a638, tid=0x0, dn=0x0, e=0x40d8eb14, matched=1, locker=7, lock=0x40d8eb1c) at dn2entry.c:68 #12 0x080b4ae9 in bdb_search (op=0x822a638, rs=0x40e4fc9c) at search.c:374 #13 0x0805ec5b in fe_op_search (op=0x822a638, rs=0x40e4fc9c) at search.c:355 #14 0x0805e43f in do_search (op=0x822a638, rs=0x40e4fc9c) at search.c:217 #15 0x0805cae2 in connection_operation (ctx=0x7, arg_v=0x822a638) at connection.c:1133 #16 0x080fd514 in ldap_int_thread_pool_wrapper (xpool=0x81c0a88) at tpool.c:478 #17 0x4019cf60 in pthread_start_thread () from /lib/i686/libpthread.so.0 #18 0x4019d0fe in pthread_start_thread_event () from /lib/i686/libpthread.so.0 #19 0x402c5327 in clone () from /lib/i686/libc.so.6 ** Description We added some printf's and found the that sh_locker has the value 0x7 which seems to be an index but not a valid lock for the database. Note: With the following modification the tests are running: File: servers/slapd/back-bdb/id2entry.c Line: 120 #if 0 /* Use our own locker if needed */ if ( !tid && locker ) cursor->locker = locker; #endif Any help is appriated Regards Mark

1 0

Re: (ITS#5171) hdb txn_checkpoint failures
by richton＠nbcs.rutgers.edu 08 Oct '07

08 Oct '07

> It's still rather suspicious that slave4 and slave6 both had identical log > status for base1 (1/188113) but different requested locations (1/8730339 vs > 1/8730401). If they're identically configured slaves then they ought to be in > lock-step. Then again, obviously they're not identical since slave6 doesn't > show base4 in your log. Identical is relative. They've got the same OpenLDAP and supporting binaries running on the same patches of Solaris 9 running identical turn-up scripts with identical configuration files. But this is production, so we've got data changes over time. For instance, the slaves bootstrap with a slapadd -q, and the underlying slapcat could easily be different from slave4 vs. slave6 (the most recent one is automatically used). I'd imagine this would look different at the db layer, even once syncrepl eventually converged the logical data? > Do you have the db_stat output from an uncorrupted slave? What about the > master? Sure... https://www.nbcs.rutgers.edu/~richton/its5171_dbstatl2

1 0

Re: (ITS#5171) hdb txn_checkpoint failures
by hyc＠symas.com 08 Oct '07

08 Oct '07

Aaron Richton wrote: >> itself. Again, we can't really tell without single-stepping thru the BDB >> library code. It may not be worth the effort, but that's your call. > > The lock was > > env_region.c:290 MUTEX_LOCK(dbenv, &renv->mutex); > > but that wasn't making much sense....and after a couple minutes in dbx I > realized that I've been killing myself with the attempts at db_stat. > Yesterday's attempts were running db_* binaries with a wrong (but > compatible) ABI. It'd be nice if Sleepycat had some more/earlier checks > for that, but oh well... Kinda figured that that's what happened. > So anyway, I corrupted base2/slave4 by running the wrong db_stat, but that > left three other bases on slave4 and all three bases on slave6. I ran > db_stat -l on them, the output is: > > https://www.nbcs.rutgers.edu/~richton/its5171_dbstatl > BTW, this ABI screwup shouldn't be the root cause of the failures...I > haven't tried any db tools until the course of debugging this. These are > AUTOREMOVE, so db_archive is unlikely, for instance. It's still rather suspicious that slave4 and slave6 both had identical log status for base1 (1/188113) but different requested locations (1/8730339 vs 1/8730401). If they're identically configured slaves then they ought to be in lock-step. Then again, obviously they're not identical since slave6 doesn't show base4 in your log. Do you have the db_stat output from an uncorrupted slave? What about the master? -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5171) hdb txn_checkpoint failures
by richton＠nbcs.rutgers.edu 08 Oct '07

08 Oct '07

> itself. Again, we can't really tell without single-stepping thru the BDB > library code. It may not be worth the effort, but that's your call. The lock was env_region.c:290 MUTEX_LOCK(dbenv, &renv->mutex); but that wasn't making much sense....and after a couple minutes in dbx I realized that I've been killing myself with the attempts at db_stat. Yesterday's attempts were running db_* binaries with a wrong (but compatible) ABI. It'd be nice if Sleepycat had some more/earlier checks for that, but oh well... So anyway, I corrupted base2/slave4 by running the wrong db_stat, but that left three other bases on slave4 and all three bases on slave6. I ran db_stat -l on them, the output is: https://www.nbcs.rutgers.edu/~richton/its5171_dbstatl BTW, this ABI screwup shouldn't be the root cause of the failures...I haven't tried any db tools until the course of debugging this. These are AUTOREMOVE, so db_archive is unlikely, for instance.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs