openldap-bugs

openldap-bugs@openldap.org

1 participants
20964 discussions

(ITS#8227) syncprov should use more threads
by hyc＠openldap.org 27 Aug '15

27 Aug '15

Full_Name: Howard Chu Version: 2.4 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (70.87.222.79) Submitted by: hyc Currently a single background thread and output queue is used by syncprov to send persist messages to all consumers. A single slow consumer can block messages from being sent to other consumers. syncprov should be enhanced to use a separate queue and thread per consumer.

1 0

(ITS#8226) long searches are bad for back-mdb
by hyc＠openldap.org 27 Aug '15

27 Aug '15

Full_Name: Howard Chu Version: 2.4 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (70.87.222.79) Submitted by: hyc Long-lived read transactions prevent old pages from being reused in LMDB. We've seen a situation triggered by the syncprov overlay that aggravates this problem - a consumer connects and performs a syncrepl refresh. Its sync cookie is too old for the syncprov sessionlog to be used, so syncprov must generate a presentlist of all of the entryUUIDs in the database. This is a very long search (in this case, around 1.5 million objects) and writes are continuing while this happens. Also, there are multiple consumers repeating this at staggered times (due to the situation described in ITS#8225 they all tend to reconnect at closely situated times). Because of these long read transactions the incoming writes are forced to use new database pages, and so the DB file size grows drastically during these refreshes. This specific case for syncprov can be mitigated by using a much larger syncprov sessionlog (and in the future, by enhancing syncprov to use accesslog), but there are many other instances when clients may legitimately issue searches that span the entire database. The fix from ITS#7904 should be extended to release and reacquire the read transaction every N entries to make large searches friendlier to write traffic. The default value of N probably should not be zero, but could be a value (e.g. 1000, with a default sizelimit of 500) that won't affect the majority of normal search requests.

1 0

Re: (ITS#8225) writetimeout should ignore consumers
by hyc＠symas.com 27 Aug '15

27 Aug '15

hyc(a)openldap.org wrote: > The idletimeout already ignores consumer connections; writetimeout should as > well. Sites can use keepalive to handle truly hung consumers, as opposed to just > overloaded consumers. It's not clear that simply "ignoring" is the correct approach - the fact that the writetimeout triggers means slapd was unable to write any more on the socket in question. As such, the provider would need to stop attempting to send any more syncrepl messages to the consumer until the connection freed up. This would just mean more data would queue up in syncprov's output queues while waiting for the socket buffers to drain. If it takes a very long time for the consumers to finally catch up (looks like on the order of hours here) then slapd's memory consumption would increase quite a lot for that time. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8225) writetimeout should ignore consumers
by hyc＠openldap.org 27 Aug '15

27 Aug '15

Full_Name: Howard Chu Version: 2.4 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (70.87.222.79) Submitted by: hyc We found a case where a site was using regular syncrepl and replicating changes to very large groups (hundreds of thousands of members, tens of megabytes total entry size) and the socket send queues on the provider were getting filled because the consumers weren't reading or processing fast enough. At some point the writetimeout on the provider kicked in and these syncrepl connections got closed, forcing the consumers to reconnect and start the whole process all over again. This sequence of events basically meant the consumers never made any progress, because the refresh on next connect attempt had to send such a large volume of data all over again. syslogs showed consumers having their connection closed multiple times within short (5 minute) spans of time. The idletimeout already ignores consumer connections; writetimeout should as well. Sites can use keepalive to handle truly hung consumers, as opposed to just overloaded consumers.

1 0

Re: Re: (ITS#8223) Replication error when slave is down and insert on master are made
by Frank.Offermanns＠caseris.de 27 Aug '15

27 Aug '15

Dies ist eine mehrteilige Nachricht im MIME-Format. --=_alternative 003127EEC1257EAE_= Content-Type: text/plain; charset="US-ASCII" Hello, I have additional information. The problem is the entryCSN. User 1, which got replicated has entryCSN: 20150827083333.751721Z#000000#001#000000 user 2, which got NOT replicated has entryCSN: 20150827083333.119585Z#000000#001#000000, which seems to be in the past user 3, replicated, 20150827083334.847604Z#000000#001#000000 user 4, not replicated 20150827083334.228113Z#000000#001#000000, which seems to be in the past user 5, replicated 20150827083335.588191Z#000000#001#000000 user 6, replicated 20150827083335.946835Z#000000#001#000000 user 7 not replicated 20150827083335.282686Z#000000#001#000000, which seems to be in the past So in previous posts I got the answer, that microseconds exact time synchronization is only important with master master and concurrent writes. But now it also seems to be important with delta-synrepl. So I suppose this ITS can be closed, since it is not a bug, but worked as designed? So how to get a good enough time synchronization with windows (since the normal time synchronization of windows seems to be not enough)? Of course, if you close the ITS I can ask this question again in the technical list. Regards, Frank --=_alternative 003127EEC1257EAE_= Content-Type: text/html; charset="US-ASCII" Hello, I have additional information. The problem is the entryCSN. User 1, which got replicated has entryCSN: 20150827083333.751721Z#000000#001#000000 user 2, which got NOT replicated has entryCSN: 20150827083333.119585Z#000000#001#000000, which seems to be in the past user 3, replicated, 20150827083334.847604Z#000000#001#000000 user 4, not replicated 20150827083334.228113Z#000000#001#000000, which seems to be in the past user 5, replicated 20150827083335.588191Z#000000#001#000000 user 6, replicated 20150827083335.946835Z#000000#001#000000 user 7 not replicated 20150827083335.282686Z#000000#001#000000, which seems to be in the past So in previous posts I got the answer, that microseconds exact time synchronization is only important with master master and concurrent writes. But now it also seems to be important with delta-synrepl. So I suppose this ITS can be closed, since it is not a bug, but worked as designed? So how to get a good enough time synchronization with windows (since the normal time synchronization of windows seems to be not enough)? Of course, if you close the ITS I can ask this question again in the technical list. Regards, Frank --=_alternative 003127EEC1257EAE_=--

1 0

Re: Re: (ITS#8223) Replication error when slave is down and insert on master are made
by Frank.Offermanns＠caseris.de 26 Aug '15

26 Aug '15

Dies ist eine mehrteilige Nachricht im MIME-Format. --=_alternative 00253B99C1257EAE_= Content-Type: text/plain; charset="US-ASCII" Hi Quanah, as stated above I have the same problem with 2.4.41 (and lmdb). Unfortunately, I am currently not able to build 2.4.42. I will report, when I have a running 2.4.42. But due to the fact, that the problem was in 2.4.25 and is still in 2.4.41, I would think, that it will also be in 2.4.42. Best regards, Frank --=_alternative 00253B99C1257EAE_= Content-Type: text/html; charset="US-ASCII" Hi Quanah, as stated above I have the same problem with 2.4.41 (and lmdb). Unfortunately, I am currently not able to build 2.4.42. I will report, when I have a running 2.4.42. But due to the fact, that the problem was in 2.4.25 and is still in 2.4.41, I would think, that it will also be in 2.4.42. Best regards, Frank --=_alternative 00253B99C1257EAE_=--

1 0

(ITS#8224) configure --disable-slapd doesn't ignore every slapd option
by ryan＠nardis.ca 26 Aug '15

26 Aug '15

Full_Name: Ryan Tandy Version: RE24 OS: Debian URL: Submission from: (NULL) (24.68.37.4) Submitted by: ryan The configure script forcibly disables most slapd options if slapd is disabled, but a few are missing: --enable-cleartext --enable-crypt --enable-lmpasswd --enable-spasswd --enable-slp Actually that whole section seems like it could be cleaned up/de-duplicated a bit...

1 0

Re: (ITS#8223) Replication error when slave is down and insert on master are made
by quanah＠zimbra.com 26 Aug '15

26 Aug '15

--On Wednesday, August 26, 2015 8:33 AM +0000 Frank.Offermanns(a)caseris.de wrote: > Full_Name: Frank Offermanns > Version: 2.4.25 and 2.4.41 > OS: Windows > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (217.7.149.50) 2.4.25 is known to have numerous replication bugs. Why are you wasting your time doing testing with it? Run the /same/ version on both master and replica, preferably 2.4.42, and then test. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#8223) Replication error when slave is down and insert on master are made
by Frank.Offermanns＠caseris.de 26 Aug '15

26 Aug '15

Full_Name: Frank Offermanns Version: 2.4.25 and 2.4.41 OS: Windows URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (217.7.149.50) Hello, when I have two ldapserver (master/slave with delta syncrepl) and do insert on the master while the slave is down and then start the slave, only some entries get replicated. This problem is reproducable (every time). I did my test with a build of 2.4.25 and mingw and with 2.4.41 build with visual studio. The version 2.4.25 is 32 bit and the 2.4.41 is 64 bit. Operation system is Windows. Version 2.4.25 runs with hdb, version 2.4.41 with lmdb. Here is a description in detail: Master and slave are fully synchronized. Then I stop the slave. Add 100 users to the master LDAP. I wait a few seconds, then I start the service at the slave. Now only some users get replicated. Here is my configuration of master (of the hdb try): ###################################################################### database config rootdn cn=config rootpw secret # Accesslog database definitions database hdb suffix cn=accesslog checkpoint 1024 5 cachesize 10000 directory "c:/mydir/accessdata" dbconfig set_cachesize 0 30000000 1 dbconfig set_flags DB_LOG_AUTOREMOVE dbconfig set_lg_regionmax 1048576 dbconfig set_lg_max 10485760 dbconfig set_lg_bsize 2097152 rootdn cn=accesslog index default eq index entryCSN,objectClass,reqEnd,reqResult,reqStart overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE limits dn.exact="cn=Replicator,ou=admins,o=caesar" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited database hdb suffix "o=caesar" checkpoint 1024 5 cachesize 10000 idlcachesize 30000 rootdn "cn=Administrator,o=caesar" rootpw secret directory "c:/mydir/data" dbconfig set_cachesize 0 100000000 1 dbconfig set_flags DB_LOG_AUTOREMOVE dbconfig set_lg_regionmax 1048576 dbconfig set_lg_max 10485760 dbconfig set_lg_bsize 2097152 # syncprov specific indexing index sn pres,eq ... index objectClass eq overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 10000 overlay accesslog logdb cn=accesslog logops writes logsuccess TRUE logpurge 07+00:00 01+00:00 sizelimit size.soft=100 size.hard=1000 size.prtotal=unlimited limits dn.exact="cn=Replicator,ou=admins,o=caesar" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited Here is my config of the slave (of the hdb test): access to * by dn.one="ou=Admins,o=caesar" write by anonymous auth ####################################################################"3%0 database config rootdn cn=config rootpw secret database hdb suffix "o=caesar" checkpoint 1024 5 cachesize 10000 idlcachesize 30000 rootdn "cn=Administrator,o=caesar" rootpw secret directory "C:/mydir/data" dbconfig set_cachesize 0 100000000 1 dbconfig set_flags DB_LOG_AUTOREMOVE dbconfig set_lg_regionmax 1048576 dbconfig set_lg_max 10485760 dbconfig set_lg_bsize 2097152 # syncprov specific indexing index sn pres,eq ... index objectClass eq sizelimit size.soft=100 size.hard=1000 size.prtotal=unlimited # Let the replica DN have limitless searches limits dn.exact="cn=Replicator,ou=admins,o=caesar" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited syncrepl rid=001 provider="ldap://masterserver.mydomain" searchbase="o=caesar" type=refreshAndPersist retry="5 3 15 +" binddn="cn=Replicator,ou=admins,o=caesar" bindmethod=simple credtitials="secret" logbase="cn=accesslog" logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" schemachecking=on syncdata=accesslog

1 0

Re: (ITS#7537) ppolicy hangs slapd on 64bit version debian
by ryan＠nardis.ca 25 Aug '15

25 Aug '15

I reproduced this with current RE24. It seems to be caused by setting pwdMaxAge to a very large value. In Marek's previous debug output you can see his policy object as returned from a search. Among other attrs it has: pwdMaxAge: 2592000000 I reproduced the report with the following setup: ### slapd.conf include ../openldap/servers/slapd/schema/core.schema include ../openldap/servers/slapd/schema/ppolicy.schema moduleload back_hdb moduleload ppolicy database hdb directory db suffix dc=example,dc=com rootdn cn=root,dc=example,dc=com rootpw secret access to * by dn="cn=admin,dc=example,dc=com" manage by * read index objectClass eq overlay ppolicy ppolicy_default cn=default,ou=policies,dc=example,dc=com ### init.ldif dn: dc=example,dc=com objectClass: domain dn: cn=admin,dc=example,dc=com objectClass: organizationalRole objectClass: simpleSecurityObject userPassword: secret dn: ou=policies,dc=example,dc=com objectClass: organizationalUnit dn: cn=default,ou=policies,dc=example,dc=com objectClass: organizationalRole objectClass: pwdPolicy pwdAttribute: userPassword pwdMaxAge: 2592000000 ### mod.ldif dn: cn=default,ou=policies,dc=example,dc=com add: pwdMaxFailure pwdMaxFailure: 7 Launched slapd and triggered via: $ ldapmodify -x -D cn=admin,dc=example,dc=com -W -f mod.ldif slapd hangs, ldapmodify is still waiting for it, and gdb says: Program received signal SIGINT, Interrupt. 0x00007ffff75f24db in pthread_join (threadid=140737286698752, thread_return=0x0) at pthread_join.c:92 92 pthread_join.c: No such file or directory. (gdb) thread apply all bt Thread 3 (Thread 0x7ffff3faf700 (LWP 25350)): #0 0x00007ffff7326653 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000004257b9 in slapd_daemon_task (ptr=0xa83a10) at daemon.c:2539 #2 0x00007ffff75f10a4 in start_thread (arg=0x7ffff3faf700) at pthread_create.c:309 #3 0x00007ffff732607d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7ffff37ae700 (LWP 25353)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffff7a4e79d in __db_pthread_mutex_lock () from /usr/lib/x86_64-linux-gnu/libdb-5.3.so #2 0x00007ffff7af74e0 in __lock_get_internal () from /usr/lib/x86_64-linux-gnu/libdb-5.3.so #3 0x00007ffff7af8719 in __lock_vec () from /usr/lib/x86_64-linux-gnu/libdb-5.3.so #4 0x00007ffff7af8e9f in __lock_vec_pp () from /usr/lib/x86_64-linux-gnu/libdb-5.3.so #5 0x000000000052f7ed in hdb_cache_entry_db_relock (bdb=0x934920, txn=0x7fffe4102810, ei=0x7fffe4101e80, rw=1, tryOnly=0, lock=0x7ffff37ac340) at cache.c:198 #6 0x0000000000531a43 in hdb_cache_modify (bdb=0x934920, e=0x9aca18, newAttrs=0x9c0690, txn=0x7fffe4102810, lock=0x7ffff37ac340) at cache.c:1231 #7 0x00000000004d85f6 in hdb_modify (op=0x7fffe4000aa0, rs=0x7ffff37adae0) at modify.c:764 #8 0x00000000004b740c in overlay_op_walk (op=0x7fffe4000aa0, rs=0x7ffff37adae0, which=op_modify, oi=0x936420, on=0x0) at backover.c:677 #9 0x00000000004b7637 in over_op_func (op=0x7fffe4000aa0, rs=0x7ffff37adae0, which=op_modify) at backover.c:730 #10 0x00000000004b776b in over_op_modify (op=0x7fffe4000aa0, rs=0x7ffff37adae0) at backover.c:769 #11 0x0000000000449541 in fe_op_modify (op=0x7fffe4000aa0, rs=0x7ffff37adae0) at modify.c:303 #12 0x0000000000448e13 in do_modify (op=0x7fffe4000aa0, rs=0x7ffff37adae0) at modify.c:177 #13 0x0000000000429a3f in connection_operation (ctx=0x7ffff37adc10, arg_v=0x7fffe4000aa0) at connection.c:1155 #14 0x0000000000429fdb in connection_read_thread (ctx=0x7ffff37adc10, argv=0xe) at connection.c:1291 #15 0x000000000058415d in ldap_int_thread_pool_wrapper (xpool=0x909fd0) at tpool.c:696 #16 0x00007ffff75f10a4 in start_thread (arg=0x7ffff37ae700) at pthread_create.c:309 #17 0x00007ffff732607d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7ffff7fec700 (LWP 25346)): #0 0x00007ffff75f24db in pthread_join (threadid=140737286698752, thread_return=0x0) at pthread_join.c:92 #1 0x00000000005855f3 in ldap_pvt_thread_join (thread=140737286698752, thread_return=0x0) at thr_posix.c:197 #2 0x0000000000426986 in slapd_daemon () at daemon.c:2932 #3 0x000000000040602d in main (argc=8, argv=0x7fffffffe598) at main.c:1017 If I change the config to use back-mdb instead, I get a crash: 55dce2bf conn=1000 op=1 MOD dn="cn=default,ou=policies,dc=example,dc=com" 55dce2bf conn=1000 op=1 MOD attr=pwdMaxFailure slapd: id2entry.c:520: mdb_opinfo_get: Assertion `!rc' failed. [New Thread 0x7ffff2caf700 (LWP 25374)] [New Thread 0x7ffff34b0700 (LWP 25371)] Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ffff2caf700 (LWP 25374)] 0x00007ffff7275107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007ffff7275107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007ffff72764e8 in __GI_abort () at abort.c:89 #2 0x00007ffff726e226 in __assert_fail_base (fmt=0x7ffff73a4d08 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x5f29e4 "!rc", file=file@entry=0x5f2975 "id2entry.c", line=line@entry=520, function=function@entry=0x5f2b30 <__PRETTY_FUNCTION__.12200> "mdb_opinfo_get") at assert.c:92 #3 0x00007ffff726e2d2 in __GI___assert_fail (assertion=0x5f29e4 "!rc", file=0x5f2975 "id2entry.c", line=520, function=0x5f2b30 <__PRETTY_FUNCTION__.12200> "mdb_opinfo_get") at assert.c:101 #4 0x0000000000553a71 in mdb_opinfo_get (op=0x7fffe4000aa0, mdb=0x7ffff7f2a010, rdonly=1, moip=0x7ffff2cacf18) at id2entry.c:520 #5 0x000000000055306b in mdb_entry_get (op=0x7fffe4000aa0, ndn=0x7fffe4000ad8, oc=0x0, at=0x0, rw=0, ent=0x7ffff2cad2c8) at id2entry.c:327 #6 0x00000000004b6a41 in overlay_entry_get_ov (op=0x7fffe4000aa0, dn=0x7fffe4000ad8, oc=0x0, ad=0x0, rw=0, e=0x7ffff2cad2c8, on=0x0) at backover.c:364 #7 0x00000000004b6b11 in over_entry_get_rw (op=0x7fffe4000aa0, dn=0x7fffe4000ad8, oc=0x0, ad=0x0, rw=0, e=0x7ffff2cad2c8) at backover.c:396 #8 0x000000000043ca86 in be_entry_get_rw (op=0x7fffe4000aa0, ndn=0x7fffe4000ad8, oc=0x0, at=0x0, rw=0, e=0x7ffff2cad2c8) at backend.c:1366 #9 0x0000000000561e6c in ppolicy_modify (op=0x7fffe4000aa0, rs=0x7ffff2caeae0) at ppolicy.c:1626 #10 0x00000000004b7362 in overlay_op_walk (op=0x7fffe4000aa0, rs=0x7ffff2caeae0, which=op_modify, oi=0x935700, on=0x9358e0) at backover.c:661 #11 0x00000000004b7637 in over_op_func (op=0x7fffe4000aa0, rs=0x7ffff2caeae0, which=op_modify) at backover.c:730 #12 0x00000000004b776b in over_op_modify (op=0x7fffe4000aa0, rs=0x7ffff2caeae0) at backover.c:769 #13 0x0000000000449541 in fe_op_modify (op=0x7fffe4000aa0, rs=0x7ffff2caeae0) at modify.c:303 #14 0x0000000000448e13 in do_modify (op=0x7fffe4000aa0, rs=0x7ffff2caeae0) at modify.c:177 #15 0x0000000000429a3f in connection_operation (ctx=0x7ffff2caec10, arg_v=0x7fffe4000aa0) at connection.c:1155 #16 0x0000000000429fdb in connection_read_thread (ctx=0x7ffff2caec10, argv=0xb) at connection.c:1291 #17 0x000000000058415d in ldap_int_thread_pool_wrapper (xpool=0x909fd0) at tpool.c:696 #18 0x00007ffff75f10a4 in start_thread (arg=0x7ffff2caf700) at pthread_create.c:309 #19 0x00007ffff732607d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 It seems like the crash does not happen if I bind as the rootdn, only when using the admin DN. The hang with back-hdb happens with both. Like I mentioned above, the problem seems to be related to the large pwdMaxAge value. If I drop a few 0s off the end of it, everything works properly. That might be related to Marek's statement that he only experienced the problem on 64-bit...

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs