delta-syncrepl stopped receiving changes
by David Hawes
My environment consists of 1 provider and 6 consumer machines using
delta-syncrepl for replication. For some reason, one of the consumer
machines stopped receiving changes for several days and did not resume
syncing until a restart (SIGKILL).
All machines are running 2.4.30 with BerkeleyDB 4.7.25 (plus patches),
the hdb backend, and tcmalloc.
I notice that 2.4.31 has several syncrepl fixes, and I intend to upgrade
in the near future, but my issue seems different than the ITS issues
listed in the change log. I'm posting here to determine if this has been
fixed or if I need to create a new ITS issue.
When I noticed one of the machines was not syncing properly I first
ensured that I could still query it. When I verified that, I decided to
change the LogLevel to 'any' to make sure I wasn't missing any logs that
would help me figure out what the issue was. Unfortunately, this LDAP
modify caused OpenLDAP to hang. The operation never finished, and new
queries using ldapsearch would simply hang as well.
At this point I got a gdb backtrace and db_stat from BDB.
db_stat -c:
=====
Default locking region information:
640 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
3000 Maximum number of locks possible
1500 Maximum number of lockers possible
1500 Maximum number of lock objects possible
80 Number of lock object partitions
47 Number of current locks
1262 Maximum number of locks at any one time
19 Maximum number of locks in any one bucket
0 Maximum number of locks stolen by for an empty partition
0 Maximum number of locks stolen for any one partition
340 Number of current lockers
341 Maximum number of lockers at any one time
44 Number of current lock objects
683 Maximum number of lock objects at any one time
6 Maximum number of lock objects in any one bucket
0 Maximum number of objects stolen by for an empty partition
0 Maximum number of objects stolen for any one partition
3799M Total number of locks requested (3799770221)
3799M Total number of locks released (3799714045)
0 Total number of locks upgraded
73 Total number of locks downgraded
73M Lock requests not available due to conflicts, for which we waited
(73026840)
5165 Lock requests not available due to conflicts, for which we did
not wait
50963 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
1MB 400KB The size of the lock region
70M The number of partition locks that required waiting (1%)
15M The maximum number of times any partition lock was waited for (0%)
15018 The number of object queue operations that required waiting (0%)
54M The number of locker allocations that required waiting (2%)
0 The number of region locks that required waiting (0%)
6 Maximum hash bucket length
=====
The number of deadlocks was pretty shocking; I've never seen that value
non-zero.
I also have db_stat -C A output if that helps.
gdb (syncrepl thread):
=====
Thread 12 (Thread 0x444a9950 (LWP 16336)):
#0 0x00007f5e087f2b99 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x00007f5e08a274fb in __db_pthread_mutex_lock ()
from /apps/local/depend/BerkeleyDB-4.7.25p4/lib/libdb-4.7.so
#2 0x00007f5e08aa5dec in __lock_get_internal ()
from /apps/local/depend/BerkeleyDB-4.7.25p4/lib/libdb-4.7.so
#3 0x00007f5e08aa6d6a in __lock_vec ()
from /apps/local/depend/BerkeleyDB-4.7.25p4/lib/libdb-4.7.so
#4 0x00007f5e08aa72db in __lock_vec_pp ()
from /apps/local/depend/BerkeleyDB-4.7.25p4/lib/libdb-4.7.so
#5 0x000000000054da72 in hdb_cache_entry_db_relock (bdb=0xaec000,
txn=0x1031c20, ei=0x3d5e240, rw=1, tryOnly=0, lock=0x444a7b30)
at cache.c:198
#6 0x000000000054fe9e in hdb_cache_modify (bdb=0xaec000, e=0x7f5ccb945f58,
newAttrs=0x7f5ccf4c1ec0, txn=0x1031c20, lock=0x444a7b30) at cache.c:1231
#7 0x00000000004fc2bd in hdb_modify (op=0x444a86d0, rs=0x444a8040)
at modify.c:711
#8 0x00000000004db66f in overlay_op_walk (op=0x444a86d0, rs=0x444a8040,
which=op_modify, oi=0xace000, on=0x0) at backover.c:671
#9 0x00000000004db899 in over_op_func (op=0x444a86d0, rs=0x444a8040,
which=op_modify) at backover.c:723
#10 0x00000000004db9de in over_op_modify (op=0x444a86d0, rs=0x444a8040)
at backover.c:762
#11 0x00000000004c8c59 in syncrepl_message_to_op (si=0xb22000,
op=0x444a86d0,
msg=0x7f5cf7df5f80) at syncrepl.c:2316
#12 0x00000000004c40dd in do_syncrep2 (op=0x444a86d0, si=0xb22000)
at syncrepl.c:986
#13 0x00000000004c61a6 in do_syncrepl (ctx=0x444a8df0, arg=0xb768c0)
at syncrepl.c:1522
#14 0x0000000000449003 in connection_read_thread (ctx=0x444a8df0, argv=0x31)
at connection.c:1288
#15 0x0000000000591269 in ldap_int_thread_pool_wrapper (xpool=0xa741c0)
at tpool.c:688
#16 0x00007f5e087ee3f7 in start_thread () from /lib/libpthread.so.0
#17 0x00007f5e0794dbbd in clone () from /lib/libc.so.6
#18 0x0000000000000000 in ?? ()
=====
gdb (thread modifying cn=config):
=====
Thread 3 (Thread 0x48cb2950 (LWP 29597)):
#0 0x00007f5e087f2b99 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x0000000000592806 in ldap_pvt_thread_cond_wait (cond=0xa74220,
mutex=0xa741c8) at thr_posix.c:277
#2 0x000000000059162f in handle_pause (tpool=0x8b5420, pause_type=96)
at tpool.c:788
#3 0x00000000005916fa in ldap_pvt_thread_pool_pause (tpool=0x8b5420)
at tpool.c:831
#4 0x0000000000433ac4 in config_back_modify (op=0x7f5cdc507800,
rs=0x48cb1c90)
at bconfig.c:5837
#5 0x0000000000468e52 in fe_op_modify (op=0x7f5cdc507800, rs=0x48cb1c90)
at modify.c:303
#6 0x0000000000468761 in do_modify (op=0x7f5cdc507800, rs=0x48cb1c90)
at modify.c:177
#7 0x0000000000448a65 in connection_operation (ctx=0x48cb1df0,
arg_v=0x7f5cdc507800) at connection.c:1150
#8 0x0000000000448fe7 in connection_read_thread (ctx=0x48cb1df0, argv=0x15)
at connection.c:1286
#9 0x0000000000591269 in ldap_int_thread_pool_wrapper (xpool=0xa741c0)
at tpool.c:688
#10 0x00007f5e087ee3f7 in start_thread () from /lib/libpthread.so.0
#11 0x00007f5e0794dbbd in clone () from /lib/libc.so.6
#12 0x0000000000000000 in ?? ()
=====
I also have bt full output if needed.
Since restarting I have seen no issues with any of the instances and the
failed instance synced without issue.
Let me know if I should create an ITS.
Thanks,
Dave
11 years, 1 month
write an acl with sets and compare connecting peername.ip with ipHost entry (ipHostNumber)
by Uwe Werler
Hallo all,
is it possible to write an ACL (with sets) which extracts the peername.ip from within an existing entry of ipHost an then compares the connecting peername.ip?
My idea is to only allow access to this entry by connecting peername.ip 192.168.1.1:
dn: cn=myhost.wdf.sap.corp,ou=HOSTS
objectClass: ipHost
objectClass: device
cn: myhost.wdf.sap.corp
ipHostNumber: 192.168.1.1
Background: I want to use ovleray nssov and therefore I need all host information at each host locally in ldap. 'cause we have several thousands of hosts I dont want to replicate all ipHosts to each local database.
Thanks for any advice/hint.
Regards Uwe
11 years, 1 month
OpenLDAP and virtual directory
by Gilles Tual
Hi,
I was wondering if I could use OpenLDAP backend to simulate a virtual
directory for a project.
Is it possible ? If yes, is there documentation on it, or help (i'm not a
pro on LDAP and virtual directory).
Thanks a lot, have a nice day !
11 years, 1 month
Re:
by Gavin Henry
Excellent. Thanks for following up.
--
Kind Regards,
Gavin Henry.
Managing Director.
T +44 (0) 1224 279484
M +44 (0) 7930 323266
F +44 (0) 1224 824887
E ghenry(a)suretec.co.uk
Open Source. Open Solutions(tm).
http://www.suretecsystems.com/
Suretec Systems is a limited company registered in Scotland. Registered
number: SC258005. Registered office: 24 Cormack Park, Rothienorman,
Inverurie,
Aberdeenshire, AB51 8GL.
Subject to disclaimer at http://www.suretecgroup.com/disclaimer.html
Do you know we have our own VoIP provider called SureVoIP? See
http://www.surevoip.co.uk
On 14 Aug 2012, at 10:25, "非人協會" <hkuhaorg(a)gmail.com> wrote:
Dear Henry,
Thanks for the kindness follow up.
The problem has fixed. The root cause of the problem is the wrongly
configured slapd.conf
Thanks and Regards,
Donald
2012/8/10 Gavin Henry <ghenry(a)suretecsystems.com>
> On 8 August 2012 04:59, 非人協會 <hkuhaorg(a)gmail.com> wrote:
> > Dear All,
> > I am new to ldap, I would like to have your kindness assistance in
> setting
> > up the directory.
> >
> > We are working on move our old ldap server to a new openldap server, I
> have
> > install the openldap in Solaris 10 x86 successfully. However I am not
> able
> > to browse the content using the ldap browser, it shows "Invalid
> > Credentials".
>
> Hi Donald,
>
> All fixed?
>
> --
> Kind Regards,
>
> Gavin Henry.
> Managing Director.
>
> T +44 (0) 1224 279484
> M +44 (0) 7930 323266
> F +44 (0) 1224 824887
> E ghenry(a)suretecsystems.com
>
> Open Source. Open Solutions(tm).
>
> http://www.suretecsystems.com/
>
> Suretec Systems is a limited company registered in Scotland. Registered
> number: SC258005. Registered office: 24 Cormack Park, Rothienorman,
> Inverurie,
> Aberdeenshire, AB51 8GL.
>
> Subject to disclaimer at http://www.suretecgroup.com/disclaimer.html
>
> Do you know we have our own VoIP provider called SureVoIP? See
> http://www.surevoip.co.uk
>
> Did you see our API news?
> http://www.surevoip.co.uk/news-events/surevoip-launches-innovative-api
>
11 years, 1 month
Re: representing file pathnames
by Chuck Lever
Hi-
On Aug 9, 2012, at 7:22 PM, Howard Chu wrote:
> Michael Ströder wrote:
>> Chuck Lever wrote:
>>> On Aug 9, 2012, at 2:46 PM, Howard Chu wrote:
>>>> Michael Ströder wrote:
>>>>> Chuck Lever wrote:
>>>>>> We could also use an NFS URL, which would allow us to express the server
>>>>>> hostname, a port number, and the pathname in a single string. But both the
>>>>>> hostname and pathname are enocded in US-ASCII, not UTF-8, and the NFS URL
>>>>>> format employs a fixed pathname separator character.
>>>>>
>>>>> That's what I would prefer. Think of file browsers which can open the NFS
>>>>> mount point just by clicking on it. Same encoding steps as with file URLs.
>>>>
>>>> This seems the most obvious and natural solution (NFS URL). After all, you are
>>>> specifying an NFS resource...
>>>
>>> I've looked more closely at this idea. While it's got some surface appeal,
>>> NFS URLs (RFC 2224) don't specify a generic NFS resource. They specify a
>>> webDAV like resource that can be accessed with NFS, called WebNFS (RFC
>>> 2054, RFC 20550), which gives clients access via a so-called "public file
>>> handle," which is a degenerate NFS FH.
>>>
>>> WebNFS is defined only for legacy versions of NFS, not for NFSv4.
>>> Referrals are supported only in NFSv4. In fact, section 4 of RFC 2224
>>> specifies that clients try version 3 then version 2. NFS version 4 is not
>>> discussed.
>>>
>>> Thus, the form of an NFS URL might be rich enough, but the existing
>>> semantics are not equivalent.
>>
>> I see no reason why it should not be able to use NFS URLs and define the exact
>> usage of them for NFSv4. Maybe I'm overlooking something though.
>
> Agreed. AFAICS the semantics of the URL are for your actual application to
> define. We're not talking about URLs that are meant to be served up by an HTTP
> server.
>
> IMO RFC2224 is defective in that it specifies both the URL format and its
> associated access methods. URL formats are meant to be "Uniform" which should
> mean they are independent of their access methods.
In order to use the NFS URL format, I think we would be compelled to correct RFC 2224, probably by issuing an RFC that supersedes it. While certainly possible, I'm not sure the NFSv4 working group is prepared to hold up the NSDB draft for such an undertaking at this point. Naturally there may be simpler alternatives.
That said, I can present this design choice (the use of an NFS URL to represent an fs_locations entry in LDAP) to them for discussion.
Thanks for the excellent feedback.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
11 years, 1 month
AW: overlay nssov + SASL
by Uwe Werler
-----Ursprüngliche Nachricht-----
An: openldap-technical(a)openldap.org;
Von: Uwe Werler <uwe.werler(a)retiolum.eu>
Gesendet: Mo 13.08.2012 09:50
Betreff: overlay nssov + SASL
> Hello list,
>
> I want to use overlay nssov and it works great in my test setup. Up to now we
> use pass through authentication with saslauthd to authenticate against an
> AD-Domain.
>
> Internally nssov uses simple binds against the local database so I can't use
> pass through authentication. With overlay pbind I can do simple binds against
> an external ldap server. Is there another possibility to use overlay nssov in
> combination with saslauthd?
>
> Thanks for any hints.
>
> Regards Uwe
Ok, I found two ways. First use back relay + overlay pbind and second use back-meta/back-ldap. Both within the same naming context as the real database but configured with "hidden on" to avoid naming context issues. Both work now.
Another question is if it's possible to use overlay nssov within multiple databases like one for hosts and another for users?
Thanks!
Regards Uwe
11 years, 1 month
How to enable LDAP ports in iptables for OpenLDAP client node
by Qian Zhang
Hi All,
I have a RHEL 6.2 machine which is set up as an OpenLDAP client, and I
can log into it with LDAP user.
Now for security concern, I need to prohibit any not-root user to
access the network:
# /etc/init.d/iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 REJECT all -- 0.0.0.0/0 0.0.0.0/0 !
owner UID match 0 reject-with icmp-port-unreachable
But if I did this in iptables, LDAP has problems, "getent passwd" can
not get any LDAP users, and I can no longer log into this machine with
LDAP user. So I think I need to open LDAP ports in iptables, what I
did is:
# /etc/init.d/iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp
spt:389 dpt:389
2 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp
spt:389 dpt:389
3 REJECT all -- 0.0.0.0/0 0.0.0.0/0 !
owner UID match 0 reject-with icmp-port-unreachable
But it did not work, any ports I missed? Or what I set up in iptables
are not correct? My /etc/openldap/ldap.conf:
URI ldap://172.17.27.159:389
BASE dc=base,dc=com
TLS_CACERTDIR /etc/openldap/cacerts
Regards,
Qian
11 years, 1 month
overlay nssov + SASL
by Uwe Werler
Hello list,
I want to use overlay nssov and it works great in my test setup. Up to now we use pass through authentication with saslauthd to authenticate against an AD-Domain.
Internally nssov uses simple binds against the local database so I can't use pass through authentication. With overlay pbind I can do simple binds against an external ldap server. Is there another possibility to use overlay nssov in combination with saslauthd?
Thanks for any hints.
Regards Uwe
11 years, 1 month
Windows SSPI, NTLMSSP and OpenLDAP with SASL/GSSAPI
by Diego Morales
Hello,
I have a proprietary windows application trying to bind on my OpenLDAP
server using GSSAPI with NTLMSSP mechanism, instead of Kerberos. Is it
possible to support this on a (unix) OpenLDAP server?
Another option would be to make the software use GSSAPI + Kerberos
instead. Let me further explain:
I have a working samba + openldap setup with many windows
workstations. The said proprietary app has LDAP auth support, and
according to its maker it works with Active Directory and Novell NDS.
It does not support simple bind, nor LDAPS, (and probably not StartTLS
either). We don't have access to the app's source code and help from
its developers/tech-support is pretty unavailable.
Checking slapd's debug, we saw the app trying to use SASL+GSSAPI to
bind. So we went on and configured a minimal Kerberos setup and
SASL+GSSAPI support for OpenLDAP on a test ldap server. It seems to be
working perfectly. We acquire a ticket and run ldapsearch from another
machine using -Y GSSAPI bind and it works. Logs from slapd debug seem
ok.
But that evil app still fails. Here's a piece from slapd debug log:
conn=1000 op=1 do_bind
ber_scanf fmt ({imt) ber:
ber_dump: buf=0x7f73f6af8810 ptr=0x7f73f6af8813 end=0x7f73f6af8856 len=67
0000: 60 84 00 00 00 3d 02 01 03 04 00 a3 84 00 00 00 `....=..........
0010: 32 04 06 47 53 53 41 50 49 04 28 4e 54 4c 4d 53 2..GSSAPI.(NTLMS
0020: 53 50 00 01 00 00 00 97 82 08 e2 00 00 00 00 00 SP..............
0030: 00 00 00 00 00 00 00 00 00 00 00 06 01 b1 1d 00 ................
0040: 00 00 0f ...
ber_scanf fmt ({m) ber:
ber_dump: buf=0x7f73f6af8810 ptr=0x7f73f6af881e end=0x7f73f6af8856 len=56
0000: 00 84 00 00 00 32 04 06 47 53 53 41 50 49 04 28 .....2..GSSAPI.(
0010: 4e 54 4c 4d 53 53 50 00 01 00 00 00 97 82 08 e2 NTLMSSP.........
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0030: 06 01 b1 1d 00 00 00 0f ........
ber_scanf fmt (m) ber:
ber_dump: buf=0x7f73f6af8810 ptr=0x7f73f6af882c end=0x7f73f6af8856 len=42
0000: 00 28 4e 54 4c 4d 53 53 50 00 01 00 00 00 97 82 .(NTLMSSP.......
0010: 08 e2 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0020: 00 00 06 01 b1 1d 00 00 00 0f ..........
ber_scanf fmt (}}) ber:
ber_dump: buf=0x7f73f6af8810 ptr=0x7f73f6af8856 end=0x7f73f6af8856 len=0
>>> dnPrettyNormal: <>
<<< dnPrettyNormal: <>, <>
conn=1000 op=1 BIND dn="" method=163
do_bind: dn () SASL mech GSSAPI
==> sasl_bind: dn="" mech=GSSAPI datalen=40
SASL [conn=1000] Failure: GSSAPI Error: An unsupported mechanism was
requested (Unknown error)
send_ldap_result: conn=1000 op=1 p=3
send_ldap_result: err=49 matched="" text="SASL(-13): authentication
failure: GSSAPI Failure: gss_accept_sec_context"
send_ldap_response: msgid=11 tag=97 err=49
ber_flush2: 87 bytes to sd 13
0000: 30 55 02 01 0b 61 50 0a 01 31 04 00 04 49 53 41 0U...aP..1...ISA
0010: 53 4c 28 2d 31 33 29 3a 20 61 75 74 68 65 6e 74 SL(-13): authent
0020: 69 63 61 74 69 6f 6e 20 66 61 69 6c 75 72 65 3a ication failure:
0030: 20 47 53 53 41 50 49 20 46 61 69 6c 75 72 65 3a GSSAPI Failure:
0040: 20 67 73 73 5f 61 63 63 65 70 74 5f 73 65 63 5f gss_accept_sec_
0050: 63 6f 6e 74 65 78 74 context
ldap_write: want=87, written=87
0000: 30 55 02 01 0b 61 50 0a 01 31 04 00 04 49 53 41 0U...aP..1...ISA
0010: 53 4c 28 2d 31 33 29 3a 20 61 75 74 68 65 6e 74 SL(-13): authent
0020: 69 63 61 74 69 6f 6e 20 66 61 69 6c 75 72 65 3a ication failure:
0030: 20 47 53 53 41 50 49 20 46 61 69 6c 75 72 65 3a GSSAPI Failure:
0040: 20 67 73 73 5f 61 63 63 65 70 74 5f 73 65 63 5f gss_accept_sec_
0050: 63 6f 6e 74 65 78 74 context
conn=1000 op=1 RESULT tag=97 err=49 text=SASL(-13): authentication
failure: GSSAPI Failure: gss_accept_sec_context
(btw, this is slapd 2.4.21, from a 10.04 ubuntu package)
I believe the application uses Windows SSPI, and I known SSPI supports
several GSSAPI mechanisms, including NTLMSSP and Kerberos. I'm afraid
Windows is auto selecting NTLMSSP cause its running on a pre-windows
2000 domain (non AD, in this case, Samba). Hoping to make windows use
Kerberos instead, I've also tried publishing some SRV records on DNS.
I have sniffed DNS queries from the workstation while the app tries to
login, caught only one _ldap._tcp SRV request, registered that ... and
nothing has changed.
I don't know how could I force the app to use GSSAPI + kerberos
without touching its source code. And I can't find much about a unix
NTLM(SSP)-as-a-mechanism-of-GSSAPI implementation. Maybe there's
something inside samba4 or in Likewise software, but I haven't found
it yet.
So ... does somebody have any advice or info?
Thanks in advance,
Diego Morales
+55 (51) 3024-3568
Propus Informática LTDA.
http://www.propus.com.br
11 years, 1 month
