LMDB persistent snapshots
by Michael Conrad
Hi, I'm scoping out my options for databases that support snapshotting. From what I've read so far, this is a natural feature of LMDB, however all the documentation talks about making sure not to let read transactions linger around. What if I *want* to hold onto a read transaction long-term, hours or even months? Also, I haven't yet seen a way for a new client to re-open the database as of the point in time of an existing transaction. Is this possible?
If I manage to do something like this, will the storage overhead be roughly equivalent to the quantity of changed data between the snapshot and the latest write? or will the overhead be more like the sum of all writes + overwrites that happened since the read transaction started? i.e. if a read transaction is held starting from t0, then data is written at t1, then overwritten at t2 can the blocks affected by t1 be reclaimed for a write at t3?
Thanks for any insight!
-Mike
6 years, 4 months
Re: back-ldap and ldaps not working
by Quanah Gibson-Mount
--On Friday, July 07, 2017 6:27 PM +0000 Jon C Kidder <jckidder(a)aep.com>
wrote:
> olcDbStartTLS: ldaps starttls=no
> tls_cacert="/appl/openldap/etc/openldap/tls/cacerts.cer"
> tls_reqcert=demand tls_crlcheck=none
There is no such option as "starttls" for the olcDbStartTLS parameter.
>From the man page:
tls {[try-]start|[try-]propagate|ldaps}
[tls_cert=<file>]
[tls_key=<file>] [tls_cacert=<file>]
[tls_cacertdir=<path>]
[tls_reqcert=never|allow|try|demand]
[tls_cipher_suite=<ciphers>] [tls_crlcheck=none|peer|all]
Specify the use of TLS when a regular connection is
initialized.
The StartTLS extended operation will be used unless the
URI
directive protocol scheme is ldaps://. In that case this
keyword
may only be set to "ldaps" and the StartTLS operation will
not
be used. propagate issues the StartTLS operation only if
the
original connection did. The try- prefix instructs the proxy
to
continue operations if the StartTLS operation failed; its use
is
not recommended.
My guess then is everything past that point is ignored, since you've
provided invalid configuration data.
The attribute is also poorly named, as it may or may not have anything to
do with startTLS. More likely it should have just been called olcDbTLS
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 4 months
Re: [EXTERNAL] Re: back-ldap and ldaps not working
by Howard Chu
Jon C Kidder wrote:
> Thanks Michael. The message is clear but the solution isn't. I think you missed the part about this exact same slapd node being a replication consumer replicating successfully using the exact same certificate/TLS setup. Just for added validation the masters have been active for a couple years serving a very active test environment with ~100 test clients connecting via ldaps. Something appears to be hinky with the configuration processing or certificate validation processing in back-ldap. My gut is that olcDBStartTLS isn't being converted/formatted correctly or isn't being parsed correctly and the option to set the path to the CA cert file is being ignored.
In that case you could use strace to verify whether the CA cert file is ever
being opened.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6 years, 4 months
RE: [EXTERNAL] Re: back-ldap and ldaps not working
by Quanah Gibson-Mount
--On Friday, July 07, 2017 9:39 PM +0000 Jon C Kidder <jckidder(a)aep.com>
wrote:
> Yeah, that's actually how I started and where the starttls=no setting
> came from.
>
> This .conf section
>
> overlay chain
> chain-uri "ldaps://ds2-q.global.aep.com"
> chain-rebind-as-user TRUE
> chain-idassert-bind bindmethod=simple
> binddn="cn=syncuser,ou=Automatons,ou=Users,dc=Global,dc=aep,dc=com"
> credentials=<redacted> mode="self" chain-tls ldaps
> tls_cacert=/appl/openldap/etc/openldap/tls/cacerts.cer chain-return-error
> TRUE
Hm, if the conversion is adding that "starttls=no" to the cn=config
database, that seems like a serious bug in the conversion process.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 4 months
RE: [EXTERNAL] Re: back-ldap and ldaps not working
by Quanah Gibson-Mount
--On Friday, July 07, 2017 8:10 PM +0000 Jon C Kidder <jckidder(a)aep.com>
wrote:
> I've removed the starttls=no syntax and the line now reads.
>
> olcDbStartTLS: ldaps
> tls_cacert="/appl/openldap/etc/openldap/tls/cacerts.cer "
> tls_reqcert=demand tls_crlcheck=none
>
> I have verified the change propagated to the configuration directory and
> restarted the instance. I saw no errors during configuration parsing in
> the log. I am still seeing this error when the chain overlay tries to
> follow the referral but no complaints when syncrepl connects.
I'm not sure how you do this with cn=config. With slapd.conf, it would be
done via using "chain-tls" and not "tls", as per the man page:
There are very few chain overlay specific directives;
however,
directives related to the instances of the ldap backend that may
be
implicitly instantiated by the overlay may assume a special
meaning
when used in conjunction with this overlay. They are described
in
slapd-ldap(5), and they also need to be prefixed by chain-.
It may be worthwhile to set up a slapd.conf where "chain-tls" is specified,
and see what happens with that on conversion.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 4 months
back-ldap and ldaps not working
by Jon C Kidder
I have the following attributes set in my ldap backend for the chain overlay.
olcDbURI: "ldaps://ds2-q.global.aep.com:636"
olcDbStartTLS: ldaps starttls=no tls_cacert="/appl/openldap/etc/openldap/tls/cacerts.cer" tls_reqcert=demand tls_crlcheck=none
The referenced file is the exact same file I use in this global attribute
olcTLSCACertificateFile: /appl/openldap/etc/openldap/tls/cacerts.cer
This is a 2.4.44 replication consumer using the following replication attribute
olcSyncrepl: {1}rid=112 provider=ldaps://ds2-q.global.aep.com:636 binddn="cn=syncuser,ou=Automatons,ou=Users,dc=Global,dc=aep,dc=com" bindmethod=simple credentials=<redacted> searchbase="dc=Global,dc=aep,dc=com" type=refreshAndPersist retry="5 5 300 +" timeout=1
Replication works perfectly and changes to the referenced master are replicated to this slave. I can see successful connections for rid=112 to this master in the log. The problem is when the chain overlay tries to follow referrals to this same master I get the following error:
595fbb1c conn=1000 op=1 ldap_chain_op: ref="ldaps://ds2-q.global.aep.com:636/uid=s012235,ou=Employees,ou=Users,dc=Global,dc=aep,dc=com" -> "ldaps://ds2-q.global.aep.com:636"
595fbb1c conn=1000 op=1 ldap_chain_op: ref="ldaps://ds2-q.global.aep.com:636/uid=s012235,ou=Employees,ou=Users,dc=Global,dc=aep,dc=com": URI="ldaps://ds2-q.global.aep.com:636" found in cache
ldap_create
ldap_url_parse_ext(ldaps://ds2-q.global.aep.com:636)
595fbb1c =>ldap_back_getconn: conn=1000 op=1: lc=0x10180430 inserted refcnt=1 rc=0
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP ds2-q.global.aep.com:636
ldap_new_socket: 16
ldap_prepare_socket: 16
ldap_connect_to_host: Trying 10.92.127.52:636
ldap_pvt_connect: fd: 16 tm: -1 async: 0
attempting to connect:
connect success
TLS trace: SSL_connect:before/connect initialization
TLS trace: SSL_connect:SSLv2/v3 write client hello A
TLS trace: SSL_connect:SSLv3 read server hello A
TLS certificate verification: depth: 1, err: 19, subject: /C=US/ST=Ohio/L=Columbus/O=American Electric Power/OU=Complex - Middleware/CN=AEP Root CA (2014)/emailAddress=middleware(a)aep.com, issuer: /C=US/ST=Ohio/L=Columbus/O=American Electric Power/OU=Complex - Middleware/CN=AEP Root CA (2014)/emailAddress=middleware(a)aep.com
TLS certificate verification: Error, self signed certificate in certificate chain
TLS trace: SSL3 alert write:fatal:unknown CA
TLS trace: SSL_connect:error in error
TLS trace: SSL_connect:error in error
TLS: can't connect: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (self signed certificate in certificate chain).
595fbb1c send_ldap_result: conn=1000 op=1 p=3
595fbb1c send_ldap_result: conn=1000 op=1 p=3
595fbb1c send_ldap_response: msgid=2 tag=103 err=52
ber_flush2: 14 bytes to sd 15
595fbb1c conn=1000 op=1 RESULT tag=103 err=52 text=
So, is there something wrong with the value of the olcDBStartTLS attribute that I'm not seeing??
-Jon C. Kidder
American Electric Power
Complex - Middleware Engineering
6 years, 4 months
RE: Using TLS
by Quanah Gibson-Mount
--On Friday, July 07, 2017 4:29 PM +0000 Daniel Le <daniel.le(a)exfo.com>
wrote:
> Right, ldap_set_option doesn't take string input argument. I implied an
> integer value such as LDAP_OPT_X_TLS_NEVER, LDAP_OPT_X_TLS_ALLOW, etc.
>
> I have no doubt it works nicely in your case. Just trying to understand
> why it doesn't on my side.
Hard to say without the full context of your code. It clearly works not
just with what I did, but with other client code as well, whether it's the
bits that part ldap.conf/.ldaprc to the syncrepl client code, etc.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 4 months
RE: Using TLS
by Quanah Gibson-Mount
--On Friday, July 07, 2017 4:05 PM +0000 Daniel Le <daniel.le(a)exfo.com>
wrote:
> Hi Quanah,
>
> I tried ldap_int_tls_config for RE24 in my app program as shown below and
> it didn't work.
Interesting. ;/ I wonder if there's something more in what you're doing
outside of that code snippet that's causing the context to be lost. It
clearly works with the code I've done (and in general with the client
libraries).
> Additionally, I'm curious why you use ldap_int_tls_config instead of
> ldap_set_option?
ldap_set_option can't take "never" as an argument to
LDAP_OPT_X_TLS_REQUIRE_CERT, but ldap_int_tls_config can. If I had used
ldap_set_option, I would have needed to do a ton of code duplication for
error checking. Similar for the LDAP_OPT_X_TLS_PROTOCOL_MIN and
LDAP_OPT_X_TLS_CRLCHECK options. Much better to do code re-use. ;)
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 4 months
Re: [Bad Attachment] Slave won't send referrals
by Quanah Gibson-Mount
--On Friday, July 07, 2017 3:10 PM +0000 Jon C Kidder <jckidder(a)aep.com>
wrote:
>
>
> Ok, this should be simple right? I have a 2.4.44 slave with olcUpdateRef
> pointing to the master. Replication works fine and the slave reflects
> any changes made to the master. The slave is also accepting modify
> operations without complaining and without returning a referral. I've
> compared my configuration with the configuration generated for test022. I
> have the needful extra stuff in my configuration and everything looks
> correct to me. I've attached the modify command line and result, the
> slave configuration, and the log of the operation for reference.
My email provider still scrubbed your attachment. ;)
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 4 months
