Use-case specific changes to LMDB
by martin＠urbackup.org
This post outlines a few changes to LMDB I had to do to make it work in a specific use case. I’d like to see those changes upstream, but I understand that they may be/are not relevant for e.g. OpenLDAP.
The use case is multiple databases on disks with long running large write transactions.
1. Option to not use custom memory allocator/page pool
LMDB has a custom malloc() implementation that re-uses pages (me_dpages). I understand that this improves the performance at bit (depending on the malloc implementation). But there should at least be the option to not do that (for many reasons). I would even make not using it the default.
2. Large transactions and spilling
In a large write transaction, it will use a lot of memory per default (512MiB) which won’t get freed when the transaction commits (see 1.). If one has a lot of databases it uses a lot of memory that never gets freed.
Alternatively, one can use MDB_WRITEMAP, but (i) per default Linux isn’t tuned to delay writing pages to disk and (ii) before commit LMDB has to remove a dirty bit, so each page is written twice.
Both problems would be fixed by making when pages get spilled configurable (mt_dirty_room as MDB_IDL_UM_MAX currently) and reducing the default non-spill memory amount for at least the MDB_WRITEMAP case. If this memory amount is low mt_spill_pgs gets sorted often so maybe this needs to be converted to a different data structure (e.g. red-black tree).
3. LMDB causes crashes if database is corrupted
If the database is corrupted it can cause the application to crash. I have fixed those cases when they (randomly) occurred. Properly fixing this would probably be best done with some fuzzing.
4. Allow LMDB to reside on a device
I used dm-cache to improve LMDB read performance. It needed a bit of adjustment to get the correct size of the device via ioctl BLKGETSIZE64.
--
I’ve fixed those issues w.r.t. my application. If there is interest in any of those application specific changes, I’ll clean them up and post them.
2 months, 4 weeks
Reset admin pass
by Клеусов Владимир Сергеевич
Hi.
How do I change the admin password correctly and not break replication ?
=)
here when setting up replication the password was mentioned
dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcRootPW
olcRootPW: 123 And
dn: olcDatabase={0}config,cn=config
changetype: modify
replace: olcSyncRepl
olcSyncRepl: rid=001
provider=ldaps://ldap1.domain.com
searchbase="cn=config"
bindmethod=simple
credentials=123
binddn="cn=admin,cn=config"
tls_reqcert=never
type=refreshAndPersist
retry="30 +"
timeout=1
olcSyncRepl: rid=002
provider=ldaps://ldap2.domain.com
searchbase="cn=config"
bindmethod=simple
credentials=123
binddn="cn=admin,cn=config"
tls_reqcert=never type=refreshAndPersist
retry="30 +"
timeout=1
tls_reqcert=never type=refreshAndPersist
retry="5 5 300 5"
timeout=1
-
replace: olcMirrorMode
olcMirrorMode: TRUE
4 months, 1 week
Other error (80) when adding olcTLSCipherSuite
by Benjamin Schneider
Hi all, I'm running version 2.4.49 on Ubuntu 20.04. I've been unable to add
the olcTLSCipherSuite configuration attribute.
# ldapmodify -H ldapi:// -Y EXTERNAL -f set-ciphersuite.ldif
returns:
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
modifying entry "cn=config"
ldap_modify: Other (e.g., implementation specific) error (80)
set-ciphersuite.ldif contains the following:
dn: cn=config
changetype: modify
add: olcTLSCipherSuite
olcTLSCipherSuite: ALL
I was able to successfully configure (and confirmed working) TLS by setting
the following attributes:
olcTLSCACertificateFile: /etc/ssl/certs/mydomain.fullchain.pem
olcTLSCertificateFile: /etc/ssl/certs/mydomain.cert.pem
olcTLSCertificateKeyFile: /etc/ssl/private/mydomain.privkey.pem
and was just looking to limit which ciphers would be offered.
I've found several discussions (here, on stackoverflow, etc.) that mention
this error, but those discussions concerned these latter TLS attributes
(which I had no problem adding) and not the olcTLSCipherSuite attribute.
They also pointed to file permissions being the issue for the certificate
files, which I've confirmed is not an issue. I would be grateful if anyone
could point me in the right direction
Ben
4 months, 2 weeks
LDAP proxy backend does not drop idle connections
by Tero Saarni
Hi,
I understood from slapd-ldap(5) description of "idle-timeout" that cached
connections towards remote LDAP server would be automatically dropped after
<time> seconds.
Problem: cached connections that are idle do not get dropped.
Questions:
(1) Is this expected?
(2) Are idle connections kept due to limitation in the implementation:
when connection is idle, back-ldap does not have a trigger that could be used
to drop idle connections?
Background:
While experimenting with this, it seems that idle timeout is only checked when
there is new activity towards the cached connection i.e. connection needs to
become active before idle timeout is checked. If the connection just remains
idle, nothing will happen.
I'm trying to study the timeout handling in back-ldap code, and I believe I
found relevant code at the end of ldap_back_getconn() in bind.c. It will
eventually trigger unbind and disconnect, but only when new activity happens
after the idle period is reached. I did not find other paths that could
trigger unbind of cached connection.
--
Tero
4 months, 2 weeks
HAProxy protocol support?
by Paul B. Henson
So management is insisting that we migrate our openLDAP systems from on
premise into the cloud <sigh>. Specifically, AWS behind one of their
load balancers.
However, we currently rely upon some level of IP address based access
control to distinguish between on-campus and off-campus clients. The
Amazon load balancers do client NAT, so the back end servers have no
idea who is connecting at the TCP/IP level.
They do support the haproxy in band protocol for supplying this
information from the load balancer to the server, but that requires
specific support from the server to do. I don't see any such support in
openldap or any evidence of past discussion regarding it.
Is this something that would be considered as a possible feature to be
included at some point, or something not desired as part of the code base?
Thanks...
4 months, 2 weeks
Antw: [EXT] Re: HAProxy protocol support?
by Ulrich Windl
>>> "Dr. Ogg" <ogg(a)sr375.com> schrieb am 18.11.2020 um 17:55 in Nachricht
<DM5PR06MB32906E48D22C0F65570D9BD0F0E10(a)DM5PR06MB3290.namprd06.prod.outlook.com>
> http://www.haproxy.org/download/1.8/doc/proxy‑protocol.txt
>
> for reference.
>
>
> From: Howard Chu <hyc(a)symas.com>
> Date: Wednesday, November 18, 2020 at 8:51 AM
> To: Paul B. Henson <henson(a)acm.org>, openldap‑technical(a)openldap.org
> <openldap‑technical(a)openldap.org>
> Subject: Re: HAProxy protocol support?
> Paul B. Henson wrote:
>> So management is insisting that we migrate our openLDAP systems from on
> premise into the cloud <sigh>. Specifically, AWS behind one of their load
> balancers.
>>
>> However, we currently rely upon some level of IP address based access
> control to distinguish between on‑campus and off‑campus clients. The Amazon
> load balancers
>> do client NAT, so the back end servers have no idea who is connecting at
the
> TCP/IP level.
>>
>> They do support the haproxy in band protocol for supplying this information
> from the load balancer to the server, but that requires specific support
from
> the
>> server to do. I don't see any such support in openldap or any evidence of
> past discussion regarding it.
>>
>> Is this something that would be considered as a possible feature to be
> included at some point, or something not desired as part of the code base?
>
> Depends on what that feature actually looks like. Feel free to submit a
> proposal
> on the ‑devel mailing list, including background info on what HAproxy
> protocol
> looks like, and what exact behaviors you want it to provide.
I wonder: Would it be possible to use a specific named bind for on-campus
hosts, and use the name used for binding to controll further access?
>
> ‑‑
> ‑‑ Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
5 months
NTLMv2 credentials over ldap
by moore chestnut
Hello.
I have a proxy application acting as a ntlm server, that supports a NTLM
handshake between web based clients.
If using NTLMv1, sending the NTLM credential blob to an Active Directory
over LDAP using openldap client works.
openldap client Version. 2.4.32
Basically just taking the ntlm response from the NTLM 3 message from client
and sending over LDAP.
However using NTLMv2 , the active directory always issues invalid
credentials even though the user name and password that the client entered
are good. It passes with successful ldap bind with NTLMv1.
Can or should this work with ntlmv2? It seems that when EPA and MIC is
present from client in the NTLM3, that the ldap exchange does not work. I
guess that may be an active directory issue but wanted to check if experts
here think it should work.
Thanks for your review.
5 months
LMDB alignment
by Alec Matusis
Our code gets references to 8 or 16 byte structs by casting pointers into
LMDB data.
1. In a database with no DUPs and 8 byte (u64) keys, can we expect the
corresponding value to have alignment 8?
2. In a database with DUPs and 8 byte (u64) keys, if a DUP value (stored as
a key internally) is 16 bytes, does it mean that its alignment is 16? Does
the 8 byte key size impact the alignment of its DUPs as it does in 1.?
5 months
how to add group in the slave/consumer node
by 张连生
Hi，
My company maintains an openldap server which stores the information of all the employees. All company internal systems authenticate with it when users login.
My department is responsible for software developing/testing and divided into many teams. I want to add the employees of my department to corresponding team in openldap so that I can manage user permission based on teams in jira/confluence/gerrit/gitlab/svn/jenkins and so on. However, I have no permission to add team or group to company openldap server.
My plan is to :
1. set up a new openldap server inside my department.
2. synchronize the user data necessary from the company openldap server to my department openldap server.
3. create groups in department openldap server.
4. add users to corresponding group in department openldap server.
5. jira/confluence/gerrit/gitlab/svn/jenkins with authenticate with department openldap server instead of the company one.
How to configure openldap to achieve this？I have googled for two days about replication/meta-directory, but still have no idea.
BTW, I know Jira have similar functionality and can authenticate for confluence, but Jira can not authenticate for other sofeware such as gerrit/gitlab/svn/jenkins.
Any help is appreciated.
BRs,
Liansheng
5 months
