Re: Antw: [EXT] Re: Unexpected LMDB RSS /performance difference on similar machines
by Howard Chu
Ulrich Windl wrote:
>>>> Howard Chu <hyc(a)symas.com> schrieb am 07.06.2020 um 22:44 in Nachricht
> <14412_1591562670_5EDD51AD_14412_294_1_79b319e0-fa23-a622-893b-b1b558a9385c@syma
> .com>:
>> Alec Matusis wrote:
>>> 2. dd reads the entire environment file into system file buffers (93GB).
>> Then when the entire environment is cached, I run the binary with
>> MDB_NORDAHEAD, but now it reads 80GB into shared memory, like when
>> MDB_NORDAHEAD is not set. Is this expected? Can it be prevented?
>>
>> It's not reading anything, since the data is already cached in memory.
>>
>> Is this expected? Yes - the data is already present, and LMDB always
>> requests a single mmap for the entire size of the environment. Since
>> the physical memory is already assigned, the mmap contains it all.
>>
>> Can it be prevented - why does it matter? If any other process needs
>> to use the RAM, it will get it automatically.
>
> While reading this: The amount of memory could suggest that using the
> hugepages feature could speed up things, especially if most of the mmapped data
> is expected to reside in RAM. Hugepages need to be enabled using
> vm.nr_hugepages=... in /etc/sysctl.conf (or corresponding). However I don't
> know whether LMDB can use them.
>
> A current AMD CPU offers these page sizes: 4k, 2M, and 1G, but some VMs (like
> Xen) can't use it. On the system I see, hugepages are 2MB on size. I don't know
> what the internal block size of LMDB is, but likely it would benefit to match
> the hugepage size if using it...
https://git.openldap.org/openldap/openldap/-/blob/13f3bcd59c2055d53e4759b...
We have already had this discussion, and your suggestion was irrelevant back then too.
https://www.openldap.org/lists/openldap-technical/201401/msg00213.html
Please stop posting disinformation.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
2 years, 12 months
Unexpected LMDB RSS /performance difference on similar machines
by Alec Matusis
We have encountered an unexpected performance impact only by moving an LMDB
to a different linux machine with similar hardware characteristics.
We have a 93GB LMDB environment with 3 databases. The database in question
is 13GB. The test executable loops over key/value pairs in the 13GB database
with a read-only cursor. For the same executable, we observe two different
behaviors on different machines (the same lmdb environment was copied to
the machines with scp). First machine has 148GB RAM and the second has 105GB
RAM and the same CPU.
The expected and desired behavior on linux kernel 3.13 / eglibc 2.19 shows
that the process takes 13GB of shared memory (seen by top and confirmed
with /proc/<pid>/smaps below)). In the alternate behavior on kernel 4.15 /
glibc 2.27 the process reads from disk 83GB into shared memory (which takes
16 minutes instead of 8 minutes on the correct machine in the corresponding
initial runs).
/proc/<pid>/smaps
Machine with expected behavior:
7f3595da8000-7f4cde517000 r--s 00000000 fb:10 6442473002
/fusionio1/lmdb/db.0/dbgraph/data.mdb
Size: 97656252 kB
Rss: 13203648 kB
Pss: 13203648 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 13203648 kB
Private_Dirty: 0 kB
Referenced: 13203648 kB
Anonymous: 0 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
VmFlags: rd sh mr mw me ms sd
Machine with excessive Rss and slower read time:
7f55990aa000-7f6ce1819000 r--s 00000000 fc:02 7077908
/lmdbguest/lmdb/db.0/dbgraph/data.mdb
Size: 97656252 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 82587036 kB
Pss: 82587036 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 82587036 kB
Private_Dirty: 0 kB
Referenced: 82587036 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
2 years, 12 months
ppolicy: attempting to load ppolicy module
by jalexander＠concentricsky.com
I have an OpenLDAP server on CentOS 7.8 and am trying to get the ppolicy overlay loaded.
I'm attempting to load in the module with this command:
ldapmodify -Y EXTERNAL -H ldapi:/// -f ppolicy_load.ldif
and I get this error:
add olcModuleLoad:
ppolicy
modifying entry "cn=module{0},cn=config"
ldap_modify: Type or value exists (20)
additional info: modify/add: olcModuleLoad: value #0 already exists
The contents of ppolicy.ldif:
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: ppolicy
cn=module{0},cn=config.ldif contents:
dn: cn=module{0}
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/lib64/openldap
olcModuleLoad: {0}back_bdb
olcModuleLoad: {1}syncprov
Can anyone provide guidance on how to get this working? I configured this on test servers that I derived from my main ldap server with no problem.
2 years, 12 months
pwdChangedTime not defined when creating new entry
by Manuela Mandache
Hello all,
We have a directory running on OpenLDAP 2.4.44 with the ppolicy overlay on the main database. When a new entry with a userPassword defined is created, pwdChangedTime is not defined, so this initial userPassword never expires.
The directory has been migrated from its OpenLDAP 2.3.34 instance (yes, we missed some steps...), and there the pwdChangedTime is set, and naturally equal to createTimestamp.
The overlay is configured as follows:
dn: olcOverlay={2}ppolicy,olcDatabase={2}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcPPolicyConfig
olcOverlay: {2}ppolicy
olcPPolicyDefault: ou=ppolicy,dc=example,dc=com
olcPPolicyHashCleartext: TRUE
olcPPolicyUseLockout: TRUE
Is there a parameter I missed which would switch on setting pwdChangedTime at entry creation? Do I have to provide some other configuration elements?
Or is it unreasonable to expect this initialisation of the attribute this way, and only a password change can set it? I think the setting at creation is rather handy... Using pwdMustChange would be difficult, we have a lot of client apps which would be forced to check and probably adapt their authentication procedures.
Thank you and regards,
Manuela
Sent with [ProtonMail](https://protonmail.com) Secure Email.
3 years
Re: ssl certificate chain
by Dale Thompson - NOAA Federal
I'm not certain the hack redhat added to force openldap to use nss actually
causes openldap to use the nss cert store. My rhel6 openldap servers appear
to just use the PEM certs they would have used as if redhat never messed
with forcing openldap to use nss, but rather left it at openssl. I did
check and slapd is linked against the nss libs, but using the pem file in
/etc/openldap/cacerts.
The fix for this might be as simple as linking the PEM version of the
updated cert store into the directory where openldap is looking.
On Wed, Jun 3, 2020 at 11:32 AM Quanah Gibson-Mount <quanah(a)symas.com>
wrote:
>
>
> --On Wednesday, June 3, 2020 8:15 PM +0300 Леонид Юрьев
> <leo(a)yuriev.ru> wrote:
>
> > Seems this is
> > https://www.agwa.name/blog/post/fixing_the_addtrust_root_expiration.
>
> Since RHEL6 is in use here, specifically see the linked tweet for
> Fedora/RHEL in the above post.
>
> Regards,
> Quanah
>
> --
>
> Quanah Gibson-Mount
> Product Architect
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>
>
--
Dale James Thompson, NWS
Radar Operations Center
IT Specialist, Configuration Management Team
1313 Halley Circle
Norman, OK 73069
Voice (405) 573-3472
Fax (405) 573-3480
Dale.J.Thompson(a)noaa.gov
3 years
userPassword is not replicated
by razvanpopescu＠hotmail.com
Hi,
I have set up a replication master/slave between 2 openldap 2.4.44 on rhel 7.x.
On the slave server, the userPassword attribute is not replicated by syncrepl, all other attributes are replicated OK
The replication has been set up as follow:
On master server (provider), I have set up :
# replication
moduleload syncprov
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
On slave server (consumer), I've set up in the /etc/openldap/slapd.conf:
# replication
syncrepl rid=100
provider=ldaps://fr-te-ldap-x1.intra.commercial-union.fr
type=refreshAndPersist
searchbase="dc=aviva,dc=fr"
scope=sub
schemachecking=on
bindmethod=simple
filter="(objectClass=*)"
binddn="cn=admin,dc=aviva,dc=fr"
credentials=redhat
retry="15 +"
index entryUUID,entryCSN eq
sizelimit 100000
On both server ( master, slave) , the ACL has been set up as follow :
access to attrs=userPassword
by self write
by anonymous auth
by * read
access to *
by self read
by users read
by anonymous read
Please help me !
What is wrong in this configuration and why the userPassword attribute is not replicated on slave side ?
Please advice me,
Thank,
Razvan
3 years
ssl certificate chain
by phei＠isc.upenn.edu
Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate.
Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
But if we run an 'openssl s_client -showcerts' against the host and port 636, we continue to see the expired root certificate even though it's not in the nss store configured chain. This is causing issues for some applications (mainly java based) so we're just trying to understand where the expired root would be coming from if it's not in the openldap server configuration.
Thanks,
Peter
relevant bits:
#slapd.conf
# TLS/ssl
TLSCACertificatePath /etc/openldap/certs
TLSCertificateFile DirectoryLdap
#sudo certutil -d /etc/openldap/certs -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
intermediate-2024 ,,
PennGroupsLdap
#sudo certutil -d /etc/openldap/certs -L -n PennGroupsLdap
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
2b:27:6c:70:ac:b4:5c:3d:11:05:17:d9:15:59:24:af
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: "CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbo
r,ST=MI,C=US"
Validity:
Not Before: Tue Feb 05 00:00:00 2019
Not After : Thu Feb 04 23:59:59 2021
Subject: "CN=penngroups-dev.net.isc.upenn.edu,OU=ISC: N&T - NES - Ide
ntity and Access Management (IAM),O=University of Pennsylvania,ST
REET=3451 Walnut Street,L=Philadelphia,ST=PA,postalCode=19104,C=U
S"
*snip*
#sudo certutil -d /etc/openldap/certs -L -n intermediate-2024
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
47:20:d0:fa:85:46:1a:7e:17:a1:64:02:91:84:63:74
Signature Algorithm: PKCS #1 SHA-384 With RSA Encryption
Issuer: "CN=USERTrust RSA Certification Authority,O=The USERTRUST Net
work,L=Jersey City,ST=New Jersey,C=US"
Validity:
Not Before: Mon Oct 06 00:00:00 2014
Not After : Sat Oct 05 23:59:59 2024
Subject: "CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arb
or,ST=MI,C=US"
*snip*
gather the openssl s_client info; why are 4 certs (depth 0->3) presented instead of the expected 2 (dept h0->1)?
#openssl s_client -host localhost -port 636 -showcerts 2>local.certs >> local.certs
#grep -A1 "s:" local.certs
0 s:/C=US/postalCode=19104/ST=PA/L=Philadelphia/street=3451 Walnut Street/O=University of Pennsylvania/OU=ISC: N&T - NES - Identity and Access Management (IAM)/CN=penngroups-dev.net.isc.upenn.edu
i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
--
1 s:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
--
2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
--
3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
0 s:/C=US/postalCode=19104/ST=PA/L=Philadelphia/street=3451 Walnut Street/O=University of Pennsylvania/OU=ISC: N&T - NES - Identity and Access Management (IAM)/CN=penngroups-dev.net.isc.upenn.edu
i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
--
1 s:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
--
2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
--
3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
3 years
