openldap-bugs

openldap-bugs@openldap.org

1 participants
20967 discussions

Re: (ITS#8823) LMDB on ramdisk
by hyc＠symas.com 21 Mar '18

21 Mar '18

luca.foppiano(a)inria.fr wrote: > Full_Name: Luca Foppiano > Version: 2.4 > OS: linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (37.169.254.52) > > > Dear OpenLDAP team, > I'm using one of your component (LMDB) via a java JNDI bindings > implementation (https://github.com/deephacks/lmdbjni) and I'm having an issue > when I deploy my LMDB file on a tempfs filesystem in RAM. The ITS is for bug reports, not for help requests. Use the -technical mailing list. Closing this ITS. > > The issue do not occur when the LMDB files are stored on a "normal" filesystem. > When the data is in the tempfs ramdisk all the allocated memory ends up being in > the Dirty area (it has not been written back to the Filesytem). > > Here an example using the ramdisk: > > 7ce320000000-7cfc20000000 r--s 00000000 00:26 2459 > /ramfs/nerd/data/db/db-en/entityEmbeddings/data.mdb > Size: 104857600 kB > Rss: 1255680 kB > Pss: 1255680 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 1255680 kB <--- > Referenced: 1255680 kB > Anonymous: 0 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd sh mr mw me ms sd > > and here an example without: > > 7ca4fc000000-7cbdfc000000 r--s 00000000 fd:00 11154951 > /data/workspace/shared/nerd-data/db/db-en/entityEmbeddings/data.mdb > Size: 104857600 kB > Rss: 838124 kB > Pss: 838124 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 838124 kB <---- > Private_Dirty: 0 kB > Referenced: 764872 kB > Anonymous: 0 kB > AnonHugePages: 0 kB > ShmemPmdMapped: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd sh mr mw me ms sd > > > According to my understanding the memory is dirty when 1)there are open > transactions, 2) the data has not been written back to the filesystem > > What I don't understand is why there is a difference between filesystem and > ramdisk? > Is there any reason? The application (listed above) is not writing on the lmdb, > but just reading (using reading transaction). > > Thank you > Luca > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8823) LMDB on ramdisk
by luca.foppiano＠inria.fr 21 Mar '18

21 Mar '18

Full_Name: Luca Foppiano Version: 2.4 OS: linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (37.169.254.52) Dear OpenLDAP team, I'm using one of your component (LMDB) via a java JNDI bindings implementation (https://github.com/deephacks/lmdbjni) and I'm having an issue when I deploy my LMDB file on a tempfs filesystem in RAM. The issue do not occur when the LMDB files are stored on a "normal" filesystem. When the data is in the tempfs ramdisk all the allocated memory ends up being in the Dirty area (it has not been written back to the Filesytem). Here an example using the ramdisk: 7ce320000000-7cfc20000000 r--s 00000000 00:26 2459 /ramfs/nerd/data/db/db-en/entityEmbeddings/data.mdb Size: 104857600 kB Rss: 1255680 kB Pss: 1255680 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 1255680 kB <--- Referenced: 1255680 kB Anonymous: 0 kB AnonHugePages: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd sh mr mw me ms sd and here an example without: 7ca4fc000000-7cbdfc000000 r--s 00000000 fd:00 11154951 /data/workspace/shared/nerd-data/db/db-en/entityEmbeddings/data.mdb Size: 104857600 kB Rss: 838124 kB Pss: 838124 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 838124 kB <---- Private_Dirty: 0 kB Referenced: 764872 kB Anonymous: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd sh mr mw me ms sd According to my understanding the memory is dirty when 1)there are open transactions, 2) the data has not been written back to the filesystem What I don't understand is why there is a difference between filesystem and ramdisk? Is there any reason? The application (listed above) is not writing on the lmdb, but just reading (using reading transaction). Thank you Luca

1 0

Re: (ITS#8822) Conan package
by hyc＠symas.com 21 Mar '18

21 Mar '18

zamazan4ik(a)tut.by wrote: > Full_Name: Alexander Zaitsev > Version: None > OS: Linux > URL: > Submission from: (NULL) (128.140.241.11) > > > Hello, > Do you know about Conan(https://github.com/conan-io/conan)? > Conan is modern dependency manager for C++. And will be great if your library > will be available via package manager for other developers. > > On https://github.com/bincrafters/conan-templates) you can find example, how you > can create package for the library. > > If you have any questions, just ask :-) > > The OpenLDAP Project provides source code, not packages. Feel free to create whatever packages you want. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8822) Conan package
by zamazan4ik＠tut.by 21 Mar '18

21 Mar '18

Full_Name: Alexander Zaitsev Version: None OS: Linux URL: Submission from: (NULL) (128.140.241.11) Hello, Do you know about Conan(https://github.com/conan-io/conan)? Conan is modern dependency manager for C++. And will be great if your library will be available via package manager for other developers. On https://github.com/bincrafters/conan-templates) you can find example, how you can create package for the library. If you have any questions, just ask :-)

1 0

Re: (ITS#8821) memory leak during slap_send_ldap_result()
by hyc＠symas.com 20 Mar '18

20 Mar '18

valdemar.pavesi(a)nokia.com wrote: > Full_Name: VALDEMAR PAVESI > Version: 2.4.40 > OS: 3.10.0-514.26.2.el7.x86_64 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (131.228.48.70) > > > hello, > > during add an delete there is a memory leak. This could be ITS#8031 or ITS#8690. 2.4.40 is nearly 4 years old. Please use a newer release and report back if you're still seeing the problem. > got 1000 times memory leaked: > > > 0x00007fc8000f8f70 libc-2.17.so __malloc()+0 > 0x00007fc8017cf2b5 liblber-2.4.so ber_memalloc_x()+53 > 0x00007fc8017cf5de liblber-2.4.so ber_dupbv_x()+62 > 0x00007fc801ea1c23 slapd > 0x00007fc801ea2172 slapd attrs_dup()+66 > 0x00007fc801ea551b slapd entry_dup2()+75 > 0x00007fc7fc6f1403 syncprov-2.4.so > 0x00007fc7fc6f20c2 syncprov-2.4.so > 0x00007fc801ea91f3 slapd > 0x00007fc801ea97b0 slapd > 0x00007fc801eaa342 slapd slap_send_ldap_result()+114 > 0x00007fc801f1807d slapd bdb_modify()+1325 > 0x00007fc801f07926 slapd overlay_op_walk()+134 > 0x00007fc801f07a94 slapd > 0x00007fc801efe7bf slapd > 0x00007fc801f008eb slapd > 0x00007fc801e999a1 slapd > 0x00007fc8019e9fba libldap_r-2.4.so > 0x00007fc800caddc5 libpthread-2.17.so start_thread()+197 > 0x00007fc80017076d libc-2.17.so __clone()+109 > > +++++++++++++++++++++++++++++++++ -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8821) memory leak during slap_send_ldap_result()
by valdemar.pavesi＠nokia.com 20 Mar '18

20 Mar '18

Full_Name: VALDEMAR PAVESI Version: 2.4.40 OS: 3.10.0-514.26.2.el7.x86_64 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (131.228.48.70) hello, during add an delete there is a memory leak. fsuseradd CESNAT4 -G _nokfsuipmgroot -p Gok9zt0Fy4FZYWhW/fGHLseWjPj/Dgv1 fsuserdel -r remove CESNAT4 there is a success for both commands. +++++++++++++++++++++++++++++++++++++++ memory keep increasing... Every 5.0s: ps -p 29064 o vsz,rss Wed Mar 21 02:34:22 2018 VSZ RSS 831368 83980 +++++++++++++++++++++++++++++++++ Every 5.0s: ps -p 29064 o vsz,rss Wed Mar 21 02:36:12 2018 VSZ RSS 831368 83984 +++++++++++++++++++++++++++++++++ got 1000 times memory leaked: 0x00007fc8000f8f70 libc-2.17.so __malloc()+0 0x00007fc8017cf2b5 liblber-2.4.so ber_memalloc_x()+53 0x00007fc8017cf5de liblber-2.4.so ber_dupbv_x()+62 0x00007fc801ea1c23 slapd 0x00007fc801ea2172 slapd attrs_dup()+66 0x00007fc801ea551b slapd entry_dup2()+75 0x00007fc7fc6f1403 syncprov-2.4.so 0x00007fc7fc6f20c2 syncprov-2.4.so 0x00007fc801ea91f3 slapd 0x00007fc801ea97b0 slapd 0x00007fc801eaa342 slapd slap_send_ldap_result()+114 0x00007fc801f1807d slapd bdb_modify()+1325 0x00007fc801f07926 slapd overlay_op_walk()+134 0x00007fc801f07a94 slapd 0x00007fc801efe7bf slapd 0x00007fc801f008eb slapd 0x00007fc801e999a1 slapd 0x00007fc8019e9fba libldap_r-2.4.so 0x00007fc800caddc5 libpthread-2.17.so start_thread()+197 0x00007fc80017076d libc-2.17.so __clone()+109 +++++++++++++++++++++++++++++++++ if keep running both commands, there is one time that we will get this error: # fsuseradd CESNAT -G _nokfsuipmgroot -p Gok9zt0Fy4FZYWhW/fGHLseWjPj/Dgv1 invalid parameter: CESNAT already exists. use -g option to assign user to CESNAT # fsuserdel -r remove CESNAT Username CESNAT does not exist in LDAP. # fsuseradd CESNAT -G _nokfsuipmgroot -p Gok9zt0Fy4FZYWhW/fGHLseWjPj/Dgv1 invalid parameter: CESNAT already exists. use -g option to assign user to CESNAT # fsuserdel -r remove CESNAT Username CESNAT does not exist in LDAP. # fsuserdel -r remove CESNAT Username CESNAT does not exist in LDAP. # fsuseradd CESNAT -G _nokfsuipmgroot -p Gok9zt0Fy4FZYWhW/fGHLseWjPj/Dgv1 invalid parameter: CESNAT already exists. use -g option to assign user to CESNAT +++++++++++++++++++++++++++++++++++++++++ [root@santos-santoscmm-necc0 ~]# more /opt/aaa/ldap/conf/ldap.conf # $OpenLDAP: pkg/ldap/servers/slapd/slapd.conf,v 1.8.8.7 2001/09/27 20:00:31 kurt Exp $ # # See slapd.conf(5) for details on configuration options. # This file should NOT be world readable. # include /opt/aaa/ldap/schema/fsAll.schema #schemacheck off # Do not enable referrals until AFTER you have a working directory # service AND an understanding of referrals. #referral ldap://root.openldap.org pidfile /var/run/slapd/slapd.pid #argsfile /var/run/openldap/slapd.args # Load dynamic backend modules: # modulepath /usr/local/libexec/openldap # moduleload back_ldap.la # moduleload back_ldbm.la # moduleload back_passwd.la # moduleload back_shell.la # Define global ACLs to disable default read access. include /opt/aaa/ldap/conf/security.conf allows bind_v2 # Log level here doesn't really have any effect since the service is started with -d option loglevel 256 logfile /var/log/ldap/ldap.log # Set the max size of primary thread pool threads 20 ####################################################################### # ldbm database definitions ####################################################################### sizelimit unlimited # main database ####################################################### database bdb suffix "" rootdn "fsClusterId=ClusterRoot" rootpw {CRYPT}$6$cKyVbTD/bEiX$DOOeUCXHkmOTdlbRq0n.hGzgv7rhD2MG4.hAt8s.8Gqr73gUKw7swD2xR7zvidcfpvbkVjOkruY4BM9UprSFH/ password-hash {CRYPT} password-crypt-salt-format "$6$%.12s" checkpoint 1024 60 directory /var/lib/ldap/aaa index objectClass,entryCSN,entryUUID eq index uid,uidNumber,gidNumber,fssecAssignedGroupRoleRef,memberUid eq moduleload /usr/lib64/openldap/ppolicy.la overlay ppolicy ppolicy_default "fsFragmentId=Policy,fsFragmentId=Configurations,fsFragmentId=Security,fsClusterId=ClusterRoot" #overlay glue #cachesize 30000 TLSCertificateFile local.local security ssf=128 TLSCipherSuite HIGH:TLS_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_256_CBC_SHA:-SSLv2:-SSLv3 TLSCACertificatePath /etc/openldap/internal_certs TLSVerifyClient never include /opt/nokia/aaa/ldap/conf/sync.conf # +++++++++++++++++++++++++++++++++++++++++ regards! Valdemar

1 0

Re: (ITS#8819) LMDB seg fault with MDB_DUPSORT on -O3
by hyc＠symas.com 20 Mar '18

20 Mar '18

nic(a)nicwatson.org wrote: > That's news to me. Then I googled it. You're right. > >>From https://wiki.sei.cmu.edu/confluence/display/c/EXP36-C.+Do+not+cast+pointers… > > The C Standard, 6.3.2.3, paragraph 7 [ISO/IEC 9899:2011], states > > A pointer to an object or incomplete type may be converted to a > pointer to a different object or incomplete type. If the resulting > pointer is not correctly aligned for the referenced type, the behavior > is undefined. > > Nic > > On Tue, Mar 20, 2018 at 9:41 AM, Hallvard Breien Furuseth > <h.b.furuseth(a)usit.uio.no> wrote: >> Looks like another type aliasing problem to me. The data is accessed >> through an MDB_page* variable. This tells the compiler that the data >> is word-aligned, like struct MDB_page. Fix: Use a void/char pointer, >> don't lie to the compiler. Good catch. We once discussed padding odd-length keys to make sure the data was still word-aligned. Maybe should do that in LMDB 1.0. This particular crash is now fixed in mdb.master. I've left other derefs of *fp alone for the moment but may need to revisit that later; older ARM and SPARC would probably choke on them. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8819) LMDB seg fault with MDB_DUPSORT on -O3
by nic＠nicwatson.org 20 Mar '18

20 Mar '18

That's news to me. Then I googled it. You're right. >From https://wiki.sei.cmu.edu/confluence/display/c/EXP36-C.+Do+not+cast+pointers… The C Standard, 6.3.2.3, paragraph 7 [ISO/IEC 9899:2011], states A pointer to an object or incomplete type may be converted to a pointer to a different object or incomplete type. If the resulting pointer is not correctly aligned for the referenced type, the behavior is undefined. Nic On Tue, Mar 20, 2018 at 9:41 AM, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > Looks like another type aliasing problem to me. The data is accessed > through an MDB_page* variable. This tells the compiler that the data > is word-aligned, like struct MDB_page. Fix: Use a void/char pointer, > don't lie to the compiler. >

1 0

Re: (ITS#8819) LMDB seg fault with MDB_DUPSORT on -O3
by h.b.furuseth＠usit.uio.no 20 Mar '18

20 Mar '18

Looks like another type aliasing problem to me. The data is accessed through an MDB_page* variable. This tells the compiler that the data is word-aligned, like struct MDB_page. Fix: Use a void/char pointer, don't lie to the compiler.

1 0

Re: (ITS#8819) LMDB seg fault with MDB_DUPSORT on -O3
by nic＠nicwatson.org 20 Mar '18

20 Mar '18

--000000000000b2e0cc0567d7decb Content-Type: text/plain; charset="UTF-8" Summary of findings so far: Attached is a make script that applies the minimal set of optimizations (at least on gcc 7) that still triggers the seg fault. It fails the first time that address is executed, when NUMKEYS(fp) hits 10 or 11 (I *think* depending on the alignment of fp). It looks like the autovectorization logic is split in three parts: one unaligned non-SSE copy, an aligned copy that uses SSE, and then an unaligned non-SSE copy. It makes sense that the SSE logic path is only triggered when the size is greater than some threshold. We should coordinate filing a gcc bug report. Nic Watson On Fri, Mar 16, 2018 at 3:11 PM, <openldap-its(a)openldap.org> wrote: > > *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** > > Thanks for your report to the OpenLDAP Issue Tracking System. Your > report has been assigned the tracking number ITS#8819. > > One of our support engineers will look at your report in due course. > Note that this may take some time because our support engineers > are volunteers. They only work on OpenLDAP when they have spare > time. > > If you need to provide additional information in regards to your > issue report, you may do so by replying to this message. Note that > any mail sent to openldap-its(a)openldap.org with (ITS#8819) > in the subject will automatically be attached to the issue report. > > mailto:openldap-its@openldap.org?subject=(ITS#8819) > > You may follow the progress of this report by loading the following > URL in a web browser: > http://www.OpenLDAP.org/its/index.cgi?findid=8819 > > Please remember to retain your issue tracking number (ITS#8819) > on any further messages you send to us regarding this report. If > you don't then you'll just waste our time and yours because we > won't be able to properly track the report. > > Please note that the Issue Tracking System is not intended to > be used to seek help in the proper use of OpenLDAP Software. > Such requests will be closed. > > OpenLDAP Software is user supported. > http://www.OpenLDAP.org/support/ > > -------------- > Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. > --000000000000b2e0cc0567d7decb Content-Type: application/x-shellscript; name="make.sh" Content-Disposition: attachment; filename="make.sh" Content-Transfer-Encoding: base64 X-Attachment-Id: f_jezny59e0 TE1EQl9ESVI9fi9zcmMvbG1kYi9saWJyYXJpZXMvbGlibG1kYgoKCk5PT1BUUzE9IgotZm5vLWF1 dG8taW5jLWRlYwotZm5vLWJyYW5jaC1jb3VudC1yZWcKLWZuby1jb21iaW5lLXN0YWNrLWFkanVz dG1lbnRzCi1mbm8tY29tcGFyZS1lbGltCi1mbm8tY3Byb3AtcmVnaXN0ZXJzCi1mbm8tZGNlCi1m bm8tZGVmZXItcG9wCi1mbm8tZGVsYXllZC1icmFuY2gKLWZuby1kc2UKLWZuby1mb3J3YXJkLXBy b3BhZ2F0ZQotZm5vLWd1ZXNzLWJyYW5jaC1wcm9iYWJpbGl0eQotZm5vLWlmLWNvbnZlcnNpb24y Ci1mbm8taWYtY29udmVyc2lvbgotZm5vLWlubGluZS1mdW5jdGlvbnMtY2FsbGVkLW9uY2UKLWZu by1pcGEtcHVyZS1jb25zdAotZm5vLWlwYS1wcm9maWxlCi1mbm8taXBhLXJlZmVyZW5jZQotZm5v LW1lcmdlLWNvbnN0YW50cwotZm5vLW1vdmUtbG9vcC1pbnZhcmlhbnRzCi1mbm8tb21pdC1mcmFt ZS1wb2ludGVyCi1mbm8tcmVvcmRlci1ibG9ja3MKLWZuby1zaHJpbmstd3JhcAotZm5vLXNocmlu ay13cmFwLXNlcGFyYXRlCi1mbm8tc3BsaXQtd2lkZS10eXBlcwotZm5vLXNzYS1iYWNrcHJvcAot Zm5vLXNzYS1waGlvcHQKLWZuby10cmVlLWJpdC1jY3AKLWZuby10cmVlLWNjcAotZm5vLXRyZWUt Y29hbGVzY2UtdmFycwotZm5vLXRyZWUtY29weS1wcm9wCi1mbm8tdHJlZS1kY2UKLWZuby10cmVl LWRvbWluYXRvci1vcHRzCi1mbm8tdHJlZS1kc2UKLWZuby10cmVlLWZvcndwcm9wCi1mbm8tdHJl ZS1waGlwcm9wCi1mbm8tdHJlZS1zaW5rCi1mbm8tdHJlZS1zbHNyCi1mbm8tdHJlZS1zcmEKLWZu by10cmVlLXB0YQotZm5vLXRyZWUtdGVyCi1mbm8tdW5pdC1hdC1hLXRpbWUKLWZ0cmVlLWZyZQot ZnRyZWUtY2gKIgojIE5vdGUgbGFzdCBlbnRyeSBpcyAqbm90KiBkaXNhYmxlZAoKT1BUUzI9Jwot ZnN0cmljdC1hbGlhc2luZwonCgpybSBtZGJfY19leGUKZ2NjIC1PICROT09QVFMxICRPUFRTMiAt ZnRyZWUtdmVjdG9yaXplIC1mdmVjdC1jb3N0LW1vZGVsPXVubGltaXRlZCAtZ2dkYiAtc3RkPWMx MSAtV2FsbCAtSSAkTE1EQl9ESVIgIGxtZGJfY3Jhc2gyLmMgJExNREJfRElSL3ttZGIuYyxtaWRs LmN9IC1scHRocmVhZCAtbyBtZGJfY19leGUKcm0gLXIgZm9vLmxtZGIKbWtkaXIgLXAgZm9vLmxt ZGIK --000000000000b2e0cc0567d7decb--

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs