openldap-technical

openldap-technical@openldap.org

14 participants
9796 discussions

Re: Can't open env with required mapsize on windows .
by Howard Chu 30 Dec '15

30 Dec '15

Mohammed Muneer wrote: > > "Windows is like that." > > I know. Please keep your replies on the mailing list. > "Have no idea what you're actually doing, so it's impossible to answer. > Learn to ask better questions and provide better information." > > Sorry that I didn't sound intelligent. I was in a hurry when I wrote it. > I will now try to give you a detailed account of what I am doing. > > > "What revision of LMDB?" > > The latest hosted at https://github.com/LMDB/lmdb. "The latest" in which branch? Note that mdb.master is the main development branch and contains work-in-progress code that is probably not stable. As a non-LMDB-developer you should only be using actual tagged releases. You may occasionally want to use the mdb.RE/xx release engineering branches if they contain a fix for a specific issue that you encountered. > "Which version of Windows?" > > All are Windows 7 Ultimate x64 > > > "How much free disk space is there?" > > Machine 1 has around 26 GB free. Machine 2 has around 40 GB free. > Virtualbox guest has only has around 10 GB free. > > > "What else is your application code doing?" > > This is silly. My application was trying to open an env on windows. Here > is the code for it. > > > #define MB * 1024 * 1024 > #define GB * 1024 * 1024 * 1024 > > MDB_env *env = 0; > > bool openEnv() { > mdb_size_t mapSize = 390 MB; > MDB_dbi maxNamedDatabases = 100; > > > if (mdb_env_create(&env) != MDB_SUCCESS) > return false; > > if (mdb_env_set_mapsize(env, mapSize) != MDB_SUCCESS) > return false; > > if (mdb_env_set_maxdbs(env, maxNamedDatabases) != MDB_SUCCESS) > return false; > > int err = mdb_env_open(env, "test", 0, 0664); > > if (err != MDB_SUCCESS) { > fprintf(stderr, "Error opening env %i", err); > return false; > } > fprintf(stderr, "Success opening env %i", err); > > return true; > } > > int main() { > > if (!openEnv()) > return 1; > > } > > > > > "How do you know that only XX MB were "opened" > > Trial and error on windows showed that mdb_env_open returns success when > I give a mapSize of 390 MB on Machine 1, etc... > But on further testing, we can open 2 GB sometimes. But after a system > restart, situation is still the same. > > > "what stats are you checking? " > > Well as you can see from the above code, I am checking the return value > of mdb_env_open(). > > > > "What other differences exist between machine 1 and machine 2?" > > Not much. One being a core-i3 and the other core-i5. Both have 4 GB RAM. > Both are similiar in that none of them have > so many applications installed and all have used memory around 500 MB > > > > "What other processes are running," > > As I stated its mostly a default installation of Windows x64 on all > machines. > > > > "how much memory are they consuming on each machine?" > > Are you referring to other processes or my application. Assuming you are > talking about my application, it is just a > main function. So that might be irrelevant > > > > "Note that just because you create a 1GB memory map of a file doesn't > mean that it will actually consume 1GB of RAM. > That's not how memory mapping works." > > Never said that. Didn't even imply it. Don't know where you are getting > this. I know how memory map works. Having implemented > one in a custom hobby OS 5 years ago. Your implementation leveraging COW > along with performance and MVCC was the whole reason I was > interested in your project. > > > > "Read your Windows developer documentation, MSDN is always there for you > to read. We're not here to teach you how your OS works." > > You don't have to teach me how my OS works. Sorry if I gave you that > impression. Apparently we do. See below. > "Hangs where, exactly? Have you gotten a stack trace from such a hang? > That's the obvious first step that any application developer should know to do." > > On windows, if I dont set env_set_mapsize(env, 0) just before beginning > a write transaction, then first insert will complete successfully. On the second > write transaction while the application is running, and we try to for > eg: mdb_cursor_get(), it doesn't return. This can be reproduced with the below > stated toolchain. > > > ----------- > > For further notes, I am using official QtSDK-mingw on windows which is a > 32 bit compiler. But sometimes I can get LMDB to open the env > succesffully for 2 gb, etc. But on a system restart, the situation is > still the same. You can't address more than 2GB of memory in 32 bit code. You generally won't be able to create a 2GB mmap in a 32 bit process because other DLLs will already be occupying large chunks of the 32 bit address space. > But I downloaded an unofficial QtSDK-mingw-64 with the same result. But > when I change > mdb_size_t mapSize = 2147483648; // 2 GB > to > mdb_size_t mapSize = 2147483648LL; > > it works. > > > But setting 2147483648LL does not work (meaning does not return > MDB_SUCCESS) most of the times on mingw-32-bit That is expected. Again, you should already know how your OS works. > So the quesstion really is how to get it working when compiled on a > 32-bit compiler and executed on both 32 and 64 bit Windows? > > Note: I am not referring to vl32 branch because I don't want to have to > use more than 2GB as the size of my env file. It can't be done. > > > On Mon, Dec 28, 2015 at 2:17 PM, Howard Chu <hyc(a)symas.com > <mailto:hyc@symas.com>> wrote: > > Mohammed Muneer wrote: > > > I can env_set_mapsize(env, 1 GB) without any problems in linux. > > But on windows, it is a different story > > > Windows is like that. > > Machine 1) Windows 64 only opens with upto 10 MB (apporx) > Machine 2 Windows 64 only opens with upto 500 MB (apporx) > Machine 3) Virtual box Windows 64 running on linux works with 1 GB > > Q1) So why is this happening. Am I doing anything wrong. > > > Have no idea what you're actually doing, so it's impossible to answer. > Learn to ask better questions and provide better information. > > What revision of LMDB? Which version of Windows? How much free disk space > is there? What else is your application code doing? How do you know that > only XX MB were "opened", what stats are you checking? What other > differences exist between machine 1 and machine 2? > > What other processes are running, how much memory are they consuming on > each machine? Note that just because you create a 1GB memory map of a file > doesn't mean that it will actually consume 1GB of RAM. That's not how > memory mapping works. Read your Windows developer documentation, MSDN is > always there for you to read. We're not here to teach you how your OS works. > > Q2) And setting env_set_mapsize before calling open is not enough. I > need to set > env_set_mapsize(env, 0) just before beginning a write transaction > inorder to > insert something or otherwise the application indefinitely > hangs on the > second insert transaction. > > > Hangs where, exactly? Have you gotten a stack trace from such a hang? > That's the obvious first step that any application developer should know > to do. > > > Both the above questions concerns just Windows. Everything is fine > with linux. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 3

Re: Regarding ITS 8268 slapd-ldap quarantine
by Nikos Voutsinas 29 Dec '15

29 Dec '15

On Tuesday, 29 December 2015, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Tuesday, December 29, 2015 5:22 PM +0200 Nikos Voutsinas < > nvoutsin(a)gmail.com> wrote: > > >> >> >> Hi, >> >> >> May I ask how come and ITS 8268 never made it to software bugs? Isn't the >> case well described? >> > > OpenLDAP is a volunteer project. People work on items as they have > time/availability. If you feel an issue is urgent, you may wish to provide > a fix/patch as per the contribution guidelines, or engage support with a > company such as Symas. > > --Quanah > > > Understood, no need to clarify this. Obviously not so many from the community are using slapd-ldap quarantine. Anyway, let's say that the first step has been made and a bug report has been filed. That would have saved me some time, if someone else had done it before. Best Regards, Nikos

1 0

Re: openldap 2.4.43 keeps crashing
by Jephte Clain 29 Dec '15

29 Dec '15

Le 29/12/2015 00:55, David Hawes a écrit : > On 28 December 2015 at 01:31, Jephte Clain <jephte.clain(a)univ-reunion.fr> wrote: >> hello all, >> >> I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze >> 64bits lts >> it is built from sources with the fix from ITS#8330 >> >> Since the upgrade, it keeps crashing regularly (I would say once every 2 to >> 5 days) >> However, there isn't anything in the logs: no message, no error, no nothing >> >> slapd no longer runs, I restart it, and it runs ok until the next crash >> >> now I don't know how to debug that. >> I could downgrade to the previous version (2.4.41) which worked fine, but as >> our university is closed until 01/11, this is the perfect time to >> investigate. I still don't know if it's a bug in 2.4.43 or a >> misconfiguration for which previous versions was forgiving >> >> What are your advice to track down the cause for the crash? >> >> PS: here is a bit of context: >> one master with 2 mdb databases: one accesslog, one for the data >> two slaves (I mean complete replication, even cn=config) >> the master has crashed only once >> the slaves keep crashing regularly >> the slaves are complete replicas of the master, and I wonder if it could be >> a source of problems: I'll give details in another thread > Do you have accesslog purge configured? > > I ran into this on an accesslog purge with 2.4.43: > > http://www.openldap.org/its/index.cgi/Software%20Bugs?id=8336;selectid=8336 > > If you start slapd with '-d 1', you should be able to see if it is the issue. > > The fix referenced in that ITS worked for me. hello, thanks for the response. no, purge is not configured, but this is a nice suggestion... without purge, I guess the accesslog database would keep growing forever :-( thanks again, regards, Jephté CLAIN -- *Jephté CLAIN | Développeur, Intégrateur d'applications* Service Systèmes d'Information Direction des Systèmes d'Information <http://dsi.univ-reunion.fr> Tél: +262 262 93 86 31 <tel:+262262938631> || Gsm: +262 692 29 58 24 <tel:+262692295824> www.univ-reunion.fr <http://www.univ-reunion.fr> || Facebook <http://www.facebook.com/pages/Universit%C3%A9-de-La-R%C3%A9union-OFFICIEL/1…> || Twitter <http://twitter.com/univ_reunion>

2 3

Regarding ITS 8268 slapd-ldap quarantine
by Nikos Voutsinas 29 Dec '15

29 Dec '15

Hi, May I ask how come and ITS 8268 never made it to software bugs? Isn't the case well described? Anyway, while we have already worked around the problem, actually by disabling the feature, I was curious if the dev team or anyone else has tested slapd-ldap quarantine with a recent Openldap version ( we have noticed similar behavior both with 2.4.41 and 2.4.42) http://www.openldap.org/its/index.cgi/Incoming?id=8268 Best Regards, Nikos

2 1

what databases are to be replicated for a slave?
by Jephte Clain 29 Dec '15

29 Dec '15

hello, I've had problems with an upgrade to openldap 2.4.43, and I reviewed my configuration to check for errors I have two slaves which replicate all databases from the master, including cn=config. My reasoning is in case of a severe crash, I'd like one of the slaves to be able to become the new master. To allow such a scenario, each database in the master have a syncrepl directive to replicate from itself. It obviously have no effect on the master, but the slave that replicate this configuration get the right directives to replicate from the master. so far so good... now this is my question: should/could the accesslog configured this way? or should the accesslog be strictly local to a server? I mean, should I remove the syncrepl directive from the accesslog database? This configuration have run without errors until 2.4.41, but I got a weird error about accesslog corruption when I upgraded to 2.4.43 (sorry I didn't write the error down), hence my question Thanks in advance for any response. best regards, Jephté CLAIN PS: I hope it's clear enough. English is not my native language -- *Jephté CLAIN | Développeur, Intégrateur d'applications* Service Systèmes d'Information Direction des Systèmes d'Information <http://dsi.univ-reunion.fr> Tél: +262 262 93 86 31 <tel:+262262938631> || Gsm: +262 692 29 58 24 <tel:+262692295824> www.univ-reunion.fr <http://www.univ-reunion.fr> || Facebook <http://www.facebook.com/pages/Universit%C3%A9-de-La-R%C3%A9union-OFFICIEL/1…> || Twitter <http://twitter.com/univ_reunion>

2 5

openldap 2.4.43 keeps crashing
by Jephte Clain 29 Dec '15

29 Dec '15

hello all, I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze 64bits lts it is built from sources with the fix from ITS#8330 Since the upgrade, it keeps crashing regularly (I would say once every 2 to 5 days) However, there isn't anything in the logs: no message, no error, no nothing slapd no longer runs, I restart it, and it runs ok until the next crash now I don't know how to debug that. I could downgrade to the previous version (2.4.41) which worked fine, but as our university is closed until 01/11, this is the perfect time to investigate. I still don't know if it's a bug in 2.4.43 or a misconfiguration for which previous versions was forgiving What are your advice to track down the cause for the crash? PS: here is a bit of context: one master with 2 mdb databases: one accesslog, one for the data two slaves (I mean complete replication, even cn=config) the master has crashed only once the slaves keep crashing regularly the slaves are complete replicas of the master, and I wonder if it could be a source of problems: I'll give details in another thread -- *Jephté CLAIN | Développeur, Intégrateur d'applications* Service Systèmes d'Information Direction des Systèmes d'Information <http://dsi.univ-reunion.fr> Tél: +262 262 93 86 31 <tel:+262262938631> || Gsm: +262 692 29 58 24 <tel:+262692295824> www.univ-reunion.fr <http://www.univ-reunion.fr> || Facebook <http://www.facebook.com/pages/Universit%C3%A9-de-La-R%C3%A9union-OFFICIEL/1…> || Twitter <http://twitter.com/univ_reunion>

4 6

RE: lmdb Handle Leak with VL32 on Windows
by Jeremiah Morrill 28 Dec '15

28 Dec '15

Submitted (seems like #8342). Apologies for not doing this before. I will go through ITS with reports in the future. -Jer From: Howard Chu<mailto:hyc@symas.com> Sent: Monday, December 28, 2015 12:12 AM To: Jeremiah Morrill<mailto:Jeremiah.Morrill@econnect.tv>; openldap-technical(a)openldap.org<mailto:openldap-technical@openldap.org> Subject: Re: lmdb Handle Leak with VL32 on Windows Jeremiah Morrill wrote: > I noticed what I believe may be a handle leak against the latest (commit > 7b9928c) on github, with the MDB_VL32 on Windows. After some poking around, > it seems to be the file mapping handle, env->me_fmh. I threw it to a > CloseHandle in the env_close0(…) and it seemed to resolve it. Please submit this to the ITS. > > Here’s a quick diff: > > diff --git a/src/lmdb.c b/src/mdb.c > > index a564d40..190c860 100644 > > --- a/src/mdb.c > > +++ b/src/mdb.c > > @@ -5350,6 +5350,7 @@ mdb_env_close0(MDB_env *env, int excl) > > } > > #ifdef MDB_VL32 > > #ifdef _WIN32 > > + if (env->me_fmh) CloseHandle(env->me_fmh); > > if (env->me_rpmutex) CloseHandle(env->me_rpmutex); > > #else > > pthread_mutex_destroy(&env->me_rpmutex); > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Can't open env with required mapsize on windows .
by Mohammed Muneer 28 Dec '15

28 Dec '15

I can env_set_mapsize(env, 1 GB) without any problems in linux. But on windows, it is a different story Machine 1) Windows 64 only opens with upto 10 MB (apporx) Machine 2 Windows 64 only opens with upto 500 MB (apporx) Machine 3) Virtual box Windows 64 running on linux works with 1 GB Q1) So why is this happening. Am I doing anything wrong. Q2) And setting env_set_mapsize before calling open is not enough. I need to set env_set_mapsize(env, 0) just before beginning a write transaction inorder to insert something or otherwise the application indefinitely hangs on the second insert transaction. Both the above questions concerns just Windows. Everything is fine with linux.

2 2

lmdb Handle Leak with VL32 on Windows
by Jeremiah Morrill 28 Dec '15

28 Dec '15

I noticed what I believe may be a handle leak against the latest (commit 7b9928c) on github, with the MDB_VL32 on Windows. After some poking around, it seems to be the file mapping handle, env->me_fmh. I threw it to a CloseHandle in the env_close0(…) and it seemed to resolve it. Here’s a quick diff: diff --git a/src/lmdb.c b/src/mdb.c index a564d40..190c860 100644 --- a/src/mdb.c +++ b/src/mdb.c @@ -5350,6 +5350,7 @@ mdb_env_close0(MDB_env *env, int excl) } #ifdef MDB_VL32 #ifdef _WIN32 + if (env->me_fmh) CloseHandle(env->me_fmh); if (env->me_rpmutex) CloseHandle(env->me_rpmutex); #else pthread_mutex_destroy(&env->me_rpmutex);

2 1

Authenticating clients
by bluethundr＠gmail.com 25 Dec '15

25 Dec '15

Hey guys, I've recently disabled anonymous binds and required TLS on my OpenLDAP sever in order to log in. Back when I was testing and anonymous binds were enabled and TLS was off I could get LDAP accounts to login to the servers. And using a simple tool like "authconfig-tui" under CentOS was all I needed to use to set that up. Where can I find the docs to configure a client server to allow LDAP account logins with anonymous binds disabled and TLS required on the OpenLDAP server? I need to do this for both Red Hat/CentOS machines and Ubuntu clients as well. I've been googling and haven't found anything helpful for both types. Thanks, Tim Sent from my iPhone

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical