openldap-technical

openldap-technical@openldap.org

5 participants
9899 discussions

LMDB write/read simultaneously over different threads
by Vasilis Lasdas 29 Aug '14

29 Aug '14

Hi all, I am a new user to LMDB. Please forgive me for my ignorance or mistakes you might encounter. However, I have spent quite some time trying to resolve my issues but without a success. That's why I am writing to you asking for your help to shade some light. To the point: 1) I am having a database where there is only one writer and a few readers(e.g. 4 readers). 2) The database-writer lives on different thread than the database-reader(s). 3) The writer-thread keeps on running/writing all the time to the database. 4) At random intervals readers from different threads will request to extract data from the database. The problem that occurs is the following: If the writer is active( during writing), If I try to fetch any data from the db using the reader, it fails, with error MDB_NOTFOUND. As a remark here, I make sure that the reader-thread is launched AFTER the writer thread to ensure that the data exist in the db. In order to simulate this kind of behaviour(write-read) over different threads, I have written a small test application, that tries to depict my case/usecase. In main() I simply launch two threads/functions, the writer-thread and after some time, the reader thread. The common enviroment is constructed() in main and passed as parameter to the auxilliary functions. If the reader starts reading while writing is not finished, then the application crashes or returns MDB_NOTFOUND. This contradicts with the capabilities of the library, that is, parallel write/read from different threads. If the writer has finished, then the reader works properly. Some general questions. i) How many times do i need to open the database? Currently I open the (unique) database twice: one in the writer and one in the reader. The doc mentions that it has to be opened only one time. Even in that case, my app does not work. ii) What is the practical usage of mdb_txn_reset() and mdb_txn_renew()? Does it need to be used in my case? iii) Since I am only ADDING data in the db and only READING them, what kind of optimizations can be applied? iv) Regarding for-loops, what is a proper way of using the trasnactions and commit/abort/reset/etc of transactions? Please any helpfull information is highly apprecited or if someone can add some examples with different threads that would be awesome. #include <iostream> #include <thread> #include <assert.h> #include <string> #include "lmdb.h" #include <unistd.h> using namespace std; void writeLMDB( MDB_env * env, size_t nrElems ) { cout << "BEGIN WRITING." << endl; MDB_txn *txn; MDB_dbi dbi; MDB_stat status; // create transaction assert( MDB_SUCCESS == mdb_txn_begin( env, NULL, 0, &txn ) ); // open db assert( MDB_SUCCESS == mdb_open( txn, NULL, MDB_DUPSORT , &dbi ) ); // this commit-transaction is needed since we are creating the db for the first time assert( MDB_SUCCESS == mdb_txn_commit( txn ) ); MDB_val key, data; // begin (again a new transaction) assert( MDB_SUCCESS == mdb_txn_begin( env, NULL, 0, &txn ) ); // keep on writing elements in the db for( size_t i = 0; i < nrElems ; ++i ) { key.mv_size = sizeof( size_t ); key.mv_data = &i; string value = "test"; data.mv_size = value.size(); data.mv_data = (void*)value.data(); cout << "WRITER-KEY:" << *(size_t*)key.mv_data << " VALUE:" ; cout.write( (char*)data.mv_data, data.mv_size ); cout << endl; assert( MDB_SUCCESS == mdb_put( txn, dbi, &key, &data, 0 ) ); } assert( MDB_SUCCESS == mdb_txn_commit( txn ) ); assert( MDB_SUCCESS == mdb_env_stat( env, &status ) ); mdb_close( env, dbi ); cout << "FINISHED WRITING." << endl; } void readLMDB( MDB_env * _env, size_t start, size_t end ) { cout << "BEGIN READING.." << endl; MDB_dbi dbi; MDB_txn *txn = NULL; // lmdb operations assert( MDB_SUCCESS == mdb_txn_begin( _env, NULL, MDB_RDONLY, &txn ) ); assert( MDB_SUCCESS == mdb_open( txn, NULL, 0, &dbi ) ); // key+data MDB_val key, data; for ( size_t i = start; i <= end ; i++ ) { key.mv_size = sizeof( size_t ); key.mv_data = &i; int res = mdb_get( txn, dbi , &key, &data ); //cout << "READER-KEY:" << *(size_t*)key.mv_data << " VALUE:" ; cout.write( (char*)data.mv_data, data.mv_size ); cout << endl; if( res != 0 ) cout << "error fetching data" << endl; } // close db mdb_close( _env, dbi ); cout << "FINISHED READING.." << endl; } int main(int argc, char *argv[] ) { // db location(folder must exist) string dbLocation = "/work/vasilis/databases/lmdbtest/"; size_t maxdbSize = 10485760; system( "rm -rf /work/vasilis/databases/lmdbtest/*" ); //create enviroment:ONE enviroment only for one process MDB_env * _env; assert( MDB_SUCCESS == ( mdb_env_create( &_env ) ) ); //set the size of the database assert( MDB_SUCCESS == mdb_env_set_mapsize( _env, maxdbSize ) ); // open the enviroment: ready to be used by everyone else assert( MDB_SUCCESS == mdb_env_open( _env, dbLocation.c_str(), 0 , 0664 ) ); // writer-thread: write 1000 elements in the database thread writethread( writeLMDB, _env, 1000 ); usleep( 1000 * 100 ); // read from - to thread readthread( readLMDB, _env, 0, 20 ); // wait for threads to finish writethread.join(); readthread.join(); cout << "THREADS COMPLETED " << endl; // close the enviroment mdb_env_close( _env ); return 0; } regards, Vasilis

1 0

Passwords, Hashing, and Binds
by Bram Cymet 29 Aug '14

29 Aug '14

Hi, I am storing users passwords in a userPassword attribute. When the passwords are hashed with MD5 I can bind as the user just fine. If I hash the password with sha-256 I get invalid credentials. Is there something I have to change in my client? Is there something I have to change on the server? Is binding a user with a password stored with sha-256 (or at least something better then md5) even possible? Thanks, -- Bram Cymet Software Developer Canadian Bank Note Co. Ltd. 613-608-9752

4 4

keepalive parameter setting in openldap for consumer doesnot work
by pramod kulkarni 29 Aug '14

29 Aug '14

Hi, I unplugged network cable of provider,added new users but it never synchronizes. with the consumer.Consumer never sends the keepalive signals I am using 2.4.39 version of Openldap on windows machine this is my consumer setting olcSyncrepl: {0}rid=002 provider=ldap://ipaddress:389 bindmethod=sasl timeou t=0 network-timeout=0 saslmech=external keepalive=1:1:10 starttls=critical tl s_cert="CN=Consumer3, O=CRAP" tls_cacert="CN=TestCA, O=CRAP" tls_reqcert=demand tls_cipher_suite=HIGH:MEDIUM:-SSLv2 filter="(objectclass=*)" searchbase="dc= crap,dc=com" scope=sub attrs="*,+" schemachecking=off type=refreshAndPersis t retry="60 +" if I make fresh setup of consumer & provider,it works & everything gets replicated properly. In the earlier version of openldap it was based on system setting like I used to get keepalive every 2 hours. What is the minimum value to be set for keepalive parameter in syncrepl? Can you please tell me do I need to make any changes on consumer side or provider ?

3 2

3rd try : Bug ? ldap_start_tls_s successful even after removing ca certificate from ca cert dir
by SOMA SEKHAR 28 Aug '14

28 Aug '14

On Sat, Aug 23, 2014 at 1:08 PM, SOMA SEKHAR <somasekhar44(a)gmail.com> wrote: > link to question on stackoverflow > <http://stackoverflow.com/questions/25457034/starttls-succesful-even-after-d…> > > > I'm having trouble verifying the correct behavior of my software. Here are > the steps I am performing to verify correct operation: > > 1. I have sample code that uses openldap library and doing a start tls > to a ldap server. > 2. I have set the global option for ca cert directory and tlx context > for the first time. > 3. After that I did ldap init and ldap start tls to a server. This is > succesful as expected. > 4. I did an ldap_unbind_s > 5. I deleted the CA cert that signed the ldap server's certificate > from the ca cert directory of the client. > 6. Again did ldap_init and ldap_start_tls_s . > 7. I expected this call to fail , as I have removed the ca cert. But > what I observe is that , server sends the certificate but start_tls is > returning success. > > I am using openldap 2.4 with libssl.0.9.8 > > LDAP *ld;int desired_version=3; > if ((ld = ldap_init(<hostname>, <server_port>)) == NULL ) { > printf("ldap_init failed\n"); > exit(0);} > > ldap_set_option(ld, LDAP_OPT_PROTOCOL_VERSION, &desired_version); > ldap_set_option(NULL, LDAP_OPT_X_TLS_CTX, NULL); > ldap_set_option(NULL, LDAP_OPT_X_TLS_CACERTDIR,"<ca dir>"); > if(ldap_start_tls_s(ld, NULL, NULL) != LDAP_SUCCESS){ > printf("start tls failed.\n"); > exit(0);} > ...... <do bind and search>... > > ldap_unbind_s(ld); ... > // DELETE the CA certificate from the ca dir. // Try to do start tls again > if ((ld = ldap_init(hostname, server_port)) == NULL ) { > printf("ldap_init failed , after deleting CA\n"); > exit(0);} > // This goes fine even after deleting the CAif (ldap_start_tls_s(ld, NULL, NULL) != LDAP_SUCCESS){ > printf("start tls failed after deleting CA.\n"); > exit(0);} > > > -- > Thanks&Regards, > SomaSekhar. > > -- Thanks&Regards, SomaSekhar.

3 5

best practices WRT resizing a MDB backend?
by Brian Reichert 28 Aug '14

28 Aug '14

Other than this thread: http://t23307.network-openldap-technical.opennetworks.info/lmdb-growing-the… I don't see a discussion of changing the 'maxsize' setting after a LMDB database has been generated. This thread includes this response about growing the database: http://www.openldap.org/lists/openldap-technical/201402/msg00302.html On Windows, Linux, and FreeBSD, there's no problem increasing the mapsize and preserving the existing data. (I'm making a wild assumption that 'mapsize' is a typo, and 'maxsize' was intended.) Can 'maxsize' ever be reduced after the fact? If so, is their guidance as to how much it can change (perhaps based on mdb_stat)? The problem I'm trying to solve: For my $job, we provide OpenLDAP-backed clustered appliances to customers. The hardware doesn't vary, but the size of individual customers' databases. - Our strategy for adding members to the cluster involves managing backups (compressed tarballs). Our prior use of the now-ancient bdb backend let these backups be lightweight things to manage for smaller customers, and larger customers would take the hit for having big databases. - Also, upgrading appliances means importing data from the customers' bdb-based server. My naive use of the LDMB backend has me assume the worst case, and now everyone is equally punished for having a 'big' (albeit sparse) database. My hope was to, given awareness of either the data in an LDIF extract, or data about the legacy bdb database itself, we could make a more conservative guess as to a reasonable size for the mdb backend. Has anyone written up some strategies on these topics, or in the position to provide any recommendation? -- Brian Reichert <reichert(a)numachi.com> BSD admin/developer at large

5 11

LMDB: Compare function implementation
by Venkat Murty 28 Aug '14

28 Aug '14

Hi all, I am inserting a non-zero length key into a database where I provide a compare function. But I am seeing that MDB_val->mv_size = 0 in the compare function. Is that possible? More details (LMDB 0.9.11 Release): I have a database where I want to delete all elements that have the same prefix. void eraseElements (KeyPrefix prefix, MDB_txn *txn, MDB_dbi database) { MDB_val k; k.mv_data = &prefix; k.mv_size = sizeof (KeyPrefix); MDB_val v; MDB_cursor *cur; mdb_cursor_open (txn, database, &cur); while (! mdb_cursor_get (cur, &k, &v, MDB_SET_RANGE)) { if (memcmp (&prefix, k.mv_data, sizeof (KeyPrefix)) != 0) break; mdb_cursor_del (cur, 0); k.mv_data = &prefix; k.mv_size = sizeof (KeyPrefix); } mdb_cursor_close (cur); } Relevant stack trace DB::compare (a, b) and b->mv_size == 0 mdb_cursor_set(mc, key, data, op=MDB_SET_RANGE, exactp=0x0) at mdb.c:5348 mdb_cursor_get(mc,key,data,op=MDB_SET_RANGE) at mdb.c:5655 eraseElements(..)

3 2

OpenLdap Access Control
by Mehmet Sağdıç 28 Aug '14

28 Aug '14

Hello, I installed OpenLDAP server on Ubuntu 12.04. Phpldapadmin using the interface groups and users have added. Connection with the ftp server on the LDAP user did running. I want to limitation of the user on openldap. I selected one group of users are able to access privileges such as read or write. OlcAccess tried to do in the checklist user restrictions, but did not. Please help

1 0

From: Suneet Shah
by Suneet Shah 27 Aug '14

27 Aug '14

Hi openldap http://anon.gooffcampus.com/love.php?ugwun3106ehvfty suneetshah2000(a)gmail.com

1 0

Bug ? ldap_start_tls_s successful even after removing ca certificate from ca cert dir
by SOMA SEKHAR 27 Aug '14

27 Aug '14

link to question on stackoverflow <http://stackoverflow.com/questions/25457034/starttls-succesful-even-after-d…> I'm having trouble verifying the correct behavior of my software. Here are the steps I am performing to verify correct operation: 1. I have sample code that uses openldap library and doing a start tls to a ldap server. 2. I have set the global option for ca cert directory and tlx context for the first time. 3. After that I did ldap init and ldap start tls to a server. This is succesful as expected. 4. I did an ldap_unbind_s 5. I deleted the CA cert that signed the ldap server's certificate from the ca cert directory of the client. 6. Again did ldap_init and ldap_start_tls_s . 7. I expected this call to fail , as I have removed the ca cert. But what I observe is that , server sends the certificate but start_tls is returning success. I am using openldap 2.4 with libssl.0.9.8 LDAP *ld;int desired_version=3; if ((ld = ldap_init(<hostname>, <server_port>)) == NULL ) { printf("ldap_init failed\n"); exit(0);} ldap_set_option(ld, LDAP_OPT_PROTOCOL_VERSION, &desired_version); ldap_set_option(NULL, LDAP_OPT_X_TLS_CTX, NULL); ldap_set_option(NULL, LDAP_OPT_X_TLS_CACERTDIR,"<ca dir>"); if(ldap_start_tls_s(ld, NULL, NULL) != LDAP_SUCCESS){ printf("start tls failed.\n"); exit(0);} ...... <do bind and search>... ldap_unbind_s(ld); ... // DELETE the CA certificate from the ca dir. // Try to do start tls again if ((ld = ldap_init(hostname, server_port)) == NULL ) { printf("ldap_init failed , after deleting CA\n"); exit(0);} // This goes fine even after deleting the CAif (ldap_start_tls_s(ld, NULL, NULL) != LDAP_SUCCESS){ printf("start tls failed after deleting CA.\n"); exit(0);} -- Thanks&Regards, SomaSekhar.

1 1

ldapi:/// without TLS; ldap:// with TLS?
by Tom 26 Aug '14

26 Aug '14

I'm running OpenLDAP 2.4 on CentOS. I'm trying to set it up so clients can use the ldapi:/// socket without TLS, but any clients using ldap:// must use TLS. I believe that the relevant olc variables are olcLocalSSF and olcSecurity. I can't get it to work - either TLS is required no matter which URI I use, or clients can connect without TLS at all. According to the docs, if I set olcLocalSSF to 128, and olcSecurity to ssf=128, it should work, but it's not. I can only connect without TLS if I delete the olcSecurity attribute, which allows anyone to connect without TLS. What am I dong wrong?

2 3

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical