openldap-technical

openldap-technical@openldap.org

9851 discussions

lmdb as a Lua database
by Russell Haley 04 Oct '17

04 Oct '17

Hi, So I'm using LMDB to create a simple 'database' system for lua that will allow me to directly serialize and store lua table data. So far I have used the wonderful lmdb wrapper lightningmdb and created a simple API to hide some of the mechanisms. Honestly, your API is so well thought out, I'm doing a disservice by removing a lot of the powerful features. However, I press ahead and beg forgiveness. :) I have two questions: 1) There is a simplified vs cursor API. I am starting to suspect that the simplified functions are just basic wrappers around the cursor API? I ask because if this is the case, I could just remove my support for the simplified API and only have to support one set of functions. 2) The key lookup is blazing fast (so awesome, this thing is going to smoke!). I wanted to create a way of indexing the data elements within the database by allowing users to create lua functions that parse the lua table values. --a basic table that would be serialized into a lmdb database called t1 t1 = { 1={col_a="test",col_b="item1"}, 2={col_a="not-a-test",col_b="item2"}, 3={col_a="test",col_b="item3"} } --The function that would be called per-row to create an index. local function(k,v) return v.col_a, k end The idea would be to use the dupsort feature to create an 'index' database with data structured as: not-a-test, 2 test, 1 test, 3

1 1

Can't get lmdb to build on FreeBSD 12
by Russell Haley 04 Oct '17

04 Oct '17

Hi, Sorry for the silly question, but I've stumped myself and my C is weak. Normally lmdb builds fine without issue on most of the platforms I try (Debian Jessie, FreeBSD 10.3 & 12), but it's stopped compiling on my TrueOS box, which is FreeBSD 12. I've tried gcc and clang 4.0 and 3.9. ------------------------ russellh@prescott:~/Git/lmdb/libraries/liblmdb% make cc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -Wuninitialized -v mdb_stat.c -o mdb_stat FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Target: x86_64-unknown-freebsd12.0 Thread model: posix InstalledDir: /usr/bin "/usr/bin/cc" -cc1 -triple x86_64-unknown-freebsd12.0 -emit-obj -disable-free -main-file-name mdb_stat.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -v -dwarf-column-info -debug-info-kind=standalone -dwarf-version=2 -debugger-tuning=gdb -resource-dir /usr/bin/../lib/clang/4.0.0 -O2 -W -Wall -Wno-unused-parameter -Wbad-function-cast -Wuninitialized -fdebug-compilation-dir /usr/home/russellh/Git/lmdb/libraries/liblmdb -ferror-limit 19 -fmessage-length 237 -pthread -fobjc-runtime=gnustep -fdiagnostics-show-option -vectorize-loops -vectorize-slp -o /tmp/mdb_stat-852384.o -x c mdb_stat.c clang -cc1 version 4.0.0 based upon LLVM 4.0.0 default target x86_64-unknown-freebsd12.0 #include "..." search starts here: #include <...> search starts here: /usr/bin/../lib/clang/4.0.0/include /usr/include End of search list. "/usr/bin/ld" --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o mdb_stat /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib /tmp/mdb_stat-852384.o -lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o /tmp/mdb_stat-852384.o: In function `main': /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:104: undefined reference to `mdb_env_create' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:106: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:111: undefined reference to `mdb_env_set_maxdbs' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:114: undefined reference to `mdb_env_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:116: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:121: undefined reference to `mdb_env_stat' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:122: undefined reference to `mdb_env_info' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:136: undefined reference to `mdb_reader_list' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:139: undefined reference to `mdb_reader_check' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:141: undefined reference to `mdb_reader_list' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:147: undefined reference to `mdb_txn_begin' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:149: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:257: undefined reference to `mdb_env_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:160: undefined reference to `mdb_cursor_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:162: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:165: undefined reference to `mdb_stat' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:171: undefined reference to `mdb_cursor_get' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:171: undefined reference to `mdb_cursor_get' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:199: undefined reference to `mdb_cursor_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:203: undefined reference to `mdb_dbi_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:205: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:209: undefined reference to `mdb_stat' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:(.text+0x5d6): undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:255: undefined reference to `mdb_txn_abort' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:221: undefined reference to `mdb_cursor_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:223: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:226: undefined reference to `mdb_cursor_get' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:234: undefined reference to `mdb_dbi_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:239: undefined reference to `mdb_stat' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:245: undefined reference to `mdb_dbi_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:226: undefined reference to `mdb_cursor_get' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:247: undefined reference to `mdb_cursor_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:253: undefined reference to `mdb_dbi_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:241: undefined reference to `mdb_strerror' cc: error: linker command failed with exit code 1 (use -v to see invocation) *** Error code 1 Stop. make: stopped in /usr/home/russellh/Git/lmdb/libraries/liblmdb ---------------------------------- The full output is here with uname, git status and compiler info: http://termbin.com/io9d Any help would be grand. Thanks, Russ

1 0

Re: Initialize ldap with mdb
by Howard Chu 04 Oct '17

04 Oct '17

Quanah Gibson-Mount wrote: > --On Tuesday, October 03, 2017 11:47 AM -0700 rammohan ganapavarapu > <rammohanganap(a)gmail.com> wrote: > >> >> Ok, but are there any casses where DB size can grow bigger then RAM and >> what will happen in that case? > > Same thing that happens in any such case.. it'll start swapping. LMDB never uses swap. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

Re: Initialize ldap with mdb
by Quanah Gibson-Mount 04 Oct '17

04 Oct '17

--On Wednesday, October 04, 2017 2:18 AM +0200 Michael Ströder <michael(a)stroeder.com> wrote: > rammohan ganapavarapu wrote: >> You mean we can resize db any time? > > You can increase but *not decrease* maxsize later. I've been able to decrease it just fine, as long as it wasn't below the actual size of the DB. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: Initialize ldap with mdb
by rammohan ganapavarapu 04 Oct '17

04 Oct '17

You mean we can resize db any time? Wouldn't it be nice to have no max limit and grow db dynamically? On Oct 3, 2017 4:05 PM, "Quanah Gibson-Mount" <quanah(a)symas.com> wrote: --On Tuesday, October 03, 2017 5:04 PM -0700 rammohan ganapavarapu < rammohanganap(a)gmail.com> wrote: > Ok, i am just testing it, i will follow those recommendations, but this > is what i did as work around: > > > So i have updated the olcDbMaxSize to below value and now i am able to do > ldapadd, but i was in the impression that once u set the maxsize at the > time of ldap initialization you can't update that, right? > Incorrect. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

2 1

Re: Initialize ldap with mdb
by Quanah Gibson-Mount 04 Oct '17

04 Oct '17

--On Tuesday, October 03, 2017 5:12 PM -0700 rammohan ganapavarapu <rammohanganap(a)gmail.com> wrote: > > You mean we can resize db any time? Wouldn't it be nice to have no max > limit and grow db dynamically? MDB needs a max size at startup, so you have to define one. Generally, it's recommended to set a large value. I wrote a script once to resize MDB upwards as it got close (20%) to its maxsize. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: Initialize ldap with mdb
by Quanah Gibson-Mount 04 Oct '17

04 Oct '17

--On Tuesday, October 03, 2017 5:04 PM -0700 rammohan ganapavarapu <rammohanganap(a)gmail.com> wrote: > > Ok, i am just testing it, i will follow those recommendations, but this > is what i did as work around: > > > So i have updated the olcDbMaxSize to below value and now i am able to do > ldapadd, but i was in the impression that once u set the maxsize at the > time of ldap initialization you can't update that, right? Incorrect. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: Initialize ldap with mdb
by rammohan ganapavarapu 04 Oct '17

04 Oct '17

Ok, i am just testing it, i will follow those recommendations, but this is what i did as work around: So i have updated the olcDbMaxSize to below value and now i am able to do ldapadd, but i was in the impression that once u set the maxsize at the time of ldap initialization you can't update that, right? echo "dn: olcDatabase={1}mdb,cn=config changetype: modify replace: olcDbMaxSize olcDbMaxSize: 21474836480" |ldapmodify -x -D "cn=config" -H ldap://localhost:10389 -w So the above operation will resize the db? Thanks, Ram On Tue, Oct 3, 2017 at 4:01 PM, Quanah Gibson-Mount <quanah(a)symas.com> wrote: > --On Tuesday, October 03, 2017 4:53 PM -0700 rammohan ganapavarapu < > rammohanganap(a)gmail.com> wrote: > > >> Ok, this is what i see in ldap logs, this could be because of >> maxsize? olcDbMaxSize: 102400 >> >> >> mdb_idl_insert_keys: c_put id failed: MDB_MAP_FULL: Environment mapsize >> limit reached (-30792) >> > > Yes. Again, as I said before, you need to set a *large* maxsize. Like > the one I recommended previously. > > > --Quanah > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> > >

1 0

Re: Initialize ldap with mdb
by Quanah Gibson-Mount 04 Oct '17

04 Oct '17

--On Tuesday, October 03, 2017 4:53 PM -0700 rammohan ganapavarapu <rammohanganap(a)gmail.com> wrote: > > Ok, this is what i see in ldap logs, this could be because of > maxsize? olcDbMaxSize: 102400 > > > mdb_idl_insert_keys: c_put id failed: MDB_MAP_FULL: Environment mapsize > limit reached (-30792) Yes. Again, as I said before, you need to set a *large* maxsize. Like the one I recommended previously. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: Initialize ldap with mdb
by rammohan ganapavarapu 04 Oct '17

04 Oct '17

There are two mdb files {data,lock}.mdb both owned by processes user. On Tue, Oct 3, 2017 at 3:41 PM, Quanah Gibson-Mount <quanah(a)symas.com> wrote: > --On Tuesday, October 03, 2017 3:53 PM -0700 rammohan ganapavarapu < > rammohanganap(a)gmail.com> wrote: > > and now i am trying to add below and getting index generation failed >> error, db dir and slapd.d owned by slapd processes user. >> > > What about the *.mdb files themselves? > > > --Quanah > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> > >

1 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical