openldap-technical

openldap-technical@openldap.org

7 participants
9864 discussions

Re: lmdb as a Lua database
by Howard Chu 04 Oct '17

04 Oct '17

Russell Haley wrote: > Hi, > > So I'm using LMDB to create a simple 'database' system for lua that > will allow me to directly serialize and store lua table data. So far I > have used the wonderful lmdb wrapper lightningmdb and created a simple > API to hide some of the mechanisms. > > Honestly, your API is so well thought out, I'm doing a disservice by > removing a lot of the powerful features. However, I press ahead and > beg forgiveness. :) > > I have two questions: > > 1) There is a simplified vs cursor API. I am starting to suspect that > the simplified functions are just basic wrappers around the cursor > API? I ask because if this is the case, I could just remove my support > for the simplified API and only have to support one set of functions. Yes, everything uses cursors internally. > 2) The key lookup is blazing fast (so awesome, this thing is going to > smoke!). I wanted to create a way of indexing the data elements within > the database by allowing users to create lua functions that parse the > lua table values. > --a basic table that would be serialized into a lmdb database called t1 > t1 = { > 1={col_a="test",col_b="item1"}, > 2={col_a="not-a-test",col_b="item2"}, > 3={col_a="test",col_b="item3"} > } > > --The function that would be called per-row to create an index. > local function(k,v) > > return v.col_a, k > end > > The idea would be to use the dupsort feature to create an 'index' > database with data structured as: > > not-a-test, 2 > test, 1 > test, 3 Sure, that's generally what dupsort is for. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: Initialize ldap with mdb
by Howard Chu 04 Oct '17

04 Oct '17

rammohan ganapavarapu wrote: > Howard, > > If it doesnt swap, what will it do in case of db size is larger than actual > RAM? also how does it manage the physical RAM? meaning what % system RAM does > it use? It doesn't manage physical RAM at all, the OS does. There is no fixed % of system RAM that it will use, the OS will let it use whatever is available. > is it configurable ? No. > will it case any OOM issues? No. > i am trying to > migrate from hdb to mdb in prod so trying to understand mdb. How does it work > in case of reads and writes? > > Thanks, > Ram > > On Tue, Oct 3, 2017 at 4:15 PM, Howard Chu <hyc(a)symas.com > <mailto:hyc@symas.com>> wrote: > > Quanah Gibson-Mount wrote: > > --On Tuesday, October 03, 2017 11:47 AM -0700 rammohan ganapavarapu > <rammohanganap(a)gmail.com <mailto:rammohanganap@gmail.com>> wrote: > > > Ok, but are there any casses where DB size can grow bigger then > RAM and > what will happen in that case? > > > Same thing that happens in any such case.. it'll start swapping. > > > LMDB never uses swap. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > <http://www.openldap.org/project/> > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Multiple index for a node
by Ervin Hegedüs 04 Oct '17

04 Oct '17

Hi there, is there any way to use multiple keys for a node in a LPD tree? I mean, there are several subtree-s: ou=company1,dc=foo,dc=com ou=company2,dc=foo,dc=com and I have to store the users under these subtrees. Sometimes the users have same names, eg. John Smith, and the nodes will be: uid=jsmith,ou=company1,dc=foo,dc=com uid=jsmith,ou=company2,dc=foo,dc=com but the any other attributes (sn, cn, ...) also the same. How do I set up the indexes? Thanks, a.

2 4

lmdb as a Lua database
by Russell Haley 04 Oct '17

04 Oct '17

Hi, So I'm using LMDB to create a simple 'database' system for lua that will allow me to directly serialize and store lua table data. So far I have used the wonderful lmdb wrapper lightningmdb and created a simple API to hide some of the mechanisms. Honestly, your API is so well thought out, I'm doing a disservice by removing a lot of the powerful features. However, I press ahead and beg forgiveness. :) I have two questions: 1) There is a simplified vs cursor API. I am starting to suspect that the simplified functions are just basic wrappers around the cursor API? I ask because if this is the case, I could just remove my support for the simplified API and only have to support one set of functions. 2) The key lookup is blazing fast (so awesome, this thing is going to smoke!). I wanted to create a way of indexing the data elements within the database by allowing users to create lua functions that parse the lua table values. --a basic table that would be serialized into a lmdb database called t1 t1 = { 1={col_a="test",col_b="item1"}, 2={col_a="not-a-test",col_b="item2"}, 3={col_a="test",col_b="item3"} } --The function that would be called per-row to create an index. local function(k,v) return v.col_a, k end The idea would be to use the dupsort feature to create an 'index' database with data structured as: not-a-test, 2 test, 1 test, 3

1 1

Can't get lmdb to build on FreeBSD 12
by Russell Haley 04 Oct '17

04 Oct '17

Hi, Sorry for the silly question, but I've stumped myself and my C is weak. Normally lmdb builds fine without issue on most of the platforms I try (Debian Jessie, FreeBSD 10.3 & 12), but it's stopped compiling on my TrueOS box, which is FreeBSD 12. I've tried gcc and clang 4.0 and 3.9. ------------------------ russellh@prescott:~/Git/lmdb/libraries/liblmdb% make cc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -Wuninitialized -v mdb_stat.c -o mdb_stat FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Target: x86_64-unknown-freebsd12.0 Thread model: posix InstalledDir: /usr/bin "/usr/bin/cc" -cc1 -triple x86_64-unknown-freebsd12.0 -emit-obj -disable-free -main-file-name mdb_stat.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -v -dwarf-column-info -debug-info-kind=standalone -dwarf-version=2 -debugger-tuning=gdb -resource-dir /usr/bin/../lib/clang/4.0.0 -O2 -W -Wall -Wno-unused-parameter -Wbad-function-cast -Wuninitialized -fdebug-compilation-dir /usr/home/russellh/Git/lmdb/libraries/liblmdb -ferror-limit 19 -fmessage-length 237 -pthread -fobjc-runtime=gnustep -fdiagnostics-show-option -vectorize-loops -vectorize-slp -o /tmp/mdb_stat-852384.o -x c mdb_stat.c clang -cc1 version 4.0.0 based upon LLVM 4.0.0 default target x86_64-unknown-freebsd12.0 #include "..." search starts here: #include <...> search starts here: /usr/bin/../lib/clang/4.0.0/include /usr/include End of search list. "/usr/bin/ld" --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o mdb_stat /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib /tmp/mdb_stat-852384.o -lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o /tmp/mdb_stat-852384.o: In function `main': /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:104: undefined reference to `mdb_env_create' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:106: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:111: undefined reference to `mdb_env_set_maxdbs' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:114: undefined reference to `mdb_env_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:116: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:121: undefined reference to `mdb_env_stat' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:122: undefined reference to `mdb_env_info' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:136: undefined reference to `mdb_reader_list' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:139: undefined reference to `mdb_reader_check' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:141: undefined reference to `mdb_reader_list' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:147: undefined reference to `mdb_txn_begin' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:149: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:257: undefined reference to `mdb_env_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:160: undefined reference to `mdb_cursor_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:162: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:165: undefined reference to `mdb_stat' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:171: undefined reference to `mdb_cursor_get' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:171: undefined reference to `mdb_cursor_get' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:199: undefined reference to `mdb_cursor_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:203: undefined reference to `mdb_dbi_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:205: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:209: undefined reference to `mdb_stat' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:(.text+0x5d6): undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:255: undefined reference to `mdb_txn_abort' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:221: undefined reference to `mdb_cursor_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:223: undefined reference to `mdb_strerror' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:226: undefined reference to `mdb_cursor_get' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:234: undefined reference to `mdb_dbi_open' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:239: undefined reference to `mdb_stat' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:245: undefined reference to `mdb_dbi_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:226: undefined reference to `mdb_cursor_get' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:247: undefined reference to `mdb_cursor_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:253: undefined reference to `mdb_dbi_close' /usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:241: undefined reference to `mdb_strerror' cc: error: linker command failed with exit code 1 (use -v to see invocation) *** Error code 1 Stop. make: stopped in /usr/home/russellh/Git/lmdb/libraries/liblmdb ---------------------------------- The full output is here with uname, git status and compiler info: http://termbin.com/io9d Any help would be grand. Thanks, Russ

1 0

Re: Initialize ldap with mdb
by Howard Chu 04 Oct '17

04 Oct '17

Quanah Gibson-Mount wrote: > --On Tuesday, October 03, 2017 11:47 AM -0700 rammohan ganapavarapu > <rammohanganap(a)gmail.com> wrote: > >> >> Ok, but are there any casses where DB size can grow bigger then RAM and >> what will happen in that case? > > Same thing that happens in any such case.. it'll start swapping. LMDB never uses swap. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

Re: Initialize ldap with mdb
by Quanah Gibson-Mount 03 Oct '17

03 Oct '17

--On Wednesday, October 04, 2017 2:18 AM +0200 Michael Ströder <michael(a)stroeder.com> wrote: > rammohan ganapavarapu wrote: >> You mean we can resize db any time? > > You can increase but *not decrease* maxsize later. I've been able to decrease it just fine, as long as it wasn't below the actual size of the DB. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: Initialize ldap with mdb
by rammohan ganapavarapu 03 Oct '17

03 Oct '17

You mean we can resize db any time? Wouldn't it be nice to have no max limit and grow db dynamically? On Oct 3, 2017 4:05 PM, "Quanah Gibson-Mount" <quanah(a)symas.com> wrote: --On Tuesday, October 03, 2017 5:04 PM -0700 rammohan ganapavarapu < rammohanganap(a)gmail.com> wrote: > Ok, i am just testing it, i will follow those recommendations, but this > is what i did as work around: > > > So i have updated the olcDbMaxSize to below value and now i am able to do > ldapadd, but i was in the impression that once u set the maxsize at the > time of ldap initialization you can't update that, right? > Incorrect. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

2 1

Re: Initialize ldap with mdb
by Quanah Gibson-Mount 03 Oct '17

03 Oct '17

--On Tuesday, October 03, 2017 5:12 PM -0700 rammohan ganapavarapu <rammohanganap(a)gmail.com> wrote: > > You mean we can resize db any time? Wouldn't it be nice to have no max > limit and grow db dynamically? MDB needs a max size at startup, so you have to define one. Generally, it's recommended to set a large value. I wrote a script once to resize MDB upwards as it got close (20%) to its maxsize. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: Initialize ldap with mdb
by Quanah Gibson-Mount 03 Oct '17

03 Oct '17

--On Tuesday, October 03, 2017 5:04 PM -0700 rammohan ganapavarapu <rammohanganap(a)gmail.com> wrote: > > Ok, i am just testing it, i will follow those recommendations, but this > is what i did as work around: > > > So i have updated the olcDbMaxSize to below value and now i am able to do > ldapadd, but i was in the impression that once u set the maxsize at the > time of ldap initialization you can't update that, right? Incorrect. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical