lmdb as a Lua database
by Russell Haley
Hi,
So I'm using LMDB to create a simple 'database' system for lua that
will allow me to directly serialize and store lua table data. So far I
have used the wonderful lmdb wrapper lightningmdb and created a simple
API to hide some of the mechanisms.
Honestly, your API is so well thought out, I'm doing a disservice by
removing a lot of the powerful features. However, I press ahead and
beg forgiveness. :)
I have two questions:
1) There is a simplified vs cursor API. I am starting to suspect that
the simplified functions are just basic wrappers around the cursor
API? I ask because if this is the case, I could just remove my support
for the simplified API and only have to support one set of functions.
2) The key lookup is blazing fast (so awesome, this thing is going to
smoke!). I wanted to create a way of indexing the data elements within
the database by allowing users to create lua functions that parse the
lua table values.
--a basic table that would be serialized into a lmdb database called t1
t1 = {
1={col_a="test",col_b="item1"},
2={col_a="not-a-test",col_b="item2"},
3={col_a="test",col_b="item3"}
}
--The function that would be called per-row to create an index.
local function(k,v)
return v.col_a, k
end
The idea would be to use the dupsort feature to create an 'index'
database with data structured as:
not-a-test, 2
test, 1
test, 3
5 years, 11 months
Can't get lmdb to build on FreeBSD 12
by Russell Haley
Hi,
Sorry for the silly question, but I've stumped myself and my C is
weak. Normally lmdb builds fine without issue on most of the platforms
I try (Debian Jessie, FreeBSD 10.3 & 12), but it's stopped compiling
on my TrueOS box, which is FreeBSD 12. I've tried gcc and clang 4.0
and 3.9.
------------------------
russellh@prescott:~/Git/lmdb/libraries/liblmdb% make
cc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast
-Wuninitialized -v mdb_stat.c -o mdb_stat
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on
LLVM 4.0.0)
Target: x86_64-unknown-freebsd12.0
Thread model: posix
InstalledDir: /usr/bin
"/usr/bin/cc" -cc1 -triple x86_64-unknown-freebsd12.0 -emit-obj
-disable-free -main-file-name mdb_stat.c -mrelocation-model static
-mthread-model posix -mdisable-fp-elim -masm-verbose
-mconstructor-aliases -munwind-tables -target-cpu x86-64 -v
-dwarf-column-info -debug-info-kind=standalone -dwarf-version=2
-debugger-tuning=gdb -resource-dir /usr/bin/../lib/clang/4.0.0 -O2 -W
-Wall -Wno-unused-parameter -Wbad-function-cast -Wuninitialized
-fdebug-compilation-dir /usr/home/russellh/Git/lmdb/libraries/liblmdb
-ferror-limit 19 -fmessage-length 237 -pthread -fobjc-runtime=gnustep
-fdiagnostics-show-option -vectorize-loops -vectorize-slp -o
/tmp/mdb_stat-852384.o -x c mdb_stat.c
clang -cc1 version 4.0.0 based upon LLVM 4.0.0 default target
x86_64-unknown-freebsd12.0
#include "..." search starts here:
#include <...> search starts here:
/usr/bin/../lib/clang/4.0.0/include
/usr/include
End of search list.
"/usr/bin/ld" --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1
--hash-style=both --enable-new-dtags -o mdb_stat /usr/lib/crt1.o
/usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib /tmp/mdb_stat-852384.o
-lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc
--as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o
/tmp/mdb_stat-852384.o: In function `main':
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:104:
undefined reference to `mdb_env_create'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:106:
undefined reference to `mdb_strerror'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:111:
undefined reference to `mdb_env_set_maxdbs'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:114:
undefined reference to `mdb_env_open'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:116:
undefined reference to `mdb_strerror'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:121:
undefined reference to `mdb_env_stat'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:122:
undefined reference to `mdb_env_info'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:136:
undefined reference to `mdb_reader_list'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:139:
undefined reference to `mdb_reader_check'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:141:
undefined reference to `mdb_reader_list'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:147:
undefined reference to `mdb_txn_begin'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:149:
undefined reference to `mdb_strerror'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:257:
undefined reference to `mdb_env_close'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:160:
undefined reference to `mdb_cursor_open'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:162:
undefined reference to `mdb_strerror'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:165:
undefined reference to `mdb_stat'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:171:
undefined reference to `mdb_cursor_get'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:171:
undefined reference to `mdb_cursor_get'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:199:
undefined reference to `mdb_cursor_close'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:203:
undefined reference to `mdb_dbi_open'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:205:
undefined reference to `mdb_strerror'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:209:
undefined reference to `mdb_stat'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:(.text+0x5d6):
undefined reference to `mdb_strerror'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:255:
undefined reference to `mdb_txn_abort'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:221:
undefined reference to `mdb_cursor_open'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:223:
undefined reference to `mdb_strerror'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:226:
undefined reference to `mdb_cursor_get'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:234:
undefined reference to `mdb_dbi_open'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:239:
undefined reference to `mdb_stat'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:245:
undefined reference to `mdb_dbi_close'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:226:
undefined reference to `mdb_cursor_get'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:247:
undefined reference to `mdb_cursor_close'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:253:
undefined reference to `mdb_dbi_close'
/usr/home/russellh/Git/lmdb/libraries/liblmdb/mdb_stat.c:241:
undefined reference to `mdb_strerror'
cc: error: linker command failed with exit code 1 (use -v to see
invocation)
*** Error code 1
Stop.
make: stopped in /usr/home/russellh/Git/lmdb/libraries/liblmdb
----------------------------------
The full output is here with uname, git status and compiler info:
http://termbin.com/io9d
Any help would be grand.
Thanks,
Russ
5 years, 11 months
Re: Initialize ldap with mdb
by Howard Chu
Quanah Gibson-Mount wrote:
> --On Tuesday, October 03, 2017 11:47 AM -0700 rammohan ganapavarapu
> <rammohanganap(a)gmail.com> wrote:
>
>>
>> Ok, but are there any casses where DB size can grow bigger then RAM and
>> what will happen in that case?
>
> Same thing that happens in any such case.. it'll start swapping.
LMDB never uses swap.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5 years, 11 months
Re: Initialize ldap with mdb
by Quanah Gibson-Mount
--On Wednesday, October 04, 2017 2:18 AM +0200 Michael Ströder
<michael(a)stroeder.com> wrote:
> rammohan ganapavarapu wrote:
>> You mean we can resize db any time?
>
> You can increase but *not decrease* maxsize later.
I've been able to decrease it just fine, as long as it wasn't below the
actual size of the DB.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 11 months
Re: Initialize ldap with mdb
by rammohan ganapavarapu
You mean we can resize db any time? Wouldn't it be nice to have no max
limit and grow db dynamically?
On Oct 3, 2017 4:05 PM, "Quanah Gibson-Mount" <quanah(a)symas.com> wrote:
--On Tuesday, October 03, 2017 5:04 PM -0700 rammohan ganapavarapu <
rammohanganap(a)gmail.com> wrote:
> Ok, i am just testing it, i will follow those recommendations, but this
> is what i did as work around:
>
>
> So i have updated the olcDbMaxSize to below value and now i am able to do
> ldapadd, but i was in the impression that once u set the maxsize at the
> time of ldap initialization you can't update that, right?
>
Incorrect.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 11 months
Re: Initialize ldap with mdb
by Quanah Gibson-Mount
--On Tuesday, October 03, 2017 5:12 PM -0700 rammohan ganapavarapu
<rammohanganap(a)gmail.com> wrote:
>
> You mean we can resize db any time? Wouldn't it be nice to have no max
> limit and grow db dynamically?
MDB needs a max size at startup, so you have to define one. Generally,
it's recommended to set a large value. I wrote a script once to resize MDB
upwards as it got close (20%) to its maxsize.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 11 months
Re: Initialize ldap with mdb
by Quanah Gibson-Mount
--On Tuesday, October 03, 2017 5:04 PM -0700 rammohan ganapavarapu
<rammohanganap(a)gmail.com> wrote:
>
> Ok, i am just testing it, i will follow those recommendations, but this
> is what i did as work around:
>
>
> So i have updated the olcDbMaxSize to below value and now i am able to do
> ldapadd, but i was in the impression that once u set the maxsize at the
> time of ldap initialization you can't update that, right?
Incorrect.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 11 months
Re: Initialize ldap with mdb
by rammohan ganapavarapu
Ok, i am just testing it, i will follow those recommendations, but this is
what i did as work around:
So i have updated the olcDbMaxSize to below value and now i am able to do
ldapadd, but i was in the impression that once u set the maxsize at the
time of ldap initialization you can't update that, right?
echo "dn: olcDatabase={1}mdb,cn=config
changetype: modify
replace: olcDbMaxSize
olcDbMaxSize: 21474836480" |ldapmodify -x -D "cn=config" -H
ldap://localhost:10389 -w
So the above operation will resize the db?
Thanks,
Ram
On Tue, Oct 3, 2017 at 4:01 PM, Quanah Gibson-Mount <quanah(a)symas.com>
wrote:
> --On Tuesday, October 03, 2017 4:53 PM -0700 rammohan ganapavarapu <
> rammohanganap(a)gmail.com> wrote:
>
>
>> Ok, this is what i see in ldap logs, this could be because of
>> maxsize? olcDbMaxSize: 102400
>>
>>
>> mdb_idl_insert_keys: c_put id failed: MDB_MAP_FULL: Environment mapsize
>> limit reached (-30792)
>>
>
> Yes. Again, as I said before, you need to set a *large* maxsize. Like
> the one I recommended previously.
>
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Product Architect
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>
>
>
5 years, 11 months
Re: Initialize ldap with mdb
by Quanah Gibson-Mount
--On Tuesday, October 03, 2017 4:53 PM -0700 rammohan ganapavarapu
<rammohanganap(a)gmail.com> wrote:
>
> Ok, this is what i see in ldap logs, this could be because of
> maxsize? olcDbMaxSize: 102400
>
>
> mdb_idl_insert_keys: c_put id failed: MDB_MAP_FULL: Environment mapsize
> limit reached (-30792)
Yes. Again, as I said before, you need to set a *large* maxsize. Like the
one I recommended previously.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 11 months
Re: Initialize ldap with mdb
by rammohan ganapavarapu
There are two mdb files {data,lock}.mdb both owned by processes user.
On Tue, Oct 3, 2017 at 3:41 PM, Quanah Gibson-Mount <quanah(a)symas.com>
wrote:
> --On Tuesday, October 03, 2017 3:53 PM -0700 rammohan ganapavarapu <
> rammohanganap(a)gmail.com> wrote:
>
> and now i am trying to add below and getting index generation failed
>> error, db dir and slapd.d owned by slapd processes user.
>>
>
> What about the *.mdb files themselves?
>
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Product Architect
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>
>
>
5 years, 11 months
