LMDB and HP-UX Itanium
by Kristian Amlie
I'm wondering if the LMDB database has been tested on HP-UX with the
Itanium processor? We are trying to use the database there and are
seeing strange errors that don't occur on other platforms. Examples are
assertions when trying to do cross process access and values not making
it into the database correctly.
I can provide more details, but I thought I would ask about the current
status first.
--
Kristian
7 years, 11 months
LMDB error
by Shu, Xinxin
Hi list ,
I am working on intergrating LMDB with ceph, I met follow error "Invalid argument" when I commit transaction to LMDB, details are descripted as following.
Two threads operate on LMDB, each thread is responsible for create write transaction and read transaction , the write transaction will update LMDB, and read transaction is for iterator that read values from LMDB, the two threads share same MDB_env. I check mdb_txn_commit source code , there are two places to return "EINVAL",
1. if (txn == NULL || txn->mt_env == NULL)
return EINVAL;
2. if (txn != env->me_txn) {
DPUTS("attempt to commit unknown transaction");
rc = EINVAL;
goto fail;
}
I check txn and txn->mt_env and rule out the first case , so only the second case cause "EINVAL", and variable env->me_txn is only initialized when you start a write transaction with mdb_txn_open, and there is a lock to prevent several writer from operating on LMDB. Btw , I tried to add lock to synchronize the two threads , everything seems ok.
My question is in what situation can cause "txn != env-> me_txn",.
I'm very new to LMDB, if there is anything wrong, please correct me.
Cheers,
xinxin
7 years, 11 months
Re: Very slow ldapserach
by Saša-Stjepan Bakša
On 8 April 2015 at 18:05, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote:
> --On Wednesday, April 08, 2015 11:58 AM +0200 Saša-Stjepan Bakša <
> ssbaksa(a)gmail.com> wrote:
>
>
>>
>> I have tested search with your suggestion. No crash - slapd crashes only
>> when deref is on or in best case it is slow (10 to 20 seconds for
>> answer). As I have said, when database is small (relative size) it is
>> quick as we expected.
>>
>>
>> So, which alternatives do I have? Back to hdb? Uh, I don't like it to
>> much.
>>
>> Is this long standing issue a big problem to solve? I am not developer so
>> maybe this was a rude question (sorry!).
>>
>
> Please keep replies on the list if you want help.
>
> My guess is this is ITS#7657.
>
>
> --Quanah
>
>
I am sorry for this mistake with answering to you and not to the group. It
was unintentional mistake.
Ok, I will check ITS#7657.
Sasa
7 years, 11 months
two namingContexts problem
by Wang, Hui
Hi, All,
I got two nameContexts, but I only set up one in slapd.conf which is o=csun. Is there any way to find out where the other nameContexts coming from? Is this normal?
I also searched all the files on Linux, I didn’t find any file contains “cn=changelog” in the file content.
[root@oldaptest01 init.d]# ldapsearch -x -b '' -s base '(objectclass=*)' namingContexts
# extended LDIF
#
# LDAPv3
# base <> with scope baseObject
# filter: (objectclass=*)
# requesting: namingContexts
#
#
dn:
namingContexts: cn=changelog
namingContexts: o=csun
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
# find -name "*.conf" -print | xargs grep -i "cn=change"
No result
Thank you
Hui
7 years, 11 months
[lmdb] MDB_BAD_DBI error upon mdb_txn_commit (newbie question)
by HLaw
Dear list,
I am new to lmdb and my apologies if this is on something obvious.
I encountered the MDB_BAD_DBI error in a simple single process,
multi-threaded program using lmdb. The scenario is :
* One thread doing a relatively long write transaction
mdb_txn_begin -> mdb_dbi_open on several database -> (write) ->
mdb_txn_commit
* Another thread (or other threads) of the same process doing short
read-only transactions
mdb_txn_begin -> mdb_dbi_open on some of the database above ->
(read) -> mdb_txn_commit
As per the advice in the document I do not close the database directly.
Also the environment created / opened is shared by all threads, being in
the same process.
All is fine when these threads execute separately, but if the read
transaction thread(s) open the database and commit within the time span
of the write transaction in another thread, the above error occurs upon
attempt to commit the write transaction.
I have done a search on the MDB_BAD_DBI error and noticed the commit below.
https://gitorious.org/mdb/mdb/commit/0401f2deed75a83d2de790b8a1313e1792e5...
Upon a brief look into this and the source of lmdb, my understanding is
that the error is due to the increment of lmdb's internal sequence
numbers of the database kept by the environment (by the read thread when
the read transaction commits). When the write thread subsequently tries
to commit, the database sequence number kept privately by the write
transaction (copied from the environment when the transaction began)
does not match that of the environment, causing the error.
To avoid this in a single process multi-threaded environment, it seems
that no read transactions could commit on databases while they are
involved in a write transaction, i.e. I need to complete all read
transactions before a write transaction on the same database could
begin. Alternatively, to have concurrent read transactions with a
write transaction, the read transactions have to take place in a
separate process (such that they would not share the same environment
and caught by the above error).
Grateful if anyone could kindly enlighten me on whether the
understanding is correct, or if I am doing something contrary to the how
lmdb should be used.
Thank you very much.
- H Law
7 years, 11 months
Re: Very slow ldapserach
by Quanah Gibson-Mount
--On Wednesday, April 08, 2015 11:58 AM +0200 Saša-Stjepan Bakša
<ssbaksa(a)gmail.com> wrote:
>
>
>
>
>
> On 7 April 2015 at 21:14, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote:
>
> --On Tuesday, April 07, 2015 1:10 PM -0700 Quanah Gibson-Mount
> <quanah(a)zimbra.com> wrote:
>
>
>
> ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w password -s sub
> -a always -b dc=SPR objectClass=*
>
>
> Sounds like there's something wrong with your openldap build or server.
> Zimbra has clients with DBs of various sizes and entry counts, often in
> the multi-GBs, and we're not seeing any such issues.
>
>
> Actually, you are doing alias deref. I believe there's a known issue
> with back-mdb and alias deref. Do you see the same behavior if you
> select -a never instead?
>
>
>
> --Quanah
>
>
>
>
> Yes, we are depending on aliases. Without them we will need at least 2
> searches to get data we need. In new version even more because we have
> data which is similar to many users and we use another aliases inside
> each user to fetch that data for particular user. Let we say that each
> user have 3 aliases to 3 groups of data which can have at leas 30
> different possibilities for each user. Without aliases data size for each
> user will be to big and in summary database will be huge.
>
>
> I have tested search with your suggestion. No crash - slapd crashes only
> when deref is on or in best case it is slow (10 to 20 seconds for
> answer). As I have said, when database is small (relative size) it is
> quick as we expected.
>
>
> So, which alternatives do I have? Back to hdb? Uh, I don't like it to
> much.
>
> Is this long standing issue a big problem to solve? I am not developer so
> maybe this was a rude question (sorry!).
Please keep replies on the list if you want help.
My guess is this is ITS#7657.
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
7 years, 11 months
Openldap NON responsive frequently
by MOHAMED Akbar
Hello,
Current openldap setup is REPLICATION, with MirrorMode on. Host is solaris11.
Frequently the clients(mostly solaris10), complain of : libsldap: Status: 91 Mesg: openConnection: simple bind failed - Can't connect to the LDAP server
Restarting the opeldap service fixes this
How can I fix this issue?
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
7 years, 11 months
Definition of an object.
by dE
Hi!
I was reading RFC 4512, here there is a mention of 'object' for the
first time in "Object identifiers (OIDs) [X.680] are represented in....".
Question is what is an object?
Is it an entry (aka directory) in the server?
7 years, 11 months
Very slow ldapserach
by Saša-Stjepan Bakša
Hi,
Year ago we have tested openldap with back_mdb and it was fantastic. Search
worked as a charm. Database was filled with 20 mil. users and serach
returned some 20 k results per sec (my colegue did the test).
Now we need that setup for some tests and we encountered very slow response
- 1 search for user data with some aliased data need 8 to 20 seconds to be
retrieved.
ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w test -s sub -a
always -b num=1234563123,dc=num,dc=SPR ObjectClass=*
num=1234563123,dc=num,dc=SPR is alias to uid
aliasedObjectName: uid=1234563123,ds=USERS,o=STANDARD,dc=spr
We build our openldap from git source. We have tried new as older versions
as well and no change is seen.
Hardware: SuperMicro, 2xQuad core, 32 GB RAM, RAID 10 storage.
HP blade 2xQuad core, 64 GB RAM sorage 2 disks in mirror.
Results are the same and not depending on hardware.
Openldap ver:
root@centdevel openldap# git log
commit 68d9aa207f51b4d1ef29bb9876e7da8c7eaf0eee
Author: Quanah Gibson-Mount <quanah(a)openldap.org>
Date: Tue Apr 8 21:16:52 2014 -0500
ITS#7430, ITS#6359
OS is Centos 6.4 (also tryed on Centos 6.6)numx
mdb config part is:
[root@spr2 cn=config]# cat olcDatabase\=\{1\}mdb.ldif
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.
# CRC32 2c245069
dn: olcDatabase={1}mdb
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcDbDirectory: /opt/openldap/var/openldap-data
olcSuffix: dc=spr
olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by
anonymou
s auth by dn="cn=admin,dc=spr" write by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to dn.base="" by * read
olcAccess: {3}to * by self write by dn="cn=admin,dc=spr" write by * read
olcLastMod: TRUE
olcRootDN: cn=admin,dc=spr
olcRootPW:: xyzdgsdsadeew
olcDbCheckpoint: 4096 10
olcDbNoSync: TRUE
olcDbIndex: objectClass eq
olcDbIndex: uid eq
olcDbIndex: num eq
olcDbIndex: numx eq
olcDbIndex: Username eq
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
olcDbIndex: contextCSN eq
olcDbMaxSize: 16106127360
structuralObjectClass: olcMdbConfig
entryUUID: 21ac150c-6b30-1034-9009-81396a683c5e
creatorsName: cn=admin,cn=config
createTimestamp: 20150330135513Z
entryCSN: 20150330135513.544218Z#000000#000#000000
modifiersName: cn=admin,cn=config
modifyTimestamp: 20150330135513Z
MDB database stat:
[root@spr2 openldap]# /opt/openldap/sbin/mdb_stat
/opt/openldap/var/openldap-data/ -e -rr -a
Environment Info
Map address: (nil)
Map size: 16106127360
Page size: 4096
Max pages: 3932160
Number of pages used: 1523336
Last transaction ID: 16058165
Max readers: 126
Number of readers used: 0
Reader Table Status
(no active readers)
0 stale readers cleared.
(no active readers)
Status of Main DB
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 11
Status of ad2i
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 38
Status of contextCSN
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of dn2i
Tree depth: 4
Branch pages: 2937
Leaf pages: 333338
Overflow pages: 0
Entries: 16000069
Status of entryCSN
Tree depth: 3
Branch pages: 3
Leaf pages: 307
Overflow pages: 0
Entries: 8000034
Status of entryUUID
Tree depth: 3
Branch pages: 259
Leaf pages: 62932
Overflow pages: 0
Entries: 8000034
Status of id2e
Tree depth: 4
Branch pages: 4446
Leaf pages: 1000005
Overflow pages: 0
Entries: 8000034
Status of numx
Tree depth: 3
Branch pages: 128
Leaf pages: 22295
Overflow pages: 0
Entries: 2000004
Status of num
Tree depth: 3
Branch pages: 129
Leaf pages: 22325
Overflow pages: 0
Entries: 2000004
Status of objectClass
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 29
Status of Username
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of uid
Tree depth: 3
Branch pages: 34
Leaf pages: 7883
Overflow pages: 0
Entries: 1000004
Build config:
make clean
./configure --enable-hdb=no \
--enable-bdb=no \
--enable-monitor=yes \
--prefix=/opt/openldap \
--enable-local=yes \
--enable-accesslog=yes \
--enable-syncprov=yes \
--enable-debug=yes
make depend
make #STRIP=''
rm -r /opt/openldap/etc/openldap/schema
make install #STRIP=''
removing debug has no efect
Do you have any hint for us?
Br
Sasa
7 years, 11 months
Multiple programs not able to read LMDB concurrently
by Sravan Kumar Reddy Javaji
Hello Everyone,
I am trying to access same LMDB source using multiple programs at the same
time. I set max_readers to 2 at the time of creating environment by the
first program, but still second program is not able to read the LMDB. Could
some one please let me know how could I implement this feature?
-
Thanks and Regards,
Sravan
7 years, 11 months
