Re: (ITS#7658) LMDB: Crash when mdb_del should ignore data, back trace and patch provided
by hyc@symas.com
sog(a)msg.com.mx wrote:
> Full_Name: Salvador Ortiz
> Version: 24
> OS: Linux
> URL: ftp://ftp.msg.com.mx/pub/varios/mdb_del-must-ignore-data.patch
> Submission from: (NULL) (187.162.45.111)
>
>
> According to the documentation, mdb_del should ignore the data parameter if the
> dbi was not opened with MDB_DUPSORT, but this check is missing causing a
> segfault deep in mdb_cursor_set when a NULL dcmp is called.
>
> I discovered this in the following back trace:
>
> #0 0x0000000000000000 in ?? ()
> #1 0x00007ffff1080f69 in mdb_cursor_set (mc=mc@entry=0x7fffffffd220,
> key=key@entry=0x7fffffffd5f0, data=0x7fffffffd210, op=MDB_GET_BOTH,
> exactp=exactp@entry=0x7fffffffd20c) at mdb.c:5149
> #2 0x00007ffff10862af in mdb_del (txn=txn@entry=0xe60a10, dbi=1,
> key=key@entry=0x7fffffffd5f0, data=data@entry=0x7fffffffd600) at mdb.c:7164
> ...
>
> A simple patch against gitorious's mdb.master provided.
Thanks, committed.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 4 months
(ITS#7658) LMDB: Crash when mdb_del should ignore data, back trace and patch provided
by sog@msg.com.mx
Full_Name: Salvador Ortiz
Version: 24
OS: Linux
URL: ftp://ftp.msg.com.mx/pub/varios/mdb_del-must-ignore-data.patch
Submission from: (NULL) (187.162.45.111)
According to the documentation, mdb_del should ignore the data parameter if the
dbi was not opened with MDB_DUPSORT, but this check is missing causing a
segfault deep in mdb_cursor_set when a NULL dcmp is called.
I discovered this in the following back trace:
#0 0x0000000000000000 in ?? ()
#1 0x00007ffff1080f69 in mdb_cursor_set (mc=mc@entry=0x7fffffffd220,
key=key@entry=0x7fffffffd5f0, data=0x7fffffffd210, op=MDB_GET_BOTH,
exactp=exactp@entry=0x7fffffffd20c) at mdb.c:5149
#2 0x00007ffff10862af in mdb_del (txn=txn@entry=0xe60a10, dbi=1,
key=key@entry=0x7fffffffd5f0, data=data@entry=0x7fffffffd600) at mdb.c:7164
...
A simple patch against gitorious's mdb.master provided.
10 years, 4 months
Re: (ITS#7657) Alias dereferencing with MDB slow compared with BDB
by quanah@zimbra.com
--On August 8, 2013 5:12:50 PM +0100 Mark Cairney <Mark.Cairney(a)ed.ac.uk>
wrote:
> Hi Quanah,
>
> I've transferred one of the nodes to back-hdb and the performance appears
> to broadly match back-bdb.
>
> Is there a simple test I can perform to confirm that it is indeed using
> back-hdb?
>
> For completeness here's the result of my timed search:
>
> bash-4.1$ time ldapsearch -H ldaps://rowan.authorise.is.ed.ac.uk:636 -b
> 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always
> "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))"
>
># numResponses: 425
># numEntries: 424
>
> real 0m0.902s
> user 0m0.054s
> sys 0m0.025s
No, that's enough, thanks. So it is MDB specific then. ;)
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 4 months
Re: (ITS#7657) Alias dereferencing with MDB slow compared with BDB
by Mark.Cairney@ed.ac.uk
Hi Quanah,
I've transferred one of the nodes to back-hdb and the performance
appears to broadly match back-bdb.
Is there a simple test I can perform to confirm that it is indeed using
back-hdb?
For completeness here's the result of my timed search:
bash-4.1$ time ldapsearch -H ldaps://rowan.authorise.is.ed.ac.uk:636 -b
'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always
"(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))"
# numResponses: 425
# numEntries: 424
real 0m0.902s
user 0m0.054s
sys 0m0.025s
On 08/08/13 16:45, Quanah Gibson-Mount wrote:
> --On August 8, 2013 4:34:32 PM +0100 Mark Cairney
> <Mark.Cairney(a)ed.ac.uk> wrote:
>
>
>>> back-bdb or back-hdb? "BDB" is just the storage database for those two
>>> backends.
>>
>>
>> back-bdb.
>
> Thanks. I would note that back-mdb is based on back-hdb, and that can
> have significant differences in performance profile for some operations
> as compared to back-bdb. If it were possible, I'd be curious to know if
> you see the same issue with back-hdb as you do with back-mdb.
>
> --Quanah
>
--
/****************************
Mark Cairney
ITI UNIX Section
Information Services
University of Edinburgh
Tel: 0131 650 6565
Email: Mark.Cairney(a)ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
10 years, 4 months
Re: (ITS#7657) Alias dereferencing with MDB slow compared with BDB
by quanah@zimbra.com
--On August 8, 2013 4:34:32 PM +0100 Mark Cairney <Mark.Cairney(a)ed.ac.uk>
wrote:
>> back-bdb or back-hdb? "BDB" is just the storage database for those two
>> backends.
>
>
> back-bdb.
Thanks. I would note that back-mdb is based on back-hdb, and that can have
significant differences in performance profile for some operations as
compared to back-bdb. If it were possible, I'd be curious to know if you
see the same issue with back-hdb as you do with back-mdb.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 4 months
Re: (ITS#7657) Alias dereferencing with MDB slow compared with BDB
by Mark.Cairney@ed.ac.uk
On 08/08/13 16:27, Howard Chu wrote:
>
> How many aliases?
>
In that OU 435, in total across the whole database: 29179.
--
/****************************
Mark Cairney
ITI UNIX Section
Information Services
University of Edinburgh
Tel: 0131 650 6565
Email: Mark.Cairney(a)ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
10 years, 4 months
Re: (ITS#7657) Alias dereferencing with MDB slow compared with BDB
by Mark.Cairney@ed.ac.uk
On 08/08/13 16:32, Quanah Gibson-Mount wrote:
>
>
> --On August 8, 2013 3:02:51 PM +0000 Mark.Cairney(a)ed.ac.uk wrote:
>
>> Full_Name: Mark Cairney
>> Version: 2.4.35-RE24
>> OS: SL 6.4 64-bit
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (129.215.200.23)
>>
>>
>> Having migrated the database of our production service running 2.4.35
>> from BDB to MDB alias de-referencing stopped working completely (this has
>> already been reported in ITS# 7557).
>
> back-bdb or back-hdb? "BDB" is just the storage database for those two
> backends.
back-bdb.
>
> Thanks,
> Quanah
>
--
/****************************
Mark Cairney
ITI UNIX Section
Information Services
University of Edinburgh
Tel: 0131 650 6565
Email: Mark.Cairney(a)ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
10 years, 4 months
Re: (ITS#7657) Alias dereferencing with MDB slow compared with BDB
by quanah@zimbra.com
--On August 8, 2013 3:02:51 PM +0000 Mark.Cairney(a)ed.ac.uk wrote:
> Full_Name: Mark Cairney
> Version: 2.4.35-RE24
> OS: SL 6.4 64-bit
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (129.215.200.23)
>
>
> Having migrated the database of our production service running 2.4.35
> from BDB to MDB alias de-referencing stopped working completely (this has
> already been reported in ITS# 7557).
back-bdb or back-hdb? "BDB" is just the storage database for those two
backends.
Thanks,
Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 4 months
Re: (ITS#7657) Alias dereferencing with MDB slow compared with BDB
by hyc@symas.com
Mark.Cairney(a)ed.ac.uk wrote:
> Full_Name: Mark Cairney
> Version: 2.4.35-RE24
> OS: SL 6.4 64-bit
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (129.215.200.23)
>
>
> Having migrated the database of our production service running 2.4.35 from BDB
> to MDB alias de-referencing stopped working completely (this has already been
> reported in ITS# 7557).
>
> Upgrading to the RE 24 now has alias derefencing working but it is considerably
> slower than BDB was for the same search.
>
> To confirm this I downgraded one of our nodes (we run 4 nodes in multi-master)
> and on that node the search results are once more acceptable:
>
> With MDB:
> time ldapsearch -H ldaps://alder.authorise.is.ed.ac.uk:636 -b
> 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always
> "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))"
>
> # numResponses: 425
> # numEntries: 424
>
> real 0m26.490s
> user 0m0.028s
> sys 0m0.032s
>
>
> with BDB:
>
> time ldapsearch -H ldaps://oak.authorise.is.ed.ac.uk:636 -b
> 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always
> "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))"
>
> real 0m2.605s
> user 0m0.054s
> sys 0m0.006s
>
> If there's anything additional I can supply (log files, strace output) let me
> know.
>
> Our OpenLDAP DB currently has around 400,000 user accounts if that's relevant.
How many aliases?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 4 months
(ITS#7657) Alias dereferencing with MDB slow compared with BDB
by Mark.Cairney@ed.ac.uk
Full_Name: Mark Cairney
Version: 2.4.35-RE24
OS: SL 6.4 64-bit
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (129.215.200.23)
Having migrated the database of our production service running 2.4.35 from BDB
to MDB alias de-referencing stopped working completely (this has already been
reported in ITS# 7557).
Upgrading to the RE 24 now has alias derefencing working but it is considerably
slower than BDB was for the same search.
To confirm this I downgraded one of our nodes (we run 4 nodes in multi-master)
and on that node the search results are once more acceptable:
With MDB:
time ldapsearch -H ldaps://alder.authorise.is.ed.ac.uk:636 -b
'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always
"(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))"
# numResponses: 425
# numEntries: 424
real 0m26.490s
user 0m0.028s
sys 0m0.032s
with BDB:
time ldapsearch -H ldaps://oak.authorise.is.ed.ac.uk:636 -b
'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always
"(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))"
real 0m2.605s
user 0m0.054s
sys 0m0.006s
If there's anything additional I can supply (log files, strace output) let me
know.
Our OpenLDAP DB currently has around 400,000 user accounts if that's relevant.
10 years, 4 months