Re: (ITS#7531) Syncrepl push delete operation does not recover when slave is unavailable
by quanah＠zimbra.com
--On Wednesday, March 27, 2013 8:42 AM +0000
hans.moser(a)ofd-z.niedersachsen.de wrote:
>> Sorry, I was in test054, not test045. But as base 054 seems to be just
>> fine. I hope, I did the patch right.
> Hm, I was not included. :(
> Is there something wrong with the patch?
Hi Marc,
Simply an oversight, sorry.
I've patched test054 with your bits on my local install, on a 2.4.34 build
without the fix for ITS#7531, and it passes for me. I would assume since I
haven't patched 2.4.34 for the fix, it should fail? I.e., it seems this
patch doesn't actually catch the problem. :/
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 2 months
Re: (ITS#7531) Syncrepl push delete operation does not recover when slave is unavailable
by hans.moser＠ofd-z.niedersachsen.de
Hi,
hans.moser(a)ofd-z.niedersachsen.de schrieb (14.03.2013 17:22 Uhr):
> hyc(a)symas.com schrieb (14.03.2013 16:24 Uhr):
>> Marc Patermann wrote:
>>> hyc(a)symas.com schrieb (14.03.2013 15:14 Uhr):
>>>> hans.moser(a)ofd-z.niedersachsen.de wrote:
>>>>> I think we had situations like this before...
>>>>>
>>>>> How about having a test for this?
>>>>> What is the test which is nearest to this situation where a test could
>>>>> be based on?
>>>> test045. Feel free to submit patches for this test script.
>>> If I'm not mistaken,
>>> - in lines 262-264 the consumer is stopped
>>> - in lines 266-284 changes are made to the provider
>>> - in lines 286-294 the consumer is started again.
>> Sounds reasonable but your line numbers don't match anything in our tree.
>> Please send an actual patch, e.g. from git format-patch (or just diff -u.)
> Sorry, I was in test054, not test045. But as base 054 seems to be just fine.
> I hope, I did the patch right.
Hm, I was not included. :(
Is there something wrong with the patch?
Marc
10 years, 2 months
RE: (ITS#7545) Problem with v2.4.30
by quanah＠zimbra.com
--On Tuesday, March 26, 2013 7:45 PM +0000 "Swenson_CNTR, Carl E."
<Carl.Swenson(a)dodiis.mil> wrote:
> All,
>
> Thank you for the replies. Yes, Oracle makes the hardware, but as you
> may have experienced when dealing with Oracle, if there is anything on
> the server that is not "Oracle", then the problem must be with that
> software. (This is the response you will get from Oracle on just about
> anything).
BDB is owned by Oracle, so they should be able to help you with it. The
creation of the BDB log files is *purely* a function internal to BDB. The
fact that they are exploding would be a function of BDB. That is why I've
repeatedly said there is no indication of an OpenLDAP bug here. That is
why Howard directed you to Oracle. They are the ones who own BDB, they
should be able to provide you insight into why it has broken on your new
server.
BDB logs are created when changes are made to the database. If you see
your logs "exploding" without corresponding massive changes being made to
the OpenLDAP server, then you are essentially ensured that the issue is
with BDB. I would also suggest you run the db_archive utility to see what
log(s) it says can or can't be archived. It is possible that BDB is
holding open a write or read transaction for some reason (again, a BDB bug).
As for MDB, it is a new database written by Howard Chu, the primary
OpenLDAP Developer. You can read up on it at <http://symas.com/mdb/>. It
is significantly better than BDB in numerous ways. I disagree with your
assessment on the recent change log as well. It is almost all entirely
around mdb.
OpenLDAP 2.4.34 Release (2013/03/03)
Fixed liblmdb mdb_env_open flag handling (ITS#7453)
Fixed liblmdb mdb_midl_sort array optimization (ITS#7432)
Fixed liblmdb freelist with large entries (ITS#7455)
Fixed liblmdb to check for filled dirty page list (ITS#7491)
Fixed liblmdb to validate data limits (ITS#7485)
Fixed liblmdb mdb_update_key for large keys (ITS#7505)
Fixed slapd-mdb to reopen attr DBs after env reopen (ITS#7416)
Fixed slapd-mdb handling of missing entries (ITS#7483,7496)
Fixed slapd-mdb environment flag setting (ITS#7452)
Fixed slapd-mdb with sub db slapcat (ITS#7469)
Fixed slapd-mdb to correctly work with toolthreads > 2 (ITS#7488,ITS#7527)
Fixed slapd-mdb subtree search speed (ITS#7473)
I see zero changes in 2.4.34 related to BDB at all.
OpenLDAP 2.4.33 Release (2012/10/10)
Fixed liblmdb POSIX semaphore cleanup on environment close (ITS#7364)
Fixed liblmdb mdb_page_split (ITS#7385, ITS#7229)
Fixed slapd-mdb slapadd -q -w double free (ITS#7356)
Fixed slapd-mdb to close read txn in reindex commit (ITS#7386)
Again, no changes related to BDB.
The last release with any fixes specific to BDB was 2.4.32.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 2 months
RE: (ITS#7545) Problem with v2.4.30
by Carl.Swenson＠dodiis.mil
All,
Thank you for the replies. Yes, Oracle makes the hardware, but as you may have experienced when dealing with Oracle, if there is anything on the server that is not "Oracle", then the problem must be with that software. (This is the response you will get from Oracle on just about anything).
We have many security layers, and an "isolated network" that has zero chance of getting to the internet / outside world that is of very little risk from malware. It is true we have never had to compile the packages being used so the install defaults referred to BDB, and that is what we used. I realize BDB 4.7.25.NC may not be the "bleeding edge" but it is the release referred to on sunfreeware.com (they have v2.4.32 packaged for download), and when you look on OpenLDAP's web site there are references to get the compiled packages from sunfreeware.com. I see on Oracle's web site there is a v5.3 download that can try.
As far as "mdb" goes, we don't know what that is, and most references even in the v2.4.34 download talk about BDB. The purpose we use OpenLDAP for is a very simple authentication with WebLogic. Nothing heavy duty.
I agree there could be some "strangeness" when mixing these packages with a new T4, but it is still unknown.
Quanah,
"There certainly isn't enough detail in your report to really offer you any particular direction to pursue."
What is it that would be useful, if I can capture it, and "secure" transfer it then I will. But I can't get most detail reports through security de-class scan to be able to post it.
Carl
===========
Well... Oracle now owns your hardware manufacturer and the software in question. They'd be the obvious place to ask for help.
Frankly I'm surprised that a military site on a classified network is allowed to install freeware binaries from any internet site. You have no idea how those binaries were built, and certainly no recourse if they fail in any way or happen to have trojans/malware embedded within.
My first step would be to compile current source code. If the behavior still exists on current code, then start debugging/diagnosing. BDB 4.7.25 is quite old, as Quanah already pointed out. We've never seen any other cases reported like this, but it's always possible for a mixture of old binaries and new hardware to go bad. For all you know it's an instruction set incompatibility and recompiling BDB for your T4 will make it go away.
>
>
> Carl
>
>
> -----Original Message-----
> From: Quanah Gibson-Mount [mailto:quanah@zimbra.com]
> Sent: Wednesday, March 20, 2013 2:20 PM
> To: Swenson_CNTR, Carl E.; openldap-its(a)openldap.org
> Subject: RE: (ITS#7545) Problem with v2.4.30
>
> --On Tuesday, March 19, 2013 8:57 PM +0000 "Swenson_CNTR, Carl E."
> <Carl.Swenson(a)dodiis.mil> wrote:
>
>> Quanah,
>>
>> Thank you for the reply.
>>
>> I double checked my versions on some of my systems just to be sure.
>> One inconsistency I noticed was in the name of the DB package
>> Possibly downloaded at 2 different times from sunfreeware.com.
>>
>> Example -
>> Sun Enterprise SPARC T5220 (everything works on this system) pkginfo
>> -l SMColdap - version 2.4.30 pkginfo -l SMCossl - version 1.0.1c
>> pkginfo -l SMCdb - version 4.7.25.NC
>>
>> Sun Enterprise SPARC T4-1 (the logs go crazy on this system) pkginfo
>> -l SMColdap - version 2.4.30 pkginfo -l SMCossl - version 1.0.1c
>> pkginfo -l SMCdb47 - version 4.7.25.NC
>>
>> IDK how this would affect the system, but based on your suggestion
>> there is some difference in the package as the "pkginst" name on one
>> system is SMCdb, and SMCdb47 on the other while the version is exactly the same.
>
> I have no idea. I abandoned Slowaris years ago... I abandoned using BDB with OpenLDAP last year. However, no one has reported any issues like this. It really appears to be a bug in BDB rather than OpenLDAP however.
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Sr. Member of Technical Staff
> Zimbra, Inc
> A Division of VMware, Inc.
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 2 months
Re: (ITS#7545) Problem with v2.4.30
by hyc＠symas.com
Carl.Swenson(a)dodiis.mil wrote:
> Quanah,
>
> Fine, I get it, you hate Solaris, BDB, etc...... But this is what I have
> to
work with, and I still have an issue that I have never heard of, nor do I know
what to even look for. Google searches turn up very little.
>
> Is there an "OpenLDAP forum" where you can get help? I have tried removing
the package labeled SMCdb47, and loaded SMCdb. The result is still the same. I
know you don't like BDB, (you didn't mention what you do prefer) but in any
case we are approved to use OpenLDAP w/ BDB. Normally we download all the
supporting packages from sunfreeware.com, and if I remember correctly, it is
defaulted to use BDB.
>
> So I have to use:
> Solaris 10
> BDB 4.7.25.NC
> OpenSSL 1.0.1c
> OpenLDAP 2.4.30
>
> What can I do. I have configured hundreds of servers using these packages,
and their predecessors. We have never seen this behavior.
Well... Oracle now owns your hardware manufacturer and the software in
question. They'd be the obvious place to ask for help.
Frankly I'm surprised that a military site on a classified network is allowed
to install freeware binaries from any internet site. You have no idea how
those binaries were built, and certainly no recourse if they fail in any way
or happen to have trojans/malware embedded within.
My first step would be to compile current source code. If the behavior still
exists on current code, then start debugging/diagnosing. BDB 4.7.25 is quite
old, as Quanah already pointed out. We've never seen any other cases reported
like this, but it's always possible for a mixture of old binaries and new
hardware to go bad. For all you know it's an instruction set incompatibility
and recompiling BDB for your T4 will make it go away.
>
>
> Carl
>
>
> -----Original Message-----
> From: Quanah Gibson-Mount [mailto:quanah@zimbra.com]
> Sent: Wednesday, March 20, 2013 2:20 PM
> To: Swenson_CNTR, Carl E.; openldap-its(a)openldap.org
> Subject: RE: (ITS#7545) Problem with v2.4.30
>
> --On Tuesday, March 19, 2013 8:57 PM +0000 "Swenson_CNTR, Carl E."
> <Carl.Swenson(a)dodiis.mil> wrote:
>
>> Quanah,
>>
>> Thank you for the reply.
>>
>> I double checked my versions on some of my systems just to be sure.
>> One inconsistency I noticed was in the name of the DB package
>> Possibly downloaded at 2 different times from sunfreeware.com.
>>
>> Example -
>> Sun Enterprise SPARC T5220 (everything works on this system) pkginfo
>> -l SMColdap - version 2.4.30 pkginfo -l SMCossl - version 1.0.1c
>> pkginfo -l SMCdb - version 4.7.25.NC
>>
>> Sun Enterprise SPARC T4-1 (the logs go crazy on this system) pkginfo
>> -l SMColdap - version 2.4.30 pkginfo -l SMCossl - version 1.0.1c
>> pkginfo -l SMCdb47 - version 4.7.25.NC
>>
>> IDK how this would affect the system, but based on your suggestion
>> there is some difference in the package as the "pkginst" name on one
>> system is SMCdb, and SMCdb47 on the other while the version is exactly the same.
>
> I have no idea. I abandoned Slowaris years ago... I abandoned using BDB with OpenLDAP last year. However, no one has reported any issues like this. It really appears to be a bug in BDB rather than OpenLDAP however.
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Sr. Member of Technical Staff
> Zimbra, Inc
> A Division of VMware, Inc.
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 2 months
RE: (ITS#7545) Problem with v2.4.30
by quanah＠zimbra.com
--On Tuesday, March 26, 2013 6:09 PM +0000 "Swenson_CNTR, Carl E."
<Carl.Swenson(a)dodiis.mil> wrote:
> Quanah,
>
> Fine, I get it, you hate Solaris, BDB, etc...... But this is what I have
> to work with, and I still have an issue that I have never heard of, nor
> do I know what to even look for. Google searches turn up very little.
>
> Is there an "OpenLDAP forum" where you can get help? I have tried
> removing the package labeled SMCdb47, and loaded SMCdb. The result is
> still the same. I know you don't like BDB, (you didn't mention what you
> do prefer) but in any case we are approved to use OpenLDAP w/ BDB.
> Normally we download all the supporting packages from sunfreeware.com,
> and if I remember correctly, it is defaulted to use BDB.
As I initially stated, there is nothing in your bug report to indicate a
bug with OpenLDAP. I would note that BDB 4.7.25 is ancient. I would note
that OpenLDAP 2.4.30 is fairly old. Who knows how OpenLDAP or BDB was
built with those packages you are using. There certainly isn't enough
detail in your report to really offer you any particular direction to
pursue. If it was *me*, I would build the latest OpenLDAP with a current
version of BDB, where I could control the compile options and flags.
By the way, BDB support in OpenLDAP is essentially deprecated. All ongoing
development is with the new MDB backend.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 2 months
RE: (ITS#7545) Problem with v2.4.30
by Carl.Swenson＠dodiis.mil
Quanah,
Fine, I get it, you hate Solaris, BDB, etc...... But this is what I have to work with, and I still have an issue that I have never heard of, nor do I know what to even look for. Google searches turn up very little.
Is there an "OpenLDAP forum" where you can get help? I have tried removing the package labeled SMCdb47, and loaded SMCdb. The result is still the same. I know you don't like BDB, (you didn't mention what you do prefer) but in any case we are approved to use OpenLDAP w/ BDB. Normally we download all the supporting packages from sunfreeware.com, and if I remember correctly, it is defaulted to use BDB.
So I have to use:
Solaris 10
BDB 4.7.25.NC
OpenSSL 1.0.1c
OpenLDAP 2.4.30
What can I do. I have configured hundreds of servers using these packages, and their predecessors. We have never seen this behavior.
Carl
-----Original Message-----
From: Quanah Gibson-Mount [mailto:quanah@zimbra.com]
Sent: Wednesday, March 20, 2013 2:20 PM
To: Swenson_CNTR, Carl E.; openldap-its(a)openldap.org
Subject: RE: (ITS#7545) Problem with v2.4.30
--On Tuesday, March 19, 2013 8:57 PM +0000 "Swenson_CNTR, Carl E."
<Carl.Swenson(a)dodiis.mil> wrote:
> Quanah,
>
> Thank you for the reply.
>
> I double checked my versions on some of my systems just to be sure.
> One inconsistency I noticed was in the name of the DB package
> Possibly downloaded at 2 different times from sunfreeware.com.
>
> Example -
> Sun Enterprise SPARC T5220 (everything works on this system) pkginfo
> -l SMColdap - version 2.4.30 pkginfo -l SMCossl - version 1.0.1c
> pkginfo -l SMCdb - version 4.7.25.NC
>
> Sun Enterprise SPARC T4-1 (the logs go crazy on this system) pkginfo
> -l SMColdap - version 2.4.30 pkginfo -l SMCossl - version 1.0.1c
> pkginfo -l SMCdb47 - version 4.7.25.NC
>
> IDK how this would affect the system, but based on your suggestion
> there is some difference in the package as the "pkginst" name on one
> system is SMCdb, and SMCdb47 on the other while the version is exactly the same.
I have no idea. I abandoned Slowaris years ago... I abandoned using BDB with OpenLDAP last year. However, no one has reported any issues like this. It really appears to be a bug in BDB rather than OpenLDAP however.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 2 months
Re: (ITS#7554) mdb_cursor_put() flag combinations broken
by hyc＠symas.com
h.b.furuseth(a)usit.uio.no wrote:
> Full_Name: Hallvard B Furuseth
> Version: mdb.master bbb27cde4af7
> OS:
> URL:
> Submission from: (NULL) (193.69.163.163)
> Submitted by: hallvard
>
>
> mdb_cursor_put() and mdb_cursor_del() sometimes tests the
> flags argument with '==' or '!=' instead of '&'. That breaks
> flag combinations like MDB_CURRENT | MDB_RESERVE:
The cursor_put flags are not combinable. The doc already says only one value
may be set. Closing this ITS.
>
> grep -n 'flags [!=]= MDB_' mdb.c
> 4884: if (flags != MDB_CURRENT && (key->mv_size == 0 || key->mv_size >
> MDB_MAXKEYSIZE))
> 4900: if (flags == MDB_CURRENT) {
> 4962: if (flags == MDB_CURRENT) {
> 4977: if (flags == MDB_CURRENT)
> 4992: return (flags == MDB_NODUPDATA) ? MDB_KEYEXIST : MDB_SUCCESS;
> 5025: if (flags == MDB_CURRENT) {
> 5274: if (flags != MDB_NODUPDATA) {
>
> The mdb_cursor_del() test is not a problem yet since only
> one flag is valid, but it'll become broken if more flags
> get supported.
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 2 months
(ITS#7554) mdb_cursor_put() flag combinations broken
by h.b.furuseth＠usit.uio.no
Full_Name: Hallvard B Furuseth
Version: mdb.master bbb27cde4af7
OS:
URL:
Submission from: (NULL) (193.69.163.163)
Submitted by: hallvard
mdb_cursor_put() and mdb_cursor_del() sometimes tests the
flags argument with '==' or '!=' instead of '&'. That breaks
flag combinations like MDB_CURRENT | MDB_RESERVE:
grep -n 'flags [!=]= MDB_' mdb.c
4884: if (flags != MDB_CURRENT && (key->mv_size == 0 || key->mv_size >
MDB_MAXKEYSIZE))
4900: if (flags == MDB_CURRENT) {
4962: if (flags == MDB_CURRENT) {
4977: if (flags == MDB_CURRENT)
4992: return (flags == MDB_NODUPDATA) ? MDB_KEYEXIST : MDB_SUCCESS;
5025: if (flags == MDB_CURRENT) {
5274: if (flags != MDB_NODUPDATA) {
The mdb_cursor_del() test is not a problem yet since only
one flag is valid, but it'll become broken if more flags
get supported.
10 years, 2 months
Re: (ITS#7553) mdb_cursor_del() crash when deleting last key
by hyc＠symas.com
dw(a)botanicus.net wrote:
> Full_Name: David Wilson
> Version: master
> OS: OS X
> URL: http://pastie.org/7114932
> Submission from: (NULL) (109.149.47.172)
>
>
> Please find attached a crash repro affecting current MDB master, where seeking
> to the last database key followed by double mdb_cursor_del() results in a
> crash.
Thanks for the report, fixed in mdb.master/master
>
> The program supplied crashed with NULL+0x10 pointer dereference, however another
> crash triggered by the same set of steps results in a separate stack trace:
>
>
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_PROTECTION_FAILURE at address: 0x0000000100fffff0
> 0x00000001005e566c in memmove$VARIANT$sse3x ()
> (gdb) bt
> #0 0x00000001005e566c in memmove$VARIANT$sse3x ()
> #1 0x0000000101dd81d0 in mdb_node_del (mp=0x10111ce20, ksize=10, indx=52758) at
> mdb.c:5582
> #2 0x0000000101dda308 in mdb_cursor_del (mc=0x10202ed80, flags=0) at
> mdb.c:6422
> #3 0x0000000101dd51a4 in _cffi_f_mdb_cursor_del (self=0x10111ce20,
> args=0x10111ce20) at _cffi__xfbbb05bexaac0bea3.c:648
> #4 0x000000010008bd77 in PyEval_EvalFrameEx ()
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 2 months
