(ITS#8781) "connection_read(248): no connection!" on 2.4.44
by mramitsingh85@gmail.com
Full_Name: Amit kumar singh
Version: 2.4.44
OS: CentOS release 6.9 (Final)
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (14.141.104.38)
Issue on replication and get below error.
Nov 30 17:03:55 host slapd[709]: connection_read(346): no connection!
Nov 30 17:03:55 host slapd[709]: connection_read(346): no connection!
Nov 30 17:03:55 host[709]: connection_read(362): no connection!
I was searching issue and found that fixed ref: ITS#6548 on 2.4.27. But strange
thing is that it comes up again in 2.4.44. Not sure If I am correct but need
your help to look into it.
6 years
Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by hyc@symas.com
Markus Junginger wrote:
> Thanks for the report. Unfortunately your suggested fix means the rpage
> test will always be skipped on read-only cursors, which will break the
> rpage tracking if the txn is still alive.
>
>
> Could you maybe explain a bit about rpages? Why does mdb_cursor_close need to
> unref pages, but not mdb_cursor_renew? If closing a transaction does not clean
> up rpages, I would have expected mdb_cursor_renew would have to do. And why
> does setting MDB_VL32 affect rpages in the first place? To my understanding,
> MDB_VL32 is about using 64 bits also on 32 bit CPUs, so a differences on page
> handling seem surprising at first sight.
Think about how LMDB works.
LMDB was originally written for use on 64bit CPUs, with 64bit virtual address
spaces. It expects to map an entire DB into virtual memory using a single mmap
call.
Obviously this cannot be done when trying to use a DB larger than 2GB on a
machine with only a 32bit virtual address space.
When using MDB_VL32 to access a 64bit DB on a 32bit CPU, the DB cannot fit
into a single mmap. It must be mapped in chunks, and the chunks must be
mapped, unmapped, and remapped on demand. The rpages thus need to be tracked,
to know which are currently mapped. None of this rpage tracking exists in the
native non-VL32 build because it's completely unnecessary.
Also note that the MDB_VL32 feature is not officially released. You're free to
use it if you want, but there is no expectation that it is suitable for
general use. When you use unreleased features, you're expected to know how to
support yourself. When you use LMDB at all, you're expected to know how
virtual addresses work.
Ideally, you should quit using obsolete 32bit CPUs.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6 years
Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by hyc@symas.com
Markus Junginger wrote:
> Thanks for your fast response.
>
>> Thanks for the report. Unfortunately your suggested fix means
>> the rpage test will always be skipped on read-only cursors, which
>> will break the rpage tracking if the txn is still alive.
>
> Does that mean it's not trivial to fix and will take some time?
Correct. And I'm still on holiday this week, won't be looking deeper into this
for a while.
> It's a last outstanding issue for a release, so I guess we have to find a
> work around until it's properly fixed?
Yes. Best is to use mdb_txn_reset/renew instead of txn_abort.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6 years
Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by markus@greenrobot.de
--94eb2c123fa4650778055ee78f61
Content-Type: text/plain; charset="UTF-8"
>
> Thanks for the report. Unfortunately your suggested fix means the rpage
> test will always be skipped on read-only cursors, which will break the
> rpage tracking if the txn is still alive.
Could you maybe explain a bit about rpages? Why does mdb_cursor_close need
to unref pages, but not mdb_cursor_renew? If closing a transaction does not
clean up rpages, I would have expected mdb_cursor_renew would have to do.
And why does setting MDB_VL32 affect rpages in the first place? To my
understanding, MDB_VL32 is about using 64 bits also on 32 bit CPUs, so a
differences on page handling seem surprising at first sight.
Thanks,
Markus
--94eb2c123fa4650778055ee78f61
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
:1px solid rgb(204,204,204);padding-left:1ex">Thanks for the report. Unfort=
unately your suggested fix means the rpage test will always be skipped on r=
ead-only cursors, which will break the rpage tracking if the txn is still a=
live.</blockquote><div><br></div><div>Could you maybe explain a bit about r=
pages? Why does=C2=A0<span style=3D"font-size:12.8px">mdb_cursor_close=C2=
=A0</span>need to unref pages, but not mdb_cursor_renew? If closing a trans=
action does not clean up rpages, I would have expected mdb_cursor_renew wou=
ld have to do. And why does setting MDB_VL32 affect rpages in the first pla=
ce? To my understanding, MDB_VL32=C2=A0 is about using 64 bits also on 32 b=
it CPUs, so a differences on page handling seem surprising at first sight.<=
/div><div><br></div><div>Thanks,</div><div>Markus</div><div><br></div></div=
></div></div>
--94eb2c123fa4650778055ee78f61--
6 years
(ITS#8778) Phone numbers "-" and " " fail normalization
by h.b.furuseth@usit.uio.no
Full_Name: Hallvard B Furuseth
Version: 2.4.45, master
OS: Linux x86_64
URL:
Submission from: (NULL) (193.90.52.167)
Submitted by: hallvard
telephoneNumberNormalize() fails for a non-empty value
of only spaces+hyphens. The validate function,
printableStringValidate(), succeeds. They should agree.
I'll make Normalize return one space. That seems the
minimal change.
6 years
Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by markus@greenrobot.de
Thanks for your fast response.
> Thanks for the report. Unfortunately your suggested fix means
> the rpage test will always be skipped on read-only cursors, which
> will break the rpage tracking if the txn is still alive.
Does that mean it's not trivial to fix and will take some time?
It's a last outstanding issue for a release, so I guess we have to find a
work around until it's properly fixed?
Thanks,
Markus
6 years
Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by hyc@symas.com
markus(a)greenrobot.de wrote:
> Full_Name: Markus Junginger
> Version: mdb.master
> OS: Ubuntu
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (77.190.230.93)
>
>
> Read cursors may outlive a transaction to be renewed with another. Thus, cursors
> should be correctly closed even after their last transaction was aborted. But
> they access the (already freed) transaction during closing when MDB_VL32 is used
> (mdb.master branch).
>
> ASan captures this like this:
> ==19722==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100001b140
> at pc 0x000000ea2387 bp 0x7ffd36e3cd20 sp 0x7ffd36e3cd18
> READ of size 8 at 0x61100001b140 thread T0
> #0 0xea2386 in mdb_cursor_unref mdb.c:2082:18
> #1 0xec577b in mdb_cursor_close mdb.c:8538:3
>
> 0x61100001b140 is located 128 bytes inside of 250-byte region
> [0x61100001b0c0,0x61100001b1ba)
> freed by thread T0 here:
> #0 0x74f840 in __interceptor_cfree.localalias.0 (...)
> #1 0xe8ea98 in mdb_txn_end mdb.c:3380:3
> #2 0xe8ecc5 in mdb_txn_abort mdb.c:3405:2
>
>
> Possible fix for line mdb.c:2082 (applying what I found in mdb_cursor_close):
> - if (mc->mc_txn->mt_rpages[0].mid) {
> + if ((mc->mc_flags & C_UNTRACK) && mc->mc_txn->mt_rpages[0].mid) {
>
>
> Code to reproduce using MDB_VL32:
> err = mdb_txn_begin(env, nullptr, MDB_RDONLY, &tx);
> err = mdb_cursor_open(tx, dbi, &cursor);
> // We must have data so get will set C_INITIALIZED
> err = mdb_cursor_get(cursor, &key, &value, MDB_SET_KEY);
> mdb_txn_abort(tx);
> mdb_cursor_close(cursor); // <- still uses tx
Thanks for the report. Unfortunately your suggested fix means the rpage test
will always be skipped on read-only cursors, which will break the rpage
tracking if the txn is still alive.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6 years