Re: AW: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by hyc＠symas.com
markus(a)greenrobot.de wrote:
>> When using MDB_VL32 to access a 64bit DB on a 32bit CPU, the DB cannot fit
>> into a single mmap.
>> [...]
>
> Long story short: I did not understand MDB_VL32 in the first place. I
> thought it would just enable 64 bit sizes.
Pointless, if you're still limited to 2GB. It would only eat up 4 extra bytes
overhead in every internal integer.
>> Ideally, you should quit using obsolete 32bit CPUs.
>
> Of course. Think about mobile and IoT however: Here, there are still 32 bit
> CPUs and it's not uncommon to even make 64 bit ARM CPUs operate in 32 bit
> mode to reduce binary size etc.
>
> I'd like to propose a different approach to 64 bits on 32 bit CPUs;
> something like this:
> #ifdef MDB_SIZE64
> typedef uint64_t mdb_size_t;
> ...
>
> Why? MDB_VL32 adds complexity and it would take additional coding and
> testing efforts to get it production ready.
If you had used the API as intended, with txn_reset/renew as I originally
suggested, there would be no problem.
>
> I think MDB_SIZE64 might be a better trade-off. Of course this would only
> work with smaller DBs files (< 2GB/4GB). But I don't see a huge scenario for
> 32 bit CPUs being able to work with huge files anyway.
The reason MDB_VL32 exists at all is because of exactly this - people wanted
to use 32 bit CPUs with 64 bit databases.
> The main reason I see
> to use 64 bits consistently across CPU architecture is binary compatibility.
> This is a big thing for Android (shipping apps with prefilled DBs). Also it
> enables getting DB files from 32 bit devices and open those on the desktop,
> etc.
>
> What do you think?
Bad idea. MDB_VL32 provides *full* compatibility between 32bit and 64bit CPUs
*regardless of filesize*. Your suggestion will only cause confusion, as people
wonder why their DB dies after reaching 2GB in size. It offers none of the
benefits of 64bit capability, while still paying all of the cost in terms of
storage overhead.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5 years, 3 months
AW: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by markus＠greenrobot.de
> When using MDB_VL32 to access a 64bit DB on a 32bit CPU, the DB cannot fit
> into a single mmap.
> [...]
Long story short: I did not understand MDB_VL32 in the first place. I
thought it would just enable 64 bit sizes.
> Ideally, you should quit using obsolete 32bit CPUs.
Of course. Think about mobile and IoT however: Here, there are still 32 bit
CPUs and it's not uncommon to even make 64 bit ARM CPUs operate in 32 bit
mode to reduce binary size etc.
I'd like to propose a different approach to 64 bits on 32 bit CPUs;
something like this:
#ifdef MDB_SIZE64
typedef uint64_t mdb_size_t;
...
Why? MDB_VL32 adds complexity and it would take additional coding and
testing efforts to get it production ready.
I think MDB_SIZE64 might be a better trade-off. Of course this would only
work with smaller DBs files (< 2GB/4GB). But I don't see a huge scenario for
32 bit CPUs being able to work with huge files anyway. The main reason I see
to use 64 bits consistently across CPU architecture is binary compatibility.
This is a big thing for Android (shipping apps with prefilled DBs). Also it
enables getting DB files from 32 bit devices and open those on the desktop,
etc.
What do you think?
Thanks,
Markus
PS.: Here's the full patch (adding MDB_SIZE64), which runs fine at least
with our test suite:
lmdb.h
@@ -187,7 +187,15 @@
# define MDB_FMT_Z "z" /**< printf/scanf format modifier for size_t */
#endif
-#ifndef MDB_VL32
+#if defined(MDB_VL32) && defined(MDB_SIZE64)
+#error either define MDB_VL32 OR MDB_SIZE64
+#endif
+#ifdef MDB_SIZE64
+typedef uint64_t mdb_size_t;
+# define MDB_SIZE_MAX SIZE_MAX /**< max #mdb_size_t */
+# define MDB_PRIy(t) PRI##t##64
+# define MDB_SCNy(t) SCN##t##64
+#elif !defined(MDB_VL32))
/** Unsigned type used for mapsize, entry counts and page/transaction IDs.
*
* It is normally size_t, hence the name. Defining MDB_VL32 makes it
5 years, 3 months
(ITS#8802) ldappasswd ppolicy
by matsl＠irf.se
Full_Name: Mats Luspa
Version: openldap-2.4.40+dfsg
OS: 3.16.0-4-686-pae #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) i686 GNU/Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (2001:6b0:27:cc:2740:e692:a5b1:4b0f)
Hello!
When you are using ppolicy password changed are recorded in pwdHistory
attribute.
ldappasswd can't be used due to that. It checks of some reason that pwdHistory
not exists before it changes that password. If pwdHistory exists then the
ldappaswd can't change the password.
Here's the log file:
2018-02-08T09:42:45+01:00 mailserver slapd[725]: bdb_modify_internal: replace
userPassword
2018-02-08T09:42:45+01:00 mailserver slapd[725]: bdb_modify_internal: replace
pwdChangedTime
2018-02-08T09:42:45+01:00 mailserver slapd[725]: bdb_modify_internal: add
pwdHistory
2018-02-08T09:42:45+01:00 mailserver slapd[725]: bdb_modify_internal: replace
pwdChangedTime
2018-02-08T09:42:45+01:00 mailserver slapd[725]: bdb_modify_internal: add
pwdHistory
2018-02-08T09:42:45+01:00 mailserver slapd[725]: bdb_modify_internal: 20
modify/add: pwdHistory: value #0 already exists
2018-02-08T09:42:45+01:00 mailserver slapd[725]: send_ldap_result: err=20
matched="" text="modify/add: pwdHistory: value #0 already exists"
/Regards Mats
5 years, 3 months
Re: (ITS#8801) Fix pending CSN queue
by Howard Chu
The problem was noticed because the accesslog overlay was invoking a spurious
slap_graduate_commit_csn() on a CSN that had already been graduated by the
underlying backend.
5 years, 3 months
(ITS#8801) Fix pending CSN queue
by hyc＠openldap.org
Full_Name: Howard Chu
Version: 2.4
OS:
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (83.136.45.75)
Submitted by: hyc
The CSN queue tracks in-progress write operations. It uses connID and opID to
identify a CSN queue entry that corresponds to a given operation. This is
error-prone in the case of syncrepl, which always uses the same connID and opID
for all of its internal operations. Should just use the Operation pointer, which
is always unique and unambiguous.
5 years, 3 months
