https://bugs.openldap.org/show_bug.cgi?id=9736
Issue ID: 9736
Summary: pwrite bug in OSX breaking LMDB promise about the
maximum value size
Product: LMDB
Version: unspecified
Hardware: All
OS: Mac OS
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: renault.cle(a)gmail.com
Target Milestone: ---
Hi,
I was working with LMDB and found an issue when trying to write a value of
approximately 3.3GiB in the database, I dive into the LMDB source code of the
mdb_put method using the lldb debugger and found out that it was not related to
an issue in LMDB itself but rather a bug in the pwrite function of the Mac OS
libc implementation.
The pwrite function is given four parameters, the file descriptor, the buffer,
the count of bytes to write from the buffer and, the offset of where to write
it in the file. On Mac OS the count of bytes is a size_t that must be a 64bits
unsigned integer but when you call pwrite with a number bigger or equal to 2^31
it returns an error 22 (invalid argument). LMDB was returning a 22 error from
the mdb_put call and not an EINVAL because the error was cause by an internal
issue and not something catchable by LMDB.
I am not sure about what we can do, can we implement this single pwrite [1] as
multiple pwrite with counts smaller than 2^31 in a loop, just for Mac OS? Like
for Windows where we do specific things for this operating system too?
I also found this issue on the RocksDB repository [2] about a similar problem
they have with pwrite and write on Mac OS it seems. I understand that this is
not a real promise that LMDB is specifying but rather an "in theory" rule [3].
Thank you for your time,
kero
[1]:
https://github.com/LMDB/lmdb/blob/01b1b7dc204abdf3849536979205dc9e3a0e3ece/…
[2]: https://github.com/facebook/rocksdb/issues/5169
[3]: http://www.lmdb.tech/doc/group__mdb.html#structMDB__val
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10054
Issue ID: 10054
Summary: Value size limited to 2,147,479,552 bytes
Product: LMDB
Version: unspecified
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: louis(a)meilisearch.com
Target Milestone: ---
Hello,
According to the documentation[0], a database that is not using `MDB_DUPSORT`
can store values up to `0xffffffff` bytes (around 4GB).
In practice, under Linux, the actual limit is `0x7ffff000` though (2^31 - 4096,
so around 2GB).
This is due to the write loop in `mdb_page_flush`. The `wsize` value
determining how many bytes will be written can be as big as
`4096*dp->mp_pages`[1], and the number of overflow pages grows with the size of
the value put inside the DB.
The `wsize` is not split in smaller chunks in the case where there are many
overflow pages to write, and as a result the call to `pwrite`[2] does not
perform a full write, but only a "short" write of 2147479552 bytes (the maximum
allowed on a call to `pwrite` on Linux[3]).
This would be OK if the short write condition was handled by looping and
performing another `pwrite` with the rest of the data, but instead `EIO` is
returned[4].
There seems to be a related, but different issue on macOS when trying to
`pwrite` more the 2^31 bytes, that was already reported[5].
This issue was reported to me by a Meilisearch user because it causes their
database indexing to fail[6]. I had to investigate a bit because their setup
was peculiar (high number of documents in their database) and the `EIO` error
code is not very descriptive of the underlying issue.
I join a C reproducer of the issue that attempts to add a 2147479553 bytes
value to the DB and fails with `EIO` (decreasing `nb_items` to a smaller value
such as `2107479552` does succeed)[7].
Thank you for making LMDB!
Louis Dureuil.
[0]:
https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/lmdb.h#LL284…
[1]:
https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#LL3770…
[2]: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#L3820
[3]:
https://stackoverflow.com/questions/70368651/why-cant-linux-write-more-than…
[4]: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#L3840
[5]: https://bugs.openldap.org/show_bug.cgi?id=9736
[6]: https://github.com/meilisearch/meilisearch/issues/3654
[7]: https://github.com/dureuill/lmdb_3654/tree/main
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9431
Issue ID: 9431
Summary: back-mdb: Always have an equality index for
objectClass
Product: OpenLDAP
Version: 2.5
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Data storage backends require an equality index on objectClass to function
correctly. As this is a hard requirement it should be automatic with back-mdb.
Why this wasn't done in the past with other backends isn't exactly clear, it
may have been due to their requirements to have additional cache layers. That
however is not necessary with back-mdb.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9612
Issue ID: 9612
Summary: Change index_hash64 default to on
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Change the default value of index_hash64. By default this means slapd won't
run on a 32-bit CPU (It will continue to work on 32-bit OSes running on 64-bit
CPUs).
If someone needs to run slapd on a 32-bit CPU they can turn this option off.
In the documentation, mark the option as deprecated for eventual removal in a
future release.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9667
Issue ID: 9667
Summary: 2.6 to 2.7 upgrade documentation
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: documentation
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Need to document any upgrade information for going from 2.6 to 2.7
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9596
Issue ID: 9596
Summary: Python test suite
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: build
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
The bash test suite is extremely limited, hard to write for and slow. We can't
lose it as it is also portable, but something should be introduced for
developers/CI on more modern systems and increase coverage.
A Python 3 seed for one is in development in MR!347.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9398
Issue ID: 9398
Summary: Stale accesslog cookie due to unclean shutdown
Product: OpenLDAP
Version: 2.4.56
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
If slapd terminates uncleanly, a checkpoint will be lost on the accesslog db.
Depending on the syncprov overlay checkpoint settings (usually no checkpointing
is enabled on the accesslog db) this can cause the system to refuse engage in
replication at startup.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9967
Issue ID: 9967
Summary: Please register my company on the support page, again
(www.openldap.org)
Product: website
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: website
Assignee: bugs(a)openldap.org
Reporter: sjsong(a)aboutdap.kr
Target Milestone: ---
Howard Chu <hyc(a)openldap.org>
송상준 <sjsong(a)aboutdap.kr>
Hi,
you can submit a ticket against "website" in the ITS for this. Thanks.
송상준 wrote:
>
> Hello. openldap page administrater.
> My name is sang jun song.
>
> I want to register my company on the support page.
> I registered before, but it seems to have disappeared.
> Our company has been attending ldapcon since 2015.
>
> The sysmas employees who attended the LDAPCON in 2019 may remember me.
>
> Please register my company on the support page. Please contact me if you need additional information..
>
> thank you.
>
> Song.
>
> Registration phrase
>
> Seojindsa Co., Ltd. (Aboutdap Co., Ltd.)- Korea
>
> Provides consultancy, development, training and user support for OpenLDAP software in Korea.
>
> URL : seojindsa : www.seojindsa.kr
>
> ------------------------------------------------------
> (주) 어바웃답 기술영업팀 송상준 부장
> TEL. 010-9780-6746
> email: sjsong(a)aboutdap.kr
> homepage: www.aboutdap.kr
> -------------------------------------------------------
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
--
You are receiving this mail because:
You are on the CC list for the issue.