[Issue 9619] New: mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

List overview All Threads
Download

newer

older

[Issue 10518] New: Every write...

[Issue 8346] lmdb memory leak with...

openldap-its＠openldap.org

28 Jul 2021 28 Jul '21

6:09 p.m.

https://bugs.openldap.org/show_bug.cgi?id=9619

Issue ID: 9619 Summary: mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file Product: LMDB Version: 0.9.29 Hardware: All OS: Windows Status: UNCONFIRMED Severity: normal Priority: --- Component: liblmdb Assignee: bugs@openldap.org Reporter: kriszyp@gmail.com Target Milestone: ---

When copying an LMDB database with mdb_env_copy2 with the MDB_CP_COMPACT with mdb.master3, the resulting mdb file seems to be corrupt and when using it in LMDB, I get segmentation faults. Copying without the compacting flag seems to work fine. I apologize, I know this is not a very good issue report, as I haven't had a chance to actually narrow this down to a more reproducible/isolated case, or look for how to patch. I thought I would report in case there are any ideas on what could cause this. The segmentation faults always seem to be memory write faults (as opposed to try fault on trying to read). Or perhaps the current backup/copying functionality is eventually going to be replaced by incremental backup/copying anyway (https://twitter.com/hyc_symas/status/1315651814096875520). I'll try to update this if I get a chance to investigate more, but otherwise feel free to ignore/consider low-priority since the work around is easy.

-- You are receiving this mail because: You are on the CC list for the issue.

Show replies by date

openldap-its＠openldap.org

27 Aug 27 Aug

12:41 p.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

Quanah Gibson-Mount quanah@openldap.org changed:

What |Removed |Added ---------------------------------------------------------------------------- Keywords| |needs_review

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

14 Sep 14 Sep

12:47 p.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

Quanah Gibson-Mount quanah@openldap.org changed:

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

15 Apr 15 Apr

3:09 p.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

Howard Chu hyc@openldap.org changed:

--- Comment #1 from Howard Chu hyc@openldap.org --- Was it perhaps related to records with large values? Maybe same as ITS#10346.

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

23 Apr 23 Apr

4:08 a.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

Srinarasi bhargavabhat@gmail.com changed:

What |Removed |Added ---------------------------------------------------------------------------- CC| |bhargavabhat@gmail.com

--- Comment #2 from Srinarasi bhargavabhat@gmail.com --- Created attachment 1143 --> https://bugs.openldap.org/attachment.cgi?id=1143&action=edit Test to reproduce the corruption

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

4:09 a.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

--- Comment #3 from Srinarasi bhargavabhat@gmail.com --- I can still reproduce this corruption.

Here's what I'm doing: 1) Open the env and create a DB. 2) Write one entry with 2019 bytes of value and a 7 byte key. 3) Compact with `mdb_env_copy2(MDB_CP_COMPACT)` 4) Close the env. 5) Remove the old db, replace it with the compacted file. 6) Open the env and the db 7) Try to write a 4000 byte value to the key that I was using in step 2.

I get MDB_KEYEXIST in the test I'm attaching. But I have seen it crash in some cases.

I see that value being >2019 bytes is the tipping point. The DB is fine if I write any less than that.

I'm attaching the test I'm using.

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

6:45 a.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

Howard Chu hyc@openldap.org changed:

--- Comment #4 from Howard Chu hyc@openldap.org --- Thanks for the test case, fixed now in git 9aa3ac00567db8d5222abea46325a8b882937350

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

10:25 a.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

--- Comment #5 from Howard Chu hyc@openldap.org --- Created attachment 1144 --> https://bugs.openldap.org/attachment.cgi?id=1144&action=edit Testcase rewritten in C

For future reference, I had to rewrite the testcase from C++ to C because the C++ executable omitted a bunch of liblmdb's debug symbols.

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

29 Apr 29 Apr

10:56 a.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

--- Comment #6 from Srinarasi bhargavabhat@gmail.com --- Created attachment 1147 --> https://bugs.openldap.org/attachment.cgi?id=1147&action=edit A patch to assign pgno after compaction

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

10:56 a.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

--- Comment #7 from Srinarasi bhargavabhat@gmail.com --- I still get an assert `mdb_cassert(mc, mp->mp_pgno != pgno);` in `mdb_page_touch()` on `mdb_put()` after compaction in some of my tests. My guess is that it's because `mp_pgno` is not set to the new compact `pgno` in `mdb_env_cwalk()`. I have attached a patch which seems to solve the problem in my tests. But I don't understand the code well enough to know if I'm missing something.

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

1:31 p.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

--- Comment #8 from Howard Chu hyc@openldap.org --- (In reply to Srinarasi from comment #7)

...

I still get an assert `mdb_cassert(mc, mp->mp_pgno != pgno);` in `mdb_page_touch()` on `mdb_put()` after compaction in some of my tests. My guess is that it's because `mp_pgno` is not set to the new compact `pgno` in `mdb_env_cwalk()`. I have attached a patch which seems to solve the problem in my tests. But I don't understand the code well enough to know if I'm missing something.

Thanks, the patch makes sense.

Missed it because of some other in-progress work to remove the page headers from the overflow pages.

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

6 May 6 May

11:52 a.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

Quanah Gibson-Mount quanah@openldap.org changed:

--- Comment #9 from Quanah Gibson-Mount quanah@openldap.org --- mdb.master3:

• 9aa3ac00 by Howard Chu at 2026-04-23T11:44:34+01:00 ITS#9619 LMDB: fix mdb_env_copy2(MDB_COMPACT) for overflow pages

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

12:45 p.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

--- Comment #10 from Quanah Gibson-Mount quanah@openldap.org --- mdb.RE/1.0:

• 340f0459 by Markus Junginger at 2026-04-27T19:49:12+01:00 ITS#9291 LMDB: quick sanity check on env open

mdb.master3:

• 5639419b by Howard Chu at 2026-04-29T18:28:57+01:00 ITS#9619 LMDB: more for prev commit

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

30 Jun 30 Jun

12:46 p.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

Quanah Gibson-Mount quanah@openldap.org changed:

What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |VERIFIED

-- You are receiving this mail because: You are on the CC list for the issue.

openldap-its＠openldap.org

8:53 p.m.

New subject: [Issue 9619] mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3 produces corrupt mdb file

https://bugs.openldap.org/show_bug.cgi?id=9619

Quanah Gibson-Mount quanah@openldap.org changed:

What |Removed |Added ---------------------------------------------------------------------------- Resolution|TEST |FIXED

-- You are receiving this mail because: You are on the CC list for the issue.

Age (days ago)

1812

Last active (days ago)

openldap-bugs@openldap.org

14 comments

1 participants

tags (0)

participants (1)

openldap-its＠openldap.org