GitLab

at 2014-10-01T20:35:12+01:00

Add MDB_ROBUST

Free write mutex earlier in txn_reset0

MDB_ROBUST: Use MDB_PID_T. Drop a wmutex variable.

Must do dlist_free before releasing Wmutex

Use SysV semaphores instead of POSIX

Since they can cleanup after themselves on process exit

Use robust mutexes by default

Making it optional on pthreads just complicates things;
they're always robust on other implementations

Don't use -fPIC for static lib

ITS#7952 .gitignore mdb_dump/load

ITS#7953 .gitignore *.lo

ITS#7955, #7671 fix MDB_PREV_DUP

ITS#7956 fix compact of empty env

ITS#7841 trivial cleanup

TRIVIA - lmdb: clean testdb-dir while "make test".

ITS#7959 Check fstat return value

Signed-off-by: Jean-Christophe DUBOIS <jcd@tribudubois.net>

ITS#7961 fix txn init

More fallout from 4d02c741b120786df1b87ee9ed49c1d3f9bc7522

ITS#7959 fix prev commit

fstat returns -1, not an errno. Move code into its own function
for reuse.

ITS#7969 LMDB: volatile & __synchronize().

Globally shared fields of meta-data are not 'volatile'.

ITS#7970 LMDB: Critical Heisenbug

Inconsistent reading & SIGSEGV due to the race condition.

ITS#7971 LMDB: clarification in mdb_txn_renew0().

ITS#7961,#7987 Re-fix txn init.

More fallout from 4d02c741b120786df1b87ee9ed49c1d3f9bc7522:
Don't modify another thread's write txn. Reinit me_txn0 fully.

Simplify recent changes a bit.

Fix robust mutexes - repair mti_txnid.

Broken by e3b6c359a935283c21e1fd7d03b790d87d53b933, if
commit() crashes in write_meta before setting mti_txnid.

Clean up SysV semaphore code.

Change MDB_LOCK_FORMAT to differ from Posix semaphores/mutexes.
Do not use array/member length 0 for MDB_txninfo.mt2.

Fix comments

mdb_strerror: Silence warning @ Windows

Fix WIN32 -> _WIN32

For ITS#7789: Ensure mapsize >= pages in use.

Check new mapsizes against mm_last_pg.  Move
mdb_env_init_meta0() so it can set mm_last_pg earlier.

Fix SysV semaphores - repair mti_txnid.

Similar to 443a7e40c4280cfd07cbf80c5de62ea072b136e3.

ITS#7994 Access to current transaction ID.

I, David Barbour, hereby place the following modifications to OpenLDAP
Software (and only these modifications) into the public domain. Hence,
these modifications may be freely used and/or redistributed for any
purpose with or without attribution and/or other notice.

ITS#7994 fix prev commit

Strip docs promising behaviors that we don't promise to hold.
Strip MDB_txnid_t typedef.

ITS#7987 fix excessive space for single write txn

C90-compatible code for SysV semaphores

Clarify mdb_dbi_open doc

The concurrency restriction is on threads within a single process.
Multiple processes can use mdb_dbi_open without affecting each other.

Try to avoid an invalid datafile after failed init

make clean: remove *.lo

Fix MDB_APPEND doc

We actually compare the new key against the last key, to prevent
corruption.

Hack for potential ext3/ext4 corruption issue

Use regular fsync() if we think this commit grew the DB file.

Fix prev commit for env_sync0

Revert "Fix prev commit for env_sync0"

This reverts commit 985bbbbdd5d64e57f55249ffdeb7c08035b240b2.

Revert "Hack for potential ext3/ext4 corruption issue"

This reverts commit 0018eeb2c3b2239c30def9d47c9d194a4ebf35fe.

Simpler fdatasync hack

Note MDB_SAFE_FDATASYNC

Revert "Note MDB_SAFE_FDATASYNC"

This reverts commit 293d6bb47f7f0d54a45d57ce2be3a9595e806dc0.

Revert "Simpler fdatasync hack"

This reverts commit 9585c012335b2a28fbfdd267bcdeeeea91dc0f08.

fdatasync hack, again

Check for ext3/ext4 fs, then check kernel version.

ITS#8021 env_sync is invalid in RDONLY env

Tweak conditionals for fdatasync hack

More cleanup for fdatasync hack

ITS#8021 doc: Don't mix MDB_WRITEMAP + non-WRITEMAP

Make SysV semaphores robust. Cleanup MDB_ROBUST.

Whitespace (align with mdb.RE)

Fix potential null deref (coverity)

mdb_env_set_flags(): Reject CHANGELESS flags.

Reverts part of a2ac10107e2fb845c4a38a339239063ec4407d84.
MDB_ROBUST needed to be accepted, but that flag is gone.

ITS#7971 Fix reader allocation and me_numreaders

Fix EACCES description

Support MDB_NOSYNC on mdb_txn_begin()

Also support NOMETASYNC on txn_begin

Happy New Year

Happy New Year

ITS#8066 fix mdb_load with large values

ITS#8062 fix uninit'd cursor index

mdb_txn_renew(): Clear error from previous txn

Renumber MDB_TXN_* flags, simplify flag code

Fix per-txn MDB_NOMETASYNC, MDB_NOSYNC.

Fallout from 4d02c741b120786df1b87ee9ed49c1d3f9bc7522. The flags
were ignored: mdb_txn_renew0/begin cleared and/or did not set them.

Cleanup MDB_env.me_txn0.

More fallout from 4d02c741b120786df1b87ee9ed49c1d3f9bc7522.

Reformat mdb_dbi_open() doc for clarity

Cleanup mtest* and sample-*.

Fix mdb_txn_begin(&read-only txn) calls. Test mdb_env_set_maxreaders().
Rename DBI open/close functions. Move mdb_dbi_close() out of txn.

ITS#8062 fix rebalance

(Probably fixes the ITS, definitely fixes a bug) when collapsing
the root page, fixups of other cursors was incomplete.

ITS#8062 also handle subcursors

ITS#8106 retry writes on EINTR

ITS#8109 fix mdb_cursor_del0 on empty DB

ITS#8117 fix INTEGERDUP compare

ITS#8117 cleanup prev commit

Revert "ITS#8117 cleanup prev commit"

This reverts commit ce7ba0ceb69fd2cd56d557c1a877d046ec9f246c.

Revert "ITS#8117 fix INTEGERDUP compare"

This reverts commit ed13d4aa59cf06a1bb6b096fe19af63fc6f8737a.

ITS#8117 better fix

Don't change mc_dbx because we would need to undo the change if
the txn aborts. Make the fix (for get) match existing code for put.

ITS#8117 Fix MDB_INTEGERDUP keysize doc + md_dcmp

mtest*.c: Fix MDB_NOOVERWRITE, plug cursor leak

Set/clear mp_pad, md_pad (MDB_DUPFIXED data size).

mdb_xcursor_init1(): md_pad is only used when MDB_DUPFIXED.
mdb_page_split():    Copy mp_pad too. Used by mdb_page_list().

mdb_txn_renew(): Drop pointless diff from mdb.RE.

Makes the code friendlier to "git cherry-pick".

ITS#8157 mdb_txn_renew0(): init after error checks

ITS#8156 Fix MDB_MAXKEYSIZE doc

Clarify mdb_env_open() doc of 'mode' param

Use ftok() for semget(), and only lower mode bits

Bump version to 0.9.15

(not really, since this is the dev/non-release branch)

CHANGES only belongs in mdb.RE

Add MDB_USE_POSIX_MUTEX.

So far just to help aligning mdb.master and mdb.RE.
We'll make it an option later.

Fix mdb_mutex_t, add mdb_mutexref_t, drop MDB_MUTEX().

Fixes the types of Windows HANDLEs, they became HANDLE*.
MDB_MUTEX() took a magic r/w arg which it's nice to get rid of.

Restore support for Posix semaphores

ITS#7969 Wrap unportable __sync_synchronize in #if

mdb_env_get_flags(): Hide internal flags

Use mdb_cmp_long() for FREE_DBI

mdb_drop0(): Omit scanning DUPSORT sub-DB leaves

mdb_txn_begin() cleanup

Simpler mdb_txn_commit().

mt_env is always set.
Commit(mt_child) resets mt_child, so parent need not.

Simpler mdb_node_shrink()

Simpler mdb_drop().

MDB_DBI_CHANGED(,MAIN_DBI) is never true.

Simpler flag/DBI checks and MDB_DEBUG

Add DB_USRVALID, to avoid 'dbi == FREE_DBI' tests

More ESECT declarations

ITS#8190 fix cursor EOF bug

ITS#8181 Verify that records are/aren't DBs.

Except we don't catch the user passing F_SUBDATA to
mdb_cursor_<put/del>, like an internal LMDB call.

Simpler mdb_node_add()

Factor me_metas[toggle] out to mdb_env_pick_meta()

ITS#8200 Fix mdb_midl_shrink() usage, return void

Move code into mdb_txn_end(). Was mdb_txn_reset0.

Side effects:
* Clean txn up a bit even before freeing it.
* Tweak DEBUG output at txn end. Add DEBUG after commit(writer).

Catch most uses of finished/parent txns.

* Add MDB_TXN_FINISHED, MDB_TXN_HAS_CHILD, MDB_TXN_BLOCKED.
* Clear mt_numdbs in writers, for TXN_DBI_EXIST() to catch.
  We already do in readers.

Cleanup: Drop !mt_dbxs hack, use MDB_TXN_FINISHED

ITS#7377 Always notice env error on txn startup.

Move the check to the end of txn startup.  Catches env
breakage which happens while the new txn waits for a lock.

ITS#7377 Catch mdb_put() to blocked/read-only txns

...early enough that txn state is left unchanged.

Kill magic numbers for NUM_METAS, databases

Simpler mdb_env_write_meta()

ITS#8192 fix reference to EINTR on WIN32 from ITS#8106

ITS#8192 cleanup unused label on WIN32

ITS#8192 fix typo

Bump version to 0.9.16

ITS#8067 add ssize_t typedef for MSVC

ITS#8069 char* strings on Windows are ASCII

Another MDB_APPEND doc tweak

Missed this in 7ce29b9edbdaf34b7aeb545324008ed4dff62952

Doc tweak - MDB_RESERVE / DUPSORT incompat

For those people who insist on ignoring the obvious.

ITS#7377 mdb_env_init_meta(): Catch calloc error

ITS#8237 fix ITS#7589 regression

ITS#8256 Create install directories if needed

ITS#8221 don't merge branch pages needlessly

ITS#8221 cleanup prev commit

ITS#8221 fix typos

ITS#8259 fix usage ordering

Fix typo in mdb_tassert().

No effect on current code.

ITS#8258 fix rebalance/split

The tree height can also increase during rebalance, not just shrink.
This can happen if update_key needs to split a parent branch page.

ITS#8264 fix cursor_del cursor tracking

Some destination fixups need to happen immediately after nodes
are moved, before rebalancing

ITS#8263 fix cursor tracking in cursor_put

ITS#8263 streamline prev patch

ITS#7771 fix cursor tracking on fake pages

node_del shifts nodes around, cursors pointing at fake pages
need to have their mc_pg[0] corrected.

ITS#7771 more for prev commit

ITS#8270 win32: fix conversion error with MSVC

Fix MDB_NOSYNC,MDB_NOMETASYNC thinko

Add MDB_NO_ROBUST to disable Robust Mutexes

Use MDB_USE_ROBUST not MDB_NO_ROBUST

ITS#8299 fix page_merge cursor fixup

The parent's mc_ki has changed. We need to fix that up as well
in other cursors.

ITS#8238 fix DUPFIXED page_split

Parent mc_ki wasn't adjusted if new_indx was > split point

ITS#8300 fix rebalance after node_move

ITS#8258, ITS#7829 fixes checked parent index to see if we were moving
from a left neighbor. Should have just checked to see if current index
was 0, meaning we added on the left. (Parent index may not tell us
anything meaningful after a nested rebalance.)

ITS#8300 fix node_move

Don't adjust other cursors when we added a node on the right.

ITS#8300 more for node_move fixups

When moving a node from the right neighbor, a different
adjustment is needed.

ITS#8300 simplify

ITS#8300 more for node_move

When moving a node from one page to another, make sure other cursors'
parent index gets adjusted too.

ITS#8304 fix page_merge

Was using the pre-touch dst page pointer instead of the touched page

Tweak robust mutex detection for glibc

ITS#8310 fix mdb_rebalance cursor fixup

When collapsing root, must adjust entire cursor stack

ITS#7209 Default MDB_DSYNC = O_SYNC if no O_DSYNC

ITS#8311 fix page_split from update_key

Check for top of stack. Usually the cursor only has height 1 when
calling page_split, but not always.

ITS#8311 add comment

ITS#8312 Fix loose pages in commit(nested txn)

ITS#8313 more for ITS#8062

dummy flags must be init'd due to 3d46d550

ITS#8315 fix ovpage_free

Keep dirty_room sync'd with dirty_list

ITS#8316 cursor fixup in page_merge

Deleting the merged page requires bumping down other ki's
of the page's parent.

ITS#7992 assume Windows paths are UTF-8

Refix root split check from 5da67968afb599697d7557c13b65fb961ec408dd

Silence some valgrind uninit warnings

ITS#8300 more for prev commit

Just tell explicitly which direction we moved/merged from

ITS#8316 more for 60c34d14ec3ceb0762d620eef77db73dd1e911dc

ITS#8321 Fix del/dupsort

When deleting a dupsort key, if other cursors pointed at that key,
set them to uninit'd, not EOF. They no longer have anything to
point at.

ITS#8321 don't skip fixups on splitting cursors

Adjustments can't be skipped, in recursive calls each level must
fixup their own level.

ITS#8321 fix mdb_cursor_chk()

It was reporting spurious errors due to uninit'd cursors

ITS#8321 fix mdb_cursor_shadow()

Set a valid txn so that cursor fixup code works on the shadows

ITS#8321 fix mdb_cursor_put

Ignore sub-cursors that shouldn't be fixed up

ITS#8321 track temporary cursors

In rebalance/split operations, temporary cursors need to be visible
to propagate fixups

ITS#8321 simplify page_split fixups

ITS#8321 reorganize page_split fixups

DUPFIXED fixups needed to occur after separator update.
MDB_RESERVE handling moved after split fixup.

ITS#8319 mdb_load: explain readline and mdb_cursor_put errors

ITS#8320 mdb_load: fix loading data from simple text files

mdb_load -T was supposed to read escaped text, but 21b51cb7 "Add mdb_load"
made it read hex.

CURSOR_TMP_[UN]TRACK() -> WITH_CURSOR_TRACKING()

ITS#8323 Fix nested commit

Must remove our spilled pages from parent's dirty list

ITS#8321 deinit empty cursors

Always unset C_INIT flag if the cursor's target DB has been deleted

ITS#8321 Fix mdb_cursor_set

Always reinit mc_pg[0] if cursor is not C_INITIALIZED
It might have a stale value when using nested txns

ITS#8321 mdb_put cursor needs tracking too

ITS#8321 page_touch - don't fixup the cursor we just touched

ITS#8321 More cursor fixup

Based on page_touch fixup from ITS#7594 but expanded:
make sure sub-cursors agree with main cursors.

Cleanup C_DEL flag usage

Only set it if the cursor's current position was deleted

ITS#8321 fix ambiguity in cursor_put fixup

After delete/add of a node, other nodes may no longer be
pointing at the data they intended. This can confuse subsequent
fixups.

ITS#8168 Allow passing AR to make

This is helpful when the ar executable is named differently, for
example with an arch specific prefix.

ITS#8169 Allow passing mandir to make install

The motivation for this change is my distribution moving to a multiarch
layout. While the architecture specific stuff (binaries, libraries,
etc.) is installed under /usr/${host}/{bin,lib,...} architecture-independent
data should still be installed to /usr/share/.

ITS#8321 cleanup unused var

from da67af1a4bf1a6d3f5d173d9daf74bb4ec66175a

ITS#8324 incremental DB file growth for Windows

ITS#8169 more for prev patch

Missed the mkdir command

ITS#8169 more Makefile tweaks

Use all the same vars as main OpenLDAP makefiles

ITS#8330 Fix robust mutex detection for glibc 2.10 and 2.11

pthread_mutexattr_setrobust and pthread_mutex_consistent are provided since 2.12 not 2.10:
https://sourceware.org/git/?p=glibc.git;a=commit;h=402cd98775db1478f64c9b0dbe00664b89eb2773
https://sourceware.org/git/?p=glibc.git;a=commit;h=78ee21859939ff75ccf8bbe00499b0c462df2e2d

ITS#8324 additional tweaks

VirtualAlloc is only needed with WRITEMAP. Regular writes
already extend the mmap implictly.

ITS#8334 Fix MDB_APPENDDUP vs. rewrite(single item)

ITS#8336 fix page_search_root assert on FreeDB

Let "illegal" branch pages thru on the FreeDB - the condition
is only temporary and will be fixed by the time rebalance finishes.

ITS#8324 fix for read-only envs

mdb_dbi_open(): Catch strdup failure

ITS#7992 cleanup

check for utf8_to_utf16 failures

Add Getting Started doc

Fix typos

MDB_VL32 preparation

Use 64 bit types consistently. This keeps database structures the
same size for MDB_VL32 and native 64 bit builds.

MDB_VL32 support 64 bit DBs on 32 bit processors

Note that we're prepping for 1.0

ITS#8324 fix for WRITEMAP

We called FlushViewOfFile with (map,mapsize) which worked fine
when we had allocated the entire map already. Now we have to make
sure to only flush as much as was actually written. Add a numpgs
argument to tell how much to flush in env_sync0().

Pass cursor to mdb_page_get(), mdb_node_read().

No change in behavior.

Prep for next commit: C_WRITEMAP, C_ORIG_RDONLY.

No change in behavior.

mdb_page_alloc(): Use original snapshot of freeDB.

I can't help thinking this should be safer, and useful somewhere.

MDB_VL32 - track overflow pages too

Otherwise they'll fill up the txn's pageref list when traversing
a DB with lots of overflow records.

MDB_VL32 - fix for Win32 read-only envs

We can't map with MEM_RESERVE because that requires write access
to the underlying file/section. Mapping with the default (MEM_COMMIT)
requires that we don't map past the end of the file.

MDB_VL32 - fix prev commit

Only tweak length for read-only envs

MDB_RESERVE doc

Add mdb_put text to mdb_cursor_put description for people who
fail to draw logical conclusions.

MDB_VL32 - Fix me_rpmutex usage

MDB_VL32 - prevent mixing with non-VL32 builds

MDB_VL32 tweak prev commit

ITS#8342 MDB_VL32/WIN32 - close file mapping handle in env_close

MDB_VL32 fix cursor_unref - ignore cursor with empty stack

Fixups for env_copy with large files

wsize was being truncated to 32bits on Windows.
Only try to write 1GB at a time on Windows64;
larger writes fail with ERROR_WORKING_SET_QUOTA.

Add MDB_PREV_MULTIPLE

Logical counterpart to GET_MULTIPLE, NEXT_MULTIPLE

ITS#8346 free last txn0->mt_rpages in env_close

ITS#8347 fix off-by-one in VL32 purge

ITS#8346 fix typo in prev commit

lmdb.h Caveats: Reserved vs. actual mem/disk usage

ITS#8324 fix c4e31434c7773ee54f2ffdeb545e5740f56492a1

Actually, there is no guarantee that Windows will map newly written
data, so we need VirtualAlloc even for non-WRITEMAP.

WIN64 needs off_t redefined too

Update WRITEMAP doc

ITS#8355 fix subcursors

make sure C_DEL gets reset in subcursor after it moves.

ITS#8324 fix a6ccef73ed288271f9b5871909d14a2e481c81ae

Removing the WRITEMAP test dropped this into the MDB_VL32 code path,
which was wrong.

MDB_VL32 more for 1ba5adb2ec262405f9207d6015d4f29eea548d25

fix 32bit overflow in set_mapsize

ITS#8363 Fix off-by-one in mdb_midl_shrink

MDB_VL32 change overflow page scan

Just check the requested page, don't worry about any other pages

Fix MDB_VL32 mdb_cursor_count()/entrycount types

Happy New Year

ITS#7992 Fix memleak in prev change

ITS#8324 Map NTAPI result codes to WIN32 codes

Tweak Win32 errmsg buffer

Tweak MDB_PREV_MULTIPLE for uninit'd cursor

MDB_VL32 - increase max write txn size

ITS#8393 fix MDB_GET_BOTH on non-dup record

mdb_drop optimization

If we know there are no sub-DBs and no overflow pages, skip leaf scan.

ITS#8406 fix xcursors after cursor_del

Don't leave them uninit'd if they now point at a valid DUP node

ITS#8412 fix NEXT_DUP after cursor_del

Comment ovpage code in mdb_cursor_put()

ITS#8424 init cursor in mdb_env_cwalk

ITS#8339 Solaris 10/11 robust mutex fixes

Check for PTHREAD_MUTEX_ROBUST_NP definition (this doesn't work
on Linux/glibc because they used an enum). Zero out mutex before
initing.

Init "locked" flag for SysV semaphores

mdb_env_setup_locks: Plug mutexattr leak on error

ITS#8209 MDB_CP_COMPACT fixes

Handle errors.  Fix cond_wait condition so mc_new
is the sole control var.  Drop specious cond_waits.
Do not look at 'mo' while copythr writes it.

ITS#8209 MDB_CP_COMPACT: Handle empty or broken DB

Preserve DB flags (use metapage#1) when main DB is empty.
Fail if metapage root != actual root in output file.

ITS#8209 Tweak previous fixes

Some _aligned_malloc() doc seems to think arg NULL = user error.
Don't know if posix_memalign() pointer is defined after failure.

Add error MDB_PROBLEM, replace some MDB_CORRUPTED

When problem is most likely in txn, not on disk.

Factor out MDB_SIZE_MAX, MDB_FMT_Y, MDB_FMT_Z

Factor some MDB_VL32-related '#if's out to macros

Add MC_OVPG() + MC_SET_OVPG(), NEED_CMP_CLONG(), MDB_CURSOR_UNREF().

Fix size_t/formats -> mdb_size_t for MDB_VL32

Use mdb_size_t for line numbers in mdb_load

This matches the mdb_size_t entry counts.

Fix MDB_INTEGERKEY doc of integer types

MDB_VL32: Match mdb_size_t type with format modifier.

When using format modifier "ll" or "I64", use the matching
type unsigned <long long / __int64> rather than uint64_t.

Refactor mdb_page_get()

MDB_VL32: Switch to mdb_size_t formats PRIu64 & co

Drop macro Y=MDB_FMT_Y, add Yu/Yd=MDB_PRIy(). Replace
Y"d..." -> Yd"...", Y"u..." -> Yu"..." / MDB_SCNy(u)"...".

mdb_size comments

Doc fixes: VALID_FLAGS, mm_last_pg, mt_loose_count

Comment MDB_page

MDB_VL32 plug rpage leak

mdb_cursor_set wasn't unref'ing as intended.

ITS#8481 make shared lib suffix overridable

Fix mdb_page_list() message

Clean up MDB_USE_ROBUST #defines

Make explicit and default nonzero equivalent. Parenthesize.

ITS#8489 reset cursor EOF flag in cursor_set

It usually gets done anyway, but one of the fastpath shortcuts
bypassed this step.

Silence warning for initializer "mdb_copy my = {0}"

1st struct member was not a scalar.

ITS#7682 F_NOCACHE: Allow error, skip any O_DIRECT

We can run without F_NOCACHE if it fails. And we do not know
what combining it with O_DIRECT means, if a system has both.

Drop spurious Errcode() call

Clean up strange fcntl result check

...and check !MDB_CLOEXEC in an 'if' rather than '#if'
to match its non-zero usage.

ITS#7992 Tighter utf8_to_utf16(), fix errcodes

The 0xFFFD check seems due to misleading MultiByteToWideChar() doc.
Bad UTF-8 gives 0xFFFD in the output string, not the return value.

Factor filename handling out to mdb_fname_*()

No change in functionality, except needs less mallocing.

Move opening files to mdb_fopen()

No change in functionality.

ITS#8505 Set FD_CLOEXEC for me_mfd,env_copy as well

ITS#8505 Protect parent from fork()-pthread_exit()

ITS#8505 Clarify fork() caveat, mdb_env_get_fd(), flock->fcntl.

Only set me_mfd if needed. Drop unused read access.

More for ITS#8406

Revert excess cursor fixup

More for ITS#8406

xcursor fixup depends on init state

Factor out refreshing sub-page pointers

Fix mdb_env_open() with MDB_VL32

Init me_rpmutex independently of MDB_NOLOCK.
Plug leaks on mdb_env_open() failure.
Tweak mdb_env_close0() to handle the rearranged mdb_env_open().

MDB_VL32 mdb_page_get(): Set MDB_TXN_ERROR on failure.

Note functions which must set MDB_TXN_ERROR on failure

Other functions depend on them to do so.
For mdb_node_read(), instead remove such a dependence.

doxygen cleanup

MDB_CP_COMPACT comments

Cleanup: Add flag DB_DUPDATA, drop DB_DIRTY hack

ITS#8542 mdb_dbi_open(): Protect mainDB cursors

MDB_VL32 cleanup: MDB_env.mm_mapsize type

ITS#8355 cleanup

Drop ~C_EOF, pointless after 8c215aa970215a58ee0df458813c0405ad27a6e9

Clean up and comment C_UNTRACK

Don't use it as a "cursor is tracked" hint in mdb_pages_xkeep().
It's been harmless so far, but would break after mdb_cursor_copy().
Checking m0 directly short-circuits better anyway.

ITS#7377 Catch mdb_cursor_sibling() error

ITS#8504 mdb_env_copyfd2(): Don't abort on SIGPIPE

Return EPIPE instead.

ITS#8504 Fix prev commit: mc_error, #ifdef SIGPIPE

Never clear mc_error, we could lose a failure in the other thread.

Mention MDB_PREV_MULTIPLE along with MDB_NEXT_MULTIPLE

Doxygen fixes. Use DISTRIBUTE_GROUP_DOC.

- DISTRIBUTE_GROUP_DOC makes doxygen give several fields the
  same doc: mn_hi + mn_lo in MDB_node.
- With mdb_mutex_t + mdb_mutexref_t, instead split them up.
- Don't hide a doxygen #name inside double quotes.

More MDB_node doc

ITS#8554 kFreeBSD is like BSD

Doesn't have POSIX robust mutexes - GNU userland on BSD kernel

ITS#8558 fix mdb_load with escaped plaintext

ITS#8557 fix mdb_cursor_last

Optimize mdb_page_search_root(PS_LAST) when cursor is already near
last position, ignoring C_EOF flag for now.

Tweak cursor_next C_EOF check

Allow C_EOF flag to be stale

ITS#8557 fix prev commit

More C_EOF tweaks

Happy New Year

Fix f8ce8a82717ddefdc912fa47c07f1bdee2a3336b

GET_MULTIPLE was broken

Further fix f8ce8a82717ddefdc912fa47c07f1bdee2a3336b

Fully revert the change to GET_MULTIPLE

Workaround VL32 cursor refcounting miscount

Don't try to deref cursor page if txn's pagelist is empty

ITS#8582 keep mutex at end of struct

since it's variable size on Linux/glibc

ITS#8582 MDB_LOCK_VERSION = 2 due to format change

Store lock ID instead of pathname in lockfile

This limits the namespace which the user can meddle with for
POSIX semaphores and Windows mutexes.  Their names change a
bit, they no longer have fixed lengths.

Simplify mdb_hash_val() -> mdb_hash()

Simpler usage since it's only called once, rename to match new usage,
and drop 3 loop pessimizations which were optimizations 20 years ago.

Clear any struct padding in idbuf

ITS#8582 Fill in MDB_LOCK_FORMAT

Attempt to stop liblmdb variants compiled with conflicting options
from using the lockfile at the same time and thus breaking it.

ITS#8622 fix xcursor after cursor_del

Re-fix 6b1df0e4c7fadd21d1233d7157229b2d89ccaa04 from ITS#8406

MDB_VL32

Use the same size dirty list for both 64 and 32 bit.

Fix Android recognition

The official macro is __ANDROID__; ANDROID may or may not be defined.

Add new MDB_RPAGE_CACHE def

Separate 32/64 dependency from rpage buffer mechanism

RPAGE_CACHE is now dynamically selectable

Behavior is controlled by MDB_REMAP_CHUNKS envflag
Remapping is always enabled in MDB_VL32

Optimization for mdb_rpage_get()

The caller already knows if it's using an overflow page, so
pass the number of expected pages in instead of having to
map the page first and check the count there.

Rename mdb_env_close0() -> mdb_env_close_active()

Hopefully we'll remember now what goes in which function.

Page-level encryption support

Currently encrypts all but the meta pages
Still needs to store/retrieve the initialization vector

Add test code for remap / encryption

Plug some information leaks

Zero out decrypted pages before freeing them.
Do proper init on reused loose pages.

Save/restore enc IV in page 0

Get flags from $LMDB_FLAGS when MDB_TEST

Now we don't need to tweak the code of callers to test encryption.

Trivial encryption when $LMDB_FLAGS = "e"

Data format change: add txnid to page header

checkpoint - moving overflow page headers

Moving headers outside of overflow page.

Scaled back on overflow page work

Still keeping page header at top of overflow page for now

Fix last commit - sizeof(pgno_t) and C90 compat

Fix broken mdb_page_get(ovpage) optimization

Move misplaced OVERFLOW_NOTYET code

...so it would set pgno,txnid in the MDB_dovpage, not the actual ovpage

Fix mdb_enctest()

Accept a partial ovpage.  I.e. decryption of the beginning
should not depend on the data at the end.
Make the key and IV less regular. (Divisor 67 has period>64.)

mdb_page_touch: Always set MDB_TXN_ERROR on error

Verify final loose_count in mdb_txn_commit()

Mismatch may indicate that pages leaked or got used twice
in the same snapshot.

Use mdb_page_loose() more

Protect freelist at end of mdb_freelist_save()

Skip unnecessary mdb_page_get()s when spilling

Do not bring in pages merely to see if they should be skipped.

Drop unnecessary searches in the spill lists

Check with IS_MUTABLE() if an MDB_page is spilled, instead of
searching spill lists.  When unspilling, skip parent spill lists.

Divide page flags in P_ADM_FLAGS and the rest

P_ADM_FLAGS flags are tied to the page, even through page_loose-alloc.
OTOH mdb_page_split() should only duplicate the relevant flags.

This is just the code for the feature, P_ADM_FLAGS is 0 for now.
We'll need it for P_HIDESPILL later.

Demo - move P_DIRTY to P_ADM_FLAGS, page_dirty()

...to have something substantial to test P_ADM_FLAGS with,
at least until we drop the P_DIRTY flag.

mp_txnid = page state, avoids searching dirty_list

In nested txns, check mp_txnid to see which txn dirtied the page.
This change will also let us remove the P_DIRTY flag, and keep
some flags in (dirty page).mp_txnid if we should need it.

Drop P_DIRTY, and MDB_WRITEMAP dirty/spill lists

mt_workid = mt_txnid when WRITEMAP, so dirty pages == "spilled"
pages and mdb_page_flush() does nothing.

Fix 418ea8b94e SEGV on spilled ovpg

Don't spill an ovpg that is referenced in a cursor

ITS#8699 more for cursor_del ITS#8622

Set C_DEL flag on reinit'd subcursor

Keep MC_OVPG() == NULL when not MDB_REMAPPING()

Needed since 418ea8b94e14567ba2be9f9772f38f563a0d7e9c introduced
option MDB_REMAP_CHUNKS.

Less hungry mdb_pages_xkeep()

Skip pages that are dirty only in an ancestor txn

Doc fixes

Drop mdb_page_get() arg 'level', it's always NULL

mdb_page_get() can ignore the toplevel spill list

...when we search dirty list before instead of after spill list.

Rename mdb_env_getflags->envflags, clarify options

The old name resembled existing function mdb_env_get_flags().

Expose MDB_ENCRYPT flag

ITS#8704 add MDB_PREVMETA flag to mdb_env_open

used to open the previous meta page, in case the latest one
is corrupted

From https://github.com/LMDB/lmdb/pull/12

ITS#8704 Add PREVMETA to envflags()

ITS#8704 Add MDB_PREVMETA support to tools

Drop always-false MDB_WRITEMAP test when IS_DIRTY_NW()

Unref ovpages when deleting them

ITS#8704 Fix PREVMETA, rename to PREVSNAPSHOT

and enforce exclusive access to environment. Also fix txn_begin/pick_meta
to use correct meta page, and reset the flag after successful commit.

mdb_dbi_open tweak

Always save the terminating NUL of the incoming DB name
Stop using the old mdb_open/mdb_close function names
 .. should consider renaming mdb_stat/mdb_drop to mdb_dbi_... too.

Add mdb_cursor_is_db()

Return 1 if the cursor is pointing to a named DB record

Doc tweaks

Note about DB names being C strings

ITS#8722 fix FIRST_DUP/LAST_DUP cursor bounds check

ITS#8728 fix MDB_VL32 freeing overflow page

XCURSOR_REFRESH() fixups/cleanup

* Check NUMKEYS(), similar to f34b61f9471d1c03fe0517b9d817c50c920e378a
  "ITS#8722 fix FIRST_DUP/LAST_DUP cursor bounds check".
* Move XCURSOR_INITED() into XCURSOR_REFRESH().  This adds a check in
  mdb_cursor_put, below /* converted, write the original data first */.
* Factor mc_ki[] out to XCURSOR_REFRESH().
* Replace an mc_pg[] with mp which is equal (mdb_cursor_del0).

Tweak ITS#8722 fix: Use XCURSOR_REFRESH()

This checks XCURSOR_INITED() and fixes the mn_flags check.

ITS#8612 Fix Solaris builds with liblmdb

This patch fixes liblmdb builds on Solaris and derivatives by defining
_POSIX_PTHREAD_SEMANTICS

Add -a append option to mdb_load

To allow reloading of custom-sorted DBs from mdb_dump

ITS#8760 fix regression in 0.9.19

Add mdb_drop tool

ITS#8324 More for Win32 NTDLL junk

Use GetProcAddress at runtime, avoid buildtime NTDLL link issues

ITS#8819 can't use fakepage mp_ptrs directly

Happy New Year

ITS#8831 move flag init into readhdr

Avoid stomping on flags from 1st readhr invocation

ITS#8844 use getpid() in mdb_env_close0()

ITS#8756 remove loose pg from dirty list in freelist_save

ITS#8891 fix M$ WINAPI typedefs

ITS#8908 DOC: GET_MULTIPLE etc don't return the key

Unnecessary since these are DUPs, the key will always be the same

ITS#8857 document mdb_cursor_del does not invalidate the cursor

Happy New Year!

Happy New Year

IDLs for VL32 must be same size as for 64bit

ITS#8969 tweak mdb_page_split

Bump up number of keys for which we use fine-grained splitpoint search

ITS#8978 Fix mdb_env_open2() failing when getting handle for NTDLL.dll

Always call GetModuleHandleW() with Unicode string, as mdb_fopen() is
calling CreateFileW() already.

ITS#9007 don't free loose writemap pages

Broken in ITS#8756

ITS#9068 fix backslash escaping

mdb_load wasn't properly inserting escaped backslashes into the data.
mdb_dump wasn't escaping backslashes when generating printable output.

ITS#8704 add missing opt flags in prev commit

ITS#9118 add MAP_NOSYNC for FreeBSD

Silence spurious fallthru warning

Happy New Year!

ITS#9155 lmdb: free mt_spill_pgs in non-nested txn on end

ITS#9017 LMDB: improve Windows sync commit perf

ITS#9017 LMDB: allow using fixed file size on Windows

ITS#9017 LMDB: fix off_t usage on Windows

ITS#9017 doxygen comment for MDB_FIXEDSIZE

ITS#9017 cleanup Windows off_t

ITS#9278 fix robust mutex cleanup for FreeBSD

FreeBSD 11 supports robust process-shared POSIX mutexes,
but requires them to be explicitly destroyed before munmap

Preliminary raw partition support

Autodetects that a block device is being used.

More RAWPART support

Use mmap to read and initialize the meta pages, raw device
may not support read/write syscalls.

Add mdb_env_set_pagesize()

Fix fallthru warning

Fix rawpart flag collision

ITS#9017 fixes for encryption

ITS#9364 rework crypto API

And add support for per-page checksums. Reserve space for checksum
at tail of page. Pass pgno+txnid as IV input for encryption.

ITS#9364 add error code texts

ITS#9364 add sample for authenticated encryption

ITS#9364 tweak sample

Cheat with OpenSSL 1.1 internal structures to avoid malloc/free per page

ITS#9364 tweak crypto sample again

ITS#9364 use crypto table properly

ITS#9364 Add crypto support to all tools

Using dynamically loaded crypto modules

ITS#9364 Add docs for crypto modules

ITS#9364 fix doxygen comment

Support db_pagesize in mdb_load

Since setting was added in de08119a296a4e8569fa04d7bfcbc7b3c2b41f2f

ITS#9376 Fixes for repeated deletes with xcursor

On DUPSORT DBs, must initialize xcursor regardless of whether
caller requested its data. Also in cursor_prev must check whether
cursor index is still within range before using it.

ITS#9371 Fix unknown size of void compilation errors

ITS#9372 Default to using POSIX semaphores on MacOS

Unless robust is specified (then use SysV)

Happy New Year!

ITS#9461 refix ITS#9376

Was setting C_DEL flag gratuitously

ITS#9461 fix typo

ITS#9469 - Typo fixes

ITS#9376 simplify

ITS#9500 fix regression from ITS#8662

ITS#9496 fix mdb_env_open bug from #8704

Broken in 3585a1eb977326c7e178c53f4eef1fdc81b46e63

ITS#9574 add mdb_drop to .gitignore

ITS#9618 fix Windows WRITEMAP flush

Revert back to using standard FlushViewOfFile/FlushFileBuffers
to sync data with WRITEMAP mode on Windows

ITS#9385 fix using MDB_NOSUBDIR with nonexistent file

ITS#9861 fix readonly regression from #9017

ITS#9910 fix undefined MDB_FDATASYNC on MacOSX

Broken by d85fe32 ITS#9372

ITS#9524 fix loose page tracking

Fixes commit 0e17ba43a818f6bdab7759586e247bae12692c25, loose pages
that have been dropped from dirty list should no longer be counted.

ITS#9910 fix typo

in ad3f8367b0ef913dbff074ddd0fdcbcaa86fdbb0

ITS#9030 - Use sys/cachectl.h rather than asm/cachectl.h on mips

It also contains the cacheflush function declaration.

ITS#9919 - Mark infrequently used functions as cold rather than manually putting them in a separate section

ITS#9806 LMDB page_split: key threshold depends on page size

32 was chosen for page size of 4KB. Not large nough for 16KB pages.

ITS#9916 liblmdb: use alternate MDB_page2 struct for some accesses

fakepage pointers are only guaranteed to be 2-byte aligned. Use a
2-byte aligned struct definition when referencing 2-byte page members
if a page pointer possibly points to a fakepage.

ITS#9920 must account for size of authentication data

When computing amount of free space or fill factor in a page

ITS#9961 LMDB: fix MSVC error

ITS#10095 partial revert of ITS#9278 2fd44e325195ae81664eb5dc36e7d265927c5ebc

The patch was incorrect and introduced numerous race conditions.
The original problem was a FreeBSD bug, subsequently fixed:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269277

ITS#10125 mdb_load: fix cursor reinit in Append mode

ITS#10137 LMDB: Allow users to define MDB_IDL_LOGN

crypto demo: fixup OpenSSL 3 compat

More crypto fixups

ITS#9378 Add explicit replay logging

Logs essential ops so they can be replayed.
Ignores read ops for now.

ITS#9378 Add replay tool

Reads a replay log and executes all the write ops

ITS#9037 mdb_page_search: fix error code when DBI record is missing

Use the more relevant MDB_BAD_DBI instead of MDB_NOTFOUND error code

ITS#10198 Win32 mdb_strerror - stop passing "ignored" parameter

The M$ docs say the parameter is ignored, but it actually isn't,
and will cause a SEGV if the pointed memory isn't an init'd va_list.

ITS#10212 LMDB: init txnid for read-only DBs

ITS#10212 LMDB: better fix

ITS#10222 LMDB: Updated mdb_dump man page

The -a option to mdb_load makes the previous text obsolete.

LMDB: tweak mdb_load.1 manpage

Add missing -a option to Synopsis

ITS#9378 don't try to print oversized keys in debug msg

ITS#10244 win32: Fix passed ptr type

From dfb3bbed656132456001c5aaca246fd4430e5ef5 ITS#9017

ITS#9920 lmdb: fix page_split of encrypted page

ITS#9920 lmdb: cleanup prev commit

LMDB: Add a separate mlm_setup0() to separate loading from setup

Another fix for OpenSSL 3.0+

Should rewrite this module to use libsodium instead.

ITS#9920 crypto: use libsodium instead of OpenSSL

Note: even though the same algorithms were used, the data produced by
this module doesn't work with the OpenSSL-based module and vice versa.

ITS#9920 lmdb: fix data races in rpage decryption

ITS#9920 lmdb: fix page offset when converting subpage to real page

offset needs to account for additional space at page tail

Move the crypto module helpers into main liblmdb.

Just makes things easier than using the individual mdb_env_set APIs.

Delete unused module code

Cleanup redundant include

Keep module code in a separate object file

That way programs that don't use these functions won't needlessly
depend on -ldl

ITS#10275 mdb_load: add -Q option to use NOSYNC for faster loading

ITS#10296 lmdb: fix fdatasync on MacOS

Patch from kero, with corrections by hyc

ITS#10296 lmdb: fix typo in prev commit

ITS#10024 lmdb: fix MDB_PREVSNAPSHOT txnid initialization

More for rawpart

It was ignoring the mapsize recorded in the meta page, and was
redundantly mapping the meta pages.

ITS#10342 lmdb: fix potential memleak in child txn_begin

ITS#10346 lmdb: fix compacting copy with large values

ITS#10355 lmdb mplay: don't assign to stdin/stdout

ITS#10396 lmdb: fix mdb_cursor_del0 with multiple DUPSORT cursors

Use the correct stack index when adjusting other cursors pointing to
the affected page and the DB has a subDB. Broken in ITS#8406.

ITS#9564 lmdb: fix race condition freeing spilled pages at end of transaction

Move the freeing of the spilled pages list inside the protection of the
transaction mutex so there is no race condition with another transaction

ITS#8988 lmdb: make all keys and data 2-byte aligned

This is an incompatible DB format change. Previously, only a single
pad byte was used if key+data was an odd size. Now, separate padding
is used for both odd keys and odd data sizes.

Note: also compile with -DMISALIGNED_OK=0 on x86 to avoid all
misaligned pgno accesses.

ITS#10421 mdb_load: check for malicious input

ITS#10419 LMDB: add support for NetBSD

ITS#10420 LMDB: add support for Haiku

ITS#9224 lmdb: add support for PREPARE/2-phase commit

ITS#10395 LMDB: Allow multiple nested read txns from a write txn

ITS#10054 lmdb: break up large overflow page writes

We were ignoring MAX_WRITE when a single overflow page was larger.
Now we guarantee that all writes are <= MAX_WRITE (1GB).

ITS#10054 lmdb: more for prev commit

ITS#8386 lmdb: be a bit more precise that mdb_get retrieves data in intro.doc

ITS#8824 mdb_dump: cleanup check for MDB_SUCCESS

ITS#10108 mdb_dump: fix comment about -a option

No change in behavior, manpage doc was always correct.

ITS#10454 lmdb: can't rely on O_DSYNC on MacOS

So we must explicitly sync writes to the metapage.

ITS#8739 lmdb: No fdatasync on FreeBSD 11.0 and older

ITS#10275 mdb_load: add -Q to usage message

ITS#9223 LMDB: add incremental dump/load

ITS#9223 LMDB: cleanup prev commit

Fix age check of meta pages.
Use MAX_WRITE consistently in copy functions.

ITS#8165 LMDB: add soname, lib version

ITS#8165 more for prev commit

Fixup .so symlinks, fixup install, add MacOSX linker option

ITS#8174 LMDB: fixes for mdb_drop(MAIN_DBI)

mdb_dump: fix symbol clash on Windows

rename byte() to dobyte()

ITS#9027 LMDB: expose address of memory map

Returned in MDB_envinfo.

ITS#8590 LMDB: Use F_SETNOSIGPIPE on OS X

It seems like OS X delivers SIGPIPE to the whole process, instead of the generating thread,
as on other UNIXes. Therefore the current code sometimes works, but mostly doesn't, since
the call to sigwait doesn't happen quickly enough.

Instead of masking SIGPIPE, we use an OS X specific fcntl to disable SIGPIPE for that particular fd.

ITS#9223 more cleanup for prev commit

ITS#9619 LMDB: fix mdb_env_copy2(MDB_COMPACT) for overflow pages

Wasn't resetting txnid in page header

ITS#8579 LMDB: all descriptors should have O_CLOEXEC

There's nothing useful that can be done with them if they're kept around.

ITS#8192 LMDB: define new error codes to avoid errno abuse

And avoid confusion with Windows error codes

ITS#8192 fix prev commit

ITS#8250 LMDB: add option MDB_ROTXN_RESET flag for txn_begin

Create a readonly txn in a reset state, it cannot be used until
mdb_txn_renew() is called.

ITS#8803 LMDB: add tool option to disable locking

For use with environments that use their own locking mechanism
instead of LMDB's internal locking.

Revert "ITS#8250 LMDB: add option MDB_ROTXN_RESET flag for txn_begin"

This reverts commit 4cd8e451679511c8fbeff68f8a97cbe7b1268b48.
Cost/benefit makes no sense.

ITS#7772 LMDB: fix sub-page growth

Check for available space before deciding to grow the sub-page.
The test program will still leave some unused space, but after the
first iteration the extra space will be used instead of growing more.

ITS#10454 LMDB: fix prev commit for Mac OSX + WRITEMAP

ITS#9388 LMDB: fix mdb_stat page counts for dupsort DBs

ITS#8335 LMDB: reject MDB_MULTIPLE put with 0 items

Also check for overflow of count and size.

ITS#10203 LMDB: add lmdb.pc to Makefile

ITS#9291 LMDB: quick sanity check on env open

Validate root page numbers and page sum against file size while opening env.

Creating LMDB 1.0 release branch

ITS#9619 LMDB: more for prev commit

Also needed to set overflow pgno.

ITS#9011 LMDB: add mdb_txn_flags()

Whitespace cleanup

ITS#9011 LMDB: fix typo in prev commit

Add changes file for LMDB 1.0 series

ITS#8605 - spelling fixes

ITS#9009 LMDB: update some version and copyright stamps

Merge branch 'mdb.RE/1.0' into ITS9009

Quanah Gibson-Mount pushed to branch master at openldap / OpenLDAP

Commits:

28 changed files: