test001 still crashes if you insert assert(a->mv_size == sizeof(int) && b->mv_size == sizeof(int)); in mdb_cmp_int(). Using RE24 commit 98f70f1841f85d93 "ITS#8120 Move final...".
I don't see "same size" in the INTEGERDUP doc, it's looser than for INTEGERKEY integers. Or have I gone blind again? Anyway, it would be clearer to cut some doc and say data behave like INTEGERKEY keys. Ignoring that, though:
Looking yet again at the doc, mdb_dbi_open():INTEGERKEY says "typically" sizeof(int/size_t), I can't see it requires these sizes. Maybe you copied that from a suggestion from me back when I asked for clarifications.
"keys must all be of the same size" in the doc is unclear. I gather it should be "... in the DB's lifetime". But it can be read as "...at a given time": Write size_t-sized integers, empty the DB, then write int-sized integers. That does not work, since me_dbxs[].md_dcmp is updated.
"mc->mc_dbx->md_dcmp = mdb_cmp_clong;" looks wrong. That updates env->me_dbxs[], I believe. But:
If we mdb_txn_abort(), the txn has not been a no-op: It will require next txn and even existing txns to use size_t-sized INTEGERDUP data.
The assignment may be non-atomic. Another txn may be reading the pointer. I expect IL32P64 architectures are examples, and also function pointers can be quite large. (OpenLDAP expects void*-sized function pointers though.)