mdb_dcmp() returns EINVAL if no dupsort function is set. I'm not sure it does the caller any favors in doing so: It claims (A>B && B>A) if it gets called with both (A,B) and (B,A). That can confuse the caller, e.g. a sort function.
I expect the sensible results without a dcmp function are 0 or a crash. Plus to document this clearly.
If that EINVAL is useful, i.e. if there are times when the user wishes to ask if a dupsort function is set, then mdb can grow an mdb_dbi_info(env, dbi, &result) function to describe the database: Name, flags, sort funcs.