--001a114390bed95e010567661549 Content-Type: text/plain; charset="UTF-8"
Some more details:
Transaction read only pages are running out in mdb_rpage_get(). MDB_txn. mt_rpages[0] is at 4096 and thus MDB_TXN_FULL is returned. This happens after around 100K mdb_cursor_del() calls.
How can we get millions of deletes to work in a single transaction?
It's an essential feature to us and I am happy to contribute under some guidance.
So, here's my hope: given that the cleanup involves only consecutive keys, we could consider a bulk delete function (delete an entire range from position 1 to position 2). This should be much more efficient. Basically, for most of the data, there's no need to operate on the node level. Deleting entire leaf pages should be relatively simple (rebalance on the parent branch page)? Not sure if rebalancing could also cope with cutting off entire branches (superefficient if that would work).
Thanks,
Markus
--001a114390bed95e010567661549 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; charset= =3Dus-ascii"><meta name=3D"Generator" content=3D"Microsoft Word 14 (filtere= d medium)"><style><!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.E-MailFormatvorlage17 {mso-style-type:personal-compose; font-family:"Calibri","sans-serif"; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 70.85pt 2.0cm 70.85pt;} div.WordSection1 {page:WordSection1;} --></style></head><body lang=3D"DE" link=3D"blue" vlink=3D"purple"><div cla= ss=3D"WordSection1"><p class=3D"MsoNormal"><span lang=3D"EN-US">Some more d= etails:</span></p><p class=3D"MsoNormal"><span lang=3D"EN-US">Transaction r= ead only pages are running out in mdb_rpage_get(). MDB_txn. mt_rpages[0] is= at 4096 and thus MDB_TXN_FULL is returned. This happens after around 100K = mdb_cursor_del() calls.</span></p><p class=3D"MsoNormal"><span lang=3D"EN-U= S"> </span></p><p class=3D"MsoNormal"><span lang=3D"EN-US">How can we get m= illions of deletes to work in a single transaction?</span></p><p class=3D"M= soNormal"><span lang=3D"EN-US">=C2=A0</span></p><p class=3D"MsoNormal"><spa= n lang=3D"EN-US">It's an essential feature to us and I am happy to cont= ribute under some guidance.</span></p><p class=3D"MsoNormal"><span lang=3D"= EN-US">=C2=A0</span></p><p class=3D"MsoNormal"><span lang=3D"EN-US">So, her= e's my hope: given that the cleanup involves only consecutive keys, we = could consider a bulk delete function (delete an entire range from position= 1 to position 2). This should be much more efficient. Basically, for most = of the data, there's no need to operate on the node level. Deleting ent= ire leaf pages should be relatively simple (rebalance on the parent branch = page)? Not sure if rebalancing could also cope with cutting off entire bran= ches (superefficient if that would work).</span></p><p class=3D"MsoNormal">= <span lang=3D"EN-US">=C2=A0</span></p><p class=3D"MsoNormal"><span lang=3D"= EN-US">Thanks,</span></p><p class=3D"MsoNormal"><span lang=3D"EN-US">Markus= </span></p></div></body></html>
--001a114390bed95e010567661549--