--047d7bfcf1deec63470514165cd2 Content-Type: text/plain; charset=UTF-8
My Testing does show it is not worth it for the data size in the db.
Also there is a race condition using sendfile on linux that could happen. It is avoidable but it would require changes to the protocol to avoid it which I am not willing to do.
So dismiss my patch with extreme prejudice
On Wed, Apr 15, 2015 at 10:37 PM, Howard Chu hyc@symas.com wrote:
rouzier@gmail.com wrote:
Full_Name: James Rouzier Version: OS: URL: ftp://ftp.openldap.org/incoming/james-rouzier-150331.patch Submission from: (NULL) (192.222.129.11)
I am working on a proof of concept of a simple key value store service using lmdb as the back end. I am using sendfile to avoid an extra copy. I accomplished this by creating function to extract get the mmap ptr from the env so I can calculate the offset of data.
I'm quite reluctant to return the mmap pointer. In a WRITEMAP environment that invites all manner of undetectable/unpreventable corruptions.
Have you actually measured the performance difference between using sendfile() here and plain write() or send()? The kernel doesn't physically copy the written data from user space to kernel space, it just maps in a particular page. Since LMDB data is already mmap'd, the kernel has very little work to do since the relevant pages are already present in the kernel's page tables.
Ultimately sending data over TCP involves byte-by-byte iteration anyway, since TCP checksums must be computed on all outgoing data. Doesn't seem to me like sendfile can be a very significant optimization here.
I figured that function might be useful to the bigger community.
If it does not fit in the bigger vision of lmdb it is fine not to include it.
I, James Rouzier, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--047d7bfcf1deec63470514165cd2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr">My Testing doe= s show it is not worth it for the data size in the db.<br><br><div>Also the= re is a race condition using sendfile on linux that could happen.<br>It is = avoidable but it would require changes to=C2=A0the protocol to avoid it whi= ch I am not willing to do.<br><br>So dismiss my patch with extreme prejudic= e<br><br><br></div></div><div><div><div class=3D"gmail_extra"><br><div clas= s=3D"gmail_quote">On Wed, Apr 15, 2015 at 10:37 PM, Howard Chu <span dir=3D= "ltr"><<a href=3D"mailto:hyc@symas.com" target=3D"_blank">hyc@symas.com<= /a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:= 0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><a href=3D"mailto:r= ouzier@gmail.com" target=3D"_blank">rouzier@gmail.com</a> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: James Rouzier<br> Version:<br> OS:<br> URL: <a href=3D"ftp://ftp.openldap.org/incoming/james-rouzier-150331.patch"= target=3D"_blank">ftp://ftp.openldap.org/incoming/james-rouzier-150331.pat= ch</a><br> Submission from: (NULL) (192.222.129.11)<br> <br> <br> I am working on a proof of concept of a simple key value store service usin= g<br> lmdb as the back end.<br> I am using sendfile to avoid an extra copy.<br> I accomplished this by creating function to extract get the mmap ptr from t= he<br> env so I can calculate the offset of data.<br> </blockquote> <br> I'm quite reluctant to return the mmap pointer. In a WRITEMAP environme= nt that invites all manner of undetectable/unpreventable corruptions.<br> <br> Have you actually measured the performance difference between using sendfil= e() here and plain write() or send()? The kernel doesn't physically cop= y the written data from user space to kernel space, it just maps in a parti= cular page. Since LMDB data is already mmap'd, the kernel has very litt= le work to do since the relevant pages are already present in the kernel= 9;s page tables.<br> <br> Ultimately sending data over TCP involves byte-by-byte iteration anyway, si= nce TCP checksums must be computed on all outgoing data. Doesn't seem t= o me like sendfile can be a very significant optimization here.<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <br> I figured that function might be useful to the bigger community.<br> <br> If it does not fit in the bigger vision of lmdb it is fine not to include i= t.<br> <br> <br> I, James Rouzier, hereby place the following modifications to OpenLDAP Soft= ware<br> (and only these modifications) into the public domain. Hence, these<br> modifications may be freely used and/or redistributed for any purpose with = or<br> without attribution and/or other notice.<br> <br> <br> <br><span><font color=3D"#888888"> </font></span></blockquote><span><font color=3D"#888888"> <br> <br> -- <br> =C2=A0 -- Howard Chu<br> =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"= http://www.symas.com" target=3D"_blank">http://www.symas.com</a><br> =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland= sun.com/hyc/" target=3D"_blank">http://highlandsun.com/hyc/</a><br> =C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p= roject/" target=3D"_blank">http://www.openldap.org/project/</a><br> </font></span></blockquote></div><br></div> </div></div></div><br></div>
--047d7bfcf1deec63470514165cd2--