Re: thread pools, performance

Howard Chu wrote: > Finally we got to running the actual authrate tests, which yielded a > peak rate > of 4526 auths/second with 40 client threads. The rate declined from > there as > more clients were added; AD clearly isn't capable of handling very many > concurrent sessions. Correction, the peak rate was with only 24 client threads. The rate dropped to 4400/sec and stayed there as more client threads were added after that point. > It's enlightening to look at the actual CPU time used during the > import tasks. > For ldifde on W2K3 we got: > > time ldifde.exe -i -f examp3.ldif -h -q 8 > 261.10u 140.73s 4:23:46.85 2.5% > > For slapadd on FC6 we got: > > time .slapadd -f slapd.conf.slam -q -l example.ldif.1mil > 260.75u 80.86s 7:05.17 80% > > One interesting part here is that the amount of user CPU time is nearly > identical in both cases. That implies that both slapadd and ldifde are > doing > about the same amount of work to parse the input LDIF. Eh, I take that back. The slapadd time includes both parsing and all of the BDB manipulation, while the ldifde time just includes parsing and encoding to BER. The database manipulation time in the server isn't reflected at all in these numbers. So really there's not much comparable in these figures at all... > Comparing the rest of the time isn't really fair since it seems that > ldifde > just feeds data into a running server using LDAP, while slapadd simply > writes > to the DB directly. I guess for the sake of fairness we'll have to > time an > OpenLDAP import using ldapadd next. For completeness we would have to time both ldapadd against AD and ldifde against OpenLDAP. Since ldifde only exists for Windows we'd have to do those runs all on Windows, against a Windows build of OpenLDAP. I guess that can come later; we already know that ldapadd in OpenLDAP is now pretty well optimized. One interesting feature of ldifde is that it also supports multiple threads, unlike our ldapadd/ldapmodify. Of course, the MS implementation is pretty braindead: if you have your entire tree in a single LDIF file, and try to import it with multiple threads in parallel, it will fail because there's no check to make sure that the thread importing a parent entry completes before the threads that import child entries start. So, to take advantage of the multithreading, you need to break out all of the parent entries into a separate file, import them single-threaded, and then do all the children in a separate invocation. It seems that whoever added that feature to ldifde didn't really think about how Directories work, or what's actually useful for a directory admin.