We are getting some clients which will occationally retrieve ~2.5M data in one search operation. If I understand correctly, that can block the thread running the search operation until the client-side socket accepts the data. So I'm wondering what to do, unless I misunderstand the above completely:
Are paged results reliable now? Browsing the mailinglists I see a lot of older problems with them. Also, what limit is reasonable to use? I imagine it makes sense to try to keep a page below the socket send buffer size.
Threads might be safer too, in case many slow clients do this at once.
Regarding the send buffer size, "/sbin/sysctl net.ipv4.tcp_wmem" currently says the default is 16384 bytes, while tcp_rmem (socket read buffer size) is 87380 bytes. How do I pick good defaults for these? We care about LDAP read operation speed but not write operation speed, so maybe it'd make sense to swap these two numbers at least for slapd.
Current stats: Dedicated LDAP server host, Linux x86_64. Max 8192 connections (ulimit -n 8192), 8G memory. 60-90 new connections and 300-500 requests per second. 1G *.bdb files for 340000 entries in 5 databases.
Hallvard B Furuseth wrote:
We are getting some clients which will occationally retrieve ~2.5M data in one search operation. If I understand correctly, that can block the thread running the search operation until the client-side socket accepts the data. So I'm wondering what to do, unless I misunderstand the above completely:
Is that really a problem? How often is "occasionally"? To me 2.5 MB does not sound so much to justify thinking about changing the client app in such a network- and data-specific way. Does it block other operations from different apps?
Ciao, Michael.
Michael Ströder writes:
Is that really a problem? How often is "occasionally"?
Don't know, and don't know.
To me 2.5 MB does not sound so much to justify thinking about changing the client app in such a network- and data-specific way.
OK, good. I've no experience with that kind of search result sizes myself. Though the client side change is just a config directive so far. getgrent() with nss_ldap. Others may come later.
Does it block other operations from different apps?
Don't know yet, that's what I was wondering about. Like I said I imagine it can, if the threads get blocked. We've just multiplied the server-side sizelimit with 200 to accomodate the change:-( So it's a jump in what degree the server is at the mercy of the clients.
Hallvard B Furuseth wrote:
Michael Ströder writes:
Is that really a problem? How often is "occasionally"?
Don't know, and don't know.
To me 2.5 MB does not sound so much to justify thinking about changing the client app in such a network- and data-specific way.
OK, good. I've no experience with that kind of search result sizes myself.
I can only speak of situations where I retrieve the whole directory (up to 300000 entries) for syncing. But this does not happen very often and my sync scripts call ldap_result() quite soon and process results as they come in.
getgrent() with nss_ldap. Others may come later.
Hmm, maybe that's what Volker Lendecke was talking about at LDAPcon 2007 regarding enumeration of groups. See his slides:
http://www.guug.de/veranstaltungen/ldapcon2007/slides/ldapcon_lendecke.pdf
Does it block other operations from different apps?
Don't know yet, that's what I was wondering about. Like I said I imagine it can, if the threads get blocked. We've just multiplied the server-side sizelimit with 200 to accomodate the change:-(
I guess user-specific limits won't help much.
Ciao, Michael.
Michael Ströder wrote:
Hallvard B Furuseth wrote:
Michael Ströder writes:
Is that really a problem? How often is "occasionally"?
Don't know, and don't know.
To me 2.5 MB does not sound so much to justify thinking about changing the client app in such a network- and data-specific way.
OK, good. I've no experience with that kind of search result sizes myself.
I can only speak of situations where I retrieve the whole directory (up to 300000 entries) for syncing. But this does not happen very often and my sync scripts call ldap_result() quite soon and process results as they come in.
getgrent() with nss_ldap. Others may come later.
Hmm, maybe that's what Volker Lendecke was talking about at LDAPcon 2007 regarding enumeration of groups. See his slides:
http://www.guug.de/veranstaltungen/ldapcon2007/slides/ldapcon_lendecke.pdf
Does it block other operations from different apps?
Don't know yet, that's what I was wondering about. Like I said I imagine it can, if the threads get blocked. We've just multiplied the server-side sizelimit with 200 to accomodate the change:-(
I guess user-specific limits won't help much.
Right. It would be possible to have slapd internally break up the results to prevent the thread from getting blocked. There's some question about whether it's worthwhile to do so (e.g. for a slow client, the time slapd spends blocked should outweigh the time it takes slapd to requeue the result processing.)
Howard Chu wrote:
It would be possible to have slapd internally break up the results to prevent the thread from getting blocked. There's some question about whether it's worthwhile to do so (e.g. for a slow client, the time slapd spends blocked should outweigh the time it takes slapd to requeue the result processing.)
What does blocking mean? Could you please elaborate on this? Are other LDAP operations not processed anymore at all?
Ciao, Michael.
On Donnerstag, 10. April 2008, Michael Ströder wrote:
Hallvard B Furuseth wrote:
Michael Ströder writes:
Is that really a problem? How often is "occasionally"?
Don't know, and don't know.
To me 2.5 MB does not sound so much to justify thinking about changing the client app in such a network- and data-specific way.
OK, good. I've no experience with that kind of search result sizes myself.
I can only speak of situations where I retrieve the whole directory (up to 300000 entries) for syncing. But this does not happen very often and my sync scripts call ldap_result() quite soon and process results as they come in.
getgrent() with nss_ldap. Others may come later.
Hmm, maybe that's what Volker Lendecke was talking about at LDAPcon 2007 regarding enumeration of groups. See his slides:
http://www.guug.de/veranstaltungen/ldapcon2007/slides/ldapcon_lendecke.pdf
If that really is the problem. How about disabling getgrent for nss_ldap. nss_ldap supports that since some versions. Have a look for "nss_getgrent_skipmembers" in the man-page.
openldap-software@openldap.org
-
Hallvard B Furuseth -
Howard Chu -
Michael Ströder -
Ralf Haferkamp