Pierangelo Masarati wrote:
I'd like to mention a different scenario, where proxy databases need to deal with a mix of slow and fast targets. What we experienced is that concurrency can be heavily penalized by this sort of mix of targets when few threads are available, because inevitably operations affecting slow targets eat up resources that remain idle in ldap_result() while they could be used to deal with fast target related operations, while now they have to remain pending. In some cases, we had to use up to 128 threads (we even experimented with 512) with big waste of system resources.
A solution could be to redesign the proxies so that request and response are handled independently by different threads, using "client" connections that detect activities on persistent connection handlers towards the targets. Together with a customer, we quickly prototyped something like this (back-aldap, standing for "asynchronous ldap"), which is just a toy right now, but it showed some potential.
I remember thinking that we need to do this; the syncrepl consumer certainly works better with this approach.