Hi,
In case of ldap_search_ext_s:
We can pass,how many entries we want. In my case i can retrieve only 100 entries at a time due to the size limit of my buffer.
But the ldap search which i am using will return 1000 entries. Is there any way through which i can retrieve 1000 entries in chunks of 100.
first time i will fetch 100 entries then again next 100 entries then again next 100 entries..... ... ...
In this way i will retrieve all the entries. Please tell me the way of fetching entries from LDAP.
Rakesh Yadav wrote:
Hi,
In case of ldap_search_ext_s:
We can pass,how many entries we want. In my case i can retrieve only 100 entries at a time due to the size limit of my buffer.
But the ldap search which i am using will return 1000 entries. Is there any way through which i can retrieve 1000 entries in chunks of 100.
first time i will fetch 100 entries then again next 100 entries then again next 100 entries..... ... ...
In this way i will retrieve all the entries. Please tell me the way of fetching entries from LDAP.
Your problem could be solved in many ways: using a band-aid, or designing your client to get rid of self-imposed constraints.
Usually there are two cases: 1) entries can be processed one by one, regardless of the order they come in 2) entries need to be processed all at once
There's also an intermediate case 3) entries need to be processed all at once, but pre-processing can start as soon as data comes in
The latter is a mix of the two, and it is basically a matter of optimization (Real Programmers (TM) usually use the asynchronous call anyway, because it allows finer grain control on things like select(2) timeouts and so).
In case (1) you should use an asynchronous call, like ldap_search_ext(3), so that you get entries one by one as soon as they're available, using ldap_result(3), and dispose of them as soon as you're done.
In case (2) you can use a synchronous call, like ldap_search_ext_s(3), which will return all entries in a bunch, malloc'ed and linked list'ed. In that case, you don't need any fixed size buffer, so if you design your client with a fixed buffer for search entries while you plan to use a synchronous call you are definitely trying to figure out the best way to shoot in your own foot.
For completeness, I have to say that if you're really committed to pulling the trigger, then you could have a look at paged results control (RFC 2696). I wouldn't go that way, though.
Examples of using asynchronous search and even paged results control are available in clients/tools/ldapsearch.c.
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati@sys-net.it ---------------------------------------
... Your problem could be solved in many ways: using a band-aid, or designing your client to get rid of self-imposed constraints. ...
There is a real use case for paged results (at least I had a similar problem, too): As we had to write a piece of software for a BIG company with thousands of accounts stored in an Active Directory server, we had to fetch all accounts from the AD server. Starting using asynchronous calls, all went fine, until we noticed, that the SERVER side had a restriction to not give out more than 1000 results per single query.
We could not find a "nice" workaround for this restriction (I dont know if the AD server CAN not give out more than X results per query, or whether the admins did not WANT to remove this limit).
We solved this by splitting the "big" query in 26 single queries (all accounts starting with "A", all accounts starting with "B" and so on) - but this solution is ugly and does not scale very well.
So a "real" solution for paged results would have been nice (until now I did not know RFC 2696).
regards -stefan-
Stefan Palme wrote:
... Your problem could be solved in many ways: using a band-aid, or designing your client to get rid of self-imposed constraints. ...
There is a real use case for paged results (at least I had a similar problem, too): As we had to write a piece of software for a BIG company with thousands of accounts stored in an Active Directory server, we had to fetch all accounts from the AD server. Starting using asynchronous calls, all went fine, until we noticed, that the SERVER side had a restriction to not give out more than 1000 results per single query.
We could not find a "nice" workaround for this restriction (I dont know if the AD server CAN not give out more than X results per query, or whether the admins did not WANT to remove this limit).
Just because Microsoft doesn't know how to implement the real LDAP protocol, and doesn't know how to write a server that can behave as a real LDAP server, doesn't make this a "real use"...
Note https://www.openldap.org/its/index.cgi/Historical?id=5092
especially followup #8 - it is *possible* to increase the sizelimit in AD, but it is strongly discouraged because increasing the limit will severely impair AD's performance. And of course, looking here http://connexitor.com/blog/pivot/entry.php?id=185
AD's performance is already miserable to begin with, so when Microsoft themselves warns you that it will be impaired, one can only imagine how much worse it could possibly get.
By the way, the time to retrieve the DN's of all 1 million entries, using paged results, in OpenLDAP was only 18 seconds. Against AD it took 34 seconds. So even without the overhead of many threads and requests running at once, OpenLDAP is much faster.
OpenLDAP: time ./ldapsearch -x -H ldap://sihu -D dc=example,dc=com -w "secret" -LLL -b ou=people,dc=example,dc=com -E pr=1000/noprompt 1.1 > dn1
real 0m17.766s user 0m5.337s sys 0m7.831s
ADAM: time ./ldapsearch -x -H ldap://sihu -D cn=bench,cn=users,dc=example,dc=com -w "123foo#" -LLL -b ou=people,dc=example,dc=com -E pr=1000/noprompt 1.1 > dn1
real 0m33.875s user 0m5.618s sys 0m3.608s
(In both cases the same client was used, querying the exact same server machine, just with a different OS/directory server.) In fact OpenLDAP can scan its entire 1M entry DB in under 2 seconds; for AD it takes around 25 seconds. (That's running a search that returns no data, so there's no network turnaround overhead like there is in this pagedresult case.)
We solved this by splitting the "big" query in 26 single queries (all accounts starting with "A", all accounts starting with "B" and so on) - but this solution is ugly and does not scale very well.
Naturally, since AD is ugly and does not scale very well.
So a "real" solution for paged results would have been nice (until now I did not know RFC 2696).
For a BIG company with thousands of accounts, a real solution would use a real LDAP server, not Microsoft garbage.
On Jan 31, 2008, at 11:37 , Howard Chu wrote:
For a BIG company with thousands of accounts, a real solution would use a real LDAP server, not Microsoft garbage.
In many companies it's simply impossible to overcome the M$ bias and inertia, even in the face of hard comparison data. In most cases the decision-makers don't really know the first thing about technology, anyway.
I wish there was a way to slap some sense into people who don't listen to technical arguments, or even business arguments, out of pure ignorance and inertia.
jens
On Thu, 2008-01-31 at 02:37 -0800, Howard Chu wrote:
...
For a BIG company with thousands of accounts, a real solution would use a real LDAP server, not Microsoft garbage.
The same what I think - but in this case the existing infrastructure had to be taken as-is, so we had to deal with the AD problems and could not just throw away the AD and replace it with an OpenLDAP server :-)
Thanks and regards -stefan-
Stefan Palme skrev, on 31-01-2008 14:06:
For a BIG company with thousands of accounts, a real solution would use a real LDAP server, not Microsoft garbage.
The same what I think - but in this case the existing infrastructure had to be taken as-is, so we had to deal with the AD problems and could not just throw away the AD and replace it with an OpenLDAP server :-)
If this were a large company "with thousands of accounts" and willing to pay for a solution that works, I think I might be looking at at least one dedicated OpenLDAP machine pulling from AD and serving what's necessary of LDAP to clients.
That might lead to a whole lot of schema hassle, but from what you described originally I think maybe not.
Like Howard I was once a confirmed Microsoft user but encountered so much that stuck in my throat on the way that I became a Unix person and looked to giving up Microsoft. In this life I have to look at ways of pulling information from the one and feeding it to the other.
Best,
--Tonni
On Thu, 2008-01-31 at 14:29 +0100, Tony Earnshaw wrote:
Stefan Palme skrev, on 31-01-2008 14:06:
For a BIG company with thousands of accounts, a real solution would use a real LDAP server, not Microsoft garbage.
The same what I think - but in this case the existing infrastructure had to be taken as-is, so we had to deal with the AD problems and could not just throw away the AD and replace it with an OpenLDAP server :-)
If this were a large company "with thousands of accounts" and willing to pay for a solution that works, I think I might be looking at at least one dedicated OpenLDAP machine pulling from AD and serving what's necessary of LDAP to clients.
That might lead to a whole lot of schema hassle, but from what you described originally I think maybe not.
Like Howard I was once a confirmed Microsoft user but encountered so much that stuck in my throat on the way that I became a Unix person and looked to giving up Microsoft. In this life I have to look at ways of pulling information from the one and feeding it to the other.
I absolutely agree (I'm afraid we are far off-topic at the moment ;-), but you speak to the wrong person... If I had anything to say in that company there would be no MS based server today. When I had the AD- problem I was a kind of student who was paid to solve one very special problem in that company (not directly related to LDAP / AD). Fetching all the users from the AD server was just one step to solve this problem...
What I want to say - I absolutely agree to all of your arguments, but the point is: sometimes there ARE situations where the original problem (the need for paged results) exists - even if the REASON for this need is a poor one (e.g. using MS software for large server installations). When you have to SOLVE the problem, you can not always start to solve the problems beginning at the root (replacing AD by OpenLDAP), but you have to find a solution that works in the given context (and with the available amount of money ;-)
regards -stefan-
Jens Vagelpohl wrote:
On Jan 31, 2008, at 11:37 , Howard Chu wrote:
For a BIG company with thousands of accounts, a real solution would use a real LDAP server, not Microsoft garbage.
In many companies it's simply impossible to overcome the M$ bias and inertia, even in the face of hard comparison data. In most cases the decision-makers don't really know the first thing about technology, anyway.
I wish there was a way to slap some sense into people who don't listen to technical arguments, or even business arguments, out of pure ignorance and inertia.
Inertia isn't all bad. Those same decision-makers have managed to defer or avoid migrating to Vista... In any business, it comes down to cost. Some small companies can survive the costs associated with running an unscalable infrastructure. For others, limitations like this http://technet.microsoft.com/en-us/library/aa998536.aspx will severely hamper their productivity, and that racks up fast. When they run into a wall, they'll have no choice but to look for an alternate solution. If they never run into a wall, then there's no point in worrying about them.
openldap-technical@openldap.org
-
Howard Chu -
Jens Vagelpohl -
Pierangelo Masarati -
Rakesh Yadav -
Stefan Palme -
Tony Earnshaw