RE: (ITS#7698) Multiple Paged search requests on one connection fail
by quanah＠zimbra.com
--On Tuesday, September 17, 2013 7:52 PM +0000 john.unsworth(a)cp.net wrote:
> AFAIK it is not the standard LDAP changelog - let me know if I'm wrong. We
> don't want to write specific code for each LDAP server that doesn't
> conform to the LDAP (draft I know) standard.
Please fix your client to stop top-posting.
With openldap, you have two choices. I don't know which draft you are
referring to, so I can't say if either conforms with what the other
directory servers do.
slapo-auditlog
slapo-accesslog
Pick your path.
--Quanah
--
Quanah Gibson-Mount
Lead Engineer
Zimbra Software, LLC
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years
RE: (ITS#7698) Multiple Paged search requests on one connection fail
by john.unsworth＠cp.net
>> There are no other directory servers in existence that can scale to the
level that OpenLDAP does and perform well at that scale.
How do you justify/prove that statement?
-----Original Message-----
From: Howard Chu [mailto:hyc@symas.com]
Sent: 17 September 2013 20:52
To: john.unsworth(a)cp.net; openldap-its(a)openldap.org
Subject: Re: (ITS#7698) Multiple Paged search requests on one connection
fail
john.unsworth(a)cp.net wrote:
> Products other than browsers use LDAP servers. Our product is a meta
> directory that watches for changes on multiple data sources (LDAP,
> Databases, SAP, ...) and reconciles any changes across the complete
> set of sources according to data filtering and transformation rules.
> However sometimes it is necessary to read the whole data set - for
> example when a new data server is added, or when changes may have been
> missed for any reason, or when an 'audit' type operation is required
> to ensure that all data sources are consistent. LDAP servers we
> connect to typically contain thousands if not millions of entries.
OpenLDAP is regularly used in production by large telcos with billions of
entries. There are no other directory servers in existence that can scale to
the level that OpenLDAP does and perform well at that scale.
> Reading the lot without paging -
> although we can do it - is very inefficient both in terms of LDAP
> resources and application resources.
That's utter nonsense. The amount of data is the same no matter how many
pieces you slice it into. It is *more* inefficient to slice it into multiple
requests.
> The directory is read using multiple parallel paged searches across
> different branches on a single connection. Every directory we have
> used except OpenLDAP can manage this effectively.
> In the case of OpenLDAP there is no LDAP changelog and so the only way
> we have of discovering changes is to read the whole directory and
> compare with what was read last time.
So you're saying you're connecting to OpenLDAP servers from version 2.1 or
older, from before syncrepl or the accesslog or changelog overlays existed.
> -----Original Message-----
> From: Michael Ströder [mailto:michael@stroeder.com]
> Sent: 17 September 2013 20:20
> To: john.unsworth(a)cp.net; openldap-its(a)openldap.org
> Subject: Re: (ITS#7698) Multiple Paged search requests on one
> connection fail
>
> john.unsworth(a)cp.net wrote:
>> I would also be interested to understand why " paged results is
>> inherently flawed".
>
> Although being the author of an interactive LDAP UI client I still
> wonder why people want paged results (or tree browsing).
>
> If you have so many results you should narrow the search criteria.
> Browsing/paging is pretty inefficient working style.
>
> Ciao, Michael.
>
>
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years
RE: (ITS#7698) Multiple Paged search requests on one connection fail
by john.unsworth＠cp.net
AFAIK it is not the standard LDAP changelog - let me know if I'm wrong. We
don't want to write specific code for each LDAP server that doesn't conform
to the LDAP (draft I know) standard.
-----Original Message-----
From: Quanah Gibson-Mount [mailto:quanah@zimbra.com]
Sent: 17 September 2013 20:49
To: john.unsworth(a)cp.net; openldap-its(a)openldap.org
Subject: RE: (ITS#7698) Multiple Paged search requests on one connection
fail
--On Tuesday, September 17, 2013 7:34 PM +0000 john.unsworth(a)cp.net wrote:
> In the case of OpenLDAP there is no LDAP changelog and so the only way
> we have of discovering changes is to read the whole directory and
> compare with what was read last time.
If you think that, then apparently you haven't spent time reading the
documentation. OpenLDAP has had a changelog since at least OpenLDAP 2.3, if
you enable it. I use it for auditing changes all the time.
--Quanah
--
Quanah Gibson-Mount
Lead Engineer
Zimbra Software, LLC
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years
Re: (ITS#7698) Multiple Paged search requests on one connection fail
by hyc＠symas.com
john.unsworth(a)cp.net wrote:
> Products other than browsers use LDAP servers. Our product is a meta
> directory that watches for changes on multiple data sources (LDAP,
> Databases, SAP, ...) and reconciles any changes across the complete set of
> sources according to data filtering and transformation rules. However
> sometimes it is necessary to read the whole data set - for example when a
> new data server is added, or when changes may have been missed for any
> reason, or when an 'audit' type operation is required to ensure that all
> data sources are consistent. LDAP servers we connect to typically contain
> thousands if not millions of entries.
OpenLDAP is regularly used in production by large telcos with billions of
entries. There are no other directory servers in existence that can scale to
the level that OpenLDAP does and perform well at that scale.
> Reading the lot without paging -
> although we can do it - is very inefficient both in terms of LDAP resources
> and application resources.
That's utter nonsense. The amount of data is the same no matter how many
pieces you slice it into. It is *more* inefficient to slice it into multiple
requests.
> The directory is read using multiple parallel
> paged searches across different branches on a single connection. Every
> directory we have used except OpenLDAP can manage this effectively.
> In the case of OpenLDAP there is no LDAP changelog and so the only way we
> have of discovering changes is to read the whole directory and compare with
> what was read last time.
So you're saying you're connecting to OpenLDAP servers from version 2.1 or
older, from before syncrepl or the accesslog or changelog overlays existed.
> -----Original Message-----
> From: Michael Ströder [mailto:michael@stroeder.com]
> Sent: 17 September 2013 20:20
> To: john.unsworth(a)cp.net; openldap-its(a)openldap.org
> Subject: Re: (ITS#7698) Multiple Paged search requests on one connection
> fail
>
> john.unsworth(a)cp.net wrote:
>> I would also be interested to understand why " paged results is inherently
>> flawed".
>
> Although being the author of an interactive LDAP UI client I still wonder
> why
> people want paged results (or tree browsing).
>
> If you have so many results you should narrow the search criteria.
> Browsing/paging is pretty inefficient working style.
>
> Ciao, Michael.
>
>
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years
RE: (ITS#7698) Multiple Paged search requests on one connection fail
by quanah＠zimbra.com
--On Tuesday, September 17, 2013 7:34 PM +0000 john.unsworth(a)cp.net wrote:
> In the case of OpenLDAP there is no LDAP changelog and so the only way we
> have of discovering changes is to read the whole directory and compare
> with what was read last time.
If you think that, then apparently you haven't spent time reading the
documentation. OpenLDAP has had a changelog since at least OpenLDAP 2.3,
if you enable it. I use it for auditing changes all the time.
--Quanah
--
Quanah Gibson-Mount
Lead Engineer
Zimbra Software, LLC
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years
RE: (ITS#7698) Multiple Paged search requests on one connection fail
by john.unsworth＠cp.net
Products other than browsers use LDAP servers. Our product is a meta
directory that watches for changes on multiple data sources (LDAP,
Databases, SAP, ...) and reconciles any changes across the complete set of
sources according to data filtering and transformation rules. However
sometimes it is necessary to read the whole data set - for example when a
new data server is added, or when changes may have been missed for any
reason, or when an 'audit' type operation is required to ensure that all
data sources are consistent. LDAP servers we connect to typically contain
thousands if not millions of entries. Reading the lot without paging -
although we can do it - is very inefficient both in terms of LDAP resources
and application resources. The directory is read using multiple parallel
paged searches across different branches on a single connection. Every
directory we have used except OpenLDAP can manage this effectively.
In the case of OpenLDAP there is no LDAP changelog and so the only way we
have of discovering changes is to read the whole directory and compare with
what was read last time.
-----Original Message-----
From: Michael Ströder [mailto:michael@stroeder.com]
Sent: 17 September 2013 20:20
To: john.unsworth(a)cp.net; openldap-its(a)openldap.org
Subject: Re: (ITS#7698) Multiple Paged search requests on one connection
fail
john.unsworth(a)cp.net wrote:
> I would also be interested to understand why " paged results is inherently
> flawed".
Although being the author of an interactive LDAP UI client I still wonder
why
people want paged results (or tree browsing).
If you have so many results you should narrow the search criteria.
Browsing/paging is pretty inefficient working style.
Ciao, Michael.
10 years
Re: (ITS#7698) Multiple Paged search requests on one connection fail
by michael＠stroeder.com
This is a cryptographically signed message in MIME format.
--------------ms010600090306020303050000
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
john.unsworth(a)cp.net wrote:
> I would also be interested to understand why " paged results is inheren=
tly
> flawed".
Although being the author of an interactive LDAP UI client I still wonder=
why
people want paged results (or tree browsing).
If you have so many results you should narrow the search criteria.
Browsing/paging is pretty inefficient working style.
Ciao, Michael.
--------------ms010600090306020303050000
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFfzCC
BXswggNjoAMCAQICAwxOfTANBgkqhkiG9w0BAQUFADB5MRAwDgYDVQQKEwdSb290IENBMR4w
HAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmlu
ZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAeFw0xMjEw
MDIyMDE3MDlaFw0xNDEwMDIyMDE3MDlaMD8xGDAWBgNVBAMUD01pY2hhZWwgU3Ry9mRlcjEj
MCEGCSqGSIb3DQEJARYUbWljaGFlbEBzdHJvZWRlci5jb20wggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDo2SKth5GhtaDrCyfGtyUG+/hAAa/J52L0NFN4SSRvTtdGf9HfWwwd
NCtgae0TVGWk2lKDbXA9d5vmyIiRhuwxd90H6FLErhRBeB9G67qtw87E8WUoXt2DwPQEUTWV
hqHpPadlmgFw3+i3TGQQTe3O3W9MMMd4GJNhObem2VGRuCD37OXnzBksTcq0FPJgcWAhe3d/
0ItOkNWBqgq8Mf3p7WFBhaQ0a27BC/mKtH8fI3kPcS305imPRja69Msq3EwUZBc9ToVp6FRQ
NYKjfOBybDUzVkmRZl3H8xutQP2w8Zxb8m5f7Q1BfLLrIFScfYvIDgOERxTCd4lab8+/09XH
AgMBAAGjggFEMIIBQDAMBgNVHRMBAf8EAjAAMFYGCWCGSAGG+EIBDQRJFkdUbyBnZXQgeW91
ciBvd24gY2VydGlmaWNhdGUgZm9yIEZSRUUgaGVhZCBvdmVyIHRvIGh0dHA6Ly93d3cuQ0Fj
ZXJ0Lm9yZzAOBgNVHQ8BAf8EBAMCA6gwQAYDVR0lBDkwNwYIKwYBBQUHAwQGCCsGAQUFBwMC
BgorBgEEAYI3CgMEBgorBgEEAYI3CgMDBglghkgBhvhCBAEwMgYIKwYBBQUHAQEEJjAkMCIG
CCsGAQUFBzABhhZodHRwOi8vb2NzcC5jYWNlcnQub3JnMDEGA1UdHwQqMCgwJqAkoCKGIGh0
dHA6Ly9jcmwuY2FjZXJ0Lm9yZy9yZXZva2UuY3JsMB8GA1UdEQQYMBaBFG1pY2hhZWxAc3Ry
b2VkZXIuY29tMA0GCSqGSIb3DQEBBQUAA4ICAQC9ouXq3p/bDWMM4tBKgD3tl4HY5H0eECl8
q9/nqk0UL6YeWkrCiQdrDtNPW7DcGqNYtzdgtzmyTr1GhiAX+igrOjdk/ge5NRcQOpONK/4b
zrmpQEcIUyxSSDKLWh211/kcFfxxLEiJ5teF4GL8Fc1qbrLP4+DCvJXWfYaaR5NLjZMqm2VP
yKTv3qpXWnGohiRkGTwS/11QM2XCfIGdRsQT9a8mO4m2fn2tGPp2TEIoCLrDDrbGVeDWaOWB
OIeTrp4wa3Q4OI6yCptJhEqKvjhV96IBRYgM76nTBqsqnDzwxExAyhhWiUS5DunRHOr/+NyF
pUpD4883RBLO0g9kUEGOhtZNF1u+8zEL0YgMGvifAom9JEklLOXZuqj0MThypKs/3d/OyOQb
4gURnu6oZwcKZ7LskytWnlRKUxF6o0A8grtmyKkqe14TS7cQbg0NTaIYXPkHR+dfFmb3uEqn
BBjvpJXFcEtWI2lQXC/ET+au991pK797ExBOmpQwjIn3SjiW80vw/UoL6DMvqY/6JhVhyNTP
MJ2W5AX5kc27DIbVtVGZs8J4AYhuNALJUq9N9Ka7rPRj3RcYDrfehDLOkM5iMnarpmtuOpLK
d1SvZhqj/0N/JWGIDpPSTkTFOPP6ZN9I9Rqyf+9NGqb2sjo4DkIiZcHxt735/GJLwus5KLBl
2DGCA6EwggOdAgEBMIGAMHkxEDAOBgNVBAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93
d3cuY2FjZXJ0Lm9yZzEiMCAGA1UEAxMZQ0EgQ2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8G
CSqGSIb3DQEJARYSc3VwcG9ydEBjYWNlcnQub3JnAgMMTn0wCQYFKw4DAhoFAKCCAfUwGAYJ
KoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTMwOTE3MTkyMDAwWjAj
BgkqhkiG9w0BCQQxFgQUpv3q6AMfQ1O8+1srFtSPkVL3xygwbAYJKoZIhvcNAQkPMV8wXTAL
BglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDAN
BggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBkQYJKwYBBAGCNxAEMYGD
MIGAMHkxEDAOBgNVBAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9y
ZzEiMCAGA1UEAxMZQ0EgQ2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJARYS
c3VwcG9ydEBjYWNlcnQub3JnAgMMTn0wgZMGCyqGSIb3DQEJEAILMYGDoIGAMHkxEDAOBgNV
BAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9yZzEiMCAGA1UEAxMZ
Q0EgQ2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJARYSc3VwcG9ydEBjYWNl
cnQub3JnAgMMTn0wDQYJKoZIhvcNAQEBBQAEggEAGjYerDwPbwx9TRVxTwWTs+XC+rIW5OBo
aslumvg3olNfz8t+Tw+Mjg23Gr9JCoAeJNW3F/YyUnvBD2hiWqDFfG/GJrRoEcpoXTMjNe+k
OBjGqU84SK/a8OwFQWgY2FN5pB88Zz56HBqSur7QmXhVu4ijKwGQGlVZxJFJ5JFeoNcKEJfo
ln/4LsMgQ8g/ouvokX3tRsbb8u8dT4EaEwN1nyMaLK5el/DhGuVwuwh0tqS6BV82NPG1NRAb
YdIs5DQAqrqXJBrFhvABVSwZQ73rzipMZrFXz42uuaG/5ronuSPMFIPAGOxuF4/zUsN5qcq8
AQ7mVAQ37eOdfyhv8jKVFwAAAAAAAA==
--------------ms010600090306020303050000--
10 years
RE: (ITS#7698) Multiple Paged search requests on one connection fail
by john.unsworth＠cp.net
I would also be interested to understand why " paged results is inherently
flawed".
-----Original Message-----
From: John Unsworth [mailto:john.unsworth@cp.net]
Sent: 17 September 2013 12:07
To: 'Pierangelo Masarati'
Cc: 'openldap-its(a)openldap.org'
Subject: RE: (ITS#7698) Multiple Paged search requests on one connection
fail
Thank you for your reply.
>> Technically speaking, paged results consists of perfectly
>> independent operations from the protocol point of view
Yes but the results have not all been retrieved - only the first page full -
and further operations are needed to get the additional results. Hence paged
searches are not really independent but are linked by the cookie. The
results are supposed to remain available so that additional searches using
the same cookie can retrieve them. Every other LDAP server I have come
across allows multiple concurrent paged searches on a single connection.
The RFC does seem to allow the OpenLDAP behaviour but it seems extremely
restrictive to me. It should at least be well documented.
-----Original Message-----
From: Pierangelo Masarati [mailto:pierangelo.masarati@polimi.it]
<snip>
No, it's correct (at least, that's the intended behavior). The operation
structure is only persistent for the duration of the operation, whereas the
connection structure persists for the duration of the connection.
Technically speaking, paged results consists of perfectly independent
operations from the protocol point of view, so persistent state is saved in
the connection structure. Of course, multiple cookies could be stored, but
since paged results is inherently flawed, it was decided to handle it this
way. Clients need to use a dedicated connection for search operations that
use the paged results control.
p.
--
Pierangelo Masarati
Associate Professor
Dipartimento di Scienze e Tecnologie Aerospaziali Politecnico di Milano
10 years
RE: (ITS#7698) Multiple Paged search requests on one connection fail
by john.unsworth＠cp.net
Thank you for your reply.
>> Technically speaking, paged results consists of perfectly independent
operations from the protocol point of view
Yes but the results have not all been retrieved - only the first page full -
and further operations are needed to get the additional results. Hence paged
searches are not really independent but are linked by the cookie. The
results are supposed to remain available so that additional searches using
the same cookie can retrieve them. Every other LDAP server I have come
across allows multiple concurrent paged searches on a single connection.
The RFC does seem to allow the OpenLDAP behaviour but it seems extremely
restrictive to me. It should at least be well documented.
-----Original Message-----
From: Pierangelo Masarati [mailto:pierangelo.masarati@polimi.it]
Sent: 17 September 2013 11:54
To: john.unsworth(a)cp.net
Cc: openldap-its(a)openldap.org
Subject: Re: (ITS#7698) Multiple Paged search requests on one connection
fail
On 09/17/2013 12:36 PM, john.unsworth(a)cp.net wrote:
> Full_Name: John Unsworth
> Version: 2.4.34
> OS: Windows 7
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (81.139.153.176)
>
>
> If I issue multiple paged requests on a single connection then when
> requesting the second page for the first request I receive 'paged
> results cookie is invalid'. I suspect from looking at the code that
> only a single paged request is allowed to be active at once on a
connection. Can someone confirm please.
Confirmed.
>
> parse_paged_cookie( Operation *op, SlapReply *rs )
>
> if ( ps->ps_cookieval.bv_len ) {
> ...
> } else {
> /* we're going to use ps_cookie */
> op->o_conn->c_pagedresults_state.ps_cookie = 0;
>
> It seems to me that the cookie is held against the connection (o_conn)
> and not the operation. Surely this is wrong?
No, it's correct (at least, that's the intended behavior). The operation
structure is only persistent for the duration of the operation, whereas the
connection structure persists for the duration of the connection.
Technically speaking, paged results consists of perfectly independent
operations from the protocol point of view, so persistent state is saved in
the connection structure. Of course, multiple cookies could be stored, but
since paged results is inherently flawed, it was decided to handle it this
way. Clients need to use a dedicated connection for search operations that
use the paged results control.
p.
--
Pierangelo Masarati
Associate Professor
Dipartimento di Scienze e Tecnologie Aerospaziali Politecnico di Milano
10 years
Re: (ITS#7698) Multiple Paged search requests on one connection fail
by pierangelo.masarati＠polimi.it
On 09/17/2013 12:36 PM, john.unsworth(a)cp.net wrote:
> Full_Name: John Unsworth
> Version: 2.4.34
> OS: Windows 7
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (81.139.153.176)
>
>
> If I issue multiple paged requests on a single connection then when requesting
> the second page for the first request I receive 'paged results cookie is
> invalid'. I suspect from looking at the code that only a single paged request is
> allowed to be active at once on a connection. Can someone confirm please.
Confirmed.
>
> parse_paged_cookie( Operation *op, SlapReply *rs )
>
> if ( ps->ps_cookieval.bv_len ) {
> ...
> } else {
> /* we're going to use ps_cookie */
> op->o_conn->c_pagedresults_state.ps_cookie = 0;
>
> It seems to me that the cookie is held against the connection (o_conn) and not
> the operation. Surely this is wrong?
No, it's correct (at least, that's the intended behavior). The
operation structure is only persistent for the duration of the
operation, whereas the connection structure persists for the duration of
the connection. Technically speaking, paged results consists of
perfectly independent operations from the protocol point of view, so
persistent state is saved in the connection structure. Of course,
multiple cookies could be stored, but since paged results is inherently
flawed, it was decided to handle it this way. Clients need to use a
dedicated connection for search operations that use the paged results
control.
p.
--
Pierangelo Masarati
Associate Professor
Dipartimento di Scienze e Tecnologie Aerospaziali
Politecnico di Milano
10 years
