https://bugs.openldap.org/show_bug.cgi?id=10570
Issue ID: 10570 Summary: Humble request for comment about cached pagination Product: OpenLDAP Version: 2.6.8 Hardware: x86_64 OS: Linux Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: overlays Assignee: bugs@openldap.org Reporter: f.castagnan@gmail.com Target Milestone: ---
Hello folks, I hope you are well and thank you for providing a forum for users to raise inquiries, I think the structured format is quite good. (it's my first time doing raising an issue on your platform)
Context: I have this query shape that is quite expensive, coming in from python ldap3 lib used in many of my clients. I decided to put openldap using the pcache overlay between my clients are my AD servers to cache the results.
```python # The python code using ldap3 python lib # Below expands to (&(objectClass=user)(memberOf:1.2.840.113556.1.4.1941:=cn=<group>,ou=REDACTED,ou=REDACTED,ou=REDACTED,ou=REDACTED,dc=REDACTED,dc=com))
for entry in conn.extend.standard.paged_search( search_base, search_filter, scope, attributes=list(attributes), paged_size=page_size, generator=True, ): ```
I am using these attribute sets and templates (cropped for brevity)
``` ... database ldap suffix "dc=REDACTED" uri ldap://REDACTED.com/ overlay pcache pcache mdb 100000 6 1000 100 pcacheAttrset 0 uidNumber gidNumber sAMAccountName loginShell sn pcacheAttrset 1 member uidNumber gidNumber loginShell sAMAccountName memberOf cn name pcacheAttrset 2 whenCreated mail sAMAccountName cn objectclass whenChanged userprincipalname uidNumber objectGUID sn memberOf userPrincipalName userAccountControl pwdLastSet loginShell pcacheAttrset 3 cn name sAMAccountName memberOf uidNumber gidNumber distinguishedName unixHomeDirectory REDACTED pcacheAttrset 4 gidNumber sAMAccountName pcacheAttrset 5 name member gidNumber sAMAccountName cn
pcacheTemplate (&(objectClass=)(gidNumber=)) 0 10800 pcacheTemplate (&(objectClass=)(gidNumber=)) 4 10800 pcacheTemplate (&(objectClass=)(cn=)) 1 10800 pcacheTemplate (&(objectClass=group)(cn=)) 1 10800 pcacheTemplate (sAMAccountName=) 0 3600 ... ```
Problem observed
I can't seem to get the paged queries to be cached and it only returns my AD server max limit of 1000 items even when there are more results leading to inacurrate results in my clients.
``` # LOGS 6a7cd2f0.2447015f 0x7f65effff640 conn=53266 op=3 SRCH base="ou=REDACTED,dc=com" scope=2 deref=3 filter="(&(objectClass=user)(!(objectClass=computer)))" 6a7cd2f0.2447c7b1 0x7f65effff640 conn=53266 op=3 SRCH attr=unixHomeDirectory sAMAccountName REDACTED uidNumber gidNumber name cn distinguishedName memberOf 6a7cd2f0.24480fd2 0x7f65effff640 conn=53266 op=3: non-critical pagedResults control disabled with proxy cache; stripped. 6a7cd2f0.244819ea 0x7f65effff640 conn=53266 op=3: non-critical control "1.2.840.113556.1.4.1339" not supported; stripped. 6a7cd2f0.24484d6b 0x7f65effff640 query template of incoming query = (&(objectClass=)(!(objectClass=))) 6a7cd2f0.24485ba0 0x7f65effff640 QUERY NOT ANSWERABLE 6a7cd2f0.24486181 0x7f65effff640 QUERY NOT CACHEABLE ```
I tried a number of permutation to fix this, but upon inspecting the code it seems that pagination is explicitly not supported. From this line: https://github.com/openldap/openldap/blob/fe9c74b257398deab91ef2ec69011df3c6...
Questions: 1. Is there an accepted design for adding pagedResults support to pcache (cookie handling, stable ordering, cursor state)?
2. Is there a recommended way to cache paged searches or to internally page cached results while forwarding a single non‑paged search to the backend?
3. Are there existing patches or modules that solve this (or similar) in OpenLDAP?
Many thanks! Francis