(ITS#7565) unacceptable growth on delete with back-mdb
by quanah＠OpenLDAP.org
Full_Name: Quanah Gibson-Mount
Version: 2.4.35
OS: Linux 2.6
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.39.181)
With back-mdb, if you add a large number of users to the database (via ldapadd),
and then delete them (via ldapmodify), the DB explodes in size.
After the initial ldapadd of 30 million users, the DB is 17GB in size.
After deleting the 30 million users, the DB is 52GB in size. This is a 3.05
times larger than the DBs initial size.
With back-hdb, there is no change in size of the DB (it is static).
Clearly this is not a desirable state of affairs.
This appears to be a bug with the freelist. After the add is complete, we
have:
zimbra@zre-ldap001:~/data/ldap/mdb/db$ mdb_stat -f .
Freelist Status
Tree depth: 3
Branch pages: 19
Leaf pages: 2007
Overflow pages: 5155
Entries: 15879
Free pages: 2612121
My delete run is not yet complete, but it is now:
zimbra@zre-ldap001:~/data/ldap/mdb/db$ mdb_stat -f .
Freelist Status
Tree depth: 3
Branch pages: 99
Leaf pages: 11096
Overflow pages: 20891
Entries: 82932
Free pages: 9381096
(and growing)
--Quanah
10 years, 8 months
Re: (ITS#7563) slapd modifies attribute value of pwdAttribute
by michael＠stroeder.com
This is a cryptographically signed message in MIME format.
--------------ms070208010706030608090702
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Howard, please try yourself first before answering.
Dieter is right here.
Example:
dn: cn=3DDefault Password Policy,ou=3DPolicies,dc=3Dstroeder,dc=3Dde
changetype: modify
delete: pwdAttribute
pwdAttribute: userPassword
-
add: pwdAttribute
pwdAttribute: 2.5.4.35
-
Reading it:
dn: cn=3DDefault Password Policy,ou=3DPolicies,dc=3Dstroeder,dc=3Dde
cn: Default Password Policy
[..]
pwdAttribute: userPassword
[..]
I vaguely remember I had to implement a work-around in web2ldap to deal w=
ith
that when generating delta modification data when the user edits such an =
entry.
Ciao, Michael.
--------------ms070208010706030608090702
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIILHzCC
BT8wggQnoAMCAQICDwCmSwABAAIAivjZQ8SBvzANBgkqhkiG9w0BAQUFADB8MQswCQYDVQQG
EwJERTEcMBoGA1UEChMTVEMgVHJ1c3RDZW50ZXIgR21iSDElMCMGA1UECxMcVEMgVHJ1c3RD
ZW50ZXIgQ2xhc3MgMSBMMSBDQTEoMCYGA1UEAxMfVEMgVHJ1c3RDZW50ZXIgQ2xhc3MgMSBM
MSBDQSBJWDAeFw0xMjA2MDYxOTAyMTZaFw0xMzA2MDcxOTAyMTZaMCgxCzAJBgNVBAYTAkRF
MRkwFwYDVQQDDBBNaWNoYWVsIFN0csO2ZGVyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAxXZGav40rnGNLxEggBW94MILWHlfC8a23Jew5U1gPlfRTXOjjzmoaZ1uCyGdgF6M
VvuO9T1aTQNGH+OdeGe3P7Tfc/NsLJFJ2wtd8blvhmodUgse2eypiWjNOd4gZuhalBhgsQ0K
b5D6/1foghII4E264iZlJ7AJ+UYcO+GxvFWT0YMTbLckgDkZk7c3qwTozdhYvXarvqx+8Ou/
kuxpQQhac/ebzxpu0N+RHSf2KIUS0g0tEGnPtGv6iL+9QNHc4JKo9Y9KKVw3tQy+Re+FQLxB
1fPE5F+qxuD3AUENpOwkMsqWLM94ohtx3CFqLpxfUPrnKFLAHOhHEbByYGvFPwIDAQABo4IC
EDCCAgwwgaUGCCsGAQUFBwEBBIGYMIGVMFEGCCsGAQUFBzAChkVodHRwOi8vd3d3LnRydXN0
Y2VudGVyLmRlL2NlcnRzZXJ2aWNlcy9jYWNlcnRzL3RjX2NsYXNzMV9MMV9DQV9JWC5jcnQw
QAYIKwYBBQUHMAGGNGh0dHA6Ly9vY3NwLml4LnRjY2xhc3MxLnRjdW5pdmVyc2FsLWkudHJ1
c3RjZW50ZXIuZGUwHwYDVR0jBBgwFoAU6bgoHUbP/M34TpvF7ktg69g7P9EwDAYDVR0TAQH/
BAIwADBKBgNVHSAEQzBBMD8GCSqCFAAsAQEBATAyMDAGCCsGAQUFBwIBFiRodHRwOi8vd3d3
LnRydXN0Y2VudGVyLmRlL2d1aWRlbGluZXMwDgYDVR0PAQH/BAQDAgTwMB0GA1UdDgQWBBS2
KAWfTfgJ/JQ63qLGwTXYLnI+LzBiBgNVHR8EWzBZMFegVaBThlFodHRwOi8vY3JsLml4LnRj
Y2xhc3MxLnRjdW5pdmVyc2FsLWkudHJ1c3RjZW50ZXIuZGUvY3JsL3YyL3RjX0NsYXNzMV9M
MV9DQV9JWC5jcmwwMwYDVR0lBCwwKgYIKwYBBQUHAwIGCCsGAQUFBwMEBggrBgEFBQcDBwYK
KwYBBAGCNxQCAjAfBgNVHREEGDAWgRRtaWNoYWVsQHN0cm9lZGVyLmNvbTANBgkqhkiG9w0B
AQUFAAOCAQEAQ3bvVUpEq+cQrLpcogyt5BJNk/WvUvOHqhzyj28M9pg9hcDl1+MYl5qqj6tR
GSTLPQZyf287pcmbMwbcTGZO/gbW9v7RYcut6RauWdwKMCUmKC3J4fVfDq9ZETA2WOV68ef4
B3Gzdhghsbp3Rhp5dDmrCVKAHlafm6ZwJrEQ9P76fxnQZzRLgeKpZep5ePH5YHUB3+YaOQvJ
FG0bOXvfHhRiRG7/HW2G+yDgjHSxDz8AFzMWL/RFePqZ4pn6T/SM/qU6WEpW39MWyJNoH/Kx
QDYK8gGYuesn1ciMCTnjrvZQj0fonGTO4SfWekJRkuGrJ7dYSZRjYbDcWBBkdFLWzzCCBdgw
ggTAoAMCAQICDgboAAEAAkqWLSQM/sXJMA0GCSqGSIb3DQEBBQUAMHkxCzAJBgNVBAYTAkRF
MRwwGgYDVQQKExNUQyBUcnVzdENlbnRlciBHbWJIMSQwIgYDVQQLExtUQyBUcnVzdENlbnRl
ciBVbml2ZXJzYWwgQ0ExJjAkBgNVBAMTHVRDIFRydXN0Q2VudGVyIFVuaXZlcnNhbCBDQSBJ
MB4XDTA5MTEwMzE0MDgxOVoXDTI1MTIzMTIxNTk1OVowfDELMAkGA1UEBhMCREUxHDAaBgNV
BAoTE1RDIFRydXN0Q2VudGVyIEdtYkgxJTAjBgNVBAsTHFRDIFRydXN0Q2VudGVyIENsYXNz
IDEgTDEgQ0ExKDAmBgNVBAMTH1RDIFRydXN0Q2VudGVyIENsYXNzIDEgTDEgQ0EgSVgwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC75pBuz2Lp6QuqthDVR+V8XSsncZpozVVt
5KLv5P7yemMRwleKyH3PjmYfZUVL64Biab1GjovFblqVGCrep/EfdRonq20yU+P7TVhiLP8Z
5cegDZotIYhZhM0d8cPIij6w5d4IJM/8QCy6QSOUu4ASiTVItoYE4AFPjLqpmPwcie0fiqHH
hpgmHnJla/7PZdkMZEsaCfVDEWBmJuMzVprJPT40anjG5VBLyM2I5DlsUCaeQCy2O3w3sqf1
3dyzUcv03IICuNc63towXA31Qt0TaVNU6YAmQjMepdfMbspmCZ+G8D2+xophEPPR/1vkstst
smUMqX0XrLonTUJczglPAgMBAAGjggJZMIICVTCBmgYIKwYBBQUHAQEEgY0wgYowUgYIKwYB
BQUHMAKGRmh0dHA6Ly93d3cudHJ1c3RjZW50ZXIuZGUvY2VydHNlcnZpY2VzL2NhY2VydHMv
dGNfdW5pdmVyc2FsX3Jvb3RfSS5jcnQwNAYIKwYBBQUHMAGGKGh0dHA6Ly9vY3NwLnRjdW5p
dmVyc2FsLUkudHJ1c3RjZW50ZXIuZGUwHwYDVR0jBBgwFoAUkqR1LKSevoFE63n8isWVpesQ
dXMwEgYDVR0TAQH/BAgwBgEB/wIBADBSBgNVHSAESzBJMAYGBFUdIAAwPwYJKoIUACwBAQEB
MDIwMAYIKwYBBQUHAgEWJGh0dHA6Ly93d3cudHJ1c3RjZW50ZXIuZGUvZ3VpZGVsaW5lczAO
BgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFOm4KB1Gz/zN+E6bxe5LYOvYOz/RMIH9BgNVHR8E
gfUwgfIwge+ggeyggemGRmh0dHA6Ly9jcmwudGN1bml2ZXJzYWwtSS50cnVzdGNlbnRlci5k
ZS9jcmwvdjIvdGNfdW5pdmVyc2FsX3Jvb3RfSS5jcmyGgZ5sZGFwOi8vd3d3LnRydXN0Y2Vu
dGVyLmRlL0NOPVRDJTIwVHJ1c3RDZW50ZXIlMjBVbml2ZXJzYWwlMjBDQSUyMEksTz1UQyUy
MFRydXN0Q2VudGVyJTIwR21iSCxPVT1yb290Y2VydHMsREM9dHJ1c3RjZW50ZXIsREM9ZGU/
Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlPzANBgkqhkiG9w0BAQUFAAOCAQEAOcjE
m+6+mO5Icm+N53G2DpCM07LBFSGoRpBoX0oE8TrJaIQh2KXmBHVdn9LU8kt3QzLclctgvwJV
0KwcsMUUl5tlCsMPpR3s2Ek5lbWpvvr0HqtW56blAQiINV9nBd1EJFASIkRjefGbV2nOq9Yz
UU+N8HA7jq1ROhd/NZZraGhjthwKyfjfHV7PKxGlY+3M0MbTIG+q/GhIfm0euDpFqhKG88e9
ALXr/uoSn3MzeOcoOWjTpW3adtFO4VWVgKbgG7jNrFbvRVlHmFLbOm4msjE5aXWxLiTwpJ2X
iF4zKca1vAdAOgw9us90jEtOeiH6GzjNxEMvb7TfeO6Zkuc6HDGCA84wggPKAgEBMIGPMHwx
CzAJBgNVBAYTAkRFMRwwGgYDVQQKExNUQyBUcnVzdENlbnRlciBHbWJIMSUwIwYDVQQLExxU
QyBUcnVzdENlbnRlciBDbGFzcyAxIEwxIENBMSgwJgYDVQQDEx9UQyBUcnVzdENlbnRlciBD
bGFzcyAxIEwxIENBIElYAg8ApksAAQACAIr42UPEgb8wCQYFKw4DAhoFAKCCAhMwGAYJKoZI
hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTMwNDA1MTYzODUwWjAjBgkq
hkiG9w0BCQQxFgQUHxv/K+bf1DIc3hHVPnUWsdJ/pvEwbAYJKoZIhvcNAQkPMV8wXTALBglg
hkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggq
hkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBoAYJKwYBBAGCNxAEMYGSMIGP
MHwxCzAJBgNVBAYTAkRFMRwwGgYDVQQKExNUQyBUcnVzdENlbnRlciBHbWJIMSUwIwYDVQQL
ExxUQyBUcnVzdENlbnRlciBDbGFzcyAxIEwxIENBMSgwJgYDVQQDEx9UQyBUcnVzdENlbnRl
ciBDbGFzcyAxIEwxIENBIElYAg8ApksAAQACAIr42UPEgb8wgaIGCyqGSIb3DQEJEAILMYGS
oIGPMHwxCzAJBgNVBAYTAkRFMRwwGgYDVQQKExNUQyBUcnVzdENlbnRlciBHbWJIMSUwIwYD
VQQLExxUQyBUcnVzdENlbnRlciBDbGFzcyAxIEwxIENBMSgwJgYDVQQDEx9UQyBUcnVzdENl
bnRlciBDbGFzcyAxIEwxIENBIElYAg8ApksAAQACAIr42UPEgb8wDQYJKoZIhvcNAQEBBQAE
ggEAFn6FSF038qyMGUYl1VtsoCpXsoQU8AP1E9C6x5iOg3kEK/0hXv0OZq7XuAdDZCryRBxj
CzUs25YPHdGANcXOJkNpq8EvYFiIgUFZj3jrc8bl2qFs0bm9YN955CsQIXkG32wVs8QcpTup
EnuTssA2/zNePYZLpUKiOjyuEiD0EzfiThJA2UlrFzXGWxSIN+wTfv7hJuIpNttYiKo19PQf
ITI6C2+LtiGMnnZ2AREgTgRhkJyrYPGwOFoEKmuwo7f6GKnA1WMa8+LEuceQXvGL9YObolq0
PVd6W+GRxWhXixQMkEk5Jwi7XV3tyRkV45c5ZA79ZiD/ldjV/k+2HcSF7gAAAAAAAA==
--------------ms070208010706030608090702--
10 years, 8 months
(ITS#7564) LMDB library bug: reopening LMDB may fail, fixed-address mmap
by tamas.kenez＠kishonti.net
Full_Name: Tamás Kenéz
Version: LMDB 8eef7a42
OS: windows 7
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (82.141.191.242)
Using the LMDB library (commit 8eef7a4275eda) on windows 7:
To reproduce: Run mtest multiple times.
The test sometimes fails with the with the windows error code 487
(ERROR_INVALID_ADDRESS) in mdb_env_open -> mdb_env_open2 at this line
env->me_map = MapViewOfFileEx(mh, flags & MDB_WRITEMAP ?
FILE_MAP_WRITE : FILE_MAP_READ,
0, 0, env->me_mapsize, meta.mm_address);
The reason is that when windows fails to map at the specified address
(meta.mm_address) it returns 0.
I tried this quick workaround:
if ( !env->me_map )
env->me_map = MapViewOfFileEx(mh, flags & MDB_WRITEMAP ?
FILE_MAP_WRITE : FILE_MAP_READ,
0, 0, env->me_mapsize, NULL);
which does return valid address, but then it fails later here:
else if (meta.mm_address && env->me_map != meta.mm_address) {
/* Can happen because the address argument to mmap() is just a
* hint. mmap() can pick another, e.g. if the range is in use.
* The MAP_FIXED flag would prevent that, but then mmap could
* instead unmap existing pages to make room for the new map.
*/
return EBUSY; /* TODO: Make a new MDB_* error code? */
10 years, 8 months
Re: (ITS#7515) Nested liblmdb transaction bugs
by hyc＠symas.com
h.b.furuseth(a)usit.uio.no wrote:
> I wrote:
>>> Can't use a DBI from an aborted child txn, unlike aborted main txn:
>> (...)
>> Well, closing the DBI in mdb_txn_abort() caused problems. A fix
>> in the opposite direction is easy, if we want DBIs to work that
>> way: mdb_env_open() can open the DBI in any parent txns too.
>
> To get more consistent behavior, if that wasn't clear: Any parent
> txn will receive the DBI independently of commit/abort, just like
> the MDB_env. Future children of a preexisting txn still differ,
> which is probably for the best. Changing that would take locking.
>
After much discussion, this part has been reverted. The rationale is simple:
nothing created inside a TXN may be visible outside the TXN until the TXN
successfully commits. On an abort or failed commit, all DBIs opened in the TXN
are automatically closed, because technically they never existed.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 8 months
Re: (ITS#7563) slapd modifies attribute value of pwdAttribute
by hyc＠symas.com
dieter(a)dkluenter.de wrote:
> Full_Name:
> Version: 2.4.33
> OS: openSuSE-12.3-x86_64
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (91.65.235.202)
>
>
> the pwdAttribute type requires a syntax of 1.3.6.1.4.1.1466.115.121.1.38,
> according to man slapo-ppolicy and ppolicy.schema.
> when adding a policy, the value of pwdAttribute gets changend from OID 2.5.4.35
> to userPassword.
You are mistaken. slapd never changes this attribute from what the user stored.
> In a replicated system syncrepl complaints about
> syncrepl_message_to_entry: rid=001 mods check (pwdAttribute: value #0 invalid
> per syntax) do_syncrepl: rid=001 rc 21 retrying.
This error will go away if you configure the ppolicy overlay on the consumer.
Closing this ITS.
>
> -Dieter
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 8 months
(ITS#7563) slapd modifies attribute value of pwdAttribute
by dieter＠dkluenter.de
Full_Name:
Version: 2.4.33
OS: openSuSE-12.3-x86_64
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (91.65.235.202)
the pwdAttribute type requires a syntax of 1.3.6.1.4.1.1466.115.121.1.38,
according to man slapo-ppolicy and ppolicy.schema.
when adding a policy, the value of pwdAttribute gets changend from OID 2.5.4.35
to userPassword.
In a replicated system syncrepl complaints about
syncrepl_message_to_entry: rid=001 mods check (pwdAttribute: value #0 invalid
per syntax) do_syncrepl: rid=001 rc 21 retrying.
-Dieter
10 years, 8 months
(ITS#7561) mdb_drop bugs
by h.b.furuseth＠usit.uio.no
Full_Name: Hallvard B Furuseth
Version: mdb.master 8eef7a4275eda8f2fa2e0d1e67c1d5cbcd91607e
OS: Linux x86_64
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (193.69.163.163)
Submitted by: hallvard
mdb_drop() does not seem to return overflow pages to the freelist.
Does not call cursor_del or anything, nor 'ovpages' loop like it has.
E.g. the enclosed program which writes some pages and then drops the DB
shows 59 free of 62 useed pages with non-overflows, but just
3 free of 38-39 with overflows.
#include <lmdb.h>
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
MDB_env *env;
MDB_txn *txn;
MDB_dbi dbi;
int rc, sz, sum;
# define DBPATH "test.mdb"
# define E(e) ((void) (rc = (e)), assert(!rc))
for (sz = 1500; sz < 9000; sz *= 2) {
MDB_val key = {sizeof(sum), &sum}, data = {sz, NULL};
printf("======== datasize %d ========\n", sz);
fflush(NULL);
remove(DBPATH);
E(mdb_env_create(&env));
E(mdb_env_set_maxdbs(env, 1));
E(mdb_env_open(env, DBPATH, MDB_NOSUBDIR, 0666));
E(mdb_txn_begin(env, 0, 0, &txn)); {
E(mdb_dbi_open(txn, "foo", MDB_CREATE, &dbi));
for (sum = 0; (sum += sz) < 100000; )
E(mdb_put(txn, dbi, &key, &data, MDB_RESERVE));
E(mdb_txn_commit(txn));
}
E(mdb_txn_begin(env, 0, 0, &txn)); {
E(mdb_drop(txn, dbi, 1));
E(mdb_txn_commit(txn));
}
mdb_env_close(env);
system("../mdb_stat -naef " DBPATH " | grep -v ': 0$'");
}
return 0;
}
10 years, 8 months
Re: (ITS#7447) backsql and german umlaute
by masarati＠aero.polimi.it
On 04/04/2013 08:13 AM, tomas.novosad(a)linuxbox.cz wrote:
> Hello,
>
> i got exactly same problem.
>
> Only the discussed character is different ;-)).
> When ThunderBird (or ldapsearch, it doesnt matter) send search query to LDAP with some UTF-8 character,
> the result query to DB (PGSQL in this case) is like
> (upper(last_name) LIKE '%šEV%')
>
> where the search parameter is:
> %<lower case utf8 character>EV%
>
> obviously backsql does not correctly handle UTF8 characters.
>
> I can't find any way how to avoid this.
> If only back-sql would leave the upper case conversion on DB - like
> this:
> (upper(last_name) LIKE upper('%šEV%'T
The solution is to augment table ldap_attr_mappings (with non-trivial
implications on DN searching and matching) with a field that specifies
the encoding for a particular attribute, and convert back and forth any
time an operation affects those attributes. Not trivial, but
contributions are welcome.
p.
--
Pierangelo Masarati
Associate Professor
Dipartimento di Scienze e Tecnologie Aerospaziali
Politecnico di Milano
10 years, 8 months
(ITS#7447) backsql and german umlaute
by tomas.novosad＠linuxbox.cz
Hello,
i got exactly same problem.
Only the discussed character is different ;-)).
When ThunderBird (or ldapsearch, it doesnt matter) send search query to LDAP with some UTF-8 character,
the result query to DB (PGSQL in this case) is like
(upper(last_name) LIKE '%šEV%')
where the search parameter is:
%<lower case utf8 character>EV%
obviously backsql does not correctly handle UTF8 characters.
I can't find any way how to avoid this.
If only back-sql would leave the upper case conversion on DB - like
this:
(upper(last_name) LIKE upper('%šEV%'))
or use ILIKE
Anyone has any suggestion how to workaround this?
Thanks in advance
--
Tomáš Novosad
10 years, 8 months
