Re: (ITS#7457) residentialPerson contains preferredDeliveryMethod twice
by kurt＠OpenLDAP.org
Per RFC 4519. See list archives for discussion. -- Kurt
On Nov 30, 2012, at 10:50 AM, peter(a)adpm.de wrote:
> Full_Name: Peter Marschall
> Version: 2.4.33
> OS: Linux
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (178.2.172.201)
>
>
> Hi,
>
> trying to convert the residentialPerson objectclass to a format for another
> directory server type, I noticed that it contains the attribute
> preferredDeliveryMethod twice in its MAY list.
>
> I do not know if this is an issue, but wanted to report it nonetheless.
>
> Thanks for OpenLDAP
> Peter
>
10 years, 3 months
(ITS#7457) residentialPerson contains preferredDeliveryMethod twice
by peter＠adpm.de
Full_Name: Peter Marschall
Version: 2.4.33
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (178.2.172.201)
Hi,
trying to convert the residentialPerson objectclass to a format for another
directory server type, I noticed that it contains the attribute
preferredDeliveryMethod twice in its MAY list.
I do not know if this is an issue, but wanted to report it nonetheless.
Thanks for OpenLDAP
Peter
10 years, 3 months
Re: (ITS#7456) slapo-smbk5pwd: Makefile with -L
by fumiyas＠osstech.jp
Hi,
At Wed, 28 Nov 2012 20:04:41 GMT,
michael(a)stroeder.com wrote:
> I'd like to propose the following patch to Makefile of slapo-smbk5pwd to ease
> building it without installed OpenLDAP client libs:
This ITS is a dup of #7309.
http://www.openldap.org/its/index.cgi?findid=7309
And see also:
http://www.openldap.org/its/index.cgi?findid=7308
This problem is fixed in master, but not in 2.4 branch...
> diff --git a/contrib/slapd-modules/smbk5pwd/Makefile
> b/contrib/slapd-modules/smbk5pwd/Makefile
> index 46b340a..01c7bc2 100644
> --- a/contrib/slapd-modules/smbk5pwd/Makefile
> +++ b/contrib/slapd-modules/smbk5pwd/Makefile
> @@ -26,7 +26,7 @@ INCS=$(LDAP_INC) $(HEIMDAL_INC) $(SSL_INC)
>
> HEIMDAL_LIB=-L/usr/heimdal/lib -lkrb5 -lkadm5srv
> SSL_LIB=-lcrypto
> -LDAP_LIB=-lldap_r -llber
> +LDAP_LIB=-L../../../libraries/libldap_r/.libs/ -lldap_r -llber
> LIBS=$(LDAP_LIB) $(HEIMDAL_LIB) $(SSL_LIB)
>
> prefix=/usr/local
--
-- Name: SATOH Fumiyasu (fumiyas @ osstech co jp)
-- Business Home: http://www.OSSTech.co.jp/
-- GitHub Home: https://GitHub.com/fumiyas/
10 years, 3 months
Re: (ITS#7455) MDB database grows without bound
by hyc＠symas.com
This is a multi-part message in MIME format.
--------------040901020606020603090609
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Howard Chu wrote:
> quanah(a)OpenLDAP.org wrote:
>> Full_Name: Quanah Gibson-Mount
>> Version: 2.4.33
>> OS: Linux 2.6
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (74.196.25.250)
>>
>>
>> I have a very small DB (about 25MB from a fresh slapadd). However, the data.mdb
>> file grows by about 50MB a day. I.e., the database size on disk *doubles* every
>> day. It is now up to 571MB in size after
>>
>>
>> Here is the DB after a fresh slapadd:
>> zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
>> 25M data.mdb
>>
>> Here is the DB on the production server:
>> [zimbra@ldap01-zcs db]$ du -c -h data.mdb
>> 571M data.mdb
>
> Based on the mdb_stat output you pasted, this is simply a case of overflow
> pages not reusing freelist pages. The significant info here is the freelist
> info and the number of overflow pages used in the id2e database.
Here's the patch we're currently testing for this issue.
It appears to work but is maybe not being aggressive enough in reclaiming
space. We may want to increase the number of retries a bit more.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
--------------040901020606020603090609
Content-Type: text/plain; charset=UTF-8;
name="diff.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="diff.txt"
diff --git a/libraries/libmdb/mdb.c b/libraries/libmdb/mdb.c
index 251ab6a..117b402 100644
--- a/libraries/libmdb/mdb.c
+++ b/libraries/libmdb/mdb.c
@@ -1242,6 +1242,7 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp)
MDB_page *np;
pgno_t pgno = P_INVALID;
MDB_ID2 mid;
+ txnid_t oldest = 0, last;
int rc;
*mp = NULL;
@@ -1254,12 +1255,11 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp)
if (!txn->mt_env->me_pghead &&
txn->mt_dbs[FREE_DBI].md_root != P_INVALID) {
/* See if there's anything in the free DB */
- int j;
MDB_reader *r;
MDB_cursor m2;
MDB_node *leaf;
MDB_val data;
- txnid_t *kptr, last;
+ txnid_t *kptr;
mdb_cursor_init(&m2, txn, FREE_DBI, NULL);
if (!txn->mt_env->me_pgfirst) {
@@ -1282,15 +1282,21 @@ again:
last = *(txnid_t *)key.mv_data;
}
- /* Unusable if referred by a meta page or reader... */
- j = 1;
- if (last < txn->mt_txnid-1) {
- j = txn->mt_env->me_txns->mti_numreaders;
- r = txn->mt_env->me_txns->mti_readers + j;
- for (j = -j; j && (last<r[j].mr_txnid || !r[j].mr_pid); j++) ;
+ {
+ unsigned int i, nr;
+ txnid_t mr;
+ oldest = txn->mt_txnid - 1;
+ nr = txn->mt_env->me_txns->mti_numreaders;
+ r = txn->mt_env->me_txns->mti_readers;
+ for (i=0; i<nr; i++) {
+ if (!r[i].mr_pid) continue;
+ mr = r[i].mr_txnid;
+ if (mr < oldest)
+ oldest = mr;
+ }
}
- if (!j) {
+ if (oldest > last) {
/* It's usable, grab it.
*/
MDB_oldpages *mop;
@@ -1331,29 +1337,108 @@ none:
if (txn->mt_env->me_pghead) {
MDB_oldpages *mop = txn->mt_env->me_pghead;
if (num > 1) {
- /* FIXME: For now, always use fresh pages. We
- * really ought to search the free list for a
- * contiguous range.
- */
- ;
+ MDB_cursor m2;
+ int retry = 2, readit = 0, n2 = num-1;
+ unsigned int i, j, k;
+
+ /* If current list is too short, must fetch more and coalesce */
+ if (mop->mo_pages[0] < (unsigned)num)
+ readit = 1;
+
+ mdb_cursor_init(&m2, txn, FREE_DBI, NULL);
+ do {
+ if (readit) {
+ MDB_val key, data;
+ MDB_oldpages *mop2;
+ pgno_t *idl;
+ int exact;
+
+ last = mop->mo_txnid + 1;
+
+ /* We haven't hit the readers list yet? */
+ if (!oldest) {
+ MDB_reader *r;
+ unsigned int nr;
+ txnid_t mr;
+
+ oldest = txn->mt_txnid - 1;
+ nr = txn->mt_env->me_txns->mti_numreaders;
+ r = txn->mt_env->me_txns->mti_readers;
+ for (i=0; i<nr; i++) {
+ if (!r[i].mr_pid) continue;
+ mr = r[i].mr_txnid;
+ if (mr < oldest)
+ oldest = mr;
+ }
+ }
+
+ /* There's nothing we can use on the freelist */
+ if (oldest - last < 1)
+ break;
+
+ exact = 0;
+ key.mv_data = &last;
+ key.mv_size = sizeof(last);
+ rc = mdb_cursor_set(&m2, &key, &data, MDB_SET, &exact);
+ if (rc)
+ return rc;
+ idl = (MDB_ID *) data.mv_data;
+ mop2 = malloc(sizeof(MDB_oldpages) + MDB_IDL_SIZEOF(idl) - 2*sizeof(pgno_t) + MDB_IDL_SIZEOF(mop->mo_pages));
+ if (!mop2)
+ return ENOMEM;
+ /* merge in sorted order */
+ i = idl[0]; j = mop->mo_pages[0]; mop2->mo_pages[0] = k = i+j;
+ mop->mo_pages[0] = P_INVALID;
+ while (i>0 || j>0) {
+ if (i && idl[i] < mop->mo_pages[j])
+ mop2->mo_pages[k--] = idl[i--];
+ else
+ mop2->mo_pages[k--] = mop->mo_pages[j--];
+ }
+ txn->mt_env->me_pglast = last;
+ mop2->mo_txnid = last;
+ mop2->mo_next = mop->mo_next;
+ txn->mt_env->me_pghead = mop2;
+ free(mop);
+ mop = mop2;
+ /* Keep trying to read until we have enough */
+ if (mop->mo_pages[0] < (unsigned)num) {
+ continue;
+ }
+ }
+
+ /* current list has enough pages, but are they contiguous? */
+ for (i=mop->mo_pages[0]; i>=(unsigned)num; i--) {
+ if (mop->mo_pages[i-n2] == mop->mo_pages[i] + n2) {
+ pgno = mop->mo_pages[i];
+ i -= n2;
+ /* move any stragglers down */
+ for (j=i+num; j<=mop->mo_pages[0]; j++)
+ mop->mo_pages[i++] = mop->mo_pages[j];
+ mop->mo_pages[0] -= num;
+ break;
+ }
+ }
+
+ /* Stop if we succeeded, or no more retries */
+ if (!retry || pgno != P_INVALID)
+ break;
+ readit = 1;
+ retry--;
+
+ } while (1);
} else {
/* peel pages off tail, so we only have to truncate the list */
pgno = MDB_IDL_LAST(mop->mo_pages);
- if (MDB_IDL_IS_RANGE(mop->mo_pages)) {
- mop->mo_pages[2]++;
- if (mop->mo_pages[2] > mop->mo_pages[1])
- mop->mo_pages[0] = 0;
+ mop->mo_pages[0]--;
+ }
+ if (MDB_IDL_IS_ZERO(mop->mo_pages)) {
+ txn->mt_env->me_pghead = mop->mo_next;
+ if (mc->mc_dbi == FREE_DBI) {
+ mop->mo_next = txn->mt_env->me_pgfree;
+ txn->mt_env->me_pgfree = mop;
} else {
- mop->mo_pages[0]--;
- }
- if (MDB_IDL_IS_ZERO(mop->mo_pages)) {
- txn->mt_env->me_pghead = mop->mo_next;
- if (mc->mc_dbi == FREE_DBI) {
- mop->mo_next = txn->mt_env->me_pgfree;
- txn->mt_env->me_pgfree = mop;
- } else {
- free(mop);
- }
+ free(mop);
}
}
}
--------------040901020606020603090609--
10 years, 3 months
Re: (ITS#7432) slapd crashes when removing members from a large group
by hyc＠symas.com
hyc(a)symas.com wrote:
> marvin.mundry(a)uni-hamburg.de wrote:
>> Full_Name: Marvin Mundry
>> Version: 2.4.33
>> OS: Ubuntu 12.10
>> URL: https://idmswiki.rrz.uni-hamburg.de:8005/debug.tar.bz
>> Submission from: (NULL) (134.100.2.183)
>>
>
> Thanks for the report. The crash has been fixed in git, but your test runs
> into another (known) issue in MDB.
>
> You're working with a very large entry, which libmdb stores in overflow pages.
> In the current version of libmdb, freespace management for overflow pages is
> not fully implemented, so every time you update the entry libmdb will always
> use new pages (instead of reusing old pages). Thus, after a few hundred
> operations, your 1GB map will be exhausted.
>
> It looks like you won't be able to use back-mdb until this feature is fully
> implemented in libmdb.
The overflow page freespace management issue is being addressed in ITS#7455. A
patch is currently being tested.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 3 months
Re: (ITS#7444) Check the return value of ldap_pvt_thread_set_concurrency
by hyc＠symas.com
Tianyin Xu wrote:
>
> On Sun, Nov 25, 2012 at 7:54 PM, Howard Chu <hyc(a)symas.com
> <mailto:hyc@symas.com>> wrote:
>
> tixu(a)cs.ucsd.edu <mailto:tixu@cs.ucsd.edu> wrote:
>
> Full_Name: Tianyin Xu
> Version: 2.4.33
> OS: Ubuntu 12.04 (actually doesn't matter)
> URL:
> Submission from: (NULL) (2607:f720:1300:1241:590b:49c:__889f:7b21)
>
>
> Hi, all,
>
> Still the verbosity idea. In the current version of slapd, when
> handling users'
> "concurrency" setting, slapd does not check the return value of
> "ldap_pvt_thread_set___concurrency" (essentially the return value of
> "pthread_setconcurrency") to see whether the setting is successful or not.
>
> So when the user setting is inappropriate (i.e., too big), slapd says
> nothing.
> But users have no way to be aware of such configuration failure. I
> suggest to
> check and notify users in this case.
>
>
> pthread_setconcurrency is a no-op on Linux systems. Even on systems where
> this setting might have an effect, it is generally unused. We have more
> important things to worry about than this.
>
>
> Yes, I understand that user configuration is never a high-priority issue.
> That's why I attached my patch which simply checks the return value, and
> prints out a log messge to notify users. The patch itself does not have any
> side effect to other part of code.
>
> I personally think the verbosity idea can improve the system reactions to
> users' configurations including this one and its #7443. It is safe (no side
> effect) and easy to do (simply adding some log messages), but can greatly
> improve the usability and friendlyness to configurations. If you agree, I'm
> happy to write patches and corresponding code.
We appreciate your desire to help, but as I said before, these two directives
are essentially unused in 99.999% of deployments. If you're looking for code
to improve, I suggest you review some of the other open reports in the ITS.
>
> Thanks,
> Tianyin
>
> Here's the patch:
>
> --- ../openldap-2.4.33/servers/__slapd/bconfig.c 2012-10-10
> 05:18:49.000000000 -0700
> +++ servers/slapd/bconfig.c 2012-11-16 14:52:37.211421828 -0800
> @@ -1530,7 +1530,14 @@
> break;
>
> case CFG_CONCUR:
> - ldap_pvt_thread_set___concurrency(c->value_int);
> +
> +
> if(ldap_pvt_thread_set___concurrency(c->value_int) != 0)
> {
> + snprintf( c->cr_msg, sizeof( c->cr_msg ),
> + "concurrency=%d is not
> valid; using
> the default setting",
> + c->value_int );
> + Debug(LDAP_DEBUG_ANY, "%s: %s.\n",
> c->log,
> c->cr_msg, 0 );
> +
> + }
> break;
>
> case CFG_THREADS:
>
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 3 months
Re: (ITS#7455) MDB database grows without bound
by hyc＠symas.com
quanah(a)OpenLDAP.org wrote:
> Full_Name: Quanah Gibson-Mount
> Version: 2.4.33
> OS: Linux 2.6
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (74.196.25.250)
>
>
> I have a very small DB (about 25MB from a fresh slapadd). However, the data.mdb
> file grows by about 50MB a day. I.e., the database size on disk *doubles* every
> day. It is now up to 571MB in size after
>
>
> Here is the DB after a fresh slapadd:
> zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
> 25M data.mdb
>
> Here is the DB on the production server:
> [zimbra@ldap01-zcs db]$ du -c -h data.mdb
> 571M data.mdb
Based on the mdb_stat output you pasted, this is simply a case of overflow
pages not reusing freelist pages. The significant info here is the freelist
info and the number of overflow pages used in the id2e database.
Environment Info
Map address: (nil)
Map size: 85899345920
Page size: 4096
Max pages: 20971520
Number of pages used: 146230
Last transaction ID: 192037
Max readers: 126
Number of readers used: 10
Freelist Status
Tree depth: 3
Branch pages: 7
Leaf pages: 668
Overflow pages: 32
Entries: 10671
Free pages: 141005
Status of Main DB
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 38
...
Status of ad2i
Tree depth: 2
Branch pages: 1
Leaf pages: 20
Overflow pages: 0
Entries: 976
Status of cn
Tree depth: 2
Branch pages: 1
Leaf pages: 231
Overflow pages: 0
Entries: 30224
Status of displayName
Tree depth: 2
Branch pages: 1
Leaf pages: 91
Overflow pages: 0
Entries: 13625
Status of dn2i
Tree depth: 2
Branch pages: 1
Leaf pages: 114
Overflow pages: 0
Entries: 5935
Status of entryCSN
Tree depth: 2
Branch pages: 1
Leaf pages: 37
Overflow pages: 0
Entries: 2967
Status of entryUUID
Tree depth: 2
Branch pages: 1
Leaf pages: 26
Overflow pages: 0
Entries: 2967
Status of givenName
Tree depth: 2
Branch pages: 1
Leaf pages: 24
Overflow pages: 0
Entries: 3318
Status of id2e
Tree depth: 3
Branch pages: 7
Leaf pages: 721
Overflow pages: 1937
Entries: 2968
Status of mail
Tree depth: 2
Branch pages: 1
Leaf pages: 141
Overflow pages: 0
Entries: 33250
Status of objectClass
Tree depth: 2
Branch pages: 1
Leaf pages: 9
Overflow pages: 0
Entries: 9353
Status of sn
Tree depth: 2
Branch pages: 1
Leaf pages: 53
Overflow pages: 0
Entries: 7502
Status of uid
Tree depth: 2
Branch pages: 1
Leaf pages: 13
Overflow pages: 0
Entries: 3148
...
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
10 years, 4 months
Re: (ITS#7455) MDB database grows without bound
by quanah＠zimbra.com
--On Wednesday, November 28, 2012 6:50 PM +0000 quanah(a)OpenLDAP.org wrote:
> Here is the DB after a fresh slapadd:
> zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
> 25M data.mdb
>
> Here is the DB on the production server:
> [zimbra@ldap01-zcs db]$ du -c -h data.mdb
> 571M data.mdb
In 1 week:
24,108 MODs
14 Adds
7 Deletes
For a client with a large database of several gigs, this will quickly lead
to significant problems, even with a large amount of free disk space. If I
have a client with a 12GB DB, then they'll hit their DB maxsize of 80GB in
just a few days.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10 years, 4 months
(ITS#7456) slapo-smbk5pwd: Makefile with -L
by michael＠stroeder.com
Full_Name:
Version: HEAD
OS:
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (79.227.167.175)
I'd like to propose the following patch to Makefile of slapo-smbk5pwd to ease
building it without installed OpenLDAP client libs:
diff --git a/contrib/slapd-modules/smbk5pwd/Makefile
b/contrib/slapd-modules/smbk5pwd/Makefile
index 46b340a..01c7bc2 100644
--- a/contrib/slapd-modules/smbk5pwd/Makefile
+++ b/contrib/slapd-modules/smbk5pwd/Makefile
@@ -26,7 +26,7 @@ INCS=$(LDAP_INC) $(HEIMDAL_INC) $(SSL_INC)
HEIMDAL_LIB=-L/usr/heimdal/lib -lkrb5 -lkadm5srv
SSL_LIB=-lcrypto
-LDAP_LIB=-lldap_r -llber
+LDAP_LIB=-L../../../libraries/libldap_r/.libs/ -lldap_r -llber
LIBS=$(LDAP_LIB) $(HEIMDAL_LIB) $(SSL_LIB)
prefix=/usr/local
10 years, 4 months
(ITS#7455) MDB database grows without bound
by quanah＠OpenLDAP.org
Full_Name: Quanah Gibson-Mount
Version: 2.4.33
OS: Linux 2.6
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (74.196.25.250)
I have a very small DB (about 25MB from a fresh slapadd). However, the data.mdb
file grows by about 50MB a day. I.e., the database size on disk *doubles* every
day. It is now up to 571MB in size after
Here is the DB after a fresh slapadd:
zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
25M data.mdb
Here is the DB on the production server:
[zimbra@ldap01-zcs db]$ du -c -h data.mdb
571M data.mdb
10 years, 4 months
