Re: (ITS#5662) Comments in schema declarations separated by semicolon
by h.b.furuseth@usit.uio.no
michael(a)stroeder.com writes:
>hyc(a)symas.com wrote:
>>Michael Ströder wrote:
>>> The comments wouldn't have to persist in cn=config.
>>
>> Perhaps not, but that makes the data in cn=config less comprehensible
>> than it would be in slapd.conf.
Well, that's no different from #comments at the beginning of lines.
Or the default values which back-config inserts. One can't tell
whether a value is present because the admin knew and thought about
it or because cn=config defaulted it.
However:
>> Who benefits from this feature?
>
> An admin copying&pasting a schema from an standard document which uses
> this format. I'm currently looking at such a document with ~500
> occurences of OIDs used in declarations instead of NAMEs.
Which one? It's not RFC 4512 format. RFC 4512 uses ';' for comments
_about_ the syntax of schema elements, not _in_ their syntax.
--
Hallvard
15 years, 3 months
Re: (ITS#5664) Deadlocks when writing in parallell (two processes)
by stelios.xx.grigoriadis@ericsson.com
tom.bjorkholm(a)aastra.com wrote:
> Full_Name: Stelios Grigoriadis & Tom Björkholm
> Version: 2.3.39
> OS: Novell SLES 10
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (194.237.142.7)
>
>
> We get a lot of DB_LOCK_DEADLOCK when using client programs that for a period of
> time continuously writes to OpenLDAP.
> Version is 2.3.39.
>
> The information added is of the form:
> ebcmdCustomer=0+ebcmdDir=220xx,ou=AuthCodes,ebcmdVersion=0,ebcmdProduct=ebcmd,dc=example,dc=com
> where xx varies.
>
> Snippet of the output:
> Mar 27 13:03:21 ldapt1 slapd[7589]: => bdb_dn2id_add: subtree
> (ebcmdCustomer=0+ebcmdDir=22037,ou=authcodes,ebcmdVersion=0,ebcmdProduct=ebcmd,dc=example,dc=com)
> put failed: -30995
> Mar 27 13:03:26 ldapt1 slapd[7589]: => bdb_idl_insert_key: c_put id failed:
> DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
> Mar 27 13:03:26 ldapt1 slapd[7589]: => bdb_dn2id_add: parent
> (ou=authcodes,ebcmdVersion=0,ebcmdProduct=ebcmd,dc=example,dc=com) insert
> failed: -30995
> Mar 27 13:03:28 ldapt1 slapd[7589]: => bdb_idl_insert_key: c_put id failed:
> DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
> Mar 27 13:03:28 ldapt1 slapd[7589]: => bdb_dn2id_add: parent
> (ou=authcodes,ebcmdVersion=0,ebcmdProduct=ebcmd,dc=example,dc=com) insert
> failed: -30995
> Mar 27 13:03:36 ldapt1 slapd[7589]: => bdb_idl_insert_key: c_put id failed:
> DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
> Mar 27 13:03:36 ldapt1 slapd[7589]: => bdb_dn2id_add: parent
> (ou=authcodes,ebcmdVersion=0,ebcmdProduct=ebcmd,dc=example,dc=com) insert
> failed: -30995
> Mar 27 13:03:38 ldapt1 slapd[7589]: => bdb_idl_insert_key: c_put id failed:
> DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
>
>
>
We've temporarily fixed the problem by introducing a static mutex before
any add/update operation.
By doing so, we have effectively serialized the add/update operations
from within slapd. This is just
intended as a temporary solution as we hope the issue will be resolved
in future releases.
The patch:
# This patch file is derived from OpenLDAP Software. All of the
modifications to OpenLDAP Software
# represented in the following patch(es) were developed by Stelios
Grigoriadis stelios.xx.grigoriadis(a)ericsson.com.
# These modifications are not subject to any license of Ericsson AB.
# I, Stelios Grigoriadis, hereby place the following modifications to
OpenLDAP Software (and only these modifications)
# into the public domain. Hence, these modifications may be freely used
and/or redistributed for any purpose with or
# without attribution and/or other notice.
# Bug Fix - This patch fixes the bug ITS#5133.
# The fix works as follows. A periodic check in the runqueue (called
do_mastercheck). The intervall is determined by
# a slapd.conf parameter (mastercheckint) in the syncrepl section and is
optional. If it's not specified, it's not
# inserted in the runqueue.
--- servers/slapd/connection.c 2007-06-15 01:49:38.000000000 +0200
+++ connection.c 2008-06-10 16:30:08.000000000 +0200
@@ -1052,19 +1052,24 @@
/* FIXME: returns 0 in case of failure */
ldap_pvt_mp_add_ulong(slap_counters.sc_ops_initiated, 1);
ldap_pvt_thread_mutex_unlock( &slap_counters.sc_ops_mutex );
+ static pthread_mutex_t op_upd_mutex = PTHREAD_MUTEX_INITIALIZER;
+ int upd_tag=0;
op->o_threadctx = ctx;
#ifdef LDAP_DEVEL
op->o_tid = ldap_pvt_thread_pool_tid( ctx );
#endif /* LDAP_DEVEL */
switch ( tag ) {
- case LDAP_REQ_BIND:
- case LDAP_REQ_UNBIND:
case LDAP_REQ_ADD:
case LDAP_REQ_DELETE:
case LDAP_REQ_MODDN:
case LDAP_REQ_MODIFY:
+ ldap_pvt_thread_mutex_lock( &op_upd_mutex );
+ upd_tag=1;
+ break;
+ case LDAP_REQ_BIND:
+ case LDAP_REQ_UNBIND:
case LDAP_REQ_COMPARE:
case LDAP_REQ_SEARCH:
case LDAP_REQ_ABANDON:
@@ -1178,6 +1183,10 @@
connection_resched( conn );
ldap_pvt_thread_mutex_unlock( &conn->c_mutex );
+ if (upd_tag) {
+ ldap_pvt_thread_mutex_unlock( &op_upd_mutex );
+ }
+
return NULL;
}
15 years, 3 months
Re: (ITS#5662) Comments in schema declarations separated by semicolon
by michael@stroeder.com
hyc(a)symas.com wrote:
> Michael Ströder wrote:
>> Howard Chu wrote:
>>> As for truncating the comments on input - kind of pointless. In
>>> particular, that means those comments won't be persisted in cn=config.
>> The comments wouldn't have to persist in cn=config.
>
> Perhaps not, but that makes the data in cn=config less comprehensible than it
> would be in slapd.conf. On the other hand, OID macros *are* preserved in
> cn=config, which is yet another reason to use them.
>
> Who benefits from this feature?
An admin copying&pasting a schema from an standard document which uses
this format. I'm currently looking at such a document with ~500
occurences of OIDs used in declarations instead of NAMEs.
> The comment is there for the benefit of a
> human admin who has to read this definition. If the comment is stripped, then
> the benefit is lost; anyone inspecting the schema remotely will just be faced
> with the unwieldy numeric OIDs.
Well, in web2ldap's schema viewer I'm always trying to display the NAME
instead of the OID if possible. slapd could also normalize the schema
declarations to list the NAME instead of OID if available (but that
would be another ITS).
> It still wouldn't persist because e.g. at_unparse() (which provides the value
> that actually gets stored) works by constructing the string out of the LDAP
> Attribute structure. So again, all of the comments would be gone, and there
> would be no embedded line endings.
My request was simply to truncate and ignore the comments.
Ciao, Michael.
15 years, 3 months
Re: (ITS#5662) Comments in schema declarations separated by semicolon
by hyc@symas.com
Michael Ströder wrote:
> Howard Chu wrote:
>> As for truncating the comments on input - kind of pointless. In
>> particular, that means those comments won't be persisted in cn=config.
>
> The comments wouldn't have to persist in cn=config.
Perhaps not, but that makes the data in cn=config less comprehensible than it
would be in slapd.conf. On the other hand, OID macros *are* preserved in
cn=config, which is yet another reason to use them.
Who benefits from this feature? The comment is there for the benefit of a
human admin who has to read this definition. If the comment is stripped, then
the benefit is lost; anyone inspecting the schema remotely will just be faced
with the unwieldy numeric OIDs.
Remember also, slapd.conf is going to be phased out down the road. It's better
not to add features to it that people are going to rely on, if the analogous
feature won't be exposed in cn=config. Instead, we should be making use of
whatever features already exist that are supported in both.
Hallvard wrote:
> back-config could accept
> attributeTypes:: base64("description with line endings and comments")
> if it doesn't already.
It still wouldn't persist because e.g. at_unparse() (which provides the value
that actually gets stored) works by constructing the string out of the LDAP
Attribute structure. So again, all of the comments would be gone, and there
would be no embedded line endings.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
15 years, 3 months
Re: (ITS#5662) Comments in schema declarations separated by semicolon
by h.b.furuseth@usit.uio.no
Michael Ströder writes:
>> but someone who schema via some LDAP-friendly tool might.
>
> ???
>
> My schema viewer always trys to display NAME(s) of any schema element
> linked.
Oops, some words disappeard. I meant someone who reads/writes schema
using a LDAP-friendly (and LDIF-friendly) tool, so they'd see the
base64-decoded version of the attribute.
--
Hallvard
15 years, 3 months
Re: (ITS#5662) Comments in schema declarations separated by semicolon
by michael@stroeder.com
h.b.furuseth(a)usit.uio.no wrote:
> Personally I wouldn't write such a file anytime
> soon,
Me neither. But if you just copy&paste schema declarations from a RFC or
I-D it would be handy to accept this.
> but someone who schema via some LDAP-friendly tool might.
???
My schema viewer always trys to display NAME(s) of any schema element
linked.
Ciao, Michael.
15 years, 3 months
Re: (ITS#5662) Comments in schema declarations separated by semicolon
by hyc@symas.com
Michael Ströder wrote:
> hyc(a)symas.com wrote:
>> michael(a)stroeder.com wrote:
>>> It might be helpful if the schema config file parser would accept lines with
>>> comments at the end separated by semicolon.
>> The parser doesn't know anything about line endings. Remember, lines with
>> leading white space are continuations of the previous line. The schema parser
>> is just fed a single string as one long line, the line endings were removed
>> long before it ever got there.
>
> Maybe lines with ; could be truncated before passing it to the schema
> parser? I don't know if that has any impact on processing slapd.conf in
> general.
> The slapd.conf parser has to already know in advance that this
> part is a schema declaration (started by key-word objectclass etc.) and
> could change pre-processing?
Actually no, the parser doesn't know there's anything special about schema
declarations. It doesn't parse the lines any differently than any other
directives. Nor should it.
As for truncating the comments on input - kind of pointless. In particular,
that means those comments won't be persisted in cn=config.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
15 years, 3 months
Re: (ITS#5662) Comments in schema declarations separated by semicolon
by h.b.furuseth@usit.uio.no
hyc(a)symas.com writes:
> It would be smarter to just standardize OID macros for each of these
> OIDs, the way X.500 did, and not require the humans reading these
> specs to memorize these numeric OIDs...
Yup. slapd/schema/syntaxes.schema?
Syntax names like olsDirectoryString or just DirectoryString?
(I don't know if OID macro names are a separate namespace or not.)
> I don't see any straightforward way to modify our parser as you're
> suggesting. Unless someone else feels like jumping in, I think we
> should reject this ITS.
back-config could accept
attributeTypes:: base64("description with line endings and comments")
if it doesn't already. Personally I wouldn't write such a file anytime
soon, but someone who schema via some LDAP-friendly tool might.
--
Hallvard
15 years, 3 months
Re: (ITS#5662) Comments in schema declarations separated by semicolon
by michael@stroeder.com
hyc(a)symas.com wrote:
> michael(a)stroeder.com wrote:
>> It might be helpful if the schema config file parser would accept lines with
>> comments at the end separated by semicolon.
>
> The parser doesn't know anything about line endings. Remember, lines with
> leading white space are continuations of the previous line. The schema parser
> is just fed a single string as one long line, the line endings were removed
> long before it ever got there.
Maybe lines with ; could be truncated before passing it to the schema
parser? I don't know if that has any impact on processing slapd.conf in
general. The slapd.conf parser has to already know in advance that this
part is a schema declaration (started by key-word objectclass etc.) and
could change pre-processing?
Ciao, Michael.
15 years, 3 months