Re: (ITS#7589) MDB nodesize issues
by hyc@symas.com
h.b.furuseth(a)usit.uio.no wrote:
> I wrote:
>> OTOH if you add a bunch of slightly smaller nodes, mdb will put
>> most of them in separate pages anyway without MDB_APPEND.
>
> ...because mdb_page_split() has been wasteful since 48ef27b6f5c804eca6a9
> "ITS#7385 fix mdb_page_split (again)". When a txn put()s ascending
> keys with nodes of the same size, the new item goes in the fullest page.
>
> E.g. put data size 1010 with 'int' keys 1,2,3... to an MDB_INTEGERKEY
> DB. As the txn progresses, (page: #key #key...) get distributed thus:
>
> Page 2: #1.
> Page 2: #1 #2.
> Page 2: #1 #2 #3.
> Page 2: #1. Page 3: #2 #3 #4.
> Page 2: #1. Page 3: #2. Page 5: #3 #4 #5.
> Page 2: #1. Page 3: #2. Page 5: #3. Page 6: #4 #5 #6.
>
> 2/3 wasted space. Descending put() work better:
>
> Page 2: #6.
> Page 2: #5 #6.
> Page 2: #4 #5 #6.
> Page 2: #3 #4. Page 3: #5 #6.
> Page 2: #2 #3 #4. Page 3: #5 #6.
> Page 2: #2 #1. Page 3: #5 #6. Page 5: #3 #4.
>
> Ascending put() with datasize 1348, so only 2 nodes fit in a page:
>
> Page 2: #1.
> Page 2: #1 #2.
> Page 2: #1. Page 3: #2 #3.
> Page 2: #1. Page 3: #2. Page 5: #3 #4.
>
> Half of the space is wasted. Descending order does not help.
>
> page_split() cannot know which split is best in this case. But it'll
> help to guess that the next put() key sometimes will be near this one,
> and ensure that the node with the new key will be the smallest. That
> will also avoid touching the old page when the nodes are that large,
> since the "split" will keep all old nodes in the old page.
Fixed now in mdb.master. On a large DB, the previous code used 3276587 pages
in slapadd -q, and the new code uses 3272633 pages. It's only a 0.13% savings
in this case, it seems the frequency of these insert patterns is quite rare.
The runtime is also 1.1% faster going from
real 41m35.8s user 50m57.6s sys 5m11.4s
to
real 41m1.7s user 50m25.8s sys 4m55.3s
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9 years, 11 months
Re: (ITS#7723) slapd crashes on multi core machines if a search request is *immediately* followed by an unbind
by hyc@symas.com
hyc(a)symas.com wrote:
> michael.vishchers(a)7p-group.com wrote:
>> Full_Name: Michael Vishchers
>> Version: 2.4.36
>> OS: RHEL 6.3
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (109.41.86.33)
>>
>>
>> The bug mentioned in ITS #7716 unfortunately still persists in 2.4.36.
>>
>> We now found a way to reproduce it:
If you want anyone to spend time on this:
Send the stack trace from the crash. Also send your slapd config, sample data,
and the actual query used in your client.
>>
>> - make sure that threads can run freely (i.e., provide enough cpus)
>
>> - start slapd with the rwm overlay
>> - run a client loop that
>> -- binds as a valid user
>> -- sends a valid search request that is *immediately* (i.e., don't wait for any
>> answers) followed by an unbind request
>>
>> after several iterations (with 2.4.23 it was between 2 and 7, with 2.4.36 it was
>> about 30) slapd crashes in malloc.c
>>
>>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9 years, 11 months
Re: (ITS#7723) slapd crashes on multi core machines if a search request is *immediately* followed by an unbind
by hyc@symas.com
michael.vishchers(a)7p-group.com wrote:
> Full_Name: Michael Vishchers
> Version: 2.4.36
> OS: RHEL 6.3
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (109.41.86.33)
>
>
> The bug mentioned in ITS #7716 unfortunately still persists in 2.4.36.
>
> We now found a way to reproduce it:
>
> - make sure that threads can run freely (i.e., provide enough cpus)
If your client loop is running as a single thread then the number of CPUs and
running threads should be irrelevant. If the number of threads and CPUs matter
then you're misusing the API. Closing this ITS.
> - start slapd with the rwm overlay
> - run a client loop that
> -- binds as a valid user
> -- sends a valid search request that is *immediately* (i.e., don't wait for any
> answers) followed by an unbind request
>
> after several iterations (with 2.4.23 it was between 2 and 7, with 2.4.36 it was
> about 30) slapd crashes in malloc.c
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9 years, 11 months
Re: (ITS#7724) Invalid FREE suspected in unbind.c
by hyc@symas.com
michael.vishchers(a)7p-group.com wrote:
> Full_Name: Michael Vishchers
> Version: 2.4.36
> OS: RHEL 6.3
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (109.41.86.33)
>
>
> This may be connected to ITS #7723:
>
> in unbind.c, there is new code (newer than 2.4.23, i.e.) that takes care not to
> free the ld struct if some other thread is using it (line 82ff)
>
> Unfortunately, in line 102, after decrementing the ref count in this struct,
> this struct is freed nonetheless.
The struct is freed because it should only be used by one thread.
Read draft-zeilenga-ldap-c-api-concurrency. Closing this ITS.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9 years, 11 months
Re: (ITS#7720) 2.4.36 build failure on HP-UX
by h.b.furuseth@usit.uio.no
On 2013-10-08 19:42, hyc(a)symas.com wrote:
> Thanks for the report. Interesting, this should also have been flagged
> as
> illegal in gcc but wasn't. Fixed now in master.
void* pointer arithmetic is a gcc extension, it's treated as char*.
gcc warns about it if you use -pedantic or -Wpointer-arith.
--
Hallvard
9 years, 11 months
(ITS#7724) Invalid FREE suspected in unbind.c
by michael.vishchers@7p-group.com
Full_Name: Michael Vishchers
Version: 2.4.36
OS: RHEL 6.3
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (109.41.86.33)
This may be connected to ITS #7723:
in unbind.c, there is new code (newer than 2.4.23, i.e.) that takes care not to
free the ld struct if some other thread is using it (line 82ff)
Unfortunately, in line 102, after decrementing the ref count in this struct,
this struct is freed nonetheless.
9 years, 11 months
(ITS#7723) slapd crashes on multi core machines if a search request is *immediately* followed by an unbind
by michael.vishchers@7p-group.com
Full_Name: Michael Vishchers
Version: 2.4.36
OS: RHEL 6.3
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (109.41.86.33)
The bug mentioned in ITS #7716 unfortunately still persists in 2.4.36.
We now found a way to reproduce it:
- make sure that threads can run freely (i.e., provide enough cpus)
- start slapd with the rwm overlay
- run a client loop that
-- binds as a valid user
-- sends a valid search request that is *immediately* (i.e., don't wait for any
answers) followed by an unbind request
after several iterations (with 2.4.23 it was between 2 and 7, with 2.4.36 it was
about 30) slapd crashes in malloc.c
9 years, 11 months
Re: (ITS#7710) contextCSN values not updated by internal non-replicated ops
by quanah@zimbra.com
--On Thursday, October 10, 2013 11:22 AM +0000 michael(a)stroeder.com wrote:
> On Thu, 10 Oct 2013 11:05:28 GMT hyc(a)symas.com wrote
>> michael(a)stroeder.com wrote:
>> > With current master and re24 which have the fix for this ITS I now see
>> > seg faults in test023-refint test057-memberof-refint.
>>
>> Fixed now in master.
>
> Thanks. These tests are passing now.
>
> @Quanah: Would be nice if this last fix could be also ported to RE24.
> Thanks in advance.
Not particularly necessary to make requests like this. I read all ITS mail
and commit traffic.
--Quanah
--
Quanah Gibson-Mount
Architect - Server
Zimbra Software, LLC
--------------------
Zimbra :: the leader in open source messaging and collaboration
9 years, 11 months
Re: (ITS#7694) cldap fails with IPv6 due to wrong size sockaddr
by stefw@redhat.com
On 10.10.2013 13:59, Howard Chu wrote:
> Stef Walter wrote:
>> On 10.10.2013 12:59, Howard Chu wrote:
>>> stefw(a)redhat.com wrote:
>>>> Full_Name: Stef Walter
>>>> Version: 2.4.35
>>>> OS: Fedora 19
>>>> Submission from: (NULL) (46.5.2.70)
>>>>
>>>>
>>>> Connectionless LDAP (ie: cldap enabled with -DLDAP_CONNECTIONLESS) is
>>>> broken for
>>>> IPv6 for current versions of openldap. Tested with version 2.4.35
>>>>
>>>> It's not clear if this ever worked properly.
>>>
>>> No, clearly not, the code was written and deprecated before IPv6
>>> existed. Nobody should be using this code today.
>>
>> Interesting. FWIW, the code is packaged by RHEL and Fedora, and is in
>> use by several projects.
>
> Can you list any of these, offhand? The original spec, RFC1798, is long
> obsoleted. There is no such thing as CLDAP in LDAPv3. Support in
> OpenLDAP was first removed back in 2000. (commit
> 25a9f7427ddc1b584a721ceb0e12690a96d3639e )
> Any apps using this must be quite ancient code and in serious need of a
> rewrite.
Well, there's still lots of libldap client code around to support LDAP
over UDP. Guarded with LDAP_CONNECTIONLESS #defines, and one can use
"cldap://xxxx" urls with ldap_initialize() and do basic cldap searches
and so on.
Windows Server is accessed via CLDAP during discovery. Although there is
normative documentation for this, it's easier to understand via these
descriptions:
http://wiki.wireshark.org/MS-CLDAP
https://fedorahosted.org/sssd/wiki/DesignDocs/ActiveDirectoryDNSSites#Sen...
So things like samba, IPA, realmd, adcli, and so on ... use and support
cldap for talking with AD. I know Samba has reimplemented cldap but the
others use libldap for this.
Cheers,
Stef
9 years, 11 months
Re: (ITS#7694) cldap fails with IPv6 due to wrong size sockaddr
by hyc@symas.com
Stef Walter wrote:
> On 10.10.2013 12:59, Howard Chu wrote:
>> stefw(a)redhat.com wrote:
>>> Full_Name: Stef Walter
>>> Version: 2.4.35
>>> OS: Fedora 19
>>> Submission from: (NULL) (46.5.2.70)
>>>
>>>
>>> Connectionless LDAP (ie: cldap enabled with -DLDAP_CONNECTIONLESS) is
>>> broken for
>>> IPv6 for current versions of openldap. Tested with version 2.4.35
>>>
>>> It's not clear if this ever worked properly.
>>
>> No, clearly not, the code was written and deprecated before IPv6
>> existed. Nobody should be using this code today.
>
> Interesting. FWIW, the code is packaged by RHEL and Fedora, and is in
> use by several projects.
Can you list any of these, offhand? The original spec, RFC1798, is long
obsoleted. There is no such thing as CLDAP in LDAPv3. Support in OpenLDAP was
first removed back in 2000. (commit 25a9f7427ddc1b584a721ceb0e12690a96d3639e )
Any apps using this must be quite ancient code and in serious need of a rewrite.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9 years, 11 months