Bill MacAllister wrote:

Howard,

We just upgraded to 2.3.42 and are seeing the problem again in our production environment. I have attached a backtrace and db_stat output. This output if from the busy server. I will disconnect the network next and see what happens.

The db_stat output seems to indicate that entries that are being fetched for read access are not getting unlocked. This behavior is new since 2.3.4x for you? What's the last release version where this does not occur?

Bill

--On Tuesday, May 13, 2008 02:45:06 AM -0700 Howard Chuhyc@symas.com wrote:

...

Bill MacAllister wrote:

...

Attached is the output of db4.2_stat -CA of the database.

Thanks for looking at this.

So far it just looks like a very busy server. Can you turn off the network access to it and see if it settles down when the query traffic stops?

It's a bit odd that a single transaction has so many pages of the suPrivilegeGroup index locked.

The backtrace is somewhat suspicious, there are several<value optimized out> items in the trace. In thread 8, frames 5 and 6 the locker value is odd; usually in BDB the locker ID associated with a transaction has bit 31 set, yielding a very large 32 bit number. Also there is no locker with that ID in the db_stat output you provided.

It looks like you'll have to try this again with a non-optimized binary to get a reliable backtrace.

...

Bill

--On Tuesday, May 13, 2008 01:20:49 AM -0700 Howard Chuhyc@symas.com wrote:

...

whm@stanford.edu wrote:

...

Full_Name: Bill MacAllister Version: 2.3.41-1su2 OS: debian etch kernel 2.6.18-4-amd64 URL: http://www.stanford.edu/~whm/ldap-test1-bt.txt Submission from: (NULL) (171.64.19.165)

The slapd process will sometimes consume all of available CPU. We observed this when we upgraded our production servers from 2.3.35-2su2 to 2.3.41-1su2. The problem was bad enough that we downgraded the production servers to 2.3.35-2su2. We have been trying to provoke the problem in our test environment and have not been successful in making it happen on demand. Today, we noticed that one of our test servers went completely CPU bound. I took a backtrace. It is available at the URL below. The interesting thing about the problem is that although top shows a pinned CPU and a high load the server is still responsive and continues to answer LDAP searches. The test server that exhibits the problem is still CPU bound and has been for 2-3 hours now. We will leave this server in this state in case there is other information that we should harvest in resolving the problem.

Please also provide the output from db_stat -CA on the database in question, thanks.

+-------------------------------------------------------- | Bill MacAllisterwhm@stanford.edu | Systems Software Programmer, ITS Unix Systems, Stanford University +---------------------------------------------------------- | "My heart is warm with the friends I make, And better | friends I'll not be knowing; Yet there isn't a train | I wouldn't take, No matter where it's going." | Edna St. Vincent Millay