Looks like the problem was a corrupted database, at least the problem wasn't reproducible on a know working database. BTW, I am using BDB 4.6.21. Thanks for the help. Roy
-----Original Message----- From: Aaron Richton [mailto:richton@nbcs.rutgers.edu] Sent: Tuesday, March 11, 2008 6:39 PM To: Marantz, Roy Cc: openldap-software@openldap.org Subject: Re: Help with server hang
On Tue, 11 Mar 2008, Marantz, Roy wrote:
I'm running OpenLDAP 2.4.8 with Berkeley DB 2.4.6 on Solaris-10
compiled
with Sun's compiler.
Are you sure you haven't typoed the bdb version number here? I mean, the most significant digit should be a "4", if nothing else...
Look for the slapd log message from bdb_open (-d trace) to find your Sleepycat version.
If I start and stop slapd in short succession it will hang after the
2nd
or 3rd time. Following are the syslog message, a pstack (backtrace), and sanitized copy of slapd.conf. Any help in debugging this would be appreciated. Mar 11 16:00:33 master.nyc.deshaw.com slapd[3995]: [ID 100111 local4.debug] slapd starting Mar 11 16:00:39 master.nyc.deshaw.com slapd[3995]: [ID 543694 local4.debug] daemon: shutdown requested and initiated. Mar 11 16:00:39 master.nyc.deshaw.com slapd[3995]: [ID 542995 local4.debug] slapd shutdown: waiting for 0 threads to terminate
Your point is that you're hanging on a shutdown that you initiated, right? Or is it that slapd refuses to fully start and/or is exiting on its own volition shortly after startup or.....?
13476: /usr/local/openldap/libexec/slapd -u ldap -g ldap ----------------- lwp# 1 / thread# 1 -------------------- fe001117 lwp_wait (2, 80476f8) fdffd326 _thrp_join (2, 0, 0, 1) + 5a fdffd4a5 pthread_join (2, 0, 80871e0, 0) + 2b 08088b02 slapd_daemon () + 7a ----------------- lwp# 2 / thread# 2 -------------------- fe00040b lwp_park (0, 0, 0) fdffac7a cond_wait_queue (83114ac, 8311494, 0, 0) + 68 fdffb146 _cond_wait (83114ac, 8311494) + 66 fdffb188 cond_wait (83114ac, 8311494) + 21 fdffb1c1 pthread_cond_wait (83114ac, 8311494, a7, 821603c) + 1b 08196c6c ldap_pvt_thread_pool_destroy (fe000542, 0, 83114cc, fdee8ec0, fdc70000, 0) + e4 083114c8 ???????? ()
Well, easy enough, figure out lock it's waiting for ;)
I think the "shutdown you asked for" interpretation is right, so under that theory:
Seeing as you're at shutdown, my guess would be it's trying to close the bdb database in
directory /var/openldap/nyc.example.com/data
so go there, run db_stat -CA, and see what locks are still held. Assuming that you've initiated a shutdown, there really shouldn't be anything left...although there might be more going on. For example, syncrepl might still be going even if you haven't done anything "by hand." (Although your stack trace and syslogs belie that...)
But hey, why guess...turn up debugging, do you see the database being closed right?