On 26/11/2014 18:23, Ulrich Windl wrote:
What you could try is (assuming you have debug symbols in your binary) attaching gdb to the running slapd to see where it is spending CPU time. I can't remember the commands precisely, but "bt (backtrace)" and "info threads" should point you in the right direction. Note that slapd will likely stop responding while gdb is attached, so you could also try to force a core dump for offline examination.
I'm not sure I follow here. How would this work? I attach gdb to the running slapd I get, but if it stops how does that help me? I've only had a little bit of experience with gdb...
How would I get a core dump, as well? That sounds like it might be more useful.
Maybe the gurus get a clue what may be wrong then...
Regards, Ulrich
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
01:31 in Nachricht 54751F73.3080902@herveybayaustralia.com.au:
I'm trying to get openldap to play nice with mdb given that it is the "recommended" database backend for it now- although the conf wasn't an issue excepting I'm playing with the new cn=config setup we're expected to use now as well (even though it is mainly broken).
My issue is that it seems to not respond like the older bdb/hdb databases. And when I say respond, I mean it hangs the ldapadd and makes slapd go into conniptions. I see slapd go to 100% WCPU and not come down as well as going into a uwait state. I've left it going for 10 minutes or more with no change, and I'm only adding 1 small entry of less the 10 lines. Strangely, I can still view other entries in the specific db as well access the rest of the server, which I won't complain about (aren't threads a wonderful invention? ).
So coming to the experts - got a fix at all? Or should I just go back to ye olde db backends? At this point I have a db I can't add anything to.
And before anyone asks, there is practically nothing in the logs that I can see; and I set the logging to everything (-1). I see recognition of the user in the acl and then nothing. The only possible curious entry is some blank lines and a number (that changes each time), so nothing informative.
I set it up using the cn=config (and I'm still not entirely convinced that I will keep cn=config, but apparently it could be gone next version according to the grapevine, so the consensus is to suck it up and get used to it or your panties will get in a bunch and around your ankles when the upgrade comes along), and I've got only olcDBMaxSize. olcSizeLimit (not sure exactly which of these 2 can go just yet), olcDBMode, olcDBDirectory, and olcDatabase and the obviously root attrs. My max size I've set larger than 50M (so 7 digits) which is larger than what I have in another db so far, and I figure I can add more if needed
- currently it is sitting at 64k.
I'm using FreeBSD 9.1, ports Openldap version is 2.4.40_1 with bdb/hdb and mdb set in config. But I notice lmdb is not installed as a dependency - is this right?
I've been on this for near a week now with no further advancement so any help would be very welcome at this point. No googling seems to find anything remotely similar either.
TIA
Am Wed, 26 Nov 2014 19:41:39 +1000 schrieb Da Rock openldap-technical@herveybayaustralia.com.au:
On 26/11/2014 18:23, Ulrich Windl wrote:
What you could try is (assuming you have debug symbols in your binary) attaching gdb to the running slapd to see where it is spending CPU time. I can't remember the commands precisely, but "bt (backtrace)" and "info threads" should point you in the right direction. Note that slapd will likely stop responding while gdb is attached, so you could also try to force a core dump for offline examination.
I'm not sure I follow here. How would this work? I attach gdb to the running slapd I get, but if it stops how does that help me? I've only had a little bit of experience with gdb...
How would I get a core dump, as well? That sounds like it might be more useful.
[...] ulimit -c unlimited
-Dieter
On 26/11/2014 11:41 πμ, Da Rock wrote:
How would I get a core dump, as well? That sounds like it might be more useful.
See for example: http://www.openldap.org/lists/openldap-technical/201111/msg00243.html
Nick
If gdb is attached to the process ("attach <PID>"), you can examine the CPU registers and the memory. Specifically the call stack. If a specific routine uses all the CPU, it's very likely that gdb will find that active routine and where it was called from. Together with local variables (info locals), a developer could find out what's wrong. Maybe you'll have to combine "-g" with "-O0" for more reliable results.
You'll get a coredump by something like kill -BUS <pid> (check ulimit -c also before starting the process), and you can detach gdb from the process after inspection.
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
10:41 in Nachricht 5475A053.7060209@herveybayaustralia.com.au:
On 26/11/2014 18:23, Ulrich Windl wrote:
What you could try is (assuming you have debug symbols in your binary)
attaching gdb to the running slapd to see where it is spending CPU time. I can't remember the commands precisely, but "bt (backtrace)" and "info threads" should point you in the right direction.
Note that slapd will likely stop responding while gdb is attached, so you
could also try to force a core dump for offline examination. I'm not sure I follow here. How would this work? I attach gdb to the running slapd I get, but if it stops how does that help me? I've only had a little bit of experience with gdb...
How would I get a core dump, as well? That sounds like it might be more useful.
Maybe the gurus get a clue what may be wrong then...
Regards, Ulrich
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
01:31 in Nachricht 54751F73.3080902@herveybayaustralia.com.au:
I'm trying to get openldap to play nice with mdb given that it is the "recommended" database backend for it now- although the conf wasn't an issue excepting I'm playing with the new cn=config setup we're expected to use now as well (even though it is mainly broken).
My issue is that it seems to not respond like the older bdb/hdb databases. And when I say respond, I mean it hangs the ldapadd and makes slapd go into conniptions. I see slapd go to 100% WCPU and not come down as well as going into a uwait state. I've left it going for 10 minutes or more with no change, and I'm only adding 1 small entry of less the 10 lines. Strangely, I can still view other entries in the specific db as well access the rest of the server, which I won't complain about (aren't threads a wonderful invention? ).
So coming to the experts - got a fix at all? Or should I just go back to ye olde db backends? At this point I have a db I can't add anything to.
And before anyone asks, there is practically nothing in the logs that I can see; and I set the logging to everything (-1). I see recognition of the user in the acl and then nothing. The only possible curious entry is some blank lines and a number (that changes each time), so nothing informative.
I set it up using the cn=config (and I'm still not entirely convinced that I will keep cn=config, but apparently it could be gone next version according to the grapevine, so the consensus is to suck it up and get used to it or your panties will get in a bunch and around your ankles when the upgrade comes along), and I've got only olcDBMaxSize. olcSizeLimit (not sure exactly which of these 2 can go just yet), olcDBMode, olcDBDirectory, and olcDatabase and the obviously root attrs. My max size I've set larger than 50M (so 7 digits) which is larger than what I have in another db so far, and I figure I can add more if needed
- currently it is sitting at 64k.
I'm using FreeBSD 9.1, ports Openldap version is 2.4.40_1 with bdb/hdb and mdb set in config. But I notice lmdb is not installed as a dependency - is this right?
I've been on this for near a week now with no further advancement so any help would be very welcome at this point. No googling seems to find anything remotely similar either.
TIA
On 27/11/2014 01:39, Ulrich Windl wrote:
If gdb is attached to the process ("attach <PID>"), you can examine the CPU registers and the memory. Specifically the call stack. If a specific routine uses all the CPU, it's very likely that gdb will find that active routine and where it was called from. Together with local variables (info locals), a developer could find out what's wrong. Maybe you'll have to combine "-g" with "-O0" for more reliable results.
You'll get a coredump by something like kill -BUS <pid> (check ulimit -c also before starting the process), and you can detach gdb from the process after inspection.
Apologies for the delay - had some technical difficulties to take care of.
So here's what I have so far:
GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Attaching to program: /usr/local/libexec/slapd, process 1326 Reading symbols from /usr/local/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libldap_r-2.4.so.2 Reading symbols from /usr/local/lib/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/liblber-2.4.so.2 Reading symbols from /usr/local/lib/libltdl.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libltdl.so.7 Reading symbols from /usr/local/lib/libdb-5.3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libdb-5.3.so.0 Reading symbols from /usr/local/lib/libsasl2.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libsasl2.so.3 Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.5 Reading symbols from /usr/lib/libwrap.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libwrap.so.6 Reading symbols from /usr/local/lib/libssl.so.8...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libssl.so.8 Reading symbols from /usr/local/lib/libcrypto.so.8...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libcrypto.so.8 Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done. [New Thread 802ce9800 (LWP 100233/slapd)] [New Thread 802ce9400 (LWP 100232/slapd)] [New Thread 802ce9000 (LWP 100231/slapd)] [New Thread 802ce8c00 (LWP 100230/slapd)] [New Thread 802ce8800 (LWP 100229/slapd)] [New Thread 802ce8400 (LWP 100228/slapd)] [New Thread 802ce8000 (LWP 100227/slapd)] [New Thread 802ce7c00 (LWP 100226/slapd)] [New Thread 8093a3400 (LWP 100210/slapd)] [New Thread 802ce7800 (LWP 100209/slapd)] [New Thread 802ce7400 (LWP 100197/slapd)] [New Thread 802ce7000 (LWP 100196/slapd)] [New Thread 802ce6c00 (LWP 100194/slapd)] [New Thread 802ce6800 (LWP 100090/slapd)] [New Thread 802ce6400 (LWP 100089/slapd)] [New Thread 802c0dc00 (LWP 100088/slapd)] [New Thread 802c07400 (LWP 100081/slapd)] Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/lib/libssl.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl.so.6 Reading symbols from /lib/libcrypto.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.6 Reading symbols from /usr/local/lib/sasl2/libanonymous.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libanonymous.so.3 Reading symbols from /usr/local/lib/sasl2/libcrammd5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libcrammd5.so.3 Reading symbols from /usr/local/lib/sasl2/libdigestmd5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libdigestmd5.so.3 Reading symbols from /usr/local/lib/sasl2/liblogin.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/liblogin.so.3 Reading symbols from /usr/local/lib/sasl2/libscram.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libscram.so.3 Reading symbols from /usr/local/lib/sasl2/libntlm.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libntlm.so.3 Reading symbols from /usr/local/lib/sasl2/libotp.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libotp.so.3 Reading symbols from /usr/lib/libopie.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libopie.so.7 Reading symbols from /lib/libmd.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libmd.so.5 Reading symbols from /usr/local/lib/sasl2/libplain.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libplain.so.3 Reading symbols from /usr/local/lib/sasl2/libsasldb.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libsasldb.so.3 Reading symbols from /usr/local/lib/sasl2/libgssapiv2.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libgssapiv2.so Reading symbols from /usr/lib/libgssapi.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi.so.10 Reading symbols from /usr/lib/libheimntlm.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libheimntlm.so.10 Reading symbols from /usr/local/lib/libkrb5.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libkrb5.so Reading symbols from /usr/lib/libhx509.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libhx509.so.10 Reading symbols from /usr/local/lib/libcom_err.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libcom_err.so Reading symbols from /usr/lib/libasn1.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libasn1.so.10 Reading symbols from /usr/lib/libroken.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libroken.so.10 Reading symbols from /usr/lib/libkrb5.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.10 Reading symbols from /usr/local/lib/libk5crypto.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libk5crypto.so Reading symbols from /usr/local/lib/libkrb5support.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libkrb5support.so Reading symbols from /usr/local/lib/libintl.so.9...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libintl.so.9 Reading symbols from /usr/lib/libcom_err.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcom_err.so.5 Reading symbols from /usr/local/lib/libiconv.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /usr/local/libexec/openldap/back_mdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_mdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_bdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_bdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_hdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_hdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_ldap-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_ldap-2.4.so.2 Reading symbols from /usr/lib/libz.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 [Switching to Thread 802ce9800 (LWP 100233/slapd)] 0x000000080200187c in pthread_kill () from /lib/libthr.so.3 (gdb) bt #0 0x000000080200187c in pthread_kill () from /lib/libthr.so.3 #1 0x0000000801ffbb25 in pthread_getschedparam () from /lib/libthr.so.3 #2 0x0000000802003c8d in pthread_cond_signal () from /lib/libthr.so.3 #3 0x000000080097c868 in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #4 0x0000000801ffa0a4 in pthread_getprio () from /lib/libthr.so.3 #5 0x0000000000000000 in ?? () Error accessing memory address 0x7ffff73ed000: Bad address.
That occurs before and after adding the new data.
I'm assuming now that I have to rebuild with debug, but I'm not sure exactly how I do that with ports yet and that will take more time. I also can't seem to remember how to access threads... or at least determine which one to follow.
At any rate, is any of this looking familiar at all?
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
10:41 in Nachricht 5475A053.7060209@herveybayaustralia.com.au:
On 26/11/2014 18:23, Ulrich Windl wrote:
What you could try is (assuming you have debug symbols in your binary)
attaching gdb to the running slapd to see where it is spending CPU time. I can't remember the commands precisely, but "bt (backtrace)" and "info threads" should point you in the right direction.
Note that slapd will likely stop responding while gdb is attached, so you
could also try to force a core dump for offline examination. I'm not sure I follow here. How would this work? I attach gdb to the running slapd I get, but if it stops how does that help me? I've only had a little bit of experience with gdb...
How would I get a core dump, as well? That sounds like it might be more useful.
Maybe the gurus get a clue what may be wrong then...
Regards, Ulrich
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
01:31 in Nachricht 54751F73.3080902@herveybayaustralia.com.au:
I'm trying to get openldap to play nice with mdb given that it is the "recommended" database backend for it now- although the conf wasn't an issue excepting I'm playing with the new cn=config setup we're expected to use now as well (even though it is mainly broken).
My issue is that it seems to not respond like the older bdb/hdb databases. And when I say respond, I mean it hangs the ldapadd and makes slapd go into conniptions. I see slapd go to 100% WCPU and not come down as well as going into a uwait state. I've left it going for 10 minutes or more with no change, and I'm only adding 1 small entry of less the 10 lines. Strangely, I can still view other entries in the specific db as well access the rest of the server, which I won't complain about (aren't threads a wonderful invention? ).
So coming to the experts - got a fix at all? Or should I just go back to ye olde db backends? At this point I have a db I can't add anything to.
And before anyone asks, there is practically nothing in the logs that I can see; and I set the logging to everything (-1). I see recognition of the user in the acl and then nothing. The only possible curious entry is some blank lines and a number (that changes each time), so nothing informative.
I set it up using the cn=config (and I'm still not entirely convinced that I will keep cn=config, but apparently it could be gone next version according to the grapevine, so the consensus is to suck it up and get used to it or your panties will get in a bunch and around your ankles when the upgrade comes along), and I've got only olcDBMaxSize. olcSizeLimit (not sure exactly which of these 2 can go just yet), olcDBMode, olcDBDirectory, and olcDatabase and the obviously root attrs. My max size I've set larger than 50M (so 7 digits) which is larger than what I have in another db so far, and I figure I can add more if needed
- currently it is sitting at 64k.
I'm using FreeBSD 9.1, ports Openldap version is 2.4.40_1 with bdb/hdb and mdb set in config. But I notice lmdb is not installed as a dependency - is this right?
I've been on this for near a week now with no further advancement so any help would be very welcome at this point. No googling seems to find anything remotely similar either.
TIA
Hi!
"Error accessing memory address 0x7ffff73ed000: Bad address." looks like either corrupted stack, or compiler optimization to me. Maybe you could try to compile with -O0, just for debugging purposes. Also as there a many threads active, you could try for each thread: "thread n" where n is the thread ID of gdb "bt"
Regards, Ulrich
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 29.11.2014 um
13:49 in Nachricht 5479C0C5.4040506@herveybayaustralia.com.au:
On 27/11/2014 01:39, Ulrich Windl wrote:
If gdb is attached to the process ("attach <PID>"), you can examine the CPU
registers and the memory. Specifically the call stack. If a specific routine uses all the CPU, it's very likely that gdb will find that active routine and where it was called from. Together with local variables (info locals), a developer could find out what's wrong. Maybe you'll have to combine "-g" with "-O0" for more reliable results.
You'll get a coredump by something like kill -BUS <pid> (check ulimit -c also
before starting the process), and you can detach gdb from the process after inspection. Apologies for the delay - had some technical difficulties to take care of.
So here's what I have so far:
GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Attaching to program: /usr/local/libexec/slapd, process 1326 Reading symbols from /usr/local/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libldap_r-2.4.so.2 Reading symbols from /usr/local/lib/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/liblber-2.4.so.2 Reading symbols from /usr/local/lib/libltdl.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libltdl.so.7 Reading symbols from /usr/local/lib/libdb-5.3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libdb-5.3.so.0 Reading symbols from /usr/local/lib/libsasl2.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libsasl2.so.3 Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.5 Reading symbols from /usr/lib/libwrap.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libwrap.so.6 Reading symbols from /usr/local/lib/libssl.so.8...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libssl.so.8 Reading symbols from /usr/local/lib/libcrypto.so.8...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libcrypto.so.8 Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done. [New Thread 802ce9800 (LWP 100233/slapd)] [New Thread 802ce9400 (LWP 100232/slapd)] [New Thread 802ce9000 (LWP 100231/slapd)] [New Thread 802ce8c00 (LWP 100230/slapd)] [New Thread 802ce8800 (LWP 100229/slapd)] [New Thread 802ce8400 (LWP 100228/slapd)] [New Thread 802ce8000 (LWP 100227/slapd)] [New Thread 802ce7c00 (LWP 100226/slapd)] [New Thread 8093a3400 (LWP 100210/slapd)] [New Thread 802ce7800 (LWP 100209/slapd)] [New Thread 802ce7400 (LWP 100197/slapd)] [New Thread 802ce7000 (LWP 100196/slapd)] [New Thread 802ce6c00 (LWP 100194/slapd)] [New Thread 802ce6800 (LWP 100090/slapd)] [New Thread 802ce6400 (LWP 100089/slapd)] [New Thread 802c0dc00 (LWP 100088/slapd)] [New Thread 802c07400 (LWP 100081/slapd)] Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/lib/libssl.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl.so.6 Reading symbols from /lib/libcrypto.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.6 Reading symbols from /usr/local/lib/sasl2/libanonymous.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libanonymous.so.3 Reading symbols from /usr/local/lib/sasl2/libcrammd5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libcrammd5.so.3 Reading symbols from /usr/local/lib/sasl2/libdigestmd5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libdigestmd5.so.3 Reading symbols from /usr/local/lib/sasl2/liblogin.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/liblogin.so.3 Reading symbols from /usr/local/lib/sasl2/libscram.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libscram.so.3 Reading symbols from /usr/local/lib/sasl2/libntlm.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libntlm.so.3 Reading symbols from /usr/local/lib/sasl2/libotp.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libotp.so.3 Reading symbols from /usr/lib/libopie.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libopie.so.7 Reading symbols from /lib/libmd.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libmd.so.5 Reading symbols from /usr/local/lib/sasl2/libplain.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libplain.so.3 Reading symbols from /usr/local/lib/sasl2/libsasldb.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libsasldb.so.3 Reading symbols from /usr/local/lib/sasl2/libgssapiv2.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libgssapiv2.so Reading symbols from /usr/lib/libgssapi.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi.so.10 Reading symbols from /usr/lib/libheimntlm.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libheimntlm.so.10 Reading symbols from /usr/local/lib/libkrb5.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libkrb5.so Reading symbols from /usr/lib/libhx509.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libhx509.so.10 Reading symbols from /usr/local/lib/libcom_err.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libcom_err.so Reading symbols from /usr/lib/libasn1.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libasn1.so.10 Reading symbols from /usr/lib/libroken.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libroken.so.10 Reading symbols from /usr/lib/libkrb5.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.10 Reading symbols from /usr/local/lib/libk5crypto.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libk5crypto.so Reading symbols from /usr/local/lib/libkrb5support.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libkrb5support.so Reading symbols from /usr/local/lib/libintl.so.9...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libintl.so.9 Reading symbols from /usr/lib/libcom_err.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcom_err.so.5 Reading symbols from /usr/local/lib/libiconv.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /usr/local/libexec/openldap/back_mdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_mdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_bdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_bdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_hdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_hdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_ldap-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_ldap-2.4.so.2 Reading symbols from /usr/lib/libz.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 [Switching to Thread 802ce9800 (LWP 100233/slapd)] 0x000000080200187c in pthread_kill () from /lib/libthr.so.3 (gdb) bt #0 0x000000080200187c in pthread_kill () from /lib/libthr.so.3 #1 0x0000000801ffbb25 in pthread_getschedparam () from /lib/libthr.so.3 #2 0x0000000802003c8d in pthread_cond_signal () from /lib/libthr.so.3 #3 0x000000080097c868 in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #4 0x0000000801ffa0a4 in pthread_getprio () from /lib/libthr.so.3 #5 0x0000000000000000 in ?? () Error accessing memory address 0x7ffff73ed000: Bad address.
That occurs before and after adding the new data.
I'm assuming now that I have to rebuild with debug, but I'm not sure exactly how I do that with ports yet and that will take more time. I also can't seem to remember how to access threads... or at least determine which one to follow.
At any rate, is any of this looking familiar at all?
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
10:41 in Nachricht 5475A053.7060209@herveybayaustralia.com.au:
On 26/11/2014 18:23, Ulrich Windl wrote:
What you could try is (assuming you have debug symbols in your binary)
attaching gdb to the running slapd to see where it is spending CPU time. I can't remember the commands precisely, but "bt (backtrace)" and "info threads" should point you in the right direction.
Note that slapd will likely stop responding while gdb is attached, so you
could also try to force a core dump for offline examination. I'm not sure I follow here. How would this work? I attach gdb to the running slapd I get, but if it stops how does that help me? I've only had a little bit of experience with gdb...
How would I get a core dump, as well? That sounds like it might be more useful.
Maybe the gurus get a clue what may be wrong then...
Regards, Ulrich
> Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
01:31 in Nachricht 54751F73.3080902@herveybayaustralia.com.au:
I'm trying to get openldap to play nice with mdb given that it is the "recommended" database backend for it now- although the conf wasn't an issue excepting I'm playing with the new cn=config setup we're expected to use now as well (even though it is mainly broken).
My issue is that it seems to not respond like the older bdb/hdb databases. And when I say respond, I mean it hangs the ldapadd and makes slapd go into conniptions. I see slapd go to 100% WCPU and not come down as well as going into a uwait state. I've left it going for 10 minutes or more with no change, and I'm only adding 1 small entry of less the 10 lines. Strangely, I can still view other entries in the specific db as well access the rest of the server, which I won't complain about (aren't threads a wonderful invention? ).
So coming to the experts - got a fix at all? Or should I just go back to ye olde db backends? At this point I have a db I can't add anything to.
And before anyone asks, there is practically nothing in the logs that I can see; and I set the logging to everything (-1). I see recognition of the user in the acl and then nothing. The only possible curious entry is some blank lines and a number (that changes each time), so nothing informative.
I set it up using the cn=config (and I'm still not entirely convinced that I will keep cn=config, but apparently it could be gone next version according to the grapevine, so the consensus is to suck it up and get used to it or your panties will get in a bunch and around your ankles when the upgrade comes along), and I've got only olcDBMaxSize. olcSizeLimit (not sure exactly which of these 2 can go just yet), olcDBMode, olcDBDirectory, and olcDatabase and the obviously root attrs. My max size I've set larger than 50M (so 7 digits) which is larger than what I have in another db so far, and I figure I can add more if needed
- currently it is sitting at 64k.
I'm using FreeBSD 9.1, ports Openldap version is 2.4.40_1 with bdb/hdb and mdb set in config. But I notice lmdb is not installed as a dependency - is this right?
I've been on this for near a week now with no further advancement so any help would be very welcome at this point. No googling seems to find anything remotely similar either.
TIA
On 01/12/2014 17:28, Ulrich Windl wrote:
Hi!
"Error accessing memory address 0x7ffff73ed000: Bad address." looks like either corrupted stack, or compiler optimization to me. Maybe you could try to compile with -O0, just for debugging purposes. Also as there a many threads active, you could try for each thread: "thread n" where n is the thread ID of gdb "bt"
Sorry, but it has been exceptionally busy.
I've tried examination of threads and before and after the add, but it is difficult to see any particular differences. In direct comparison nothing really seems to change at all. And it seems that the whole system freezes while in debug anyway, so I can't seem to make head or tail of anything.
The list of threads shows 18 threads of which one is in select, and the rest are in pthread_kill. Backtraces show only the same as already mentioned, and even after the add takes place; but as I said gdb is blocking any actions (even a search) so I don't know exactly how effective it really is either.
I'm seriously considering whether I might not need to just start again with this particular database. Unless someone can come up with any other examinations or specific instructions for me to try?
Regards, Ulrich
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 29.11.2014 um
13:49 in Nachricht 5479C0C5.4040506@herveybayaustralia.com.au:
On 27/11/2014 01:39, Ulrich Windl wrote:
If gdb is attached to the process ("attach <PID>"), you can examine the CPU
registers and the memory. Specifically the call stack. If a specific routine uses all the CPU, it's very likely that gdb will find that active routine and where it was called from. Together with local variables (info locals), a developer could find out what's wrong. Maybe you'll have to combine "-g" with "-O0" for more reliable results.
You'll get a coredump by something like kill -BUS <pid> (check ulimit -c also
before starting the process), and you can detach gdb from the process after inspection. Apologies for the delay - had some technical difficulties to take care of.
So here's what I have so far:
GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Attaching to program: /usr/local/libexec/slapd, process 1326 Reading symbols from /usr/local/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libldap_r-2.4.so.2 Reading symbols from /usr/local/lib/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/liblber-2.4.so.2 Reading symbols from /usr/local/lib/libltdl.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libltdl.so.7 Reading symbols from /usr/local/lib/libdb-5.3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libdb-5.3.so.0 Reading symbols from /usr/local/lib/libsasl2.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libsasl2.so.3 Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.5 Reading symbols from /usr/lib/libwrap.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libwrap.so.6 Reading symbols from /usr/local/lib/libssl.so.8...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libssl.so.8 Reading symbols from /usr/local/lib/libcrypto.so.8...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libcrypto.so.8 Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done. [New Thread 802ce9800 (LWP 100233/slapd)] [New Thread 802ce9400 (LWP 100232/slapd)] [New Thread 802ce9000 (LWP 100231/slapd)] [New Thread 802ce8c00 (LWP 100230/slapd)] [New Thread 802ce8800 (LWP 100229/slapd)] [New Thread 802ce8400 (LWP 100228/slapd)] [New Thread 802ce8000 (LWP 100227/slapd)] [New Thread 802ce7c00 (LWP 100226/slapd)] [New Thread 8093a3400 (LWP 100210/slapd)] [New Thread 802ce7800 (LWP 100209/slapd)] [New Thread 802ce7400 (LWP 100197/slapd)] [New Thread 802ce7000 (LWP 100196/slapd)] [New Thread 802ce6c00 (LWP 100194/slapd)] [New Thread 802ce6800 (LWP 100090/slapd)] [New Thread 802ce6400 (LWP 100089/slapd)] [New Thread 802c0dc00 (LWP 100088/slapd)] [New Thread 802c07400 (LWP 100081/slapd)] Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/lib/libssl.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl.so.6 Reading symbols from /lib/libcrypto.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.6 Reading symbols from /usr/local/lib/sasl2/libanonymous.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libanonymous.so.3 Reading symbols from /usr/local/lib/sasl2/libcrammd5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libcrammd5.so.3 Reading symbols from /usr/local/lib/sasl2/libdigestmd5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libdigestmd5.so.3 Reading symbols from /usr/local/lib/sasl2/liblogin.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/liblogin.so.3 Reading symbols from /usr/local/lib/sasl2/libscram.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libscram.so.3 Reading symbols from /usr/local/lib/sasl2/libntlm.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libntlm.so.3 Reading symbols from /usr/local/lib/sasl2/libotp.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libotp.so.3 Reading symbols from /usr/lib/libopie.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libopie.so.7 Reading symbols from /lib/libmd.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libmd.so.5 Reading symbols from /usr/local/lib/sasl2/libplain.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libplain.so.3 Reading symbols from /usr/local/lib/sasl2/libsasldb.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libsasldb.so.3 Reading symbols from /usr/local/lib/sasl2/libgssapiv2.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/sasl2/libgssapiv2.so Reading symbols from /usr/lib/libgssapi.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi.so.10 Reading symbols from /usr/lib/libheimntlm.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libheimntlm.so.10 Reading symbols from /usr/local/lib/libkrb5.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libkrb5.so Reading symbols from /usr/lib/libhx509.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libhx509.so.10 Reading symbols from /usr/local/lib/libcom_err.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libcom_err.so Reading symbols from /usr/lib/libasn1.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libasn1.so.10 Reading symbols from /usr/lib/libroken.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libroken.so.10 Reading symbols from /usr/lib/libkrb5.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.10 Reading symbols from /usr/local/lib/libk5crypto.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libk5crypto.so Reading symbols from /usr/local/lib/libkrb5support.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libkrb5support.so Reading symbols from /usr/local/lib/libintl.so.9...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libintl.so.9 Reading symbols from /usr/lib/libcom_err.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcom_err.so.5 Reading symbols from /usr/local/lib/libiconv.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /usr/local/libexec/openldap/back_mdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_mdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_bdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_bdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_hdb-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_hdb-2.4.so.2 Reading symbols from /usr/local/libexec/openldap/back_ldap-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/local/libexec/openldap/back_ldap-2.4.so.2 Reading symbols from /usr/lib/libz.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 [Switching to Thread 802ce9800 (LWP 100233/slapd)] 0x000000080200187c in pthread_kill () from /lib/libthr.so.3 (gdb) bt #0 0x000000080200187c in pthread_kill () from /lib/libthr.so.3 #1 0x0000000801ffbb25 in pthread_getschedparam () from /lib/libthr.so.3 #2 0x0000000802003c8d in pthread_cond_signal () from /lib/libthr.so.3 #3 0x000000080097c868 in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #4 0x0000000801ffa0a4 in pthread_getprio () from /lib/libthr.so.3 #5 0x0000000000000000 in ?? () Error accessing memory address 0x7ffff73ed000: Bad address.
That occurs before and after adding the new data.
I'm assuming now that I have to rebuild with debug, but I'm not sure exactly how I do that with ports yet and that will take more time. I also can't seem to remember how to access threads... or at least determine which one to follow.
At any rate, is any of this looking familiar at all?
Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
10:41 in Nachricht 5475A053.7060209@herveybayaustralia.com.au:
On 26/11/2014 18:23, Ulrich Windl wrote:
What you could try is (assuming you have debug symbols in your binary)
attaching gdb to the running slapd to see where it is spending CPU time. I can't remember the commands precisely, but "bt (backtrace)" and "info threads" should point you in the right direction.
Note that slapd will likely stop responding while gdb is attached, so you
could also try to force a core dump for offline examination. I'm not sure I follow here. How would this work? I attach gdb to the running slapd I get, but if it stops how does that help me? I've only had a little bit of experience with gdb...
How would I get a core dump, as well? That sounds like it might be more useful.
Maybe the gurus get a clue what may be wrong then...
Regards, Ulrich
>> Da Rock openldap-technical@herveybayaustralia.com.au schrieb am 26.11.2014 um
01:31 in Nachricht 54751F73.3080902@herveybayaustralia.com.au:
I'm trying to get openldap to play nice with mdb given that it is the "recommended" database backend for it now- although the conf wasn't an issue excepting I'm playing with the new cn=config setup we're expected to use now as well (even though it is mainly broken).
My issue is that it seems to not respond like the older bdb/hdb databases. And when I say respond, I mean it hangs the ldapadd and makes slapd go into conniptions. I see slapd go to 100% WCPU and not come down as well as going into a uwait state. I've left it going for 10 minutes or more with no change, and I'm only adding 1 small entry of less the 10 lines. Strangely, I can still view other entries in the specific db as well access the rest of the server, which I won't complain about (aren't threads a wonderful invention? ).
So coming to the experts - got a fix at all? Or should I just go back to ye olde db backends? At this point I have a db I can't add anything to.
And before anyone asks, there is practically nothing in the logs that I can see; and I set the logging to everything (-1). I see recognition of the user in the acl and then nothing. The only possible curious entry is some blank lines and a number (that changes each time), so nothing informative.
I set it up using the cn=config (and I'm still not entirely convinced that I will keep cn=config, but apparently it could be gone next version according to the grapevine, so the consensus is to suck it up and get used to it or your panties will get in a bunch and around your ankles when the upgrade comes along), and I've got only olcDBMaxSize. olcSizeLimit (not sure exactly which of these 2 can go just yet), olcDBMode, olcDBDirectory, and olcDatabase and the obviously root attrs. My max size I've set larger than 50M (so 7 digits) which is larger than what I have in another db so far, and I figure I can add more if needed
- currently it is sitting at 64k.
I'm using FreeBSD 9.1, ports Openldap version is 2.4.40_1 with bdb/hdb and mdb set in config. But I notice lmdb is not installed as a dependency - is this right?
I've been on this for near a week now with no further advancement so any help would be very welcome at this point. No googling seems to find anything remotely similar either.
TIA
openldap-technical@openldap.org
-
Da Rock -
Dieter Klünter -
Nick Milas -
Ulrich Windl -
Леонид Юрьев