Re: (ITS#7928) Segfault when cancelling syncprov search
by hyc＠symas.com
ben(a)morrow.me.uk wrote:
> Full_Name: Ben Morrow
> Version: 2.4.39 and git head
> OS: FreeBSD
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (86.140.232.115)
>
>
> I believe I have found a crashing bug in the syncprov overlay. If I
> start a refreshAndPersist search, wait for the persist phase to start
> and at least one change to be returned, and then cancel the search,
> slapd dies with SIGSEGV. (I don't know how many of these conditions are
> necessary. I suspect that cancelling any syncprov search will cause the
> same crash, but I haven't investigated that.)
>
> I also believe the following change to
> servers/slapd/overlays/syncprov.c:syncprov_ab_cleanup fixes the prlelem:
>
> slap_callback *sc = op->o_callback;
> op->o_callback = sc->sc_next;
> - syncprov_drop_psearch( op->o_callback->sc_private, 0 );
> + syncprov_drop_psearch( sc->sc_private, 0 );
>
> The crash I see actually occurs in libthr (FreeBSD's threading library),
> because drop_psearch passes its first argument to free_syncop which
> starts by trying to lock a mutex; but in fact the object being passed is
> not the right type at all, and where there is supposed to be a mutex
> there is instead a function pointer.
Thanks, looks like you're right. Fixed in git master.
>
> Ben
>
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9 years, 1 month
(ITS#7928) Segfault when cancelling syncprov search
by ben＠morrow.me.uk
Full_Name: Ben Morrow
Version: 2.4.39 and git head
OS: FreeBSD
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (86.140.232.115)
I believe I have found a crashing bug in the syncprov overlay. If I
start a refreshAndPersist search, wait for the persist phase to start
and at least one change to be returned, and then cancel the search,
slapd dies with SIGSEGV. (I don't know how many of these conditions are
necessary. I suspect that cancelling any syncprov search will cause the
same crash, but I haven't investigated that.)
I also believe the following change to
servers/slapd/overlays/syncprov.c:syncprov_ab_cleanup fixes the prlelem:
slap_callback *sc = op->o_callback;
op->o_callback = sc->sc_next;
- syncprov_drop_psearch( op->o_callback->sc_private, 0 );
+ syncprov_drop_psearch( sc->sc_private, 0 );
The crash I see actually occurs in libthr (FreeBSD's threading library),
because drop_psearch passes its first argument to free_syncop which
starts by trying to lock a mutex; but in fact the object being passed is
not the right type at all, and where there is supposed to be a mutex
there is instead a function pointer.
Ben
9 years, 1 month
Re: (ITS#7926) modifying olcListenerThreads crashes slapd
by quanah＠zimbra.com
--On Monday, August 25, 2014 9:53 PM +0000 quanah(a)openldap.org wrote:
> Full_Name: Quanah Gibson-Mount
> Version: 2.4.39
> OS: Linux 3.13
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (75.111.58.125)
>
>
> After modifying olcListenerThreads for a running slapd, the process died
> and I had to restart ldap. Going to see if cores were enabled.
>
Trivial to reproduce:
(gdb) cont
Continuing.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f963ca41700 (LWP 12048)]
0x0000000000432279 in slapd_clr_read (s=13, wake=0) at daemon.c:981
981 daemon.c: No such file or directory.
in daemon.c
Thread 6 (Thread 0x7f963ca41700 (LWP 12048)):
#0 0x0000000000432279 in slapd_clr_read (s=13, wake=0) at daemon.c:981
rc = 1
id = 1
#1 0x000000000043ab9c in connection_read_activate (s=13) at
connection.c:1285
rc = 62
#2 0x0000000000436ddb in slapd_daemon_task (ptr=0x1ba1d28) at daemon.c:2769
rc = 1
fd = 13
w = 0
r = 1
ns = 1
at = 0
nfds = 18
revents = 0x1c40000
tvp = 0x0
cat = {tv_sec = 0, tv_usec = 0}
i = 0
nwriters = 0
now = 1409005820
tv = {tv_sec = 0, tv_usec = 0}
tdelta = 1
rtask = 0x0
l = 2
last_idle_check = 1408061664
ebadf = 0
tid = 0
#3 0x0000003e130079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x0000003e12ce8b5d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 5 (Thread 0x7f963c240700 (LWP 12052)):
#0 0x0000003e1300b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fa28ef29787 in ldap_pvt_thread_cond_wait (cond=0x1bdc038,
mutex=0x1bdc010) at thr_posix.c:277
No locals.
#2 0x00007fa28ef27fa2 in ldap_int_thread_pool_wrapper (xpool=0x1bdc000) at
tpool.c:927
pq = 0x1bdc000
pool = 0x1da0180
task = 0x0
work_list = 0x1bdc070
ctx = {ltu_pq = 0x1bdc000, ltu_id = 140283230816000, ltu_key =
{{ltk_key = 0x43a106, ltk_data = 0x1da4e00, ltk_free = 0x439f4a
<conn_counter_destroy>}, {ltk_key = 0x4ad967,
ltk_data = 0x3d9c2c0, ltk_free = 0x4ad78c
<slap_sl_mem_destroy>}, {ltk_key = 0x455375, ltk_data = 0x20a03c0, ltk_free
= 0x4552c8 <slap_op_q_destroy>}, {
ltk_key = 0x7fa28b554533, ltk_data = 0x419e000, ltk_free =
0x7fa28b554510 <search_stack_free>}, {ltk_key = 0x7fa28b5515ab, ltk_data =
0x3e9e000,
ltk_free = 0x7fa28b551563 <scope_chunk_free>}, {ltk_key =
0x1bdd6c0, ltk_data = 0x1fe2200, ltk_free = 0x7fa28b55f195
<mdb_reader_free>}, {ltk_key = 0x0,
ltk_data = 0x9106400, ltk_free = 0}, {ltk_key = 0x0, ltk_data
= 0x0, ltk_free = 0} <repeats 25 times>}}
kctx = 0x0
i = 32
keyslot = 380
hash = 3641033084
pool_lock = 0
freeme = 0
__PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
#3 0x0000003e130079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x0000003e12ce8b5d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 4 (Thread 0x7f963ba3f700 (LWP 12183)):
#0 0x0000003e1300b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fa28ef29787 in ldap_pvt_thread_cond_wait (cond=0x1bdc038,
mutex=0x1bdc010) at thr_posix.c:277
No locals.
#2 0x00007fa28ef27fa2 in ldap_int_thread_pool_wrapper (xpool=0x1bdc000) at
tpool.c:927
pq = 0x1bdc000
pool = 0x1da0180
task = 0x0
work_list = 0x1bdc070
ctx = {ltu_pq = 0x1bdc000, ltu_id = 140283222423296, ltu_key =
{{ltk_key = 0x43a106, ltk_data = 0x1da4700, ltk_free = 0x439f4a
<conn_counter_destroy>}, {ltk_key = 0x4ad967,
ltk_data = 0x3d9c9c0, ltk_free = 0x4ad78c
<slap_sl_mem_destroy>}, {ltk_key = 0x1bdd6c0, ltk_data = 0x52dc000,
ltk_free = 0x7fa28b55f195 <mdb_reader_free>}, {
ltk_key = 0x7fa28b554533, ltk_data = 0x55e6000, ltk_free =
0x7fa28b554510 <search_stack_free>}, {ltk_key = 0x7fa28b5515ab, ltk_data =
0x52e6000,
ltk_free = 0x7fa28b551563 <scope_chunk_free>}, {ltk_key =
0x455375, ltk_data = 0x237b840, ltk_free = 0x4552c8 <slap_op_q_destroy>},
{ltk_key = 0x0, ltk_data = 0x66ed600,
ltk_free = 0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0}
<repeats 25 times>}}
kctx = 0x0
i = 32
keyslot = 138
hash = 3676775562
pool_lock = 0
freeme = 0
__PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
#3 0x0000003e130079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x0000003e12ce8b5d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x7f963b23e700 (LWP 24606)):
#0 0x0000003e1300b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fa28ef29787 in ldap_pvt_thread_cond_wait (cond=0x1bdc038,
mutex=0x1bdc010) at thr_posix.c:277
No locals.
#2 0x00007fa28ef27fa2 in ldap_int_thread_pool_wrapper (xpool=0x1bdc000) at
tpool.c:927
pq = 0x1bdc000
pool = 0x1da0180
task = 0x0
work_list = 0x1bdc070
ctx = {ltu_pq = 0x1bdc000, ltu_id = 140283214030592, ltu_key =
{{ltk_key = 0x43a106, ltk_data = 0x2155100, ltk_free = 0x439f4a
<conn_counter_destroy>}, {ltk_key = 0x4ad967,
ltk_data = 0x66d6b40, ltk_free = 0x4ad78c
<slap_sl_mem_destroy>}, {ltk_key = 0x455375, ltk_data = 0x2379a40, ltk_free
= 0x4552c8 <slap_op_q_destroy>}, {ltk_key = 0x1bdd6c0,
ltk_data = 0x6a60000, ltk_free = 0x7fa28b55f195
<mdb_reader_free>}, {ltk_key = 0x7fa28b554533, ltk_data = 0x6d6e000,
ltk_free = 0x7fa28b554510 <search_stack_free>}, {
ltk_key = 0x7fa28b5515ab, ltk_data = 0x6a6e000, ltk_free =
0x7fa28b551563 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x9108800,
ltk_free = 0}, {ltk_key = 0x0,
ltk_data = 0x0, ltk_free = 0} <repeats 25 times>}}
kctx = 0x0
i = 32
keyslot = 250
hash = 907520250
pool_lock = 0
freeme = 0
__PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
#3 0x0000003e130079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x0000003e12ce8b5d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 2 (Thread 0x7f963aa3d700 (LWP 24607)):
#0 0x0000003e1300b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fa28ef29787 in ldap_pvt_thread_cond_wait (cond=0x1bdc038,
mutex=0x1bdc010) at thr_posix.c:277
No locals.
#2 0x00007fa28ef27fa2 in ldap_int_thread_pool_wrapper (xpool=0x1bdc000) at
tpool.c:927
pq = 0x1bdc000
pool = 0x1da0180
task = 0x0
work_list = 0x1bdc070
ctx = {ltu_pq = 0x1bdc000, ltu_id = 140283205637888, ltu_key =
{{ltk_key = 0x43a106, ltk_data = 0x2154f00, ltk_free = 0x439f4a
<conn_counter_destroy>}, {ltk_key = 0x4ad967,
ltk_data = 0x66d6b80, ltk_free = 0x4ad78c
<slap_sl_mem_destroy>}, {ltk_key = 0x455375, ltk_data = 0x66c6000, ltk_free
= 0x4552c8 <slap_op_q_destroy>}, {ltk_key = 0x1bdd6c0,
ltk_data = 0x6a61a00, ltk_free = 0x7fa28b55f195
<mdb_reader_free>}, {ltk_key = 0x7fa28b554533, ltk_data = 0x806e000,
ltk_free = 0x7fa28b554510 <search_stack_free>}, {
ltk_key = 0x7fa28b5515ab, ltk_data = 0x7d6e000, ltk_free =
0x7fa28b551563 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x9108800,
ltk_free = 0}, {ltk_key = 0x0,
ltk_data = 0x0, ltk_free = 0} <repeats 25 times>}}
kctx = 0x0
i = 32
keyslot = 41
hash = 1003389993
pool_lock = 0
freeme = 0
__PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
#3 0x0000003e130079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x0000003e12ce8b5d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 1 (Thread 0x7fa28e26a720 (LWP 12047)):
#0 0x0000003e1300822d in pthread_join () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fa28ef296c8 in ldap_pvt_thread_join (thread=140283239208704,
thread_return=0x0) at thr_posix.c:197
No locals.
#2 0x0000000000437587 in slapd_daemon () at daemon.c:2907
i = 0
rc = 0
#3 0x0000000000414afb in main (argc=9, argv=0x7fff33ea3258) at main.c:1012
i = 9
no_detach = 0
rc = 0
urls = 0x1bb0000 "ldap://zre-ldap003.eng.zimbra.com:389 ldapi:///"
username = 0x1ba0010 "root"
groupname = 0x0
sandbox = 0x0
syslogUser = 128
pid = 0
waitfds = {9, 10}
g_argc = 9
g_argv = 0x7fff33ea3258
configfile = 0x0
configdir = 0x1ba4020 "/opt/zimbra/data/ldap/config"
serverName = 0x7fff33ea4d76 "slapd"
serverMode = 1
scp = 0x0
scp_entry = 0x0
debug_unknowns = 0x0
syslog_unknowns = 0x0
serverNamePrefix = 0x4f19c8 ""
l = 5182640
slapd_pid_file_unlink = 1
slapd_args_file_unlink = 1
firstopt = 0
__PRETTY_FUNCTION__ = "main"
(gdb)
--
Quanah Gibson-Mount
Server Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
9 years, 1 month
Re: (ITS#7925) mdb_from_db Berkeley DB importer
by hyc＠symas.com
graehl(a)gmail.com wrote:
> Full_Name: Jonathan Graehl
> Version: commit 8d346721a60684aeaa7b1e3b2111c972393bfad3
> OS: linux
> URL:
> Submission from: (NULL) (75.85.99.117)
>
>
> mdb_from_db Berkeley DB->LMDB import utility
Interesting, thanks for the submission.
> +Additionally,
> +.B mdb_from_db
> +may write in the
> +.B -T
> +plain text format understood by
> +.BR mdb_load (1)
> +which can only understand a single subdatabase at a time.
Not true. mdb_load can load multiple DBs from a single input file.
Also, I'm not sure if it's your mailer or ours, but this submission text is
riddled with corruptions.
> +
> B2B.SH OPTION0D0D
In general I don't think including this utility into LMDB is a good idea since
it has an obvious binary dependency on BerkeleyDB. In particular, with the
change of BerkeleyDB's license to AGPL it would not be possible for anyone to
distribute any binaries for this package.
Better to just use BDB db_dump piped into mdb_load.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9 years, 1 month
(ITS#7925) mdb_from_db Berkeley DB importer
by graehl＠gmail.com
Full_Name: Jonathan Graehl
Version: commit 8d346721a60684aeaa7b1e3b2111c972393bfad3
OS: linux
URL:
Submission from: (NULL) (75.85.99.117)
mdb_from_db Berkeley DB->LMDB import utility
See also https://github.com/openldap/openldap/pull/1
>From 32f6c10570bf7ede64cdb734775e97ea2afe1011 Mon Sep 17 00:00:00 2001
From: graehl <graehl(a)gmail.com>
Date: Sun, 24 Aug 2014 16:02:41 -0700
Subject: [PATCH] mdb_from_db Berkeley DB->LMDB import
---
libraries/liblmdb/Makefile | 3 +-
libraries/liblmdb/mdb_from_db.1 | 104 ++++++++
libraries/liblmdb/mdb_from_db.c | 548
+++++++++++++++++++++++++++++++++++++B%B++
3 files changed, 654 insertions(+), 1 deletion(-)
create mode 100644 libraries/liblmdb/mdb_from_db.1
create mode 100644 libraries/liblmdb/mdb_from_db.c
diff --git a/libraries/liblmdb/Makefile b/libraries/liblmdb/Makefile
index 25c1095..196ed08 100644
--- a/libraries/liblmdb/Makefile
+++ b/libraries/liblmdb/Makefile
@@ -29,7 +29,7 @@ prefix = /usr/local
IHDRS = lmdb.h
ILIBS = liblmdb.a liblmdb.so
-IPROGS = mdb_stat mdb_copy mdb_dump mdb_load
+IPROGS = mdb_stat mdb_copy mdb_dump mdb_load mdb_from_db
IDOCS = mdb_stat.1 mdb_copy.1 mdb_dump.1 mdb_load.1
PROGS = $(IPROGS) mtest mtest2 mtest3 mtest4 mtest5
all: $(ILIBS) $(PROGS)
@@ -58,6 +58,7 @@ mdb_stat: mdb_stat.o liblmdb.a
mdb_copy: mdb_copy.o liblmdb.a
mdb_dump: mdb_dump.o liblmdb.a
mdb_load: mdb_load.o liblmdb.a
+mdb_from_db: mdb_from_db.o liblmdb.a
mtest: mtest.o liblmdb.a
mtest2: mtest2.o liblmdb.a
mtest3: mtest3.o liblmdb.a
diff --git a/libraries/liblmdb/mdb_from_db.1 b/libraries/liblmdb/mdb_from_db.1
new file mode 100644
index 0000000..dbd6797
--- /dev/null
+++ b/libraries/liblmdb/mdb_from_db.1
@@ -0,0 +1,104 @@
+.TH MDB_FROM_DB 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply. See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_from_db \- LMDB environment translate from Berkeley DB environment tool
+.SH SYNOPSIS
+.B mdb_from_db
+.BR \ berkeley.db
+.BR \ envpath
+[\c
+.BR \-V ]
+[\c
+.BR \-n ]
+[\c
+.BI \-s \ subdb\fR]
+[\c
+.B%5\-b \ bahshsize\fR]
+[\c
+.BI \-h \ berkeley-db-homedir\fR]
+[\c
+.BR \-N ]
+[\c
+.BR \-T ]
+.SH DESCRIPTION
+The
+.B mdb_from_db
+utility reads from a Berkeley DB environment
+.BR berkeley.db
+and from_dbs all its subdatabases, or just the specified
+.BR subdb
+, into the
+LMDB environment
+.BR envpath .
+
+Additionally,
+.B mdb_from_db
+may write in the
+.B -T
+plain text format understood by
+.BR mdb_load (1)
+which can only understand a single subdatabase at a time.
+
B2B.SH OPTION0D0D
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-n
+From_Db an LMDB database which does not use subdirectories.
+.TP
+.BR \-s \ subdb
+From_Db a specific subdatabase. If no database is specified, data is from_dbed
into the main database.
+.TP
+.BR \-N
+Don't overwrite existing records when from_dbing into an already existing
database; just skip them.
+.TP
+.BR \-b \ sz
+Commit LMDB records
+.B sz
+at a time.
+.TP
+.BR \-h \ db_homedir
+Treat input db path as relative to this homedir (see the Berkeley DB docs).
Default is '.'
+.TP
+.BR \-B
+Perform a nonblocking Berkeley DB open.
+.TP
+.BR \-T
+Write the key/data into a single simple text file (stderr messages
+would allow segmenting the output into separate files for each
+subdatabase). The input will be paired lines of text, where the first
+line of the pair is the key item, and the second line of the pair is
+its corresponding data item. If more than one database is read then
+refer to the counts reported on stderr.
+
+A simple escape mechanism, where newline and backslash (\\) characters
+are special, is applied to the text input. Newline characters are
+interpreted as record separators. Backslash characters in the text
+will be interpreted in one of two ways: If the backslash character
+precedes another backslash character, the pair will be interpreted as
+a literal backslash. If the backslash character precedes any other
+character, the two characters following the backslash will be
+interpreted as a hexadecimal specification of a single character; for
+example, \\0a is a newline character in the ASCII character set.
+
+For this reason, any backslash or newline characters that naturally
+occur in the text input must be escaped to avoid misinterpretation by
+.BR mdb_load.
+
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+
+Information about each subdatabase processed, and the total number of
+records is also written to standard error.
+
+.SH "SEE ALSO"
+.BR mdb_load (1)
+.BR mdb_dump (1)
+.BR db_dump (1)
+
+.SH AUTHOR
+Jonathan Graehl <graehl(a)gmail.com>
diff --git a/libraries/liblmdb/mdb_from_db.c b/libraries/liblmdb/mdb_from_db.c
new file mode 100644
index 0000000..ee81db0
--- /dev/null
+++ b/libraries/liblmdb/mdb_from_db.c
@@ -0,0 +1,548 @@
+/* mdb_from_db.c - translate Berkeley DB to memory-mapped database(s) */
+/*
+ * Copyright 2014 Jonathan Graeh2C2C 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <string.h>
+#include <ctype.h>
+#include <unistd.h>
+#include "lmdb.h"
+#include <db.h>
+#include <stdbool.h>
+#include <sys/stat.h>
+
+static int datasize = 64*1024;
+static int batchsize = 100;
+
+static char *subname = NULL;
+
+static char *progB0D
+
+static MDB_val kbuf, dbuf;
+
+#ifdef _WIN32
+#define Z "I"
+#else
+#define Z "z"
+#endif
+
+
+char *usagestr =
+ "path.input.berkeley.db [path.output.mdb|T-txt] [-P dbpasswd] [-V] [-l]
[-n] [-s subdbname] [-N] [-B] [-T] [-v] [-h homedirpath] [-b write-batchsize]
[-f redirect_stdout.txt]\n"
+ " (-T prints to stdout key/val in mdb_load format)\n"
+ " (-l: only list (to stdout) database names, -N: don't overwrite existing
keys; -B nonblocking db open; -n: create single mdb file instead of dir)\n"
+ ;
+
+/**
+ fail() and shutdown(): everything that might need to be cleaned up on
error->exit
+ (conceptually some of these are locals, but having them global lets us call
+ fail() from anywhere)
+*/A%A+static MDB_env *env;
+static MDB_txn *txn;
+static MDB_dbi dbi;
+
+static DB *dbp;
+static DBC *dbcp;
+static DB *parent_dbp;
+static DBC *bdb_subdbcursor;
+static char *subdbname;
+static DB_TXN *dbtxn;
+
+void bdb_close() {
+ if (dbcp)
+ dbcp->cse%2(dbcp);
+ dbcp = 0;
+ if (dbp)
+ dbp->close(dbp, 0);
+ dbp = 0;
+}
+void shutdown() {
+ if (bdb_subdbcursor)
+ bdb_subdbcursor->close(bdb_subdbcursor);
+ bdb_subdbcursor = 0;
+ bdb_close();
+ if (parent_dbp)
+ parent_dbp->close(parent_dbp, 0);
+ parent_dbp = 0;
+ if (txn)
+ mdb_txn_abort(txn);
+ txn = 0;
+ if (dbi)
+ mdb_dbi_close(env, dbi);
+ if (env)
+ mdb_env_close(env);
+ env = 0;
+ if (subdbname)
+ free(subdbname);
+ subdbname =%0;
+}
+
+void fail() {
+ shutdown();
+ exit(EXIT_FAILURE);
+}
+
+static void usage(void)
+{
+ fprintf(stderr, "usage: %s %s", prog, usagestr);
+ fail();
+}
+
+/**
+ BDB env
+*/
+static char *dbhome;
+static DB_ENV *dbenv;
+static u_int32_t dbcache = 1024*1024;
+static char *dbpasswd;
+static bool dbnonblocking;
+void strfill(char *str, char fill) {
+ while(*str)
+ *str++ = fill;
+}
+void bdb_err(char *fn, int rc) {
+ fprintf(stderr, "%s: ", prog);
+ if (dbenv)
+ dbenv->err(dbenv, rc, fn);
+ else
+ fprintf(stderr, "%s\n", db_strerror(rc));
+ fail();
+}
+
+DBT dbkey, dbdata;
+void bdb_init_dbenv() {
+ int rc;
+ if ((rc = db_env_create(&dbenv, 0)) != 0)
+ bdb_err("db_env_create", rc);
+ dbenv->set_errfile(dbenv, stderr);
+ dbv-v->set_errpfx(dbenv, prog);
+ if (dbpasswd != NULL) {
+ rc = dbenv->set_encrypt(dbenv, dbpasswd, DB_ENCRYPT_AES);
+ strfill(dbpasswd, '\0');
+ if (rc)
+ bdb_err("dbenv::set_encrypt", rc);
+ }
+ if (dbnonblocking) {
+ if ((rc = dbenv->set_flags(dbenv, DB_NOLOCKING, 1)))
+ bdb_err("DB_NOLOCKING", rc);
+ if ((rc = dbenv->set_flags(dbenv, DB_NOPANIC, 1)))
+ bdb_err("DB_NOPANIC", rc);
+ }
+ if ((rc = dbenv->set_cachesize(dbenv, 0,bdbcache, 1)))
+ bdb_err("dbenv::set_cachesize", rc);
+ if ((rc = dbenv->open(dbenv, dbhome,
+ DB_CREATE | DB_INIT_MPOOL | DB_PRIVATE |
DB_USE_ENVIRON, 0)))
+ bdb_err("dbenv::open", rc);
+ dbdata.flags = DB_DBT_USERMEM;
+ if "1%2(dbdata.data = malloc(dbdata.ulen = dbcache)))
+ fail();
+}
+
+/**
+ BDB db.
+*/
+static char *dbfilename;
+static DBT keyret, dataret;
+static bool bdb_is_recno;
+static db_recno_t bdb_recno;
+static DB_HEAP_RID bdb_heaprid;
+static int bdb_get_flags;B2Bstatic void *pointer_get;
+
+void bdb_open(char *dbname) {
+ int rc;
+ bdb_close();
+ if ((rc = db_create(&dbp, dbenv, 0)))
+ bdb_err("db_create", rc);
+ if ((rc = dbp->open(dbp, dbtxn, dbfilename, dbname,
+ DB_UNKNOWN, (parent_dbp ? 0 : DB_RDWRMASTER)|DB_RDONLY,
0))) {
+ fprintf(stderr, "db open %s : %s\n", dbfilename, dbname);
+ bdb_err(dbfilename, rc);
+ }
+}
+
+void bdb_start_chunks() {
+ int rc;
+ bdb_get_flags = DB_NEXT | DB_MULTIPLE_KEY;
+ if ((bdb_is_recno = (dbp->type == DB_RECNO || dbp->type == DB_QUEUE)))
+ keyret.size = sizeof(*(keyret.data = &bdb_recno));
+ else if (dbp->type == DB_HEAP) {
+ bdb_get_flags = DB_NEXT;
+ dbkey.flags = DB_DBT_USERMEM;
+ dbkey.ze % = dbkey.ulen = sizeof(*(dbkey.data = &bdb_heaprid));
+ }
+ if ((rc = dbp->cursor(dbp, NULL, &dbcp, 0)))
+ bdb_err("cursor", rc);
+}
+
+unsigned align(unsigned req, unsigned granule) {
+ return ((req + granule - 1) / granule) * granule;
+}
+
+/**
+ \return true if there's another chunk of records.
+*/
+bool bdb_read_chunk() {
+ int rc;
+ if ((rc = dbcp->get(dbcp, &dbkey, &dbdata, bdb_get_flags))) {
+ if (rc == DB_NOTFOUND)
+ return false;
+ if (rc == DB_BUFFER_SMALL) {
+ dbdata.ulen = dbdata.size = align(dbdata.size, 4096);
+ if (!(dbdata.data = realloc(dbdata.data, dbdata.size)))
+ fail();
+ rc = dbcp->get(dbcp, &dbkey, &dbdata, bdb_get_flags);
+ }
+ if (rc)
+ bdb_err("get chunk", rc);
+ }
+ DB_MULTIPLE_INIT(pointer_get, &dbdata);
+ return true;
+}
+
+/**
+ \return true if there was another record; sets keyret and dataret.
+*/
+bool bdb_next_record_in_chunk() {
+ if (bdb_is_recno)
+ DB_MULTIPLE_RECNO_NEXT(pointer_get, &dbdata,
+ bdb_recno, dataret.data, dataret.size);
+ else
+ DB_MULTIPLE_KEY_NEXT(pointer_get, &dbdata,
+ keyret.data, keyret.size,
+ dataret.data, dataret.size);
+ return dataret.data;
+}
+
+static char hexc_[] = "01234567890ABCDEF";
+
+char hexc(unsigned char i) {
+ return hexc_[i];
+}
+
+void putchar_T(unsigned char c) {
+ if (c >= 32 && c < 127 && c != '\\') {
+ putchar(c);
+ } else {
+ putchar('\\');
+ putchar(hexc(c >> 4));
+ putchar(hexc(c & 0xf));
+ }
+}
+
+/**
+ TODO: could fwrite chunks of no-escape-needed bytes, or probably faster,
+ encode in memory then write once
+*/
+void print_T(char *data, unsigned len) {
+ unsigned i = 0;
+ for (; i < len; ++i)
+ putchar_T(data[i]);
+}
+
+/**
+ Paired lines of text, where the first line of the pair is the key item, and
the
+ second line of the pair is its corresponding data item.
+
+ A simple escape mechanism, where newline and backslash (\\) characters are
special, is
+ applied to the text input. Newline characters are interpreted as record
separators.
+ Backslash characters in the text will be interpreted in one of two ways: If
the backslash
+ character precedes another backslash character, the pair will be interpreted
as a literal
+ backslash. If the backslash character precedes any other character, the two
characters
+ following the backslash will be interpreted as a hexadecimal specification
of a single
+ character; for example, \\0a is a newline character in the ASCII character
set.
+
+ For this reason, any backslash or newline characters that naturally occur in
the text
+ input must be escaped to avoid misinterpretation by
+*/
+void print_record_T() {
+ print_T(keyret.data, keyret.size);
+ putchar('\n');
+ print_T(dataret.data, dataret.size);
+ putchar('\n');
+}
+
+
+char *bdb_open_subdb(DBT key) {
+ if (!(subdbname = malloc(key.size + 1)))
+ fail();
+ memcpy(subdbname, key.data, key.size);
+ subdbname[key.size] = '\0';
+ bdb_open(subdbname);
+ return subdbname;
+}
+
+bool isdir(char *path) {
+ struct stat s;
+ if (stat(path, &s)) {
+ perror("path");
+ fail();
+ }
+ return S_ISDIR(s.st_mode);
+}
+
+void mkdir_if_needed(char *path) {
+ if (mkdir(path, 0755))
+ if (errno != EEXIST) {
+ perror(path);
+ fail();
+ }
+ if (!isdir(path)) {
+ fprintf(stderr, "%s is not a directory and can't mkdir it. try with -n for
no-subdir (to store as a file)", path);
+ fail();
+ }
+}
+
+int main(int argc, char *argv[])
+{
+ int i, rc;
+ MDB_cursor *mc;
+ int envflags = 0, putflags = 0;
+ int textflag = false;
+ bool havemultiple;
+
+ prog = argv[0];
+
+ if (argc < 2) {
+ usage();
+ }
+
+ /* -n: use NOSUBDIR flag on env_open
+ * -S do not use NOSUBDIR
+ * -s subDB: translate just named subDB (default: all)
+ * -N: use NOOVERWRITE on puts
+ * -V: print version and exit
+ * -T: print -s database in format suitable for mdb_load -T (then output not
required)
+ * -b N: batch size=N (default 100)
+ * -f stdout_file: write stdout here instead
+
+ * db_dump-like options:
+ * '-h dir: ('home' dir for relative db filenames default .)
+ * -B: nonblocking db open
+ */
+ bool subdir = true;
+ bool nodup = true;
+ bool listdbs = false;
+
+ while ((i = getopt(argc, argv, "P:h:s:b:lnvVTNS")) != EOF) {
+ switch(i) {
+ case 'b':
+ i = sscanf(optarg, "%d", &batchsize);
+ if (i != 1) {
+ fprintf(stderr, "ERROR: -b '%s' was not int\n", optarg);
+ usage();
+ }
+ break;
+ case 'f':
+ if (freopen(optarg, "w", stdout) == NULL) {
+ fprintf(stderr, "%s: %s: reopen: %s\n",
+ prog, optarg, strerror(errno));
+ exit(EXIT_FAILURE);
+ }
+ break;
+ case 'V':
+ printf("%s\n", MDB_VERSION_STRING);
+ printf("%s\n", db_version(NULL, NULL, NULL));
+ exit(0);
+ break;
+ case 'S':
+ subdir = true;
+ break;
+ case 'n':
+ subdir = false;
+ break;
+ case 's':
+ subname = strdup(optarg);
+ break;
+ case 'N':
+ nodup = true;
+ putflags = MDB_NOOVERWRITE|MDB_NODUPDATA;
+ break;
+ case 'B':
+ dbnonblocking = true;
+ break;
+ case 'T':
+ textflag = true;
+ break;
+ case 'h':
+ dbhome = optarg;
+ break;
+ case 'P':
+ /**
+ we XXX password immediately on init, to hide from top etc. but
would
+ be better to get from stdin (XXX earlier would still be insecure)
+ */
+ dbpasswd = optarg;
+ break;
+ case 'l':
+ listdbs = true;
+ break;
+ case '?':
+ default:
+ usage();
+ }
+ }
+
+ if (!subdir)
+ envflags |= MDB_NOSUBDIR;
+ bool haveout = optind == argc - 2;
B2B if (opndnd >= argc)
+ usage();
+ dbfilename = argv[optind++];
+ char *mdboutpath = haveout ? argv[optind++] : NULL;
+ if (mdboutpath) {
+ if (subdir)
+ mkdir_if_needed(mdboutpath);
+ }
+ if (listdbs) {
+ if (textflag)
+ fprintf(stderr, "disabling -T (print key/val lines) because -l (list dbs)
was specified\n");
+ textflag = false;
+ }
+
+ /**
+ args parsed.
+
+ init BDB:
+ */
+ bdb_init_dbenv();
+ bdb_open(subname);
+
+ /**
+ init MDB:D0D
+ */
+#undef MDB_OK
+#define MDB_OK(call) \
+ if (rc) { \
+ fprintf(stderr, #call " failed - error %d %s\n", rc, mdb_strerror(rc)); \
+ goto shutdown; \
+ } else {}
+
+ if (mdboutpath) {
+ rc = mdb_env_create(&env);
+ MDB_OK(mdb_env_create);
+
+ rc = mdb_env_set_maxdbs(env, 2);
+ MDB_OK(mdb_env_set_maxdbs);
+
% rc = mdb_env_open(env, mdboutpath, envflags, 0664);
+ MDB_OK(mdb_env_open);
+
+ kbuf.mv_size = mdb_env_get_maxkeysize(env) * 2 + 2;
+ kbuf.mv_data = malloc(kbuf.mv_size);
+
+ dbuf.mv_size = datasize;
+ dbuf.mv_data = malloc(dbuf.mv_size);
+ }
+
+ havemultiple = !subname && dbp->get_multiple(dbp);
+ if (havemultiple) {
+ parent_dbp = dbp;
+ dbp = 0;
+ if ((rc = parent_dbp->cursor(parent_dbp, NULL, &bdb_subdbcursor, 0)))
+ bdb_err("cursor(sub-dbs)", rc);
+ }
+
+ unsigned long long wnrecords, wnrecordsall = 0;
+ unsigned long long nrecords, nrecordsall = 0;
+ unsigned ndbs = 0;
+ bool const reading = textflag || mdboutpath;
+ for (;;) {
+ MDB_val key, data;
+ int batch = 0;
+ if (havemultiple) {
+ if ((rc = bdb_subdbcursor->get(bdb_subdbcursor, &dbkey, &dbdata, DB_NEXT
| DB_IGNORE_LEASE))) {
+ if (rc != DB_NOTFOUND)
+ bdb_err("get-next-subdb", rc);
+ else
+ rc = 0;
+ break;
+ }
+ subname = bdb_open_subdb(dbkey);
+ }
+
+ ++ndbs;
+ nrecords = 0;
+ wnrecords = 0;
+ if (subname) {
+ if (listdbs)
+ printf("%s\n", subname);
+ if (reading)
+ fprintf(stderr, "reading DB %s ... ", subname);
+ } ee e {
+ listdbs = false;
+ fprintf(stderr, "reading unnamed DB ... ");
+ }
+
+ if (mdboutpath) {
+ rc = mdb_txn_begin(env, NULL, 0, &txn);
+ MDB_OK(mdb_txn_begin);
+ rc = mdb_open(txn, subname, MDB_CREATE, &dbi);
+ MDB_OK(mdb_open);
+ rc = mdb_cursor_open(txn, dbi, &mc);
+ MDB_OK(mdb_cursor_open);
+ }
+
+ if (reading) {
+ bdb_start_chunks();
+ while (bdb_read_chunk()) {
+ while (bdb_next_record_in_chunk()) {
+ ++nrecords;%%0
+ if (textflag)
+ print_record_T();
+ if (mdboutpath) {
+ key.mv_data = keyret.data;
+ key.mv_size = keyret.size;
+ data.mv_data = dataret.data;
+ data.mv_size = dataret.size;
+ rc = mdb_cursor_put(mc, &key, &data, putflags);
+ if (rc == MDB_KEYEXIST && nodup)
+ continue;
+ ++wnrecords;
+ MDB_OK(mdb_cursor_put);
+ if (++batch == batchsize) {
+ batch = 0;
+ rc = mdb_txn_commit(txn);
+ MDB_OK(mdb_txn_commit);
+ rc = mdb_txn_begin(env, NULL, 0, &txn);
+ MDB_OK(mdb_txn_begin);
+ rc = mdb_cursor_open(txn, dbi, &mc);
+ MDB_OK(mdb_cursor_open);
+ }
+ }
+ }
+ }
+ if (mdboutpath) {
+ rc = mdb_txn_commit(txn);
+ txn = 0;
+ MDB_OK(mdb_txn_commit);
+ mdb_dbi_close(env, dbi);
+ dbi = 0;
+ }
+ nrecordsall += nrecords;
+ wnrecordsall += wnrecords;
+ fprintf(stderr, "%llu records (stored %llu).\n", nrecords, wnrecords);
+ }
+ if (!havemultiple)
+ break;
+ }
+ fprintf(stderr, "uound %u Berkeley DB(s) in input file %s - read %llu
records", ndbs, dbfilename, nrecordsall);
+ if (mdboutpath)
+ fprintf(stderr, " (stored %llu to MDB %s).\n", wnrecordsall, mdboutpath);
+ fprintf(stderr, "\n");
+shutdown:
+ shutdown();
+ return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}
--
1.8.3.GIT
9 years, 1 month
(ITS#7924) perl-backend rfc2696 support...
by info-openldap＠ch2o.info
Full_Name: Mathieu CARBONNEAUX
Version: 2.4.37
OS: Centos 6.4
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (86.73.162.197)
hi,
there any solution to support rfc2696 "Simple Paged Results Manipulation" in
perl-backend ?
actualy they respond err=12 (LDAP_UNAVAILABLE_CRITICAL_EXTENSION) without
calling perl script...
Best Regards,
Mathieu
9 years, 1 month
Re: (ITS#7923) constraint_close / free problems on olcDbDirectory change
by hyc＠symas.com
ebackes(a)symas.com wrote:
> Full_Name: Emily Backes
> Version: RE24
> OS:
> URL:
> Submission from: (NULL) (50.113.67.84)
>
>
> Changing olcDbDirectory on a live server with overlays makes a mess; much like
> the recent ITS#7906, constraint has a problem with open/close management.
>
> constraint_close calls constraint_free( ap, 1 ) which wipes everything out, but
> the state is not restored when things are restarted. In particular, a second
> olcDbDirectory mod results in constraint_free being called again with garbage.
Should note this is related to ITS#7906. I guess we have no choice now but to
revisit all of the remaining overlays in RE24.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9 years, 1 month
