Howard Chu wrote:
Stelios Grigoriadis wrote:
The setup to reproduce the error is as follows: 1 master, 3 replicas.
I've been traveling and haven't yet been able to run this test. In the meantime, can you run slapd with debug level sync, and upload those logs somewhere for this test?
- Start the replicas.
- Start the program that adds persons (parallell_stress_simple.sh). Actually a script that starts a number of processes (add_person.c) on different machines that add persons.
- Start the master.
- When the script completes, compare the number of added entries in the master and replicas.
To Quanah Gibson-Mount: The slapd.conf i also provided.
/Stelios
The output from one run. I haven't had the time to look at it yet.
ldapt1:~/openldap-2.3.38/servers/slapd # ./slapd -d 16384 @(#) $OpenLDAP: slapd 2.3.38 (Oct 5 2007 14:08:48) $ root@ldapt1:/root/openldap-2.3.38/servers/slapd /etc/openldap/slapd.conf: line 101: rootdn is always granted unlimited privileges. /etc/openldap/slapd.conf: line 135: rootdn is always granted unlimited privileges. slapd starting => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6107-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6172-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6139-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6123-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6080-0,dc=example,dc=com) put failed: -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: subtree (cn=dlt30715-1,dc=example,dc=com) put failed: -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6172-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6080-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6107-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6123-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: subtree (cn=dlt6139-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: subtree (cn=dlt6123-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6107-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6080-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: subtree (cn=dlt6139-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: subtree (cn=dlt6123-0,dc=example,dc=com) put failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995) => bdb_dn2id_add: parent (dc=example,dc=com) insert failed: -30995 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 Entry cn=dlt30604-0,dc=example,dc=com CSN 20071005135356Z#00000e#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21127-0,dc=example,dc=com CSN 20071005135356Z#00001d#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6107-0,dc=example,dc=com CSN 20071005135356Z#00001f#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6172-0,dc=example,dc=com CSN 20071005135356Z#000020#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30678-0,dc=example,dc=com CSN 20071005135356Z#000021#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6080-0,dc=example,dc=com CSN 20071005135356Z#000024#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21126-0,dc=example,dc=com CSN 20071005135356Z#000025#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21125-0,dc=example,dc=com CSN 20071005135356Z#000026#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9984-0,dc=example,dc=com CSN 20071005135356Z#000027#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30715-1,dc=example,dc=com CSN 20071005135356Z#000028#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30604-1,dc=example,dc=com CSN 20071005135357Z#000000#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9934-0,dc=example,dc=com CSN 20071005135357Z#000001#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9952-1,dc=example,dc=com CSN 20071005135357Z#000002#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21127-1,dc=example,dc=com CSN 20071005135357Z#000003#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9984-1,dc=example,dc=com CSN 20071005135357Z#000004#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30678-1,dc=example,dc=com CSN 20071005135357Z#000005#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9923-1,dc=example,dc=com CSN 20071005135357Z#000006#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21125-1,dc=example,dc=com CSN 20071005135358Z#000000#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6172-1,dc=example,dc=com CSN 20071005135358Z#000001#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30616-1,dc=example,dc=com CSN 20071005135358Z#000002#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6107-1,dc=example,dc=com CSN 20071005135358Z#000003#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9934-1,dc=example,dc=com CSN 20071005135358Z#000004#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30604-0,dc=example,dc=com CSN 20071005135356Z#00000e#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21127-0,dc=example,dc=com CSN 20071005135356Z#00001d#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6107-0,dc=example,dc=com CSN 20071005135356Z#00001f#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6172-0,dc=example,dc=com CSN 20071005135356Z#000020#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30678-0,dc=example,dc=com CSN 20071005135356Z#000021#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6080-0,dc=example,dc=com CSN 20071005135356Z#000024#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21126-0,dc=example,dc=com CSN 20071005135356Z#000025#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21125-0,dc=example,dc=com CSN 20071005135356Z#000026#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9984-0,dc=example,dc=com CSN 20071005135356Z#000027#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30715-1,dc=example,dc=com CSN 20071005135356Z#000028#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30604-1,dc=example,dc=com CSN 20071005135357Z#000000#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9934-0,dc=example,dc=com CSN 20071005135357Z#000001#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9952-1,dc=example,dc=com CSN 20071005135357Z#000002#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21127-1,dc=example,dc=com CSN 20071005135357Z#000003#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9984-1,dc=example,dc=com CSN 20071005135357Z#000004#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30678-1,dc=example,dc=com CSN 20071005135357Z#000005#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9923-1,dc=example,dc=com CSN 20071005135357Z#000006#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21125-1,dc=example,dc=com CSN 20071005135358Z#000000#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6172-1,dc=example,dc=com CSN 20071005135358Z#000001#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30616-1,dc=example,dc=com CSN 20071005135358Z#000002#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6107-1,dc=example,dc=com CSN 20071005135358Z#000003#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9934-1,dc=example,dc=com CSN 20071005135358Z#000004#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 => bdb_dn2id_add: put failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock -30995 Entry cn=dlt30604-0,dc=example,dc=com CSN 20071005135356Z#00000e#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21127-0,dc=example,dc=com CSN 20071005135356Z#00001d#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6107-0,dc=example,dc=com CSN 20071005135356Z#00001f#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6172-0,dc=example,dc=com CSN 20071005135356Z#000020#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30678-0,dc=example,dc=com CSN 20071005135356Z#000021#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6080-0,dc=example,dc=com CSN 20071005135356Z#000024#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21126-0,dc=example,dc=com CSN 20071005135356Z#000025#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21125-0,dc=example,dc=com CSN 20071005135356Z#000026#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9984-0,dc=example,dc=com CSN 20071005135356Z#000027#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30715-1,dc=example,dc=com CSN 20071005135356Z#000028#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30604-1,dc=example,dc=com CSN 20071005135357Z#000000#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9934-0,dc=example,dc=com CSN 20071005135357Z#000001#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9952-1,dc=example,dc=com CSN 20071005135357Z#000002#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21127-1,dc=example,dc=com CSN 20071005135357Z#000003#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9984-1,dc=example,dc=com CSN 20071005135357Z#000004#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30678-1,dc=example,dc=com CSN 20071005135357Z#000005#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9923-1,dc=example,dc=com CSN 20071005135357Z#000006#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21125-1,dc=example,dc=com CSN 20071005135358Z#000000#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6172-1,dc=example,dc=com CSN 20071005135358Z#000001#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt30616-1,dc=example,dc=com CSN 20071005135358Z#000002#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt6107-1,dc=example,dc=com CSN 20071005135358Z#000003#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt9934-1,dc=example,dc=com CSN 20071005135358Z#000004#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000 Entry cn=dlt21126-1,dc=example,dc=com CSN 20071005135358Z#000005#00#000000 greater than snapshot 20071005135356Z#00000d#00#000000