Re: (ITS#7172) SEGFAULT in openldap-2.4.28 & 2.4.29
by david.coutadeur＠linagora.com
Hi,
I finally had the opportunity to get a valgrind analysis of my problem.
You can find it below. Thank you for any help.
D.
==19496== 12 bytes in 1 blocks are definitely lost in loss record 16 of 52
==19496== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==19496== by 0x4056977: ber_memalloc_x (memory.c:228)
==19496== by 0x8095D39: ch_malloc (ch_malloc.c:54)
==19496== by 0x80E0297: do_syncrepl (syncrepl.c:706)
==19496== by 0x4016A23: ldap_int_thread_pool_wrapper (tpool.c:688)
==19496== by 0x821831: start_thread (in /lib/libpthread-2.5.so)
==19496== by 0x76146D: clone (in /lib/libc-2.5.so)
==19496==
==19496== 155 (32 direct, 123 indirect) bytes in 1 blocks are definitely
lost in loss record 47 of 52
==19496== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==19496== by 0x4056977: ber_memalloc_x (memory.c:228)
==19496== by 0x4057054: ber_bvarray_dup_x (memory.c:756)
==19496== by 0x80E0212: do_syncrepl (syncrepl.c:699)
==19496== by 0x4016A23: ldap_int_thread_pool_wrapper (tpool.c:688)
==19496== by 0x821831: start_thread (in /lib/libpthread-2.5.so)
==19496== by 0x76146D: clone (in /lib/libc-2.5.so)
==19496==
==19496== 270 (228 direct, 42 indirect) bytes in 3 blocks are definitely
lost in loss record 49 of 52
==19496== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==19496== by 0x4056977: ber_memalloc_x (memory.c:228)
==19496== by 0x8095D39: ch_malloc (ch_malloc.c:54)
==19496== by 0x8195E91: syncprov_op_search (syncprov.c:2473)
==19496== by 0x80E3333: overlay_op_walk (backover.c:661)
==19496== by 0x80E3A09: over_op_func (backover.c:723)
==19496== by 0x807BE85: fe_op_search (search.c:402)
==19496== by 0x807C772: do_search (search.c:247)
==19496== by 0x807988B: connection_operation (connection.c:1150)
==19496== by 0x807A0FC: connection_read_thread (connection.c:1286)
==19496== by 0x4016A23: ldap_int_thread_pool_wrapper (tpool.c:688)
==19496== by 0x821831: start_thread (in /lib/libpthread-2.5.so)
Le 20/02/2012 10:32, David Coutadeur a écrit :
>
> Hi,
>
> Concerning the cpu :
> model name : Intel(R) Xeon(R) CPU L5640 @ 2.27GHz
> However, OpenLDAP is used in a virtual machine, with only one dedicated
> core. This is 32 bits version of OpenLDAP, on a 32 bits OS.
> Linux OS :
> Red Hat Enterprise Linux Server release 5.7 (Tikanga)
> kernel :
> 2 2.6.18-274.3.1.el5
>
> You can get the server configuration below.
>
> Concerning the client, you can find the source code here :
> http://loadtesting.sourceforge.net/index.php?lang=en
>
>
>
> #
> ----------------------------------------------------------------------------
> # Global section
> #
> ----------------------------------------------------------------------------
>
> sizelimit 15000
>
> allow bind_v2
> password-hash {SSHA}
> threads 8
> loglevel 256
>
> serverID 2
>
> include /opt/openldap/etc/openldap/inc_shemas/slapd-schemas.conf
>
> pidfile /opt/openldap/var/run/slapd.pid
> argsfile /opt/openldap/var/run/slapd.args
>
> # ACLs definition (cutted here)
>
> #######################################################################
> # SSL definitions
> #######################################################################
> # Definition of cert files
> # ---------------------------------
> TLSCipherSuite HIGH:MEDIUM:+SSLv2
>
> TLSCACertificateFile /usr/local/openldap/etc/certs/ca.cert
> TLSCertificateFile /usr/local/openldap/etc/certs/ldap-master-2.cert
> TLSCertificateKeyFile /usr/local/openldap/etc/certs/ldap-master-2.key
>
> TLSVerifyClient never
>
>
>
> #######################################################################
> # BDB database definitions
> #######################################################################
> database monitor
>
> #######################################################################
> # suffix dc=example,dc=com
> # --------------------
> #######################################################################
> database bdb
> directory /opt/openldap/var/openldap-data
> # some indexes
>
> #
> ----------------------------------------------------------------------------
> # root suffix
> #
> ----------------------------------------------------------------------------
> suffix "dc=example,dc=com"
> checkpoint 512 10
>
> #
> ----------------------------------------------------------------------------
> # cache settings
> #
> ----------------------------------------------------------------------------
>
> cachesize 60000
> dncachesize 60000
> idlcachesize 60000
> cachefree 100
>
> rootdn "cn=Manager,dc=example,dc=com"
> rootpw secret
>
> # Password policy : hash the clear passwords
> overlay ppolicy
> ppolicy_hash_cleartext
>
>
> #
> ----------------------------------------------------------------------------
> # REPLICATION definitions
> #
> ----------------------------------------------------------------------------
> #######################################################################
>
> # I am a master (provider)
> # ------------------------------------------
> overlay syncprov
> syncprov-checkpoint 100 10
> syncprov-sessionlog 100
>
> # syncrepl directive (consummer infos)
> # ------------------------------------------
> syncrepl rid=001
> provider=ldap://ldap-master-1.example.com
> bindmethod=simple
> binddn="uid=ReplicationMaster,ou=Replication,ou=Special
> Users,dc=example,dc=com"
> credentials=secret
> searchbase="dc=example,dc=com"
> schemachecking=on
> type=refreshAndPersist
> retry="60 +"
>
> mirrormode on
>
>
> #
> ----------------------------------------------------------------------------
> # limits
> #
> ----------------------------------------------------------------------------
>
> limits dn.exact="uid=ReplicationMaster,ou=Replication,ou=Special
> Users,dc=example,dc=com" size=unlimited time=unlimited
> limits dn.exact="uid=ReplicationHub,ou=Replication,ou=Special
> Users,dc=example,dc=com" size=unlimited time=unlimited
> limits dn.exact="uid=ReplicationLSC,ou=Replication,ou=Special
> Users,dc=example,dc=com" size=unlimited time=unlimited
>
>
> #######################################################################
> # suffix o=edition
> # ----------------
> #######################################################################
> database bdb
>
> directory /opt/openldap/var/openldap-data-edition
> # some indexes
>
>
> #
> ----------------------------------------------------------------------------
> # root suffix
> #
> ----------------------------------------------------------------------------
> suffix "o=edition"
> checkpoint 512 10
>
> #
> ----------------------------------------------------------------------------
> # cache settings
> #
> ----------------------------------------------------------------------------
>
> cachesize 50000
> dncachesize 50000
> idlcachesize 50000
> cachefree 100
>
>
>
> #
> ----------------------------------------------------------------------------
> # rootdn
> #
> ----------------------------------------------------------------------------
> # Cleartext passwords, especially for the rootdn, should
> # be avoid. See slappasswd(8) and slapd.conf(5) for details.
> # Use of strong authentication encouraged.
> rootdn "cn=Manager,o=edition"
> rootpw secret
>
> # Password policy : hash the clear passwords
> overlay ppolicy
> ppolicy_hash_cleartext
>
>
> #######################################################################
> # REPLICATION definitions
> #######################################################################
>
> # I am a master (provider)
> # ------------------------------------------
> overlay syncprov
> syncprov-checkpoint 100 10
> syncprov-sessionlog 100
>
> # syncrepl directive (consummer infos)
> # ------------------------------------------
> syncrepl rid=002
> provider=ldap://vspar-ldap-master-1.example.com
> bindmethod=simple
> binddn="uid=ReplicationMaster,ou=Replication,ou=Special
> Users,dc=example,dc=com"
> credentials=secret
> searchbase="o=edition"
> schemachecking=on
> type=refreshAndPersist
> retry="60 +"
>
> mirrormode on
>
> #
> ----------------------------------------------------------------------------
> # Limits for current BDB
> #
> ----------------------------------------------------------------------------
>
> limits dn.exact="uid=ReplicationMaster,ou=Replication,ou=Special
> Users,dc=example,dc=com" size=unlimited time=unlimited
> limits dn.exact="uid=ReplicationHub,ou=Replication,ou=Special
> Users,dc=example,dc=com" size=unlimited time=unlimited
> limits dn.exact="uid=ReplicationLSC,ou=Replication,ou=Special
> Users,dc=example,dc=com" size=unlimited time=unlimited
>
>
>
>
>
> Le 17/02/2012 19:00, Howard Chu a écrit :
>> dcoutadeur(a)linagora.com wrote:
>>> Full_Name: dcoutadeur
>>> Version: 2.4.28
>>> OS: Red Hat Enterprise Linux Server release 5.7 (Tikanga)
>>> URL: ftp://ftp.openldap.org/incoming/
>>> Submission from: (NULL) (109.197.176.10)
>>>
>>>
>>>
>>> Hello,
>>>
>>>
>>> I had a segfault in the last git version of OpenLDAP, after 10 to 15
>>> tests, each
>>> interrupted by Ctrl+C. (see what's a test below)
>>> The segfault is also reproduced in version 2.4.28.
>>>
>>> I think I won't be able to reproduce the bug with Valgrind.
>>
>> If you expect us to try to reproduce the bug, you'll have to provide
>> more information. Since this crash is in syncprov there's obviously at
>> least two servers involved; what are their configurations? What hardware
>> are they running on (in particular, how many CPU cores per server)?
>> Since you're using a custom client, can you provide the client source code?
>>>
>>> Thank you in advance for any help.
>>>
>>> D.
>>>
>>>
>>> Note : A test is 100 times 100 threads, each doing a bind, an add, a
>>> modify, a
>>> delete, and a logout.
>>>
>>>
>>>
>>>
>>>
>>> (gdb) bt full
>>> #0 sp_avl_cmp (c1=0x8b4004c8, c2=0xa37cf28) at syncprov.c:366
>>> rc =<value optimized out>
>>> #1 0x081afe3e in avl_delete (root=0xa255648, data=0x8b4004c8,
>>> fcmp=0x81948a0<sp_avl_cmp>) at avl.c:197
>>> p =<value optimized out>
>>> q =<value optimized out>
>>> r =<value optimized out>
>>> top =<value optimized out>
>>> side =<value optimized out>
>>> side_bf =<value optimized out>
>>> shorter =<value optimized out>
>>> nside =<value optimized out>
>>> pptr = {0x89908, 0x0, 0x0, 0x0, 0xe8043c, 0x0, 0xfdc,
>>> 0x8d994858, 0xe7b95c, 0xfdc, 0xa372570, 0x0, 0xa288350,
>>> 0xe8043c, 0xa372570, 0x8d994878, 0xe7c324, 0xfdc, 0xa372570,
>>> 0x0, 0xe7af2c, 0x8cb9136e, 0x81b3634, 0x0, 0xe8043c,
>>> 0xe, 0xa3d3a40, 0x8d9948b8, 0xe7d081, 0xa3e3618, 0x8cb91358,
>>> 0x823c27}
>>> pdir =
>>> "\000\000\000\000lI\231\215\064\066\033\bn>\000\000\244\345t\000\310\004@\213\b\346\067\n\310H\231\215"
>>>
>>> depth = 0
>>> #2 0x08199f7f in syncprov_op_cleanup (op=0xa37e608, rs=0x8d995108) at
>>> syncprov.c:1401
>>> cb = 0x8cb91258
>>> opc = 0x8cb91268
>>> si = 0xa255610
>>> sm = 0xa255688
>>> snext =<value optimized out>
>>> mt = 0x8b4004c8
>>> #3 0x08089654 in slap_cleanup_play (op=0xa37e608, rs=0x8d995108) at
>>> result.c:541
>>> sc_next = 0x8d994dec
>>> sc = 0x8cb91258
>>> scp = 0x8d994928
>>> #4 0x0808a150 in send_ldap_response (op=0xa37e608, rs=0x8d995108) at
>>> result.c:733
>>> berbuf = {
>>> buffer = "\000\000\001\000\000\001\000\000\377\377\377\377",
>>> '\000'<repeats 12 times>,
>>> "f\023\271\214\064#\271\214\000\000\000\000f\023\271\214p%7\n\000\000\000\000\314I\231\215\001\000\000\000\000\000\000\000\314mK\236x\271\347\000\001\000\000\000`+@\213D`K\236\230\063\066\n\250<6\n\000\000\000\000\000\000\000\000\005\000\000\000P7@\213`\343\070\n\000\000\000\000\n\000\000\000(\234\200\330\000\000\000\000\000\000\000\000@4\"\000\000\000\000\000(\234\200\330\210J\231\215\270\214
>>>
>>> \000\230\063\066\n`+@\213\314mK\236\r\000\000\000\001\000\000\000\021\217;O(\234\200\330\000\000\000\000`&%\n`&%\n8J\231\215b\f\"\000\224mK\236\230\063\066\n(\234\200أ\347\022\b\a",
>>>
>>> '\000'<repeats 31 times>, "D'%\n\224mK\236\000\000\000",
>>> ialign = 65536, lalign = 65536, falign = 9.18354962e-41,
>>> dalign = 5.4323095486619588e-312,
>>> palign = 0x10000<Address 0x10000 out of bounds>}
>>> ber =<value optimized out>
>>> rc = 32768
>>> bytes = 14
>>> __PRETTY_FUNCTION__ = "send_ldap_response"
>>> #5 0x0808af1f in slap_send_ldap_result (op=0xa37e608, rs=0x8d995108) at
>>> result.c:860
>>> tmp = 0x0
>>> otext = 0x0
>>> oref = 0x0
>>> __PRETTY_FUNCTION__ = "slap_send_ldap_result"
>>> #6 0x0812bde5 in bdb_add (op=0xa37e608, rs=0x8d995108) at add.c:511
>>> pdn = {bv_len = 23, bv_val = 0x8b40372f
>>> "ou=people,dc=afp,dc=com"}
>>> p = 0x8fc4c0fc
>>> oe = 0x8fc4c804
>>> ei = 0xa37d1c8
>>> textbuf = "\000\000\000\000\320O\"\n", '\000'<repeats 48
>>> times>, "\001", '\000'<repeats 198 times>
>>> children = 0xa223b20
>>> entry = 0xa223980
>>> ltid = 0x0
>>> lt2 = 0x8b402bf0
>>> eid = 57976
>>> opinfo = {boi_oe = {oe_next = {sle_next = 0x8d99509c}, oe_key =
>>> 0x0}, boi_txn = 0x8b402b60, boi_locks = 0x0,
>>> boi_err = 0, boi_acl_cache = 0 '\000', boi_flag = 0 '\000'}
>>> lock = {off = 133260, ndx = 772, gen = 2004, mode =
>>> DB_LOCK_READ}
>>> num_retries = 0
>>> success = 0
>>> postread_ctrl = 0x0
>>> ctrls = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
>>> num_ctrls = 0
>>> #7 0x080e33a1 in overlay_op_walk (op=0xa37e608, rs=0x8d995108,
>>> which=op_add, oi=0xa254ff0, on=0xa255508) at backover.c:671
>>> rc = 32768
>>> #8 0x080e3a0a in over_op_func (op=0xa37e608, rs=0x8d995108,
>>> which=op_add) at backover.c:723
>>> oi = 0xa254ff0
>>> on = 0xa255508
>>> be = 0xa252560
>>> db = {bd_info = 0x821d41c, bd_self = 0xa252560,
>>> be_ctrls =
>>> "\000\001\001\001\000\001\000\000\001\000\000\001\001\000\001\001\000\000\000\000\001\000\001\000\000\000\000\000\000\000\000\000\001",
>>>
>>> be_flags = 563464, be_restrictops = 0, be_requires = 0, be_ssf_set =
>>> {sss_ssf = 0,
>>> sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf
>>> = 0, sss_update_transport = 0, sss_update_tls = 0,
>>> sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix =
>>> 0xa288350, be_nsuffix = 0xa288368, be_schemadn = {
>>> bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0,
>>> bv_val = 0x0}, be_rootdn = {bv_len = 24,
>>> bv_val = 0xa287648 "cn=Manager,dc=afp,dc=com"}, be_rootndn =
>>> {bv_len = 24,
>>> bv_val = 0xa2876d0 "cn=manager,dc=afp,dc=com"}, be_rootpw =
>>> {bv_len = 38,
>>> bv_val = 0xa2876f0
>>> "{SSHA}rEmMhg3MU5xkQX5Ng92tH4WzGMlA+nGU"}, be_max_deref_depth = 15,
>>> be_def_limit = {
>>> lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = 15000,
>>> lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0,
>>> lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits =
>>> 0xa255748, be_acl = 0x0, be_dfltaccess = ACL_READ,
>>> be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val =
>>> 0x0}, be_update_refs = 0x0,
>>> be_pending_csn_list = 0xa363388, be_pcl_mutex = {__data =
>>> {__lock = 0, __count = 0, __owner = 0, __kind = 0,
>>> __nusers = 0, {__spins = 0, __list = {__next = 0x0}}},
>>> __size = '\000'<repeats 23 times>, __align = 0},
>>> be_syncinfo = 0xa28aec8, be_pb = 0x0, be_cf_ocs = 0x821f840,
>>> be_private = 0xa252660, be_next = {
>>> stqe_next = 0xa288538}}
>>> cb = {sc_next = 0x0, sc_response = 0x80e30e0
>>> <over_back_response>, sc_cleanup = 0, sc_private = 0xa254ff0}
>>> sc =<value optimized out>
>>> rc =<value optimized out>
>>> __PRETTY_FUNCTION__ = "over_op_func"
>>> #9 0x08081129 in fe_op_add (op=0xa37e608, rs=0x8d995108) at add.c:334
>>> repl_user = 0
>>> rc =<value optimized out>
>>> bd = 0x82234c0
>>> textbuf =
>>> "\000\000\000\000\000\000\000\000\060[\231\215\000\000\000\000\035\000\000\000\020\070@\213\001\000\000\000xN\231\215\270\026@\213(I\"\n\002\000\000\000\250N\231\215\255\214\v\b\270\026@\213\224N\231\215\001\000\000\000\000\000\000\000x9@\213\000\000\000\000\n\000\000\000\001\000\000\000\340\067@\213\n\000\000\000\060\070@\213\320\026@\213(I\"\n\270\026@\213\370N\231\215oc\t\b\002\000\000\000X(
>>>
>>> \n\370N\231\215\321_\t\bh\234!\n\240\066@\213'<\202\000\000\000\000\000\f\000\000\000W.@\213n>\000\000\244\345t\000\320O\"\n\320O\"\n\370N\231\215\035\205q\000
>>>
>>> .\"\b\314h\032\216\030O\231\215\245\063\b\b
>>> .\"\b\240\066@\213\000\000\000\000\270\026@\213\244i\032\216\000\000\000\000HO\231\215\267\r\b\b\320O\"\n\320O\"\n\001\000\000\000HO\231\215\020\000\000\000\340h\032\216\377\377\377\377"
>>>
>>> __PRETTY_FUNCTION__ = "fe_op_add"
>>> #10 0x08081a13 in do_add (op=0xa37e608, rs=0x8d995108) at add.c:194
>>> ber =<value optimized out>
>>> last = 0x8b402e71 ""
>>> dn = {bv_len = 38, bv_val = 0x8b402d98
>>> "uid=dcoutadeur,ou=People,dc=afp,dc=com"}
>>> len = 28
>>> tag =<value optimized out>
>>> modlist = 0x8b4015f0
>>> modtail = 0x8b403694
>>> tmp = {sml_mod = {sm_desc = 0x80ce5ca, sm_values = 0x8b4036a0,
>>> sm_nvalues = 0x0, sm_numvals = 2375635128,
>>> sm_op = 0, sm_flags = 0, sm_type = {bv_len = 12, bv_val =
>>> 0x8b402e57 "userPassword"}}, sml_next = 0x823c27}
>>> textbuf =
>>> "\025\000\000\000\310\031@\213\b\026@\213\006\340(\000\220[\231\215\000\000\000\000\000\000\000\000\020\000@\213\025\000\000\000\310\031@\213\310+@\213\005\070/\000\200O@\213T\245(\000\000\000\000\000\020\000@\213\364\237\067\000\220[\231\215\000\000\000\000\233\071@\213@:=\n\370O\231\215T\213\202\000b\213\202\000;\334\347\000\"\000\000\000\233\071@\213\b\000\000\000\201\354(\000\fP\231\215<\004\350\000\270P\231\215\312\315\347\000\370\326\070\n\233\071@\213\b\000\000\000\001\200\255\373\b\347\067\n@\000\000\000\243P\231\215@\000@\213\026\347\067\n@\000@\213\b\347\067\n@\261\067\000,\000\000\000\020\000@\213",
>>>
>>> '\000'<repeats 20 times>,
>>> "5\000\000\000@\000@\213\000\000\000\000\340\021@\213\000\000\000\000\000\000\000\000\260+@\213\000\000\000\000\001\000\000\000\004\000\020\000\350Q\231\215\310P\231\215"
>>>
>>> rc =<value optimized out>
>>> freevals =<value optimized out>
>>> oex = {oe = {oe_next = {sle_next = 0x0}, oe_key = 0x8081330},
>>> oe_db = 0x0}
>>> #11 0x0807988c in connection_operation (ctx=0x8d9951e8, arg_v=0xa37e608)
>>> at connection.c:1150
>>> rc =<value optimized out>
>>> cancel =<value optimized out>
>>> rs = {sr_type = REP_RESULT, sr_tag = 105, sr_msgid = 2, sr_err =
>>> 0, sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0,
>>> sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0,
>>> r_attr_flags = 0, r_operational_attrs = 0x0, r_attrs = 0x0,
>>> r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata =
>>> 0x0}, sru_extended = {r_rspoid = 0x0,
>>> r_rspdata = 0x0}}, sr_flags = 0}
>>> tag = 104
>>> opidx = SLAP_OP_ADD
>>> conn = 0xb7f3bc10
>>> memctx = 0xa372570
>>> memctx_null = 0x0
>>> __PRETTY_FUNCTION__ = "connection_operation"
>>> #12 0x0807a0fd in connection_read_thread (ctx=0x8d9951e8, argv=0x22) at
>>> connection.c:1286
>>> s =<value optimized out>
>>> #13 0x00717a24 in ldap_int_thread_pool_wrapper (xpool=0xa2265c8) at
>>> tpool.c:688
>>> task = 0xa382e10
>>> work_list =<value optimized out>
>>> ctx = {ltu_id = 2375637904, ltu_key = {{ltk_key = 0x80ce400,
>>> ltk_data = 0xa372570,
>>> ltk_free = 0x80ce430<slap_sl_mem_destroy>}, {ltk_key =
>>> 0xa363398, ltk_data = 0xa371a88,
>>> ltk_free = 0x812e4c0<bdb_reader_free>}, {ltk_key =
>>> 0x8078320, ltk_data = 0xa37de68,
>>> ltk_free = 0x80783f0<conn_counter_destroy>}, {ltk_key =
>>> 0x808dde0, ltk_data = 0x0,
>>> ltk_free = 0x808dbf0<slap_op_q_destroy>}, {ltk_key = 0x0,
>>> ltk_data = 0x0, ltk_free = 0}<repeats 28 times>}}
>>> kctx =<value optimized out>
>>> keyslot = 241
>>> hash = 5278961
>>> __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
>>> #14 0x00821832 in start_thread () from /lib/libpthread.so.0
>>> No symbol table info available.
>>> #15 0x002f746e in clone () from /lib/libc.so.6
>>> No symbol table info available.
>>>
>>>
>>>
>>>
>>>
>>
>>
