https://bugs.openldap.org/show_bug.cgi?id=9626
Issue ID: 9626 Summary: Segmentation fault on mdb_midl_append_list Product: LMDB Version: 0.9.29 Hardware: x86_64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: --- Component: liblmdb Assignee: bugs@openldap.org Reporter: carlos.velasco@nimastelecom.com Target Milestone: ---
Hello,
Using LMDB for modsecurity 3 I get segmentation fauls of httpd every few hours. Core debugging shows it ocurrs in mdb_midl_append_list in LMDB lib.
# gdb /usr/sbin/httpd core.httpd.25.127c0e0a8a1e468f8d5749d995f81381.204107.1628497829000000000000 GNU gdb (GDB) 9.2 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/httpd... (No debugging symbols found in /usr/sbin/httpd) [New LWP 204177] [New LWP 204154] [New LWP 204152] [New LWP 204151] [New LWP 204107] [New LWP 204169] [New LWP 204147] [New LWP 204149] [New LWP 204170] [New LWP 204173] [New LWP 204186] [New LWP 204185] [New LWP 204181] [New LWP 204189] [New LWP 204184] [New LWP 204171] [New LWP 204172] [New LWP 204178] [New LWP 204175] [New LWP 204187] [New LWP 204174] [New LWP 204176] [New LWP 204179] [New LWP 204180] [New LWP 204182] [New LWP 204183] [New LWP 204188] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/httpd -k start'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 175 midl.c: No such file or directory. [Current thread is 1 (Thread 0x7f2a09ffb640 (LWP 204177))] (gdb) bt #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 #1 0x00007f2a32a325bf in mdb_txn_commit (txn=0xf9bda0) at mdb.c:3485 #2 0x00007f2a32eb8904 in modsecurity::collection::backend::LMDB::storeOrUpdateFirst (this=0x1fe28b0, key=..., value=...) at collection/backend/lmdb.cc:245 #3 0x00007f2a32e97bb8 in modsecurity::collection::Collection::storeOrUpdateFirst (value=..., compartment2=..., compartment=..., key=..., this=0x1fe28b0) at ../headers/modsecurity/collection/collection.h:99 #4 modsecurity::variables::Ip_DynamicElement::storeOrUpdateFirst (value=..., var=..., t=<optimized out>) at ../src/variables/ip.h:110 #5 modsecurity::actions::SetVar::evaluate (this=0x30acfc0, rule=<optimized out>, t=<optimized out>) at actions/set_var.cc:144 #6 0x00007f2a32e641bc in modsecurity::RuleWithActions::executeActionsIndependentOfChainedRuleResult (this=this@entry=0x30c9f50, trans=trans@entry=0x7f29f8036e40, containsBlock=containsBlock@entry=0x7f2a09ff94ef, ruleMessage=...) at rule_with_actions.cc:199 #7 0x00007f2a32e6dc33 in modsecurity::RuleWithOperator::evaluate (this=<optimized out>, trans=<optimized out>, ruleMessage=...) at /usr/include/c++/11.2.0/ext/atomicity.h:109 #8 0x00007f2a32e66e59 in modsecurity::RuleWithActions::evaluate (this=0x30c9f50, transaction=0x7f29f8036e40) at /usr/include/c++/11.2.0/ext/atomicity.h:111 #9 0x00007f2a32e5cd3c in modsecurity::RulesSet::evaluate (this=<optimized out>, phase=phase@entry=3, t=t@entry=0x7f29f8036e40) at rules_set.cc:210 #10 0x00007f2a32e41793 in modsecurity::Transaction::processRequestBody (this=0x7f29f8036e40) at transaction.cc:942 #11 0x00007f2a32fa0a28 in hook_request_late () from /usr/lib64/httpd/modules/mod_security3.so #12 0x000000000045616b in ap_process_request_internal () #13 0x0000000000476ef3 in ap_process_async_request () #14 0x0000000000473150 in ap_process_http_connection () #15 0x00000000004695bf in ap_run_process_connection () #16 0x00007f2a33492831 in process_socket () from /usr/lib64/httpd/modules/mod_mpm_event.so #17 0x00007f2a33493307 in worker_thread () from /usr/lib64/httpd/modules/mod_mpm_event.so #18 0x00007f2a33703fd6 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f2a336241df in clone () from /lib64/libc.so.6 (gdb)
Regards, Carlos Velasco
https://bugs.openldap.org/show_bug.cgi?id=9626
--- Comment #1 from carlos.velasco@nimastelecom.com --- It seems ids get corrupted.
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 175 if (ids[0] + app[0] >= ids[-1]) { [Current thread is 1 (Thread 0x7f2a09ffb640 (LWP 204177))] (gdb) bt #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 #1 0x00007f2a32a325bf in mdb_txn_commit (txn=0xf9bda0) at mdb.c:3485 #2 0x00007f2a32eb8904 in modsecurity::collection::backend::LMDB::storeOrUpdateFirst (this=0x1fe28b0, key=..., value=...) at collection/backend/lmdb.cc:245 #3 0x00007f2a32e97bb8 in modsecurity::collection::Collection::storeOrUpdateFirst (value=..., compartment2=..., compartment=..., key=..., this=0x1fe28b0) at ../headers/modsecurity/collection/collection.h:99 #4 modsecurity::variables::Ip_DynamicElement::storeOrUpdateFirst (value=..., var=..., t=<optimized out>) at ../src/variables/ip.h:110 #5 modsecurity::actions::SetVar::evaluate (this=0x30acfc0, rule=<optimized out>, t=<optimized out>) at actions/set_var.cc:144 #6 0x00007f2a32e641bc in modsecurity::RuleWithActions::executeActionsIndependentOfChainedRuleResult (this=this@entry=0x30c9f50, trans=trans@entry=0x7f29f8036e40, containsBlock=containsBlock@entry=0x7f2a09ff94ef, ruleMessage=...) at rule_with_actions.cc:199 #7 0x00007f2a32e6dc33 in modsecurity::RuleWithOperator::evaluate (this=<optimized out>, trans=<optimized out>, ruleMessage=...) at /usr/include/c++/11.2.0/ext/atomicity.h:109 #8 0x00007f2a32e66e59 in modsecurity::RuleWithActions::evaluate (this=0x30c9f50, transaction=0x7f29f8036e40) at /usr/include/c++/11.2.0/ext/atomicity.h:111 #9 0x00007f2a32e5cd3c in modsecurity::RulesSet::evaluate (this=<optimized out>, phase=phase@entry=3, t=t@entry=0x7f29f8036e40) at rules_set.cc:210 #10 0x00007f2a32e41793 in modsecurity::Transaction::processRequestBody (this=0x7f29f8036e40) at transaction.cc:942 #11 0x00007f2a32fa0a28 in hook_request_late () from /usr/lib64/httpd/modules/mod_security3.so #12 0x000000000045616b in ap_process_request_internal () #13 0x0000000000476ef3 in ap_process_async_request () #14 0x0000000000473150 in ap_process_http_connection () #15 0x00000000004695bf in ap_run_process_connection () #16 0x00007f2a33492831 in process_socket () from /usr/lib64/httpd/modules/mod_mpm_event.so #17 0x00007f2a33493307 in worker_thread () from /usr/lib64/httpd/modules/mod_mpm_event.so #18 0x00007f2a33703fd6 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f2a336241df in clone () from /lib64/libc.so.6 (gdb) print ids $1 = (MDB_IDL) 0x7069746c756d7c20 (gdb) print ids[0] Cannot access memory at address 0x7069746c756d7c20
https://bugs.openldap.org/show_bug.cgi?id=9626
--- Comment #2 from Howard Chu hyc@openldap.org --- (In reply to carlos.velasco from comment #1)
It seems ids get corrupted.
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 175 if (ids[0] + app[0] >= ids[-1]) { [Current thread is 1 (Thread 0x7f2a09ffb640 (LWP 204177))] (gdb) bt #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 #1 0x00007f2a32a325bf in mdb_txn_commit (txn=0xf9bda0) at mdb.c:3485
In frame 1 can you get the output for: info locals print *txn print *parent
Does your code actually use nested transactions?
#2 0x00007f2a32eb8904 in modsecurity::collection::backend::LMDB::storeOrUpdateFirst (this=0x1fe28b0, key=..., value=...) at collection/backend/lmdb.cc:245 #3 0x00007f2a32e97bb8 in modsecurity::collection::Collection::storeOrUpdateFirst (value=..., compartment2=..., compartment=..., key=..., this=0x1fe28b0) at ../headers/modsecurity/collection/collection.h:99 #4 modsecurity::variables::Ip_DynamicElement::storeOrUpdateFirst (value=..., var=..., t=<optimized out>) at ../src/variables/ip.h:110 #5 modsecurity::actions::SetVar::evaluate (this=0x30acfc0, rule=<optimized out>, t=<optimized out>) at actions/set_var.cc:144 #6 0x00007f2a32e641bc in modsecurity::RuleWithActions::executeActionsIndependentOfChainedRuleResult (this=this@entry=0x30c9f50, trans=trans@entry=0x7f29f8036e40, containsBlock=containsBlock@entry=0x7f2a09ff94ef, ruleMessage=...) at rule_with_actions.cc:199 #7 0x00007f2a32e6dc33 in modsecurity::RuleWithOperator::evaluate (this=<optimized out>, trans=<optimized out>, ruleMessage=...) at /usr/include/c++/11.2.0/ext/atomicity.h:109 #8 0x00007f2a32e66e59 in modsecurity::RuleWithActions::evaluate (this=0x30c9f50, transaction=0x7f29f8036e40) at /usr/include/c++/11.2.0/ext/atomicity.h:111 #9 0x00007f2a32e5cd3c in modsecurity::RulesSet::evaluate (this=<optimized out>, phase=phase@entry=3, t=t@entry=0x7f29f8036e40) at rules_set.cc:210 #10 0x00007f2a32e41793 in modsecurity::Transaction::processRequestBody (this=0x7f29f8036e40) at transaction.cc:942 #11 0x00007f2a32fa0a28 in hook_request_late () from /usr/lib64/httpd/modules/mod_security3.so #12 0x000000000045616b in ap_process_request_internal () #13 0x0000000000476ef3 in ap_process_async_request () #14 0x0000000000473150 in ap_process_http_connection () #15 0x00000000004695bf in ap_run_process_connection () #16 0x00007f2a33492831 in process_socket () from /usr/lib64/httpd/modules/mod_mpm_event.so #17 0x00007f2a33493307 in worker_thread () from /usr/lib64/httpd/modules/mod_mpm_event.so #18 0x00007f2a33703fd6 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f2a336241df in clone () from /lib64/libc.so.6 (gdb) print ids $1 = (MDB_IDL) 0x7069746c756d7c20 (gdb) print ids[0] Cannot access memory at address 0x7069746c756d7c20
https://bugs.openldap.org/show_bug.cgi?id=9626
--- Comment #3 from carlos.velasco@nimastelecom.com --- (In reply to Howard Chu from comment #2)
(In reply to carlos.velasco from comment #1)
It seems ids get corrupted.
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 175 if (ids[0] + app[0] >= ids[-1]) { [Current thread is 1 (Thread 0x7f2a09ffb640 (LWP 204177))] (gdb) bt #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 #1 0x00007f2a32a325bf in mdb_txn_commit (txn=0xf9bda0) at mdb.c:3485
In frame 1 can you get the output for: info locals print *txn print *parent
(gdb) f 1 #1 0x00007f2a32a325bf in mdb_txn_commit (txn=0xf9bda0) at mdb.c:3485 3485 rc = mdb_midl_append_list(&parent->mt_free_pgs, txn->mt_free_pgs); (gdb) info locals parent = 0x7f29f8041aeb lp = 0x7f2a09ff9120 dst = 0x7f2a09ff90e0 pspill = 0x7f2a09ff90b0 y = 167743792 len = 0 src = 0x40009000d0006 x = 32554 ps_len = 32554 rc = 32554 i = 167743728 end_mode = 2097201 env = 0x1fd9d40 __func__ = "mdb_txn_commit" (gdb) p *txn $1 = {mt_parent = 0x7f29f8041aeb, mt_child = 0x0, mt_next_pgno = 255, mt_txnid = 47589, mt_env = 0x1fd9d40, mt_free_pgs = 0x25fa538, mt_loose_pgs = 0x0, mt_loose_count = 0, mt_spill_pgs = 0x0, mt_u = { dirty_list = 0x26fa540, reader = 0x26fa540}, mt_dbxs = 0x1fd9e40, mt_dbs = 0xf9be28, mt_dbiseqs = 0xf9be98, mt_cursors = 0xf9be88, mt_dbflags = 0xf9bea0 "\b\030", mt_numdbs = 2, mt_flags = 524288, mt_dirty_room = 131067} (gdb) p *parent $2 = {mt_parent = 0x9b99d10000000000, mt_child = 0x8d000007f2e0a, mt_next_pgno = 3276218095827364344, mt_txnid = 7814437141411426150, mt_env = 0x7c6465646f636e65, mt_free_pgs = 0x7069746c756d7c20, mt_loose_pgs = 0x6d726f662f747261, mt_loose_count = 1952539693, mt_spill_pgs = 0x72617069746c756d, mt_u = {dirty_list = 0x6574616c65722f74, reader = 0x6574616c65722f74}, mt_dbxs = 0x747865747c207c64, mt_dbs = 0x617c207c6c6d782f, mt_dbiseqs = 0x69746163696c7070, mt_cursors = 0x207c6c6d782f6e6f, mt_dbflags = 0x6163696c7070617c <error: Cannot access memory at address 0x6163696c7070617c>, mt_numdbs = 1852795252, mt_flags = 1634693935, mt_dirty_room = 1836591984} (gdb)
Does your code actually use nested transactions?
I don't know, the code is from modsecurity: https://github.com/SpiderLabs/ModSecurity
https://bugs.openldap.org/show_bug.cgi?id=9626
--- Comment #4 from Howard Chu hyc@openldap.org --- (In reply to carlos.velasco from comment #3)
(In reply to Howard Chu from comment #2)
(In reply to carlos.velasco from comment #1)
It seems ids get corrupted.
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 175 if (ids[0] + app[0] >= ids[-1]) { [Current thread is 1 (Thread 0x7f2a09ffb640 (LWP 204177))] (gdb) bt #0 0x00007f2a32a4109f in mdb_midl_append_list (idp=0x7f29f8041b13, app=0x25fa538) at midl.c:175 #1 0x00007f2a32a325bf in mdb_txn_commit (txn=0xf9bda0) at mdb.c:3485
In frame 1 can you get the output for: info locals print *txn print *parent
(gdb) f 1 #1 0x00007f2a32a325bf in mdb_txn_commit (txn=0xf9bda0) at mdb.c:3485 3485 rc = mdb_midl_append_list(&parent->mt_free_pgs, txn->mt_free_pgs); (gdb) info locals parent = 0x7f29f8041aeb lp = 0x7f2a09ff9120 dst = 0x7f2a09ff90e0 pspill = 0x7f2a09ff90b0 y = 167743792 len = 0 src = 0x40009000d0006 x = 32554 ps_len = 32554 rc = 32554 i = 167743728 end_mode = 2097201 env = 0x1fd9d40 __func__ = "mdb_txn_commit" (gdb) p *txn $1 = {mt_parent = 0x7f29f8041aeb, mt_child = 0x0, mt_next_pgno = 255, mt_txnid = 47589, mt_env = 0x1fd9d40, mt_free_pgs = 0x25fa538, mt_loose_pgs = 0x0, mt_loose_count = 0, mt_spill_pgs = 0x0, mt_u = { dirty_list = 0x26fa540, reader = 0x26fa540}, mt_dbxs = 0x1fd9e40, mt_dbs = 0xf9be28, mt_dbiseqs = 0xf9be98, mt_cursors = 0xf9be88, mt_dbflags = 0xf9bea0 "\b\030", mt_numdbs = 2, mt_flags = 524288, mt_dirty_room = 131067} (gdb) p *parent $2 = {mt_parent = 0x9b99d10000000000, mt_child = 0x8d000007f2e0a, mt_next_pgno = 3276218095827364344, mt_txnid = 7814437141411426150, mt_env = 0x7c6465646f636e65, mt_free_pgs = 0x7069746c756d7c20, mt_loose_pgs = 0x6d726f662f747261, mt_loose_count = 1952539693, mt_spill_pgs = 0x72617069746c756d, mt_u = {dirty_list = 0x6574616c65722f74, reader = 0x6574616c65722f74}, mt_dbxs = 0x747865747c207c64, mt_dbs = 0x617c207c6c6d782f, mt_dbiseqs = 0x69746163696c7070, mt_cursors = 0x207c6c6d782f6e6f, mt_dbflags = 0x6163696c7070617c <error: Cannot access memory at address 0x6163696c7070617c>, mt_numdbs = 1852795252, mt_flags = 1634693935, mt_dirty_room = 1836591984} (gdb)
Does your code actually use nested transactions?
I don't know, the code is from modsecurity: https://github.com/SpiderLabs/ModSecurity
The txn->mt_parent pointer is clearly invalid. Most likely some other code overwrote it, probably in ModSecurity. This is extremely unlikely to be an actual LMDB bug.
https://bugs.openldap.org/show_bug.cgi?id=9626
--- Comment #5 from carlos.velasco@nimastelecom.com ---
The txn->mt_parent pointer is clearly invalid. Most likely some other code overwrote it, probably in ModSecurity. This is extremely unlikely to be an actual LMDB bug.
Ok. I will report it there and reference this case number. Thank you very much for your prompt support.
https://bugs.openldap.org/show_bug.cgi?id=9626
Howard Chu hyc@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |INVALID Status|UNCONFIRMED |RESOLVED
--- Comment #6 from Howard Chu hyc@openldap.org --- (In reply to carlos.velasco from comment #5)
The txn->mt_parent pointer is clearly invalid. Most likely some other code overwrote it, probably in ModSecurity. This is extremely unlikely to be an actual LMDB bug.
Ok. I will report it there and reference this case number. Thank you very much for your prompt support.
I have examined their code and reported a number of flaws in their code. https://github.com/SpiderLabs/ModSecurity/issues/2601
-
openldap-its＠openldap.org