forked from apache/doris
-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update flink-cdc-to-doris.md #1
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
个 -> 隔 那 -> 哪
HappenLee
pushed a commit
that referenced
this pull request
Jul 17, 2022
* [Schema Change] support fast add/drop column (apache#49) * [feature](schema-change) support fast schema change. coauthor: yixiutt * [schema change] Using columns desc from fe to read data. coauthor: Lchangliang * [feature](schema change) schema change optimize for add/drop columns. 1.add uniqueId field for class column. 2.schema change for add/drop columns directly update schema meta Co-authored-by: yixiutt <yixiu@selectdb.com> Co-authored-by: SWJTU-ZhangLei <1091517373@qq.com> [Feature](schema change) fix write and add regression test (apache#69) Co-authored-by: yixiutt <yixiu@selectdb.com> [schema change] be ssupport that delete use newest schema add delete regression test fix regression case (apache#107) tmp [feature](schema change) light schema change exclude rollup and agg/uniq/dup key type. [feature](schema change) fe olapTable maxUniqueId write in disk. [feature](schema change) add rpc iface for sc add column. [feature](schema change) add columnsDesc to TPushReq for ligtht sc. resolve the deadlock when schema change (apache#124) fix columns from fe don't has bitmap_index flag (apache#134) add update/delete case construct MATERIALIZED schema from origin schema when insert fix not vectorized compaction coredump use segment cache choose newest schema by schema version when compaction (apache#182) [bugfix](schema change) fix ligth schema change problem. [feature](schema change) light schema change add alter job. (#1) fix be ut [bug] (schema change) unique drop key column should not light schema change [feature](schema change) add schema change regression-test. fix regression test [bugfix](schema change) fix multi alter clauses for light schema change. (#2) [bugfix](schema change) fix multi clauses calculate column unique id (#3) modify PushTask process (apache#217) [Bugfix](schema change) fix jobId replay cause bdbje exception. [bug](schema change) fix max col unique id repeatitive. (apache#232) [optimize](schema change) modify pendingMaxColUniqueId generate rule. fix compaction error * fix be ut * fix snapshot load core fix unique_id error (apache#278) [refact](fe) remove redundant code for light schema change. (#4) [refact](fe) remove redundant code for light schema change. (#4) format fe core format be core fix be ut modify fe meta version fix rebase error flush schema into rowset_meta in old table [refactor](schema change) refact fe light schema change. (apache#5) delete the change of schemahash and support get max version schema * modify for review * fix be ut * fix schema change test
HappenLee
pushed a commit
that referenced
this pull request
Sep 7, 2022
Currently, explain string print all expression as slot id, e.g. `<slot 1>`. This PR, print its name with slot id instead, e.g. `column_a[#1]`. For details: - print qualified table name for OlapScanNode - print NamedExpression name with SlotId instead of just SlotId - OlapScanNode's node name use "OlapScanNode" instead of table name
HappenLee
pushed a commit
that referenced
this pull request
Mar 6, 2023
… convert function (apache#17032) sql select convert('a' using utf8) cause coredump of BE: Thread 338 "FragmentMgrThre" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f0b2c21a700 (LWP 2865259)] 0x00007fff04f8a015 in __gconv_close () from /lib64/libc.so.6 (gdb) bt #0 0x00007fff04f8a015 in __gconv_close () from /lib64/libc.so.6 #1 0x00007fff04f899e3 in iconv_close () from /lib64/libc.so.6 #2 0x000055556f92f5a7 in doris::vectorized::FunctionConvertTo::close (this=0x606003f28f50, context=0x6020039fe710, scope=doris_udf::FunctionContext::THREAD_LOCAL) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/functions/function_string.h:2442 #3 0x000055556db0598d in doris::vectorized::DefaultFunction::close (this=0x607000748f00, context=0x6020039fe710, scope=doris_udf::FunctionContext::THREAD_LOCAL) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/functions/function.h:524 #4 0x000055556da00b66 in doris::vectorized::VExpr::close_function_context (this=0x619004ca1d80, context=0x606003fad300, scope=doris_udf::FunctionContext::FRAGMENT_LOCAL, function=...) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exprs/vexpr.cpp:373 apache#5 0x000055556da16455 in doris::vectorized::VectorizedFnCall::close (this=0x619004ca1d80, state=0x61c00026c880, context=0x606003fad300, scope=doris_udf::FunctionContext::FRAGMENT_LOCAL) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exprs/vectorized_fn_call.cpp:92 apache#6 0x000055556da26afd in doris::vectorized::VExprContext::close (this=0x606003fad300, state=0x61c00026c880) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exprs/vexpr_context.cpp:72 apache#7 0x000055556d9fcff1 in doris::vectorized::VExpr::close (ctxs=..., state=0x61c00026c880) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exprs/vexpr.cpp:242 apache#8 0x000055556aa32c9a in doris::vectorized::VSortExecExprs::close (this=0x618000345f00, state=0x61c00026c880) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/common/sort/vsort_exec_exprs.cpp:77 apache#9 0x000055556aa26dae in doris::vectorized::VSortNode::close (this=0x618000345c80, state=0x61c00026c880) at /mnt/disk2/tengjianping/doris-1.2/be/src/vec/exec/vsort_node.cpp:129 apache#10 0x000055556794b5ef in doris::PlanFragmentExecutor::close (this=0x6170005a35f0) at /mnt/disk2/tengjianping/doris-1.2/be/src/runtime/plan_fragment_executor.cpp:676 apache#11 0x00005555678ce1fb in doris::FragmentExecState::execute (this=0x6170005a3580) at /mnt/disk2/tengjianping/doris-1.2/be/src/runtime/fragment_mgr.cpp:258 apache#12 0x00005555678d6633 in doris::FragmentMgr::_exec_actual(std::shared_ptr<doris::FragmentExecState>, std::function<void (doris::PlanFragmentExecutor*)>) (this=0x627000003900, exec_state=..., cb=...) at /mnt/disk2/tengjianping/doris-1.2/be/src/runtime/fragment_mgr.cpp:497 apache#13 0x00005555678d8aa3 in operator() (__closure=0x60800014cda0) at /mnt/disk2/tengjianping/doris-1.2/be/src/runtime/fragment_mgr.cpp:721
HappenLee
pushed a commit
that referenced
this pull request
Mar 6, 2023
… memory apache#17189 #0 doris::MemTracker::consumption (this=0x0) at /root/doris/be/src/runtime/memory/mem_tracker.h:65 #1 doris::PushHandler::_convert_v2 (this=this@entry=0x7ffec1e17f90, cur_tablet=..., cur_rowset=cur_rowset@entry=0x7ffec1e177a0, tablet_schema=...) at /root/doris/be/src/olap/push_handler.cpp:247 #2 0x000055555a271bc6 in doris::PushHandler::_do_streaming_ingestion (this=this@entry=0x7ffec1e17f90, tablet=..., request=..., push_type=push_type@entry=doris::PUSH_NORMAL_V2, tablet_info_vec=tablet_info_vec@entry=0x7ffec1e185d0) at /var/local/ldb-toolchain/include/c++/11/ext/atomicity.h:109 #3 0x000055555a27222c in doris::PushHandler::process_streaming_ingestion
HappenLee
pushed a commit
that referenced
this pull request
Mar 22, 2023
…d to nullable within a block (apache#17944) 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so 2# JVM_handle_linux_signal in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so 3# signalHandler(int, siginfo*, void*) in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so 4# 0x00007F4051C9F400 in /lib64/libc.so.6 5# memcpy at /root/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219 6# doris::vectorized::ColumnString::deserialize_and_insert_from_arena(char const*) at /root/doris/be/src/vec/columns/column_string.cpp:226 7# doris::vectorized::ColumnString::deserialize_vec_with_null_map(std::vector<StringRef, std::allocator<StringRef> >&, unsigned long, unsigned char const*) at /root/doris/be/src/vec/columns/column_string.cpp:283 8# void doris::vectorized::AggregationNode::_serialize_with_serialized_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*)::{lambda(auto:1&&)#1}::operator()<doris::vectorized::AggregationMethodSerialized<PHHashMap<StringRef, char*, DefaultHash<StringRef, void>, false> >&>(doris::vectorized::AggregationMethodSerialized<PHHashMap<StringRef, char*, DefaultHash<StringRef, void>, false> >&) const at /root/doris/be/src/vec/exec/vaggregation_node.cpp:1232 9# doris::vectorized::AggregationNode::_serialize_with_serialized_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/vaggregation_node.cpp:1294 10# std::_Function_handler<doris::Status (doris::RuntimeState*, doris::vectorized::Block*, bool*), std::_Bind_result<doris::Status, doris::Status (doris::vectorized::AggregationNode::*(doris::vectorized::AggregationNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)> >::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:293 11# doris::vectorized::AggregationNode::get_next(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/vaggregation_node.cpp:508 12# doris::ExecNode::get_next_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/exec/exec_node.cpp:852 13# doris::PlanFragmentExecutor::get_vectorized_internal(doris::vectorized::Block**) at /root/doris/be/src/runtime/plan_fragment_executor.cpp:352 14# doris::PlanFragmentExecutor::open_vectorized_internal() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:300 15# doris::PlanFragmentExecutor::open() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:253 16# doris::FragmentExecState::execute() at /root/doris/be/src/runtime/fragment_mgr.cpp:251 17# doris::FragmentMgr::_exec_actual(std::shared_ptr<doris::FragmentExecState>, std::function<void (doris::PlanFragmentExecutor*)>) at /root/doris/be/src/runtime/fragment_mgr.cpp:498 18# std::_Function_handler<void (), doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::PlanFragmentExecutor*)>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291 19# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:542 20# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:455 21# start_thread in /lib64/libpthread.so.0 22# clone in /lib64/libc.so.6
HappenLee
pushed a commit
that referenced
this pull request
Mar 22, 2023
… order by keys (apache#17913) if the order by keys are not simple slot in sort node, the order by exprs have to been added to sort node's output tuple. In that case, we need add a project node above sort node to eliminate the unused order by exprs. for example: ```sql WITH t0 AS (SELECT DATE_FORMAT(date, '%Y%m%d') AS date FROM cir_1756_t1 ), t3 AS (SELECT date_format(date, '%Y%m%d') AS `date` FROM `cir_1756_t2` GROUP BY date_format(date, '%Y%m%d') **ORDER BY date_format(date, '%Y%m%d')** ) SELECT t0.date FROM t0 LEFT JOIN t3 ON t0.date = t3.date; ``` before: ``` +--------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +--------------------------------------------------------------------------------------------------------------------------------------------------+ | LogicalProject[159] ( distinct=false, projects=[date#1], excepts=[], canEliminate=true ) | | +--LogicalJoin[158] ( type=LEFT_OUTER_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(date#1 = date#3)], otherJoinConjuncts=[] ) | | |--LogicalProject[151] ( distinct=false, projects=[date_format(date#0, '%Y%m%d') AS `date`#1], excepts=[], canEliminate=true ) | | | +--LogicalOlapScan ( qualified=default_cluster:bugfix.cir_1756_t1, indexName=cir_1756_t1, selectedIndexId=412339, preAgg=ON ) | | +--LogicalSort[157] ( orderKeys=[date_format(cast(date#3 as DATETIME), '%Y%m%d') asc null first] ) | | +--LogicalAggregate[156] ( groupByExpr=[date#3], outputExpr=[date#3], hasRepeat=false ) | | +--LogicalProject[155] ( distinct=false, projects=[date_format(date#2, '%Y%m%d') AS `date`#3], excepts=[], canEliminate=true ) | | +--LogicalOlapScan ( qualified=default_cluster:bugfix.cir_1756_t2, indexName=cir_1756_t2, selectedIndexId=412352, preAgg=ON ) | +--------------------------------------------------------------------------------------------------------------------------------------------------+ ``` after: ``` +--------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +--------------------------------------------------------------------------------------------------------------------------------------------------+ | LogicalProject[171] ( distinct=false, projects=[date#2], excepts=[], canEliminate=true ) | | +--LogicalJoin[170] ( type=LEFT_OUTER_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(date#2 = date#4)], otherJoinConjuncts=[] ) | | |--LogicalProject[162] ( distinct=false, projects=[date_format(date#0, '%Y%m%d') AS `date`#2], excepts=[], canEliminate=true ) | | | +--LogicalOlapScan ( qualified=default_cluster:bugfix.cir_1756_t1, indexName=cir_1756_t1, selectedIndexId=1049812, preAgg=ON ) | | +--LogicalProject[169] ( distinct=false, projects=[date#4], excepts=[], canEliminate=false ) | | +--LogicalSort[168] ( orderKeys=[date_format(cast(date#4 as DATETIME), '%Y%m%d') asc null first] ) | | +--LogicalAggregate[167] ( groupByExpr=[date#4], outputExpr=[date#4], hasRepeat=false ) | | +--LogicalProject[166] ( distinct=false, projects=[date_format(date#3, '%Y%m%d') AS `date`#4], excepts=[], canEliminate=true ) | | +--LogicalOlapScan ( qualified=default_cluster:bugfix.cir_1756_t2, indexName=cir_1756_t2, selectedIndexId=1049825, preAgg=ON ) | +--------------------------------------------------------------------------------------------------------------------------------------------------+ ```
HappenLee
pushed a commit
that referenced
this pull request
Apr 24, 2023
…d to nullable within a block (apache#17944) 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so 2# JVM_handle_linux_signal in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so 3# signalHandler(int, siginfo*, void*) in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so 4# 0x00007F4051C9F400 in /lib64/libc.so.6 5# memcpy at /root/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219 6# doris::vectorized::ColumnString::deserialize_and_insert_from_arena(char const*) at /root/doris/be/src/vec/columns/column_string.cpp:226 7# doris::vectorized::ColumnString::deserialize_vec_with_null_map(std::vector<StringRef, std::allocator<StringRef> >&, unsigned long, unsigned char const*) at /root/doris/be/src/vec/columns/column_string.cpp:283 8# void doris::vectorized::AggregationNode::_serialize_with_serialized_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*)::{lambda(auto:1&&)#1}::operator()<doris::vectorized::AggregationMethodSerialized<PHHashMap<StringRef, char*, DefaultHash<StringRef, void>, false> >&>(doris::vectorized::AggregationMethodSerialized<PHHashMap<StringRef, char*, DefaultHash<StringRef, void>, false> >&) const at /root/doris/be/src/vec/exec/vaggregation_node.cpp:1232 9# doris::vectorized::AggregationNode::_serialize_with_serialized_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/vaggregation_node.cpp:1294 10# std::_Function_handler<doris::Status (doris::RuntimeState*, doris::vectorized::Block*, bool*), std::_Bind_result<doris::Status, doris::Status (doris::vectorized::AggregationNode::*(doris::vectorized::AggregationNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)> >::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:293 11# doris::vectorized::AggregationNode::get_next(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/vaggregation_node.cpp:508 12# doris::ExecNode::get_next_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/exec/exec_node.cpp:852 13# doris::PlanFragmentExecutor::get_vectorized_internal(doris::vectorized::Block**) at /root/doris/be/src/runtime/plan_fragment_executor.cpp:352 14# doris::PlanFragmentExecutor::open_vectorized_internal() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:300 15# doris::PlanFragmentExecutor::open() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:253 16# doris::FragmentExecState::execute() at /root/doris/be/src/runtime/fragment_mgr.cpp:251 17# doris::FragmentMgr::_exec_actual(std::shared_ptr<doris::FragmentExecState>, std::function<void (doris::PlanFragmentExecutor*)>) at /root/doris/be/src/runtime/fragment_mgr.cpp:498 18# std::_Function_handler<void (), doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::PlanFragmentExecutor*)>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291 19# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:542 20# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:455 21# start_thread in /lib64/libpthread.so.0 22# clone in /lib64/libc.so.6
HappenLee
pushed a commit
that referenced
this pull request
Jun 1, 2023
…ugh sort (apache#20161) Support the rewrite rule for push-down filter through sort. We can directly push-down the filter through sort without any conditions check. Before this PR: ``` mysql> explain select * from (select * from t1 order by a) t2 where t2.b > 2; +-------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#2] | | b[#3] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 3:VSELECT | | | predicates: b[#3] > 2 | | | | | 2:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: a[#0] | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | UNPARTITIONED | | | | 1:VTOP-N | | | order by: a[#2] ASC | | | offset: 0 | | | | | 0:VOlapScanNode | | TABLE: default_cluster:test.t1(t1), PREAGGREGATION: ON | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | +-------------------------------------------------------------+ 30 rows in set (0.06 sec) ``` After this PR: ``` mysql> explain select * from (select * from t1 order by a) t2 where t2.b > 2; +-------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#2] | | b[#3] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 2:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: a[#0] | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | UNPARTITIONED | | | | 1:VTOP-N | | | order by: a[#2] ASC | | | offset: 0 | | | | | 0:VOlapScanNode | | TABLE: default_cluster:test.t1(t1), PREAGGREGATION: ON | | PREDICATES: b[#1] > 2 | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | +-------------------------------------------------------------+ 28 rows in set (0.40 sec) ```
HappenLee
pushed a commit
that referenced
this pull request
Jun 12, 2023
…scan node _conjuncts (apache#20534) =3073084==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60601897db80 at pc 0x55b2c993666e bp 0x7d1fbbfb66b0 sp 0x7d1fbbfb66a8 READ of size 8 at 0x60601897db80 thread T610 (_scanner_scan) #0 0x55b2c993666d in std::__shared_ptr<doris::vectorized::VExprContext, (__gnu_cxx::_Lock_policy)2>::get() const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291:16 #1 0x55b2dae86ec5 in doris::vectorized::VExprContext::clone(doris::RuntimeState*, std::shared_ptr<doris::vectorized::VExprContext>&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exprs/vexpr_context.cpp:98:5 #2 0x55b2e757b6d8 in doris::vectorized::VScanner::prepare(doris::RuntimeState*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>> const&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/vscanner.cpp:47:13 #3 0x55b2e78e8155 in doris::vectorized::NewOlapScanner::init() /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/new_olap_scanner.cpp:109:5 #4 0x55b2e7551c81 in doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/scanner_scheduler.cpp:279:27 apache#5 0x55b2e7554d5e in doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()::operator()() const /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/scanner_scheduler.cpp:202:31 apache#6 0x55b2e7554c14 in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#7 0x55b2e7554bb4 in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>(doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#8 0x55b2e7554a1c in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()>::_M_invoke(std::_Any_data const&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#9 0x55b2c80f2cd2 in std::function<void ()>::operator()() const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#10 0x55b2e755f3e4 in doris::PriorityWorkStealingThreadPool::work_thread(int) /mnt/disk2/tengjianping/doris-master/be/src/util/priority_work_stealing_thread_pool.hpp:135:17 apache#11 0x55b2e7563c72 in void std::__invoke_impl<void, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_memfun_deref, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 apache#12 0x55b2e7563b44 in std::__invoke_result<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>::type std::__invoke<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#13 0x55b2e7563b14 in decltype(std::__invoke((*this)._M_pmf, std::forward<doris::PriorityWorkStealingThreadPool*&>(fp), std::forward<int&>(fp))) std::_Mem_fn_base<void (doris::PriorityWorkStealingThreadPool::*)(int), true>::operator()<doris::PriorityWorkStealingThreadPool*&, int&>(doris::PriorityWorkStealingThreadPool*&, int&) const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:131:11 apache#14 0x55b2e7563ae4 in void std::__invoke_impl<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_other, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#15 0x55b2e7563a54 in std::enable_if<is_invocable_r_v<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>, void>::type std::__invoke_r<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#16 0x55b2e75639c3 in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::__call<void, 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570:11 apache#17 0x55b2e756382d in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::operator()<>() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:629:17 apache#18 0x55b2e7563744 in void std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#19 0x55b2e7563704 in std::__invoke_result<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>::type std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#20 0x55b2e75636dc in void std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253:13 apache#21 0x55b2e75636b4 in std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::operator()() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260:11 apache#22 0x55b2e7563638 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>>::_M_run() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 apache#23 0x55b2eb41d0ef in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18 apache#24 0x7f1dfd4e1179 in start_thread pthread_create.c apache#25 0x7f1dfdd7bdf2 in clone (/lib64/libc.so.6+0xfcdf2) (BuildId: 20ee73ce1b6ac38a52440bab82ec7e28f0f5c5b9)
HappenLee
pushed a commit
that referenced
this pull request
Jun 30, 2023
…1350) For sql like select cast("627492340" as datetime); the string is an invalid datetime, function DateV2Value<T>::from_date_str cast it as datetime 2062-74-92 23:40:00, with an out-of-range month and day value, which cause memory violation in function DateV2Value<T>::format_datetime when trying to access s_days_in_month. ==256444==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55a7c1a5cff8 at pc 0x55a7e5aa3d2a bp 0x7f3b805f0370 sp 0x7f3b805f0368 READ of size 4 at 0x55a7c1a5cff8 thread T390 (FragmentMgrThre) #0 0x55a7e5aa3d29 in doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>::format_datetime(unsigned int*, bool*) const /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdatetime_value.cpp:1821:31 #1 0x55a7e5aa3052 in doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>::from_date_str(char const*, int, int) /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdatetime_value.cpp:1968:5 #2 0x55a7d48f0c49 in bool doris::vectorized::read_datetime_v2_text_impl<unsigned long>(unsigned long&, doris::vectorized::ReadBuffer&, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/io/io_helper.h:309:19 #3 0x55a7ddb21642 in bool doris::vectorized::try_read_datetime_v2_text<unsigned long>(unsigned long&, doris::vectorized::ReadBuffer&, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/io/io_helper.h:409:12 #4 0x55a7ddb215ec in bool doris::vectorized::try_parse_impl<doris::vectorized::DataTypeDateTimeV2, unsigned int, void*>(doris::vectorized::DataTypeDateTimeV2::FieldType&, doris::vectorized::ReadBuffer&, DateLUTImpl const*, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:839:16 apache#5 0x55a7ddb21c84 in auto doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)::operator()<std::integral_constant<bool, false>, std::integral_constant<bool, true>>(void*, auto) const /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1340:38 apache#6 0x55a7ddb216f7 in void* std::__invoke_impl<doris::Status, doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(std::__invoke_other, auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#7 0x55a7ddb2167f in std::__invoke_result<void*, std::integral_constant<bool, false>, std::integral_constant<bool, true>>::type std::__invoke<doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(void*&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#8 0x55a7ddb20d14 in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<doris::Status> (*)(doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&)>, std::integer_sequence<unsigned long, 0ul, 1ul>>::__visit_invoke(doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013:11 apache#9 0x55a7ddb20c15 in decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<doris::Status>, doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1714:14 apache#10 0x55a7ddb20b6a in decltype(auto) std::visit<doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(void*&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1769:9 apache#11 0x55a7ddb205ff in doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1321:23 apache#12 0x55a7ddb1f2c7 in doris::vectorized::FunctionConvertFromString<doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1417:20
HappenLee
pushed a commit
that referenced
this pull request
Jul 7, 2023
…tion (apache#21317) explain WITH cte_0 AS ( SELECT 1 AS a ) SELECT * from cte_0 t1 join cte_0 t2 on true WHERE false; before: ``` +----------------------------+ | Explain String | +----------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#1] | | a[#2] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 1:VEMPTYSET | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | a[#0] | | PARTITION: UNPARTITIONED | | | | MultiCastDataSinks | | | | 0:VUNION | | constant exprs: | | 1 | +----------------------------+ ``` after: ``` +----------------------------+ | Explain String | +----------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#0] | | a[#1] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 0:VEMPTYSET | +----------------------------+ ```
HappenLee
pushed a commit
that referenced
this pull request
Jul 11, 2023
…1350) For sql like select cast("627492340" as datetime); the string is an invalid datetime, function DateV2Value<T>::from_date_str cast it as datetime 2062-74-92 23:40:00, with an out-of-range month and day value, which cause memory violation in function DateV2Value<T>::format_datetime when trying to access s_days_in_month. ==256444==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55a7c1a5cff8 at pc 0x55a7e5aa3d2a bp 0x7f3b805f0370 sp 0x7f3b805f0368 READ of size 4 at 0x55a7c1a5cff8 thread T390 (FragmentMgrThre) #0 0x55a7e5aa3d29 in doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>::format_datetime(unsigned int*, bool*) const /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdatetime_value.cpp:1821:31 #1 0x55a7e5aa3052 in doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>::from_date_str(char const*, int, int) /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdatetime_value.cpp:1968:5 #2 0x55a7d48f0c49 in bool doris::vectorized::read_datetime_v2_text_impl<unsigned long>(unsigned long&, doris::vectorized::ReadBuffer&, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/io/io_helper.h:309:19 #3 0x55a7ddb21642 in bool doris::vectorized::try_read_datetime_v2_text<unsigned long>(unsigned long&, doris::vectorized::ReadBuffer&, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/io/io_helper.h:409:12 #4 0x55a7ddb215ec in bool doris::vectorized::try_parse_impl<doris::vectorized::DataTypeDateTimeV2, unsigned int, void*>(doris::vectorized::DataTypeDateTimeV2::FieldType&, doris::vectorized::ReadBuffer&, DateLUTImpl const*, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:839:16 apache#5 0x55a7ddb21c84 in auto doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)::operator()<std::integral_constant<bool, false>, std::integral_constant<bool, true>>(void*, auto) const /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1340:38 apache#6 0x55a7ddb216f7 in void* std::__invoke_impl<doris::Status, doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(std::__invoke_other, auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#7 0x55a7ddb2167f in std::__invoke_result<void*, std::integral_constant<bool, false>, std::integral_constant<bool, true>>::type std::__invoke<doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(void*&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#8 0x55a7ddb20d14 in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<doris::Status> (*)(doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&)>, std::integer_sequence<unsigned long, 0ul, 1ul>>::__visit_invoke(doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013:11 apache#9 0x55a7ddb20c15 in decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<doris::Status>, doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1714:14 apache#10 0x55a7ddb20b6a in decltype(auto) std::visit<doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*)::'lambda'(void*, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(void*&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1769:9 apache#11 0x55a7ddb205ff in doris::Status doris::vectorized::ConvertThroughParsing<doris::vectorized::DataTypeString, doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute<void*>(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long, bool, void*) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1321:23 apache#12 0x55a7ddb1f2c7 in doris::vectorized::FunctionConvertFromString<doris::vectorized::DataTypeDateTimeV2, doris::vectorized::NameCast>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long>> const&, unsigned long, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/functions/function_cast.h:1417:20
HappenLee
pushed a commit
that referenced
this pull request
Jul 11, 2023
…tion (apache#21317) explain WITH cte_0 AS ( SELECT 1 AS a ) SELECT * from cte_0 t1 join cte_0 t2 on true WHERE false; before: ``` +----------------------------+ | Explain String | +----------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#1] | | a[#2] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 1:VEMPTYSET | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | a[#0] | | PARTITION: UNPARTITIONED | | | | MultiCastDataSinks | | | | 0:VUNION | | constant exprs: | | 1 | +----------------------------+ ``` after: ``` +----------------------------+ | Explain String | +----------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#0] | | a[#1] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 0:VEMPTYSET | +----------------------------+ ```
HappenLee
pushed a commit
that referenced
this pull request
Jul 20, 2023
fix: apache#21136 mem tracker group uses class static variables instead of global variables https://stackoverflow.com/questions/2204608/does-c-call-destructors-for-global-and-class-static-variables TODO: A mem tracker manager is required, don't use global variables, it will sad ==3623982==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0000056b8 at pc 0x56478bbe3ae0 bp 0x7f20953d2270 sp 0x7f20953d2268 READ of size 8 at 0x60f0000056b8 thread T41 (memory_tracker_) *** Query id: 0-0 *** *** Aborted at 1689749969 (unix time) try "date -d @1689749969" if you are using GNU date *** *** Current BE git commitID: b3e9cad *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3623982 (TID 3624277 OR 0x7f19e06dd640) from PID 0; stack trace: *** #0 0x56478bbe3adf in std::__shared_ptr::operator bool() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1295:16 #1 0x56478bbe306e in doris::MemTracker::refresh_profile_counter() /doris/be/src/runtime/memory/mem_tracker.h:149:13 #2 0x56478bbec669 in doris::MemTrackerLimiter::refresh_all_tracker_profile() /doris/be/src/runtime/memory/mem_tracker_limiter.cpp:119:22 #3 0x564788f53fa0 in doris::Daemon::memory_tracker_profile_refresh_thread() /doris/be/src/common/daemon.cpp:295:9 #4 0x564788f5d04b in doris::Daemon::start()::$_4::operator()() const /doris/be/src/common/daemon.cpp:473:30 apache#5 0x564788f5cff6 in void std::__invoke_impl(std::__invoke_other, doris::Daemon::start()::$_4&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#6 0x564788f5cf78 in std::enable_if, void>::type std::__invoke_r(doris::Daemon::start()::$_4&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#7 0x564788f5cdae in std::_Function_handler::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#8 0x56478903f576 in std::function::operator()() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#9 0x56478c4a35af in doris::Thread::supervise_thread(void*) /doris/be/src/util/thread.cpp:465:5 apache#10 0x7f217c8a244f in start_thread nptl/pthread_create.c:473:8 apache#11 0x7f217cb27d52 in __clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x60f0000056b8 is located 56 bytes inside of 168-byte region [0x60f000005680,0x60f000005728) freed by thread T0 here: #0 0x564788e7280d in operator delete(void*) (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x1758280d) (BuildId: 219493cc924323ee) #1 0x56478acec1d5 in std::default_delete::operator()(doris::MemTrackerLimiter*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #2 0x56478ace9faf in std::unique_ptr >::~unique_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 #3 0x56478ace1471 in doris::ShardedLRUCache::~ShardedLRUCache() /doris/be/src/olap/lru_cache.cpp:581:1 #4 0x56478ace14c8 in doris::ShardedLRUCache::~ShardedLRUCache() /doris/be/src/olap/lru_cache.cpp:572:37 apache#5 0x56478acd0984 in std::default_delete::operator()(doris::Cache*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 apache#6 0x56478acceddf in std::unique_ptr >::~unique_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 apache#7 0x56478ad96dc6 in doris::StoragePageCache::~StoragePageCache() /doris/be/src/olap/page_cache.h:78:7 apache#8 0x7f217ca54146 in __run_exit_handlers stdlib/exit.c:108:8 previously allocated by thread T0 here: #0 0x564788e71fad in operator new(unsigned long) (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x17581fad) (BuildId: 219493cc924323ee) #1 0x56478ace9c90 in std::_MakeUniq::__single_object std::make_unique, std::allocator > const&>(doris::MemTrackerLimiter::Type&&, std::__cxx11::basic_string, std::allocator > const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:30 #2 0x56478acde930 in doris::ShardedLRUCache::ShardedLRUCache(std::__cxx11::basic_string, std::allocator > const&, unsigned long, doris::LRUCacheType, unsigned int, unsigned int) /doris/be/src/olap/lru_cache.cpp:526:20 #3 0x56478ace22e1 in doris::new_lru_cache(std::__cxx11::basic_string, std::allocator > const&, unsigned long, doris::LRUCacheType, unsigned int) /doris/be/src/olap/lru_cache.cpp:670:16 #4 0x56478ad91da2 in doris::StoragePageCache::StoragePageCache(unsigned long, int, long, unsigned int) /doris/be/src/olap/page_cache.cpp:47:17 apache#5 0x56478ad9156e in doris::StoragePageCache::create_global_cache(unsigned long, int, long, unsigned int) /doris/be/src/olap/page_cache.cpp:31:29 apache#6 0x56478b98b3d3 in doris::ExecEnv::_init_mem_env() /doris/be/src/runtime/exec_env_init.cpp:251:5 apache#7 0x56478b98946c in doris::ExecEnv::_init(std::vector > const&) /doris/be/src/runtime/exec_env_init.cpp:182:5 apache#8 0x56478b987139 in doris::ExecEnv::init(doris::ExecEnv*, std::vector > const&) /doris/be/src/runtime/exec_env_init.cpp:98:17 apache#9 0x564788e79b50 in main /doris/be/src/service/doris_main.cpp:429:5 apache#10 0x7f217ca38564 in __libc_start_main csu/../csu/libc-start.c:332:16 Thread T41 (memory_tracker_) created by T0 here: #0 0x564788e1fcaa in pthread_create (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x1752fcaa) (BuildId: 219493cc924323ee) #1 0x56478c4a2366 in doris::Thread::start_thread(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::function const&, unsigned long, scoped_refptr*) /doris/be/src/util/thread.cpp:419:15 #2 0x564788f59b91 in doris::Status doris::Thread::create(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, doris::Daemon::start()::$_4 const&, scoped_refptr*) /doris/be/src/util/thread.h:50:16 #3 0x564788f58165 in doris::Daemon::start() /doris/be/src/common/daemon.cpp:471:10 #4 0x564788e79a96 in main /doris/be/src/service/doris_main.cpp:420:12 apache#5 0x7f217ca38564 in __libc_start_main csu/../csu/libc-start.c:332:16
HappenLee
pushed a commit
that referenced
this pull request
Jul 28, 2023
fix: apache#21136 mem tracker group uses class static variables instead of global variables https://stackoverflow.com/questions/2204608/does-c-call-destructors-for-global-and-class-static-variables TODO: A mem tracker manager is required, don't use global variables, it will sad ==3623982==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0000056b8 at pc 0x56478bbe3ae0 bp 0x7f20953d2270 sp 0x7f20953d2268 READ of size 8 at 0x60f0000056b8 thread T41 (memory_tracker_) *** Query id: 0-0 *** *** Aborted at 1689749969 (unix time) try "date -d @1689749969" if you are using GNU date *** *** Current BE git commitID: b3e9cad *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3623982 (TID 3624277 OR 0x7f19e06dd640) from PID 0; stack trace: *** #0 0x56478bbe3adf in std::__shared_ptr::operator bool() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1295:16 #1 0x56478bbe306e in doris::MemTracker::refresh_profile_counter() /doris/be/src/runtime/memory/mem_tracker.h:149:13 #2 0x56478bbec669 in doris::MemTrackerLimiter::refresh_all_tracker_profile() /doris/be/src/runtime/memory/mem_tracker_limiter.cpp:119:22 #3 0x564788f53fa0 in doris::Daemon::memory_tracker_profile_refresh_thread() /doris/be/src/common/daemon.cpp:295:9 #4 0x564788f5d04b in doris::Daemon::start()::$_4::operator()() const /doris/be/src/common/daemon.cpp:473:30 apache#5 0x564788f5cff6 in void std::__invoke_impl(std::__invoke_other, doris::Daemon::start()::$_4&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#6 0x564788f5cf78 in std::enable_if, void>::type std::__invoke_r(doris::Daemon::start()::$_4&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#7 0x564788f5cdae in std::_Function_handler::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#8 0x56478903f576 in std::function::operator()() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#9 0x56478c4a35af in doris::Thread::supervise_thread(void*) /doris/be/src/util/thread.cpp:465:5 apache#10 0x7f217c8a244f in start_thread nptl/pthread_create.c:473:8 apache#11 0x7f217cb27d52 in __clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x60f0000056b8 is located 56 bytes inside of 168-byte region [0x60f000005680,0x60f000005728) freed by thread T0 here: #0 0x564788e7280d in operator delete(void*) (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x1758280d) (BuildId: 219493cc924323ee) #1 0x56478acec1d5 in std::default_delete::operator()(doris::MemTrackerLimiter*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #2 0x56478ace9faf in std::unique_ptr >::~unique_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 #3 0x56478ace1471 in doris::ShardedLRUCache::~ShardedLRUCache() /doris/be/src/olap/lru_cache.cpp:581:1 #4 0x56478ace14c8 in doris::ShardedLRUCache::~ShardedLRUCache() /doris/be/src/olap/lru_cache.cpp:572:37 apache#5 0x56478acd0984 in std::default_delete::operator()(doris::Cache*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 apache#6 0x56478acceddf in std::unique_ptr >::~unique_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 apache#7 0x56478ad96dc6 in doris::StoragePageCache::~StoragePageCache() /doris/be/src/olap/page_cache.h:78:7 apache#8 0x7f217ca54146 in __run_exit_handlers stdlib/exit.c:108:8 previously allocated by thread T0 here: #0 0x564788e71fad in operator new(unsigned long) (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x17581fad) (BuildId: 219493cc924323ee) #1 0x56478ace9c90 in std::_MakeUniq::__single_object std::make_unique, std::allocator > const&>(doris::MemTrackerLimiter::Type&&, std::__cxx11::basic_string, std::allocator > const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:30 #2 0x56478acde930 in doris::ShardedLRUCache::ShardedLRUCache(std::__cxx11::basic_string, std::allocator > const&, unsigned long, doris::LRUCacheType, unsigned int, unsigned int) /doris/be/src/olap/lru_cache.cpp:526:20 #3 0x56478ace22e1 in doris::new_lru_cache(std::__cxx11::basic_string, std::allocator > const&, unsigned long, doris::LRUCacheType, unsigned int) /doris/be/src/olap/lru_cache.cpp:670:16 #4 0x56478ad91da2 in doris::StoragePageCache::StoragePageCache(unsigned long, int, long, unsigned int) /doris/be/src/olap/page_cache.cpp:47:17 apache#5 0x56478ad9156e in doris::StoragePageCache::create_global_cache(unsigned long, int, long, unsigned int) /doris/be/src/olap/page_cache.cpp:31:29 apache#6 0x56478b98b3d3 in doris::ExecEnv::_init_mem_env() /doris/be/src/runtime/exec_env_init.cpp:251:5 apache#7 0x56478b98946c in doris::ExecEnv::_init(std::vector > const&) /doris/be/src/runtime/exec_env_init.cpp:182:5 apache#8 0x56478b987139 in doris::ExecEnv::init(doris::ExecEnv*, std::vector > const&) /doris/be/src/runtime/exec_env_init.cpp:98:17 apache#9 0x564788e79b50 in main /doris/be/src/service/doris_main.cpp:429:5 apache#10 0x7f217ca38564 in __libc_start_main csu/../csu/libc-start.c:332:16 Thread T41 (memory_tracker_) created by T0 here: #0 0x564788e1fcaa in pthread_create (/mnt/hdd01/dorisTestEnv/NEREIDS_ASAN/be/lib/doris_be+0x1752fcaa) (BuildId: 219493cc924323ee) #1 0x56478c4a2366 in doris::Thread::start_thread(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::function const&, unsigned long, scoped_refptr*) /doris/be/src/util/thread.cpp:419:15 #2 0x564788f59b91 in doris::Status doris::Thread::create(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, doris::Daemon::start()::$_4 const&, scoped_refptr*) /doris/be/src/util/thread.h:50:16 #3 0x564788f58165 in doris::Daemon::start() /doris/be/src/common/daemon.cpp:471:10 #4 0x564788e79a96 in main /doris/be/src/service/doris_main.cpp:420:12 apache#5 0x7f217ca38564 in __libc_start_main csu/../csu/libc-start.c:332:16
HappenLee
pushed a commit
that referenced
this pull request
Aug 28, 2023
consider sql "select * from test_simplify_in_predicate_t where a in ('1992-01-31', '1992-02-01', '1992-02-02', '1992-02-03', '1992-02-04');" before: ``` | 0:VOlapScanNode | | TABLE: default_cluster:bugfix.test_simplify_in_predicate_t(test_simplify_in_predicate_t), PREAGGREGATION: OFF. Reason: No aggregate on scan. | | PREDICATES: CAST(a[#0] AS DATETIMEV2(0)) IN ('1992-01-31 00:00:00', '1992-02-01 00:00:00', '1992-02-02 00:00:00', '1992-02-03 00:00:00', '1992-02-04 00:00:00') AND __DORIS_DELETE_SIGN__[#1] = 0 | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | projections: a[#0] | | project output tuple id: 1 | | tuple ids: 0 ``` after: ``` | 0:VOlapScanNode | | TABLE: default_cluster:bugfix.test_simplify_in_predicate_t(test_simplify_in_predicate_t), PREAGGREGATION: OFF. Reason: No aggregate on scan. | | PREDICATES: a[#0] IN ('1992-01-31', '1992-02-01', '1992-02-02', '1992-02-03', '1992-02-04') AND __DORIS_DELETE_SIGN__[#1] = 0 | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | projections: a[#0] | | project output tuple id: 1 | | tuple ids: 0 ```
HappenLee
pushed a commit
that referenced
this pull request
Aug 31, 2023
consider sql "select * from test_simplify_in_predicate_t where a in ('1992-01-31', '1992-02-01', '1992-02-02', '1992-02-03', '1992-02-04');" before: ``` | 0:VOlapScanNode | | TABLE: default_cluster:bugfix.test_simplify_in_predicate_t(test_simplify_in_predicate_t), PREAGGREGATION: OFF. Reason: No aggregate on scan. | | PREDICATES: CAST(a[#0] AS DATETIMEV2(0)) IN ('1992-01-31 00:00:00', '1992-02-01 00:00:00', '1992-02-02 00:00:00', '1992-02-03 00:00:00', '1992-02-04 00:00:00') AND __DORIS_DELETE_SIGN__[#1] = 0 | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | projections: a[#0] | | project output tuple id: 1 | | tuple ids: 0 ``` after: ``` | 0:VOlapScanNode | | TABLE: default_cluster:bugfix.test_simplify_in_predicate_t(test_simplify_in_predicate_t), PREAGGREGATION: OFF. Reason: No aggregate on scan. | | PREDICATES: a[#0] IN ('1992-01-31', '1992-02-01', '1992-02-02', '1992-02-03', '1992-02-04') AND __DORIS_DELETE_SIGN__[#1] = 0 | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | projections: a[#0] | | project output tuple id: 1 | | tuple ids: 0 ```
HappenLee
pushed a commit
that referenced
this pull request
Oct 25, 2023
Template arguments for decimalv3 type is wrong when creating MinMaxNumFunc, which result in wrong decimal type like vectorized::Decimal<doris::vectorized::Decimal128I> : class MinmaxFunctionTraits { public: using BasePtr = MinMaxFuncBase*; template <PrimitiveType type, size_t N> static BasePtr get_function() { return new MinMaxNumFunc<std::conditional_t< type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 || type == TYPE_DECIMAL128I, vectorized::Decimal<typename PrimitiveTypeTraits<type>::CppType>, typename PrimitiveTypeTraits<type>::CppType>>(); } }; (gdb) bt #0 doris::vectorized::Decimal<__int128>::operator<=>(doris::vectorized::Decimal<__int128> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 #1 doris::vectorized::Decimal<doris::vectorized::Decimal128I>::operator<=>(doris::vectorized::Decimal<doris::vectorized::Decimal128I> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 #2 doris::MinMaxNumFunc<doris::vectorized::Decimal<doris::vectorized::Decimal128I> >::insert (this=0x7ef19de903c0, data=0x7c612928d860) at /mnt/disk2/user/doris-1.1/be/src/exprs/minmax_predicate.h:61 #3 0x0000558465c2844a in doris::RuntimePredicateWrapper::insert (this=0x7f47a2e3d720, value=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:492 #4 doris::RuntimePredicateWrapper::insert_batch (this=0x7f47a2e3d720, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:504 apache#5 0x0000558465c1f6c3 in doris::IRuntimeFilter::insert_batch (this=<optimized out>, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:1060 apache#6 0x000055846883dafd in doris::VRuntimeFilterSlots::insert (this=0x7d7e767f4e70, datas=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter_slots.h:188 apache#7 0x0000558468840535 in doris::vectorized::ProcessRuntimeFilterBuild::operator()<doris::vectorized::MethodOneNumber<doris::vectorized::UInt128, PartitionedHashMapTable<HashMapTable<doris::vectorized::UInt128, HashMapCell<doris::vectorized::UInt128, doris::vectorized::RowRefList, HashCRC32<doris::vectorized::UInt128>, HashTableNoState>, HashCRC32<doris::vectorized::UInt128>, PartitionedHashTableGrower<8ul>, Allocator<true, true, false> > > >, doris::vectorized::HashJoinNode> ( this=this@entry=0x7e7c6a5bee40, state=0x7ef19df00400, hash_table_ctx=..., parent=0x7ef19dee3c00) at /mnt/disk2/user/doris-1.1/be/src/vec/exec/join/vhash_join_node.h:92
HappenLee
pushed a commit
that referenced
this pull request
Dec 21, 2023
…_in_blocks` (apache#28536) Otherwise using the stale `_row_in_blocks` will result in heap-buffer-overflow ``` ==2695213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62900122e210 at pc 0x56524744aecf bp 0x7f62c595ef7 0 sp 0x7f62c595ef68 READ of size 8 at 0x62900122e210 thread T1627 (MemTableFlushTh) #0 0x56524744aece in doris::vectorized::ColumnVector<long>::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_vector.cpp:378:33 #1 0x5652472a7538 in doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_nullable.cpp:310:25 #2 0x56524782a62a in doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/core/block.cpp:961:14 #3 0x565233f187ae in doris::MemTable::_put_into_output(doris::vectorized::Block&) /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:248:27 #4 0x565233f1db66 in doris::MemTable::to_block() /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:496:13 apache#5 0x565233efae60 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:121:62 apache#6 0x565233efc8d6 in doris::FlushToken::_flush_memtable(doris::MemTable*, int, long) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:150:16 apache#7 0x565233f0c5eb in doris::MemtableFlushTask::run() /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:58:23 ```
HappenLee
pushed a commit
that referenced
this pull request
Dec 29, 2023
…f got failed status (apache#29076) Test in regression-test/suites/datatype_p0/decimalv3/test_decimalv3_overflow.groovy::249 sometimes failed when there are multiple BEs and FE process report status slowly for some reason. explain select k1, k2, k1 * k2 from test_decimal128_overflow2 order by 1,2,3 -------------- +----------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | k1[apache#5] | | k2[apache#6] | | (k1 * k2)[apache#7] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 111:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: k1[#0], k2[#1] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 111 | | UNPARTITIONED | | | | 108:VSORT | | | order by: k1[apache#5] ASC, k2[apache#6] ASC, (k1 * k2)[apache#7] ASC | | | offset: 0 | | | | | 102:VOlapScanNode | | TABLE: regression_test_datatype_p0_decimalv3.test_decimal128_overflow2(test_decimal128_overflow2), PREAGGREGATION: ON | | partitions=1/1 (test_decimal128_overflow2), tablets=8/8, tabletList=22841,22843,22845 ... | | cardinality=6, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | projections: k1[#0], k2[#1], (k1[#0] * k2[#1]) | | project output tuple id: 1 | +----------------------------------------------------------------------------------------------------------------------------+ 36 rows in set (0.03 sec) Why failed: Multiple BEs Fragments 0 and 1 are MUST on different BEs Pipeline task of VOlapScanNode which executes k1*k2 failed sets query status to cancelled Pipeline task of VSort call try close, send Cancelled status to VMergeExchange sort_curso did not throw exception when it meets error
HappenLee
pushed a commit
that referenced
this pull request
Jan 12, 2024
…_in_blocks` (apache#28536) Otherwise using the stale `_row_in_blocks` will result in heap-buffer-overflow ``` ==2695213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62900122e210 at pc 0x56524744aecf bp 0x7f62c595ef7 0 sp 0x7f62c595ef68 READ of size 8 at 0x62900122e210 thread T1627 (MemTableFlushTh) #0 0x56524744aece in doris::vectorized::ColumnVector<long>::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_vector.cpp:378:33 #1 0x5652472a7538 in doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_nullable.cpp:310:25 #2 0x56524782a62a in doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/core/block.cpp:961:14 #3 0x565233f187ae in doris::MemTable::_put_into_output(doris::vectorized::Block&) /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:248:27 #4 0x565233f1db66 in doris::MemTable::to_block() /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:496:13 apache#5 0x565233efae60 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:121:62 apache#6 0x565233efc8d6 in doris::FlushToken::_flush_memtable(doris::MemTable*, int, long) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:150:16 apache#7 0x565233f0c5eb in doris::MemtableFlushTask::run() /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:58:23 ```
HappenLee
pushed a commit
that referenced
this pull request
Jan 12, 2024
…f got failed status (apache#29076) Test in regression-test/suites/datatype_p0/decimalv3/test_decimalv3_overflow.groovy::249 sometimes failed when there are multiple BEs and FE process report status slowly for some reason. explain select k1, k2, k1 * k2 from test_decimal128_overflow2 order by 1,2,3 -------------- +----------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | k1[apache#5] | | k2[apache#6] | | (k1 * k2)[apache#7] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 111:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: k1[#0], k2[#1] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 111 | | UNPARTITIONED | | | | 108:VSORT | | | order by: k1[apache#5] ASC, k2[apache#6] ASC, (k1 * k2)[apache#7] ASC | | | offset: 0 | | | | | 102:VOlapScanNode | | TABLE: regression_test_datatype_p0_decimalv3.test_decimal128_overflow2(test_decimal128_overflow2), PREAGGREGATION: ON | | partitions=1/1 (test_decimal128_overflow2), tablets=8/8, tabletList=22841,22843,22845 ... | | cardinality=6, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | projections: k1[#0], k2[#1], (k1[#0] * k2[#1]) | | project output tuple id: 1 | +----------------------------------------------------------------------------------------------------------------------------+ 36 rows in set (0.03 sec) Why failed: Multiple BEs Fragments 0 and 1 are MUST on different BEs Pipeline task of VOlapScanNode which executes k1*k2 failed sets query status to cancelled Pipeline task of VSort call try close, send Cancelled status to VMergeExchange sort_curso did not throw exception when it meets error
HappenLee
pushed a commit
that referenced
this pull request
Jun 1, 2024
…l_backend() (apache#35634) ## Proposed changes ```cpp ==1825462==ERROR: AddressSanitizer: heap-use-after-free on address 0x5030012ea880 at pc 0x555cc645e1aa bp 0x7fe4a9407e20 sp 0x7fe4a94075e0 READ of size 22 at 0x5030012ea880 thread T2250 (REPORT_DISK_STA) #0 0x555cc645e1a9 in __asan_memcpy (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22cb01a9) (BuildId: 76fba423c52f4da5) #1 0x555cc64b6990 in std::char_traits<char>::copy(char*, char const*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/char_traits.h:445:33 #2 0x555cc64b68a1 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_S_copy(char*, char const*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:420:4 #3 0x555cc64c526e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:291:6 #4 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8 apache#5 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15 apache#6 0x555ccb05a722 in doris::TBackend::TBackend(doris::TBackend const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5755:8 apache#7 0x555cc9c9234e in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:79:12 apache#8 0x555cc6586e5a in doris::report_disk_callback(doris::StorageEngine&, doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1021:27 apache#9 0x555cc64f2a0a in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:190:133 apache#10 0x555cc64f2994 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#11 0x555cc64f2944 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#12 0x555cc64f280c in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#13 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#14 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13 apache#15 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#16 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#17 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#18 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#19 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5 apache#20 0x555cc645de0a in asan_thread_start(void*) crtstuff.c apache#21 0x7fef061601c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#22 0x7fef06b4fe72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x5030012ea880 is located 0 bytes inside of 31-byte region [0x5030012ea880,0x5030012ea89f) freed by thread T2249 (REPORT_TASK-182) here: #0 0x555cc649d43d in operator delete(void*) (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22cef43d) (BuildId: 76fba423c52f4da5) #1 0x555cc64b6bbc in std::__new_allocator<char>::deallocate(char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:168:2 #2 0x555cc64b6b5d in std::allocator<char>::deallocate(char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:210:25 #3 0x555cc64b6b5d in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:516:13 #4 0x555cc64b6b5d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:289:9 apache#5 0x555cc64b6a56 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:283:4 apache#6 0x555cc64c51ee in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:285:8 apache#7 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8 apache#8 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15 apache#9 0x555ccb058510 in doris::TBackend::__set_host(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5584:14 apache#10 0x555cc9c92239 in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:75:14 apache#11 0x555cc6585b9d in doris::report_task_callback(doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1004:27 apache#12 0x555cc64f24b9 in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:187:112 apache#13 0x555cc64f2474 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#14 0x555cc64f2424 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#15 0x555cc64f22ec in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#16 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#17 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13 apache#18 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#19 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#20 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#21 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#22 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5 apache#23 0x555cc645de0a in asan_thread_start(void*) crtstuff.c previously allocated by thread T2250 (REPORT_DISK_STA) here: #0 0x555cc649cbdd in operator new(unsigned long) (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22ceebdd) (BuildId: 76fba423c52f4da5) #1 0x555cc64b684e in std::__new_allocator<char>::allocate(unsigned long, void const*) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:147:27 #2 0x555cc64b67b0 in std::allocator<char>::allocate(unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:198:32 #3 0x555cc64b67b0 in std::allocator_traits<std::allocator<char>>::allocate(std::allocator<char>&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:482:20 #4 0x555cc64b67b0 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_S_allocate(std::allocator<char>&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:126:16 apache#5 0x555cc64b63b1 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_create(unsigned long&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:155:14 apache#6 0x555cc64c51de in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:284:24 apache#7 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8 apache#8 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15 apache#9 0x555ccb058510 in doris::TBackend::__set_host(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5584:14 apache#10 0x555cc9c92239 in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:75:14 apache#11 0x555cc6586e5a in doris::report_disk_callback(doris::StorageEngine&, doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1021:27 apache#12 0x555cc64f2a0a in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:190:133 apache#13 0x555cc64f2994 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#14 0x555cc64f2944 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#15 0x555cc64f280c in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#16 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#17 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13 apache#18 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#19 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#20 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#21 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#22 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5 apache#23 0x555cc645de0a in asan_thread_start(void*) crtstuff.c ```
HappenLee
pushed a commit
that referenced
this pull request
Jun 13, 2024
…apache#35458) ## Proposed changes Move memtable core when use multi table load: ``` 0x51f000c73860 is located 3040 bytes inside of 3456-byte region [0x51f000c72c80,0x51f000c73a00) freed by thread T4867 (FragmentMgrThre) here: #0 0x558f6ad7f43d in operator delete(void*) (/mnt/hdd01/STRESS_ENV/be/lib/doris_be+0x22eec43d) (BuildId: b46f73d1f76dfcd6) #1 0x558f6e6cea2c in std::__new_allocator<doris::PTabletID>::deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:168:2 #2 0x558f6e6ce9e7 in std::allocator<doris::PTabletID>::deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:210:25 #3 0x558f6e6ce9e7 in std::allocator_traits<std::allocator<doris::PTabletID>>::deallocate(std::allocator<doris::PTabletID>&, doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:516:13 #4 0x558f6e6ce9e7 in std::_Vector_base<doris::PTabletID, std::allocator<doris::PTabletID>>::_M_deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_vector.h:387:4 apache#5 0x558f6e6d0780 in void std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>::_M_range_insert<__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>>(__gnu_cxx::__normal_iterator<doris::PTabletID*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, std::forward_iterator_tag) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/vector.tcc:832:3 apache#6 0x558f6e6c54c5 in __gnu_cxx::__normal_iterator<doris::PTabletID*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>> std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>::insert<__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, void>(__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_vector.h:1483:4 apache#7 0x558f9b4b214f in doris::LoadStreamMap::save_tablets_to_commit(long, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>> const&) /mnt/disk2/xujianxu/doris/be/src/vec/sink/load_stream_map_pool.cpp:90:13 apache#8 0x558f9b7258dd in doris::vectorized::VTabletWriterV2::_calc_tablets_to_commit() /mnt/disk2/xujianxu/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:650:27 apache#9 0x558f9b7229f1 in doris::vectorized::VTabletWriterV2::close(doris::Status) /mnt/disk2/xujianxu/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:547:9 ``` Multiple sinks with different table loads use the load id, causing confusion in the use of shared data structures between sinks.
HappenLee
pushed a commit
that referenced
this pull request
Jun 17, 2024
…he#36086) ## Proposed changes ``` /root/doris/be/src/vec/common/pod_array.h:448:13: runtime error: applying non-zero offset 16 to null pointer #0 0x562f904159f3 in void doris::vectorized::PODArray<doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::PatternAction, 64ul, AllocatorWithStackMemory<Allocator<false, false, false>, 64ul, 8ul>, 0ul, 0ul>::emplace_back<doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::PatternActionType>(doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::PatternActionType&&) /root/doris/be/src/vec/common/pod_array.h:448:13 #1 0x562f90412d6a in doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::parse_pattern() /root/doris/be/src/vec/aggregate_functions/aggregate_function_sequence_match.h:208:17 #2 0x562f9040a824 in doris::vectorized::AggregateFunctionSequenceMatchData<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long, doris::vectorized::AggregateFunctionSequenceMatch<doris::DateV2Value<doris::DateTimeV2ValueType>, unsigned long> >::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long) /root/doris/be/src/vec/aggregate_functions/aggregate_function_sequence_match.h:95:13 ``` this->c_end may be nullptr
HappenLee
pushed a commit
that referenced
this pull request
Jun 17, 2024
…he#36207) The NormalizeRepeat rule can change the output of agg. For example: ```sql SELECT col_int_undef_signed2 AS C1 , col_int_undef_signed2 FROM normalize_repeat_name_unchanged GROUP BY GROUPING SETS ( (col_int_undef_signed2), (col_int_undef_signed2)) ``` Before fixing the bug, the plan is: ```sql LogicalResultSink[97] ( outputExprs=[C1#7, col_int_undef_signed2#1] ) +--LogicalProject[94] ( distinct=false, projects=[C1#7, C1#7], excepts=[] ) +--LogicalAggregate[93] ( groupByExpr=[C1#7, GROUPING_ID#8], outputExpr=[C1#7, GROUPING_ID#8], hasRepeat=true ) +--LogicalRepeat ( groupingSets=[[C1#7], [C1#7]], outputExpressions=[C1#7, GROUPING_ID#8] ) +--LogicalProject[91] ( distinct=false, projects=[col_int_undef_signed2#1 AS `C1`apache#7], excepts=[] ) +--LogicalOlapScan ( ) ``` This can lead to column not found in LogicalResultSink, report error: Input slot(s) not in childs output: col_int_undef_signed2#1 in plan: LogicalResultSink[97] ( outputExprs=[C1#7, col_int_undef_signed2#1] ) child output is: [C1#7] This pr makes agg output unchanged after normalized repeat. After fixing, the plan is: ```sql LogicalResultSink[97] ( outputExprs=[C1#7, col_int_undef_signed2#1] ) +--LogicalProject[94] ( distinct=false, projects=[C1#7, C1#7 as `col_int_undef_signed2`#1], excepts=[] ) +--LogicalAggregate[93] ( groupByExpr=[C1#7, GROUPING_ID#8], outputExpr=[C1#7, GROUPING_ID#8], hasRepeat=true ) +--LogicalRepeat ( groupingSets=[[C1#7], [C1#7]], outputExpressions=[C1#7, GROUPING_ID#8] ) +--LogicalProject[91] ( distinct=false, projects=[col_int_undef_signed2#1 AS `C1`apache#7], excepts=[] ) +--LogicalOlapScan ( ) ``` --------- Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
HappenLee
pushed a commit
that referenced
this pull request
Jun 17, 2024
## Proposed changes If group commit time interval is larger than the load timeout, and there is no new client load to reuse the internal group commit load, the group commit can not cancel in time because it stuck in wait: ``` #0 0x00007f33937a47aa in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005651105dbd05 in __gthread_cond_timedwait(pthread_cond_t*, pthread_mutex_t*, timespec const*) () #2 0x000056511063f385 in std::__condvar::wait_until(std::mutex&, timespec&) () #3 0x000056511063dc2e in std::cv_status std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () #4 0x000056511063cedf in std::cv_status std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () apache#5 0x0000565110824f48 in std::cv_status std::condition_variable::wait_for<long, std::ratio<1l, 1000l> >(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&) () apache#6 0x0000565113b5612a in doris::LoadBlockQueue::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*, bool*) () apache#7 0x000056513f900941 in doris::pipeline::GroupCommitOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) () apache#8 0x000056513c69c0b6 in doris::pipeline::ScanOperatorX<doris::pipeline::GroupCommitLocalState>::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) () apache#9 0x000056514009d5f1 in doris::pipeline::PipelineTask::execute(bool*) () apache#10 0x00005651400fb24a in doris::pipeline::TaskScheduler::_do_work(unsigned long) () ```
HappenLee
pushed a commit
that referenced
this pull request
Jun 17, 2024
… fail (apache#36269) ## Proposed changes ``` *** Current BE git commitID: 5a8ea30 *** *** SIGSEGV address not mapped to object (@0x18) received by PID 3726857 (TID 3727585 OR 0x7f0129e83700) from PID 24; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/xujianxu/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F01D9E87090 in /lib/x86_64-linux-gnu/libc.so.6 4# std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_State_baseV2::_Setter<doris::Status, doris::Status const&> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 5# std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) at /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/future:593 6# __pthread_once_slow at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_once.c:118 7# std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) at /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/future:428 8# doris::io::MultiTablePipe::_handle_consumer_finished() at /mnt/disk2/xujianxu/doris/be/src/io/fs/multi_table_pipe.cpp:334 9# doris::io::MultiTablePipe::exec_plans<doris::TPipelineFragmentParams>(doris::ExecEnv*, std::vector<doris::TPipelineFragmentParams, std::allocator<doris::TPipelineFragmentParams> >)::{lambda(doris::RuntimeState*, doris::Status*)#1}::operator()(doris::RuntimeState*, doris::Status*) const at /mnt/disk2/xujianxu/doris/be/src/io/fs/multi_table_pipe.cpp:253 10# doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext() at /mnt/disk2/xujianxu/doris/be/src/pipeline/pipeline_fragment_context.cpp:131 11# std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() at /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:199 12# doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) at /mnt/disk2/xujianxu/doris/be/src/pipeline/task_scheduler.cpp:95 13# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /mnt/disk2/xujianxu/doris/be/src/pipeline/task_scheduler.cpp:168 14# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/STRESS_ENV/be/lib/doris_be 15# doris::Thread::supervise_thread(void*) at /mnt/disk2/xujianxu/doris/be/src/util/thread.cpp:499 16# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478 17# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 ``` BE will core when multi table load pipe finish fail. For exec_task will return if finish fail, causing ctx was deconstructed. Wait all table finish to solve this problem.
HappenLee
pushed a commit
that referenced
this pull request
Aug 30, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
HappenLee
pushed a commit
that referenced
this pull request
Aug 30, 2024
…he#39237) ## Proposed changes which led to accessing uninitialized data. Moved the profile initialization into the init method. ``` /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9: runtime error: member call on null pointer of type 'doris::RuntimeProfile::Counter' #0 0x563f0fb8b6c9 in doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper>>::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 #1 0x563f0fb78a45 in doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325:16 #2 0x563f09610366 in doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196:20 #3 0x563f0fb63709 in doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchange_source_operator.cpp:59:24 #4 0x563f09728e80 in doris::pipeline::OperatorXBase::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245:28 apache#5 0x563f0fd3fb58 in doris::pipeline::PipelineTask::close(doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459:28 apache#6 0x563f0fdb5315 in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91:27 apache#7 0x563f0fdb6573 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125:13 apache#8 0x563f0fdb9d6a in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#9 0x563f0fdb9cee in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#10 0x563f0fdb9c4e in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#11 0x563f0fdb99d5 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#12 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#13 0x563ec4bd6db4 in doris::FunctionRunnable::run() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48:27 apache#14 0x563ec4bbc1b5 in doris::ThreadPool::dispatch_thread() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543:24 apache#15 0x563ec4bf9a53 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14 apache#16 0x563ec4bf9858 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#17 0x563ec4bf9790 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11 apache#18 0x563ec4bf9585 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17 apache#19 0x563ec4bf947e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#20 0x563ec4bf93be in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#21 0x563ec4bf8e55 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#22 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#23 0x563ec4b78d91 in doris::Thread::supervise_thread(void*) /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498:5 apache#24 0x563ebdb2fe0a in asan_thread_start(void*) crtstuff.c apache#25 0x7feeac9e21c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#26 0x7feead3d1e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 in *** Query id: ea174401bc134452-bd8a35522726a96a *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1723455006 (unix time) try "date -d @1723455006" if you are using GNU date *** *** Current BE git commitID: c47399c *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3055435 (TID 3060341 OR 0x7fe320acf700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# 0x00007FEEAD3E6B50 in /lib64/libc.so.6 2# doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper> >::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100 3# doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325 4# doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196 5# doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 6# doris::pipeline::OperatorXBase::close(doris::RuntimeState*) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245 7# doris::pipeline::PipelineTask::close(doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459 8# doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91 9# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125 10# doris::pipeline::TaskScheduler::start()::$_0::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64 11# void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 12# std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 13# std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 14# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 15# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 16# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 17# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 18# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 19# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 20# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 21# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 22# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 23# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 24# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 25# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 26# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 27# start_thread in /lib64/libpthread.so.0 28# __clone in /lib64/libc.so.6 ``` <!--Describe your changes.-->
HappenLee
pushed a commit
that referenced
this pull request
Aug 30, 2024
…ort (apache#39340) Since some variables are only initialized when enable_java_support is enabled, not adding this check here would result in accessing a nullptr. ``` *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1723618805 (unix time) try "date -d @1723618805" if you are using GNU date *** *** Current BE git commitID: 8d64cfa563 *** *** SIGSEGV address not mapped to object (@0x0) received by PID 2579243 (TID 2583082 OR 0x7f07ae942700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 2# JVM_handle_linux_signal in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 3# 0x00007F0DA7188B50 in /lib64/libc.so.6 4# jni_CallVoidMethodV in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 5# JNIEnv_::CallVoidMethod(_jobject*, _jmethodID*, ...) at /mnt/disk2/yanxuecheng/java17/include/jni.h:1059 6# doris::JniUtil::clean_udf_class_load_cache(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 7# doris::clean_udf_cache_callback(doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/doris/be/src/agent/task_worker_pool.cpp:2072 8# auto doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const at /mnt/disk2/yanxuecheng/doris/be/src/agent/agent_server.cpp:184 9# void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 10# std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 11# std::_Function_handler<void (doris::TAgentTaskRequest const&), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19>::_M_invoke(std::_Any_data const&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 12# std::function<void (doris::TAgentTaskRequest const&)>::operator()(doris::TAgentTaskRequest const&) const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 13# doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/agent/task_worker_pool.cpp:541 14# void std::__invoke_impl<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>(std::__invoke_other, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 15# std::enable_if<is_invocable_r_v<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>, void>::type std::__invoke_r<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>(doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 16# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 17# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 18# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 19# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 20# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 21# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 22# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 23# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 24# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 25# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 26# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 27# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 28# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 29# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 30# start_thread in /lib64/libpthread.so.0 31# __clone in /lib64/libc.so.6 ```
HappenLee
pushed a commit
that referenced
this pull request
Aug 30, 2024
in fe we will check the params same with be in be we throw exception if params is not support otherwise we will meet coredump like this ``` start BE in local mode /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/array/function_array_aggregation.cpp:150:26: runtime error: member call on null pointer of type 'doris::vectorized::IAggregateFunction' #0 0x55dc04c6b409 in doris::vectorized::ArrayAggregateImpl<(doris::vectorized::AggregateOperation)1>::get_return_type(std::vector<std::shared_ptr<doris::vectorized::IDataType const>, std::allocator<std::shared_ptr<doris::vectorized::IDataType const>>> const&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/array/function_array_aggregation.cpp:150:26 #1 0x55dc04c6a41f in doris::vectorized::FunctionArrayMapped<doris::vectorized::ArrayAggregateImpl<(doris::vectorized::AggregateOperation)1>, doris::vectorized::NameArrayMax>::get_return_type_impl(std::vector<std::shared_ptr<doris::vectorized::IDataType const>, std::allocator<std::shared_ptr<doris::vectorized::IDataType const>>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/array/function_array_mapped.h:71:16 #2 0x55dc049b06cb in doris::vectorized::FunctionBuilderImpl::get_return_type_impl(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:339:16 #3 0x55dc049c1385 in doris::vectorized::DefaultFunctionBuilder::get_return_type_impl(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:579:26 #4 0x55dc0b0a5652 in doris::vectorized::FunctionBuilderImpl::get_return_type_without_low_cardinality(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:283:12 apache#5 0x55dc0b0a5fd4 in doris::vectorized::FunctionBuilderImpl::get_return_type(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:298:17 apache#6 0x55dc04997050 in doris::vectorized::FunctionBuilderImpl::build(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&, std::shared_ptr<doris::vectorized::IDataType const> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:296:47 apache#7 0x55dbde3d727f in doris::vectorized::SimpleFunctionFactory::get_function(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, int) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/simple_function_factory.h:193:32 apache#8 0x55dc04984b4d in doris::vectorized::VectorizedFnCall::prepare(doris::RuntimeState*, doris::RowDescriptor const&, doris::vectorized::VExprContext*) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/exprs/vectorized_fn_call.cpp:111:55 apache#9 0x55dc04ab9b7a in doris::vectorized::VExprContext::prepare(doris::RuntimeState*, doris::RowDescriptor const&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/exprs/vexpr_context.cpp:64:5 apache#10 0x55dbdf625845 in doris::Status doris::FoldConstantExecutor::_prepare_and_open<doris::vectorized::VExprContext>(doris::vectorized::VExprContext*) /mnt/disk1/wangqiannan/amory/doris/be/src/runtime/fold_constant_executor.cpp:185:5 apache#11 0x55dbdf60db76 in doris::FoldConstantExecutor::fold_constant_vexpr(doris::TFoldConstantParams const&, doris::PConstantExprResult*) /mnt/disk1/wangqiannan/amory/doris/be/src/runtime/fold_constant_executor.cpp:94:13 apache#12 0x55dbdf4f58d6 in doris::PInternalService::_fold_constant_expr(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, doris::PConstantExprResult*) /mnt/disk1/wangqiannan/amory/doris/be/src/service/internal_service.cpp:1498:5 apache#13 0x55dbdf544e3e in doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/service/internal_service.cpp:1476:14 apache#14 0x55dbdf54431e in void std::__invoke_impl<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>(std::__invoke_other, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#15 0x55dbdf54425e in std::enable_if<is_invocable_r_v<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>, void>::type std::__invoke_r<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>(doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#16 0x55dbdf543f35 in std::_Function_handler<void (), doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#17 0x55dbd8cd425a in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#18 0x55dbdf59ee14 in doris::WorkThreadPool<false>::work_thread(int) /mnt/disk1/wangqiannan/amory/doris/be/src/util/work_thread_pool.hpp:158:17 @@@ ```
HappenLee
pushed a commit
that referenced
this pull request
Aug 30, 2024
…che#39566) ==17355==ERROR: AddressSanitizer: heap-use-after-free on address 0x61c00388c269 at pc 0x55b47f79e5c4 bp 0x7fde7dd98e30 sp 0x7fde7dd98e28 READ of size 1 at 0x61c00388c269 thread T604 (SpillIOThreadPo) #0 0x55b47f79e5c3 in doris::QueryContext::is_nereids() const /root/doris/be/src/runtime/query_context.h:229:38 #1 0x55b47f79e5c3 in doris::RuntimeState::is_nereids() const /root/doris/be/src/runtime/runtime_state.cpp:542:24 #2 0x55b47f89afed in doris::AttachTask::AttachTask(doris::RuntimeState*) /root/doris/be/src/runtime/thread_context.cpp:48:50 #3 0x55b4b039e7e5 in doris::pipeline::SpillRunnable::run() /root/doris/be/src/pipeline/exec/spill_utils.h:43:9 #4 0x55b47fdf739d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#5 0x55b47fdd05de in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#6 0x55b47fdd05de in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#7 0x7fe1bcb5e608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 apache#8 0x7fe1bce0b132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Co-authored-by: yiguolei <yiguolei@gmail.com>
HappenLee
pushed a commit
that referenced
this pull request
Sep 6, 2024
… enough rdlock inside tablet internal function calls. (apache#39211) ## Proposed changes There's a recurse rdlock inside get_cooldown;when using libcxx,libcxx‘s shared_mutex dosen't support recursive lock, so causes hangs below: #0 0x00007f788b101a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005636b8ff996f in std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) () #2 0x00005636b8ffad8b in std::__1::__shared_mutex_base::lock_shared() () #3 0x00005636addc524c in doris::Tablet::_has_data_to_cooldown() () #4 0x00005636addc353b in doris::Tablet::pick_cooldown_rowset() () apache#5 0x00005636addc56b9 in doris::Tablet::need_cooldown(long*, unsigned long*) () apache#6 0x00005636adde52f7 in doris::TabletManager::get_cooldown_tablets(std::__1::vector<std::__1::shared_ptr<doris::Tablet>, std::__1::allocator<std::__1::shared_ptr<doris::Tablet> > >*, std::__1::vector<std::__1::shared_ptr<doris::Rowset>, std::__1::allocator<std::__1::shared_ptr<doris::Rowset> > >*, std::__1::function<bool (std::__1::shared_ptr<doris::Tablet> const&)>) () apache#7 0x00005636adb0ed81 in doris::StorageEngine::_cooldown_tasks_producer_callback() () apache#8 0x00005636ae17a253 in doris::Thread::supervise_thread(void*) () apache#9 0x00007f788b0fdea5 in start_thread () from /lib64/libpthread.so.0 apache#10 0x00007f788b916b0d in clone () from /lib64/libc.so.6 review the code, there's no need a rdlock inside TabletManager::get_cooldown_tablets
HappenLee
pushed a commit
that referenced
this pull request
Sep 10, 2024
We have dead lock when submit scanner to scheduler failed. pstack looks like ```txt Thread 2012 (Thread 0x7f87363fb700 (LWP 4179707) "Pipe_normal [wo"): #0 0x00007f8b8f3dc82d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f8b8f3d5ad9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x000055b20f333e7a in __gthread_mutex_lock (__mutex=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default .h:749 #3 std::mutex::lock (this=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:100 #4 std::lock_guard<std::mutex>::lock_guard (__m=..., this=<optimized out>) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:229 apache#5 doris::vectorized::ScannerContext::append_block_to_queue (this=<optimized out>, scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:234 apache#6 0x000055b20f32c0f9 in doris::vectorized::ScannerScheduler::submit (this=<optimized out>, ctx=..., scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:209 apache#7 0x000055b20f3338fc in doris::vectorized::ScannerContext::submit_scan_task (this=this@entry=0x7f8733d96010, scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:217 apache#8 0x000055b20f3346cd in doris::vectorized::ScannerContext::get_block_from_queue (this=0x7f8733d96010, state=<optimized out>, block=0x7f871f728de0, eos=0x7f871abce470, id=<optimized out>) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:290 apache#9 0x000055b214cb4f13 in doris::pipeline::ScanOperatorX<doris::pipeline::OlapScanLocalState>::get_block (this=<optimized out>, state=0x7f872f0eb400, block=0x7f8b8f3dc82d <__lll_lock_wait+29>, eos=0x7f871abce470) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/scan_operator.cpp:1292 apache#10 0x000055b2142b5772 in doris::pipeline::ScanOperatorX<doris::pipeline::OlapScanLocalState>::get_block_after_projects (this=0x80, state=0x0, block=0x7f8b8f3dc82d <__lll_lock_wait+29>, eos=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/scan_operator.h:363 apache#11 0x000055b2142e7880 in doris::pipeline::StatefulOperatorX<doris::pipeline::StreamingAggLocalState>::get_block (this=0x7f871f9bee00, state=0x7f872f0eb400, block=0x7f8716d49060, eos=0x7f87363f4937) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/operator.cpp:587 ``` Deallock happens with following ```cpp Status ScannerContext::get_block_from_queue { std::unique_lock l(_transfer_lock); ... if (scan_task->is_eos()) { ... } else { // resubmit current running scanner to read the next block submit_scan_task(scan_task); } } ScannerContext::submit_scan_task(std::shared_ptr<ScanTask> scan_task) { _scanner_scheduler->submit(shared_from_this(), scan_task); } void ScannerScheduler::submit(std::shared_ptr<ScannerContext> ctx, std::shared_ptr<ScanTask> scan_task) { ... if (auto ret = sumbit_task(); !ret) { scan_task->set_status(Status::InternalError( "Failed to submit scanner to scanner pool reason:" + std::string(ret.msg()) + "|type:" + std::to_string(type))); ctx->append_block_to_queue(scan_task); return; } } void ScannerContext::append_block_to_queue(std::shared_ptr<ScanTask> scan_task) { ... std::lock_guard<std::mutex> l(_transfer_lock); ... } ``` Since mutex in cpp is not re-enterable, so the scanner thread will deadlock with itself. This pr fix the problem by making `ScannerScheduler::submit` return a Status instead of doing append failed task to the ScannerContext. The caller itself will decide where resubmit the scanner or just abort the execution of the query.
HappenLee
pushed a commit
that referenced
this pull request
Sep 13, 2024
…#39201) ## Proposed changes pick apache#39191 Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
HappenLee
pushed a commit
that referenced
this pull request
Sep 13, 2024
…he#39237) (apache#39266) ## Proposed changes apache#39237 which led to accessing uninitialized data. Moved the profile initialization into the init method. ``` /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9: runtime error: member call on null pointer of type 'doris::RuntimeProfile::Counter' #0 0x563f0fb8b6c9 in doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper>>::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 #1 0x563f0fb78a45 in doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325:16 #2 0x563f09610366 in doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196:20 #3 0x563f0fb63709 in doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchange_source_operator.cpp:59:24 #4 0x563f09728e80 in doris::pipeline::OperatorXBase::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245:28 apache#5 0x563f0fd3fb58 in doris::pipeline::PipelineTask::close(doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459:28 apache#6 0x563f0fdb5315 in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91:27 apache#7 0x563f0fdb6573 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125:13 apache#8 0x563f0fdb9d6a in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#9 0x563f0fdb9cee in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#10 0x563f0fdb9c4e in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#11 0x563f0fdb99d5 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#12 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#13 0x563ec4bd6db4 in doris::FunctionRunnable::run() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48:27 apache#14 0x563ec4bbc1b5 in doris::ThreadPool::dispatch_thread() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543:24 apache#15 0x563ec4bf9a53 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14 apache#16 0x563ec4bf9858 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#17 0x563ec4bf9790 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11 apache#18 0x563ec4bf9585 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17 apache#19 0x563ec4bf947e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#20 0x563ec4bf93be in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#21 0x563ec4bf8e55 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#22 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#23 0x563ec4b78d91 in doris::Thread::supervise_thread(void*) /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498:5 apache#24 0x563ebdb2fe0a in asan_thread_start(void*) crtstuff.c apache#25 0x7feeac9e21c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#26 0x7feead3d1e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 in *** Query id: ea174401bc134452-bd8a35522726a96a *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1723455006 (unix time) try "date -d @1723455006" if you are using GNU date *** *** Current BE git commitID: c47399c *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3055435 (TID 3060341 OR 0x7fe320acf700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# 0x00007FEEAD3E6B50 in /lib64/libc.so.6 2# doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper> >::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100 3# doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325 4# doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196 5# doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 6# doris::pipeline::OperatorXBase::close(doris::RuntimeState*) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245 7# doris::pipeline::PipelineTask::close(doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459 8# doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91 9# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125 10# doris::pipeline::TaskScheduler::start()::$_0::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64 11# void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 12# std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 13# std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 14# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 15# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 16# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 17# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 18# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 19# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 20# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 21# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 22# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 23# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 24# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 25# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 26# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 27# start_thread in /lib64/libpthread.so.0 28# __clone in /lib64/libc.so.6 ``` <!--Describe your changes.--> ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
HappenLee
pushed a commit
that referenced
this pull request
Sep 18, 2024
…che#40273) ## Proposed changes Issue Number: close #xxx cur_reader pointer can be null in VFileScanner. can cause BE crash [ RUN ] VfileScannerExcepTest.failure_case AddressSanitizer:DEADLYSIGNAL ================================================================= ==1892247==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5cdbe26eb5f8 bp 0x7ffed6728610 sp 0x7ffed67277a0 T0) ==1892247==The signal is caused by a READ memory access. ==1892247==Hint: address points to the zero page. #0 0x5cdbe26eb5f8 in doris::vectorized::VFileScanner::_get_next_reader() /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:980:9 #1 0x5cdbe26e3512 in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:286:25 #2 0x5cdbe26e2e46 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:252:17 #3 0x5cdbe28e05ce in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 #4 0x5cdbc405a922 in doris::vectorized::VfileScannerExcepTest_failure_case_Test::TestBody() /root/doris/workspace/doris/be/test/vec/exec/vfile_scanner_excep_test.cpp:309:24 apache#5 0x5cdbff5b191a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51eb491a) (BuildId: 21d41a2d207823b9) apache#6 0x5cdbff59f989 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51ea2989) (BuildId: 21d41a2d207823b9) apache#7 0x5cdbff57a9c2 in testing::Test::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7d9c2) (BuildId: 21d41a2d207823b9) apache#8 0x5cdbff57b708 in testing::TestInfo::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7e708) (BuildId: 21d41a2d207823b9) apache#9 0x5cdbff57bec3 in testing::TestSuite::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7eec3) (BuildId: 21d41a2d207823b9) After fix: I20240902 09:11:07.722273 1946048 run_all_tests.cpp:67] init config 1 Note: Google Test filter = VfileScannerE* [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from VfileScannerExceptionTest [ RUN ] VfileScannerExceptionTest.failure_case msg = [INTERNAL_ERROR]cur path: . Failed to create reader for file format: 11 [ OK ] VfileScannerExceptionTest.failure_case (3 ms) [----------] 1 test from VfileScannerExceptionTest (3 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (3 ms total) [ PASSED ] 1 test. === Finished. Gtest output: /root/doris/workspace/doris/be/ut_build_ASAN/gtest_output
HappenLee
pushed a commit
that referenced
this pull request
Sep 18, 2024
…#40643) ## Proposed changes ### Issue ``` *** is nereids: 1 *** tablet id: 4 Abort at 1725864966 (unix time) try "date -d @1725864966" if you are using GNU date *** *** Set a breakpoint in static void __GI_abort() to debug *** PC: @ 0x7f007fb4090a04 *** SIGSEGV (address not mapped to object 0xa0fa868a41d6) received by PID 404737 (TID 274135 OR 0x7ece29df700) from PID 1755584205; stack trace: *** #0 __GI_raise #1 __GI_abort #2 sig_handler #3 _sigaction #4 JVM_handle_linux_signal apache#5 _sigaction apache#6 doris::vectorized::ByteArrayDictDecoder::set_dict(std::unique_ptr<unsigned char[], std::default_delete<unsigned char[]>> &&, int, unsigned long) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/byte_array_dict_decoder.cpp:41 apache#7 doris::vectorized::ColumnChunkReader::_decode_dict_page() at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp:258 apache#8 doris::vectorized::ColumnChunkReader::next_page() at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp:105 apache#9 doris::vectorized::ParquetColumnReader::_read_column_data(doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:508 apache#10 doris::vectorized::ScalarColumnReader::_next_value(doris::vectorized::ICollumn*, unsigned long, unsigned long*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:699 apache#11 doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &, std::vector<doris::vectorized::ColumnSelectVector>*, unsigned long, unsigned long*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:425 apache#12 doris::vectorized::RowGroupReader::get_next_block(doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:311 apache#13 doris::vectorized::ParquetReader::get_next(doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_reader.cpp:533 apache#14 doris::vectorized::VFileScanner::_get_next_reader_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/vfile_scanner.cpp:368 apache#15 doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/vfile_scanner.cpp:411 apache#16 doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/vscanner.cpp:431 apache#17 doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/vscanner.cpp:96 apache#18 doris::vectorized::ScannerScheduler::submit(doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::ScanTask>) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/scanner_context.cpp:96 apache#19 doris::Thread::supervise_thread(void*) at /mnt/disk1/yy/git/enterprise-core/be/src/util/thread.cpp:499 apache#20 start_thread apache#21 clone in /lib64/libc.so.6 ``` ### Solution It is not known why the parquet dictionary page will be null in this case, causing a crash. This PR adds defensive code to prevent the crash.
HappenLee
pushed a commit
that referenced
this pull request
Sep 18, 2024
1. Add a new class MemCounter. 2. MemTracker and MemTrackerLimiter no longer have inheritance and dependencies. 3. MemTrackerLimiter is used to count process memory, the BE web page `/mem_tracker` also only displays MemTrackerLimiter. 4. MemTracker is used to count the memory of operators and some data structures. It is not counted in the process memory and is used for logic control and profile. In addition, it seems that the crash is caused by memory abnormality, not sure whether this PR can fix the problem, but this will help locate the problem and more elegant. ``` ==6641==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000980e70 at pc 0x559be21880be bp 0x7fd7cfc75070 sp 0x7fd7cfc75068 READ of size 8 at 0x603000980e70 thread T1453 (memory_maintena) #0 0x559be21880bd in std::__atomic_base<long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 #1 0x559be21880bd in doris::MemTracker::MemCounter::current_value() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:139:63 #2 0x559be21880bd in doris::MemTracker::consumption() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:165:56 #3 0x559be3985133 in doris::MemTrackerLimiter::refresh_global_counter() /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker_limiter.cpp:245:59 #4 0x559bdfcd52d5 in doris::Daemon::memory_maintenance_thread() /home/zcp/repo_center/doris_master/doris/be/src/common/daemon.cpp:239:13 apache#5 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#6 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#7 0x7fdf5647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x603000980e74 is located 0 bytes after 20-byte region [0x603000980e60,0x603000980e74) 04:19:48 freed by thread T1491 (Scan_normal [wo) here: 04:19:48 #0 0x559bdfb0dd9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x33546d9d) (BuildId: f170e92ad3c55512) 04:19:48 #1 0x559bdfb1d42b in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:139:2 04:19:48 #2 0x559bdfb1d42b in std::allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:187:27 04:19:48 #3 0x559bdfb1d42b in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:492:13 04:19:48 #4 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:237:9 04:19:48 apache#5 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:232:4 04:19:48 apache#6 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:658:9 04:19:48 apache#7 0x559bdfb22501 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 04:19:48 apache#8 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 04:19:48 apache#9 0x559bdfb22501 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 04:19:48 apache#10 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 04:19:48 apache#11 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 04:19:48 apache#12 0x559bdfb22501 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2 04:19:48 apache#13 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9867:1 04:19:48 apache#14 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9866:36 04:19:48 apache#15 0x559be0047ee6 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:372:5 04:19:48 apache#16 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13 04:19:48 apache#17 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28 04:19:48 apache#18 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18 04:19:48 apache#19 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18 04:19:48 apache#20 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5 04:19:48 apache#21 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32 04:19:48 apache#22 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 04:19:48 apache#23 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 04:19:48 apache#24 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 04:19:48 apache#25 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 04:19:48 apache#26 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 04:19:48 apache#27 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 04:19:48 apache#28 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 04:19:48 apache#29 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 04:19:48 apache#30 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 04:19:48 04:19:48 previously allocated by thread T1491 (Scan_normal [wo) here: 04:19:48 #0 0x559bdfb0d53d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x3354653d) (BuildId: f170e92ad3c55512) 04:19:48 #1 0x559bdfb33f74 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14 04:19:48 #2 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct_aux<char*>(char*, char*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11 04:19:48 #3 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4 04:19:48 #4 0x559bdfb32705 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:451:9 04:19:48 apache#5 0x559bdfc326e6 in decltype(::new((void*)(0)) std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>(std::declval<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>())) std::construct_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39 04:19:48 apache#6 0x559bdfc326e6 in void std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4 04:19:48 apache#7 0x559bdfc326e6 in void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:449:4 04:19:48 apache#8 0x559be0032206 in doris::DeleteHandler::parse_condition(doris::DeleteSubPredicatePB const&, doris::TCondition*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:300:33 04:19:48 apache#9 0x559be0047aa3 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:355:9 04:19:48 apache#10 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13 04:19:48 apache#11 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28 04:19:48 apache#12 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18 04:19:48 apache#13 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18 04:19:48 apache#14 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5 04:19:48 apache#15 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32 04:19:48 apache#16 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 04:19:48 apache#17 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 04:19:48 apache#18 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 04:19:48 apache#19 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 04:19:48 apache#20 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 04:19:48 apache#21 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 04:19:48 apache#22 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 04:19:48 apache#23 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 04:19:48 apache#24 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 ```
HappenLee
pushed a commit
that referenced
this pull request
Sep 18, 2024
…0857) ## Proposed changes ### Issue ``` /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef' #0 0x55ee63eb0418 in std::vector<doris::StringRef, std::allocator<doris::StringRef>>::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2 #1 0x55ee63eb0418 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1172:39 #2 0x55ee63ea2685 in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1124:16 #3 0x55ee63e97e7a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1365:16 #4 0x55ee63b0e450 in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1532:5 apache#5 0x55ee63e99622 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1410:9 apache#6 0x55ee63b0e450 in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1532:5 apache#7 0x55ee63ad4f86 in doris::vectorized::OrcReader::get_next_block_impl(doris::vectorized::Block*, unsigned long*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1714:13 apache#8 0x55ee63ad093b in doris::vectorized::OrcReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1547:5 ``` ### Solution [Fix] (orc-reader) Fix StringRef nullptr data in orc-reader. When string is empty in orc row batch, the data can point anything, maybe nullptr, StringRef has undefined behavior when data is nullptr. Related with apache#37845.
HappenLee
pushed a commit
that referenced
this pull request
Oct 1, 2024
…o null pointer (apache#41486) ## Proposed changes ``` /root/doris/be/src/vec/common/pod_array.h:406:13: runtime error: applying non-zero offset 16 to null pointer #0 0x55827f4bbf0c in void doris::vectorized::PODArray<__int128, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 0ul, 0ul>::push_back<__int128&>(__int128&) /root/doris/be/src/vec/common/pod_array.h:406:13 #1 0x55827f4baa40 in doris::Counts<__int128>::increment(__int128) /root/doris/be/src/util/counts.h:160:36 #2 0x55827f4baa40 in doris::vectorized::PercentileState<__int128>::add(__int128, doris::vectorized::PODArray<double, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul> const&, int) /root/doris/be/src/vec/aggregate_functions/aggregate_function_percentile.h:634:27 #3 0x55827f6ca40f in doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionPercentileArray<__int128>, false>::add(char*, doris::vectorized::IColumn const**, long, doris::vectorized::Arena*) const /root/doris/be/src/vec/aggregate_functions/aggregate_function_null.h:324:32 #4 0x55827f6cef0e in doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionPercentileArray<__int128>, false> >::add_batch(unsigned long, char**, unsigned long, doris::vectorized::IColumn const**, doris::vectorized::Arena*, bool) const /root/doris/be/src/vec/aggregate_functions/aggregate_function.h:273:22 apache#5 0x5582a0c41059 in doris::vectorized::AggFnEvaluator::execute_batch_add(doris::vectorized::Block*, unsigned long, char**, doris::vectorized::Arena*, bool) /root/doris/be/src/vec/exprs/vectorized_agg_fn.cpp:241:16 apache#6 0x5582a0c39037 in doris::Status doris::pipeline::AggSinkLocalState::_execute_with_serialized_key_helper<false>(doris::vectorized::Block*)::'lambda'()::operator()() const /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:481:17 apache#7 0x5582a0b51e1e in doris::Status doris::pipeline::AggSinkLocalState::_execute_with_serialized_key_helper<false>(doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:497:13 apache#8 0x5582a0a014e1 in doris::pipeline::AggSinkLocalState::_execute_with_serialized_key(doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:437:16 apache#9 0x5582a0c37811 in doris::pipeline::AggSinkLocalState::Executor<false, false>::execute(doris::pipeline::AggSinkLocalState*, doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.h:61:41 ``` <!--Describe your changes.-->
HappenLee
pushed a commit
that referenced
this pull request
Oct 10, 2024
…apache#41495) ## Proposed changes Fix problem "Runtime Error: Null pointer passed to 'StringTypeInvertedIndexReader::query', which requires a non-null argument." ``` Stack Trace: #0 doris::segment_v2::StringTypeInvertedIndexReader::query(...) inverted_index_reader.cpp:473 #1 doris::segment_v2::InvertedIndexIterator::read_from_inverted_index(...) inverted_index_reader.cpp:1237 #2 doris::vectorized::FunctionArraysOverlap::evaluate_inverted_index(...) function_arrays_overlap.h:192 #3 doris::vectorized::DefaultFunction::evaluate_inverted_index(...) function.h:532 #4 doris::vectorized::VExpr::_evaluate_inverted_index(...) vexpr.cpp:708 apache#5 doris::vectorized::VectorizedFnCall::evaluate_inverted_index(...) vectorized_fn_call.cpp:143 apache#6 doris::vectorized::VExprContext::evaluate_inverted_index(...) vexpr_context.cpp:126 ```
HappenLee
pushed a commit
that referenced
this pull request
Oct 16, 2024
…#41743) ==40376==ERROR: AddressSanitizer: heap-use-after-free on address 0x616002419ed0 at pc 0x56132d2dc34a bp 0x7fe20150ba70 sp 0x7fe20150ba68 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27557_27557&logView=flowAware) WRITE of size 4 at 0x616002419ed0 thread T1474 (Pipe_normal [wo) [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27558_27558&logView=flowAware) #0 0x56132d2dc349 in doris::pipeline::PipelineTask::update_queue_level(int) /root/doris/be/src/pipeline/pipeline_task.h:174:67 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27559_27559&logView=flowAware) #1 0x56132d2dc349 in doris::pipeline::PriorityTaskQueue::_try_take_unprotected(bool) /root/doris/be/src/pipeline/task_queue.cpp:79:15 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27560_27560&logView=flowAware) #2 0x56132d2dc615 in doris::pipeline::PriorityTaskQueue::try_take(bool) /root/doris/be/src/pipeline/task_queue.cpp:97:12 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27561_27561&logView=flowAware) #3 0x56132d2de69f in doris::pipeline::MultiCoreTaskQueue::take(int) /root/doris/be/src/pipeline/task_queue.cpp:164:50 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27562_27562&logView=flowAware) #4 0x56132d2ee0e3 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /root/doris/be/src/pipeline/task_scheduler.cpp:102:35 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27563_27563&logView=flowAware) apache#5 0x5612fc23029d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27564_27564&logView=flowAware) apache#6 0x5612fc20880e in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27565_27565&logView=flowAware) apache#7 0x5612fc20880e in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27566_27566&logView=flowAware) apache#8 0x7fe9405ca608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27567_27567&logView=flowAware) apache#9 0x7fe940877132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27568_27568&logView=flowAware) [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27569_27569&logView=flowAware) 0x616002419ed0 is located 80 bytes inside of 632-byte region [0x616002419e80,0x61600241a0f8) [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27570_27570&logView=flowAware) freed by thread T1463 (Pipe_normal [wo) here: [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27571_27571&logView=flowAware) #0 0x5612f7ac280d in operator delete(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2ed1180d) (BuildId: 04f75ee9f41a400a) [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27572_27572&logView=flowAware) #1 0x56132d2519d4 in std::default_delete<doris::pipeline::PipelineTask>::operator()(doris::pipeline::PipelineTask*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27573_27573&logView=flowAware) #2 0x56132d2519d4 in std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27574_27574&logView=flowAware) #3 0x56132d2519d4 in void std::destroy_at<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27575_27575&logView=flowAware) #4 0x56132d2519d4 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27576_27576&logView=flowAware) apache#5 0x56132d2519d4 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*>(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27577_27577&logView=flowAware) apache#6 0x56132d2519d4 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*>(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27578_27578&logView=flowAware) apache#7 0x56132d2519d4 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27579_27579&logView=flowAware) apache#8 0x56132d2519d4 in std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27580_27580&logView=flowAware) apache#9 0x56132d1d717c in void std::destroy_at<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27581_27581&logView=flowAware) apache#10 0x56132d1d717c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27582_27582&logView=flowAware) apache#11 0x56132d1d717c in void std::_Destroy_aux<false>::__destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27583_27583&logView=flowAware) apache#12 0x56132d1d717c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27584_27584&logView=flowAware) apache#13 0x56132d1d717c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27585_27585&logView=flowAware) apache#14 0x56132d1d717c in std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > > >::_M_erase_at_end(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1796:6 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27586_27586&logView=flowAware) apache#15 0x56132d1d717c in std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > > >::clear() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1499:9 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27587_27587&logView=flowAware) apache#16 0x56132d1d717c in doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext() /root/doris/be/src/pipeline/pipeline_fragment_context.cpp:143:12 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27588_27588&logView=flowAware) apache#17 0x5612f7aeabdc in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27589_27589&logView=flowAware) apache#18 0x56132d2ed705 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27590_27590&logView=flowAware) apache#19 0x56132d2ed705 in std::__shared_ptr<doris::TaskExecutionContext, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27591_27591&logView=flowAware) apache#20 0x56132d2ed705 in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /root/doris/be/src/pipeline/task_scheduler.cpp:98:1 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27592_27592&logView=flowAware) apache#21 0x56132d2efce9 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /root/doris/be/src/pipeline/task_scheduler.cpp:181:17 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27593_27593&logView=flowAware) apache#22 0x5612fc23029d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
## Proposed changes ``` F20240628 01:49:16.382710 4183685 delete_handler.cpp:388] Check failed: !_is_inited reinitialize delete handler. *** Check failure stack trace: *** @ 0x55700470e3c6 google::LogMessage::SendToLog() @ 0x55700470ae10 google::LogMessage::Flush() @ 0x55700470ec09 google::LogMessageFatal::~LogMessageFatal() @ 0x556fccf40e64 doris::DeleteHandler::init() @ 0x556fcff46678 doris::TabletReader::_init_delete_condition() @ 0x556fcff3a2dd doris::TabletReader::_init_params() @ 0x556fcff39432 doris::TabletReader::init() @ 0x556fffb8c2dd doris::vectorized::BlockReader::init() @ 0x557002cca96a doris::vectorized::NewOlapScanner::open() @ 0x556fe892d565 doris::vectorized::ScannerScheduler::_scanner_scan() @ 0x556fe8931a0f _ZNSt17_Function_handlerIFvvEZZN5doris10vectorized16ScannerScheduler6submitESt10shared_ptrINS2_14ScannerContextEES4_INS2_8ScanTaskEEENK3$_1clEvEUlvE_E9_M_invokeERKSt9_Any_data @ 0x556fd0ed95dc doris::ThreadPool::dispatch_thread() @ 0x556fd0eb1288 doris::Thread::supervise_thread() @ 0x7f95143b5609 start_thread @ 0x7f9514662133 clone @ (nil) (unknown) *** Query id: c389fc2a1ff6473c-a06f032b8970810c *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1719510556 (unix time) try "date -d @1719510556" if you are using GNU date *** *** Current BE git commitID: b13c17d *** *** SIGABRT unknown detail explain (@0x3fca33) received by PID 4180531 (TID 4183685 OR 0x7f89734a5700) from PID 4180531; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421 1# 0x00007F9514586090 in /lib/x86_64-linux-gnu/libc.so.6 2# raise at ../sysdeps/unix/sysv/linux/raise.c:51 3# abort at /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81 4# 0x0000557004718C9D in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 5# 0x000055700470B2DA in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 6# google::LogMessage::SendToLog() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 7# google::LogMessage::Flush() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 9# doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta> > > const&, long, bool) at /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:388 10# doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:654 11# doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:295 12# doris::TabletReader::init(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:128 13# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 14# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:219 15# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250 16# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 17# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 18# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 19# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478 20# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 ``` related PRs: apache#36090, apache#36101, apache#36314
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
## Proposed changes - Null pointer of type 'doris::StringRef' in orc reader. The root cause is error will throw when `num_values == 0` in `_decode_string_non_dict_encoded_column`. ``` /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef' #0 0x562516fa9770 in std::vector<doris::StringRef, std::allocator<doris::StringRef> >::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2 #1 0x562516fa9770 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1164:39 #2 0x562516f9c08b in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1116:16 #3 0x562516f91d73 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1357:16 #4 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 apache#5 0x562516f9339a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1402:9 apache#6 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 ... ``` - Shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') in parquet reader. The root cause is error will throw when `len == 0`. ``` /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27: runtime error: shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') #0 0x56251760fbc7 in doris::vectorized::parquet::StringToDecimal<doris::vectorized::Decimal128V3, (doris::vectorized::DecimalScaleParams::ScaleType)1>::physical_convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27 #1 0x562517290dc4 in doris::vectorized::parquet::PhysicalToLogicalConverter::convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::TypeDescriptor, std::shared_ptr<doris::vectorized::IDataType const> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, bool) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:209:9 #2 0x562517284a6d in doris::vectorized::ScalarColumnReader::read_column_data(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, unsigned long, unsigned long*, bool*, bool) /root/doris/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569:24 #3 0x56251725ae7e in doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::vectorized::ColumnSelectVector&) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:421:13 #4 0x56251724d6d2 in doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:321:9 apache#5 0x56251708eb97 in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:530:36 apache#6 0x56253036772d in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:311:13 apache#7 0x562530366549 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:253:17 apache#8 0x5625176e79c8 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 apache#9 0x5625176e6fc1 in doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:84:12 apache#10 0x562517698047 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250:5 apache#11 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:172:25 apache#12 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:171:35 ... ```
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
… StreamLoadContext when operate_txn_2pc (apache#38033) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:384:14: runtime error: load of value 3200171710, which is not a valid value for type 'TLoadType::type' #0 0x55be4e2cf808 in doris::StreamLoadExecutor::collect_load_stat(doris::StreamLoadContext*, doris::TTxnCommitAttachment*) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:384:14 #1 0x55be4e2d2666 in doris::StreamLoadExecutor::rollback_txn(doris::StreamLoadContext*) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:364:9 #2 0x55be80413748 in doris::CloudStreamLoadExecutor::operate_txn_2pc(doris::StreamLoadContext*) /root/doris/be/src/cloud/cloud_stream_load_executor.cpp:80:33 #3 0x55be50cad893 in doris::StreamLoad2PCAction::handle(doris::HttpRequest*) /root/doris/be/src/http/action/stream_load_2pc.cpp:85:49
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
…out range of _buf (apache#37936) if we select nested type such as map/array/struct after large string , when string type in mysql_row_buf reserve make buffer size is not large enough , which will lead nested type open_dynamic_mode make _pos pointer out range of mysql_row_buf, then nested type call push_string, and reserve() will make heap_buffer_overflow ``` ==200769==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0051c12ba at pc 0x55a77788692d bp 0x7fb52f474a30 sp 0x7fb52f4741f8 READ of size 36541 at 0x62d0051c12ba thread T2309 (Pipe_normal [wo) #0 0x55a77788692c in __asan_memcpy (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c1c92c) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f622fa in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:140:5 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a21f16eb in doris::Status doris::vectorized::DataTypeMapSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:410:21 #4 0x55a7a21e4c1e in doris::vectorized::DataTypeMapSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:478:12 apache#5 0x55a7a22070e6 in doris::Status doris::vectorized::DataTypeNullableSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:300:9 apache#6 0x55a7a21fbc5e in doris::vectorized::DataTypeNullableSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:317:12 apache#7 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 apache#8 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 apache#9 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 apache#10 0x55a7c99a4b27 in doris::pipeline::PipelineTask::execute(bool*) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:364:22 apache#11 0x55a7c9a23a2b in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:138:9 apache#12 0x55a7c9a269ca in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#13 0x55a7c9a2694e in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#14 0x55a7c9a268ae in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#15 0x55a7c9a26635 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#16 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#17 0x55a77e95ec94 in doris::FunctionRunnable::run() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:48:27 apache#18 0x55a77e941015 in doris::ThreadPool::dispatch_thread() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:543:24 apache#19 0x55a77e97eb23 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 apache#20 0x55a77e97e928 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#21 0x55a77e97e860 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 apache#22 0x55a77e97e655 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 apache#23 0x55a77e97e54e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#24 0x55a77e97e48e in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#25 0x55a77e97dd85 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#26 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#27 0x55a77e8fb841 in doris::Thread::supervise_thread(void*) /mnt/disk1/wangqiannan/amory/doris/be/src/util/thread.cpp:498:5 apache#28 0x7fc1c3a111c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#29 0x7fc1c4400e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x62d0051c12ba is located 0 bytes after 36538-byte region [0x62d0051b8400,0x62d0051c12ba) allocated by thread T2309 (Pipe_normal [wo) here: #0 0x55a7778c20bd in operator new[](unsigned long) (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c580bd) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f621c1 in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:137:21 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a1fd0d75 in doris::Status doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:260:16 #4 0x55a7a1fccc1e in doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:215:16 apache#5 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 apache#6 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 apache#7 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
…pache#38935) ``` 1 __pthread_kill_internal (signo=6, threadid=140314099000896) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140314099000896, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007fa1e2e8e476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007fa1e2e747f3 in __GI_abort () at ./stdlib/abort.c:79 apache#5 0x000055f77d0e71ea in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 apache#6 0x000055f77d0e5956 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 apache#7 0x000055f77d0e59c1 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58 apache#8 0x000055f76ffc5fbe in __clang_call_terminate () apache#9 0x000055f77667ef44 in doris::Defer<doris::vectorized::FunctionVariantElement::get_element_column(doris::vectorized::ColumnObject const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*)::{lambda()#1}>::~Defer() (this=0x7f9d6c06a900) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/defer_op.h:37 ```
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
…he#39237) (apache#39264) ## Proposed changes apache#39237 which led to accessing uninitialized data. Moved the profile initialization into the init method. ``` /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9: runtime error: member call on null pointer of type 'doris::RuntimeProfile::Counter' #0 0x563f0fb8b6c9 in doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper>>::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 #1 0x563f0fb78a45 in doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325:16 #2 0x563f09610366 in doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196:20 #3 0x563f0fb63709 in doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchange_source_operator.cpp:59:24 #4 0x563f09728e80 in doris::pipeline::OperatorXBase::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245:28 apache#5 0x563f0fd3fb58 in doris::pipeline::PipelineTask::close(doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459:28 apache#6 0x563f0fdb5315 in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91:27 apache#7 0x563f0fdb6573 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125:13 apache#8 0x563f0fdb9d6a in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#9 0x563f0fdb9cee in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#10 0x563f0fdb9c4e in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#11 0x563f0fdb99d5 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#12 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#13 0x563ec4bd6db4 in doris::FunctionRunnable::run() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48:27 apache#14 0x563ec4bbc1b5 in doris::ThreadPool::dispatch_thread() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543:24 apache#15 0x563ec4bf9a53 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14 apache#16 0x563ec4bf9858 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#17 0x563ec4bf9790 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11 apache#18 0x563ec4bf9585 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17 apache#19 0x563ec4bf947e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#20 0x563ec4bf93be in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#21 0x563ec4bf8e55 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#22 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#23 0x563ec4b78d91 in doris::Thread::supervise_thread(void*) /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498:5 apache#24 0x563ebdb2fe0a in asan_thread_start(void*) crtstuff.c apache#25 0x7feeac9e21c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#26 0x7feead3d1e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 in *** Query id: ea174401bc134452-bd8a35522726a96a *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1723455006 (unix time) try "date -d @1723455006" if you are using GNU date *** *** Current BE git commitID: c47399c *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3055435 (TID 3060341 OR 0x7fe320acf700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# 0x00007FEEAD3E6B50 in /lib64/libc.so.6 2# doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper> >::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100 3# doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325 4# doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196 5# doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 6# doris::pipeline::OperatorXBase::close(doris::RuntimeState*) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245 7# doris::pipeline::PipelineTask::close(doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459 8# doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91 9# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125 10# doris::pipeline::TaskScheduler::start()::$_0::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64 11# void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 12# std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 13# std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 14# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 15# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 16# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 17# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 18# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 19# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 20# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 21# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 22# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 23# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 24# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 25# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 26# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 27# start_thread in /lib64/libpthread.so.0 28# __clone in /lib64/libc.so.6 ``` <!--Describe your changes.--> ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
…ort (apache#39340) Since some variables are only initialized when enable_java_support is enabled, not adding this check here would result in accessing a nullptr. ``` *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1723618805 (unix time) try "date -d @1723618805" if you are using GNU date *** *** Current BE git commitID: 8d64cfa563 *** *** SIGSEGV address not mapped to object (@0x0) received by PID 2579243 (TID 2583082 OR 0x7f07ae942700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 2# JVM_handle_linux_signal in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 3# 0x00007F0DA7188B50 in /lib64/libc.so.6 4# jni_CallVoidMethodV in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 5# JNIEnv_::CallVoidMethod(_jobject*, _jmethodID*, ...) at /mnt/disk2/yanxuecheng/java17/include/jni.h:1059 6# doris::JniUtil::clean_udf_class_load_cache(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 7# doris::clean_udf_cache_callback(doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/doris/be/src/agent/task_worker_pool.cpp:2072 8# auto doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const at /mnt/disk2/yanxuecheng/doris/be/src/agent/agent_server.cpp:184 9# void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 10# std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 11# std::_Function_handler<void (doris::TAgentTaskRequest const&), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19>::_M_invoke(std::_Any_data const&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 12# std::function<void (doris::TAgentTaskRequest const&)>::operator()(doris::TAgentTaskRequest const&) const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 13# doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/agent/task_worker_pool.cpp:541 14# void std::__invoke_impl<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>(std::__invoke_other, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 15# std::enable_if<is_invocable_r_v<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>, void>::type std::__invoke_r<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>(doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 16# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 17# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 18# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 19# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 20# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 21# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 22# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 23# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 24# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 25# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 26# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 27# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 28# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 29# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 30# start_thread in /lib64/libpthread.so.0 31# __clone in /lib64/libc.so.6 ```
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
…che#39566) ==17355==ERROR: AddressSanitizer: heap-use-after-free on address 0x61c00388c269 at pc 0x55b47f79e5c4 bp 0x7fde7dd98e30 sp 0x7fde7dd98e28 READ of size 1 at 0x61c00388c269 thread T604 (SpillIOThreadPo) #0 0x55b47f79e5c3 in doris::QueryContext::is_nereids() const /root/doris/be/src/runtime/query_context.h:229:38 #1 0x55b47f79e5c3 in doris::RuntimeState::is_nereids() const /root/doris/be/src/runtime/runtime_state.cpp:542:24 #2 0x55b47f89afed in doris::AttachTask::AttachTask(doris::RuntimeState*) /root/doris/be/src/runtime/thread_context.cpp:48:50 #3 0x55b4b039e7e5 in doris::pipeline::SpillRunnable::run() /root/doris/be/src/pipeline/exec/spill_utils.h:43:9 #4 0x55b47fdf739d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#5 0x55b47fdd05de in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#6 0x55b47fdd05de in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#7 0x7fe1bcb5e608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 apache#8 0x7fe1bce0b132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Co-authored-by: yiguolei <yiguolei@gmail.com>
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
We have dead lock when submit scanner to scheduler failed. pstack looks like ```txt Thread 2012 (Thread 0x7f87363fb700 (LWP 4179707) "Pipe_normal [wo"): #0 0x00007f8b8f3dc82d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f8b8f3d5ad9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x000055b20f333e7a in __gthread_mutex_lock (__mutex=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default .h:749 #3 std::mutex::lock (this=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:100 #4 std::lock_guard<std::mutex>::lock_guard (__m=..., this=<optimized out>) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:229 apache#5 doris::vectorized::ScannerContext::append_block_to_queue (this=<optimized out>, scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:234 apache#6 0x000055b20f32c0f9 in doris::vectorized::ScannerScheduler::submit (this=<optimized out>, ctx=..., scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:209 apache#7 0x000055b20f3338fc in doris::vectorized::ScannerContext::submit_scan_task (this=this@entry=0x7f8733d96010, scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:217 apache#8 0x000055b20f3346cd in doris::vectorized::ScannerContext::get_block_from_queue (this=0x7f8733d96010, state=<optimized out>, block=0x7f871f728de0, eos=0x7f871abce470, id=<optimized out>) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:290 apache#9 0x000055b214cb4f13 in doris::pipeline::ScanOperatorX<doris::pipeline::OlapScanLocalState>::get_block (this=<optimized out>, state=0x7f872f0eb400, block=0x7f8b8f3dc82d <__lll_lock_wait+29>, eos=0x7f871abce470) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/scan_operator.cpp:1292 apache#10 0x000055b2142b5772 in doris::pipeline::ScanOperatorX<doris::pipeline::OlapScanLocalState>::get_block_after_projects (this=0x80, state=0x0, block=0x7f8b8f3dc82d <__lll_lock_wait+29>, eos=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/scan_operator.h:363 apache#11 0x000055b2142e7880 in doris::pipeline::StatefulOperatorX<doris::pipeline::StreamingAggLocalState>::get_block (this=0x7f871f9bee00, state=0x7f872f0eb400, block=0x7f8716d49060, eos=0x7f87363f4937) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/operator.cpp:587 ``` Deallock happens with following ```cpp Status ScannerContext::get_block_from_queue { std::unique_lock l(_transfer_lock); ... if (scan_task->is_eos()) { ... } else { // resubmit current running scanner to read the next block submit_scan_task(scan_task); } } ScannerContext::submit_scan_task(std::shared_ptr<ScanTask> scan_task) { _scanner_scheduler->submit(shared_from_this(), scan_task); } void ScannerScheduler::submit(std::shared_ptr<ScannerContext> ctx, std::shared_ptr<ScanTask> scan_task) { ... if (auto ret = sumbit_task(); !ret) { scan_task->set_status(Status::InternalError( "Failed to submit scanner to scanner pool reason:" + std::string(ret.msg()) + "|type:" + std::to_string(type))); ctx->append_block_to_queue(scan_task); return; } } void ScannerContext::append_block_to_queue(std::shared_ptr<ScanTask> scan_task) { ... std::lock_guard<std::mutex> l(_transfer_lock); ... } ``` Since mutex in cpp is not re-enterable, so the scanner thread will deadlock with itself. This pr fix the problem by making `ScannerScheduler::submit` return a Status instead of doing append failed task to the ScannerContext. The caller itself will decide where resubmit the scanner or just abort the execution of the query.
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
… enough rdlock inside tablet internal function calls. (apache#39211) ## Proposed changes There's a recurse rdlock inside get_cooldown;when using libcxx,libcxx‘s shared_mutex dosen't support recursive lock, so causes hangs below: #0 0x00007f788b101a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005636b8ff996f in std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) () #2 0x00005636b8ffad8b in std::__1::__shared_mutex_base::lock_shared() () #3 0x00005636addc524c in doris::Tablet::_has_data_to_cooldown() () #4 0x00005636addc353b in doris::Tablet::pick_cooldown_rowset() () apache#5 0x00005636addc56b9 in doris::Tablet::need_cooldown(long*, unsigned long*) () apache#6 0x00005636adde52f7 in doris::TabletManager::get_cooldown_tablets(std::__1::vector<std::__1::shared_ptr<doris::Tablet>, std::__1::allocator<std::__1::shared_ptr<doris::Tablet> > >*, std::__1::vector<std::__1::shared_ptr<doris::Rowset>, std::__1::allocator<std::__1::shared_ptr<doris::Rowset> > >*, std::__1::function<bool (std::__1::shared_ptr<doris::Tablet> const&)>) () apache#7 0x00005636adb0ed81 in doris::StorageEngine::_cooldown_tasks_producer_callback() () apache#8 0x00005636ae17a253 in doris::Thread::supervise_thread(void*) () apache#9 0x00007f788b0fdea5 in start_thread () from /lib64/libpthread.so.0 apache#10 0x00007f788b916b0d in clone () from /lib64/libc.so.6 review the code, there's no need a rdlock inside TabletManager::get_cooldown_tablets
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
…che#40273) ## Proposed changes Issue Number: close #xxx cur_reader pointer can be null in VFileScanner. can cause BE crash [ RUN ] VfileScannerExcepTest.failure_case AddressSanitizer:DEADLYSIGNAL ================================================================= ==1892247==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5cdbe26eb5f8 bp 0x7ffed6728610 sp 0x7ffed67277a0 T0) ==1892247==The signal is caused by a READ memory access. ==1892247==Hint: address points to the zero page. #0 0x5cdbe26eb5f8 in doris::vectorized::VFileScanner::_get_next_reader() /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:980:9 #1 0x5cdbe26e3512 in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:286:25 #2 0x5cdbe26e2e46 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:252:17 #3 0x5cdbe28e05ce in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 #4 0x5cdbc405a922 in doris::vectorized::VfileScannerExcepTest_failure_case_Test::TestBody() /root/doris/workspace/doris/be/test/vec/exec/vfile_scanner_excep_test.cpp:309:24 apache#5 0x5cdbff5b191a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51eb491a) (BuildId: 21d41a2d207823b9) apache#6 0x5cdbff59f989 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51ea2989) (BuildId: 21d41a2d207823b9) apache#7 0x5cdbff57a9c2 in testing::Test::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7d9c2) (BuildId: 21d41a2d207823b9) apache#8 0x5cdbff57b708 in testing::TestInfo::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7e708) (BuildId: 21d41a2d207823b9) apache#9 0x5cdbff57bec3 in testing::TestSuite::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7eec3) (BuildId: 21d41a2d207823b9) After fix: I20240902 09:11:07.722273 1946048 run_all_tests.cpp:67] init config 1 Note: Google Test filter = VfileScannerE* [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from VfileScannerExceptionTest [ RUN ] VfileScannerExceptionTest.failure_case msg = [INTERNAL_ERROR]cur path: . Failed to create reader for file format: 11 [ OK ] VfileScannerExceptionTest.failure_case (3 ms) [----------] 1 test from VfileScannerExceptionTest (3 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (3 ms total) [ PASSED ] 1 test. === Finished. Gtest output: /root/doris/workspace/doris/be/ut_build_ASAN/gtest_output
HappenLee
pushed a commit
that referenced
this pull request
Oct 24, 2024
…o null pointer (apache#41486) ## Proposed changes ``` /root/doris/be/src/vec/common/pod_array.h:406:13: runtime error: applying non-zero offset 16 to null pointer #0 0x55827f4bbf0c in void doris::vectorized::PODArray<__int128, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 0ul, 0ul>::push_back<__int128&>(__int128&) /root/doris/be/src/vec/common/pod_array.h:406:13 #1 0x55827f4baa40 in doris::Counts<__int128>::increment(__int128) /root/doris/be/src/util/counts.h:160:36 #2 0x55827f4baa40 in doris::vectorized::PercentileState<__int128>::add(__int128, doris::vectorized::PODArray<double, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul> const&, int) /root/doris/be/src/vec/aggregate_functions/aggregate_function_percentile.h:634:27 #3 0x55827f6ca40f in doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionPercentileArray<__int128>, false>::add(char*, doris::vectorized::IColumn const**, long, doris::vectorized::Arena*) const /root/doris/be/src/vec/aggregate_functions/aggregate_function_null.h:324:32 #4 0x55827f6cef0e in doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionPercentileArray<__int128>, false> >::add_batch(unsigned long, char**, unsigned long, doris::vectorized::IColumn const**, doris::vectorized::Arena*, bool) const /root/doris/be/src/vec/aggregate_functions/aggregate_function.h:273:22 apache#5 0x5582a0c41059 in doris::vectorized::AggFnEvaluator::execute_batch_add(doris::vectorized::Block*, unsigned long, char**, doris::vectorized::Arena*, bool) /root/doris/be/src/vec/exprs/vectorized_agg_fn.cpp:241:16 apache#6 0x5582a0c39037 in doris::Status doris::pipeline::AggSinkLocalState::_execute_with_serialized_key_helper<false>(doris::vectorized::Block*)::'lambda'()::operator()() const /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:481:17 apache#7 0x5582a0b51e1e in doris::Status doris::pipeline::AggSinkLocalState::_execute_with_serialized_key_helper<false>(doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:497:13 apache#8 0x5582a0a014e1 in doris::pipeline::AggSinkLocalState::_execute_with_serialized_key(doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:437:16 apache#9 0x5582a0c37811 in doris::pipeline::AggSinkLocalState::Executor<false, false>::execute(doris::pipeline::AggSinkLocalState*, doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.h:61:41 ``` <!--Describe your changes.-->
HappenLee
pushed a commit
that referenced
this pull request
Oct 25, 2024
…dir when index file writer open index (apache#42207) ## Proposed changes Fix UAF as below ``` ==16442==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0008718a0 at pc 0x5586663cea7a bp 0x7f02673d0530 sp 0x7f02673d0528 READ of size 8 at 0x60f0008718a0 thread T490 (CumuCompactionT) #0 0x5586663cea79 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::finish() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:549:63 #1 0x5586663360bc in doris::segment_v2::ScalarColumnWriter::write_inverted_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:640:41 #2 0x5586662ce160 in doris::segment_v2::SegmentWriter::_write_inverted_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1435:9 #3 0x5586662ccb13 in doris::segment_v2::SegmentWriter::finalize_columns_index(unsigned long*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1317:5 #4 0x55866653c074 in doris::VerticalBetaRowsetWriter::_flush_columns(doris::segment_v2::SegmentWriter*, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:127:5 apache#5 0x558666534e3f in doris::VerticalBetaRowsetWriter::flush_columns(bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:153:5 apache#6 0x558665a8d8b2 in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:333:5 apache#7 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#8 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#9 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#10 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#11 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#12 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#13 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#14 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#15 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#16 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#17 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#18 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 apache#19 0x7f0639e0d84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x60f0008718a0 is located 0 bytes inside of 168-byte region [0x60f0008718a0,0x60f000871948) freed by thread T490 (CumuCompactionT) here: #0 0x558663259d9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/cluster0/be/lib/doris_be+0x337a1d9d) (BuildId: fa8094411569cb8d) #1 0x558665a655eb in std::default_delete::operator()(lucene::store::Directory*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #2 0x558665a655eb in std::unique_ptr>::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 #3 0x558665f83dfe in std::pair, std::allocator>> const, std::unique_ptr>>::~pair() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:2379:12 #4 0x558665f83dfe in void std::destroy_at, std::allocator>> const, std::unique_ptr>>>(std::pair, std::allocator>> const, std::unique_ptr>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 apache#5 0x558665f83dfe in void std::allocator_traits, std::allocator>> const, std::unique_ptr>>>>>::destroy, std::allocator>> const, std::unique_ptr>>>(std::allocator, std::allocator>> const, std::unique_ptr>>>>&, std::pair, std::allocator>> const, std::unique_ptr>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4 apache#6 0x558665f83dfe in std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_destroy_node(std::_Rb_tree_node, std::allocator>> const, std::unique_ptr>>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:623:2 apache#7 0x558665f83dfe in std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_drop_node(std::_Rb_tree_node, std::allocator>> const, std::unique_ptr>>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:631:2 apache#8 0x558665f9e9b8 in std::pair, std::allocator>> const, std::unique_ptr>>>, bool> std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_emplace_unique, std::allocator>>, std::unique_ptr>>(std::pair, std::allocator>>&&, std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:2391:6 apache#9 0x558665f86e29 in std::pair, std::allocator>> const, std::unique_ptr>>>, bool> std::map, std::allocator>>, std::unique_ptr>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::emplace, std::allocator>>, std::unique_ptr>>(std::pair, std::allocator>>&&, std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:577:16 apache#10 0x558665f86e29 in doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:60:19 apache#11 0x5586663d0423 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::open_index_directory() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:160:16 apache#12 0x5586663d00f7 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init_bkd_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146:16 apache#13 0x5586663cb2d9 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:107:24 apache#14 0x5586663b1201 in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr>*, doris::segment_v2::InvertedIndexFileWriter*, doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:697:27 apache#15 0x55866632cf8d in doris::segment_v2::ScalarColumnWriter::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483:13 apache#16 0x5586662a5e52 in doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:267:5 apache#17 0x5586662a71b4 in doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector> const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:316:9 apache#18 0x5586662a2be3 in doris::segment_v2::SegmentWriter::init(std::vector> const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:285:5 apache#19 0x5586665333b2 in doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector> const&, bool, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96:17 apache#20 0x558665a8cb9b in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:309:9 apache#21 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#22 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#23 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#24 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#25 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#26 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#27 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#28 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#29 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#30 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#31 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#32 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 previously allocated by thread T490 (CumuCompactionT) here: #0 0x55866325953d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/cluster0/be/lib/doris_be+0x337a153d) (BuildId: fa8094411569cb8d) #1 0x558665fb7d7b in doris::segment_v2::DorisFSDirectoryFactory::getDirectory(std::shared_ptr const&, char const*, bool, lucene::store::LockFactory*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_fs_directory.cpp:797:15 #2 0x558665f86b5c in doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:58:17 #3 0x5586663d0423 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::open_index_directory() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:160:16 #4 0x5586663d00f7 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init_bkd_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146:16 apache#5 0x5586663cb2d9 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:107:24 apache#6 0x5586663b1201 in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr>*, doris::segment_v2::InvertedIndexFileWriter*, doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:697:27 apache#7 0x55866632cf8d in doris::segment_v2::ScalarColumnWriter::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483:13 apache#8 0x5586662a5e52 in doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:267:5 apache#9 0x5586662a71b4 in doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector> const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:316:9 apache#10 0x5586662a2be3 in doris::segment_v2::SegmentWriter::init(std::vector> const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:285:5 apache#11 0x5586665333b2 in doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector> const&, bool, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96:17 apache#12 0x558665a8cb9b in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:309:9 apache#13 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#14 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#15 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#16 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#17 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#18 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#19 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#20 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#21 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#22 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#23 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#24 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
个 -> 隔
那 -> 哪
Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to Doris?
Put an
x
in the boxes that applyChecklist
Put an
x
in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...