[Bug] core dump using vectorized schema change

[Gabriel39]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master

What's Wrong?

================================================================
==164133==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f4323ab99a4 at pc 0x5624d25cd871 bp 0x7f4323aaf450 sp 0x7f4323aaf440
READ of size 1 at 0x7f4323ab99a4 thread T342 (TaskWorkerPool.)
#0 0x5624d25cd870 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/incubator-doris/be/src/olap/rowset/beta_rowset_reader.cpp:201
#1 0x5624d19f9b0c in doris::VSchemaChangeDirectly::_inner_process(std::shared_ptrdoris::RowsetReader, doris::RowsetWriter*, std::shared_ptrdoris::Tablet, std::shared_ptrdoris::Tablet) /root/incubator-doris/be/src/olap/schema_change.cpp:1278
#2 0x5624d1a21bad in doris::SchemaChange::process(std::shared_ptrdoris::RowsetReader, doris::RowsetWriter*, std::shared_ptrdoris::Tablet, std::shared_ptrdoris::Tablet) /root/incubator-doris/be/src/olap/schema_change.h:108
#3 0x5624d1a0cfca in doris::SchemaChangeHandler::_convert_historical_rowsets(doris::SchemaChangeHandler::SchemaChangeParams const&) /root/incubator-doris/be/src/olap/schema_change.cpp:2073
#4 0x5624d1a08f00 in doris::SchemaChangeHandler::_do_process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) /root/incubator-doris/be/src/olap/schema_change.cpp:1946
#5 0x5624d1a0470e in doris::SchemaChangeHandler::process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) /root/incubator-doris/be/src/olap/schema_change.cpp:1688
#6 0x5624d255bfe6 in doris::EngineAlterTabletTask::execute() /root/incubator-doris/be/src/olap/task/engine_alter_tablet_task.cpp:40
#7 0x5624d17b2fcd in doris::StorageEngine::execute_task(doris::EngineTask*) /root/incubator-doris/be/src/olap/storage_engine.cpp:1058
#8 0x5624d184af07 in doris::TaskWorkerPool::_alter_tablet(doris::TAgentTaskRequest const&, long, doris::TTaskType::type, doris::TFinishTaskRequest*) /root/incubator-doris/be/src/agent/task_worker_pool.cpp:538
#9 0x5624d184a2d3 in doris::TaskWorkerPool::_alter_tablet_worker_thread_callback() /root/incubator-doris/be/src/agent/task_worker_pool.cpp:496
#10 0x5624d189180d in void std::__invoke_impl<void, void (doris::TaskWorkerPool::&)(), doris::TaskWorkerPool&>(std::__invoke_memfun_deref, void (doris::TaskWorkerPool::&)(), doris::TaskWorkerPool&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
#11 0x5624d1890fa6 in std::enable_if<is_invocable_r_v<void, void (doris::TaskWorkerPool::&)(), doris::TaskWorkerPool&>, void>::type std::__invoke_r<void, void (doris::TaskWorkerPool::&)(), doris::TaskWorkerPool&>(void (doris::TaskWorkerPool::&)(), doris::TaskWorkerPool&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
#12 0x5624d188ff0d in void std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:570
#13 0x5624d188d978 in void std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>::operator()<>() /var/local/ldb-toolchain/include/c++/11/functional:629
#14 0x5624d188b4bf in void std::__invoke_impl<void, std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&>(std::__invoke_other, std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
#15 0x5624d1886ae6 in std::enable_if<is_invocable_r_v<void, std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&>, void>::type std::__invoke_r<void, std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&>(std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
#16 0x5624d1880a8f in std::_Function_handler<void (), std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
#17 0x5624d26accb9 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
#18 0x5624d2edc915 in doris::FunctionRunnable::run() /root/incubator-doris/be/src/util/threadpool.cpp:45
#19 0x5624d2ed7c72 in doris::ThreadPool::dispatch_thread() /root/incubator-doris/be/src/util/threadpool.cpp:540
#20 0x5624d2ef8ea9 in void std::__invoke_impl<void, void (doris::ThreadPool::&)(), doris::ThreadPool&>(std::__invoke_memfun_deref, void (doris::ThreadPool::&)(), doris::ThreadPool&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
#21 0x5624d2ef8748 in std::__invoke_result<void (doris::ThreadPool::&)(), doris::ThreadPool&>::type std::__invoke<void (doris::ThreadPool::&)(), doris::ThreadPool&>(void (doris::ThreadPool::&)(), doris::ThreadPool&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
#22 0x5624d2ef7ae7 in void std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
#23 0x5624d2ef65ea in void std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
#24 0x5624d2ef3151 in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
#25 0x5624d2ef06ad in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&>(std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
#26 0x5624d2eeb9b0 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
#27 0x5624d26accb9 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
#28 0x5624d2eb75f9 in doris::Thread::supervise_thread(void*) /root/incubator-doris/be/src/util/thread.cpp:425
#29 0x7f43c0effea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
#30 0x7f43c1212b0c in clone (/lib64/libc.so.6+0xfeb0c)

Address 0x7f4323ab99a4 is located in stack of thread T342 (TaskWorkerPool.) at offset 2468 in frame
#0 0x5624d1a049d7 in doris::SchemaChangeHandler::_do_process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) /root/incubator-doris/be/src/olap/schema_change.cpp:1702

This frame has 85 object(s):
[48, 56) '__for_begin' (line 1753)
[80, 88) '__for_end' (line 1753)
[112, 120) 'base_tablet_lock' (line 1771)
[144, 152) 'new_tablet_lock' (line 1772)
[176, 184) 'base_tablet_wlock' (line 1773)
[208, 216) 'new_tablet_wlock' (line 1774)
[240, 248) '__for_begin' (line 1826)
[272, 280) '__for_end' (line 1826)
[304, 312) '__for_begin' (line 1847)
[336, 344) '__for_end' (line 1847)
[368, 376) 'end_version' (line 1853)
[400, 408) '__for_begin' (line 1854)
[432, 440) '__for_end' (line 1854)
[464, 472) '__for_begin' (line 1883)
[496, 504) '__for_end' (line 1883)
[528, 536) '__for_begin' (line 1906)
[560, 568) '__for_end' (line 1906)
[592, 600) 'wrlock' (line 1943)
[624, 632) ''
[656, 664) 'wrlock' (line 1948)
[688, 696) ''
[720, 728) 'new_wlock' (line 1955)
[752, 768) 'base_tablet' (line 1704)
[784, 800) ''
[816, 832) 'new_tablet' (line 1712)
[848, 864) ''
[880, 896) ''
[912, 928) ''
[944, 960) ''
[976, 992) 'base_migration_rlock' (line 1735)
[1008, 1024) 'new_migration_rlock' (line 1739)
[1040, 1056) 'base_tablet_schema' (line 1749)
[1072, 1088) ''
[1104, 1120) 'max_rowset' (line 1799)
[1136, 1152) ''
[1168, 1184) ''
[1200, 1216) ''
[1232, 1248) ''
[1264, 1280) ''
[1296, 1312) ''
[1328, 1344) ''
[1360, 1376) ''
[1392, 1408) ''
[1424, 1440) ''
[1456, 1472) ''
[1488, 1504) ''
[1520, 1536) ''
[1552, 1568) ''
[1584, 1608) 'versions_to_be_changed' (line 1744)
[1648, 1672) 'rs_readers' (line 1745)
[1712, 1736) 'return_columns' (line 1748)
[1776, 1800) 'rowsets_to_delete' (line 1818)
[1840, 1864) 'version_rowsets' (line 1819)
[1904, 1928) 'empty_vec' (line 1840)
[1968, 2000) 'delete_handler' (line 1747)
[2032, 2216) 'sc_params' (line 1897)
[2288, 2504) 'reader_context' (line 1785) <== Memory access at offset 2468 is inside this variable
[2576, 2856) 'reader_params' (line 1759)
[2928, 2960) ''
[2992, 3024) ''
[3056, 3088) ''
[3120, 3152) ''
[3184, 3216) ''
[3248, 3280) ''
[3312, 3344) ''
[3376, 3408) ''
[3440, 3552) 'item' (line 1906)
[3584, 3696) 'mv_param' (line 1907)
[3728, 3872) ''
[3936, 4136) ''
[4208, 5816) 'reader' (line 1758)
[5952, 8000) 'res' (line 1703)
[8128, 10176) ''
[10304, 12352) ''
[12480, 14528) ''
[14656, 16704) ''
[16832, 18880) ''
[19008, 21056) ''
[21184, 23232) '_s' (line 1870)
[23360, 25408) ''
[25536, 27584) ''
[27712, 29760) ''
[29888, 31936) ''
[32064, 34112) ''
[34240, 36288) ''
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions are supported)
Thread T342 (TaskWorkerPool.) created by T0 here:
#0 0x5624d0c42061 in __interceptor_pthread_create (/mnt/disk1/gabriel/Workspace/dev/doris/output/be/lib/doris_be+0xb039061)
#1 0x5624d2eb68c1 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&, unsigned long, scoped_refptrdoris::Thread) /root/incubator-doris/be/src/util/thread.cpp:379
#2 0x5624d2ee0f8b in doris::Status doris::Thread::create<void (doris::ThreadPool::)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptrdoris::Thread) /root/incubator-doris/be/src/util/thread.h:54
#3 0x5624d2ed9492 in doris::ThreadPool::create_thread() /root/incubator-doris/be/src/util/threadpool.cpp:603
#4 0x5624d2ed2f29 in doris::ThreadPool::init() /root/incubator-doris/be/src/util/threadpool.cpp:266
#5 0x5624d2ecf768 in doris::ThreadPoolBuilder::build(std::unique_ptr<doris::ThreadPool, std::default_deletedoris::ThreadPool >) const /root/incubator-doris/be/src/util/threadpool.cpp:77
#6 0x5624d1843ee9 in doris::TaskWorkerPool::start() /root/incubator-doris/be/src/agent/task_worker_pool.cpp:215
#7 0x5624d2beaee8 in doris::AgentServer::AgentServer(doris::ExecEnv*, doris::TMasterInfo const&) /root/incubator-doris/be/src/agent/agent_server.cpp:77
#8 0x5624d2bd078b in doris::BackendService::BackendService(doris::ExecEnv*) /root/incubator-doris/be/src/service/backend_service.cpp:69
#9 0x5624d2bd08ea in doris::BackendService::create_service(doris::ExecEnv*, int, doris::ThriftServer**) /root/incubator-doris/be/src/service/backend_service.cpp:72
#10 0x5624d0ce882b in main /root/incubator-doris/be/src/service/doris_main.cpp:408
#11 0x7f43c1136554 in __libc_start_main (/lib64/libc.so.6+0x22554)

SUMMARY: AddressSanitizer: stack-use-after-scope /root/incubator-doris/be/src/olap/rowset/beta_rowset_reader.cpp:201 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*)
Shadow bytes around the buggy address:
0x0fe8e474f2e0: f8 f2 f2 f2 f2 f2 f8 f8 f8 f2 f2 f2 f2 f2 f8 f8
0x0fe8e474f2f0: f8 f2 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00
0x0fe8e474f300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0fe8e474f310: 00 00 00 00 00 f2 f2 f2 f2 f2 f2 f2 f2 f2 f8 f8
0x0fe8e474f320: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
=>0x0fe8e474f330: f8 f8 f8 f8[f8]f8 f8 f8 f8 f2 f2 f2 f2 f2 f2 f2
0x0fe8e474f340: f2 f2 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0fe8e474f350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0fe8e474f360: 00 00 00 00 00 f2 f2 f2 f2 f2 f2 f2 f2 f2 f8 f8
0x0fe8e474f370: f8 f8 f2 f2 f2 f2 f8 f8 f8 f8 f2 f2 f2 f2 f8 f8
0x0fe8e474f380: f8 f8 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==164133==ABORTING

What You Expected?

fix

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct