使用braft过程中遇到braft deadlock 线程堆栈如下 #309
Comments
cl079074282
changed the title
|
补充一点：我们在log_manager中做了callback业务代码的动作 |
|
在LogManager::append_entries 里面，bthread::execution_queue_execute(_disk_queue, done) 提交磁盘任务，这里是异步的，提交任务后会立即返回，继续向下执行wakeup_all_waiter，这里会释放掉LogManager::_mutex，并返回到NodeImpl::handle_append_entries_request 释放掉NodeImpl::_mutex，应该是不会发生死锁。 |
execution_queue_execute是异步执行的没错，但新的bthread去申请 LogManager::_mutex就发生了死锁，所以肯定还是某个bthread 一直持有mutex没有释放，上面的thread bt也是这样 |
|
bthread::pthread_mutex_lock_impl (mutex=) mutex 变量里有一个owner字段，是持有者的线程id，看一下这个线程在做什么 |
如上 我查看了所有死锁的thread mutex被占用情况 占用mutex的thread 也是lock_wait 所以才一直释放不出来 |
Thread 26 (Thread 0x7fa51effd700 (LWP 6893)):
#0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa51effb6a0, clockid=, expected=0, futex_word=0x7fa524001048) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7fa51effb6a0, clockid=, mutex=0x7fa524000ff8, cond=0x7fa524001020) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7fa524001020, mutex=0x7fa524000ff8, abstime=0x7fa51effb6a0) at pthread_cond_wait.c:656
#3 0x00007fa584d00bf2 in __gthread_cond_timedwait (__cond=0x7fa524001020, __mutex=0x7fa524000ff8, __abs_timeout=0x7fa51effb6a0) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
#4 0x00007fa584d0333d in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa524001020, __lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188
#5 0x00007fa584d027e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa524001020, __lock=..., __atime=...) at /usr/include/c++/9/condition_variable:121
#6 0x00007fa584d01eac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa524001020, __lock=..., __rtime=...) at /usr/include/c++/9/condition_variable:152
#7 0x00007fa584cfe916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa524c5a0b8) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45
#8 0x00007fa584d007e8 in std::__invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#9 0x00007fa584d0073b in std::__invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#10 0x00007fa584d00699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa524c5a0b8) at /usr/include/c++/9/thread:244
#11 0x00007fa584d0061e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa524c5a0b8) at /usr/include/c++/9/thread:251
#12 0x00007fa584d005e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa524c5a0b0) at /usr/include/c++/9/thread:195
#13 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#15 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 25 (Thread 0x7fa51f7fe700 (LWP 6892)):
#0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa51f7fc740, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa51f7fc740, remaining=remaining@entry=0x0) at nanosleep.c:27
#2 0x00007fa58345d9bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32
#3 0x00007fa58507c516 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa524c42250) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734
#4 0x00007fa5850ad5a3 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (__f=@0x7fa524c7af70: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> * const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, __t=@0x7fa524c7af68: 0x7fa524c42250) at /usr/include/c++/9/bits/invoke.h:73
#5 0x00007fa5850a8211 in std::__invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (__fn=@0x7fa524c7af70: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> * const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95
#6 0x00007fa5850a344c in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa524c7af68) at /usr/include/c++/9/thread:244
#7 0x00007fa58509a098 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa524c7af68) at /usr/include/c++/9/thread:251
#8 0x00007fa585098d8f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa524c7af60) at /usr/include/c++/9/thread:195
#9 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#11 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 24 (Thread 0x7fa538ff9700 (LWP 6890)):
#0 0x0000559a49f333ad in spdk_nvme_qpair_process_completions (qpair=qpair@entry=0x20000eca4d18, max_completions=max_completions@entry=0) at nvme_qpair.c:684
#1 0x0000559a49f2e1cc in nvme_pcie_poll_group_process_completions (tgroup=0x559a4b01b390, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair*, void*)>) at nvme_pcie.c:2536
#2 0x0000559a49f36cc9 in nvme_transport_poll_group_process_completions (tgroup=tgroup@entry=0x559a4b01b390, completions_per_qpair=completions_per_qpair@entry=0, disconnected_qpair_cb=disconnected_qpair_cb@entry=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair*, void*)>) at nvme_transport.c:499
#3 0x0000559a49f40c79 in spdk_nvme_poll_group_process_completions (group=, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair*, void*)>) at nvme_poll_group.c:127
#4 0x00007fa584d07dfb in lse::chunkserver::SpdkNamespace::Complete (this=0x559a4b19d610) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:408
#5 0x00007fa585079ddc in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x559a4afcfcb0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1207
#6 0x00007fa585074d24 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x559a4afcfcb0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152
#7 0x00007fa5850ad6be in std::__invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#8 0x00007fa5850a83c8 in std::__invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#9 0x00007fa5850a34f9 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa524c41a28) at /usr/include/c++/9/thread:244
#10 0x00007fa58509a16e in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa524c41a28) at /usr/include/c++/9/thread:251
#11 0x00007fa585098dfb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa524c41a20) at /usr/include/c++/9/thread:195
#12 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#14 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 23 (Thread 0x7fa5397fa700 (LWP 6887)):
#0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa5397f86f0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa5397f86f0, remaining=remaining@entry=0x0) at nanosleep.c:27
#2 0x00007fa58345d9bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32
#3 0x00007fa583a4d178 in bthread_usleep (microseconds=microseconds@entry=1000000) at /home/leo/code/thirdparty/brpc/src/bthread/bthread.cpp:363
#4 0x00007fa583b40862 in brpc::Server::RunUntilAskedToQuit (this=0x7fa524c5b280) at /home/leo/code/thirdparty/brpc/src/brpc/server.cpp:1693
#5 0x00007fa585068779 in lse::chunkserver::Container::execute (this=0x559a4b01b750) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:182
#6 0x00007fa5850858a0 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (__f=@0x559a4afbd7d0: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container * const)) 0x7fa585067818 lse::chunkserver::Container::execute(), __t=@0x559a4afbd7c8: 0x559a4b01b750) at /usr/include/c++/9/bits/invoke.h:73
#7 0x00007fa58507f616 in std::__invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container*> (__fn=@0x559a4afbd7d0: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container * const)) 0x7fa585067818 lse::chunkserver::Container::execute()) at /usr/include/c++/9/bits/invoke.h:95
#8 0x00007fa585077e9e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container*> >::_M_invoke<0ul, 1ul> (this=0x559a4afbd7c8) at /usr/include/c++/9/thread:244
#9 0x00007fa58507430e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x559a4afbd7c8) at /usr/include/c++/9/thread:251
#10 0x00007fa58506f2cf in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x559a4afbd7c0) at /usr/include/c++/9/thread:195
#11 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#13 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 22 (Thread 0x7fa539ffb700 (LWP 6886)):
#0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa539ff96a0, clockid=, expected=0, futex_word=0x7fa530001048) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7fa539ff96a0, clockid=, mutex=0x7fa530000ff8, cond=0x7fa530001020) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7fa530001020, mutex=0x7fa530000ff8, abstime=0x7fa539ff96a0) at pthread_cond_wait.c:656
#3 0x00007fa584d00bf2 in __gthread_cond_timedwait (__cond=0x7fa530001020, __mutex=0x7fa530000ff8, __abs_timeout=0x7fa539ff96a0) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
#4 0x00007fa584d0333d in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa530001020, __lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188
#5 0x00007fa584d027e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa530001020, __lock=..., __atime=...) at /usr/include/c++/9/condition_variable:121
#6 0x00007fa584d01eac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa530001020, __lock=..., __rtime=...) at /usr/include/c++/9/condition_variable:152
#7 0x00007fa584cfe916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa530c5a0b8) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45
#8 0x00007fa584d007e8 in std::__invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#9 0x00007fa584d0073b in std::__invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#10 0x00007fa584d00699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa530c5a0b8) at /usr/include/c++/9/thread:244
#11 0x00007fa584d0061e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa530c5a0b8) at /usr/include/c++/9/thread:251
#12 0x00007fa584d005e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa530c5a0b0) at /usr/include/c++/9/thread:195
#13 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#15 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 21 (Thread 0x7fa53a7fc700 (LWP 6885)):
#0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa53a7fa740, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa53a7fa740, remaining=remaining@entry=0x0) at nanosleep.c:27
#2 0x00007fa58345d9bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32
#3 0x00007fa58507c516 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa530c42250) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734
#4 0x00007fa5850ad5a3 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (__f=@0x7fa530c7af70: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> * const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, __t=@0x7fa530c7af68: 0x7fa530c42250) at /usr/include/c++/9/bits/invoke.h:73
#5 0x00007fa5850a8211 in std::__invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (__fn=@0x7fa530c7af70: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> * const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95
#6 0x00007fa5850a344c in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa530c7af68) at /usr/include/c++/9/thread:244
#7 0x00007fa58509a098 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa530c7af68) at /usr/include/c++/9/thread:251
#8 0x00007fa585098d8f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa530c7af60) at /usr/include/c++/9/thread:195
#9 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#11 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 20 (Thread 0x7fa53b7fe700 (LWP 6883)):
#0 0x0000559a49f33368 in nvme_qpair_resubmit_requests (qpair=qpair@entry=0x2000014a7918, num_requests=num_requests@entry=0) at nvme_qpair.c:1002
#1 0x0000559a49f3343e in spdk_nvme_qpair_process_completions (qpair=qpair@entry=0x2000014a7918, max_completions=max_completions@entry=0) at nvme_qpair.c:735
#2 0x0000559a49f2e1cc in nvme_pcie_poll_group_process_completions (tgroup=0x559a4b01b5b0, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair*, void*)>) at nvme_pcie.c:2536
#3 0x0000559a49f36cc9 in nvme_transport_poll_group_process_completions (tgroup=tgroup@entry=0x559a4b01b5b0, completions_per_qpair=completions_per_qpair@entry=0, disconnected_qpair_cb=disconnected_qpair_cb@entry=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair*, void*)>) at nvme_transport.c:499
#4 0x0000559a49f40c79 in spdk_nvme_poll_group_process_completions (group=, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair*, void*)>) at nvme_poll_group.c:127
#5 0x00007fa584d07dfb in lse::chunkserver::SpdkNamespace::Complete (this=0x559a4b19ca10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:408
#6 0x00007fa585079ddc in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x559a4b020460) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1207
#7 0x00007fa585074d24 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x559a4b020460) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152
#8 0x00007fa5850ad6be in std::__invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#9 0x00007fa5850a83c8 in std::__invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#10 0x00007fa5850a34f9 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa530c41a28) at /usr/include/c++/9/thread:244
#11 0x00007fa58509a16e in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa530c41a28) at /usr/include/c++/9/thread:251
#12 0x00007fa585098dfb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa530c41a20) at /usr/include/c++/9/thread:195
#13 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#15 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 19 (Thread 0x7fa53bfff700 (LWP 6880)):
#0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa53bffd6f0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa53bffd6f0, remaining=remaining@entry=0x0) at nanosleep.c:27
#2 0x00007fa58345d9bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32
#3 0x00007fa583a4d178 in bthread_usleep (microseconds=microseconds@entry=1000000) at /home/leo/code/thirdparty/brpc/src/bthread/bthread.cpp:363
#4 0x00007fa583b40862 in brpc::Server::RunUntilAskedToQuit (this=0x7fa530c5b280) at /home/leo/code/thirdparty/brpc/src/brpc/server.cpp:1693
#5 0x00007fa585068779 in lse::chunkserver::Container::execute (this=0x559a4b01d8f0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:182
#6 0x00007fa5850858a0 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (__f=@0x559a4aff1850: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container * const)) 0x7fa585067818 lse::chunkserver::Container::execute(), __t=@0x559a4aff1848: 0x559a4b01d8f0) at /usr/include/c++/9/bits/invoke.h:73
#7 0x00007fa58507f616 in std::__invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container*> (__fn=@0x559a4aff1850: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container * const)) 0x7fa585067818 lse::chunkserver::Container::execute()) at /usr/include/c++/9/bits/invoke.h:95
#8 0x00007fa585077e9e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container*> >::_M_invoke<0ul, 1ul> (this=0x559a4aff1848) at /usr/include/c++/9/thread:244
#9 0x00007fa58507430e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x559a4aff1848) at /usr/include/c++/9/thread:251
#10 0x00007fa58506f2cf in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x559a4aff1840) at /usr/include/c++/9/thread:195
#11 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#13 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 18 (Thread 0x7fa560ff9700 (LWP 6879)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=__mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa5110d06c0, this=0x7fa5110d06c0) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa5110d06c0) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa5110d0840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656
#9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315
#10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188
#11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164
#12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210
#13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa5523d4630, request=0x7fa50000bb20, response=0x7fa553c2db70, done=0x7fa550dc3b80) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352
#14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa5523d4630, request=0x7fa50000bb20, response=0x7fa553c2db70, done=0x7fa550dc3b80) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467
#15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54003ed70) at /usr/include/c++/9/bits/unique_ptr.h:381
#16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135
#17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141
#18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292
#19 brpc::InputMessenger::OnNewMessages (m=0x7fa50453f3d0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186
#20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa50453f3d0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017
#21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#23 0x0000000000000000 in ?? ()
Thread 17 (Thread 0x7fa5617fa700 (LWP 6878)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=__mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa5116d66c0, this=0x7fa5116d66c0) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa5116d66c0) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa5116d6840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656
#9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315
#10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188
#11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164
#12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210
#13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa500118190, request=0x7fa550255150, response=0x7fa553adab50, done=0x7fa500be5690) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352
#14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa500118190, request=0x7fa550255150, response=0x7fa553adab50, done=0x7fa500be5690) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467
#15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54003ed70) at /usr/include/c++/9/bits/unique_ptr.h:381
#16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135
#17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141
#18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292
#19 brpc::InputMessenger::OnNewMessages (m=0x7fa50453e7d0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186
#20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa50453e7d0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017
#21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#23 0x0000000000000000 in ?? ()
Thread 16 (Thread 0x7fa561ffb700 (LWP 6877)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa55406eca0, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa55406eca0) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=0x7fa55406eca0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa5839b46e6 in butil::Mutex::lock (this=) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa4d4a4a480, this=0x7fa4d4a4a480) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa4d4a4a480) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::FollowerStableClosure::run (this=this@entry=0x7fa55219f760) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2139
#9 0x00007fa5839b4ba4 in braft::FollowerStableClosure::Run (this=0x7fa55219f760) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2123
#10 0x00007fa583995f25 in braft::AppendBatcher::flush (this=this@entry=0x7fa4d4a4a5f0) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:515
#11 0x00007fa58398eb63 in braft::LogManager::disk_thread (meta=0x7fa55407fe50, iter=...) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:619
#12 0x00007fa583a4e070 in bthread::ExecutionQueueBase::_execute (this=0x7fa55406fe50, head=0x7fa5480370c0, high_priority=, niterated=0x0) at /home/leo/code/thirdparty/brpc/src/bthread/execution_queue.cpp:272
#13 0x00007fa583a4fb90 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at /home/leo/code/thirdparty/brpc/src/bthread/execution_queue.cpp:151
#14 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#15 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#16 0x0000000000000000 in ?? ()
Thread 15 (Thread 0x7fa5627fc700 (LWP 6876)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa540059e88, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540059e88) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=__mutex@entry=0x7fa540059e88) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa58398c7b5 in butil::Mutex::lock (this=0x7fa540059e88) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa51c6db560, this=0x7fa51c6db560) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa51c6db560) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::LogManager::set_disk_id (this=0x7fa540059e70, disk_id=...) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:809
#9 0x00007fa58398eb6e in braft::LogManager::disk_thread (meta=0x7fa540059e70, iter=...) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:620
#10 0x00007fa583a4e070 in bthread::ExecutionQueueBase::_execute (this=0x7fa540049e70, head=0x7fa54c0d2230, high_priority=, niterated=0x0) at /home/leo/code/thirdparty/brpc/src/bthread/execution_queue.cpp:272
#11 0x00007fa583a4fb90 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at /home/leo/code/thirdparty/brpc/src/bthread/execution_queue.cpp:151
#12 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#13 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#14 0x0000000000000000 in ?? ()
Thread 14 (Thread 0x7fa562ffd700 (LWP 6875)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=__mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa5113d36c0, this=0x7fa5113d36c0) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa5113d36c0) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa5113d3840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656
#9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315
#10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188
#11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164
#12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210
#13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa551e61300, request=0x7fa500c000e0, response=0x7fa550813ce0, done=0x7fa553110ad0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352
#14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa551e61300, request=0x7fa500c000e0, response=0x7fa550813ce0, done=0x7fa553110ad0) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467
#15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54003ed70) at /usr/include/c++/9/bits/unique_ptr.h:381
#16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135
#17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141
#18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292
#19 brpc::InputMessenger::OnNewMessages (m=0x7fa50453edd0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186
#20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa50453edd0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017
#21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#23 0x0000000000000000 in ?? ()
Thread 13 (Thread 0x7fa5637fe700 (LWP 6874)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=__mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa510bcb6c0, this=0x7fa510bcb6c0) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa510bcb6c0) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa510bcb840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656
#9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315
#10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188
#11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164
#12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210
#13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa50009f1f0, request=0x7fa50000a720, response=0x7fa5513070a0, done=0x7fa550b14a20) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352
#14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa50009f1f0, request=0x7fa50000a720, response=0x7fa5513070a0, done=0x7fa550b14a20) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467
#15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54003ed70) at /usr/include/c++/9/bits/unique_ptr.h:381
#16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135
#17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141
#18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292
#19 brpc::InputMessenger::OnNewMessages (m=0x7fa50453fdd0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186
#20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa50453fdd0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017
#21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#23 0x0000000000000000 in ?? ()
Thread 12 (Thread 0x7fa563fff700 (LWP 6873)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=__mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa4e4ece6c0, this=0x7fa4e4ece6c0) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa4e4ece6c0) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa4e4ece840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656
#9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315
#10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188
#11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164
#12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210
#13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa558c53fa0, request=0x7fa55826f700, response=0x7fa558c54220, done=0x7fa558c54250) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352
#14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa558c53fa0, request=0x7fa55826f700, response=0x7fa558c54220, done=0x7fa558c54250) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467
#15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54b7076b0) at /usr/include/c++/9/bits/unique_ptr.h:381
#16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54b7076b0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135
#17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54b7076b0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141
#18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292
#19 brpc::InputMessenger::OnNewMessages (m=0x7fa53c097030) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186
#20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa53c097030) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017
#21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#23 0x0000000000000000 in ?? ()
Thread 11 (Thread 0x7fa570d59700 (LWP 6872)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa55406eca0, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa55406eca0) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=__mutex@entry=0x7fa55406eca0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa5839a567a in butil::Mutex::lock (this=0x7fa55406eca0) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa51dbf0a10, this=0x7fa51dbf0a10) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa51dbf0a10) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::NodeImpl::handle_append_entries_request (this=0x7fa55406ea00, cntl=cntl@entry=0x7fa53d3ce460, request=request@entry=0x7fa5492fff70, response=response@entry=0x7fa53d546e50, done=done@entry=0x7fa53db80dd0, from_append_entries_cache=from_append_entries_cache@entry=false) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2208
#9 0x00007fa5839c546a in braft::RaftServiceImpl::append_entries (this=, cntl_base=0x7fa53d3ce460, request=0x7fa5492fff70, response=0x7fa53d546e50, done=0x7fa53db80dd0) at /usr/local/include/brpc/closure_guard.h:55
#10 0x00007fa583a08611 in braft::RaftService::CallMethod (this=, method=, controller=, request=, response=, done=) at /home/leo/code/thirdparty/braft/bld/braft/raft.pb.cc:4813
#11 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa5400423f0) at /usr/include/c++/9/bits/unique_ptr.h:381
#12 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa5400423f0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135
#13 0x00007fa583a98214 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa5400423f0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141
#14 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292
#15 brpc::InputMessenger::OnNewMessages (m=0x7fa54ab261a0) at /usr/include/c++/9/bits/unique_ptr.h:286
#16 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa54ab261a0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017
#17 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#18 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#19 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x7fa57155a700 (LWP 6871)):
#0 __lll_lock_wait (futex=futex@entry=0x7fa540059e88, private=0) at lowlevellock.c:52
#1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540059e88) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549
#3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812
#4 pthread_mutex_lock (__mutex=__mutex@entry=0x7fa540059e88) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811
#5 0x00007fa58399072c in butil::Mutex::lock (this=0x7fa540059e88) at /usr/local/include/butil/synchronization/lock.h:69
#6 std::unique_lockbutil::Mutex::lock (this=0x7fa4d4b4b570, this=0x7fa4d4b4b570) at /usr/include/c++/9/bits/unique_lock.h:141
#7 std::unique_lockbutil::Mutex::unique_lock (__m=..., this=0x7fa4d4b4b570) at /usr/include/c++/9/bits/unique_lock.h:71
#8 braft::LogManager::get_term (this=0x7fa540059e70, index=1154599) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:737
#9 0x00007fa5839cbd46 in braft::Replicator::_fill_common_fields (this=0x7fa54e57d640, request=0x7fa54029a4e0, prev_log_index=, is_heartbeat=) at /home/leo/code/thirdparty/braft/src/braft/replicator.cpp:517
#10 0x00007fa5839d14bf in braft::Replicator::_send_entries (this=0x7fa54e57d640) at /home/leo/code/thirdparty/braft/src/braft/replicator.cpp:640
#11 0x00007fa5839d3718 in braft::Replicator::_on_rpc_returned (id=, cntl=0x7fa5090c1210, request=0x7fa54f3ad4a0, response=, rpc_send_time=) at /home/leo/code/thirdparty/braft/src/braft/replicator.cpp:510
#12 0x00007fa5839d4e3c in brpc::internal::FunctionClosure5<unsigned long, brpc::Controller*, braft::AppendEntriesRequest*, braft::AppendEntriesResponse*, long>::Run (this=0x7fa54dc796f0) at /usr/local/include/brpc/callback.h:371
#13 0x00007fa583a7523b in brpc::Controller::EndRPC (this=0x7fa5090c1210, info=...) at /home/leo/code/thirdparty/brpc/src/brpc/controller.cpp:912
#14 0x00007fa583a76a73 in brpc::Controller::OnVersionedRPCReturned (this=this@entry=0x7fa5090c1210, info=..., new_bthread=new_bthread@entry=false, saved_error=saved_error@entry=0) at /home/leo/code/thirdparty/brpc/src/brpc/controller.cpp:695
#15 0x00007fa583a9fe83 in brpc::ControllerPrivateAccessor::OnResponse (this=, saved_error=0, id=...) at /home/leo/code/thirdparty/brpc/src/brpc/details/controller_private_accessor.h:48
#16 brpc::policy::ProcessRpcResponse (msg_base=0x7fa54c0bacf0) at /home/leo/code/thirdparty/brpc/src/brpc/policy/baidu_rpc_protocol.cpp:619
#17 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54c0bacf0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135
#18 0x00007fa583a98214 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54c0bacf0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141
#19 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292
#20 brpc::InputMessenger::OnNewMessages (m=0x7fa54ab28fa0) at /usr/include/c++/9/bits/unique_ptr.h:286
#21 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa54ab28fa0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017
#22 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295
#23 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so
#24 0x0000000000000000 in ?? ()
Thread 9 (Thread 0x7fa571d5b700 (LWP 6870)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fa583a68948 in bthread::futex_wait_private (timeout=, expected=25900, addr1=0x7fa578c5f338) at /home/leo/code/thirdparty/brpc/src/bthread/sys_futex.h:40
#2 bthread::TimerThread::run (this=) at /home/leo/code/thirdparty/brpc/src/bthread/timer_thread.cpp:431
#3 0x00007fa583a6940d in bthread::TimerThread::run_this (arg=) at /home/leo/code/thirdparty/brpc/src/bthread/timer_thread.cpp:120
#4 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#5 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 8 (Thread 0x7fa57255c700 (LWP 6869)):
#0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa57255a6a0, clockid=, expected=0, futex_word=0x7fa578001048) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7fa57255a6a0, clockid=, mutex=0x7fa578000ff8, cond=0x7fa578001020) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7fa578001020, mutex=0x7fa578000ff8, abstime=0x7fa57255a6a0) at pthread_cond_wait.c:656
#3 0x00007fa584d00bf2 in __gthread_cond_timedwait (__cond=0x7fa578001020, __mutex=0x7fa578000ff8, __abs_timeout=0x7fa57255a6a0) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
#4 0x00007fa584d0333d in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa578001020, __lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188
#5 0x00007fa584d027e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa578001020, __lock=..., __atime=...) at /usr/include/c++/9/condition_variable:121
#6 0x00007fa584d01eac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa578001020, __lock=..., __rtime=...) at /usr/include/c++/9/condition_variable:152
#7 0x00007fa584cfe916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa578c5a0a8) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45
#8 0x00007fa584d007e8 in std::__invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#9 0x00007fa584d0073b in std::__invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#10 0x00007fa584d00699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa578c5a0a8) at /usr/include/c++/9/thread:244
#11 0x00007fa584d0061e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa578c5a0a8) at /usr/include/c++/9/thread:251
#12 0x00007fa584d005e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa578c5a0a0) at /usr/include/c++/9/thread:195
#13 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#15 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 7 (Thread 0x7fa573fff700 (LWP 6868)):
#0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa573ffd740, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa573ffd740, remaining=remaining@entry=0x0) at nanosleep.c:27
#2 0x00007fa58345d9bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32
#3 0x00007fa58507c516 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa5781a2240) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734
#4 0x00007fa5850ad5a3 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (__f=@0x7fa578c7af80: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> * const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, __t=@0x7fa578c7af78: 0x7fa5781a2240) at /usr/include/c++/9/bits/invoke.h:73
#5 0x00007fa5850a8211 in std::__invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (__fn=@0x7fa578c7af80: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> * const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95
#6 0x00007fa5850a344c in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa578c7af78) at /usr/include/c++/9/thread:244
#7 0x00007fa58509a098 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa578c7af78) at /usr/include/c++/9/thread:251
#8 0x00007fa585098d8f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa578c7af70) at /usr/include/c++/9/thread:195
#9 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#11 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 6 (Thread 0x7fa57355e700 (LWP 6866)):
#0 0x0000559a49f36cce in nvme_transport_poll_group_process_completions (tgroup=tgroup@entry=0x559a4b0214f0, completions_per_qpair=completions_per_qpair@entry=0, disconnected_qpair_cb=disconnected_qpair_cb@entry=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair*, void*)>) at nvme_transport.c:499
#1 0x0000559a49f40c79 in spdk_nvme_poll_group_process_completions (group=, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair*, void*)>) at nvme_poll_group.c:127
#2 0x00007fa584d07dfb in lse::chunkserver::SpdkNamespace::Complete (this=0x559a4b01f9b0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:408
#3 0x00007fa585079ddc in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x559a4adb9f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1207
#4 0x00007fa585074d24 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x559a4adb9f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152
#5 0x00007fa5850ad6be in std::__invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#6 0x00007fa5850a83c8 in std::__invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#7 0x00007fa5850a34f9 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa578c45c08) at /usr/include/c++/9/thread:244
#8 0x00007fa58509a16e in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa578c45c08) at /usr/include/c++/9/thread:251
#9 0x00007fa585098dfb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa578c45c00) at /usr/include/c++/9/thread:195
#10 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#12 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 5 (Thread 0x7fa580dee700 (LWP 6863)):
#0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa580dec6f0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa580dec6f0, remaining=remaining@entry=0x0) at nanosleep.c:27
#2 0x00007fa58345d9bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32
#3 0x00007fa583a4d178 in bthread_usleep (microseconds=microseconds@entry=1000000) at /home/leo/code/thirdparty/brpc/src/bthread/bthread.cpp:363
#4 0x00007fa583b40862 in brpc::Server::RunUntilAskedToQuit (this=0x7fa578c5b2c0) at /home/leo/code/thirdparty/brpc/src/brpc/server.cpp:1693
#5 0x00007fa585068779 in lse::chunkserver::Container::execute (this=0x559a4b022420) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:182
#6 0x00007fa5850858a0 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (__f=@0x559a4b015460: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container * const)) 0x7fa585067818 lse::chunkserver::Container::execute(), __t=@0x559a4b015458: 0x559a4b022420) at /usr/include/c++/9/bits/invoke.h:73
#7 0x00007fa58507f616 in std::__invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container*> (__fn=@0x559a4b015460: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container * const)) 0x7fa585067818 lse::chunkserver::Container::execute()) at /usr/include/c++/9/bits/invoke.h:95
#8 0x00007fa585077e9e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container*> >::_M_invoke<0ul, 1ul> (this=0x559a4b015458) at /usr/include/c++/9/thread:244
#9 0x00007fa58507430e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x559a4b015458) at /usr/include/c++/9/thread:251
#10 0x00007fa58506f2cf in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x559a4b015450) at /usr/include/c++/9/thread:195
#11 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#13 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 4 (Thread 0x7fa581bf2700 (LWP 6862)):
#0 0x00007fa58374849f in __libc_accept (fd=12, addr=..., len=0x0) at ../sysdeps/unix/sysv/linux/accept.c:26
#1 0x00007fa582c4d8eb in socket_listener () from /lib/librte_telemetry.so.20.0
#2 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#3 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 3 (Thread 0x7fa5823f3700 (LWP 6861)):
#0 0x00007fa5834675ce in epoll_wait (epfd=10, events=0x7fa5823f1800, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007fa58323ff4a in eal_intr_thread_main () from /lib/librte_eal.so.20.0
#2 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#3 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 2 (Thread 0x7fa582bf4700 (LWP 6860)):
#0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa582bf27d0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa582bf27d0, remaining=remaining@entry=0x0) at nanosleep.c:27
#2 0x00007fa58345d9bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32
#3 0x00007fa583a3a76a in bvar::detail::SamplerCollector::run (this=0x559a4adbb440) at /home/leo/code/thirdparty/brpc/src/bvar/detail/sampler.cpp:180
#4 0x00007fa583a3c40d in bvar::detail::SamplerCollector::sampling_thread (arg=) at /home/leo/code/thirdparty/brpc/src/bvar/detail/sampler.cpp:110
#5 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477
#6 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 1 (Thread 0x7fa582c09d80 (LWP 6857)):
#0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffcfb885a80, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7ffcfb885a80, remaining=remaining@entry=0x0) at nanosleep.c:27
#2 0x00007fa58345d9bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32
#3 0x00007fa583a4d178 in bthread_usleep (microseconds=microseconds@entry=1000000) at /home/leo/code/thirdparty/brpc/src/bthread/bthread.cpp:363
#4 0x00007fa583b40862 in brpc::Server::RunUntilAskedToQuit (this=0x7ffcfb885e00) at /home/leo/code/thirdparty/brpc/src/brpc/server.cpp:1693
#5 0x0000559a49f1c8a9 in main (argc=2, argv=0x7ffcfb886478) at /root/jenkins_S/workspace/CI-CKS-Daily/src/main_control.cpp:125
Detaching from program: /home/leo/deploy/CKS/chunkserver, process 6857
[Inferior 1 (process 6857) detached]
经初步分析每个LWP持有mutex的竞争关系如下：
6871->6876
6872->6877
6873->6876
6874->6876
6875->6876
6876->6876
6877->6877
6878->6876
6879->6876
通过上面thread bt和锁的竞争关系 得出一种可能的执行序列导致死锁情况
bthread 1 调度执行NodeImpl::handle_append_entries_request 获得NodeImpl::_mutex
LogManager::append_entries 获得LogManager::_mutex
然后通过bthread::execution_queue_execute(_disk_queue, done) 向_disk_queue提交任务
bthread 2 执行LogManager::disk_thread 调用AppendBatcher::flush 执行FollowerStableClosure::run 去获取NodeImpl::_mutex失败
bthread 3 执行LogManager::set_disk_id 获取LogManager::_mutex失败
至此 其他bthread 获取NodeImpl::_mutex和LogManager::_mutex都会失败 导致braft全面死锁
看来问题是bthread 1同时获得了NodeImpl::_mutex LogManager::_mutex锁 而又不能继续得到调度执行 导致braft死锁
braft可能存在bthread不能被调度的场景吗
期待大神分析解答
The text was updated successfully, but these errors were encountered: