Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](cache) File cache async init #39036

Merged
merged 9 commits into from
Aug 15, 2024

Conversation

suxiaogang223
Copy link
Contributor

Proposed changes

Do load_cache_info_into_memory() asynchronously in a background thread in LRUFileCache::initialize().
When the cache is not ready, LRUFileCache::get_or_set() will return the FileBlock which state is SKIP_CACHE.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/io/cache/block/block_lru_file_cache.cpp Outdated Show resolved Hide resolved
be/src/io/cache/block/block_lru_file_cache.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/io/cache/block/block_lru_file_cache.cpp Outdated Show resolved Hide resolved
be/src/io/cache/block/block_lru_file_cache.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

github-actions bot commented Aug 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@suxiaogang223
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Aug 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@suxiaogang223
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Aug 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@suxiaogang223
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@suxiaogang223
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@morningman morningman merged commit 0680c8d into apache:branch-2.1 Aug 15, 2024
19 of 21 checks passed
@suxiaogang223 suxiaogang223 deleted the file_cache_async_init branch August 15, 2024 08:47
morningman pushed a commit to morningman/doris that referenced this pull request Aug 18, 2024
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
3 tasks
yiguolei pushed a commit that referenced this pull request Oct 25, 2024
come from: #39036

Crash stack:
```
(gdb) bt
#0  0x00007ff5c8c6c387 in raise () from /lib64/libc.so.6
#1  0x00007ff5c8c6da78 in abort () from /lib64/libc.so.6
#2  0x0000561eb0a5e38a in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x0000561eb0a5caf6 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#4  0x0000561eb0a5cb61 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
#5  0x0000561ea4028552 in doris::io::LRUFileCache::~LRUFileCache (this=0x7ff540fed000) at /root/be/src/io/cache/block/block_lru_file_cache.h:62
#6  0x0000561ea402857e in doris::io::LRUFileCache::~LRUFileCache (this=0x33688) at /root/be/src/io/cache/block/block_lru_file_cache.h:54
#7  0x0000561ea4251cd2 in std::default_delete<doris::io::IFileCache>::operator() (this=0x7ff54101d000, __ptr=0x33688)
    at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85
#8  std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >::~unique_ptr (this=0x7ff54101d000)
    at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361
#9  std::destroy_at<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> > > (__location=0x7ff54101d000)
    at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88
#10 std::_Destroy<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> > > (__pointer=0x7ff54101d000)
    at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138
#11 std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >*> (__first=0x7ff54101d000, 
    __last=0x7ff54101d008) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152
#12 std::_Destroy<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >*> (__first=<optimized out>, __last=0x7ff54101d008)
    at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184
#13 std::_Destroy<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >*, std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> > > (__first=<optimized out>, __last=0x7ff54101d008)
    at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746
#14 std::vector<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> >, std::allocator<std::unique_ptr<doris::io::IFileCache, std::default_delete<doris::io::IFileCache> > > >::~vector (this=0x7ff596a24ba8)
    at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680
#15 doris::io::FileCacheFactory::~FileCacheFactory (this=0x7ff596a24b80) at /root/be/src/io/cache/block/block_file_cache_factory.h:42
#16 doris::ExecEnv::destroy (this=0x561eb130b800 <doris::ExecEnv::GetInstance()::s_exec_env>) at /root/be/src/runtime/exec_env_init.cpp:651
#17 0x0000561ea35c3fe3 in main (argc=<optimized out>, argv=<optimized out>) at /root/be/src/service/doris_main.cpp:628
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants