Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve spdb memtable performance #298

Closed
ayulas opened this issue Dec 18, 2022 · 8 comments · Fixed by #299
Closed

Improve spdb memtable performance #298

ayulas opened this issue Dec 18, 2022 · 8 comments · Fixed by #299
Assignees
@Yuval-Ariel @erez-speedb @ayulas
Labels
enhancement New feature or request performance Working on it A feature request that is being actively worked on
Milestone
V2.3.0

Comments

@ayulas
Copy link
Contributor

ayulas commented Dec 18, 2022

Improve spdb memtable performance
ayulas added a commit that referenced this issue Dec 18, 2022
@ayulas
Improve spdb memtable performnace (#298)
b062d89
@ayulas ayulas linked a pull request Dec 18, 2022 that will close this issue
Improve spdb memtable performance (#298) #299
Merged
ayulas added a commit that referenced this issue Dec 18, 2022
@ayulas
Improve spdb memtable performnace (#298)
e5a788b
ayulas added a commit that referenced this issue Dec 19, 2022
@ayulas
Improve spdb memtable performnace (#298)
36bc5c3
ayulas added a commit that referenced this issue Dec 19, 2022
@ayulas
Improve spdb memtable performnace (#298)
704ca30
ayulas added a commit that referenced this issue Dec 20, 2022
@ayulas
Improve spdb memtable performnace (#298)
8a2c585
ayulas added a commit that referenced this issue Dec 20, 2022
@ayulas
Improve spdb memtable performnace (#298)
98a49f2
@erez-speedb
Copy link

Got seg fault during large obj tests.
The test log shows
#long op: 1630358 micros
long op: 1845646 micros
long op: 1630375 micros
long op: 1845607 micros
long op: 1630371 micros
long op: 1630053 micros
long op: 1630390 micros
long op: 1630379 micros
long op: 1630344 micros
long op: 1630396 micros
long op: 1630370 micros
long op: 1630366 micros
long op: 1630376 micros
long op: 1630374 micros
long op: 1630358 micros
long op: 1630387 micros
long op: 1630376 micros

Received signal 11 (Segmentation fault)
#0 /home/ubuntu/speedb/db_bench(+0x519196) [0x55ff1bfd5196] ?? ??:0
#1 /home/ubuntu/speedb/db_bench(+0x51a172) [0x55ff1bfd6172] ?? ??:0
#2 /home/ubuntu/speedb/db_bench(+0x2486ed) [0x55ff1bd046ed] ?? ??:0
#3 /home/ubuntu/speedb/db_bench(+0x2d1aa0) [0x55ff1bd8daa0] ?? ??:0
#4 /home/ubuntu/speedb/db_bench(+0x2d2d57) [0x55ff1bd8ed57] ?? ??:0
#5 /home/ubuntu/speedb/db_bench(+0x2c6560) [0x55ff1bd82560] ?? ??:0
#6 /home/ubuntu/speedb/db_bench(+0x2c6cd2) [0x55ff1bd82cd2] ?? ??:0
#7 /home/ubuntu/speedb/db_bench(+0x2c8ecf) [0x55ff1bd84ecf] ?? ??:0
#8 /home/ubuntu/speedb/db_bench(+0x1e5d5c) [0x55ff1bca1d5c] ?? ??:0
#9 /home/ubuntu/speedb/db_bench(+0x1e8368) [0x55ff1bca4368] ?? ??:0
#10 /home/ubuntu/speedb/db_bench(+0x1e9f14) [0x55ff1bca5f14] ?? ??:0
#11 /home/ubuntu/speedb/db_bench(+0x1ea24a) [0x55ff1bca624a] ?? ??:0
#12 /home/ubuntu/speedb/db_bench(+0x1ea3d4) [0x55ff1bca63d4] ?? ??:0
#13 /home/ubuntu/speedb/db_bench(+0x106f67) [0x55ff1bbc2f67] ?? ??:0
#14 /home/ubuntu/speedb/db_bench(+0xf2bb6) [0x55ff1bbaebb6] ?? ??:0
#15 /home/ubuntu/speedb/db_bench(+0x2f5309) [0x55ff1bdb1309] ?? ??:0
#16 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6766457609] ?? ??:0
#17 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6765f05133] ?? ??:0
bash: line 1: 13576 Segmentation fault (core dumped)

@erez-speedb
Copy link

from gdb
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
--Type for more, q to quit, c to continue without paging--
Core was generated by `/home/ubuntu/speedb/db_bench --compression_type=None -db=/data/ -num=200000000'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 rocksdb::(anonymous namespace)::SpdbVector::Add (key=0x7f665480ee88 "\030", this=0x7f65d45bcd30) at /usr/include/c++/9/bits/stl_vector.h:1040
1040 operator[](size_type __n) _GLIBCXX_NOEXCEPT
[Current thread is 1 (Thread 0x7f6734ff9700 (LWP 13619))]
(gdb) bt
#0 rocksdb::(anonymous namespace)::SpdbVector::Add (key=0x7f665480ee88 "\030", this=0x7f65d45bcd30) at /usr/include/c++/9/bits/stl_vector.h:1040
#1 rocksdb::(anonymous namespace)::SpdbVectorContainer::InternalInsert (this=this@entry=0x7f673004b920, key=key@entry=0x7f665480ee88 "\030") at plugin/speedb/memtable/hash_spd_rep.cc:272
#2 0x000055ff1bfd6172 in rocksdb::(anonymous namespace)::SpdbVectorContainer::Insert (key=0x7f665480ee88 "\030", this=0x7f673004b920) at plugin/speedb/memtable/hash_spd_rep.cc:280
#3 rocksdb::(anonymous namespace)::HashSpdRep::InsertKey (this=0x7f662c047100, handle=0x7f665480ee80) at plugin/speedb/memtable/hash_spd_rep.cc:549
#4 0x000055ff1bd046ed in rocksdb::MemTable::Add (this=this@entry=0x7f66a55a4930, s=1010856646, type=type@entry=rocksdb::kTypeValue, key=..., value=..., kv_prot_info=0x0, allow_concurrent=true,
post_process_info=0x7f65d0036ba8, hint=0x0) at /usr/include/c++/9/ext/atomicity.h:69
#5 0x000055ff1bd8daa0 in rocksdb::(anonymous namespace)::MemTableInserter::PutCFImpl (this=0x7f6734ff73f0, column_family_id=0, key=..., value=..., value_type=, kv_prot_info=0x0)
at /usr/include/c++/9/bits/stl_tree.h:273
#6 0x000055ff1bd8ed57 in rocksdb::(anonymous namespace)::MemTableInserter::PutCF (this=0x7f6734ff73f0, column_family_id=0, key=..., value=...) at db/write_batch.cc:2108
#7 0x000055ff1bd82560 in rocksdb::WriteBatchInternal::Iterate (wb=0x7f6734ff85a0, handler=0x7f6734ff73f0, begin=12, end=1032) at db/write_batch.cc:537
#8 0x000055ff1bd82cd2 in rocksdb::WriteBatch::Iterate (this=, handler=handler@entry=0x7f6734ff73f0) at db/write_batch.cc:476
#9 0x000055ff1bd84ecf in rocksdb::WriteBatchInternal::InsertInto (writer=writer@entry=0x7f6734ff77a0, sequence=, memtables=memtables@entry=0x7f6734ff7760,
flush_scheduler=flush_scheduler@entry=0x55ff1d95f120, trim_history_scheduler=trim_history_scheduler@entry=0x55ff1d95f128, ignore_missing_column_families=false, log_number=0, db=0x55ff1d95dfc0,
concurrent_memtable_writes=true, seq_per_batch=false, batch_cnt=0, batch_per_txn=true, hint_per_batch=false) at db/write_batch.cc:2885
#10 0x000055ff1bca1d5c in rocksdb::DBImpl::PipelinedWriteImpl (this=0x55ff1d95dfc0, write_options=..., my_batch=, callback=, log_used=,
log_ref=, disable_memtable=, seq_used=) at db/db_impl/db_impl_write.cc:789
#11 0x000055ff1bca4368 in rocksdb::DBImpl::WriteImpl (this=0x55ff1d95dfc0, write_options=..., my_batch=0x7f6734ff85a0, callback=0x0, log_used=0x0, log_ref=0, disable_memtable=false, seq_used=0x0,
batch_cnt=0, pre_release_callback=0x0, post_memtable_callback=0x0) at db/db_impl/db_impl_write.cc:283
#12 0x000055ff1bca5f14 in rocksdb::DBImpl::Write (this=0x55ff1d95dfc0, write_options=..., my_batch=0x7f6734ff85a0) at db/db_impl/db_impl_write.cc:126
#13 0x000055ff1bca624a in rocksdb::DB::Put (this=0x55ff1d95dfc0, opt=..., column_family=0x55ff1daef2d0, key=..., value=...) at db/db_impl/db_impl_write.cc:2257
#14 0x000055ff1bca63d4 in rocksdb::DBImpl::Put (this=this@entry=0x55ff1d95dfc0, o=..., column_family=0x55ff1daef2d0, key=..., val=...) at db/db_impl/db_impl_write.cc:28
#15 0x000055ff1bbc2f67 in rocksdb::DB::Put (value=..., key=..., options=..., this=0x55ff1d95dfc0) at ./include/rocksdb/db.h:403
#16 rocksdb::Benchmark::SeekRandomWriteRandom (this=0x55ff1d957cf0, thread=0x55ff23972530) at tools/db_bench_tool.cc:7968
#17 0x000055ff1bbaebb6 in rocksdb::Benchmark::ThreadBody (v=0x55ff1dc20a40) at tools/db_bench_tool.cc:4138
#18 0x000055ff1bdb1309 in rocksdb::(anonymous namespace)::StartThreadWrapper (arg=0x55ff1db9a860) at env/env_posix.cc:461
#19 0x00007f6766457609 in start_thread (arg=) at pthread_create.c:477
#20 0x00007f6765f05133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

@erez-speedb
Copy link

Running test with small obj, 4 threads show major improvement in write but major degradation with seeks.
https://admin.speedb.io/performance?items=vgfoZlxfwtWKIpooxsHi&items=V5L3IaL6MKbbo8bEvMQc&colors=%23F06292&colors=%2300796b

@Guyme Guyme modified the milestones: v2.2.0, V2.3.0 Dec 22, 2022
@erez-speedb
Copy link

Testing new hash 5be5400
Small obj
https://admin.speedb.io/performance?items=7dCfIWJQa7aVt8yutCsU&items=1OOiaGLj012pDn0zUeXg&colors=%23F06292&colors=%2300796b

Small obj 4 threads shows major improvements.
https://admin.speedb.io/performance?items=OHZys0wyTuvcmLCxiKnf&items=V5L3IaL6MKbbo8bEvMQc&colors=%23F06292&colors=%2300796b

@Yuval-Ariel please run your test

@Yuval-Ariel
Copy link
Contributor

@ayulas , plz update the PR with these changes and assign a reviewer

@Yuval-Ariel Yuval-Ariel removed their assignment Jan 8, 2023
ayulas added a commit that referenced this issue Jan 15, 2023
@ayulas
Improve spdb memtable performnace (#298)
01bb530
@Guyme
Copy link

Guyme commented Jan 16, 2023

@Yuval-Ariel - Please test
@erez-speedb - Please test

ayulas added a commit that referenced this issue Jan 16, 2023
@ayulas
Improve spdb memtable performnace (#298)
831e18a
ayulas added a commit that referenced this issue Jan 17, 2023
@ayulas
Improve spdb memtable performnace (#298)
62efec5
@Guyme Guyme added the enhancement New feature or request label Jan 18, 2023
@Guyme Guyme added the Working on it A feature request that is being actively worked on label Jan 18, 2023
@erez-speedb
Copy link

Results for
root@5088f598036d:~/speedb# git log -1 origin/298-improve-spdb-memtable-perf
commit 62efec5 (origin/298-improve-spdb-memtable-perf)
Small DB
https://admin.speedb.io/performance?items=J20KlYqf6OnCEbCB4GVQ&items=MTnDOa9oyhGa33CyQgci&colors=%23F06292&colors=%2300796b
Small Obj here there are improvements with seeks

https://admin.speedb.io/performance?items=zBkp6lRINrcLODRvDp5b&items=zmqpQ9ogatg10p1lqXkC&colors=%23F06292&colors=%2300796b

Large obj
https://admin.speedb.io/performance?items=n1Fh8zykdfRlz5gkqZr1&items=DQbr0ntUjYJDz3RLExzO&colors=%23F06292&colors=%2300796b

Huge DB
https://admin.speedb.io/performance?items=aPugftlYzT0xNTrrCrUT&items=hKWqp0Aknv7eR3CDlnnG&colors=%23F06292&colors=%2300796b

The most gain is with the 4 threads test
https://admin.speedb.io/performance?items=du9shO7S2wG615D7Xn55&items=QTiAOcC8otvMijbozkHu&colors=%23F06292&colors=%2300796b

Passed perf tests.

ayulas added a commit that referenced this issue Jan 23, 2023
@ayulas
Improve spdb memtable performnace (#298)
a00bfae
ayulas added a commit that referenced this issue Jan 24, 2023
@ayulas
Improve spdb memtable performnace (#298)
b4b3aa8
ayulas added a commit that referenced this issue Jan 31, 2023
@ayulas
Improve spdb memtable performnace (#298)
2ff2ce1
ayulas added a commit that referenced this issue Jan 31, 2023
@ayulas
Improve spdb memtable performnace (#298)
ca71de9
@erez-speedb
Copy link

Tested on hash: ca71de9
Performance test passed

Small DB
https://admin.speedb.io/performance?items=gxJV87JBKf3gY14kBMh2&items=JL3z5Hj1Lo4KUDQ[…]f3tRld0hY2&colors=%23F06292&colors=%2342a5f5&colors=%2300796b
Small Obj
https://admin.speedb.io/performance?items=gP5xP9cIT01eb6ALogzc&items=Bc3j4cCnRfqvzPp[…]Q0f995TfC8&colors=%23F06292&colors=%2303a9f4&colors=%2300796b
Large Obj
https://admin.speedb.io/performance?items=y3BkugLStB855HxPRApy&items=q6Mlo7XawmREan8[…]C8l8P1H3ZH&colors=%23F06292&colors=%2342a5f5&colors=%2300796b

The extra memory is in all tests.

ayulas added a commit that referenced this issue Feb 1, 2023
@ayulas
Spdb memtable: improve performance (#298)
8c37223 
Reduce lock access between switch writes vectors
Avoid unnecessary lock between read and writes
Decrease the writes vectors numbers
Yuval-Ariel pushed a commit that referenced this issue Feb 1, 2023
@ayulas @Yuval-Ariel
Spdb memtable: improve performance (#298)
1247fca 
Reduce lock access between switch writes vectors
Avoid unnecessary lock between read and writes
Decrease the writes vectors numbers
ayulas added a commit that referenced this issue Feb 26, 2023
@ayulas
Spdb memtable: improve performance (#298)
22bcc3d 
Reduce lock access between switch writes vectors
Avoid unnecessary lock between read and writes
Decrease the writes vectors numbers
Yuval-Ariel pushed a commit that referenced this issue May 1, 2023
@ayulas @Yuval-Ariel
Spdb memtable: improve performance (#298)
79c20ea 
Reduce lock access between switch writes vectors
Avoid unnecessary lock between read and writes
Decrease the writes vectors numbers
Yuval-Ariel pushed a commit that referenced this issue May 4, 2023
@ayulas @Yuval-Ariel
Spdb memtable: improve performance (#298)
ed8320d 
Reduce lock access between switch writes vectors
Avoid unnecessary lock between read and writes
Decrease the writes vectors numbers
udi-speedb pushed a commit that referenced this issue Nov 13, 2023
@ayulas @udi-speedb
Spdb memtable: improve performance (#298)
3e77609 
Reduce lock access between switch writes vectors
Avoid unnecessary lock between read and writes
Decrease the writes vectors numbers
udi-speedb pushed a commit that referenced this issue Nov 15, 2023
@ayulas @udi-speedb
Spdb memtable: improve performance (#298)
6213b63 
Reduce lock access between switch writes vectors
Avoid unnecessary lock between read and writes
Decrease the writes vectors numbers
udi-speedb pushed a commit that referenced this issue Dec 4, 2023
@ayulas @udi-speedb
Spdb memtable: improve performance (#298)
cedb39f 
Reduce lock access between switch writes vectors
Avoid unnecessary lock between read and writes
Decrease the writes vectors numbers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Yuval-Ariel Yuval-Ariel

@erez-speedb erez-speedb

@ayulas ayulas

Labels
enhancement New feature or request performance Working on it A feature request that is being actively worked on
Projects
None yet
Milestone
V2.3.0
Development

Successfully merging a pull request may close this issue.

Improve spdb memtable performance (#298) speedb-io/speedb
5 participants
@Yuval-Ariel @erez-speedb @bosmatt @ayulas @Guyme