Snapshot optimization #35
Comments
|
An experimental implementation is in https://github.com/speedb-io/speedb/tree/exp/snapshot-reuse
|
|
New WIP branch that tries to tackle these issues, but is currently incomplete and experiences data races between reference subtraction and |
Guyme
added
the
Future
ofriedma
pushed a commit
to ofriedma/speedb
that referenced
this issue
Jun 8, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature adds new atomic_shared_ptr that used mutex in order to synchronize the shared_ptr. In order to cache the snapshots, there is last_snapshot_ (atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jun 8, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature adds new atomic_shared_ptr that used mutex in order to synchronize the shared_ptr. In order to cache the snapshots, there is last_snapshot_ (atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jun 11, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature adds new atomic_shared_ptr that used mutex in order to synchronize the shared_ptr. In order to cache the snapshots, there is last_snapshot_ (atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jun 14, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jun 14, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jun 14, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
bosmatt
added
Working on it
ofriedma
pushed a commit
to ofriedma/speedb
that referenced
this issue
Jul 5, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 6, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 6, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 6, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 6, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 19, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 19, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 19, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 20, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 23, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 23, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
ofriedma
pushed a commit
that referenced
this issue
Jul 23, 2023
Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
|
@erez-speedb - Please run standard tests. |
|
This Feature must have folly library installed. How to install folly on Ubuntu: sudo apt install libssl-dev libfmt-dev
git clone https://github.com/facebook/folly
cd folly
sudo ./build/fbcode_builder/getdeps.py install-system-deps --recursive
mkdir build_
cd build_
cmake .. -DBUILD_SHARED_LIBS=ON
make -j $(nproc) installIf there is an error with the compile process above that is related with fmt, sed -i "s/format_to/fmt::format_to/g" /usr/include/fmt/chrono.hTo compile speedb with this enhancement please compile it using cmake: git clone https://github.com/speedb-io/speedb
cd speedb
mkdir build
cd build
cmake .. -DWITH_SNAP_OPTIMIZATION=ON -DBUILD_SHARED_LIBS=ON
make -j $(nproc) |
|
Performance tests pass, the results are the same as current main. |
Yuval-Ariel
pushed a commit
that referenced
this issue
Aug 2, 2023
* Snapshot Optimization (#35) Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
bosmatt
removed
the
Working on it
udi-speedb
pushed a commit
that referenced
this issue
Nov 22, 2023
* Snapshot Optimization (#35) Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
udi-speedb
pushed a commit
that referenced
this issue
Dec 5, 2023
* Snapshot Optimization (#35) Motivation: The most important information inside a snapshot is its Sequence number, which allows the compaction to know if the key-value should be deleted or not. The sequence number is being changed when modification happens in the db. This feature allows the db to take a snapshot without acquiring db mutex when the last snapshot has the same sequence number as a new one. In transactional db with mostly read operations, it should improve performance when used with multithreaded environment and as well other scenarios of taking large amount of snapshots with mostly read operations. This Feature must have folly library installed. In order to cache the snapshots, there is last_snapshot_ (folly::atomic_shared_ptr, lock free atomic_shared_ptr) in order to access the last_snapshot_ created and point to it. For every GetSnapshotImpl call (where snapshots are being created), the function checks if the sequence number is different than last_snapshot_, if no, it creates new snapshot and inside this snapshot it adds a reference to last_snapshot_ (the reference is cached_snapshot), so this sequence number will remain inside SnapshotList (SnapshotList is the list of the snapshots in the system and used in compaction to show which snapshots are being used), if there are still snapshots holding this sequence number. If the sequence number as changed or the last_snapshot_ is nullptr it will create the snapshot while acquiring db_mutex. For ReleaseSnapshotImpl (deleting a snapshot). We will unref the last_snapshot_ (using comapre_exchange_weak) and if the refcount becomes 0, it will call Deleter and remove this snapshot entirely from the SnapshotList and continue with taking the db mutex. If there are still references, it will return without taking it out from the SnapshotList nor taking the db mutex
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Why:
Improve performance during read heavy snapshot workload
What :
In Read only workload, when taking a snapshot while Read, don’t create a new snap if it hasn’t changed, use the last one.
In Read only workload, don’t use mutex when the last snapshot is still valid.
The text was updated successfully, but these errors were encountered: