Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mvcc: reduce count-only range overhead #11771

Merged
merged 4 commits into from
Apr 10, 2020

Conversation

tangcong
Copy link
Contributor

@tangcong tangcong commented Apr 8, 2020

image
if there are too much keys(> 1 millions),count-only range is very slow(slice expansion) and cost too much memory. it is not necessary to return revpairs,so we can do a small optimization to reduce overhead.

@tangcong tangcong force-pushed the optimize-range-count-only-performance branch from 449baa2 to 498291b Compare April 8, 2020 14:57
@codecov-io
Copy link

codecov-io commented Apr 8, 2020

Codecov Report

Merging #11771 into master will decrease coverage by 0.09%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #11771      +/-   ##
==========================================
- Coverage   66.92%   66.83%   -0.10%     
==========================================
  Files         403      403              
  Lines       36868    36881      +13     
==========================================
- Hits        24675    24649      -26     
- Misses      10706    10740      +34     
- Partials     1487     1492       +5     
Impacted Files Coverage Δ
mvcc/index.go 93.75% <100.00%> (+0.51%) ⬆️
mvcc/kvstore_txn.go 75.00% <100.00%> (+0.29%) ⬆️
pkg/transport/timeout_conn.go 60.00% <0.00%> (-20.00%) ⬇️
auth/simple_token.go 82.35% <0.00%> (-6.73%) ⬇️
clientv3/namespace/watch.go 87.87% <0.00%> (-6.07%) ⬇️
proxy/grpcproxy/watcher.go 89.79% <0.00%> (-4.09%) ⬇️
clientv3/leasing/cache.go 87.77% <0.00%> (-3.89%) ⬇️
etcdserver/api/rafthttp/peer.go 78.91% <0.00%> (-1.81%) ⬇️
etcdserver/api/v2http/client.go 88.32% <0.00%> (-1.28%) ⬇️
etcdserver/server.go 75.90% <0.00%> (-1.14%) ⬇️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7eae024...2ecbcb8. Read the comment docs.

Copy link
Contributor

@jingyih jingyih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea. Not sure if we want to change the existing function signature or create a new method. If the request is for count, can we create a method CountRevisions?

@tangcong
Copy link
Contributor Author

tangcong commented Apr 9, 2020

I like the idea. Not sure if we want to change the existing function signature or create a new method. If the request is for count, can we create a method CountRevisions?

yes,it will be more clear and concise. I will take the time to update pr.

@tangcong tangcong force-pushed the optimize-range-count-only-performance branch 2 times, most recently from e815396 to 2ecbcb8 Compare April 10, 2020 11:34
@tangcong
Copy link
Contributor Author

@jingyih updated. PTAL. thanks.

Copy link
Contributor

@jingyih jingyih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@gyuho gyuho merged commit 59f5fb2 into etcd-io:master Apr 10, 2020
@gyuho
Copy link
Contributor

gyuho commented Apr 10, 2020

@tangcong Curious, what's your use case for counting API?

@tangcong
Copy link
Contributor Author

tangcong commented Apr 10, 2020

Low frequency use, but we benefit a lot from it:

  1. A dimension of consistency monitoring (report etcd_key_diff metric by comparing every member total key diff at the same time)
  2. In the configuration system and k8s scenarios, quickly view the number of resources with a specified prefix.
    Through this optimization, we tested that the latency can be shortened by nearly 3 times in the ten million key scenarios, and there is no expensive memory overhead. Thanks. @gyuho

@gyuho
Copy link
Contributor

gyuho commented Apr 10, 2020

@tangcong Makes sense. Thanks!

@tangcong tangcong deleted the optimize-range-count-only-performance branch February 26, 2021 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants