Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add backoff metrics for grpc #1684

Merged
merged 9 commits into from
Jun 6, 2022
Merged

Add backoff metrics for grpc #1684

merged 9 commits into from
Jun 6, 2022

Conversation

hlts2
Copy link
Contributor

@hlts2 hlts2 commented Jun 2, 2022

Description:

NOTE : This PR is mirror of #1666

WHAT

I have added the logic to be able to get backoff metrics.
This allows collection of backoff metrics (retry count) within each RPC request when observability and backoff are enabled

WHY

We are in the process of implementing CircuitBreaker, but first we need to get proper metrics on the backoffs that occur inside RPC requests.

Related Issue:

How Has This Been Tested?:

Environment:

  • Go Version: 1.18.2
  • Docker Version: 20.10.8
  • Kubernetes Version: 1.22.0
  • NGT Version: 1.14.5

Types of changes:

  • Bug fix [type/bug]
  • New feature [type/feature]
  • Add tests [type/test]
  • Security related changes [type/security]
  • Add documents [type/documentation]
  • Refactoring [type/refactoring]
  • Update dependencies [type/dependency]
  • Update benchmarks and performances [type/bench]
  • Update CI [type/ci]

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

Checklist:

  • I have read the CONTRIBUTING document.
  • I have checked open Pull Requests for the similar feature or fixes?
  • I have added tests and benchmarks to cover my changes.
  • I have ensured all new and existing tests passed.
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly.

hlts2 added 3 commits June 2, 2022 15:44
Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com>
Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com>
Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com>
@vdaas-ci
Copy link
Collaborator

vdaas-ci commented Jun 2, 2022

[CHATOPS:HELP] ChatOps commands.

  • 🙆‍♀️ /approve - approve
  • 💌 /changelog - replace the PR body by changelog details
  • 🍱 /format - format codes and add licenses
  • /gen-test - generate test codes
  • 🏷️ /label - add labels
  • /rebase - rebase master
  • 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

@@ -31,7 +31,7 @@ import (
"github.com/vdaas/vald/internal/test/goleak"
)

func TestDo(t *testing.T) {
func TestDo_for_race(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have fixed function name because of CI fails.
The reason why CI fails was because there was a same function in to the same package.

https://github.com/vdaas/vald/blob/master/internal/runner/runner_test.go#L284

@hlts2 hlts2 requested review from kpango and datelier June 2, 2022 06:54
@hlts2 hlts2 self-assigned this Jun 2, 2022
@codecov
Copy link

codecov bot commented Jun 2, 2022

Codecov Report

Merging #1684 (95db66c) into master (bcaac99) will decrease coverage by 0.03%.
The diff coverage is 5.15%.

@@            Coverage Diff             @@
##           master    #1684      +/-   ##
==========================================
- Coverage   31.30%   31.26%   -0.04%     
==========================================
  Files         374      377       +3     
  Lines       32591    32679      +88     
==========================================
+ Hits        10202    10218      +16     
- Misses      22002    22078      +76     
+ Partials      387      383       -4     
Impacted Files Coverage Δ
internal/client/v1/client/discoverer/discover.go 0.00% <0.00%> (ø)
internal/net/grpc/client.go 0.00% <0.00%> (ø)
internal/net/grpc/context.go 0.00% <0.00%> (ø)
internal/observability/metrics/backoff/backoff.go 0.00% <0.00%> (ø)
pkg/gateway/filter/handler/grpc/handler.go 0.00% <0.00%> (ø)
pkg/gateway/lb/handler/grpc/handler.go 0.00% <0.00%> (ø)
pkg/gateway/lb/usecase/vald.go 0.00% <0.00%> (ø)
pkg/manager/index/service/indexer.go 0.00% <0.00%> (ø)
pkg/manager/index/usecase/indexer.go 0.00% <0.00%> (ø)
internal/backoff/backoff.go 73.26% <5.55%> (-14.69%) ⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7203cd3...95db66c. Read the comment docs.

datelier
datelier previously approved these changes Jun 2, 2022
internal/backoff/backoff_test.go Show resolved Hide resolved
internal/backoff/context_test.go Show resolved Hide resolved
internal/backoff/context_test.go Show resolved Hide resolved
internal/backoff/backoff_test.go Show resolved Hide resolved
internal/backoff/context_test.go Show resolved Hide resolved
internal/backoff/backoff.go Show resolved Hide resolved
internal/backoff/backoff.go Show resolved Hide resolved
Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com>
Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com>
pkg/agent/core/ngt/handler/grpc/handler.go Show resolved Hide resolved
pkg/agent/core/ngt/handler/grpc/handler.go Show resolved Hide resolved
internal/backoff/context.go Show resolved Hide resolved
internal/net/grpc/context.go Show resolved Hide resolved
Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com>
@@ -2004,7 +2004,7 @@
}

func (s *server) Remove(ctx context.Context, req *payload.Remove_Request) (res *payload.Object_Location, err error) {
ctx, span := trace.StartSpan(ctx, apiName+".Remove")
ctx, span := trace.StartSpan(ctx, apiName+"/"+vald.RemoveRPCName)

Check warning

Code scanning / CodeQL

Useless assignment to local variable

This definition of ctx is never used.
@@ -2140,7 +2140,7 @@
}

func (s *server) MultiRemove(ctx context.Context, reqs *payload.Remove_MultiRequest) (res *payload.Object_Locations, err error) {
ctx, span := trace.StartSpan(ctx, apiName+".MultiRemove")
ctx, span := trace.StartSpan(ctx, apiName+"/"+vald.MultiRemoveRPCName)

Check warning

Code scanning / CodeQL

Useless assignment to local variable

This definition of ctx is never used.
Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com>
@vankichi vankichi requested review from kpango and datelier June 6, 2022 02:01
Copy link
Collaborator

@kpango kpango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hlts2 hlts2 merged commit d1ade8c into master Jun 6, 2022
@hlts2 hlts2 deleted the feature/add-bo-metrics branch June 6, 2022 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants