chore: profiling via Dagger #1873

markphelps · 2023-07-16T12:09:40Z

Adds mage hack:loadTest command to run a series of loadtest and output the profiling data to be able to view in Pyroscope

codecov · 2023-07-16T12:20:59Z

Codecov Report

Merging #1873 (2c5c575) into eval-v2 (a1ec0b6) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##           eval-v2    #1873   +/-   ##
========================================
  Coverage    71.68%   71.68%           
========================================
  Files           63       63           
  Lines         6142     6142           
========================================
  Hits          4403     4403           
  Misses        1491     1491           
  Partials       248      248

Impacted Files	Coverage Δ
internal/server/auth/method/kubernetes/verify.go	`66.36% <100.00%> (ø)`
internal/telemetry/telemetry.go	`62.35% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

build/hack/cmd/loadtest/main.go

GeorgeMac

Nice. Couple suggestions, I have a gut feeling that the instance has no state in it.

GeorgeMac · 2023-07-16T12:38:39Z

build/hack/loadtest.go

+	// import some test data
+	_, err := flipt.WithEnvVariable("UNIQUE", uuid.New().String()).
+		WithFile("import.yaml", seed).
+		WithExec(importCmd).
+		ExitCode(ctx)
+
+	if err != nil {
+		return err
+	}


I think this would technically result in the layer produced at 17 having the state in it.
However, the built instance of the flipt container you build down on line 34 is built from the reference of flipt passed into this function.

I think the following might work (if not we can do what I do in the other tests and run import in a foreground container with flipt as a running service and go through the API):

Suggested change

// import some test data

_, err := flipt.WithEnvVariable("UNIQUE", uuid.New().String()).

WithFile("import.yaml", seed).

WithExec(importCmd).

ExitCode(ctx)

if err != nil {

return err

}

// import some test data

flipt, err = flipt.WithEnvVariable("UNIQUE", uuid.New().String()).

WithFile("import.yaml", seed).

WithExec(importCmd).

Sync(ctx)

if err != nil {

return err

}

thanks i wasnt sure what the difference was between Sync and ExitCode and when you would use one over the other

much better!

Nice yeah that looks legit

build/hack/loadtest.go

GeorgeMac

One thing and its good to go 👍

build/hack/loadtest.go

yquansah

This is great. I see changes in the go.work.sum, probably not too significant.

markphelps · 2023-07-17T13:39:01Z

This is great. I see changes in the go.work.sum, probably not too significant.

yeah ill checkout go.mod* and go.work.sum from the base branch and then run go mod tidy to see if all the changes are needed

* Feat: eval v2 sqlite (#1739) * feat(wip): rollout for boolean flags * feat(wip): strategy impl * feat: rollout rules > rollout strategy * chore: add namespace_key to tables * feat: list rules * feat: impl delete * chore: rm debug logs * chore: rename things * chore: compiles, start adding storage tests * feat(wip): update rollout * chore: compiles * chore: compiles/finish sqlite rollout tests for now * chore: fix countRollouts call * chore: rebase eval-v2 * chore: checkout ui from main * chore: replace ui with ui from main * chore: fix buflint * chore: fix lint errors * chore: rowserrcheck * chore: mage proto * feat: implement gRPC server for the Variant RPC method * chore: rename Server field to evaluator * chore: move around tests, and other fixes * chore: DRY up middleware and add extra field to otel metrics * feat: add boolean evaluation logic for the store * chore: add tests for boolean evaluate and address PR comments from upstream * chore: add batch evaluation functionality * chore: address comments and add more tests for behavior * feat: implement gRPC server for the Variant RPC method (#1798) * feat: implement gRPC server for the Variant RPC method * chore: rename Server field to evaluator * chore: move around tests, and other fixes * chore: DRY up middleware and add extra field to otel metrics * test(integration): ensure Evaluation.Variant returns as expected (#1800) * test(integration/api): ensure Evaluation.Variant returns as expected * test(integration/readonly): ensure Evaluation.Variant returns as expected * fix(build/testing): set production yaml version to 1.0 * fix(build/testing/api): ensure disabled flag is created and listed * fix(build/testing/api): fix disabled flag asserts and delete before ns is deleted * fix(build/testdata): typo on name field case * fix(server/middleware): set request IDs on evaluation req/resp type * fix(build/integration): ensure 51 flags exist * fix(build/integration): ensure flag not found error is returned * fix(build/integration): explicitly set enabled false in testdata * chore: rearrange some files and make some new declarations * chore: move around some structures and address other feedback around passing flag into Evaluate * chore: set timestamps and request id on Variant responses * chore: add test and move invalid flag type to evaluator --------- Co-authored-by: George <me@georgemac.com> * chore: readd tests and other couple fixes * chore: add middleware bits for timestamp and duration * chore: remove storage implementation of getting rollouts * chore: use lowercase package names, and return nil instead of real value * chore: add tests for coverage * chore: add error response for batch evaluation * fix(api/rollouts): ensure tx rollback (#1805) * test(integration/api): ensure Boolean type flag create * fix(api/rollouts): ensure tx rollback and drop create rollout type * fix(storage/rollout): ensure tx rollback and fixed rollout type on update * fix(storage/sql/rollout): validate rule is supplied on create * chore(rpc): add rollout request types validation * fix(integration/readonly): add flag_disabled to end of set * chore: commit proto version comment updates * chore: use different enum for batch evaluation responses * chore: make small optimization for struct initialization * refactor!(rpc/flipt): rename RolloutPercentage to RolloutThreshold * fix(tests): change assertions to match threshold not percentage enum string * refactor(storage): rename RolloutPercentage to RolloutThreshold * chore: change proto definition for evaluation and error evaluation reasons * chore: implement storage queries for rollout evaluation * chore: add integration test for boolean, batch * feat(ui): add flag type radio (#1830) * feat(wip): flag type select/render * chore: make disabled on update * chore: fix default checked * chore: address PR comments * feat(ext): add support for flag type and rollout definitions (#1826) * test(ext): add failing cases for new version 1.1 and rollouts * feat(ext): add support for flag type and rollout definitions * fix(ext): adding missing error not-nil check * test(readonly): bump testdata import version * fix(ext): support rename from rollout percentage to threshold * fix(ext): ensure minimum version enforced as expected * test(readonly): update suite with boolean flag assertions * feat(ext): support implicit rank in flag state version 1.1 * chore(ext): correct typos in comments * fix(test/cli): update assertion for import/export * feat(validate): add support for state version 1.1 * chore(build): address comment fix suggestions * fix(test/migration): use an import file valid for latest * fix(test/migration): import using latest build not tip * feat: add fs storage implentation and readonly tests * feat: address all the missing links with rollouts in regards to fs store * feat: create eval rollout threshold (#1835) * feat(wip): ui rollout threshold crud * feat(wip): show new rollout button * chore: add type to flag table * feat(wip): threshold rule form * feat(wip): rollout thresholds * chore: rename to 'rules' for now * chore: rm un-needed prop * chore: fix small issues * chore: extract some more constants * feat: add missing description, set max/min for threshold client side * fix: typing * chore: type the form values * chore: move value to common form * chore: disable evaluation tab for boolean flag * chore(fs/rollouts): address PR comments and add relevant tests * fix: broken ui build (#1838) * feat: add audit logging for rollout actions (#1834) * feat: add audit logging for rollout actions * chore(tests): add unit tests for delete in the middleware * feat: eval rollout segment ui (#1839) * feat(wip): start * feat(ui): finish segment rollout creation form * fix(rollouts): add order rollouts and validate requests (#1850) * feat(rollouts): add order rollouts RPC * fix(rollouts): implement server.OrderRollouts and add unit tests * fix(proto): add gateway routes for order rollouts * test(readonly): ensure list rollout pagination * chore(rpc/flipt): add validation around create and update rollouts * fix(ext): add variant description to testdata yaml * fix(build/testing/migration): use previous release versions test suite (#1852) * fix(build/testing/migration): use previous release versions test suite * refactor(build/testing/migration): use readonly test suite data from latest release * feat(middleware): support caching new evaluation RPCs (#1853) * feat(middleware): support caching evaluation.VariantEvaluationResponse * feat(middleware): support caching evaluation.BooleanEvaluationResponse * feat(eval/rollouts): write benchmark tests for evaluations * chore(benchmarks): move around logic * feat(auth): ability to skip authentication for top-level api prefixes (#1854) * feat(auth): ability to skip authentication for top-level api prefixes * fix(config): add mapstructure tags * fix(cmd/grpc): thread authentication options * Rule List refactor (#1855) * feat: redo rules list styles * feat: done tweaking the ui * chore: drag drop * chore: show actual variant key; fix border color on drag * chore: make responsive * chore(sql): format query * feat: real rollout list/quick edit/delete (#1857) * feat(wip): rollout list ui + quickedit * chore: update some styles * chore: fix selected segment * feat: finish update/delete of rollout list * chore: fix linter * chore: add percent icon, hide slider in mobile * feat: make reset work: * chore: clean up benchmarks to not have random elements in the requests * feat(console-evaluation): use v2 evaluation in console rather than v1 evaluation * feat: rollout edit + typescript enum redo (#1859) * chore: address comments * feat: Quick edit rules (#1862) * feat(metrics): Add otel and general metrics for boolean evaluation * chore: pass in context from the outside for boolean match * fix(build): use relative replace for rpc/flipt (#1881) * fix(build): use relative replace for rpc/flipt * chore(build): replace root with relative directory * chore: profiling via Dagger (#1873) * chore: rebase on eval-v2 * chore: rm binary * feat(wip): attempt to run profiling / pyroscope in dagger * chore: go.sum things * chore: more mod tidy * chore: give proper user * chore: finish profile run * chore: finish loadtest via dagger * chore: bump to 60s, add more to README * chore: fix loadtest to actually import / use state * chore: rm un-needed export * feat(ui): eval default rollout (#1866) * chore: change boolean eval to use false as default * chore: check for default namespace * chore: change back to use enabled as default, rename boolean eval response field * chore: return reason from matchConstraints * feat: add default rollout in UI * chore: fix api IT * chore(sqlite): add custom logic for sqlite for returning errors on specific cases * feat(db): Add the rest of sql files for rollout migrations * chore: runs prettier after dependency update (#1883) * chore: go mod tidy * chore(mysql): address comments and make query squirrel builder * chore(mysql): disable sql_mode for testing * chore: change property of Value to Enabled * chore: add default value to current_timestamp as well * chore: remove unique index * fix: validation of rollout rule values in forms (#1889) * chore: rollouts UI its (#1888) * chore: small ui tweaks for rules/rollouts (#1890) * chore: prettier package.json * chore: add namespace key to error reason; rename error enum (#1902) * chore: boolean eval screenshots (#1904) * chore: fix screenshots for new ui * chore: mv screenshots to subdir * feat(wip): concepts docs * chore: rm old screenshots func * chore: finish most of the screenshots * chore: add rollouts * chore: add readonly screenshot * chore: add missing eval console screen * chore: fix scrollToBottom for screenshots (#1911) * chore: fix scrollToBottom for screenshots * chore: increase sleep by 1s * chore: mod tidy * chore: loadtest cleanup * chore: rework interceptor config * chore: fix cacher interceptor ordering (#1914) * chore: fix cacher interceptor ordering * chore: fix storage cacher tests (#1916) * chore: why are you changing go.work.sum * chore: fixup cacher tests into new eval-v2 branch * chore: rm unknown file --------- Co-authored-by: Mark Phelps <209477+markphelps@users.noreply.github.com> Co-authored-by: Yoofi Quansah <ybquansah@gmail.com>

markphelps added 9 commits July 14, 2023 13:34

chore: rebase on eval-v2

4afdc4a

chore: rm binary

27c8cb0

feat(wip): attempt to run profiling / pyroscope in dagger

e3aa215

chore: go.sum things

3bd4a1d

chore: more mod tidy

9d14f07

chore: give proper user

438cce6

chore: finish profile run

56596f9

chore: finish loadtest via dagger

f282cfe

chore: bump to 60s, add more to README

3ed45cb

markphelps requested a review from a team as a code owner July 16, 2023 12:09

GeorgeMac reviewed Jul 16, 2023

View reviewed changes

build/hack/cmd/loadtest/main.go Show resolved Hide resolved

GeorgeMac requested changes Jul 16, 2023

View reviewed changes

markphelps added 2 commits July 16, 2023 09:23

chore: fix loadtest to actually import / use state

7410d4b

Merge branch 'eval-v2' into hack-profiling

3e60a73

markphelps requested a review from GeorgeMac July 16, 2023 13:23

GeorgeMac requested changes Jul 17, 2023

View reviewed changes

build/hack/loadtest.go Outdated Show resolved Hide resolved

markphelps added 2 commits July 17, 2023 08:02

chore: rm un-needed export

2331ca8

chore: merge

fb6e441

markphelps requested review from GeorgeMac and a team July 17, 2023 12:14

yquansah approved these changes Jul 17, 2023

View reviewed changes

markphelps added 2 commits July 17, 2023 09:59

chore: merge upstream

2c5c575

chore: merge

cb5d8f4

markphelps merged commit 6dc4824 into eval-v2 Jul 17, 2023
14 of 18 checks passed

markphelps deleted the hack-profiling branch July 17, 2023 16:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: profiling via Dagger #1873

chore: profiling via Dagger #1873

markphelps commented Jul 16, 2023 •

edited

Loading

codecov bot commented Jul 16, 2023 •

edited

Loading

GeorgeMac left a comment

GeorgeMac Jul 16, 2023

markphelps Jul 16, 2023

markphelps Jul 16, 2023

GeorgeMac Jul 17, 2023

GeorgeMac left a comment

yquansah left a comment

markphelps commented Jul 17, 2023

chore: profiling via Dagger #1873

chore: profiling via Dagger #1873

Conversation

markphelps commented Jul 16, 2023 • edited Loading

codecov bot commented Jul 16, 2023 • edited Loading

Codecov Report

GeorgeMac left a comment

Choose a reason for hiding this comment

GeorgeMac Jul 16, 2023

Choose a reason for hiding this comment

markphelps Jul 16, 2023

Choose a reason for hiding this comment

markphelps Jul 16, 2023

Choose a reason for hiding this comment

GeorgeMac Jul 17, 2023

Choose a reason for hiding this comment

GeorgeMac left a comment

Choose a reason for hiding this comment

yquansah left a comment

Choose a reason for hiding this comment

markphelps commented Jul 17, 2023

markphelps commented Jul 16, 2023 •

edited

Loading

codecov bot commented Jul 16, 2023 •

edited

Loading