Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add monitoring/benchmarking tools #3

Closed
24 tasks done
darcyclarke opened this issue Oct 15, 2019 · 0 comments
Closed
24 tasks done

Add monitoring/benchmarking tools #3

darcyclarke opened this issue Oct 15, 2019 · 0 comments
Assignees

Comments

@darcyclarke
Copy link
Contributor

darcyclarke commented Oct 15, 2019

  • Add a benchmark suite to test against our builds (ex. pnpm benchmarks)
  • Create a build pipeline/process, separate from the cli builds, to run the benchmarks & generate a report (use GitHub actions if possible)
Requirements

Requirements

  • Implement benchmark/ folder with script to run benchmarking commands [R]
    • create execution pattern
    • create scenario's paradigm
  • Create github action to run benchmark script [R]
    • link current commit of repo to global node_modules/ [R]
    • run benchmark script which will run scenarios [R]
    • output benchmark results in console [R]
  • Make output from benchmark run more usable [R]
    • write output of scenarios to a results.yml file [R]
    • write benchmark output to comment in pull-request [R]
    • update the same comment in pull-request [S]
  • Integrate benchmarking run into release process [R]
    • write benchmark results to file and commit it [R]
Legend

[R]: required
[S]: stretch goal

Todos

TODO:

  • update comments in scenarios.js[DOCS]

The information about order is incorrect. The ordering doesn't matter as the result of each run is a full cache, full node_modules, and full lockfile. Each next scenario begins with "everything", and must remove the things it doesn't need to complete itself.

  • remove delay function [CHORE]

Was used for testing

  • Properly comment index.js [DOCS]; remove commented code [CHORE]

Description of what a "scenario" is would be helpful for adding new scenarios in the future. Commented code was a previous iteration and can be removed.

  • collect results from each scenario into an object in index.js; write this object into a yaml file in benchmarks/results [FEAT]

Currently just writing the results to stdout. We need to keep track of the results for comparison between releases.

  • update GitHub Action workflow to only run on pull-request [FIX]

Currently the workflow runs on push which is everything: branches and pull-requests. Need to narrow scope to just pull-requests so that we can be sure to be able to write the results to a comment on said pull-request.

  • write "results.yml" into a comment on a pull-request [FEAT]

Using actions/...@v1 should be able to write a comment for the pull-request this action ran for.

  • update release process to include creating "results.yml" and committing it to the repository [FEAT]

A final "results.yml" file should be generated at release time, the "released commit" has an idea of it's benchmark.

Stretch Goal Tasks

Stretch Goal Tasks

  • if pull-request comment exists; update said comment

We want to avoid spamming pull-requests with all the results from each commit pushed to a branch with a pull-request. We should update the same comment; if a comment exists, other write a new comment.

  • compare results from current run with those inside benchmarks/results

Need to handle the edge case where that file doesn't exist.

Pull-Requests

pnpm forked repository which has ci update to commit results to repo upon merge into master branch

pull-request to add ci workflow for pull-request events, and pushes to latest, release-next branches

pull-request for benchmarking work into the npm/benchmark repo

@darcyclarke darcyclarke changed the title Add monitoring/benmarking to builds Add monitoring/benchmarking to builds Oct 15, 2019
@darcyclarke darcyclarke changed the title Add monitoring/benchmarking to builds Add monitoring/benchmarking tools Oct 15, 2019
@settings settings bot removed the enhancement label Oct 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants