- Add the concept of
archived
to Branches, Testbeds, Benchmarks, and Measures - Remove Alert stats endpoint
- Fix
+ Add
API token UI permissions bug - Filter for
archived
resources in the Console UI - Archive and unarchive resources in the Console UI
- Make Start Points for Branches fallible
- Parse results for multiple tools for
iai-callgrind
adapter (Thank you @OmarTawfik)
- Revamp UI for looking at resource details
- Remove cached Organization when logging out
- Track Branches for pinned Plots across changes in their Start Point
- Update pinned Plots when theme changes
- Remove 15 second timeout for all CLI commands
- Show total number of pages for paginated lists in Console UI
- Add the ability to create and delete Organizations
- Fix 404 error for dedicated Metric pages in Console
- Create Branch and Testbed, if they do not exist for a Report on the API server
- Remove Branch and Testbed
soft
creation - Add Docker daemon check to CLI commands
- Fix iai-callgrind adapter bug
- Obfuscate user emails in API error messages
- Include Metric UUID in Report and Perf results
- Add a
GET
Metric API endpoint and CLI subcommand - Add a dedicated Metric page when clicking on a Metric dot in a Perf Plot
- Enable
busy_timeout
for SQLite connections to preventdatabase is locked
errors - Add replica settings for disaster recovery
- Add
Required
to required query params in docs (Thank you eddyashton) - Auto-log out in Console UI for expired sessions
- Fix wordmark color for default theme initialization
- Add
bencher plot
subcommand to manage pinned plots to CLI - Add
--api-env
/--console-env
and--api-volume
/--console-volume
tobencher up
CLI command (Thank you ricardaxel) - Add
service
argument tobencher up
andbencher down
to only control a single container
- Fix cross-project query bug
- Reduce Console navbar, menu, and panel flickering on page load
- Fix
Add
,Edit
,Delete
button permissions bug (Thank you eddyashton for reporting this bug) - Add the ability to "pin" Perf Plots to create a dashboard of pinned plots
- Add a button to clear each dimension separately in the Perf Plot
- Expand
--file-size
option forbencher run
to be used multiple times to track multiple file sizes - Add dark mode to Bencher Console UI
- Automatically find the
git
branch name forbencher run
if one is not specified - Add the
--file-size
option tobencher run
to track file size (ie binary size) - Allow custom Measures in Report results to be created on the fly
- Fix parsing new libtest bench Metrics as floats instead of integers (#390)
- Add new user onboarding flow
- Fix server stats endpoint bug
- A very special thank you to hamirmahal for their first contribution!
- Redirect to public Perf Page for Bencher Console links if user is not logged in
- Fix jittering in Perf Plot tabs when loading and changing pages
- Fix PR comment tags when changing branch start points
- Move benchmark results under a
details
tag for HTML comment output - Add an Alert summary table for HTML comment output
- Add a warning if no Thresholds are found for HTML comment output
- Deprecated:
--ci-no-metrics
flag no longer effects HTML comment output
- Add perf plot images as OpenGraph images
- Add GET
/v0/users
endpoint for server admins - Improve perf plot query performance by 120x or more by adding indexes and a materialized view
- Move Bencher Cloud out of BETA
- Add tab loading progress bar to perf pages
- Add
--branch-start-point
and--branch-start-point-hash
options tobencher run
to track rebasing and other branch changes - Deprecated: The
bencher run
options and flags--if-branch
,--else-if-branch
,--else-branch
, and--endif-branch
have been deprecated in favor of using--branch
and--branch-start-point
- BREAKING CHANGE
--if-branch
will now hard fail instead of skipping the run if not specified or if an empty string is provided, and it will always be created even without--else-branch
set - BREAKING CHANGE
--else-if-branch
will only use the first branch specified as the start point and it must exist, if it is a non-empty string - Update branch date time plot to use local time and version plot to be discrete along the x-axis
- Fix resource contention during large Perf queries
- Remove
public
query param from/v0/projects
endpoint - BREAKING CHANGE Move all CLI commands using
--org
,--project
, and--user
over to positional arguments, exceptbencher run
andbencher threshold create
- Add a
bencher report create
CLI subcommand bencher run
now tries to auto-fill the git commit hash for the report- BREAKING CHANGE Rename
statistic
tomodel
and remove API endpoint - Add "Click to view Threshold" for all boundary limits and add a public view Threshold page
- Add PATCH endpoint for user account settings
- BREAKING CHANGE Remove
/v0/server/ping
endpoint - Deprecate
/v0/server/endpoint
in favor of/v0/server/config/console
and BREAKING CHANGE removebencher server endpoint
CLI command
- BREAKING CHANGE Limit branch names to 256 characters and all slug lengths to 64 characters. Any existing values greater than these new limits are set to their respective UUID.
- Add shortcut URLs for user pages:
/console/users/tokens
->/console/users/USER-SLUG/tokens
/console/users/tokens/add
->/console/users/USER-SLUG/tokens/add
/console/users/settings
->/console/users/USER-SLUG/settings
- Improve
bencher run
error messages to include the command being run (idea credit to dholroyd: #335) - Prevent pwn requests when using
bencher run
- Add an API version check to all Bencher CLI commands that use the backend
- Fix Perf Plot view reduce bug
- Add a
View
button to Perf Plot for Reports, Branches, Testbeds, and Benchmarks - Add
search
query param toGET
list endpoints for all named resources - Add search for a named resources in Console UI and Perf Plot dimensions
- Add date based search for Reports in Console UI and Perf Plot
- Add search for public projects
- Move Bencher API server to "distroless" Docker image,
bencher-api
that is 10x smaller - Move Bencher Console UI to "distroless" Docker image,
bencher-console
that is 1/3 smaller - Fix Console server side rendering bug when using local URLs inside of Docker
- First time contributor JonasWanke added the Iai-Callgrind benchmark harness adapter
- Add a "Show all"/"Hide all" toggle for perf plot results (idea credit to jerome-benoit)
- Add an
--exec
flag tobencher run
to run commands as executables (no shell) and by default when there is more than one argument - Add new Thresholds: Static, Percentage, Log Normal, Interquartile Range, and Delta Interquartile Range
- Include Iteration number for Report results
- Redirect to newly created resources in Console UI
- BREAKING CHANGES Remove backwards compatibility for renames:
Metric
(lower_bound
=>lower_value
andupper_bound
=>upper_value
)Measure
(metric_kind
=>measure
inReport
andThreshold
;metric_kinds
=>measures
inPerf
)Boundary
(average
=>baseline
)
- Add
bencher up
,bencher down
, andbencher logs
commands to CLI to easily run Bencher Self-Hosted - Branches and Testbeds are now created if they do not exit for
bencher run
if the value passed is a slug or name (UUIDs are still required to already exist)
- Add OAuth2 based login for GitHub
- Allow users to accept invitations while logged in
- Run smoke tests on Linux, MacOS, and Windows
- Release Linux, MacOS, and Windows CLI builds for both x86_64 and ARM64
- Add convenience script to install
bencher
CLI on Linux, MacOS, and Windows
- Allow custom CI ID to be an unbounded length
- Add URL fragment links to all headings 🔗
- Fix browser navigation to replace history for automatic query args, like pagination
- Scroll to top of page when changing paginated pages
- Allow Branch, Testbed, and Measure names to be used in addition to slugs and UUIDs for Report (
GET
list &POST
), Threshold (GET
list &POST
), and Branch (POST
start point) endpoints - Validate Branch and Testbed exist before running benchmarks with
bencher run
- Move CLI over to relaxed response parsing and add
--strict
flag to enable strict response parsing - Add
--quiet
flag tobencher run
to enable quite mode and only output the final Report JSON
- The Boundary Limits object is now optional for Perf query results and Reports
- Update all
POST
API endpoints with creation semantics to return a201
status code - Update all
PUT
andPATCH
API endpoints with immediate update semantics to return a200
status code - Update all
DELETE
API endpoints to return a204
status code and an empty body
- Add Hyperfine adapter
- Rename
Metric Kind
toMeasure
- Rename
average
tobaseline
for the Boundary Limits
- Include the
average
used for the Boundary Limits - Remove
--ci-public-links
forbencher run
and simply use the project visibility to determine PR links - Replace
--ci-with-metrics
with--ci-no-metrics
forbencher run
and make showing benchmark metrics and boundary limits on PRs the default
- Add public Alert pages
- Add a
--ci-public-links
option tobencher run
to only post public links to PRs - Add a
--ci-with-metrics
option tobencher run
to display benchmark metrics and boundary limits on PRs - Update GitHub Actions PR documentation
- Fix word break CSS in console UI
- Add status bar for perf plot loading
- Fix perf plot x-axis label
- Improve embedded perf plot layout
- Improve
bencher_client
ergonomics for use inrustls
- Fix navbar bug on public pages while logged in
- Add Bencher Learn section to the site
- Set max perf query permutations to
255
- BREAKING CHANGE: Move from single
metric_kind
to array ofmetric_kinds
for perf plot query
- Embedded Perf Plots
- Move default Threshold Boundaries from
0.95
to0.98
- Add server statistics collection to Bencher Self-Hosted instances
- Move default Threshold Boundaries from
0.9
to0.95
- Redirect to login for an authorized users in Console UI
- Improve API server error messages
- Fixed the UI y-axis scale issue... again
- Add Bencher Plus licenses
- Add ability for
bencher run
to read output from a file without running a command argument - Improve Report creation error messages
- Add filtering Reports by Branch, Testbed, and start/end time
- Add one month historical default when viewing Reports in the Perf Plot
- Add endpoint
/v0/server/spec
to view OpenAPI spec JSON - Remove old UI source and scripts
- Change Metric
lower_bound
andupper_bound
tolower_value
andupper_value
respectively - Add ability to visualize
lower_value
andupper_value
in Perf Plot
- Add strongly typed IDs for database entities
- Remove deprecated configuration keys (
endpoint
=>console.url
andsecret_key
=>security.secret_key
)
- Add optional error monitoring with Sentry
- Start API error message improvement
- Fix Threshold dimension selection bug in UI
- Display benchmark harness results in realtime via
bencher run
- A special thank you to @robertorosmaninho for bug hunting 👾
- Move UI over to Astro with stronger TypeScript
- Add check for failing Netlify deployments
- Move default Threshold Boundaries from
0.95
to0.9
- Segment PR comments by the combination of Project, Branch, Testbed, and Adapter
- Add a custom PR comment ID argument
--ci-id
tobencher run
for complex scenarios - Another special thank you to @GitGab19 for all the feedback
- Fix slug length validation bug
- A special thank you to @GitGab19 for all the feedback
- Fix multi-Metric Kind bug for Perf Plot Report selection
- Require minimum and maximum sample sizes to be greater than or equal to two
- Allow system Metric Kinds (ie Latency, Throughput, etc) to be deleted and recreated via a later Report
- Comment on GitHub pull requests using
--github-actions
argument forbencher run
- Only post CI integrations when a Threshold is set for a Metric Kind, Branch, and Testbed with
ci-only-thresholds
- Only start posting CI integrations when an Alert is generated with
--ci-only-on-alert
- Fix Threshold boundary calculation bug when standard deviation is zero
- Add a warning
⚠️ in the Perf Plot UI when a Boundary Limit is not calculated - A special thank you to @untitaker for all the feedback
- Visualize Threshold lower and upper boundaries and Alerts in Perf Plot
- Add ability to dismiss and reactivate Alerts
- Add ability to update Thresholds in the UI
- Fix Threshold
PUT
bug - Add deletion capabilities to the UI
- Add help page to public UI
- Revamp Quick Start tutorial
- Add Docker Self-Hosted tutorial
- Fix public perf pages bug
- Add Reports tab to perf pages
- Add ability to clear the perf plot
- Add ability to set Project URL to
null
- Add Project permissions API endpoint
- Remove redundant Project API endpoints
- Add ability to update Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks in UI
- Move API server config key
endpoint
toconsole.url
- Change API from
/v0/server/config/endpoint
to/v0/server/endpoint
to getconsole.url
- Add dedicated
POST
API endpoint for Benchmarks - Add
PATCH
API endpoints for Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks - Add
PUT
API endpoint for Thresholds - Add
DELETE
API endpoint for Metric Kinds, Branches, Testbeds, Benchmarks, and Thresholds - Create two default Thresholds for all new Projects
- Add a Go benchmark example
- Fix Go auxillary data parse error
- Decrement all proceeding Branch versions after the final Report is deleted for a Branch version
- Add pagination to all GET list API endpoints,
bencher
CLI, and console UI - Move all
bencher
CLI API calls to using client auto-generated from OpenAPI spec - Improve
bencher
CLI error messages
- Add
--backdate
argument tobencher run
for backdating Reports - Add
created
andmodified
timestamps as applicable to all entities and their JSON - Add ability to delete a Report
- Add ability to delete a Project
- A special thank you to @wuputah and the Hydra team for all the feedback
- Add demo video to landing page
- Add
bencher run
documentation - Refactor Report, Alert, and Perf JSON to be more powerful
- All Report and Perf metrics now show the boundary limits used if a Threshold is set
- Display URL to visualize report in UI after
bencher run
- Change
magnitude
topow
forbencher mock
- A very special thank you to @MichaelScofield for being Bencher's second community contributor!
- Update roadmap
- Add Iai benchmark harness adapter
- Move from GitPod over to Dev Containers/GitHub Codespaces
- Fix thresholds with no historical metrics bug
- A very special thank you to @osiewicz for being Bencher's first community contributor!
- Add servers to Swagger API docs
- Fix perf view refresh bug
- Add an order of
--magnitude
argument tobencher mock
command - Add an API endpoint for threshold statistics and a CLI
statistic
sub-command - Upgrade the
bencher perf
CLI flag--table
to accept an optional style parameter - Fix long text wrapping in UI deck view, both header and card contents
- Autogenerate TypeScript types from Rust type definitions
- Massive thank you to @vient for all the feedback
- Fix UI expired token error
- Move the CLI linker over to use Zig in order to pin a specific
glibc
version - Move linker in CI over to
mold
for increased parallelism - Fix cargo workspace
default-features = false
warnings - Add project stewardship commitments
- Massive thank you to @brainstorm for all the feedback
- Add a
--table
option to thebencher perf
command to display query results as a table - Add the ability to toggle the perf plot x-axis from report date time to branch version number
- Create and publish Bencher CLI GitHub Action
- Fix both UI and perf image y-axis sizing once and for all... hopefully
- Move perf plot images from a screenshot to server side rendering
- Add a share mechanism to perf pages with a perf image preview
- Add an optional
--file
argument tobencher run
to read file output - Move perf query API endpoint from a POST to GET
- Add an endpoint to GET a screenshot of a public perf plot
- Update readme to add Discord invite
- Allow certain benchmarks to be ignored by Threshold checks
- BREAKING CHANGE: Make Project names unique within an organization and delete existing duplicates
- Add paid plans and metered billing for Bencher Cloud
- Change perf pages to use milliseconds as opposed to date/time stamp
- Fix CLI runtime dependency error over the past five releases
- BREAKING CHANGE: Make Metric Kind, Branch, and Testbed names unique and delete existing duplicates
- Add a soft branch creation option to eliminate race conditions in CI
- Add supported languages to landing page
- Lots of new docs!
- Add new
average
setting for adapters to choose between mean/standard deviation and median/interquartile range when possible - Increase CLI default retry to ten attempts three seconds apart (30 second window)
- Add ability to clone thresholds when creating a new branch from a start point and make this the default behavior for
--else-if-branch
inbencher run
CLI - Moved the plot key to below the plot all the time in preparation for screenshots/embedding
- Add STARTTLS option for SMTP connections
- Add
throughput
as a new default metric kind withoperations / second (ops/s)
as units - Add Java JMH, C# BenchmarkDotNet, JavaScript Benchmark.js, JavaScript console.time/timeEnd, Python airspeed velocity, Python pytest-benchmark, and Ruby Benchmark adapters
- Add resend email button to UI auth confirm page
- Change C++ Catch2 adapter to use standard deviation for
lower_bound
andupper_bound
- Fix table and perf panel formatting for long, single word values
- Prevent leaking secrets by obfuscating them in the server logs (ie
************
) - Move run failure detection to the CLI, adapter parsing will no longer attempt error detection
- New harness specific adapter targets (ie
rust_bench
andrust_criterion
) - Add additional options to
bencher mock
,--fail
and--flaky
to easily test parsing failing and flaky benchmarks respectively - Wrote continuous benchmarking and how to install the CLI pages
- Update server framework, dropshot, and Rust to
1.67.0
- Two new C++ adapters for Google Benchmark and Catch2
- Add an adapter for
go test -bench
- Added
--if-else-branch
,--else-branch
, and--endif-branch
flags to tobencher run
CLI - Wrote a branch management explanation page for using
bencher run
- Completely refactored docs UI configuration to make it easier to add new pages, one place instead of three
- Made the Diataxis convention explicit in the docs
- Fixed bounds bug in
bencher mock
- Added a
/v0/server/backup
route andbencher server backup
CLI command to perform database backups with options to compress, push to a remote data store (currently only AWS S3), and remove the local copy
- Fixed and refixed Criterion empty benchmark name bug
- Type level checking for benchmark names and metric kind resource IDs
- Added documentation: Quick Start, Prior Art, and GitHub Actions how to
- Prettier and eslint added to CI/CD
- Stubbed out UI tests with
vitest
in CI/CD
- Centralized all validation logic between frontend and backend via Rust + WASM modules
- All pages in the UI now have basic functionality
- Added public project perf pages
- Fixed the UI y-axis scale cutoff issue! Note to self: RTFD
- Fixed the UI documentation + routing bug! It turns out
SwaggerUIBundle
was the solution. - Added Metric Kind list, add, and view to UI
- In perf queries allowed for both UUIDs and slugs for branches and testbeds
- Added bencher.dev to Google Search Console and Analytics
- Changed API server default URL to always be localhost:3000
- Moved Docker UI build stage into its own
builder.Dockerfile
, mirroring the API setup - Created a better default for the UI's backend API URL, basically
this-host:61016
- Created a walking skeleton of adding Rust + WASM modules to the UI in both dev and CI/CD
- Fixed UI logout + routing "too many redirects" bug
- Changed notifications from state based to URL query param based in the UI
- Refactored how pathname and redirects are handled in the UI to increase encapsulation
- Added the current version to the UI
BETA
tag - Created a repo shortcut at bencher.dev/repo