ci(benchmarks): run only the last release

[P403n1x87]

We speed up the benchmarks CI run by only running the latest release and dev. Any issues caught in this way can be analysed locally if needed.

[github-actions]

Austin Benchmarks

Running Austin benchmarks with Python 3.10.13

Benchmark Summary

Comparison of dev against 3.5.0.

The following scenarios show a statistically significant difference in performance between the two versions.

	Sample Rate	Saturation	Error Rate	Sampling Speed
CPU time [sampling interval: 10]	🟡	🟡	🟢	🟡
RSA keygen [sampling interval: 100]	🟡	🟡	🟡	🔴
Full metrics [sampling interval: 1]	🟡	🟡	🟡	🟢

Benchmark Results

Wall time [sampling interval: 1]

Wall time [sampling interval: 1]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	118000 ± 3000	1 ± 0	1e-05 ± 6e-06	13.2 ± 0.4
dev	117000 ± 3000	1 ± 0	5e-06 ± 4e-06	13.1 ± 0.3

Wall time [sampling interval: 10]

Wall time [sampling interval: 10]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	116000 ± 3000	0.543 ± 0.002	1e-05 ± 4e-06	13.1 ± 0.3
dev	116000 ± 6000	0.544 ± 0.002	6e-06 ± 3e-06	13.3 ± 0.5

Wall time [sampling interval: 100]

Wall time [sampling interval: 100]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	10500 ± 400	0.001 ± 0.0005	3e-05 ± 3e-05	14.7 ± 0.7
dev	10400 ± 200	0.0008 ± 0.0001	3e-05 ± 3e-05	14.6 ± 0.7

Wall time [sampling interval: 1000]

Wall time [sampling interval: 1000]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	1860 ± 30	0.0001 ± 0.0003	0 ± 0	17.9 ± 0.9
dev	1850 ± 20	0 ± 0	2e-05 ± 7e-05	18.2 ± 0.4

CPU time [sampling interval: 1]

CPU time [sampling interval: 1]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	52000 ± 4000	1 ± 0	1.6e-05 ± 1e-05	22.4 ± 0.8
dev	51000 ± 5000	1 ± 0	2e-05 ± 1e-05	23 ± 2

CPU time [sampling interval: 10]

CPU time [sampling interval: 10]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	50000 ± 5000	0.9975 ± 0.0002	2.7e-05 ± 1e-05	23 ± 1
dev	51000 ± 5000	0.997 ± 0.001	2e-05 ± 1e-05	23 ± 2

CPU time [sampling interval: 100]

CPU time [sampling interval: 100]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	5300 ± 100	0.0012 ± 0.0006	8e-05 ± 6e-05	22.3 ± 0.5
dev	5400 ± 100	0.002 ± 0.001	5e-05 ± 8e-05	22.7 ± 0.8

CPU time [sampling interval: 1000]

CPU time [sampling interval: 1000]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	949 ± 2	0 ± 0	0.0001 ± 0.0003	26.9 ± 0.6
dev	950 ± 3	0.0 ± 0.0001	0 ± 0	26.6 ± 0.5

RSA keygen [sampling interval: 1]

RSA keygen [sampling interval: 1]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	40000 ± 1000	1 ± 0	0.0003 ± 0.0002	24.2 ± 0.6
dev	39700 ± 900	1 ± 0	0.0004 ± 0.0004	24.4 ± 0.7

RSA keygen [sampling interval: 10]

RSA keygen [sampling interval: 10]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	39000 ± 1000	0.98 ± 0.03	0.0004 ± 0.0004	24.5 ± 0.7
dev	39900 ± 700	0.98 ± 0.01	0.0002 ± 0.0003	24.3 ± 0.7

RSA keygen [sampling interval: 100]

RSA keygen [sampling interval: 100]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	6280 ± 30	0.002 ± 0.001	0.0006 ± 0.0006	27 ± 2
dev	6260 ± 60	0.003 ± 0.003	0.0005 ± 0.0007	29 ± 2

RSA keygen [sampling interval: 1000]

RSA keygen [sampling interval: 1000]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	941 ± 5	0 ± 0	0.001 ± 0.001	31 ± 2
dev	942 ± 2	0 ± 0	0.001 ± 0.003	32 ± 1

Full metrics [sampling interval: 1]

Full metrics [sampling interval: 1]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	51000 ± 2000	1 ± 0	3e-05 ± 1e-05	30 ± 1
dev	52300 ± 900	1 ± 0	2e-05 ± 1e-05	29.3 ± 0.5

Full metrics [sampling interval: 10]

Full metrics [sampling interval: 10]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	51000 ± 2000	1 ± 0	2e-05 ± 1e-05	30 ± 2
dev	51000 ± 3000	1 ± 0	1.6e-05 ± 1e-05	30 ± 2

Full metrics [sampling interval: 100]

Full metrics [sampling interval: 100]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	10000 ± 300	0.002 ± 0.0004	6e-05 ± 5e-05	33.1 ± 1.0
dev	10000 ± 200	0.003 ± 0.003	5e-05 ± 6e-05	32.9 ± 0.7

Full metrics [sampling interval: 1000]

Full metrics [sampling interval: 1000]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	1850 ± 20	0.0002 ± 0.0004	0.0001 ± 0.0001	37.8 ± 0.9
dev	1860 ± 20	0.0002 ± 0.0002	2e-05 ± 7e-05	37 ± 1

Multiprocess wall time [sampling interval: 1]

Multiprocess wall time [sampling interval: 1]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	3300 ± 200	1 ± 0	0.00016 ± 3e-05	290 ± 10
dev	3300 ± 200	1 ± 0	0.00014 ± 3e-05	290 ± 20

Multiprocess wall time [sampling interval: 10]

Multiprocess wall time [sampling interval: 10]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	3350 ± 60	1 ± 0	0.00014 ± 4e-05	286 ± 6
dev	3270 ± 90	1 ± 0	0.00012 ± 4e-05	292 ± 9

Multiprocess wall time [sampling interval: 100]

Multiprocess wall time [sampling interval: 100]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	3200 ± 100	0.05 ± 0.02	7e-05 ± 4e-05	300 ± 10
dev	3100 ± 400	0.04 ± 0.02	6e-05 ± 4e-05	270 ± 80

Multiprocess wall time [sampling interval: 1000]

Multiprocess wall time [sampling interval: 1000]

	Sample Rate	Saturation	Error Rate	Sampling Speed
3.5.0	3100 ± 200	0.008 ± 0.001	2e-05 ± 3e-05	67 ± 6
dev	3000 ± 100	0.007 ± 0.002	1e-05 ± 2e-05	70 ± 10

[codecov]

Codecov Report

Patch coverage has no change and project coverage change: +0.31% 🎉

Comparison is base (6455c21) 68.37% compared to head (0050447) 68.68%.

Additional details and impacted files

@@            Coverage Diff             @@
##            devel     #203      +/-   ##
==========================================
+ Coverage   68.37%   68.68%   +0.31%     
==========================================
  Files          27       27              
  Lines        2523     2523              
  Branches      775      775              
==========================================
+ Hits         1725     1733       +8     
+ Misses        460      459       -1     
+ Partials      338      331       -7     

see 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.