More improvements for ReferenceUpdater

[ffreyer]

Description

Changes:

• pick latest CI run if multiple are present (e.g. due to closing and reopening a pr)
• add tab for removing reference images of missing recordings (e.g. to clean up reference images for renamed or deleted tests)
• Use latest reference images when updating, rather than the ones bundled with the CI run. This should prevent reference images from getting overwritten for outdated CI runs. make ReferenceUpdater safe to use from an outdated CI run (currently this uses old reference images from the CI run, rather than the latest uploaded ones.)
• use table layout for new (and missing) ref images
• add some documentation to the page
• add a "toggle all" for added and removed refimgs

Type of change

• tooling

[jkrumbiegel]

make ReferenceUpdater safe to use from an outdated CI run (currently this uses old reference images from the CI run, rather than the latest uploaded ones.

Yeah that's currently not great that you can clobber already updated reference images with the old ones from a previous CI run. I did this at the time because I was focusing on showing the changes that CI picked up in that specific run (that's what the scores were computed for etc.). How did you want to change this, compute scores again against the freshly downloaded reference set?

[ffreyer]

I just wanted to download the latest set for merging/updating.

I'm not sure if recomputing scores is worth it. I usually don't merge outdated prs if I'm not decently confident that they are compatible with things that got merged since. So I'd usually let CI run again anyway.

[MakieBot]

Compile Times benchmark

Note, that these numbers may fluctuate on the CI servers, so take them with a grain of salt. All benchmark results are based on the mean time and negative percent mean faster than the base branch. Note, that GLMakie + WGLMakie run on an emulated GPU, so the runtime benchmark is much slower. Results are from running:

using_time = @ctime using Backend
# Compile time
create_time = @ctime fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @ctime Makie.colorbuffer(display(fig))
# Runtime
create_time = @benchmark fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @benchmark Makie.colorbuffer(fig)

	using	create	display	create	display
GLMakie	4.79s (4.75, 4.81) 0.02+-	112.48ms (111.49, 115.08) 1.26+-	422.08ms (417.13, 427.80) 3.99+-	8.33ms (8.29, 8.41) 0.04+-	26.19ms (26.13, 26.28) 0.05+-
master	4.78s (4.74, 4.80) 0.02+-	111.02ms (110.08, 112.78) 0.88+-	417.98ms (414.66, 423.80) 2.87+-	8.89ms (8.31, 9.48) 0.52+-	26.20ms (26.12, 26.32) 0.07+-
evaluation	1.00x invariant, 0.01s (0.23d, 0.67p, 0.02std)	0.99x slower X, 1.46ms (1.34d, 0.03p, 1.07std)	0.99x slower X, 4.11ms (1.18d, 0.05p, 3.43std)	1.07x faster✅, -0.56ms (-1.53d, 0.03p, 0.28std)	1.00x invariant, -0.0ms (-0.07d, 0.89p, 0.06std)
CairoMakie	4.57s (4.48, 4.61) 0.04+-	113.22ms (110.27, 117.98) 2.75+-	171.55ms (167.72, 176.89) 3.58+-	9.33ms (9.06, 9.65) 0.22+-	1.15ms (1.14, 1.17) 0.01+-
master	4.61s (4.53, 4.68) 0.06+-	116.30ms (112.63, 126.90) 5.21+-	173.56ms (166.08, 178.49) 4.38+-	9.39ms (9.24, 9.60) 0.12+-	1.15ms (1.13, 1.16) 0.01+-
evaluation	1.01x invariant, -0.04s (-0.85d, 0.14p, 0.05std)	1.03x invariant, -3.08ms (-0.74d, 0.20p, 3.98std)	1.01x invariant, -2.02ms (-0.50d, 0.37p, 3.98std)	1.01x invariant, -0.06ms (-0.33d, 0.56p, 0.17std)	1.00x invariant, 0.01ms (0.56d, 0.32p, 0.01std)
WGLMakie	5.20s (5.14, 5.32) 0.06+-	110.31ms (108.85, 112.68) 1.61+-	4.98s (4.85, 5.21) 0.14+-	11.52ms (11.36, 11.92) 0.20+-	118.56ms (115.58, 123.50) 2.52+-
master	5.22s (5.15, 5.32) 0.07+-	109.50ms (107.99, 113.26) 1.75+-	4.96s (4.88, 5.08) 0.07+-	11.88ms (11.32, 13.94) 0.94+-	119.67ms (116.95, 122.24) 2.00+-
evaluation	1.00x invariant, -0.02s (-0.24d, 0.66p, 0.06std)	0.99x invariant, 0.82ms (0.49d, 0.38p, 1.68std)	1.00x invariant, 0.02s (0.20d, 0.71p, 0.11std)	1.03x invariant, -0.35ms (-0.52d, 0.36p, 0.57std)	1.01x invariant, -1.11ms (-0.49d, 0.38p, 2.26std)

[ffreyer]

Updating should not download the lastest refimg set and merge with that, instead of using the one attached to a pr.

The page now looks like this: (at 80% zoom to fit more on a screenshot)

[jkrumbiegel]

Maybe take out the couple console.logs if they aren't needed, otherwise this looks good to me!

[ffreyer]

Oh I forgot about those