Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework projection code #3040

Draft
wants to merge 24 commits into
base: master
Choose a base branch
from
Draft

Rework projection code #3040

wants to merge 24 commits into from

Conversation

ffreyer
Copy link
Collaborator

@ffreyer ffreyer commented Jul 2, 2023

Since this pr is shaping out to be breaking I'm planning on handling #3013, #2881 and #1730 here.

Description

This pr has two main goals. The first is to reduce the amount of (more or less) redundant code dealing with projecting positions. The second is to add missing space options. The pipeline of coordinate spaces/transformations we have looks like this

spaces

where transformed, world and eye space are not accessible through space. (See also #2766) These are added here.

Something I'm not sure about yet is whether pixel and relative space should ignore Transformations. It might be a bit weird that things like scale!() or polar transforms don't work with them. If we do allow them to affect pixel and relative space it might make sense to separate them completely from space. Otherwise naming will get confusing I think. We should also disallow transform propagation if spaces don't match.

I'm also considering restarting #1730 here, as it may change how some things need to work.

Closes #3013
Closes #2766 since this also implements transformed space.

TODO

  • Rework project to be more useful. It now includes model and transform_func depending on input space and can deal with vector inputs.
  • Add projection_obs to simplify projecting data in observables
  • add:transformed, :world and :eye space to make every coordinate space accessible
  • Remove redundant code from Makie
  • Update GLMakie
  • Rework CairoMakies projection code
  • Update WGLMakie
  • Update RPRMakie
  • fix axis scale problem (bracket causes error due to late update?)
  • figure out how to deal with transformations vs camera projections (both controlled via space or just projections)
  • split spaces
  • update bounding boxes with new project code
  • consider doing boundingbox cleanup #2881 here
  • Documentation
    • new page
    • docstrings, e.g. hvlines, project functions, ...
  • Cleanup of old code and notes

Other changes

  • fixed duplicate update issues with projectionview
  • allow transform_func to be passed directly as a plot kwarg
  • remove transformation and transform_func from plot attributes on construction

Type of change

  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

While project may or may not be internal, I think it is fairly likely to be used outside in similar situations as errorbars, h/vlines, h/vspan, bracket, etc. (I.e. whether part of or the full plot needs to be in pixel space.)

Checklist

  • Added an entry in NEWS.md (for new features and breaking changes)
  • Added or changed relevant sections in the documentation
  • Added unit tests for new algorithms, conversion methods, etc.
  • Added reference image tests for new plotting functions, recipes, visual options, etc.

@MakieBot
Copy link
Collaborator

MakieBot commented Jul 2, 2023

Compile Times benchmark

Note, that these numbers may fluctuate on the CI servers, so take them with a grain of salt. All benchmark results are based on the mean time and negative percent mean faster than the base branch. Note, that GLMakie + WGLMakie run on an emulated GPU, so the runtime benchmark is much slower. Results are from running:

using_time = @ctime using Backend
# Compile time
create_time = @ctime fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @ctime Makie.colorbuffer(display(fig))
# Runtime
create_time = @benchmark fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @benchmark Makie.colorbuffer(display(fig))
using create display create display
GLMakie 14.13s (13.86, 14.34) 0.15+- 1.44s (1.38, 1.47) 0.03+- 861.90ms (822.69, 911.27) 28.17+- 12.64ms (12.54, 12.73) 0.05+- 165.63ms (163.12, 166.49) 1.16+-
master 14.05s (13.80, 14.31) 0.17+- 1.49s (1.44, 1.55) 0.04+- 865.02ms (841.99, 899.06) 20.07+- 12.65ms (12.48, 12.91) 0.15+- 166.40ms (164.53, 169.52) 1.78+-
evaluation +0.59%, 0.08s invariant (0.52d, 0.35p, 0.16std) -3.12%, -0.04s faster ✓ (-1.19d, 0.05p, 0.04std) -0.36%, -3.12ms invariant (-0.13d, 0.82p, 24.12std) -0.12%, -0.01ms invariant (-0.13d, 0.82p, 0.10std) -0.46%, -0.77ms invariant (-0.51d, 0.36p, 1.47std)
CairoMakie 10.74s (10.61, 10.82) 0.07+- 1.26s (1.24, 1.27) 0.01+- 255.34ms (253.20, 256.87) 1.52+- 10.73ms (10.57, 10.93) 0.12+- 6.37ms (6.30, 6.46) 0.05+-
master 10.89s (10.79, 11.00) 0.06+- 1.25s (1.23, 1.26) 0.01+- 229.42ms (224.54, 232.97) 3.04+- 10.68ms (10.51, 10.80) 0.10+- 4.55ms (4.51, 4.58) 0.02+-
evaluation -1.41%, -0.15s faster ✓ (-2.31d, 0.00p, 0.07std) +0.70%, 0.01s invariant (0.88d, 0.13p, 0.01std) +10.15%, 25.92ms slower❌ (10.78d, 0.00p, 2.28std) +0.48%, 0.05ms invariant (0.47d, 0.39p, 0.11std) +28.59%, 1.82ms slower❌ (44.14d, 0.00p, 0.04std)
WGLMakie 16.42s (16.13, 16.65) 0.19+- 1.69s (1.63, 1.86) 0.08+- 14.66s (14.47, 14.95) 0.17+- 18.44ms (17.13, 20.58) 1.17+- 1.42s (1.38, 1.48) 0.03+-
master 16.35s (16.14, 16.56) 0.12+- 1.66s (1.58, 1.74) 0.05+- 14.76s (14.33, 15.28) 0.29+- 18.17ms (17.01, 19.49) 0.97+- 1.42s (1.37, 1.46) 0.04+-
evaluation +0.44%, 0.07s invariant (0.45d, 0.41p, 0.16std) +1.44%, 0.02s invariant (0.35d, 0.52p, 0.07std) -0.64%, -0.09s invariant (-0.40d, 0.47p, 0.23std) +1.44%, 0.27ms invariant (0.25d, 0.65p, 1.07std) +0.65%, 0.01s invariant (0.26d, 0.64p, 0.04std)

Comment on lines +257 to +261
function to_ndim(trg::VT, src::VecTypes{N2}) where {N, N2, VT <: VecTypes{N}}
VT(ntuple(Val(N)) do i
@inbounds i > N2 ? trg[i] : src[i]
end)
end
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did some benchmarking on to_ndim while working on this pr. It's relatively slow if the type isn't known at compile time. You can see this by benchmarking the function directly:

T = Point4f
p2 = Point2f(1, 2)
fillval = 1f0
@benchmark Makie.to_ndim($T, $p2, $fillval) # ~74ns

And comparing with a wrapped version where the type is hardcoded:

f2(p, v) = Makie.to_ndim(Point4f, p, v)
@benchmark f2($p2, $fillval) # 1.4ns

Using a function barrier does not improve performance:

f1(T, p, v) = Makie.to_ndim(T, p, v)
@benchmark f1($T, $p2, $fillval) # ~74ns

However if you use a instantiated object rather than passing a type things become fast with a "dynamic" type:

p4 = Point4f(1)
@benchmark Makie.to_ndim($p4, $p2) # 1.4ns

This also has a real effect on Makie.project. With 1000 Point2f's it cuts down 150µs to about 9.6µs for me.

Comment on lines 966 to 1052
# For correct z-ordering we need to be in eye (camera), clip, pixel or
# relative space
if get_value(primitive, :space, :data) in (:data, :transformed, :world)
zs = last.(Makie.project(primitive, pos, output_space = :eye))
zorder = sortperm(zs, rev = false)
elseif length(pos[1]) > 2
zs = getindex.(zs, 3)
zorder = sortperm(zs, rev = false)
else
zorder = eachindex(pos)
end
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I benchmarked this with some simplifications (no matrix-matrix product, just one random matrix). The old version takes about 90µs for me, the new about 17µs with 1000 Point3f's. Using project with a transform_func drops it to about 22µs.

Comment on lines +97 to 101
lock = Observables.listeners(camera.projectionview)[1][2]::ObservableLocker
lock!(lock)
camera.view[] = view
unlock!(lock)
camera.projection[] = projection
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively pull observables or JuliaGizmos/Observables.jl#109

@ffreyer
Copy link
Collaborator Author

ffreyer commented Jul 7, 2023

I've been thinking about whether Transformations should be part of what space controls or not, and I'm leaning towards separating them.

Conceptually I see Transformation, i.e. the transform_func, translate!, rotate! and scale! (or the model matrix) as transformations of the input data, and transformations derived from :world, :eye, :pixel, :relative and :clip space as transformations of the "world". In other words Transformation describes how data is placed in the world and the latter group describes the coordinate system of the world (or how the world maps to something displayable).

In practice it seems reasonable to me that any plot can be scaled, translated, rotated or even have a nonllinear transform_func like a polar transform before being placed into a world, even if that world is in pixel space or clip space. (This is also the way it works if we use a different camera rather than adjusting space.)

Detaching space and Transformations would allow you to define a pixel space box for example, and use translate!() it to where it needs to be without directly updating the box. This might be useful when updating that box involves expensive computations. Another example would a polar transform in pixel or relative space. You could effectively change the units of the radius with this.

Changes I'm thinking about making:

  • add transform_func as a kwarg to plot! (which gets moved to Transformation, not attributes)
  • maybe model decomposition into rotation, translation, scale so that it can be absorbed into Transformation (only static model matrices)
  • transform = :full / :linear / :nonlinear / :none as an attribute for controlling transform_func and model application (transformation is used for Transformation)
  • maybe detach Transformation from parents if spaces do not match

@ffreyer
Copy link
Collaborator Author

ffreyer commented Jul 8, 2023

With higher test accuracy this fails

@reference_test "Surface + wireframe + contour" begin
N = 51
x = range(-2, stop=2, length=N)
y = x
z = (-x .* exp.(-x.^2 .- (y').^2)) .* 4
fig, ax, surfaceplot = surface(x, y, z)
xm, ym, zm = minimum(data_limits(ax.scene))
contour!(ax, x, y, z, levels=15, linewidth=2, transformation=(:xy, zm))
wireframe!(ax, x, y, z, transparency=true, color=(:black, 0.1))
center!(ax.scene) # center the Scene on the display
fig
end

because the axis limits have become a bit larger, changing the generated grid. I assume this happens because the data_limits of the contour plot changed to include the z-translation it has. Not sure what caused this specifically, but I think this is something we probably want anyway.

@ffreyer ffreyer mentioned this pull request Jul 15, 2023
25 tasks
@ffreyer ffreyer mentioned this pull request Aug 24, 2023
5 tasks
@ffreyer ffreyer changed the title Clean up projection code Rework projection code Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cleanup projection code
2 participants