feat: Extend .interactive() with tooltip and legend

[joelostblom]

These are some initial ideas for how we could extend the .interactive() method to become more useful. I think having a clickable legend and a tooltip by default would cover many of the basic interactive needs for situations when one does not want more fine-grained control.

vscode-zoom-bug-2024-04-04_09.01.58.mp4

We would still need to add this to the concat charts etc, but I thought I would add to #3393 by creating an example of what this could look like in the code.

I think the addition of the tooltip is pretty straightforward and harmless. The addition of the legend raises the question of what would happen when there is an existing opacity encoding. My suggestion would be to detect this and print a message that the interactive legend has been automatically disabled (and setting legend=False) would suppress that message.

I chose to link the legend click to opacity as I thought this was less intrusive than color or a transform filter, but happy to hear other thoughts (I know that e.g. plotly filters points based on legend clicks)

close #3393

[joelostblom]

I am thinking it's more elegant if we set the legend encodings only to encodings that actually exist in the spec. However, to_dict(validate=False) currently does not work for channels as per #3075.

It is possible to do something like this instead, but it seems clunky:

[
    enc for enc in vars(chart.encoding)['_kwds']
    if not isinstance(chart.encoding[enc], alt.utils.schemapi.UndefinedType)
]

It is an order of magnitude faster thought, so maybe that it the way to go:

[binste]

I'll need to find some more time to review and be helpful but I really like the idea, thanks for getting started on this! I received the feedback from Plotly users that Plotly does more "interactive" things out of the box so I think there could be demand for this.

[dwootton]

Really excited to see the progress on .interactive! Stefan makes a great point – ease of "getting to first interaction" makes for a positive first impression. This aligns with the user feedback we gathered during Altair Express testing.

Since this PR is still a draft, consider the following discussion points rather than direct implementation feedback:

Behavior Across Concatenated Views: How should filtering behave in multi-view charts? In Altair Express, confusion arose when interactions were chart-specific. Propagating interactions across the entire spec proved more intuitive (ie alx.highlight_brush() + my_multi_view_chart added a brush to each chart and cross filtered the entire visualization. Would a similar approach make sense here?

For example, I would probably expect that my histogram should filter to just the selected Origins:

concat_views_behavior.mp4

Interaction Effect: I love the opacity fade effect for scatterplots. Feels intuitive and looks nice when plots are relatively balanced like the cars datasets. However, for broader chart types, I've found filtering generally provides better usability. For example with bar charts, fades don’t rescale the y axis and as such they're less helpful for investigating lower-count bars.

altair_bar_filter.mp4

Filtering out elements could be more useful and would align with plotly's behavior. In our user study, people seemed to prefer filtering when there were outliers in their data.

outlier_filter.mp4

Great work bringing up changes for .interactive; Let me know your thoughts on these points. Happy to brainstorm potential on trade-offs. Longer term might make sense to roll up some of altair express's code to .interactive.

[joelostblom]

Thanks both! I agree that this first interactive impression is important and it's great to hear that this is backed up by your research findings @dylan. Thank you so much for sharing some of the insights from your research!

Interaction Effect: I love the opacity fade effect for scatterplots. Feels intuitive and looks nice when plots are relatively balanced like the cars datasets. However, for broader chart types, I've found filtering generally provides better usability. ...

I 100% agree with this. I think the fade looks nicer and it doesn't have the effect of the axes domain rescaling upon selection with creates a somewhat jarring/jumpy behavior at times. But it is not as general and as you point out, it is not at all useful when there are outliers compressing the rest of the data. I actually switched a personal project over to highlighting to filtering just yesterday for this exact reason and I will update the PR do do the same thing here.

Behavior Across Concatenated Views: How should filtering behave in multi-view charts? ...

This is a great insight and something I didn't even think about. Thanks for sharing it! My preference would be propagation across the entire spec and with filtering instead of opacity highlighting that would be more straightforward.

Longer term might make sense to roll up some of altair express's code to .interactive.

I'm very receptive to plan ahead for this if there are things you already have in mind and you think we should take into account when designing this PR (and you're more than welcome to help out implementing it too if you have time available).

---

What do you all think about keeping the highlighting behavior on control clicking the legend and having filtering on regular click? On the one hand I worry that we are overloading the legend interaction, but on the other hand, I don't think this would get in the way for people who only clicks the legend without holding control. I can't help but to really like the visual effect of highlighting...

If either of you have more ideas of what we could do here, I would be happy to adjust this PR further with the goal of giving the most helpful default interactivity for everyone who is not interested in setting up their own bindings. One aspect I think would be useful would be to have the cursor change to a "clickable hand" when hovering the legend to indicate that it is indeed. It seems as if this could be a simple change as marks in Vega/Vega-Lite already supports a cursor parameter, but after a brief glance at the source, I can't say that it is immediately obvious what to do so maybe we have to wait for vega/vega-lite#4155...

[mattijn]

Given the extensive research that has been put in place on this topic by @dwootton, and his offer to investigate how we can integrate this into the .interactive() method. I think we should not rush this and come up with a deliberate approach that is in-line with the best practices discovered by altair-express.

[joelostblom]

@dwootton I was working on this today to implement your suggestion, but ran into the issue of how to keep the legend entries constant when the legend is used to filter the data. I opened vega/vega-lite#9360 with more details, but thought I would also mention it here in case you have a clever way to solve this in altair express.

[dwootton]

Ooh yeah I remember that. The crux of the issue is that in the data filtering occurs upstream from other options in the dataflow graph. As such, when the color scale is recomputed, it is recomputed over the filtered data, resulting in the behavior seen. Three ways to get around this:

1. Compute color domain non-reactively.
   When you set reactive=False for a parameter, the parameter will not update in response to changes in their upstream dependencies. By adding two parameters and then setting the color domain to be driven off the non-reactive parameter, you can reference the initial color legend.

"params":[....
      {
         "name":"param4", // this parameter will grab the domain from the color scale. 
         "expr":"domain('color')" // note: may need fancier logic to handle cases where there are independent color scales.
      },
      {
         "name":"color_domain", // stores the domain of the original color scale at initial load 
         "react":false, 
         "expr":"param4"
      }]

As this breaks VL's schema, I'm not sure this approach would be suitable for the altair project. However, out of the other options, this feels the cleanest as it doesn't require major modifications to the specification or outside computation.

2. Create a layered visualization using their provided view with opacity=0 marks.

This will keep the color scale set to unfiltered data but hide any marks from appearing on the visualization.

3. (Outside of VL) Precompute the domain in python and add it to the color encoding field.

This is probably the easiest option, but from the chart object, you can grab the field from the color encoding and then precompute the unfiltered domain as the domain for the color encoding.

[joelostblom]

Thank you for the detailed and helpful answer @dwootton ! Sorry it took me a while to get back to you, I wanted to find time to see if I could add your first option to Vega-Lite and finally managed to put together a PR in vega/vega-lite#9374. I think your first option is the cleanest and it would be great to have this supported in the official Vega-Lite grammar. Since the data can be passed as an URL etc, we must make assumption if precomputing the color domain and also rely on a dataframe library, which we are trying to avoid (but this is what I use most of the time personally).

[mattijn]

VL5.20 was released including vega/vega-lite#9374 🙌️. Thanks for releasing @kanitw!

[joelostblom]

Rebased on #3525 and implemented the suggestion above with using react: false in the parameter setup to create the constant color domain. Tasks left to take care of:

• Sort the legend. I can't find a way to sort the array produced bydomain().
• Figure out how to support ordinal encodings as it seems like using domain() is incompatible with ordinal color schemes Marks do not show when domain() is used with an ordinal scale range vega#3959. Converting to nominal and sorting would be one solution but the colorscale would then be the nominal one instead of the expected ordinal one.
• Support the size encoding. For some unknown reason, when the size encoding is used, no marks are produced in the graph. It works when it is quantitative but not nominal.
• Decide if the tooltip parameter should take a list of encoding channels or just be a boolean
  • (edit) Thinking about this more, I believe that a boolean is fine since it is easy to just switch to the tooltip encoding when wanting to pass a list
• Write tests
• Update docs

[mattijn]

While testing this option with zoomable scales and fixed legends in VL I noticed that the selection of the legend is emptied of all values (resetting the domain) upon a mouseup event on pan. Raising here, as that seems undesirable? See following animated gif (Open the Chart in the Vega Editor):

And when trying the following Altair specification using this PR:

import altair as alt
from vega_datasets import data

source = data.movies.url
predicate = (alt.datum.IMDB_Rating == None) | (alt.datum.Rotten_Tomatoes_Rating == None)

chart = alt.Chart(source).mark_point(invalid=None).encode(
    x="IMDB_Rating:Q",
    y="Rotten_Tomatoes_Rating:Q",
    color=alt.when(predicate).then(alt.value("grey"))
)
chart.interactive()   

It now will give the following error:

TypeError: Then.__init__() got an unexpected keyword argument 'condition'

I don't think this option is currently covered in the new tests added within this PR.

[dangotbanned]

@mattijn I will take a look tomorrow, just added a commit to force a fail.

Seems to be related to #3543 more than Chart.interactive I think

[dangotbanned]

@mattijn just pushed a fix in #3553

[joelostblom]

@mattijn in response to your comment:

I noticed that the selection of the legend is emptied of all values (resetting the domain) upon a mouseup event on pan. Raising here, as that seems undesirable?

I agree that it is undesireable, but it is a general issue in Vega-Lite rather than specific to this PR, see for example this example.. I think the solution would be to fire the legend selection event only on pointerup events that are not part of dragleave. I don't know if it is possible to combine and negate selectors like that so I asked on slack if anyone knows.

[dwootton]

Got this working in vegalite! This approach uses an event stream filtered to legend marks such that only click events on the legend are captured for the point selection. It also adds a clear behavior on double click, similar to the clear behavior for interval selections.

Working example from above.

[joelostblom]

More VL magic from @dwootton, thank you! Double clicking to clear is a good idea, but it seems like it is not possible to select multiple legend items with this solution, do you think there is a tweak that allows that to happen?

[dwootton]

You should be able to use Shift + click to do a multi-select! I'd recommend keeping this approach as its the same behavior that's instrumented across all point selections by default.

If you wanted to get multi-select without shift+click, you can add the toggle property to allow for points to be added with just clicks like:

Example

[joelostblom]

That's odd, it doesn't work for me to shift + click to select multiple with your example (in either Firefox of Chromium). If I remove the two occurrences of "markname": "Origin_legend_symbols", shift + clicking works as expected

[dwootton]

Hmm... I just tried with safari, chrome, edge, and firefox on my mac and it seems to be working. Do any errors or logs appear when you shift click on the first example?

[joelostblom]

So interesting... I don't see any errors or logs in the Vega console. The browser debug console just shows the following (and it shows the same for the working spec):

I'm on Linux, so that is a difference, but it doesn't seem like that should impact anything here. Let's hear from someone else if it is working for them or not.

[mattijn]

Thanks for pushing this further @dwootton, pretty awesome! I also think I understand why you have mixed results. I noticed that @dwootton is clicking on the symbols in the legend and I think that @joelostblom is clicking on the labels in the legend.

If you change "markname": "Origin_legend_symbols" to "markname": "Origin_legend_labels" (2x) than you can do multi-select with shift + click on the legend labels. Clicking on the legend symbols behaves as if there is a toggle property (no need to hold shift for multi-select in that case).

See the Chart in the Vega Editor

I also changed "clear": false to "clear": "dblclick" for the zoomable_scales selection parameter. I think that is consistent with current behavior.

[joelostblom]

Great catch @mattijn! That's it! Ideally we would want both the legend label and symbol to trigger the same selection behavior. I guess we can duplicate the color_selection parameter and legend binding once each for "Origin_legend_symbols" and "Origin_legend_labels", but maybe there is a cleaner way?

I still see some unexpected behavior with the spec you posted @mattijn. I start by clicking the "USA" label to select only that to only and then shift+click the label of "Japan" so that I have those two labels selected. Now, if I regular click the "Europe" label, only "Europe" would be selected (as expected). However, if I instead would have regular clicked the "Europe" symbol, all three labels are selected (so it behaves as a shift click for some reason; I would expect only Europe to be selected in this case too, just as when the label is clicked)

I also changed "clear": false to "clear": "dblclick" for the zoomable_scales selection parameter. I think that is consistent with current behavior.

+1