modal widgets

[cmyr]

Doing modal dialogs correctly involves a lot of platform coordination, and so I would like as a stopgap to add support for modal widgets; these will be usable for things like a drop-down combo box or a setting panel, allowing us to implement working versions of these things much more easily than if we try and hook platform windows for all of them.

There are some API questions: for instance, a combo box needs to be presented with its current selection aligned at a particular location, and also needs to be maximally visible since we won't be able to draw outside the bounds of the containing window.

The general architecture, though, should be pretty simple: Only one modal will be allowed at a time, it will exist at the window level, and it will communicate back to the main widget tree by sending Commands when an item is selected. We could also add life cycle events for showing and hiding a modal.

I'm not sure where exactly this is on the roadmap, but it's probably something we should do soon. I know that @xStrom has been working on some related stuff, and so if they're interested in exploring this they would be very welcome; otherwise (unless anyone else is interested) I'll take it up sometime after the life cycle work.

[luleyleo]

I decided to write down my thoughts on the topic for #1009 and it got a bit longer and less related to the specific implementation proposed there than expected, thus I'm posting this here instead.

Requirements

The use cases for modals that come to my mind are tool-tips, dialogs, pop-up menus and general overlays such as drawers or completions lists for example.

Due to those, modal should allow nesting, for example a dialog containing a combo box or a button with a tool-tip.

Positioning

There are three types of positioning that would be required to fulfill the mentioned use cases:

• Whole window (dialogs)

• Widget relative (pop-ups, overlays)

• Cursor relative (tool-tips, context menus)

The max size of the modal would be pretty much unbound for relative ones and the windows size for those like dialogs.

For the relative ones, an automatic 'keep-in-window' / 'keep-in-screen' would be needed. Maybe we could do this by passing the remaining down-right space as max bounds and adjust position if it returns a size larger than those max bounds? This would be fairly easy but somewhat breaking the idea of max bounds, maybe there is a better solution.

Input Events

Obviously, modals must receive input events first and should then be able to decide about their propagation. Tool-tips for example should never swallow events and there are cases such as a completion list for a text box, which would only want to catch mouse input but not block the keyboard

Focus cycling (tab) should be contained inside the modal. For this we will probably need a separate focus list for each modal.

Painting

In theory, painting is pretty normal with the only difference that modal always paint over (after) their parent.

Also, dialogs often want to dim the main window, while tool-tips don't and would need a way to control their background.

API

My preferred modal API is a bit different to what #1009 is proposing and would allow modals to be normal, in-tree widgets, representing the same data as the containing widget. This would be the least surprising to users and would avoid doing the app-state-dance which we need to do with menus already and hopefully can get rid of at some point.

For this druid would provide a ModalHost, which can either contain just a modal in the case of dialogs for example, but also a modal plus the widget the modal is associated with.

This dialog will be provided with some way to open and close the modal, which could either be a function Data -> bool or a selector.

To give some ideas what I imagine:

fn modal_dialog() -> impl Widget<String> {
    Modal::dialog(
        slectors::SHOW_MY_DIALOG,
        Label::dynamic(|data, _| data.clone())
    )
}

fn button_with_menu() -> impl Widget<bool> {
    Modal::relative(
        |data| data,
        Menu::from(description),
        Button::new("Menu"),
    )
}

fn button_with_tooltip() -> impl Widget<()> {
    Modal::cursor(
        selector::HOVERED,
        Label::new("Tooltip"),
        Button::new("Button"),
    )
    .controller(OnHover(selector::HOVERED))
}

Implementation

There are two ways how we can implement modals, either simulate modals inside a single window or using platform windows for modals, allowing them to leave the window they are attached to.

One thing I'm uncertain about is whether the single window version has any advantage over native windows, or if we can replace it completely if native windows are available.

Implementation wise I see a couple differences:

Single Window	Native Windows
All events must be routed to the modal first in order to find out if it is interested.	Modals should have access to events in their parent window.
Painting has to be deferred artificially until all parents have done theirs.	Due to paining on a different surface, parents are not affected by modals.
When no space is left in the window, relative modals must be repositioned in the window. Also, dialogs can never be larger than the window containing them.	Same as for single window modals, but in relation to the screen instead of the window. Dialogs could be larger than their parent window, at least in theory.
Window decoration must be offered by druid.	Decoration can be provided by the platform (Wayland being special).
Transparency and blur should be doable with piet normally. So no blur right now, to my knowledge at least.	Transparency and blur might be offered by the platform or not. If transparency is offered but no blur, piet won't be able to do it either.

I would imagine menu bars to be another issue, but as a long time Gnome user I'm certainly not the best one to give advice about how those should work.

The single window version will probably be overall easier to implement but it causes some issues with painting. Native windows will be quite challenging to make reliable and iron out the platform specific quirks, but those aside they seem rather straight forward if we have single window modals working (famous last words).

I've had some experiments with this, and it started fairly well. The lifecycle and update work out of the box, events can be passed to modals first by adding an InternalEvent::RouteModalEvent.

Layout also seems rather easy with this approach. Relative positioning would be just like normal layout only that the modal would be able to leave its parents box. For window relative modals like dialogs, I think it would be sufficient if we add some way to position a widget relative to the window instead of its parent.

The major pain point with in-tree, single window modals is painting. The issue here is that we somehow need to defer it until all parents finished painting, which makes in-tree modals currently impossible. What we would need here is either support for layers which would allow us to composite them in the order we want after all the drawing was done normally, or we implement some sort of 'buffering' in piet, e.g. we record what would have been drawn and replay that later. This feels rather hacky though and I would certainly prefer using layers for this. If we can resolve this issue, then I think in-tree modals are feasible.

[futurepaul]

In regards to deferring painting, isn't that what z-index does?

[jneem]

This sounds good! I think an in-tree API will be much nicer than the global version in #1009 (which I'll close). I didn't follow the API part, though -- based on the tooltip example, my understanding was that Modal would be a wrapper around a normal widget, which might occasionally also decide to display a modal widget in addition. But that understanding doesn't work with the dialog example -- where does that widget live in the tree?

I also didn't understand how the selectors are supposed to work. Is the intention that on a hover, the OnHover controller sends a command to its child? If that's the case, why not have OnHover take a callback instead (which would get a &mut EventCtx, a &mut ChildWidget, and maybe some Data and Env?

[luleyleo]

@jneem

my understanding was that Modal would be a wrapper around a normal widget, which might occasionally also decide to display a modal widget in addition.

My idea (which is nowhere near final) was that a modal is first and foremost just that, a modal. If the modal is associated with some content, which is only relevant for relative modals such as tooltips, then the modal would act as a wrapper. Writing this down it sounds a bit irritating though, probably we can do better / more consistent.

That thing with selectors was just some spit balling really. The idea was that you could provide some sort of activation selector, that can be submitted to show the modal. So instead of the modal mandating what selector should be used to show it, you tell it what selector it should react to. The OnHover was just some ad-hoc controller that emits the provided selector when the widget is hovered, purely meant as an example, not as part of the API.

---

@futurepaul

In regards to deferring painting, isn't that what z-index does?

The problem I head with that was, that I wanted to draw an entire child tree of widgets on a different z-axis which I though was not possible. After some more experimentation it turns out to be possible, but pretty awful as I have to wrap the content widget in a Rc<RefCell<..>> and clone + move that as well as data and the Env into the paint closure. But I got something working with that approach and realized that invalidation might be the most troublesome offender with my (admittedly still very hacky) design, that will need some more investigation though.

[cmyr]

Excuse my having skimmed a bit, but I want to suggest that tooltips and modals are fundamentally different. As I see things, a tooltip is just something a widget draws that can occlude other widgets, and (possibly) be drawn outside the window, but is basically part of that widget. Importantly it doesn't have any influence on event handling.

Modals are basically entirely separate widget graphs (and with platform modals, separate windows) that have their own event handling behaviour, their own focus chains, and fundamentally behave at the level of a druid::Window. The ModalHost solution from runebender is a hacky way of getting a fairly simple 'modal-like' behaviour, but is definitely not the ideal solution.

[xStrom]

Great summary @Finnerale!

Single window vs platform windows

I wasn't a fan of the single window apporach before and reading the summary here I think I'm even less of a fan. It just seems such a detour. To enable proper window-overflowing behavior we will need platform windows eventually anyway and it seems that for the single window approach there are just too many problems that need solving which won't be relevant for the eventual platform window based solution. Thus I personally wouldn't touch the single window approach at all, but if someone wants to I would suggest at least being super hacky and having a destined-for-the-bin attitude about it. No sense in solving a bunch of the issues for code that will exist for just a few months at most.

Full-blown modal dialogs

I'm not sure that they should be anything different than a bool option when creating a new window. That's what they are for the platform and this would keep the number of concepts down that the developers have to learn.

Giving that modal dialog access to only a subset of Data can be useful, but the same is true with non-modal windows, and thus I think this is something that can be thought about in a more generic multi-window context.

Modal elements (comboboxes, tooltips etc)

I think having them look to the developer as close to a normal widget as possible would be awesome - again it keeps the number of concepts low.

In that sense the API that should be the most ergonomic is the one that developers are mostly using.

Button::new("Button").tooltip("Click me!");
Combobox::new(vec!["One", "Two"]);
// etc ..

The API underneath that which enables this high-level API could trade some ergonomics for power because it would only be used by us and by developers building advanced custom widgets.

Input events

I think the modal windows could just receive all input events (as they mostly do by default already) and then decide which they will send to their parent window.

[cmyr]

I do agree that if we can reasonably just use platform windows, that's great.

I have one concern, which is the web. It feels like on the web we're going to need to implement our own modal shim, so we may have to do the 'single window' work anyway, and so if the platform window stuff was going to be a lot of work, doing the single window stuff first could make sense.

[xStrom]

The web modal solution could conceivably live in the druid-shell layer though. I guess this just ends up coming back to the question of what a window means for the web backend.

[luleyleo]

Yeah, I think it will be best to go straight for platform windows, because the single window approach comes with just as many problems and would be wasted work once we have support for platform windows.

It would probably be the best if druid-shell offers modal windows on all platforms and emulates them when they are absent (web, Android), which seems to be the plan with #1055.

---

@xStrom

In that sense the API that should be the most ergonomic is the one that developers are mostly using.

I would expect two levels of API:

1. A low-level API which can open modal windows and control their position, size and content, probably through contexts.
2. A high-level API which offers convenient widgets such as dialogs and tooltips.

I'm not sure that they should be anything different than a bool option when creating a new window. That's what they are for the platform and this would keep the number of concepts down that the developers have to learn.

Sure, I such a command based API can always be done through a custom utility widget, if you want it in your app.

[rjwittams]

I’ve done some work on this for general ‘sub windows’ at https://github.com/rjwittams/druid/tree/sub-window
It is not limited to modals or any particular kind of window. The ‘data context’ of the sub window is determined by the “parent pod” ie where it was launched from, and bidirectional syncing of that data is working via commands

[madbonkey]

One thing I'm uncertain about is whether the single window version has any advantage over native windows

I'm currently proof-of-concepting a UI with Druid for a machine controller (works great by the way, love this library). The UI will run in "kiosk mode" and be displayed on a touchscreen device, which makes me wonder about the whole multiwindow stuff. The design spec for the main menu says it should pop out from a sidebar over the current screen's content, similar to a modal/overlay but only covering part of the viewport. The design also specifies tooltips, dialogs and other modal-like things like notification bars. Overall, these widgets will be pretty web-app-like in appearance and behavior, and have different characteristics of "modal"-ness.

Currently, my PoC assumes a singular window in fullscreen/kiosk mode (I haven't actually run the UI on the target device, so I've yet to see how this would work at all and if my assumptions are right). Ideally I'd like the choice over having modals work as actual windows or as widgets living in the window's tree, as it seems to me there could be implications to having it exclusively one way or the other.

[cmyr]

@madbonkey what backend are you using for the controller?

If we end up pushing forward with the web backend for druid it will require the ability to have modals that are rendered within the main window, so ideally that work could be shared.

[madbonkey]

@cmyr I'm using the regular one (I'm a total newbie to the rust ecosystem, I don't know the words yet), so no WASM or anything. That might be an additional use case in the future, but the one I have now is somewhat special, too, in that the UI would run fullscreen on a touch panel with no way for the user to "escape" it. Modals/overlays would need to be able to draw on top of that - if there's new windows involved they must be customizable and controllable to allow for the semantics we need (modal != dialog != overlays regarding their intent).

I just have a tingling of lizard-brain-buzz thinking about expressing these semantics between widget trees across windows, it seems like it could be quite the hassle to get a smooth "kiosk-mode" experience going. I might be totally wrong but I wanted to add this kind of application to the discussion.

[cmyr]

Sorry, to clarify when I say backend: The druid crate runs on various platforms, with each platform having an implementation in druid-shell. This is generally selected automatically based on the deployment target. Is your controller running linux?

In any case, the situation you're describing would be implemented in druid-shell in some way; it would handle creating the 'windows' which would actually be backed by a single underlying 'render target'.

[madbonkey]

The actual deployment is a bit complicated, but yes, it'll all run on a Debian host in the end (or something similar), development largely happens on macos.

[cmyr]

and you're using x11 or something, not some custom windowing stack?

[madbonkey]

Correct, it's X11 :)