Long callback refactor

[T4rk1n]

Fix raising errors in long callback triggering an infinite update loop. Fix #1821

• raise PreventUpdate output a no_content
• Other errors are printed to console and appears in dev tools ui errors.
• Handle NoUpdate in celery long callbacks. Fix [BUG] Long Callbacks: dash.no_update not supported with Celery/Redis implementations #1974

Replace long callback interval polling

• Fix short interval loop by handling the status of the long callback internally in the renderer when sending callback requests if not completed if it will refetch after interval milliseconds.
  • Fix [BUG] long_callback short interval makes job run in loop #1769
  • Fix [BUG] Long Callback interval_time causing infinite loop #1852
• Add long=False to dash.callback to use instead of app.long_callback.
  • Prefix previous app.long_callback arguments with long_ (interval, running, cancel, progress, progress_default, cache_args_to_ignore, manager)

Long callback parity:

• Support pattern matching.
• Support callback context.

[alexcjohnson]

Now that long callbacks are their own category to the renderer rather than creating more regular callbacks, a couple of extra questions:

• What happens if one of the extra outputs (progress or running) is an input to another callback? I guess that callback should fire every time that input changes, even if that's many times during one execution of the long callback, including after it completes. Does it?
• Do those extra outputs appear in the callback graph viz in the devtools? I'm not sure what this should look like, probably not the same as the final outputs of the long callback but they should be included somehow. Not a super high priority though, we can come back to that if it's more than a quick adjustment.
• What about PMC inputs & outputs? I think you mentioned you expect these are now supported, but there are a number of situations to consider (and test):
  • MATCH - can we make a list of multiple independent input/output sets, each triggering its own instance of the long callback?
  • And does that include the extra inputs and outputs, like can you use MATCH to associate one progress bar and one cancel button with each long callback?
  • ALL - what happens if you trigger a long callback, then add or remove an item in the set? Does the running callback get canceled automatically?

[T4rk1n]

@alexcjohnson

What happens if one of the extra outputs (progress or running) is an input to another callback? I guess that callback should fire every time that input changes, even if that's many times during one execution of the long callback, including after it completes. Does it?

Tried an input on the progress, doesn't update, I used dispatch(updateProps(... for the side updates, what action should it use?

Do those extra outputs appear in the callback graph viz in the devtools?

No.

[T4rk1n]

What about PMC inputs & outputs?

Pattern matching callbacks are now supported for regular inputs/outputs.

And does that include the extra inputs and outputs, like can you use MATCH to associate one progress bar and one cancel button with each long callback?

No, progress/running/cancel pattern matching are not supported and will throw an error.

ALL - what happens if you trigger a long callback, then add or remove an item in the set? Does the running callback get canceled automatically?

No, the long callback continues until canceled/completed, not sure how that would be done.

[alexcjohnson]

Do those extra outputs appear in the callback graph viz in the devtools?

No.

OK. It's going to be a little bit tricky to figure out what that should look like so let's make an issue to come back to this and any other limitations we think could potentially be removed later.

What about PMC inputs & outputs?

Pattern matching callbacks are now supported for regular inputs/outputs.

🌟

And does that include the extra inputs and outputs, like can you use MATCH to associate one progress bar and one cancel button with each long callback?

No, progress/running/cancel pattern matching are not supported and will throw an error.

OK - as long as it's a clear error we can leave that for later as well.

ALL - what happens if you trigger a long callback, then add or remove an item in the set? Does the running callback get canceled automatically?

No, the long callback continues until canceled/completed, not sure how that would be done.

I guess for now I don't really care whether the long callback is allowed to continue (although I'd consider that another limitation, at some point we should make it be canceled as soon as it's known to be moot), but it's important that the first result is ignored when it finally is returned, if at that point there's a different set of inputs.

Here's a sample app that shows the desired behavior: if I type something, then change the number of inputs before the output callback has returned, I should not see the intermediate result appear, only the final state. That's how it works right now with regular callbacks, we should ensure it works the same with long callbacks.

from dash import Input, Output, State, Dash, html, dcc, callback, ALL
import time


app = Dash(__name__)

app.layout = html.Div(
    [
        dcc.Dropdown(id="dropdown", value=1, options=list(range(10))),
        html.Div([], id="container"),
        html.Div(id="out")
    ],
)

@callback(Output("container", "children"), Input("dropdown", "value"), State("container", "children"))
def make_inputs(n, prev_children):
    prev_len = len(prev_children)
    if n > prev_len:
        return prev_children + [dcc.Input(id={"n": n}, value="") for n in range(prev_len, n)]
    return prev_children[:n]


@callback(Output("out", "children"), Input({"n": ALL}, "value"))
def out(vals):
    time.sleep(3)
    return "+".join(vals)


if __name__ == "__main__":
    app.run_server(debug=True)

[T4rk1n]

I guess for now I don't really care whether the long callback is allowed to continue (although I'd consider that another limitation, at some point we should make it be canceled as soon as it's known to be moot), but it's important that the first result is ignored when it finally is returned, if at that point there's a different set of inputs.

It looked like the previous output was discarded in the case of regular callbacks (but job still running). For long pattern matching inputs, the output were not discarded. I've put a check on the output to auto cancel previous jobs.

[alexcjohnson]

💃 Looks great! Once the tests pass, the only other thing I'd like to see before merging is an issue listing the remaining known TODO items:

• pattern-matching with progress / cancel
• progress/cancel in devtools callback graph viz
• do all obsolete jobs now get canceled ASAP, or just the PMC jobs?
• anything else?

[T4rk1n]

do all obsolete jobs now get canceled ASAP, or just the PMC jobs?

Yes

I'll create issues for devtools and pattern matching for progress/cancel/running.