Show within-target progress when using tar_watch for dynamic branching

[mattwarkentin]

Prework

• Read and agree to the code of conduct and contributing guidelines.
• If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• For any problems you identify, post a minimal reproducible example like this one so the maintainer can troubleshoot. A reproducible example is:
  • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
  • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
  • Readable: format your code according to the tidyverse style guide.

Question

Hi @wlandau,

I was wondering if it would be reasonable to have it so the dependency graph in tar_watch() could somehow show within-target progress when using dynamic branching. In other words, somehow show how many branches out of N total branches have been built.

I suppose this would mean adding some functionality to tar_visnetwork() since this powers tar_watch(). But it seems to me that at runtime a few things are known:

1. How many total branches belong to a given target

2. How many of those branches requiring being built/re-built

tar_watch() knows once ALL branches are built for a target, but not at a more granular level.

If this information could be shared in some way with tar_visnetwork(), it would be nice to present it to give a sense of within-target progress. Also, with dynamic branching sometimes we may not know how many branches we are even creating, so one cannot easily figure out the progress just by looking at how many branches have finished so far by the textual progress updates.

Thoughts?

[ginolhac]

A bit geeky, but that would be really great (and fun)!

[mattwarkentin]

I'm still thinking this over but maybe tar_visnetwork() should be left alone. Perhaps another option would be to add another bs4Card to the same column in the dashboard, and have it display BS4 progress bars?

Something like:

I really like the dev version of BS4 also, so I took that for a spin below...

Dev version of bs4Dash

Screen.Recording.2021-01-20.at.2.56.04.PM.mov

[wlandau]

I like #273 (comment). We will need to add more information to _targets/meta/progress to make that happen, but it's doable.

[mattwarkentin]

I think it could be a good addition to the dashboard given how important dynamic branching is to targets as a whole - so long as adding branch-wise progress updates to _targets/meta/progress isn't too much of a hassle.

[wlandau]

Yeah, I'm actually really stoked for this! Most of the hassle is refactoring the methods of the progress class to accept whole targets (so more detailed progress information can be written to _targets/meta/progress). That appears to be working in my local copy.

[wlandau]

#276 should expose enough progress info to build this new feature in the app.

[wlandau]

We now have specific information on errored and cancelled branches too. How should we display that?

[wlandau]

Maybe just a colored/stacked bar chart in ggplot2? As much as I like the barber bar animation in #273 (comment), the progress indicators will not move until the whole app refreshes anyway.

[mattwarkentin]

I do like the formal progress bars, they don't need to be animated, even thought the animation just gives the visual perception of activity (the bars only physically progress when the app is updated and there is progress to add).

Want me to pull in the changes are try to reproduce something akin to what I hacked together above? I feel obliged (and I have some ideas in my head) to offer a PR since I did suggest this idea.

[wlandau]

Sure, those earlier sketches were really nice. I think that could help our pre-PR discussion.

the animation just gives the visual perception of activity

That's an interesting issue. tar_watch() currently does not know if the main process is actually running, and it's tricky to implement that because the session could crash, and it might not even be running on the same node. I doubt we will be able to completely reliably indicate whether the main process is actually running. So perhaps it is best not to give a false impression?

If we do static visuals, here are a couple thoughts:

1. Stacked bar charts like I mentioned before.
2. Separate boxes of Show within-target progress when using tar_watch for dynamic branching #273 (comment) for errored and cancelled targets. The cancelled and errored ones could be collapsed by default.
3. Separate panels within a single box for (2).

I don't like (1) as much as before. (2) would look nice because we could match the boxes with the existing color scheme. (3) could conserve real estate but may awkwardly cram a lot together.

[mattwarkentin]

Ahh yes, good point about the false impression of activity. I don't think the animation is necessary anyway, so we can forget that.

I agree that something like 2/3 would be good. I've already started to play around with some aesthetics, but I think I need to wait for #276 to be merged before I can actually try implementing anything.

[wlandau]

Should be merged now. Thanks for the help.

[wlandau]

Another consideration: for the sake of performance, targets does not write to _targets/meta/progress for skipped stems or branches. That means the progress bars may never get to 100% even for a successful pipeline. Maybe the visuals shouldn't create that expectation either?

[wlandau]

I was working on a tar_progress_branches() function to make the visualization easier

> tar_progress_branches()
# A tibble: 2 x 6
  name  branches running built canceled errored
  <chr>    <int>   <int> <int>    <int>   <int>
1 y            2       0     2        0       0
2 z            2       0     1        0       1

And now I think we should just stick this table in the app instead of a graph. I know it is not as fancy, but it is clear, concise, and complete. And we can always add colors to the cells.

[ginolhac]

for the table in the shiny, it could be a DT one with barplot inside like in the documentation https://rstudio.github.io/DT/functions.html. That also always the sorting on the user side.

[wlandau]

That's a cool feature, I didn't know about it.

[wlandau]

But I would actually prefer gt because it is simple and crisp.

[wlandau]

I got started on the app part in #278. @mattwarkentin, I know we had discussed a PR from you before, but today I ended up with such a clear idea of what I want this to look like. Sorry about that. If you have specific stylistic changes, please let me know or feel free to submit downstream PRs.

[mattwarkentin]

That's okay, you get final say. Does the branches column value for a type=='pattern' returned from tar_progress(fields = NULL) not represent the total number of branches?

#> # A tibble: 4 x 5
#>   name       type    parent branches progress
#>   <chr>      <chr>   <chr>     <int> <chr>   
#> 1 x          stem    x             0 built   
#> 2 y_29239c8a branch  y             0 built   
#> 3 y_7cc32924 branch  y             0 built   
#> 4 y          pattern y             2 built

[wlandau]

It should be the total number of branches that targets is planning to check or run.

[mattwarkentin]

Doesn't the table also suffer from the same issue that it only includes what tar_progress() knows about?

[wlandau]

Is there other information we would want to include?

[mattwarkentin]

Maybe not, but I thought the motivation for the table was that progress bars would be incomplete or misleading.

[wlandau]

The table includes the same data (potentially more actually) but fewer assumptions: for example, that the pipeline is running or the progress bars are going to reach 100%.

[mattwarkentin]

Hmm, fair enough. I would've thought tar_watch()/tar_progress() is really only interesting and relevant in the context of running a pipeline. But perhaps there is a compelling use-case for calling tar_watch() outside of running a pipeline that I haven't run into yet.

[wlandau]

Hmm, fair enough. I would've thought tar_watch()/tar_progress() is really only interesting and relevant in the context of running a pipeline.

They are only interesting when the pipeline is running, but they cannot promise that a pipeline is running. The hard guarantee is all I want to avoid.