Prototype for module-internal chain of tasks #363

makortel · 2019-07-01T15:14:31Z

PR description:

This PR adds helpers for a chain of TBB tasks internal to the ExternalWork module. These helpers can be used to implement e.g. a chain of tasks that alternate CPU and GPU work, or a loop of GPU work where the decision of continuing is done in the CPU. This PR also adds CUDAScopedContextAnalyze mentioned in #355 (comment).

In short, the CUDAScopedContextAcquire can be used to queue a "next task" instead of the produce() with

void ...::acquire(...) {
  CUDAScopedContextAcquire ctx{...};

  ctx.insertNextTask([this](WaitingTaskWithArenaHolder h) {
    CUDAScopedContextTask ctx{this->ctxState_, std::move(h)};
     ...
   }):
}

The commit 203cd7d experiments with the interface by providing directly the CUDAScopedContextTask to the user at the cost of interface inconsistency between acquire() and the function

void ...::acquire(...) {
  CUDAScopedContextAcquire ctx{...};

  ctx.insertNextTask([](CUDAScopedContextTask ctx) {
     ...
   }):
}

I would like to hear opinions of others which one would you prefer.

PR validation:

Added test runs.

This guarantees proper exception delivery from the function(s) to the WaitingTaskWithArenaHolder.

…ngTaskHolder()

makortel · 2019-07-01T16:13:53Z

@vkhristenko this is the PR I mentioned

makortel · 2019-07-01T16:14:23Z

@VinInn this mechanism can be used also for "re-running a producer with more memory when initial memory was not sufficient"

…lities taking just a lambda

makortel · 2019-07-05T13:45:44Z

FWCore/Concurrency/interface/WaitingTaskWithArenaHolder.h

+            holder.doneWaiting(*excptr);
+            return;
+          }
+          func();


I'm a bit unhappy about the replication of WaitingTaskWithArenaHolder here, especially since the code-to-be-copied would be only 4 lines.

fwyzard · 2019-07-10T06:07:08Z

Validation summary

Reference release CMSSW_10_6_0 at b45186e
Development branch CMSSW_10_6_X_Patatrack at f340840
Testing PRs:

Prototype for module-internal chain of tasks #363 at 7895569

`makeTrackValidationPlots.py` plots

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

/RelValZMM_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_design_v3-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

logs and `nvprof`/`nvvp` profiles

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.51
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.52
- ✔️ step3.py: log, visual profile and summary
- ✔️ profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
testing release, workflow 10824.51
- ✔️ step3.py: log, visual profile and summary
testing release, workflow 10824.52
- ✔️ step3.py: log, visual profile and summary
- ✔️ profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

/RelValZMM_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.51
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.52
- ✔️ step3.py: log, visual profile and summary
- ✔️ profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
testing release, workflow 10824.51
- ✔️ step3.py: log, visual profile and summary
testing release, workflow 10824.52
- ✔️ step3.py: log, visual profile and summary
- ✔️ profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_design_v3-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.51
- ✔️ step3.py: log, visual profile and summary
development release, workflow 10824.52
- ✔️ step3.py: log, visual profile and summary
- ✔️ profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- ✔️ step3.py: log, visual profile and summary
testing release, workflow 10824.51
- ✔️ step3.py: log, visual profile and summary
testing release, workflow 10824.52
- ✔️ step3.py: log, visual profile and summary
- ✔️ profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

Logs

The full log is available at https://fwyzard.web.cern.ch/fwyzard/patatrack/pulls/125cfeca1ba29106d05e2d5c98456f59118c5442/log .

fwyzard · 2019-09-09T15:26:17Z

About the insertNextTask() syntax, I prefer the second variant, as it is leaner and avoids repeating the same couple of lines.

makortel · 2019-09-09T16:24:55Z

Couple of items from a chat with @fwyzard

Change the order insertNextTask() and CUDA async work launches to be more intuitive (the order doesn't really matter)
Make the LIFO stack property of the "next task" more clear in the interface (insertNextTask() -> pushNextTask()) and examples
Make it clear in the documentation that the caller of acquire()/lambda has a copy of the WaitingTaskWithArenaHolder guaranteeing that the next task will be run only after the current one has finished

…entation

makortel · 2019-09-09T21:52:36Z

@fwyzard I've addressed the points from our discussion (#363 (comment)), could you take a look again?

fwyzard · 2019-09-09T23:28:09Z

Thanks, the latest set of changes make it clear what the behaviour of the tasks should be.

fwyzard · 2019-09-09T23:29:38Z

The only aspect that it's not clear to me is why

Note that the CUDAService is not available (nor is any other service) in these intermediate tasks.

I thought the Services were available to the underlying thread, not to the individual task ?

makortel · 2019-09-10T14:02:27Z

@fwyzard Services are a rather complex subsystem, approximately requiring a valid ServiceToken on the thread calling Services. The framework holds a ServiceToken during the execution of any module methods, but such token does not exist on arbitrary TBB tasks and thus Services can not be used.

Here I decided to be lazy and not support Services in these tasks (and see how far we get with it).

makortel added 7 commits July 1, 2019 17:00

Introduce make_waiting_task_with_holder() function

143329d

This guarantees proper exception delivery from the function(s) to the WaitingTaskWithArenaHolder.

Introduce CUDAScopedContextTask, and inserNextTask() and replaceWaiti…

c0f99ee

…ngTaskHolder()

Make TestCUDAProducerGPUEW asynchronous

42c800d

Add TestCUDAProducerGPUEWTask example

f82c4fc

Add CUDAScopedContextAnalyze

40ad83d

Document the chain of tasks

203cd7d

Experiment passing CUDAScopedContextTask to the intermediate tasks

6926a6c

makortel mentioned this pull request Jul 2, 2019

Move event and stream caches, and caching allocators out from CUDAService #364

Merged

Add also make_lambda_with_holder() that is useful with other task uti…

7895569

…lities taking just a lambda

makortel commented Jul 5, 2019

View reviewed changes

Rename insertNextTask() to pushNextTask(), clarify examples and docum…

0bd7874

…entation

fwyzard merged commit 703ee96 into cms-patatrack:CMSSW_10_6_X_Patatrack Sep 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prototype for module-internal chain of tasks #363

Prototype for module-internal chain of tasks #363

makortel commented Jul 1, 2019

makortel commented Jul 1, 2019

makortel commented Jul 1, 2019

makortel Jul 5, 2019

fwyzard commented Jul 10, 2019 •

edited

Loading

fwyzard commented Sep 9, 2019

makortel commented Sep 9, 2019 •

edited

Loading

makortel commented Sep 9, 2019

fwyzard commented Sep 9, 2019

fwyzard commented Sep 9, 2019 •

edited

Loading

makortel commented Sep 10, 2019

Prototype for module-internal chain of tasks #363

Prototype for module-internal chain of tasks #363

Conversation

makortel commented Jul 1, 2019

PR description:

PR validation:

makortel commented Jul 1, 2019

makortel commented Jul 1, 2019

makortel Jul 5, 2019

Choose a reason for hiding this comment

fwyzard commented Jul 10, 2019 • edited Loading

Validation summary

makeTrackValidationPlots.py plots

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_design_v3-v1/GEN-SIM-DIGI-RAW

logs and nvprof/nvvp profiles

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_design_v3-v1/GEN-SIM-DIGI-RAW

Logs

fwyzard commented Sep 9, 2019

makortel commented Sep 9, 2019 • edited Loading

makortel commented Sep 9, 2019

fwyzard commented Sep 9, 2019

fwyzard commented Sep 9, 2019 • edited Loading

makortel commented Sep 10, 2019

fwyzard commented Jul 10, 2019 •

edited

Loading

`makeTrackValidationPlots.py` plots

logs and `nvprof`/`nvvp` profiles

makortel commented Sep 9, 2019 •

edited

Loading

fwyzard commented Sep 9, 2019 •

edited

Loading