Work around atomicAdd problem on Volta #180

fwyzard · 2018-09-30T18:18:27Z

As reported here, atomicAdd() seems to introduce a synchronisation problem; fortunately it seems that atomicInc() is OK.

fwyzard · 2018-09-30T18:47:48Z

Validation summary

Reference release CMSSW_10_2_5 at a8a031d
Development branch CMSSW_10_2_X_Patatrack at b5135e6
Testing PRs:

`makeTrackValidationPlots.py` plots

/RelValTTbar_13/CMSSW_10_2_2-PU25ns_102X_upgrade2018_realistic_v11-v2/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.8
tracking validation plots and summary for workflow 10824.7
tracking validation plots and summary for workflow 10824.9

/RelValZMM_13/CMSSW_10_2_2-102X_upgrade2018_realistic_v11-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.8
tracking validation plots and summary for workflow 10824.7
tracking validation plots and summary for workflow 10824.9

DQM GUI plots

/RelValTTbar_13/CMSSW_10_2_2-PU25ns_102X_upgrade2018_realistic_v11-v2/GEN-SIM-DIGI-RAW

reference DQM plots for reference release, workflow 10824.5
DQM plots for development release, workflow 10824.5
DQM plots for development release, workflow 10824.8
DQM plots for development release, workflow 10824.7
DQM plots for development release, workflow 10824.9
DQM plots for testing release, workflow 10824.5
DQM plots for testing release, workflow 10824.8
DQM plots for testing release, workflow 10824.7
DQM plots for testing release, workflow 10824.9
DQM comparison for reference workflow 10824.5
DQM comparison for workflow 10824.8
DQM comparison for workflow 10824.7
DQM comparison for workflow 10824.9

/RelValZMM_13/CMSSW_10_2_2-102X_upgrade2018_realistic_v11-v1/GEN-SIM-DIGI-RAW

reference DQM plots for reference release, workflow 10824.5
DQM plots for development release, workflow 10824.5
DQM plots for development release, workflow 10824.8
DQM plots for development release, workflow 10824.7
DQM plots for development release, workflow 10824.9
DQM plots for testing release, workflow 10824.5
DQM plots for testing release, workflow 10824.8
DQM plots for testing release, workflow 10824.7
DQM plots for testing release, workflow 10824.9
DQM comparison for reference workflow 10824.5
DQM comparison for workflow 10824.8
DQM comparison for workflow 10824.7
DQM comparison for workflow 10824.9

logs and `nvprof`/`nvvp` profiles

/RelValTTbar_13/CMSSW_10_2_2-PU25ns_102X_upgrade2018_realistic_v11-v2/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 10824.7
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.9
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.7
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.9
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

/RelValZMM_13/CMSSW_10_2_2-102X_upgrade2018_realistic_v11-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ❌ cuda-memcheck --tool synccheck (report, log) found no CUDA-MEMCHECK results
development release, workflow 10824.7
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.9
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.7
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.9
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

Logs

The full log is available at https://fwyzard.web.cern.ch/fwyzard/patatrack/pulls/5ffdb2df3fe7e11c424be64ce5ae9b2c0e10b9c8/log .

fwyzard · 2018-09-30T19:47:38Z

The single crash was due to cudaErrorMemoryAllocation: out of memory because the validation was running too many jobs at the same time...

fwyzard · 2018-09-30T19:54:39Z

@VinInn looking at the validation results, #179 and #180 taken together seem to reduce the number of duplicate tracks. Just a random fluctuation, or do you think it may have actually have improved ?

Work around atomicAdd synchronisation problem on Volta

ba3123e

fwyzard mentioned this pull request Sep 30, 2018

fix mask in warp #179

Merged

fwyzard added the bug label Sep 30, 2018

fwyzard changed the title ~~Work around atomicAdd problem on volta~~ Work around atomicAdd problem on Volta Sep 30, 2018

fwyzard merged commit e269bdd into cms-patatrack:CMSSW_10_2_X_Patatrack Sep 30, 2018

fwyzard added the fixed label Sep 30, 2018

fwyzard self-assigned this Sep 30, 2018

fwyzard mentioned this pull request Sep 30, 2018

Disable vertex reconstruction on GPU #177

Merged

fwyzard added this to the CMSSW_10_2_6_Patatrack milestone Oct 31, 2018

fwyzard deleted the Work_around_atomicAdd_problem_on_Volta branch January 26, 2019 07:54

fwyzard added a commit that referenced this pull request Oct 8, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

6837428

fwyzard mentioned this pull request Oct 8, 2020

Patatrack integration - Pixel local reconstruction (9/N) cms-sw/cmssw#31721

Merged

fwyzard added a commit that referenced this pull request Oct 19, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

c24e243

fwyzard added a commit that referenced this pull request Oct 20, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

634ec59

fwyzard added a commit that referenced this pull request Oct 23, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

0b24f18

fwyzard added a commit that referenced this pull request Nov 6, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

22c6d45

fwyzard added a commit that referenced this pull request Nov 16, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

be8f851

fwyzard pushed a commit that referenced this pull request Dec 25, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

3535847

fwyzard added a commit that referenced this pull request Dec 29, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

b5a80db

fwyzard added a commit that referenced this pull request Dec 29, 2020

Work around atomicAdd synchronisation problem on Volta (#180)

e843016

fwyzard mentioned this pull request Jan 13, 2021

Patatrack integration - Pixel vertex reconstruction (11/N) cms-sw/cmssw#31723

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work around atomicAdd problem on Volta #180

Work around atomicAdd problem on Volta #180

fwyzard commented Sep 30, 2018

fwyzard commented Sep 30, 2018 •

edited

Loading

fwyzard commented Sep 30, 2018

fwyzard commented Sep 30, 2018

Work around atomicAdd problem on Volta #180

Work around atomicAdd problem on Volta #180

Conversation

fwyzard commented Sep 30, 2018

fwyzard commented Sep 30, 2018 • edited Loading

Validation summary

makeTrackValidationPlots.py plots

/RelValTTbar_13/CMSSW_10_2_2-PU25ns_102X_upgrade2018_realistic_v11-v2/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_2_2-102X_upgrade2018_realistic_v11-v1/GEN-SIM-DIGI-RAW

DQM GUI plots

/RelValTTbar_13/CMSSW_10_2_2-PU25ns_102X_upgrade2018_realistic_v11-v2/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_2_2-102X_upgrade2018_realistic_v11-v1/GEN-SIM-DIGI-RAW

logs and nvprof/nvvp profiles

/RelValTTbar_13/CMSSW_10_2_2-PU25ns_102X_upgrade2018_realistic_v11-v2/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_2_2-102X_upgrade2018_realistic_v11-v1/GEN-SIM-DIGI-RAW

Logs

fwyzard commented Sep 30, 2018

fwyzard commented Sep 30, 2018

fwyzard commented Sep 30, 2018 •

edited

Loading

`makeTrackValidationPlots.py` plots

logs and `nvprof`/`nvvp` profiles