Fix access to uninitialised memory in `RawToDigi_kernel` #206

fwyzard · 2018-11-29T09:48:12Z

Reported by cuda-memcheck --tool initcheck:

CUDA-MEMCHECK
Host API memory access error at host access to 0x7fe311800000 of size 112660 bytes
    Uninitialized access at 0x7fe311811720 on access by cudaMemcopy source.
    Saved host backtrace up to driver entry point at error
    ...
    Host Frame:.../pluginRecoLocalTrackerSiPixelClusterizerPlugins.so (pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(SiPixelFedCablingMapGPU const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, unsigned int, unsigned int, bool, bool, bool, bool, bool, cuda::stream_t<false>&) + 0x1d87) [0x6e827]
    Host Frame:.../pluginRecoLocalTrackerSiPixelClusterizerPlugins.so (SiPixelRawToClusterHeterogeneous::acquireGPUCuda(edm::HeterogeneousEvent const&, edm::EventSetup const&, cuda::stream_t<false>&) + 0x768) [0x58618]
    ...

Reported by cuda-memcheck --tool initcheck: CUDA-MEMCHECK Host API memory access error at host access to 0x7fe311800000 of size 112660 bytes Uninitialized access at 0x7fe311811720 on access by cudaMemcopy source. Saved host backtrace up to driver entry point at error ... Host Frame:.../pluginRecoLocalTrackerSiPixelClusterizerPlugins.so (pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(SiPixelFedCablingMapGPU const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, unsigned int, unsigned int, bool, bool, bool, bool, bool, cuda::stream_t<false>&) + 0x1d87) [0x6e827] Host Frame:.../pluginRecoLocalTrackerSiPixelClusterizerPlugins.so (SiPixelRawToClusterHeterogeneous::acquireGPUCuda(edm::HeterogeneousEvent const&, edm::EventSetup const&, cuda::stream_t<false>&) + 0x768) [0x58618] ...

fwyzard · 2018-11-29T09:48:59Z

Validation summary

Reference release CMSSW_10_4_0_pre2 at c6061f4
Development branch CMSSW_10_4_X_Patatrack at b8334a2
Testing PRs:

Fix access to uninitialised memory in RawToDigi_kernel #206 at 53a5b1f

`makeTrackValidationPlots.py` plots

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.8
tracking validation plots and summary for workflow 10824.7
tracking validation plots and summary for workflow 10824.9

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.8
tracking validation plots and summary for workflow 10824.7
tracking validation plots and summary for workflow 10824.9

DQM GUI plots

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

reference DQM plots for reference release, workflow 10824.5
DQM plots for development release, workflow 10824.5
DQM plots for development release, workflow 10824.8
DQM plots for development release, workflow 10824.7
DQM plots for development release, workflow 10824.9
DQM plots for testing release, workflow 10824.5
DQM plots for testing release, workflow 10824.8
DQM plots for testing release, workflow 10824.7
DQM plots for testing release, workflow 10824.9
DQM comparison for reference workflow 10824.5
DQM comparison for workflow 10824.8
DQM comparison for workflow 10824.7
DQM comparison for workflow 10824.9

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

reference DQM plots for reference release, workflow 10824.5
DQM plots for development release, workflow 10824.5
DQM plots for development release, workflow 10824.8
DQM plots for development release, workflow 10824.7
DQM plots for development release, workflow 10824.9
DQM plots for testing release, workflow 10824.5
DQM plots for testing release, workflow 10824.8
DQM plots for testing release, workflow 10824.7
DQM plots for testing release, workflow 10824.9
DQM comparison for reference workflow 10824.5
DQM comparison for workflow 10824.8
DQM comparison for workflow 10824.7
DQM comparison for workflow 10824.9

logs and `nvprof`/`nvvp` profiles

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ❌ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) found 2423 errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 10824.7
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.9
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.7
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.9
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 10824.7
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
development release, workflow 10824.9
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.5
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.8
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.7
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
testing release, workflow 10824.9
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors

Logs

The full log is available at https://fwyzard.web.cern.ch/fwyzard/patatrack/pulls/03488ef0452a185595c78ca7d23ee413eb03c8bb/log .

fwyzard · 2018-11-29T10:33:25Z

No negative impact on runtime performance.
Before:

Running 4 times over 4200 events with 1 jobs, each with 8 threads, 8 streams and 1 GPUs
  1833.6 ±   2.6 ev/s (4000 events)
  1840.2 ±   2.6 ev/s (4000 events)
  1812.8 ±   2.1 ev/s (4000 events)
  1848.8 ±   2.7 ev/s (4000 events)

After:

  1851.5 ±   2.1 ev/s (4000 events)
  1850.1 ±   2.8 ev/s (4000 events)
  1874.4 ±   2.7 ev/s (4000 events)
  1873.6 ±   2.2 ev/s (4000 events)

fwyzard · 2018-11-29T10:51:59Z

Comparison of the summaries look good.
The initcheck errors are gone.

Reported by cuda-memcheck --tool initcheck: CUDA-MEMCHECK Host API memory access error at host access to 0x7fe311800000 of size 112660 bytes Uninitialized access at 0x7fe311811720 on access by cudaMemcopy source. Saved host backtrace up to driver entry point at error ... Host Frame:.../pluginRecoLocalTrackerSiPixelClusterizerPlugins.so (pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(SiPixelFedCablingMapGPU const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, unsigned int, unsigned int, bool, bool, bool, bool, bool, cuda::stream_t<false>&) + 0x1d87) [0x6e827] Host Frame:.../pluginRecoLocalTrackerSiPixelClusterizerPlugins.so (SiPixelRawToClusterHeterogeneous::acquireGPUCuda(edm::HeterogeneousEvent const&, edm::EventSetup const&, cuda::stream_t<false>&) + 0x768) [0x58618] ...

fwyzard merged commit da30349 into cms-patatrack:CMSSW_10_4_X_Patatrack Nov 29, 2018

fwyzard deleted the fix_memory_RawToDigi_kernel branch November 29, 2018 10:53

fwyzard added this to the CMSSW_10_4_0_pre3_Patatrack milestone Dec 7, 2018

fwyzard added fixed bug and removed bug labels Dec 7, 2018

fwyzard mentioned this pull request Oct 8, 2020

Patatrack integration - Pixel local reconstruction (9/N) cms-sw/cmssw#31721

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix access to uninitialised memory in `RawToDigi_kernel` #206

Fix access to uninitialised memory in `RawToDigi_kernel` #206

fwyzard commented Nov 29, 2018

fwyzard commented Nov 29, 2018 •

edited

Loading

fwyzard commented Nov 29, 2018

fwyzard commented Nov 29, 2018

Fix access to uninitialised memory in RawToDigi_kernel #206

Fix access to uninitialised memory in RawToDigi_kernel #206

Conversation

fwyzard commented Nov 29, 2018

fwyzard commented Nov 29, 2018 • edited Loading

Validation summary

makeTrackValidationPlots.py plots

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

DQM GUI plots

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

logs and nvprof/nvvp profiles

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

Logs

fwyzard commented Nov 29, 2018

fwyzard commented Nov 29, 2018

Fix access to uninitialised memory in `RawToDigi_kernel` #206

Fix access to uninitialised memory in `RawToDigi_kernel` #206

fwyzard commented Nov 29, 2018 •

edited

Loading

`makeTrackValidationPlots.py` plots

logs and `nvprof`/`nvvp` profiles