BeamSpotOnline in DQM beamspot clients

[francescobrivio]

PR description:

This PR introduces changes in the DQM code related to BeamSpot.

• The BeamSpot fit results are now also stored in a BeamSpotOnlineObjects (introduced in New BeamSpot Object and Records #29662 ) and saved in a payload using the service cond::service::OnlineDBOutputService.
• The DQM/BeamMonitor/python/*py configuration files are updated to include the default value of the BeamSpot Online record name for the upload to CondDB.
• Both beamspot python clients DQM/Integration/python/clients/beam*_dqm_sourceclient-live_cfg.py are updated to handle the upload of the payload to the database always using OnlineDBOutputService
• No changes in the actual fitting of the beamspot are introduced.
• Nothing is removed/deleted from the BeamMonitor.cc plugin, only the changes described above are added

There are two still open questions for which I might need help from DQM and DB experts:

1. For DQM experts:
   Is it ok to have options = VarParsing.VarParsing() under DQM/Integration/python/clients? I only see usage of this in DQM/Integration/python/config, so I was wondering if it is correct or it might cause problems. The options are needed to properly setup the OnlineDBOutputService.

2. For DB experts:
   I'm not sure how to properly configure these lines in OnlineDBOutputService:

• authenticationPath
• lastLumiFile
• connect and preLoadConnectionString

Backporting:

Once this PR is completed and merged, it will need a backport to CMSSW_11_1

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30448/16574

• This PR adds an extra 40KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File DQM/BeamMonitor/plugins/BeamMonitor.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936

[cmsbuild]

A new Pull Request was created by @francescobrivio for master.

It involves the following packages:

DQM/BeamMonitor
DQM/Integration

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@threus, @batinkov, @battibass this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

[francescobrivio]

Let me tag @ggovi as well, since I worked with him on the preparation of this PR and he might be interested (and he may be able to answer some of the questions).

[jfernan2]

@francescobrivio May ask you to add yourself along with your git-username in the comments to the corresponding DQM/Validation e-group in:
https://twiki.cern.ch/twiki/bin/viewauth/CMS/DQMContacts
for future reference? Thanks

[ggovi]

@francescobrivio
authenticationPath,lastLumiFile,connect and preLoadConnectionString params will be set according to the deployment needs. For this PR, I would suggest to leave authenticationPath and lastLumiFile empty ( string '' ) .

[francescobrivio]

preLoadConnectionString

Ok I will put an empty string in authenticationPath and lastLumiFile.
What about connect and preLoadConnectionString? Are they ok like they are now?

[francescobrivio]

@francescobrivio May ask you to add yourself along with your git-username in the comments to the corresponding DQM/Validation e-group in:
https://twiki.cern.ch/twiki/bin/viewauth/CMS/DQMContacts
for future reference? Thanks

I subscribed to Tracker-Online DQM Developer(s). If I understand correctly the twiki is auto-generated and should be updated once my subscription is approved.

[mmusich]

I subscribed to Tracker-Online DQM Developer(s). If I understand correctly the twiki is auto-generated and should be updated once my subscription is approved.

BeamSpot is more a responsibility of Tracking than Tracker, but do to lack of an "online tracking" set of developers, I guess that's fine. This somehow highlights the fact that the DQM developer table does not consider many grey area cases (such as this one, where a DQM package is just a place where really an AlCa/DB operation happens).

[cmsbuild]

The code-checks are being triggered in jenkins.

[jfernan2]

@francescobrivio May ask you to add yourself along with your git-username in the comments to the corresponding DQM/Validation e-group in:
https://twiki.cern.ch/twiki/bin/viewauth/CMS/DQMContacts
for future reference? Thanks

I subscribed to Tracker-Online DQM Developer(s). If I understand correctly the twiki is auto-generated and should be updated once my subscription is approved.

Thanks @francescobrivio Your understanding is correct

[jfernan2]

About "Is it ok to have options = VarParsing.VarParsing() under DQM/Integration/python/clients? "
It is fine if it works, but we don't know if ORP has some objection, however those clients are run in Online DQM so there should be no problem

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30448/16582

• This PR adds an extra 40KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File DQM/BeamMonitor/plugins/BeamMonitor.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936

[cmsbuild]

Pull request #30448 was updated. @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please check and sign again.

[francescobrivio]

After discussion with @ggovi and @gennai I added an option to handle the upload of the BeamSpot conditions to the DB:

• By default the client only creates the (local) .db file
  • the BS fit result is anyway also stored in a txt file so the run2-like workflow is preserved
• The upload to the DB is performed only if the option noDB=False is passed from command line

@jfernan2 this should now also work in the P5 playback, could you test it?

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30448/16786

• This PR adds an extra 28KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File DQM/BeamMonitor/plugins/BeamMonitor.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936

[cmsbuild]

Pull request #30448 was updated. @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please check and sign again.

[jfernan2]

@francescobrivio we tested it in Online and it works fine

[jfernan2]

please test

[cmsbuild]

The tests are being triggered in jenkins.

• CMSSW_11_2_X_2020-07-06-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7723/console Started: 2020/07/06 23:41

[cmsbuild]

+1
Tested at: 9d6c7b3
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17b66c/7723/summary.html
CMSSW: CMSSW_11_2_X_2020-07-06-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17b66c/7723/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 37
• DQMHistoTests: Total histograms compared: 2787364
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2787313
• DQMHistoTests: Total skipped: 50
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
• Checked 154 log files, 17 edm output root files, 37 DQM output files

[jfernan2]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1