Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DQM: Allow end lumi work in DQMOneEDAnalyzer #30326

Conversation

schneiml
Copy link
Contributor

@schneiml schneiml commented Jun 22, 2020

PR description:

This PR adds dqmEndLuminosityBlock to DQMOneEDAnalyzer.

Turns out we can set EndLuminosityBlockProducer without using WatchLuminosityBlocks?
(Edit: no, this also blocks concurrent lumis. Back to the drawing board then.)

That gives an option to enable dqmEndLuminosityBlock by default.

This is once again risky, since the semantics now are subtly different compared to 2018:
The same callbacks exist, but events of multiple lumis can arrive alternatingly.
This could lead to subtle bugs. The good thing is, we have eliminated almost
all usages of the old API, so we know there are not many places where such bugs
could happen.

globalEndLuminosityBlock in DQM modules is banned in favor of dqmEndLuminosityBlock now for consistency, and all existing users changed. The semantics might be slightly different (the EDM docs hint that some lumi products might not be availale in ennLuminosityBlockProduce, but in globalEndLuminostyBlock, let's see if this survives validation.

PR validation:

Compiles.

Further validation needed.

Turns out we can set EndLuminosityBlockProducer without using WatchLuminosityBlocks?

That gives an option to enable dqmEndLuminosityBlock by default.

This is once again risky, since the semantics now are subtly different compared to 2018:
The same callbacks exist, but events of multiple lumis can arrive alternatingly.
This could lead to subtle bugs. The good thing is, we have eliminated almost
all usages of the old API, so we know there are not many places where such bugs
could happen.

This is sort-of required to allow 'per-lumi ME with lumi data' with concurrent lumis.
@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30326/16320

Code check has found code style and quality issues which could be resolved by applying following patch(s)

The result does not compile.
@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@schneiml
Copy link
Contributor Author

Ok, the result of the code checks does not compile, and the version that compiles does not code-check. I feared sth. along those lines.

I'll see it there is a better solution, @Dr15Jones @makortel @smuzaffar let me know if you have any ideas.

@cmsbuild
Copy link
Contributor

-code-checks

ERROR: Build errors found during clang-tidy run.

DQMServices/Core/interface/DQMOneEDAnalyzer.h:84:86: error: only virtual member functions can be marked 'final' [clang-diagnostic-error]
  void globalEndLuminosityBlock(const edm::LuminosityBlock&, const edm::EventSetup&) final{};
                                                                                     ^~~~~
DQM/L1TMonitor/interface/L1TStage2CaloLayer1.h:80:36: note: in instantiation of template class 'DQMOneEDAnalyzer<edm::one::WatchLuminosityBlocks>' requested here
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:128: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

@smuzaffar
Copy link
Contributor

@schneiml , may be inlining is issue here?

@Dr15Jones
Copy link
Contributor

@schneiml the compiler is telling you about a real problem. You made DQMOneEDAnalyzer always be an EndLuminosityBlockProducer which is fine. However, the function

void globalEndLuminosityBlock(const edm::LuminosityBlock&, const edm::EventSetup&) final{};

is not part of the EndLuminosityBlockProducer interface, it is part of the edm::LuminosityBlockCache<> interface. The reason L1TStage2CaloLayer1 is failing is it does not use a edm::LuminosityBlockCache<>.

@schneiml
Copy link
Contributor Author

schneiml commented Jun 22, 2020

@Dr15Jones The error now is obvious. The problem is this:
I want to ban using globalEndLuminosityBlock in classes derived from DQMOneEDAnalyzer, so I declare a final globalEndLuminosityBlock there.

But the edm::one::LuminosityBlockCache that enables this feature in EDM is 1) optional, and 2) depends on a parameter from the derived class that DQMOneEDAnalyzer just forwards.

Now, the version from 49cce0f works fine: It declares a virtual final globalEndLuminosityBlock in all instances, with the wanted effect. But, in the cases where there is no LuminosityBlockCache, this class needs to be virtual final since it cannot be only final, but then in the cases where a LuminosityBlockCache is added, the compiler warns about the redundant virtual.

Edit to add: The options I see are 1) simply not banning globalEndLuminosityBlock (quite error prone, since code using MonitorElements there will fail for non-obvious reasons), 2) doing what i did and somehow suppress the error, or 3) only implement the globalEndLuminosityBlock in DQMOneEDAnalyzer if there is a lumi cache (exceeds my template meta-programming skills).

I removed the cache type where it was not used.
@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@schneiml
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 23, 2020

The tests are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

+1
Tested at: 3a6a90f
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14ffb3/7275/summary.html
CMSSW: CMSSW_11_2_X_2020-06-22-2300
SCRAM_ARCH: slc7_amd64_gcc820

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14ffb3/7275/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 36
  • DQMHistoTests: Total histograms compared: 2778811
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2778760
  • DQMHistoTests: Total skipped: 50
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
  • Checked 152 log files, 16 edm output root files, 36 DQM output files

@schneiml
Copy link
Contributor Author

hold

Turns out this does block concurrent lumis. Nevermind, then.

@cmsbuild
Copy link
Contributor

Pull request has been put on hold by @schneiml
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

@cmsbuild cmsbuild added the hold label Jun 23, 2020
@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30326/16361

@cmsbuild
Copy link
Contributor

Pull request #30326 was updated. @benkrikler, @kmaeshima, @schneiml, @andrius-k, @cmsbuild, @rekovic, @jfernan2, @fioriNTU can you please check and sign again.

@makortel
Copy link
Contributor

Turns out this does block concurrent lumis. Nevermind, then.

Correct (I was actually first surprised as well, but it actually makes sense). I clarified in the documentation that the Begin/EndLuminosityBlockProducer extension leads to system effectively processing events from one lumi at a time.

The only way to support concurrent lumis with any of WatchLuminosityBlocks/BeginLuminosityBlockProducer/EndLuminosityBlockProducer is to use LuminosityBlockCache.

@schneiml
Copy link
Contributor Author

@makortel I think I'll give it another try with the caches and global transitions. That will require yet another DQM*Analyzer base class, but should work.

@schneiml
Copy link
Contributor Author

Actually, I don't plan to try this with global transitions, it might work but would violate the "DQM behaves like products" principle. So, as long as nobody has a real need for this, I'll close it.

@schneiml schneiml closed this Jun 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants