REMINIAOD workflows crash

[silviodonato]

The following workflows are crashing since CMSSW_11_1_X_2020-05-13-1100
https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_11_1/2020-05-13-1100?selectedArchs=slc7_amd64_gcc820&selectedFlavors=X&selectedStatus=failed

136.76111 RunJetHT2016E_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM
136.88811 RunJetHT2018D_reminiaodUL+REMINIAOD_data2018UL+HARVEST2018_REMINIAOD_data2018UL
1325.516 TTbar_13_reminiaod2016UL_preVFP_INPUT+REMINIAOD_mc2016UL_preVFP+HARVESTDR2_REMINIAOD_mc2016UL_preVFP
1325.5161 TTbar_13_reminiaod2016UL_postVFP_INPUT+REMINIAOD_mc2016UL_postVFP+HARVESTDR2_REMINIAOD_mc2016UL_postVFP 

with the error

----- Begin Fatal Exception 13-May-2020 14:12:45 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 18 event: 1707 stream: 0
   [1] Prefetching for module PFCandidateDQMAnalyzer/'ElectronValidation1'
   [2] Prefetching for module PATElectronSlimmer/'slimmedElectrons'
   [3] Prefetching for module PATElectronSelector/'selectedPatElectrons'
   [4] Prefetching for module PATElectronProducer/'patElectrons'
   [5] Prefetching for module CITKPFIsolationSumProducerForPUPPI/'egmElectronPUPPIIsolation'
   [6] Calling method for module PATPackedCandidateProducer/'packedPFCandidates'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::ValueMap<float>
Looking for module label: puppi
Looking for productInstanceName: 

----- End Fatal Exception -------------------------------------------------
%MSG-w MemoryCheck:  PATPackedCandidateProducer:packedPFCandidates  13-May-2020 14:12:45 CEST Run: 1 Event:

The workflows were added by #29756

Workflows 1325.517 and 1325.518 are crashing as well with a DAS error, but I think this was expected (see #29756 (comment))

[silviodonato]

assign reconstruction

[cmsbuild]

New categories assigned: reconstruction

@slava77,@perrotta you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @silviodonato Silvio Donato.

@Dr15Jones, @silviodonato, @dpiparo, @smuzaffar, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[slava77]

@silviodonato
This is a case of concurrent/conflicting change entering the IB after the last test of #29756; otherwise we'd catch the issue sooner, before merging.

I'll try to get a fix in the next 24 hours.
Please let me know if it's OK to possibly have the wfs broken until then and no revert is needed.
Thank you.

[silviodonato]

@silviodonato
This is a case of concurrent/conflicting change entering the IB after the last test of #29756; otherwise we'd catch the issue sooner, before merging.

I'll try to get a fix in the next 24 hours.
Please let me know if it's OK to possibly have the wfs broken until then and no revert is needed.
Thank you.

Yes, of course, getting a fix in the next 24 hours is perfectly fine, thanks.

[slava77]

Yes, of course, getting a fix in the next 24 hours is perfectly fine, thanks.

well, apparently I didn't think of the broken wf in the short matrix.
@smuzaffar are we masking wfs broken in the IB reference already or do they block the tests?

[slava77]

I'm testing something now, this should be in time for the 2300 IB

[smuzaffar]

@slava77 , no we are not masking broken wfs of IB during PR tests.
Do do mask wfs for comparisons though but this information is also reported in the comparison result comment.

[slava77]

fixed in #29826