Fix the bug of using push time to identify new created segment

[Jackie-Jiang]

Currently we use the push time as the creation time to identify whether a segment is newly created in the cluster. It doesn't apply to real-time segments, in which case we should use creation time instead. Note that for uploaded segment, we should still use push time because creation time is the time when the segment file is created instead of when the segment is uploaded.
Also added more logs in SegmentPartitionMetadataManager to make it easier to debug

[snleee]

LGTM.

• Mostly changing push -> creation. I see that only logic change is to honor creation time when the push time is not available.

Before this change, do we directly mark the newly added realtime segments as old segment and this change will now honer created time so it first becomes a new segment and then old segment?

[Jackie-Jiang]

Before this change, do we directly mark the newly added realtime segments as old segment and this change will now honer created time so it first becomes a new segment and then old segment?

Yes, all newly added consuming segment will be identified as old segment, and void the handling of new segments

[codecov-commenter]

Codecov Report

Merging #11599 (45eb40d) into master (8abe86b) will decrease coverage by 0.14%.
Report is 2 commits behind head on master.
The diff coverage is 76.47%.

@@             Coverage Diff              @@
##             master   #11599      +/-   ##
============================================
- Coverage     63.05%   62.92%   -0.14%     
  Complexity     1106     1106              
============================================
  Files          2326     2326              
  Lines        124974   124981       +7     
  Branches      19078    19080       +2     
============================================
- Hits          78803    78644     -159     
- Misses        40569    40723     +154     
- Partials       5602     5614      +12     

Flag	Coverage Δ
integration	<0.01% <0.00%> (ø)
integration1	<0.01% <0.00%> (ø)
integration2	0.00% <0.00%> (ø)
java-11	14.49% <70.58%> (-48.53%)	⬇️
java-17	62.89% <76.47%> (-0.02%)	⬇️
java-20	62.91% <76.47%> (+0.01%)	⬆️
temurin	62.92% <76.47%> (-0.14%)	⬇️
unittests	62.91% <76.47%> (-0.14%)	⬇️
unittests1	67.27% <100.00%> (-0.19%)	⬇️
unittests2	14.50% <70.58%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...not/common/metadata/segment/SegmentZKMetadata.java	57.69% <ø> (+2.56%)	⬆️
.../org/apache/pinot/query/routing/WorkerManager.java	69.92% <ø> (+0.50%)	⬆️
...mentpartition/SegmentPartitionMetadataManager.java	70.94% <42.85%> (-2.67%)	⬇️
...routing/instanceselector/BaseInstanceSelector.java	93.58% <100.00%> (+0.03%)	⬆️
...ker/routing/instanceselector/InstanceSelector.java	100.00% <100.00%> (ø)
...oker/routing/instanceselector/NewSegmentState.java	100.00% <100.00%> (ø)
...ceselector/StrictReplicaGroupInstanceSelector.java	98.41% <100.00%> (ø)
...va/org/apache/pinot/common/utils/SegmentUtils.java	19.04% <100.00%> (+19.04%)	⬆️

... and 20 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more