assign segments for upsert table with respect to ideal state

[klsince]

This PR adds more checks when assigning new segments for upsert table, in order to process queries correctly. The extra checks make sure that at any time the idealState has all segments from the same table partition hosted on same server. Basically, it honors the assignment in the current idealState over the one as calculated by InstancePartition.

The segment is assigned according to InstancePartition, which can be ahead of IdealState if table rebalance fails to update idealState completely according to the new InstancePartition or is in middle of updating the idealState. If the new segment assignment is simply set in IdealState as done today, it's possible new segments from the same partition start to go to new server, causing queries to return wrong results.The new checks added here avoid such issue by checking if the assignment in the current idealState for the table partition is same as the one calculated from InstancePartition and favor the idealState if mismatch.

[codecov-commenter]

Codecov Report

Merging #11628 (c12ee9a) into master (9bcbef8) will decrease coverage by 0.06%.
Report is 10 commits behind head on master.
The diff coverage is 75.32%.

@@             Coverage Diff              @@
##             master   #11628      +/-   ##
============================================
- Coverage     63.10%   63.04%   -0.06%     
- Complexity     1120     1121       +1     
============================================
  Files          2335     2343       +8     
  Lines        125212   125753     +541     
  Branches      19209    19321     +112     
============================================
+ Hits          79018    79286     +268     
- Misses        40560    40818     +258     
- Partials       5634     5649      +15     

Flag	Coverage Δ
integration	<0.01% <0.00%> (-0.01%)	⬇️
integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration2	0.00% <0.00%> (ø)
java-11	63.01% <75.32%> (-0.08%)	⬇️
java-17	62.90% <75.32%> (-0.06%)	⬇️
java-20	62.89% <75.32%> (-0.07%)	⬇️
temurin	63.04% <75.32%> (-0.06%)	⬇️
unittests	63.04% <75.32%> (-0.06%)	⬇️
unittests1	67.20% <0.00%> (-0.12%)	⬇️
unittests2	14.44% <75.32%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...g/apache/pinot/common/metrics/ControllerMeter.java	0.00% <0.00%> (ø)
...nment/segment/StrictRealtimeSegmentAssignment.java	66.00% <66.00%> (ø)
...e/assignment/segment/SegmentAssignmentFactory.java	87.50% <80.00%> (-12.50%)	⬇️
...apache/pinot/controller/BaseControllerStarter.java	78.16% <100.00%> (ø)
...ntroller/helix/core/PinotHelixResourceManager.java	63.61% <100.00%> (+0.04%)	⬆️
...core/assignment/segment/BaseSegmentAssignment.java	97.77% <100.00%> (+0.05%)	⬆️
.../assignment/segment/RealtimeSegmentAssignment.java	92.47% <100.00%> (+0.08%)	⬆️
.../core/realtime/PinotLLCRealtimeSegmentManager.java	60.73% <100.00%> (+0.16%)	⬆️
...ntroller/helix/core/rebalance/TableRebalancer.java	68.98% <100.00%> (+0.13%)	⬆️

... and 29 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Jackie-Jiang]

LGTM