Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling SegmentGenerationAndPushTask to push segment to realtime table #12084

Merged
merged 7 commits into from
Dec 4, 2023

Conversation

swaminathanmanish
Copy link
Contributor

@swaminathanmanish swaminathanmanish commented Dec 1, 2023

Description:

Following are the changes . Segment upload to realtime was already in this PR

  1. Updated segment push utilities to pass in tableType parameter in order to push segments to realtime tables. The tableType parameter has to be passed, without which default of OFFLINE gets used by the controller API that pushes segments to tables. makeTableParam is the method called by push utilities
    pushSegments
    sendSegmentUris
    sendSegmentUriAndMetadata
  2. Minor changes in SegmentGenerationAndPushTask to allow segments to be pushed to realtime table.

Testing:

Integration tests for SegmentGenerationAndPushTask that validates ingestion into realtime table.

@swaminathanmanish
Copy link
Contributor Author

@snleee - Please review.

@codecov-commenter
Copy link

codecov-commenter commented Dec 1, 2023

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (d7c76b9) 61.62% compared to head (069e1bd) 61.62%.
Report is 1 commits behind head on master.

Files Patch % Lines
...andpush/SegmentGenerationAndPushTaskGenerator.java 0.00% 11 Missing ⚠️
...e/pinot/common/utils/FileUploadDownloadClient.java 0.00% 6 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #12084      +/-   ##
============================================
- Coverage     61.62%   61.62%   -0.01%     
+ Complexity     1152     1147       -5     
============================================
  Files          2389     2389              
  Lines        129824   129823       -1     
  Branches      20083    20082       -1     
============================================
- Hits          80009    80008       -1     
+ Misses        43989    43987       -2     
- Partials       5826     5828       +2     
Flag Coverage Δ
custom-integration1 <0.01% <0.00%> (ø)
integration <0.01% <0.00%> (ø)
integration1 <0.01% <0.00%> (ø)
integration2 0.00% <0.00%> (ø)
java-11 61.59% <0.00%> (-0.02%) ⬇️
java-21 61.49% <0.00%> (-0.01%) ⬇️
skip-bytebuffers-false 61.61% <0.00%> (-0.01%) ⬇️
skip-bytebuffers-true 61.46% <0.00%> (-0.01%) ⬇️
temurin 61.62% <0.00%> (-0.01%) ⬇️
unittests 61.62% <0.00%> (-0.01%) ⬇️
unittests1 46.88% <0.00%> (-0.03%) ⬇️
unittests2 27.62% <0.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@swaminathanmanish swaminathanmanish changed the title Enabling push to RealTime tables Segment push from Minion to Realtime table Dec 1, 2023
@swaminathanmanish swaminathanmanish changed the title Segment push from Minion to Realtime table Enabling SegmentGenerationAndPushTask to push segment to realtime table Dec 1, 2023
Copy link
Contributor

@snleee snleee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@snleee snleee merged commit 69fbeef into apache:master Dec 4, 2023
19 checks passed
@snleee snleee added enhancement release-notes Referenced by PRs that need attention when compiling the next release notes labels Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement ingestion release-notes Referenced by PRs that need attention when compiling the next release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants