Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proto null fix #11723

Merged
merged 6 commits into from
Oct 3, 2023
Merged

Proto null fix #11723

merged 6 commits into from
Oct 3, 2023

Conversation

gortiz
Copy link
Contributor

@gortiz gortiz commented Oct 2, 2023

This PR fixes the way nulls are read in Protobuf and add some test that verify the correct behavior.

See #11712 to see the same tests failing in current master branch.

PS: See https://docs.google.com/document/d/1xfYVL8V600lPV7U6ZO4NpGIV3gOVp8cTlKn2Zj0bCOI

@gortiz
Copy link
Contributor Author

gortiz commented Oct 2, 2023

cc @swaminathanmanish

@codecov-commenter
Copy link

codecov-commenter commented Oct 2, 2023

Codecov Report

Merging #11723 (ffd2c6d) into master (54a1ca0) will decrease coverage by 48.61%.
Report is 3 commits behind head on master.
The diff coverage is 100.00%.

@@              Coverage Diff              @@
##             master   #11723       +/-   ##
=============================================
- Coverage     63.05%   14.44%   -48.61%     
+ Complexity     1118      201      -917     
=============================================
  Files          2342     2342               
  Lines        125787   125802       +15     
  Branches      19335    19336        +1     
=============================================
- Hits          79314    18176    -61138     
- Misses        40813   106093    +65280     
+ Partials       5660     1533     -4127     
Flag Coverage Δ
integration <0.01% <0.00%> (ø)
integration1 <0.01% <0.00%> (ø)
integration2 0.00% <0.00%> (ø)
java-11 14.44% <100.00%> (-48.57%) ⬇️
java-17 0.00% <0.00%> (-62.91%) ⬇️
java-20 14.43% <100.00%> (-48.50%) ⬇️
temurin 14.44% <100.00%> (-48.61%) ⬇️
unittests 14.44% <100.00%> (-48.61%) ⬇️
unittests1 ?
unittests2 14.44% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
.../inputformat/protobuf/ProtoBufRecordExtractor.java 84.78% <100.00%> (+5.43%) ⬆️

... and 1510 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Contributor

@swaminathanmanish swaminathanmanish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks ! Your doc that explains the behavior makes sense
https://docs.google.com/document/d/1xfYVL8V600lPV7U6ZO4NpGIV3gOVp8cTlKn2Zj0bCOI/edit

Copy link
Contributor

@walterddr walterddr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me. minor question. maybe we can copy paste the tl;dr from the google doc or link the google doc here

@@ -62,7 +62,7 @@ private Object getFieldValue(Descriptors.FieldDescriptor fieldDescriptor, Messag
// Note w.r.t proto3 - If a field is not declared with optional keyword, there's no way to distinguish
// if its explicitly set to a proto default or not been set at all i.e hasField() returns false
// and we would use null.
if (fieldDescriptor.isRepeated() || !fieldDescriptor.isOptional() || message.hasField(fieldDescriptor)) {
if (fieldDescriptor.isRepeated() || !fieldDescriptor.hasPresence() || message.hasField(fieldDescriptor)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should we change the note above? ("not declared with optional keyword" --> "is set as hasPresence flag")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right. I'm opening a new PR to change that

@walterddr walterddr merged commit af310b3 into apache:master Oct 3, 2023
21 checks passed
@gortiz gortiz deleted the proto-null-fix branch October 4, 2023 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants