New API added to referential integrity to allow for row level annotation #466
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- The provided primary data frame will have a column added to it to indicate a true/false value. - The value will be true if the values of the provided columns for that row exist in the reference dataframe. Otherwise, the value will be false. - Added various tests, including tests for nested columns. - Refactored the parameter validation from the regular row level check, so that the same validation can be done in the new function as well. - Updated scaladoc of the two APIs.
eycho-am
reviewed
Apr 13, 2023
| val result = ReferentialIntegrity.subsetCheckRowLevel(ds1, cols, ds2, cols, Some(outcomeCol)) | ||
| assert(result.isRight) | ||
| val outcomes = result.right.get.select(outcomeCol).collect().toSeq.map { r => r.get(0) } | ||
| assert(outcomes == Seq(true, true, false, true)) |
There was a problem hiding this comment.
I'm a little confused as to why it's true, true, false, true when the last row is the incorrect one
mentekid
reviewed
Apr 13, 2023
| val ds2 = rdd2.toDF("id", "state name", "state") | ||
|
|
||
| val cols = Seq("state name", "state") | ||
| val outcomeCol = "row_level_outcome" |
There was a problem hiding this comment.
Can you add an assertion for the columns returned? It should be exactly ("id", "state name", "state", "row_level_outcome")
| val result = ReferentialIntegrity.subsetCheckRowLevel(ds1, cols, ds2, cols, Some(outcomeCol)) | ||
| assert(result.isRight) | ||
| val outcomes = result.right.get.select(outcomeCol).collect().toSeq.map { r => r.get(0) } | ||
| assert(outcomes == Seq(true, true, false, true)) |
There was a problem hiding this comment.
maybe also assert that if I drop "row_level_outcome" I get a dataframe with exactly what's in rdd1?
|
Would it be possible to also add tests in the |
- The order of the outcomes needed to be updated, to reflect the order of the data that was setup. - Added more assertions based on feedback.
This is a standalone utility and we are not using it in the VerificationSuite. Once the verification suite supports multiple dataframes, we will add this check and integrate the row level results. |
rdsharma26
added a commit
that referenced
this pull request
Apr 27, 2023
…ion (#466) * New API added to referential integrity to allow for row level annotation - The provided primary data frame will have a column added to it to indicate a true/false value. - The value will be true if the values of the provided columns for that row exist in the reference dataframe. Otherwise, the value will be false. - Added various tests, including tests for nested columns. - Refactored the parameter validation from the regular row level check, so that the same validation can be done in the new function as well. - Updated scaladoc of the two APIs. * Updated tests - The order of the outcomes needed to be updated, to reflect the order of the data that was setup. - Added more assertions based on feedback.
rdsharma26
added a commit
that referenced
this pull request
Apr 16, 2024
…ion (#466) * New API added to referential integrity to allow for row level annotation - The provided primary data frame will have a column added to it to indicate a true/false value. - The value will be true if the values of the provided columns for that row exist in the reference dataframe. Otherwise, the value will be false. - Added various tests, including tests for nested columns. - Refactored the parameter validation from the regular row level check, so that the same validation can be done in the new function as well. - Updated scaladoc of the two APIs. * Updated tests - The order of the outcomes needed to be updated, to reflect the order of the data that was setup. - Added more assertions based on feedback.
rdsharma26
added a commit
that referenced
this pull request
Apr 16, 2024
…ion (#466) * New API added to referential integrity to allow for row level annotation - The provided primary data frame will have a column added to it to indicate a true/false value. - The value will be true if the values of the provided columns for that row exist in the reference dataframe. Otherwise, the value will be false. - Added various tests, including tests for nested columns. - Refactored the parameter validation from the regular row level check, so that the same validation can be done in the new function as well. - Updated scaladoc of the two APIs. * Updated tests - The order of the outcomes needed to be updated, to reflect the order of the data that was setup. - Added more assertions based on feedback.
rdsharma26
added a commit
that referenced
this pull request
Apr 16, 2024
…ion (#466) * New API added to referential integrity to allow for row level annotation - The provided primary data frame will have a column added to it to indicate a true/false value. - The value will be true if the values of the provided columns for that row exist in the reference dataframe. Otherwise, the value will be false. - Added various tests, including tests for nested columns. - Refactored the parameter validation from the regular row level check, so that the same validation can be done in the new function as well. - Updated scaladoc of the two APIs. * Updated tests - The order of the outcomes needed to be updated, to reflect the order of the data that was setup. - Added more assertions based on feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.