Allow fillna to validate for CategoricalColumn.fillna
#15683
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
galipremsagar
added
improvement
mroeschke
approved these changes
May 7, 2024
galipremsagar
added
the
5 - Ready to Merge
|
/merge |
wence-
reviewed
May 8, 2024
Comment on lines
+1079
to
+1080
| # Validation of `fill_value` will have to be performed | ||
| # before returning self. |
There was a problem hiding this comment.
nit: this comment suggests that validation of the value must still be performed. But I think it has been in the code above.
So perhaps
Even if there are no nulls, we must validate that the provided value is valid (done above) to match pandas
WDYT?
| ) | ||
| or method is not None | ||
| or ( | ||
| isinstance(col, cudf.core.column.CategoricalColumn) |
There was a problem hiding this comment.
nit: This special-cases categorical columns for fillna both to not care about whether the column has nulls and whether the column name is in the provided dict-like (?) of values.
It so happens that the _is_null_host_scalar check will be true in the latter case, but it's somewhat confusing.
Do you want to just provide a carve-out for categorical columns in the first condition as:
col_name in value
and (col.has_nulls(include_nan=True) or isinstance(col, CategoricalColumn))
and not libcudf.scalar._is_null_host_scalar(replace_val)
?
rapids-bot bot
pushed a commit
to rapidsai/cugraph
that referenced
this pull request
May 10, 2024
…fillna()` based on latest cuDF changes (#4408) This handles a [recent cuDF change](rapidsai/cudf#15683) by applying non-dict and non-Series values for a `fillna()` call on `PropertyGraph` instances only to the user-defined columns, with the assumption that savvy users that intend to update the "internal" columns, or users that are aware of their own categorical dtype columns, will use a dict or Series value to properly apply dtypes as needed. This also updates code in `cugraph-service` that serializes dataframes to numpy bytes to properly convert NA values when categoricals are present. Notes: * This is only applied to the SG `PropertyGraph` class. The MG class needs further review as to how to best apply the same policy (and because there are other MG failing tests that need addressed). Since this is blocking CI for the SG case only, this PR is being submitted now and MG will be addressed later, which should be okay since `PropertyGraph` is experimental. * This could be considered a breaking change if `PropertyGraph` was not experimental. Authors: - Rick Ratzel (https://github.com/rlratzel) Approvers: - Alex Barghi (https://github.com/alexbarghi-nv) URL: #4408
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes: #15666
This PR validates values passed to
fillnaeven if there are no null values in a categorical column.Forks from #14534
Checklist