Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update conftest for backwards compat and new api for to parquet call #1784

Merged
merged 2 commits into from
Mar 17, 2023

Conversation

jperez999
Copy link
Contributor

For cudf version 22.10.00+ the to_parquet api has changed. It no longer support the chunk_size call. It was replaced with row_group_size_rows, which can take minimum of 5000 rows. We have updated conftest call to reflect this change while maintaining backwards compatibility. We need this for all future versions.

Copy link
Contributor

@karlhigley karlhigley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks fine so I'm optimistically approving, but if @rjzamora has concerns we can open another PR to address them

@karlhigley karlhigley merged commit f726194 into NVIDIA-Merlin:main Mar 17, 2023
@rjzamora
Copy link
Collaborator

For cudf version 22.10.00+ the to_parquet api has changed. It no longer support the chunk_size call. It was replaced with row_group_size_rows

Hmm - I may be mistaken, but I don't think chunksize was ever a supported argument to cudf.to_parquet. I suspect that older cudf versions were just silently ignoring chunk_size=1000.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants