Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encode UUID as FixedLenByteArray in parquet_derive #5773

Merged
merged 4 commits into from
May 18, 2024

Conversation

conradludgate
Copy link
Contributor

Which issue does this PR close?

Closes #5254.

Rationale for this change

UUIDs are specified to be FixedByteArray(16), not ByteArray.

What changes are included in this PR?

Adds support for defining type builders with a length field. Fixes mapping for uuid to encoded type.

Are there any user-facing changes?

This is likely a breaking change as it is no longer compatible with prior parquet encodings generated with this lib.

@tustvold tustvold added the api-change Changes to the arrow API label May 16, 2024
@tustvold tustvold changed the title fix uuid derive Encode UUID as FixedByteArray in parquet_derive May 16, 2024
@tustvold tustvold changed the title Encode UUID as FixedByteArray in parquet_derive Encode UUID as FixedLenByteArray in parquet_derive May 16, 2024
@tustvold tustvold merged commit 30762e8 into apache:master May 18, 2024
12 checks passed
@tustvold
Copy link
Contributor

Thank you

conradludgate added a commit to neondatabase/neon that referenced this pull request May 19, 2024
## Summary of changes

Updates the parquet lib. one change left that we need is in an open PR
against upstream, hopefully we can remove the git dependency by 52.0.0
apache/arrow-rs#5773

I'm not sure why the parquet files got a little bit bigger. I tested
them and they still open fine. 🤷

side effect of the update, chrono updated and added yet another
deprecation warning (hence why the safekeepers change)
a-masterov pushed a commit to neondatabase/neon that referenced this pull request May 20, 2024
## Summary of changes

Updates the parquet lib. one change left that we need is in an open PR
against upstream, hopefully we can remove the git dependency by 52.0.0
apache/arrow-rs#5773

I'm not sure why the parquet files got a little bit bigger. I tested
them and they still open fine. 🤷

side effect of the update, chrono updated and added yet another
deprecation warning (hence why the safekeepers change)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api-change Changes to the arrow API
Projects
None yet
Development

Successfully merging this pull request may close these issues.

parquet_derive: UUID should not be a ByteArray
2 participants