Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-457] Don't write GeometryUDT into org.apache.spark.sql.parquet.row.metadata when writing GeoParquet files #1164

Merged
merged 2 commits into from
Dec 29, 2023

Conversation

Kontinuation
Copy link
Member

@Kontinuation Kontinuation commented Dec 28, 2023

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

Spark SQL primarily uses org.apache.spark.sql.parquet.row.metadata to infer the schema of parquet files. It will fall back to using the native parquet schema only when org.apache.spark.sql.parquet.row.metadata is absent. Writing the schema of dataframes with GeometryUDT columns into org.apache.spark.sql.parquet.row.metadata may cause compatibility problems. Please refer to the JIRA ticket for more details.

This patch replaces the GeometryUDT written into the metadata with binary type, since this is the physical data type for representing geometry values.

How was this patch tested?

Add assertions to verify that the metadata written by Spark SQL does not contain GeometryUDT.

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the docs.

@Kontinuation Kontinuation marked this pull request as ready for review December 28, 2023 06:54
@jiayuasu
Copy link
Member

jiayuasu commented Dec 29, 2023

This PR will fix OvertureMaps/data#89

@jiayuasu jiayuasu merged commit 7bb0ece into apache:master Dec 29, 2023
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants