[SEDONA-133] Allow user-defined schemas in Adapter.toDf() #655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There is no way to distinguish between having no user data (in which case it is set to "null") and having one column of user data that may take on null values (in which case the string representation would also be "null"). But in the second case, we want to preserve it and not drop the null values.
brianrice2
commented
Aug 5, 2022
Comment on lines
+286
to
+315
| if (data == "null") { | ||
| return desiredType match { | ||
| case _: ByteType => null.asInstanceOf[Byte] | ||
| case _: ShortType => null.asInstanceOf[Short] | ||
| case _: IntegerType => null.asInstanceOf[Integer] | ||
| case _: LongType => null.asInstanceOf[Long] | ||
| case _: FloatType => null.asInstanceOf[Float] | ||
| case _: DoubleType => null.asInstanceOf[Double] | ||
| case _: DateType => null.asInstanceOf[Date] | ||
| case _: TimestampType => null.asInstanceOf[Timestamp] | ||
| case _: BooleanType => null.asInstanceOf[Boolean] | ||
| case _: StringType => null.asInstanceOf[String] | ||
| } | ||
| } | ||
|
|
||
| desiredType match { | ||
| case _: ByteType => data.toByte | ||
| case _: ShortType => data.toShort | ||
| case _: IntegerType => data.toInt | ||
| case _: LongType => data.toLong | ||
| case _: FloatType => data.toFloat | ||
| case _: DoubleType => data.toDouble | ||
| case _: DateType => Date.valueOf(data) | ||
| case _: TimestampType => Timestamp.valueOf(data) | ||
| case _: BooleanType => data.toBoolean | ||
| case _: StringType => data | ||
| case _: StructType => | ||
| val desiredStructSchema = desiredType.asInstanceOf[StructType] | ||
| new GenericRowWithSchema(parseStruct(data, desiredStructSchema), desiredStructSchema) | ||
| } |
There was a problem hiding this comment.
I couldn't find an elegant way to perform the null-safe conversion, so I have these long match statements
|
|
jiayuasu
approved these changes
Aug 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Yes, I have read Contributor Rules and Contributor Development Guide
Is this PR related to a JIRA ticket?
Yes, the URL of the assoicated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-XXX. The PR name follows the format
[SEDONA-XXX] my subject.Link to original ticket.
What changes were proposed in this PR?
This expands the Adapter API to allow for users to convert to DataFrames with a given schema (for both SpatialRDD and JavaPairRDD).
User data is still stored in String format, so these new methods parse/cast the strings to whichever new data type is requested. This is similar to Spark's UnivocityParser, which is used to parse CSV files, but unfortunately that functionality is not exposed publicly so I created a barebones version here. I didn't cover all the data types, but tried to cover the key ones. This page details the encoders that Spark uses and may be helpful to understand the appropriate data types for conversion. We could expand to cover more data types later, or I'm open to it now if you request it.
This also adds some helper private methods and refactors a few common operations into their own functions.
How was this patch tested?
Added unit tests to confirm that SpatialRDD/JavaPairRDD -> DataFrame conversion works as expected.
Note: It doesn't seem to be common practice to test private methods, so I didn't add unit tests for the private methods I introduced. Their behavior is tested implicitly by the public functions.
Did this PR include necessary documentation updates?
Yes, I am adding a new API. I am using the current SNAPSHOT version number in since
vX.Y.Zformat.Note: I don't see another appropriate place to change documentation. Please let me know if I missed this!
Questions
1.
Is the following behavior intentional? I don't have a strong geospatial background.
In the JavaPairRDD -> DataFrame test case (called "can convert JavaPairRDD to DataFrame with user-supplied schema"), you may notice that the left and right dataframes get switched. The SpatialJoinQuery has
pointRDDon the left andpolygonRDDon the right, but the final output hasleftGeometryof type POLYGON, followed by thepolygonRDDuser data fields, andrightGeometryof type POINT, followed by thepointRDDuser data (null).