[SEDONA-314] Support optimized join on ST_HausdorffDistance #878
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit c7f6236.
This reverts commit 85ae113.
Removed note/tip block elements that caused numbering to be reset
…entation_update_Nilesh
Changed generic Exception to IllegalArgumentException in ST_NumPoints implementation and its corresponding test
# Conflicts: # common/src/main/java/org/apache/sedona/common/Functions.java # common/src/test/java/org/apache/sedona/common/FunctionsTest.java # flink/src/main/java/org/apache/sedona/flink/Catalog.java # flink/src/main/java/org/apache/sedona/flink/expressions/Functions.java # flink/src/test/java/org/apache/sedona/flink/FunctionTest.java # python/sedona/sql/st_functions.py # python/tests/sql/test_function.py # sql/common/src/main/scala/org/apache/sedona/sql/UDF/Catalog.scala # sql/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/Functions.scala # sql/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/st_functions.scala # sql/common/src/test/scala/org/apache/sedona/sql/dataFrameAPITestScala.scala # sql/common/src/test/scala/org/apache/sedona/sql/functionTestScala.scala
Refactored function name Made java tests more comprehensive by checking both nDims and WKT of returned geometry Added more test cases in scala test cases Updated documentation with empty geometry case and more examples
…ut. Added a note explaining the same
Update java/scala test execution
…GN5117/sedona into develop_Nilesh_1.4.1_Translate # Conflicts: # common/src/main/java/org/apache/sedona/common/Functions.java # common/src/main/java/org/apache/sedona/common/utils/GeomUtils.java # common/src/test/java/org/apache/sedona/common/FunctionsTest.java # flink/src/main/java/org/apache/sedona/flink/Catalog.java # flink/src/main/java/org/apache/sedona/flink/expressions/Functions.java # flink/src/test/java/org/apache/sedona/flink/FunctionTest.java # python/sedona/sql/st_functions.py # python/tests/sql/test_function.py # sql/common/src/main/scala/org/apache/sedona/sql/UDF/Catalog.scala # sql/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/Functions.scala # sql/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/st_functions.scala # sql/common/src/test/scala/org/apache/sedona/sql/dataFrameAPITestScala.scala # sql/common/src/test/scala/org/apache/sedona/sql/functionTestScala.scala
Added geom collection test cases for force3D
This reverts commit 19016ae.
# Conflicts: # common/src/test/java/org/apache/sedona/common/FunctionsTest.java # flink/src/main/java/org/apache/sedona/flink/Catalog.java # flink/src/main/java/org/apache/sedona/flink/expressions/Functions.java # python/sedona/sql/st_functions.py # python/tests/sql/test_function.py # sql/common/src/main/scala/org/apache/sedona/sql/UDF/Catalog.scala # sql/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/Functions.scala # sql/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/st_functions.scala # sql/common/src/test/scala/org/apache/sedona/sql/dataFrameAPITestScala.scala # sql/common/src/test/scala/org/apache/sedona/sql/functionTestScala.scala
Revert accidental incorrect removals from Catalog while merging with master
# Conflicts: # common/src/main/java/org/apache/sedona/common/utils/GeomUtils.java # common/src/test/java/org/apache/sedona/common/FunctionsTest.java # flink/src/main/java/org/apache/sedona/flink/expressions/Functions.java # python/tests/sql/test_dataframe_api.py # sql/common/src/test/scala/org/apache/sedona/sql/dataFrameAPITestScala.scala # sql/common/src/test/scala/org/apache/sedona/sql/functionTestScala.scala
jiayuasu
reviewed
Jun 29, 2023
| val expectedDensityIntersects = bruteForceDistanceJoinHausdorff(sampleCount, distance, densityFrac, true) | ||
| val distanceDensityIntersectsDF = inputPoint.alias("pointDF").join(inputPolygon.alias("polygonDF"), expr(s"ST_HausdorffDistance(pointDF.pointshape, polygonDF.polygonshape, $densityFrac) <= $distance")) | ||
| assert(distanceDensityIntersectsDF.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1) | ||
| assert(distanceDensityIntersectsDF.count() == expectedDensityIntersects) |
There was a problem hiding this comment.
I wonder what the number of expected intersects is in these test cases? Can you report the numbers here?
There was a problem hiding this comment.
Sure, both intersects (<=) and non-intersects (<) return a count of 100 for this dataset. The polygon-point pairs are also exactly same since there is no pair with frechetDistance == 1
There was a problem hiding this comment.
Can you add more distanceCandidates? Most importantly, add some candidates that lead to results > 100.
There was a problem hiding this comment.
Have added candidates 1, 2, 5, 10. These yield result 100, 298, 688, 1258 respectively for a sample size of 100
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a JIRA ticket?
[SEDONA-XXX] my subject.What changes were proposed in this PR?
How was this patch tested?
Did this PR include necessary documentation updates?