Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-223] Add ST_Split #742

Merged
merged 17 commits into from
Jan 6, 2023
Merged

[SEDONA-223] Add ST_Split #742

merged 17 commits into from
Jan 6, 2023

Conversation

douglasdennis
Copy link
Contributor

Did you read the Contributor Guide?

  • Yes

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

  1. Add ST_Split function to sedona-common, sedona-sql, sedona-flink, and the dataframe/python API.
  2. Created FunctionsTest.java to house tests for functions being added to sedona-common.

How was this patch tested?

  1. Comprehensive unit tests in sedona-common
  2. Integration unit tests in sedona-sql, sedona-flink, and python.

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation in docs/api/sql/Function.md.

@douglasdennis
Copy link
Contributor Author

Ugh. I noticed that my VS Code seems to have gone wild on deleting leading white spaces. I hope that is ok. If the edits seem too messy though then I can try to revert them.

return (MultiLineString)ensureMultiGeometryOfDimensionN(inputLines, 1);
} else if (intersectionWithBlade.getDimension() != 0) {
// FIXME need way to access the logger
System.err.println("Colinear sections detected between source and blade geometry. Returned null.");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure of a good way to access a log in sedona-common so I defaulted to standard error out. Is a logging mechanism something that needs to be added? @jiayuasu mentioned Log.error in the Jira ticket but I couldn't find the Log class/object. I'm happy to change this prior to merge if someone has a better way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was looking into standardizing the logging dependencies a little bit as a follow on to my last PR, can submit one for that soon. Right now log4j is available when spark is included, but not in common or flink

Copy link
Member

@jiayuasu jiayuasu Jan 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we don't need to include log4j in the common module. Instead, functions in the common module only add throw exception in their func signatures and sedona-flink / sedona-spark catch them and produce log.error using log4j?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a new type: SedonaFunctionException. With a constructor like: SedonaFunctionException(String message, Object result). Where message gets written to the appropriate log and result is the result of the function? I feel like there may sometimes be situations where a log entry needs to be made but a non-null result is still viable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made #743 if we want to just be able to log in the common module

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And if you're going to add logging it might be a good chance to start using slf4j instead of log4j directly

@jiayuasu
Copy link
Member

jiayuasu commented Jan 5, 2023

@douglasdennis PR #743 has been merged. If you pull from the upstream, you can actually now use log.error to handle the issue. Adam might need to make some changes later on but the dependencies should be the same.

@douglasdennis
Copy link
Contributor Author

@jiayuasu Added it in. Let me know if I was supposed to do it some other way. The output I see looks like this:
23/01/04 23:52:03 ERROR GeometrySplitter: Colinear sections detected between source and blade geometry. Returned null.

@jiayuasu jiayuasu removed this from the sedona-.1.3.2 milestone Jan 5, 2023
@jiayuasu jiayuasu added this to the sedona-1.4.0 milestone Jan 5, 2023
// blade and inputLines are disjoint so just return the input as a multilinestring
return (MultiLineString)ensureMultiGeometryOfDimensionN(inputLines, 1);
} else if (intersectionWithBlade.getDimension() != 0) {
Logger.getLogger(GeometrySplitter.class).error("Colinear sections detected between source and blade geometry. Returned null.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 things:
Can you try changing the common dependency from

        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-1.2-api</artifactId>
        </dependency>

to

            <dependency>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </dependency>

and try using the slf4j logger API instead? I think doing that in the common module would be a good place to try to stop using the log4j 1.x API directly.

Create the Logger as a static variable at the top of the class, I think that's the recommended way.

And kinda just my opinion, but if it doesn't break/stop the execution, I feel like it should be.a warning message and not an error message

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Kimahriman Double check what I just pushed, please :)

I followed this which seemed to recommend using this LoggerFactory class as well. I don't mind iterating on this till you're good with it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep that's exactly what I was picturing!

@jiayuasu
Copy link
Member

jiayuasu commented Jan 6, 2023

Awesome. Will merge.

@jiayuasu jiayuasu merged commit 6c6e8ce into apache:master Jan 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants