Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-244][SEDONA-246] R align spark_read/write functions #779

Merged
merged 23 commits into from
Feb 28, 2023

Conversation

gregleleu
Copy link
Contributor

@gregleleu gregleleu commented Feb 28, 2023

https://issues.apache.org/jira/projects/SEDONA/issues/SEDONA-244
https://issues.apache.org/jira/projects/SEDONA/issues/SEDONA-246

What changes were proposed in this PR?

In the R package:

  • Split sedona_read|write_* functions returning RDDs and spark_read|write_* functions returning dataframes (R tbl)
  • Add geoparquet and geotiff
  • R package documentation updates
  • Soft deprecated typed RDD functions

How was this patch tested?

Translated Scala tests for geoparquet and geotiff

Did this PR include necessary documentation updates?

R documentation only

gregleleu and others added 23 commits February 17, 2023 11:31
DESCRIPTION cleanup: bump to dev version .9000, move dplyr to Suggests (never imported, a dependency of sparklyr anyway)
Fixes to documentation
More geoparquet tests (copied from scala tests)
Create functions
Tests to read/write (based on java tests)
Some read and RS functions tests
Co-authored-by: zzs-wherobots <zzs-wherobots@ZongsideMac-Studio.local>
Co-authored-by: Zongsi Zhang <zongsi@Zongsis-MacBook-Pro.local>
Adding test on multi-geom geoparquet
Using lifecycle (switch to markdown documentation)
Soft deprecation warning for typed RDD functions
Fix arguments propagation for spark_write_geoparquet
Change way to count files in geotiff test
Changed test to accomodate bug in Spark 3.2.1 (can't reuse registered view names)
R/DESCRIPTION Show resolved Hide resolved
R/DESCRIPTION Show resolved Hide resolved
R/apache.sedona.Rproj Show resolved Hide resolved
R/R/apache.sedona-package.R Show resolved Hide resolved
@jiayuasu jiayuasu added this to the sedona-1.4.0 milestone Feb 28, 2023
@jiayuasu jiayuasu changed the title [SEDONA-244 & SEDONA-246] R align spark_read/write functions [SEDONA-244][SEDONA-246] R align spark_read/write functions Feb 28, 2023
@gregleleu
Copy link
Contributor Author

Also, man pages have so far not been committed to git. Which means i) someone needs to run devtools::document before building and ii) that could cause an issue if the "figures" folder is not there (typically it contains the badges). Other projects usually commit man pages.

BTW, the absence of man pages is the reason for one of the warnings in the GH Actions ("Warning: Undocumented code objects:" at the "Check" phase)

@jiayuasu
Copy link
Member

Can you create a separate PR to upload the man page? I think this PR is ready to go.

@gregleleu
Copy link
Contributor Author

Sure

@jiayuasu jiayuasu merged commit a67b8df into apache:master Feb 28, 2023
@gregleleu gregleleu deleted the r-spark_read branch March 4, 2023 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants