Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Docker build improvements #1425

Merged
merged 9 commits into from
May 24, 2024

Conversation

Kontinuation
Copy link
Member

@Kontinuation Kontinuation commented May 24, 2024

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

  • No, this is a CI update. The PR name follows the format [CI] my subject.

What changes were proposed in this PR?

  • Supports building docker images for Spark 3.5
  • Automatically pickup GeoTools version when building docker images for latest (probably snapshot) version of Sedona
  • Add dockerignore file to reduce the size of build context
  • Check if RAM is sufficient to run driver + executor and show hints for tuning DRIVER_MEM and EXECUTOR_MEM
  • Add SPARK_WORKER_MEMORY config to ensure the worker has enough memory to be allocated
  • Allow terminating the container using Ctrl-C
  • Fix matrix declaration for docker build workflow
  • Spark-extension version now matches with the Spark version

How was this patch tested?

Tested locally and passing CI.

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the documentation.

@Kontinuation Kontinuation force-pushed the improve-docker-build branch 2 times, most recently from 896e911 to 69863bc Compare May 24, 2024 03:09
@Kontinuation Kontinuation force-pushed the improve-docker-build branch 2 times, most recently from 255bdad to 02992e0 Compare May 24, 2024 03:28
@Kontinuation Kontinuation marked this pull request as ready for review May 24, 2024 05:24
Copy link
Member

@jiayuasu jiayuasu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@jiayuasu jiayuasu merged commit 2de9160 into apache:master May 24, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants