Skip to content

Asynchronously replicates Spark shuffle files in a distributed storage layer.

License

Notifications You must be signed in to change notification settings

eltorocorp/async-shuffle-upload

 
 

Repository files navigation

Spark Async Shuffle Upload Plugin

An implementation of SPARK-25299 that continues to write shuffle files to local disk, but asynchronously replicates shuffle files to a distributed storage layer.

More details to follow!

Gradle Tasks

./gradlew tasks - to get the list of gradle tasks

Start Developing

Run one of the following commands:

  • ./gradlew idea for IntelliJ
  • ./gradlew eclipse for Eclipse

About

Asynchronously replicates Spark shuffle files in a distributed storage layer.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Languages

  • Java 91.3%
  • Scala 8.7%