Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize S3 storage writing for MSQ durable storage #16481

Merged
merged 34 commits into from
Jun 7, 2024
Merged

Optimize S3 storage writing for MSQ durable storage #16481

merged 34 commits into from
Jun 7, 2024

Commits on Jun 4, 2024

  1. Optimise S3 storage writing for MSQ durable storage

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    38f81eb View commit details
    Browse the repository at this point in the history

  2. Get rid of static ConcurrentHashMap

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    bf518d8 View commit details
    Browse the repository at this point in the history

  3. Fix static checks

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    a920f74 View commit details
    Browse the repository at this point in the history

  4. Fix tests

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    38dc51c View commit details
    Browse the repository at this point in the history

  5. Remove unused constructor parameter chunkValidation + relevant cleanup

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    934ecd1 View commit details
    Browse the repository at this point in the history

  6. Assert etags as String instead of Integer

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    82fa878 View commit details
    Browse the repository at this point in the history

  7. Fix flaky test

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    bc55f0a View commit details
    Browse the repository at this point in the history

  8. Inject executor service

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    438008e View commit details
    Browse the repository at this point in the history

  9. Make threadpool size dynamic based on number of cores

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    f5a1514 View commit details
    Browse the repository at this point in the history

  10. Fix S3StorageDruidModuleTest

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    c266a9e View commit details
    Browse the repository at this point in the history

  11. Fix S3StorageConnectorProviderTest

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    ce60f7e View commit details
    Browse the repository at this point in the history

  12. Fix injection issues

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    c382daf View commit details
    Browse the repository at this point in the history

  13. Add S3UploadConfig to manage maximum number of concurrent chunks dyna… 

    …mically based on chunk size
    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    2224d11 View commit details
    Browse the repository at this point in the history

  14. Address the minor review comments

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    262182c View commit details
    Browse the repository at this point in the history

  15. Refactor S3UploadConfig + ExecutorService into S3UploadManager

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    6d62bf9 View commit details
    Browse the repository at this point in the history

  16. Address review comments

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    a705294 View commit details
    Browse the repository at this point in the history

  17. Make updateChunkSizeIfGreater() synchronized instead of recomputeMaxC… 

    …oncurrentNumChunks()
    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    1c0bfe5 View commit details
    Browse the repository at this point in the history

  18. Address the minor review comments

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    7418790 View commit details
    Browse the repository at this point in the history

  19. Fix intellij-inspections check

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    fbe71e3 View commit details
    Browse the repository at this point in the history

  20. Refactor code to use futures for maxNumConcurrentChunks. Also use exe… 

    …cutor service with blocking queue for backpressure semantics.
    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    5c83514 View commit details
    Browse the repository at this point in the history

  21. Update javadoc

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    8d6442b View commit details
    Browse the repository at this point in the history

  22. Get rid of cyclic dependency injection between S3UploadManager and S3… 

    …OutputConfig
    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    295689f View commit details
    Browse the repository at this point in the history

  23. Fix RetryableS3OutputStreamTest

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    6ec023f View commit details
    Browse the repository at this point in the history

  24. Remove unnecessary synchronization parts from RetryableS3OutputStream

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    f43a0c9 View commit details
    Browse the repository at this point in the history

  25. Update javadoc

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    79a22ed View commit details
    Browse the repository at this point in the history

  26. Add S3UploadManagerTest

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    da5e57a View commit details
    Browse the repository at this point in the history

  27. Revert back to S3StorageConnectorProvider extends S3OutputConfig

    @Akshat-Jain
    Akshat-Jain committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    059a5e9 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Address Karan's review comments

    @Akshat-Jain
    Akshat-Jain committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    70c7352 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. Address Kashif's review comments

    @Akshat-Jain
    Akshat-Jain committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    749b20d View commit details
    Browse the repository at this point in the history

  2. Change a log message to debug

    @Akshat-Jain
    Akshat-Jain committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    258376e View commit details
    Browse the repository at this point in the history

  3. Address review comments

    @Akshat-Jain
    Akshat-Jain committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    8e7882e View commit details
    Browse the repository at this point in the history

  4. Fix intellij-inspections check

    @Akshat-Jain
    Akshat-Jain committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    f3df9c7 View commit details
    Browse the repository at this point in the history

  5. Fix checkstyle

    @Akshat-Jain
    Akshat-Jain committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a474008 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. Merge branch 'master' into s3-output-stream

    @asdf2014
    asdf2014 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    8ef6086 View commit details
    Browse the repository at this point in the history