Skip to content

Creating a S3 bucket with lifecycle policy enabled and uploading a large file > 5 GB using S3 multipart upload.

Notifications You must be signed in to change notification settings

subhamay-cloudworks/0054-hyacinth-tf

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 

Repository files navigation

Project Hyacinth: S3 Multipart Upload

A user executes a shell script to split a large file (typically 100 MB or greater) into multiple part files. Then the script initiates the multipart upload and uploads each part file one by one. Once the part files upoload is complte the script completes the multipart upload by merging the part files into the original file.

Description

Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object. In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation.

Project Hyacinth - Design Diagram

Getting Started

Dependencies

  • Create a Customer Managed KMS Key in the region where you want to create the stack..
  • Modify the KMS Key Policy to let the IAM user encrypt / decrypt using any resource using the created KMS Key.

Installing

  • Clone the repository in a local hard drive.
  • Give execute permission to the bash script s3_multipart_upload.sh
  • Create a S3 bucket.

Executing program

  • Change directory where you have cloned the repository and give execute permission to the script
cd <script directory>
  • Provide execute permission to the bash script s3_multipart_upload.sh
chmod 755 s3_multipart_upload.sh
  • Execute the bash script from your local system using the following command
sh ./s3_multipart_upload.sh <Filename to be uploaded> <Split size in MB> <Target Bucket Name>

Help

Post message in my blog (https://blog.subhamay.com)

Authors

Contributors names and contact info

Subhamay Bhattacharyya - subhamay.aws@gmail.com

Version History

  • 0.1
    • Initial Release

License

This project is licensed under Subhamay Bhattacharyya. All Rights Reserved.

Acknowledgments

Inspiration, code snippets, etc.

About

Creating a S3 bucket with lifecycle policy enabled and uploading a large file > 5 GB using S3 multipart upload.

Topics

Resources

Stars

Watchers

Forks

Packages

No packages published