s3scripts

bash scripts to interact with S3

I have written these scripts a few times at my past couple jobs using my memory and my stackoverflow answer. So, now thanks to Grokker, I am able to release these. Hopefully they will help someone who needs to manipulate S3 objects from bash.

---

Usage:

All of the scripts take an AWSID and Secret as the first two arguments. Most of them then take a file and bucket as the next two.

• s3_get.sh - copy an object from S3 into a local file
• s3_put.sh - copy a local file into an S3 object
• s3_head.sh - get some information on an S3 object
• s3_delete.sh - remove an object from S3
• s3_list.sh - list the contents of an S3 bucket
• s3_make_public.sh - change the acl on an S3 object into 'public'
• s3_copy_bucket.sh - copy the objects from one S3 bucket into another, uses s3_get.sh and s3_put.sh to ensure large objects (>5GB) get copied
• s3_list_multipart_uploads.sh - list unfinished multipart uploads including the uploadid. Objects greater then 5GB require the use of AWS's multipart upload process
• s3_abort_multipart_upload.sh - abort any unfinished multipart upload

### s3_get.sh ### s3_get.sh [local file] - Amazon Web Services ID - Amazon Web Services Secret - The object key (file) in the bucket to retrieve - The S3 bucket which contains the object [local file] - An optional local file to store the object in, by default it saves it in (any directories included) ### s3_put.sh ### s3_put.sh [object key] [mime type] - Amazon Web Services ID - Amazon Web Services Secret - The file to put into an object in S3 - The S3 bucket which will contain the object [object key] - An optional object key to store the object within, by default it will be the basename of the local file (no directories) [mime type] - An optional mime type to be set, by default it will use *file* to automatically determine the mime type. *file* fails on .js .css and .svg, so those are hard coded. [local file] - An optional local file to store the object in, by default it saves it in (any directories included) ### s3_head.sh ### s3_head.sh - Amazon Web Services ID - Amazon Web Services Secret - The object key (file) in the bucket to get information about - The S3 bucket which contains the object ### s3_delete.sh ### s3_delete.sh - Amazon Web Services ID - Amazon Web Services Secret - The object key (file) in the bucket to remove - The S3 bucket which contains the object ### s3_list.sh ### s3_list.sh [all?] [marker] - Amazon Web Services ID - Amazon Web Services Secret - The S3 bucket from which you want a listing of object [all?] - Optional, if specified and not an empty string it will list all of the objects in the bucket, otherwise it will only list the first 1,000 [marker] - Optional, if specified it will start listing objects after this object key ### s3_make_public.sh ### s3_make_public.sh - Amazon Web Services ID - Amazon Web Services Secret - The object key (file) in the bucket to make publicly available - The S3 bucket which contains the object ### s3_copy_bucket.sh ### s3_copy_bucket.sh - Amazon Web Services ID - Amazon Web Services Secret - The S3 bucket which contains the objects you want copied - The S3 bucket into which you want the objects copied ### s3_list_multipart_uploads.sh ### s3_list_multipart_uploads.sh [all?] [marker] - Amazon Web Services ID - Amazon Web Services Secret - The S3 bucket from which you want a listing of object [all?] - Optional, if specified and not an empty string it will list all of the current multipart uploads not finished in the bucket, otherwise it will only list the first 1,000 [marker] - Optional, if specified it will start listing unfinished multipart uploads after this object key ### s3_abort_multipart_upload.sh ### s3_abort_multipart_upload.sh - Amazon Web Services ID - Amazon Web Services Secret - The object key (file) in the bucket to remove - The S3 bucket which would contain the object - The uploadid supplied for the multipart upload. You can also get the uploadid from s3_list_multipart_uploads.sh