010_s3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/rzl-ds/gu511/blob/master/010_s3.ipynb\" target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# `aws s3` (Simple Storage Service)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "when it comes to managing your data, you can get pretty far with just an `ec2` server and a virtual hard drive.\n",
    "\n",
    "for example, from 2018 - 2019 I had a `postgres` server running on an `ec2` instance with a 64G hard drive, and that server polled the [`wmata` train position api](https://developer.wmata.com/docs/services/5763fa6ff91823096cac1057/operations/5763fb35f91823096cac1058) every 10 seconds for about 1 year. that is about 2.5B records. not exactly big data, but not exactly small data either."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "as another example, the long-running project scraping power outage information (mentioned last lecture) in 15 minute intervals. the resulting `json` files are saved to the `/data` directory directly on that machine, and (if nothing changes) will keep on downloading and storing those files forever"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are disadvantages to on-disk storage, though, like:\n",
    "\n",
    "?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "1. access\n",
    "    1. you have to log in to the linux server to access the files\n",
    "    2. this means I have to grant selective access to people\n",
    "2. cost\n",
    "    1. disk space (specifically: EBS (elastic block store)) on an `ec2` service is expensive, running about $0.10 per GB-month\n",
    "3. manual administration\n",
    "    1. if I want to grow the hard drive size, I have to do it myself\n",
    "    2. I have to know *when* that's going to happen, or I could fill my hard drive without paying attention\n",
    "    3. I could intentionally set up backup policies and redundancy mechanisms on my `ec2` server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`s3` is a service offered by `aws` to be a central file storage location, accessible from everywhere via standard REST api requests (`GET, PUT, COPY, POST, LIST`), which addresses some of those disadvantages:\n",
    "\n",
    "1. access\n",
    "    1. you can control access to any \"bucket\" (basically, a top-level directory in the file system) from the web console\n",
    "    2. you can control permissions down to the individual file level\n",
    "    3. you can be as permissive or as restrictive as you desire\n",
    "2. cost\n",
    "    1. standard storage starts at about $0.023 per GB-month (so about 23% the EBS `ec2` hard drive cost) for the first 50 TB, and gets *cheaper* from there\n",
    "3. manual administration\n",
    "    1. will grow on the fly without administration\n",
    "    2. redundancy and backup is a built-in service option\n",
    "    3. I can have easy-access logging and version information\n",
    "    4. I can just host a static webpage with a click of a button\n",
    "\n",
    "it's not perfect, but it's pretty cool"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and what about cons?\n",
    "\n",
    "1. size limit\n",
    "    1. there is a 5T size limit for individual files\n",
    "    2. there is a 5G size limit for individual `PUT` requests\n",
    "    3. there is a separate process recommended for loading large files: a [multipart upload](http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html)\n",
    "2. small changes to files are inefficient\n",
    "    1. require full download and re-upload\n",
    "3. sub-optimal as mounted drives on a server\n",
    "4. charged *per read*\n",
    "    1. not just charged for taking up space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## buckets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "open your `aws` console and go to the `s3` service\n",
    "\n",
    "https://s3.console.aws.amazon.com/s3/home?region=us-east-1\n",
    "\n",
    "everywhere you look: \"Buckets\". a *bucket* is effectively a top-level directory for a family of related files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you *could* create a single \"data\" bucket and keep everything in there, but you probably don't want to. why?\n",
    "\n",
    "1. a broad interpretation of the \"separation of concerns\" principle applies\n",
    "    1. you don't want to mix different files that are doing different things for different purposes\n",
    "2. permissions will be dicey\n",
    "    1. eventually you may want to be very restrictive or very permissive\n",
    "    2. you *can* control permissions on a per-file basis in `s3`, so it is *possible*\n",
    "    3. *but* other project stakeholders may not approve of this at all, or it may just be more onerous to set permissions on a per-file basis than on a bucket-wide level\n",
    "3. descriptive names actually make for better code\n",
    "    1. long path names that have nothing to do with a task or project are wasted words\n",
    "    2. descriptive names can help you understand exactly what you're looking for from the main page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: create an `s3` bucket</div>**\n",
    "\n",
    "1. pick a name -- it has to be *globally* unique (that is, across *all* of `s3`, not just *your* buckets)\n",
    "    + no uppercase letters and no underscore characters (`_`)\n",
    "    + one recommendation: `test-bucket.[YOUR GU ID GOES HERE].gu511`\n",
    "    + a common convention: name it like a \"url\", e.g. something like `*****.com`\n",
    "        + I often use `***********.lamberty.io` bucket names because I own that domain\n",
    "    + it doesn't have to be like either of the above -- it just has to be unique\n",
    "1. go with the `us-east-1` NOVA region\n",
    "1. leave all other values at their default\n",
    "1. submit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">tour the `s3` web console</div>**\n",
    "\n",
    "*note: this walkthrough is for the \"new\" console look; check the notes slide below for the same from the \"old\" console*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ main page\n",
    "    + pretty self-explanatory: a list of buckets and the ability to create new ones\n",
    "    + click on the radio button on a single bucket *line* (not the name link)\n",
    "    + top menu gives you the ability to\n",
    "        + copy the ARN of that bucket (e.g. `arn:aws:s3:::test1.rzl.gu511.com`)\n",
    "        + empty the contents of that bucket\n",
    "        + delete that bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ bucket page (off of any bucket name link)\n",
    "    + overview tab\n",
    "        + allows you to upload a file, create a directory, and set object permissions\n",
    "        + if you have files, you can click on those lines and bring up a right context menu\n",
    "            + includes information about size, versions, permission info, and path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ bucket page (cont)\n",
    "    + properties tab\n",
    "        + versioning: you have the ability to record version histories of every file in your bucket (off by default)\n",
    "        + server, object logging: log (to `s3`, $ for `aws`) access and file creation / manipulation log records, or specific `rest`ful requests\n",
    "        + static website hosting: host a webpage straight out out of your `s3` bucket (not bad huh?)\n",
    "        + encryption: set up server-side encryption (that is, scrambled on `aws`'s servers) for your files\n",
    "        + object lock: like it says, prevent an object from being deleted\n",
    "        + tags: good for collecting resources for shared purpose or project\n",
    "        + transfer acceleration: pay for faster access\n",
    "        + events: create a trigger when a file gets dropped to a location\n",
    "        + requester pays: like calling collect, but for the internet age"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ bucket page (cont)\n",
    "    + permissions tab\n",
    "        + block public access\n",
    "        + access control list: control bucket-level permissions for `aws` users and the public\n",
    "        + object ownership: specify whether the *object writer* or the *bucket owner* gets to be owner of new objects\n",
    "        + bucket policy: generate an `iam` policy which you can later attach to `iam` users or roles\n",
    "        + CORS configuration: advanced; allows you to allow other web services to access these files as if they were local to that web service (forbidden by default)\n",
    "    + management tab\n",
    "        + lifecycle: making sure you pay less for old stuff you don't use\n",
    "        + replication: making sure you have multiple copies of your most important things\n",
    "        + a suite of other analytics and monitoring tools; not useful at this time since we have no information in these buckets\n",
    "    + access points\n",
    "        + a menu for creating advanced ways to access these files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "\"old\" console walkthrough\n",
    "\n",
    "1. main page\n",
    "    1. pretty self-explanatory: a list of buckets and the ability to create new ones\n",
    "    1. click on any bucket *line* and a right context menu comes up\n",
    "        1. basic descriptions of the three types of configuration values (properties, permissions, management), and the ability to quickly access the `arn` (amazon resource name)\n",
    "1. bucket page (off of any bucket name link)\n",
    "    1. overview tab\n",
    "        1. allows you to upload a file, create a directory, and set object permissions\n",
    "        1. if you have files, you can click on those lines and bring up a right context menu\n",
    "    1. properties tab\n",
    "        1. versioning: you have the ability to record version histories of every file in your bucket (off by default)\n",
    "        1. server, object logging: log (to `s3`, $ for `aws`) access and file creation / manipulation log records, or specific `rest`ful requests\n",
    "        1. static website hosting: host a webpage straight out out of your `s3` bucket (not bad huh?)\n",
    "        1. encryption: set up server-side encryption (that is, scrambled on `aws`'s servers) for your files\n",
    "        1. tags: good for collecting resources for shared purpose or project\n",
    "        1. transfer acceleration: pay for faster access\n",
    "        1. events: create a trigger when a file gets dropped to a location\n",
    "        1. requester pays: like calling collect, but for the internet age\n",
    "    1. permissions tab\n",
    "        1. block public access\n",
    "        1. access control list: control bucket-level permissions for `aws` users and the public\n",
    "        1. bucket policy: generate an `iam` policy which you can later attach to `iam` users or roles\n",
    "        1. CORS configuration: advanced; allows you to allow other web services to access these files as if they were local to that web service (forbidden by default)\n",
    "    1. management tab\n",
    "        1. lifecycle: making sure you pay less for old stuff you don't use\n",
    "        1. replication: making sure you have multiple copies of your most important things\n",
    "        1. a suite of other analytics and monitoring tools; not useful at this time since we have no information in these buckets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### putting things in the bucket: web interface\n",
    "\n",
    "putting things into a bucket via the web interface is pretty intuitive\n",
    "\n",
    "1. click on the bucket name\n",
    "2. create a directory if you want\n",
    "3. click the upload button\n",
    "\n",
    "while uploading, you'll see a couple of different dialogs. let's upload a file together, and discuss some of the dialogs while we're doing it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: add a file to the bucket via the web console</div>**\n",
    "\n",
    "1. create a text file on your *local* computer (not your `ec2` server)\n",
    "    1. do it any way you want -- something super simple.\n",
    "2. in your bucket, create a folder called \"helloworld\"\n",
    "3. click the \"Upload\" button\n",
    "    1. select your file and click next\n",
    "        + note: callouts to the REST api\n",
    "    2. leave permissions as-is\n",
    "        + note: here is one place you could grant others -- or everyone! -- access\n",
    "    3. leave properties as-is\n",
    "        1. storage class discussion (notes below)\n",
    "        2. encryption discussion (notes below)\n",
    "        3. metadata discussion (notes below)\n",
    "    4. review and upload"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### storage classes\n",
    "\n",
    "there are different ways of storing files in `s3`, and the primary dimensions of difference are\n",
    "\n",
    "1. cost per GB of storage space used\n",
    "2. cost per request for or upload of a file\n",
    "3. availability (how often will the system fail to find your file at a given moment)\n",
    "4. redundancy (how many backup copies of your file exist across all of the aws internal storage system)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "given these dimensions of difference, the options available are\n",
    "\n",
    "+ standard\n",
    "    + default behavior\n",
    "+ intelligent tiering\n",
    "    + halfway between standard (above) and standard-ia (below)\n",
    "    + wait for your users to tell you how rarely or frequently you need files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ standard-ia\n",
    "    + `ia` for \"Infrequent Access\"\n",
    "    + cheaper storage but more expensive per access request\n",
    "    + good for long-term storage of files\n",
    "    + you can rotate files from `standard` to `standard-ia`\n",
    "+ one-zone ia\n",
    "    + this option is a little bit cheaper and made for things you are \"more comfortable\" losing\n",
    "    + still not extremely likely to lose files this way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ `glacier`\n",
    "    + this is a \"deeper\" level of storage than `standard-ia`\n",
    "    + much cheaper per `GB`\n",
    "    + requests for files are neither instant nor cheap\n",
    "    + this is basically just tape archives of your files\n",
    "+ `glacier` deep archive\n",
    "    + go even deeper -- in`s3`ption\n",
    "    + if you can afford to wait hours to get something every few months, this is the storage level for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### encryption\n",
    "\n",
    "the files you are loading to and downloading from `s3` can be encrypted or unencrypted (*aka* in plain text) during different stages of the process\n",
    "\n",
    "+ on upload or download: the text sent over the internet in the upload or download request could be plain text or decrypted\n",
    "+ in storage: the actual item stored in `s3` could be encrypted or unencrypted\n",
    "\n",
    "there are basically two times a malicious agent could \"steal\" your data:\n",
    "\n",
    "1. in transit (during upload or download over the internet)\n",
    "2. at rest (while it's located on some aws machine hosting the `s3` service)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the different types of encryption policies and schemes basically boil down to which of those options you try to prevent.\n",
    "\n",
    "if you don't care at all, you're not performing any encryption.\n",
    "\n",
    "now suppose you only care to prevent the latter (at rest, people \"hacking\" `s3` as a service), and you don't care if you're sending that information over the internet un-encrypted. schemes that allow un-encrypted information to be transferred from your local computer to the `s3` service and *then* encrypt the information for `s3` storage, and which decrypt that information *prior* to sending it back to you (the client) are referred to as *server-side encryption*, because all of the information about how files are encrypted / decrypted resides on the server side of the transaction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "now suppose you want to make sure your data never leaves your physical network (e.g. your computer, or your government-agency-wide network). schemes which focus on encrypting files before upload and after download are referred to as *client-side encryption* because the knowledge about how files are encrypted / decrypted resides on the client (your local computer) side of the transaction.\n",
    "\n",
    "if you're a paranoid sadist, you could do both -- have the server \"double\" encrypt the encrypted message."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "when you're selecting the encryption option on the file upload dialog, you are *only* talking about server-side encryption. This will not affect how data is transferred to and from `s3`, but rather under what conditions it will be saved within `s3` itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are three options\n",
    "\n",
    "+ none\n",
    "    + the file will be saved in `s3` exactly as it was presented to `s3`\n",
    "    + this is the chosen option for both no encryption and client-side encryption\n",
    "+ amazon `s3` master-key\n",
    "    + this server-side encryption option will manage the keys within the `s3` service itself\n",
    "    + each file will get a unique key\n",
    "+ amazon `kms` master-key\n",
    "    + this also does server-side encryption\n",
    "    + encryption keys are controlled via a separate service, the `kms` service\n",
    "        + you access this service as part of the `iam` service, `iam` > \"encryption keys\"\n",
    "    + for the `cli` and the `sdk`s, there are some \"signature\" issues (the signature of the files returned in `GET` requests is not the `md5` sum of the file), but they are trivial to fix\n",
    "    + the file returned to you is in plain text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### metadata\n",
    "\n",
    "there are several options in terms of meta-data. there are effectively three types of metadata fields on offer here:\n",
    "\n",
    "1. standard `http` header message metadata\n",
    "    1. these control how web-aware applications will handle your `s3` files\n",
    "    2. of these, the one most worth knowing about from the start is the `content-type`, a field which offers a hint at how applications are supposed to interact with the file based on its type\n",
    "2. `aws s3`-specific metadata\n",
    "3. user-specific metadata\n",
    "    1. for whatever information I want to tag these files with\n",
    "    2. can be useful for bulk processing of your files, for example, based on header data queries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "now that you've created files, let's investigate the items available in the web console\n",
    "\n",
    "1. click on the file *line* and review the side panel\n",
    "    1. click on the URL in the side panel\n",
    "    2. go back in your browser (to the file overview tab)\n",
    "2. the file overview tab\n",
    "    1. the URL of the file is the same as the previous link\n",
    "    2. you can click \"open\", though, to actually open this file\n",
    "    3. you could download from here\n",
    "3. click on the \"test\" link in the breadcrumbs menu at the top\n",
    "4. right click one of the line items for a full context menu of options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### putting things in the bucket: `cli`\n",
    "\n",
    "that's all well and good, but eventually we're going to want to be interacting with the `s3` service via applications and from the command line. fortunately, both the `aws cli` and the `python boto3` libraries make that straight-forward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "log in to your `ec2` server and check out the things that the `aws cli` can do with the `s3` service:\n",
    "\n",
    "```bash\n",
    "aws s3 help\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: add a file to the bucket via the `aws cli`</div>**\n",
    "\n",
    "1. create a text file on your `ec2` sever\n",
    "2. upload the file to the `s3` service\n",
    "    1. use the `aws s3 cp` command\n",
    "    2. post it to the bucket and directory you created above\n",
    "3. verify (in the web console) that your file was posted successfully\n",
    "\n",
    "```bash\n",
    "echo \"hello world\" >> /tmp/helloworld.txt\n",
    "aws s3 cp /tmp/helloworld.txt s3://test-bucket-2020.rzl5.gu511/test/helloworld.cli.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### putting things in the bucket: `python` and `boto3`\n",
    "\n",
    "under the hood, the `cli` is just using the `boto3` library -- so we should be able to easily reproduce this behavior within a `python` script / shell.\n",
    "\n",
    "with `s3` we can *upload* or *download* file objects, and with `python` we can do it from/to files or `python` `str`s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for starters, let's go ahead and enter a `python` shell. on your `ec2`:\n",
    "\n",
    "```bash\n",
    "python\n",
    "```\n",
    "\n",
    "and then\n",
    "\n",
    "```python\n",
    "import boto3\n",
    "session = boto3.session.Session(region_name='us-east-1')\n",
    "s3 = session.resource('s3')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# import boto3\n",
    "\n",
    "# s3 = boto3.client('s3')\n",
    "# help(s3.upload_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: upload a local file to the bucket using the `boto3` `python` library</div>**\n",
    "\n",
    "1. create an `s3` `resource` object\n",
    "2. get the `bucket` object corresponding to our desired bucket\n",
    "3. use the `bucket.upload_file` function\n",
    "4. verify it worked on the `s3` console\n",
    "\n",
    "```python\n",
    "import boto3\n",
    "\n",
    "session = boto3.session.Session(region_name='us-east-1')\n",
    "\n",
    "s3 = session.resource('s3')\n",
    "bucket = s3.Bucket('test1.rzl.gu511.com')\n",
    "bucket.upload_file(Filename='/tmp/helloworld.txt',\n",
    "                   Key='test/helloworld.boto3.txt')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: download an `s3` file to the local disk</div>**\n",
    "\n",
    "1. create an `s3` `resource` object\n",
    "2. get the `bucket` object corresponding to our desired bucket\n",
    "3. use the `bucket.download_file` method\n",
    "\n",
    "```python\n",
    "import boto3\n",
    "\n",
    "session = boto3.session.Session(region_name='us-east-1')\n",
    "\n",
    "s3 = session.resource('s3')\n",
    "bucket = s3.Bucket('test1.rzl.gu511.com')\n",
    "bucket.download_file(Key='test/helloworld.boto3.txt',\n",
    "                     Filename='/tmp/helloworld.cli.downloadedtxt')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: create an `s3` file and update contents from a `str` (no local file)</div>**\n",
    "\n",
    "1. create an `s3` `resource` object\n",
    "2. get the `bucket` object corresponding to our desired bucket\n",
    "3. use the `Object` method to create a new file object on that bucket, and set the body of that file to be a hard-coded string message\n",
    "4. verify it worked on the `s3` console\n",
    "\n",
    "```python\n",
    "import boto3\n",
    "\n",
    "session = boto3.session.Session(region_name='us-east-1')\n",
    "\n",
    "s3 = session.resource('s3')\n",
    "bucket = s3.Bucket('test1.rzl.gu511.com')\n",
    "s3file = bucket.Object(key='test/goodbyeworld.boto3.txt')\n",
    "s3file.put(Body='goodbye, world!',\n",
    "           ContentType='text/plain')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: loading the contents of an `s3` file to a `str`</div>**\n",
    "\n",
    "1. create an `s3` `resource` object\n",
    "2. get the `bucket` object corresponding to our desired bucket\n",
    "3. use the `bucket.Object` method to access a file object\n",
    "4. use the file object's `get` method to get a `json` representation of the file\n",
    "5. use the `read` method of the `get` response's `Body` value to read the contents to a string\n",
    "\n",
    "```python\n",
    "import boto3\n",
    "\n",
    "session = boto3.session.Session(region_name='us-east-1')\n",
    "\n",
    "s3 = session.resource('s3')\n",
    "bucket = s3.Bucket('test1.rzl.gu511.com')\n",
    "s3file = bucket.Object(key='test/goodbyeworld.boto3.txt')\n",
    "\n",
    "response = s3file.get()\n",
    "print(response)\n",
    "\n",
    "response['Body'].read().decode()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<strong><em><div align=\"center\">sssuper</div></em></strong>\n",
    "<div align=\"center\"><img src=\"https://vice-images.vice.com/images/content-images-crops/2016/07/27/that-s-thing-everyone-drew-in-school-what-is-it-body-image-1469592131-size_1000.jpg?output-quality=75\" width=\"500px\"></div>\n",
    "\n",
    "# END OF LECTURE\n",
    "\n",
    "next lecture: [`aws` serverless function architecture `lambda`](011_lambda.ipynb)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}