2017_11_09_b.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises due by EOD 2017.11.09"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## goal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in this homework assignment we will focus on `aws lambda` function deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## method of delivery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as mentioned in our first lecture, the method of delivery may change from assignment to assignment. we will include this section in every assignment to provide an overview of how we expect homework results to be submitted, and to provide background notes or explanations for \"new\" delivery concepts or methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this week you will be submitting the results of your homework via upload to two different `s3` buckets. the first is one I created for you and will be used for question 1, and the second is the same homework bucket you have been using now for several weeks.\n",
    "\n",
    "summary:\n",
    "\n",
    "| exercise | deliverable                               | method of delivery                                  |\n",
    "|----------|-------------------------------------------|-----------------------------------------------------|\n",
    "| 1        | deployment archive `airportweather.zip`   | upload to our `s3` bucket: `2017.fall.gu511.{GUID}` |\n",
    "| 1        | deployment command `how_i_deployed_it.sh` | upload to our `s3` bucket: `2017.fall.gu511.{GUID}` |\n",
    "| 2        | a test message in a database              | `INSERT`ed into a shared `aws rds` database         |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# exercise 1: create a `lambda` function from a deployment package\n",
    "\n",
    "let's create a `zip` archive containing all the material we need to execute a `lambda` function.\n",
    "\n",
    "*note*: for this assignment, you must perform all of your work in a `linux` envrionment. the `python` code we will use will be OS-agnostic but the external library (`requests`) may not be. your mac or windows laptops are not guaranteed to produce the same results and I haven't tested or verified whether they would.\n",
    "\n",
    "it's also worth noting: the `lambda` function code will run on the `amazon linux ami`, so you may want to consider creating a new `ec2` instance using that `ami` for your work (though I can confirm that the `ubuntu ami` we used for our `ec2` instances *will* work in this instance).\n",
    "\n",
    "\n",
    "## get the `python` files\n",
    "\n",
    "create an isolated directory on your `linux` instance with the name `airportweather`, and download into this new directory the following two `python` files:\n",
    "\n",
    "+ `airportweather.py`: a file implementing a simple `requests HTTP` call to our dca weather api\n",
    "    + https://s3.amazonaws.com/shared.rzl.gu511.com/airportweather.py \n",
    "+ `utils.py`: generalized / abstracted utility functions for defining future `aws lambda` functions\n",
    "    + https://s3.amazonaws.com/shared.rzl.gu511.com/utils.py\n",
    "\n",
    "using a `python 3.6` environment with the `requests` library installed (via `conda install requests`), verify that you have correctly downloaded the file by running\n",
    "\n",
    "```bash\n",
    "python airportweather.py\n",
    "```\n",
    "\n",
    "and verifying that the `json` output of the dca weather api is printed to the screen. you should see something like\n",
    "\n",
    "```json\n",
    "{'statusCode': '200', 'body': {'delay': 'false', 'IATA': 'DCA', 'state': 'District of Columbia', 'name': 'Ronald Reagan Washington National', 'weather': {'visibility': 10.0, 'weather': 'Overcast', 'meta': {'credit': \"NOAA's National Weather Service\", 'updated': '4:52 PM Local', 'url': 'http://weather.gov/'}, 'temp': '61.0 F (16.1 C)', 'wind': 'Southwest at 5.8mph'}, 'ICAO': 'KDCA', 'city': 'Washington', 'status': {'reason': 'No known delays for this airport.', 'closureBegin': '', 'endTime': '', 'minDelay': '', 'avgDelay': '', 'maxDelay': '', 'closureEnd': '', 'trend': '', 'type': ''}}, 'headers': {'Content-Type': 'application/json'}}\n",
    "```\n",
    "\n",
    "\n",
    "## get a copy of the `requests` library\n",
    "\n",
    "if we were to just `zip` up the two `python` files right now and deploy those, our script would fail. why?\n",
    "\n",
    "we use the `requests` library, and our `python 3.6` runtime **does not come with** that library. `requests`, in turn, depends on several *other* non-standard libraries. we need to include them all in our deployment package\n",
    "\n",
    "one standard way of building up this deployment package is to do the following:\n",
    "\n",
    "1. create a brand new `conda` environment named `airportweather` with `python 3.6`\n",
    "2. `activate` that environment\n",
    "3. `install requests` (and automatically its dependencies)\n",
    "    1. this last command will install many packages -- make note of them!\n",
    "\n",
    "for example, when I installed in `ubuntu`:\n",
    "\n",
    "```bash\n",
    "asn1crypto:   0.22.0-py36h265ca7c_1\n",
    "cffi:         1.10.0-py36had8d393_1\n",
    "chardet:      3.0.4-py36h0f667ec_1 \n",
    "cryptography: 2.0.3-py36ha225213_1 \n",
    "idna:         2.6-py36h82fb2a8_1   \n",
    "pycparser:    2.18-py36hf9f622e_1  \n",
    "pyopenssl:    17.2.0-py36h5cc804b_0\n",
    "pysocks:      1.6.7-py36hd97a5b1_1 \n",
    "requests:     2.18.4-py36he2e5f8d_1\n",
    "six:          1.11.0-py36h372c433_1\n",
    "urllib3:      1.22-py36hbe7ace6_0  \n",
    "```\n",
    "\n",
    "we need to provide these extra libraries, and the way we will do that is by taking the files that define it (in our `conda` environment) and copying them into the same folder our other `python` code lives in -- they will then be included in our deployment archive.\n",
    "\n",
    "you may already know where those files are located, but if not, you can find out. in a `python` session, execute the following:\n",
    "\n",
    "```python\n",
    "import requests\n",
    "print(requests.__file__)\n",
    "```\n",
    "\n",
    "the result should be something like\n",
    "\n",
    "```bash\n",
    "/path/to/conda/env/airportweather/lib/python3.6/site-packages/requests/__init__.py\n",
    "```\n",
    "\n",
    "many packages and files live in `/path/to/conda/env/airportweather/lib/python3.6/site-packages/` -- these files form the basis of our `python` environment.\n",
    "\n",
    "copy each of them to the `airportweather` directory:\n",
    "\n",
    "```bash\n",
    "# this assumes your working directory (pwd) is your `airportweather` directory\n",
    "cp -r /path/to/conda/env/airportweather/lib/python3.6/site-packages/* .\n",
    "```\n",
    "\n",
    "this means that your `airportweather` directory should have many items in it:\n",
    "\n",
    "1. `airportweather.py`\n",
    "2. `utils.py`\n",
    "3. the `requests` folder\n",
    "4. several other `python` library folders and files\n",
    "\n",
    "*note*: many of these files and folders are unecessary. The `pip`, `wheel`, `certifi`, `OpenSSL`, `pkg_resources`, and `setuptools` directories were all included *before* we `install`ed `requests` -- they are *built-in* packages, so they will already exist in our `aws lambda` runtime. in a true deployment setting, we would prune these libraries before moving on. for now, though, let's just leave them in for simplicity's sake\n",
    "\n",
    "\n",
    "## `zip` it\n",
    "\n",
    "zip the contents of `airportweather` into a `zip` archive named `airportweather.zip`:\n",
    "\n",
    "```bash\n",
    "zip -r9 airportweather.zip ./*\n",
    "```\n",
    "\n",
    "now, right besides the two `python` library folders and two `python` files, you should *also* have one `zip` archive.\n",
    "\n",
    "\n",
    "## deploy it\n",
    "\n",
    "using the `aws lambda create-function` subcommand, *deploy* this `zip` archive as a `lambda` function named `airportweather`. create a new `iam role` for this or re-use one from previous homeworks or the lecture -- that much is up to you! you will need to determine values for the following command line flags:\n",
    "\n",
    "+ `function-name`\n",
    "+ `zip-file`\n",
    "+ `role`\n",
    "+ `handler`\n",
    "+ `runtime`\n",
    "\n",
    "write whatever `aws` command(s) you use to deploy this `lambda` function into a plain-text shell script file `how_i_deployed_it.sh`\n",
    "\n",
    "\n",
    "## test it\n",
    "\n",
    "this command should have created a new `lambda` function for you named `airportweather`. test that `lambda` function from the command line using the `aws lambda invoke` command:\n",
    "\n",
    "```bash\n",
    "aws lambda invoke  \\\n",
    "    --function-name airportweather  \\\n",
    "    --payload '{\"airportcode\": \"DCA\"}'  \\\n",
    "    /tmp/dcaweather.json\n",
    "```\n",
    "\n",
    "this command should print to the terminal\n",
    "\n",
    "```bash\n",
    "{\n",
    "    \"StatusCode\": 200\n",
    "}\n",
    "```\n",
    "\n",
    "and the contents of `/tmp/dcaweather.json` should be the familiar DCA weather report in `json` form.\n",
    "\n",
    "if you see the same thing, congratulations -- you've deployed a complex `lambda` function!\n",
    "\n",
    "\n",
    "##### upload `airportweather.zip` and `how_i_deployed_it.sh` to the new `2017.fall.gu511.{GUID} s3` bucket (which we own but on which you should have full permissions). make sure you add flags `--acl bucket-owner-full-control`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exercise 2: connect and write to a `postgres` database\n",
    "\n",
    "we have set up a shared `postgres` database with the following connection details:\n",
    "\n",
    "| parameter  | value                                                       |\n",
    "|------------|-------------------------------------------------------------|\n",
    "| `host`     | `rzl-gu511-shared.cdmknaubrmaw.us-east-1.rds.amazonaws.com` |\n",
    "| `port`     | `5432`                                                      |\n",
    "| `username` | `gu511shared`                                               |\n",
    "| `db`       | `gu511`                                                     |\n",
    "| `password` | written to `s3://2017.fall.gu511.{GUID}/dbpw.txt`           |\n",
    "\n",
    "we would like for you to install the `psql` client on your `ec2 ubuntu` instances, use it to connect to this shared database, and `insert` a value into a shared table.\n",
    "\n",
    "\n",
    "## install `psql`\n",
    "\n",
    "on your `ec2 ubuntu` instances, install `psql` using the command\n",
    "\n",
    "```bash\n",
    "sudo apt install postgresql-client\n",
    "```\n",
    "\n",
    "you can verify that your installation worked be checking\n",
    "\n",
    "```bash\n",
    "which psql\n",
    "```\n",
    "\n",
    "\n",
    "## connect to the database\n",
    "\n",
    "using the values we provided in the table above, the course notes, and `man psql`, figure out how to connect to our shared database with the `psql` command\n",
    "\n",
    "\n",
    "## `insert` a new record\n",
    "\n",
    "the following command will insert a record into a shared database we've created called `gu511_shared_table`. the table has two columns:\n",
    "\n",
    "1. `guid`: your georgetown user id\n",
    "2. `message`: a test message to verify you were able to successfully insert your record\n",
    "\n",
    "once you've successfully connect to the `postgres` database, insert a record with the following query (replacing `MY_GUID` and `MY_MESSAGE` with real values, of course):\n",
    "\n",
    "```sql\n",
    "INSERT INTO gu511_shared_table (guid, message) VALUES ('MY_GUID', 'MY_MESSAGE');\n",
    "```\n",
    "\n",
    "*note*: the single quotes in `'MY_GUID', 'MY_MESSAGE'` are required. they indicate that the values being passed in are strings, which is the datatype of those two coumns in our table\n",
    "\n",
    "\n",
    "##### the submission is the record in the shared database -- we will check results by running `SELECT * FROM gu511_shared_table;`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}