2017_09_28.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises due by EOD 2017.09.28"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## goal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this set of exercises, we will\n",
    "\n",
    "+ use the `conda` environment manager to launch a `jupyter` notebook, do a bit o' data science, and create some plots.\n",
    "+ create an `iam` policy and role to allow external access to some read-only properties of our `iam` configurations\n",
    "+ perform some simple `aws cli` and `boto3` actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## method of delivery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as mentioned in our first lecture, the method of delivery may change from assignment to assignment. we will include this section in every assignment to provide an overview of how we expect homework results to be submitted, and to provide background notes or explanations for \"new\" delivery concepts or methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this week you will be submitting the results of your homework via email.\n",
    "\n",
    "summary:\n",
    "\n",
    "| number | deliverable                                              | method of delivery |\n",
    "|--------|----------------------------------------------------------|--------------------|\n",
    "| 1      | none                                                     | none               |\n",
    "| 2      | none                                                     | none               |\n",
    "| 3      | completed jupyter notebook, converted to `html` document | include in email   |\n",
    "| 4      | `aws arn` of a specially created `iam role`              | include in email   |\n",
    "| 5      | `aws_commands.sh`, a list of four `aws cli` commands     | attach to email    |\n",
    "| 6      | list of `iam policy`s (corresponding to `aws` commands)  | include in email   |\n",
    "\n",
    "details: \n",
    "\n",
    "+ exercise 1\n",
    "    + in this exercise you will make sure you have an environment ready to launch a `jupyter notebook` and the ability to connect to that notebook service\n",
    "    + nothing to turn in here\n",
    "+ exercise 2\n",
    "    + in this exercise you will download a `jupyter notebook` and run it locally using your `conda` environment from exercise 1\n",
    "    + nothing to turn in here\n",
    "+ exercise 3\n",
    "    + you will fill in the missing elements from the template `jupyter notebook`\n",
    "    + once the code snippets have been added, export that notebook as an `html` document (\"File > Download As > HTML\")\n",
    "    + attached that html document to your submission email\n",
    "+ exercise 4\n",
    "    + after all of the `iam` setup, you will be left with a `role`. collect the `arn` of that role and send it in the body of your submission email\n",
    "+ exercise 5\n",
    "    + you will construct `aws cli` commands to do four different things\n",
    "    + these commands will be written to a `sh` file and attached to the submission email\n",
    "+ exercise 6\n",
    "    + you will collect `iam policy`s to grant a user or role the necessary permissions to perform the `aws cli` commands in exercise 5\n",
    "    + include these policy or permission names in the body of the submission email"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 1: preparing a `conda` environment for launching a `jupyter notebook`\n",
    "\n",
    "in some of the following exercises, we are going to be running a `jupyter notebook` server to edit a `jupyter notebook`. we need to create a `python` environment which is capable of doing just that.\n",
    "\n",
    "to do this, we must create a conda environment called with the following packages installed:\n",
    "\n",
    "+ `python` version 3.x\n",
    "+ `jupyter`\n",
    "+ `matplotlib`\n",
    "+ `numpy`\n",
    "+ `pandas`\n",
    "+ `plotly`\n",
    "+ `scikit-learn`\n",
    "+ `seaborn`\n",
    "\n",
    "note: as I said above, you're going to be using this `conda` environment and the `jupyter` package you install into it to create a notebook server, and you're going to connect to that notebook server with your web browser to open and edit notebook files. you need to choose whether you want to run that `jupyter notebook` service *locally* or *remotely*\n",
    "\n",
    "\n",
    "#### local `jupyter notebook` service\n",
    "\n",
    "*if* you have `conda` installed on your laptop or desktop, you have created the environment locally, and launched the `jupyter` notebook server locally, you will be able to access the notebooks being served up by that local service at the url http://localhost:888/\n",
    "\n",
    "\n",
    "#### remote (`ec2`) `jupyter notebook` service\n",
    "\n",
    "*if, however* you have decided to do all of the above on your `ec2` server, you will need to do one of the following:\n",
    "\n",
    "1. allow \"outsiders\" to connect to your notebook server\n",
    "    1. launch `jupyter notebook` with the `--ip` parameter set to `127.0.0.1`\n",
    "        + this causes your `jupyter` service to *listen* for requests from outside the `ec2` server as well as from within (the default behavior is within only)\n",
    "    2. open the port (default: 8888) to your ip or the world (via the `ec2` console)\n",
    "    3. access the service at http://your.ip.add.ress:8888/\n",
    "2. set up `ssh` tunnel port forwarding\n",
    "    1. this is a way of suggesting that requests on your *local* port 8888 be redirected over the ssh tunnel to your `ec2` server, where they are routed to the `ec2` port 8888.\n",
    "    2. from linux or mac, the command is: `ssh -N -f -L localhost:[PORT YOU WANT LOCALLY]:localhost:[PORT JUPYTER IS RUNNING ON REMOTELY] [USER@HOSTNAME or SSH_CONFIG_ALIAS]`\n",
    "    3. reasonable default values: `ssh -N -f -L localhost:8888:localhost:8888 [USER@HOSTNAME or SSH_CONFIG_ALIAS]`\n",
    "    3. it that worked, you can access the service at http://localhost:PORT_YOU_WANT_LOCALLY, so probably http://localhost:8888/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 2: download and launch a `jupyter` notebook\n",
    "\n",
    "download the [ds_pipeline_template](ds_pipeline_template.ipynb) notebook file to either your local machine (your laptop) or your `ec2` server.\n",
    "\n",
    "in your terminal, change into the directory into which that `ipynb` file was saved, and launch a `jupyter` notebook in or above the directory that holds that `ipynb` file.\n",
    "\n",
    "navigate to the notebook file in your web browser."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 3: fill in the `jupyter` notebook template\n",
    "\n",
    "the notebook you just downloaded and launched in a `jupyter` notebook server contains an outline of a simple data science project using the standard `python` data science stack. it includes `import` statements and notes of the general tasks to perform.\n",
    "\n",
    "it also includes several code cells which simply read\n",
    "\n",
    "```python\n",
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #\n",
    "```\n",
    "\n",
    "you should... you know... fill them in.\n",
    "\n",
    "after you have done that, **export this notebook to an html document and include it in your submission email**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 4: giving us read access to your `iam` users, group, and roles \n",
    "\n",
    "in this exercise we will set up an `iam` role to grant list access to your `iam` users, groups, and roles to *an entirely differen aws account*.\n",
    "\n",
    "This sort of cross-account permission wrangling is common when you have separate `aws` accounts for separate teams, department, or companies working on the same project, or a separate production, UAT, or development environment.\n",
    "\n",
    "because we have covered only `ec2` and `iam` for now, we'll use `iam` permissions. In the future we will share `s3` permissions in a nearly identical way.\n",
    "\n",
    "\n",
    "### 4.a: create a policy to allow listing\n",
    "\n",
    "use the `iam` dashboard to create a new policy with the following properties (as represented in the `aws` policy `json` structure):\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"Version\": \"2012-10-17\",\n",
    "  \"Statement\": [\n",
    "    {\n",
    "      \"Effect\": \"Allow\",\n",
    "      \"Action\": [\n",
    "        \"iam:ListGroups\",\n",
    "        \"iam:ListRoles\",\n",
    "        \"iam:ListUsers\"\n",
    "      ],\n",
    "      \"Resource\": [\n",
    "        \"arn:aws:iam::YOUR_ACCT_NUMBER_HERE:*\"\n",
    "      ]\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "what does the above policy say?\n",
    "\n",
    "\n",
    "### 4.b: create an `iam role` with the above policy\n",
    "\n",
    "create a `role` for me.\n",
    "\n",
    "my account number is `134461086921`. create a role of type \"Another AWS account\" and use this account number.\n",
    "\n",
    "call this role whatever you would like, and add to that role the permission policy you created in part `a`.\n",
    "\n",
    "\n",
    "### 4.c: collect the `arn` of the role\n",
    "\n",
    "after all of this, you should be able to open the role's summary page to see the role `arn`, and that `arn` should have a value like\n",
    "\n",
    "```\n",
    "arn:aws:iam::YOUR_ACCT_NUMBER_HERE:role/THE_ROLE_NAME\n",
    "```\n",
    "\n",
    "**include that `arn` in your email to us**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 5: `aws cli` help and information\n",
    "\n",
    "take the following four tasks:\n",
    "\n",
    "1. collect information about your `ec2` instances, and return that information as a `json` string\n",
    "2. collect information about your `ec2` elastic ip address. return answers as a tab delimited table\n",
    "3. change the rules for who can connect to your `ec2` server so that someone at ip address 1337.1337.1337.1337 (for a cidr, append the characters /32) can connect via `ssh`\n",
    "4. list your `iam` roles in a fancy table format\n",
    "\n",
    "each of the above requests can be done using the `aws cli`. \n",
    "\n",
    "using the help command\n",
    "\n",
    "```\n",
    "aws ec2 help\n",
    "\n",
    "# or\n",
    "\n",
    "aws iam help\n",
    "```\n",
    "\n",
    "or the [online documentation](http://docs.aws.amazon.com/cli/latest/reference/) ([`ec2`](http://docs.aws.amazon.com/cli/latest/reference/ec2/index.html) and [`iam`](http://docs.aws.amazon.com/cli/latest/reference/iam/index.html)), construct the `aws cli` statements that would perform those tasks. \n",
    "\n",
    "let's assume your region is `us-east-1` (this is NoVa, which we chose at the beginning of the class, and is the default), this means four commands that all start with\n",
    "\n",
    "```bash\n",
    "aws --region us-east-1 [FILL IN THIS PART]\n",
    "```\n",
    "\n",
    "**write these four commands to a file called `aws_commands.sh` and attach it to your email**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 6: `iam` policies needed for `aws cli` activity\n",
    "\n",
    "in order to execute the above commands, you must have certain policies or specific permissions. For each of the above, list *an `iam` policy or set of policies* which would be sufficient to allow the `cli` to execute that task.\n",
    "\n",
    "an example: if I wanted to list the contents of an `s3` bucket using the `aws s3` command, the `aws cli` would have to operate in an `iam role` which has `s3` read permissions. One policy which will grant that permission is the pre-packaged `AmazonS3FullAccess` policy. you can access [that `policy`](https://console.aws.amazon.com/iam/home?region=us-east-1#/policies/arn:aws:iam::aws:policy/AmazonS3FullAccess$serviceLevelSummary) via the `iam policies` page if you are logged in to the `aws` console.\n",
    "\n",
    "**include these four policies in the body of the email you send to us**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}