2017_09_28_answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Answers for Exercises due by EOD 2017.09.28"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 1: preparing a `conda` environment for launching a `jupyter notebook`\n",
    "\n",
    "in some of the following exercises, we are going to be running a `jupyter notebook` server to edit a `jupyter notebook`. we need to create a `python` environment which is capable of doing just that.\n",
    "\n",
    "to do this, we must create a conda environment called with the following packages installed:\n",
    "\n",
    "+ `python` version 3.x\n",
    "+ `jupyter`\n",
    "+ `matplotlib`\n",
    "+ `numpy`\n",
    "+ `pandas`\n",
    "+ `plotly`\n",
    "+ `scikit-learn`\n",
    "+ `seaborn`\n",
    "\n",
    "note: as I said above, you're going to be using this `conda` environment and the `jupyter` package you install into it to create a notebook server, and you're going to connect to that notebook server with your web browser to open and edit notebook files. you need to choose whether you want to run that `jupyter notebook` service *locally* or *remotely*\n",
    "\n",
    "\n",
    "#### local `jupyter notebook` service\n",
    "\n",
    "*if* you have `conda` installed on your laptop or desktop, you have created the environment locally, and launched the `jupyter` notebook server locally, you will be able to access the notebooks being served up by that local service at the url http://localhost:888/\n",
    "\n",
    "\n",
    "#### remote (`ec2`) `jupyter notebook` service\n",
    "\n",
    "*if, however* you have decided to do all of the above on your `ec2` server, you will need to do one of the following:\n",
    "\n",
    "1. allow \"outsiders\" to connect to your notebook server\n",
    "    1. launch `jupyter notebook` with the `--ip` parameter set to `127.0.0.1`\n",
    "        + this causes your `jupyter` service to *listen* for requests from outside the `ec2` server as well as from within (the default behavior is within only)\n",
    "    2. open the port (default: 8888) to your ip or the world (via the `ec2` console)\n",
    "    3. access the service at http://your.ip.add.ress:8888/\n",
    "2. set up `ssh` tunnel port forwarding\n",
    "    1. this is a way of suggesting that requests on your *local* port 8888 be redirected over the ssh tunnel to your `ec2` server, where they are routed to the `ec2` port 8888.\n",
    "    2. from linux or mac, the command is: `ssh -N -f -L localhost:[PORT YOU WANT LOCALLY]:localhost:[PORT JUPYTER IS RUNNING ON REMOTELY] [USER@HOSTNAME or SSH_CONFIG_ALIAS]`\n",
    "    3. reasonable default values: `ssh -N -f -L localhost:8888:localhost:8888 [USER@HOSTNAME or SSH_CONFIG_ALIAS]`\n",
    "    3. it that worked, you can access the service at http://localhost:PORT_YOU_WANT_LOCALLY, so probably http://localhost:8888/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 1\n",
    "\n",
    "the following commands will create this environment:\n",
    "\n",
    "1. `conda create -n ENVIRONMENT_NAME python=3`\n",
    "2. `source activate ENVIRONMENT_NAME`\n",
    "3. `conda install jupyter matplotlib numpy pandas plotly scikit-learn seaborn`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 2: download and launch a `jupyter` notebook\n",
    "\n",
    "download the [ds_pipeline_template](ds_pipeline_template.ipynb) notebook file to either your local machine (your laptop) or your `ec2` server.\n",
    "\n",
    "in your terminal, change into the directory into which that `ipynb` file was saved, and launch a `jupyter` notebook in or above the directory that holds that `ipynb` file.\n",
    "\n",
    "navigate to the notebook file in your web browser."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 2\n",
    "\n",
    "assume I have already gone through the web-console-based process of downloading the file to a directory on my machine `~/code/hw`. I can launch `jupyter` notebook via\n",
    "\n",
    "```bash\n",
    "# some env with jupyter installed, such as the one from q1\n",
    "source activate ENVIRONMENT_NAME\n",
    "\n",
    "cd ~/code/hw/\n",
    "\n",
    "jupyter notebook\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 3: fill in the `jupyter` notebook template\n",
    "\n",
    "the notebook you just downloaded and launched in a `jupyter` notebook server contains an outline of a simple data science project using the standard `python` data science stack. it includes `import` statements and notes of the general tasks to perform.\n",
    "\n",
    "it also includes several code cells which simply read\n",
    "\n",
    "```python\n",
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #\n",
    "```\n",
    "\n",
    "you should... you know... fill them in.\n",
    "\n",
    "after you have done that, **export this notebook to an html document and include it in your submission email**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 3\n",
    "\n",
    "see the neighboring `ds_pipeline_template_answers.ipynb` notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 4: giving us read access to your `iam` users, group, and roles \n",
    "\n",
    "in this exercise we will set up an `iam` role to grant list access to your `iam` users, groups, and roles to *an entirely differen aws account*.\n",
    "\n",
    "This sort of cross-account permission wrangling is common when you have separate `aws` accounts for separate teams, department, or companies working on the same project, or a separate production, UAT, or development environment.\n",
    "\n",
    "because we have covered only `ec2` and `iam` for now, we'll use `iam` permissions. In the future we will share `s3` permissions in a nearly identical way.\n",
    "\n",
    "\n",
    "### 4.a: create a policy to allow listing\n",
    "\n",
    "use the `iam` dashboard to create a new policy with the following properties (as represented in the `aws` policy `json` structure):\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"Version\": \"2012-10-17\",\n",
    "  \"Statement\": [\n",
    "    {\n",
    "      \"Effect\": \"Allow\",\n",
    "      \"Action\": [\n",
    "        \"iam:ListGroups\",\n",
    "        \"iam:ListRoles\",\n",
    "        \"iam:ListUsers\"\n",
    "      ],\n",
    "      \"Resource\": [\n",
    "        \"arn:aws:iam::YOUR_ACCT_NUMBER_HERE:*\"\n",
    "      ]\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "what does the above policy say?\n",
    "\n",
    "\n",
    "### 4.b: create an `iam role` with the above policy\n",
    "\n",
    "create a `role` for me.\n",
    "\n",
    "my account number is `134461086921`. create a role of type \"Another AWS account\" and use this account number.\n",
    "\n",
    "call this role whatever you would like, and add to that role the permission policy you created in part `a`.\n",
    "\n",
    "\n",
    "### 4.c: collect the `arn` of the role\n",
    "\n",
    "after all of this, you should be able to open the role's summary page to see the role `arn`, and that `arn` should have a value like\n",
    "\n",
    "```\n",
    "arn:aws:iam::YOUR_ACCT_NUMBER_HERE:role/THE_ROLE_NAME\n",
    "```\n",
    "\n",
    "**include that `arn` in your email to us**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 4\n",
    "\n",
    "an `aws`-specific walkthrough is [here](http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html). the instructions above should be sufficient -- just *make sure that the role `arn` you send is formatted as* `arn:aws:iam::YOUR_ACCT_NUMBER_HERE:role/THE_ROLE_NAME`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 5: `aws cli` help and information\n",
    "\n",
    "our goal here is simply to use the available help menus (online or the `help` commands on the command line) to identify which `aws ec2` or `iam` **subcommands** are the correct subcommands to accomplish a task.\n",
    "\n",
    "this is how it often goes in programming and software development -- you know what you want to *do* but you don't know the functions that will allow you to do it, and perhaps you don't know the proper words or semantics. the only way forward is to gain skills and confidence in navigating documentation and stack overflow.\n",
    "\n",
    "take the following four tasks:\n",
    "\n",
    "1. collect information about your `ec2` instances, and return that information as a `json` string\n",
    "2. collect information about your `ec2` elastic ip address. return answers as a tab delimited table\n",
    "3. change the rules for who can connect to your `ec2` server so that someone at ip address 1337.1337.1337.1337 (for a cidr, append the characters /32) can connect via `ssh`\n",
    "    1. note: it's not good practice to just add connections to random ip addresses. this one is fake, but don't try doing this with real ip addresses unless you know you trust that IP address!\n",
    "4. list your `iam` roles in a fancy table format\n",
    "\n",
    "in a vacuum, it's possible some of them may not be doable with the `cli`. that's not the case here -- each of the above requests can be done using the `aws cli`. \n",
    "\n",
    "using the help command\n",
    "\n",
    "```\n",
    "aws ec2 help\n",
    "\n",
    "# or\n",
    "\n",
    "aws iam help\n",
    "```\n",
    "\n",
    "or the [online documentation](http://docs.aws.amazon.com/cli/latest/reference/) ([`ec2`](http://docs.aws.amazon.com/cli/latest/reference/ec2/index.html) and [`iam`](http://docs.aws.amazon.com/cli/latest/reference/iam/index.html)), figure out which sub-command of the `aws cli` statements would perform those tasks. \n",
    "\n",
    "let's also do a quick mockup of how we would invoke those commands. you don't need to test that they work -- the `1337` ip address doesn't even exist so not all of them even *could* work -- you just need to put down the right service, subcommand, and a reasonable stab at the appropriate flags. \n",
    "\n",
    "let's assume your region is part of your configuration file so we don't need to worry about the `--region` flag. this means four commands that all look like the below, filling in the 3 bracketed pieces\n",
    "\n",
    "```bash\n",
    "aws [ec2 or iam] [subcommand] [flags]\n",
    "```\n",
    "\n",
    "**write these four commands to a file called `aws_commands.sh` and attach it to your email**\n",
    "\n",
    "make sure that shell script is in *plain text* -- the simple editors on mac, for example, will still sometimes save files in \"rich text format\" (rtf). you want *plain text*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 5\n",
    "\n",
    "the contents of `aws_commands.sh` are basically as follows:\n",
    "\n",
    "```bash\n",
    "aws ec2 describe-instances --output json\n",
    "aws ec2 describe-addresses --output text\n",
    "aws ec2 authorize-security-group-ingress --group-name SOME_GROUP_NAME --protocol tcp --port 22 --cidr 1337.1337.1337.1337/32\n",
    "aws iam list-roles --output table\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 6: `iam` policies needed for `aws cli` activity\n",
    "\n",
    "in order to execute the above commands, you must have certain policies or specific permissions. For each of the above, list *an `iam` policy or set of policies* which would be sufficient to allow the `cli` to execute that task.\n",
    "\n",
    "an example: if I wanted to list the contents of an `s3` bucket using the `aws s3` command, the `aws cli` would have to operate in an `iam role` which has `s3` read permissions. One policy which will grant that permission is the pre-packaged `AmazonS3FullAccess` policy. you can access [that `policy`](https://console.aws.amazon.com/iam/home?region=us-east-1#/policies/arn:aws:iam::aws:policy/AmazonS3FullAccess$serviceLevelSummary) via the `iam policies` page if you are logged in to the `aws` console.\n",
    "\n",
    "**include these four policies in the body of the email you send to us**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 6\n",
    "\n",
    "the below are a *strict super-set* of the policies needed, so they are sufficient while also granting extra permissions:\n",
    "\n",
    "1. AmazonEC2ReadOnlyAccess\n",
    "2. AmazonEC2ReadOnlyAccess\n",
    "3. AmazonEC2FullAccess\n",
    "4. IAMReadOnlyAccess\n",
    "\n",
    "for any of the above, if you replace \"ReadOnly\" with \"Full\" you will still have a correct answer."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}