2017_09_21_answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Answers for  Exercises due by EOD 2017.09.21"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 1: creating a useful bash script "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "both of the following will be graded equivalently, so choose based on your familiarity with linux or desire for a challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### exercise 1.A: creating a \"useful\" bash script (linux beginners)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we're going to write a bash script that will download current weather information at DCA (Reagan National Airport). we'll do this in stages:\n",
    "\n",
    "1. create a directory to hold our data\n",
    "2. download the current weather and delay status for DCA (Reagan Washington National airport)\n",
    "3. print a status message indicating whether or not we were successful to a log file\n",
    "\n",
    "to create this script, we will move one step at a time; the final script will just be all of the commands put together into one script.\n",
    "\n",
    "along the way, we will want to make sure that all of the commands we execute are *repeatable*: we should be able to run this script a *first* time (and it will do any setup we may need that first time), and then *again* (so it will be okay that this setup is already done, and not fail)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### create a directory\n",
    "\n",
    "write a command to make a directory `~/data/weather/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### make sure your \"create a directory\" command is *repeatable*\n",
    "\n",
    "try running the command you just wrote *again* -- what happens?\n",
    "\n",
    "in order to make this command repeatable, you will need to specify some flags to this command such that it will:\n",
    "\n",
    "1. create both `~/data` and `~/data/weather` if they don't exist\n",
    "    1. this is necessary the *first* time the script runs\n",
    "2. to not to throw an error if that directory already exists\n",
    "    1. this is necesary the *other* times the script runs\n",
    "\n",
    "*hint: if you know how to make a directory, try `man [COMMAND]` to see how to make sure no error is thrown if a directory already exists*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### download the current weather and delay status for DCA\n",
    "\n",
    "the FAA (Federal Aviation Administration) has created [a RESTful `xml` and `json` formatted endpoint](http://services.faa.gov/docs/services/airport/) for basic information about airports -- thanks, FFA!\n",
    "\n",
    "the endpoint of that API is http://services.faa.gov/docs/services/airport/airportCode, and it expects one of two values for the \"format\" method:\n",
    "\n",
    "+ `application/xml`\n",
    "+ `application/json`\n",
    "\n",
    "let's open DCA's `json` formatted output. head to http://services.faa.gov/airport/status/DCA?format=application/json in your browser.\n",
    "\n",
    "using a command line tool, download the json results of that API call to a file named \n",
    "\n",
    "`~/data/weather/dca.weather.json`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### get the status code from the download request\n",
    "\n",
    "you just successfully wrote a linux command that can download the DCA json information from the API and wrote it to a file. any time that command runs, it will either be *successful* or *unsuccessful*.\n",
    "\n",
    "after you run that command, get the **exit status** of that command and print it to the terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### print a status message to a log file\n",
    "\n",
    "let's get the following for a status message:\n",
    "\n",
    "1. the current time\n",
    "2. the result of the previous command (the download command) -- just as an error code, nothing more complicated than that\n",
    "\n",
    "the end result should be a line formatted like\n",
    "\n",
    "```\n",
    "YYYY-mm-dd HH:MM:SS    gu511_download_A.sh    command status code was: [status code here]\n",
    "```\n",
    "\n",
    "write a command to save the current time to a variable called `$NOW`.\n",
    "\n",
    "once you can construct such a line, *append* that line to a log file at `~/data/weather/download.log`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### combine all of the above into a bash script\n",
    "\n",
    "create a file called `gu511_download_A.sh` by filling in the following template:\n",
    "\n",
    "```bash\n",
    "#!/usr/bin/bash\n",
    "# when this script is run, the line above tells the\n",
    "# command line what program (binary) to use to\n",
    "# execute the commands\n",
    "\n",
    "# the following line(s) creates the directory \n",
    "# ~/data/weather if needed\n",
    "FILL THIS IN\n",
    "\n",
    "# the following line(s) downloads the current weather \n",
    "# and delay status for DCA into ~/data/weather\n",
    "FILL THIS IN\n",
    "\n",
    "# the following line(s) write a log message to file \n",
    "# indicating status code of previous line \n",
    "FILL THIS IN\n",
    "\n",
    "# exit with the most recent error code -- you can\n",
    "# leave this line alone\n",
    "exit $?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### submit this file: see exercise 4 below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### answer 1.A\n",
    "\n",
    "```bash\n",
    "#!/usr/bin/bash\n",
    "# when this script is run, the line above tells the\n",
    "# command line what program (binary) to use to\n",
    "# execute the commands\n",
    "\n",
    "# the following line(s) creates the directory \n",
    "# ~/data/weather if needed\n",
    "mkdir -p ~/data/weather\n",
    "\n",
    "# the following line(s) downloads the current weather \n",
    "# and delay status for DCA into ~/data/weather\n",
    "curl --silent -o ~/data/weather/dca.weather.json http://services.faa.gov/airport/status/DCA?format=application/json\n",
    "\n",
    "# the following line(s) write a log message to file \n",
    "# indicating status code of previous line \n",
    "STATUS_CODE=$?\n",
    "echo \"$(date +'%Y-%m-%d %H:%M:%S')    gu511_download_A.sh    $STATUS_CODE\"\n",
    "\n",
    "# exit with the most recent error code -- you can\n",
    "# leave this line alone\n",
    "exit $?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### exercise 1.B: create a *useful* bash script (advanced linux users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we're going to write a bash script that will download an arbitrary number of urls from a text file in a highly parallel way. we'll write this script in stages:\n",
    "\n",
    "1. create a directory to hold our downloaded data\n",
    "2. download a list of urls from a text file\n",
    "\n",
    "to create this script, we will move one step at a time; the final script will just be all of the commands put together into one script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### create a test csv\n",
    "\n",
    "execute the following commands to create a list of test urls for downloading:\n",
    "\n",
    "```bash\n",
    "echo www.google.com >> /tmp/test.urls\n",
    "echo www.georgetown.edu >> /tmp/test.urls\n",
    "echo www.elderresearch.com >> /tmp/test.urls\n",
    "echo www.twitter.com >> /tmp/test.urls\n",
    "echo www.facebook.com >> /tmp/test.urls\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### create a directory\n",
    "\n",
    "write a command to make a directory `~/data/weather/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### make sure your \"create a directory\" command is *repeatable*\n",
    "\n",
    "try running the command you just wrote *again* -- what happens?\n",
    "\n",
    "in order to make this command repeatable, you will need to specify some flags to this command such that it will:\n",
    "\n",
    "1. create both `~/data` and `~/data/weather` if they don't exist\n",
    "    1. this is necessary the *first* time the script runs\n",
    "2. to not to throw an error if that directory already exists\n",
    "    1. this is necesary the *other* times the script runs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### write a command to print the contents of `test.csv` of urls to `stdout`\n",
    "\n",
    "print the contents of `test.csv` to the terminal (for piping to a later function)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### use `xargs` to pipe the contents of `test.urls` to the `echo` function\n",
    "\n",
    "soon we will write a function which will take a *single* url and download it. to pass many urls to this script and to create several forks (separate processes which will work in parallel) we will use the `xargs` command.\n",
    "\n",
    "let's get some practice with the `xargs` command before trying to use it for our download function. in particular, let's look at the following flags:\n",
    "\n",
    "1. `-P` or `--max-procs`: specify the maximum number of separate processes we should start (default is 1, 0 is interpreted as \"maximum number possible\")\n",
    "2. `-n`: in conjunction with `-P`, the number of items passed to each process\n",
    "3. `-I`: specify which sequence of characters in the command to follow should be replaced with the item passed in by `xargs`. a somewhat common option is `{}` because it is unlikely to be meaningful in any command that follows. that must be escaped, though -- see below\n",
    "\n",
    "as an example, check out the results of the following:\n",
    "\n",
    "```bash\n",
    "cat /tmp/test.urls | xargs -P 100 -n 3 -I{} echo url is \\{\\}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `curl` one of those urls\n",
    "\n",
    "take one of those urls -- say, www.google.com -- and download it to a file. do the following:\n",
    "\n",
    "1. run it in \"silent\" mode\n",
    "2. cap the maximum time the whole download operation should take at 10 seconds\n",
    "3. write the contents of that download to a file in `~/data/downloads` with a the same name as the final portion (the `basename` of that url)\n",
    "\n",
    "*hint*: suppose we have the url is a bash variable `$URL`. we could write\n",
    "\n",
    "```bash\n",
    "curl [silent flag and maximum download time flag] $URL > ~/data/downloads/$(basename $URL)\n",
    "```\n",
    "\n",
    "the `basename` piece is necessary for urls which are more complicated than just `www.xxxxxxxx.com`, such as `www.xxxxxxxx.com/a/longer/path/with?stuff=x&other_stuff=y`\n",
    "\n",
    "verify that the downloaded contents for one test url match the source on the corresponding webpage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### export that `curl` statement as a function\n",
    "\n",
    "you can create a bash function using the syntax\n",
    "\n",
    "```bash\n",
    "function my_function_name {\n",
    "    # do bash stuff\n",
    "}\n",
    "```\n",
    "\n",
    "arguments are passed to this function as bash variables `$1`, `$2`, and so on, such that if you write\n",
    "\n",
    "```bash\n",
    "my_function_name arg1 arg2 arg3 arg4\n",
    "```\n",
    "\n",
    "these will be \"available\" within the body of the function as\n",
    "\n",
    "| variable name | value |\n",
    "|---------------|-------|\n",
    "| `$1`          | arg1  |\n",
    "| `$2`          | arg2  |\n",
    "| `$3`          | arg3  |\n",
    "| `$4`          | arg4  |\n",
    "\n",
    "for example, if we wanted to turn our echo command up above into a super l33t re-usable function, we could write\n",
    "\n",
    "```bash\n",
    "function l33t_url_echo {\n",
    "    echo \"the url is $1\"\n",
    "}\n",
    "\n",
    "# test it out\n",
    "l33t_url_echo www.google.com\n",
    "```\n",
    "\n",
    "we could also make this available in other bash shells be `export`-ing it:\n",
    "\n",
    "```bash\n",
    "export -f l33t_url_echo\n",
    "```\n",
    "\n",
    "so, let's talk about **what you should actually do**:\n",
    "\n",
    "1. convert your `curl` statement from before into a bash function that will take a url as a parameter\n",
    "2. export it for use in other bash sessions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### use that function with `xargs` on your test urls\n",
    "\n",
    "for each of the urls filtered by `xargs` we want to run the newly-minted `bash` function with that url as the argument.\n",
    "\n",
    "for example, if we wanted to use our `l33t_url_echo` function from above, we could write:\n",
    "\n",
    "```bash\n",
    "# ...it pays to read ahead...\n",
    "cat /tmp/test.urls | xargs -P 100 -n 3 -I{} bash -c l33t_url_echo\\ \\{\\}\n",
    "```\n",
    "\n",
    "in the above, the actual *command* we are executing with `xargs` is the `bash` command, which\n",
    "\n",
    "1. starts a new `bash` shell\n",
    "2. executes the *command* following flag `-c` (that's what the `-c` flag *is*)\n",
    "3. replaces the occurrence of `\\{\\}` with whatever url is available\n",
    "4. special characters such as spaces and braces need to be escaped to be passed in using the `-c` command\n",
    "\n",
    "write your own version of the command above, replacing `l33t_url_echo` with the function you created previously.\n",
    "\n",
    "delete all of the items in `~/data/downloads` to start from scratch, and run the whole `cat + xargs + your_function` line. verify it downloads each test url."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### replace `/tmp/test.urls` with a variable path name\n",
    "\n",
    "create a variable `$URL_FILE` with a value of `/tmp/test.urls`, and invoke the previous `cat` + `xargs` + `your_function` line using the variable name instead of the hard-coded path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### understand command line arguments\n",
    "\n",
    "the way that bash handles command line arguments to a shell script is identical to the way functions receive them -- the first word (first in a space-separated list) is stored to a variable `$1`, the second to `$2`, and so on. \n",
    "\n",
    "a common convention for command line arguments is to supply a default value, and this can be done with a bash variable resolution construct:\n",
    "\n",
    "```bash\n",
    "MY_VAR=${TRY_THIS_FIRST:-USE_THIS_IF_NOTHING_FOUND}\n",
    "```\n",
    "\n",
    "if `$TRY_THIS_FIRST` exists, bash resolves that expression to the value of `$TRY_THIS_FIRST` and uses it to set the value of `$MY_VAR`. if it does not, it will then try evaluating the *exact string* following the `:-` characters.\n",
    "\n",
    "In the example above,\n",
    "\n",
    "+ if `$TRY_THIS_FIRST` is set to some value, `MY_VAR` will be set to that value\n",
    "+ if `$TRY_THIS_FIRST` is *not* set to some value, `MY_VAR` will be set to the `USE_THIS_IF_NOTHING_FOUND`\n",
    "    + if `USE_THIS_IF_NOTHING_FOUND` is *itself* a variable expression (e.g. `$USER`), it will be resolved and then assigned to the variable `MY_VAR`\n",
    "    \n",
    "a common use of this is setting default command line argument values. for example, suppose I create a file `my_script.sh` that contains the following:\n",
    "\n",
    "```bash\n",
    "!/usr/bin/bash\n",
    "\n",
    "FIRST_ARGUMENT=${1:-defaultval}\n",
    "\n",
    "echo $FIRST_ARGUMENT\n",
    "```\n",
    "\n",
    "if I call\n",
    "\n",
    "```bash\n",
    "bash my_script.sh\n",
    "```\n",
    "\n",
    "there is no argument passed and therefore `$1` will not be set. This will result in `FIRST_ARGUMENT` being set to the default value `defaultval`, and the script will print `defaultval` to the terminal.\n",
    "\n",
    "if, on the other hand, I call\n",
    "\n",
    "```bash\n",
    "bash my_script.sh \"print me\"\n",
    "```\n",
    "\n",
    "bash will create a variable `$1` with a value `print me`, and the script will end up printing `print me` to the terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### combine all of the above into a bash script\n",
    "\n",
    "create a file called `gu511_download_B.sh` to the following format:\n",
    "\n",
    "```bash\n",
    "#!/usr/bin/bash\n",
    "# when this script is run, the line above tells the\n",
    "# command line what program (binary) to use to\n",
    "# execute the commands\n",
    "\n",
    "# allow the executing user to pass their own list of urls,\n",
    "# but keep /tmp/test.urls as a default\n",
    "URL_FILE=${1:-/tmp/test.urls}\n",
    "\n",
    "# the following line(s) creates the directory \n",
    "# ~/data/downloads if needed\n",
    "FILL THIS IN\n",
    "\n",
    "# the following line(s) define our single-url curl\n",
    "# download function\n",
    "FILL THIS IN\n",
    "\n",
    "# the following line(s) export that function for use\n",
    "# in other bash session\n",
    "FILL THIS IN\n",
    "\n",
    "# the following line is the \"cat + xargs + your_function\"\n",
    "# line from the previous step\n",
    "FILL THIS IN\n",
    "\n",
    "# exit with the most recent error code -- you can\n",
    "# leave this line alone\n",
    "exit $?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### postscript\n",
    "\n",
    "*if everything went according to plan, this script should be among the fastest download programs I've ever come across (no exageration there). it was useful enough that I put it and some variants on a github repo I own.*\n",
    "\n",
    "*...it **really** pays to read ahead...*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### submit this file: see exercise 4 below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### answer 1.B\n",
    "\n",
    "I have a few versions in [my github repo](https://github.com/RZachLamberty/zshell) (as mentioned above), but this is basically what we want to do\n",
    "\n",
    "```bash\n",
    "#!/usr/bin/bash\n",
    "# when this script is run, the line above tells the\n",
    "# command line what program (binary) to use to\n",
    "# execute the commands\n",
    "\n",
    "# allow the executing user to pass their own list of urls,\n",
    "# but keep /tmp/test.urls as a default\n",
    "URL_FILE=${1:-/tmp/test.urls}\n",
    "\n",
    "# the following line(s) creates the directory \n",
    "# ~/data/downloads if needed\n",
    "mkdir -p ~/data/downloads\n",
    "\n",
    "# the following line(s) define our single-url curl\n",
    "# download function\n",
    "function l33t_url_echo {\n",
    "    curl -s \"$1\" > ~/data/downloads/$(basename $1)\n",
    "}\n",
    "\n",
    "# the following line(s) export that function for use\n",
    "# in other bash session\n",
    "export -f l33t_url_echo\n",
    "\n",
    "# the following line is the \"cat + xargs + your_function\"\n",
    "# line from the previous step\n",
    "cat /tmp/test.urls | xargs -P 100 -n 3 -I{} bash -c l33t_url_echo\\ \\{\\}\n",
    "\n",
    "# exit with the most recent error code -- you can\n",
    "# leave this line alone\n",
    "exit $?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 2: installing `miniconda` on your `ec2` server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this will be a straightforward list of steps to execute from the terminal of your `ec2` server in order to install the `anaconda python` distribution.\n",
    "\n",
    "1. on your laptop\n",
    "    1. in your browser, go to [the `miniconda` download page](https://conda.io/miniconda.html)\n",
    "    2. find the `python 3.6` installer for `64-bit linux`\n",
    "    3. *copy the download link address*, don't just click\n",
    "        1. at the time of writing, this link was: `https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh`\n",
    "2. on your `ec2` server\n",
    "    1. download this file: `wget [THE URL FROM ABOVE]`. this will create a file (say) `Miniconda3-latest-Linux-x86_64.sh` in your current working directory.\n",
    "    2. execute that bash script: `bash Miniconda3-latest-Linux-x86_64.sh`\n",
    "        1. read the license. scroll through.... yes... yes... okay... fine... sure... yes... okay... type `yes` to accept\n",
    "        2. the default installation directory is fine but change it if you'd like\n",
    "        3. I will recommend you *do* change your `PATH` -- the default is `no`, so you have to actively type `yes` at this prompt.\n",
    "    3. if you experience an error or abort, you may have created the install directory. if you see an error: `ERROR: File or directory already exists: /home/ubuntu/miniconda3`, you can and should `rm` that directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 2\n",
    "\n",
    "the instructions above should be sufficient -- reach out if you have any issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 3: creating a `conda` environment and an `environment.yml` file\n",
    "\n",
    "first, read [the documentation on creating and managing conda environments](https://conda.io/docs/user-guide/tasks/manage-environments.html).\n",
    "\n",
    "once you've done that, create a conda environment called `gu511` with `python` version 3.5 and install into *that environment* (not your root environment) the following packages:\n",
    "\n",
    "+ `jupyter`\n",
    "+ `pandas`\n",
    "+ `plotly`\n",
    "+ `scikit-learn`\n",
    "\n",
    "use the [environment sharing `export` command](https://conda.io/docs/user-guide/tasks/manage-environments.html#sharing-an-environment) to create an `environment.yml` file.\n",
    "\n",
    "view that file with `less` and figure out what it is saying about your `conda` environment, and how some one might use that file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### submit this file: see exercise 4 below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 3\n",
    "\n",
    "the following sequence of commands will create the environment, install the desired packages, and create the `environment.yml` file:\n",
    "\n",
    "1. `conda create -n gu511 python=3.5`\n",
    "2. `source activate gu511`\n",
    "3. `conda install jupyter pandas plotly scikit-learn`\n",
    "4. `conda env export > environment.yml`\n",
    "\n",
    "\n",
    "this is the contents of the `environment.yml` file for the environment created *on a linux machine* (there may be slight differences if you create yours on a mac, fwiw).\n",
    "\n",
    "```yaml\n",
    "name: gu511\n",
    "channels:\n",
    "- defaults\n",
    "dependencies:\n",
    "- asn1crypto=0.22.0=py35h0d675fe_1\n",
    "- bleach=2.0.0=py35h055c768_0\n",
    "- ca-certificates=2017.08.26=h1d4fec5_0\n",
    "- certifi=2017.7.27.1=py35h19f42a1_0\n",
    "- cffi=1.10.0=py35h796c292_1\n",
    "- chardet=3.0.4=py35hb6e9ddf_1\n",
    "- cryptography=2.0.3=py35hef72dfd_1\n",
    "- dbus=1.10.22=h3b5a359_0\n",
    "- decorator=4.1.2=py35h3a268aa_0\n",
    "- entrypoints=0.2.3=py35h48174a2_2\n",
    "- expat=2.2.4=hc00ebd1_1\n",
    "- fontconfig=2.12.4=h88586e7_1\n",
    "- freetype=2.8=h52ed37b_0\n",
    "- glib=2.53.6=hc861d11_1\n",
    "- gmp=6.1.2=hb3b607b_0\n",
    "- gst-plugins-base=1.12.2=he3457e5_0\n",
    "- gstreamer=1.12.2=h4f93127_0\n",
    "- html5lib=0.999999999=py35h0543385_0\n",
    "- icu=58.2=h211956c_0\n",
    "- idna=2.6=py35h8605a33_1\n",
    "- intel-openmp=2018.0.0=h15fc484_7\n",
    "- ipykernel=4.6.1=py35h29d130c_0\n",
    "- ipython=6.1.0=py35h1b71439_1\n",
    "- ipython_genutils=0.2.0=py35hc9e07d0_0\n",
    "- ipywidgets=7.0.0=py35h8ebd919_0\n",
    "- jedi=0.10.2=py35hc33c70f_0\n",
    "- jinja2=2.9.6=py35h90b8645_1\n",
    "- jpeg=9b=habf39ab_1\n",
    "- jsonschema=2.6.0=py35h4395190_0\n",
    "- jupyter=1.0.0=py35hd38625c_0\n",
    "- jupyter_client=5.1.0=py35h2bff583_0\n",
    "- jupyter_console=5.2.0=py35h4044a63_1\n",
    "- jupyter_core=4.3.0=py35he2f7985_0\n",
    "- libedit=3.1=heed3624_0\n",
    "- libffi=3.2.1=h4deb6c0_3\n",
    "- libgcc-ng=7.2.0=h7cc24e2_2\n",
    "- libgfortran-ng=7.2.0=h9f7466a_2\n",
    "- libpng=1.6.32=hda9c8bc_2\n",
    "- libsodium=1.0.13=h31c71d8_2\n",
    "- libstdcxx-ng=7.2.0=h7a57d05_2\n",
    "- libxcb=1.12=h84ff03f_3\n",
    "- libxml2=2.9.4=h6b072ca_5\n",
    "- markupsafe=1.0=py35h4f4fcf6_1\n",
    "- mistune=0.7.4=py35hfd0f961_0\n",
    "- mkl=2018.0.0=hb491cac_4\n",
    "- nbconvert=5.3.1=py35hc5194e3_0\n",
    "- nbformat=4.4.0=py35h12e6e07_0\n",
    "- ncurses=6.0=h06874d7_1\n",
    "- notebook=5.0.0=py35h65c930e_2\n",
    "- numpy=1.13.3=py35hd829ed6_0\n",
    "- openssl=1.0.2l=h077ae2c_5\n",
    "- pandas=0.20.3=py35h85c2c75_2\n",
    "- pandoc=1.19.2.1=hea2e7c5_1\n",
    "- pandocfilters=1.4.2=py35h1565a15_1\n",
    "- pcre=8.41=hc71a17e_0\n",
    "- pexpect=4.2.1=py35h8b56cb4_0\n",
    "- pickleshare=0.7.4=py35hd57304d_0\n",
    "- pip=9.0.1=py35haa8ec2a_3\n",
    "- plotly=2.1.0=py35hac5c16f_0\n",
    "- prompt_toolkit=1.0.15=py35hc09de7a_0\n",
    "- ptyprocess=0.5.2=py35h38ce0a3_0\n",
    "- pycparser=2.18=py35h61b3040_1\n",
    "- pygments=2.2.0=py35h0f41973_0\n",
    "- pyopenssl=17.2.0=py35h1d2a76c_0\n",
    "- pyqt=5.6.0=py35h0e41ada_5\n",
    "- pysocks=1.6.7=py35h6aefbb0_1\n",
    "- python=3.5.4=he2c66cf_20\n",
    "- python-dateutil=2.6.1=py35h90d5b31_1\n",
    "- pytz=2017.2=py35h9225bff_1\n",
    "- pyzmq=16.0.2=py35h4be1f71_2\n",
    "- qt=5.6.2=h974d657_12\n",
    "- qtconsole=4.3.1=py35h4626a06_0\n",
    "- readline=7.0=hac23ff0_3\n",
    "- requests=2.18.4=py35hb9e6ad1_1\n",
    "- scikit-learn=0.19.0=py35h25e8076_2\n",
    "- scipy=0.19.1=py35ha8f041b_3\n",
    "- setuptools=36.5.0=py35ha8c1747_0\n",
    "- simplegeneric=0.8.1=py35h2ec4104_0\n",
    "- sip=4.18.1=py35h9eaea60_2\n",
    "- six=1.10.0=py35h5312c1b_1\n",
    "- sqlite=3.20.1=h6d8b0f3_1\n",
    "- terminado=0.6=py35hce234ed_0\n",
    "- testpath=0.3.1=py35had42eaf_0\n",
    "- tk=8.6.7=h5979e9b_1\n",
    "- tornado=4.5.2=py35hf879e1d_0\n",
    "- traitlets=4.3.2=py35ha522a97_0\n",
    "- urllib3=1.22=py35h2ab6e29_0\n",
    "- wcwidth=0.1.7=py35hcd08066_0\n",
    "- webencodings=0.5.1=py35hb6cf162_1\n",
    "- wheel=0.29.0=py35h601ca99_1\n",
    "- widgetsnbextension=3.0.2=py35h0be620c_1\n",
    "- xz=5.2.3=h2bcbf08_1\n",
    "- zeromq=4.2.2=hb0b69da_1\n",
    "- zlib=1.2.11=hfbfcf68_1\n",
    "- pip:\n",
    "  - ipython-genutils==0.2.0\n",
    "  - jupyter-client==5.1.0\n",
    "  - jupyter-console==5.2.0\n",
    "  - jupyter-core==4.3.0\n",
    "  - prompt-toolkit==1.0.15\n",
    "prefix: /home/ubuntu/miniconda3/envs/gu511\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## exercise 4: submitting your homework\n",
    "\n",
    "### tangent about how your `ssh` access was set up\n",
    "\n",
    "in last week's exercises you created a public key and sent it to me along with a desired user name and an ip address.\n",
    "\n",
    "after receiving them, I used the following script to create your users and configure `ssh` access:\n",
    "\n",
    "```bash\n",
    "#!/bin/bash\n",
    "\n",
    "# command line args\n",
    "USERNAME=${1}\n",
    "HOME=/home/$USERNAME\n",
    "IP=${2}\n",
    "PUBKEY=${3}\n",
    "\n",
    "# create user and set up home / .ssh director\n",
    "adduser --disabled-password $USERNAME\n",
    "mkdir -p $HOME/.ssh\n",
    "chown $USERNAME:$USERNAME $HOME/.ssh\n",
    "\n",
    "# add public key to authorized_keys\n",
    "echo $PUBKEY >> $HOME/.ssh/authorized_keys\n",
    "chown -R $USERNAME:$USERNAME $HOME/.ssh/\n",
    "chmod 700 $HOME/.ssh\n",
    "chmod 600 $HOME/.ssh/authorized_keys\n",
    "\n",
    "# use awscli to update ec2 port settings\n",
    "aws --region us-east-1 ec2 authorize-security-group-ingress \\\n",
    "    --group-name ssh_for_hw \\\n",
    "    --protocol tcp \\\n",
    "    --port 22 \\\n",
    "    --cidr $IP/32\n",
    "```\n",
    "\n",
    "I then sent you the information you need to sign in:\n",
    "\n",
    "1. the user name you requested and received\n",
    "2. the server's ip address\n",
    "\n",
    "you should then be able to log in to my `ec2` server with the command\n",
    "\n",
    "```bash\n",
    "ssh -i /path/to/your/private/key [YOUR USER NAME HERE]@[MY EC2 IP ADDRESS HERE]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### actually doing exercise 4\n",
    "\n",
    "the point of this exercise is to use `scp` (the SSH copy command) or some secure copy application (e.g. WinSCP or Filezilla) to copy your bash script file to my `ec2` server.\n",
    "\n",
    "you should copy it into your home directory (`~`, `/home/[YOUR USER NAME HERE]`) and keep the file name as `gu511_download_A.sh` or `gu511_download_B.sh`, depending on whether you completed `1.A` or `1.B` above.\n",
    "\n",
    "if you are using `scp`, the general structure of the command is\n",
    "\n",
    "```bash\n",
    "# copying a *local* file to a *remote* machine\n",
    "scp -i /path/to/your/private/key [local files to copy] [user name]@[host name or ip]:[path on remote machine]\n",
    "```\n",
    "\n",
    "to go in the other direction (*i.e.* copy remote files to your local machine), just flip the order between the `[local files to copy]` element and the `[user name]@[host name or ip]:[path on remote machine]` element.\n",
    "\n",
    "so for this particular copy operation:\n",
    "\n",
    "```bash\n",
    "scp -i /path/to/your/private/key /path/to/your/gu511_download_A.sh [your user name here]@[my aws ec2 ip]:~/gu511_download_A.sh\n",
    "\n",
    "# or\n",
    "\n",
    "scp -i /path/to/your/private/key /path/to/your/gu511_download_B.sh [your user name here]@[my aws ec2 ip]:~/gu511_download_B.sh\n",
    "```\n",
    "\n",
    "and then\n",
    "\n",
    "```bash\n",
    "scp -i /path/to/your/private/key /path/to/your/environment.yaml [your user name here]@[my aws ec2 ip]:~/\n",
    "```\n",
    "\n",
    "the final evaluation will be me running your script and creating `conda` environments using your `environment.yml` file and verifying that both behave as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## answer 4\n",
    "\n",
    "simply following the steps above is sufficient"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}