2017_10_12.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises due by EOD 2017.10.12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## goal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in this assignment, we will learn about additional features of the `s3` service, will work on `ec2` and `s3` integration, and will practice using and building `xpath` and `css selector` expressions. we will do the follwoing:\n",
    "\n",
    "1. create a public webpage using `s3`\n",
    "2. develop a simple `python` script to print messages to `s3`\n",
    "3. schedule a `cron` job of that `python` script\n",
    "4. practice `xpath` and `css selector` expressions on a simple `html` document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## method of delivery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as mentioned in our first lecture, the method of delivery may change from assignment to assignment. we will include this section in every assignment to provide an overview of how we expect homework results to be submitted, and to provide background notes or explanations for \"new\" delivery concepts or methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this week you will be submitting the results of your homework via email and upload to the `s3` bucket you created in last week's assignment.\n",
    "\n",
    "summary:\n",
    "\n",
    "| exercise | deliverable                               | method of delivery     |\n",
    "|----------|-------------------------------------------|------------------------|\n",
    "| 1        | one url                                   | email us               |\n",
    "| 2        | a bash script (`what_i_ran.sh`)           | upload to s3 hw bucket |\n",
    "| 2        | one url                                   | email us               |\n",
    "| 3        | a plain text file (`alarm_clock.crontab`) | upload to s3 hw bucket |\n",
    "| 4        | a csv file (`xpath_and_css.csv`)          | upload to s3 hw bucket |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exercise 1: create a static webpage\n",
    "\n",
    "let's make a webpage!\n",
    "\n",
    "\n",
    "### get an `html` file\n",
    "\n",
    "we're going to need some `html` (hypertext markup language) for our webpage. If you have a page you really want to show off to the world, feel free to use it -- otherwise, feel free to use [our example](https://s3.amazonaws.com/shared.rzl.gu511.com/index.html). edit it. go wild.\n",
    "\n",
    "\n",
    "### create a static webpage bucket\n",
    "\n",
    "you may remember, but when we were creating buckets in class we mentioned that it was possible to configure a bucket such that it could be used for \"static website hosting\". create a new bucket (name it whatever you want), and after creating it, configure it to host a static website.\n",
    "\n",
    "upload the `html` file from the previous step, and point to it as the \"Index document\". We'll add an error document in step 6 below, but feel free to do that now if you already know what you want to do with that.\n",
    "\n",
    "leave the rest of the configurations as-is. grab the \"endpoint\" from the configuration window\n",
    "\n",
    "\n",
    "### try it out\n",
    "\n",
    "navigate to that endpoint you were given while configuring the bucket to be a static webpage. what do you see?\n",
    "\n",
    "\n",
    "### read the gosh darn manual\n",
    "\n",
    "without advanced preparation (and kudos to you if you did it already!) the default behavior will be to return to you a `403 FORBIDDEN` error. For my bucket, for example:\n",
    "\n",
    "```\n",
    "403 Forbidden\n",
    "\n",
    "Code: AccessDenied\n",
    "Message: Access Denied\n",
    "RequestId: A7BA5343504C695B\n",
    "HostId: 7KtvPPnjmQAk2Ry4CeYn58+I1IL1+W+tV633d2/SX5c6XmIFqvewLMTUGwKxrgaY33tzlOF0jek=\n",
    "```\n",
    "\n",
    "\n",
    "First, read [what a 403 error is](https://en.wikipedia.org/wiki/HTTP_403) (or [any `html` code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes), for that matter). \n",
    "\n",
    "After this, read [the static website hosting documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/HowDoIWebsiteConfiguration.html) for details on how to configure permissions to allow users to access this site.\n",
    "\n",
    "*Note*: the documentation tells you how to open *an entire bucket*, so keep in mind this will make all the items in the configured bucket public. It is possible to make a single file public from the file summary page, fwiw.\n",
    "\n",
    "\n",
    "### fix permissions\n",
    "\n",
    "using the information from the documentation, make sure that the endpoint you tried above (and possibly received a 403 for) is now publicly accessible\n",
    "\n",
    "\n",
    "### add an error page\n",
    "\n",
    "copy the `html` file you are using as your index page to a new file called `error.html`, and edit that new file to contain an error message. This might be as simple as replacing the header (the `<h1>` element) and first paragraph (the `<p>` element) so that they contain warnings that a \"page is missing\" or \"this url was an error\". If you have your own `error.html` file, feel free to use that instead.\n",
    "\n",
    "upload that new `error.html` file, and then go back to the static webpage configuration for your bucket (where we *didn't* enter an `error.html` file before), and add the newly uploaded file. \n",
    "\n",
    "\n",
    "### verify that missing pages redirect to `error.html`\n",
    "\n",
    "verify that a url that doesn't exist takes you to that `error.html` page. Take the ip address from before (for example, mine is: http://wp.rzl.gu511.com.s3-website-us-east-1.amazonaws.com/) and add a meaningless url path to the end of that. Again, for example: http://wp.rzl.gu511.com.s3-website-us-east-1.amazonaws.com/pagedontexistyo.php\n",
    "\n",
    "verify that the page that is displayed is your error page.\n",
    "\n",
    "\n",
    "### deliverables:\n",
    "\n",
    "send us the url of your static webpage. We will visit\n",
    "\n",
    "+ the path itself, and\n",
    "+ a path that doesn't exist\n",
    "\n",
    "to verify that both the index and error pages are available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exercise 2: a really sad alarm clock\n",
    "\n",
    "\n",
    "### getting familiar with the script\n",
    "\n",
    "download [this `python` file](https://s3.amazonaws.com/shared.rzl.gu511.com/alarm_clock.py) and review it to figure out what it does.\n",
    "\n",
    "the elements below the `command line` comment block implement a command line interface (`cli`) for this python script. check out the `cli` options by trying out\n",
    "\n",
    "```bash\n",
    "python alarm_clock.py --help\n",
    "```\n",
    "\n",
    "*note*: `boto3` is a prerequisite, so you have to execute the above command in an environment where `boto3` is installed.\n",
    "\n",
    "\n",
    "### create an alarm clock bucket\n",
    "\n",
    "create a *new* `s3` bucket (i.e. don't use your homework submission `s3` bucket) and make it visible to the public. we (Carlos and I) will be looking at the contents of this bucket for other assignments, so it's important that anyone can access it.\n",
    "\n",
    "going forward, I will refer to this as \"the alarm clock bucket\". email us the `url` for this bucket!\n",
    "\n",
    "\n",
    "### post a message\n",
    "\n",
    "use the `alarm_clock.py` file to post a message to that new bucket you created in the previous step.\n",
    "\n",
    "\n",
    "### send us some proof!\n",
    "\n",
    "write the exact `bash` command you called to run the `alarm_clock.py` script to a file called `what_i_ran.sh` and upload it to the public homework submission `s3` bucket you shared with us in the last homework assignment.\n",
    "\n",
    "also, get the link to the text file this command created on `s3`, and include that url in your submission email to us. note: verify that the `url` works for other users, or for you when not signed in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exercise 3: turn a `python` script into a `cron` job on the `ec2` box\n",
    "\n",
    "the above `bash` command is something that could be put into a [`crontab`](http://www.adminschoice.com/crontab-quick-reference) entry on a linux machine, and run with some frequency by [`cron`](https://en.wikipedia.org/wiki/Cron). let's do that!\n",
    "\n",
    "\n",
    "### breaking it down\n",
    "\n",
    "as we said in lectures, `cron` is a linux command scheduling service. it is constantly running, and every minute it checks to see if there are commands which must be executed at that time. users can request that commands be executed with some fixed frequency (up to minute resolution).\n",
    "\n",
    "the way that users add elements to \"the schedule\" is by editing a file called a `crontab`. the name `crontab` is short of `cron` table, because the items are expected to be entered in a sort of tabular format (more on that below).\n",
    "\n",
    "users can use the linux command `crontab` to interact with these files -- try `man crontab` and `crontab -h` on your `ec2` server to see the available options, or jump right in \n",
    "\n",
    "a `cron` entry consists of an execution frequency statement and a command to execute\n",
    "\n",
    "\n",
    "#### execution frequency statement\n",
    "\n",
    "an entry in a `crontab` file will start with a 5-element sequence that dictates how frequently a command should be executed. you can see several examples on the [`cron` wikipedia page](https://en.wikipedia.org/wiki/Cron), and check out [this cool webapp](https://crontab.guru/) to see what options are available and what a given statement means in \"human\" terms.\n",
    "\n",
    "as an example, we could choose to run a job every day at midnight using the frequency statement:\n",
    "\n",
    "```\n",
    "0 0 * * *\n",
    "```\n",
    "\n",
    "this is at\n",
    "\n",
    "1. the 0 minute\n",
    "2. of the 0 hour\n",
    "3. on every calendar day\n",
    "4. on every month\n",
    "5. and on every day of the week\n",
    "\n",
    "one thing to be aware of: our `ubuntu` computers are using `UTC` for time -- type the `date` command to see what time your `ec2` server thinks it is.\n",
    "\n",
    "**develop a `crontab` frequency expression to execute our command: every day at 6:00 AM EST**\n",
    "\n",
    "don't worry about daylight savings time or any of that right now -- just make sure that the timestamp on the file the next time it runs (i.e. any day within the next month) is 6:00 AM EST.\n",
    "\n",
    "\n",
    "#### command statement\n",
    "\n",
    "`cron` is effectively just executing shell statements for you, so anything you can type in a `bash` shell is fair game -- including the `python alarm_clock.py` statement from the previous exercise.\n",
    "\n",
    "\n",
    "##### `cron` and your `PATH` variable\n",
    "\n",
    "there is an important difference between how you normally execute `bash` commands and how `cron` will do that: whenever you *manually* start up a shell session via `ssh`, your `bash` shell will `source` the contents of your `.bashrc` file.\n",
    "\n",
    "`cron` **will not do this**.\n",
    "\n",
    "this is important for a lot of reasons, but the most important right now is that `source`-ing your `.bashrc` will populate your `PATH` variable. We've discussed this several times in class, but it bears repeating: `conda` and `conda` environments depend critically on *changing* this `PATH` variable so that when you type `python` the shell session *resolves* that command to be the particular `python` executable you want to use.\n",
    "\n",
    "when you executed the `python alarm_clock.py` command in the previous exercise, `python` resolved to something depending on your `PATH`. after you've successfully run that command, \n",
    "\n",
    "1. execute `which python` to see which `python` executable you used\n",
    "2. execute `echo $PATH` to see what your `PATH` variable is \n",
    "\n",
    "depending on actions you took (e.g. `source activate myenv`), your `PATH` variable would change. on your `ec2` server,\n",
    "\n",
    "+ if no changes have been made to your `PATH` at all, `python` will resolve to nothing and `python3` will resolve to `/usr/bin/python3`\n",
    "+ if your `PATH` was updated by `conda` in your `.bashrc` to be `PATH=/home/ubuntu/miniconda3/bin:[a bunch of other paths]` (this is how we configured it in class), `python` will resolve to `/home/ubuntu/miniconda3/bin/python`\n",
    "+ if you had run `source activate myenv`, this would have updated your `PATH` to be `PATH=/home/ubunut/miniconda3/envs/myenv/bin:[a bunch of other paths]`, `python` will resolve to `/home/ubuntu/miniconda3/envs/myenv/bin/python`\n",
    "\n",
    "to summarize:\n",
    "\n",
    "| previous action taken          | changes to `PATH` variable                          | `python` resolves to...                                                       |\n",
    "|--------------------------------|-----------------------------------------------------|------------------------------------------------------------------------------|\n",
    "| default                        | no change                                           | nothing (`python3 --> /usr/bin/python`)                               |\n",
    "| `conda` statement in `.bashrc` | `PATH=/home/ubuntu/miniconda3/bin:$PATH`            | `/home/ubuntu/miniconda3/bin/python`            |\n",
    "| `source activate myenv`        | `PATH=/home/ubuntu/miniconda3/envs/myenv/bin:$PATH` | `/home/ubuntu/miniconda3/envs/myenv/bin/python` |\n",
    "\n",
    "\n",
    "###### why does this matter?\n",
    "\n",
    "`cron` will not do anything special with your `PATH`. if you want a special `PATH` for a special `python` (hint hint hint: you do), you can add a line anywhere in the `crontab` file before your commands:\n",
    "\n",
    "```bash\n",
    "PATH=/the/path/i/want/to/add:$PATH\n",
    "```\n",
    "\n",
    "so to use the base `conda` installation, you would have to add:\n",
    "\n",
    "```bash\n",
    "PATH=/home/ubuntu/miniconda3/bin:$PATH\n",
    "```\n",
    "\n",
    "and to use the a `conda` environment `myenv`, you would have to add:\n",
    "\n",
    "```bash\n",
    "PATH=/home/ubuntu/miniconda3/envs/myenv/bin:$PATH\n",
    "```\n",
    "\n",
    "(the actual, exact path you want to use here is the directory your `python` executable was in when you ran `which python` after a successful run, and which was in the `PATH` you got from the `echo $PATH` command you ran at that time as well)\n",
    "\n",
    "\n",
    "### developing and testing\n",
    "\n",
    "at first, it is helpful to have the command run every minute, and to write all output to a file\n",
    "\n",
    "```bash\n",
    "PATH=/my/special/path:$PATH\n",
    "\n",
    "* * * * *  mycommand.sh >> /home/ubuntu/mycommand.log\n",
    "```\n",
    "\n",
    "\n",
    "### deliverables\n",
    "\n",
    "1. upload a plain text file named `alarm_clock.crontab` to your `s3` homework submission bucket with two lines in it:\n",
    "    1. the `PATH` statement you wrote\n",
    "    2. the one-line `crontab` entry to run `alarm_clock.py` every day at 6:00 AM EST\n",
    "2. we will look for at least one file successfully created in the public \"alarm clock\" bucket (from the previous exercise) at 6:00 AM EST\n",
    "    1. because this bucket is public, we will be able to see it ourselves -- no need to send us anything"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exercise 4: `xpath` and `css` selectors in a controlled environment\n",
    "\n",
    "take the following `html` document (also [available via `s3`](https://s3.amazonaws.com/shared.rzl.gu511.com/example.html) if you want to use chrome or firefox Inspect mode):\n",
    "\n",
    "```html\n",
    "<html>\n",
    "    <head></head>\n",
    "    <body>\n",
    "        <div id=\"tablediv\">\n",
    "            <table id=\"important_table\" class=\"very_pretty\">\n",
    "                <thead>\n",
    "                    <tr>\n",
    "                        <th>column a</th>\n",
    "                        <th>column b</th>\n",
    "                        <th>column c</th>\n",
    "                    </tr>\n",
    "                </thead>\n",
    "                <tbody>\n",
    "                    <tr class=\"oddrow\">\n",
    "                        <td>1</td>\n",
    "                        <td>4</td>\n",
    "                        <td>5</td>\n",
    "                    </tr>\n",
    "                    <tr class=\"evenrow\">\n",
    "                        <td>0</td>\n",
    "                        <td>2</td>\n",
    "                        <td>4</td>\n",
    "                    </tr>\n",
    "                </tbody>\n",
    "            </table>\n",
    "            <ul>\n",
    "                <li>just to be tricky</li>\n",
    "            </ul>\n",
    "        </div>\n",
    "        <div>\n",
    "            <ul class=\"very_pretty\">\n",
    "                <li>hello</li>\n",
    "                <li class=\"active\">world</li>\n",
    "            </ul>\n",
    "            <ol class=\"kinda_ugly\">\n",
    "                <li>howya</li>\n",
    "                <li class=\"inactive\">doin</li>\n",
    "            </ol>\n",
    "        </div>\n",
    "    </body>\n",
    "</html>\n",
    "```\n",
    "\n",
    "in the following, there are no trick questions. there will always be at least one element selected in 1 and 2, and at least one valid path in 3 and 4. also, remember that you can enter these `xpath` and `css selector` expressions directly in the developer tools (highlight the html elements window and press `Ctrl + F` or `Command + F`) to see the number of matches and to cycle through them\n",
    "\n",
    "\n",
    "### 4.1\n",
    "\n",
    "for each of the below `xpath` expressions, idenfity the number of elements matched by that expression:\n",
    "\n",
    "1. `/html/body/div/ul`\n",
    "2. `/html/body/div/ul/li`\n",
    "3. `/html/body/div/*/li`\n",
    "4. `/html/body/div/*/li[@class]`\n",
    "5. `/html/body/div/*/li[@class=\"active\"]`\n",
    "\n",
    "\n",
    "### 4. 2\n",
    "\n",
    "for each of the below `css` selectors, identify the number of elements matched by that expression:\n",
    "\n",
    "1. `tr`  (*note: text search in developer tools might be tricky here, since \"tr\" is a common string...*)\n",
    "2. `tr.evenrow`\n",
    "3. `#important_table`\n",
    "4. `.very_pretty`\n",
    "5. `div > ul`\n",
    "\n",
    "\n",
    "### 4. 3\n",
    "\n",
    "for each of the below, develop the appropriate `xpath` expression\n",
    "\n",
    "1. use an *absolute* path to select *only* the element `<li class=\"active\">world</li>`\n",
    "2. use a non-absolute path to select *only* that same element `<li class=\"active\">world</li>` which uses the `class` attribute\n",
    "3. select all `td` elements\n",
    "4. select all `td` element in a row with `class=\"evenrow\"`\n",
    "5. select the `<table id=\"important_table\" class=\"very_pretty\">` element using its `class` attribute\n",
    "6. select the `<table id=\"important_table\" class=\"very_pretty\">` element using its `id` attribute\n",
    "\n",
    "\n",
    "### 4. 4\n",
    "\n",
    "now for each of the below, develop the appropriate `css` selector\n",
    "\n",
    "1. use a *direct descendant* selector to select *only* the element `<ul class=\"very_pretty\">`\n",
    "2. use an *any descendant* selector to *only* select the four `<li>` elements in the *second* `div` block\n",
    "3. select all `td` elements\n",
    "4. select all `td` element in a row with `class=\"evenrow\"`\n",
    "5. select the `<table id=\"important_table\" class=\"very_pretty\">` element using its `class` attribute\n",
    "6. select the `<table id=\"important_table\" class=\"very_pretty\">` element using its `id` attribute\n",
    "\n",
    "\n",
    "## deliverable\n",
    "\n",
    "fill in the following table, save it as a `csv` with name `xpath_and_css.csv`, and upload that `csv` to your `s3` homework submission bucket\n",
    "\n",
    "| exercise | answer |\n",
    "|----------|--------|\n",
    "| 4.1.1    |        |\n",
    "| 4.1.2    |        |\n",
    "| 4.1.3    |        |\n",
    "| 4.1.4    |        |\n",
    "| 4.1.5    |        |\n",
    "| 4.2.1    |        |\n",
    "| 4.2.2    |        |\n",
    "| 4.2.3    |        |\n",
    "| 4.2.4    |        |\n",
    "| 4.2.5    |        |\n",
    "| 4.3.1    |        |\n",
    "| 4.3.2    |        |\n",
    "| 4.3.3    |        |\n",
    "| 4.3.4    |        |\n",
    "| 4.3.5    |        |\n",
    "| 4.3.6    |        |\n",
    "| 4.4.1    |        |\n",
    "| 4.4.2    |        |\n",
    "| 4.4.3    |        |\n",
    "| 4.4.4    |        |\n",
    "| 4.4.5    |        |\n",
    "| 4.4.6    |        |"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}