spot_price_history.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `ec2` spot price history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "import boto3\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## preamble: what is spot pricing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "when we created our `ec2` instances, we went with the \"free tier\" option, so we didn't care much about what the instance we built cost. That's not always an option -- for example, suppose you want to have a computer with more than one processor and more than 1 GiB of memory (not an unreasonable ask!). you might end up spending quite a bit for an *on-demand* instance -- one you can access any time you want -- with better specs.\n",
    "\n",
    "check out the pricing information [here](https://aws.amazon.com/ec2/pricing/on-demand/) for more info about these costs.\n",
    "\n",
    "there are other options, however:\n",
    "\n",
    "1. reserved instances: you pay up front to have one machine with some properties *for as long as you want*. the up-front cose is high, but the day-to-day cost is much lower\n",
    "2. spot prices: you agree to pay *up to a certain price*. you can use your instance up until the market demand for that resource passes your limit price, after which point your machine becomes unavailable to you until the market price goes back down.\n",
    "\n",
    "spot pricing provides an interesting opportunity: suppose I want to have a very powerful machine, but I don't anticipate I will use it often, and I'm okay if it is unavailable for time to time (a good example: distributed computing with \"worker nodes\" in a hadoop environment). Maybe I can save a lot of money by picking a price much lower than the \"on-demand\" price?\n",
    "\n",
    "let's download the timeseries of spot prices to get an idea about just what type of savings are possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## downloading spot price information\n",
    "\n",
    "we know from [the `ec2 api` documentation](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Welcome.html) that we can get spot pricing via the `aws api` using the [`DescribeSpotPriceHistory`](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSpotPriceHistory.html) endpoint.\n",
    "\n",
    "let's use the `boto3 python` library to hit up that `api` and do a bit of analysis.\n",
    "\n",
    "in the following, we will go through the steps required to download spot price history and load it into a dataframe. once we have an end-to-end process set up, we'll use the intermediate steps to create a single function that we can use to pull down spot pricing programatically and repeatedly.\n",
    "\n",
    "### setup\n",
    "\n",
    "first things first: let's create a `boto session` object named `session`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### investigate the `ec2 resource`\n",
    "\n",
    "as we mentioned in lecture, it is always preferrable to use the `ec2 resource` whenever possible. let's use the `session.resource` method of the `session` object we created above to build an `ec2` resource object, and explore the methods of that resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #\n",
    "dir(ec2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I could read through that list looking for the word \"spot\", but I'm *pretty* lazy. I'll let `python` do it for me. if I wanted to find any element in the `dir` of `ec2` which involves `vpc`s, I could do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[elem for elem in dir(ec2) if 'vpc' in elem.lower()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "do the same, but for things related to spot pricing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "huh... nothing.\n",
    "\n",
    "at this point, I would crack open [the `boto3 ec2 resource`](http://boto3.readthedocs.io/en/latest/reference/services/ec2.html) documentation to see if there is any discussion of `spot` pricing. there is, but only associated with `client` objects -- not `resource` objects.\n",
    "\n",
    "whenever we\n",
    "\n",
    "1. can't make a `resource` object for a particular service, or\n",
    "2. can't use a `resource` method or attribute to obtain information the information we desire\n",
    "\n",
    "then and only then we will use an `ec2 client` and an exact endpoint instead"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `ec2 client` function\n",
    "\n",
    "use the `session.client` method of our `session` object to create a *new* `ec2` item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### exploring `ec2`'s `describe_spot_price_history`\n",
    "\n",
    "as with all `boto` clients, the member functions of the `ec2` client we just created are in a one-to-one mapping with the available `api` endpoints. We are looking to describe spot price history, and sure enough:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "help(ec2.describe_spot_price_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "it looks like it accepts many parameters, but there are a few that jump out at me:\n",
    "\n",
    "##### `StartTime` and `EndTime`\n",
    "\n",
    "we can pull all spot prices between these two `datetime` objects\n",
    "\n",
    "##### `InstanceTypes`\n",
    "\n",
    "we can select spot pricing for various instance types\n",
    "\n",
    "##### `MaxResults` and `NextToken`\n",
    "\n",
    "if you read the documentation, you'll see that the response you get from this request will not be *all* spot prices in your time window, but the first `MaxResults` items. The returned message will also give you a `NextToken`, which you can use on your *next* call to the `api` to effectively say \"I've already received `MaxResults` records, so start there\".\n",
    "\n",
    "this arrangement -- where you receive information in chunks and have to keep track of which chunk you last received and which you need next -- is often called \"pagination\" (because you are receiving date one \"page\" at a time).\n",
    "\n",
    "let's look at the spot prices for the first 12 hours of September 29th, and for a particularly beefy instance type: [`m4.16xlarge`](https://aws.amazon.com/ec2/pricing/on-demand/). this instance type has 64 virtual `cpu`s, 258 `GiB` of memory, and usually costs about $3.2 per hour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "t0 = datetime.datetime(2017, 9, 29)\n",
    "t1 = t0 + datetime.timedelta(hours=12)\n",
    "instancetypes = ['m4.16xlarge']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resp = ec2.describe_spot_price_history(\n",
    "    StartTime=t0,\n",
    "    EndTime=t1,\n",
    "    InstanceTypes=instancetypes\n",
    ")\n",
    "resp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "note: we got a `NextToken` value, which indicates that we didn't finish our request just yet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resp['NextToken']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we also got 1000 prices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prices = resp['SpotPriceHistory']\n",
    "len(prices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the first price in that list is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prices[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### pagination and `paginator`s\n",
    "\n",
    "at this point, we could create a loop which would take the `NextToken` from one request and use it in the next request, and that would be perfectly fine.\n",
    "\n",
    "however, because this paradigm and process is so common, `boto3` has implemented a special way of handling that via `paginator` objects. a `paginator` is effectively a wrapper around a given endpoint (here, e.g., `describe_spot_price_history`) which handles this `NextToken` iteration logic for us -- pretty cool.\n",
    "\n",
    "let's create a `paginator` for this `api` endpoint and look at the first two pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "help(ec2.get_paginator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "paginator = ec2.get_paginator('describe_spot_price_history')\n",
    "help(paginator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "note that the only method available to `paginator` is `paginate`. the documentation for that `paginate` method is effectively just the documentation for the `describe_spot_price_history` api, except that the discussion about `NextToken` has dropped out.\n",
    "\n",
    "the `paginate` method creates an iterator object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pageiter = paginator.paginate(\n",
    "    StartTime=t0,\n",
    "    EndTime=t1,\n",
    "    InstanceTypes=instancetypes\n",
    ")\n",
    "pageiter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### iterating through spot price pages\n",
    "\n",
    "paginator is *an `iterator`*, so whenever we use it we will be looping through it:\n",
    "\n",
    "```python\n",
    "for page in pageiter:\n",
    "    # do something ...\n",
    "```\n",
    "\n",
    "it also means that it is *stateful*, so if we should re-create it each time we want to use it. in this sense it is very different from a list (which you could iterate through as many times as you want). you can't re-use an iterato  after it's been used once because *internally* it thinks it has done all of the work it can, and will no longer return anything.\n",
    "\n",
    "It is possible to get the first item in any iterator by beginning to iterate through it (in a `for` loop) and immediately `break`ing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for page in pageiter:\n",
    "    break\n",
    "page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so, the individual elements in the `paginator.paginate` iterator are effectively identical to the regular `api` endpoint response items -- cool.\n",
    "\n",
    "let's practice using one of these iterators one time, just printing how many prices we get for each page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we create a new iterator every time\n",
    "pageiter = paginator.paginate(\n",
    "    StartTime=t0,\n",
    "    EndTime=t1,\n",
    "    InstanceTypes=instancetypes\n",
    ")\n",
    "\n",
    "for page in pageiter:\n",
    "    print(len(page['SpotPriceHistory']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "cool, 2649 spot prices in those 12 hours, delivered in 1000-price chunks!\n",
    "\n",
    "we can use a list comprehension to load all 2649 of those items into a single list of dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we have to create a new iterator every time\n",
    "pageiter = paginator.paginate(\n",
    "    StartTime=t0,\n",
    "    EndTime=t1,\n",
    "    InstanceTypes=instancetypes\n",
    ")\n",
    "\n",
    "pricehistory = [\n",
    "    price\n",
    "    for page in pageiter\n",
    "    for price in page['SpotPriceHistory']\n",
    "]\n",
    "\n",
    "len(pricehistory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pricehistory[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### loading a list of `dict` items into a `pandas` dataframe\n",
    "\n",
    "it just so happens that lists of dictionary items (such as the `pricehistory` list we just created) are one of the most basic input structures for a `pandas` dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(pricehistory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print out the first 10 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "there's only one weird thing going on here, and it's not immediately obvious from the above. let's look at the `dtypes` (data types) attribute of our data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the `dtype` of the `SpotPrice` column is \"object\", which is the `dtype` of *strings*, not numbers, in `pandas`. for more details on `dtypes`, please refer to [the `pandas` documentation](https://pandas.pydata.org/pandas-docs/stable/basics.html#dtypes)\n",
    "\n",
    "the reason these items appear as `object` and not `float` is because the `api` returns them as quoted strings.\n",
    "\n",
    "let's fix this.\n",
    "\n",
    "recall from the last homework that whenever we want to update a column in a dataframe we should use the `df.loc` indexer:\n",
    "\n",
    "```python\n",
    "df.loc[:, 'column_to_be_updated'] = newvalues\n",
    "```\n",
    "\n",
    "let's use the `astype` method of the `df.SpotPrice` column to replace the string values in column `SpotPrice` with `float` values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print the first 10 records and the dtypes to see if this has changed as expected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "finally, let's figure out what the `shape` of our dataframe is -- we know there are 2649 prices, so we should see that reflected here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## putting it all together into a function\n",
    "\n",
    "given all the various steps we took above, let's build a function to create a dataframe object with spot prices on an arbitrary set of instance types and between arbitrary start and end times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def spot(t0, t1, instancetypes):\n",
    "    # construct an s3 session\n",
    "    # --------------- #\n",
    "    # FILL ME IN !!!! #\n",
    "    # --------------- #\n",
    "    \n",
    "    # construct an ec2 client object\n",
    "    # --------------- #\n",
    "    # FILL ME IN !!!! #\n",
    "    # --------------- #\n",
    "    \n",
    "    # create a paginator object\n",
    "    # --------------- #\n",
    "    # FILL ME IN !!!! #\n",
    "    # --------------- #\n",
    "    \n",
    "    # create a pageiter object\n",
    "    # --------------- #\n",
    "    # FILL ME IN !!!! #\n",
    "    # --------------- #\n",
    "    \n",
    "    # use the pageiter object to create a list of \n",
    "    # spot price dictionary objects\n",
    "    # --------------- #\n",
    "    # FILL ME IN !!!! #\n",
    "    # --------------- #\n",
    "    \n",
    "    # create a dataframe from the list of dicts above\n",
    "    # --------------- #\n",
    "    # FILL ME IN !!!! #\n",
    "    # --------------- #\n",
    "\n",
    "    # convert the elements of column `SpotPrice` to `float`s\n",
    "    # --------------- #\n",
    "    # FILL ME IN !!!! #\n",
    "    # --------------- #\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "above we saw that there were 2649 prices in the first 12 hours of September 29th -- let's confirm that our function creates a dataframe with that many items as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = spot(t0=t0, t1=t1, instancetypes=instancetypes)\n",
    "\n",
    "# this will produce an error if the right hand side\n",
    "# is not true, and be silent otherwise (so no news\n",
    "# is good news!)\n",
    "assert df.shape[0] == 2649"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## downloading and investigating data\n",
    "\n",
    "use your `spot` function above to create a dataframe `df29` with the spot pricing *for the entire day* of September 29th (not just the first 12 hours) for the same instance type (`m4.16xlarge`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "t0 = datetime.datetime(2017, 9, 29)\n",
    "t1 = datetime.datetime(2017, 9, 30)\n",
    "instancetypes = ['m4.16xlarge']\n",
    "\n",
    "df29 = spot(t0, t1, instancetypes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df29.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df29.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use the `df29.groupby` method to group records by `AvailabilityZone` and `InstanceType`, and then perform aggregation calculations on the `SpotPrice` values within those groups using the `agg` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "help(df29.groupby)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df29.groupby('AvailabilityZone').SpotPrice.agg(['min', 'max', 'mean', 'std'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "use the `df29.groupby` method to group records based on both `ProductDescription` and `AvailabilityZone` (hint: look at the examples in the documentation above), and perform the same aggregation calculations on the `SpotPrice` columns within those groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## plotting data\n",
    "\n",
    "If you were able to do the above, one thing you will have noticed is that the average spot price for different product descriptions was *hugely* different (Windows machines cost over `$`4, but Linux options cost less than $1), and within a given `ProductDescription` (e.g. just Linux/UNIX machines) there was even a fair amount from availability zone to availability zone.\n",
    "\n",
    "what if one availability zone is always cheaper than another? What if the pricing is cyclical, so some times in the day is always cheapest? that'd be good to know!\n",
    "\n",
    "let's plot those different timeseries values and see if anything pops out.\n",
    "\n",
    "first, let's limit ourselves to just Linux/UNIX machines. create a dataframe `dflinux` which has only the elements of `df29` above where the `ProductDescription` is `\"Linux/UNIX\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# --------------- #\n",
    "# FILL ME IN !!!! #\n",
    "# --------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now, let's set up our plotly graphing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import plotly.graph_objs as go\n",
    "import plotly.offline\n",
    "\n",
    "plotly.offline.init_notebook_mode(connected=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we'd like to create a separate line plot for each availabilty zone's timeseries.\n",
    "\n",
    "To do this, we can *iterate* through `groupby` objects using the following syntax:\n",
    "\n",
    "```python\n",
    "for (grpIndexValues, groupRecordsDataframe) in df.groupby('myGroupbyColumn'):\n",
    "    # do something with the common index\n",
    "    # do something with the records for that common index\n",
    "```\n",
    "\n",
    "to get an idea of what the `grpIndexValues` and `groupRecordsDataframe` in the above look like, we can do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for (idx, grp) in df29.groupby('AvailabilityZone'):\n",
    "    break\n",
    "\n",
    "idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grp.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grp.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so in availability zone `us-east-1a` we have 1006 price records. the `groupby` iteration gives us the index that defines this group (the availability zone, `us-east-1a`), and also the \"chunk\" of records (as a dataframe) that is all of the records in `dflinux` with that availability zone.\n",
    "\n",
    "as an example where we do things with the index and group chunks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for (idx, grp) in df29.groupby('AvailabilityZone'):\n",
    "    print(idx)\n",
    "    print(grp.shape)\n",
    "    print(grp.SpotPrice.mean())\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use this fact to create a different `plotly` [line object using the `Scatter` method](https://plot.ly/python/line-and-scatter/#line-and-scatter-plots) for each `AvailabilityZone` value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = [\n",
    "    go.Scatter(\n",
    "        x=grp.Timestamp,\n",
    "        y=grp.SpotPrice,\n",
    "        name=idx\n",
    "    )\n",
    "    # we need an iterator here that will produce\n",
    "    # AvailabilityZone values as \"idx\" and chunks of\n",
    "    # records for that AZ as \"grp\"\n",
    "    # --------------- #\n",
    "    # FILL ME IN !!!! #\n",
    "    # --------------- #\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now, we render the plot!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layout = go.Layout(\n",
    "    title=\"Spot pricing of Linux machines in us-east-1, September 29th\",\n",
    "    xaxis={'title': 'time (UTC)'},\n",
    "    yaxis={'title': 'price (USD)'}\n",
    ")\n",
    "\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "\n",
    "plotly.offline.iplot(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## choosing your favorite option"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "two questions, and just write them in here:\n",
    "\n",
    "first: which availabilty zone would you pick *just based on this single day's price history*?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FILL ME IN!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "second: roughly how much would you save if you used that instance vs. the quoted price online of $3.20 (no need to do a calculation, just a rough estimate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FILL ME IN!!!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}