006_environments_1_anaconda.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/rzl-ds/gu511/blob/master/006_environments_1_anaconda.ipynb\" target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# environment management: `anaconda`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## wait, what class is this?\n",
    "\n",
    "why are we talking about environments?\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://news-media.stanford.edu/wp-content/uploads/2016/11/10165436/environment_GettyImages-501231894.jpg\" width=\"800px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in the computer science world, the phrase \"environment\" is often thrown around with slightly ambiguous meaning. in the broadest sense, it can be the \"computing\" environment or the \"operating\" environment -- the combination of hardware and software that a user interacts with; the whole enchilada.\n",
    "\n",
    "in discussions about specific applications and for certain programming languages, it can be filtered down to the \"runtime\" environment -- the relevant aspects of the hardware (from that application's point of view) and the codebase which defines that application or language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "generally speaking, when I talk about the **environment** I'm focusing on the software (literal files, on your computer's hard drive) that define how *something* behaves. for example..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## your `python` environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "your `python` environment is the tools and packages available to you for use within the `python` programming language, and the way those tools and packages behave. this is completely determined by the literal files defining the `python` language on your computer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### current system `python` environments\n",
    "\n",
    "let's do a quick `python` version check:\n",
    "\n",
    "on your `ec2` instance, what `python` version do you have installed?\n",
    "\n",
    "```sh\n",
    "python --version\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "```sh\n",
    "# grrrrr.......\n",
    "python3 --version\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "python --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "different versions of `python` (and different versions of installed packages) correspond to different files defining the language's behavior and thus different levels of compatibility. personally, I think knowing that these files exist is among the more important pieces of information in my `python` learning.\n",
    "\n",
    "***the way that the code you wrote behaves depends on these files***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "recall that the `bash` command `which` will tell us the path of the executable that will actually be called when we type in a command\n",
    "\n",
    "```sh\n",
    "which python3\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "which python3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "your out-of-the-box `ec2` instances will likely return `/usr/bin/python3`. so when you type `python3` on the command line, you will actually call the executable file `/usr/bin/python3`.\n",
    "\n",
    "the same sort of thing is going on for individual `python` modules we import. Every module has a \"private\" member `__file__` which lists the path to the file used to define that module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.__file__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's look at that file!\n",
    "\n",
    "```sh\n",
    "# for you, it is:\n",
    "less /usr/lib/python3.6/os.py\n",
    "\n",
    "#for me, right now, it'll be different -- hence the craziness below. sorry!\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "OS_FILE=$(python -c \"import os; print(os.__file__)\")\n",
    "cat $OS_FILE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "if you change that file, or your friend (who is running your code) doesn't have that same file, the code that uses `os` will be different.\n",
    "\n",
    "the same caveat goes for every file or environment variable used by your python process on any machine. this collection of files defines what is often called the \"`python` environment\", and it can be different on any system. `sudo apt install` could totally change it.\n",
    "\n",
    "yikes!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in the real world, the implication is immediate: if one of my programs only works for version 1.2, and another only works for version 2.1, and the `GOVERNMENT AGENCY NAME REDACTED` sysad just installed library 1.0 and *that* took two years, this  will probably be a problem.\n",
    "\n",
    "It would be nice if this problem was solved..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### virtual environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "on our `ec2` instances, there is only one `python`: the executable file located at `/usr/bin`.\n",
    "\n",
    "there is also only one `os.py` file, one `datetime.py` file, one `pickle.py` file, etc. the code that quite literally defines what you can do after you run\n",
    "\n",
    "```python\n",
    "import os\n",
    "```\n",
    "\n",
    "is in the `/usrlib/python3.6` directory.\n",
    "\n",
    "what else is in that directory?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for a vanilla installation of `python` on any machine -- like the installation on this `ec2` instance -- there is only one \"environment\": this collection of `python`-related files.\n",
    "\n",
    "there is a fixed version in time that defines these files and packages, and no matter who you are or what you are doing on this `ec2`, `python` \"looks\" the same way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "that consistency is essential for all of the various `python` programs running on your current `ec2` instance at the root level to coordinate and work together. it's a good thing!\n",
    "\n",
    "but what if you want to do things slightly differently?\n",
    "\n",
    "what if you want to access a different version of `python` than the one installed?\n",
    "\n",
    "what if you want to install a package that requires a different version than the version the rest of the system wants to use?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you *could* try and update the system's environment, using `sudo apt get` or `pip` to install different versions... but that's potentially dangerous, because you're changing things for **everyone**, including the root user that runs and maintains the whole operating system. ideally you'd be prevented from breaking everything, but we are very powerful, after all, and I'm sure we can find a way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the solution to problems like this -- wanting to have a `python` environment that is different than the base `python` environment without changing that base `python` environment\n",
    "\n",
    "\"virtual environments\" are ways of isolating out the contents (the files) of libraries you're installing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this is something you've actually probably (*kind of*) done in `R`, actually, without knowing it. if you've ever tried installing a package but didn't have admin rights, the `R` interpreter prompts you to see if there's some other place you'd like to install things (usually in your home directory).\n",
    "\n",
    "that is a system-level isolation of the files you want to install. When the interpreter is told to load a package, it looks first for your local copy to see if you have anything spicy, and then it checks for a global copy, and then it cries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so, generalize that idea: let's make *multiple* separate environments (collections of files defining how our `python` code behaves).\n",
    "\n",
    "we can generalize this beyond just \"global\" and \"user\" (as with `R`), even creating a separate environment for each process or code base."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "on a very basic level, all we're doing here is re-installing packages into a special sub-directory somewhere on the machine, and then telling `python` (through environment variables like the `PATH` variable) where to look to find them.\n",
    "\n",
    "instead of using the `python` at `/usr/bin/python3` and the module files at `/usr/lib/python3.6/*.py`, we create some special folder (say, `~/my-virtual-environment`) and install different versions of `python` or `python` packages in\n",
    "\n",
    "```\n",
    "~/my-virtual-environment/bin/python3.9001\n",
    "~/my-virtual-environment/lib/python3.9001/*.py\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "then we edit the `PATH` variable to have `~/my-virtual-environment` at the front and now suddenly when type\n",
    "\n",
    "```sh\n",
    "python3\n",
    "```\n",
    "\n",
    "our `bash` session finds out special `~/my-virtual-environment` files instead of the \"regular\" ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we're tricking `python` into doing the right thing. and `python` is cool about it; once it realizes it's been tricked it's not even mad or anything, in fact it's laughing about it and really *you're* the mad one when you think about it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "once we have the ability to control the versions of the packages we use when we run `python`, it opens up some avenues for getting *really* specific about how the code we use is defined -- specific down to the version of the packages it imports.\n",
    "\n",
    "often times finished `python` projects will ship with a `requirements.txt` file, which lists each `python` package which should be installed and the exact version that it was tested against, and it is expected that it will be executed by a system with the same packages and versions.\n",
    "\n",
    "the \"virtual environment\" is an isolated set of packages that will meet that requirement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "prior to the advent of `conda`, the primary way of creating a virtual environment was to use the python utility `virtualenv`, which is awesome and worth checking out.\n",
    "\n",
    "that being said, however, it's not what I'll recommend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## generalizing virtual environments: `conda`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`conda`, short for `anaconda`, is a *distribution* of python. it takes the virtual environment concept above and adds a special wrinkle: while most virtual environment managers allow you to install different versions of `python` *packages*, `conda` allows you to install different versions of `python` *itself*.\n",
    "\n",
    "this should help you deal with any `python2` vs. `python3` problems you may experience, as well as allow you to use the newer features in the `python` language even if the computer you are working on is stuck at `python3.5`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so, let's go ahead and install `conda`, create a virtual environment, and install something.\n",
    "\n",
    "*note: I would recommend you install `conda` (specifically `miniconda`) on both your laptop and your `ec2` instance, but we will **require** you to install it on your `ec2` instance (it's part of the homework), so you may want to use that instance to do all of this right now*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### installing `conda`\n",
    "\n",
    "`conda`, by default, comes with many of the most commonly downloaded `python` packages. This is great because it gives you a pretty solid working base without any modification, *BUT* given our time and bandwidth limits, I'm going to recommend you install the `miniconda` version (the bare bones) and install packages *as needed* instead of up front.\n",
    "\n",
    "+ [`conda`](https://www.continuum.io/downloads): a big installation, which will take a few minutes, and pre-installs several of the \"must haves\" (many of the above, and maybe more)\n",
    "+ [`miniconda`](https://conda.io/miniconda.html): a bare-bones implementation of the above for the *discerning* gentleprogrammer\n",
    "\n",
    "click on that `miniconda` link (https://conda.io/miniconda.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the decision of what to install depends on a few variables:\n",
    "\n",
    "+ which operating system are you on?\n",
    "    + `win`, `osx`, and `linux` are the options\n",
    "+ which \"architecture\" does your machine have?\n",
    "    + the options are `32` and `64`\n",
    "    + this is the number of bits in memory addresses the processor can understand\n",
    "    + basically, 32 bit processors can handle up to 4GB of RAM at one time, 64 bit processors up to 16 BILLION GB of RAM\n",
    "+ which version of `python` do you want to use\n",
    "    + options right now are `2.7`, `3.7`, and `3.8` (depending on your OS)\n",
    "\n",
    "we will go with `linux`, `64` bit, and `python3.8`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: everyone installs `conda` on their `ec2`</div>**\n",
    "\n",
    "on your `ec2` instance:\n",
    "\n",
    "```sh\n",
    "cd ~\n",
    "wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n",
    "bash Miniconda3-latest-Linux-x86_64.sh\n",
    "\n",
    "# when prompted, we do the following:\n",
    "# press ENTER to read the license\n",
    "#     press `d` to scroll *d*own\n",
    "# yes: approve the license\n",
    "# ENTER: we are okay with this location\n",
    "# yes: run conda init so that your PATH *always* includes conda\n",
    "```\n",
    "\n",
    "then log out and back in and run\n",
    "\n",
    "```sh\n",
    "rm ~/Miniconda3-latest-Linux-x86_64.sh\n",
    "conda update -n base -c defaults conda\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "recall that we previously called\n",
    "\n",
    "```sh\n",
    "which python3\n",
    "```\n",
    "\n",
    "and got `/usr/bin/python3`, and we also checked the file path to the `os` package (from within a `python` shell):\n",
    "\n",
    "```python\n",
    "import os\n",
    "os.__file__\n",
    "```\n",
    "\n",
    "what do we get now, after installing `conda`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: see how the `python` file paths have changed after `conda` was installed</div>**\n",
    "\n",
    "+ on your `ec2` instance, from the `bash` prompt\n",
    "    + run `which python3`\n",
    "+ actually execute the command `python3`\n",
    "+ from the `python` command prompt, run\n",
    "    + `import os; os.__file__`\n",
    "+ exit `python`\n",
    "+ look at the contents of those directories on your machine and compare them to the directories we examined from the base environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "*everything* the `conda` command creates or installs is put into one and only one directory. \"uninstalling\" `conda` is equivalent to simply deleting that directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "take a step back and think about the **python environments** you have now:\n",
    "\n",
    "1. our vanilla `ubuntu` `python` installation (came with the `ec2` instance)\n",
    "1. this new `anaconda`-created environment\n",
    "    + this environment is called the `anaconda` `base` environment\n",
    "\n",
    "try the command\n",
    "\n",
    "```sh\n",
    "conda env list\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "why stop at only two environments?\n",
    "\n",
    "we can use the `conda` command to *create* new environments as well. let's try that right now:\n",
    "\n",
    "```sh\n",
    "conda create -n l33tmode python=3\n",
    "```\n",
    "\n",
    "this will use `conda` to create a new environment named \"`l33tmode`\" with `python` version 3 installed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`conda create` creates a new environment inside of new folder under the `env` sub-directory in that main `conda` directory, and installing all of our required packages there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "as the little dialog will state after you create the environment, you have to \"activate\" that environment if you want to use it. You have to do this any time you want to use a virtual environment.\n",
    "\n",
    "what we're *actually* doing here is updating the `PATH` environment variable to \"point\" `python` to our newly created set of files. Now, when we wish to use `python`, we will be using our specialized, isolated versions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So let's do that:\n",
    "\n",
    "```sh\n",
    "conda activate l33tmode\n",
    "```\n",
    "\n",
    "This should have made our terminal prompt 10 times l33t3r."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: see how the `l33tmode` environment file paths are different than the `base` conda environment file paths</div>**\n",
    "\n",
    "+ same steps as above:\n",
    "+ on your `ec2` instance, from the `bash` prompt\n",
    "    + run `which python3`\n",
    "+ actually execute the command `python3`\n",
    "+ from the `python` command prompt, run\n",
    "    + `import os; os.__file__`\n",
    "+ exit `python`\n",
    "\n",
    "how do the file paths you saw differ from the `base` file paths?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```sh\n",
    "which python3\n",
    "```\n",
    "\n",
    "should return\n",
    "\n",
    "```\n",
    "/home/ubuntu/miniconda3/envs/l33tmode/bin/python3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and\n",
    "\n",
    "```python\n",
    "import os\n",
    "os.__file__\n",
    "```\n",
    "\n",
    "should return\n",
    "\n",
    "```\n",
    "'/home/ubuntu/miniconda3/envs/l33tmode/lib/python3.8/os.py'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "these environments are ours to do with as we wish -- we know that we can do anything to them and we won't break anything important on our system. so let's live dangerously! let's install something (or, two somethings) fun:\n",
    "\n",
    "```sh\n",
    "conda install ipython pandas\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and then try it out. from your `bash` command line run\n",
    "\n",
    "```sh\n",
    "ipython\n",
    "```\n",
    "\n",
    "this should open a fancier python interpreter (`ipython`). inside, run\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "\n",
    "pd.__version__\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## installing packages with `pip`\n",
    "\n",
    "within a `conda` environment, you should **always** try to install new packages with `conda` via `conda install [YOUR PACKAGE NAME HERE]`\n",
    "\n",
    "that being said, not every package is available to install via `conda`. some packages are only installable with `pip` -- luckily for us, `pip install` works with our `conda` environments as well!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "just as `conda` points our `bash` session to a special `python` executable:\n",
    "\n",
    "```sh\n",
    "which python\n",
    "```\n",
    "\n",
    "it also points us to a special `pip`, which will install its packages in our current environment:\n",
    "\n",
    "```sh\n",
    "which pip\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: `pip` install a package into our `conda` environment</div>**\n",
    "\n",
    "make sure you have `l33tmode` activated:\n",
    "\n",
    "```sh\n",
    "conda activate l33tmode\n",
    "```\n",
    "\n",
    "then run the commands\n",
    "\n",
    "```sh\n",
    "pip install requests\n",
    "```\n",
    "\n",
    "afterwards, open a `python` shell and run\n",
    "\n",
    "```python\n",
    "import requests\n",
    "requests.__file__\n",
    "exit()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## freezing and sharing environments\n",
    "\n",
    "one of the purposes of working with a `python` environment manager like `conda` was to enable us to install whatever we want, but the *reason* we wanted to be able to do that was so that we could make sure that no matter what computer we run our code on we have the same behavior"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "if we want to do that, we need to be able to\n",
    "\n",
    "+ **specify** what our environment is when our code is working, and\n",
    "+ **recreate** that environment in other places\n",
    "\n",
    "`conda` can help us do both of these things easily."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "recall back to when we chose which `miniconda` installer script to download: we needed to choose our operating system (`win`, `osx`, or `linux`) and our architecture (`32` or `64`). to share our environment, we need to answer one question first:\n",
    "\n",
    "> does the computer where we want to re-create this environment have ***the same*** architecture and os? or are they ***different****?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### architecture and/or os are *the same*: specify and recreate with `conda list --explicit`\n",
    "\n",
    "within the same OS and architecture, if every package we care about was `conda install`ed we can be incredibly explicit about what should be installed to re-create our *exact* environment, to the file: we use\n",
    "\n",
    "```sh\n",
    "conda list --explicit\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it is common when running this command to write the output to a file `spec-file.txt` which is then used by other users on other (identical os / architecture) computers to create a matching environment\n",
    "\n",
    "```sh\n",
    "# create an environments txt file\n",
    "conda list --explicit > spec-file.txt\n",
    "\n",
    "# look at the contents\n",
    "cat spec-file.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: creating a `spec-file.txt`</div>**\n",
    "\n",
    "make sure you have `l33tmode` activated:\n",
    "\n",
    "```sh\n",
    "conda activate l33tmode\n",
    "```\n",
    "\n",
    "then run the commands\n",
    "\n",
    "```sh\n",
    "conda list --explicit > spec-file.txt\n",
    "cat spec-file.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "anyone on a `linux-64` machine can now create a new environment identical to yours by using this file:\n",
    "\n",
    "```sh\n",
    "conda create --name l33tmode-clone --file spec-file.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### important caveat about `pip` installed packages\n",
    "\n",
    "we mentioned above that you *can* install packages via `pip`. we installed `requests` this way (`pip install requests`).\n",
    "\n",
    "`conda list` will ***not*** include any packages we installed with `pip install`. if you look at the contents of `spec-file.txt`, you will see that `requests` is not in it.\n",
    "\n",
    "theoretically, if you want to have both install types, you *could* have all the `conda` packages installed this way, and then separately install the `pip` packages.\n",
    "\n",
    "in practice, though, if this is your situation it is more common to use the next method (`conda env export`), which supports both `conda` and `pip` installed packages at the same time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### architecture and/or os are *different*: specify and recreate with `conda env export`\n",
    "\n",
    "when the architecture or the OS of the computer trying to recreate an environment is different, it can't install *exactly* the same files -- it has to install a version of those files that was built for *its* architecture and OS.\n",
    "\n",
    "this means we don't need to know the exact file names, we just need to know the versions of the installed packages, and we can figure out what the file names are for our setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we can use the `conda env export` command to list the packages installed in a given `conda` environment and possibly the versions at which they were installed.\n",
    "\n",
    "there are two modes for running the command:\n",
    "\n",
    "1. `conda env export`: lists *every* package installed and *every* version number\n",
    "    + this includes the packages we explicitly installed, as well as ones installed because they were dependencies of those packages\n",
    "1. `conda env export --from-history`: lists *only* the packages we installed explicitly, and only versions if we installed specific versions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: what `--from-history` means</div>**\n",
    "\n",
    "make sure you have `l33tmode` activated:\n",
    "\n",
    "```sh\n",
    "conda activate l33tmode\n",
    "```\n",
    "\n",
    "then run the commands\n",
    "\n",
    "```sh\n",
    "conda env export\n",
    "conda env export --from-history\n",
    "```\n",
    "\n",
    "what is the difference?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`conda env export --from-history` lists out the packages we chose to `conda install` (e.g. `pandas` and `ipython`).\n",
    "\n",
    "`conda env export` (no flag) lists out\n",
    "\n",
    "+ all of those packages *and* their current version numbers (e.g. `pandas=1.1.1` and `ipython=7.18.1`)\n",
    "+ all the things that our particular OS and architecture needed to get `pandas` and `ipython` installed (e.g. `numpy=1.19.1` and `sqlite=3.33.0`)\n",
    "+ all of the `pip install`ed packages (at the bottom, under a heading `- pip:`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "which one of these options you want to use depends on context:\n",
    "\n",
    "+ if you want to share environments across different OS, you should probably use `--from-history`\n",
    "    + if this is the case, you probably want to *explicitly* set versions when you run `conda install`\n",
    "        + e.g. `conda install ipython=7.18.1 pandas=1.1.1`, so that the versions are in the `--from-history` output\n",
    "    + you will lose any `pip` install values this way\n",
    "+ if you will be mostly within one OS, you may want the more explicit output of the flag-less command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini exercise: creating an `environment.yml` file</div>**\n",
    "\n",
    "run the commands\n",
    "\n",
    "```sh\n",
    "# if you haven't already\n",
    "conda activate l33tmode\n",
    "conda env export > environment.yml\n",
    "cat environment.yml\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this `environment.yml` file can be sent to other users or re-used by you on future `ec2` instances to create a new but completely identical environment:\n",
    "\n",
    "```sh\n",
    "conda env create --name l33tmode-clone-2 --file environment.yml\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `conda env export` vs. `conda list -e`: basic differences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the differences between these two are minor but important:\n",
    "\n",
    "| command | os- and arch-specific | includes exact versions | includes `pip` files |\n",
    "|-|-|-|-|\n",
    "| `conda list -e` | yes* | yes | no |\n",
    "| `conda env export` | yes* | yes | yes |\n",
    "| `conda env export --from-history` | no | no | no |\n",
    "\n",
    "*: technically these may work on other os and arch, but it is not guaranteed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### specify and recreate with `pip freeze`\n",
    "\n",
    "the `environment.yml` and `spec-file.txt` files you created above can be read by `conda`, but not by other `python` virtual environment or package managers. there is a format for specifying packages to install that is much more broadly recognized in the `python` world -- a `requirements.txt` file. this is the sort of file you could use to install all packages using the basic `pip` package installer, for example.\n",
    "\n",
    "to create a `requirements.txt` file, you can simply execute\n",
    "\n",
    "```sh\n",
    "pip freeze > requirements.txt\n",
    "\n",
    "# look at the contents\n",
    "cat requirements.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you can use this on any system which has `pip` installed to install the listed packages into the active environment with\n",
    "\n",
    "```sh\n",
    "pip install -r requirements.txt\n",
    "```\n",
    "\n",
    "note that the packages installed this way *are not* `conda` packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div align=\"center\"><img src=\"https://i.ytimg.com/vi/BX1EIlwtQvU/maxresdefault.jpg\" width=\"800px\"></div>\n",
    "\n",
    "# END OF LECTURE\n",
    "\n",
    "next lecture: [environment management pt. 2: `docker`](006_environments_2_docker.ipynb)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}