10c_fp16_jcat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training in mixed precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from exp.nb_10b import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A little bit of theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=1318)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuing the documentation on the fastai_v1 development here is a brief piece about mixed precision training. A very nice and clear introduction to it is [this video from NVIDIA](http://on-demand.gputechconf.com/gtc/2018/video/S81012/).\n",
    "\n",
    "### What's half precision?\n",
    "In neural nets, all the computations are usually done in single precision, which means all the floats in all the arrays that represent inputs, activations, weights... are 32-bit floats (FP32 in the rest of this post). An idea to reduce memory usage (and avoid those annoying cuda errors) has been to try and do the same thing in half-precision, which means using 16-bits floats (or FP16 in the rest of this post). By definition, they take half the space in RAM, and in theory could allow you to double the size of your model and double your batch size.\n",
    "\n",
    "Another very nice feature is that NVIDIA developed its latest GPUs (the Volta generation) to take fully advantage of half-precision tensors. Basically, if you give half-precision tensors to those, they'll stack them so that each core can do more operations at the same time, and theoretically gives an 8x speed-up (sadly, just in theory).\n",
    "\n",
    "So training at half precision is better for your memory usage, way faster if you have a Volta GPU (still a tiny bit faster if you don't since the computations are easiest). How do we do it? Super easily in pytorch, we just have to put .half() everywhere: on the inputs of our model and all the parameters. Problem is that you usually won't see the same accuracy in the end (so it happens sometimes) because half-precision is... well... not as precise ;).\n",
    "\n",
    "### Problems with half-precision:\n",
    "To understand the problems with half precision, let's look briefly at what an FP16 looks like (more information [here](https://en.wikipedia.org/wiki/Half-precision_floating-point_format)).\n",
    "\n",
    "![half float](images/half.png)\n",
    "\n",
    "The sign bit gives us +1 or -1, then we have 5 bits to code an exponent between -14 and 15, while the fraction part has the remaining 10 bits. Compared to FP32, we have a smaller range of possible values (2e-14 to 2e15 roughly, compared to 2e-126 to 2e127 for FP32) but also a smaller *offset*.\n",
    "\n",
    "For instance, between 1 and 2, the FP16 format only represents the number 1, 1+2e-10, 1+2*2e-10... which means that 1 + 0.0001 = 1 in half precision. That's what will cause a certain numbers of problems, specifically three that can occur and mess up your training.\n",
    "1. The weight update is imprecise: inside your optimizer, you basically do w = w - lr * w.grad for each weight of your network. The problem in performing this operation in half precision is that very often, w.grad is several orders of magnitude below w, and the learning rate is also small. The situation where w=1 and lr*w.grad is 0.0001 (or lower) is therefore very common, but the update doesn't do anything in those cases.\n",
    "2. Your gradients can underflow. In FP16, your gradients can easily be replaced by 0 because they are too low.\n",
    "3. Your activations or loss can overflow. The opposite problem from the gradients: it's easier to hit nan (or infinity) in FP16 precision, and your training might more easily diverge.\n",
    "\n",
    "### The solution: mixed precision training\n",
    "\n",
    "To address those three problems, we don't fully train in FP16 precision. As the name mixed training implies, some of the operations will be done in FP16, others in FP32. This is mainly to take care of the first problem listed above. For the next two there are additional tricks.\n",
    "\n",
    "The main idea is that we want to do the forward pass and the gradient computation in half precision (to go fast) but the update in single precision (to be more precise). It's okay if w and grad are both half floats, but when we do the operation w = w - lr * grad, we need to compute it in FP32. That way our 1 + 0.0001 is going to be 1.0001. \n",
    "\n",
    "This is why we keep a copy of the weights in FP32 (called master model). Then, our training loop will look like:\n",
    "1. compute the output with the FP16 model, then the loss\n",
    "2. back-propagate the gradients in half-precision.\n",
    "3. copy the gradients in FP32 precision\n",
    "4. do the update on the master model (in FP32 precision)\n",
    "5. copy the master model in the FP16 model.\n",
    "\n",
    "Note that we lose precision during step 5, and that the 1.0001 in one of the weights will go back to 1. But if the next update corresponds to add 0.0001 again, since the optimizer step is done on the master model, the 1.0001 will become 1.0002 and if we eventually go like this up to 1.0005, the FP16 model will be able to tell the difference.\n",
    "\n",
    "That takes care of problem 1. For the second problem, we use something called gradient scaling: to avoid the gradients getting zeroed by the FP16 precision, we multiply the loss by a scale factor (scale=512 for instance). That way we can push the gradients to the right in the next figure, and have them not become zero. Note that scaling the *loss function* also scales the *gradients*\n",
    "\n",
    "![half float representation](images/half_representation.png)\n",
    "\n",
    "Of course we don't want those 512-scaled gradients to be in the weight update, so after converting them into FP32, we can divide them by this scale factor (once they have no risks of becoming 0). This changes the loop to:\n",
    "1. compute the output with the FP16 model, then the loss.\n",
    "2. multiply the loss by scale then back-propagate the gradients in half-precision.\n",
    "3. copy the gradients in FP32 precision then divide them by scale.\n",
    "4. do the update on the master model (in FP32 precision).\n",
    "5. copy the master model in the FP16 model.\n",
    "\n",
    "For the last problem, the tricks offered by NVIDIA are to leave the batchnorm layers in single precision (they don't have many weights so it's not a big memory challenge) and compute the loss in single precision (which means converting the last output of the model in single precision before passing it to the loss).\n",
    "\n",
    "![Mixed precision training](images/Mixed_precision.jpeg)\n",
    "\n",
    "Implementing all of this in the new callback system is surprisingly easy, let's dig into this!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Util functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before going in the main `Callback` we will need some helper functions. We will refactor using the [APEX library](https://github.com/NVIDIA/apex) util functions. The python-only build is enough for what we will use here if you don't manage to do the CUDA/C++ installation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export \n",
    "import apex.fp16_utils as fp16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting the model to FP16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=1425)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need a function to convert all the layers of the model to FP16 precision except the BatchNorm-like layers (since those need to be done in FP32 precision to be stable). We do this in two steps: first we convert the model to FP16, then we loop over all the layers and put them back to FP32 if they are a BatchNorm layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "bn_types = (nn.BatchNorm1d, nn.BatchNorm2d, nn.BatchNorm3d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bn_to_float(model):\n",
    "    # edge case where the model consists of only a batchnorm\n",
    "    if isinstance(model, bn_types): model.float()\n",
    "    # normal case\n",
    "    for child in model.children():  bn_to_float(child)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_to_half(model):\n",
    "    model = model.half()\n",
    "    return bn_to_float(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = nn.Sequential(nn.Linear(10,30), nn.BatchNorm1d(30), nn.Linear(30,2)).cuda()\n",
    "model = model_to_half(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_weights(model):\n",
    "    for i,t in enumerate([torch.float16, torch.float32, torch.float16]):\n",
    "        assert model[i].weight.dtype == t\n",
    "        assert model[i].bias.dtype   == t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_weights(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Apex, the function that does this for us is `convert_network`. We can use it to put the model in FP16 or back to FP32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = nn.Sequential(nn.Linear(10,30), nn.BatchNorm1d(30), nn.Linear(30,2)).cuda()\n",
    "model = fp16.convert_network(model, torch.float16)\n",
    "check_weights(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the master copy of the parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From our model parameters (mostly in FP16), we'll want to create a copy in FP32 (master parameters) that we will use for the step in the optimizer. Optionally, we concatenate all the parameters to do one flat big tensor, which can make that step a little bit faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn.utils import parameters_to_vector\n",
    "\n",
    "def get_master(model, flat_master=False):\n",
    "    model_params = [param for param in model.parameters() if param.requires_grad]\n",
    "    if flat_master:\n",
    "        master_param = parameters_to_vector([param.data.float() for param in model_params])\n",
    "        master_param = torch.nn.Parameter(master_param, requires_grad=True)\n",
    "        if master_param.grad is None: master_param.grad = master_param.new(*master_param.size())\n",
    "        return model_params, [master_param]\n",
    "    else:\n",
    "        # clone each parameter and detach the clone from the computation graph of the original parameters\n",
    "        master_params = [param.clone().float().detach() for param in model_params]\n",
    "        # put the cloned parameters into their own computation graph\n",
    "        for param in master_params: param.requires_grad_(True)\n",
    "        return model_params, master_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The util function from Apex to do this is `prep_param_lists`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# these commands accomplish the same task \n",
    "model_p,master_p = get_master(model)\n",
    "model_p1,master_p1 = fp16.prep_param_lists(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to check properties of two lists\n",
    "def same_lists(ps1, ps2):\n",
    "    assert len(ps1) == len(ps2)\n",
    "    for (p1,p2) in zip(ps1,ps2): \n",
    "        assert p1.requires_grad == p2.requires_grad\n",
    "        assert torch.allclose(p1.data.float(), p2.data.float())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "same_lists(model_p,model_p1)\n",
    "same_lists(model_p,master_p)\n",
    "same_lists(master_p,master_p1)\n",
    "same_lists(model_p1,master_p1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can't use flat_master when there is a mix of FP32 and FP16 parameters (like batchnorm here)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model1 = nn.Sequential(nn.Linear(10,30), nn.Linear(30,2)).cuda()\n",
    "model1 = fp16.convert_network(model1, torch.float16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_p,master_p = get_master(model1, flat_master=True)\n",
    "model_p1,master_p1 = fp16.prep_param_lists(model1, flat_master=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "same_lists(model_p,model_p1)\n",
    "same_lists(master_p,master_p1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(master_p[0]) == 10*30 + 30 + 30*2 + 2\n",
    "assert len(master_p1[0]) == 10*30 + 30 + 30*2 + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(master_p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The thing is that we don't always want all the parameters of our model in the same parameter group, because we might:\n",
    "- want to do transfer learning and freeze some layers\n",
    "- apply discriminative learning rates\n",
    "- don't apply weight decay to some layers (like BatchNorm) or the bias terms\n",
    "\n",
    "So we actually need a function that splits the parameters of an optimizer (and not a model) according to the right parameter groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_master(opt, flat_master=False):\n",
    "    \n",
    "    # create a list of parameter groups\n",
    "    model_params = [[param for param in pg if param.requires_grad] for pg in opt.param_groups]\n",
    "    \n",
    "    if flat_master:\n",
    "        master_params = []\n",
    "        for pg in model_params:\n",
    "            # convert parameters in each group to a list of floats, then create a parameters vector from the list\n",
    "            mp = parameters_to_vector([param.data.float() for param in pg])\n",
    "            # initialize the Parameter object to the parameters vector and put it on the computations graph\n",
    "            mp = torch.nn.Parameter(mp, requires_grad=True)\n",
    "            # if there is no grad property, then initialize it to template of zeros of the right size\n",
    "            if mp.grad is None: mp.grad = mp.new(*mp.size())\n",
    "            # add the parameter group to the master_params list\n",
    "            master_params.append(mp)\n",
    "    else:\n",
    "        # create the master parameter list\n",
    "        master_params = [[param.clone().float().detach() for param in pg] for pg in model_params]\n",
    "        # add all the parameters on the computation graph\n",
    "        for pg in master_params:\n",
    "            for param in pg: param.requires_grad_(True)\n",
    "    return model_params, master_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the backward pass, all gradients must be copied to the master params before the optimizer step can be done in FP32. We need a function for that (with a bit of adjustement if we have flat master)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_master_grads(model_params, master_params, flat_master:bool=False)->None:\n",
    "    if flat_master:\n",
    "        if master_params[0].grad is None: master_params[0].grad = master_params[0].data.new(*master_params[0].data.size())\n",
    "        master_params[0].grad.data.copy_(parameters_to_vector([p.grad.data.float() for p in model_params]))\n",
    "    else:\n",
    "        for model, master in zip(model_params, master_params):\n",
    "            if model.grad is not None:\n",
    "                if master.grad is None: master.grad = master.data.new(*master.data.size())\n",
    "                master.grad.data.copy_(model.grad.data)\n",
    "            else: master.grad = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corresponding function in the Apex utils is `model_grads_to_master_grads`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.randn(20,10).half().cuda()\n",
    "z = model(x)\n",
    "loss = F.cross_entropy(z, torch.randint(0, 2, (20,)).cuda())\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_master_grads(model_p, master_p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_grads(m1, m2):\n",
    "    for p1,p2 in zip(m1,m2): \n",
    "        if p1.grad is None: assert p2.grad is None\n",
    "        else: assert torch.allclose(p1.grad.data, p2.grad.data) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_grads(model_p, master_p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "fp16.model_grads_to_master_grads(model_p, master_p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_grads(model_p, master_p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy the master params to the model params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the step, we need to copy back the master parameters to the model parameters for the next update."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch._utils import _unflatten_dense_tensors\n",
    "\n",
    "def to_model_params(model_params, master_params, flat_master:bool=False)->None:\n",
    "    if flat_master:\n",
    "        for model, master in zip(model_params, _unflatten_dense_tensors(master_params[0].data, model_params)):\n",
    "            model.data.copy_(master)\n",
    "    else:\n",
    "        for model, master in zip(model_params, master_params): model.data.copy_(master.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corresponding function in Apex is `master_params_to_model_params`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### But we need to handle param groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The thing is that we don't always want all the parameters of our model in the same parameter group, because we might:\n",
    "- want to do transfer learning and freeze some layers\n",
    "- apply discriminative learning rates\n",
    "- don't apply weight decay to some layers (like BatchNorm) or the bias terms\n",
    "\n",
    "So we actually need a function that splits the parameters of an optimizer (and not a model) according to the right parameter groups and the following functions need to handle lists of lists of parameters (one list of each param group in `model_pgs` and `master_pgs`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export \n",
    "def get_master(opt, flat_master=False):\n",
    "    model_pgs = [[param for param in pg if param.requires_grad] for pg in opt.param_groups]\n",
    "    if flat_master:\n",
    "        master_pgs = []\n",
    "        for pg in model_pgs:\n",
    "            mp = parameters_to_vector([param.data.float() for param in pg])\n",
    "            mp = torch.nn.Parameter(mp, requires_grad=True)\n",
    "            if mp.grad is None: mp.grad = mp.new(*mp.size())\n",
    "            master_pgs.append([mp])\n",
    "    else:\n",
    "        master_pgs = [[param.clone().float().detach() for param in pg] for pg in model_pgs]\n",
    "        for pg in master_pgs:\n",
    "            for param in pg: param.requires_grad_(True)\n",
    "    return model_pgs, master_pgs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export \n",
    "def to_master_grads(model_pgs, master_pgs, flat_master:bool=False)->None:\n",
    "    for (model_params,master_params) in zip(model_pgs,master_pgs):\n",
    "        fp16.model_grads_to_master_grads(model_params, master_params, flat_master=flat_master)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export \n",
    "def to_model_params(model_pgs, master_pgs, flat_master:bool=False)->None:\n",
    "    for (model_params,master_params) in zip(model_pgs,master_pgs):\n",
    "        fp16.master_params_to_model_params(model_params, master_params, flat_master=flat_master)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The main Callback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=1452)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MixedPrecision(Callback):\n",
    "    _order = 99\n",
    "    def __init__(self, loss_scale=512, flat_master=False):\n",
    "        assert torch.backends.cudnn.enabled, \"Mixed precision training requires cudnn.\"\n",
    "        self.loss_scale,self.flat_master = loss_scale,flat_master\n",
    "\n",
    "    def begin_fit(self):\n",
    "        self.run.model = fp16.convert_network(self.model, dtype=torch.float16)\n",
    "        self.model_pgs, self.master_pgs = get_master(self.opt, self.flat_master)\n",
    "        #Changes the optimizer so that the optimization step is done in FP32.\n",
    "        self.run.opt.param_groups = self.master_pgs #Put those param groups inside our runner.\n",
    "        \n",
    "    def after_fit(self): self.model.float()\n",
    "\n",
    "    def begin_batch(self): self.run.xb = self.run.xb.half() #Put the inputs to half precision\n",
    "    def after_pred(self):  self.run.pred = self.run.pred.float() #Compute the loss in FP32\n",
    "    def after_loss(self):  self.run.loss *= self.loss_scale #Loss scaling to avoid gradient underflow\n",
    "\n",
    "    def after_backward(self):\n",
    "        #Copy the gradients to master and unscale\n",
    "        to_master_grads(self.model_pgs, self.master_pgs, self.flat_master)\n",
    "        for master_params in self.master_pgs:\n",
    "            for param in master_params:\n",
    "                if param.grad is not None: param.grad.div_(self.loss_scale)\n",
    "\n",
    "    def after_step(self):\n",
    "        #Zero the gradients of the model since the optimizer is disconnected.\n",
    "        self.model.zero_grad()\n",
    "        #Update the params from master to model.\n",
    "        to_model_params(self.model_pgs, self.master_pgs, self.flat_master)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's test this on Imagenette"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = datasets.untar_data(datasets.URLs.IMAGENETTE_160)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfms = [make_rgb, ResizeFixed(128), to_byte_tensor, to_float_tensor]\n",
    "bs = 64\n",
    "\n",
    "il = ImageList.from_files(path, tfms=tfms)\n",
    "sd = SplitData.split_by_func(il, partial(grandparent_splitter, valid_name='val'))\n",
    "ll = label_by_func(sd, parent_labeler, proc_y=CategoryProcessor())\n",
    "data = ll.to_databunch(bs, c_in=3, c_out=10, num_workers=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfs = [32,64,128,256,512]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_learner(nfs, data, lr, layer, loss_func=F.cross_entropy,\n",
    "                cb_funcs=None, opt_func=adam_opt(), **kwargs):\n",
    "    model = get_cnn_model(data, nfs, layer, **kwargs)\n",
    "    init_cnn(model)\n",
    "    return Learner(model, data, loss_func, lr=lr, cb_funcs=cb_funcs, opt_func=opt_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training without mixed precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "cbfs = [partial(AvgStatsCallback,accuracy),\n",
    "        ProgressCallback,\n",
    "        CudaCallback,\n",
    "        partial(BatchTransformXCallback, norm_imagenette)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = get_learner(nfs, data, 1e-2, conv_layer, cb_funcs=cbfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.872608</td>\n",
       "      <td>0.383744</td>\n",
       "      <td>1.453536</td>\n",
       "      <td>0.544000</td>\n",
       "      <td>00:47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.872766</td>\n",
       "      <td>0.397394</td>\n",
       "      <td>1.813968</td>\n",
       "      <td>0.518000</td>\n",
       "      <td>00:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.905525</td>\n",
       "      <td>0.389251</td>\n",
       "      <td>1.418059</td>\n",
       "      <td>0.522000</td>\n",
       "      <td>00:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.972752</td>\n",
       "      <td>0.366062</td>\n",
       "      <td>1.556963</td>\n",
       "      <td>0.484000</td>\n",
       "      <td>00:44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.896609</td>\n",
       "      <td>0.376842</td>\n",
       "      <td>1.608963</td>\n",
       "      <td>0.464000</td>\n",
       "      <td>00:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.854175</td>\n",
       "      <td>0.406235</td>\n",
       "      <td>1.945591</td>\n",
       "      <td>0.456000</td>\n",
       "      <td>00:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.878600</td>\n",
       "      <td>0.394525</td>\n",
       "      <td>1.598111</td>\n",
       "      <td>0.476000</td>\n",
       "      <td>00:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.929414</td>\n",
       "      <td>0.369319</td>\n",
       "      <td>1.776040</td>\n",
       "      <td>0.458000</td>\n",
       "      <td>00:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.864859</td>\n",
       "      <td>0.393129</td>\n",
       "      <td>1.517715</td>\n",
       "      <td>0.498000</td>\n",
       "      <td>00:42</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.947703</td>\n",
       "      <td>0.373972</td>\n",
       "      <td>2.158857</td>\n",
       "      <td>0.494000</td>\n",
       "      <td>00:42</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.863666</td>\n",
       "      <td>0.391810</td>\n",
       "      <td>1.392280</td>\n",
       "      <td>0.532000</td>\n",
       "      <td>00:42</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for jj in range(10):\n",
    "    learn = get_learner(nfs, data, 1e-2, conv_layer, cb_funcs=cbfs)\n",
    "    learn.fit(1)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training with mixed precision: why isn't it faster than training without mixed precision? Because I'm not using the GPU in Windows 10?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "cbfs = [partial(AvgStatsCallback,accuracy),\n",
    "        CudaCallback,\n",
    "        ProgressCallback,\n",
    "        partial(BatchTransformXCallback, norm_imagenette),\n",
    "        MixedPrecision]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = get_learner(nfs, data, 1e-2, conv_layer, cb_funcs=cbfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_eq(next(learn.model.parameters()).type(), 'torch.cuda.FloatTensor')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.920720</td>\n",
       "      <td>0.380952</td>\n",
       "      <td>1.406080</td>\n",
       "      <td>0.504000</td>\n",
       "      <td>00:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.879001</td>\n",
       "      <td>0.392043</td>\n",
       "      <td>1.776441</td>\n",
       "      <td>0.494000</td>\n",
       "      <td>00:47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "RuntimeError",
     "evalue": "CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 8.00 GiB total capacity; 297.02 MiB already allocated; 20.42 MiB free; 46.98 MiB cached)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-40-3a3366754326>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mjj\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m     \u001b[0mlearn\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_learner\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnfs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m1e-2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mconv_layer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcb_funcs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcbfs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m     \u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\fastai\\course-v3\\nbs\\dl2\\exp\\nb_09b.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, epochs, cbs, reset_opt)\u001b[0m\n\u001b[0;32m     70\u001b[0m             \u001b[1;32mfor\u001b[0m \u001b[0mepoch\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepochs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     71\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdo_begin_epoch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepoch\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 72\u001b[1;33m                 \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'begin_epoch'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mall_batches\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     73\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     74\u001b[0m                 \u001b[1;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\fastai\\course-v3\\nbs\\dl2\\exp\\nb_09b.py\u001b[0m in \u001b[0;36mall_batches\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     49\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0miters\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     50\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 51\u001b[1;33m             \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mxb\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0myb\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mone_batch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mxb\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0myb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     52\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mCancelEpochException\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'after_cancel_epoch'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     53\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\fastai\\course-v3\\nbs\\dl2\\exp\\nb_09b.py\u001b[0m in \u001b[0;36mone_batch\u001b[1;34m(self, i, xb, yb)\u001b[0m\n\u001b[0;32m     40\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mloss\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mloss_func\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpred\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0myb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m;\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'after_loss'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     41\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0min_train\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;32mreturn\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 42\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mloss\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m;\u001b[0m                           \u001b[0mself\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'after_backward'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     43\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m;\u001b[0m                                \u001b[0mself\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'after_step'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     44\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\torch\\tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[1;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[0;32m    116\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[1;33m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[1;33m.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    117\u001b[0m         \"\"\"\n\u001b[1;32m--> 118\u001b[1;33m         \u001b[0mtorch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    119\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    120\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\torch\\autograd\\__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[1;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[0;32m     91\u001b[0m     Variable._execution_engine.run_backward(\n\u001b[0;32m     92\u001b[0m         \u001b[0mtensors\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 93\u001b[1;33m         allow_unreachable=True)  # allow_unreachable flag\n\u001b[0m\u001b[0;32m     94\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     95\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mRuntimeError\u001b[0m: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 8.00 GiB total capacity; 297.02 MiB already allocated; 20.42 MiB free; 46.98 MiB cached)"
     ]
    }
   ],
   "source": [
    "for jj in range(10):\n",
    "    learn = get_learner(nfs, data, 1e-2, conv_layer, cb_funcs=cbfs)\n",
    "    learn.fit(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dynamic loss scaling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only annoying thing with the previous implementation of mixed precision training is that it introduces one new hyper-parameter to tune, the value of the loss scaling. Fortunately for us, there is a way around this. We want the loss scaling to be as high as possible so that our gradients can use the whole range of representation, so let's first try a really high value. In all likelihood, this will cause our gradients or our loss to overflow, and we will try again with half that big value, and again, until we get to the largest loss scale possible that doesn't make our gradients overflow.\n",
    "\n",
    "This value will be perfectly fitted to our model and can continue to be dynamically adjusted as the training goes, if it's still too high, by just halving it each time we overflow. After a while though, training will converge and gradients will start to get smaller, so we also need a mechanism to get this dynamic loss scale larger if it's safe to do so. The strategy used in the Apex library is to multiply the loss scale by 2 each time we had a given number of iterations without overflowing.\n",
    "\n",
    "To check if the gradients have overflowed, we check their sum (computed in FP32). If one term is nan, the sum will be nan. Interestingly, on the GPU, it's faster than checking `torch.isnan`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=1472)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export \n",
    "def test_overflow(x):\n",
    "    s = float(x.float().sum())\n",
    "    return (s == float('inf') or s == float('-inf') or s != s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.randn(512,1024).cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_overflow(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[123,145] = float('inf')\n",
    "test_overflow(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Compare test_overflow vs. PyTorch way: PyTorch is faster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "252 µs ± 116 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit test_overflow(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "214 µs ± 11.7 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit torch.isnan(x).any().item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "88 µs ± 1.63 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit torch.isnan(x).any()#.item() -- don't need to convert to boolean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can use it in the following function that checks for gradient overflow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export \n",
    "def grad_overflow(param_groups):\n",
    "    for group in param_groups:\n",
    "        for p in group:\n",
    "            if p.grad is not None:\n",
    "                s = float(p.grad.data.float().sum())\n",
    "                if s == float('inf') or s == float('-inf') or s != s: return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can write a new version of the `Callback` that handles dynamic loss scaling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export \n",
    "class MixedPrecision(Callback):\n",
    "    _order = 99\n",
    "    def __init__(self, loss_scale=512, flat_master=False, dynamic=True, max_loss_scale=2.**24, div_factor=2.,\n",
    "                 scale_wait=500):\n",
    "        assert torch.backends.cudnn.enabled, \"Mixed precision training requires cudnn.\"\n",
    "        self.flat_master,self.dynamic,self.max_loss_scale = flat_master,dynamic,max_loss_scale\n",
    "        self.div_factor,self.scale_wait = div_factor,scale_wait\n",
    "        self.loss_scale = max_loss_scale if dynamic else loss_scale\n",
    "\n",
    "    def begin_fit(self):\n",
    "        self.run.model = fp16.convert_network(self.model, dtype=torch.float16)\n",
    "        self.model_pgs, self.master_pgs = get_master(self.opt, self.flat_master)\n",
    "        #Changes the optimizer so that the optimization step is done in FP32.\n",
    "        self.run.opt.param_groups = self.master_pgs #Put those param groups inside our runner.\n",
    "        if self.dynamic: self.count = 0\n",
    "\n",
    "    def begin_batch(self): self.run.xb = self.run.xb.half() #Put the inputs to half precision\n",
    "    def after_pred(self):  self.run.pred = self.run.pred.float() #Compute the loss in FP32\n",
    "    def after_loss(self):  \n",
    "        if self.in_train: self.run.loss *= self.loss_scale #Loss scaling to avoid gradient underflow\n",
    "\n",
    "    def after_backward(self):\n",
    "        #First, check for an overflow\n",
    "        if self.dynamic and grad_overflow(self.model_pgs):\n",
    "            #Divide the loss scale by div_factor, zero the grad (after_step will be skipped)\n",
    "            self.loss_scale /= self.div_factor\n",
    "            self.model.zero_grad()\n",
    "            return True #skip step and zero_grad\n",
    "        #Copy the gradients to master and unscale\n",
    "        to_master_grads(self.model_pgs, self.master_pgs, self.flat_master)\n",
    "        for master_params in self.master_pgs:\n",
    "            for param in master_params:\n",
    "                if param.grad is not None: param.grad.div_(self.loss_scale)\n",
    "        #Check if it's been long enough without overflow\n",
    "        if self.dynamic:\n",
    "            self.count += 1\n",
    "            if self.count == self.scale_wait:\n",
    "                self.count = 0\n",
    "                self.loss_scale *= self.div_factor\n",
    "\n",
    "    def after_step(self):\n",
    "        #Zero the gradients of the model since the optimizer is disconnected.\n",
    "        self.model.zero_grad()\n",
    "        #Update the params from master to model.\n",
    "        to_model_params(self.model_pgs, self.master_pgs, self.flat_master)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "cbfs = [partial(AvgStatsCallback,accuracy),\n",
    "        CudaCallback,\n",
    "        ProgressCallback,\n",
    "        partial(BatchTransformXCallback, norm_imagenette),\n",
    "        MixedPrecision]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = get_learner(nfs, data, 1e-2, conv_layer, cb_funcs=cbfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.012661</td>\n",
       "      <td>0.348302</td>\n",
       "      <td>1.668833</td>\n",
       "      <td>0.438000</td>\n",
       "      <td>00:44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss scale used is way higher than our previous number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65536.0"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.cbs[-1].loss_scale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.965432</td>\n",
       "      <td>0.355824</td>\n",
       "      <td>1.594996</td>\n",
       "      <td>0.484000</td>\n",
       "      <td>00:45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65536.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.934876</td>\n",
       "      <td>0.379246</td>\n",
       "      <td>1.487156</td>\n",
       "      <td>0.498000</td>\n",
       "      <td>00:44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16384.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.992089</td>\n",
       "      <td>0.349853</td>\n",
       "      <td>2.006961</td>\n",
       "      <td>0.428000</td>\n",
       "      <td>00:44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65536.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.945281</td>\n",
       "      <td>0.362339</td>\n",
       "      <td>2.005281</td>\n",
       "      <td>0.358000</td>\n",
       "      <td>00:44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65536.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.882761</td>\n",
       "      <td>0.391733</td>\n",
       "      <td>1.424370</td>\n",
       "      <td>0.522000</td>\n",
       "      <td>00:44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65536.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.952187</td>\n",
       "      <td>0.375601</td>\n",
       "      <td>1.522131</td>\n",
       "      <td>0.508000</td>\n",
       "      <td>00:45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32768.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.961801</td>\n",
       "      <td>0.370482</td>\n",
       "      <td>1.602143</td>\n",
       "      <td>0.484000</td>\n",
       "      <td>00:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32768.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.885039</td>\n",
       "      <td>0.384210</td>\n",
       "      <td>1.472521</td>\n",
       "      <td>0.474000</td>\n",
       "      <td>00:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32768.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.977835</td>\n",
       "      <td>0.358616</td>\n",
       "      <td>1.551121</td>\n",
       "      <td>0.462000</td>\n",
       "      <td>00:45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65536.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.887953</td>\n",
       "      <td>0.396541</td>\n",
       "      <td>1.556678</td>\n",
       "      <td>0.494000</td>\n",
       "      <td>00:45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16384.0\n"
     ]
    }
   ],
   "source": [
    "for jj in range(10):\n",
    "    learn = get_learner(nfs, data, 1e-2, conv_layer, cb_funcs=cbfs)\n",
    "    learn.fit(1)\n",
    "    print(learn.cbs[-1].loss_scale)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !./notebook2script.py 10c_fp16.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python notebook2script.py 10c_fp16.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}