trends/data_2/CapsuleNetwork_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6hnVCrd9wXbg"
   },
   "source": [
    "# Capsule Networks HW"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This homework is strongly based on https://github.com/higgsfield/Capsule-Network-Tutorial/blob/master/Capsule%20Network.ipynb. \n",
    "\n",
    "In case you want to check CapsNet on MNIST feel free to do that (link above)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6QkLRp28woWk"
   },
   "source": [
    "After you have uploaded this notebook to Google Collaboratory you may start running it (if you have fast GPU, you can run it locally). \n",
    "\n",
    "First of all you need to install pytorch. The code below does that for you.\n",
    "\n",
    "Every 24 hours Google Collaboratory reboots. Make sure you either run operations that require less than 24 hours of computation or you log your progress so that you can avoid starting from scratch every time.\n",
    "\n",
    "The notebooks are stored in your root Google Drive folder and you can save other files there as well (requires some googling)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "n1XrJ3IcE4DV"
   },
   "outputs": [],
   "source": [
    "# http://pytorch.org/\n",
    "from os import path\n",
    "from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
    "platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
    "\n",
    "accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'\n",
    "\n",
    "!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.3.0.post4-{platform}-linux_x86_64.whl torchvision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "jFNhTBo-8S_F"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "from torch.optim import Adam\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "from time import time\n",
    "\n",
    "USE_CUDA = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6nQlfiY0w1Gn"
   },
   "source": [
    "## CIFAR-10 data loader/generator.\n",
    "\n",
    "Code below setups image generators from folder './data' (this folder is not saved and will be erased every 24 hours). \n",
    "\n",
    "Normalization values for CIFAR10 are taken from pytorch website (usual normalization values for the task)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "OdImV-LJwuOL"
   },
   "outputs": [],
   "source": [
    "class Cifar10:\n",
    "    def __init__(self, batch_size):\n",
    "        dataset_transform = transforms.Compose([\n",
    "                       transforms.ToTensor(),\n",
    "                       transforms.Normalize((0.4914, 0.4822, 0.4465), (0.247, 0.243, 0.261))\n",
    "                   ])\n",
    "\n",
    "        train_dataset = datasets.CIFAR10('./data', train=True, download=True, transform=dataset_transform)\n",
    "        test_dataset = datasets.CIFAR10('./data', train=False, download=True, transform=dataset_transform)\n",
    "        \n",
    "        self.train_loader  = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "        self.test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YJry2wSsHUa8"
   },
   "source": [
    "## Network\n",
    "\n",
    "Recall the architecture of CapsNet. This tutorial walks you through building process of it. Note that actual values of parameters such as \"number of capsules\", \"number of filters in the first layer\" etc. are not taken from MNIST implementation in the original paper, but instead from CIFAR10 implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j0Z5bAZezXuY"
   },
   "source": [
    "![alt text](https://cdn-images-1.medium.com/max/2000/1*uItEGzY1I9NK6hl1u4hPYg.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NhW37DmbH-7N"
   },
   "source": [
    "### Pre-capsule layer\n",
    "\n",
    "This is a usual convolution layer that extracts basic features from images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "N1afD72K8S_X"
   },
   "outputs": [],
   "source": [
    "class ConvLayer(nn.Module):\n",
    "    def __init__(self, in_channels=3, out_channels=256, kernel_size=9):\n",
    "        super(ConvLayer, self).__init__()\n",
    "\n",
    "        self.conv = nn.Conv2d(in_channels=in_channels,\n",
    "                               out_channels=out_channels,\n",
    "                               kernel_size=kernel_size,\n",
    "                               stride=1\n",
    "                             )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return F.relu(self.conv(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tB21rTXeICUI"
   },
   "source": [
    "### First capsule layer (PrimaryCaps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PiDlqFB94Cad"
   },
   "source": [
    "This is the second layer of the network and the first one which contains capsules (recall that capsules are just groups of neurons).\n",
    "\n",
    "The squash operation is the following one:\n",
    "\n",
    "\\begin{align}\n",
    "v_j & = \\frac{(\\|s_j\\|^2)}{(1 + \\|s_j\\|^2)} \\frac{s_j}{\\|s_j\\|}\\\\\n",
    "\\end{align}\n",
    "\n",
    "It takes a vector s_j as input, normalizes it to unit norm and then adds some non-linearity so that large vectors become close to 1 and small vectors close to zero. Recall that it is needed to enforce the property of v_j's norm being the probability (or certainty) that object is detected by the capsule v_j."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "55zyRgn18S_c"
   },
   "outputs": [],
   "source": [
    "class PrimaryCaps(nn.Module):\n",
    "    def __init__(self, num_capsules=8, in_channels=256, out_channels=64, kernel_size=9):\n",
    "        super(PrimaryCaps, self).__init__()\n",
    "\n",
    "        self.capsules = nn.ModuleList([\n",
    "            nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=2, padding=0) \n",
    "                          for _ in range(num_capsules)])\n",
    "    \n",
    "    def forward(self, x):\n",
    "        u = [capsule(x) for capsule in self.capsules]\n",
    "        u = torch.stack(u, dim=1)\n",
    "        u = u.view(x.size(0), 64 * 8 * 8, -1)\n",
    "        return self.squash(u)\n",
    "    \n",
    "    def squash(self, input_tensor):\n",
    "        squared_norm = (input_tensor ** 2).sum(-1, keepdim=True)\n",
    "        output_tensor = squared_norm *  input_tensor / ((1. + squared_norm) * torch.sqrt(squared_norm))\n",
    "        return output_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mtw2-UyJIHQ4"
   },
   "source": [
    "### Second capsule layer (DigitCaps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_EUmV1wa3-G1"
   },
   "source": [
    "This is the final layer of the network and the one that contains digit-capsules (or in case of CIFAR10 - class-capsules) which predict the class on the image.\n",
    "\n",
    "Below you may see the dynamic routing algorithm from the original paper under the forward section of the layer.\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/2000/1*ukE9EQ6Yd6IPIu1cLJWSEQ.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "Lew0uSA-8S_g"
   },
   "outputs": [],
   "source": [
    "class DigitCaps(nn.Module):\n",
    "    def __init__(self, num_capsules=10, num_routes=64 * 8 * 8, in_channels=8, out_channels=16):\n",
    "        super(DigitCaps, self).__init__()\n",
    "\n",
    "        self.in_channels = in_channels\n",
    "        self.num_routes = num_routes\n",
    "        self.num_capsules = num_capsules\n",
    "\n",
    "        self.W = nn.Parameter(torch.randn(1, num_routes, num_capsules, out_channels, in_channels))\n",
    "\n",
    "    def forward(self, x):\n",
    "        batch_size = x.size(0)\n",
    "        x = torch.stack([x] * self.num_capsules, dim=2).unsqueeze(4)\n",
    "\n",
    "        W = torch.cat([self.W] * batch_size, dim=0)\n",
    "        u_hat = torch.matmul(W, x)\n",
    "\n",
    "        b_ij = Variable(torch.zeros(1, self.num_routes, self.num_capsules, 1))\n",
    "        if USE_CUDA:\n",
    "            b_ij = b_ij.cuda()\n",
    "\n",
    "        num_iterations = 3\n",
    "        for iteration in range(num_iterations):\n",
    "            c_ij = F.softmax(b_ij)\n",
    "            c_ij = torch.cat([c_ij] * batch_size, dim=0).unsqueeze(4)\n",
    "\n",
    "            s_j = (c_ij * u_hat).sum(dim=1, keepdim=True)\n",
    "            v_j = self.squash(s_j)\n",
    "            \n",
    "            if iteration < num_iterations - 1:\n",
    "                a_ij = torch.matmul(u_hat.transpose(3, 4), torch.cat([v_j] * self.num_routes, dim=1))\n",
    "                b_ij = b_ij + a_ij.squeeze(4).mean(dim=0, keepdim=True)\n",
    "\n",
    "        return v_j.squeeze(1)\n",
    "    \n",
    "    def squash(self, input_tensor):\n",
    "        squared_norm = (input_tensor ** 2).sum(-1, keepdim=True)\n",
    "        output_tensor = squared_norm *  input_tensor / ((1. + squared_norm) * torch.sqrt(squared_norm))\n",
    "        return output_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TwrZ5H7qIUWn"
   },
   "source": [
    "### Reconstruction part of network (decoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3vjTImHJ5rQq"
   },
   "source": [
    "This is the second task for the network, namely, to reconstruct the image from the final class-capsules. \n",
    "\n",
    "This is a useful technique of regularization to prevent overfitting and also to enforce the property of capsules representing the 'instantiation parameters' of the object. In other words, final capsule should contain information about the class it predicts and that information (implicitly) may be: rotation angle, distortion, illumination etc.\n",
    "\n",
    "The reconstruction is done by a simple decoder (stack of fully-connected layers). Below is the picture for MNIST.\n",
    "\n",
    "![alt text](https://github.com/raw/deepblacksky/capsnet-tensorflow/master/images/recong.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "Wr4UA8_k8S_l"
   },
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Decoder, self).__init__()\n",
    "        \n",
    "        self.reconstraction_layers = nn.Sequential(\n",
    "            nn.Linear(16 * 10, 512),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(512, 1024),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(1024, 1024*3),\n",
    "            nn.Sigmoid()\n",
    "        )\n",
    "        \n",
    "    def forward(self, x, data):\n",
    "        classes = torch.sqrt((x ** 2).sum(2))\n",
    "        classes = F.softmax(classes)\n",
    "        \n",
    "        _, max_length_indices = classes.max(dim=1)\n",
    "        masked = Variable(torch.sparse.torch.eye(10))\n",
    "        if USE_CUDA:\n",
    "            masked = masked.cuda()\n",
    "        masked = masked.index_select(dim=0, index=F.Variable(max_length_indices.squeeze(1).data))\n",
    "        \n",
    "        reconstructions = self.reconstraction_layers((x * masked[:, :, None, None]).view(x.size(0), -1))\n",
    "        reconstructions = reconstructions.view(-1, 3, 32, 32)\n",
    "        \n",
    "        return reconstructions, masked"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KWHxjIKjIaYg"
   },
   "source": [
    "### Full network (CapsNet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gYDG4nCl7l5Q"
   },
   "source": [
    "This is a final forward pass for the whole network. The only new part here is the custom loss from the original paper.\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/1493/1*y-bVFuiLReqSSdmdZ6wAmA.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "9maeKxss8S_p"
   },
   "outputs": [],
   "source": [
    "class CapsNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CapsNet, self).__init__()\n",
    "        self.conv_layer = ConvLayer()\n",
    "        self.primary_capsules = PrimaryCaps()\n",
    "        self.digit_capsules = DigitCaps()\n",
    "        self.decoder = Decoder()\n",
    "        \n",
    "        self.mse_loss = nn.MSELoss()\n",
    "        \n",
    "    def forward(self, data):\n",
    "        output = self.digit_capsules(self.primary_capsules(self.conv_layer(data)))\n",
    "        reconstructions, masked = self.decoder(output, data)\n",
    "        return output, reconstructions, masked\n",
    "    \n",
    "    def loss(self, data, x, target, reconstructions):\n",
    "        return self.margin_loss(x, target) + self.reconstruction_loss(data, reconstructions)\n",
    "    \n",
    "    def margin_loss(self, x, labels, size_average=True):\n",
    "        batch_size = x.size(0)\n",
    "\n",
    "        v_c = torch.sqrt((x**2).sum(dim=2, keepdim=True))\n",
    "\n",
    "        left = (F.relu(0.9 - v_c)**2).view(batch_size, -1)\n",
    "        right = (F.relu(v_c - 0.1)**2).view(batch_size, -1)\n",
    "\n",
    "        loss = labels * left + 0.5 * (1.0 - labels) * right\n",
    "        loss = loss.sum(dim=1).mean()\n",
    "\n",
    "        return loss\n",
    "    \n",
    "    def reconstruction_loss(self, data, reconstructions):\n",
    "        loss = self.mse_loss(reconstructions.view(reconstructions.size(0), -1), data.view(reconstructions.size(0), -1))\n",
    "        return loss * 0.0005"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dpK01eKY8JHS"
   },
   "source": [
    "Here the model is compiled with Adam optimizer with basic parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "4IZH_lMV8S_s"
   },
   "outputs": [],
   "source": [
    "capsule_net = CapsNet()\n",
    "if USE_CUDA:\n",
    "    capsule_net = capsule_net.cuda()\n",
    "optimizer = Adam(capsule_net.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PF6hxTxDHrsk"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zoBoVHvR8VeX"
   },
   "source": [
    "Note that one epoch takes a lot of time even on GPU, so don't rush and plan everything ahead and try to justify your ideas prior to implementing them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 496,
     "output_extras": [
      {
       "item_id": 2
      },
      {
       "item_id": 4
      },
      {
       "item_id": 21
      }
     ]
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "goSEK6q28S_y",
    "outputId": "96ef6916-bafb-4138-823c-fc578005f88d"
   },
   "outputs": [],
   "source": [
    "batch_size = 100\n",
    "# dataset = Mnist(batch_size)\n",
    "dataset = Cifar10(batch_size)\n",
    "\n",
    "n_epochs = 10\n",
    "\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    ep_start = time()\n",
    "    capsule_net.train()\n",
    "    train_loss = 0\n",
    "    train_accuracy = 0\n",
    "    for batch_id, (data, target) in enumerate(dataset.train_loader):\n",
    "        st = time()\n",
    "\n",
    "        target = torch.sparse.torch.eye(10).index_select(dim=0, index=target)\n",
    "        data, target = Variable(data), Variable(target)\n",
    "\n",
    "        if USE_CUDA:\n",
    "            data, target = data.cuda(), target.cuda()\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        output, reconstructions, masked = capsule_net(data)\n",
    "        loss = capsule_net.loss(data, output, target, reconstructions)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        train_loss += loss.data[0]\n",
    "        \n",
    "        tr_accuracy = sum(np.argmax(masked.data.cpu().numpy(), 1) == \n",
    "                          np.argmax(target.data.cpu().numpy(), 1)) / float(batch_size)\n",
    "        train_accuracy += tr_accuracy\n",
    "        if batch_id % 100 == 0 or batch_id == 499:\n",
    "            print \"train accuracy [batch {}]:\".format(batch_id), tr_accuracy\n",
    "        en = time()\n",
    "#         print 'Sec per batch', round(en-st, 2)\n",
    "    ep_end = time()\n",
    "    print 'Total train loss', train_loss / len(dataset.train_loader)\n",
    "    print 'Total train accuracy', train_accuracy / len(dataset.train_loader)\n",
    "    print 'Total time for training an epoch: {}'.format(int(ep_end - ep_start))\n",
    "        \n",
    "    capsule_net.eval()\n",
    "    test_loss = 0\n",
    "    test_accuracy = 0\n",
    "    for batch_id, (data, target) in enumerate(dataset.test_loader):\n",
    "\n",
    "        target = torch.sparse.torch.eye(10).index_select(dim=0, index=target)\n",
    "        data, target = Variable(data), Variable(target)\n",
    "\n",
    "        if USE_CUDA:\n",
    "            data, target = data.cuda(), target.cuda()\n",
    "\n",
    "        output, reconstructions, masked = capsule_net(data)\n",
    "        loss = capsule_net.loss(data, output, target, reconstructions)\n",
    "\n",
    "        test_loss += loss.data[0]\n",
    "        ts_accuracy = sum(np.argmax(masked.data.cpu().numpy(), 1) == \n",
    "                          np.argmax(target.data.cpu().numpy(), 1)) / float(batch_size)\n",
    "        test_accuracy += ts_accuracy\n",
    "        if batch_id % 25 == 0 or batch_id == 99:\n",
    "            print \"test accuracy [batch {}]:\".format(batch_id), ts_accuracy\n",
    "    \n",
    "    print 'Total test loss', test_loss / len(dataset.test_loader)\n",
    "    print 'Total test accuracy', test_accuracy / len(dataset.test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XaYfSSjpHyDf"
   },
   "source": [
    "## Reconstructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2MXG6YWW8w2k"
   },
   "source": [
    "Here you can view the reconstructions obtained by your CapsNet. Nothing special here, just fun to visualize them. Actually, for mnist the reconstructions are great, however for CIFAR10 they are rubbish (see the original paper for clues on that).\n",
    "\n",
    "Be careful when running reconstructions after `keyboard_interrupt`, because this may result in wrong input-target values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "1UjRCmmI8S_7"
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_images_separately(images):\n",
    "    \"Plot the six MNIST images separately.\"\n",
    "    fig = plt.figure()\n",
    "    for j in xrange(1, 7):\n",
    "        ax = fig.add_subplot(1, 6, j)\n",
    "        ax.matshow(images[j-1], cmap = matplotlib.cm.binary)\n",
    "        plt.xticks(np.array([]))\n",
    "        plt.yticks(np.array([]))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 95,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 769,
     "status": "ok",
     "timestamp": 1520067617847,
     "user": {
      "displayName": "Nikita Seleznev",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "101390845716372350574"
     },
     "user_tz": -180
    },
    "id": "F5_f1El78S_9",
    "outputId": "13cf3b8f-d951-44bf-c868-2f38d34dadc5"
   },
   "outputs": [],
   "source": [
    "plot_images_separately(data[:6,0].data.cpu().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 95,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 713,
     "status": "ok",
     "timestamp": 1520067620289,
     "user": {
      "displayName": "Nikita Seleznev",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "101390845716372350574"
     },
     "user_tz": -180
    },
    "id": "sqg4U_AJ8TAB",
    "outputId": "bfd2ef66-9331-48f2-a9bb-1f7e1bc6e3ed"
   },
   "outputs": [],
   "source": [
    "plot_images_separately(reconstructions[:6,0].data.cpu().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "B3IApzaZ8TAE"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pWTldL4t9QYt"
   },
   "source": [
    "## Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XBmwmhZP9UDs"
   },
   "source": [
    "Your task is to **get at least 75% test accuracy on CIFAR10 (in the original paper ~90%) in no more than 10 epochs**. For this you may consider tuning various parameters.\n",
    "\n",
    "<br>\n",
    "Start with some **obvious stuff**:\n",
    "\n",
    "- Stack more convolutional layers before capsule layers.\n",
    "- Increase the size of the capsule layers (more capsules, larger capsules etc.). Note that it may take a lot of time.\n",
    "- Play with number of routing iterations in forward pass.\n",
    "- Play with kernel size of convolutions in the first layer (don't forget to change parameters of subsequent layers).\n",
    "- Play with kernel size of capsules in the second layer (again, pay attention to the parameters of subsequent computations).\n",
    "\n",
    "Then, you may consider more **advansed ideas**.\n",
    "\n",
    "- Try different variants of original implementation's loss function (change m+, m-, lambda, get rid of square etc.).\n",
    "- Try different loss functions (make it pure Hinge or pure MSE, maybe even cross-entropy!).\n",
    "- Try different implementation of capsules (not usual convolution operation, but maybe fully connected groups of neurons).\n",
    "- Try different non-linearities for capsules (changing ^2 to ^4 doesn't count!).\n",
    "- Try different weights for reconstruction loss.\n",
    "\n",
    "<br>\n",
    "For each of the tested ideas **supply a mini-report (create a markdown cell above the implementation)**. This mini-report should cover what is the idea (ex: _\"I increased the number of routing iterations from 3 to 10.\"_), and if the idea is not obvious (like stacking more layers), please supply 1/2-sentence justification for it (ex: _\"I changed m+ to 0.7 because I think that we really do not need such high certainty in classification. I speculate that it will speed-up training, taking into account that pictures in CIFAR10 are much less alike than in MNIST we really do not need to be 90% sure, just 70% and it will lead to the same result.\"_).\n",
    "\n",
    "<br>\n",
    "The final subtask is to **propose an idea in which capsules may be used in future and briefly explain motivation for it**. \n",
    "\n",
    "Ex: _\"The capsule representation of the object may be a great input for a GAN if coupled with usual GAN input (random noise vector). I think that capsules' generalized representations of objects may serve as an additional guide for a GAN so as to get expected results in image generation and not random ones. If we tune some coordinates in a capsule vector a little, we vary the reconstruction result (demonstrated in the paper), so, if we add up random noise vector and capsule representation (also tweaked to our desire) we can get various representations of the same object. For example we vary the coordinate 0 of the capsule and supply it to GAN we may get red apple, yellow apple, green apple etc. the random noise is needed to get different red apples, different yellow apples etc.\"_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SANDCkeQfXOA"
   },
   "source": [
    "## Grading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nvvqaMcJfZqB"
   },
   "source": [
    "The **deadline is March 16, 2018 20.00**\n",
    "\n",
    "75% test accuracy in 10 epochs gives you 7 points.\n",
    "\n",
    "Playing with at least 2 parameters in the advanced section gives you another 2 points in case you supplied mini-report with justification for it.\n",
    "\n",
    "Proposing not-so-obvious research idea will give you another 1 point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HiXkxhDNueVt"
   },
   "source": [
    "## Contacts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4Os4SO3Cukpe"
   },
   "source": [
    "Should you have any questions regarding the assignment contact Nikita Seleznev: http://telegram.me/stleznev\n",
    "\n",
    "However, please, avoid an avalanche of messages during the last hours before the deadline :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "DBjxzFCQ9SxO"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "CapsuleNetwork.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}