tensorflow1_ipynb/mechanics/file-queues.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.6.1\n",
      "IPython 6.1.0\n",
      "\n",
      "tensorflow 1.1.0\n",
      "numpy 1.12.1\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p tensorflow,numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Queue Runners to Feed Images Directly from Disk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow provides users with multiple options for providing data to the model. One of the probably most common methods is to define placeholders in the TensorFlow graph and feed the data from the current Python session into the TensorFlow `Session` using the `feed_dict` parameter. Using this approach, a large dataset that does not fit into memory is most conveniently and efficiently stored using NumPy archives as explained in [Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives](image-data-chunking-npz.ipynb) or HDF5 data base files ([Storing an Image Dataset for Minibatch Training using HDF5](image-data-chunking-hdf5.ipynb)).\n",
    "\n",
    "Another approach, which is often preferred when it comes to computational efficiency, is to do the \"data loading\" directly in the graph using input queues from so-called TFRecords files, which is illustrated in the [Using Input Pipelines to Read Data from TFRecords Files](tfrecords.ipynb) notebook. \n",
    "\n",
    "This notebook will introduce an alternative approach which is similar to the TFRecords approach as we will be using input queues to load the data directly on the graph. However, here we are going to read the images directly from JPEG files, which is a useful approach if disk space is a concern and we don't want to create a large TFRecords file from our \"large\" image database.\n",
    "\n",
    "Beyond the examples in this notebook, you are encouraged to read more in TensorFlow's \"[Reading Data](https://www.tensorflow.org/programmers_guide/reading_data)\" guide."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. The Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's pretend we have a directory of images containing two subdirectories with images for training, validation, and testing. The following function will create such a dataset of images in JPEG format locally for demonstration purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./train-images-idx3-ubyte.gz\n",
      "Extracting ./train-labels-idx1-ubyte.gz\n",
      "Extracting ./t10k-images-idx3-ubyte.gz\n",
      "Extracting ./t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# Note that executing the following code \n",
    "# cell will download the MNIST dataset\n",
    "# and save all the 60,000 images as separate JPEG\n",
    "# files. This might take a few minutes depending\n",
    "# on your machine.\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "# load utilities from ../helper.py\n",
    "import sys\n",
    "sys.path.insert(0, '..') \n",
    "from helper import mnist_export_to_jpg\n",
    "\n",
    "np.random.seed(123)\n",
    "mnist_export_to_jpg(path='./')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `mnist_export_to_jpg` function called above creates 3 directories, mnist_train, mnist_test, and mnist_validation. Note that the names of the subdirectories correspond directly to the class label of the images that are stored under it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mnist_train subdirectories ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']\n",
      "mnist_valid subdirectories ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']\n",
      "mnist_test subdirectories ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "for i in ('train', 'valid', 'test'): \n",
    "    dirs = [d for d in os.listdir('mnist_%s' % i) if not d.startswith('.')]\n",
    "    print('mnist_%s subdirectories' % i, dirs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make sure that the images look okay, the snippet below plots an example image from the subdirectory `mnist_train/9/`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEkpJREFUeJzt3W2MVGWWB/D/4cUGupGX7V5AB22MZlQwMkmFbMSY0XFG\nxkyCYxSHmJE1ZtoP7GQnmZhVNsbXGLPuQPywkjALGdjMOrMRVDS6K5AJZgKZ0BBWcFyVxcYBge4G\nku7mRWj67Ie+uK32PaetW3VvFef/SwjddepWPV1d/67qPvd5HlFVEFE8o4oeABEVg+EnCorhJwqK\n4ScKiuEnCorhJwqK4ScKiuEnCorhJwpqTJ531tzcrK2trXneJVFVeGfGikhOI/myjo4OdHd3j+jO\nM4VfRBYAeBHAaAD/qqrPW9dvbW1Fe3t7lrusS+fPnzfr3hNl1KjqvUEr8klc7fu2bj/rbff395v1\nMWNyfV39QqlUGvF1y35WichoAP8C4IcArgewWESuL/f2iChfWV5S5gHYp6r7VfUsgN8BWFiZYRFR\ntWUJ/+UA/jLk84PJZV8iIm0i0i4i7V1dXRnujogqqep/7VfVVapaUtVSS0tLte+OiEYoS/gPAZg5\n5PNvJZcRUR3IEv4dAK4RkVkicgmAnwDYWJlhEVG1ld2PUNV+Efk7AP+FwVbfGlV9v2Iju4iMHj06\n0/Feq3BgYCC1NnbsWPPYrC2vc+fOmXXr/k+ePGke642tsbGx7OO9x9RrrxbVyqukTF+Bqr4F4K0K\njYWIcsTTe4mCYviJgmL4iYJi+ImCYviJgmL4iYKq/2ZlHejr6zPrTU1NZt07T8Cqnzp1KtNtNzQ0\nmHXvPALrHATv686qt7c3teZ9XVnPzagHfOUnCorhJwqK4ScKiuEnCorhJwqK4ScKiq2+HGRtaWVZ\n5XbChAmZ7tvjTY21xt7T01P2sQAwZcoUsz5x4kSzbjlz5oxZ91qBXgu0FvCVnygohp8oKIafKCiG\nnygohp8oKIafKCiGnygo9vlz4PXCvR1fvZ5ylmWkT58+bda9Xrs3NdYa2+TJk81jPWfPnjXrVq/d\nWxZ83LhxZY2pnvCVnygohp8oKIafKCiGnygohp8oKIafKCiGnyioTH1+EekA0AvgPIB+VS1VYlAX\nm1peBtrr03tbVXusefHV7qVn2X7cO//BG3vWrc/zUImTfG5V1e4K3A4R5Yhv+4mCyhp+BbBZRHaK\nSFslBkRE+cj6tv9mVT0kIn8NYJOI/I+qvjv0CskPhTYAuOKKKzLeHRFVSqZXflU9lPzfCeBVAPOG\nuc4qVS2paqmlpSXL3RFRBZUdfhFpFJGJFz4G8AMAeys1MCKqrixv+6cBeDVpaYwB8O+q+p8VGRUR\nVV3Z4VfV/QBurOBYLlrnzp0z69Vc4/3AgQNmffPmzWZ99erVZn379u1m3Vo739pCeyQeeeQRs/7U\nU0+l1saPH28em2WvhHrBVh9RUAw/UVAMP1FQDD9RUAw/UVAMP1FQXLo7B1mW1gaA48ePm/WXXnop\ntbZ27Vrz2P3795t1rw3Z2Nho1q12nnfGZ1dXl1l/4YUXzLq1JPrjjz9uHutt/+0tx17L07gv4Cs/\nUVAMP1FQDD9RUAw/UVAMP1FQDD9RUAw/UVC59vlV1Zze6vWUrd6q11c9deqUWZ8wYYJZt2SdsvvZ\nZ5+Z9dtuu82sf/jhh2bd4m2Tfeutt5r1+fPnm/XZs2eXfaw33fjuu+826xs2bEitPfPMM+ax3nRj\na6pyveArP1FQDD9RUAw/UVAMP1FQDD9RUAw/UVAMP1FQufb5RcTseVvzrwF7XvzZs2fNY7NuB33y\n5MnUmjenfceOHWb9jjvuMOve12Z57LHHzPqyZcvMelNTk1n3xmbVvdv2eI+7de7HsWPHzGO9reWK\nXI69UvjKTxQUw08UFMNPFBTDTxQUw08UFMNPFBTDTxSU2+cXkTUAfgSgU1XnJJdNBfB7AK0AOgAs\nUtUTWQdz5swZs271hS+55JJM9+31q62e8u7du81jb7nlFrPures/apT9M3rTpk2ptdtvv908tru7\n26x7vXjvcc/yfdm6datZP336tFm/7rrrUmveuvyeeujje0byyv8bAAu+ctmjALao6jUAtiSfE1Ed\nccOvqu8C+OqWMQsBXNgKZi2Auyo8LiKqsnJ/55+mqoeTj48AmFah8RBRTjL/wU9VFYCm1UWkTUTa\nRaTd23uNiPJTbviPisgMAEj+70y7oqquUtWSqpa8jRmJKD/lhn8jgCXJx0sAvF6Z4RBRXtzwi8jL\nALYD+LaIHBSRhwA8D+D7IvIxgNuTz4mojrh9flVdnFL6XoXH4vazBwYGyj7Wm3/t9dqt459++mnz\nWG9PAe++vX73DTfcYNYtzc3NZr2np8esX3rppWXf97Zt28z6+vXrzbr1fACAGTNmpNa8dfe98z6y\nnldSC3iGH1FQDD9RUAw/UVAMP1FQDD9RUAw/UVC5Lt0NDG7TncZbXts6NiuvVfjaa6+l1t544w3z\nWG/654MPPmjWr732WrMuIqm1LMuhA/7W5d735MSJ9Jnezz77rHnswYMHzbo33dg6o9Qbt9fK6+vr\nM+tZlyXPA1/5iYJi+ImCYviJgmL4iYJi+ImCYviJgmL4iYLKvc9v8Xrtlt7eXrPuTeH0vPLKK6k1\nr1c+efJks758+fKyxjQSnZ2piywBAC677DKz7n1t3nkETzzxRGrt7bffNo9taGgw695U6fvuuy+1\n5k3Z9e7bq9cDvvITBcXwEwXF8BMFxfATBcXwEwXF8BMFxfATBVVT8/mteekebxnnrKZPn55aGz9+\nvHmsN3f8k08+MeuzZs0y6xavj+9ti+6tsfDAAw+YdW+tA4v3fPAe1xtvvLHs+/a2//a+5/WAr/xE\nQTH8REEx/ERBMfxEQTH8REEx/ERBMfxEQbl9fhFZA+BHADpVdU5y2ZMAfgagK7naMlV9y7stVc20\nzbY1B3vSpEne3Zu8ddjvvffe1NqKFSsy3XepVDLr8+fPN+v33HNPam327NnmsevWrTPrK1euNOve\n1udz5sxJrXlr2+/Zs8ese+skWGPz9lLw1gq4GIzklf83ABYMc/kKVZ2b/HODT0S1xQ2/qr4L4HgO\nYyGiHGX5nf/nIvKeiKwRkSkVGxER5aLc8K8EcBWAuQAOA/hV2hVFpE1E2kWkvbu7u8y7I6JKKyv8\nqnpUVc+r6gCAXwOYZ1x3laqWVLXU3Nxc7jiJqMLKCr+IzBjy6Y8B7K3McIgoLyNp9b0M4LsAmkXk\nIIAnAHxXROYCUAAdAB6u4hiJqArc8Kvq4mEuXl2FsbisPdO9frO3/rzXc7766qtTa9u3bzePXbp0\nqVnftWuXWffmxL/zzjupNW/Ou7d+vTdv/eGH7Z/7zz33XFk1ANi3b59Z99Y5yLI+hPVcA4Dz58+b\n9Xo4T4Bn+BEFxfATBcXwEwXF8BMFxfATBcXwEwVVU1t0V1OWtg8AtLS0pNYaGxvNY3fu3GnW33zz\nTbPe0dFh1q1W36effmoee//995t1ayozALS2tpr1EydOpNa2bdtmHuudDn7TTTeZda+9a/Fax96U\n4HrAV36ioBh+oqAYfqKgGH6ioBh+oqAYfqKgGH6ioHLt84tIpt6rxZtC2d/fb9azjGvChAllHwsA\nCxYMtzjy//PG1tbWllrzpvQ2NDSYdY93+/v370+teUtzexYtWlT2sd64vfrFgK/8REEx/ERBMfxE\nQTH8REEx/ERBMfxEQTH8REFdNPP5ve29vfn83hLW1vFZ53Z7y0B7Y/eWmc7iyJEjZn369Olm3Vp2\n/OTJk+ax1vbeADB37lyzbvXqvT6+95haW80D/vOxFtT+CImoKhh+oqAYfqKgGH6ioBh+oqAYfqKg\nGH6ioNw+v4jMBLAOwDQACmCVqr4oIlMB/B5AK4AOAItUNX2R9hHw+t1W79TrhWftlVdzfvepU6fM\n+pQpU8q+7WPHjpn1qVOnmnWvj+/16rdu3Zpa83rlV155pVn31miw1njwjvWeD/XQx/eM5CvoB/BL\nVb0ewN8AWCoi1wN4FMAWVb0GwJbkcyKqE274VfWwqu5KPu4F8AGAywEsBLA2udpaAHdVa5BEVHnf\n6L2LiLQC+A6APwGYpqqHk9IRDP5aQER1YsThF5EmAOsB/EJVe4bWdPAX4mF/KRaRNhFpF5H2rq6u\nTIMlosoZUfhFZCwGg/9bVd2QXHxURGYk9RkAOoc7VlVXqWpJVUvWZpdElC83/DL4Z/LVAD5Q1eVD\nShsBLEk+XgLg9coPj4iqZSRTeucD+CmAPSKyO7lsGYDnAfyHiDwE4ACA8tdRTnjtE6sV6B2btTWT\ndYtvS5ZWnmf8+PFmPevX5W1Pbn3Pxo0bZx7rtdtOnz5t1idOnJha87bgruY06Vrhhl9V/wgg7Rny\nvcoOh4jyUv9nKhBRWRh+oqAYfqKgGH6ioBh+oqAYfqKgamrpbq/nnGWKpjdd2Nvi2+JNTfXqXk/5\nzJkzZt0au7d9uNfv9qbsTpo0yaxbS6J73zNvGrV3DoPF+3573zNvbFmeT3nhKz9RUAw/UVAMP1FQ\nDD9RUAw/UVAMP1FQDD9RULn2+QcGBsw52FnmnmfdJjsLb60Aryf8+eefm3Vv3rvF23rce9wmT55s\n1g8cOGDWP/roo9Sad/6Cdx7AmDH209d63LM8poA/tnrAV36ioBh+oqAYfqKgGH6ioBh+oqAYfqKg\nGH6ioHLt848aNSrTHOx6lXV78Cz97qzrz/f19Zn1hoYGs27N9z9xwt7R3dsTwGM97lnPIfDq9YCv\n/ERBMfxEQTH8REEx/ERBMfxEQTH8REEx/ERBuc1KEZkJYB2AaQAUwCpVfVFEngTwMwBdyVWXqepb\n1RroxczbU8BbD8BaY95ba8DjnZfhrU9v9eqnT59uHjtz5kyznmUvhqx9em+dhKznV+RhJI9AP4Bf\nquouEZkIYKeIbEpqK1T1n6s3PCKqFjf8qnoYwOHk414R+QDA5dUeGBFV1zd6TygirQC+A+BPyUU/\nF5H3RGSNiExJOaZNRNpFpL2rq2u4qxBRAUYcfhFpArAewC9UtQfASgBXAZiLwXcGvxruOFVdpaol\nVS21tLRUYMhEVAkjCr+IjMVg8H+rqhsAQFWPqup5VR0A8GsA86o3TCKqNDf8Mjg1ajWAD1R1+ZDL\nZwy52o8B7K388IioWkby1/75AH4KYI+I7E4uWwZgsYjMxWD7rwPAw1UZYQBeOy7rdtFZeK08rxW4\nd2/6a0JPT495rLe8tjc2a0l0r9Xn3XbWFmotGMlf+/8IYLiJ0ezpE9Wx+v/xRURlYfiJgmL4iYJi\n+ImCYviJgmL4iYKq//WHLwJezzjLFuDeOQLesuJePYumpiaz7p2/4E3pterekuMeLt1NRHWL4ScK\niuEnCorhJwqK4ScKiuEnCorhJwpKqjkX/Gt3JtIF4MCQi5oBdOc2gG+mVsdWq+MCOLZyVXJsV6rq\niNbLyzX8X7tzkXZVLRU2AEOtjq1WxwVwbOUqamx8208UFMNPFFTR4V9V8P1banVstTougGMrVyFj\nK/R3fiIqTtGv/ERUkELCLyILRORDEdknIo8WMYY0ItIhIntEZLeItBc8ljUi0ikie4dcNlVENonI\nx8n/w26TVtDYnhSRQ8ljt1tE7ixobDNF5A8i8mcReV9E/j65vNDHzhhXIY9b7m/7RWQ0gI8AfB/A\nQQA7ACxW1T/nOpAUItIBoKSqhfeEReQWAH0A1qnqnOSyfwJwXFWfT35wTlHVf6iRsT0JoK/onZuT\nDWVmDN1ZGsBdAP4WBT52xrgWoYDHrYhX/nkA9qnqflU9C+B3ABYWMI6ap6rvAjj+lYsXAlibfLwW\ng0+e3KWMrSao6mFV3ZV83Avgws7ShT52xrgKUUT4LwfwlyGfH0RtbfmtADaLyE4RaSt6MMOYlmyb\nDgBHAEwrcjDDcHduztNXdpaumceunB2vK41/8Pu6m1V1LoAfAliavL2tSTr4O1sttWtGtHNzXobZ\nWfoLRT525e54XWlFhP8QgJlDPv9WcllNUNVDyf+dAF5F7e0+fPTCJqnJ/50Fj+cLtbRz83A7S6MG\nHrta2vG6iPDvAHCNiMwSkUsA/ATAxgLG8TUi0pj8IQYi0gjgB6i93Yc3AliSfLwEwOsFjuVLamXn\n5rSdpVHwY1dzO16rau7/ANyJwb/4/y+AfyxiDCnjugrAfyf/3i96bABexuDbwHMY/NvIQwD+CsAW\nAB8D2Axgag2N7d8A7AHwHgaDNqOgsd2Mwbf07wHYnfy7s+jHzhhXIY8bz/AjCop/8CMKiuEnCorh\nJwqK4ScKiuEnCorhJwqK4ScKiuEnCur/ABj13FTNxtqQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11a05a3c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "\n",
    "some_img = os.path.join('./mnist_train/9/', os.listdir('./mnist_train/9/')[0])\n",
    "\n",
    "img = mpimg.imread(some_img)\n",
    "print(img.shape)\n",
    "plt.imshow(img, cmap='binary');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The JPEG format introduces a few artifacts that we can see in the image above. In this case, we use JPEG instead of PNG. Here, JPEG is used for demonstration purposes since that's still format many image datasets are stored in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section provides an example of how to use the [`tf.WholeFileReader`](https://www.tensorflow.org/api_docs/python/tf/WholeFileReader) and a filename queue to read in the images from the `mnist_train` directory. Also, we will be extracting the class labels directly from the file paths and convert the images to a one-hot encoded format that we will use in the later sections to train a multilayer neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image shape: (28, 28, 1)\n",
      "File name: b'mnist_train/6/22340.jpg'\n",
      "Class label: 6\n",
      "One-hot class label: [ 0.  0.  0.  0.  0.  0.  1.  0.  0.  0.]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "g = tf.Graph()\n",
    "with g.as_default():\n",
    "    \n",
    "    filename_queue = tf.train.string_input_producer(\n",
    "        tf.train.match_filenames_once('mnist_train/*/*.jpg'),\n",
    "        seed=123,\n",
    "        shuffle=True)\n",
    "\n",
    "    image_reader = tf.WholeFileReader()\n",
    "\n",
    "    file_name, image_raw = image_reader.read(filename_queue)\n",
    "    file_name = tf.identity(file_name, name='file_name')\n",
    "    image = tf.image.decode_jpeg(image_raw, name='image')\n",
    "    image = tf.cast(image, tf.float32)\n",
    "    label = tf.string_split([file_name], '/').values[1]\n",
    "    label = tf.string_to_number(label, tf.int32, name='label')\n",
    "    onehot_label = tf.one_hot(indices=label, \n",
    "                              depth=10, \n",
    "                              name='onehot_label')\n",
    "\n",
    "    \n",
    "\n",
    "with tf.Session(graph=g) as sess:\n",
    "\n",
    "    sess.run(tf.local_variables_initializer())\n",
    "\n",
    "    coord = tf.train.Coordinator()\n",
    "    threads = tf.train.start_queue_runners(sess=sess,\n",
    "                                           coord=coord)\n",
    "    \n",
    "    image_tensor, file_name, class_label, ohe_label =\\\n",
    "            sess.run(['image:0', \n",
    "                      'file_name:0', \n",
    "                      'label:0',\n",
    "                      'onehot_label:0'])\n",
    "\n",
    "    print('Image shape:', image_tensor.shape)\n",
    "    print('File name:', file_name)    \n",
    "    print('Class label:', class_label)  \n",
    "    print('One-hot class label:', ohe_label)  \n",
    "\n",
    "    coord.request_stop()\n",
    "    coord.join(threads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `tf.train.string_input_producer` produces a filename queue that we iterate over in the session. Note that we need to call `sess.run(tf.local_variables_initializer())` for our filename queue. y.\" \n",
    "\n",
    "- The `tf.train.start_queue_runners` function uses a queue runner that uses a separate thread to load the filenames from the `queue` that we defined in the graph without blocking the reader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that it is important to shuffle the dataset so that we can later make use of TensorFlow's [`tf.train.shuffle_batch`](https://www.tensorflow.org/api_docs/python/tf/train/shuffle_batch) function and don't need to load the whole dataset into memory to shuffle epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Reading in batches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the previous section illustrated how we can use input pipelines to read images one by one, we rarely (want to) train neural networks with one datapoint at a time but use minibatches instead. TensorFlow also has some really convenient utility functions to do the batching conveniently. In the following code example, we will use the [`tf.train.shuffle_batch`](https://www.tensorflow.org/api_docs/python/tf/train/shuffle_batch) function to load the images and labels in batches of size 64.\n",
    "\n",
    "Also, let us put the code for processing the images and labels into a function, `read_images_from_disk`, that we can reuse later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image batch shape: (64, 28, 28, 1)\n",
      "Label batch shape: (64, 10)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "def read_images_from_disk(filename_queue, image_dimensions, normalize=True):\n",
    "\n",
    "    image_reader = tf.WholeFileReader()\n",
    "    file_name, image_raw = image_reader.read(filename_queue)\n",
    "    file_name = tf.identity(file_name, name='file_name')\n",
    "    image = tf.image.decode_jpeg(image_raw, name='image')\n",
    "    image.set_shape(image_dimensions)\n",
    "    image = tf.cast(image, tf.float32)\n",
    "\n",
    "    if normalize:\n",
    "        # normalize to [0, 1] range\n",
    "        image = image / 255.\n",
    "        \n",
    "    label = tf.string_split([file_name], '/').values[1]\n",
    "    label = tf.string_to_number(label, tf.int32)\n",
    "    onehot_label = tf.one_hot(indices=label,\n",
    "                              depth=10, \n",
    "                              name='onehot_label')\n",
    "    return image, onehot_label\n",
    "    \n",
    "    \n",
    "    \n",
    "\n",
    "g = tf.Graph()\n",
    "with g.as_default():\n",
    "    \n",
    "\n",
    "    filename_queue = tf.train.string_input_producer(\n",
    "            tf.train.match_filenames_once('mnist_train/*/*.jpg'),\n",
    "            seed=123)\n",
    "     \n",
    "    image, label = read_images_from_disk(filename_queue,\n",
    "                                         image_dimensions=[28, 28, 1])\n",
    "    \n",
    "    \n",
    "    image_batch, label_batch = tf.train.shuffle_batch([image, label], \n",
    "                                                       batch_size=64,\n",
    "                                                       capacity=2000,\n",
    "                                                       min_after_dequeue=1000,\n",
    "                                                       num_threads=1,\n",
    "                                                       seed=123)\n",
    "\n",
    "\n",
    "    \n",
    "with tf.Session(graph=g) as sess:\n",
    "\n",
    "    sess.run(tf.local_variables_initializer())\n",
    "\n",
    "    coord = tf.train.Coordinator()\n",
    "    threads = tf.train.start_queue_runners(sess=sess,\n",
    "                                           coord=coord)\n",
    "    \n",
    "    multipe_images, multiple_labels = sess.run([image_batch, label_batch])\n",
    "    print('Image batch shape:', multipe_images.shape)\n",
    "    print('Label batch shape:', label_batch.shape)\n",
    "\n",
    "    coord.request_stop()\n",
    "    coord.join(threads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other relevant arguments we provided to `tf.train.shuffle_batch` are described below:\n",
    "\n",
    "- `capacity`: An integer that defines the maximum number of elements in the queue.\n",
    "- `min_after_dequeue`: The minimum number elements in the queue after a dequeue, which is used to ensure that a minimum number of data points have been loaded for shuffling.\n",
    "- `num_threads`: The number of threads for enqueuing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Use queue runners to train a neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will take the concepts that were introduced in the previous sections and train a multilayer perceptron using the concepts introduced in the previous sections: the `read_images_from_disk` function, a filename queue, and the `tf.train.shuffle_batch` function.\n",
    "                                  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001 | AvgCost: 0.007\n",
      "Epoch: 002 | AvgCost: 0.481\n",
      "Epoch: 003 | AvgCost: 0.234\n",
      "Epoch: 004 | AvgCost: 0.180\n",
      "Epoch: 005 | AvgCost: 0.147\n",
      "Epoch: 006 | AvgCost: 0.125\n",
      "Epoch: 007 | AvgCost: 0.108\n",
      "Epoch: 008 | AvgCost: 0.094\n",
      "Epoch: 009 | AvgCost: 0.084\n",
      "Epoch: 010 | AvgCost: 0.075\n",
      "Epoch: 011 | AvgCost: 0.067\n",
      "Epoch: 012 | AvgCost: 0.061\n",
      "Epoch: 013 | AvgCost: 0.055\n",
      "Epoch: 014 | AvgCost: 0.049\n",
      "Epoch: 015 | AvgCost: 0.046\n"
     ]
    }
   ],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Hyperparameters\n",
    "learning_rate = 0.1\n",
    "batch_size = 128\n",
    "n_epochs = 15\n",
    "n_iter = n_epochs * (45000 // batch_size)\n",
    "\n",
    "# Architecture\n",
    "n_hidden_1 = 128\n",
    "n_hidden_2 = 256\n",
    "height, width = 28, 28\n",
    "n_classes = 10\n",
    "\n",
    "\n",
    "\n",
    "##########################\n",
    "### GRAPH DEFINITION\n",
    "##########################\n",
    "\n",
    "g = tf.Graph()\n",
    "with g.as_default():\n",
    "    \n",
    "    tf.set_random_seed(123)\n",
    "\n",
    "    # Input data\n",
    "    filename_queue = tf.train.string_input_producer(\n",
    "            tf.train.match_filenames_once('mnist_train/*/*.jpg'),\n",
    "            seed=123)\n",
    "     \n",
    "    image, label = read_images_from_disk(filename_queue,\n",
    "                                         image_dimensions=[28, 28, 1])\n",
    "    image = tf.reshape(image, (width*height,))\n",
    "    \n",
    "    \n",
    "    image_batch, label_batch = tf.train.shuffle_batch([image, label], \n",
    "                                                       batch_size=batch_size,\n",
    "                                                       capacity=2000,\n",
    "                                                       min_after_dequeue=1000,\n",
    "                                                       num_threads=1,\n",
    "                                                       seed=123)\n",
    "    \n",
    "    tf_images = tf.placeholder_with_default(image_batch,\n",
    "                                            shape=[None, 784], \n",
    "                                            name='images')\n",
    "    tf_labels = tf.placeholder_with_default(label_batch, \n",
    "                                            shape=[None, 10], \n",
    "                                            name='labels')\n",
    "\n",
    "    # Model parameters\n",
    "    weights = {\n",
    "        'h1': tf.Variable(tf.truncated_normal([height*width, n_hidden_1], stddev=0.1)),\n",
    "        'h2': tf.Variable(tf.truncated_normal([n_hidden_1, n_hidden_2], stddev=0.1)),\n",
    "        'out': tf.Variable(tf.truncated_normal([n_hidden_2, n_classes], stddev=0.1))\n",
    "    }\n",
    "    biases = {\n",
    "        'b1': tf.Variable(tf.zeros([n_hidden_1])),\n",
    "        'b2': tf.Variable(tf.zeros([n_hidden_2])),\n",
    "        'out': tf.Variable(tf.zeros([n_classes]))\n",
    "    }\n",
    "\n",
    "    # Multilayer perceptron\n",
    "    layer_1 = tf.add(tf.matmul(tf_images, weights['h1']), biases['b1'])\n",
    "    layer_1 = tf.nn.relu(layer_1)\n",
    "    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])\n",
    "    layer_2 = tf.nn.relu(layer_2)\n",
    "    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']\n",
    "\n",
    "    # Loss and optimizer\n",
    "    loss = tf.nn.softmax_cross_entropy_with_logits(logits=out_layer, labels=tf_labels)\n",
    "    cost = tf.reduce_mean(loss, name='cost')\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "    train = optimizer.minimize(cost, name='train')\n",
    "\n",
    "    # Prediction\n",
    "    prediction = tf.argmax(out_layer, 1, name='prediction')\n",
    "    correct_prediction = tf.equal(tf.argmax(label_batch, 1), tf.argmax(out_layer, 1))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy')\n",
    "    \n",
    "    \n",
    "    \n",
    "with tf.Session(graph=g) as sess:\n",
    "    sess.run(tf.local_variables_initializer())\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver0 = tf.train.Saver()\n",
    "    \n",
    "    coord = tf.train.Coordinator()\n",
    "    threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n",
    "    \n",
    "    avg_cost = 0.\n",
    "    iter_per_epoch = n_iter // n_epochs\n",
    "    epoch = 0\n",
    "\n",
    "    for i in range(n_iter):\n",
    "        _, cost = sess.run(['train', 'cost:0'])\n",
    "        avg_cost += cost\n",
    "        \n",
    "        if not i % iter_per_epoch:\n",
    "            epoch += 1\n",
    "            avg_cost /= iter_per_epoch\n",
    "            print(\"Epoch: %03d | AvgCost: %.3f\" % (epoch, avg_cost))\n",
    "            avg_cost = 0.\n",
    "            \n",
    "        \n",
    "    coord.request_stop()\n",
    "    coord.join(threads)\n",
    "    \n",
    "    saver0.save(sess, save_path='./mlp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After looking at the graph above, you probably wondered why we used [`tf.placeholder_with_default`](https://www.tensorflow.org/api_docs/python/tf/placeholder_with_default) to define the two placeholders:\n",
    "\n",
    "```python\n",
    "tf_images = tf.placeholder_with_default(image_batch,\n",
    "                                            shape=[None, 784], \n",
    "                                            name='images')\n",
    "tf_labels = tf.placeholder_with_default(label_batch, \n",
    "                                        shape=[None, 10], \n",
    "                                        name='labels')\n",
    "```      \n",
    "\n",
    "In the training session above, these placeholders are being ignored if we don't feed them via a session's `feed_dict`, or in other words \"[A `tf.placeholder_with_default` is a] placeholder op that passes through input when its output is not fed\" (https://www.tensorflow.org/api_docs/python/tf/placeholder_with_default).\n",
    "\n",
    "However, these placeholders are useful if we want to feed new data to the graph and make predictions after training as in a real-world application, which we will see in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Feeding new datapoints through placeholders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To demonstrate how we can feed new data points to the network that are not part of the training queue, let's use the test dataset and load the images into Python and pass it to the graph using a `feed_dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./mlp\n",
      "Test accuracy: 97.1%\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.image as mpimg\n",
    "import numpy as np\n",
    "import glob\n",
    "\n",
    "\n",
    "img_paths = np.array([p for p in glob.iglob('mnist_test/*/*.jpg')])\n",
    "labels = np.array([int(path.split('/')[1]) for path in img_paths])\n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    \n",
    "    saver1 = tf.train.import_meta_graph('./mlp.meta')\n",
    "    saver1.restore(sess, save_path='./mlp')\n",
    "    \n",
    "    num_correct = 0\n",
    "    cnt = 0\n",
    "    for path, lab in zip(img_paths, labels):\n",
    "        cnt += 1\n",
    "        image = mpimg.imread(path)\n",
    "        image = image.reshape(1, -1)\n",
    "        \n",
    "        pred = sess.run('prediction:0', \n",
    "                         feed_dict={'images:0': image})\n",
    "\n",
    "        num_correct += int(lab == pred[0])\n",
    "    acc = num_correct / cnt * 100\n",
    "\n",
    "print('Test accuracy: %.1f%%' % acc)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}