00_tensorflow_fundamentals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/mrdbourke/tensorflow-deep-learning/blob/main/00_tensorflow_fundamentals.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ielzxn52r2CB"
   },
   "source": [
    "# 00. Getting started with TensorFlow: A guide to the fundamentals\n",
    "\n",
    "## What is TensorFlow?\n",
    "\n",
    "[TensorFlow](https://www.tensorflow.org/) is an open-source end-to-end machine learning library for preprocessing data, modelling data and serving models (getting them into the hands of others).\n",
    "\n",
    "## Why use TensorFlow?\n",
    "\n",
    "Rather than building machine learning and deep learning models from scratch, it's more likely you'll use a library such as TensorFlow. This is because it contains many of the most common machine learning functions you'll want to use.\n",
    "\n",
    "## What we're going to cover\n",
    "\n",
    "TensorFlow is vast. But the main premise is simple: turn data into numbers (tensors) and build machine learning algorithms to find patterns in them.\n",
    "\n",
    "In this notebook we cover some of the most fundamental TensorFlow operations, more specificially:\n",
    "* Introduction to tensors (creating tensors)\n",
    "* Getting information from tensors (tensor attributes)\n",
    "* Manipulating tensors (tensor operations)\n",
    "* Tensors and NumPy\n",
    "* Using @tf.function (a way to speed up your regular Python functions)\n",
    "* Using GPUs with TensorFlow\n",
    "* Exercises to try\n",
    "\n",
    "Things to note:\n",
    "* Many of the conventions here will happen automatically behind the scenes (when you build a model) but it's worth knowing so if you see any of these things, you know what's happening.\n",
    "* For any TensorFlow function you see, it's important to be able to check it out in the documentation, for example, going to the Python API docs for all functions and searching for what you need: https://www.tensorflow.org/api_docs/python/ (don't worry if this seems overwhelming at first, with enough practice, you'll get used to navigating the documentaiton).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VArmDlu06sH0"
   },
   "source": [
    "## Introduction to Tensors\n",
    "\n",
    "If you've ever used NumPy, [tensors](https://www.tensorflow.org/guide/tensor) are kind of like NumPy arrays (we'll see more on this later).\n",
    "\n",
    "For the sake of this notebook and going forward, you can think of a tensor as a multi-dimensional numerical representation (also referred to as n-dimensional, where n can be any number) of something. Where something can be almost anything you can imagine: \n",
    "* It could be numbers themselves (using tensors to represent the price of houses). \n",
    "* It could be an image (using tensors to represent the pixels of an image).\n",
    "* It could be text (using tensors to represent words).\n",
    "* Or it could be some other form of information (or data) you want to represent with numbers.\n",
    "\n",
    "The main difference between tensors and NumPy arrays (also an n-dimensional array of numbers) is that tensors can be used on [GPUs (graphical processing units)](https://blogs.nvidia.com/blog/2009/12/16/whats-the-difference-between-a-cpu-and-a-gpu/) and [TPUs (tensor processing units)](https://en.wikipedia.org/wiki/Tensor_processing_unit). \n",
    "\n",
    "The benefit of being able to run on GPUs and TPUs is faster computation, this means, if we wanted to find patterns in the numerical representations of our data, we can generally find them faster using GPUs and TPUs.\n",
    "\n",
    "Okay, we've been talking enough about tensors, let's see them.\n",
    "\n",
    "The first thing we'll do is import TensorFlow under the common alias `tf`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Z7ieIu8t9ijY",
    "outputId": "7548088b-91d5-412f-8544-3f6b3487e33a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.4.1\n"
     ]
    }
   ],
   "source": [
    "# Import TensorFlow\n",
    "import tensorflow as tf\n",
    "print(tf.__version__) # find the version number (should be 2.x+)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JNtV5t1qz0VP"
   },
   "source": [
    "### Creating Tensors with `tf.constant()`\n",
    "\n",
    "As mentioned before, in general, you usually won't create tensors yourself. This is because TensorFlow has modules built-in (such as [`tf.io`](https://www.tensorflow.org/api_docs/python/tf/io) and [`tf.data`](https://www.tensorflow.org/guide/data)) which are able to read your data sources and automatically convert them to tensors and then later on, neural network models will process these for us.\n",
    "\n",
    "But for now, because we're getting familar with tensors themselves and how to manipulate them, we'll see how we can create them ourselves.\n",
    "\n",
    "We'll begin by using [`tf.constant()`](https://www.tensorflow.org/api_docs/python/tf/constant)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nC7aQgqi0M_Z",
    "outputId": "89250b3e-86a9-4d9b-b15f-8ae0812dde5b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=7>"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a scalar (rank 0 tensor)\n",
    "scalar = tf.constant(7)\n",
    "scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o6fXE5dXkO_3"
   },
   "source": [
    "A scalar is known as a rank 0 tensor. Because it has no dimensions (it's just a number).\n",
    "\n",
    "> 🔑 **Note:** For now, you don't need to know too much about the different ranks of tensors (but we will see more on this later). The important point is knowing tensors can have an unlimited range of dimensions (the exact amount will depend on what data you're representing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1sgUNKoFkJ21",
    "outputId": "5b244f42-05bc-4702-dc5c-a91716443c4c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 3,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the number of dimensions of a tensor (ndim stands for number of dimensions)\n",
    "scalar.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "irtCo2fs0V_o",
    "outputId": "e424c961-7a1a-4544-ed5a-becc1cc1d1bd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=int32, numpy=array([10, 10], dtype=int32)>"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a vector (more than 0 dimensions)\n",
    "vector = tf.constant([10, 10])\n",
    "vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7DDc36pvmOse",
    "outputId": "f6fbc397-5058-4f2f-8f27-9d7c6adf4f95"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the number of dimensions of our vector tensor\n",
    "vector.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HXf5A5360V7A",
    "outputId": "3f773144-81c5-49d2-819b-192ff0153c03"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[10,  7],\n",
       "       [ 7, 10]], dtype=int32)>"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a matrix (more than 1 dimension)\n",
    "matrix = tf.constant([[10, 7],\n",
    "                      [7, 10]])\n",
    "matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Asmn6YghlT6u",
    "outputId": "aa08a360-2917-4be3-e263-b4cea62ed569"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix.ndim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qvQb7RT2s9Te"
   },
   "source": [
    "By default, TensorFlow creates tensors with either an `int32` or `float32` datatype.\n",
    "\n",
    "This is known as [32-bit precision](https://en.wikipedia.org/wiki/Precision_(computer_science) (the higher the number, the more precise the number, the more space it takes up on your computer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aEgthLq80V2u",
    "outputId": "f64d9ac7-1a83-4fc3-9511-a02e5e19dd6b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float16, numpy=\n",
       "array([[10.,  7.],\n",
       "       [ 3.,  2.],\n",
       "       [ 8.,  9.]], dtype=float16)>"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create another matrix and define the datatype\n",
    "another_matrix = tf.constant([[10., 7.],\n",
    "                              [3., 2.],\n",
    "                              [8., 9.]], dtype=tf.float16) # specify the datatype with 'dtype'\n",
    "another_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "v-Y-lXdOlXRg",
    "outputId": "7f7c851b-9487-4ce8-de4d-5a747b9912e3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Even though another_matrix contains more numbers, its dimensions stay the same\n",
    "another_matrix.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fAy7J6fT0Vwz",
    "outputId": "77c91572-9b0a-4e0d-d39f-91455f41ee1e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=\n",
       "array([[[ 1,  2,  3],\n",
       "        [ 4,  5,  6]],\n",
       "\n",
       "       [[ 7,  8,  9],\n",
       "        [10, 11, 12]],\n",
       "\n",
       "       [[13, 14, 15],\n",
       "        [16, 17, 18]]], dtype=int32)>"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# How about a tensor? (more than 2 dimensions, although, all of the above items are also technically tensors)\n",
    "tensor = tf.constant([[[1, 2, 3],\n",
    "                       [4, 5, 6]],\n",
    "                      [[7, 8, 9],\n",
    "                       [10, 11, 12]],\n",
    "                      [[13, 14, 15],\n",
    "                       [16, 17, 18]]])\n",
    "tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FhIsj108mFOS",
    "outputId": "64bf5633-c17f-42b7-ce0a-14e3df8a2dec"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor.ndim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w5MGwSpA100u"
   },
   "source": [
    "This is known as a rank 3 tensor (3-dimensions), however a tensor can have an arbitrary (unlimited) amount of dimensions.\n",
    "\n",
    "For example, you might turn a series of images into tensors with shape (224, 224, 3, 32), where:\n",
    "* 224, 224 (the first 2 dimensions) are the height and width of the images in pixels.\n",
    "* 3 is the number of colour channels of the image (red, green blue).\n",
    "* 32 is the batch size (the number of images a neural network sees at any one time).\n",
    "\n",
    "All of the above variables we've created are actually tensors. But you may also hear them referred to as their different names (the ones we gave them):\n",
    "* **scalar**: a single number.\n",
    "* **vector**: a number with direction (e.g. wind speed with direction).\n",
    "* **matrix**: a 2-dimensional array of numbers.\n",
    "* **tensor**: an n-dimensional arrary of numbers (where n can be any number, a 0-dimension tensor is a scalar, a 1-dimension tensor is a vector). \n",
    "\n",
    "To add to the confusion, the terms matrix and tensor are often used interchangably.\n",
    "\n",
    "Going forward since we're using TensorFlow, everything we refer to and use will be tensors.\n",
    "\n",
    "For more on the mathematical difference between scalars, vectors and matrices see the [visual algebra post by Math is Fun](https://www.mathsisfun.com/algebra/scalar-vector-matrix.html).\n",
    "\n",
    "![difference between scalar, vector, matrix, tensor](https://github.com/raw/mrdbourke/tensorflow-deep-learning/main/images/00-scalar-vector-matrix-tensor.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TZMfDFKC0Cl7"
   },
   "source": [
    "### Creating Tensors with `tf.Variable()`\n",
    "\n",
    "You can also (although you likely rarely will, because often, when working with data, tensors are created for you automatically) create tensors using [`tf.Variable()`](https://www.tensorflow.org/api_docs/python/tf/Variable).\n",
    "\n",
    "The difference between `tf.Variable()` and `tf.constant()` is tensors created with `tf.constant()` are immutable (can't be changed, can only be used to create a new tensor), where as, tensors created with `tf.Variable()` are mutable (can be changed)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bv1SBbDe4TxN",
    "outputId": "2af43a4e-9d28-4788-f5f9-ab33c95a6551"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Variable 'Variable:0' shape=(2,) dtype=int32, numpy=array([10,  7], dtype=int32)>,\n",
       " <tf.Tensor: shape=(2,), dtype=int32, numpy=array([10,  7], dtype=int32)>)"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the same tensor with tf.Variable() and tf.constant()\n",
    "changeable_tensor = tf.Variable([10, 7])\n",
    "unchangeable_tensor = tf.constant([10, 7])\n",
    "changeable_tensor, unchangeable_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A4WrQ8c2ux15"
   },
   "source": [
    "Now let's try to change one of the elements of the changable tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 200
    },
    "id": "dfDwbF6i5Sy3",
    "outputId": "087b7aa3-d371-4741-d798-9aef73895635"
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-14-daecfbad2415>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Will error (requires the .assign() method)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mchangeable_tensor\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m7\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mchangeable_tensor\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'ResourceVariable' object does not support item assignment"
     ]
    }
   ],
   "source": [
    "# Will error (requires the .assign() method)\n",
    "changeable_tensor[0] = 7\n",
    "changeable_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oWP-kZEVvGm8"
   },
   "source": [
    "To change an element of a `tf.Variable()` tensor requires the `assign()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FJV3iwvG4jg4",
    "outputId": "d110b678-03ff-49e8-dee6-9974ab75ba24"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(2,) dtype=int32, numpy=array([7, 7], dtype=int32)>"
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Won't error\n",
    "changeable_tensor[0].assign(7)\n",
    "changeable_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9UiV1Z0XvZ_B"
   },
   "source": [
    "Now let's try to change a value in a `tf.constant()` tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 200
    },
    "id": "5j_rOo8X5N9f",
    "outputId": "095cb0dc-dda0-41bc-e8dc-995934535957"
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-16-3947b974feb9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Will error (can't change tf.constant())\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0munchangeable_tensor\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0massign\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m7\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0munchangleable_tensor\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'assign'"
     ]
    }
   ],
   "source": [
    "# Will error (can't change tf.constant())\n",
    "unchangeable_tensor[0].assign(7)\n",
    "unchangleable_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1t21IcYpverQ"
   },
   "source": [
    "Which one should you use? `tf.constant()` or `tf.Variable()`?\n",
    "\n",
    "It will depend on what your problem requires. However, most of the time, TensorFlow will automatically choose for you (when loading data or modelling data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pAAT59Ay0J0l"
   },
   "source": [
    "### Creating random tensors\n",
    "\n",
    "Random tensors are tensors of some abitrary size which contain random numbers.\n",
    "\n",
    "Why would you want to create random tensors? \n",
    "\n",
    "This is what neural networks use to intialize their weights (patterns) that they're trying to learn in the data.\n",
    "\n",
    "For example, the process of a neural network learning often involves taking a random n-dimensional array of numbers and refining them until they represent some kind of pattern (a compressed way to represent the original data).\n",
    "\n",
    "**How a network learns**\n",
    "![how a network learns](https://github.com/raw/mrdbourke/tensorflow-deep-learning/main/images/00-how-a-network-learns.png)\n",
    "*A network learns by starting with random patterns (1) then going through demonstrative examples of data (2) whilst trying to update its random patterns to represent the examples (3).*\n",
    "\n",
    "We can create random tensors by using the [`tf.random.Generator`](https://www.tensorflow.org/guide/random_numbers#the_tfrandomgenerator_class) class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yZ7Zu5Z178JL",
    "outputId": "79b4497d-0404-4171-f42f-050c854b9019"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       " array([[-0.7565803 , -0.06854702],\n",
       "        [ 0.07595026, -1.2573844 ],\n",
       "        [-0.23193765, -1.8107855 ]], dtype=float32)>,\n",
       " <tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       " array([[-0.7565803 , -0.06854702],\n",
       "        [ 0.07595026, -1.2573844 ],\n",
       "        [-0.23193765, -1.8107855 ]], dtype=float32)>,\n",
       " <tf.Tensor: shape=(3, 2), dtype=bool, numpy=\n",
       " array([[ True,  True],\n",
       "        [ True,  True],\n",
       "        [ True,  True]])>)"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create two random (but the same) tensors\n",
    "random_1 = tf.random.Generator.from_seed(42) # set the seed for reproducibility\n",
    "random_1 = random_1.normal(shape=(3, 2)) # create tensor from a normal distribution \n",
    "random_2 = tf.random.Generator.from_seed(42)\n",
    "random_2 = random_2.normal(shape=(3, 2))\n",
    "\n",
    "# Are they equal?\n",
    "random_1, random_2, random_1 == random_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k6Od5fpZ-S--"
   },
   "source": [
    "The random tensors we've made are actually [pseudorandom numbers](https://www.computerhope.com/jargon/p/pseudo-random.htm) (they appear as random, but really aren't).\n",
    "\n",
    "If we set a seed we'll get the same random numbers (if you've ever used NumPy, this is similar to `np.random.seed(42)`). \n",
    "\n",
    "Setting the seed says, \"hey, create some random numbers, but flavour them with X\" (X is the seed).\n",
    "\n",
    "What do you think will happen when we change the seed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9eStLqr1F4ZP",
    "outputId": "cf545015-ce96-46dd-9d43-fbb148a04782"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       " array([[-0.7565803 , -0.06854702],\n",
       "        [ 0.07595026, -1.2573844 ],\n",
       "        [-0.23193765, -1.8107855 ]], dtype=float32)>,\n",
       " <tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       " array([[ 0.2730574 , -0.29925638],\n",
       "        [-0.3652325 ,  0.61883307],\n",
       "        [-1.0130816 ,  0.2829171 ]], dtype=float32)>,\n",
       " <tf.Tensor: shape=(3, 2), dtype=bool, numpy=\n",
       " array([[ True,  True],\n",
       "        [ True,  True],\n",
       "        [ True,  True]])>,\n",
       " <tf.Tensor: shape=(3, 2), dtype=bool, numpy=\n",
       " array([[False, False],\n",
       "        [False, False],\n",
       "        [False, False]])>)"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create two random (and different) tensors\n",
    "random_3 = tf.random.Generator.from_seed(42)\n",
    "random_3 = random_3.normal(shape=(3, 2))\n",
    "random_4 = tf.random.Generator.from_seed(11)\n",
    "random_4 = random_4.normal(shape=(3, 2))\n",
    "\n",
    "# Check the tensors and see if they are equal\n",
    "random_3, random_4, random_1 == random_3, random_3 == random_4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nji9AdFRIhBi"
   },
   "source": [
    "What if you wanted to shuffle the order of a tensor?\n",
    "\n",
    "Wait, why would you want to do that?\n",
    "\n",
    "Let's say you working with 15,000 images of cats and dogs and the first 10,000 images of were of cats and the next 5,000 were of dogs. This order could effect how a neural network learns (it may overfit by learning the order of the data), instead, it might be a good idea to move your data around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sl4HYEWMBI6x",
    "outputId": "195d45c3-9c2e-4e7d-ea71-f3382b4dc6b0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[10,  7],\n",
       "       [ 2,  5],\n",
       "       [ 3,  4]], dtype=int32)>"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shuffle a tensor (valuable for when you want to shuffle your data)\n",
    "not_shuffled = tf.constant([[10, 7],\n",
    "                            [3, 4],\n",
    "                            [2, 5]])\n",
    "# Gets different results each time\n",
    "tf.random.shuffle(not_shuffled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-HYn0ME_H1SY",
    "outputId": "1a5c9383-9ca8-47c2-90a8-d2002dbf4ef8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[ 2,  5],\n",
       "       [ 3,  4],\n",
       "       [10,  7]], dtype=int32)>"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shuffle in the same order every time using the seed parameter (won't acutally be the same)\n",
    "tf.random.shuffle(not_shuffled, seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GmC3qGIHjAx6"
   },
   "source": [
    "Wait... why didn't the numbers come out the same?\n",
    "\n",
    "It's due to rule #4 of the [`tf.random.set_seed()`](https://www.tensorflow.org/api_docs/python/tf/random/set_seed) documentation.\n",
    "\n",
    "> \"4. If both the global and the operation seed are set: Both seeds are used in conjunction to determine the random sequence.\"\n",
    "\n",
    "`tf.random.set_seed(42)` sets the global seed, and the `seed` parameter in `tf.random.shuffle(seed=42)` sets the operation seed.\n",
    "\n",
    "Because, \"Operations that rely on a random seed actually derive it from two seeds: the global and operation-level seeds. This sets the global seed.\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cM6S8set-ixV",
    "outputId": "3ab2f553-5818-46eb-d276-a14853693bdc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[10,  7],\n",
       "       [ 3,  4],\n",
       "       [ 2,  5]], dtype=int32)>"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shuffle in the same order every time\n",
    "\n",
    "# Set the global random seed\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# Set the operation random seed\n",
    "tf.random.shuffle(not_shuffled, seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xKJOsdE8yCn4",
    "outputId": "9cd2788b-95a7-4f31-997c-313a2297541d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[ 3,  4],\n",
       "       [ 2,  5],\n",
       "       [10,  7]], dtype=int32)>"
      ]
     },
     "execution_count": 22,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the global random seed\n",
    "tf.random.set_seed(42) # if you comment this out you'll get different results\n",
    "\n",
    "# Set the operation random seed\n",
    "tf.random.shuffle(not_shuffled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ouZ1fpJk0R3h"
   },
   "source": [
    "### Other ways to make tensors\n",
    "\n",
    "Though you might rarely use these (remember, many tensor operations are done behind the scenes for you), you can use [`tf.ones()`](https://www.tensorflow.org/api_docs/python/tf/ones) to create a tensor of all ones and [`tf.zeros()`](https://www.tensorflow.org/api_docs/python/tf/zeros) to create a tensor of all zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aG8QNZP7kEe1",
    "outputId": "9ab2ec05-8b1d-45b0-a886-392e63d69149"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.]], dtype=float32)>"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a tensor of all ones\n",
    "tf.ones(shape=(3, 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GQKiWrB9kprj",
    "outputId": "8cd8ccee-2546-4473-ca89-0789100fbc1c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a tensor of all zeros\n",
    "tf.zeros(shape=(3, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "slcLTK5D7kc2"
   },
   "source": [
    "You can also turn NumPy arrays in into tensors.\n",
    "\n",
    "Remember, the main difference between tensors and NumPy arrays is that tensors can be run on GPUs.\n",
    "\n",
    "> 🔑 **Note:** A matrix or tensor is typically represented by a capital letter (e.g. `X` or `A`) where as a vector is typically represented by a lowercase letter (e.g. `y` or `b`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "C0XP37xi7mn4",
    "outputId": "f59af147-58d8-4239-bbc1-1f140a858118"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,\n",
       "        18, 19, 20, 21, 22, 23, 24], dtype=int32),\n",
       " <tf.Tensor: shape=(2, 4, 3), dtype=int32, numpy=\n",
       " array([[[ 1,  2,  3],\n",
       "         [ 4,  5,  6],\n",
       "         [ 7,  8,  9],\n",
       "         [10, 11, 12]],\n",
       " \n",
       "        [[13, 14, 15],\n",
       "         [16, 17, 18],\n",
       "         [19, 20, 21],\n",
       "         [22, 23, 24]]], dtype=int32)>)"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "numpy_A = np.arange(1, 25, dtype=np.int32) # create a NumPy array between 1 and 25\n",
    "A = tf.constant(numpy_A,  \n",
    "                shape=[2, 4, 3]) # note: the shape total (2*4*3) has to match the number of elements in the array\n",
    "numpy_A, A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J1JLXa2P0wpx"
   },
   "source": [
    "## Getting information from tensors (shape, rank, size)\n",
    "\n",
    "There will be times when you'll want to get different pieces of information from your tensors, in particuluar, you should know the following tensor vocabulary:\n",
    "* **Shape:** The length (number of elements) of each of the dimensions of a tensor.\n",
    "* **Rank:** The number of tensor dimensions. A scalar has rank 0, a vector has rank 1, a matrix is rank 2, a tensor has rank n.\n",
    "* **Axis** or **Dimension:** A particular dimension of a tensor.\n",
    "* **Size:** The total number of items in the tensor.\n",
    "\n",
    "You'll use these especially when you're trying to line up the shapes of your data to the shapes of your model. For example, making sure the shape of your image tensors are the same shape as your models input layer.\n",
    "\n",
    "We've already seen one of these before using the `ndim` attribute. Let's see the rest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qhckrmovCaAA",
    "outputId": "ac4307f8-d850-4c45-eaff-7cfa8a3919e9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3, 4, 5), dtype=float32, numpy=\n",
       "array([[[[0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0.]]]], dtype=float32)>"
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a rank 4 tensor (4 dimensions)\n",
    "rank_4_tensor = tf.zeros([2, 3, 4, 5])\n",
    "rank_4_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ImJdhWnLtZ_2",
    "outputId": "92b9e814-51b1-4374-ae81-516dcc2e0c49"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([2, 3, 4, 5]), 4, <tf.Tensor: shape=(), dtype=int32, numpy=120>)"
      ]
     },
     "execution_count": 27,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rank_4_tensor.shape, rank_4_tensor.ndim, tf.size(rank_4_tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Vvb-4ZYdpI9f",
    "outputId": "0f1b35eb-b93f-422e-8979-17260c621ffb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datatype of every element: <dtype: 'float32'>\n",
      "Number of dimensions (rank): 4\n",
      "Shape of tensor: (2, 3, 4, 5)\n",
      "Elements along axis 0 of tensor: 2\n",
      "Elements along last axis of tensor: 5\n",
      "Total number of elements (2*3*4*5): 120\n"
     ]
    }
   ],
   "source": [
    "# Get various attributes of tensor\n",
    "print(\"Datatype of every element:\", rank_4_tensor.dtype)\n",
    "print(\"Number of dimensions (rank):\", rank_4_tensor.ndim)\n",
    "print(\"Shape of tensor:\", rank_4_tensor.shape)\n",
    "print(\"Elements along axis 0 of tensor:\", rank_4_tensor.shape[0])\n",
    "print(\"Elements along last axis of tensor:\", rank_4_tensor.shape[-1])\n",
    "print(\"Total number of elements (2*3*4*5):\", tf.size(rank_4_tensor).numpy()) # .numpy() converts to NumPy array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S0SMO2ZOqL0G"
   },
   "source": [
    "You can also index tensors just like Python lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CFzOo-7QqLJf",
    "outputId": "2629d691-a8de-4488-915a-b06868f5218d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2, 2, 2), dtype=float32, numpy=\n",
       "array([[[[0., 0.],\n",
       "         [0., 0.]],\n",
       "\n",
       "        [[0., 0.],\n",
       "         [0., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0.],\n",
       "         [0., 0.]],\n",
       "\n",
       "        [[0., 0.],\n",
       "         [0., 0.]]]], dtype=float32)>"
      ]
     },
     "execution_count": 29,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the first 2 items of each dimension\n",
    "rank_4_tensor[:2, :2, :2, :2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "weQe2bBUqknd",
    "outputId": "16c61a8c-2ba3-42a3-d4b8-d19185553300"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 1, 1, 5), dtype=float32, numpy=array([[[[0., 0., 0., 0., 0.]]]], dtype=float32)>"
      ]
     },
     "execution_count": 30,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the dimension from each index except for the final one\n",
    "rank_4_tensor[:1, :1, :1, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YQKcZWz5rFXG",
    "outputId": "15a0bee6-6467-46b7-c315-e5ec963de8bd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=int32, numpy=array([7, 4], dtype=int32)>"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a rank 2 tensor (2 dimensions)\n",
    "rank_2_tensor = tf.constant([[10, 7],\n",
    "                             [3, 4]])\n",
    "\n",
    "# Get the last item of each row\n",
    "rank_2_tensor[:, -1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cLZq3sHKrVdS"
   },
   "source": [
    "You can also add dimensions to your tensor whilst keeping the same information present using `tf.newaxis`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KuEEEQa4w1id",
    "outputId": "998dcc25-889f-4c5b-e417-f9e45dcaa350"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       " array([[10,  7],\n",
       "        [ 3,  4]], dtype=int32)>,\n",
       " <tf.Tensor: shape=(2, 2, 1), dtype=int32, numpy=\n",
       " array([[[10],\n",
       "         [ 7]],\n",
       " \n",
       "        [[ 3],\n",
       "         [ 4]]], dtype=int32)>)"
      ]
     },
     "execution_count": 32,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add an extra dimension (to the end)\n",
    "rank_3_tensor = rank_2_tensor[..., tf.newaxis] # in Python \"...\" means \"all dimensions prior to\"\n",
    "rank_2_tensor, rank_3_tensor # shape (2, 2), shape (2, 2, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a5_KyB-6r7z7"
   },
   "source": [
    "You can achieve the same using [`tf.expand_dims()`](https://www.tensorflow.org/api_docs/python/tf/expand_dims)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HpPTBqt4rvr9",
    "outputId": "47a95be1-ba16-475f-b2cd-437f115cd5ac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2, 1), dtype=int32, numpy=\n",
       "array([[[10],\n",
       "        [ 7]],\n",
       "\n",
       "       [[ 3],\n",
       "        [ 4]]], dtype=int32)>"
      ]
     },
     "execution_count": 33,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.expand_dims(rank_2_tensor, axis=-1) # \"-1\" means last axis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4EHae9iA04Ok"
   },
   "source": [
    "## Manipulating tensors (tensor operations)\n",
    "\n",
    "Finding patterns in tensors (numberical representation of data) requires manipulating them.\n",
    "\n",
    "Again, when building models in TensorFlow, much of this pattern discovery is done for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-NzdbYDqs1Ex"
   },
   "source": [
    "### Basic operations\n",
    "\n",
    "You can perform many of the basic mathematical operations directly on tensors using Pyhton operators such as, `+`, `-`, `*`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tu3zJirLsMVw",
    "outputId": "7dce405b-292e-4d7a-efdd-eeda89807887"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[20, 17],\n",
       "       [13, 14]], dtype=int32)>"
      ]
     },
     "execution_count": 34,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can add values to a tensor using the addition operator\n",
    "tensor = tf.constant([[10, 7], [3, 4]])\n",
    "tensor + 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_smxbIkYwYY3"
   },
   "source": [
    "Since we used `tf.constant()`, the original tensor is unchanged (the addition gets done on a copy)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BhJn3puhwOlM",
    "outputId": "92ce1cbe-e4c8-4f4e-c50b-c21d4ea57d43"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[10,  7],\n",
       "       [ 3,  4]], dtype=int32)>"
      ]
     },
     "execution_count": 35,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Original tensor unchanged\n",
    "tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5N6tUM16xq9d"
   },
   "source": [
    "Other operators also work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6TW0_ZC_xoEC",
    "outputId": "f6aa4f99-a9e8-4e13-c06c-5c9450ef99c4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[100,  70],\n",
       "       [ 30,  40]], dtype=int32)>"
      ]
     },
     "execution_count": 36,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Multiplication (known as element-wise multiplication)\n",
    "tensor * 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MN6XjwWfxu66",
    "outputId": "ca0a775c-e2e3-4a53-fbae-e4ced3f2c91c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[ 0, -3],\n",
       "       [-7, -6]], dtype=int32)>"
      ]
     },
     "execution_count": 37,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Subtraction\n",
    "tensor - 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1kMJe4GlyuZR"
   },
   "source": [
    "You can also use the equivalent TensorFlow function. Using the TensorFlow function (where possible) has the advantage of being sped up later down the line when running as part of a [TensorFlow graph](https://www.tensorflow.org/tensorboard/graphs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "R2NDjqYIyyMc",
    "outputId": "b811b7c6-2c4a-4f89-fae6-89ceaf8dfa5e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[100,  70],\n",
       "       [ 30,  40]], dtype=int32)>"
      ]
     },
     "execution_count": 38,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use the tensorflow function equivalent of the '*' (multiply) operator\n",
    "tf.multiply(tensor, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lKEuDBFD49w7",
    "outputId": "99300b75-0f81-432b-f3d0-b32e4c651b5b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[10,  7],\n",
       "       [ 3,  4]], dtype=int32)>"
      ]
     },
     "execution_count": 39,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The original tensor is still unchanged\n",
    "tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8M-0dhjtzHoZ"
   },
   "source": [
    "### Matrix mutliplication\n",
    "\n",
    "One of the most common operations in machine learning algorithms is [matrix multiplication](https://www.mathsisfun.com/algebra/matrix-multiplying.html).\n",
    "\n",
    "TensorFlow implements this matrix multiplication functionality in the [`tf.matmul()`](https://www.tensorflow.org/api_docs/python/tf/linalg/matmul) method.\n",
    "\n",
    "The main two rules for matrix multiplication to remember are:\n",
    "1. The inner dimensions must match:\n",
    "  * `(3, 5) @ (3, 5)` won't work\n",
    "  * `(5, 3) @ (3, 5)` will work\n",
    "  * `(3, 5) @ (5, 3)` will work\n",
    "2. The resulting matrix has the shape of the inner dimensions:\n",
    " * `(5, 3) @ (3, 5)` -> `(3, 3)`\n",
    " * `(3, 5) @ (5, 3)` -> `(5, 5)`\n",
    "\n",
    "> 🔑 **Note:** '`@`' in Python is the symbol for matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pbpwVJrAsPpA",
    "outputId": "580b001a-f0c1-45ef-9918-cc5538b0fb1e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[10  7]\n",
      " [ 3  4]], shape=(2, 2), dtype=int32)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[121,  98],\n",
       "       [ 42,  37]], dtype=int32)>"
      ]
     },
     "execution_count": 40,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Matrix multiplication in TensorFlow\n",
    "print(tensor)\n",
    "tf.matmul(tensor, tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9vpDnpb10G7U",
    "outputId": "6693b6f2-e529-41f3-b08b-c19a90df1137"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[121,  98],\n",
       "       [ 42,  37]], dtype=int32)>"
      ]
     },
     "execution_count": 41,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Matrix multiplication with Python operator '@'\n",
    "tensor @ tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TAV3S5YV0cDQ"
   },
   "source": [
    "Both of these examples work because our `tensor` variable is of shape (2, 2).\n",
    "\n",
    "What if we created some tensors which had mismatched shapes?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UXSE6q1o0amm",
    "outputId": "77694ca8-b6c5-4997-8e0e-9cb43bd78145"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       " array([[1, 2],\n",
       "        [3, 4],\n",
       "        [5, 6]], dtype=int32)>, <tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       " array([[ 7,  8],\n",
       "        [ 9, 10],\n",
       "        [11, 12]], dtype=int32)>)"
      ]
     },
     "execution_count": 42,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create (3, 2) tensor\n",
    "X = tf.constant([[1, 2],\n",
    "                 [3, 4],\n",
    "                 [5, 6]])\n",
    "\n",
    "# Create another (3, 2) tensor\n",
    "Y = tf.constant([[7, 8],\n",
    "                 [9, 10],\n",
    "                 [11, 12]])\n",
    "X, Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "id": "3J4DGQa309Hc",
    "outputId": "d7a867bd-6c5f-4ef9-b9a9-b00c89b1cba1"
   },
   "outputs": [
    {
     "ename": "InvalidArgumentError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-43-62e1e4702ffd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Try to matrix multiply them (will error)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mX\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py\u001b[0m in \u001b[0;36mbinary_op_wrapper\u001b[0;34m(x, y)\u001b[0m\n\u001b[1;32m   1162\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname_scope\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mop_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1163\u001b[0m       \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1164\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1165\u001b[0m       \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mTypeError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1166\u001b[0m         \u001b[0;31m# Even if dispatching the op failed, the RHS may be a tensor aware\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    199\u001b[0m     \u001b[0;34m\"\"\"Call target, and fall back on dispatchers if there is a TypeError.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    200\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 201\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mtarget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    202\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mTypeError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m       \u001b[0;31m# Note: convert_to_eager_tensor currently raises a ValueError, not a\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py\u001b[0m in \u001b[0;36mmatmul\u001b[0;34m(a, b, transpose_a, transpose_b, adjoint_a, adjoint_b, a_is_sparse, b_is_sparse, name)\u001b[0m\n\u001b[1;32m   3313\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3314\u001b[0m       return gen_math_ops.mat_mul(\n\u001b[0;32m-> 3315\u001b[0;31m           a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)\n\u001b[0m\u001b[1;32m   3316\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3317\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/gen_math_ops.py\u001b[0m in \u001b[0;36mmat_mul\u001b[0;34m(a, b, transpose_a, transpose_b, name)\u001b[0m\n\u001b[1;32m   5530\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0m_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5531\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0m_core\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5532\u001b[0;31m       \u001b[0m_ops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_from_not_ok_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   5533\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0m_core\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_FallbackException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5534\u001b[0m       \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py\u001b[0m in \u001b[0;36mraise_from_not_ok_status\u001b[0;34m(e, name)\u001b[0m\n\u001b[1;32m   6860\u001b[0m   \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\" name: \"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   6861\u001b[0m   \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 6862\u001b[0;31m   \u001b[0msix\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_from\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_status_to_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   6863\u001b[0m   \u001b[0;31m# pylint: enable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   6864\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/six.py\u001b[0m in \u001b[0;36mraise_from\u001b[0;34m(value, from_value)\u001b[0m\n",
      "\u001b[0;31mInvalidArgumentError\u001b[0m: Matrix size-incompatible: In[0]: [3,2], In[1]: [3,2] [Op:MatMul]"
     ]
    }
   ],
   "source": [
    "# Try to matrix multiply them (will error)\n",
    "X @ Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v27FQ59v1N-H"
   },
   "source": [
    "Trying to matrix multiply two tensors with the shape `(3, 2)` errors because the inner dimensions don't match.\n",
    "\n",
    "We need to either:\n",
    "* Reshape X to `(2, 3)` so it's `(2, 3) @ (3, 2)`.\n",
    "* Reshape Y to `(3, 2)` so it's `(3, 2) @ (2, 3)`.\n",
    "\n",
    "We can do this with either:\n",
    "* [`tf.reshape()`](https://www.tensorflow.org/api_docs/python/tf/reshape) - allows us to reshape a tensor into a defined shape.\n",
    "* [`tf.transpose()`](https://www.tensorflow.org/api_docs/python/tf/transpose) - switches the dimensions of a given tensor.\n",
    "\n",
    "![lining up dimensions for dot products](https://github.com/raw/mrdbourke/tensorflow-deep-learning/main/images/00-lining-up-dot-products.png)\n",
    "\n",
    "Let's try `tf.reshape()` first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZwvVl-k_2W9u",
    "outputId": "e408fd19-4705-476d-f1c5-302ff6859e54"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=int32, numpy=\n",
       "array([[ 7,  8,  9],\n",
       "       [10, 11, 12]], dtype=int32)>"
      ]
     },
     "execution_count": 44,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example of reshape (3, 2) -> (2, 3)\n",
    "tf.reshape(Y, shape=(2, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1jBKPQVn1Nep",
    "outputId": "19906c7b-3e3b-4611-a639-7c8a3fff1d6a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 3), dtype=int32, numpy=\n",
       "array([[ 27,  30,  33],\n",
       "       [ 61,  68,  75],\n",
       "       [ 95, 106, 117]], dtype=int32)>"
      ]
     },
     "execution_count": 45,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try matrix multiplication with reshaped Y\n",
    "X @ tf.reshape(Y, shape=(2, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CWAgHKNk2_TO"
   },
   "source": [
    "It worked, let's try the same with a reshaped `X`, except this time we'll use [`tf.transpose()`](https://www.tensorflow.org/api_docs/python/tf/transpose) and `tf.matmul()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qA2rCnik2OnQ",
    "outputId": "f23907d3-f052-4702-dcbb-371e0ccb8df3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=int32, numpy=\n",
       "array([[1, 3, 5],\n",
       "       [2, 4, 6]], dtype=int32)>"
      ]
     },
     "execution_count": 46,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example of transpose (3, 2) -> (2, 3)\n",
    "tf.transpose(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zR8YdMfh3G0S",
    "outputId": "cd2fbe33-97c6-4e54-c27f-8febe63c8e4f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[ 89,  98],\n",
       "       [116, 128]], dtype=int32)>"
      ]
     },
     "execution_count": 47,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try matrix multiplication \n",
    "tf.matmul(tf.transpose(X), Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SL45P1cC5tnJ",
    "outputId": "b2aa00a3-c1e2-44a5-a574-48a0a721f0ee"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[ 89,  98],\n",
       "       [116, 128]], dtype=int32)>"
      ]
     },
     "execution_count": 48,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can achieve the same result with parameters\n",
    "tf.matmul(a=X, b=Y, transpose_a=True, transpose_b=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AqE518TJ3GXG"
   },
   "source": [
    "Notice the difference in the resulting shapes when tranposing `X` or reshaping `Y`.\n",
    "\n",
    "This is because of the 2nd rule mentioned above:\n",
    " * `(3, 2) @ (2, 3)` -> `(2, 2)` done with `tf.matmul(tf.transpose(X), Y)`\n",
    " * `(2, 3) @ (3, 2)` -> `(3, 3)` done with `X @ tf.reshape(Y, shape=(2, 3))`\n",
    "\n",
    "This kind of data manipulation is a reminder: you'll spend a lot of your time in machine learning and working with neural networks reshaping data (in the form of tensors) to prepare it to be used with various operations (such as feeding it to a model).\n",
    "\n",
    "### The dot product\n",
    "\n",
    "Multiplying matrices by eachother is also referred to as the dot product.\n",
    "\n",
    "You can perform the `tf.matmul()` operation using [`tf.tensordot()`](https://www.tensorflow.org/api_docs/python/tf/tensordot). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qfSJHDpe2Oe9",
    "outputId": "67aab203-89c1-4b2f-afe5-b96416f043d9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[ 89,  98],\n",
       "       [116, 128]], dtype=int32)>"
      ]
     },
     "execution_count": 49,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform the dot product on X and Y (requires X to be transposed)\n",
    "tf.tensordot(tf.transpose(X), Y, axes=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "waJcSOGf_Fg7"
   },
   "source": [
    "You might notice that although using both `reshape` and `tranpose` work, you get different results when using each.\n",
    "\n",
    "Let's see an example, first with `tf.transpose()` then with `tf.reshape()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AAzB-F4l6Dc0",
    "outputId": "f2c1545c-e2e5-4e03-9319-65985cd299da"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 3), dtype=int32, numpy=\n",
       "array([[ 23,  29,  35],\n",
       "       [ 53,  67,  81],\n",
       "       [ 83, 105, 127]], dtype=int32)>"
      ]
     },
     "execution_count": 50,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform matrix multiplication between X and Y (transposed)\n",
    "tf.matmul(X, tf.transpose(Y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "s-kQH7qh69PV",
    "outputId": "f8ff25a7-a7e6-4fbc-c1b7-91059ca8cfa7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 3), dtype=int32, numpy=\n",
       "array([[ 27,  30,  33],\n",
       "       [ 61,  68,  75],\n",
       "       [ 95, 106, 117]], dtype=int32)>"
      ]
     },
     "execution_count": 51,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform matrix multiplication between X and Y (reshaped)\n",
    "tf.matmul(X, tf.reshape(Y, (2, 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2eCDnCX6AhbF"
   },
   "source": [
    "Hmm... they result in different values.\n",
    "\n",
    "Which is strange because when dealing with `Y` (a `(3x2)` matrix), reshaping to `(2, 3)` and tranposing it result in the same shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "P_RLV373ATAb",
    "outputId": "202e1c47-504a-4254-bcab-faac96ab9f86"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([3, 2]), TensorShape([2, 3]), TensorShape([2, 3]))"
      ]
     },
     "execution_count": 52,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check shapes of Y, reshaped Y and tranposed Y\n",
    "Y.shape, tf.reshape(Y, (2, 3)).shape, tf.transpose(Y).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OilqUMBKAevX"
   },
   "source": [
    "But calling `tf.reshape()` and `tf.transpose()` on `Y` don't necessarily result in the same values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "B5_aYjqeA_w_",
    "outputId": "39d8c934-c06e-4442-8a9d-fb91a7fb82e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal Y:\n",
      "tf.Tensor(\n",
      "[[ 7  8]\n",
      " [ 9 10]\n",
      " [11 12]], shape=(3, 2), dtype=int32) \n",
      "\n",
      "Y reshaped to (2, 3):\n",
      "tf.Tensor(\n",
      "[[ 7  8  9]\n",
      " [10 11 12]], shape=(2, 3), dtype=int32) \n",
      "\n",
      "Y transposed:\n",
      "tf.Tensor(\n",
      "[[ 7  9 11]\n",
      " [ 8 10 12]], shape=(2, 3), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# Check values of Y, reshape Y and tranposed Y\n",
    "print(\"Normal Y:\")\n",
    "print(Y, \"\\n\") # \"\\n\" for newline\n",
    "\n",
    "print(\"Y reshaped to (2, 3):\")\n",
    "print(tf.reshape(Y, (2, 3)), \"\\n\")\n",
    "\n",
    "print(\"Y transposed:\")\n",
    "print(tf.transpose(Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g9g3l45LFO7K"
   },
   "source": [
    "As you can see, the outputs of `tf.reshape()` and `tf.transpose()` when called on `Y`, even though they have the same shape, are different.\n",
    "\n",
    "This can be explained by the default behaviour of each method:\n",
    "* [`tf.reshape()`](https://www.tensorflow.org/api_docs/python/tf/reshape) - change the shape of the given tensor (first) and then insert values in order they appear (in our case, 7, 8, 9, 10, 11, 12).\n",
    "* [`tf.transpose()`](https://www.tensorflow.org/api_docs/python/tf/transpose) - swap the order of the axes, by default the last axis becomes the first, however the order can be changed using the [`perm` parameter](https://www.tensorflow.org/api_docs/python/tf/transpose)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fzjcZ4FHCOb5"
   },
   "source": [
    "So which should you use?\n",
    "\n",
    "Again, most of the time these operations (when they need to be run, such as during the training a neural network, will be implemented for you).\n",
    "\n",
    "But generally, whenever performing a matrix multiplication and the shapes of two matrices don't line up, you will transpose (not reshape) one of them in order to line them up.\n",
    "\n",
    "### Matrix multiplication tidbits\n",
    "* If we transposed `Y`, it would be represented as $\\mathbf{Y}^\\mathsf{T}$ (note the capital T for tranpose).\n",
    "* Get an illustrative view of matrix multiplication [by Math is Fun](https://www.mathsisfun.com/algebra/matrix-multiplying.html).\n",
    "* Try a hands-on demo of matrix multiplcation: http://matrixmultiplication.xyz/ (shown below).\n",
    "\n",
    "![visual demo of matrix multiplication](https://github.com/raw/mrdbourke/tensorflow-deep-learning/main/images/00-matrix-multiply-crop.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bK8Kc94SL-JL"
   },
   "source": [
    "### Changing the datatype of a tensor\n",
    "\n",
    "Sometimes you'll want to alter the default datatype of your tensor. \n",
    "\n",
    "This is common when you want to compute using less precision (e.g. 16-bit floating point numbers vs. 32-bit floating point numbers). \n",
    "\n",
    "Computing with less precision is useful on devices with less computing capacity such as mobile devices (because the less bits, the less space the computations require).\n",
    "\n",
    "You can change the datatype of a tensor using [`tf.cast()`](https://www.tensorflow.org/api_docs/python/tf/cast)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jRkZhW35Lge_",
    "outputId": "a205ee88-9f2f-485c-e6e5-3be0b080fb6f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(2,), dtype=float32, numpy=array([1.7, 7.4], dtype=float32)>,\n",
       " <tf.Tensor: shape=(2,), dtype=int32, numpy=array([1, 7], dtype=int32)>)"
      ]
     },
     "execution_count": 54,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new tensor with default datatype (float32)\n",
    "B = tf.constant([1.7, 7.4])\n",
    "\n",
    "# Create a new tensor with default datatype (int32)\n",
    "C = tf.constant([1, 7])\n",
    "B, C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kdhV1rgcNpUP",
    "outputId": "83a85f1c-fdd3-44e9-f854-75c36b94af3b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float16, numpy=array([1.7, 7.4], dtype=float16)>"
      ]
     },
     "execution_count": 55,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change from float32 to float16 (reduced precision)\n",
    "B = tf.cast(B, dtype=tf.float16)\n",
    "B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Px2E1ANeNxRv",
    "outputId": "be744bf6-beca-4b27-e63d-ae1255be0593"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([1., 7.], dtype=float32)>"
      ]
     },
     "execution_count": 56,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change from int32 to float32\n",
    "C = tf.cast(C, dtype=tf.float32)\n",
    "C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1F3L1BDuQVJz"
   },
   "source": [
    "### Getting the absolute value\n",
    "Sometimes you'll want the absolute values (all values are positive) of elements in your tensors.\n",
    "\n",
    "To do so, you can use [`tf.abs()`](https://www.tensorflow.org/api_docs/python/tf/math/abs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "plNBFi51QOvW",
    "outputId": "79dbe7c8-e212-4b0d-9ee1-965c496afe28"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=int32, numpy=array([ -7, -10], dtype=int32)>"
      ]
     },
     "execution_count": 57,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create tensor with negative values\n",
    "D = tf.constant([-7, -10])\n",
    "D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bOSYmX37QFHS",
    "outputId": "8aa0a217-54e1-4edf-b812-4b0eb05f61a5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=int32, numpy=array([ 7, 10], dtype=int32)>"
      ]
     },
     "execution_count": 58,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the absolute values\n",
    "tf.abs(D)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G4cnALehQ2CE"
   },
   "source": [
    "### Finding the min, max, mean, sum (aggregation)\n",
    "\n",
    "You can quickly aggregate (perform a calculation on a whole tensor) tensors to find things like the minimum value, maximum value, mean and sum of all the elements.\n",
    "\n",
    "To do so, aggregation methods typically have the syntax `reduce()_[action]`, such as:\n",
    "* [`tf.reduce_min()`](https://www.tensorflow.org/api_docs/python/tf/math/reduce_min) - find the minimum value in a tensor.\n",
    "* [`tf.reduce_max()`](https://www.tensorflow.org/api_docs/python/tf/math/reduce_max) - find the maximum value in a tensor (helpful for when you want to find the highest prediction probability).\n",
    "* [`tf.reduce_mean()`](https://www.tensorflow.org/api_docs/python/tf/math/reduce_mean) - find the mean of all elements in a tensor.\n",
    "* [`tf.reduce_sum()`](https://www.tensorflow.org/api_docs/python/tf/math/reduce_sum) - find the sum of all elements in a tensor.\n",
    "* **Note:** typically, each of these is under the `math` module, e.g. `tf.math.reduce_min()` but you can use the alias `tf.reduce_min()`.\n",
    "\n",
    "Let's see them in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kjW5_6i6Q7oo",
    "outputId": "7145a7b4-d9da-4120-a2cb-5ebe2bbc0f17"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(50,), dtype=int64, numpy=\n",
       "array([79, 33, 64, 90, 16,  3, 70, 51, 27, 54, 71, 65,  4,  8, 78, 84, 11,\n",
       "       23, 15,  2, 55, 74, 26, 58,  9, 74, 33, 49, 48,  4, 30, 41, 69, 17,\n",
       "       39, 37,  3, 63, 13, 37, 72, 80, 93, 29, 61, 82, 85, 40, 79, 14])>"
      ]
     },
     "execution_count": 59,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a tensor with 50 random values between 0 and 100\n",
    "E = tf.constant(np.random.randint(low=0, high=100, size=50))\n",
    "E"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "slwMVgT-Rac0",
    "outputId": "34cf2c05-53b9-44eb-9df5-28df227794d4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int64, numpy=2>"
      ]
     },
     "execution_count": 60,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the minimum\n",
    "tf.reduce_min(E)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "voLqafGCRYqO",
    "outputId": "8bec8619-45e8-4b95-970b-4ee3b7c55d77"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int64, numpy=93>"
      ]
     },
     "execution_count": 61,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the maximum\n",
    "tf.reduce_max(E)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MPqwvy6TRYN3",
    "outputId": "905c4bf8-5d3f-443f-f92b-74752800987a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int64, numpy=45>"
      ]
     },
     "execution_count": 62,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the mean\n",
    "tf.reduce_mean(E)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lqwCXeD6RhyE",
    "outputId": "4ce00693-124a-4875-a942-5a48672ace3c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int64, numpy=2262>"
      ]
     },
     "execution_count": 63,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the sum\n",
    "tf.reduce_sum(E)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zXGyiVPiRqgO"
   },
   "source": [
    "You can also find the standard deviation ([`tf.reduce_std()`](https://www.tensorflow.org/api_docs/python/tf/math/reduce_std)) and variance ([`tf.reduce_variance()`](https://www.tensorflow.org/api_docs/python/tf/math/reduce_variance)) of elements in a tensor using similar methods.\n",
    "\n",
    "### Finding the positional maximum and minimum\n",
    "\n",
    "How about finding the position a tensor where the maximum value occurs?\n",
    "\n",
    "This is helpful when you want to line up your labels (say `['Green', 'Blue', 'Red']`) with your prediction probabilities tensor (e.g. `[0.98, 0.01, 0.01]`).\n",
    "\n",
    "In this case, the predicted label (the one with the highest prediction probability) would be `'Green'`.\n",
    "\n",
    "You can do the same for the minimum (if required) with the following:\n",
    "* [`tf.argmax()`](https://www.tensorflow.org/api_docs/python/tf/math/argmax) - find the position of the maximum element in a given tensor.\n",
    "* [`tf.argmin()`](https://www.tensorflow.org/api_docs/python/tf/math/argmin) - find the position of the minimum element in a given tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PspO0Vjp3Nm6",
    "outputId": "74c67bd9-a883-4b67-a7a3-55654d506e77"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(50,), dtype=float64, numpy=\n",
       "array([0.55725069, 0.66589953, 0.51717871, 0.23663905, 0.03861408,\n",
       "       0.25753377, 0.58184723, 0.05496164, 0.97498269, 0.52503693,\n",
       "       0.68152575, 0.8151567 , 0.25854921, 0.15479351, 0.43657365,\n",
       "       0.34710353, 0.18739747, 0.7859743 , 0.11497802, 0.68242342,\n",
       "       0.74991327, 0.16043629, 0.5636553 , 0.4110329 , 0.62245869,\n",
       "       0.58129871, 0.44792704, 0.91472587, 0.76207828, 0.36502321,\n",
       "       0.96601855, 0.03094982, 0.53464709, 0.73241468, 0.0856144 ,\n",
       "       0.22923332, 0.66802859, 0.68604535, 0.53128266, 0.48685057,\n",
       "       0.53320669, 0.38020824, 0.81706951, 0.33855374, 0.94007405,\n",
       "       0.05419892, 0.10267899, 0.6020753 , 0.78038818, 0.87655176])>"
      ]
     },
     "execution_count": 64,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a tensor with 50 values between 0 and 1\n",
    "F = tf.constant(np.random.random(50))\n",
    "F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ADbAMm9N3Zlb",
    "outputId": "6fc95755-3b77-495e-f382-0a8b7f2cdef2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int64, numpy=8>"
      ]
     },
     "execution_count": 65,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the maximum element position of F\n",
    "tf.argmax(F)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aQrv1nVE3ckx",
    "outputId": "0a78d68f-6804-4125-da7c-d12a6811258e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int64, numpy=31>"
      ]
     },
     "execution_count": 66,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the minimum element position of F\n",
    "tf.argmin(F)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yFHzARFwLmIf",
    "outputId": "2fbf3ccb-5ae2-4f20-c351-ed98401a135d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The maximum value of F is at position: 8\n",
      "The maximum value of F is: 0.9749826885110175\n",
      "Using tf.argmax() to index F, the maximum value of F is: 0.9749826885110175\n",
      "Are the two max values the same (they should be)? True\n"
     ]
    }
   ],
   "source": [
    "# Find the maximum element position of F\n",
    "print(f\"The maximum value of F is at position: {tf.argmax(F).numpy()}\") \n",
    "print(f\"The maximum value of F is: {tf.reduce_max(F).numpy()}\") \n",
    "print(f\"Using tf.argmax() to index F, the maximum value of F is: {F[tf.argmax(F)].numpy()}\")\n",
    "print(f\"Are the two max values the same (they should be)? {F[tf.argmax(F)].numpy() == tf.reduce_max(F).numpy()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vSGIuNwm5QHM"
   },
   "source": [
    "### Squeezing a tensor (removing all single dimensions)\n",
    "\n",
    "If you need to remove single-dimensions from a tensor (dimensions with size 1), you can use `tf.squeeze()`.\n",
    "\n",
    "* [`tf.squeeze()`](https://www.tensorflow.org/api_docs/python/tf/squeeze) - remove all dimensions of 1 from a tensor.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6xDZLtNu5wUZ",
    "outputId": "b86ca328-b539-4701-e729-b7dc5c80dc49"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([1, 1, 1, 1, 50]), 5)"
      ]
     },
     "execution_count": 68,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a rank 5 (5 dimensions) tensor of 50 numbers between 0 and 100\n",
    "G = tf.constant(np.random.randint(0, 100, 50), shape=(1, 1, 1, 1, 50))\n",
    "G.shape, G.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "oS91XOgO6lai",
    "outputId": "0359d710-ca66-40c1-ac48-cd60c164f5fe"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([50]), 1)"
      ]
     },
     "execution_count": 69,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Squeeze tensor G (remove all 1 dimensions)\n",
    "G_squeezed = tf.squeeze(G)\n",
    "G_squeezed.shape, G_squeezed.ndim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "46cKe32W65Ox"
   },
   "source": [
    "### One-hot encoding\n",
    "\n",
    "If you have a tensor of indicies and would like to one-hot encode it, you can use [`tf.one_hot()`](https://www.tensorflow.org/api_docs/python/tf/one_hot).\n",
    "\n",
    "You should also specify the `depth` parameter (the level which you want to one-hot encode to)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FlRkMjL-646U",
    "outputId": "f49a6bee-e62a-4175-b84c-e4f8b5500d4f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 4), dtype=float32, numpy=\n",
       "array([[1., 0., 0., 0.],\n",
       "       [0., 1., 0., 0.],\n",
       "       [0., 0., 1., 0.],\n",
       "       [0., 0., 0., 1.]], dtype=float32)>"
      ]
     },
     "execution_count": 70,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a list of indices\n",
    "some_list = [0, 1, 2, 3]\n",
    "\n",
    "# One hot encode them\n",
    "tf.one_hot(some_list, depth=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xYxYV0r08THk"
   },
   "source": [
    "You can also specify values for `on_value` and `off_value` instead of the default `0` and `1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FZluadm88EcN",
    "outputId": "a129f06b-b23f-40d8-ed3a-ddbee35be070"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 4), dtype=string, numpy=\n",
       "array([[b\"We're live!\", b'Offline', b'Offline', b'Offline'],\n",
       "       [b'Offline', b\"We're live!\", b'Offline', b'Offline'],\n",
       "       [b'Offline', b'Offline', b\"We're live!\", b'Offline'],\n",
       "       [b'Offline', b'Offline', b'Offline', b\"We're live!\"]], dtype=object)>"
      ]
     },
     "execution_count": 71,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Specify custom values for on and off encoding\n",
    "tf.one_hot(some_list, depth=4, on_value=\"We're live!\", off_value=\"Offline\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-E-I1jFC84Qi"
   },
   "source": [
    "### Squaring, log, square root\n",
    "\n",
    "Many other common mathematical operations you'd like to perform at some stage, probably exist.\n",
    "\n",
    "Let's take a look at:\n",
    "* [`tf.square()`](https://www.tensorflow.org/api_docs/python/tf/math/square) - get the square of every value in a tensor. \n",
    "* [`tf.sqrt()`](https://www.tensorflow.org/api_docs/python/tf/math/sqrt) - get the squareroot of every value in a tensor (**note:** the elements need to be floats or this will error).\n",
    "* [`tf.math.log()`](https://www.tensorflow.org/api_docs/python/tf/math/log) - get the natural log of every value in a tensor (elements need to floats)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KTOvziCBLqhZ",
    "outputId": "9ef46ed1-6bd2-4e61-9068-11f5bb183150"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(9,), dtype=int64, numpy=array([1, 2, 3, 4, 5, 6, 7, 8, 9])>"
      ]
     },
     "execution_count": 72,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new tensor\n",
    "H = tf.constant(np.arange(1, 10))\n",
    "H"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qvPS3xbk9JBK",
    "outputId": "999851d2-5a61-4768-8a43-1b1862ffb0bd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(9,), dtype=int64, numpy=array([ 1,  4,  9, 16, 25, 36, 49, 64, 81])>"
      ]
     },
     "execution_count": 73,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Square it\n",
    "tf.square(H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 275
    },
    "id": "SJibI0GO9uf4",
    "outputId": "0a50bff8-6654-470c-dad8-7177cdf29d7b"
   },
   "outputs": [
    {
     "ename": "InvalidArgumentError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-74-d7db039da8bb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Find the squareroot (will error), needs to be non-integer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mH\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    199\u001b[0m     \u001b[0;34m\"\"\"Call target, and fall back on dispatchers if there is a TypeError.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    200\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 201\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mtarget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    202\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mTypeError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m       \u001b[0;31m# Note: convert_to_eager_tensor currently raises a ValueError, not a\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py\u001b[0m in \u001b[0;36msqrt\u001b[0;34m(x, name)\u001b[0m\n\u001b[1;32m   4901\u001b[0m     \u001b[0mA\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0mof\u001b[0m \u001b[0msame\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtype\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0msparsity\u001b[0m \u001b[0;32mas\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mx\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4902\u001b[0m   \"\"\"\n\u001b[0;32m-> 4903\u001b[0;31m   \u001b[0;32mreturn\u001b[0m \u001b[0mgen_math_ops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4904\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4905\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/gen_math_ops.py\u001b[0m in \u001b[0;36msqrt\u001b[0;34m(x, name)\u001b[0m\n\u001b[1;32m  10035\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0m_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m  10036\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0m_core\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m> 10037\u001b[0;31m       \u001b[0m_ops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_from_not_ok_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m  10038\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0m_core\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_FallbackException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m  10039\u001b[0m       \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py\u001b[0m in \u001b[0;36mraise_from_not_ok_status\u001b[0;34m(e, name)\u001b[0m\n\u001b[1;32m   6860\u001b[0m   \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\" name: \"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   6861\u001b[0m   \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 6862\u001b[0;31m   \u001b[0msix\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_from\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_status_to_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   6863\u001b[0m   \u001b[0;31m# pylint: enable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   6864\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/six.py\u001b[0m in \u001b[0;36mraise_from\u001b[0;34m(value, from_value)\u001b[0m\n",
      "\u001b[0;31mInvalidArgumentError\u001b[0m: Value for attr 'T' of int64 is not in the list of allowed values: bfloat16, half, float, double, complex64, complex128\n\t; NodeDef: {{node Sqrt}}; Op<name=Sqrt; signature=x:T -> y:T; attr=T:type,allowed=[DT_BFLOAT16, DT_HALF, DT_FLOAT, DT_DOUBLE, DT_COMPLEX64, DT_COMPLEX128]> [Op:Sqrt]"
     ]
    }
   ],
   "source": [
    "# Find the squareroot (will error), needs to be non-integer\n",
    "tf.sqrt(H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zlxRWy6Q-NHK",
    "outputId": "f61c7c31-34da-4c8f-8040-be761e939f95"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(9,), dtype=float32, numpy=array([1., 2., 3., 4., 5., 6., 7., 8., 9.], dtype=float32)>"
      ]
     },
     "execution_count": 75,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change H to float32\n",
    "H = tf.cast(H, dtype=tf.float32)\n",
    "H"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "S73eO0p--TtN",
    "outputId": "6ae3c553-1df5-4f5a-a5a1-e3597300c583"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(9,), dtype=float32, numpy=\n",
       "array([1.       , 1.4142135, 1.7320508, 2.       , 2.236068 , 2.4494898,\n",
       "       2.6457512, 2.828427 , 3.       ], dtype=float32)>"
      ]
     },
     "execution_count": 76,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the square root\n",
    "tf.sqrt(H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RyCo55vz9u4f",
    "outputId": "3a8f9f11-4a00-482b-dde5-b5cebee8dcde"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(9,), dtype=float32, numpy=\n",
       "array([0.       , 0.6931472, 1.0986123, 1.3862944, 1.609438 , 1.7917595,\n",
       "       1.9459102, 2.0794415, 2.1972246], dtype=float32)>"
      ]
     },
     "execution_count": 77,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the log (input also needs to be float)\n",
    "tf.math.log(H)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "urq3bx3l_Y4K"
   },
   "source": [
    "### Manipulating `tf.Variable` tensors\n",
    "\n",
    "Tensors created with `tf.Variable()` can be changed in place using methods such as:\n",
    "\n",
    "* [`.assign()`](https://www.tensorflow.org/api_docs/python/tf/Variable#assign) - assign a different value to a particular index of a variable tensor.\n",
    "* [`.add_assign()`](https://www.tensorflow.org/api_docs/python/tf/Variable#assign_add) - add to an existing value and reassign it at a particular index of a variable tensor.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tV7_uzdR_F4c",
    "outputId": "dbe2bd14-249b-4bf0-c2f1-1af02d98c59a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(5,) dtype=int64, numpy=array([0, 1, 2, 3, 4])>"
      ]
     },
     "execution_count": 78,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a variable tensor\n",
    "I = tf.Variable(np.arange(0, 5))\n",
    "I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ukatOuCG_4CY",
    "outputId": "a6351712-d894-4e23-c57e-2ab45537d566"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(5,) dtype=int64, numpy=array([ 0,  1,  2,  3, 50])>"
      ]
     },
     "execution_count": 79,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assign the final value a new value of 50\n",
    "I.assign([0, 1, 2, 3, 50])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3r2Pya_vAnLz",
    "outputId": "a65b7db9-a0ec-47d7-f3cf-12cb60eb31ac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(5,) dtype=int64, numpy=array([ 0,  1,  2,  3, 50])>"
      ]
     },
     "execution_count": 80,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The change happens in place (the last value is now 50, not 4)\n",
    "I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gZie_FduAo5O",
    "outputId": "0b7d34bf-76e9-429e-ecec-7bb2beedb750"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(5,) dtype=int64, numpy=array([10, 11, 12, 13, 60])>"
      ]
     },
     "execution_count": 81,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add 10 to every element in I\n",
    "I.assign_add([10, 10, 10, 10, 10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2qXODgVOBDzC",
    "outputId": "5fd97dbb-eb43-4e55-b74a-f4e94956b4c2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(5,) dtype=int64, numpy=array([10, 11, 12, 13, 60])>"
      ]
     },
     "execution_count": 82,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Again, the change happens in place\n",
    "I"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2L2IyQhl-Z73"
   },
   "source": [
    "## Tensors and NumPy\n",
    "\n",
    "We've seen some examples of tensors interact with NumPy arrays, such as, using NumPy arrays to create tensors. \n",
    "\n",
    "Tensors can also be converted to NumPy arrays using:\n",
    "\n",
    "* `np.array()` - pass a tensor to convert to an ndarray (NumPy's main datatype).\n",
    "* `tensor.numpy()` - call on a tensor to convert to an ndarray.\n",
    "\n",
    "Doing this is helpful as it makes tensors iterable as well as allows us to use any of NumPy's methods on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HLHrij0vBywD",
    "outputId": "83109e2d-e3be-4e95-e6dd-d84713e9dc70"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float64, numpy=array([ 3.,  7., 10.])>"
      ]
     },
     "execution_count": 83,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a tensor from a NumPy array\n",
    "J = tf.constant(np.array([3., 7., 10.]))\n",
    "J"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "P0KBe_FqCKdU",
    "outputId": "c4de5d97-8095-4e0c-86de-67fad637d88d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 3.,  7., 10.]), numpy.ndarray)"
      ]
     },
     "execution_count": 84,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert tensor J to NumPy with np.array()\n",
    "np.array(J), type(np.array(J))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xxKsJPvSCUPI",
    "outputId": "39cef6d1-2deb-4ec2-cbdf-b9e6f3b06488"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 3.,  7., 10.]), numpy.ndarray)"
      ]
     },
     "execution_count": 85,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert tensor J to NumPy with .numpy()\n",
    "J.numpy(), type(J.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SL4xHtNWDIT5"
   },
   "source": [
    "By default tensors have `dtype=float32`, where as NumPy arrays have `dtype=float64`.\n",
    "\n",
    "This is because neural networks (which are usually built with TensorFlow) can generally work very well with less precision (32-bit rather than 64-bit)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HrQoUeyPCXU9",
    "outputId": "d4790fab-fc39-41d8-bf6b-c628d0f1ada0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tf.float64, tf.float32)"
      ]
     },
     "execution_count": 86,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a tensor from NumPy and from an array\n",
    "numpy_J = tf.constant(np.array([3., 7., 10.])) # will be float64 (due to NumPy)\n",
    "tensor_J = tf.constant([3., 7., 10.]) # will be float32 (due to being TensorFlow default)\n",
    "numpy_J.dtype, tensor_J.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_ytLPBWF1wxt"
   },
   "source": [
    "## Using `@tf.function`\n",
    "\n",
    "In your TensorFlow adventures, you might come across Python functions which have the decorator [`@tf.function`](https://www.tensorflow.org/api_docs/python/tf/function).\n",
    "\n",
    "If you aren't sure what Python decorators do, [read RealPython's guide on them](https://realpython.com/primer-on-python-decorators/).\n",
    "\n",
    "But in short, decorators modify a function in one way or another.\n",
    "\n",
    "In the `@tf.function` decorator case, it turns a Python function into a callable TensorFlow graph. Which is a fancy way of saying, if you've written your own Python function, and you decorate it with `@tf.function`, when you export your code (to potentially run on another device), TensorFlow will attempt to convert it into a fast(er) version of itself (by making it part of a computation graph).\n",
    "\n",
    "For more on this, read the [Better performnace with tf.function](https://www.tensorflow.org/guide/function) guide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "URzFiwPoFDI7",
    "outputId": "a6c7bbe3-4bfb-499a-91e9-f8bd4313b6a0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(10,), dtype=int64, numpy=array([ 10,  12,  16,  22,  30,  40,  52,  66,  82, 100])>"
      ]
     },
     "execution_count": 87,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a simple function\n",
    "def function(x, y):\n",
    "  return x ** 2 + y\n",
    "\n",
    "x = tf.constant(np.arange(0, 10))\n",
    "y = tf.constant(np.arange(10, 20))\n",
    "function(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EHGbvlkXF7Gs",
    "outputId": "40f5d83c-f2fc-4c50-fdfe-f37b3fdfa75b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(10,), dtype=int64, numpy=array([ 10,  12,  16,  22,  30,  40,  52,  66,  82, 100])>"
      ]
     },
     "execution_count": 88,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the same function and decorate it with tf.function\n",
    "@tf.function\n",
    "def tf_function(x, y):\n",
    "  return x ** 2 + y\n",
    "\n",
    "tf_function(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lsiapEnTGBbH"
   },
   "source": [
    "If you noticed no difference between the above two functions (the decorated one and the non-decorated one) you'd be right.\n",
    "\n",
    "Much of the difference happens behind the scenes. One of the main ones being potential code speed-ups where possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y8AQmchkxKDn"
   },
   "source": [
    "## Finding access to GPUs\n",
    "\n",
    "We've mentioned GPUs plenty of times throughout this notebook.\n",
    "\n",
    "So how do you check if you've got one available?\n",
    "\n",
    "You can check if you've got access to a GPU using [`tf.config.list_physical_devices()`](https://www.tensorflow.org/guide/gpu)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3rFNSeODxi3L",
    "outputId": "7dd6e024-bed3-46ad-d96f-bea634317e24"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]\n"
     ]
    }
   ],
   "source": [
    "print(tf.config.list_physical_devices('GPU'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8165PtH8xyq5"
   },
   "source": [
    "If the above outputs an empty array (or nothing), it means you don't have access to a GPU (or at least TensorFlow can't find it).\n",
    "\n",
    "If you're running in Google Colab, you can access a GPU by going to *Runtime -> Change Runtime Type -> Select GPU* (**note:** after doing this your notebook will restart and any variables you've saved will be lost).\n",
    "\n",
    "Once you've changed your runtime type, run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iAvMwRbZyULE",
    "outputId": "eaab2fea-5960-43f1-899f-1eee870a76f9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.config.list_physical_devices('GPU'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J3yu0LKiyZlJ"
   },
   "source": [
    "If you've got access to a GPU, the cell above should output something like:\n",
    "\n",
    "`[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]`\n",
    "\n",
    "You can also find information about your GPU using `!nvidia-smi`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yOuVvG5kyefS",
    "outputId": "fc734522-6270-447d-cefc-34e9fbe4a89b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mon Mar 15 22:33:50 2021       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 460.56       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
      "| N/A   52C    P0    29W /  70W |    224MiB / 15109MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pER7mLFezV9u"
   },
   "source": [
    "> 🔑 **Note:** If you have access to a GPU, TensorFlow will automatically use it whenever possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HcCZxK61VIv0"
   },
   "source": [
    "## 🛠 Exercises\n",
    "\n",
    "1. Create a vector, scalar, matrix and tensor with values of your choosing using `tf.constant()`.\n",
    "2. Find the shape, rank and size of the tensors you created in 1.\n",
    "3. Create two tensors containing random values between 0 and 1 with shape `[5, 300]`.\n",
    "4. Multiply the two tensors you created in 3 using matrix multiplication.\n",
    "5. Multiply the two tensors you created in 3 using dot product.\n",
    "6. Create a tensor with random values between 0 and 1 with shape `[224, 224, 3]`.\n",
    "7. Find the min and max values of the tensor you created in 6.\n",
    "8. Created a tensor with random values of shape `[1, 224, 224, 3]` then squeeze it to change the shape to `[224, 224, 3]`.\n",
    "9. Create a tensor with shape `[10]` using your own choice of values, then find the index which has the maximum value.\n",
    "10. One-hot encode the tensor you created in 9."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EgHK5CjY3oVX"
   },
   "source": [
    "## 📖 Extra-curriculum\n",
    "\n",
    "* Read through the [list of TensorFlow Python APIs](https://www.tensorflow.org/api_docs/python/), pick one we haven't gone through in this notebook, reverse engineer it (write out the documentation code for yourself) and figure out what it does.\n",
    "* Try to create a series of tensor functions to calculate your most recent grocery bill (it's okay if you don't use the names of the items, just the price in numerical form).\n",
    "  * How would you calculate your grocery bill for the month and for the year using tensors?\n",
    "* Go through the [TensorFlow 2.x quick start for beginners](https://www.tensorflow.org/tutorials/quickstart/beginner) tutorial (be sure to type out all of the code yourself, even if you don't understand it).\n",
    "  * Are there any functions we used in here that match what's used in there? Which are the same? Which haven't you seen before?\n",
    "* Watch the video [\"What's a tensor?\"](https://www.youtube.com/watch?v=f5liqUk0ZTw) - a great visual introduction to many of the concepts we've covered in this notebook."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyNbw3GZW3WCZGOWaqvAIsnr",
   "collapsed_sections": [
    "fzjcZ4FHCOb5"
   ],
   "include_colab_link": true,
   "name": "00_tensorflow_fundamentals.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}