Tensor & Torch AutoGrad.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ddfcc38b",
   "metadata": {},
   "source": [
    "## Tensor\n",
    "-----\n",
    "\n",
    "In PyTorch, a tensor is a multi-dimensional array that can be used for various mathematical operations. It is similar to NumPy's ndarray, but with additional capabilities that make it suitable for deep learning tasks and GPU acceleration.\n",
    "\n",
    "A tensor can have different dimensions: 0-dimensional tensor (scalar), 1-dimensional tensor (vector), 2-dimensional tensor (matrix), and higher-dimensional tensors. Each dimension is referred to as an axis.\n",
    "\n",
    "**PyTorch tensors have several important properties:**\n",
    "\n",
    "- **Data type:** Tensors can store data of different types, such as float, double, int, long, and others. The data type is specified during tensor creation and affects the precision and memory usage of the tensor.\n",
    "\n",
    "- **Device:** Tensors can be stored and processed on different devices, such as CPU or GPU. By default, tensors are created on the CPU, but they can be easily moved to the GPU for accelerated computations.\n",
    "\n",
    "- **Autograd support:** PyTorch tensors have built-in support for automatic differentiation, which is crucial for training neural networks. The autograd package keeps track of operations performed on tensors and allows for automatic computation of gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "840d95c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "20fdf39b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([9.2755e-39, 1.0561e-38, 1.0653e-38])\n"
     ]
    }
   ],
   "source": [
    "x = torch.empty(3) #1d Tensor\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "40b161b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.3635e-05,  6.8944e-43, -1.3635e-05],\n",
      "        [ 6.8944e-43, -2.1595e-05,  6.8944e-43]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.empty(2,3) #2d Tensor\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3b6f80d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[-2.1772e-05,  6.8944e-43],\n",
      "         [-2.1772e-05,  6.8944e-43]],\n",
      "\n",
      "        [[-2.1764e-05,  6.8944e-43],\n",
      "         [-2.1764e-05,  6.8944e-43]]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.empty(2,2, 2) #3d Tensor\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "52f9e1fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.8155, 0.2555])\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(2) #random numbers\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a1de1388",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0., 0.],\n",
      "        [0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.zeros(2,2)\n",
    "#all the values are zero when the size of the matrix is 2x2\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "48e50311",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(2,2)\n",
    "#all the values are 1 when the size of the matrix is 2x2\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b109ab64",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.float32\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(2,2)\n",
    "print(x.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9487b86f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.int32\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(2,2, dtype=torch.int) #int/ double / float16\n",
    "print(x.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "411121cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 2])\n"
     ]
    }
   ],
   "source": [
    "print(x.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "af919db7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.5000, 0.1000])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=torch.tensor([2.5, 0.1])\n",
    "x #another way to create tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0f08c78d",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.rand(2,2)\n",
    "y = torch.rand(2,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "26fcab87",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.4818, 0.9015],\n",
      "        [0.0117, 0.3570]])\n"
     ]
    }
   ],
   "source": [
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "59baac36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0231, 0.3410],\n",
      "        [0.6202, 0.0633]])\n"
     ]
    }
   ],
   "source": [
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "92fa8002",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.5049, 1.2425],\n",
       "        [0.6319, 0.4202]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z= x+y\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ed5a50fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.5049, 1.2425],\n",
       "        [0.6319, 0.4202]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = torch.add(x,y)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "24aafb24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.5049, 1.2425],\n",
       "        [0.6319, 0.4202]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.add_(x) #here add_ \n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8deaf113",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.0231, -0.3410],\n",
       "        [-0.6202, -0.0633]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z= x-y\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a63258bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.0231, -0.3410],\n",
       "        [-0.6202, -0.0633]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = torch.sub(x,y)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d9f1cf65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.2432, 1.1201],\n",
       "        [0.0074, 0.1500]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = torch.mul(x,y)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2b861b90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.9543, 0.7256],\n",
       "        [0.0186, 0.8495]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z= x/y\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "54e20846",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0567, 0.2834, 0.6829],\n",
       "        [0.4570, 0.8197, 0.6981],\n",
       "        [0.5545, 0.1712, 0.5145],\n",
       "        [0.9089, 0.6889, 0.5640],\n",
       "        [0.4665, 0.1205, 0.0314]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Slicing\n",
    "x= torch.rand(5,3)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "708e4213",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0567, 0.4570, 0.5545, 0.9089, 0.4665])\n"
     ]
    }
   ],
   "source": [
    "print(x[:,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "54834990",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.4570)\n"
     ]
    }
   ],
   "source": [
    "print(x[1,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "de4db507",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8197047114372253\n"
     ]
    }
   ],
   "source": [
    "print(x[1,1].item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bb55fb34",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0478, 0.8703, 0.4500, 0.4190],\n",
       "        [0.2832, 0.3381, 0.1201, 0.5882],\n",
       "        [0.0512, 0.9625, 0.3585, 0.0110],\n",
       "        [0.5243, 0.5050, 0.7169, 0.1568]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Reshaping Tensor\n",
    "x=torch.rand(4,4)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "c1c1260e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.0478, 0.8703, 0.4500, 0.4190, 0.2832, 0.3381, 0.1201, 0.5882, 0.0512,\n",
       "        0.9625, 0.3585, 0.0110, 0.5243, 0.5050, 0.7169, 0.1568])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x.view(16)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "2e374b9e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0478, 0.8703, 0.4500, 0.4190, 0.2832, 0.3381, 0.1201, 0.5882],\n",
       "        [0.0512, 0.9625, 0.3585, 0.0110, 0.5243, 0.5050, 0.7169, 0.1568]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x.view(-1,8)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f4c8ed0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 8])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "fbbc2c0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Numpy to Torch\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b51339ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.ones(5) # a is a Tensor\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b85996bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1.], dtype=float32)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.numpy() #Here b is a numpy array\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4486e956",
   "metadata": {},
   "source": [
    "***If we run this code in CPU then the tensor would store in the same memory while running on GPU can allocate memory separetely. So we should keep in mind that when we update the Tensor.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "21b6bfdd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2., 2., 2., 2., 2.])\n",
      "[2. 2. 2. 2. 2.]\n"
     ]
    }
   ],
   "source": [
    "a.add_(1)\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9b78bef",
   "metadata": {},
   "source": [
    "***As we see from the output, we just added 1 to the tensor a, but both a and b have been updated as b was copied from a and they were stored in the same memory location, because this Tensor was run on CPU.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "27e5320a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 2., 2., 2., 2.])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "090faf00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2., 2., 2., 2., 2.], dtype=float32)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "860768fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 1., 1., 1., 1.], requires_grad=True)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#It says that the tensor needs to calculate gradient\n",
    "x = torch.ones(5, requires_grad= True) \n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d5515c4",
   "metadata": {},
   "source": [
    "## Torch AutoGrad\n",
    "-----\n",
    "In PyTorch, the torch.autograd module provides automatic differentiation capabilities, which is a fundamental component of deep learning frameworks. Automatic differentiation allows you to compute gradients of tensors with respect to other tensors, enabling efficient and convenient implementation of gradient-based optimization algorithms, such as backpropagation.\n",
    "\n",
    "The torch.autograd module in PyTorch defines a class called Variable (prior to PyTorch version 0.4.0) or Tensor (starting from PyTorch version 0.4.0), which represents a tensor and tracks its operations to compute gradients automatically. When you perform operations on these variables/tensors, PyTorch builds a computational graph, which is a record of the operations applied to the variables.\n",
    "\n",
    "The computational graph is used to perform backpropagation and compute gradients. Gradients are essentially derivatives, and they represent the rate of change of a tensor with respect to other tensors in the graph. The gradients are computed using the chain rule of calculus, which allows efficient calculation of gradients for complex computational graphs.\n",
    "\n",
    "By utilizing torch.autograd, you can calculate gradients automatically without explicitly deriving and implementing them yourself. This feature greatly simplifies the process of training neural networks, as you can focus on designing the network architecture and defining the forward pass, while PyTorch takes care of computing the gradients and updating the network's parameters during the backward pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "cc842379",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "import warnings as wr\n",
    "wr.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "29852abd",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(1.,requires_grad = True)\n",
    "w = torch.tensor(2.,requires_grad = True)\n",
    "b = torch.tensor(3.,requires_grad = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0d58d08",
   "metadata": {},
   "source": [
    "**Build a computational Graph:**\n",
    "y = w * x + b,  y = 2 * x +3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "18a88bae",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Computer Gradient\n",
    "y = w * x + b\n",
    "y.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "4bb0934a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2.)\n",
      "tensor(1.)\n",
      "tensor(1.)\n"
     ]
    }
   ],
   "source": [
    "print(x.grad) #x.grad = 2\n",
    "print(w.grad) #x.grad = 1\n",
    "print(b.grad) #x.grad = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "8aa7676d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a 2 layer computational graph\n",
    "x = torch.tensor(1., requires_grad = True)\n",
    "w1 = torch.tensor(2., requires_grad = True)\n",
    "b1 = torch.tensor(3., requires_grad = True)\n",
    "\n",
    "w2 = torch.tensor(4., requires_grad = True)\n",
    "b2 = torch.tensor(5., requires_grad = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "efa52d4a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(8.) tensor(4.) tensor(5.) None None\n"
     ]
    }
   ],
   "source": [
    "layer_1 = w1 * x + b\n",
    "layer_2 = w2 * layer_1 + b\n",
    "\n",
    "layer_2.backward()\n",
    "print(x.grad, w1.grad, w2.grad, b1.grad, b2.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "847eb7a5",
   "metadata": {},
   "source": [
    "#### Train a Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "7f22703f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x.shape: torch.Size([10, 5])\n",
      "y: tensor([[ 0.4620,  1.3333],\n",
      "        [ 0.2411,  0.4739],\n",
      "        [ 0.3680,  1.1487],\n",
      "        [ 0.1500,  0.4983],\n",
      "        [-0.2761,  0.5017],\n",
      "        [-0.2714, -0.3496],\n",
      "        [-0.3796, -0.0810],\n",
      "        [-1.3155, -0.1073],\n",
      "        [-1.3341,  0.6571],\n",
      "        [-1.3604,  1.4133]]), y.shape: torch.Size([10, 2])\n"
     ]
    }
   ],
   "source": [
    "#Create tensors pf shape (10,3) and (10,2)\n",
    "feat_dim = 5 \n",
    "x = torch.randn(10, feat_dim)\n",
    "y = torch.randn(10,2)\n",
    "\n",
    "print('x.shape: {}'.format(x.shape))\n",
    "print('y: {}, y.shape: {}'.format(y, y.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "e0fef84c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w: Parameter containing:\n",
      "tensor([[ 0.3820,  0.1718, -0.1179,  0.1841, -0.2726],\n",
      "        [-0.1399,  0.1207, -0.3922, -0.0857, -0.3800]], requires_grad=True) w.shape: torch.Size([2, 5])\n",
      "b: Parameter containing:\n",
      "tensor([ 0.2675, -0.1812], requires_grad=True) b.shape: torch.Size([2])\n"
     ]
    }
   ],
   "source": [
    "# Build a fully connected layer\n",
    "\n",
    "linear = nn.Linear(feat_dim, 2)\n",
    "print('w: {} w.shape: {}'.format(linear.weight, linear.weight.shape))\n",
    "print('b: {} b.shape: {}'.format(linear.bias, linear.bias.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "1aad0277",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build loss function and optimizer\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = torch.optim.SGD(linear.parameters(), lr = 0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "eb793fb0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pred.shape: torch.Size([10, 2])\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass\n",
    "pred = linear(x)\n",
    "print('pred.shape: {}'.format(pred.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "56e01bf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 1.40336012840271\n"
     ]
    }
   ],
   "source": [
    "# Compute Loss:\n",
    "loss= criterion (pred, y)\n",
    "print('loss:', loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ff623e59",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Backward pass\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "572734aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dL/dw:  tensor([[ 0.3819,  0.3409, -0.1182,  0.3733, -0.3276],\n",
      "        [-0.4214,  0.0787, -0.6373, -0.4869, -0.6445]])\n",
      "dL/dw:  tensor([ 0.9001, -0.7659])\n"
     ]
    }
   ],
   "source": [
    "# Print out the gradients\n",
    "print('dL/dw: ', linear.weight.grad)\n",
    "print('dL/dw: ', linear.bias.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "8cb8f790",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1-step gradient descent\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "ecde66da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Performing gradient descent at the low level\n",
    "# w = w - alpha * (dl/dw) #Update the weights\n",
    "# b = b- alpha * (dl/db) #Update the bias\n",
    "linear.weight.data = linear.weight.data - (0.01*linear.weight.grad.data)\n",
    "linear.bias.data = linear.bias.data - (0.01*linear.bias.grad.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "521537de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss after 1-step optimization:  1.40336012840271\n"
     ]
    }
   ],
   "source": [
    "# Printing loss after 1 - step gradient descent\n",
    "Pred = linear(x)\n",
    "loss = criterion(pred, y )\n",
    "print('Loss after 1-step optimization: ', loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c5dfb42",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}