pytorch_lightning深入浅出.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c29f0eec",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# pytorch_lightning深入浅出"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "508621a3",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "pytorch-lightning 是建立在pytorch之上的高层次模型接口。\n",
    "\n",
    "pytorch-lightning 之于 pytorch，就如同keras之于 tensorflow.\n",
    "\n",
    "pytorch-lightning 有以下一些引人注目的功能：\n",
    "\n",
    "* 可以不必编写自定义循环，只要指定loss计算方法即可。\n",
    "\n",
    "* 可以通过callbacks非常方便地添加CheckPoint参数保存、early_stopping 等功能。\n",
    "\n",
    "* 可以非常方便地在单CPU、多CPU、单GPU、多GPU乃至多TPU上训练模型。\n",
    "\n",
    "* 可以通过调用torchmetrics库，非常方便地添加Accuracy,AUC,Precision等各种常用评估指标。\n",
    "\n",
    "* 可以非常方便地实施多批次梯度累加、半精度混合精度训练、最大batch_size自动搜索等技巧，加快训练过程。\n",
    "\n",
    "* 可以非常方便地使用SWA(随机参数平均)、CyclicLR(学习率周期性调度策略)与auto_lr_find(最优学习率发现)等技巧 实现模型涨点。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b7b69e2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "一般按照如下方式 安装和 引入 pytorch-lightning 库。\n",
    "\n",
    "```bash\n",
    "#安装\n",
    "pip install pytorch-lightning\n",
    "```\n",
    "\n",
    "```python \n",
    "#引入\n",
    "import pytorch_lightning as pl \n",
    "```\n",
    "\n",
    "顾名思义，它可以帮助我们漂亮(pl)地进行深度学习研究。😋😋 \n",
    "You do the research. Lightning will do everything else.⭐️⭐️ \n",
    "\n",
    "\n",
    "参考文档：\n",
    "\n",
    "* pl_docs: https://pytorch-lightning.readthedocs.io/en/latest/starter/introduction.html\n",
    "* pl_template:https://github.com/PyTorchLightning/deep-learning-project-template\n",
    "* torchmetrics: https://torchmetrics.readthedocs.io/en/latest/pages/lightning.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6474aac3",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7f3edf91",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 一，pytorch-lightning的设计哲学"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85b6ceda",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "pytorch-lightning 的核心设计哲学是将 深度学习项目中的 研究代码(定义模型) 和 工程代码 (训练模型) 相互分离。\n",
    "\n",
    "用户只需专注于研究代码(pl.LightningModule)的实现，而工程代码借助训练工具类(pl.Trainer)统一实现。\n",
    "\n",
    "更详细地说，深度学习项目代码可以分成如下4部分：\n",
    "\n",
    "* 研究代码 (Research code)，用户继承LightningModule实现。\n",
    "* 工程代码 (Engineering code)，用户无需关注通过调用Trainer实现。\n",
    "* 非必要代码 （Non-essential research code，logging, etc...），用户通过调用Callbacks实现。\n",
    "* 数据 (Data)，用户通过torch.utils.data.DataLoader实现，也可以封装成pl.LightningDataModule。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20e2fb73",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "593da4a2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 二，pytorch-lightning使用范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10313c6",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "下面我们使用minist图片分类问题为例，演示pytorch-lightning的最佳实践。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d97bae4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 1，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4cf42760",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "from torchvision import transforms as T\n",
    "from torchvision.datasets import MNIST\n",
    "from torch.utils.data import DataLoader,random_split\n",
    "import pytorch_lightning as pl \n",
    "from torchmetrics import Accuracy \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "88231de3",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class MNISTDataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_dir: str = \"./minist/\", \n",
    "                 batch_size: int = 32,\n",
    "                 num_workers: int =4):\n",
    "        super().__init__()\n",
    "        self.data_dir = data_dir\n",
    "        self.batch_size = batch_size\n",
    "        self.num_workers = num_workers\n",
    "\n",
    "    def setup(self, stage = None):\n",
    "        transform = T.Compose([T.ToTensor()])\n",
    "        self.ds_test = MNIST(self.data_dir, train=False,transform=transform,download=True)\n",
    "        self.ds_predict = MNIST(self.data_dir, train=False,transform=transform,download=True)\n",
    "        ds_full = MNIST(self.data_dir, train=True,transform=transform,download=True)\n",
    "        self.ds_train, self.ds_val = random_split(ds_full, [55000, 5000])\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        return DataLoader(self.ds_train, batch_size=self.batch_size,\n",
    "                          shuffle=True, num_workers=self.num_workers,\n",
    "                          pin_memory=True)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return DataLoader(self.ds_val, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=True)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return DataLoader(self.ds_test, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=True)\n",
    "\n",
    "    def predict_dataloader(self):\n",
    "        return DataLoader(self.ds_predict, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=True)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b3833b60",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./minist/MNIST/raw/train-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ffc95af6e47f42f78a42d3f5559cf0a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/9912422 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./minist/MNIST/raw/train-images-idx3-ubyte.gz to ./minist/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./minist/MNIST/raw/train-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed402280a0c141c78e8e08f8ce0a0122",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/28881 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./minist/MNIST/raw/train-labels-idx1-ubyte.gz to ./minist/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./minist/MNIST/raw/t10k-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3df7a1d8d0024e34ad65c0cf1549c17e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1648877 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./minist/MNIST/raw/t10k-images-idx3-ubyte.gz to ./minist/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./minist/MNIST/raw/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7b3bdfc75d2e4fd192bee2ae31355cd8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4542 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./minist/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./minist/MNIST/raw\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data_mnist = MNISTDataModule()\n",
    "data_mnist.setup()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "caa5a7df",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 1, 28, 28])\n",
      "torch.Size([32])\n"
     ]
    }
   ],
   "source": [
    "for features,labels in data_mnist.train_dataloader():\n",
    "    print(features.shape)\n",
    "    print(labels.shape)\n",
    "    break "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "910732e7",
   "metadata": {},
   "source": [
    "```\n",
    "torch.Size([32, 1, 28, 28])\n",
    "torch.Size([32])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42c7dc3f",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "dedc209e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 2，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3f76a21d",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_size = 0.218447 M \n",
      "\n",
      "   | Name      | Type              | Params | In sizes         | Out sizes       \n",
      "---------------------------------------------------------------------------------------\n",
      "0  | net       | Sequential        | 54.0 K | [32, 1, 28, 28]  | [32, 10]        \n",
      "1  | net.0     | Conv2d            | 320    | [32, 1, 28, 28]  | [32, 32, 26, 26]\n",
      "2  | net.1     | MaxPool2d         | 0      | [32, 32, 26, 26] | [32, 32, 13, 13]\n",
      "3  | net.2     | Conv2d            | 51.3 K | [32, 32, 13, 13] | [32, 64, 9, 9]  \n",
      "4  | net.3     | MaxPool2d         | 0      | [32, 64, 9, 9]   | [32, 64, 4, 4]  \n",
      "5  | net.4     | Dropout2d         | 0      | [32, 64, 4, 4]   | [32, 64, 4, 4]  \n",
      "6  | net.5     | AdaptiveMaxPool2d | 0      | [32, 64, 4, 4]   | [32, 64, 1, 1]  \n",
      "7  | net.6     | Flatten           | 0      | [32, 64, 1, 1]   | [32, 64]        \n",
      "8  | net.7     | Linear            | 2.1 K  | [32, 64]         | [32, 32]        \n",
      "9  | net.8     | ReLU              | 0      | [32, 32]         | [32, 32]        \n",
      "10 | net.9     | Linear            | 330    | [32, 32]         | [32, 10]        \n",
      "11 | train_acc | Accuracy          | 0      | ?                | ?               \n",
      "12 | val_acc   | Accuracy          | 0      | ?                | ?               \n",
      "13 | test_acc  | Accuracy          | 0      | ?                | ?               \n",
      "---------------------------------------------------------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/liangyun/ProgramFiles/anaconda3/lib/python3.8/site-packages/pytorch_lightning/utilities/parsing.py:261: UserWarning: Attribute 'net' is an instance of `nn.Module` and is already saved during checkpointing. It is recommended to ignore them using `self.save_hyperparameters(ignore=['net'])`.\n",
      "  rank_zero_warn(\n"
     ]
    }
   ],
   "source": [
    "net = nn.Sequential(\n",
    "    nn.Conv2d(in_channels=1,out_channels=32,kernel_size = 3),\n",
    "    nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "    nn.Conv2d(in_channels=32,out_channels=64,kernel_size = 5),\n",
    "    nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "    nn.Dropout2d(p = 0.1),\n",
    "    nn.AdaptiveMaxPool2d((1,1)),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(64,32),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(32,10)\n",
    ")\n",
    "\n",
    "class Model(pl.LightningModule):\n",
    "    \n",
    "    def __init__(self,net,learning_rate=1e-3):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        self.net = net\n",
    "        self.train_acc = Accuracy()\n",
    "        self.val_acc = Accuracy()\n",
    "        self.test_acc = Accuracy() \n",
    "        \n",
    "        \n",
    "    def forward(self,x):\n",
    "        x = self.net(x)\n",
    "        return x\n",
    "    \n",
    "    \n",
    "    #定义loss\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "    \n",
    "    #定义各种metrics\n",
    "    def training_step_end(self,outputs):\n",
    "        train_acc = self.train_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"train_acc\",train_acc,prog_bar=True)\n",
    "        return {\"loss\":outputs[\"loss\"].mean()}\n",
    "    \n",
    "    #定义optimizer,以及可选的lr_scheduler\n",
    "    def configure_optimizers(self):\n",
    "        return torch.optim.Adam(self.parameters(), lr=self.hparams.learning_rate)\n",
    "    \n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "\n",
    "    def validation_step_end(self,outputs):\n",
    "        val_acc = self.val_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"val_loss\",outputs[\"loss\"].mean(),on_epoch=True,on_step=False)\n",
    "        self.log(\"val_acc\",val_acc,prog_bar=True,on_epoch=True,on_step=False)\n",
    "    \n",
    "    def test_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "    \n",
    "    def test_step_end(self,outputs):\n",
    "        test_acc = self.test_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"test_acc\",test_acc,on_epoch=True,on_step=False)\n",
    "        self.log(\"test_loss\",outputs[\"loss\"].mean(),on_epoch=True,on_step=False)\n",
    "    \n",
    "model = Model(net)\n",
    "\n",
    "#查看模型大小\n",
    "model_size = pl.utilities.memory.get_model_size_mb(model)\n",
    "print(\"model_size = {} M \\n\".format(model_size))\n",
    "model.example_input_array = [features]\n",
    "summary = pl.utilities.model_summary.ModelSummary(model,max_depth=-1)\n",
    "print(summary) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93757e61",
   "metadata": {},
   "source": [
    "```\n",
    "model_size = 0.218447 M \n",
    "\n",
    "   | Name      | Type              | Params | In sizes         | Out sizes       \n",
    "---------------------------------------------------------------------------------------\n",
    "0  | net       | Sequential        | 54.0 K | [32, 1, 28, 28]  | [32, 10]        \n",
    "1  | net.0     | Conv2d            | 320    | [32, 1, 28, 28]  | [32, 32, 26, 26]\n",
    "2  | net.1     | MaxPool2d         | 0      | [32, 32, 26, 26] | [32, 32, 13, 13]\n",
    "3  | net.2     | Conv2d            | 51.3 K | [32, 32, 13, 13] | [32, 64, 9, 9]  \n",
    "4  | net.3     | MaxPool2d         | 0      | [32, 64, 9, 9]   | [32, 64, 4, 4]  \n",
    "5  | net.4     | Dropout2d         | 0      | [32, 64, 4, 4]   | [32, 64, 4, 4]  \n",
    "6  | net.5     | AdaptiveMaxPool2d | 0      | [32, 64, 4, 4]   | [32, 64, 1, 1]  \n",
    "7  | net.6     | Flatten           | 0      | [32, 64, 1, 1]   | [32, 64]        \n",
    "8  | net.7     | Linear            | 2.1 K  | [32, 64]         | [32, 32]        \n",
    "9  | net.8     | ReLU              | 0      | [32, 32]         | [32, 32]        \n",
    "10 | net.9     | Linear            | 330    | [32, 32]         | [32, 10]        \n",
    "11 | train_acc | Accuracy          | 0      | ?                | ?               \n",
    "12 | val_acc   | Accuracy          | 0      | ?                | ?               \n",
    "13 | test_acc  | Accuracy          | 0      | ?                | ?               \n",
    "---------------------------------------------------------------------------------------\n",
    "54.0 K    Trainable params\n",
    "0         Non-trainable params\n",
    "54.0 K    Total params\n",
    "0.216     Total estimated model params size (MB)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66a49029",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1f714ad6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 3，训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0eb33a5f-5f26-417b-8cd5-1d5259677bd1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "Missing logger folder: /Users/liangyun/CodeFiles/machine_learning_AK47/2_Base_Models/lightning_logs\n",
      "\n",
      "  | Name      | Type       | Params | In sizes        | Out sizes\n",
      "-----------------------------------------------------------------------\n",
      "0 | net       | Sequential | 54.0 K | [32, 1, 28, 28] | [32, 10] \n",
      "1 | train_acc | Accuracy   | 0      | ?               | ?        \n",
      "2 | val_acc   | Accuracy   | 0      | ?               | ?        \n",
      "3 | test_acc  | Accuracy   | 0      | ?               | ?        \n",
      "-----------------------------------------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Sanity Checking: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6f699b1761d84d8987e1441bf3b7769f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "FIT Profiler Report\n",
      "\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|  Action                                                                                                                                                                                                          \t|  Mean duration (s)\t|  Num calls      \t|  Total time (s) \t|  Percentage %   \t|\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|  Total                                                                                                                                                                                                           \t|  -              \t|  750524         \t|  1035.3         \t|  100 %          \t|\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|  run_training_epoch                                                                                                                                                                                              \t|  114.28         \t|  9              \t|  1028.5         \t|  99.346         \t|\n",
      "|  run_training_batch                                                                                                                                                                                              \t|  0.046199       \t|  15471          \t|  714.75         \t|  69.04          \t|\n",
      "|  [LightningModule]Model.optimizer_step                                                                                                                                                                           \t|  0.045555       \t|  15471          \t|  704.78         \t|  68.077         \t|\n",
      "|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                         \t|  0.02216        \t|  15471          \t|  342.84         \t|  33.116         \t|\n",
      "|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                                    \t|  0.018079       \t|  15471          \t|  279.7          \t|  27.017         \t|\n",
      "|  on_train_batch_end                                                                                                                                                                                              \t|  0.0045703      \t|  15471          \t|  70.707         \t|  6.8299         \t|\n",
      "|  [LightningModule]Model.training_step_end                                                                                                                                                                        \t|  0.0020879      \t|  15471          \t|  32.301         \t|  3.1201         \t|\n",
      "|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                                  \t|  0.022605       \t|  1415           \t|  31.986         \t|  3.0897         \t|\n",
      "|  on_train_batch_start                                                                                                                                                                                            \t|  0.00065768     \t|  15471          \t|  10.175         \t|  0.98284        \t|\n",
      "|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                               \t|  0.0047436      \t|  1415           \t|  6.7123         \t|  0.64836        \t|\n",
      "|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                                  \t|  0.0003106      \t|  16886          \t|  5.2448         \t|  0.50662        \t|\n",
      "|  [LightningModule]Model.optimizer_zero_grad                                                                                                                                                                      \t|  0.00024071     \t|  15471          \t|  3.724          \t|  0.35972        \t|\n",
      "|  [LightningModule]Model.validation_step_end                                                                                                                                                                      \t|  0.002431       \t|  1415           \t|  3.4399         \t|  0.33228        \t|\n",
      "|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                                   \t|  0.043175       \t|  10             \t|  0.43175        \t|  0.041705       \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_batch_start                                                                                                                                    \t|  1.7939e-05     \t|  15471          \t|  0.27754        \t|  0.026809       \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_after_backward                                                                                                                                 \t|  1.4938e-05     \t|  15471          \t|  0.23111        \t|  0.022324       \t|\n",
      "|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                        \t|  1.3093e-05     \t|  15471          \t|  0.20256        \t|  0.019566       \t|\n",
      "|  [Callback]ModelSummary.on_batch_start                                                                                                                                                                           \t|  1.2013e-05     \t|  15471          \t|  0.18585        \t|  0.017952       \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_batch_end                                                                                                                                      \t|  1.0508e-05     \t|  15471          \t|  0.16256        \t|  0.015703       \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     \t|  1.0212e-05     \t|  15471          \t|  0.158          \t|  0.015262       \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_before_zero_grad                                                                                                                               \t|  9.7898e-06     \t|  15471          \t|  0.15146        \t|  0.01463        \t|\n",
      "|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                     \t|  9.4311e-06     \t|  15471          \t|  0.14591        \t|  0.014094       \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                          \t|  9.3942e-06     \t|  15471          \t|  0.14534        \t|  0.014039       \t|\n",
      "|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                             \t|  8.792e-05      \t|  1415           \t|  0.12441        \t|  0.012017       \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_before_backward                                                                                                                                \t|  7.7306e-06     \t|  15471          \t|  0.1196         \t|  0.011553       \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       \t|  7.4362e-06     \t|  15471          \t|  0.11504        \t|  0.011113       \t|\n",
      "|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                          \t|  7.3346e-06     \t|  15471          \t|  0.11347        \t|  0.010961       \t|\n",
      "|  [LightningModule]Model.on_train_batch_start                                                                                                                                                                     \t|  7.2084e-06     \t|  15471          \t|  0.11152        \t|  0.010772       \t|\n",
      "|  [Callback]ModelSummary.on_batch_end                                                                                                                                                                             \t|  6.8693e-06     \t|  15471          \t|  0.10627        \t|  0.010265       \t|\n",
      "|  [Callback]ModelSummary.on_after_backward                                                                                                                                                                        \t|  6.8604e-06     \t|  15471          \t|  0.10614        \t|  0.010252       \t|\n",
      "|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                                   \t|  6.6147e-06     \t|  15471          \t|  0.10234        \t|  0.0098851      \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_before_optimizer_step                                                                                                                          \t|  6.5559e-06     \t|  15471          \t|  0.10143        \t|  0.0097971      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                       \t|  6.4805e-06     \t|  15471          \t|  0.10026        \t|  0.0096846      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                            \t|  6.2774e-06     \t|  15471          \t|  0.097118       \t|  0.009381       \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  \t|  6.2459e-06     \t|  15471          \t|  0.096631       \t|  0.0093339      \t|\n",
      "|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                        \t|  0.094863       \t|  1              \t|  0.094863       \t|  0.0091632      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                \t|  6.0117e-06     \t|  15471          \t|  0.093007       \t|  0.0089839      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 \t|  5.9367e-06     \t|  15471          \t|  0.091846       \t|  0.0088718      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 \t|  0.010071       \t|  9              \t|  0.090637       \t|  0.008755       \t|\n",
      "|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                                    \t|  5.7881e-06     \t|  15471          \t|  0.089548       \t|  0.0086498      \t|\n",
      "|  [Callback]ModelSummary.on_before_backward                                                                                                                                                                       \t|  5.7598e-06     \t|  15471          \t|  0.089111       \t|  0.0086076      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                     \t|  5.7241e-06     \t|  15471          \t|  0.088557       \t|  0.0085541      \t|\n",
      "|  [Callback]ModelSummary.on_before_zero_grad                                                                                                                                                                      \t|  5.6813e-06     \t|  15471          \t|  0.087896       \t|  0.0084902      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                      \t|  5.3884e-06     \t|  15471          \t|  0.083364       \t|  0.0080525      \t|\n",
      "|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                              \t|  5.2815e-06     \t|  15471          \t|  0.08171        \t|  0.0078927      \t|\n",
      "|  [Callback]TQDMProgressBar.on_sanity_check_start                                                                                                                                                                 \t|  0.080444       \t|  1              \t|  0.080444       \t|  0.0077704      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                                \t|  5.1356e-06     \t|  15471          \t|  0.079453       \t|  0.0076747      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           \t|  5.1284e-06     \t|  15471          \t|  0.079342       \t|  0.007664       \t|\n",
      "|  [Callback]ModelSummary.on_before_optimizer_step                                                                                                                                                                 \t|  5.0537e-06     \t|  15471          \t|  0.078186       \t|  0.0075523      \t|\n",
      "|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                             \t|  4.6168e-06     \t|  15471          \t|  0.071427       \t|  0.0068994      \t|\n",
      "|  [LightningModule]Model.on_train_batch_end                                                                                                                                                                       \t|  4.4916e-06     \t|  15471          \t|  0.069489       \t|  0.0067123      \t|\n",
      "|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                                \t|  3.7312e-06     \t|  15471          \t|  0.057726       \t|  0.005576       \t|\n",
      "|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                                  \t|  0.0060883      \t|  9              \t|  0.054795       \t|  0.0052928      \t|\n",
      "|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                     \t|  0.0044036      \t|  10             \t|  0.044036       \t|  0.0042536      \t|\n",
      "|  [Callback]ModelSummary.on_validation_batch_end                                                                                                                                                                  \t|  3.0484e-05     \t|  1415           \t|  0.043135       \t|  0.0041666      \t|\n",
      "|  [LightningModule]Model.on_after_backward                                                                                                                                                                        \t|  2.7791e-06     \t|  15471          \t|  0.042995       \t|  0.0041531      \t|\n",
      "|  [LightningModule]Model.on_before_backward                                                                                                                                                                       \t|  2.3516e-06     \t|  15471          \t|  0.036382       \t|  0.0035143      \t|\n",
      "|  [LightningModule]Model.on_before_zero_grad                                                                                                                                                                      \t|  2.2216e-06     \t|  15471          \t|  0.034371       \t|  0.00332        \t|\n",
      "|  [LightningModule]Model.on_before_optimizer_step                                                                                                                                                                 \t|  2.1045e-06     \t|  15471          \t|  0.032558       \t|  0.003145       \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                               \t|  2.2432e-05     \t|  1415           \t|  0.031742       \t|  0.0030661      \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_validation_batch_start                                                                                                                         \t|  2.066e-05      \t|  1415           \t|  0.029234       \t|  0.0028238      \t|\n",
      "|  [Callback]ModelSummary.on_fit_start                                                                                                                                                                             \t|  0.026042       \t|  1              \t|  0.026042       \t|  0.0025155      \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_train_epoch_end                                                                                                                                \t|  0.0025392      \t|  9              \t|  0.022853       \t|  0.0022074      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            \t|  1.4988e-05     \t|  1415           \t|  0.021207       \t|  0.0020485      \t|\n",
      "|  [Callback]ModelSummary.on_validation_batch_start                                                                                                                                                                \t|  1.2564e-05     \t|  1415           \t|  0.017777       \t|  0.0017172      \t|\n",
      "|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                                    \t|  0.001752       \t|  9              \t|  0.015768       \t|  0.0015231      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                                 \t|  1.0898e-05     \t|  1415           \t|  0.01542        \t|  0.0014895      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          \t|  1.0686e-05     \t|  1415           \t|  0.01512        \t|  0.0014605      \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_validation_batch_end                                                                                                                           \t|  8.9751e-06     \t|  1415           \t|  0.0127         \t|  0.0012267      \t|\n",
      "|  [LightningModule]Model.on_validation_batch_start                                                                                                                                                                \t|  4.1048e-06     \t|  1415           \t|  0.0058083      \t|  0.00056105     \t|\n",
      "|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                              \t|  3.6303e-06     \t|  1415           \t|  0.0051369      \t|  0.00049619     \t|\n",
      "|  [LightningModule]Model.on_validation_batch_end                                                                                                                                                                  \t|  3.3238e-06     \t|  1415           \t|  0.0047031      \t|  0.0004543      \t|\n",
      "|  [LightningModule]Model.on_validation_model_train                                                                                                                                                                \t|  0.00038458     \t|  10             \t|  0.0038458      \t|  0.00037148     \t|\n",
      "|  [LightningModule]Model.on_validation_model_eval                                                                                                                                                                 \t|  0.000218       \t|  10             \t|  0.00218        \t|  0.00021057     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  \t|  0.00014091     \t|  10             \t|  0.0014091      \t|  0.00013611     \t|\n",
      "|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                          \t|  0.0011128      \t|  1              \t|  0.0011128      \t|  0.00010749     \t|\n",
      "|  [Callback]ModelSummary.on_validation_start                                                                                                                                                                      \t|  6.479e-05      \t|  10             \t|  0.0006479      \t|  6.2583e-05     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_sanity_check_start                                                                                                                                                   \t|  0.00044772     \t|  1              \t|  0.00044772     \t|  4.3247e-05     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_epoch_end                                                                                                                                      \t|  1.7578e-05     \t|  19             \t|  0.00033398     \t|  3.2261e-05     \t|\n",
      "|  [Callback]ModelSummary.on_train_epoch_end                                                                                                                                                                       \t|  3.562e-05      \t|  9              \t|  0.00032058     \t|  3.0966e-05     \t|\n",
      "|  [Callback]ModelSummary.on_validation_end                                                                                                                                                                        \t|  3.1521e-05     \t|  10             \t|  0.00031521     \t|  3.0447e-05     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.setup                              \t|  0.00025678     \t|  1              \t|  0.00025678     \t|  2.4804e-05     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                            \t|  1.1977e-05     \t|  19             \t|  0.00022756     \t|  2.1981e-05     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_validation_start                                                                                                                               \t|  2.1693e-05     \t|  10             \t|  0.00021693     \t|  2.0954e-05     \t|\n",
      "|  [Callback]ModelSummary.on_train_epoch_start                                                                                                                                                                     \t|  2.2876e-05     \t|  9              \t|  0.00020589     \t|  1.9887e-05     \t|\n",
      "|  [LightningModule]Model.configure_optimizers                                                                                                                                                                     \t|  0.0001787      \t|  1              \t|  0.0001787      \t|  1.7261e-05     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_epoch_start                                                                                                                                    \t|  9.1784e-06     \t|  19             \t|  0.00017439     \t|  1.6845e-05     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_validation_end                                                                                                                                 \t|  1.7215e-05     \t|  10             \t|  0.00017215     \t|  1.6628e-05     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                                    \t|  1.8298e-05     \t|  9              \t|  0.00016468     \t|  1.5907e-05     \t|\n",
      "|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                          \t|  7.6978e-06     \t|  19             \t|  0.00014626     \t|  1.4128e-05     \t|\n",
      "|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                               \t|  1.4491e-05     \t|  10             \t|  0.00014491     \t|  1.3998e-05     \t|\n",
      "|  [LightningModule]Model.on_epoch_start                                                                                                                                                                           \t|  7.5777e-06     \t|  19             \t|  0.00014398     \t|  1.3907e-05     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                          \t|  6.8797e-06     \t|  19             \t|  0.00013071     \t|  1.2626e-05     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                     \t|  1.3034e-05     \t|  10             \t|  0.00013034     \t|  1.259e-05      \t|\n",
      "|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                        \t|  6.7241e-06     \t|  19             \t|  0.00012776     \t|  1.2341e-05     \t|\n",
      "|  [Callback]ModelSummary.on_epoch_end                                                                                                                                                                             \t|  6.6082e-06     \t|  19             \t|  0.00012556     \t|  1.2128e-05     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       \t|  6.5673e-06     \t|  19             \t|  0.00012478     \t|  1.2053e-05     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     \t|  6.2478e-06     \t|  19             \t|  0.00011871     \t|  1.1467e-05     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_validation_epoch_end                                                                                                                           \t|  1.1434e-05     \t|  10             \t|  0.00011434     \t|  1.1045e-05     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                \t|  1.121e-05      \t|  10             \t|  0.0001121      \t|  1.0828e-05     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                       \t|  1.1132e-05     \t|  10             \t|  0.00011132     \t|  1.0753e-05     \t|\n",
      "|  [Callback]ModelSummary.on_epoch_start                                                                                                                                                                           \t|  5.4825e-06     \t|  19             \t|  0.00010417     \t|  1.0062e-05     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                      \t|  1.0704e-05     \t|  9              \t|  9.6338e-05     \t|  9.3057e-06     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            \t|  9.4385e-06     \t|  10             \t|  9.4385e-05     \t|  9.117e-06      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               \t|  9.0062e-06     \t|  9              \t|  8.1056e-05     \t|  7.8295e-06     \t|\n",
      "|  [Callback]ModelSummary.on_validation_epoch_end                                                                                                                                                                  \t|  7.654e-06      \t|  10             \t|  7.654e-05      \t|  7.3933e-06     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                                 \t|  6.7581e-06     \t|  10             \t|  6.7581e-05     \t|  6.5279e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_validation_epoch_start                                                                                                                         \t|  6.6193e-06     \t|  10             \t|  6.6193e-05     \t|  6.3939e-06     \t|\n",
      "|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                             \t|  6.5567e-06     \t|  10             \t|  6.5567e-05     \t|  6.3334e-06     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_sanity_check_start              \t|  6.5134e-05     \t|  1              \t|  6.5134e-05     \t|  6.2916e-06     \t|\n",
      "|  [LightningModule]Model.on_validation_epoch_end                                                                                                                                                                  \t|  6.3101e-06     \t|  10             \t|  6.3101e-05     \t|  6.0952e-06     \t|\n",
      "|  [Callback]ModelSummary.on_save_checkpoint                                                                                                                                                                       \t|  1.0263e-05     \t|  6              \t|  6.158e-05      \t|  5.9483e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_save_checkpoint                                                                                                                                \t|  1.0004e-05     \t|  6              \t|  6.0027e-05     \t|  5.7983e-06     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          \t|  5.5679e-06     \t|  10             \t|  5.5679e-05     \t|  5.3783e-06     \t|\n",
      "|  [LightningModule]Model.on_train_epoch_start                                                                                                                                                                     \t|  6.1488e-06     \t|  9              \t|  5.5339e-05     \t|  5.3454e-06     \t|\n",
      "|  [Callback]ModelSummary.on_sanity_check_start                                                                                                                                                                    \t|  5.53e-05       \t|  1              \t|  5.53e-05       \t|  5.3417e-06     \t|\n",
      "|  [Callback]ModelSummary.on_validation_epoch_start                                                                                                                                                                \t|  5.282e-06      \t|  10             \t|  5.282e-05      \t|  5.1021e-06     \t|\n",
      "|  [LightningModule]Model.on_train_epoch_end                                                                                                                                                                       \t|  5.807e-06      \t|  9              \t|  5.2263e-05     \t|  5.0483e-06     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_save_checkpoint                                                                                                                                                      \t|  8.704e-06      \t|  6              \t|  5.2224e-05     \t|  5.0445e-06     \t|\n",
      "|  [LightningModule]Model.validation_epoch_end                                                                                                                                                                     \t|  5.0435e-06     \t|  10             \t|  5.0435e-05     \t|  4.8717e-06     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                               \t|  4.9998e-06     \t|  10             \t|  4.9998e-05     \t|  4.8295e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_train_epoch_start                                                                                                                              \t|  5.2429e-06     \t|  9              \t|  4.7186e-05     \t|  4.5579e-06     \t|\n",
      "|  [LightningModule]Model.on_epoch_end                                                                                                                                                                             \t|  2.4024e-06     \t|  19             \t|  4.5646e-05     \t|  4.4091e-06     \t|\n",
      "|  [Callback]ModelSummary.on_train_start                                                                                                                                                                           \t|  4.533e-05      \t|  1              \t|  4.533e-05      \t|  4.3786e-06     \t|\n",
      "|  [Callback]TQDMProgressBar.on_save_checkpoint                                                                                                                                                                    \t|  5.8962e-06     \t|  6              \t|  3.5377e-05     \t|  3.4172e-06     \t|\n",
      "|  [LightningModule]Model.on_validation_end                                                                                                                                                                        \t|  3.4332e-06     \t|  10             \t|  3.4332e-05     \t|  3.3163e-06     \t|\n",
      "|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                                \t|  2.9985e-06     \t|  10             \t|  2.9985e-05     \t|  2.8964e-06     \t|\n",
      "|  [LightningModule]Model.on_save_checkpoint                                                                                                                                                                       \t|  4.9433e-06     \t|  6              \t|  2.966e-05      \t|  2.865e-06      \t|\n",
      "|  [LightningModule]Model.on_validation_start                                                                                                                                                                      \t|  2.8826e-06     \t|  10             \t|  2.8826e-05     \t|  2.7844e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_train_end                                                                                                                                      \t|  2.8456e-05     \t|  1              \t|  2.8456e-05     \t|  2.7487e-06     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_save_checkpoint                 \t|  4.693e-06      \t|  6              \t|  2.8158e-05     \t|  2.7199e-06     \t|\n",
      "|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                              \t|  2.759e-06      \t|  10             \t|  2.759e-05      \t|  2.665e-06      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     \t|  2.5755e-05     \t|  1              \t|  2.5755e-05     \t|  2.4878e-06     \t|\n",
      "|  [LightningModule]Model.on_validation_epoch_start                                                                                                                                                                \t|  2.3547e-06     \t|  10             \t|  2.3547e-05     \t|  2.2745e-06     \t|\n",
      "|  [Callback]TQDMProgressBar.on_sanity_check_end                                                                                                                                                                   \t|  2.2204e-05     \t|  1              \t|  2.2204e-05     \t|  2.1448e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.setup                                                                                                                                             \t|  2.1144e-05     \t|  1              \t|  2.1144e-05     \t|  2.0424e-06     \t|\n",
      "|  [Callback]ModelSummary.on_train_end                                                                                                                                                                             \t|  2.02e-05       \t|  1              \t|  2.02e-05       \t|  1.9512e-06     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                            \t|  1.9313e-05     \t|  1              \t|  1.9313e-05     \t|  1.8655e-06     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                          \t|  1.8056e-05     \t|  1              \t|  1.8056e-05     \t|  1.7441e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.teardown                                                                                                                                          \t|  1.6018e-05     \t|  1              \t|  1.6018e-05     \t|  1.5472e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_fit_end                                                                                                                                        \t|  1.2208e-05     \t|  1              \t|  1.2208e-05     \t|  1.1792e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_sanity_check_end                                                                                                                               \t|  1.1828e-05     \t|  1              \t|  1.1828e-05     \t|  1.1425e-06     \t|\n",
      "|  [Callback]TQDMProgressBar.setup                                                                                                                                                                                 \t|  1.1639e-05     \t|  1              \t|  1.1639e-05     \t|  1.1243e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_train_start                                                                                                                                    \t|  1.0905e-05     \t|  1              \t|  1.0905e-05     \t|  1.0534e-06     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_sanity_check_start                                                                                                                             \t|  9.687e-06      \t|  1              \t|  9.687e-06      \t|  9.3571e-07     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_before_accelerator_backend_setup                                                                                                               \t|  8.982e-06      \t|  1              \t|  8.982e-06      \t|  8.6761e-07     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_pretrain_routine_start                                                                                                                         \t|  8.735e-06      \t|  1              \t|  8.735e-06      \t|  8.4375e-07     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                            \t|  7.96e-06       \t|  1              \t|  7.96e-06       \t|  7.6889e-07     \t|\n",
      "|  [Callback]ModelSummary.on_sanity_check_end                                                                                                                                                                      \t|  7.892e-06      \t|  1              \t|  7.892e-06      \t|  7.6232e-07     \t|\n",
      "|  [LightningModule]Model.configure_callbacks                                                                                                                                                                      \t|  7.678e-06      \t|  1              \t|  7.678e-06      \t|  7.4165e-07     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_sanity_check_end                \t|  7.623e-06      \t|  1              \t|  7.623e-06      \t|  7.3634e-07     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_sanity_check_end                                                                                                                                                     \t|  7.397e-06      \t|  1              \t|  7.397e-06      \t|  7.1451e-07     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       \t|  7.375e-06      \t|  1              \t|  7.375e-06      \t|  7.1238e-07     \t|\n",
      "|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                            \t|  6.705e-06      \t|  1              \t|  6.705e-06      \t|  6.4766e-07     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       \t|  6.439e-06      \t|  1              \t|  6.439e-06      \t|  6.2197e-07     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_fit_start                                                                                                                                      \t|  6.282e-06      \t|  1              \t|  6.282e-06      \t|  6.068e-07      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                              \t|  6.145e-06      \t|  1              \t|  6.145e-06      \t|  5.9357e-07     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_configure_sharded_model                                                                                                                        \t|  6.03e-06       \t|  1              \t|  6.03e-06       \t|  5.8246e-07     \t|\n",
      "|  [Callback]ModelSummary.setup                                                                                                                                                                                    \t|  5.683e-06      \t|  1              \t|  5.683e-06      \t|  5.4894e-07     \t|\n",
      "|  [LightningModule]Model.on_train_start                                                                                                                                                                           \t|  5.618e-06      \t|  1              \t|  5.618e-06      \t|  5.4267e-07     \t|\n",
      "|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                            \t|  4.771e-06      \t|  1              \t|  4.771e-06      \t|  4.6085e-07     \t|\n",
      "|  [Callback]ModelSummary.on_fit_end                                                                                                                                                                               \t|  4.759e-06      \t|  1              \t|  4.759e-06      \t|  4.5969e-07     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                                   \t|  4.71e-06       \t|  1              \t|  4.71e-06       \t|  4.5496e-07     \t|\n",
      "|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                                   \t|  4.692e-06      \t|  1              \t|  4.692e-06      \t|  4.5322e-07     \t|\n",
      "|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                                   \t|  4.68e-06       \t|  1              \t|  4.68e-06       \t|  4.5206e-07     \t|\n",
      "|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                             \t|  4.495e-06      \t|  1              \t|  4.495e-06      \t|  4.3419e-07     \t|\n",
      "|  [LightningModule]Model.on_train_dataloader                                                                                                                                                                      \t|  4.478e-06      \t|  1              \t|  4.478e-06      \t|  4.3255e-07     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         \t|  4.376e-06      \t|  1              \t|  4.376e-06      \t|  4.227e-07      \t|\n",
      "|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                          \t|  4.219e-06      \t|  1              \t|  4.219e-06      \t|  4.0753e-07     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          \t|  4.192e-06      \t|  1              \t|  4.192e-06      \t|  4.0492e-07     \t|\n",
      "|  [Callback]EarlyStopping{'monitor': 'val_loss', 'mode': 'min'}.on_pretrain_routine_end                                                                                                                           \t|  4.17e-06       \t|  1              \t|  4.17e-06       \t|  4.028e-07      \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.on_before_accelerator_backend_setup\t|  4.052e-06      \t|  1              \t|  4.052e-06      \t|  3.914e-07      \t|\n",
      "|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                              \t|  4.023e-06      \t|  1              \t|  4.023e-06      \t|  3.886e-07      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                                 \t|  3.912e-06      \t|  1              \t|  3.912e-06      \t|  3.7788e-07     \t|\n",
      "|  [Callback]ModelSummary.on_before_accelerator_backend_setup                                                                                                                                                      \t|  3.831e-06      \t|  1              \t|  3.831e-06      \t|  3.7005e-07     \t|\n",
      "|  [LightningModule]Model.on_val_dataloader                                                                                                                                                                        \t|  3.815e-06      \t|  1              \t|  3.815e-06      \t|  3.6851e-07     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                     \t|  3.79e-06       \t|  1              \t|  3.79e-06       \t|  3.6609e-07     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            \t|  3.764e-06      \t|  1              \t|  3.764e-06      \t|  3.6358e-07     \t|\n",
      "|  [Callback]ModelSummary.on_pretrain_routine_start                                                                                                                                                                \t|  3.763e-06      \t|  1              \t|  3.763e-06      \t|  3.6348e-07     \t|\n",
      "|  [Callback]ModelSummary.teardown                                                                                                                                                                                 \t|  3.74e-06       \t|  1              \t|  3.74e-06       \t|  3.6126e-07     \t|\n",
      "|  [Callback]ModelSummary.on_configure_sharded_model                                                                                                                                                               \t|  3.707e-06      \t|  1              \t|  3.707e-06      \t|  3.5807e-07     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         \t|  3.7e-06        \t|  1              \t|  3.7e-06        \t|  3.574e-07      \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                               \t|  3.674e-06      \t|  1              \t|  3.674e-06      \t|  3.5489e-07     \t|\n",
      "|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                               \t|  3.515e-06      \t|  1              \t|  3.515e-06      \t|  3.3953e-07     \t|\n",
      "|  [Callback]ModelCheckpoint{'monitor': 'val_loss', 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           \t|  3.444e-06      \t|  1              \t|  3.444e-06      \t|  3.3267e-07     \t|\n",
      "|  [LightningModule]Model.prepare_data                                                                                                                                                                             \t|  3.397e-06      \t|  1              \t|  3.397e-06      \t|  3.2813e-07     \t|\n",
      "|  [Callback]ModelSummary.on_pretrain_routine_end                                                                                                                                                                  \t|  3.361e-06      \t|  1              \t|  3.361e-06      \t|  3.2465e-07     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                              \t|  3.315e-06      \t|  1              \t|  3.315e-06      \t|  3.2021e-07     \t|\n",
      "|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                                \t|  3.142e-06      \t|  1              \t|  3.142e-06      \t|  3.035e-07      \t|\n",
      "|  [LightningModule]Model.on_fit_end                                                                                                                                                                               \t|  3.09e-06       \t|  1              \t|  3.09e-06       \t|  2.9848e-07     \t|\n",
      "|  [LightningModule]Model.on_train_end                                                                                                                                                                             \t|  3.075e-06      \t|  1              \t|  3.075e-06      \t|  2.9703e-07     \t|\n",
      "|  [LightningModule]Model.configure_sharded_model                                                                                                                                                                  \t|  2.912e-06      \t|  1              \t|  2.912e-06      \t|  2.8128e-07     \t|\n",
      "|  [LightningModule]Model.setup                                                                                                                                                                                    \t|  2.681e-06      \t|  1              \t|  2.681e-06      \t|  2.5897e-07     \t|\n",
      "|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                     \t|  2.507e-06      \t|  1              \t|  2.507e-06      \t|  2.4216e-07     \t|\n",
      "|  [LightningModule]Model.on_fit_start                                                                                                                                                                             \t|  2.148e-06      \t|  1              \t|  2.148e-06      \t|  2.0748e-07     \t|\n",
      "|  [LightningModule]Model.on_pretrain_routine_end                                                                                                                                                                  \t|  2.145e-06      \t|  1              \t|  2.145e-06      \t|  2.0719e-07     \t|\n",
      "|  [LightningModule]Model.on_pretrain_routine_start                                                                                                                                                                \t|  1.893e-06      \t|  1              \t|  1.893e-06      \t|  1.8285e-07     \t|\n",
      "|  [LightningModule]Model.teardown                                                                                                                                                                                 \t|  1.312e-06      \t|  1              \t|  1.312e-06      \t|  1.2673e-07     \t|\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pl.seed_everything(1234)\n",
    "\n",
    "ckpt_callback = pl.callbacks.ModelCheckpoint(\n",
    "    monitor='val_loss',\n",
    "    save_top_k=1,\n",
    "    mode='min'\n",
    ")\n",
    "early_stopping = pl.callbacks.EarlyStopping(monitor = 'val_loss',\n",
    "               patience=3,\n",
    "               mode = 'min')\n",
    "\n",
    "# gpus=0 则使用cpu训练，gpus=1则使用1个gpu训练，gpus=2则使用2个gpu训练，gpus=-1则使用所有gpu训练，\n",
    "# gpus=[0,1]则指定使用0号和1号gpu训练， gpus=\"0,1,2,3\"则使用0,1,2,3号gpu训练\n",
    "# tpus=1 则使用1个tpu训练\n",
    "\n",
    "trainer = pl.Trainer(max_epochs=20,   \n",
    "     #gpus=0, #单CPU模式\n",
    "     gpus=0, #单GPU模式\n",
    "     #num_processes=4,strategy=\"ddp_find_unused_parameters_false\", #多CPU(进程)模式\n",
    "     #gpus=[0,1,2,3],strategy=\"dp\", #多GPU的DataParallel(速度提升效果一般)\n",
    "     #gpus=[0,1,2,3],strategy=“ddp_find_unused_parameters_false\" #多GPU的DistributedDataParallel(速度提升效果好)\n",
    "     callbacks = [ckpt_callback,early_stopping],\n",
    "     profiler=\"simple\") \n",
    "\n",
    "#断点续训\n",
    "#trainer = pl.Trainer(resume_from_checkpoint='./lightning_logs/version_31/checkpoints/epoch=02-val_loss=0.05.ckpt')\n",
    "\n",
    "#训练模型\n",
    "trainer.fit(model,data_mnist)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2e76bfd",
   "metadata": {},
   "source": [
    "```\n",
    "Epoch 8: 100%\n",
    "1876/1876 [01:44<00:00, 17.93it/s, loss=0.0603, v_num=0, train_acc=1.000, val_acc=0.985]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b09b9eb2-05fb-4cab-8889-b78ee8514e10",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "75cef2a1",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 4，评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "id": "c0f856fc-3a0c-45e1-8080-40a63d6b7864",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=11-step=20627.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=11-step=20627.ckpt\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:662: UserWarning:\n",
      "\n",
      "Your `test_dataloader` has `shuffle=True`, it is strongly recommended that you turn this off for val/test/predict dataloaders.\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "efbbb003c62b41f089b1aea84e7695a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TEST Profiler Report\n",
      "\n",
      "Action                             \t|  Mean duration (s)\t|Num calls      \t|  Total time (s) \t|  Percentage %   \t|\n",
      "---------------------------------------------------------------------------------------------------------------------------------------\n",
      "Total                              \t|  -              \t|_              \t|  404.7          \t|  100 %          \t|\n",
      "---------------------------------------------------------------------------------------------------------------------------------------\n",
      "run_training_epoch                 \t|  21.814         \t|15             \t|  327.21         \t|  80.853         \t|\n",
      "run_training_batch                 \t|  0.0076827      \t|25785          \t|  198.1          \t|  48.949         \t|\n",
      "optimizer_step_with_closure_0      \t|  0.0072715      \t|25785          \t|  187.5          \t|  46.329         \t|\n",
      "training_step_and_backward         \t|  0.0061708      \t|25785          \t|  159.11         \t|  39.316         \t|\n",
      "model_forward                      \t|  0.0041915      \t|25785          \t|  108.08         \t|  26.705         \t|\n",
      "training_step_end                  \t|  0.0028167      \t|25785          \t|  72.629         \t|  17.946         \t|\n",
      "on_train_batch_end                 \t|  0.002178       \t|25785          \t|  56.159         \t|  13.877         \t|\n",
      "backward                           \t|  0.0016287      \t|25785          \t|  41.995         \t|  10.377         \t|\n",
      "training_step                      \t|  0.0011591      \t|25785          \t|  29.888         \t|  7.3851         \t|\n",
      "evaluation_step_and_end            \t|  0.0044375      \t|4546           \t|  20.173         \t|  4.9846         \t|\n",
      "run_test_evaluation                \t|  4.4232         \t|3              \t|  13.27          \t|  3.2788         \t|\n",
      "zero_grad                          \t|  0.00031003     \t|25785          \t|  7.9942         \t|  1.9753         \t|\n",
      "get_train_batch                    \t|  0.00030696     \t|25800          \t|  7.9194         \t|  1.9568         \t|\n",
      "on_train_batch_start               \t|  0.0003015      \t|25785          \t|  7.7742         \t|  1.921          \t|\n",
      "training_batch_to_device           \t|  0.00029603     \t|25785          \t|  7.633          \t|  1.8861         \t|\n",
      "validation_step_end                \t|  0.0031315      \t|2357           \t|  7.3808         \t|  1.8238         \t|\n",
      "fetch_next_train_batch             \t|  0.00028266     \t|25800          \t|  7.2927         \t|  1.802          \t|\n",
      "test_step_end                      \t|  0.0029989      \t|2189           \t|  6.5646         \t|  1.6221         \t|\n",
      "validation_step                    \t|  0.001317       \t|2357           \t|  3.1043         \t|  0.76705        \t|\n",
      "test_step                          \t|  0.001246       \t|2189           \t|  2.7275         \t|  0.67395        \t|\n",
      "get_validate_batch                 \t|  0.00083031     \t|2370           \t|  1.9678         \t|  0.48624        \t|\n",
      "fetch_next_validate_batch          \t|  0.00080573     \t|2370           \t|  1.9096         \t|  0.47184        \t|\n",
      "evaluation_batch_to_device         \t|  0.0003088      \t|4546           \t|  1.4038         \t|  0.34687        \t|\n",
      "on_batch_start                     \t|  4.1352e-05     \t|25785          \t|  1.0663         \t|  0.26346        \t|\n",
      "on_batch_end                       \t|  3.8432e-05     \t|25785          \t|  0.99096        \t|  0.24486        \t|\n",
      "on_before_zero_grad                \t|  3.8268e-05     \t|25785          \t|  0.98675        \t|  0.24382        \t|\n",
      "on_after_backward                  \t|  3.6081e-05     \t|25785          \t|  0.93036        \t|  0.22989        \t|\n",
      "on_before_backward                 \t|  3.2031e-05     \t|25785          \t|  0.82592        \t|  0.20408        \t|\n",
      "on_before_optimizer_step           \t|  3.0111e-05     \t|25785          \t|  0.77642        \t|  0.19185        \t|\n",
      "get_test_batch                     \t|  0.00034506     \t|2192           \t|  0.75637        \t|  0.18689        \t|\n",
      "on_validation_batch_end            \t|  0.00030486     \t|2357           \t|  0.71855        \t|  0.17755        \t|\n",
      "fetch_next_test_batch              \t|  0.00032118     \t|2192           \t|  0.70402        \t|  0.17396        \t|\n",
      "on_validation_start                \t|  0.034396       \t|16             \t|  0.55034        \t|  0.13598        \t|\n",
      "on_test_batch_end                  \t|  0.00015042     \t|2189           \t|  0.32926        \t|  0.081358       \t|\n",
      "on_test_start                      \t|  0.032437       \t|3              \t|  0.097312       \t|  0.024045       \t|\n",
      "on_validation_batch_start          \t|  3.8675e-05     \t|2357           \t|  0.091158       \t|  0.022525       \t|\n",
      "on_train_epoch_end                 \t|  0.0055654      \t|15             \t|  0.083482       \t|  0.020628       \t|\n",
      "on_test_batch_start                \t|  3.6226e-05     \t|2189           \t|  0.079299       \t|  0.019594       \t|\n",
      "on_validation_end                  \t|  0.0030449      \t|16             \t|  0.048719       \t|  0.012038       \t|\n",
      "on_train_epoch_start               \t|  0.0026013      \t|15             \t|  0.03902        \t|  0.0096415      \t|\n",
      "on_train_start                     \t|  0.036383       \t|1              \t|  0.036383       \t|  0.0089901      \t|\n",
      "on_sanity_check_start              \t|  0.034045       \t|1              \t|  0.034045       \t|  0.0084122      \t|\n",
      "get_sanity_check_batch             \t|  0.0064563      \t|3              \t|  0.019369       \t|  0.0047859      \t|\n",
      "fetch_next_sanity_check_batch      \t|  0.0064196      \t|3              \t|  0.019259       \t|  0.0047587      \t|\n",
      "on_pretrain_routine_start          \t|  0.0055585      \t|1              \t|  0.0055585      \t|  0.0013735      \t|\n",
      "on_test_end                        \t|  0.0017212      \t|3              \t|  0.0051637      \t|  0.0012759      \t|\n",
      "on_validation_model_eval           \t|  0.00027574     \t|16             \t|  0.0044118      \t|  0.0010901      \t|\n",
      "on_train_end                       \t|  0.0012372      \t|1              \t|  0.0012372      \t|  0.00030571     \t|\n",
      "on_epoch_start                     \t|  3.4914e-05     \t|34             \t|  0.0011871      \t|  0.00029332     \t|\n",
      "on_epoch_end                       \t|  3.2898e-05     \t|34             \t|  0.0011185      \t|  0.00027638     \t|\n",
      "on_validation_epoch_end            \t|  4.788e-05      \t|16             \t|  0.00076608     \t|  0.00018929     \t|\n",
      "on_test_model_eval                 \t|  0.0002197      \t|3              \t|  0.00065911     \t|  0.00016286     \t|\n",
      "on_validation_epoch_start          \t|  3.1486e-05     \t|16             \t|  0.00050377     \t|  0.00012448     \t|\n",
      "configure_optimizers               \t|  0.00027334     \t|1              \t|  0.00027334     \t|  6.7542e-05     \t|\n",
      "on_configure_sharded_model         \t|  4.7993e-05     \t|4              \t|  0.00019197     \t|  4.7435e-05     \t|\n",
      "teardown                           \t|  4.1536e-05     \t|4              \t|  0.00016614     \t|  4.1053e-05     \t|\n",
      "on_before_accelerator_backend_setup\t|  3.8753e-05     \t|4              \t|  0.00015501     \t|  3.8303e-05     \t|\n",
      "on_test_epoch_end                  \t|  4.8818e-05     \t|3              \t|  0.00014645     \t|  3.6188e-05     \t|\n",
      "setup                              \t|  3.5844e-05     \t|4              \t|  0.00014338     \t|  3.5427e-05     \t|\n",
      "on_test_epoch_start                \t|  3.1454e-05     \t|3              \t|  9.4362e-05     \t|  2.3316e-05     \t|\n",
      "on_sanity_check_end                \t|  6.0066e-05     \t|1              \t|  6.0066e-05     \t|  1.4842e-05     \t|\n",
      "configure_callbacks                \t|  1.4934e-05     \t|4              \t|  5.9735e-05     \t|  1.476e-05      \t|\n",
      "configure_sharded_model            \t|  1.4916e-05     \t|4              \t|  5.9664e-05     \t|  1.4743e-05     \t|\n",
      "on_fit_end                         \t|  5.3426e-05     \t|1              \t|  5.3426e-05     \t|  1.3201e-05     \t|\n",
      "on_test_dataloader                 \t|  1.5845e-05     \t|3              \t|  4.7536e-05     \t|  1.1746e-05     \t|\n",
      "on_fit_start                       \t|  4.3138e-05     \t|1              \t|  4.3138e-05     \t|  1.0659e-05     \t|\n",
      "on_pretrain_routine_end            \t|  3.9236e-05     \t|1              \t|  3.9236e-05     \t|  9.6949e-06     \t|\n",
      "prepare_data                       \t|  8.8867e-06     \t|4              \t|  3.5547e-05     \t|  8.7834e-06     \t|\n",
      "on_train_dataloader                \t|  1.6958e-05     \t|1              \t|  1.6958e-05     \t|  4.1901e-06     \t|\n",
      "on_val_dataloader                  \t|  9.2648e-06     \t|1              \t|  9.2648e-06     \t|  2.2893e-06     \t|\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9966545701026917, 'test_loss': 0.010617421939969063}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "result = trainer.test(model,data_mnist.train_dataloader(),ckpt_path='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "997ac09a",
   "metadata": {},
   "source": [
    "```\n",
    "--------------------------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9966545701026917, 'test_loss': 0.010617421939969063}\n",
    "--------------------------------------------------------------------------------\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9360b9c3",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "id": "b14b47d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning:\n",
      "\n",
      "DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.\n",
      "\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=11-step=20627.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=11-step=20627.ckpt\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "244b108d855c45aca18b09fcb82b24ad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning:\n",
      "\n",
      "DataModule.teardown has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.teardown.\n",
      "\n",
      "TEST Profiler Report\n",
      "\n",
      "Action                             \t|  Mean duration (s)\t|Num calls      \t|  Total time (s) \t|  Percentage %   \t|\n",
      "---------------------------------------------------------------------------------------------------------------------------------------\n",
      "Total                              \t|  -              \t|_              \t|  371.98         \t|  100 %          \t|\n",
      "---------------------------------------------------------------------------------------------------------------------------------------\n",
      "run_training_epoch                 \t|  21.814         \t|15             \t|  327.21         \t|  87.966         \t|\n",
      "run_training_batch                 \t|  0.0076827      \t|25785          \t|  198.1          \t|  53.255         \t|\n",
      "optimizer_step_with_closure_0      \t|  0.0072715      \t|25785          \t|  187.5          \t|  50.405         \t|\n",
      "training_step_and_backward         \t|  0.0061708      \t|25785          \t|  159.11         \t|  42.775         \t|\n",
      "model_forward                      \t|  0.0041915      \t|25785          \t|  108.08         \t|  29.055         \t|\n",
      "training_step_end                  \t|  0.0028167      \t|25785          \t|  72.629         \t|  19.525         \t|\n",
      "on_train_batch_end                 \t|  0.002178       \t|25785          \t|  56.159         \t|  15.097         \t|\n",
      "backward                           \t|  0.0016287      \t|25785          \t|  41.995         \t|  11.29          \t|\n",
      "training_step                      \t|  0.0011591      \t|25785          \t|  29.888         \t|  8.0349         \t|\n",
      "evaluation_step_and_end            \t|  0.004534       \t|2827           \t|  12.818         \t|  3.4458         \t|\n",
      "zero_grad                          \t|  0.00031003     \t|25785          \t|  7.9942         \t|  2.1491         \t|\n",
      "get_train_batch                    \t|  0.00030696     \t|25800          \t|  7.9194         \t|  2.129          \t|\n",
      "on_train_batch_start               \t|  0.0003015      \t|25785          \t|  7.7742         \t|  2.09           \t|\n",
      "training_batch_to_device           \t|  0.00029603     \t|25785          \t|  7.633          \t|  2.052          \t|\n",
      "validation_step_end                \t|  0.0031315      \t|2357           \t|  7.3808         \t|  1.9842         \t|\n",
      "fetch_next_train_batch             \t|  0.00028266     \t|25800          \t|  7.2927         \t|  1.9605         \t|\n",
      "run_test_evaluation                \t|  1.7593         \t|2              \t|  3.5187         \t|  0.94594        \t|\n",
      "validation_step                    \t|  0.001317       \t|2357           \t|  3.1043         \t|  0.83454        \t|\n",
      "get_validate_batch                 \t|  0.00083031     \t|2370           \t|  1.9678         \t|  0.52902        \t|\n",
      "fetch_next_validate_batch          \t|  0.00080573     \t|2370           \t|  1.9096         \t|  0.51336        \t|\n",
      "test_step_end                      \t|  0.0030928      \t|470            \t|  1.4536         \t|  0.39079        \t|\n",
      "on_batch_start                     \t|  4.1352e-05     \t|25785          \t|  1.0663         \t|  0.28664        \t|\n",
      "on_batch_end                       \t|  3.8432e-05     \t|25785          \t|  0.99096        \t|  0.2664         \t|\n",
      "on_before_zero_grad                \t|  3.8268e-05     \t|25785          \t|  0.98675        \t|  0.26527        \t|\n",
      "on_after_backward                  \t|  3.6081e-05     \t|25785          \t|  0.93036        \t|  0.25011        \t|\n",
      "evaluation_batch_to_device         \t|  0.00031589     \t|2827           \t|  0.89301        \t|  0.24007        \t|\n",
      "on_before_backward                 \t|  3.2031e-05     \t|25785          \t|  0.82592        \t|  0.22203        \t|\n",
      "on_before_optimizer_step           \t|  3.0111e-05     \t|25785          \t|  0.77642        \t|  0.20873        \t|\n",
      "on_validation_batch_end            \t|  0.00030486     \t|2357           \t|  0.71855        \t|  0.19317        \t|\n",
      "test_step                          \t|  0.0013339      \t|470            \t|  0.62694        \t|  0.16854        \t|\n",
      "on_validation_start                \t|  0.034396       \t|16             \t|  0.55034        \t|  0.14795        \t|\n",
      "get_test_batch                     \t|  0.00067238     \t|472            \t|  0.31736        \t|  0.085318       \t|\n",
      "fetch_next_test_batch              \t|  0.0006473      \t|472            \t|  0.30552        \t|  0.082135       \t|\n",
      "on_validation_batch_start          \t|  3.8675e-05     \t|2357           \t|  0.091158       \t|  0.024506       \t|\n",
      "on_train_epoch_end                 \t|  0.0055654      \t|15             \t|  0.083482       \t|  0.022443       \t|\n",
      "on_test_batch_end                  \t|  0.00016392     \t|470            \t|  0.077042       \t|  0.020711       \t|\n",
      "on_test_start                      \t|  0.032143       \t|2              \t|  0.064287       \t|  0.017282       \t|\n",
      "on_validation_end                  \t|  0.0030449      \t|16             \t|  0.048719       \t|  0.013097       \t|\n",
      "on_train_epoch_start               \t|  0.0026013      \t|15             \t|  0.03902        \t|  0.01049        \t|\n",
      "on_train_start                     \t|  0.036383       \t|1              \t|  0.036383       \t|  0.009781       \t|\n",
      "on_sanity_check_start              \t|  0.034045       \t|1              \t|  0.034045       \t|  0.0091523      \t|\n",
      "get_sanity_check_batch             \t|  0.0064563      \t|3              \t|  0.019369       \t|  0.005207       \t|\n",
      "fetch_next_sanity_check_batch      \t|  0.0064196      \t|3              \t|  0.019259       \t|  0.0051774      \t|\n",
      "on_test_batch_start                \t|  3.7254e-05     \t|470            \t|  0.01751        \t|  0.0047072      \t|\n",
      "on_pretrain_routine_start          \t|  0.0055585      \t|1              \t|  0.0055585      \t|  0.0014943      \t|\n",
      "on_validation_model_eval           \t|  0.00027574     \t|16             \t|  0.0044118      \t|  0.001186       \t|\n",
      "on_test_end                        \t|  0.0020979      \t|2              \t|  0.0041958      \t|  0.001128       \t|\n",
      "on_train_end                       \t|  0.0012372      \t|1              \t|  0.0012372      \t|  0.00033261     \t|\n",
      "on_epoch_start                     \t|  3.4934e-05     \t|33             \t|  0.0011528      \t|  0.00030992     \t|\n",
      "on_epoch_end                       \t|  3.3019e-05     \t|33             \t|  0.0010896      \t|  0.00029293     \t|\n",
      "on_validation_epoch_end            \t|  4.788e-05      \t|16             \t|  0.00076608     \t|  0.00020595     \t|\n",
      "on_validation_epoch_start          \t|  3.1486e-05     \t|16             \t|  0.00050377     \t|  0.00013543     \t|\n",
      "on_test_model_eval                 \t|  0.00021777     \t|2              \t|  0.00043555     \t|  0.00011709     \t|\n",
      "configure_optimizers               \t|  0.00027334     \t|1              \t|  0.00027334     \t|  7.3484e-05     \t|\n",
      "on_configure_sharded_model         \t|  4.8459e-05     \t|3              \t|  0.00014538     \t|  3.9082e-05     \t|\n",
      "teardown                           \t|  4.0405e-05     \t|3              \t|  0.00012122     \t|  3.2587e-05     \t|\n",
      "on_before_accelerator_backend_setup\t|  3.8253e-05     \t|3              \t|  0.00011476     \t|  3.0851e-05     \t|\n",
      "setup                              \t|  3.7334e-05     \t|3              \t|  0.000112       \t|  3.011e-05      \t|\n",
      "on_test_epoch_end                  \t|  4.7239e-05     \t|2              \t|  9.4477e-05     \t|  2.5399e-05     \t|\n",
      "on_test_epoch_start                \t|  3.185e-05      \t|2              \t|  6.37e-05       \t|  1.7125e-05     \t|\n",
      "on_sanity_check_end                \t|  6.0066e-05     \t|1              \t|  6.0066e-05     \t|  1.6148e-05     \t|\n",
      "on_fit_end                         \t|  5.3426e-05     \t|1              \t|  5.3426e-05     \t|  1.4363e-05     \t|\n",
      "configure_callbacks                \t|  1.5653e-05     \t|3              \t|  4.696e-05      \t|  1.2624e-05     \t|\n",
      "configure_sharded_model            \t|  1.4702e-05     \t|3              \t|  4.4105e-05     \t|  1.1857e-05     \t|\n",
      "on_fit_start                       \t|  4.3138e-05     \t|1              \t|  4.3138e-05     \t|  1.1597e-05     \t|\n",
      "on_pretrain_routine_end            \t|  3.9236e-05     \t|1              \t|  3.9236e-05     \t|  1.0548e-05     \t|\n",
      "on_test_dataloader                 \t|  1.6155e-05     \t|2              \t|  3.2309e-05     \t|  8.6859e-06     \t|\n",
      "prepare_data                       \t|  8.8324e-06     \t|3              \t|  2.6497e-05     \t|  7.1233e-06     \t|\n",
      "on_train_dataloader                \t|  1.6958e-05     \t|1              \t|  1.6958e-05     \t|  4.5587e-06     \t|\n",
      "on_val_dataloader                  \t|  9.2648e-06     \t|1              \t|  9.2648e-06     \t|  2.4907e-06     \t|\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9865999817848206, 'test_loss': 0.042671505361795425}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "result = trainer.test(model,data_mnist.val_dataloader(),ckpt_path='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af974a6b",
   "metadata": {},
   "source": [
    "```\n",
    "--------------------------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9865999817848206, 'test_loss': 0.042671505361795425}\n",
    "--------------------------------------------------------------------------------\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "b30d7de3-91b5-465b-a266-b5fd1b532d8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=11-step=20627.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=11-step=20627.ckpt\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2580d3fa23a9484fbdb681f348add9c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TEST Profiler Report\n",
      "\n",
      "Action                             \t|  Mean duration (s)\t|Num calls      \t|  Total time (s) \t|  Percentage %   \t|\n",
      "---------------------------------------------------------------------------------------------------------------------------------------\n",
      "Total                              \t|  -              \t|_              \t|  421.76         \t|  100 %          \t|\n",
      "---------------------------------------------------------------------------------------------------------------------------------------\n",
      "run_training_epoch                 \t|  21.814         \t|15             \t|  327.21         \t|  77.583         \t|\n",
      "run_training_batch                 \t|  0.0076827      \t|25785          \t|  198.1          \t|  46.97          \t|\n",
      "optimizer_step_with_closure_0      \t|  0.0072715      \t|25785          \t|  187.5          \t|  44.456         \t|\n",
      "training_step_and_backward         \t|  0.0061708      \t|25785          \t|  159.11         \t|  37.726         \t|\n",
      "model_forward                      \t|  0.0041915      \t|25785          \t|  108.08         \t|  25.625         \t|\n",
      "training_step_end                  \t|  0.0028167      \t|25785          \t|  72.629         \t|  17.221         \t|\n",
      "on_train_batch_end                 \t|  0.002178       \t|25785          \t|  56.159         \t|  13.316         \t|\n",
      "backward                           \t|  0.0016287      \t|25785          \t|  41.995         \t|  9.9573         \t|\n",
      "training_step                      \t|  0.0011591      \t|25785          \t|  29.888         \t|  7.0865         \t|\n",
      "evaluation_step_and_end            \t|  0.0044129      \t|4859           \t|  21.442         \t|  5.084          \t|\n",
      "run_test_evaluation                \t|  3.8129         \t|4              \t|  15.252         \t|  3.6162         \t|\n",
      "zero_grad                          \t|  0.00031003     \t|25785          \t|  7.9942         \t|  1.8955         \t|\n",
      "get_train_batch                    \t|  0.00030696     \t|25800          \t|  7.9194         \t|  1.8777         \t|\n",
      "on_train_batch_start               \t|  0.0003015      \t|25785          \t|  7.7742         \t|  1.8433         \t|\n",
      "training_batch_to_device           \t|  0.00029603     \t|25785          \t|  7.633          \t|  1.8098         \t|\n",
      "test_step_end                      \t|  0.0029875      \t|2502           \t|  7.4748         \t|  1.7723         \t|\n",
      "validation_step_end                \t|  0.0031315      \t|2357           \t|  7.3808         \t|  1.75           \t|\n",
      "fetch_next_train_batch             \t|  0.00028266     \t|25800          \t|  7.2927         \t|  1.7291         \t|\n",
      "validation_step                    \t|  0.001317       \t|2357           \t|  3.1043         \t|  0.73604        \t|\n",
      "test_step                          \t|  0.0012236      \t|2502           \t|  3.0615         \t|  0.72589        \t|\n",
      "get_validate_batch                 \t|  0.00083031     \t|2370           \t|  1.9678         \t|  0.46658        \t|\n",
      "fetch_next_validate_batch          \t|  0.00080573     \t|2370           \t|  1.9096         \t|  0.45277        \t|\n",
      "evaluation_batch_to_device         \t|  0.00030573     \t|4859           \t|  1.4855         \t|  0.35222        \t|\n",
      "on_batch_start                     \t|  4.1352e-05     \t|25785          \t|  1.0663         \t|  0.25281        \t|\n",
      "on_batch_end                       \t|  3.8432e-05     \t|25785          \t|  0.99096        \t|  0.23496        \t|\n",
      "on_before_zero_grad                \t|  3.8268e-05     \t|25785          \t|  0.98675        \t|  0.23396        \t|\n",
      "on_after_backward                  \t|  3.6081e-05     \t|25785          \t|  0.93036        \t|  0.22059        \t|\n",
      "get_test_batch                     \t|  0.00035842     \t|2506           \t|  0.89821        \t|  0.21297        \t|\n",
      "fetch_next_test_batch              \t|  0.00033491     \t|2506           \t|  0.83928        \t|  0.199          \t|\n",
      "on_before_backward                 \t|  3.2031e-05     \t|25785          \t|  0.82592        \t|  0.19583        \t|\n",
      "on_before_optimizer_step           \t|  3.0111e-05     \t|25785          \t|  0.77642        \t|  0.18409        \t|\n",
      "on_validation_batch_end            \t|  0.00030486     \t|2357           \t|  0.71855        \t|  0.17037        \t|\n",
      "on_validation_start                \t|  0.034396       \t|16             \t|  0.55034        \t|  0.13049        \t|\n",
      "on_test_batch_end                  \t|  0.00014831     \t|2502           \t|  0.37107        \t|  0.087982       \t|\n",
      "on_test_start                      \t|  0.031928       \t|4              \t|  0.12771        \t|  0.030281       \t|\n",
      "on_validation_batch_start          \t|  3.8675e-05     \t|2357           \t|  0.091158       \t|  0.021614       \t|\n",
      "on_test_batch_start                \t|  3.5954e-05     \t|2502           \t|  0.089957       \t|  0.021329       \t|\n",
      "on_train_epoch_end                 \t|  0.0055654      \t|15             \t|  0.083482       \t|  0.019794       \t|\n",
      "on_validation_end                  \t|  0.0030449      \t|16             \t|  0.048719       \t|  0.011551       \t|\n",
      "on_train_epoch_start               \t|  0.0026013      \t|15             \t|  0.03902        \t|  0.0092517      \t|\n",
      "on_train_start                     \t|  0.036383       \t|1              \t|  0.036383       \t|  0.0086266      \t|\n",
      "on_sanity_check_start              \t|  0.034045       \t|1              \t|  0.034045       \t|  0.0080721      \t|\n",
      "get_sanity_check_batch             \t|  0.0064563      \t|3              \t|  0.019369       \t|  0.0045924      \t|\n",
      "fetch_next_sanity_check_batch      \t|  0.0064196      \t|3              \t|  0.019259       \t|  0.0045663      \t|\n",
      "on_test_end                        \t|  0.0017795      \t|4              \t|  0.0071179      \t|  0.0016877      \t|\n",
      "on_pretrain_routine_start          \t|  0.0055585      \t|1              \t|  0.0055585      \t|  0.0013179      \t|\n",
      "on_validation_model_eval           \t|  0.00027574     \t|16             \t|  0.0044118      \t|  0.0010461      \t|\n",
      "on_train_end                       \t|  0.0012372      \t|1              \t|  0.0012372      \t|  0.00029335     \t|\n",
      "on_epoch_start                     \t|  3.4778e-05     \t|35             \t|  0.0012172      \t|  0.00028861     \t|\n",
      "on_epoch_end                       \t|  3.2773e-05     \t|35             \t|  0.0011471      \t|  0.00027197     \t|\n",
      "on_test_model_eval                 \t|  0.00021448     \t|4              \t|  0.00085791     \t|  0.00020341     \t|\n",
      "on_validation_epoch_end            \t|  4.788e-05      \t|16             \t|  0.00076608     \t|  0.00018164     \t|\n",
      "on_validation_epoch_start          \t|  3.1486e-05     \t|16             \t|  0.00050377     \t|  0.00011945     \t|\n",
      "configure_optimizers               \t|  0.00027334     \t|1              \t|  0.00027334     \t|  6.4811e-05     \t|\n",
      "on_configure_sharded_model         \t|  4.8502e-05     \t|5              \t|  0.00024251     \t|  5.75e-05       \t|\n",
      "teardown                           \t|  4.0765e-05     \t|5              \t|  0.00020383     \t|  4.8328e-05     \t|\n",
      "on_test_epoch_end                  \t|  4.7728e-05     \t|4              \t|  0.00019091     \t|  4.5266e-05     \t|\n",
      "on_before_accelerator_backend_setup\t|  3.8085e-05     \t|5              \t|  0.00019042     \t|  4.515e-05      \t|\n",
      "setup                              \t|  3.4354e-05     \t|5              \t|  0.00017177     \t|  4.0728e-05     \t|\n",
      "on_test_epoch_start                \t|  3.0808e-05     \t|4              \t|  0.00012323     \t|  2.9219e-05     \t|\n",
      "configure_sharded_model            \t|  1.4956e-05     \t|5              \t|  7.4778e-05     \t|  1.773e-05      \t|\n",
      "configure_callbacks                \t|  1.4231e-05     \t|5              \t|  7.1157e-05     \t|  1.6872e-05     \t|\n",
      "on_test_dataloader                 \t|  1.6113e-05     \t|4              \t|  6.4451e-05     \t|  1.5282e-05     \t|\n",
      "on_sanity_check_end                \t|  6.0066e-05     \t|1              \t|  6.0066e-05     \t|  1.4242e-05     \t|\n",
      "on_fit_end                         \t|  5.3426e-05     \t|1              \t|  5.3426e-05     \t|  1.2668e-05     \t|\n",
      "prepare_data                       \t|  8.9305e-06     \t|5              \t|  4.4652e-05     \t|  1.0587e-05     \t|\n",
      "on_fit_start                       \t|  4.3138e-05     \t|1              \t|  4.3138e-05     \t|  1.0228e-05     \t|\n",
      "on_pretrain_routine_end            \t|  3.9236e-05     \t|1              \t|  3.9236e-05     \t|  9.3029e-06     \t|\n",
      "on_train_dataloader                \t|  1.6958e-05     \t|1              \t|  1.6958e-05     \t|  4.0207e-06     \t|\n",
      "on_val_dataloader                  \t|  9.2648e-06     \t|1              \t|  9.2648e-06     \t|  2.1967e-06     \t|\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.987500011920929, 'test_loss': 0.047178059816360474}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "result = trainer.test(model,data_mnist.test_dataloader(),ckpt_path='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c48b956b",
   "metadata": {},
   "source": [
    "```\n",
    "--------------------------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.987500011920929, 'test_loss': 0.047178059816360474}\n",
    "--------------------------------------------------------------------------------\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33518228",
   "metadata": {},
   "source": [
    "### 5，使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "41390bb6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-13.0112,  -2.8257,  -1.8588,  -3.6137,  -0.3307,  -5.4953, -19.7282,\n",
      "          15.9651,  -8.0379,  -2.2925],\n",
      "        [ -6.0261,  -2.5480,  13.4140,  -5.5701, -10.2049,  -6.4469,  -3.7119,\n",
      "          -6.0732,  -6.0826,  -7.7339],\n",
      "        [ -5.1793,   8.9197,  -7.2543, -13.2983,  -5.0533,  -6.4719,  -3.7550,\n",
      "          -6.6116,  -6.1681, -11.3888],\n",
      "        [ 14.4084,  -9.3049,  -0.2826,  -5.9594, -10.0195,  -1.9158,   3.8159,\n",
      "         -10.0758,  -1.9282,  -5.2118],\n",
      "        [ -9.4577,  -4.3220, -10.3566, -15.9498,  11.8866,  -7.0987,  -7.1806,\n",
      "          -1.5529,  -6.9066,   2.5904],\n",
      "        [ -8.8615,  11.8241,  -8.4843, -16.3067,  -5.3663,  -7.3302,  -6.5410,\n",
      "          -3.8980,  -8.0192, -10.7459],\n",
      "        [ -7.5349,  -2.6864,  -8.1347, -11.1265,   8.3502,  -2.3806,  -5.2649,\n",
      "          -3.3453,  -2.2118,  -2.6089],\n",
      "        [ -7.0058,  -9.8076,  -1.7974,  -4.9063,  -0.0970,  -2.0131,  -4.3724,\n",
      "          -6.8718,   1.4275,   8.6853],\n",
      "        [ -3.9027,  -9.7034,   0.1843,  -0.4189,  -6.5177,   5.7285,  -0.9670,\n",
      "          -8.5215,  -1.2326,  -4.2621],\n",
      "        [ -5.9817, -12.8820,  -2.9370, -11.9836,  -1.7086,  -1.5813, -14.7753,\n",
      "           0.4480,   1.3776,  11.3914],\n",
      "        [ 16.8340,  -8.0244,  -0.7741,  -5.0373, -11.2885,  -3.0471,   1.0007,\n",
      "          -7.0407,  -1.2350,  -6.8061],\n",
      "        [  6.8250, -12.7335,   0.1518,  -7.4281, -12.0778,   0.2939,  20.9634,\n",
      "         -20.7821,   3.9082,  -9.5157],\n",
      "        [ -8.0805, -13.0810,  -4.6054, -10.2063,  -0.9193,  -4.6293, -11.5931,\n",
      "          -4.6349,   2.1769,  15.3276],\n",
      "        [ 13.7328,  -3.8627,  -4.0112,  -7.9956,  -9.7883,  -6.5610,  -2.8990,\n",
      "          -4.4617,  -3.1956,  -2.7883],\n",
      "        [ -3.1513,  11.1374,  -5.1134,  -5.2249,  -6.0831,  -3.9673,  -7.4346,\n",
      "          -4.5471,  -4.4011,  -8.6621],\n",
      "        [ -9.6877, -15.3236,  -1.2526,   4.5638,  -7.1031,  15.3295,  -7.1941,\n",
      "          -2.5545,  -3.3236,  -7.0454],\n",
      "        [ -1.3276, -12.8454,  -3.9938, -10.3719,  -2.7715,  -5.6029,  -6.2417,\n",
      "          -3.1272,   2.6297,  10.1432],\n",
      "        [ -5.7811,  -8.4391,  -0.4687,  -3.7991,  -3.6599,  -1.2870, -19.4970,\n",
      "          16.7608,  -9.3972,  -4.0588],\n",
      "        [ -5.3566,  -7.7389,   5.9662,  12.1440,  -8.3407,   1.1119,  -6.9031,\n",
      "          -6.4835,   4.3029,  -1.3591],\n",
      "        [ -9.5364,  -4.2172, -10.2173, -13.0270,  15.0092,  -2.6132,  -5.0676,\n",
      "          -2.7529,  -4.0657,  -4.2093],\n",
      "        [-15.8513,  -3.5621,  -6.6063,  -6.6193,   0.8588,  -3.8259, -14.1176,\n",
      "           0.7407,  -4.0745,  11.5699],\n",
      "        [ -0.8086, -11.0222,  -1.0758,  -8.4852, -12.8985,   7.2650,  19.9413,\n",
      "         -17.6839,   2.1309, -10.0899],\n",
      "        [  0.2642, -14.5380,  -6.8190, -11.9187,  -5.3507,   1.6481,  14.2073,\n",
      "         -18.0123,   4.7929,  -5.3580],\n",
      "        [ -9.0779, -15.1762,  -9.6641,   1.0906, -12.2709,  20.9257,  -4.5509,\n",
      "         -10.3599,  -1.9897,  -4.2746],\n",
      "        [ -6.6996,  -4.9878,  -7.9854, -11.2790,  10.4383,  -5.9630,  -4.8909,\n",
      "          -3.5206,  -3.8640,   0.2913],\n",
      "        [ 17.2738, -11.7963,  -1.6868,  -9.3057,  -9.3189,  -5.6459,   0.5775,\n",
      "          -8.4094,  -0.2450,  -3.6427],\n",
      "        [ -4.8818,  -8.1715,   2.8342,   1.0672,  -4.5164,  -1.5684, -11.8290,\n",
      "           9.5152,  -4.6545,  -3.0746],\n",
      "        [ -6.3762,  -4.2479,  -9.9438, -16.4484,  16.2417,  -4.5141,  -7.3567,\n",
      "          -3.2876,  -5.0833,  -2.4909],\n",
      "        [ 16.5702,  -4.3689,  -0.7699,  -5.4350, -10.6765,  -6.3137,  -1.3944,\n",
      "          -7.6100,  -1.2885,  -4.8424],\n",
      "        [ -2.9474,  10.0741,  -4.6694,  -6.3517,  -3.6757,  -3.3109,  -8.0018,\n",
      "          -3.7477,  -3.9049,  -7.0626],\n",
      "        [-16.7028,  -4.9060,   0.4400,  24.4337, -12.8793,   1.5085, -17.9232,\n",
      "          -3.0839,   0.5491,   1.9846],\n",
      "        [ -5.0909,  10.1805,  -8.2528,  -9.2240,  -1.8044,  -4.0296,  -8.2297,\n",
      "          -3.1828,  -5.9361,  -4.8410]], grad_fn=<AddmmBackward0>)\n"
     ]
    }
   ],
   "source": [
    "data,label = next(iter(data_module.test_dataloader()))\n",
    "model.eval()\n",
    "prediction = model(data)\n",
    "print(prediction)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763bbd77",
   "metadata": {},
   "source": [
    "```\n",
    "tensor([[-13.0112,  -2.8257,  -1.8588,  -3.6137,  -0.3307,  -5.4953, -19.7282,\n",
    "          15.9651,  -8.0379,  -2.2925],\n",
    "        [ -6.0261,  -2.5480,  13.4140,  -5.5701, -10.2049,  -6.4469,  -3.7119,\n",
    "          -6.0732,  -6.0826,  -7.7339],\n",
    "          ...\n",
    "        [-16.7028,  -4.9060,   0.4400,  24.4337, -12.8793,   1.5085, -17.9232,\n",
    "          -3.0839,   0.5491,   1.9846],\n",
    "        [ -5.0909,  10.1805,  -8.2528,  -9.2240,  -1.8044,  -4.0296,  -8.2297,\n",
    "          -3.1828,  -5.9361,  -4.8410]], grad_fn=<AddmmBackward0>)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baf56e5d",
   "metadata": {},
   "source": [
    "### 6，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b28f47d",
   "metadata": {},
   "source": [
    "最优模型默认保存在 trainer.checkpoint_callback.best_model_path 的目录下，可以直接加载。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "9530ca81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_10/checkpoints/epoch=8-step=15470.ckpt\n",
      "tensor(0.0376, device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "print(trainer.checkpoint_callback.best_model_path)\n",
    "print(trainer.checkpoint_callback.best_model_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "453dfb18",
   "metadata": {},
   "source": [
    "```\n",
    "lightning_logs/version_10/checkpoints/epoch=8-step=15470.ckpt\n",
    "tensor(0.0376, device='cuda:0')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "ac0969d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6eafdcd1b49848558014cbcfa1d771be",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9887999892234802, 'test_loss': 0.03627564385533333}\n",
      "--------------------------------------------------------------------------------\n",
      "[{'test_acc': 0.9887999892234802, 'test_loss': 0.03627564385533333}]\n"
     ]
    }
   ],
   "source": [
    "model_clone = Model.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)\n",
    "trainer_clone = pl.Trainer(max_epochs=3,gpus=1) \n",
    "result = trainer_clone.test(model_clone,data_module.test_dataloader())\n",
    "print(result)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b47eff3d",
   "metadata": {},
   "source": [
    "```\n",
    "--------------------------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9887999892234802, 'test_loss': 0.03627564385533333}\n",
    "--------------------------------------------------------------------------------\n",
    "[{'test_acc': 0.9887999892234802, 'test_loss': 0.03627564385533333}]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fb65bc3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fb6e1ead",
   "metadata": {},
   "source": [
    "## 三，训练加速技巧"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20affb65",
   "metadata": {},
   "source": [
    "下面重点介绍pytorch_lightning 模型训练加速的一些技巧。\n",
    "\n",
    "* 1，使用多进程读取数据(num_workers=4)\n",
    "\n",
    "* 2，使用锁业内存(pin_memory=True)\n",
    "\n",
    "* 3，使用加速器(gpus=4,strategy=\"ddp_find_unused_parameters_false\")\n",
    "\n",
    "* 4，使用梯度累加(accumulate_grad_batches=6)\n",
    "\n",
    "* 5，使用半精度(precision=16,batch_size=`2*batch_size`)\n",
    "\n",
    "* 6，自动搜索最大batch_size(auto_scale_batch_size='binsearch') \n",
    "\n",
    "（注：过大的batch_size对模型学习是有害的。）\n",
    "\n",
    "详细原理，可以参考：\n",
    "\n",
    "https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html\n",
    "\n",
    "我们将训练代码封装成如下脚本形式，方便后面测试使用。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "3921bcbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mnist_cnn.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile mnist_cnn.py\n",
    "import torch \n",
    "from torch import nn \n",
    "from argparse import ArgumentParser\n",
    "\n",
    "import torchvision \n",
    "from torchvision import transforms as T\n",
    "from torchvision.datasets import MNIST\n",
    "from torch.utils.data import DataLoader,random_split\n",
    "import pytorch_lightning as pl\n",
    "from torchmetrics import Accuracy\n",
    "\n",
    "#================================================================================\n",
    "# 一，准备数据\n",
    "#================================================================================\n",
    "\n",
    "class MNISTDataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_dir: str = \"./minist/\", \n",
    "                 batch_size: int = 32,\n",
    "                 num_workers: int =4,\n",
    "                 pin_memory:bool =True):\n",
    "        super().__init__()\n",
    "        self.data_dir = data_dir\n",
    "        self.batch_size = batch_size\n",
    "        self.num_workers = num_workers\n",
    "        self.pin_memory = pin_memory\n",
    "\n",
    "    def setup(self, stage = None):\n",
    "        transform = T.Compose([T.ToTensor()])\n",
    "        self.ds_test = MNIST(self.data_dir, download=True,train=False,transform=transform)\n",
    "        self.ds_predict = MNIST(self.data_dir, download=True, train=False,transform=transform)\n",
    "        ds_full = MNIST(self.data_dir, download=True, train=True,transform=transform)\n",
    "        self.ds_train, self.ds_val = random_split(ds_full, [55000, 5000])\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        return DataLoader(self.ds_train, batch_size=self.batch_size,\n",
    "                          shuffle=True, num_workers=self.num_workers,\n",
    "                          pin_memory=self.pin_memory)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return DataLoader(self.ds_val, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=self.pin_memory)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return DataLoader(self.ds_test, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=self.pin_memory)\n",
    "\n",
    "    def predict_dataloader(self):\n",
    "        return DataLoader(self.ds_predict, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=self.pin_memory)\n",
    "    \n",
    "    @staticmethod\n",
    "    def add_dataset_args(parent_parser):\n",
    "        parser = ArgumentParser(parents=[parent_parser], add_help=False)\n",
    "        parser.add_argument('--batch_size', type=int, default=32)\n",
    "        parser.add_argument('--num_workers', type=int, default=4)\n",
    "        parser.add_argument('--pin_memory', type=bool, default=True)\n",
    "        return parser\n",
    "\n",
    "#================================================================================\n",
    "# 二，定义模型\n",
    "#================================================================================\n",
    "\n",
    "net = nn.Sequential(\n",
    "    nn.Conv2d(in_channels=1,out_channels=32,kernel_size = 3),\n",
    "    nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "    nn.Conv2d(in_channels=32,out_channels=64,kernel_size = 5),\n",
    "    nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "    nn.Dropout2d(p = 0.1),\n",
    "    nn.AdaptiveMaxPool2d((1,1)),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(64,32),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(32,10)\n",
    ")\n",
    "\n",
    "class Model(pl.LightningModule):\n",
    "    \n",
    "    def __init__(self,net,learning_rate=1e-3):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        self.net = net\n",
    "        self.train_acc = Accuracy()\n",
    "        self.val_acc = Accuracy()\n",
    "        self.test_acc = Accuracy() \n",
    "        \n",
    "        \n",
    "    def forward(self,x):\n",
    "        x = self.net(x)\n",
    "        return x\n",
    "    \n",
    "    \n",
    "    #定义loss\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "    \n",
    "    #定义各种metrics\n",
    "    def training_step_end(self,outputs):\n",
    "        train_acc = self.train_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"train_acc\",train_acc,prog_bar=True)\n",
    "        return {\"loss\":outputs[\"loss\"].mean()}\n",
    "    \n",
    "    #定义optimizer,以及可选的lr_scheduler\n",
    "    def configure_optimizers(self):\n",
    "        return torch.optim.Adam(self.parameters(), lr=self.hparams.learning_rate)\n",
    "    \n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "\n",
    "    def validation_step_end(self,outputs):\n",
    "        val_acc = self.val_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"val_loss\",outputs[\"loss\"].mean(),on_epoch=True,on_step=False)\n",
    "        self.log(\"val_acc\",val_acc,prog_bar=True,on_epoch=True,on_step=False)\n",
    "    \n",
    "    def test_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "    \n",
    "    def test_step_end(self,outputs):\n",
    "        test_acc = self.test_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"test_acc\",test_acc,on_epoch=True,on_step=False)\n",
    "        self.log(\"test_loss\",outputs[\"loss\"].mean(),on_epoch=True,on_step=False)\n",
    "    \n",
    "    @staticmethod\n",
    "    def add_model_args(parent_parser):\n",
    "        parser = ArgumentParser(parents=[parent_parser], add_help=False)\n",
    "        parser.add_argument('--learning_rate', type=float, default=1e-3)\n",
    "        return parser\n",
    "    \n",
    "\n",
    "#================================================================================\n",
    "# 三，训练模型\n",
    "#================================================================================\n",
    "    \n",
    "def main(hparams):\n",
    "    pl.seed_everything(1234)\n",
    "    \n",
    "    data_mnist = MNISTDataModule(batch_size=hparams.batch_size,\n",
    "                                 num_workers=hparams.num_workers)\n",
    "    \n",
    "    model = Model(net,learning_rate=hparams.learning_rate)\n",
    "    \n",
    "    ckpt_callback = pl.callbacks.ModelCheckpoint(\n",
    "        monitor='val_loss',\n",
    "        save_top_k=1,\n",
    "        mode='min'\n",
    "    )\n",
    "    early_stopping = pl.callbacks.EarlyStopping(monitor = 'val_loss',\n",
    "                   patience=3,\n",
    "                   mode = 'min')\n",
    "    \n",
    "    trainer = pl.Trainer.from_argparse_args( \n",
    "        hparams,\n",
    "        max_epochs=10,\n",
    "        \n",
    "        callbacks = [ckpt_callback,early_stopping]\n",
    "    ) \n",
    "\n",
    "    \n",
    "    \n",
    "    if hparams.auto_scale_batch_size is not None:\n",
    "        #搜索不发生OOM的最大batch_size\n",
    "        max_batch_size = trainer.tuner.scale_batch_size(model,data_mnist,\n",
    "                        mode=hparams.auto_scale_batch_size)\n",
    "        data_mnist.batch_size = max_batch_size\n",
    "        \n",
    "        #等价于\n",
    "        #trainer.tune(model,data_mnist)\n",
    "        \n",
    "    \n",
    "    #gpus=0, #单CPU模式\n",
    "    #gpus=1, #单GPU模式\n",
    "    #num_processes=4,strategy=\"ddp_find_unused_parameters_false\", #多CPU(进程)模式\n",
    "    #gpus=4,strategy=\"dp\", #多GPU(dp速度提升效果一般)\n",
    "    #gpus=4,strategy=“ddp_find_unused_parameters_false\" #多GPU(ddp速度提升效果好)\n",
    "\n",
    "    trainer.fit(model,data_mnist)\n",
    "    result = trainer.test(model,data_mnist,ckpt_path='best')\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    parser = ArgumentParser()\n",
    "    parser = MNISTDataModule.add_dataset_args(parser)\n",
    "    parser = Model.add_model_args(parser)\n",
    "    parser = pl.Trainer.add_argparse_args(parser)\n",
    "    hparams = parser.parse_args()\n",
    "    main(hparams)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "842109db-14fe-4d31-bd88-eb8cf463309e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f9ff1897",
   "metadata": {},
   "source": [
    "### 1，使用多进程读取数据(num_workers=4)\n",
    "\n",
    "使用多进程读取数据，可以避免数据加载过程成为性能瓶颈。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c192aa5",
   "metadata": {},
   "source": [
    "* 单进程读取数据(num_workers=0, gpus=1): 1min 18s\n",
    "\n",
    "* 多进程读取数据(num_workers=4, gpus=1): 59.7s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "fa6d4362",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Validation sanity check: 0it [00:00, ?it/s]/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 64 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  f\"The dataloader, {name}, does not have many workers which may be a bottleneck.\"\n",
      "Global seed set to 1234                                                         \n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 64 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  f\"The dataloader, {name}, does not have many workers which may be a bottleneck.\"\n",
      "Epoch 0:  92%|▉| 1719/1876 [00:21<00:01, 80.92it/s, loss=0.131, v_num=7, train_a\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Epoch 0:  92%|▉| 1724/1876 [00:21<00:01, 81.00it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  93%|▉| 1736/1876 [00:21<00:01, 81.18it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  93%|▉| 1749/1876 [00:21<00:01, 81.38it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  94%|▉| 1762/1876 [00:21<00:01, 81.56it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  95%|▉| 1775/1876 [00:21<00:01, 81.75it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  95%|▉| 1788/1876 [00:21<00:01, 81.93it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  96%|▉| 1801/1876 [00:21<00:00, 82.09it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  97%|▉| 1814/1876 [00:22<00:00, 82.26it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  97%|▉| 1827/1876 [00:22<00:00, 82.42it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0:  98%|▉| 1840/1876 [00:22<00:00, 82.60it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Validating:  77%|█████████████████████▌      | 121/157 [00:01<00:00, 116.50it/s]\u001b[A\n",
      "Epoch 0:  99%|▉| 1853/1876 [00:22<00:00, 82.78it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 0: 100%|█| 1876/1876 [00:22<00:00, 83.07it/s, loss=0.131, v_num=7, train_a\u001b[A\n",
      "Epoch 1:  92%|▉| 1719/1876 [00:22<00:02, 78.06it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Epoch 1:  92%|▉| 1729/1876 [00:22<00:01, 78.22it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  93%|▉| 1742/1876 [00:22<00:01, 78.44it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  94%|▉| 1755/1876 [00:22<00:01, 78.66it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  94%|▉| 1768/1876 [00:22<00:01, 78.88it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  95%|▉| 1781/1876 [00:22<00:01, 79.10it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  96%|▉| 1795/1876 [00:22<00:01, 79.36it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  96%|▉| 1809/1876 [00:22<00:00, 79.60it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  97%|▉| 1823/1876 [00:22<00:00, 79.82it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  98%|▉| 1837/1876 [00:22<00:00, 80.03it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Validating:  75%|█████████████████████       | 118/157 [00:00<00:00, 126.17it/s]\u001b[A\n",
      "Epoch 1:  99%|▉| 1851/1876 [00:23<00:00, 80.22it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1:  99%|▉| 1865/1876 [00:23<00:00, 80.41it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 1: 100%|█| 1876/1876 [00:23<00:00, 80.54it/s, loss=0.0759, v_num=7, train_\u001b[A\n",
      "Epoch 2:  92%|▉| 1719/1876 [00:21<00:01, 79.40it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Epoch 2:  92%|▉| 1722/1876 [00:21<00:01, 79.44it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  93%|▉| 1736/1876 [00:21<00:01, 79.65it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  93%|▉| 1750/1876 [00:21<00:01, 79.86it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  94%|▉| 1764/1876 [00:22<00:01, 80.08it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  95%|▉| 1778/1876 [00:22<00:01, 80.30it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  96%|▉| 1792/1876 [00:22<00:01, 80.50it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  96%|▉| 1806/1876 [00:22<00:00, 80.71it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  97%|▉| 1820/1876 [00:22<00:00, 80.92it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Validating:  64%|██████████████████          | 101/157 [00:00<00:00, 120.94it/s]\u001b[A\n",
      "Epoch 2:  98%|▉| 1834/1876 [00:22<00:00, 81.14it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  99%|▉| 1848/1876 [00:22<00:00, 81.35it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2:  99%|▉| 1862/1876 [00:22<00:00, 81.57it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:22<00:00, 81.76it/s, loss=0.0629, v_num=7, train_\u001b[A\n",
      "Epoch 3:  92%|▉| 1719/1876 [00:21<00:01, 80.51it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Epoch 3:  92%|▉| 1722/1876 [00:21<00:01, 80.55it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  93%|▉| 1736/1876 [00:21<00:01, 80.76it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  93%|▉| 1750/1876 [00:21<00:01, 80.98it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  94%|▉| 1764/1876 [00:21<00:01, 81.19it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  95%|▉| 1778/1876 [00:21<00:01, 81.41it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  96%|▉| 1792/1876 [00:21<00:01, 81.61it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  96%|▉| 1806/1876 [00:22<00:00, 81.81it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  97%|▉| 1820/1876 [00:22<00:00, 82.01it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Validating:  64%|██████████████████          | 101/157 [00:00<00:00, 120.40it/s]\u001b[A\n",
      "Epoch 3:  98%|▉| 1834/1876 [00:22<00:00, 82.22it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  99%|▉| 1848/1876 [00:22<00:00, 82.43it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3:  99%|▉| 1862/1876 [00:22<00:00, 82.65it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 3: 100%|█| 1876/1876 [00:22<00:00, 82.85it/s, loss=0.0536, v_num=7, train_\u001b[A\n",
      "Epoch 4:  92%|▉| 1719/1876 [00:21<00:01, 79.62it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Epoch 4:  92%|▉| 1722/1876 [00:21<00:01, 79.67it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  93%|▉| 1736/1876 [00:21<00:01, 79.89it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  93%|▉| 1750/1876 [00:21<00:01, 80.10it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  94%|▉| 1764/1876 [00:21<00:01, 80.31it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  95%|▉| 1778/1876 [00:22<00:01, 80.53it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  96%|▉| 1792/1876 [00:22<00:01, 80.74it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  96%|▉| 1806/1876 [00:22<00:00, 80.93it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Validating:  55%|████████████████             | 87/157 [00:00<00:00, 118.58it/s]\u001b[A\n",
      "Epoch 4:  97%|▉| 1820/1876 [00:22<00:00, 81.12it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  98%|▉| 1834/1876 [00:22<00:00, 81.31it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  99%|▉| 1848/1876 [00:22<00:00, 81.51it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4:  99%|▉| 1862/1876 [00:22<00:00, 81.72it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 4: 100%|█| 1876/1876 [00:22<00:00, 81.91it/s, loss=0.0731, v_num=7, train_\u001b[A\n",
      "Epoch 5:  92%|▉| 1719/1876 [00:21<00:01, 79.49it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Epoch 5:  92%|▉| 1722/1876 [00:21<00:01, 79.55it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  93%|▉| 1736/1876 [00:21<00:01, 79.79it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  93%|▉| 1750/1876 [00:21<00:01, 80.02it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  94%|▉| 1764/1876 [00:21<00:01, 80.26it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  95%|▉| 1778/1876 [00:22<00:01, 80.48it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  96%|▉| 1792/1876 [00:22<00:01, 80.70it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  96%|▉| 1806/1876 [00:22<00:00, 80.91it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  97%|▉| 1820/1876 [00:22<00:00, 81.12it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  98%|▉| 1834/1876 [00:22<00:00, 81.32it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  99%|▉| 1848/1876 [00:22<00:00, 81.53it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 5:  99%|▉| 1862/1876 [00:22<00:00, 81.71it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Validating:  91%|█████████████████████████▌  | 143/157 [00:01<00:00, 120.06it/s]\u001b[A\n",
      "Epoch 5: 100%|█| 1876/1876 [00:22<00:00, 81.90it/s, loss=0.0458, v_num=7, train_\u001b[A\n",
      "Epoch 6:  92%|▉| 1719/1876 [00:21<00:01, 79.39it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Epoch 6:  92%|▉| 1722/1876 [00:21<00:01, 79.44it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  93%|▉| 1736/1876 [00:21<00:01, 79.72it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  93%|▉| 1750/1876 [00:21<00:01, 79.97it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  94%|▉| 1764/1876 [00:21<00:01, 80.23it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  95%|▉| 1778/1876 [00:22<00:01, 80.48it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  96%|▉| 1792/1876 [00:22<00:01, 80.71it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  96%|▉| 1806/1876 [00:22<00:00, 80.95it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  97%|▉| 1820/1876 [00:22<00:00, 81.18it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  98%|▉| 1834/1876 [00:22<00:00, 81.43it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  99%|▉| 1849/1876 [00:22<00:00, 81.72it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6:  99%|▉| 1864/1876 [00:22<00:00, 81.98it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6: 100%|█| 1876/1876 [00:22<00:00, 82.15it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "Epoch 6: 100%|█| 1876/1876 [00:22<00:00, 82.14it/s, loss=0.0726, v_num=7, train_\u001b[A\n",
      "FIT Profiler Report\n",
      "\n",
      "Action                             \t|  Mean duration (s)\t|Num calls      \t|  Total time (s) \t|  Percentage %   \t|\n",
      "--------------------------------------------------------------------------------------------------------------------------------------\n",
      "Total                              \t|  -              \t|_              \t|  164.47         \t|  100 %          \t|\n",
      "--------------------------------------------------------------------------------------------------------------------------------------\n",
      "run_training_epoch                 \t|  22.878         \t|7              \t|  160.15         \t|  97.368         \t|\n",
      "run_training_batch                 \t|  0.0063689      \t|12033          \t|  76.637         \t|  46.595         \t|\n",
      "optimizer_step_with_closure_0      \t|  0.0060027      \t|12033          \t|  72.23          \t|  43.916         \t|\n",
      "training_step_and_backward         \t|  0.005106       \t|12033          \t|  61.44          \t|  37.355         \t|\n",
      "get_train_batch                    \t|  0.0040925      \t|12040          \t|  49.274         \t|  29.958         \t|\n",
      "fetch_next_train_batch             \t|  0.0040657      \t|12040          \t|  48.951         \t|  29.762         \t|\n",
      "model_forward                      \t|  0.0037554      \t|12033          \t|  45.189         \t|  27.475         \t|\n",
      "training_step_end                  \t|  0.0025311      \t|12033          \t|  30.457         \t|  18.518         \t|\n",
      "backward                           \t|  0.0011116      \t|12033          \t|  13.375         \t|  8.1321         \t|\n",
      "training_step                      \t|  0.0010423      \t|12033          \t|  12.543         \t|  7.6258         \t|\n",
      "on_train_batch_end                 \t|  0.0008786      \t|12033          \t|  10.572         \t|  6.4279         \t|\n",
      "evaluation_step_and_end            \t|  0.0048366      \t|1101           \t|  5.3251         \t|  3.2376         \t|\n",
      "get_validate_batch                 \t|  0.0038521      \t|1106           \t|  4.2605         \t|  2.5903         \t|\n",
      "fetch_next_validate_batch          \t|  0.0038265      \t|1106           \t|  4.2321         \t|  2.5731         \t|\n",
      "on_train_batch_start               \t|  0.00027346     \t|12033          \t|  3.2906         \t|  2.0007         \t|\n",
      "validation_step_end                \t|  0.0026181      \t|1101           \t|  2.8825         \t|  1.7525         \t|\n",
      "zero_grad                          \t|  0.00020552     \t|12033          \t|  2.473          \t|  1.5036         \t|\n",
      "validation_step                    \t|  0.0021413      \t|1101           \t|  2.3576         \t|  1.4334         \t|\n",
      "training_batch_to_device           \t|  0.00019259     \t|12033          \t|  2.3174         \t|  1.409          \t|\n",
      "on_batch_start                     \t|  3.9314e-05     \t|12033          \t|  0.47306        \t|  0.28762        \t|\n",
      "on_batch_end                       \t|  3.4313e-05     \t|12033          \t|  0.41289        \t|  0.25104        \t|\n",
      "on_before_zero_grad                \t|  3.3974e-05     \t|12033          \t|  0.40881        \t|  0.24856        \t|\n",
      "on_after_backward                  \t|  3.2071e-05     \t|12033          \t|  0.38591        \t|  0.23463        \t|\n",
      "on_before_backward                 \t|  2.9808e-05     \t|12033          \t|  0.35868        \t|  0.21808        \t|\n",
      "on_before_optimizer_step           \t|  2.8173e-05     \t|12033          \t|  0.339          \t|  0.20611        \t|\n",
      "evaluation_batch_to_device         \t|  0.0002006      \t|1101           \t|  0.22086        \t|  0.13428        \t|\n",
      "on_validation_batch_end            \t|  0.00011326     \t|1101           \t|  0.12469        \t|  0.075814       \t|\n",
      "on_validation_batch_start          \t|  3.4017e-05     \t|1101           \t|  0.037453       \t|  0.022771       \t|\n",
      "on_train_epoch_end                 \t|  0.0039483      \t|7              \t|  0.027638       \t|  0.016804       \t|\n",
      "get_sanity_check_batch             \t|  0.0047596      \t|3              \t|  0.014279       \t|  0.0086815      \t|\n",
      "fetch_next_sanity_check_batch      \t|  0.0047342      \t|3              \t|  0.014203       \t|  0.0086352      \t|\n",
      "on_validation_end                  \t|  0.00073308     \t|8              \t|  0.0058647      \t|  0.0035657      \t|\n",
      "on_validation_start                \t|  0.00046837     \t|8              \t|  0.003747       \t|  0.0022781      \t|\n",
      "on_train_epoch_start               \t|  0.00041611     \t|7              \t|  0.0029128      \t|  0.001771       \t|\n",
      "on_validation_model_eval           \t|  0.00018604     \t|8              \t|  0.0014883      \t|  0.00090488     \t|\n",
      "on_sanity_check_start              \t|  0.0014785      \t|1              \t|  0.0014785      \t|  0.00089891     \t|\n",
      "on_pretrain_routine_start          \t|  0.0013295      \t|1              \t|  0.0013295      \t|  0.00080836     \t|\n",
      "on_train_start                     \t|  0.00058921     \t|1              \t|  0.00058921     \t|  0.00035824     \t|\n",
      "on_epoch_start                     \t|  3.1013e-05     \t|15             \t|  0.00046519     \t|  0.00028284     \t|\n",
      "on_epoch_end                       \t|  2.9344e-05     \t|15             \t|  0.00044016     \t|  0.00026762     \t|\n",
      "on_train_end                       \t|  0.00038472     \t|1              \t|  0.00038472     \t|  0.00023391     \t|\n",
      "on_validation_epoch_end            \t|  3.6193e-05     \t|8              \t|  0.00028955     \t|  0.00017604     \t|\n",
      "configure_optimizers               \t|  0.00027934     \t|1              \t|  0.00027934     \t|  0.00016984     \t|\n",
      "on_validation_epoch_start          \t|  2.7588e-05     \t|8              \t|  0.0002207      \t|  0.00013419     \t|\n",
      "setup                              \t|  5.6159e-05     \t|1              \t|  5.6159e-05     \t|  3.4144e-05     \t|\n",
      "on_configure_sharded_model         \t|  4.634e-05      \t|1              \t|  4.634e-05      \t|  2.8174e-05     \t|\n",
      "on_sanity_check_end                \t|  4.2111e-05     \t|1              \t|  4.2111e-05     \t|  2.5603e-05     \t|\n",
      "on_fit_end                         \t|  4.1825e-05     \t|1              \t|  4.1825e-05     \t|  2.5429e-05     \t|\n",
      "on_fit_start                       \t|  3.6279e-05     \t|1              \t|  3.6279e-05     \t|  2.2057e-05     \t|\n",
      "on_before_accelerator_backend_setup\t|  3.3909e-05     \t|1              \t|  3.3909e-05     \t|  2.0617e-05     \t|\n",
      "teardown                           \t|  3.31e-05       \t|1              \t|  3.31e-05       \t|  2.0125e-05     \t|\n",
      "on_pretrain_routine_end            \t|  3.1208e-05     \t|1              \t|  3.1208e-05     \t|  1.8974e-05     \t|\n",
      "prepare_data                       \t|  1.2093e-05     \t|1              \t|  1.2093e-05     \t|  7.3526e-06     \t|\n",
      "configure_sharded_model            \t|  1.0941e-05     \t|1              \t|  1.0941e-05     \t|  6.6522e-06     \t|\n",
      "configure_callbacks                \t|  1.0638e-05     \t|1              \t|  1.0638e-05     \t|  6.4682e-06     \t|\n",
      "on_train_dataloader                \t|  8.137e-06      \t|1              \t|  8.137e-06      \t|  4.9472e-06     \t|\n",
      "on_val_dataloader                  \t|  6.9747e-06     \t|1              \t|  6.9747e-06     \t|  4.2406e-06     \t|\n",
      "\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, test_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 64 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  f\"The dataloader, {name}, does not have many workers which may be a bottleneck.\"\n",
      "Testing: 100%|██████████████████████████████▉| 312/313 [00:02<00:00, 124.68it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9857000112533569, 'test_loss': 0.04885349050164223}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|███████████████████████████████| 313/313 [00:02<00:00, 123.46it/s]\n",
      "TEST Profiler Report\n",
      "\n",
      "Action                             \t|  Mean duration (s)\t|Num calls      \t|  Total time (s) \t|  Percentage %   \t|\n",
      "---------------------------------------------------------------------------------------------------------------------------------------\n",
      "Total                              \t|  -              \t|_              \t|  167.18         \t|  100 %          \t|\n",
      "---------------------------------------------------------------------------------------------------------------------------------------\n",
      "run_training_epoch                 \t|  22.878         \t|7              \t|  160.15         \t|  95.792         \t|\n",
      "run_training_batch                 \t|  0.0063689      \t|12033          \t|  76.637         \t|  45.841         \t|\n",
      "optimizer_step_with_closure_0      \t|  0.0060027      \t|12033          \t|  72.23          \t|  43.205         \t|\n",
      "training_step_and_backward         \t|  0.005106       \t|12033          \t|  61.44          \t|  36.751         \t|\n",
      "get_train_batch                    \t|  0.0040925      \t|12040          \t|  49.274         \t|  29.473         \t|\n",
      "fetch_next_train_batch             \t|  0.0040657      \t|12040          \t|  48.951         \t|  29.28          \t|\n",
      "model_forward                      \t|  0.0037554      \t|12033          \t|  45.189         \t|  27.03          \t|\n",
      "training_step_end                  \t|  0.0025311      \t|12033          \t|  30.457         \t|  18.218         \t|\n",
      "backward                           \t|  0.0011116      \t|12033          \t|  13.375         \t|  8.0005         \t|\n",
      "training_step                      \t|  0.0010423      \t|12033          \t|  12.543         \t|  7.5024         \t|\n",
      "on_train_batch_end                 \t|  0.0008786      \t|12033          \t|  10.572         \t|  6.3238         \t|\n",
      "evaluation_step_and_end            \t|  0.0045467      \t|1414           \t|  6.4291         \t|  3.8456         \t|\n",
      "get_validate_batch                 \t|  0.0038521      \t|1106           \t|  4.2605         \t|  2.5484         \t|\n",
      "fetch_next_validate_batch          \t|  0.0038265      \t|1106           \t|  4.2321         \t|  2.5315         \t|\n",
      "on_train_batch_start               \t|  0.00027346     \t|12033          \t|  3.2906         \t|  1.9683         \t|\n",
      "validation_step_end                \t|  0.0026181      \t|1101           \t|  2.8825         \t|  1.7242         \t|\n",
      "run_test_evaluation                \t|  2.5374         \t|1              \t|  2.5374         \t|  1.5178         \t|\n",
      "zero_grad                          \t|  0.00020552     \t|12033          \t|  2.473          \t|  1.4792         \t|\n",
      "validation_step                    \t|  0.0021413      \t|1101           \t|  2.3576         \t|  1.4102         \t|\n",
      "training_batch_to_device           \t|  0.00019259     \t|12033          \t|  2.3174         \t|  1.3862         \t|\n",
      "get_test_batch                     \t|  0.0038072      \t|314            \t|  1.1955         \t|  0.71507        \t|\n",
      "fetch_next_test_batch              \t|  0.0037828      \t|314            \t|  1.1878         \t|  0.71049        \t|\n",
      "test_step_end                      \t|  0.002581       \t|313            \t|  0.80784        \t|  0.48322        \t|\n",
      "on_batch_start                     \t|  3.9314e-05     \t|12033          \t|  0.47306        \t|  0.28296        \t|\n",
      "on_batch_end                       \t|  3.4313e-05     \t|12033          \t|  0.41289        \t|  0.24697        \t|\n",
      "on_before_zero_grad                \t|  3.3974e-05     \t|12033          \t|  0.40881        \t|  0.24453        \t|\n",
      "on_after_backward                  \t|  3.2071e-05     \t|12033          \t|  0.38591        \t|  0.23083        \t|\n",
      "on_before_backward                 \t|  2.9808e-05     \t|12033          \t|  0.35868        \t|  0.21455        \t|\n",
      "on_before_optimizer_step           \t|  2.8173e-05     \t|12033          \t|  0.339          \t|  0.20278        \t|\n",
      "evaluation_batch_to_device         \t|  0.00019911     \t|1414           \t|  0.28153        \t|  0.1684         \t|\n",
      "test_step                          \t|  0.00087096     \t|313            \t|  0.27261        \t|  0.16306        \t|\n",
      "on_validation_batch_end            \t|  0.00011326     \t|1101           \t|  0.12469        \t|  0.074587       \t|\n",
      "on_validation_batch_start          \t|  3.4017e-05     \t|1101           \t|  0.037453       \t|  0.022403       \t|\n",
      "on_train_epoch_end                 \t|  0.0039483      \t|7              \t|  0.027638       \t|  0.016532       \t|\n",
      "on_test_batch_end                  \t|  7.6605e-05     \t|313            \t|  0.023977       \t|  0.014342       \t|\n",
      "get_sanity_check_batch             \t|  0.0047596      \t|3              \t|  0.014279       \t|  0.008541       \t|\n",
      "fetch_next_sanity_check_batch      \t|  0.0047342      \t|3              \t|  0.014203       \t|  0.0084954      \t|\n",
      "on_test_batch_start                \t|  3.3206e-05     \t|313            \t|  0.010393       \t|  0.0062169      \t|\n",
      "on_validation_end                  \t|  0.00073308     \t|8              \t|  0.0058647      \t|  0.003508       \t|\n",
      "on_validation_start                \t|  0.00046837     \t|8              \t|  0.003747       \t|  0.0022413      \t|\n",
      "on_train_epoch_start               \t|  0.00041611     \t|7              \t|  0.0029128      \t|  0.0017423      \t|\n",
      "on_validation_model_eval           \t|  0.00018604     \t|8              \t|  0.0014883      \t|  0.00089024     \t|\n",
      "on_sanity_check_start              \t|  0.0014785      \t|1              \t|  0.0014785      \t|  0.00088436     \t|\n",
      "on_pretrain_routine_start          \t|  0.0013295      \t|1              \t|  0.0013295      \t|  0.00079527     \t|\n",
      "on_train_start                     \t|  0.00058921     \t|1              \t|  0.00058921     \t|  0.00035244     \t|\n",
      "on_epoch_start                     \t|  3.1191e-05     \t|16             \t|  0.00049905     \t|  0.00029851     \t|\n",
      "on_epoch_end                       \t|  2.9256e-05     \t|16             \t|  0.00046809     \t|  0.00027999     \t|\n",
      "on_test_start                      \t|  0.00044506     \t|1              \t|  0.00044506     \t|  0.00026621     \t|\n",
      "on_train_end                       \t|  0.00038472     \t|1              \t|  0.00038472     \t|  0.00023012     \t|\n",
      "on_test_end                        \t|  0.00035959     \t|1              \t|  0.00035959     \t|  0.00021509     \t|\n",
      "on_validation_epoch_end            \t|  3.6193e-05     \t|8              \t|  0.00028955     \t|  0.00017319     \t|\n",
      "configure_optimizers               \t|  0.00027934     \t|1              \t|  0.00027934     \t|  0.00016709     \t|\n",
      "on_validation_epoch_start          \t|  2.7588e-05     \t|8              \t|  0.0002207      \t|  0.00013202     \t|\n",
      "on_test_model_eval                 \t|  0.00018527     \t|1              \t|  0.00018527     \t|  0.00011082     \t|\n",
      "setup                              \t|  5.4737e-05     \t|2              \t|  0.00010947     \t|  6.5483e-05     \t|\n",
      "on_configure_sharded_model         \t|  4.7909e-05     \t|2              \t|  9.5818e-05     \t|  5.7314e-05     \t|\n",
      "teardown                           \t|  3.4635e-05     \t|2              \t|  6.9271e-05     \t|  4.1435e-05     \t|\n",
      "on_before_accelerator_backend_setup\t|  3.3455e-05     \t|2              \t|  6.6911e-05     \t|  4.0023e-05     \t|\n",
      "on_sanity_check_end                \t|  4.2111e-05     \t|1              \t|  4.2111e-05     \t|  2.5189e-05     \t|\n",
      "on_fit_end                         \t|  4.1825e-05     \t|1              \t|  4.1825e-05     \t|  2.5018e-05     \t|\n",
      "on_fit_start                       \t|  3.6279e-05     \t|1              \t|  3.6279e-05     \t|  2.17e-05       \t|\n",
      "on_test_epoch_end                  \t|  3.3344e-05     \t|1              \t|  3.3344e-05     \t|  1.9945e-05     \t|\n",
      "on_pretrain_routine_end            \t|  3.1208e-05     \t|1              \t|  3.1208e-05     \t|  1.8667e-05     \t|\n",
      "on_test_epoch_start                \t|  3.0434e-05     \t|1              \t|  3.0434e-05     \t|  1.8204e-05     \t|\n",
      "configure_sharded_model            \t|  1.2611e-05     \t|2              \t|  2.5222e-05     \t|  1.5087e-05     \t|\n",
      "configure_callbacks                \t|  1.08e-05       \t|2              \t|  2.1599e-05     \t|  1.292e-05      \t|\n",
      "prepare_data                       \t|  9.8348e-06     \t|2              \t|  1.967e-05      \t|  1.1765e-05     \t|\n",
      "on_test_dataloader                 \t|  1.2081e-05     \t|1              \t|  1.2081e-05     \t|  7.2264e-06     \t|\n",
      "on_train_dataloader                \t|  8.137e-06      \t|1              \t|  8.137e-06      \t|  4.8672e-06     \t|\n",
      "on_val_dataloader                  \t|  6.9747e-06     \t|1              \t|  6.9747e-06     \t|  4.1719e-06     \t|\n",
      "\n",
      "CPU times: user 4.67 s, sys: 2.14 s, total: 6.81 s\n",
      "Wall time: 2min 50s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#单进程读取数据(num_workers=0)\n",
    "!python3 mnist_cnn.py --num_workers=0 --gpus=1 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecda78f5",
   "metadata": {},
   "source": [
    "```\n",
    "------------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9857000112533569, 'test_loss': 0.04885349050164223}\n",
    "--------------------------------------------------------------------------------\n",
    "\n",
    "CPU times: user 4.67 s, sys: 2.14 s, total: 6.81 s\n",
    "Wall time: 2min 50s\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23025ad8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "f6597ed3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  92%|▉| 1720/1876 [00:14<00:01, 115.08it/s, loss=0.125, v_num=63, train\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  92%|▉| 1734/1876 [00:15<00:01, 113.45it/s, loss=0.125, v_num=63, train\u001b[A\n",
      "Epoch 0:  93%|▉| 1754/1876 [00:15<00:01, 114.01it/s, loss=0.125, v_num=63, train\u001b[A\n",
      "Epoch 0:  95%|▉| 1776/1876 [00:15<00:00, 114.67it/s, loss=0.125, v_num=63, train\u001b[A\n",
      "Epoch 0:  96%|▉| 1798/1876 [00:15<00:00, 115.34it/s, loss=0.125, v_num=63, train\u001b[A\n",
      "Epoch 0:  97%|▉| 1820/1876 [00:15<00:00, 116.01it/s, loss=0.125, v_num=63, train\u001b[A\n",
      "Epoch 0:  98%|▉| 1842/1876 [00:15<00:00, 116.65it/s, loss=0.125, v_num=63, train\u001b[A\n",
      "Epoch 0:  99%|▉| 1865/1876 [00:15<00:00, 117.34it/s, loss=0.125, v_num=63, train\u001b[A\n",
      "Epoch 0: 100%|█| 1876/1876 [00:16<00:00, 117.00it/s, loss=0.125, v_num=63, train\u001b[A\n",
      "Epoch 1:  92%|▉| 1719/1876 [00:14<00:01, 115.00it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  92%|▉| 1725/1876 [00:15<00:01, 112.94it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Epoch 1:  93%|▉| 1748/1876 [00:15<00:01, 113.64it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Epoch 1:  94%|▉| 1771/1876 [00:15<00:00, 114.33it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Epoch 1:  96%|▉| 1794/1876 [00:15<00:00, 114.99it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Epoch 1:  97%|▉| 1817/1876 [00:15<00:00, 115.62it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Epoch 1:  98%|▉| 1840/1876 [00:15<00:00, 116.24it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Epoch 1:  99%|▉| 1863/1876 [00:15<00:00, 116.88it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Epoch 1: 100%|█| 1876/1876 [00:16<00:00, 116.61it/s, loss=0.0657, v_num=63, trai\u001b[A\n",
      "Epoch 2:  92%|▉| 1719/1876 [00:14<00:01, 116.38it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  92%|▉| 1725/1876 [00:15<00:01, 114.39it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Epoch 2:  93%|▉| 1748/1876 [00:15<00:01, 115.11it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Epoch 2:  94%|▉| 1771/1876 [00:15<00:00, 115.83it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Epoch 2:  96%|▉| 1794/1876 [00:15<00:00, 116.53it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Epoch 2:  97%|▉| 1817/1876 [00:15<00:00, 117.21it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Epoch 2:  98%|▉| 1840/1876 [00:15<00:00, 117.88it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Epoch 2:  99%|▉| 1863/1876 [00:15<00:00, 118.53it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:15<00:00, 118.23it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:15<00:00, 118.20it/s, loss=0.0752, v_num=63, trai\u001b[A\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "Testing:  96%|█████████████████████████████▋ | 300/313 [00:01<00:00, 200.61it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9764000177383423, 'test_loss': 0.0820135846734047}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|███████████████████████████████| 313/313 [00:01<00:00, 163.40it/s]\n",
      "CPU times: user 1.56 s, sys: 647 ms, total: 2.21 s\n",
      "Wall time: 59.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#多进程读取数据(num_workers=4)\n",
    "!python3 mnist_cnn.py --num_workers=4 --gpus=1 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50f7f6df",
   "metadata": {},
   "source": [
    "```\n",
    "---------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9764000177383423, 'test_loss': 0.0820135846734047}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|███████████████████████████████| 313/313 [00:01<00:00, 163.40it/s]\n",
    "CPU times: user 1.56 s, sys: 647 ms, total: 2.21 s\n",
    "Wall time: 59.7 s\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c8a8774",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9efdbe3f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e1670e6f",
   "metadata": {},
   "source": [
    "### 2，使用锁业内存(pin_memory=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "253ada52",
   "metadata": {},
   "source": [
    "锁页内存存放的内容在任何情况下都不会与主机的虚拟内存进行交换（注：虚拟内存就是硬盘）\n",
    "\n",
    "因此锁业内存比非锁业内存读写效率更高，copy到GPU上也更快速。\n",
    "\n",
    "当计算机的内存充足的时候，可以设置pin_memory=True。当系统卡住，或者交换内存使用过多的时候，设置pin_memory=False。\n",
    "\n",
    "因为pin_memory与电脑硬件性能有关，pytorch开发者不能确保每一个炼丹玩家都有高端设备，因此pin_memory默认为False。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b135184",
   "metadata": {},
   "source": [
    "* 非锁业内存存储数据(pin_memory=False, gpus=1): 1min\n",
    "\n",
    "* 锁业内存存储数据(pin_memory=True, gpus=1): 59.5s\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "2a2e9929",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  92%|▉| 1719/1876 [00:15<00:01, 110.93it/s, loss=0.141, v_num=64, train\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  92%|▉| 1727/1876 [00:15<00:01, 109.33it/s, loss=0.141, v_num=64, train\u001b[A\n",
      "Epoch 0:  93%|▉| 1750/1876 [00:15<00:01, 110.07it/s, loss=0.141, v_num=64, train\u001b[A\n",
      "Epoch 0:  95%|▉| 1773/1876 [00:16<00:00, 110.78it/s, loss=0.141, v_num=64, train\u001b[A\n",
      "Epoch 0:  96%|▉| 1796/1876 [00:16<00:00, 111.45it/s, loss=0.141, v_num=64, train\u001b[A\n",
      "Epoch 0:  97%|▉| 1819/1876 [00:16<00:00, 112.08it/s, loss=0.141, v_num=64, train\u001b[A\n",
      "Epoch 0:  98%|▉| 1842/1876 [00:16<00:00, 112.67it/s, loss=0.141, v_num=64, train\u001b[A\n",
      "Epoch 0:  99%|▉| 1865/1876 [00:16<00:00, 113.29it/s, loss=0.141, v_num=64, train\u001b[A\n",
      "Epoch 0: 100%|█| 1876/1876 [00:16<00:00, 112.80it/s, loss=0.141, v_num=64, train\u001b[A\n",
      "Epoch 1:  92%|▉| 1719/1876 [00:14<00:01, 114.87it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  92%|▉| 1725/1876 [00:15<00:01, 113.05it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Epoch 1:  93%|▉| 1748/1876 [00:15<00:01, 113.77it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Epoch 1:  94%|▉| 1771/1876 [00:15<00:00, 114.46it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Epoch 1:  96%|▉| 1794/1876 [00:15<00:00, 115.15it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Epoch 1:  97%|▉| 1817/1876 [00:15<00:00, 115.81it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Epoch 1:  98%|▉| 1840/1876 [00:15<00:00, 116.47it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Epoch 1:  99%|▉| 1863/1876 [00:15<00:00, 117.11it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Epoch 1: 100%|█| 1876/1876 [00:16<00:00, 116.92it/s, loss=0.0764, v_num=64, trai\u001b[A\n",
      "Epoch 2:  92%|▉| 1719/1876 [00:15<00:01, 112.30it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  92%|▉| 1725/1876 [00:15<00:01, 110.63it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Epoch 2:  93%|▉| 1748/1876 [00:15<00:01, 111.33it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Epoch 2:  94%|▉| 1771/1876 [00:15<00:00, 112.00it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Epoch 2:  96%|▉| 1794/1876 [00:15<00:00, 112.69it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Epoch 2:  97%|▉| 1817/1876 [00:16<00:00, 113.39it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Epoch 2:  98%|▉| 1840/1876 [00:16<00:00, 114.07it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Epoch 2:  99%|▉| 1863/1876 [00:16<00:00, 114.76it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:16<00:00, 114.54it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:16<00:00, 114.47it/s, loss=0.0576, v_num=64, trai\u001b[A\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "Testing:  99%|██████████████████████████████▊| 311/313 [00:01<00:00, 210.55it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9812999963760376, 'test_loss': 0.06231774762272835}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|███████████████████████████████| 313/313 [00:01<00:00, 171.69it/s]\n",
      "CPU times: user 1.59 s, sys: 619 ms, total: 2.21 s\n",
      "Wall time: 1min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#非锁业内存存储数据(pin_memory=False)\n",
    "!python3 mnist_cnn.py --pin_memory=False --gpus=1 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77ee73c7",
   "metadata": {},
   "source": [
    "```\n",
    "----------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9812999963760376, 'test_loss': 0.06231774762272835}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|███████████████████████████████| 313/313 [00:01<00:00, 171.69it/s]\n",
    "CPU times: user 1.59 s, sys: 619 ms, total: 2.21 s\n",
    "Wall time: 1min\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "749215e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  92%|▉| 1719/1876 [00:14<00:01, 116.58it/s, loss=0.13, v_num=66, train_\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  92%|▉| 1724/1876 [00:15<00:01, 114.70it/s, loss=0.13, v_num=66, train_\u001b[A\n",
      "Epoch 0:  93%|▉| 1746/1876 [00:15<00:01, 115.39it/s, loss=0.13, v_num=66, train_\u001b[A\n",
      "Epoch 0:  94%|▉| 1768/1876 [00:15<00:00, 115.98it/s, loss=0.13, v_num=66, train_\u001b[A\n",
      "Epoch 0:  95%|▉| 1790/1876 [00:15<00:00, 116.63it/s, loss=0.13, v_num=66, train_\u001b[A\n",
      "Epoch 0:  97%|▉| 1812/1876 [00:15<00:00, 117.26it/s, loss=0.13, v_num=66, train_\u001b[A\n",
      "Epoch 0:  98%|▉| 1834/1876 [00:15<00:00, 117.86it/s, loss=0.13, v_num=66, train_\u001b[A\n",
      "Epoch 0:  99%|▉| 1856/1876 [00:15<00:00, 118.46it/s, loss=0.13, v_num=66, train_\u001b[A\n",
      "Epoch 0: 100%|█| 1876/1876 [00:15<00:00, 118.31it/s, loss=0.13, v_num=66, train_\u001b[A\n",
      "Epoch 1:  92%|▉| 1719/1876 [00:14<00:01, 116.58it/s, loss=0.0697, v_num=66, trai\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  93%|▉| 1738/1876 [00:15<00:01, 115.16it/s, loss=0.0697, v_num=66, trai\u001b[A\n",
      "Epoch 1:  94%|▉| 1761/1876 [00:15<00:00, 115.90it/s, loss=0.0697, v_num=66, trai\u001b[A\n",
      "Epoch 1:  95%|▉| 1784/1876 [00:15<00:00, 116.64it/s, loss=0.0697, v_num=66, trai\u001b[A\n",
      "Epoch 1:  96%|▉| 1807/1876 [00:15<00:00, 117.37it/s, loss=0.0697, v_num=66, trai\u001b[A\n",
      "Epoch 1:  98%|▉| 1830/1876 [00:15<00:00, 118.04it/s, loss=0.0697, v_num=66, trai\u001b[A\n",
      "Epoch 1:  99%|▉| 1853/1876 [00:15<00:00, 118.75it/s, loss=0.0697, v_num=66, trai\u001b[A\n",
      "Epoch 1: 100%|█| 1876/1876 [00:15<00:00, 118.68it/s, loss=0.0697, v_num=66, trai\u001b[A\n",
      "Epoch 2:  92%|▉| 1719/1876 [00:15<00:01, 114.55it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  92%|▉| 1725/1876 [00:15<00:01, 112.75it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Epoch 2:  93%|▉| 1748/1876 [00:15<00:01, 113.46it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Epoch 2:  94%|▉| 1771/1876 [00:15<00:00, 114.14it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Epoch 2:  96%|▉| 1794/1876 [00:15<00:00, 114.83it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Epoch 2:  97%|▉| 1817/1876 [00:15<00:00, 115.55it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Epoch 2:  98%|▉| 1840/1876 [00:15<00:00, 116.26it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Epoch 2:  99%|▉| 1863/1876 [00:15<00:00, 116.95it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:16<00:00, 116.66it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:16<00:00, 116.64it/s, loss=0.0864, v_num=66, trai\u001b[A\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "Testing:  94%|█████████████████████████████  | 294/313 [00:01<00:00, 224.25it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9757999777793884, 'test_loss': 0.08017424494028091}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|███████████████████████████████| 313/313 [00:01<00:00, 174.58it/s]\n",
      "CPU times: user 1.54 s, sys: 677 ms, total: 2.22 s\n",
      "Wall time: 59.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#锁业内存存储数据(pin_memory=True)\n",
    "!python3 mnist_cnn.py --pin_memory=True --gpus=1 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5d9dd32",
   "metadata": {},
   "source": [
    "```\n",
    "---------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9757999777793884, 'test_loss': 0.08017424494028091}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|███████████████████████████████| 313/313 [00:01<00:00, 174.58it/s]\n",
    "CPU times: user 1.54 s, sys: 677 ms, total: 2.22 s\n",
    "Wall time: 59.5 s\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26455e53",
   "metadata": {},
   "source": [
    "### 3，使用加速器(gpus=4,strategy=\"ddp_find_unused_parameters_false\")\n",
    "\n",
    "pl 可以很方便地应用单CPU、多CPU、单GPU、多GPU乃至多TPU上训练模型。\n",
    "\n",
    "以下几种情况训练耗时统计如下：\n",
    "\n",
    "* 单CPU: 2min 17s\n",
    "\n",
    "* 单GPU:  59.4 s\n",
    "\n",
    "* 4个GPU(dp模式): 1min\n",
    "\n",
    "* 4个GPU(ddp模式): 38.9 s\n",
    "\n",
    "一般情况下，如果是单机多卡，建议使用 ddp模式，因为dp模式需要非常多的data和model传输，非常耗时。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "bd7728f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py:1585: UserWarning: GPU available but not used. Set the gpus flag in your trainer `Trainer(gpus=1)` or script `--gpus=1`.\n",
      "  \"GPU available but not used. Set the gpus flag in your trainer `Trainer(gpus=1)` or script `--gpus=1`.\"\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  92%|▉| 1719/1876 [00:39<00:03, 43.06it/s, loss=0.137, v_num=43, train_\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  92%|▉| 1726/1876 [00:40<00:03, 42.89it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  92%|▉| 1734/1876 [00:40<00:03, 42.98it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  93%|▉| 1745/1876 [00:40<00:03, 43.14it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  94%|▉| 1756/1876 [00:40<00:02, 43.28it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  94%|▉| 1767/1876 [00:40<00:02, 43.40it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  95%|▉| 1778/1876 [00:40<00:02, 43.53it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  95%|▉| 1789/1876 [00:40<00:01, 43.68it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Validating:  45%|█████████████▍                | 70/157 [00:01<00:01, 83.88it/s]\u001b[A\n",
      "Epoch 0:  96%|▉| 1800/1876 [00:41<00:01, 43.81it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  97%|▉| 1812/1876 [00:41<00:01, 43.98it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  97%|▉| 1824/1876 [00:41<00:01, 43.74it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  98%|▉| 1836/1876 [00:41<00:00, 43.92it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  99%|▉| 1848/1876 [00:41<00:00, 44.03it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0:  99%|▉| 1860/1876 [00:42<00:00, 44.21it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0: 100%|▉| 1872/1876 [00:42<00:00, 44.32it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 0: 100%|█| 1876/1876 [00:42<00:00, 44.26it/s, loss=0.137, v_num=43, train_\u001b[A\n",
      "Epoch 1:  92%|▉| 1719/1876 [00:39<00:03, 43.56it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  92%|▉| 1728/1876 [00:39<00:03, 43.41it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  93%|▉| 1740/1876 [00:39<00:03, 43.57it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Validating:  13%|████                          | 21/157 [00:00<00:02, 56.25it/s]\u001b[A\n",
      "Epoch 1:  93%|▉| 1752/1876 [00:40<00:02, 43.70it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  94%|▉| 1764/1876 [00:40<00:02, 43.83it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Validating:  30%|████████▉                     | 47/157 [00:00<00:01, 72.21it/s]\u001b[A\n",
      "Epoch 1:  95%|▉| 1776/1876 [00:40<00:02, 43.97it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  95%|▉| 1788/1876 [00:40<00:01, 44.13it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  96%|▉| 1800/1876 [00:40<00:01, 44.28it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  97%|▉| 1812/1876 [00:40<00:01, 44.44it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  97%|▉| 1824/1876 [00:40<00:01, 44.61it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  98%|▉| 1836/1876 [00:41<00:00, 44.73it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  99%|▉| 1848/1876 [00:41<00:00, 44.87it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1:  99%|▉| 1860/1876 [00:41<00:00, 45.03it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1: 100%|▉| 1872/1876 [00:41<00:00, 45.19it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 1: 100%|█| 1876/1876 [00:41<00:00, 45.16it/s, loss=0.103, v_num=43, train_\u001b[A\n",
      "Epoch 2:  92%|▉| 1719/1876 [00:38<00:03, 44.87it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  92%|▉| 1728/1876 [00:38<00:03, 44.69it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2:  93%|▉| 1740/1876 [00:38<00:03, 44.85it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Validating:  13%|████                          | 21/157 [00:00<00:02, 54.69it/s]\u001b[A\n",
      "Epoch 2:  93%|▉| 1752/1876 [00:38<00:02, 45.02it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2:  94%|▉| 1764/1876 [00:39<00:02, 44.96it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2:  95%|▉| 1776/1876 [00:39<00:02, 45.04it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2:  95%|▉| 1788/1876 [00:39<00:01, 45.19it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Validating:  44%|█████████████▏                | 69/157 [00:01<00:01, 64.20it/s]\u001b[A\n",
      "Epoch 2:  96%|▉| 1800/1876 [00:39<00:01, 45.33it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2:  97%|▉| 1812/1876 [00:39<00:01, 45.43it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2:  97%|▉| 1824/1876 [00:40<00:01, 45.56it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2:  98%|▉| 1836/1876 [00:40<00:00, 45.70it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Validating:  75%|█████████████████████▌       | 117/157 [00:01<00:00, 76.39it/s]\u001b[A\n",
      "Epoch 2:  99%|▉| 1848/1876 [00:40<00:00, 45.75it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2:  99%|▉| 1860/1876 [00:40<00:00, 45.89it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Validating:  91%|██████████████████████████▍  | 143/157 [00:02<00:00, 66.14it/s]\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:40<00:00, 45.92it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:40<00:00, 45.91it/s, loss=0.0701, v_num=43, train\u001b[A\n",
      "Testing: 100%|████████████████████████████████| 313/313 [00:05<00:00, 61.75it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9790999889373779, 'test_loss': 0.07223792374134064}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|████████████████████████████████| 313/313 [00:05<00:00, 55.95it/s]\n",
      "CPU times: user 2.67 s, sys: 740 ms, total: 3.41 s\n",
      "Wall time: 2min 17s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#单CPU\n",
    "!python3 mnist_cnn.py --gpus=0 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd62c23c",
   "metadata": {},
   "source": [
    "```\n",
    "-----------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9790999889373779, 'test_loss': 0.07223792374134064}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|████████████████████████████████| 313/313 [00:05<00:00, 55.95it/s]\n",
    "CPU times: user 2.67 s, sys: 740 ms, total: 3.41 s\n",
    "Wall time: 2min 17s\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8a2ff9ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  92%|▉| 1719/1876 [00:16<00:01, 103.63it/s, loss=0.136, v_num=3, train_\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  92%|▉| 1726/1876 [00:16<00:01, 102.26it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 0:  93%|▉| 1744/1876 [00:16<00:01, 102.70it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 0:  94%|▉| 1763/1876 [00:17<00:01, 103.21it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 0:  95%|▉| 1782/1876 [00:17<00:00, 103.72it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 0:  96%|▉| 1801/1876 [00:17<00:00, 104.21it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 0:  97%|▉| 1820/1876 [00:17<00:00, 104.69it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 0:  98%|▉| 1839/1876 [00:17<00:00, 105.16it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 0:  99%|▉| 1858/1876 [00:17<00:00, 105.62it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 0: 100%|█| 1876/1876 [00:17<00:00, 105.67it/s, loss=0.136, v_num=3, train_\u001b[A\n",
      "Epoch 1:  92%|▉| 1719/1876 [00:16<00:01, 102.50it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  92%|▉| 1729/1876 [00:17<00:01, 101.29it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Epoch 1:  93%|▉| 1750/1876 [00:17<00:01, 101.90it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Epoch 1:  94%|▉| 1771/1876 [00:17<00:01, 102.51it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Epoch 1:  96%|▉| 1792/1876 [00:17<00:00, 103.08it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Epoch 1:  97%|▉| 1813/1876 [00:17<00:00, 103.65it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Epoch 1:  98%|▉| 1834/1876 [00:17<00:00, 104.24it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Epoch 1:  99%|▉| 1855/1876 [00:17<00:00, 104.83it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Epoch 1: 100%|█| 1876/1876 [00:17<00:00, 105.02it/s, loss=0.0813, v_num=3, train\u001b[A\n",
      "Epoch 2:  92%|▉| 1719/1876 [00:15<00:01, 108.71it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                       | 0/157 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  92%|▉| 1722/1876 [00:16<00:01, 107.15it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Epoch 2:  93%|▉| 1743/1876 [00:16<00:01, 107.76it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Epoch 2:  94%|▉| 1764/1876 [00:16<00:01, 108.29it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Epoch 2:  95%|▉| 1785/1876 [00:16<00:00, 108.79it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Epoch 2:  96%|▉| 1806/1876 [00:16<00:00, 109.37it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Epoch 2:  97%|▉| 1827/1876 [00:16<00:00, 109.91it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Epoch 2:  99%|▉| 1848/1876 [00:16<00:00, 110.49it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:16<00:00, 110.80it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "Epoch 2: 100%|█| 1876/1876 [00:16<00:00, 110.73it/s, loss=0.0521, v_num=3, train\u001b[A\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Testing:  96%|█████████████████████████████▌ | 299/313 [00:01<00:00, 213.15it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9778000116348267, 'test_loss': 0.06929327547550201}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|███████████████████████████████| 313/313 [00:01<00:00, 171.04it/s]\n",
      "CPU times: user 1.83 s, sys: 488 ms, total: 2.32 s\n",
      "Wall time: 1min 3s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#单GPU\n",
    "!python3 mnist_cnn.py --gpus=1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6875397",
   "metadata": {},
   "source": [
    "```\n",
    "---------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9778000116348267, 'test_loss': 0.06929327547550201}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|███████████████████████████████| 313/313 [00:01<00:00, 171.04it/s]\n",
    "CPU times: user 1.83 s, sys: 488 ms, total: 2.32 s\n",
    "Wall time: 1min 3s\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "a987bc2a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  91%|▉| 430/470 [00:11<00:01, 37.23it/s, loss=0.16, v_num=48, train_acc\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  93%|▉| 435/470 [00:12<00:00, 36.00it/s, loss=0.16, v_num=48, train_acc\u001b[A\n",
      "Epoch 0:  94%|▉| 441/470 [00:12<00:00, 36.18it/s, loss=0.16, v_num=48, train_acc\u001b[A\n",
      "Epoch 0:  95%|▉| 447/470 [00:12<00:00, 36.35it/s, loss=0.16, v_num=48, train_acc\u001b[A\n",
      "Epoch 0:  96%|▉| 453/470 [00:12<00:00, 36.51it/s, loss=0.16, v_num=48, train_acc\u001b[A\n",
      "Epoch 0:  98%|▉| 459/470 [00:12<00:00, 36.68it/s, loss=0.16, v_num=48, train_acc\u001b[A\n",
      "Epoch 0:  99%|▉| 465/470 [00:12<00:00, 36.86it/s, loss=0.16, v_num=48, train_acc\u001b[A\n",
      "Epoch 0: 100%|█| 470/470 [00:12<00:00, 36.57it/s, loss=0.16, v_num=48, train_acc\u001b[A\n",
      "Epoch 1:  91%|▉| 430/470 [00:10<00:01, 39.66it/s, loss=0.11, v_num=48, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  92%|▉| 432/470 [00:11<00:00, 38.05it/s, loss=0.11, v_num=48, train_acc\u001b[A\n",
      "Epoch 1:  93%|▉| 438/470 [00:11<00:00, 38.22it/s, loss=0.11, v_num=48, train_acc\u001b[A\n",
      "Epoch 1:  94%|▉| 444/470 [00:11<00:00, 38.37it/s, loss=0.11, v_num=48, train_acc\u001b[A\n",
      "Epoch 1:  96%|▉| 450/470 [00:11<00:00, 38.54it/s, loss=0.11, v_num=48, train_acc\u001b[A\n",
      "Epoch 1:  97%|▉| 456/470 [00:11<00:00, 38.71it/s, loss=0.11, v_num=48, train_acc\u001b[A\n",
      "Epoch 1:  98%|▉| 462/470 [00:11<00:00, 38.88it/s, loss=0.11, v_num=48, train_acc\u001b[A\n",
      "Epoch 1: 100%|█| 470/470 [00:12<00:00, 38.65it/s, loss=0.11, v_num=48, train_acc\u001b[A\n",
      "Epoch 2:  91%|▉| 430/470 [00:10<00:01, 39.47it/s, loss=0.0776, v_num=48, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  92%|▉| 434/470 [00:11<00:00, 38.05it/s, loss=0.0776, v_num=48, train_a\u001b[A\n",
      "Epoch 2:  94%|▉| 441/470 [00:11<00:00, 38.24it/s, loss=0.0776, v_num=48, train_a\u001b[A\n",
      "Epoch 2:  95%|▉| 448/470 [00:11<00:00, 38.43it/s, loss=0.0776, v_num=48, train_a\u001b[A\n",
      "Epoch 2:  97%|▉| 455/470 [00:11<00:00, 38.62it/s, loss=0.0776, v_num=48, train_a\u001b[A\n",
      "Validating:  62%|███████████████████▍           | 25/40 [00:00<00:00, 40.04it/s]\u001b[A\n",
      "Epoch 2:  98%|▉| 462/470 [00:11<00:00, 38.79it/s, loss=0.0776, v_num=48, train_a\u001b[A\n",
      "Epoch 2: 100%|█| 470/470 [00:12<00:00, 38.58it/s, loss=0.0776, v_num=48, train_a\u001b[A\n",
      "Epoch 2: 100%|█| 470/470 [00:12<00:00, 38.54it/s, loss=0.0776, v_num=48, train_a\u001b[A\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "Testing: 100%|██████████████████████████████████| 79/79 [00:02<00:00, 38.62it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9790999889373779, 'test_loss': 0.06855566054582596}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 79/79 [00:02<00:00, 38.55it/s]\n",
      "CPU times: user 1.2 s, sys: 553 ms, total: 1.75 s\n",
      "Wall time: 1min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#多GPU，dp模式（为公平比较，batch_size=32*4）\n",
    "!python3 mnist_cnn.py --gpus=4 --strategy=\"dp\" --batch_size=128"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77073b7a",
   "metadata": {},
   "source": [
    "```\n",
    "------------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9790999889373779, 'test_loss': 0.06855566054582596}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|██████████████████████████████████| 79/79 [00:02<00:00, 38.55it/s]\n",
    "CPU times: user 1.2 s, sys: 553 ms, total: 1.75 s\n",
    "Wall time: 1min\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43018329",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "6fa1f488",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/4\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/4\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/4\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4\n",
      "----------------------------------------------------------------------------------------------------\n",
      "distributed_backend=nccl\n",
      "All distributed processes registered. Starting with 4 processes\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Epoch 0:  91%|▉| 430/470 [00:05<00:00, 83.09it/s, loss=0.201, v_num=47, train_ac\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  93%|▉| 437/470 [00:05<00:00, 79.88it/s, loss=0.201, v_num=47, train_ac\u001b[A\n",
      "Epoch 0:  97%|▉| 456/470 [00:05<00:00, 81.84it/s, loss=0.201, v_num=47, train_ac\u001b[A\n",
      "Epoch 0: 100%|█| 470/470 [00:05<00:00, 80.88it/s, loss=0.201, v_num=47, train_ac\u001b[A\n",
      "Epoch 1:  91%|▉| 430/470 [00:05<00:00, 84.31it/s, loss=0.0979, v_num=47, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  93%|▉| 437/470 [00:05<00:00, 80.97it/s, loss=0.0979, v_num=47, train_a\u001b[A\n",
      "Epoch 1:  97%|▉| 456/470 [00:05<00:00, 82.95it/s, loss=0.0979, v_num=47, train_a\u001b[A\n",
      "Epoch 1: 100%|█| 470/470 [00:05<00:00, 82.27it/s, loss=0.0979, v_num=47, train_a\u001b[A\n",
      "Epoch 2:  91%|▉| 430/470 [00:05<00:00, 84.67it/s, loss=0.0889, v_num=47, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  93%|▉| 437/470 [00:05<00:00, 81.37it/s, loss=0.0889, v_num=47, train_a\u001b[A\n",
      "Epoch 2:  97%|▉| 456/470 [00:05<00:00, 83.34it/s, loss=0.0889, v_num=47, train_a\u001b[A\n",
      "Epoch 2: 100%|█| 470/470 [00:05<00:00, 82.87it/s, loss=0.0889, v_num=47, train_a\u001b[A\n",
      "Epoch 2: 100%|█| 470/470 [00:05<00:00, 82.72it/s, loss=0.0889, v_num=47, train_a\u001b[A\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "Testing:  91%|██████████████████████████████   | 72/79 [00:00<00:00, 137.66it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9732000231742859, 'test_loss': 0.08606339246034622}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 79/79 [00:00<00:00, 85.79it/s]\n",
      "CPU times: user 784 ms, sys: 387 ms, total: 1.17 s\n",
      "Wall time: 38.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#多GPU，ddp模式\n",
    "!python3 mnist_cnn.py --gpus=4 --strategy=\"ddp_find_unused_parameters_false\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e96fe6",
   "metadata": {},
   "source": [
    "```\n",
    "---------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9732000231742859, 'test_loss': 0.08606339246034622}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|██████████████████████████████████| 79/79 [00:00<00:00, 85.79it/s]\n",
    "CPU times: user 784 ms, sys: 387 ms, total: 1.17 s\n",
    "Wall time: 38.9 s\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dacaa85b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "090dbf1f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "37f89470",
   "metadata": {},
   "source": [
    "### 4，使用梯度累加(accumulate_grad_batches=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7dbb726",
   "metadata": {},
   "source": [
    "梯度累加就是累加多个batch的梯度，然后用累加的梯度更新一次参数，使用梯度累加相当于增大batch_size.\n",
    "\n",
    "由于更新参数的计算量略大于简单梯度求和的计算量（对于大部分优化器而言），使用梯度累加会让速度略有提升。\n",
    "\n",
    "* 4个GPU(ddp模式): 38.9 s\n",
    "\n",
    "* 4个GPU(ddp模式)+梯度累加: 36.9 s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "872bfc35",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/4\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/4\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/4\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4\n",
      "----------------------------------------------------------------------------------------------------\n",
      "distributed_backend=nccl\n",
      "All distributed processes registered. Starting with 4 processes\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Epoch 0:  91%|▉| 430/470 [00:04<00:00, 95.76it/s, loss=0.639, v_num=54, train_ac\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  92%|▉| 432/470 [00:04<00:00, 90.82it/s, loss=0.639, v_num=54, train_ac\u001b[A\n",
      "Epoch 0:  96%|▉| 449/470 [00:04<00:00, 92.43it/s, loss=0.639, v_num=54, train_ac\u001b[A\n",
      "Epoch 0: 100%|█| 470/470 [00:05<00:00, 92.13it/s, loss=0.639, v_num=54, train_ac\u001b[A\n",
      "Epoch 1:  91%|▉| 430/470 [00:04<00:00, 95.65it/s, loss=0.205, v_num=54, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  94%|▉| 442/470 [00:04<00:00, 91.50it/s, loss=0.205, v_num=54, train_ac\u001b[A\n",
      "Epoch 1:  98%|▉| 460/470 [00:04<00:00, 93.21it/s, loss=0.205, v_num=54, train_ac\u001b[A\n",
      "Epoch 1: 100%|█| 470/470 [00:05<00:00, 92.20it/s, loss=0.205, v_num=54, train_ac\u001b[A\n",
      "Epoch 2:  91%|▉| 430/470 [00:04<00:00, 94.67it/s, loss=0.155, v_num=54, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/40 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  92%|▉| 432/470 [00:04<00:00, 89.72it/s, loss=0.155, v_num=54, train_ac\u001b[A\n",
      "Epoch 2:  96%|▉| 450/470 [00:04<00:00, 91.25it/s, loss=0.155, v_num=54, train_ac\u001b[A\n",
      "Epoch 2: 100%|█| 470/470 [00:05<00:00, 90.41it/s, loss=0.155, v_num=54, train_ac\u001b[A\n",
      "Epoch 2: 100%|█| 470/470 [00:05<00:00, 90.23it/s, loss=0.155, v_num=54, train_ac\u001b[A\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "Testing:  94%|██████████████████████████████▉  | 74/79 [00:00<00:00, 141.05it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9603000283241272, 'test_loss': 0.1400066614151001}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 79/79 [00:00<00:00, 89.10it/s]\n",
      "CPU times: user 749 ms, sys: 402 ms, total: 1.15 s\n",
      "Wall time: 36.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#多GPU，ddp模式, 考虑梯度累加\n",
    "!python3 mnist_cnn.py --accumulate_grad_batches=6 --gpus=4 --strategy=\"ddp_find_unused_parameters_false\" "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23e1271f",
   "metadata": {},
   "source": [
    "```\n",
    "----------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9603000283241272, 'test_loss': 0.1400066614151001}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|██████████████████████████████████| 79/79 [00:00<00:00, 89.10it/s]\n",
    "CPU times: user 749 ms, sys: 402 ms, total: 1.15 s\n",
    "Wall time: 36.9 s\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58afdb0d",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86aeb0bc",
   "metadata": {},
   "source": [
    "### 5，使用半精度(precision=16) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1d1cc94",
   "metadata": {},
   "source": [
    "通过precision可以设置 double (64), float (32), bfloat16 (\"bf16\"), half (16) 精度的训练。\n",
    "\n",
    "默认是float(32) 标准精度，bfloat16 (\"bf16\")是混合精度。\n",
    "\n",
    "如果选择 half(16) 半精度，并同时增大batch_size为原来2倍, 通常训练速度会提升3倍左右。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "626a4433",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "Using 16bit native Automatic Mixed Precision (AMP)\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.108     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  92%|▉| 860/939 [00:08<00:00, 99.68it/s, loss=0.142, v_num=73, train_ac\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/79 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  92%|▉| 868/939 [00:08<00:00, 96.98it/s, loss=0.142, v_num=73, train_ac\u001b[A\n",
      "Epoch 0:  95%|▉| 888/939 [00:09<00:00, 98.11it/s, loss=0.142, v_num=73, train_ac\u001b[A\n",
      "Epoch 0:  97%|▉| 908/939 [00:09<00:00, 99.16it/s, loss=0.142, v_num=73, train_ac\u001b[A\n",
      "Epoch 0:  99%|▉| 928/939 [00:09<00:00, 100.10it/s, loss=0.142, v_num=73, train_a\u001b[A\n",
      "Epoch 0: 100%|█| 939/939 [00:09<00:00, 99.59it/s, loss=0.142, v_num=73, train_ac\u001b[A\n",
      "Epoch 1:  92%|▉| 860/939 [00:08<00:00, 99.95it/s, loss=0.0815, v_num=73, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/79 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  94%|▉| 880/939 [00:08<00:00, 97.85it/s, loss=0.0815, v_num=73, train_a\u001b[A\n",
      "Epoch 1:  96%|▉| 900/939 [00:09<00:00, 98.96it/s, loss=0.0815, v_num=73, train_a\u001b[A\n",
      "Epoch 1:  98%|▉| 920/939 [00:09<00:00, 100.02it/s, loss=0.0815, v_num=73, train_\u001b[A\n",
      "Validating:  77%|███████████████████████▏      | 61/79 [00:00<00:00, 135.13it/s]\u001b[A\n",
      "Epoch 1: 100%|█| 939/939 [00:09<00:00, 99.97it/s, loss=0.0815, v_num=73, train_a\u001b[A\n",
      "Epoch 2:  92%|▉| 860/939 [00:08<00:00, 100.47it/s, loss=0.0806, v_num=73, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/79 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  94%|▉| 880/939 [00:08<00:00, 98.30it/s, loss=0.0806, v_num=73, train_a\u001b[A\n",
      "Validating:  25%|███████▊                       | 20/79 [00:00<00:00, 64.09it/s]\u001b[A\n",
      "Epoch 2:  96%|▉| 900/939 [00:09<00:00, 99.23it/s, loss=0.0806, v_num=73, train_a\u001b[A\n",
      "Epoch 2:  98%|▉| 920/939 [00:09<00:00, 100.13it/s, loss=0.0806, v_num=73, train_\u001b[A\n",
      "Epoch 2: 100%|█| 939/939 [00:09<00:00, 100.10it/s, loss=0.0806, v_num=73, train_\u001b[A\n",
      "Epoch 2: 100%|█| 939/939 [00:09<00:00, 99.99it/s, loss=0.0806, v_num=73, train_a\u001b[A\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "Testing: 100%|███████████████████████████████| 157/157 [00:01<00:00, 145.74it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9769999980926514, 'test_loss': 0.07660940289497375}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|███████████████████████████████| 157/157 [00:01<00:00, 130.02it/s]\n",
      "CPU times: user 943 ms, sys: 429 ms, total: 1.37 s\n",
      "Wall time: 39.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "#半精度\n",
    "!python3 mnist_cnn.py --precision=16 --batch_size=64 --gpus=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e44efe53",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "02b5844a",
   "metadata": {},
   "source": [
    "### 6，自动搜索最大batch_size(auto_scale_batch_size=\"power\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "dfac369f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py:2165: LightningDeprecationWarning: `Trainer.weights_summary` is deprecated in v1.5 and will be removed in v1.7.\n",
      "  rank_zero_deprecation(\"`Trainer.weights_summary` is deprecated in v1.5 and will be removed in v1.7.\")\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py:2170: LightningDeprecationWarning: Setting `Trainer.weights_summary` is deprecated in v1.5 and will be removed in v1.7.\n",
      "  rank_zero_deprecation(\"Setting `Trainer.weights_summary` is deprecated in v1.5 and will be removed in v1.7.\")\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 2 succeeded, trying batch size 4\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.teardown has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.teardown.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "Batch size 4 succeeded, trying batch size 8\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 8 succeeded, trying batch size 16\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 16 succeeded, trying batch size 32\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 32 succeeded, trying batch size 64\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 64 succeeded, trying batch size 128\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 128 succeeded, trying batch size 256\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 256 succeeded, trying batch size 512\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 512 succeeded, trying batch size 1024\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 1024 succeeded, trying batch size 2048\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (27) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 2048 succeeded, trying batch size 4096\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (14) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 4096 succeeded, trying batch size 8192\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (7) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 8192 succeeded, trying batch size 16384\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (4) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 16384 succeeded, trying batch size 32768\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (2) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 32768 succeeded, trying batch size 65536\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Global seed set to 1234\n",
      "Batch size 55000 succeeded, trying batch size 110000\n",
      "Finished batch size finder, will continue with full run using batch size 55000\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/scale_batch_size_temp_model_46adaf1c-f209-4c7f-910c-6fcecfcc985b.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  50%|▌| 1/2 [00:07<00:07,  7.74s/it, loss=1.91, v_num=12, train_acc=0.0\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0: 100%|█| 2/2 [00:08<00:00,  4.29s/it, loss=1.91, v_num=12, train_acc=0.0\u001b[A\n",
      "Epoch 1:  50%|▌| 1/2 [00:07<00:07,  7.83s/it, loss=1.92, v_num=12, train_acc=0.1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1: 100%|█| 2/2 [00:08<00:00,  4.34s/it, loss=1.92, v_num=12, train_acc=0.1\u001b[A\n",
      "Epoch 2:  50%|▌| 1/2 [00:07<00:07,  7.88s/it, loss=1.93, v_num=12, train_acc=0.1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2: 100%|█| 2/2 [00:08<00:00,  4.36s/it, loss=1.93, v_num=12, train_acc=0.1\u001b[A\n",
      "Epoch 3:  50%|▌| 1/2 [00:07<00:07,  7.93s/it, loss=1.94, v_num=12, train_acc=0.1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 3: 100%|█| 2/2 [00:08<00:00,  4.38s/it, loss=1.94, v_num=12, train_acc=0.1\u001b[A\n",
      "Epoch 4:  50%|▌| 1/2 [00:07<00:07,  7.71s/it, loss=1.95, v_num=12, train_acc=0.2\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 4: 100%|█| 2/2 [00:08<00:00,  4.29s/it, loss=1.95, v_num=12, train_acc=0.2\u001b[A\n",
      "Epoch 5:  50%|▌| 1/2 [00:07<00:07,  7.89s/it, loss=1.96, v_num=12, train_acc=0.2\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 5: 100%|█| 2/2 [00:08<00:00,  4.37s/it, loss=1.96, v_num=12, train_acc=0.2\u001b[A\n",
      "Epoch 6:  50%|▌| 1/2 [00:07<00:07,  7.84s/it, loss=1.97, v_num=12, train_acc=0.3\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 6: 100%|█| 2/2 [00:08<00:00,  4.34s/it, loss=1.97, v_num=12, train_acc=0.3\u001b[A\n",
      "Epoch 7:  50%|▌| 1/2 [00:07<00:07,  7.94s/it, loss=1.98, v_num=12, train_acc=0.3\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 7: 100%|█| 2/2 [00:08<00:00,  4.38s/it, loss=1.98, v_num=12, train_acc=0.3\u001b[A\n",
      "Epoch 8:  50%|▌| 1/2 [00:07<00:07,  7.95s/it, loss=2, v_num=12, train_acc=0.333,\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 8: 100%|█| 2/2 [00:08<00:00,  4.41s/it, loss=2, v_num=12, train_acc=0.333,\u001b[A\n",
      "Epoch 9:  50%|▌| 1/2 [00:07<00:07,  7.92s/it, loss=2.01, v_num=12, train_acc=0.3\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 9: 100%|█| 2/2 [00:08<00:00,  4.39s/it, loss=2.01, v_num=12, train_acc=0.3\u001b[A\n",
      "Epoch 9: 100%|█| 2/2 [00:08<00:00,  4.39s/it, loss=2.01, v_num=12, train_acc=0.3\u001b[A\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Testing: 100%|████████████████████████████████████| 1/1 [00:01<00:00,  1.49s/it]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.4447000026702881, 'test_loss': 2.1517751216888428}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|████████████████████████████████████| 1/1 [00:01<00:00,  1.49s/it]\n"
     ]
    }
   ],
   "source": [
    "!python3 mnist_cnn.py --auto_scale_batch_size=\"power\"  --gpus=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "faeda7a7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9508c293",
   "metadata": {},
   "source": [
    "## 四，训练涨分技巧"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61841977",
   "metadata": {},
   "source": [
    "pytorch_lightning 可以非常容易地支持以下训练涨分技巧: \n",
    "\n",
    "* SWA(随机参数平均): 调用pl.callbacks.stochastic_weight_avg.StochasticWeightAveraging实现。\n",
    "\n",
    "* CyclicLR(学习率周期性调度策略): 设置 lr_scheduler 为 torch.optim.lr_scheduler.CyclicLR实现。\n",
    "\n",
    "* auto_lr_find最优学习率发现: 设置 pl.Trainer(auto_lr_find = True)实现。\n",
    "\n",
    "\n",
    "参考论文:\n",
    "\n",
    "* Cyclical Learning Rates for Training Neural Networks 【https://arxiv.org/pdf/1506.01186.pdf】\n",
    "\n",
    "* Averaging Weights Leads to Wider Optima and Better Generalization【https://arxiv.org/abs/1803.05407】"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb9d5477-59cb-4fc6-880c-d61b5b7ac72d",
   "metadata": {},
   "source": [
    "我们将代码整理成如下形式，以便后续测试使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2be95dc9-63bd-4f27-9473-f6762bc600d1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 306,
   "id": "439eeec3-a229-4164-934f-bec6215187f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mnist_cnn.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile mnist_cnn.py\n",
    "import torch \n",
    "from torch import nn \n",
    "from argparse import ArgumentParser\n",
    "import numpy as np \n",
    "\n",
    "import torchvision \n",
    "from torchvision import transforms as T\n",
    "from torchvision.datasets import MNIST\n",
    "from torch.utils.data import DataLoader,random_split\n",
    "import pytorch_lightning as pl\n",
    "from torchmetrics import Accuracy\n",
    "\n",
    "#================================================================================\n",
    "# 一，准备数据\n",
    "#================================================================================\n",
    "\n",
    "class MNISTDataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_dir: str = \"./minist/\", \n",
    "                 batch_size: int = 32,\n",
    "                 num_workers: int =4,\n",
    "                 pin_memory:bool =True):\n",
    "        super().__init__()\n",
    "        self.data_dir = data_dir\n",
    "        self.batch_size = batch_size\n",
    "        self.num_workers = num_workers\n",
    "        self.pin_memory = pin_memory\n",
    "\n",
    "    def setup(self, stage = None):\n",
    "        transform = T.Compose([T.ToTensor()])\n",
    "        self.ds_test = MNIST(self.data_dir, download=True,train=False,transform=transform)\n",
    "        self.ds_predict = MNIST(self.data_dir, download=True, train=False,transform=transform)\n",
    "        ds_full = MNIST(self.data_dir, download=True, train=True,transform=transform)\n",
    "        ds_train, self.ds_val = random_split(ds_full, [59000, 1000])\n",
    "        #为加速训练，随机取10000个\n",
    "        indices = np.arange(59000)\n",
    "        np.random.shuffle(indices)\n",
    "        self.ds_train = torch.utils.data.dataset.Subset(\n",
    "            ds_train,indices = indices[:3000]) \n",
    "\n",
    "    def train_dataloader(self):\n",
    "        return DataLoader(self.ds_train, batch_size=self.batch_size,\n",
    "                          shuffle=True, num_workers=self.num_workers,\n",
    "                          pin_memory=self.pin_memory)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return DataLoader(self.ds_val, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=self.pin_memory)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return DataLoader(self.ds_test, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=self.pin_memory)\n",
    "\n",
    "    def predict_dataloader(self):\n",
    "        return DataLoader(self.ds_predict, batch_size=self.batch_size,\n",
    "                          shuffle=False, num_workers=self.num_workers,\n",
    "                          pin_memory=self.pin_memory)\n",
    "    \n",
    "    @staticmethod\n",
    "    def add_dataset_args(parent_parser):\n",
    "        parser = ArgumentParser(parents=[parent_parser], add_help=False)\n",
    "        parser.add_argument('--batch_size', type=int, default=32)\n",
    "        parser.add_argument('--num_workers', type=int, default=8)\n",
    "        parser.add_argument('--pin_memory', type=bool, default=True)\n",
    "        return parser\n",
    "\n",
    "#================================================================================\n",
    "# 二，定义模型\n",
    "#================================================================================\n",
    "\n",
    "net = nn.Sequential(\n",
    "    nn.Conv2d(in_channels=1,out_channels=32,kernel_size = 3),\n",
    "    nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "    nn.Conv2d(in_channels=32,out_channels=64,kernel_size = 5),\n",
    "    nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "    nn.Dropout2d(p = 0.1),\n",
    "    nn.AdaptiveMaxPool2d((1,1)),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(64,32),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(32,10)\n",
    ")\n",
    "\n",
    "class Model(pl.LightningModule):\n",
    "    \n",
    "    def __init__(self,net,\n",
    "                 learning_rate=1e-3,\n",
    "                 use_CyclicLR = False,\n",
    "                 epoch_size=500):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters() #自动创建self.hparams\n",
    "        self.net = net\n",
    "        self.train_acc = Accuracy()\n",
    "        self.val_acc = Accuracy()\n",
    "        self.test_acc = Accuracy() \n",
    "        \n",
    "        \n",
    "    def forward(self,x):\n",
    "        x = self.net(x)\n",
    "        return x\n",
    "    \n",
    "    \n",
    "    #定义loss\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "    \n",
    "    #定义各种metrics\n",
    "    def training_step_end(self,outputs):\n",
    "        train_acc = self.train_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"train_acc\",train_acc,prog_bar=True)\n",
    "        return {\"loss\":outputs[\"loss\"].mean()}\n",
    "    \n",
    "    #定义optimizer,以及可选的lr_scheduler\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.RMSprop(self.parameters(), lr=self.hparams.learning_rate)\n",
    "        if not self.hparams.use_CyclicLR:\n",
    "            return optimizer \n",
    "\n",
    "        max_lr = self.hparams.learning_rate\n",
    "        base_lr = max_lr/4.0\n",
    "        scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer,\n",
    "            base_lr=base_lr,max_lr=max_lr,\n",
    "            step_size_up=5*self.hparams.epoch_size,cycle_momentum=False)\n",
    "        self.print(\"set lr = \"+str(max_lr))\n",
    "        \n",
    "        return ([optimizer],[scheduler])\n",
    "    \n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "\n",
    "    def validation_step_end(self,outputs):\n",
    "        val_acc = self.val_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"val_loss\",outputs[\"loss\"].mean(),on_epoch=True,on_step=False)\n",
    "        self.log(\"val_acc\",val_acc,prog_bar=True,on_epoch=True,on_step=False)\n",
    "    \n",
    "    def test_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "        preds = self(x)\n",
    "        loss = nn.CrossEntropyLoss()(preds,y)\n",
    "        return {\"loss\":loss,\"preds\":preds.detach(),\"y\":y.detach()}\n",
    "    \n",
    "    def test_step_end(self,outputs):\n",
    "        test_acc = self.test_acc(outputs['preds'], outputs['y']).item()    \n",
    "        self.log(\"test_acc\",test_acc,on_epoch=True,on_step=False)\n",
    "        self.log(\"test_loss\",outputs[\"loss\"].mean(),on_epoch=True,on_step=False)\n",
    "    \n",
    "    @staticmethod\n",
    "    def add_model_args(parent_parser):\n",
    "        parser = ArgumentParser(parents=[parent_parser], add_help=False)\n",
    "        parser.add_argument('--learning_rate', type=float, default=7e-3)\n",
    "        parser.add_argument('--use_CyclicLR', type=bool, default=False)\n",
    "        return parser\n",
    "    \n",
    "\n",
    "#================================================================================\n",
    "# 三，训练模型\n",
    "#================================================================================\n",
    "    \n",
    "def main(hparams):\n",
    "    pl.seed_everything(1234)\n",
    "    \n",
    "    data_mnist = MNISTDataModule(batch_size=hparams.batch_size,\n",
    "                                 num_workers=hparams.num_workers)\n",
    "    data_mnist.setup()\n",
    "    epoch_size = len(data_mnist.ds_train)//data_mnist.batch_size\n",
    "    \n",
    "    model = Model(net,learning_rate=hparams.learning_rate,\n",
    "                  use_CyclicLR = hparams.use_CyclicLR,\n",
    "                  epoch_size=epoch_size)\n",
    "    \n",
    "    ckpt_callback = pl.callbacks.ModelCheckpoint(\n",
    "        monitor='val_acc',\n",
    "        save_top_k=3,\n",
    "        mode='max'\n",
    "    )\n",
    "    \n",
    "    early_stopping = pl.callbacks.EarlyStopping(monitor = 'val_acc',\n",
    "                   patience=16,\n",
    "                   mode = 'max')\n",
    "    callbacks = [ckpt_callback,early_stopping]\n",
    "    if hparams.use_swa:\n",
    "        callbacks.append(pl.callbacks.StochasticWeightAveraging())\n",
    "        \n",
    "    trainer = pl.Trainer.from_argparse_args( \n",
    "        hparams,\n",
    "        max_epochs=1000,\n",
    "        callbacks = callbacks) \n",
    "\n",
    "    \n",
    "    print(\"hparams.auto_lr_find=\",hparams.auto_lr_find)\n",
    "    if hparams.auto_lr_find:\n",
    "        \n",
    "        #搜索学习率范围\n",
    "        lr_finder = trainer.tuner.lr_find(model,\n",
    "          datamodule = data_mnist,\n",
    "          min_lr=1e-08,\n",
    "          max_lr=1,\n",
    "          num_training=100,\n",
    "          mode='exponential',\n",
    "          early_stop_threshold=4.0\n",
    "          )\n",
    "        lr_finder.plot() \n",
    "        lr = lr_finder.suggestion()\n",
    "        model.hparams.learning_rate = lr \n",
    "        print(\"suggest lr=\",lr)\n",
    "        \n",
    "        del model \n",
    "        \n",
    "        hparams.learning_rate = lr\n",
    "        model = Model(net,learning_rate=hparams.learning_rate,\n",
    "                  use_CyclicLR = hparams.use_CyclicLR,\n",
    "                  epoch_size=epoch_size)\n",
    "        \n",
    "        #等价于\n",
    "        #trainer.tune(model,data_mnist)\n",
    "        \n",
    "\n",
    "    trainer.fit(model,data_mnist)\n",
    "    train_result = trainer.test(model,data_mnist.train_dataloader(),ckpt_path='best')\n",
    "    val_result = trainer.test(model,data_mnist.val_dataloader(),ckpt_path='best')\n",
    "    test_result = trainer.test(model,data_mnist.test_dataloader(),ckpt_path='best')\n",
    "    \n",
    "    print(\"train_result:\\n\")\n",
    "    print(train_result)\n",
    "    print(\"val_result:\\n\")\n",
    "    print(val_result)\n",
    "    print(\"test_result:\\n\")\n",
    "    print(test_result)\n",
    "    \n",
    "    \n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    parser = ArgumentParser()\n",
    "    parser.add_argument('--use_swa', default=False, type=bool)\n",
    "    parser = MNISTDataModule.add_dataset_args(parser)\n",
    "    parser = Model.add_model_args(parser)\n",
    "    parser = pl.Trainer.add_argparse_args(parser)\n",
    "    hparams = parser.parse_args()\n",
    "    main(hparams)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f71c326c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bc72745f",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 1，SWA 随机权重平均 (pl.callbacks.stochastic_weight_avg.StochasticWeightAveraging)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e19ae84d-4838-4cc0-9fc6-2043d18b3cfc",
   "metadata": {},
   "source": [
    "* 平凡方式训练：test_acc = 0.9581000208854675\n",
    "\n",
    "* SWA随机权重： test_acc = 0.963100016117096"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "id": "49138e06-555d-4c44-9f05-7720d179cc0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2\n",
      "----------------------------------------------------------------------------------------------------\n",
      "distributed_backend=nccl\n",
      "All distributed processes registered. Starting with 2 processes\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Global seed set to 1234\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (47) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "Epoch 0:  75%|▋| 47/63 [00:02<00:00, 18.14it/s, loss=2.17, v_num=3, train_acc=0.\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  78%|▊| 49/63 [00:02<00:00, 16.42it/s, loss=2.17, v_num=3, train_acc=0.\u001b[A\n",
      "Epoch 0: 100%|█| 63/63 [00:03<00:00, 17.40it/s, loss=2.17, v_num=3, train_acc=0.\u001b[A\n",
      "Epoch 1:  76%|▊| 48/63 [00:02<00:00, 20.70it/s, loss=1.65, v_num=3, train_acc=0.\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  81%|▊| 51/63 [00:02<00:00, 18.81it/s, loss=1.65, v_num=3, train_acc=0.\u001b[A\n",
      "Epoch 1: 100%|█| 63/63 [00:03<00:00, 18.96it/s, loss=1.65, v_num=3, train_acc=0.\u001b[A\n",
      "Epoch 2:  76%|▊| 48/63 [00:02<00:00, 20.64it/s, loss=0.913, v_num=3, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  81%|▊| 51/63 [00:02<00:00, 18.71it/s, loss=0.913, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 2: 100%|█| 63/63 [00:03<00:00, 18.81it/s, loss=0.913, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 3:  76%|▊| 48/63 [00:02<00:00, 20.38it/s, loss=0.8, v_num=3, train_acc=0.8\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 3:  81%|▊| 51/63 [00:02<00:00, 18.47it/s, loss=0.8, v_num=3, train_acc=0.8\u001b[A\n",
      "Epoch 3: 100%|█| 63/63 [00:03<00:00, 18.70it/s, loss=0.8, v_num=3, train_acc=0.8\u001b[A\n",
      "Epoch 4:  76%|▊| 48/63 [00:02<00:00, 20.64it/s, loss=0.48, v_num=3, train_acc=0.\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 4:  81%|▊| 51/63 [00:02<00:00, 18.74it/s, loss=0.48, v_num=3, train_acc=0.\u001b[A\n",
      "Epoch 4: 100%|█| 63/63 [00:03<00:00, 18.81it/s, loss=0.48, v_num=3, train_acc=0.\u001b[A\n",
      "Epoch 5:  76%|▊| 48/63 [00:02<00:00, 20.49it/s, loss=0.34, v_num=3, train_acc=0.\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 5:  81%|▊| 51/63 [00:02<00:00, 18.44it/s, loss=0.34, v_num=3, train_acc=0.\u001b[A\n",
      "Epoch 5: 100%|█| 63/63 [00:03<00:00, 18.56it/s, loss=0.34, v_num=3, train_acc=0.\u001b[A\n",
      "Epoch 6:  76%|▊| 48/63 [00:02<00:00, 19.81it/s, loss=0.28, v_num=3, train_acc=1.\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 6:  81%|▊| 51/63 [00:02<00:00, 17.95it/s, loss=0.28, v_num=3, train_acc=1.\u001b[A\n",
      "Epoch 6: 100%|█| 63/63 [00:03<00:00, 18.27it/s, loss=0.28, v_num=3, train_acc=1.\u001b[A\n",
      "Epoch 7:  76%|▊| 48/63 [00:02<00:00, 20.28it/s, loss=0.983, v_num=3, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 7:  81%|▊| 51/63 [00:02<00:00, 18.28it/s, loss=0.983, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 7: 100%|█| 63/63 [00:03<00:00, 18.54it/s, loss=0.983, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 8:  76%|▊| 48/63 [00:02<00:00, 19.85it/s, loss=0.203, v_num=3, train_acc=1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 8:  81%|▊| 51/63 [00:02<00:00, 17.98it/s, loss=0.203, v_num=3, train_acc=1\u001b[A\n",
      "Epoch 8: 100%|█| 63/63 [00:03<00:00, 18.15it/s, loss=0.203, v_num=3, train_acc=1\u001b[A\n",
      "Epoch 9:  76%|▊| 48/63 [00:02<00:00, 19.96it/s, loss=0.225, v_num=3, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 9:  81%|▊| 51/63 [00:02<00:00, 18.00it/s, loss=0.225, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 9: 100%|█| 63/63 [00:03<00:00, 18.34it/s, loss=0.225, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 10:  76%|▊| 48/63 [00:02<00:00, 19.77it/s, loss=0.796, v_num=3, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 10:  81%|▊| 51/63 [00:02<00:00, 17.87it/s, loss=0.796, v_num=3, train_acc=\u001b[A\n",
      "Epoch 10: 100%|█| 63/63 [00:03<00:00, 18.29it/s, loss=0.796, v_num=3, train_acc=\u001b[A\n",
      "Epoch 11:  76%|▊| 48/63 [00:02<00:00, 20.05it/s, loss=0.15, v_num=3, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 11:  81%|▊| 51/63 [00:02<00:00, 18.12it/s, loss=0.15, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 11: 100%|█| 63/63 [00:03<00:00, 18.37it/s, loss=0.15, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 12:  76%|▊| 48/63 [00:02<00:00, 19.68it/s, loss=0.0767, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 12:  81%|▊| 51/63 [00:02<00:00, 17.79it/s, loss=0.0767, v_num=3, train_acc\u001b[A\n",
      "Epoch 12: 100%|█| 63/63 [00:03<00:00, 18.07it/s, loss=0.0767, v_num=3, train_acc\u001b[A\n",
      "Epoch 13:  76%|▊| 48/63 [00:02<00:00, 20.01it/s, loss=0.0841, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 13:  81%|▊| 51/63 [00:02<00:00, 18.14it/s, loss=0.0841, v_num=3, train_acc\u001b[A\n",
      "Epoch 13: 100%|█| 63/63 [00:03<00:00, 18.31it/s, loss=0.0841, v_num=3, train_acc\u001b[A\n",
      "Epoch 14:  76%|▊| 48/63 [00:02<00:00, 20.10it/s, loss=0.0821, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 14:  81%|▊| 51/63 [00:02<00:00, 18.12it/s, loss=0.0821, v_num=3, train_acc\u001b[A\n",
      "Epoch 14: 100%|█| 63/63 [00:03<00:00, 18.49it/s, loss=0.0821, v_num=3, train_acc\u001b[A\n",
      "Epoch 15:  76%|▊| 48/63 [00:02<00:00, 20.43it/s, loss=0.0548, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 15:  81%|▊| 51/63 [00:02<00:00, 18.50it/s, loss=0.0548, v_num=3, train_acc\u001b[A\n",
      "Epoch 15: 100%|█| 63/63 [00:03<00:00, 18.67it/s, loss=0.0548, v_num=3, train_acc\u001b[A\n",
      "Epoch 16:  76%|▊| 48/63 [00:02<00:00, 20.69it/s, loss=2.05, v_num=3, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 16:  81%|▊| 51/63 [00:02<00:00, 18.67it/s, loss=2.05, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 16: 100%|█| 63/63 [00:03<00:00, 18.87it/s, loss=2.05, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 17:  76%|▊| 48/63 [00:02<00:00, 20.86it/s, loss=0.116, v_num=3, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 17:  81%|▊| 51/63 [00:02<00:00, 18.83it/s, loss=0.116, v_num=3, train_acc=\u001b[A\n",
      "Epoch 17: 100%|█| 63/63 [00:03<00:00, 18.95it/s, loss=0.116, v_num=3, train_acc=\u001b[A\n",
      "Epoch 18:  76%|▊| 48/63 [00:02<00:00, 20.65it/s, loss=0.0519, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 18:  81%|▊| 51/63 [00:02<00:00, 18.73it/s, loss=0.0519, v_num=3, train_acc\u001b[A\n",
      "Epoch 18: 100%|█| 63/63 [00:03<00:00, 18.80it/s, loss=0.0519, v_num=3, train_acc\u001b[A\n",
      "Epoch 19:  76%|▊| 48/63 [00:02<00:00, 20.86it/s, loss=0.0401, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 19:  81%|▊| 51/63 [00:02<00:00, 18.79it/s, loss=0.0401, v_num=3, train_acc\u001b[A\n",
      "Epoch 19: 100%|█| 63/63 [00:03<00:00, 18.80it/s, loss=0.0401, v_num=3, train_acc\u001b[A\n",
      "Epoch 20:  76%|▊| 48/63 [00:02<00:00, 20.23it/s, loss=0.0385, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 20:  81%|▊| 51/63 [00:02<00:00, 18.19it/s, loss=0.0385, v_num=3, train_acc\u001b[A\n",
      "Epoch 20: 100%|█| 63/63 [00:03<00:00, 18.46it/s, loss=0.0385, v_num=3, train_acc\u001b[A\n",
      "Epoch 21:  76%|▊| 48/63 [00:02<00:00, 20.16it/s, loss=0.533, v_num=3, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 21:  81%|▊| 51/63 [00:02<00:00, 18.26it/s, loss=0.533, v_num=3, train_acc=\u001b[A\n",
      "Epoch 21: 100%|█| 63/63 [00:03<00:00, 18.45it/s, loss=0.533, v_num=3, train_acc=\u001b[A\n",
      "Epoch 22:  76%|▊| 48/63 [00:02<00:00, 19.86it/s, loss=0.0419, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 22:  81%|▊| 51/63 [00:02<00:00, 17.98it/s, loss=0.0419, v_num=3, train_acc\u001b[A\n",
      "Epoch 22: 100%|█| 63/63 [00:03<00:00, 18.15it/s, loss=0.0419, v_num=3, train_acc\u001b[A\n",
      "Epoch 23:  76%|▊| 48/63 [00:02<00:00, 20.05it/s, loss=0.0225, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 23:  81%|▊| 51/63 [00:02<00:00, 18.20it/s, loss=0.0225, v_num=3, train_acc\u001b[A\n",
      "Epoch 23: 100%|█| 63/63 [00:03<00:00, 18.32it/s, loss=0.0225, v_num=3, train_acc\u001b[A\n",
      "Epoch 24:  76%|▊| 48/63 [00:02<00:00, 20.55it/s, loss=0.217, v_num=3, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 24:  81%|▊| 51/63 [00:02<00:00, 18.55it/s, loss=0.217, v_num=3, train_acc=\u001b[A\n",
      "Epoch 24: 100%|█| 63/63 [00:03<00:00, 18.61it/s, loss=0.217, v_num=3, train_acc=\u001b[A\n",
      "Epoch 25:  76%|▊| 48/63 [00:02<00:00, 20.13it/s, loss=0.0376, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 25:  81%|▊| 51/63 [00:02<00:00, 18.24it/s, loss=0.0376, v_num=3, train_acc\u001b[A\n",
      "Epoch 25: 100%|█| 63/63 [00:03<00:00, 18.57it/s, loss=0.0376, v_num=3, train_acc\u001b[A\n",
      "Epoch 26:  76%|▊| 48/63 [00:02<00:00, 20.44it/s, loss=0.0398, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 26:  81%|▊| 51/63 [00:02<00:00, 18.37it/s, loss=0.0398, v_num=3, train_acc\u001b[A\n",
      "Epoch 26: 100%|█| 63/63 [00:03<00:00, 18.80it/s, loss=0.0398, v_num=3, train_acc\u001b[A\n",
      "Epoch 27:  76%|▊| 48/63 [00:02<00:00, 20.82it/s, loss=3.13, v_num=3, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 27:  81%|▊| 51/63 [00:02<00:00, 18.79it/s, loss=3.13, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 27: 100%|█| 63/63 [00:03<00:00, 19.05it/s, loss=3.13, v_num=3, train_acc=0\u001b[A\n",
      "Epoch 28:  76%|▊| 48/63 [00:02<00:00, 20.78it/s, loss=0.0784, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 28:  81%|▊| 51/63 [00:02<00:00, 18.80it/s, loss=0.0784, v_num=3, train_acc\u001b[A\n",
      "Epoch 28: 100%|█| 63/63 [00:03<00:00, 18.93it/s, loss=0.0784, v_num=3, train_acc\u001b[A\n",
      "Epoch 29:  76%|▊| 48/63 [00:02<00:00, 21.07it/s, loss=0.0301, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 29:  81%|▊| 51/63 [00:02<00:00, 19.07it/s, loss=0.0301, v_num=3, train_acc\u001b[A\n",
      "Epoch 29: 100%|█| 63/63 [00:03<00:00, 19.13it/s, loss=0.0301, v_num=3, train_acc\u001b[A\n",
      "Epoch 30:  76%|▊| 48/63 [00:02<00:00, 20.59it/s, loss=0.0128, v_num=3, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 30:  81%|▊| 51/63 [00:02<00:00, 18.68it/s, loss=0.0128, v_num=3, train_acc\u001b[A\n",
      "Epoch 30: 100%|█| 63/63 [00:03<00:00, 18.84it/s, loss=0.0128, v_num=3, train_acc\u001b[A\n",
      "Epoch 30: 100%|█| 63/63 [00:03<00:00, 18.73it/s, loss=0.0128, v_num=3, train_acc\u001b[A\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_3/checkpoints/epoch=14-step=704.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_3/checkpoints/epoch=14-step=704.ckpt\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:662: UserWarning: Your `test_dataloader` has `shuffle=True`, it is strongly recommended that you turn this off for val/test/predict dataloaders.\n",
      "  category=UserWarning,\n",
      "Testing:  83%|████████████████████████████▏     | 39/47 [00:00<00:00, 84.54it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.996666669845581, 'test_loss': 0.02135266177356243}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 47/47 [00:02<00:00, 22.11it/s]\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_3/checkpoints/epoch=14-step=704.ckpt\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_3/checkpoints/epoch=14-step=704.ckpt\n",
      "Testing: 100%|██████████████████████████████████| 16/16 [00:00<00:00, 36.09it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9620000123977661, 'test_loss': 0.14162316918373108}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 16/16 [00:01<00:00, 15.29it/s]\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.teardown has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.teardown.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_3/checkpoints/epoch=14-step=704.ckpt\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_3/checkpoints/epoch=14-step=704.ckpt\n",
      "Testing: 100%|███████████████████████████████| 157/157 [00:01<00:00, 136.93it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9581000208854675, 'test_loss': 0.14859822392463684}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|████████████████████████████████| 157/157 [00:05<00:00, 26.47it/s]\n",
      "train_result:\n",
      "\n",
      "[{'test_acc': 0.996666669845581, 'test_loss': 0.02135266177356243}]\n",
      "val_result:\n",
      "\n",
      "[{'test_acc': 0.9620000123977661, 'test_loss': 0.14162316918373108}]\n",
      "test_result:\n",
      "\n",
      "[{'test_acc': 0.9581000208854675, 'test_loss': 0.14859822392463684}]\n",
      "train_result:\n",
      "\n",
      "[{'test_acc': 0.996666669845581, 'test_loss': 0.02135266177356243}]\n",
      "val_result:\n",
      "\n",
      "[{'test_acc': 0.9620000123977661, 'test_loss': 0.14162316918373108}]\n",
      "test_result:\n",
      "\n",
      "[{'test_acc': 0.9581000208854675, 'test_loss': 0.14859822392463684}]\n"
     ]
    }
   ],
   "source": [
    "#平凡方式训练\n",
    "!python3 mnist_cnn.py --gpus=2 --strategy=\"ddp_find_unused_parameters_false\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04fd2e75",
   "metadata": {},
   "source": [
    "```\n",
    "------------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9581000208854675, 'test_loss': 0.14859822392463684}\n",
    "--------------------------------------------------------------------------------\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeee80a6-5e08-4f70-a255-87f17a6a8cf0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4623555b-ca77-4cbe-83d5-f6b0e28cd303",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "id": "46a5f55b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2\n",
      "----------------------------------------------------------------------------------------------------\n",
      "distributed_backend=nccl\n",
      "All distributed processes registered. Starting with 2 processes\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Global seed set to 1234\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (47) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "Epoch 0:  75%|▋| 47/63 [00:02<00:00, 16.73it/s, loss=2.21, v_num=5, train_acc=0.\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  78%|▊| 49/63 [00:03<00:00, 14.77it/s, loss=2.21, v_num=5, train_acc=0.\u001b[A\n",
      "Epoch 0: 100%|█| 63/63 [00:03<00:00, 15.85it/s, loss=2.21, v_num=5, train_acc=0.\u001b[A\n",
      "Epoch 1:  76%|▊| 48/63 [00:02<00:00, 18.82it/s, loss=1.93, v_num=5, train_acc=0.\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  81%|▊| 51/63 [00:03<00:00, 16.69it/s, loss=1.93, v_num=5, train_acc=0.\u001b[A\n",
      "Epoch 1: 100%|█| 63/63 [00:03<00:00, 17.12it/s, loss=1.93, v_num=5, train_acc=0.\u001b[A\n",
      "Epoch 2:  76%|▊| 48/63 [00:02<00:00, 18.84it/s, loss=1.29, v_num=5, train_acc=0.\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  81%|▊| 51/63 [00:03<00:00, 16.67it/s, loss=1.29, v_num=5, train_acc=0.\u001b[A\n",
      "Epoch 2: 100%|█| 63/63 [00:03<00:00, 16.98it/s, loss=1.29, v_num=5, train_acc=0.\u001b[A\n",
      "Epoch 3:  76%|▊| 48/63 [00:02<00:00, 18.84it/s, loss=0.801, v_num=5, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 3:  81%|▊| 51/63 [00:03<00:00, 16.53it/s, loss=0.801, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 3: 100%|█| 63/63 [00:03<00:00, 16.88it/s, loss=0.801, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 4:  76%|▊| 48/63 [00:02<00:00, 19.13it/s, loss=0.486, v_num=5, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 4:  81%|▊| 51/63 [00:03<00:00, 16.92it/s, loss=0.486, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 4: 100%|█| 63/63 [00:03<00:00, 17.19it/s, loss=0.486, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 5:  76%|▊| 48/63 [00:02<00:00, 18.84it/s, loss=0.247, v_num=5, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 5:  81%|▊| 51/63 [00:03<00:00, 16.74it/s, loss=0.247, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 5: 100%|█| 63/63 [00:03<00:00, 17.09it/s, loss=0.247, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 6:  76%|▊| 48/63 [00:02<00:00, 18.74it/s, loss=0.448, v_num=5, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 6:  81%|▊| 51/63 [00:03<00:00, 16.57it/s, loss=0.448, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 6: 100%|█| 63/63 [00:03<00:00, 16.92it/s, loss=0.448, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 7:  76%|▊| 48/63 [00:02<00:00, 18.69it/s, loss=0.192, v_num=5, train_acc=1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 7:  81%|▊| 51/63 [00:03<00:00, 16.49it/s, loss=0.192, v_num=5, train_acc=1\u001b[A\n",
      "Epoch 7: 100%|█| 63/63 [00:03<00:00, 16.85it/s, loss=0.192, v_num=5, train_acc=1\u001b[A\n",
      "Epoch 8:  76%|▊| 48/63 [00:02<00:00, 18.52it/s, loss=0.181, v_num=5, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 8:  81%|▊| 51/63 [00:03<00:00, 16.41it/s, loss=0.181, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 8: 100%|█| 63/63 [00:03<00:00, 16.88it/s, loss=0.181, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 9:  76%|▊| 48/63 [00:02<00:00, 18.79it/s, loss=0.599, v_num=5, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 9:  81%|▊| 51/63 [00:03<00:00, 16.62it/s, loss=0.599, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 9: 100%|█| 63/63 [00:03<00:00, 17.08it/s, loss=0.599, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 10:  76%|▊| 48/63 [00:02<00:00, 18.67it/s, loss=0.0882, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 10:  81%|▊| 51/63 [00:03<00:00, 16.45it/s, loss=0.0882, v_num=5, train_acc\u001b[A\n",
      "Epoch 10: 100%|█| 63/63 [00:03<00:00, 16.88it/s, loss=0.0882, v_num=5, train_acc\u001b[A\n",
      "Epoch 11:  76%|▊| 48/63 [00:02<00:00, 18.73it/s, loss=0.0923, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 11:  81%|▊| 51/63 [00:03<00:00, 16.55it/s, loss=0.0923, v_num=5, train_acc\u001b[A\n",
      "Epoch 11: 100%|█| 63/63 [00:03<00:00, 16.79it/s, loss=0.0923, v_num=5, train_acc\u001b[A\n",
      "Epoch 12:  76%|▊| 48/63 [00:02<00:00, 18.52it/s, loss=0.124, v_num=5, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 12:  81%|▊| 51/63 [00:03<00:00, 16.39it/s, loss=0.124, v_num=5, train_acc=\u001b[A\n",
      "Epoch 12: 100%|█| 63/63 [00:03<00:00, 16.70it/s, loss=0.124, v_num=5, train_acc=\u001b[A\n",
      "Epoch 13:  76%|▊| 48/63 [00:02<00:00, 18.48it/s, loss=0.271, v_num=5, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 13:  81%|▊| 51/63 [00:03<00:00, 16.28it/s, loss=0.271, v_num=5, train_acc=\u001b[A\n",
      "Epoch 13: 100%|█| 63/63 [00:03<00:00, 16.78it/s, loss=0.271, v_num=5, train_acc=\u001b[A\n",
      "Epoch 14:  76%|▊| 48/63 [00:02<00:00, 18.56it/s, loss=0.2, v_num=5, train_acc=1.\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 14:  81%|▊| 51/63 [00:03<00:00, 16.45it/s, loss=0.2, v_num=5, train_acc=1.\u001b[A\n",
      "Epoch 14: 100%|█| 63/63 [00:03<00:00, 16.83it/s, loss=0.2, v_num=5, train_acc=1.\u001b[A\n",
      "Epoch 15:  76%|▊| 48/63 [00:02<00:00, 19.29it/s, loss=0.076, v_num=5, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 15:  81%|▊| 51/63 [00:02<00:00, 17.05it/s, loss=0.076, v_num=5, train_acc=\u001b[A\n",
      "Epoch 15: 100%|█| 63/63 [00:03<00:00, 17.32it/s, loss=0.076, v_num=5, train_acc=\u001b[A\n",
      "Epoch 16:  76%|▊| 48/63 [00:02<00:00, 19.44it/s, loss=0.11, v_num=5, train_acc=1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 16:  81%|▊| 51/63 [00:02<00:00, 17.10it/s, loss=0.11, v_num=5, train_acc=1\u001b[A\n",
      "Epoch 16: 100%|█| 63/63 [00:03<00:00, 17.49it/s, loss=0.11, v_num=5, train_acc=1\u001b[A\n",
      "Epoch 17:  76%|▊| 48/63 [00:02<00:00, 19.36it/s, loss=0.131, v_num=5, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 17:  81%|▊| 51/63 [00:02<00:00, 17.08it/s, loss=0.131, v_num=5, train_acc=\u001b[A\n",
      "Epoch 17: 100%|█| 63/63 [00:03<00:00, 17.31it/s, loss=0.131, v_num=5, train_acc=\u001b[A\n",
      "Epoch 18:  76%|▊| 48/63 [00:02<00:00, 19.35it/s, loss=0.0798, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 18:  81%|▊| 51/63 [00:02<00:00, 17.14it/s, loss=0.0798, v_num=5, train_acc\u001b[A\n",
      "Epoch 18: 100%|█| 63/63 [00:03<00:00, 17.44it/s, loss=0.0798, v_num=5, train_acc\u001b[A\n",
      "Epoch 19:  76%|▊| 48/63 [00:02<00:00, 19.30it/s, loss=0.0513, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 19:  81%|▊| 51/63 [00:03<00:00, 16.99it/s, loss=0.0513, v_num=5, train_acc\u001b[A\n",
      "Epoch 19: 100%|█| 63/63 [00:03<00:00, 17.28it/s, loss=0.0513, v_num=5, train_acc\u001b[A\n",
      "Epoch 20:  76%|▊| 48/63 [00:02<00:00, 18.89it/s, loss=0.158, v_num=5, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 20:  81%|▊| 51/63 [00:03<00:00, 16.61it/s, loss=0.158, v_num=5, train_acc=\u001b[A\n",
      "Epoch 20: 100%|█| 63/63 [00:03<00:00, 16.89it/s, loss=0.158, v_num=5, train_acc=\u001b[A\n",
      "Epoch 21:  76%|▊| 48/63 [00:02<00:00, 18.67it/s, loss=0.0401, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 21:  81%|▊| 51/63 [00:03<00:00, 16.57it/s, loss=0.0401, v_num=5, train_acc\u001b[A\n",
      "Epoch 21: 100%|█| 63/63 [00:03<00:00, 16.93it/s, loss=0.0401, v_num=5, train_acc\u001b[A\n",
      "Epoch 22:  76%|▊| 48/63 [00:02<00:00, 18.54it/s, loss=0.0472, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 22:  81%|▊| 51/63 [00:03<00:00, 16.38it/s, loss=0.0472, v_num=5, train_acc\u001b[A\n",
      "Epoch 22: 100%|█| 63/63 [00:03<00:00, 16.78it/s, loss=0.0472, v_num=5, train_acc\u001b[A\n",
      "Epoch 23:  76%|▊| 48/63 [00:02<00:00, 18.60it/s, loss=0.0338, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 23:  81%|▊| 51/63 [00:03<00:00, 16.31it/s, loss=0.0338, v_num=5, train_acc\u001b[A\n",
      "Epoch 23: 100%|█| 63/63 [00:03<00:00, 16.85it/s, loss=0.0338, v_num=5, train_acc\u001b[A\n",
      "Epoch 24:  76%|▊| 48/63 [00:02<00:00, 18.66it/s, loss=0.0156, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 24:  81%|▊| 51/63 [00:03<00:00, 16.51it/s, loss=0.0156, v_num=5, train_acc\u001b[A\n",
      "Epoch 24: 100%|█| 63/63 [00:03<00:00, 16.99it/s, loss=0.0156, v_num=5, train_acc\u001b[A\n",
      "Epoch 25:  76%|▊| 48/63 [00:02<00:00, 18.61it/s, loss=0.049, v_num=5, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 25:  81%|▊| 51/63 [00:03<00:00, 16.24it/s, loss=0.049, v_num=5, train_acc=\u001b[A\n",
      "Epoch 25: 100%|█| 63/63 [00:03<00:00, 16.58it/s, loss=0.049, v_num=5, train_acc=\u001b[A\n",
      "Epoch 26:  76%|▊| 48/63 [00:02<00:00, 18.59it/s, loss=0.0388, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 26:  81%|▊| 51/63 [00:03<00:00, 16.42it/s, loss=0.0388, v_num=5, train_acc\u001b[A\n",
      "Epoch 26: 100%|█| 63/63 [00:03<00:00, 16.78it/s, loss=0.0388, v_num=5, train_acc\u001b[A\n",
      "Epoch 27:  76%|▊| 48/63 [00:02<00:00, 18.68it/s, loss=0.0194, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 27:  81%|▊| 51/63 [00:03<00:00, 16.55it/s, loss=0.0194, v_num=5, train_acc\u001b[A\n",
      "Epoch 27: 100%|█| 63/63 [00:03<00:00, 17.03it/s, loss=0.0194, v_num=5, train_acc\u001b[A\n",
      "Epoch 28:  76%|▊| 48/63 [00:02<00:00, 18.86it/s, loss=0.0535, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 28:  81%|▊| 51/63 [00:03<00:00, 16.73it/s, loss=0.0535, v_num=5, train_acc\u001b[A\n",
      "Epoch 28: 100%|█| 63/63 [00:03<00:00, 16.91it/s, loss=0.0535, v_num=5, train_acc\u001b[A\n",
      "Epoch 29:  76%|▊| 48/63 [00:02<00:00, 19.45it/s, loss=1.83, v_num=5, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 29:  81%|▊| 51/63 [00:02<00:00, 17.18it/s, loss=1.83, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 29: 100%|█| 63/63 [00:03<00:00, 17.40it/s, loss=1.83, v_num=5, train_acc=0\u001b[A\n",
      "Epoch 30:  76%|▊| 48/63 [00:02<00:00, 18.97it/s, loss=0.0553, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 30:  81%|▊| 51/63 [00:03<00:00, 16.79it/s, loss=0.0553, v_num=5, train_acc\u001b[A\n",
      "Epoch 30: 100%|█| 63/63 [00:03<00:00, 17.24it/s, loss=0.0553, v_num=5, train_acc\u001b[A\n",
      "Epoch 31:  76%|▊| 48/63 [00:02<00:00, 19.26it/s, loss=0.0287, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 31:  81%|▊| 51/63 [00:02<00:00, 17.04it/s, loss=0.0287, v_num=5, train_acc\u001b[A\n",
      "Epoch 31: 100%|█| 63/63 [00:03<00:00, 17.26it/s, loss=0.0287, v_num=5, train_acc\u001b[A\n",
      "Epoch 32:  76%|▊| 48/63 [00:02<00:00, 19.32it/s, loss=0.0206, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 32:  81%|▊| 51/63 [00:02<00:00, 17.11it/s, loss=0.0206, v_num=5, train_acc\u001b[A\n",
      "Epoch 32: 100%|█| 63/63 [00:03<00:00, 17.36it/s, loss=0.0206, v_num=5, train_acc\u001b[A\n",
      "Epoch 33:  76%|▊| 48/63 [00:02<00:00, 18.59it/s, loss=0.0221, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 33:  81%|▊| 51/63 [00:03<00:00, 16.36it/s, loss=0.0221, v_num=5, train_acc\u001b[A\n",
      "Epoch 33: 100%|█| 63/63 [00:03<00:00, 16.84it/s, loss=0.0221, v_num=5, train_acc\u001b[A\n",
      "Epoch 34:  76%|▊| 48/63 [00:02<00:00, 18.70it/s, loss=0.0704, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 34:  81%|▊| 51/63 [00:03<00:00, 16.29it/s, loss=0.0704, v_num=5, train_acc\u001b[A\n",
      "Epoch 34: 100%|█| 63/63 [00:03<00:00, 17.00it/s, loss=0.0704, v_num=5, train_acc\u001b[A\n",
      "Epoch 35:  76%|▊| 48/63 [00:02<00:00, 18.79it/s, loss=0.0251, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 35:  81%|▊| 51/63 [00:03<00:00, 16.67it/s, loss=0.0251, v_num=5, train_acc\u001b[A\n",
      "Epoch 35: 100%|█| 63/63 [00:03<00:00, 16.93it/s, loss=0.0251, v_num=5, train_acc\u001b[A\n",
      "Epoch 36:  76%|▊| 48/63 [00:02<00:00, 18.62it/s, loss=0.0133, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 36:  81%|▊| 51/63 [00:03<00:00, 16.27it/s, loss=0.0133, v_num=5, train_acc\u001b[A\n",
      "Epoch 36: 100%|█| 63/63 [00:03<00:00, 16.89it/s, loss=0.0133, v_num=5, train_acc\u001b[A\n",
      "Epoch 37:  76%|▊| 48/63 [00:02<00:00, 18.64it/s, loss=0.0389, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 37:  81%|▊| 51/63 [00:03<00:00, 16.43it/s, loss=0.0389, v_num=5, train_acc\u001b[A\n",
      "Epoch 37: 100%|█| 63/63 [00:03<00:00, 16.82it/s, loss=0.0389, v_num=5, train_acc\u001b[A\n",
      "Epoch 38:  76%|▊| 48/63 [00:02<00:00, 18.71it/s, loss=0.0219, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 38:  81%|▊| 51/63 [00:03<00:00, 16.52it/s, loss=0.0219, v_num=5, train_acc\u001b[A\n",
      "Epoch 38: 100%|█| 63/63 [00:03<00:00, 16.95it/s, loss=0.0219, v_num=5, train_acc\u001b[A\n",
      "Epoch 39:  76%|▊| 48/63 [00:02<00:00, 18.49it/s, loss=1.46, v_num=5, train_acc=1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 39:  81%|▊| 51/63 [00:03<00:00, 16.40it/s, loss=1.46, v_num=5, train_acc=1\u001b[A\n",
      "Epoch 39: 100%|█| 63/63 [00:03<00:00, 16.67it/s, loss=1.46, v_num=5, train_acc=1\u001b[A\n",
      "Epoch 40:  76%|▊| 48/63 [00:02<00:00, 18.82it/s, loss=0.0498, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 40:  81%|▊| 51/63 [00:03<00:00, 16.63it/s, loss=0.0498, v_num=5, train_acc\u001b[A\n",
      "Epoch 40: 100%|█| 63/63 [00:03<00:00, 17.13it/s, loss=0.0498, v_num=5, train_acc\u001b[A\n",
      "Epoch 41:  76%|▊| 48/63 [00:02<00:00, 18.41it/s, loss=0.0128, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 41:  81%|▊| 51/63 [00:03<00:00, 16.24it/s, loss=0.0128, v_num=5, train_acc\u001b[A\n",
      "Epoch 41: 100%|█| 63/63 [00:03<00:00, 16.69it/s, loss=0.0128, v_num=5, train_acc\u001b[A\n",
      "Epoch 42:  76%|▊| 48/63 [00:02<00:00, 18.94it/s, loss=0.0136, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 42:  81%|▊| 51/63 [00:03<00:00, 16.71it/s, loss=0.0136, v_num=5, train_acc\u001b[A\n",
      "Epoch 42: 100%|█| 63/63 [00:03<00:00, 17.30it/s, loss=0.0136, v_num=5, train_acc\u001b[A\n",
      "Epoch 43:  76%|▊| 48/63 [00:02<00:00, 19.06it/s, loss=0.0176, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 43:  81%|▊| 51/63 [00:03<00:00, 16.90it/s, loss=0.0176, v_num=5, train_acc\u001b[A\n",
      "Epoch 43: 100%|█| 63/63 [00:03<00:00, 17.17it/s, loss=0.0176, v_num=5, train_acc\u001b[A\n",
      "Epoch 44:  76%|▊| 48/63 [00:02<00:00, 19.33it/s, loss=0.131, v_num=5, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 44:  81%|▊| 51/63 [00:03<00:00, 16.98it/s, loss=0.131, v_num=5, train_acc=\u001b[A\n",
      "Epoch 44: 100%|█| 63/63 [00:03<00:00, 17.36it/s, loss=0.131, v_num=5, train_acc=\u001b[A\n",
      "Epoch 45:  76%|▊| 48/63 [00:02<00:00, 19.59it/s, loss=0.0284, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 45:  81%|▊| 51/63 [00:02<00:00, 17.27it/s, loss=0.0284, v_num=5, train_acc\u001b[A\n",
      "Epoch 45: 100%|█| 63/63 [00:03<00:00, 17.51it/s, loss=0.0284, v_num=5, train_acc\u001b[A\n",
      "Epoch 46:  76%|▊| 48/63 [00:02<00:00, 19.21it/s, loss=0.684, v_num=5, train_acc=\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 46:  81%|▊| 51/63 [00:03<00:00, 16.99it/s, loss=0.684, v_num=5, train_acc=\u001b[A\n",
      "Epoch 46: 100%|█| 63/63 [00:03<00:00, 17.40it/s, loss=0.684, v_num=5, train_acc=\u001b[A\n",
      "Epoch 47:  76%|▊| 48/63 [00:02<00:00, 19.03it/s, loss=0.0317, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 47:  81%|▊| 51/63 [00:03<00:00, 16.77it/s, loss=0.0317, v_num=5, train_acc\u001b[A\n",
      "Epoch 47: 100%|█| 63/63 [00:03<00:00, 17.16it/s, loss=0.0317, v_num=5, train_acc\u001b[A\n",
      "Epoch 48:  76%|▊| 48/63 [00:02<00:00, 19.03it/s, loss=0.00829, v_num=5, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 48:  81%|▊| 51/63 [00:03<00:00, 16.89it/s, loss=0.00829, v_num=5, train_ac\u001b[A\n",
      "Epoch 48: 100%|█| 63/63 [00:03<00:00, 16.98it/s, loss=0.00829, v_num=5, train_ac\u001b[A\n",
      "Epoch 49:  76%|▊| 48/63 [00:02<00:00, 19.09it/s, loss=0.00588, v_num=5, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 49:  81%|▊| 51/63 [00:03<00:00, 16.77it/s, loss=0.00588, v_num=5, train_ac\u001b[A\n",
      "Epoch 49: 100%|█| 63/63 [00:03<00:00, 17.24it/s, loss=0.00588, v_num=5, train_ac\u001b[A\n",
      "Epoch 50:  76%|▊| 48/63 [00:02<00:00, 18.79it/s, loss=0.0282, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 50:  81%|▊| 51/63 [00:03<00:00, 16.64it/s, loss=0.0282, v_num=5, train_acc\u001b[A\n",
      "Epoch 50: 100%|█| 63/63 [00:03<00:00, 17.06it/s, loss=0.0282, v_num=5, train_acc\u001b[A\n",
      "Epoch 51:  76%|▊| 48/63 [00:02<00:00, 18.93it/s, loss=0.0103, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 51:  81%|▊| 51/63 [00:03<00:00, 16.79it/s, loss=0.0103, v_num=5, train_acc\u001b[A\n",
      "Epoch 51: 100%|█| 63/63 [00:03<00:00, 17.00it/s, loss=0.0103, v_num=5, train_acc\u001b[A\n",
      "Epoch 52:  76%|▊| 48/63 [00:02<00:00, 19.02it/s, loss=0.0122, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 52:  81%|▊| 51/63 [00:03<00:00, 16.23it/s, loss=0.0122, v_num=5, train_acc\u001b[A\n",
      "Epoch 52: 100%|█| 63/63 [00:03<00:00, 17.08it/s, loss=0.0122, v_num=5, train_acc\u001b[A\n",
      "Epoch 53:  76%|▊| 48/63 [00:02<00:00, 17.87it/s, loss=0.0657, v_num=5, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 53:  81%|▊| 51/63 [00:03<00:00, 16.00it/s, loss=0.0657, v_num=5, train_acc\u001b[A\n",
      "Epoch 53: 100%|█| 63/63 [00:03<00:00, 16.36it/s, loss=0.0657, v_num=5, train_acc\u001b[A\n",
      "Epoch 53: 100%|█| 63/63 [00:03<00:00, 16.27it/s, loss=0.0657, v_num=5, train_acc\u001b[A\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_5/checkpoints/epoch=37-step=1785.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_5/checkpoints/epoch=37-step=1785.ckpt\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:662: UserWarning: Your `test_dataloader` has `shuffle=True`, it is strongly recommended that you turn this off for val/test/predict dataloaders.\n",
      "  category=UserWarning,\n",
      "Testing:  85%|████████████████████████████▉     | 40/47 [00:00<00:00, 78.94it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9993333220481873, 'test_loss': 0.004299486521631479}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 47/47 [00:02<00:00, 20.94it/s]\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_5/checkpoints/epoch=37-step=1785.ckpt\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_5/checkpoints/epoch=37-step=1785.ckpt\n",
      "Testing: 100%|██████████████████████████████████| 16/16 [00:00<00:00, 31.38it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9700000286102295, 'test_loss': 0.1477910727262497}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 16/16 [00:01<00:00, 14.15it/s]\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.teardown has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.teardown.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_5/checkpoints/epoch=37-step=1785.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_5/checkpoints/epoch=37-step=1785.ckpt\n",
      "Testing:  95%|█████████████████████████████▍ | 149/157 [00:01<00:00, 166.43it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.963100016117096, 'test_loss': 0.18146753311157227}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|████████████████████████████████| 157/157 [00:06<00:00, 25.87it/s]\n",
      "train_result:\n",
      "\n",
      "[{'test_acc': 0.9993333220481873, 'test_loss': 0.004299486521631479}]\n",
      "val_result:\n",
      "\n",
      "[{'test_acc': 0.9700000286102295, 'test_loss': 0.1477910727262497}]\n",
      "test_result:\n",
      "\n",
      "[{'test_acc': 0.963100016117096, 'test_loss': 0.18146753311157227}]\n",
      "train_result:\n",
      "\n",
      "[{'test_acc': 0.9993333220481873, 'test_loss': 0.004299486521631479}]\n",
      "val_result:\n",
      "\n",
      "[{'test_acc': 0.9700000286102295, 'test_loss': 0.1477910727262497}]\n",
      "test_result:\n",
      "\n",
      "[{'test_acc': 0.963100016117096, 'test_loss': 0.18146753311157227}]\n"
     ]
    }
   ],
   "source": [
    "#使用SWA随机权重\n",
    "!python3 mnist_cnn.py --gpus=2 --strategy=\"ddp_find_unused_parameters_false\" --use_swa=True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "682b75fe",
   "metadata": {},
   "source": [
    "```\n",
    "-----------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.963100016117096, 'test_loss': 0.18146753311157227}\n",
    "--------------------------------------------------------------------------------\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71d115f7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b9f16e11-6035-4f57-9576-619ca41939a3",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2，CyclicLR学习率调度策略（torch.optim.lr_scheduler.CyclicLR）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58a2c037-e15a-4631-b5b5-9af87a51f82e",
   "metadata": {},
   "source": [
    "* 平凡方式训练：test_acc = 0.9581000208854675\n",
    "\n",
    "* SWA随机权重： test_acc = 0.963100016117096\n",
    "\n",
    "* SWA随机权重 + CyClicLR学习率调度策略: test_acc = 0.9688000082969666"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a90ef18-abed-4d6e-b388-e20b7d3630e3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "id": "c39a6a28-9c38-4a61-940f-314b7611737c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2\n",
      "Global seed set to 1234\n",
      "initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2\n",
      "----------------------------------------------------------------------------------------------------\n",
      "distributed_backend=nccl\n",
      "All distributed processes registered. Starting with 2 processes\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Global seed set to 1234\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:433: UserWarning: The number of training samples (47) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  f\"The number of training samples ({self.num_training_batches}) is smaller than the logging interval\"\n",
      "Epoch 0:  75%|▋| 47/63 [00:02<00:00, 17.80it/s, loss=1.28, v_num=6, train_acc=0.\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  78%|▊| 49/63 [00:03<00:00, 15.63it/s, loss=1.28, v_num=6, train_acc=0.\u001b[A\n",
      "Epoch 0: 100%|█| 63/63 [00:03<00:00, 16.65it/s, loss=1.28, v_num=6, train_acc=0.\u001b[A\n",
      "Epoch 1:  76%|▊| 48/63 [00:02<00:00, 19.68it/s, loss=0.544, v_num=6, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  81%|▊| 51/63 [00:02<00:00, 17.35it/s, loss=0.544, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 1: 100%|█| 63/63 [00:03<00:00, 17.61it/s, loss=0.544, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 2:  76%|▊| 48/63 [00:02<00:00, 19.70it/s, loss=0.305, v_num=6, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  81%|▊| 51/63 [00:02<00:00, 17.32it/s, loss=0.305, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 2: 100%|█| 63/63 [00:03<00:00, 17.82it/s, loss=0.305, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 3:  76%|▊| 48/63 [00:02<00:00, 19.04it/s, loss=0.233, v_num=6, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 3:  81%|▊| 51/63 [00:03<00:00, 16.81it/s, loss=0.233, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 3: 100%|█| 63/63 [00:03<00:00, 17.25it/s, loss=0.233, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 4:  76%|▊| 48/63 [00:02<00:00, 19.13it/s, loss=0.188, v_num=6, train_acc=1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 4:  81%|▊| 51/63 [00:03<00:00, 16.91it/s, loss=0.188, v_num=6, train_acc=1\u001b[A\n",
      "Epoch 4: 100%|█| 63/63 [00:03<00:00, 17.36it/s, loss=0.188, v_num=6, train_acc=1\u001b[A\n",
      "Epoch 5:  76%|▊| 48/63 [00:02<00:00, 18.86it/s, loss=0.137, v_num=6, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 5:  81%|▊| 51/63 [00:03<00:00, 16.70it/s, loss=0.137, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 5: 100%|█| 63/63 [00:03<00:00, 17.03it/s, loss=0.137, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 6:  76%|▊| 48/63 [00:02<00:00, 19.12it/s, loss=0.205, v_num=6, train_acc=1\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 6:  81%|▊| 51/63 [00:03<00:00, 16.83it/s, loss=0.205, v_num=6, train_acc=1\u001b[A\n",
      "Epoch 6: 100%|█| 63/63 [00:03<00:00, 17.29it/s, loss=0.205, v_num=6, train_acc=1\u001b[A\n",
      "Epoch 7:  76%|▊| 48/63 [00:02<00:00, 18.86it/s, loss=0.111, v_num=6, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 7:  81%|▊| 51/63 [00:03<00:00, 16.71it/s, loss=0.111, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 7: 100%|█| 63/63 [00:03<00:00, 17.16it/s, loss=0.111, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 8:  76%|▊| 48/63 [00:02<00:00, 19.09it/s, loss=0.137, v_num=6, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 8:  81%|▊| 51/63 [00:03<00:00, 16.92it/s, loss=0.137, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 8: 100%|█| 63/63 [00:03<00:00, 17.38it/s, loss=0.137, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 9:  76%|▊| 48/63 [00:02<00:00, 19.07it/s, loss=0.067, v_num=6, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 9:  81%|▊| 51/63 [00:03<00:00, 16.99it/s, loss=0.067, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 9: 100%|█| 63/63 [00:03<00:00, 16.96it/s, loss=0.067, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 10:  76%|▊| 48/63 [00:02<00:00, 18.93it/s, loss=0.0757, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 10:  81%|▊| 51/63 [00:03<00:00, 16.80it/s, loss=0.0757, v_num=6, train_acc\u001b[A\n",
      "Epoch 10: 100%|█| 63/63 [00:03<00:00, 17.18it/s, loss=0.0757, v_num=6, train_acc\u001b[A\n",
      "Epoch 11:  76%|▊| 48/63 [00:02<00:00, 19.80it/s, loss=0.0624, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 11:  81%|▊| 51/63 [00:02<00:00, 17.48it/s, loss=0.0624, v_num=6, train_acc\u001b[A\n",
      "Epoch 11: 100%|█| 63/63 [00:03<00:00, 17.86it/s, loss=0.0624, v_num=6, train_acc\u001b[A\n",
      "Epoch 12:  76%|▊| 48/63 [00:02<00:00, 18.77it/s, loss=0.0417, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 12:  81%|▊| 51/63 [00:03<00:00, 16.60it/s, loss=0.0417, v_num=6, train_acc\u001b[A\n",
      "Epoch 12: 100%|█| 63/63 [00:03<00:00, 16.99it/s, loss=0.0417, v_num=6, train_acc\u001b[A\n",
      "Epoch 13:  76%|▊| 48/63 [00:02<00:00, 19.00it/s, loss=0.0757, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 13:  81%|▊| 51/63 [00:03<00:00, 16.78it/s, loss=0.0757, v_num=6, train_acc\u001b[A\n",
      "Epoch 13: 100%|█| 63/63 [00:03<00:00, 17.11it/s, loss=0.0757, v_num=6, train_acc\u001b[A\n",
      "Epoch 14:  76%|▊| 48/63 [00:02<00:00, 19.14it/s, loss=0.0935, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 14:  81%|▊| 51/63 [00:03<00:00, 16.93it/s, loss=0.0935, v_num=6, train_acc\u001b[A\n",
      "Epoch 14: 100%|█| 63/63 [00:03<00:00, 17.20it/s, loss=0.0935, v_num=6, train_acc\u001b[A\n",
      "Epoch 15:  76%|▊| 48/63 [00:02<00:00, 18.81it/s, loss=0.0433, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 15:  81%|▊| 51/63 [00:03<00:00, 16.67it/s, loss=0.0433, v_num=6, train_acc\u001b[A\n",
      "Epoch 15: 100%|█| 63/63 [00:03<00:00, 17.01it/s, loss=0.0433, v_num=6, train_acc\u001b[A\n",
      "Epoch 16:  76%|▊| 48/63 [00:02<00:00, 18.93it/s, loss=0.0359, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 16:  81%|▊| 51/63 [00:03<00:00, 16.68it/s, loss=0.0359, v_num=6, train_acc\u001b[A\n",
      "Epoch 16: 100%|█| 63/63 [00:03<00:00, 17.11it/s, loss=0.0359, v_num=6, train_acc\u001b[A\n",
      "Epoch 17:  76%|▊| 48/63 [00:02<00:00, 18.93it/s, loss=0.0191, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 17:  81%|▊| 51/63 [00:03<00:00, 16.72it/s, loss=0.0191, v_num=6, train_acc\u001b[A\n",
      "Epoch 17: 100%|█| 63/63 [00:03<00:00, 17.14it/s, loss=0.0191, v_num=6, train_acc\u001b[A\n",
      "Epoch 18:  76%|▊| 48/63 [00:02<00:00, 18.83it/s, loss=0.0576, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 18:  81%|▊| 51/63 [00:03<00:00, 16.55it/s, loss=0.0576, v_num=6, train_acc\u001b[A\n",
      "Epoch 18: 100%|█| 63/63 [00:03<00:00, 16.88it/s, loss=0.0576, v_num=6, train_acc\u001b[A\n",
      "Epoch 19:  76%|▊| 48/63 [00:02<00:00, 18.55it/s, loss=0.0218, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 19:  81%|▊| 51/63 [00:03<00:00, 16.38it/s, loss=0.0218, v_num=6, train_acc\u001b[A\n",
      "Epoch 19: 100%|█| 63/63 [00:03<00:00, 16.65it/s, loss=0.0218, v_num=6, train_acc\u001b[A\n",
      "Epoch 20:  76%|▊| 48/63 [00:02<00:00, 18.66it/s, loss=0.0469, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 20:  81%|▊| 51/63 [00:03<00:00, 16.51it/s, loss=0.0469, v_num=6, train_acc\u001b[A\n",
      "Epoch 20: 100%|█| 63/63 [00:03<00:00, 16.87it/s, loss=0.0469, v_num=6, train_acc\u001b[A\n",
      "Epoch 21:  76%|▊| 48/63 [00:02<00:00, 18.70it/s, loss=0.0203, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 21:  81%|▊| 51/63 [00:03<00:00, 16.61it/s, loss=0.0203, v_num=6, train_acc\u001b[A\n",
      "Epoch 21: 100%|█| 63/63 [00:03<00:00, 17.00it/s, loss=0.0203, v_num=6, train_acc\u001b[A\n",
      "Epoch 22:  76%|▊| 48/63 [00:02<00:00, 18.84it/s, loss=0.0164, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 22:  81%|▊| 51/63 [00:03<00:00, 16.58it/s, loss=0.0164, v_num=6, train_acc\u001b[A\n",
      "Epoch 22: 100%|█| 63/63 [00:03<00:00, 16.86it/s, loss=0.0164, v_num=6, train_acc\u001b[A\n",
      "Epoch 23:  76%|▊| 48/63 [00:02<00:00, 18.53it/s, loss=0.0107, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 23:  81%|▊| 51/63 [00:03<00:00, 16.33it/s, loss=0.0107, v_num=6, train_acc\u001b[A\n",
      "Epoch 23: 100%|█| 63/63 [00:03<00:00, 16.99it/s, loss=0.0107, v_num=6, train_acc\u001b[A\n",
      "Epoch 24:  76%|▊| 48/63 [00:02<00:00, 18.70it/s, loss=0.0298, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 24:  81%|▊| 51/63 [00:03<00:00, 16.57it/s, loss=0.0298, v_num=6, train_acc\u001b[A\n",
      "Epoch 24: 100%|█| 63/63 [00:03<00:00, 16.98it/s, loss=0.0298, v_num=6, train_acc\u001b[A\n",
      "Epoch 25:  76%|▊| 48/63 [00:02<00:00, 18.73it/s, loss=0.00768, v_num=6, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 25:  81%|▊| 51/63 [00:03<00:00, 16.61it/s, loss=0.00768, v_num=6, train_ac\u001b[A\n",
      "Epoch 25: 100%|█| 63/63 [00:03<00:00, 17.01it/s, loss=0.00768, v_num=6, train_ac\u001b[A\n",
      "Epoch 26:  76%|▊| 48/63 [00:02<00:00, 19.35it/s, loss=0.0103, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 26:  81%|▊| 51/63 [00:02<00:00, 17.14it/s, loss=0.0103, v_num=6, train_acc\u001b[A\n",
      "Epoch 26: 100%|█| 63/63 [00:03<00:00, 17.41it/s, loss=0.0103, v_num=6, train_acc\u001b[A\n",
      "Epoch 27:  76%|▊| 48/63 [00:02<00:00, 19.46it/s, loss=0.00524, v_num=6, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 27:  81%|▊| 51/63 [00:02<00:00, 17.18it/s, loss=0.00524, v_num=6, train_ac\u001b[A\n",
      "Epoch 27: 100%|█| 63/63 [00:03<00:00, 17.57it/s, loss=0.00524, v_num=6, train_ac\u001b[A\n",
      "Epoch 28:  76%|▊| 48/63 [00:02<00:00, 19.29it/s, loss=0.0278, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 28:  81%|▊| 51/63 [00:02<00:00, 17.06it/s, loss=0.0278, v_num=6, train_acc\u001b[A\n",
      "Epoch 28: 100%|█| 63/63 [00:03<00:00, 17.47it/s, loss=0.0278, v_num=6, train_acc\u001b[A\n",
      "Epoch 29:  76%|▊| 48/63 [00:02<00:00, 19.40it/s, loss=0.0438, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 29:  81%|▊| 51/63 [00:02<00:00, 17.17it/s, loss=0.0438, v_num=6, train_acc\u001b[A\n",
      "Epoch 29: 100%|█| 63/63 [00:03<00:00, 17.54it/s, loss=0.0438, v_num=6, train_acc\u001b[A\n",
      "Epoch 30:  76%|▊| 48/63 [00:02<00:00, 18.87it/s, loss=0.0217, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 30:  81%|▊| 51/63 [00:03<00:00, 16.72it/s, loss=0.0217, v_num=6, train_acc\u001b[A\n",
      "Epoch 30: 100%|█| 63/63 [00:03<00:00, 17.09it/s, loss=0.0217, v_num=6, train_acc\u001b[A\n",
      "Epoch 31:  76%|▊| 48/63 [00:02<00:00, 18.73it/s, loss=0.0139, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 31:  81%|▊| 51/63 [00:03<00:00, 16.55it/s, loss=0.0139, v_num=6, train_acc\u001b[A\n",
      "Epoch 31: 100%|█| 63/63 [00:03<00:00, 16.98it/s, loss=0.0139, v_num=6, train_acc\u001b[A\n",
      "Epoch 32:  76%|▊| 48/63 [00:02<00:00, 18.48it/s, loss=0.0471, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 32:  81%|▊| 51/63 [00:03<00:00, 16.29it/s, loss=0.0471, v_num=6, train_acc\u001b[A\n",
      "Epoch 32: 100%|█| 63/63 [00:03<00:00, 16.80it/s, loss=0.0471, v_num=6, train_acc\u001b[A\n",
      "Epoch 33:  76%|▊| 48/63 [00:02<00:00, 18.89it/s, loss=0.0247, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 33:  81%|▊| 51/63 [00:03<00:00, 16.59it/s, loss=0.0247, v_num=6, train_acc\u001b[A\n",
      "Epoch 33: 100%|█| 63/63 [00:03<00:00, 16.99it/s, loss=0.0247, v_num=6, train_acc\u001b[A\n",
      "Epoch 34:  76%|▊| 48/63 [00:02<00:00, 18.81it/s, loss=0.00486, v_num=6, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 34:  81%|▊| 51/63 [00:03<00:00, 16.63it/s, loss=0.00486, v_num=6, train_ac\u001b[A\n",
      "Epoch 34: 100%|█| 63/63 [00:03<00:00, 16.98it/s, loss=0.00486, v_num=6, train_ac\u001b[A\n",
      "Epoch 35:  76%|▊| 48/63 [00:02<00:00, 18.88it/s, loss=0.00731, v_num=6, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 35:  81%|▊| 51/63 [00:03<00:00, 16.66it/s, loss=0.00731, v_num=6, train_ac\u001b[A\n",
      "Epoch 35: 100%|█| 63/63 [00:03<00:00, 17.01it/s, loss=0.00731, v_num=6, train_ac\u001b[A\n",
      "Epoch 36:  76%|▊| 48/63 [00:02<00:00, 18.86it/s, loss=0.0105, v_num=6, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 36:  81%|▊| 51/63 [00:03<00:00, 16.68it/s, loss=0.0105, v_num=6, train_acc\u001b[A\n",
      "Epoch 36: 100%|█| 63/63 [00:03<00:00, 17.04it/s, loss=0.0105, v_num=6, train_acc\u001b[A\n",
      "Epoch 37:  76%|▊| 48/63 [00:02<00:00, 19.00it/s, loss=0.26, v_num=6, train_acc=0\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 37:  81%|▊| 51/63 [00:03<00:00, 16.74it/s, loss=0.26, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 37: 100%|█| 63/63 [00:03<00:00, 16.94it/s, loss=0.26, v_num=6, train_acc=0\u001b[A\n",
      "Epoch 38:  76%|▊| 48/63 [00:02<00:00, 18.96it/s, loss=0.00954, v_num=6, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/16 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 38:  81%|▊| 51/63 [00:03<00:00, 16.81it/s, loss=0.00954, v_num=6, train_ac\u001b[A\n",
      "Epoch 38: 100%|█| 63/63 [00:03<00:00, 17.18it/s, loss=0.00954, v_num=6, train_ac\u001b[A\n",
      "Epoch 38: 100%|█| 63/63 [00:03<00:00, 17.09it/s, loss=0.00954, v_num=6, train_ac\u001b[A\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_6/checkpoints/epoch=22-step=1080.ckpt\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_6/checkpoints/epoch=22-step=1080.ckpt\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:662: UserWarning: Your `test_dataloader` has `shuffle=True`, it is strongly recommended that you turn this off for val/test/predict dataloaders.\n",
      "  category=UserWarning,\n",
      "Testing:  87%|█████████████████████████████▋    | 41/47 [00:00<00:00, 82.49it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.999666690826416, 'test_loss': 0.0029280984308570623}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 47/47 [00:02<00:00, 20.96it/s]\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_6/checkpoints/epoch=22-step=1080.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_6/checkpoints/epoch=22-step=1080.ckpt\n",
      "Testing: 100%|██████████████████████████████████| 16/16 [00:00<00:00, 30.07it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9660000205039978, 'test_loss': 0.11626611649990082}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 16/16 [00:01<00:00, 14.53it/s]\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.teardown has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.teardown.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "Global seed set to 1234\n",
      "Global seed set to 1234\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_6/checkpoints/epoch=22-step=1080.ckpt\n",
      "LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_6/checkpoints/epoch=22-step=1080.ckpt\n",
      "Testing: 100%|███████████████████████████████| 157/157 [00:01<00:00, 138.74it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9688000082969666, 'test_loss': 0.11470437049865723}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|████████████████████████████████| 157/157 [00:06<00:00, 25.85it/s]\n",
      "train_result:\n",
      "\n",
      "[{'test_acc': 0.999666690826416, 'test_loss': 0.0029280984308570623}]\n",
      "val_result:\n",
      "\n",
      "[{'test_acc': 0.9660000205039978, 'test_loss': 0.11626611649990082}]\n",
      "test_result:\n",
      "\n",
      "[{'test_acc': 0.9688000082969666, 'test_loss': 0.11470437049865723}]\n",
      "train_result:\n",
      "\n",
      "[{'test_acc': 0.999666690826416, 'test_loss': 0.0029280984308570623}]\n",
      "val_result:\n",
      "\n",
      "[{'test_acc': 0.9660000205039978, 'test_loss': 0.11626611649990082}]\n",
      "test_result:\n",
      "\n",
      "[{'test_acc': 0.9688000082969666, 'test_loss': 0.11470437049865723}]\n"
     ]
    }
   ],
   "source": [
    "!python3 mnist_cnn.py --gpus=2 --strategy=\"ddp_find_unused_parameters_false\" --use_swa=True --use_CyclicLR=True "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "456becb1",
   "metadata": {},
   "source": [
    "```\n",
    "------------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9688000082969666, 'test_loss': 0.11470437049865723}\n",
    "--------------------------------------------------------------------------------\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fd4d835",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "57d4bbcc",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a7e9b3d4-739a-44b0-9576-5198712d7b90",
   "metadata": {},
   "source": [
    "### 3, 最优学习率搜索(auto_lr_find=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "821ac510-96f2-49b6-97a9-679303d1b844",
   "metadata": {},
   "source": [
    "* 平凡方式训练：test_acc = 0.9581000208854675\n",
    "\n",
    "* SWA随机权重： test_acc = 0.963100016117096\n",
    "\n",
    "* SWA随机权重 + CyClicLR学习率调度策略: test_acc = 0.9688000082969666\n",
    "\n",
    "* SWA随机权重 + CyClicLR学习率调度策略 + 最优学习率搜索： test_acc = 0.9693999886512756"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 307,
   "id": "6777c89a-e83e-4242-8a4f-9f572bacb899",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed set to 1234\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "hparams.auto_lr_find= True\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "set lr = 0.007\n",
      "Global seed set to 1234\n",
      "Finding best initial lr:  80%|█████████████▌   | 80/100 [00:00<00:00, 94.27it/s]\n",
      "LR finder stopped early after 80 steps due to diverging loss.\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lr_find_temp_model_5344e797-48c6-4eb4-b090-a25e9aa6635c.ckpt\n",
      "suggest lr= 0.003311311214825908\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "\n",
      "  | Name      | Type       | Params\n",
      "-----------------------------------------\n",
      "0 | net       | Sequential | 54.0 K\n",
      "1 | train_acc | Accuracy   | 0     \n",
      "2 | val_acc   | Accuracy   | 0     \n",
      "3 | test_acc  | Accuracy   | 0     \n",
      "-----------------------------------------\n",
      "54.0 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.0 K    Total params\n",
      "0.216     Total estimated model params size (MB)\n",
      "Global seed set to 1234                                                         \n",
      "Epoch 0:  75%|▋| 94/126 [00:01<00:00, 63.08it/s, loss=1.04, v_num=19, train_acc=\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 0:  78%|▊| 98/126 [00:01<00:00, 50.41it/s, loss=1.04, v_num=19, train_acc=\u001b[A\n",
      "Epoch 0:  94%|▉| 118/126 [00:02<00:00, 57.72it/s, loss=1.04, v_num=19, train_acc\u001b[A\n",
      "Epoch 0: 100%|█| 126/126 [00:02<00:00, 57.73it/s, loss=1.04, v_num=19, train_acc\u001b[A\n",
      "Epoch 1:  75%|▋| 94/126 [00:01<00:00, 64.47it/s, loss=0.598, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 1:  79%|▊| 100/126 [00:01<00:00, 51.37it/s, loss=0.598, v_num=19, train_ac\u001b[A\n",
      "Epoch 1:  95%|▉| 120/126 [00:02<00:00, 58.61it/s, loss=0.598, v_num=19, train_ac\u001b[A\n",
      "Epoch 1: 100%|█| 126/126 [00:02<00:00, 58.17it/s, loss=0.598, v_num=19, train_ac\u001b[A\n",
      "Epoch 2:  75%|▋| 94/126 [00:01<00:00, 62.16it/s, loss=0.437, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 2:  79%|▊| 100/126 [00:02<00:00, 49.87it/s, loss=0.437, v_num=19, train_ac\u001b[A\n",
      "Epoch 2:  95%|▉| 120/126 [00:02<00:00, 56.89it/s, loss=0.437, v_num=19, train_ac\u001b[A\n",
      "Epoch 2: 100%|█| 126/126 [00:02<00:00, 56.49it/s, loss=0.437, v_num=19, train_ac\u001b[A\n",
      "Epoch 3:  75%|▋| 94/126 [00:01<00:00, 61.26it/s, loss=0.307, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 3:  79%|▊| 100/126 [00:02<00:00, 49.41it/s, loss=0.307, v_num=19, train_ac\u001b[A\n",
      "Epoch 3: 100%|█| 126/126 [00:02<00:00, 56.76it/s, loss=0.307, v_num=19, train_ac\u001b[A\n",
      "Epoch 4:  75%|▋| 94/126 [00:01<00:00, 62.91it/s, loss=0.225, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 4:  83%|▊| 105/126 [00:01<00:00, 52.78it/s, loss=0.225, v_num=19, train_ac\u001b[A\n",
      "Epoch 4: 100%|█| 126/126 [00:02<00:00, 60.14it/s, loss=0.225, v_num=19, train_ac\u001b[A\n",
      "Epoch 4: 100%|█| 126/126 [00:02<00:00, 58.13it/s, loss=0.225, v_num=19, train_ac\u001b[A\n",
      "Epoch 5:  75%|▋| 94/126 [00:01<00:00, 63.91it/s, loss=0.178, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 5:  83%|▊| 105/126 [00:01<00:00, 53.01it/s, loss=0.178, v_num=19, train_ac\u001b[A\n",
      "Epoch 5: 100%|█| 126/126 [00:02<00:00, 60.51it/s, loss=0.178, v_num=19, train_ac\u001b[A\n",
      "Epoch 5: 100%|█| 126/126 [00:02<00:00, 58.48it/s, loss=0.178, v_num=19, train_ac\u001b[A\n",
      "Epoch 6:  75%|▋| 94/126 [00:01<00:00, 62.28it/s, loss=0.178, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 6:  83%|▊| 105/126 [00:02<00:00, 52.21it/s, loss=0.178, v_num=19, train_ac\u001b[A\n",
      "Epoch 6: 100%|█| 126/126 [00:02<00:00, 59.54it/s, loss=0.178, v_num=19, train_ac\u001b[A\n",
      "Epoch 6: 100%|█| 126/126 [00:02<00:00, 57.34it/s, loss=0.178, v_num=19, train_ac\u001b[A\n",
      "Epoch 7:  75%|▋| 94/126 [00:01<00:00, 64.29it/s, loss=0.128, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 7:  83%|▊| 105/126 [00:01<00:00, 52.78it/s, loss=0.128, v_num=19, train_ac\u001b[A\n",
      "Epoch 7: 100%|█| 126/126 [00:02<00:00, 60.16it/s, loss=0.128, v_num=19, train_ac\u001b[A\n",
      "Epoch 7: 100%|█| 126/126 [00:02<00:00, 57.52it/s, loss=0.128, v_num=19, train_ac\u001b[A\n",
      "Epoch 8:  75%|▋| 94/126 [00:01<00:00, 63.20it/s, loss=0.127, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 8:  83%|▊| 105/126 [00:01<00:00, 52.55it/s, loss=0.127, v_num=19, train_ac\u001b[A\n",
      "Validating:  59%|██████████████████▍            | 19/32 [00:00<00:00, 44.64it/s]\u001b[A\n",
      "Epoch 8: 100%|█| 126/126 [00:02<00:00, 57.51it/s, loss=0.127, v_num=19, train_ac\u001b[A\n",
      "Epoch 9:  75%|▋| 94/126 [00:01<00:00, 63.51it/s, loss=0.105, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 9:  83%|▊| 105/126 [00:01<00:00, 52.83it/s, loss=0.105, v_num=19, train_ac\u001b[A\n",
      "Epoch 9: 100%|█| 126/126 [00:02<00:00, 60.19it/s, loss=0.105, v_num=19, train_ac\u001b[A\n",
      "Epoch 9: 100%|█| 126/126 [00:02<00:00, 57.10it/s, loss=0.105, v_num=19, train_ac\u001b[A\n",
      "Epoch 10:  75%|▋| 94/126 [00:01<00:00, 65.49it/s, loss=0.0979, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 10:  83%|▊| 105/126 [00:01<00:00, 54.23it/s, loss=0.0979, v_num=19, train_\u001b[A\n",
      "Validating:  59%|██████████████████▍            | 19/32 [00:00<00:00, 45.49it/s]\u001b[A\n",
      "Epoch 10: 100%|█| 126/126 [00:02<00:00, 59.81it/s, loss=0.0979, v_num=19, train_\u001b[A\n",
      "Epoch 11:  75%|▋| 94/126 [00:01<00:00, 65.20it/s, loss=0.0908, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 11:  83%|▊| 105/126 [00:01<00:00, 54.31it/s, loss=0.0908, v_num=19, train_\u001b[A\n",
      "Epoch 11: 100%|█| 126/126 [00:02<00:00, 61.92it/s, loss=0.0908, v_num=19, train_\u001b[A\n",
      "Epoch 11: 100%|█| 126/126 [00:02<00:00, 59.01it/s, loss=0.0908, v_num=19, train_\u001b[A\n",
      "Epoch 12:  75%|▋| 94/126 [00:01<00:00, 67.03it/s, loss=0.0511, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 12:  83%|▊| 105/126 [00:01<00:00, 55.86it/s, loss=0.0511, v_num=19, train_\u001b[A\n",
      "Epoch 12: 100%|█| 126/126 [00:02<00:00, 61.20it/s, loss=0.0511, v_num=19, train_\u001b[A\n",
      "Epoch 13:  75%|▋| 94/126 [00:01<00:00, 67.98it/s, loss=0.0388, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 13:  83%|▊| 105/126 [00:01<00:00, 55.69it/s, loss=0.0388, v_num=19, train_\u001b[A\n",
      "Validating:  62%|███████████████████▍           | 20/32 [00:00<00:00, 47.61it/s]\u001b[A\n",
      "Epoch 13: 100%|█| 126/126 [00:02<00:00, 61.39it/s, loss=0.0388, v_num=19, train_\u001b[A\n",
      "Epoch 14:  75%|▋| 94/126 [00:01<00:00, 67.66it/s, loss=0.0438, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 14:  83%|▊| 105/126 [00:01<00:00, 55.96it/s, loss=0.0438, v_num=19, train_\u001b[A\n",
      "Epoch 14: 100%|█| 126/126 [00:01<00:00, 63.63it/s, loss=0.0438, v_num=19, train_\u001b[A\n",
      "Epoch 14: 100%|█| 126/126 [00:02<00:00, 61.33it/s, loss=0.0438, v_num=19, train_\u001b[A\n",
      "Epoch 15:  75%|▋| 94/126 [00:01<00:00, 68.00it/s, loss=0.0427, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 15:  83%|▊| 105/126 [00:01<00:00, 55.75it/s, loss=0.0427, v_num=19, train_\u001b[A\n",
      "Epoch 15: 100%|█| 126/126 [00:02<00:00, 60.84it/s, loss=0.0427, v_num=19, train_\u001b[A\n",
      "Epoch 16:  75%|▋| 94/126 [00:01<00:00, 67.85it/s, loss=0.0482, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 16:  83%|▊| 105/126 [00:01<00:00, 55.95it/s, loss=0.0482, v_num=19, train_\u001b[A\n",
      "Epoch 16: 100%|█| 126/126 [00:02<00:00, 61.39it/s, loss=0.0482, v_num=19, train_\u001b[A\n",
      "Epoch 17:  75%|▋| 94/126 [00:01<00:00, 67.91it/s, loss=0.0531, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 17:  83%|▊| 105/126 [00:01<00:00, 55.76it/s, loss=0.0531, v_num=19, train_\u001b[A\n",
      "Epoch 17: 100%|█| 126/126 [00:02<00:00, 61.47it/s, loss=0.0531, v_num=19, train_\u001b[A\n",
      "Epoch 18:  75%|▋| 94/126 [00:01<00:00, 66.77it/s, loss=0.0296, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 18:  83%|▊| 105/126 [00:01<00:00, 55.03it/s, loss=0.0296, v_num=19, train_\u001b[A\n",
      "Epoch 18: 100%|█| 126/126 [00:02<00:00, 62.71it/s, loss=0.0296, v_num=19, train_\u001b[A\n",
      "Epoch 18: 100%|█| 126/126 [00:02<00:00, 59.48it/s, loss=0.0296, v_num=19, train_\u001b[A\n",
      "Epoch 19:  75%|▋| 94/126 [00:01<00:00, 63.95it/s, loss=0.0511, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 19:  83%|▊| 105/126 [00:01<00:00, 53.19it/s, loss=0.0511, v_num=19, train_\u001b[A\n",
      "Epoch 19: 100%|█| 126/126 [00:02<00:00, 60.61it/s, loss=0.0511, v_num=19, train_\u001b[A\n",
      "Epoch 19: 100%|█| 126/126 [00:02<00:00, 57.81it/s, loss=0.0511, v_num=19, train_\u001b[A\n",
      "Epoch 20:  75%|▋| 94/126 [00:01<00:00, 62.81it/s, loss=0.0224, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 20:  83%|▊| 105/126 [00:02<00:00, 52.48it/s, loss=0.0224, v_num=19, train_\u001b[A\n",
      "Epoch 20: 100%|█| 126/126 [00:02<00:00, 59.95it/s, loss=0.0224, v_num=19, train_\u001b[A\n",
      "Epoch 20: 100%|█| 126/126 [00:02<00:00, 57.91it/s, loss=0.0224, v_num=19, train_\u001b[A\n",
      "Epoch 21:  75%|▋| 94/126 [00:01<00:00, 61.95it/s, loss=0.0223, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 21:  83%|▊| 105/126 [00:02<00:00, 51.65it/s, loss=0.0223, v_num=19, train_\u001b[A\n",
      "Epoch 21: 100%|█| 126/126 [00:02<00:00, 59.04it/s, loss=0.0223, v_num=19, train_\u001b[A\n",
      "Epoch 21: 100%|█| 126/126 [00:02<00:00, 56.54it/s, loss=0.0223, v_num=19, train_\u001b[A\n",
      "Epoch 22:  75%|▋| 94/126 [00:01<00:00, 64.44it/s, loss=0.0434, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 22:  83%|▊| 105/126 [00:01<00:00, 53.17it/s, loss=0.0434, v_num=19, train_\u001b[A\n",
      "Epoch 22: 100%|█| 126/126 [00:02<00:00, 60.58it/s, loss=0.0434, v_num=19, train_\u001b[A\n",
      "Epoch 22: 100%|█| 126/126 [00:02<00:00, 57.61it/s, loss=0.0434, v_num=19, train_\u001b[A\n",
      "Epoch 23:  75%|▋| 94/126 [00:01<00:00, 63.32it/s, loss=0.03, v_num=19, train_acc\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 23:  83%|▊| 105/126 [00:01<00:00, 52.78it/s, loss=0.03, v_num=19, train_ac\u001b[A\n",
      "Epoch 23: 100%|█| 126/126 [00:02<00:00, 60.26it/s, loss=0.03, v_num=19, train_ac\u001b[A\n",
      "Epoch 23: 100%|█| 126/126 [00:02<00:00, 57.25it/s, loss=0.03, v_num=19, train_ac\u001b[A\n",
      "Epoch 24:  75%|▋| 94/126 [00:01<00:00, 63.32it/s, loss=0.0143, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 24:  83%|▊| 105/126 [00:01<00:00, 52.74it/s, loss=0.0143, v_num=19, train_\u001b[A\n",
      "Epoch 24: 100%|█| 126/126 [00:02<00:00, 60.23it/s, loss=0.0143, v_num=19, train_\u001b[A\n",
      "Epoch 24: 100%|█| 126/126 [00:02<00:00, 57.35it/s, loss=0.0143, v_num=19, train_\u001b[A\n",
      "Epoch 25:  75%|▋| 94/126 [00:01<00:00, 63.10it/s, loss=0.0344, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 25:  83%|▊| 105/126 [00:02<00:00, 52.44it/s, loss=0.0344, v_num=19, train_\u001b[A\n",
      "Epoch 25: 100%|█| 126/126 [00:02<00:00, 59.84it/s, loss=0.0344, v_num=19, train_\u001b[A\n",
      "Epoch 25: 100%|█| 126/126 [00:02<00:00, 57.07it/s, loss=0.0344, v_num=19, train_\u001b[A\n",
      "Epoch 26:  75%|▋| 94/126 [00:01<00:00, 64.24it/s, loss=0.0579, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 26:  83%|▊| 105/126 [00:01<00:00, 52.58it/s, loss=0.0579, v_num=19, train_\u001b[A\n",
      "Epoch 26: 100%|█| 126/126 [00:02<00:00, 60.07it/s, loss=0.0579, v_num=19, train_\u001b[A\n",
      "Epoch 26: 100%|█| 126/126 [00:02<00:00, 57.95it/s, loss=0.0579, v_num=19, train_\u001b[A\n",
      "Epoch 27:  75%|▋| 94/126 [00:01<00:00, 63.36it/s, loss=0.0185, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 27:  83%|▊| 105/126 [00:01<00:00, 52.68it/s, loss=0.0185, v_num=19, train_\u001b[A\n",
      "Epoch 27: 100%|█| 126/126 [00:02<00:00, 60.14it/s, loss=0.0185, v_num=19, train_\u001b[A\n",
      "Epoch 27: 100%|█| 126/126 [00:02<00:00, 57.52it/s, loss=0.0185, v_num=19, train_\u001b[A\n",
      "Epoch 28:  75%|▋| 94/126 [00:01<00:00, 63.29it/s, loss=0.0183, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 28:  83%|▊| 105/126 [00:01<00:00, 53.09it/s, loss=0.0183, v_num=19, train_\u001b[A\n",
      "Epoch 28: 100%|█| 126/126 [00:02<00:00, 60.50it/s, loss=0.0183, v_num=19, train_\u001b[A\n",
      "Epoch 28: 100%|█| 126/126 [00:02<00:00, 57.71it/s, loss=0.0183, v_num=19, train_\u001b[A\n",
      "Epoch 29:  75%|▋| 94/126 [00:01<00:00, 62.71it/s, loss=0.0451, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 29:  83%|▊| 105/126 [00:02<00:00, 52.30it/s, loss=0.0451, v_num=19, train_\u001b[A\n",
      "Epoch 29: 100%|█| 126/126 [00:02<00:00, 59.72it/s, loss=0.0451, v_num=19, train_\u001b[A\n",
      "Epoch 29: 100%|█| 126/126 [00:02<00:00, 57.35it/s, loss=0.0451, v_num=19, train_\u001b[A\n",
      "Epoch 30:  75%|▋| 94/126 [00:01<00:00, 63.14it/s, loss=0.017, v_num=19, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 30:  83%|▊| 105/126 [00:02<00:00, 51.86it/s, loss=0.017, v_num=19, train_a\u001b[A\n",
      "Epoch 30: 100%|█| 126/126 [00:02<00:00, 59.23it/s, loss=0.017, v_num=19, train_a\u001b[A\n",
      "Epoch 30: 100%|█| 126/126 [00:02<00:00, 57.18it/s, loss=0.017, v_num=19, train_a\u001b[A\n",
      "Epoch 31:  75%|▋| 94/126 [00:01<00:00, 60.85it/s, loss=0.0127, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 31:  83%|▊| 105/126 [00:02<00:00, 50.97it/s, loss=0.0127, v_num=19, train_\u001b[A\n",
      "Validating:  62%|███████████████████▍           | 20/32 [00:00<00:00, 46.59it/s]\u001b[A\n",
      "Epoch 31: 100%|█| 126/126 [00:02<00:00, 56.00it/s, loss=0.0127, v_num=19, train_\u001b[A\n",
      "Epoch 32:  75%|▋| 94/126 [00:01<00:00, 61.13it/s, loss=0.0195, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 32:  83%|▊| 105/126 [00:02<00:00, 51.01it/s, loss=0.0195, v_num=19, train_\u001b[A\n",
      "Epoch 32: 100%|█| 126/126 [00:02<00:00, 58.22it/s, loss=0.0195, v_num=19, train_\u001b[A\n",
      "Epoch 32: 100%|█| 126/126 [00:02<00:00, 56.28it/s, loss=0.0195, v_num=19, train_\u001b[A\n",
      "Epoch 33:  75%|▋| 94/126 [00:01<00:00, 63.95it/s, loss=0.00425, v_num=19, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 33:  83%|▊| 105/126 [00:01<00:00, 53.21it/s, loss=0.00425, v_num=19, train\u001b[A\n",
      "Validating:  62%|███████████████████▍           | 20/32 [00:00<00:00, 47.47it/s]\u001b[A\n",
      "Epoch 33: 100%|█| 126/126 [00:02<00:00, 58.09it/s, loss=0.00425, v_num=19, train\u001b[A\n",
      "Epoch 34:  75%|▋| 94/126 [00:01<00:00, 65.46it/s, loss=0.0357, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 34:  83%|▊| 105/126 [00:01<00:00, 54.39it/s, loss=0.0357, v_num=19, train_\u001b[A\n",
      "Validating:  59%|██████████████████▍            | 19/32 [00:00<00:00, 46.15it/s]\u001b[A\n",
      "Epoch 34: 100%|█| 126/126 [00:02<00:00, 59.95it/s, loss=0.0357, v_num=19, train_\u001b[A\n",
      "Epoch 35:  75%|▋| 94/126 [00:01<00:00, 67.32it/s, loss=0.0234, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 35:  83%|▊| 105/126 [00:01<00:00, 55.70it/s, loss=0.0234, v_num=19, train_\u001b[A\n",
      "Epoch 35: 100%|█| 126/126 [00:02<00:00, 60.55it/s, loss=0.0234, v_num=19, train_\u001b[A\n",
      "Epoch 36:  75%|▋| 94/126 [00:01<00:00, 69.17it/s, loss=0.0287, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 36:  83%|▊| 105/126 [00:01<00:00, 56.81it/s, loss=0.0287, v_num=19, train_\u001b[A\n",
      "Validating:  59%|██████████████████▍            | 19/32 [00:00<00:00, 46.65it/s]\u001b[A\n",
      "Epoch 36: 100%|█| 126/126 [00:02<00:00, 62.47it/s, loss=0.0287, v_num=19, train_\u001b[A\n",
      "Epoch 37:  75%|▋| 94/126 [00:01<00:00, 68.60it/s, loss=0.00399, v_num=19, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 37:  83%|▊| 105/126 [00:01<00:00, 56.76it/s, loss=0.00399, v_num=19, train\u001b[A\n",
      "Epoch 37: 100%|█| 126/126 [00:02<00:00, 62.68it/s, loss=0.00399, v_num=19, train\u001b[A\n",
      "Epoch 38:  75%|▋| 94/126 [00:01<00:00, 69.03it/s, loss=0.0454, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 38:  83%|▊| 105/126 [00:01<00:00, 56.97it/s, loss=0.0454, v_num=19, train_\u001b[A\n",
      "Validating:  62%|███████████████████▍           | 20/32 [00:00<00:00, 49.34it/s]\u001b[A\n",
      "Epoch 38: 100%|█| 126/126 [00:02<00:00, 62.55it/s, loss=0.0454, v_num=19, train_\u001b[A\n",
      "Epoch 39:  75%|▋| 94/126 [00:01<00:00, 67.26it/s, loss=0.034, v_num=19, train_ac\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 39:  83%|▊| 105/126 [00:01<00:00, 55.60it/s, loss=0.034, v_num=19, train_a\u001b[A\n",
      "Validating:  62%|███████████████████▍           | 20/32 [00:00<00:00, 48.56it/s]\u001b[A\n",
      "Epoch 39: 100%|█| 126/126 [00:02<00:00, 60.97it/s, loss=0.034, v_num=19, train_a\u001b[A\n",
      "Epoch 40:  75%|▋| 94/126 [00:01<00:00, 67.34it/s, loss=0.0262, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 40:  83%|▊| 105/126 [00:01<00:00, 55.31it/s, loss=0.0262, v_num=19, train_\u001b[A\n",
      "Validating:  59%|██████████████████▍            | 19/32 [00:00<00:00, 45.36it/s]\u001b[A\n",
      "Epoch 40: 100%|█| 126/126 [00:02<00:00, 60.83it/s, loss=0.0262, v_num=19, train_\u001b[A\n",
      "Epoch 41:  75%|▋| 94/126 [00:01<00:00, 67.71it/s, loss=0.00348, v_num=19, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 41:  83%|▊| 105/126 [00:01<00:00, 56.08it/s, loss=0.00348, v_num=19, train\u001b[A\n",
      "Epoch 41: 100%|█| 126/126 [00:02<00:00, 61.10it/s, loss=0.00348, v_num=19, train\u001b[A\n",
      "Epoch 42:  75%|▋| 94/126 [00:01<00:00, 64.99it/s, loss=0.0051, v_num=19, train_a\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 42:  83%|▊| 105/126 [00:01<00:00, 53.78it/s, loss=0.0051, v_num=19, train_\u001b[A\n",
      "Epoch 42: 100%|█| 126/126 [00:02<00:00, 61.36it/s, loss=0.0051, v_num=19, train_\u001b[A\n",
      "Epoch 42: 100%|█| 126/126 [00:02<00:00, 58.60it/s, loss=0.0051, v_num=19, train_\u001b[A\n",
      "Epoch 43:  75%|▋| 94/126 [00:01<00:00, 63.08it/s, loss=0.00219, v_num=19, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 43:  83%|▊| 105/126 [00:01<00:00, 52.56it/s, loss=0.00219, v_num=19, train\u001b[A\n",
      "Epoch 43: 100%|█| 126/126 [00:02<00:00, 60.01it/s, loss=0.00219, v_num=19, train\u001b[A\n",
      "Epoch 43: 100%|█| 126/126 [00:02<00:00, 57.97it/s, loss=0.00219, v_num=19, train\u001b[A\n",
      "Epoch 44:  75%|▋| 94/126 [00:01<00:00, 64.57it/s, loss=0.00869, v_num=19, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 44:  83%|▊| 105/126 [00:01<00:00, 53.33it/s, loss=0.00869, v_num=19, train\u001b[A\n",
      "Epoch 44: 100%|█| 126/126 [00:02<00:00, 60.71it/s, loss=0.00869, v_num=19, train\u001b[A\n",
      "Epoch 44: 100%|█| 126/126 [00:02<00:00, 57.78it/s, loss=0.00869, v_num=19, train\u001b[A\n",
      "Epoch 45:  75%|▋| 94/126 [00:01<00:00, 61.46it/s, loss=0.00969, v_num=19, train_\u001b[A\n",
      "Validating: 0it [00:00, ?it/s]\u001b[A\n",
      "Validating:   0%|                                        | 0/32 [00:00<?, ?it/s]\u001b[A\n",
      "Epoch 45:  83%|▊| 105/126 [00:02<00:00, 51.96it/s, loss=0.00969, v_num=19, train\u001b[A\n",
      "Epoch 45: 100%|█| 126/126 [00:02<00:00, 59.38it/s, loss=0.00969, v_num=19, train\u001b[A\n",
      "Epoch 45: 100%|█| 126/126 [00:02<00:00, 56.84it/s, loss=0.00969, v_num=19, train\u001b[A\n",
      "Epoch 45: 100%|█| 126/126 [00:02<00:00, 56.77it/s, loss=0.00969, v_num=19, train\u001b[A\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py:470: LightningDeprecationWarning: DataModule.teardown has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.teardown.\n",
      "  f\"DataModule.{name} has already been called, so it will not be called again. \"\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=29-step=2819.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=29-step=2819.ckpt\n",
      "/usr/local/lib/python3.6/site-packages/pytorch_lightning/trainer/data_loading.py:662: UserWarning: Your `test_dataloader` has `shuffle=True`, it is strongly recommended that you turn this off for val/test/predict dataloaders.\n",
      "  category=UserWarning,\n",
      "Testing:  99%|████████████████████████████████▋| 93/94 [00:00<00:00, 143.64it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 1.0, 'test_loss': 0.0020417957566678524}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 94/94 [00:01<00:00, 89.38it/s]\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=29-step=2819.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=29-step=2819.ckpt\n",
      "Testing: 100%|██████████████████████████████████| 32/32 [00:00<00:00, 61.36it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.972000002861023, 'test_loss': 0.10358494520187378}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|██████████████████████████████████| 32/32 [00:00<00:00, 46.36it/s]\n",
      "Restoring states from the checkpoint path at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=29-step=2819.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n",
      "Loaded model weights from checkpoint at /home/hadoop-map/machine_learning_AK47/②基础模型/lightning_logs/version_19/checkpoints/epoch=29-step=2819.ckpt\n",
      "Testing: 100%|███████████████████████████████| 313/313 [00:02<00:00, 155.28it/s]--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9693999886512756, 'test_loss': 0.11024412512779236}\n",
      "--------------------------------------------------------------------------------\n",
      "Testing: 100%|███████████████████████████████| 313/313 [00:02<00:00, 137.85it/s]\n",
      "train_result:\n",
      "\n",
      "[{'test_acc': 1.0, 'test_loss': 0.0020417957566678524}]\n",
      "val_result:\n",
      "\n",
      "[{'test_acc': 0.972000002861023, 'test_loss': 0.10358494520187378}]\n",
      "test_result:\n",
      "\n",
      "[{'test_acc': 0.9693999886512756, 'test_loss': 0.11024412512779236}]\n"
     ]
    }
   ],
   "source": [
    "!python3 mnist_cnn.py --gpus=1  --auto_lr_find=True --use_swa=True --use_CyclicLR=True \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd8f4a2a",
   "metadata": {},
   "source": [
    "```\n",
    "---------------------------------------------------------------\n",
    "DATALOADER:0 TEST RESULTS\n",
    "{'test_acc': 0.9693999886512756, 'test_loss': 0.11024412512779236}\n",
    "--------------------------------------------------------------------------------\n",
    "Testing: 100%|███████████████████████████████| 313/313 [00:02<00:00, 137.85it/s]\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}