modules/transfer_mmar.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) MONAI Consortium  \n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  \n",
    "you may not use this file except in compliance with the License.  \n",
    "You may obtain a copy of the License at  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;http://www.apache.org/licenses/LICENSE-2.0  \n",
    "Unless required by applicable law or agreed to in writing, software  \n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,  \n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  \n",
    "See the License for the specific language governing permissions and  \n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BazmryhqKyZk"
   },
   "source": [
    "# Transfer learning from Clara Train models\n",
    " \n",
    "In this tutorial, we demonstrate transfer learning from a pretrained model in [Clara Train's Medical Model Archive format](https://docs.nvidia.com/clara/clara-train-sdk/pt/mmar.html).\n",
    "\n",
    "The `monai.apps.load_from_mmar` command implements the following: \n",
    "- fetch a remote model if it is not cached (from Nvidia GPU cloud https://ngc.nvidia.com/)\n",
    "- cache a local model file copy\n",
    "- instantiate a network model with pre-trained weights.\n",
    "\n",
    "The APIs enable direct access to the MMAR models in a pythonic way.\n",
    "\n",
    "\n",
    "In this demo, we show that a PyTorch native pipeline could be configured to use the pretrained models,\n",
    "and compare the following three cases:\n",
    "1. regular training with a randomly initialised model (best mean Dice: 0.84)\n",
    "<p><img src=\"../figures/mmar_1.png\" width=\"50%\" alt='mmar_1'></p>\n",
    "2. loading from a pretrained model, without any model updating (best mean Dice: 0.86)\n",
    "<p><img src=\"../figures/mmar_2.png\" width=\"50%\" alt='mmar_2'></p>\n",
    "3. loading from a pretrained model, with finetuning (best mean Dice: 0.92)\n",
    "<p><img src=\"../figures/mmar_0.png\" width=\"50%\" alt='mmar_0'></p>\n",
    "\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Project-MONAI/tutorials/blob/main/modules/transfer_mmar.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H3dniV1cKyZp"
   },
   "source": [
    "## Setup environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2709,
     "status": "ok",
     "timestamp": 1625519079692,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "fUOXcmazKyZq",
    "outputId": "1ecb32b7-afe7-406e-cd97-eb7be42ec437"
   },
   "outputs": [],
   "source": [
    "!python -c \"import monai\" || pip install -q \"monai-weekly[nibabel, lmdb, tqdm]\"\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kGQY14M9KyZr"
   },
   "source": [
    "## Setup imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1222,
     "status": "ok",
     "timestamp": 1625519080911,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "E1VgZdItKyZr",
    "outputId": "19f244ef-b133-4de6-d9a0-bd6c27c03c90",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MONAI version: 1.1.0+11.g7de6c336.dirty\n",
      "Numpy version: 1.22.2\n",
      "Pytorch version: 1.13.0+cu117\n",
      "MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False\n",
      "MONAI rev id: 7de6c33656a99087ca3b89a817b0879cf093febc\n",
      "MONAI __file__: /workspace/Code/MONAI/monai/__init__.py\n",
      "\n",
      "Optional dependencies:\n",
      "Pytorch Ignite version: 0.4.10\n",
      "Nibabel version: 4.0.2\n",
      "scikit-image version: 0.19.3\n",
      "Pillow version: 9.0.1\n",
      "Tensorboard version: 2.11.0\n",
      "gdown version: 4.6.0\n",
      "TorchVision version: 0.14.0+cu117\n",
      "tqdm version: 4.64.1\n",
      "lmdb version: 1.3.0\n",
      "psutil version: 5.9.2\n",
      "pandas version: 1.1.5\n",
      "einops version: 0.6.0\n",
      "transformers version: 4.21.3\n",
      "mlflow version: 2.0.1\n",
      "pynrrd version: 1.0.0\n",
      "\n",
      "For details about installing the optional dependencies, please visit:\n",
      "    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import shutil\n",
    "import tempfile\n",
    "import glob\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import numpy as np\n",
    "from monai.apps import download_and_extract\n",
    "from monai.config import print_config\n",
    "from monai.networks.nets import UNet\n",
    "from monai.networks.layers import Norm\n",
    "from monai.losses import DiceFocalLoss\n",
    "from monai.metrics import DiceMetric\n",
    "from monai.inferers import sliding_window_inference\n",
    "from monai.data import (\n",
    "    LMDBDataset,\n",
    "    DataLoader,\n",
    "    decollate_batch,\n",
    ")\n",
    "from monai.apps import load_from_mmar\n",
    "from monai.transforms import (\n",
    "    AsDiscrete,\n",
    "    EnsureChannelFirstd,\n",
    "    Compose,\n",
    "    LoadImaged,\n",
    "    ScaleIntensityRanged,\n",
    "    Spacingd,\n",
    "    Orientationd,\n",
    "    CropForegroundd,\n",
    "    RandCropByPosNegLabeld,\n",
    "    RandAffined,\n",
    "    RandRotated,\n",
    "    EnsureType,\n",
    "    EnsureTyped,\n",
    ")\n",
    "from monai.utils import set_determinism\n",
    "from monai.apps.mmars import RemoteMMARKeys\n",
    "\n",
    "print_config()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the flag, it will change the initial model object.\n",
    "PRETRAINED = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gybHjcBLKyZs"
   },
   "source": [
    "## Setup data directory\n",
    "\n",
    "You can specify a directory with the `MONAI_DATA_DIRECTORY` environment variable.  \n",
    "This allows you to save results and reuse downloads.  \n",
    "If not specified a temporary directory will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 4,
     "status": "ok",
     "timestamp": 1625519080912,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "IxiXEVmEKyZs",
    "outputId": "b22f9026-7df7-471d-99ff-f13b7523a32f",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/Data\n"
     ]
    }
   ],
   "source": [
    "directory = os.environ.get(\"MONAI_DATA_DIRECTORY\")\n",
    "root_dir = tempfile.mkdtemp() if directory is None else directory\n",
    "print(root_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-bFK3xLcKyZt"
   },
   "source": [
    "## Download dataset\n",
    "\n",
    "Downloads and extracts the dataset.\n",
    "The dataset comes from http://medicaldecathlon.com/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3026,
     "status": "ok",
     "timestamp": 1625519083935,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "musXbLIAKyZt",
    "outputId": "dabb2d74-285f-42f1-abc5-7b5b622ac8c1",
    "scrolled": false,
    "tags": []
   },
   "outputs": [],
   "source": [
    "resource = \"https://msd-for-monai.s3-us-west-2.amazonaws.com/Task09_Spleen.tar\"\n",
    "md5 = \"410d4a301da4e5b2f6f86ec3ddba524e\"\n",
    "\n",
    "compressed_file = os.path.join(root_dir, \"Task09_Spleen.tar\")\n",
    "download_and_extract(resource, compressed_file, root_dir, md5)\n",
    "data_dir = os.path.join(root_dir, \"Task09_Spleen\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ypEvK0aAKyZu"
   },
   "source": [
    "## Set MSD Spleen dataset path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "fQeUN3eyKyZu",
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_images = sorted(glob.glob(os.path.join(data_dir, \"imagesTr\", \"*.nii.gz\")))\n",
    "train_labels = sorted(glob.glob(os.path.join(data_dir, \"labelsTr\", \"*.nii.gz\")))\n",
    "data_dicts = [{\"image\": image_name, \"label\": label_name} for image_name, label_name in zip(train_images, train_labels)]\n",
    "train_files, val_files = data_dicts[:-9], data_dicts[-9:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UkQNk9yxKyZv"
   },
   "source": [
    "# Setup transforms for training and validation\n",
    "Relatively heavy data augmentations are used here to simulate a difficult training/validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "U1-_K-26KyZv",
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_transforms = Compose(\n",
    "    [\n",
    "        LoadImaged(keys=[\"image\", \"label\"]),\n",
    "        EnsureChannelFirstd(keys=[\"image\", \"label\"]),\n",
    "        Orientationd(keys=[\"image\", \"label\"], axcodes=\"RAS\"),\n",
    "        Spacingd(keys=[\"image\", \"label\"], pixdim=(1.5, 1.5, 2.0), mode=(\"bilinear\", \"nearest\")),\n",
    "        ScaleIntensityRanged(\n",
    "            keys=[\"image\"],\n",
    "            a_min=-57,\n",
    "            a_max=164,\n",
    "            b_min=0.0,\n",
    "            b_max=1.0,\n",
    "            clip=True,\n",
    "        ),\n",
    "        CropForegroundd(keys=[\"image\", \"label\"], source_key=\"image\"),\n",
    "        RandCropByPosNegLabeld(\n",
    "            keys=[\"image\", \"label\"],\n",
    "            label_key=\"label\",\n",
    "            spatial_size=(96, 96, 96),\n",
    "            pos=1,\n",
    "            neg=1,\n",
    "            num_samples=4,\n",
    "            image_key=\"image\",\n",
    "            image_threshold=0,\n",
    "        ),\n",
    "        RandAffined(\n",
    "            keys=[\"image\", \"label\"],\n",
    "            mode=(\"bilinear\", \"nearest\"),\n",
    "            prob=0.5,\n",
    "            spatial_size=(96, 96, 96),\n",
    "            rotate_range=(np.pi / 18, np.pi / 18, np.pi / 5),\n",
    "            scale_range=(0.05, 0.05, 0.05),\n",
    "        ),\n",
    "        EnsureTyped(keys=[\"image\", \"label\"]),\n",
    "    ]\n",
    ")\n",
    "val_transforms = Compose(\n",
    "    [\n",
    "        LoadImaged(keys=[\"image\", \"label\"]),\n",
    "        EnsureChannelFirstd(keys=[\"image\", \"label\"]),\n",
    "        Orientationd(keys=[\"image\", \"label\"], axcodes=\"RAS\"),\n",
    "        Spacingd(keys=[\"image\", \"label\"], pixdim=(1.5, 1.5, 2.0), mode=(\"bilinear\", \"nearest\")),\n",
    "        ScaleIntensityRanged(\n",
    "            keys=[\"image\"],\n",
    "            a_min=-57,\n",
    "            a_max=164,\n",
    "            b_min=0.0,\n",
    "            b_max=1.0,\n",
    "            clip=True,\n",
    "        ),\n",
    "        RandRotated(\n",
    "            keys=[\"image\", \"label\"],\n",
    "            mode=(\"bilinear\", \"nearest\"),\n",
    "            range_x=np.pi / 18,\n",
    "            range_y=np.pi / 18,\n",
    "            range_z=np.pi / 5,\n",
    "            prob=1.0,\n",
    "            padding_mode=(\"reflection\", \"reflection\"),\n",
    "        ),\n",
    "        CropForegroundd(keys=[\"image\", \"label\"], source_key=\"image\"),\n",
    "        EnsureTyped(keys=[\"image\", \"label\"]),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XMudxL8c6oG9"
   },
   "source": [
    "# Define LMDBDataset and DataLoader for training and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "8cgC2sZ6KyZw"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 32/32 [00:00<00:00, 46138.79it/s]\n",
      "100%|██████████| 9/9 [00:00<00:00, 29959.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accessing lmdb file: /workspace/mmar_cache/monai_cache.lmdb.\n",
      "Accessing lmdb file: /workspace/mmar_cache/monai_cache.lmdb.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "train_ds = LMDBDataset(data=train_files, transform=train_transforms, cache_dir=root_dir)\n",
    "# initialize cache and print meta information\n",
    "print(train_ds.info())\n",
    "\n",
    "# use batch_size=2 to load images and use RandCropByPosNegLabeld\n",
    "# to generate 2 x 4 images for network training\n",
    "train_loader = DataLoader(train_ds, batch_size=2, shuffle=True, num_workers=2)\n",
    "\n",
    "# the validation data loader will be created on the fly to ensure\n",
    "# a deterministic validation set for demo purpose.\n",
    "val_ds = LMDBDataset(data=val_files, transform=val_transforms, cache_dir=root_dir)\n",
    "# initialize cache and print meta information\n",
    "print(val_ds.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'map_addr': 0, 'map_size': 1099511627776, 'last_pgno': 885751, 'last_txnid': 41, 'max_readers': 126, 'num_readers': 0, 'size': 32, 'filename': '/workspace/mmar_cache/monai_cache.lmdb'}\n"
     ]
    }
   ],
   "source": [
    "print(train_ds.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_IbQ5UcDeSRm"
   },
   "source": [
    "# Download the pretrained model from Nvidia Clara Train\n",
    "\n",
    "By setting the global variable `PRETRAINED` to `True`, the following sections\n",
    "will use a pretrained model from the `clara_pt_spleen_ct_segmentation` MMAR.\n",
    "\n",
    "\n",
    "For more details, please see https://docs.monai.io/en/latest/apps.html#monai.apps.load_from_mmar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "9Sf7H2GWmSeS"
   },
   "outputs": [],
   "source": [
    "mmar = {\n",
    "    RemoteMMARKeys.ID: \"clara_pt_spleen_ct_segmentation_1\",\n",
    "    RemoteMMARKeys.NAME: \"clara_pt_spleen_ct_segmentation\",\n",
    "    RemoteMMARKeys.FILE_TYPE: \"zip\",\n",
    "    RemoteMMARKeys.HASH_TYPE: \"md5\",\n",
    "    RemoteMMARKeys.HASH_VAL: None,\n",
    "    RemoteMMARKeys.MODEL_FILE: os.path.join(\"models\", \"model.pt\"),\n",
    "    RemoteMMARKeys.CONFIG_FILE: os.path.join(\"config\", \"config_train.json\"),\n",
    "    RemoteMMARKeys.VERSION: 1,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3622,
     "status": "ok",
     "timestamp": 1625519087552,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "_CuUelsjOaeP",
    "outputId": "502b1e14-d5ae-41b6-f405-1d80a1f409d6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using a pretrained model.\n",
      "Expected md5 is None, skip md5 check for file ./mmar_cache/clara_pt_spleen_ct_segmentation_1.zip.\n",
      "File exists: ./mmar_cache/clara_pt_spleen_ct_segmentation_1.zip, skipped downloading.\n",
      "Non-empty folder exists in ./mmar_cache/clara_pt_spleen_ct_segmentation_1, skipped extracting.\n",
      "\n",
      "*** \"clara_pt_spleen_ct_segmentation_1\" available at ./mmar_cache/clara_pt_spleen_ct_segmentation_1.\n",
      "*** Model: <class 'monai.networks.nets.unet.UNet'>\n",
      "*** Model params: {'dimensions': 3, 'in_channels': 1, 'out_channels': 2, 'channels': [16, 32, 64, 128, 256], 'strides': [2, 2, 2, 2], 'num_res_units': 2, 'norm': 'batch'}\n",
      "\n",
      "---\n",
      "For more information, please visit https://ngc.nvidia.com/catalog/models/nvidia:med:clara_pt_spleen_ct_segmentation\n",
      "\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "if PRETRAINED:\n",
    "    print(\"using a pretrained model.\")\n",
    "    unet_model = load_from_mmar(item=mmar[RemoteMMARKeys.NAME], mmar_dir=root_dir, map_location=device, pretrained=True)\n",
    "    model = unet_model\n",
    "else:\n",
    "    print(\"using a randomly init. model.\")\n",
    "    model = UNet(\n",
    "        spatial_dims=3,\n",
    "        in_channels=1,\n",
    "        out_channels=2,\n",
    "        channels=(16, 32, 64, 128, 256),\n",
    "        strides=(2, 2, 2, 2),\n",
    "        num_res_units=2,\n",
    "        norm=Norm.BATCH,\n",
    "    )\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RdzdCygTPdc9"
   },
   "source": [
    "### Alternative transfer learning setup\n",
    "Using the MONAI modules, it is also possible to set\n",
    "the learning rates for different layers for transfer learning.\n",
    "\n",
    "```python\n",
    "from monai.networks.utils import copy_model_state\n",
    "from monai.optimizers import generate_param_groups\n",
    "\n",
    "# copy all the pretrained weights except for variables whose name matches \"model.0.conv.unit0\"\n",
    "pretrained_dict, updated_keys, unchanged_keys = copy_model_state(\n",
    "    model, unet_model, exclude_vars=\"model.0.conv.unit0\")\n",
    "print(\"num. var. using the pretrained\", len(updated_keys), \", random init\", len(unchanged_keys), \"variables.\")\n",
    "model.load_state_dict(pretrained_dict)\n",
    "\n",
    "print([x[0] for x in model.named_parameters()])\n",
    "print(unchanged_keys)\n",
    "\n",
    "# stop gradients for the pretrained weights\n",
    "for x in model.named_parameters():\n",
    "    if x[0] in updated_keys:\n",
    "        x[1].requires_grad = False\n",
    "params = generate_param_groups(\n",
    "    network=model,\n",
    "    layer_matches=[lambda x: x[0] in updated_keys],\n",
    "    match_types=[\"filter\"],\n",
    "    lr_values=[1e-4],\n",
    "    include_others=False\n",
    ")\n",
    "optimizer = torch.optim.Adam(params, 1e-5)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GcfAgnCbPLvN"
   },
   "source": [
    "## Create an optimizer and a loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "x6W5lOcMJbo8"
   },
   "outputs": [],
   "source": [
    "loss_function = DiceFocalLoss(to_onehot_y=True, softmax=True)\n",
    "optimizer = torch.optim.Adam(model.parameters(), 5e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6FQ5DHsxKyZw"
   },
   "source": [
    "## Model training\n",
    "\n",
    "Execute a typical PyTorch training that run epoch loop and step loop, and do validation after other epoch.  \n",
    "Will save the model weights to file if got best validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gmrPkMd_KyZw",
    "tags": []
   },
   "outputs": [],
   "source": [
    "max_epochs = 50\n",
    "val_interval = 2\n",
    "best_metric = -1\n",
    "best_metric_epoch = -1\n",
    "epoch_loss_values = []\n",
    "metric_values = []\n",
    "post_pred = Compose([EnsureType(), AsDiscrete(argmax=True, to_onehot=2)])\n",
    "post_label = Compose([EnsureType(), AsDiscrete(to_onehot=2)])\n",
    "dice_metric = DiceMetric(include_background=False, reduction=\"mean\", get_not_nans=False)\n",
    "\n",
    "for epoch in range(max_epochs):\n",
    "    print(\"-\" * 10)\n",
    "    print(f\"epoch {epoch + 1}/{max_epochs}\")\n",
    "    model.train()\n",
    "    epoch_loss = 0\n",
    "    step = 0\n",
    "    set_determinism(seed=42)\n",
    "    for batch_data in train_loader:\n",
    "        step += 1\n",
    "        inputs, labels = (\n",
    "            batch_data[\"image\"].to(device),\n",
    "            batch_data[\"label\"].to(device),\n",
    "        )\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(inputs)\n",
    "        loss = loss_function(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        epoch_loss += loss.item()\n",
    "        print(f\"{step}/{len(train_ds) // train_loader.batch_size}, \" f\"train_loss: {loss.item():.4f}\")\n",
    "    epoch_loss /= step\n",
    "    epoch_loss_values.append(epoch_loss)\n",
    "    print(f\"epoch {epoch + 1} average loss: {epoch_loss:.4f}\")\n",
    "\n",
    "    if (epoch + 1) % val_interval == 0:\n",
    "        model.eval()\n",
    "        with torch.no_grad():\n",
    "            set_determinism(seed=42)\n",
    "            for val_data in DataLoader(val_ds, batch_size=1, num_workers=2):\n",
    "                val_inputs, val_labels = (\n",
    "                    val_data[\"image\"].to(device),\n",
    "                    val_data[\"label\"].to(device),\n",
    "                )\n",
    "                roi_size = (160, 160, 160)\n",
    "                sw_batch_size = 4\n",
    "                val_outputs = sliding_window_inference(val_inputs, roi_size, sw_batch_size, model, overlap=0.5)\n",
    "                val_outputs = [post_pred(i) for i in decollate_batch(val_outputs)]\n",
    "                val_labels = [post_label(i) for i in decollate_batch(val_labels)]\n",
    "                dice_metric(y_pred=val_outputs, y=val_labels)\n",
    "            metric = dice_metric.aggregate().item()\n",
    "            dice_metric.reset()\n",
    "            metric_values.append(metric)\n",
    "            if metric > best_metric:\n",
    "                best_metric = metric\n",
    "                best_metric_epoch = epoch + 1\n",
    "                torch.save(model.state_dict(), os.path.join(root_dir, \"best_metric_model.pth\"))\n",
    "                print(\"saved new best metric model\")\n",
    "            print(\n",
    "                f\"current epoch: {epoch + 1} current mean dice: {metric:.4f}\"\n",
    "                f\"\\nbest mean dice: {best_metric:.4f} \"\n",
    "                f\"at epoch: {best_metric_epoch}\"\n",
    "            )\n",
    "print(f\"train completed, best_metric: {best_metric:.4f} \" f\"at epoch: {best_metric_epoch}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b-P6bmD9KyZx"
   },
   "source": [
    "## Plot the loss and metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 232
    },
    "executionInfo": {
     "elapsed": 16,
     "status": "error",
     "timestamp": 1625521679645,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "u7ge6508KyZx",
    "outputId": "2ba5aa42-fec4-4177-c93a-705d51fdd04e"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAGDCAYAAAAh/naNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAyCElEQVR4nO3deZhcd33n+/e3qnrT2rIlG2vxgi1iDCE2CAMxJA5LYhjGZm5Y7AQCE8CTTExgyJ3EzuQSQu7MXJJ5SJgbJ+AQLpAAxiEhoxDnGsIaSAAJMIttbAsvaMHWYm0tqdf6zh/ndKvctKSW1FvVeb8e1VNnqzrfU1361ad+dZbITCRJkqSqqc13AZIkSdJ8MAhLkiSpkgzCkiRJqiSDsCRJkirJICxJkqRKMghLkiSpkgzCmlURkRFx0XzXIUk6eZ3YhkfEXRFx5XzXoYXBIFwhEfFQRByJiIGW25/Md12TRcTrysb3VfNdy+mKiPPLbWnMdy2Sqici/v+IeMcU06+JiEdOp22KiM+X7dtPTJr+iXL6laf63KdYz3h7O/759mhEfDIiXtS6XGY+JTM/P5e1aeEyCFfPv83MJS23G+a7oCm8FngM+KXZeHJDqaQK+SDw6oiISdNfA3w4M0dP8/nvo6WtjogzgecAu07zeU9Hf2YuAX4C+DTwiYh43TzWowXMICxgohf2yxHxJxGxPyK+FxEvaJm/OiI2RsRjEbElIt7YMq8eEb8dEd+PiIMR8fWIWNfy9C+MiPsjYl9E3DxFg9xax3nATwPXAz8XEU8op/9ZRPyPScv+r4h4a0t9fxMRuyLiwYj49Zbl3h4RH4+Iv4qIA8DrIuLyiPjXsqYfltvd3fKYn42Ie8vX4k8j4gsR8YaW+b8cEfdExN6IuKOs+2Rf8+O9ppdHxOaIOFD2aryrnN5bbseesvZNEXH2ya5bUmX8HXAm8LzxCRGxAngp8KETtYXT8GHgVRFRL8evAz4BDLesrxYRN5afEXsi4raIOKNl/l+XvdP7I+KLEfGUlnkfKD83/qH8fPlqRFw4ncIy85HMfDfwduCdEVErn/OhiHhhOXzMz6+IuDgiPl220fdGxCtP4nVRmzAIq9WzgO8DK4HfBf62pbG6FdgGrAZeDvy3iHh+Oe+tFI3fS4BlwC8Dh1ue96XAM4GnAa8Efu44NfwSsDkz/wa4B/jFcvpHKRrbgImG/GeBW8vG7e+BbwFrgBcAb4mI1vVcA3wc6KdouMeA/1Ru63PKx/zH8rlXlsveRPEBci/wk+NPFBHXAL8N/B/AKuCfy/pO1vFe03cD787MZcCFwG3l9NcCy4F1ZW2/Ahw5hXVLqoDMPELRfrT+wvZK4HuZ+S2O0xZO0w7gbor2mHI9H5q0zJuAl1F0cqwG9gI3t8z/R2A9cBbwDYo2utW1wO8BK4AtwH89ifoA/rZ87h+bYt6Un18RsZiiN/kj5WOvBf40Ii45yXVroctMbxW5AQ8BA8C+ltsby3mvo2jQomX5r1H8fLaOorFc2jLvvwMfKIfvBa45xjoTeG7L+G3Ajcep8X7gLeXwTcC3yuEAfgD8VDn+RuCz5fCzgB9Mep6bgP+vHH478MUTvDZvAT5RDv8S8K8t8wLYCryhHP9H4PUt82sUwf+8KZ73/PI1aEyafqLX9IsUDf/KSY/7ZeBfgKfN9/vJmzdv7XEDnlu2973l+JeB/3SMZSfawnI8gYuOsezngTcAr6boDLgYuK+ctw24shy+B3hBy+POAUYmt4vlvP5yncvL8Q8A72uZ/xKKED9VPcdqb3vL6VeU4w8BLyyHp/z8Al4F/POkae8Ffne+/57eZvZmj3D1vCwz+1tuf94yb3uW/9tLD1N8e18NPJaZByfNW1MOr6PoST6WR1qGDwNLplooIq4ALqDoKYXim/iPR8SlZV23UnxzB/gFjvYanAesLn/a2xcR+yh6bFt3Gdg6aV1PiuIgikfK3SX+G0WPCOX2Tixfrntby8PPA97dsq7HKMLyGqbvRK/p64EnAd8rd394aTn9L4E7KHrCd0TEH0RE10msV1LFZOaXgN3Ay8rdCi6naF9P1BZO198CzwduoGijJjuPYj/d8TbzHoqOgLPLXRP+n3LXhAMUIZVJNUzrM+Q4xtvVx6aYd6zPr/OAZ036XPlF4AknuW4tcAZhtVozvutB6VyKXuIdwBkRsXTSvO3l8FaKn+9P12spAuWdEfEI8NWW6VD0OLy83B/3WcDftKz/wUkBf2lmvqTluVsDPsCfAd8D1mex+8Fvl+sG+CGwdnzB8jVZ2/LYrcB/mLS+vsz8l5PY1uO+ppl5f2ZeR/GT3DuBj0fE4swcyczfy8xLKHbXeCmzdFChpI7yIYq24tXAHZn5aDn9eG3htGTmYYpfyn6VqYPwVuDFk9rM3szcTtGpcQ3wQordvs4vH3NSNZzAvwN2UvT+TlXbVJ9fW4EvTKp5SWb+6gzWpQXAIKxWZwG/HhFdEfEK4MnA7Zm5leLn+P9eHqz1NIoey78qH/c+4PcjYn0UnhbFkcPTFhG9FPutXQ9c2nJ7E/ALEdHIzG9S9Gq8j6Ih31c+/GvAwYj4rYjoK3sYnhoRzzzOKpcCB4CBiLiYogEf9w8UPdEvi+IME7/G43sB3gPcNH5AR0QsL1+v4+kpX7veclu3c5zXNCJeHRGrMrNJ8ZMmQDMifiYifrw8MOUAxc+LzROsW5I+RBE230hxJolxx2sLT8ZvAz+dmQ9NMe89wH8tOzGIiFXlsRbj6x8C9gCLKHqkZ0REnB0RN1Ac83JT2Z5OdqzPr08CT4qI15SfiV0R8cyIePJM1aeFwSBcPX8fjz+P8Cda5n2V4oCF3RQHI7w8M/eU866j+Ka+g+KI4N/NzH8q572LYt/fT1E0qH8B9J1kXS+jOOjrQ1kc6ftIZj4CvB9oAFeVy32EojH/yPgDM3OMomf0UuBBjobl5cdZ3/9J0RNxEPhz4GMtz7cbeAXwBxSN8yXAZorGmsz8BEUv7a3lT3nfBV58gu0bKLdv/PZ8jv+aXgXcFREDFAfOXZvFQS9PoDiQ7wDFz4tfYOoeGEmaUAbUfwEWAxtbZh2zLTzJ599R7oIxlXeX6/xURBwEvkLxqx4UAf1his6Bu8t5p2tfRBwCvkOxT/ErMvP9x1h2ys+vcre1n6U4SG4Hxe4Z7wR6ZqA+LSDx+F1CVVVRnGPxDZn53PmuZaEpz0qxDfjFzPzcfNcjSZJmhj3C0hQi4ucioj8ieji6z9xM9FRIkqQFYlpBOCKuKk8mvSUibpxi/h9FxJ3l7b7y6EqpnT2H4kji3cC/pTjbhufrVVuIiPdHxM6I+O4x5kdE/M+yTf92RDx9rmuUpIXghLtGlAfl3Ae8iOLn4U3AdZl59zGWfxNwWWb+8gzXKkmahoj4KYr90j+UmU+dYv5LKA5EfQnFvprvzsxnTV5OkjrddHqELwe2ZOYDmTlMcS7Xa46z/HWc2lW2JEkzIDO/yNTnTB13DUVIzsz8CtAfEefMTXWStHBMJwiv4fEXI9jGMS4cUJ4a5QLgs6dfmiRplky7XZekTtaY4ee7Fvh4eTqrHxER11OcJ5bFixc/4+KLL57h1UvS7Pv617++OzNXzXcds802W1KnOFa7PZ0gvJ3iEoTj1nL0imKTXUtx8YEpZeYtwC0AGzZsyM2bN09j9ZK0sETEw/Ndw2maVrtumy2pUxyr3Z7OrhGbgPURcUFEdFOE3Y2TFyqvSLMC+NfTKVSSNOs2Ar9Unj3i2cD+zPzhfBclSXPthD3CmTlaXqLwDqAOvD8z74qIdwCbM3M8FF8L3JpeoUOS5lVEfBS4ElgZEdsoLjHbBZCZ7wFupzhjxBbgMPDv56dSSZpf09pHODNvp2g4W6e9bdL422euLEnSqcrM604wPznObmySVBVeWU6SJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJTXmuwBJUrVlJvfvHOCrDz7G/sPDDI82GRprMjTSZHisyfBoy22sydDoGMOjTQCW9DRY0tvFkp4GS3sbxXhPgyW9DZaW98W8Lpb1NljW10VvV33Wt6nZTJqZjGXSbNIynDSzGM+EpLwfn1Y+Fo4uMzg6xqGhUQaGxjg8NMrA0CiHhkY5NFxMbx0+MjLGWDMnnq91uLgVr/fj6moW42PN4tZsJqPj9Tdbbpktf7OWv9/j/phHBxv1oK+rTm9Xnb7uOn1dxa23u05fV60Y7y7mj/9Nxre59XUan95af29XnZVLejhzcTdnLunhzCXdrFxc3C/qrhMRx/zbHBwc4Yf7B4vbviPl8JGJaV31Gpeu6+fp5/Zz2bkreOLKxdRqx34+tTeDsCRpzj16YJAvb9nNl+7fzZe27GbnwaHHze9p1Ohu1Ir7+vhwne5yene9+EFzz6FhHt5zmINDowwMFkHwRLobNZb3FcF4eV8Xy/q6yvHifnlfFz1dNQ4PjwfPMQ4PPz54Hh4fHh7l8NAYw2NNMpkIlHOlXgsWd9dZ0tNgcU+Dvu469VpQi6AWEBHUatBVq1GLKMaDifm1COq1llvLeG3SeL0WPC4ORuvg0ZHxDDo61uTIyBhHhpsMjoyVw2McODLCzgNjHB4upg0OjzE4OkYQxKTamDQeUSwzODzGwaHRKV+T3q4aZy7uYeWSIiQv622w59Awj5RBd2DS4yJg5ZIeVi/v5cJVizk8PMYnv72Dj37tBwAs7+sqg/EKLju3n59Y18/yvq6Z+PNNODA4wo59R9ix7wjb9w2yY98RHtk/SDNz4u/SqBd/g0atVt5Hy33x/6J/URf95Xt4+aIu+hd109/XdcIvB1VmEJYkzbqBoVG+9uAe/vn+3Xx5y27ue3QAgDMWd3PFRSt57kVn8pMXruTsZb101eOUP7RHx5ocGhrjwOAIA2Xv6cDgKAcGRzg4OMr+IyMcGBzhwJGRYvjIKHsGhnlw96FyfITWHBsBi7sbLCrD5qKeOou6G6xa2sN5Zy4q5vXU6a7XqNWKwFaPKIdjIpTWa+OBrgx1ZagcD3a1KMZbw1+U6+/tqrO4p8Hi7uJ+SU9Rz+KeBj2NWmUDzuDIGI8dGmbPwDC7Dw2xZ2CYPQND7Dk0zO6BYnznwUHue3SEMxd388RVi7niopWcs7yXc/r7OGd5L09Y1svZy3rpbjx+T9FmM3lg9wDfeHgf39y6l288vI8//sx9ZBZ/k4tWLZkIxUt6GsXfkWLeeKAfHx//xhABR4bH2F4G3uJWhN7Job5Ri4n/C6PNx/fMj07cNxlrJiNjJ/7i1agF/YuOftHrX9TNWUt7eMLy3uJ1WF6+Hst7WVpuz3RkJoeGx9gzMMTultd/sPxlYrSZjI41W2r+0fGxsfFtap5wW8eayc8+5Qn8yk9fOK36psMgLEmaFd/dvp/P3LOTL2/ZzTd+sJfRZtLTqHH5BWfw809fy3PXr+TJT1g2oz87N+o1li+qsXzRqfXYZSYDQ6MMjjRZ3FP8lF/VoLnQ9XbVWd3fx+r+vhl/7lotuOispVx01lJe+cx1QLFLxbe37ecbD+/lm1v38em7H+W2zdtO6flXLOpidX8f5565iOdceCar+3sntmVNfx8rl/RQP4n/F81mMjg6xv4jI+w7XNz2Hxlh/5HhieF9R0bYXw4/emCQ72zfz+6Bocft5gKwuLteBuS+iaC8pKfBY4eHH/dlY89A8YVjqNxN6URqweN6s4se7hr1WjF9vMd7/FeI8flHe75r9HYFPY2ZPbzNICxJmhUf/urD3LppK09dvZw3/tQTed5FK3n6eSvmZB/dUxURLO3tYmnvfFeihWZpbxdXXLSSKy5aCRRfmrbvO8LQaLMMk+P7ffMj+3+PDxfhvZdF3TMbv2q1YFF3g0XdDc5ZPv0vBsOjTXYeHJzYbeSR/YPs2H9kYvxL9+9m58FBmgnd9RpnLukubot7uOisJVPup33Gkm4WddVp1B+/G8dC3c/aICxJmhW//oL1/ObPXcyKxd3zXYo04yKCtSsWzXcZp6W7UWPtikXH3Y7xfb2XnMQuE+3EICxJmhUn0zMlaWFq1GssrXfu2XY7d8skSZKk4zAIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSppWEI6IqyLi3ojYEhE3HmOZV0bE3RFxV0R8ZGbLlCRJkmZW40QLREQduBl4EbAN2BQRGzPz7pZl1gM3AVdk5t6IOGu2CpYkSZJmwnR6hC8HtmTmA5k5DNwKXDNpmTcCN2fmXoDM3DmzZUqSJEkzazpBeA2wtWV8Wzmt1ZOAJ0XElyPiKxFx1VRPFBHXR8TmiNi8a9euU6tYkiRJmgEzdbBcA1gPXAlcB/x5RPRPXigzb8nMDZm5YdWqVTO0akmSJOnkTScIbwfWtYyvLae12gZszMyRzHwQuI8iGEuSJEkL0nSC8CZgfURcEBHdwLXAxknL/B1FbzARsZJiV4kHZq5MSZIkaWadMAhn5ihwA3AHcA9wW2beFRHviIiry8XuAPZExN3A54D/nJl7ZqtoSZIk6XSd8PRpAJl5O3D7pGlvaxlO4K3lTZIkSVrwvLKcJEmSKskgLEkd5kRXA42IcyPicxHxzYj4dkS8ZD7qlKT5ZhCWpA7ScjXQFwOXANdFxCWTFvsdiuM9LqM4APpP57ZKSVoYDMKS1FmmczXQBJaVw8uBHXNYnyQtGNM6WE6S1DamuhrosyYt83bgUxHxJmAx8MK5KU2SFhZ7hCWpeq4DPpCZa4GXAH8ZET/yeRAR10fE5ojYvGvXrjkvUpJmm0FYkjrLdK4G+nrgNoDM/FegF1g5+Yky85bM3JCZG1atWjVL5UrS/DEIS1Jnmc7VQH8AvAAgIp5MEYTt8pVUOQZhSeog07wa6G8Ab4yIbwEfBV5XXhhJkirFg+UkqcNM42qgdwNXzHVdkrTQ2CMsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaqkaQXhiLgqIu6NiC0RceMU818XEbsi4s7y9oaZL1WSJEmaOY0TLRARdeBm4EXANmBTRGzMzLsnLfqxzLxhFmqUJEmSZtx0eoQvB7Zk5gOZOQzcClwzu2VJkiRJs2s6QXgNsLVlfFs5bbKfj4hvR8THI2LdVE8UEddHxOaI2Lxr165TKFeSJEmaGTN1sNzfA+dn5tOATwMfnGqhzLwlMzdk5oZVq1bN0KolSZKkkzedILwdaO3hXVtOm5CZezJzqBx9H/CMmSlPkiRJmh3TCcKbgPURcUFEdAPXAhtbF4iIc1pGrwbumbkSJUmSpJl3wrNGZOZoRNwA3AHUgfdn5l0R8Q5gc2ZuBH49Iq4GRoHHgNfNYs2SJEnSaTthEAbIzNuB2ydNe1vL8E3ATTNbmiRJkjR7vLKcJEmSKskgLEmSpEoyCEuSJKmSDMKSJEmqJIOwJEmSKskgLEmSpEoyCEuSJKmSDMKSJEmqJIOwJEmSKskgLEmSpEoyCEuSJKmSDMKSJEmqJIOwJEmSKskgLEkdJiKuioh7I2JLRNx4jGVeGRF3R8RdEfGRua5RkhaCxnwXIEmaORFRB24GXgRsAzZFxMbMvLtlmfXATcAVmbk3Is6an2olaX7ZIyxJneVyYEtmPpCZw8CtwDWTlnkjcHNm7gXIzJ1zXKMkLQgGYUnqLGuArS3j28pprZ4EPCkivhwRX4mIq6Z6ooi4PiI2R8TmXbt2zVK5kjR/DMKSVD0NYD1wJXAd8OcR0T95ocy8JTM3ZOaGVatWzW2FkjQHDMKS1Fm2A+taxteW01ptAzZm5khmPgjcRxGMJalSDMKS1Fk2Aesj4oKI6AauBTZOWubvKHqDiYiVFLtKPDCHNUrSgmAQlqQOkpmjwA3AHcA9wG2ZeVdEvCMiri4XuwPYExF3A58D/nNm7pmfiiVp/nj6NEnqMJl5O3D7pGlvaxlO4K3lTZIqyx5hSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlTSsIR8RVEXFvRGyJiBuPs9zPR0RGxIaZK1GSJEmaeScMwhFRB24GXgxcAlwXEZdMsdxS4M3AV2e6SEmSJGmmTadH+HJgS2Y+kJnDwK3ANVMs9/vAO4HBGaxPkiRJmhXTCcJrgK0t49vKaRMi4unAusz8hxmsTZIkSZo1p32wXETUgHcBvzGNZa+PiM0RsXnXrl2nu2pJkiTplE0nCG8H1rWMry2njVsKPBX4fEQ8BDwb2DjVAXOZeUtmbsjMDatWrTr1qiVJkqTTNJ0gvAlYHxEXREQ3cC2wcXxmZu7PzJWZeX5mng98Bbg6MzfPSsWSJEnSDDhhEM7MUeAG4A7gHuC2zLwrIt4REVfPdoGSJEnSbGhMZ6HMvB24fdK0tx1j2StPvyxJkiRpdnllOUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUnqMBFxVUTcGxFbIuLG4yz38xGREbFhLuuTpIXCICxJHSQi6sDNwIuBS4DrIuKSKZZbCrwZ+OrcVihJC4dBWJI6y+XAlsx8IDOHgVuBa6ZY7veBdwKDc1mcJC0kBmFJ6ixrgK0t49vKaRMi4unAusz8h+M9UURcHxGbI2Lzrl27Zr5SSZpnBmFJqpCIqAHvAn7jRMtm5i2ZuSEzN6xatWr2i5OkOWYQlqTOsh1Y1zK+tpw2binwVODzEfEQ8GxgowfMSaoig7AkdZZNwPqIuCAiuoFrgY3jMzNzf2auzMzzM/N84CvA1Zm5eX7KlaT5YxCWpA6SmaPADcAdwD3AbZl5V0S8IyKunt/qJGlhacx3AZKkmZWZtwO3T5r2tmMse+Vc1CRJC5E9wpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaqkaQXhiLgqIu6NiC0RceMU838lIr4TEXdGxJci4pKZL1WSJEmaOScMwhFRB24GXgxcAlw3RdD9SGb+eGZeCvwB8K6ZLnRcZs7WU0uSJKlCptMjfDmwJTMfyMxh4FbgmtYFMvNAy+hiYFbS6jU3f5nf/Pi3Z+OpJUmSVDGNaSyzBtjaMr4NeNbkhSLi14C3At3A86d6ooi4Hrge4Nxzzz3ZWumqBVv3Hj7px0mSJEmTzdjBcpl5c2ZeCPwW8DvHWOaWzNyQmRtWrVp10utYs6KPHfsGT7NSSZIkaXpBeDuwrmV8bTntWG4FXnYaNR3T6v4+frj/CM2m+wlLkiTp9EwnCG8C1kfEBRHRDVwLbGxdICLWt4z+G+D+mSvxqNX9fYyMJbsGhmbj6SVJklQhJ9xHODNHI+IG4A6gDrw/M++KiHcAmzNzI3BDRLwQGAH2Aq+djWLX9vcBsG3vEc5e1jsbq5AkSVJFTOdgOTLzduD2SdPe1jL85hmua0qryyC8Y98RnnHeirlYpSRJkjpUW11ZbnV/0Qu8fd+Rea5EkiRJ7a6tgvDS3i6W9TbYYRCWJEnSaWqrIAywZsUitu81CEuSJOn0tF8Q7u911whJkiSdtjYMwn3uGiFJkqTT1nZBeHV/HwcGRzk4ODLfpUiSJKmNtV0QXrNi/BRqXmpZkiRJp67tgvD4uYS37zs8z5VIkiSpnbVdEF47EYTtEZYkSdKpa7sgvHJJD1318BRqkiRJOi1tF4RrteCc5Z45QpIkSaen7YIwFKdQ81zCkiRJOh1tGYRXey5hSZIknaa2DMJrVvTx6IFBRsaa812KJEmS2lR7BuH+XpoJj+z3zBGSJEk6NW0ahBcBuHuEJEmSTllbBuHV/b0AHjAnSZKkU9amQXj8MssGYUmSJJ2atgzCvV11Vi7ptkdYkiRJp6wtgzCMn0vYg+UkSZJ0ato2CK/u72P73sPzXYYkSZLaVFsH4R37BsnM+S5FkiRJbahtg/Ca/j6OjIyx9/DIfJciSZKkNtS2QdgzR0iSJOl0tG0QXruiCMKeOUKSJEmnom2D8HiP8Pa9BmFJkiSdvLYNwisWddHXVXfXCEmSJJ2Stg3CEcHq/l53jZAkSdIpadsgDLBmxSJ7hCVJknRK2jsI2yMsSZKkU9TmQbiP3QPDDI6MzXcpkiRJajNtHYQ9l7AkSZJOVVsH4TUTQXhwniuRJElSu2nrIDxxLuF9h+e5EkmSJLWbtg7CT1jeSy1guz3CkjQhIq6KiHsjYktE3DjF/LdGxN0R8e2I+ExEnDcfdUrSfGvrINxVr3H2sl73EZakUkTUgZuBFwOXANdFxCWTFvsmsCEznwZ8HPiDua1SkhaGtg7CUOwe4WWWJWnC5cCWzHwgM4eBW4FrWhfIzM9l5vg+ZV8B1s5xjZK0ILR9EF7T38eO/QZhSSqtAba2jG8rpx3L64F/nNWKJGmBavsgvLq/jx/uG6TZzPkuRZLaSkS8GtgA/OEx5l8fEZsjYvOuXbvmtjhJmgNtH4TXrOhjeKzJ7oGh+S5FkhaC7cC6lvG15bTHiYgXAv8FuDozp2xAM/OWzNyQmRtWrVo1K8VK0nxq/yDc3wvANg+YkySATcD6iLggIrqBa4GNrQtExGXAeylC8M55qFGSFoS2D8Lrz1oKwD/ft3ueK5Gk+ZeZo8ANwB3APcBtmXlXRLwjIq4uF/tDYAnw1xFxZ0RsPMbTSVJHa8x3Aadr3RmLeOGTz+ID//Igb/ypC1jU3fabJEmnJTNvB26fNO1tLcMvnPOiJGkBavseYYBf+ekL2Xt4hNs2bT3xwpIkSRIdEoQ3nH8Gzzx/BX/+zw8yMtac73IkSZLUBqYVhNvhcp2/euWFbN93hL//1o65XrUkSZLa0AmDcLtcrvNnfuwsfuzspbznC9/3nMKSJEk6oen0CLfF5Tojgl+58onc9+gAn7vXswFJkiTp+KYThGfscp2zfZWilz5tNWv6+/izz39/xp9bkiRJnWVGD5Y70eU6Z/sqRV31Gm983gVsfngvmx56bMafX5IkSZ1jOkF4xi7XORde9cxzOWNxN++xV1iSJEnHMZ0g3FaX6+zrrvO6nzyfz3xvJ9975MB8liJJkqQF7IRBuB0v1/lLzzmPRd11/vRz3yfTM0hIkiTpR03resTtdrnO/kXdvOY55/HeLzwAwP/9757Kst6uea5KkiRJC8m0gnA7+s2fu5gl3Q3++DP3882te3n3tZfx9HNXzHdZkiRJWiA64hLLU6nXgje9YD23/Ydn02zCK97zr9z8uS2MebENSZIk0cFBeNwzzjuD29/8PF781Cfwh3fcy6vf91Ue3nNovsuSJEnSPOv4IAywvK+L//e6y/iDn38ad27dx5X/4/O85i++yu3f+SHDo835Lk+SJEnzoGP3EZ4sInjlM9fxvCet5GObtnLbpq38xw9/gzMXd/PyZ6zlVc9cxxNXLZnvMiVJkjRHKhOEx52zvI+3vPBJvOn56/ni/bu49Ws/4H1fepD3fvEBnnzOMl50ydm86Mln89Q1y4iI+S5XkiRJs6RyQXhcvRb8zI+dxc/82FnsPDjIxjt38Km7HuVPPns///Mz93PO8l5e+OSzef7FZ/GM81d4+jVJkqQOU9kg3Oqspb284XlP5A3PeyJ7Bob47Pd28um7H+XjX9/GX37lYWoBFz9hGZdfcAaXX3AGzzz/DFYt7ZnvsiVJknQaDMKTnLmkh1dsWMcrNqxjcGSMrz+8l689+BibHnqMj23aygf+5SEA1vT38eRzlvGU1cu4ZPUyLjlnGWtX9AEwONLk0PAoh4ZGOTQ0Rk9XjXPPWERXvRLHJkqSJLUFg/Bx9HbVueKilVxx0UoARsaafHf7fjY/tJfv7tjPXTsO8NnvPcr4qYl7GjVGxppMdariRi04f+ViLly1mIvOWsKFq5bw5HOWcdFZSwzIkiRJ88AgfBK66jUuO3cFl7Vcoe7I8Bj3PnqQu3cc4KE9h+hp1FjU3WBxT53F5f3A0Bjf3zXA93cOcP/OAf7pnp0TF/bobtS4+AlLecrq5Tx1zTKedPZSFnc36Omq0dtVp6dR3Pd11anXPHhPkiRpphiET1Nfd51L1/Vz6br+aT9meLTJw3sOcfcPD3DXjgN8d/t+/uHbO/jo135wzMfUAs5Y3MPKJd2sXFLcn7mkh6W9jTJwHw3fi7rr9HTV6e2q0dM4et/TVaO7XtxqU4TqsWYyPNqcOLfy0t7GlMtJkiR1AoPwPOhu1Fh/9lLWn72Uay5dA0Bmsm3vEbbsGmBoZIzBkSZDo2MMjTYZHBnj4OAouweG2D0wzO6BIR7+wSH2DAxzeHjslGqo14Lueo1GPRhrJkOjzR+5/HSjFpyxuAjc4wF8aW+D3q46vY1aGbaLoN2oBfVajXoNahE0asVzj/dojy/X26iTMBG4h8fGJta9qLteBPruRhHwexruNiJJkmaNQXiBiAjWnbGIdWcsOqnHjTWTw8PFQXmtB+gNjbaE6ZEmg6NjZfBsMjKaDI+NMTJW9AB31YPuRo3uep3uRo2eRo1mJo8dGmbPwDB7DhUB/MHdhxgYGmWwDOpzobtRo6/cNWRRdxGo+7rrNGpBLYJ6LYhgYvho8B7vDa/TXQ9qtaAe5X05HFG87rWA4Ojw+LL12tFbJgyVr+FQ+ToOjzYn1ttVL74IdNWDrnqNxT0NlvYUYX5JTxHs67XiS0czk9Fm0mwmY5mMjiUjY8WXgZGxnPhCsrinPvGFYEn5BaFWC7J8/FizvB9LBkfHJv4uR0aK4aA4+HPVkh6W9TVO6rzYmUkzi/viNX78Y8eayaHhUQYGi/fcwaFRhkaaLO/r4ozF3fQv6qK3qz6D7wRJkmaeQbjN1WvB0t4uls7xeY4zi17k8ZA9HuzGA1oR6h4fxAdHit7tWhRhsbtR3sqe6SPDYwwMjTIwVISrgcFRDg0Xoe7w8ChHRpocGR7jyMgoo2PJaLPJ8Bg08+i6h0abE18CBkeKdQ+Pdc5ltGvBlAdjnkhXPThzcQ8rl3ZTjyhenzI8j/8dx0P6WCY5xTrqtfJLQ8S0Lk2+uLvOisXdLO3tonv8y1ajRle9uI0H+vEvAqPl+rvL90ZPo/gy092oUa8FzUwo/pGZJFCPoFF++eiql79M1IOg/KID5X2U74+jX2aGRpsMj47xy1dcwLOeeObJv6iSpLZnENYpiYiJXR6Ws/AvNjLe+zoR9ppFj2cRrI72fjaTifkTt0wC6OmqPy6kdddrJEz05h4N582il3Sw6J0fGBrh4OAoY818XC/zeM90o/wi0Kgd3aUkk4me1ole18ERmlkE0vHAN75LyvhuJ0WPeTHcTNhzaIhdB4/uUrNnYIiEoz3n5b7jPY0a9VqNWowH3uIWcfSLRjNhLIvXr7fx+N7qJb0Neuo1DgyO8NihER47NMRjh0bYe3iYg4OjjJQ96EMjTQ4Ojk70phc96cVr0NtVo1aG7IODo+wpv9QMjzUZG0tiohe/+AUAmHjdR8aaE2F6tJkTf9dsCc6Neo2eeq3c3uJA1O5GjUPDo/P3xpQkzSuDsCqhVgtqBLPxa3295i4AkiS1I49EkiRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlTStIBwRV0XEvRGxJSJunGL+T0XENyJiNCJePvNlSpKmaxptdk9EfKyc/9WIOH8eypSkeXfCIBwRdeBm4MXAJcB1EXHJpMV+ALwO+MhMFyhJmr5pttmvB/Zm5kXAHwHvnNsqJWlhmE6P8OXAlsx8IDOHgVuBa1oXyMyHMvPbQHMWapQkTd8J2+xy/IPl8MeBF0REzGGNkrQgTCcIrwG2toxvK6dJkhae6bTZE8tk5iiwHzhzTqqTpAWkMZcri4jrgevL0YGIuPc4i68Eds9+VfOq07ex07cPOn8bO3374NS28bzZKGShOU6b7fuic7idnaMK2winvp1TttvTCcLbgXUt42vLaSctM28BbpnOshGxOTM3nMp62kWnb2Onbx90/jZ2+vZBR27jdNrs8WW2RUQDWA7smfxEx2qzO/A1+xFV2EZwOztJFbYRZn47p7NrxCZgfURcEBHdwLXAxpkqQJI0o6bTZm8EXlsOvxz4bGbmHNYoSQvCCYNwuf/YDcAdwD3AbZl5V0S8IyKuBoiIZ0bENuAVwHsj4q7ZLFqSNLXptNnAXwBnRsQW4K3Aj5xiTZKqYFr7CGfm7cDtk6a9rWV4E8XPbzNpWrtQtLlO38ZO3z7o/G3s9O2DDtzGabTZgxQdF6eq416zKVRhG8Ht7CRV2EaY4e0Mfw2TJElSFXmJZUmSJFXSggzCJ7o8aDuKiPdHxM6I+G7LtDMi4tMRcX95v2I+azwdEbEuIj4XEXdHxF0R8eZyekdsY0T0RsTXIuJb5fb9Xjn9gvIStVvKS9Z2z3etpyMi6hHxzYj4ZDneadv3UER8JyLujIjN5bSOeI/OhU5sm6Hz22fo/DZ6XFXaauj89hrmps1ecEE4pnd50Hb0AeCqSdNuBD6TmeuBz9DeB6yMAr+RmZcAzwZ+rfy7dco2DgHPz8yfAC4FroqIZ1NcmvaPykvV7qW4dG07ezPFAVbjOm37AH4mMy9tOf1Op7xHZ1UHt83Q+e0zdH4bPa4qbTVUo72GWW6zF1wQZnqXB207mflF4LFJk1svc/pB4GVzWdNMyswfZuY3yuGDFP8519Ah25iFgXK0q7wl8HyKS9RCG28fQESsBf4N8L5yPOig7TuOjniPzoGObJuh89tn6Pw2elwV2mqodHsNM/yeXYhBuEqXdD47M39YDj8CnD2fxcyUiDgfuAz4Kh20jeXPUHcCO4FPA98H9pWnq4L2f6/+MfCbQLMcP5PO2j4oPhA/FRFfj+KqadBB79FZVqW2GTr4fdGpbfS4CrTVUI32GuagzZ7TSyzr2DIzI6LtT+EREUuAvwHekpkHii+phXbfxswcAy6NiH7gE8DF81vRzImIlwI7M/PrEXHlPJczm56bmdsj4izg0xHxvdaZ7f4e1ezopPdFJ7fR4zq5rYZKtdcwB232QuwRnrFLOreBRyPiHIDyfuc813NaIqKLooH9cGb+bTm5o7YRIDP3AZ8DngP0R3GJWmjv9+oVwNUR8RDFT97PB95N52wfAJm5vbzfSfEBeTkd+B6dJVVqm6ED3xdVaaPHdWhbDRVpr2Fu2uyFGISrdEnn1sucvhb4X/NYy2kp90/6C+CezHxXy6yO2MaIWFX2LhARfcCLKPax+xzFJWqhjbcvM2/KzLWZeT7F/7nPZuYv0iHbBxARiyNi6fgw8LPAd+mQ9+gcqFLbDB32vuj0Nnpcp7fVUI32Guawzc7MBXcDXgLcR7Ffz3+Z73pmaJs+CvwQGKHYd+f1FPv0fAa4H/gn4Iz5rvM0tu+5FPvyfBu4s7y9pFO2EXga8M1y+74LvK2c/kTga8AW4K+BnvmudQa29Urgk522feW2fKu83TXetnTKe3SOXsOOa5vL7ero9rncxo5uo1u2szJtdbldHdlet2zPrLfZXllOkiRJlbQQd42QJEmSZp1BWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJUVEVdGxCfnuw5J0onZZms2GIQlSZJUSQZhLXgR8eqI+FpE3BkR742IekQMRMQfRcRdEfGZiFhVLntpRHwlIr4dEZ+IiBXl9Isi4p8i4lsR8Y2IuLB8+iUR8fGI+F5EfLi8+pIk6RTZZqudGIS1oEXEk4FXAVdk5qXAGPCLwGJgc2Y+BfgC8LvlQz4E/FZmPg34Tsv0DwM3Z+ZPAD9JcRUpgMuAtwCXUFzF5opZ3iRJ6li22Wo3jfkuQDqBFwDPADaVX/z7gJ1AE/hYucxfAX8bEcuB/sz8Qjn9g8Bfl9cqX5OZnwDIzEGA8vm+lpnbyvE7gfOBL836VklSZ7LNVlsxCGuhC+CDmXnT4yZG/F+TljvVa4UPtQyP4f8JSTodttlqK+4aoYXuM8DLI+IsgIg4IyLOo3jvvrxc5heAL2XmfmBvRDyvnP4a4AuZeRDYFhEvK5+jJyIWzeVGSFJF2GarrfhNSgtaZt4dEb8DfCoiasAI8GvAIeDyct5Oin3SAF4LvKdsNB8A/n05/TXAeyPiHeVzvGION0OSKsE2W+0mMk/11wlp/kTEQGYume86JEknZputhcpdIyRJklRJ9ghLkiSpkuwRliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJ/xtPbjbGjFGkiQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(\"train\", (12, 6))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.title(\"Epoch Average Loss\")\n",
    "x = [i + 1 for i in range(len(epoch_loss_values))]\n",
    "y = epoch_loss_values\n",
    "plt.xlabel(\"epoch\")\n",
    "plt.ylim([0.1, 0.7])\n",
    "plt.plot(x, y)\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.title(\"Val Mean Dice\")\n",
    "x = [val_interval * (i + 1) for i in range(len(metric_values))]\n",
    "y = metric_values\n",
    "plt.xlabel(\"epoch\")\n",
    "plt.ylim([0, 1.0])\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zIO0xaNDKyZ0"
   },
   "source": [
    "## Cleanup data directory\n",
    "\n",
    "Remove directory if a temporary was used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "qJpZYdCgKyZ0"
   },
   "outputs": [],
   "source": [
    "if directory is None:\n",
    "    shutil.rmtree(root_dir)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "transfer_mmar.ipynb",
   "provenance": [
    {
     "file_id": "https://github.com/Project-MONAI/tutorials/blob/main/2d_classification/mednist_tutorial.ipynb",
     "timestamp": 1607545340801
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}