S03-Segmenting-and-Patching-Slides.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f0f12162",
   "metadata": {},
   "source": [
    "# S03-Segmenting and Patching Slides\n",
    "\n",
    "## 1. CLAM\n",
    "\n",
    "It is highly recommanded to use CLAM for slide segmentating and patching, because CLAM implements efficient algorithms and flexible interfaces. \n",
    "\n",
    "To do so, there are two alternative means:\n",
    "- visit [our enhanced CLAM](./tools/CLAM) and follow this notebook and [another one](./S04-Extracting-Patch-Features.ipynb) to use CLAM for preprocessing WSIs, with detailed guidance and walkthrough.\n",
    "- visit [the official website](https://github.com/mahmoodlab/CLAM) and read the related instructions to use CLAM.\n",
    "\n",
    "\n",
    "## 2. Original CLAM vs. Our Enhanced CLAM\n",
    "\n",
    "NOTE: read this sub-section if you want to know the details regarding how CLAM implements tissue segmentating and patching. \n",
    "\n",
    "### 2.1 Original CLAM \n",
    "\n",
    "#### (1) Segmentation\n",
    "\n",
    "In CLAM, the segmentation is operated at a best level (e.g., downsampling `x64`).\n",
    "\n",
    "By default, the values of `a_t` and `a_h` have the following behavior in **segmenting slide tissues** in CLAM: \n",
    "- `a_t`: area filter threshold for tissue (positive integer, the minimum size of detected foreground contours to consider, relative to **a reference patch size of 512 x 512 at level 0**, e.g. a value 10 means only detected foreground contours of size greater than 10 512 x 512 sized patches at level 0 will be processed, `16` by default)\n",
    "- `a_h`: area filter threshold for holes (positive integer, the minimum size of detected holes/cavities in foreground contours to avoid, once again **relative to 512 x 512 sized patches at level 0**, `8` by default)\n",
    "\n",
    "Two important operations in original CLAM:\n",
    "- when segmenting the tissues, the value of filter threshold would be automatically determined in the segmentation level.\n",
    "- when the segmentation is finished, all coordinates of foreground contours of tissues and holes would be rescaled to the highest level `0`.\n",
    "\n",
    "#### (2) Patching\n",
    "\n",
    "All patches are derived from the rectangle box bounding the foreground contours of tissues.\n",
    "\n",
    "The most important parameters for patches are `patch_size`, `step_size` and `patch_level`, indicating which size at which level we want to extract. Actually when we run patching, the size and moving step is scaled to **level 0** using following code:\n",
    "\n",
    "```python\n",
    "# Patch Size\n",
    "patch_downsample = (int(self.level_downsamples[patch_level][0]), int(self.level_downsamples[patch_level][1]))\n",
    "ref_patch_size = (patch_size * patch_downsample[0], patch_size * patch_downsample[1])\n",
    "# Step Size\n",
    "step_size_x = step_size * patch_downsample[0]\n",
    "step_size_y = step_size * patch_downsample[1]\n",
    "```\n",
    "That is to say, all the patches are calculated and generated **at level 0**. Thus, the final patch coordinates (restored in `h5` files) are at the highest level 0. Also, the actual patch size is equal to `patch_size * level_downsamples[patch_level]` at the highest level 0.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22989247-f6d6-465b-8462-c96a2eb94a12",
   "metadata": {},
   "source": [
    "### 2.2 Our Enhanced CLAM (New Features)\n",
    "\n",
    "The biggest problem of original CLAM in WSI preprocessing may be that **it cannot patch WSIs at a user-specified magnification**.\n",
    "\n",
    "However, in most cases, it is desired to patch WSIs at the same magnification, especially when **the WSIs at hand are at different highest magnifications** (`20x` or `40x`). Therefore, we fix this in our enhanced CLAM, by allowing users to specify a unified magnification to patch WSIs. \n",
    "\n",
    "For example, if you set `--patch_magnification 20` and `--patch_size 256`, i.e., you want to patch all WSIs at `20x` with patch size `256 x 256`, the following things would happen. Concretely, for a given WSI,\n",
    "- when its highest magnification is `40x` at level 0, `512 x 512` patches will be generated at level 0.\n",
    "- when its highest magnification is `20x` at level 0, `256 x 256` patches will be generated at level 0.\n",
    "- otherwise, smaller patches (less than 256) will be generated at level 0 as its highest magnification cannot reach up to you specified.\n",
    "\n",
    "With the above operations, all generated patches would be `256 x 256` at the `20x` magnification."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db3322b9",
   "metadata": {},
   "source": [
    "### 2.3 Some Important Parameters of WSIs\n",
    "\n",
    "To better know and use WSIs, you should get familiar with some important parameters of them.\n",
    "- `image size`: the image size at the highest magnification.\n",
    "- `the highest magnification`: It could be seen from the parameter **\"MPP\"**. A MPP of ~0.25 usually indicates a highest magnification of `40x`, and A MPP of ~0.5 usually indicates a highest magnification of `20x`. \n",
    "- `downsample levels`: all available downsample levels in WSIs.\n",
    "\n",
    "Next, we show an example of accessing these parameters using OpenSlide (https://openslide.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b204997e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openslide\n",
    "wsi_path = '/NAS02/RawData/tcga_rcc/TCGA-KL-8336/TCGA-KL-8336-01Z-00-DX1.bfba9373-efa8-4573-8ee6-8ac961f0b65a.svs'\n",
    "wsi_object = openslide.open_slide(wsi_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a1fea8",
   "metadata": {},
   "source": [
    "`image size`: this WSI is with a size of `127655 * 53444` pixels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2fd6d0dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(127655, 53444)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wsi_object.dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0501935",
   "metadata": {},
   "source": [
    "`the highest magnification`: `'aperio.MPP': '0.2498'`, which indicates that its highest magnification is `40x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dc0d4a48",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<_PropertyMap {'aperio.AppMag': '40', 'aperio.DSR ID': 'ap1251-dsr', 'aperio.Date': '07/13/12', 'aperio.Filename': '285308', 'aperio.Focus Offset': '0.000000', 'aperio.ImageID': '285308', 'aperio.Left': '22.248177', 'aperio.LineAreaXOffset': '0.000000', 'aperio.LineAreaYOffset': '0.000000', 'aperio.LineCameraSkew': '0.000236', 'aperio.MPP': '0.2498', 'aperio.OriginalWidth': '133000', 'aperio.Originalheight': '53544', 'aperio.Parmset': 'COVERSLIP', 'aperio.ScanScope ID': 'IPTH5001', 'aperio.StripeWidth': '1000', 'aperio.Time': '22:51:01', 'aperio.Top': '16.991373', 'aperio.User': '131dd1e8-882f-4bfa-afaf-8d5807a90a47', 'openslide.comment': 'Aperio Image Library v11.0.37\\r\\n127655x53444 (256x256) JPEG/RGB Q=30;Aperio Image Library v10.0.50\\r\\n133000x53544 [0,100 127655x53444] (256x256) J2K/YUV16 Q=70|AppMag = 40|StripeWidth = 1000|ScanScope ID = IPTH5001|Filename = 285308|Date = 07/13/12|Time = 22:51:01|User = 131dd1e8-882f-4bfa-afaf-8d5807a90a47|Parmset = COVERSLIP|MPP = 0.2498|Left = 22.248177|Top = 16.991373|LineCameraSkew = 0.000236|LineAreaXOffset = 0.000000|LineAreaYOffset = 0.000000|Focus Offset = 0.000000|DSR ID = ap1251-dsr|ImageID = 285308|OriginalWidth = 133000|Originalheight = 53544', 'openslide.level-count': '4', 'openslide.level[0].downsample': '1', 'openslide.level[0].height': '53444', 'openslide.level[0].tile-height': '256', 'openslide.level[0].tile-width': '256', 'openslide.level[0].width': '127655', 'openslide.level[1].downsample': '4.0000470027888326', 'openslide.level[1].height': '13361', 'openslide.level[1].tile-height': '256', 'openslide.level[1].tile-width': '256', 'openslide.level[1].width': '31913', 'openslide.level[2].downsample': '16.001037508837925', 'openslide.level[2].height': '3340', 'openslide.level[2].tile-height': '256', 'openslide.level[2].tile-width': '256', 'openslide.level[2].width': '7978', 'openslide.level[3].downsample': '32.00207501767585', 'openslide.level[3].height': '1670', 'openslide.level[3].tile-height': '256', 'openslide.level[3].tile-width': '256', 'openslide.level[3].width': '3989', 'openslide.mpp-x': '0.24979999999999999', 'openslide.mpp-y': '0.24979999999999999', 'openslide.objective-power': '40', 'openslide.vendor': 'aperio', 'tiff.ImageDescription': 'Aperio Image Library v11.0.37\\r\\n127655x53444 (256x256) JPEG/RGB Q=30;Aperio Image Library v10.0.50\\r\\n133000x53544 [0,100 127655x53444] (256x256) J2K/YUV16 Q=70|AppMag = 40|StripeWidth = 1000|ScanScope ID = IPTH5001|Filename = 285308|Date = 07/13/12|Time = 22:51:01|User = 131dd1e8-882f-4bfa-afaf-8d5807a90a47|Parmset = COVERSLIP|MPP = 0.2498|Left = 22.248177|Top = 16.991373|LineCameraSkew = 0.000236|LineAreaXOffset = 0.000000|LineAreaYOffset = 0.000000|Focus Offset = 0.000000|DSR ID = ap1251-dsr|ImageID = 285308|OriginalWidth = 133000|Originalheight = 53544', 'tiff.ResolutionUnit': 'inch'}>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wsi_object.properties"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ec4c0c6",
   "metadata": {},
   "source": [
    "`downsample levels`: there are four available downsample levels in this WSI file:\n",
    "- downsample `1x` at a level `0`: the original image, with a size of 127655 * 53444 pixels.\n",
    "- downsample `4x` at a level `1`: the rescaled image, with a size of 31913 * 13361 pixels.\n",
    "- downsample `16x` at a level `2`: the rescaled image, with a size of 7978 * 3340 pixels.\n",
    "- downsample `32x` at a level `3`: the rescaled image, with a size of 3989 * 1670 pixels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e8bb7d07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All available downsample levels: (1.0, 4.000047002788833, 16.001037508837925, 32.00207501767585)\n",
      "All available dimension levels: ((127655, 53444), (31913, 13361), (7978, 3340), (3989, 1670))\n"
     ]
    }
   ],
   "source": [
    "print(\"All available downsample levels:\", wsi_object.level_downsamples)\n",
    "print(\"All available dimension levels:\", wsi_object.level_dimensions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cbb6b45",
   "metadata": {},
   "source": [
    "## 3. Segmenting and patching tissue regions (foreground)\n",
    "\n",
    "Next, we show you how to use our enhanced CLAM for segmenting and patching tissue regions.\n",
    "\n",
    "In fact, we have improved CLAM in terms of \n",
    "- allowing users to **specify a unified magnification for all WSIs in tissue patching**,\n",
    "- **more functionalities**: e.g., patch image normlization, image Gaussian bluring, and different architectures for feature extracting,\n",
    "- **adapting to a patient-level organization of WSIs**.\n",
    "\n",
    "### 3.1 Running Script\n",
    "\n",
    "A detailed bash script (placed at `./tools/scripts/S03-Seg-Patching.sh`) is as follows:\n",
    "\n",
    "```bash\n",
    "#!/bin/bash\n",
    "set -e\n",
    "\n",
    "# Sample patches of SIZE x SIZE at a specified magnification (MAG)\n",
    "# Typical MAG is 20 (~0.5 MMP); it can also be set to 10 (~1 MMP) or 5 (~2 MMP)\n",
    "MAG=20\n",
    "SIZE=256\n",
    "\n",
    "# Path where CLAM is installed\n",
    "DIR_REPO=../CLAM\n",
    "\n",
    "# Root path to pathology images \n",
    "DIR_READ=/NAS02/RawData/tcga_rcc\n",
    "DIR_SAVE=/NAS02/ExpData/tcga_rcc\n",
    "\n",
    "cd ${DIR_REPO}\n",
    "\n",
    "echo \"run seg & patching for all slides\"\n",
    "CUDA_VISIBLE_DEVICES=0 python3 create_patches_fp.py \\\n",
    "    --source ${DIR_READ} \\\n",
    "    --save_dir ${DIR_SAVE}/tiles-${MAG}x-s${SIZE} \\\n",
    "    --patch_size ${SIZE} \\\n",
    "    --step_size ${SIZE} \\\n",
    "    --preset tcga.csv \\\n",
    "    --patch_magnification ${MAG} \\\n",
    "    --seg --patch --stitch --save_mask \\\n",
    "    --auto_skip --in_child_dir\n",
    "```\n",
    "\n",
    "You could run this script using the following command:\n",
    "```bash\n",
    "nohup ./S03-Seg-Patching.sh > S03-Seg-Patching.log 2>&1 &\n",
    "```\n",
    "\n",
    "### 3.2 Checking Important Running Logs\n",
    "\n",
    "When it is finished, you can check by\n",
    "- `./tools/scripts/S03-Seg-Patching.log`: running details.\n",
    "- `/NAS02/ExpData/tcga_rcc/tiles-20x-s256/process_list_autogen.csv`: important parameters used in S03. \n",
    "\n",
    "Next, we check if the number of generated `.h5` files are correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5514974b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#Slides: 940\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>File ID</th>\n",
       "      <th>File Name</th>\n",
       "      <th>Data Category</th>\n",
       "      <th>Data Type</th>\n",
       "      <th>Project ID</th>\n",
       "      <th>Case ID</th>\n",
       "      <th>Sample ID</th>\n",
       "      <th>Sample Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0596623c-c2c5-4de5-b358-d5393e79120e</td>\n",
       "      <td>TCGA-B3-4103-01Z-00-DX1.76bba2e9-0a6d-460b-8ae...</td>\n",
       "      <td>Biospecimen</td>\n",
       "      <td>Slide Image</td>\n",
       "      <td>TCGA-KIRP</td>\n",
       "      <td>TCGA-B3-4103</td>\n",
       "      <td>TCGA-B3-4103-01Z</td>\n",
       "      <td>Primary Tumor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b1b3df18-1fcc-40a1-8610-143f06c9748b</td>\n",
       "      <td>TCGA-AL-3468-01Z-00-DX1.F86A4811-D60C-4845-A7A...</td>\n",
       "      <td>Biospecimen</td>\n",
       "      <td>Slide Image</td>\n",
       "      <td>TCGA-KIRP</td>\n",
       "      <td>TCGA-AL-3468</td>\n",
       "      <td>TCGA-AL-3468-01Z</td>\n",
       "      <td>Primary Tumor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>e55f0570-5c9e-4676-8b65-380ae02a8d63</td>\n",
       "      <td>TCGA-A4-7997-01Z-00-DX1.aa4e2dd8-fac9-43ae-963...</td>\n",
       "      <td>Biospecimen</td>\n",
       "      <td>Slide Image</td>\n",
       "      <td>TCGA-KIRP</td>\n",
       "      <td>TCGA-A4-7997</td>\n",
       "      <td>TCGA-A4-7997-01Z</td>\n",
       "      <td>Primary Tumor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>04ea6765-f97b-45a3-9c50-7882b2edf61a</td>\n",
       "      <td>TCGA-HE-A5NF-01Z-00-DX1.74ABE42F-E64E-4550-AD8...</td>\n",
       "      <td>Biospecimen</td>\n",
       "      <td>Slide Image</td>\n",
       "      <td>TCGA-KIRP</td>\n",
       "      <td>TCGA-HE-A5NF</td>\n",
       "      <td>TCGA-HE-A5NF-01Z</td>\n",
       "      <td>Primary Tumor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>212eed8a-ee10-4149-a5c8-7effb1d4747e</td>\n",
       "      <td>TCGA-EV-5903-01Z-00-DX1.04ef7cdf-b282-4ad3-917...</td>\n",
       "      <td>Biospecimen</td>\n",
       "      <td>Slide Image</td>\n",
       "      <td>TCGA-KIRP</td>\n",
       "      <td>TCGA-EV-5903</td>\n",
       "      <td>TCGA-EV-5903-01Z</td>\n",
       "      <td>Primary Tumor</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                File ID  \\\n",
       "0  0596623c-c2c5-4de5-b358-d5393e79120e   \n",
       "1  b1b3df18-1fcc-40a1-8610-143f06c9748b   \n",
       "2  e55f0570-5c9e-4676-8b65-380ae02a8d63   \n",
       "3  04ea6765-f97b-45a3-9c50-7882b2edf61a   \n",
       "4  212eed8a-ee10-4149-a5c8-7effb1d4747e   \n",
       "\n",
       "                                           File Name Data Category  \\\n",
       "0  TCGA-B3-4103-01Z-00-DX1.76bba2e9-0a6d-460b-8ae...   Biospecimen   \n",
       "1  TCGA-AL-3468-01Z-00-DX1.F86A4811-D60C-4845-A7A...   Biospecimen   \n",
       "2  TCGA-A4-7997-01Z-00-DX1.aa4e2dd8-fac9-43ae-963...   Biospecimen   \n",
       "3  TCGA-HE-A5NF-01Z-00-DX1.74ABE42F-E64E-4550-AD8...   Biospecimen   \n",
       "4  TCGA-EV-5903-01Z-00-DX1.04ef7cdf-b282-4ad3-917...   Biospecimen   \n",
       "\n",
       "     Data Type Project ID       Case ID         Sample ID    Sample Type  \n",
       "0  Slide Image  TCGA-KIRP  TCGA-B3-4103  TCGA-B3-4103-01Z  Primary Tumor  \n",
       "1  Slide Image  TCGA-KIRP  TCGA-AL-3468  TCGA-AL-3468-01Z  Primary Tumor  \n",
       "2  Slide Image  TCGA-KIRP  TCGA-A4-7997  TCGA-A4-7997-01Z  Primary Tumor  \n",
       "3  Slide Image  TCGA-KIRP  TCGA-HE-A5NF  TCGA-HE-A5NF-01Z  Primary Tumor  \n",
       "4  Slide Image  TCGA-KIRP  TCGA-EV-5903  TCGA-EV-5903-01Z  Primary Tumor  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import os.path as osp\n",
    "import pandas as pd\n",
    "\n",
    "# please use your own gdc samples sheet. Here is an example for illustrating\n",
    "FILEPATH_TO_GDC_SAMPLE_SHEET = \"./docs/gdc_sample_sheet.tsv\"\n",
    "\n",
    "df = pd.read_csv(FILEPATH_TO_GDC_SAMPLE_SHEET, sep='\\t')\n",
    "slide_names = [osp.splitext(df.loc[i, \"File Name\"].strip())[0] for i in df.index]\n",
    "print(\"#Slides:\", len(slide_names))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91179292",
   "metadata": {},
   "source": [
    "There are three slides that cannot be processed by CLAM, because they only contain the highest level 0, *i.e.*, the largest image view."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "51eb53e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slide TCGA-UZ-A9PQ-01Z-00-DX1.C2CB0E94-2548-4399-BCAB-E4D556D533EF is not found in generated patch files.\n",
      "The slide TCGA-5P-A9KC-01Z-00-DX1.F3D67C35-111C-4EE6-A5F7-05CF8D01E783 is not found in generated patch files.\n",
      "The slide TCGA-5P-A9KA-01Z-00-DX1.6F4914E0-AB5D-4D5F-8BF6-FB862AA63A87 is not found in generated patch files.\n"
     ]
    }
   ],
   "source": [
    "DIR_TO_PATCH_COORD = \"/NAS02/ExpData/tcga_rcc/tiles-20x-s256/patches\"\n",
    "generated_patch_filenames = []\n",
    "for f in os.listdir(DIR_TO_PATCH_COORD):\n",
    "    if f.endswith(\".h5\"):\n",
    "        generated_patch_filenames.append(osp.splitext(f)[0])\n",
    "\n",
    "filenames_not_processed = []\n",
    "for s in slide_names:\n",
    "    if s not in generated_patch_filenames:\n",
    "        filenames_not_processed.append(s)\n",
    "        print(\"The slide {} is not found in generated patch files.\".format(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b30035c7",
   "metadata": {},
   "source": [
    "Here we need to exclude these slides from the table saved for possible classification tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b51175cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slide TCGA-UZ-A9PQ-01Z-00-DX1.C2CB0E94-2548-4399-BCAB-E4D556D533EF is excluded.\n",
      "The slide TCGA-5P-A9KC-01Z-00-DX1.F3D67C35-111C-4EE6-A5F7-05CF8D01E783 is excluded.\n",
      "The slide TCGA-5P-A9KA-01Z-00-DX1.6F4914E0-AB5D-4D5F-8BF6-FB862AA63A87 is excluded.\n",
      "Saved new data in /NAS02/ExpData/tcga_rcc/table/TCGA_RCC_full_path_subtype.csv\n"
     ]
    }
   ],
   "source": [
    "PATH_TO_TABLE = \"/NAS02/ExpData/tcga_rcc/table/TCGA_RCC_path_subtype.csv\"\n",
    "PATH_TO_NEW_TABLE = \"/NAS02/ExpData/tcga_rcc/table/TCGA_RCC_full_path_subtype.csv\"\n",
    "data_tb = pd.read_csv(PATH_TO_TABLE)\n",
    "keep_idxs = []\n",
    "for i in data_tb.index:\n",
    "    if data_tb.loc[i, \"pathology_id\"] in filenames_not_processed:\n",
    "        print(\"The slide {} is excluded.\".format(data_tb.loc[i, \"pathology_id\"]))\n",
    "    else:\n",
    "        keep_idxs.append(i)\n",
    "data_to_save = data_tb.loc[keep_idxs, :]\n",
    "data_to_save.to_csv(PATH_TO_NEW_TABLE, index=False)\n",
    "print(\"Saved new data in {}\".format(PATH_TO_NEW_TABLE))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7580587",
   "metadata": {},
   "source": [
    "### Example of Running Logs \n",
    "\n",
    "The running log of the first WSI is presented as follows:\n",
    "```text\n",
    "progress: 0.00, 0/940\n",
    "processing TCGA-2K-A9WE--TCGA-2K-A9WE-01Z-00-DX1.ED8ADE3B-D49B-403B-B4EB-BD11D91DD676.svs\n",
    "Creating patches for:  TCGA-2K-A9WE-01Z-00-DX1.ED8ADE3B-D49B-403B-B4EB-BD11D91DD676 ...\n",
    "Total number of contours to process:  7\n",
    "Bounding Box: 3600 58550 2625 3345\n",
    "Contour Area: 4296064.0\n",
    "Extracted 6 coordinates\n",
    "Bounding Box: 145333 30435 3281 11666\n",
    "Contour Area: 10688768.0\n",
    "Extracted 20 coordinates\n",
    "Bounding Box: 5776 864 11121 9986\n",
    "Contour Area: 66040280.0\n",
    "Extracted 75 coordinates\n",
    "Bounding Box: 106932 0 49138 16914\n",
    "Contour Area: 145224504.0\n",
    "Extracted 186 coordinates\n",
    "Bounding Box: 68850 0 77108 92570\n",
    "Contour Area: 3868769667.0\n",
    "Extracted 3698 coordinates\n",
    "Bounding Box: 7600 0 76756 78489\n",
    "Contour Area: 3373277550.0\n",
    "Extracted 3272 coordinates\n",
    "Bounding Box: 0 0 18049 737\n",
    "Contour Area: 6355584.0\n",
    "Extracted 17 coordinates\n",
    "start stitching TCGA-2K-A9WE-01Z-00-DX1.ED8ADE3B-D49B-403B-B4EB-BD11D91DD676\n",
    "original size: 156086 x 92586\n",
    "downscaled size for stiching: 9755 x 5786\n",
    "number of patches: 7274\n",
    "patch size: 512x512 patch level: 0\n",
    "ref patch size: (512, 512)x(512, 512)\n",
    "downscaled patch size: 64x64\n",
    "progress: 0/7274 stitched\n",
    "progress: 728/7274 stitched\n",
    "progress: 1456/7274 stitched\n",
    "progress: 2184/7274 stitched\n",
    "progress: 2912/7274 stitched\n",
    "progress: 3640/7274 stitched\n",
    "progress: 4368/7274 stitched\n",
    "progress: 5096/7274 stitched\n",
    "progress: 5824/7274 stitched\n",
    "progress: 6552/7274 stitched\n",
    "segmentation took 11.164469480514526 seconds\n",
    "patching took 3.314134359359741 seconds\n",
    "stitching took 7.7146852016448975 seconds\n",
    "```\n",
    "\n",
    "From the log shown above, we can find that\n",
    "- there are 7274 patches (each with a size of 512 * 512 at the level 0) after segmenting and patching tissues,\n",
    "- and it costs about 14 seconds for processing the first WSI.\n",
    "\n",
    "The segmentation (see your `RESULTS_DIRECTORY/tiles-20x-s256/masks/`) and stitching (see your `RESULTS_DIRECTORY/tiles-20x-s256/stitches/`) results of the first WSI are shown as follows:\n",
    "\n",
    "<img src=\"./docs/S03-example-seg.png\" width=\"60%\" align='left' />\n",
    "\n",
    "<img src=\"./docs/S03-example-stitch.png\" width=\"60%\" align='left' />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "267308f9",
   "metadata": {},
   "source": [
    "## 4. Results\n",
    "\n",
    "When finishing the steps above, it is expected that your file structure is as follows:\n",
    "```txt\n",
    "RESULTS_DIRECTORY/\n",
    "\t├── masks\n",
    "    \t\t├── slide_1.png\n",
    "    \t\t├── slide_2.png\n",
    "    \t\t└── ...\n",
    "\t├── patches\n",
    "    \t\t├── slide_1.h5\n",
    "    \t\t├── slide_2.h5\n",
    "    \t\t└── ...\n",
    "\t├── stitches\n",
    "    \t\t├── slide_1.png\n",
    "    \t\t├── slide_2.png\n",
    "    \t\t└── ...\n",
    "\t└── process_list_autogen.csv\n",
    "```\n",
    "\n",
    "`RESULTS_DIRECTORY` would be `/NAS02/ExpData/tcga_rcc/tiles-20x-s256` in our example."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}