tutorials/run_benchmarks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "61f0fee5-c227-4d34-94b2-8d264480fa27",
   "metadata": {},
   "source": [
    "# Import necessary moudules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "73c7730e-d0d9-474c-ad1b-2b73d1256006",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import anndata as ad\n",
    "from typing import Optional\n",
    "from FeatureSelectionBenchmarks.benchmark.run_benchmark import run_bench"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea3136f9-ca14-4c89-9bf5-e3d179ebcd07",
   "metadata": {},
   "source": [
    "# Define custom methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2e64997a-5338-4f33-84b1-d73b18bc2249",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_select(adata: ad.AnnData, n_selected_genes: int, seed: int):\n",
    "    rng = np.random.default_rng(seed)\n",
    "    return rng.choice(adata.var_names, size=n_selected_genes)\n",
    "\n",
    "\n",
    "def random_clustering(adata: ad.AnnData, img: Optional[np.ndarray], k: int, seed: int):\n",
    "    rng = np.random.default_rng(seed)\n",
    "    return rng.integers(low=0, high=k, size=adata.n_obs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47768548-468a-48f8-9419-b745f084e4fd",
   "metadata": {},
   "source": [
    "# Three configurations about data, feature selection, and downstream analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9418a5f3-138b-4edf-a8dd-d8e1fa843148",
   "metadata": {},
   "source": [
    "Before running the benchmark, you first need to prepare three configurations stored in `dict` type: `data_cfg`, `fs_cfg`, and `cl_cfg`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e15708c-496b-45f4-a254-6abc37ef74a8",
   "metadata": {},
   "source": [
    "## 1. data_cfg\n",
    "\n",
    "Configurations of datasets. It should be a dict in the format `{'data_name': {'property_name': data_property}}`. \n",
    "Supported property names are:\n",
    "\n",
    "- 'adata_path': path to the `h5ad` file. The raw counts should be  stored in `adata.X`. The benchmark will do\n",
    "quality control and  normalization automatically when `preprocess=True`. If you want to use datasets processed\n",
    "by yourself, please set `preprocess=False`. In this case the normalized counts and raw data should be stored in\n",
    "`adata.X` and `adata.raw` respectively.\n",
    "- 'image_path': path to the `h5ad` file, optional. It will be ignored when `modality='scrna'`.\n",
    "- 'annot_key' : a key in `adata.obs` that represents annotations (cell types/domains).\n",
    "- 'to_replace': replace some values in `adata.obs['annot_key']`, optional. It should be a dict in the format\n",
    " `{'value': ['to_replace_1', 'to_replace_2']}`, where 'to_replace_x' are values that will be replaced, and\n",
    " 'value' is value to replace any values matching 'to_replace_x' with.\n",
    "- 'to_remove' :  remove cells/spots that have some values in `adata.obs['annot_key']`, optional. It should be a\n",
    "list of values to be removed.\n",
    "- 'batch': a key in `adata.obs` that represents batches, optional. Only valid when `modality='scrna'`. If it's\n",
    "specified, the benchmark will use the algorithm in Seurat to combine the features selected in each batch.\n",
    "- 'shape': Set 'hexagon' for Visium data, and 'square' for ST data when `modality='spatial'`, optional.\n",
    "Default is 'hexagon'. Currently this parameter is only used in the cluster refinement of `spaGCN`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebeaddf6-4d3d-4f20-8e83-3e0c33531de6",
   "metadata": {},
   "source": [
    "## 2. fs_cfg\n",
    "\n",
    "Configurations of feature selection methods. It should be a dict in the format\n",
    "`{fs_method: list_of_numbers_of_selected_genes}`. More specifically,\n",
    "\n",
    "- fs_method: can be either a string that represents a predefined function in this benchmark, or a custom\n",
    "function. The benchmark will call the function like `custom_fs_function(adata, n_selected_genes, **kwargs)`,\n",
    "and the return values must be an ndarray that contains features selected by the function. You can write a\n",
    "wrapper function to work around incompatible parameters/return values.\n",
    "- list_of_numbers_of_selected_genes: a list of numbers of genes needed to be selected. If the function\n",
    "internally determines the number of selected genes (e.g. GeneClust and no feature selection), write the list\n",
    "as `['auto']`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbb745c7-6ea6-49e1-9e2a-a2338cc9ea8a",
   "metadata": {},
   "source": [
    "## 3. cl_cfg\n",
    "\n",
    "Configurations of downstream cell clustering/domain detection methods. It should be a dict in the format\n",
    "`{cl_method: list_of_numbers_of_runs}`. More specifically,\n",
    "\n",
    "- cl_method: can be either a string that represents a predefined function in this benchmark, or a custom\n",
    "function. The benchmark will call the function like `custom_cl_function(fs_adata, img, **kwargs)`,\n",
    "and the return values must be an ndarray that contains cluster labels generated by the function. the parameter\n",
    "'img' is always `None` when `modality='scrna'`. You can write a wrapper function to work around incompatible\n",
    "parameters/return values.\n",
    "- list_of_numbers_of_runs: a list of numbers of times that the function will run with different random states."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a1022d5-b42a-4d1d-885a-55d845cbd87c",
   "metadata": {},
   "source": [
    "# Run benchmark on scRNA-seq data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28879850-fbf7-4bf4-b1d3-7bfa1d9e7174",
   "metadata": {},
   "source": [
    "## Evaluate predefined methods by the benchmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "85b88d86-b3d7-49d8-b88c-6df778c63890",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:18.842\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m./cache has been deleted.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────── </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">PBMC3k</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m───────────────────────────────────────────────────── \u001B[0m\u001B[1;31mPBMC3k\u001B[0m\u001B[92m ──────────────────────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:18.857\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mReading adata...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:18.858\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mNo cache for \u001B[35mPBMC3k\u001B[0m\u001B[1m. Trying to read h5ad from the given path in config...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:18.924\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFound sparse matrix in `adata.X`. Converting to ndarray...\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Before QC: <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2700</span> cells and <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">32738</span> genes.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Before QC: \u001B[1;33m2700\u001B[0m cells and \u001B[1;33m32738\u001B[0m genes.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:19.129\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mObservation names and Variables names are all unique now.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:19.417\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mDropping \u001B[33m10\u001B[0m\u001B[1m special genes from \u001B[33m32738\u001B[0m\u001B[1m genes...\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">After QC: <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2623</span> cells and <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">11057</span> genes.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "After QC: \u001B[1;33m2623\u001B[0m cells and \u001B[1;33m11057\u001B[0m genes.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────── </span>FS method: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">seurat_v3</span>, Gene Number: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">1000</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ─────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m───────────────────────────────────── \u001B[0mFS method: \u001B[1;31mseurat_v3\u001B[0m, Gene Number: \u001B[1;31m1000\u001B[0m\u001B[92m ─────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:21.062\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mseurat_v3\u001B[0m\u001B[1m feature selection with \u001B[33m1000\u001B[0m\u001B[1m features to be selected.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:22.175\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFeature selection finished.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:22.178\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected genes have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:22.279\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m0\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:22.499\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:22.518\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m1\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:22.666\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────── </span>FS method: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">seurat_v3</span>, Gene Number: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">2000</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ─────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m───────────────────────────────────── \u001B[0mFS method: \u001B[1;31mseurat_v3\u001B[0m, Gene Number: \u001B[1;31m2000\u001B[0m\u001B[92m ─────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:22.678\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mseurat_v3\u001B[0m\u001B[1m feature selection with \u001B[33m2000\u001B[0m\u001B[1m features to be selected.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:23.901\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFeature selection finished.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:23.905\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected genes have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:24.065\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m0\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:24.225\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:24.235\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m1\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:24.348\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">────────────────────────────────── </span>FS method: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">GeneClust-fast</span>, Gene Number: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">auto</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ───────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m────────────────────────────────── \u001B[0mFS method: \u001B[1;31mGeneClust-fast\u001B[0m, Gene Number: \u001B[1;31mauto\u001B[0m\u001B[92m ───────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:24.360\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mGeneClust-fast\u001B[0m\u001B[1m feature selection with \u001B[33mauto\u001B[0m\u001B[1m features to be selected.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:24.682\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mPerforming GeneClust-fast on scRNA-seq data, with 95 workers.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:24.732\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mPreprocessing data...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:25.724\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mData preprocessing done.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:25.725\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mClustering genes...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.024\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mGene clustering done!\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.491\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected 396 genes.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.492\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mGeneClust-fast finished.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.569\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFeature selection finished.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.573\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected genes have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.644\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m0\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.741\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.751\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m1\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.807\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.865\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mrecords have been saved into './2023-02 14_54_26 scrna.xlsx'.\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "data_cfg = {'PBMC3k': {\n",
    "        'adata_path': '../tests/data/scrna/pbmc3k_raw.h5ad',\n",
    "        'annot_key': 'louvain',\n",
    "        'to_replace': {'Dendritic': ['Dendritic cells']},\n",
    "        'to_remove': [np.nan, 'Megakaryocytes']\n",
    "    }}\n",
    "fs_cfg = {'seurat_v3': [1000, 2000], 'GeneClust-fast': ['auto']}\n",
    "cl_cfg = {'KMeans': 2}\n",
    "\n",
    "run_bench(data_cfg, fs_cfg, cl_cfg, ['ARI', 'NMI'], 'scrna', clean_cache=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc82a7fd-cd85-4b72-9b55-2fd587b0c2d6",
   "metadata": {},
   "source": [
    "## Evaluate custom methods defined by the user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6e2d33d9-f747-4ff0-9b7e-7251e9a30bb4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:26.935\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m./cache has been deleted.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────── </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">PBMC3k</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m───────────────────────────────────────────────────── \u001B[0m\u001B[1;31mPBMC3k\u001B[0m\u001B[92m ──────────────────────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:26.939\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mReading adata...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:26.940\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mNo cache for \u001B[35mPBMC3k\u001B[0m\u001B[1m. Trying to read h5ad from the given path in config...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:27.006\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFound sparse matrix in `adata.X`. Converting to ndarray...\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Before QC: <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2700</span> cells and <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">32738</span> genes.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Before QC: \u001B[1;33m2700\u001B[0m cells and \u001B[1;33m32738\u001B[0m genes.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:27.212\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mObservation names and Variables names are all unique now.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:27.500\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mDropping \u001B[33m10\u001B[0m\u001B[1m special genes from \u001B[33m32738\u001B[0m\u001B[1m genes...\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">After QC: <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2623</span> cells and <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">11057</span> genes.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "After QC: \u001B[1;33m2623\u001B[0m cells and \u001B[1;33m11057\u001B[0m genes.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────── </span>FS method: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">seurat_v3</span>, Gene Number: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">1000</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ─────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m───────────────────────────────────── \u001B[0mFS method: \u001B[1;31mseurat_v3\u001B[0m, Gene Number: \u001B[1;31m1000\u001B[0m\u001B[92m ─────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:29.161\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mseurat_v3\u001B[0m\u001B[1m feature selection with \u001B[33m1000\u001B[0m\u001B[1m features to be selected.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:30.212\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFeature selection finished.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:30.215\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected genes have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:30.316\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m0\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:30.434\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:30.444\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m1\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:30.526\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:30.538\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mrandom_clustering\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────── </span>FS method: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">seurat_v3</span>, Gene Number: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">2000</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ─────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m───────────────────────────────────── \u001B[0mFS method: \u001B[1;31mseurat_v3\u001B[0m, Gene Number: \u001B[1;31m2000\u001B[0m\u001B[92m ─────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:30.548\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mseurat_v3\u001B[0m\u001B[1m feature selection with \u001B[33m2000\u001B[0m\u001B[1m features to be selected.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:31.766\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFeature selection finished.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:31.769\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected genes have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:31.925\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m0\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.060\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.070\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m1\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.181\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.193\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mrandom_clustering\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">─────────────────────────────────── </span>FS method: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">random_select</span>, Gene Number: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">500</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m─────────────────────────────────── \u001B[0mFS method: \u001B[1;31mrandom_select\u001B[0m, Gene Number: \u001B[1;31m500\u001B[0m\u001B[92m ────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:32.205\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected genes have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.283\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m0\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.344\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.354\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mKMeans\u001B[0m\u001B[1m clustering with \u001B[33m7\u001B[0m\u001B[1m clusters and random state \u001B[33m1\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.413\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mKMeans\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.424\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mrandom_clustering\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.476\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mrecords have been saved into './2023-02 14_54_32 scrna.xlsx'.\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "data_cfg = {'PBMC3k': {\n",
    "        'adata_path': '../tests/data/scrna/pbmc3k_raw.h5ad',\n",
    "        'annot_key': 'louvain',\n",
    "        'to_replace': {'Dendritic': ['Dendritic cells']},\n",
    "        'to_remove': [np.nan, 'Megakaryocytes']\n",
    "    }}\n",
    "fs_cfg = {'seurat_v3': [1000, 2000], random_select: [500]}\n",
    "cl_cfg = {'KMeans': 2,  random_clustering: 1}\n",
    "\n",
    "run_bench(data_cfg, fs_cfg, cl_cfg, ['ARI', 'NMI'], 'scrna', clean_cache=True, fs_kwarg={'seed': 123}, cl_kwarg={'k':5, 'seed': 123})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e865b91d-0dac-4743-8bd7-6bf2e4a2196b",
   "metadata": {},
   "source": [
    "# Run benchmark on SRT data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dfb1649-1470-4d65-8a94-675424cbb147",
   "metadata": {},
   "source": [
    "## Evaluate predefined methods by the benchmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ca189e12-d190-4446-9560-2551f56692fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:32.549\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m./cache has been deleted.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">─────────────────────────────────────────────────── </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">mouse_brain</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ───────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m─────────────────────────────────────────────────── \u001B[0m\u001B[1;31mmouse_brain\u001B[0m\u001B[92m ───────────────────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:32.554\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mReading adata...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:32.555\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mNo cache for \u001B[35mmouse_brain\u001B[0m\u001B[1m. Trying to read h5ad from the given path in config...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:34.046\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mUsing raw data in `adata.raw`...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:34.100\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFound sparse matrix in `adata.X`. Converting to ndarray...\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Before QC: <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2688</span> cells and <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">18078</span> genes.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Before QC: \u001B[1;33m2688\u001B[0m cells and \u001B[1;33m18078\u001B[0m genes.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:34.931\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mObservation names and Variables names are all unique now.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:54:34.939\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mDropping \u001B[33m0\u001B[0m\u001B[1m special genes from \u001B[33m18078\u001B[0m\u001B[1m genes...\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">After QC: <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2688</span> cells and <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">16957</span> genes.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "After QC: \u001B[1;33m2688\u001B[0m cells and \u001B[1;33m16957\u001B[0m genes.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:39.665\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mImage has been loaded.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">────────────────────────────────────── </span>FS method: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">SPARKX</span>, Gene Number: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">1000</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ───────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m────────────────────────────────────── \u001B[0mFS method: \u001B[1;31mSPARKX\u001B[0m, Gene Number: \u001B[1;31m1000\u001B[0m\u001B[92m ───────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:54:39.668\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mSPARKX\u001B[0m\u001B[1m feature selection with \u001B[33m1000\u001B[0m\u001B[1m features to be selected.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:16.758\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFeature selection finished.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:16.763\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected genes have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:17.027\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mspaGCN\u001B[0m\u001B[1m clustering with \u001B[33m15\u001B[0m\u001B[1m clusters and random state \u001B[33m0\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:29.956\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mspaGCN\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:30.016\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mrecords have been saved into './2023-02 14_55_29 spatial.xlsx'.\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "data_cfg = {'mouse_brain': {\n",
    "    'adata_path': '../tests/data/spatial/V1_Adult_Mouse_Brain.h5ad',\n",
    "    'image_path': '../tests/data/spatial/img.jpg',\n",
    "    'annot_key': 'cluster',\n",
    "}}\n",
    "fs_cfg = {'SPARKX': [1000]}\n",
    "cl_cfg = {'spaGCN': 1}\n",
    "\n",
    "run_bench(data_cfg, fs_cfg, cl_cfg, ['ARI', 'NMI'], 'spatial', clean_cache=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "176fa638-bfe9-45d9-a4e3-fd12517c700e",
   "metadata": {},
   "source": [
    "## Evaluate custom methods defined by the user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5192d83e-b028-45ce-bfe8-1f119dc2d24e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:55:30.107\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m./cache has been deleted.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">─────────────────────────────────────────────────── </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">mouse_brain</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ───────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m─────────────────────────────────────────────────── \u001B[0m\u001B[1;31mmouse_brain\u001B[0m\u001B[92m ───────────────────────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:55:30.112\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mReading adata...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:30.112\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mNo cache for \u001B[35mmouse_brain\u001B[0m\u001B[1m. Trying to read h5ad from the given path in config...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:31.594\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mUsing raw data in `adata.raw`...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:31.650\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mFound sparse matrix in `adata.X`. Converting to ndarray...\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Before QC: <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2688</span> cells and <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">18078</span> genes.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Before QC: \u001B[1;33m2688\u001B[0m cells and \u001B[1;33m18078\u001B[0m genes.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:55:32.491\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mObservation names and Variables names are all unique now.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:32.502\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mDropping \u001B[33m0\u001B[0m\u001B[1m special genes from \u001B[33m18078\u001B[0m\u001B[1m genes...\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">After QC: <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2688</span> cells and <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">16957</span> genes.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "After QC: \u001B[1;33m2688\u001B[0m cells and \u001B[1;33m16957\u001B[0m genes.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:55:37.245\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mImage has been loaded.\n",
      "\u001B[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">─────────────────────────────────── </span>FS method: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">random_select</span>, Gene Number: <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">2000</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ───────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[92m─────────────────────────────────── \u001B[0mFS method: \u001B[1;31mrandom_select\u001B[0m, Gene Number: \u001B[1;31m2000\u001B[0m\u001B[92m ───────────────────────────────────\u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[32m2023-02-10 14:55:37.252\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mSelected genes have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:37.574\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mRunning \u001B[35mspaGCN\u001B[0m\u001B[1m clustering with \u001B[33m15\u001B[0m\u001B[1m clusters and random state \u001B[33m0\u001B[0m\u001B[1m...\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:43.715\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mspaGCN\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:43.733\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1m\u001B[35mrandom_clustering\u001B[0m\u001B[1m clustering results have been cached.\n",
      "\u001B[0m\u001B[32m2023-02-10 14:55:43.792\u001B[0m | \u001B[1mINFO \u001B[0m | \u001B[1mrecords have been saved into './2023-02 14_55_43 spatial.xlsx'.\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "data_cfg = {'mouse_brain': {\n",
    "    'adata_path': '../tests/data/spatial/V1_Adult_Mouse_Brain.h5ad',\n",
    "    'image_path': '../tests/data/spatial/img.jpg',\n",
    "    'annot_key': 'cluster',\n",
    "}}\n",
    "fs_cfg = {random_select: [2000]}\n",
    "cl_cfg = {'spaGCN': 1,  random_clustering: 1}\n",
    "\n",
    "run_bench(data_cfg, fs_cfg, cl_cfg, ['ARI', 'NMI'], 'spatial', clean_cache=True, fs_kwarg={'seed': 123}, cl_kwarg={'k':5, 'seed': 123})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bioinfo",
   "language": "python",
   "name": "bioinfo"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}