notebooks/training_demo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "omnWjd7NRKFq"
      },
      "source": [
        "# Training Demo\n",
        "\n",
        "In this notebook we will run training script for the work [*Unsupervised Change Detection of Extreme Events Using ML On-Board*](http://arxiv.org/abs/2111.02995). This work was conducted at the [FDL Europe 2021](https://fdleurope.org/fdl-europe-2021) research accelerator program. \n",
        "\n",
        "**These instructions are meant to work on your local machine** (we don't use the Google Colab environment)\n",
        "\n",
        "*Note that in practice this takes long time, so this should serve only as an orientational demo.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o93G2PiIRKFt"
      },
      "source": [
        "## 1 Preparation\n",
        "\n",
        "- Get the dataset (for this demo we also provide a tiny training dataset subset - see below)\n",
        "\n",
        "- For better visualizations log into weights and biases with: wandb init\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ofzitDKwRKFt"
      },
      "source": [
        "## 2 Libraries\n",
        "\n",
        "**Run these:**\n",
        "\n",
        "```\n",
        "make requirements\n",
        "conda activate ravaen_env\n",
        "conda install nb_conda\n",
        "jupyter notebook\n",
        "# start this notebook\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install --quiet --upgrade gdown"
      ],
      "metadata": {
        "id": "GGNRwsr8SKVj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p3RyYqyzRKFu",
        "outputId": "d9feb9bc-f0a6-4b1e-c5ca-7b5ba73538ed"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "     active environment : ravaen_env\r\n"
          ]
        }
      ],
      "source": [
        "!conda info | grep 'active environment'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gg1167yWRKFx",
        "outputId": "32651f09-03ac-4c6e-dba0-e932a6c91262"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Tue Mar  1 20:35:33 2022       \r\n",
            "+-----------------------------------------------------------------------------+\r\n",
            "| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |\r\n",
            "|-------------------------------+----------------------+----------------------+\r\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n",
            "|                               |                      |               MIG M. |\r\n",
            "|===============================+======================+======================|\r\n",
            "|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |\r\n",
            "| N/A   48C    P8     3W /  N/A |    256MiB /  7982MiB |     10%      Default |\r\n",
            "|                               |                      |                  N/A |\r\n",
            "+-------------------------------+----------------------+----------------------+\r\n",
            "                                                                               \r\n",
            "+-----------------------------------------------------------------------------+\r\n",
            "| Processes:                                                                  |\r\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\r\n",
            "|        ID   ID                                                   Usage      |\r\n",
            "|=============================================================================|\r\n",
            "|    0   N/A  N/A      1213      G   /usr/lib/xorg/Xorg                108MiB |\r\n",
            "|    0   N/A  N/A      1920      G   /usr/bin/gnome-shell               95MiB |\r\n",
            "|    0   N/A  N/A      2472      G   ...AAAAAAAAA= --shared-files       48MiB |\r\n",
            "+-----------------------------------------------------------------------------+\r\n"
          ]
        }
      ],
      "source": [
        "!nvidia-smi"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# The official training dataset is much larger, for the purpose of the demo, we provide a small subset:\n",
        "!gdown https://drive.google.com/uc?id=1rl3Clf0c7HlXnlPXO837Pjr2iCjwak0Y -O train_minisubset.zip\n",
        "!unzip -q train_minisubset.zip\n",
        "!rm train_minisubset.zip"
      ],
      "metadata": {
        "id": "OS17PWKWSMpV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwc457wzRKFx"
      },
      "source": [
        "**Edit the paths in config/config.yaml**\n",
        "\n",
        "```\n",
        "log_dir: \"/home/<USER>/results\"\n",
        "cache_dir: \"/home/<USER>/cache\"\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "633WMEtkRKFy",
        "outputId": "f375b444-7d01-44a5-eb18-1df41db51107"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "---\r\n",
            "entity: \"mlpayloads\"\r\n",
            "\r\n",
            "log_dir: \"/home/vitek/fdl_tmp/results\"\r\n",
            "cache_dir: \"/home/vitek/fdl_tmp/cache\"\r\n",
            "\r\n"
          ]
        }
      ],
      "source": [
        "!cat config/config.yaml\n",
        "\"\"\"\n",
        "Fill in:\n",
        "log_dir: \"/home/<USER>/results\"\n",
        "cache_dir: \"/home/<USER>/cache\"\n",
        "\"\"\"\n",
        "pass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EaNE7JUlRKFy",
        "outputId": "d714853b-442f-4a60-b856-376f011c4cf3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Global seed set to 42\n",
            "\n",
            "LATENT SPACE size: 128\n",
            "/home/vitek/anaconda3/envs/ravaen_env/lib/python3.9/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:338: UserWarning: ModelCheckpoint(save_last=True, save_top_k=None, monitor=None) is a redundant configuration. You can save the last checkpoint with ModelCheckpoint(save_top_k=None, monitor=None).\n",
            "  rank_zero_warn(\n",
            "ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.\n",
            "GPU available: True, used: True\n",
            "TPU available: False, using: 0 TPU cores\n",
            "Using native 16bit precision.\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mmlpayloads\u001b[0m (use `wandb login --relogin` to force relogin)\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: wandb version 0.12.11 is available!  To upgrade, please run:\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m:  $ pip install wandb --upgrade\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Tracking run with wandb version 0.12.10\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Syncing run \u001b[33msrc.models.module.Module/train_VAE_128small\u001b[0m\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: ⭐️ View project at \u001b[34m\u001b[4mhttps://wandb.ai/mlpayloads/train_VAE_128small\u001b[0m\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: 🚀 View run at \u001b[34m\u001b[4mhttps://wandb.ai/mlpayloads/train_VAE_128small/runs/3leewg5y\u001b[0m\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Run data is saved locally in /home/vitek/fdl_tmp/results/wandb/run-20220301_203551-3leewg5y\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Run `wandb offline` to turn off syncing.\n",
            "\n",
            "\n",
            "  | Name             | Type          | Params\n",
            "---------------------------------------------------\n",
            "0 | model            | DeeperVAE     | 443 K \n",
            "1 | da_transformer   | NoTransformer | 0     \n",
            "2 | eval_transformer | NoTransformer | 0     \n",
            "---------------------------------------------------\n",
            "443 K     Trainable params\n",
            "0         Non-trainable params\n",
            "443 K     Total params\n",
            "1.776     Total estimated model params size (MB)\n",
            "Validation sanity check:   0%|                            | 0/2 [00:00<?, ?it/s]/home/vitek/anaconda3/envs/ravaen_env/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
            "  warnings.warn(_create_warning_msg(\n",
            "Validation sanity check:  50%|██████████          | 1/2 [00:14<00:14, 14.64s/it]/home/vitek/Vitek/python_codes/FDL21/LOCAL_MAIN_DEV_BRANCH/change-detection/src/callbacks/visualisation_callback.py:33: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
            "  sample = [torch.tensor(s) for s in sample]\n",
            "Global seed set to 42                                                           \n",
            "/home/vitek/anaconda3/envs/ravaen_env/lib/python3.9/site-packages/pytorch_lightning/callbacks/lr_monitor.py:97: RuntimeWarning: You are using `LearningRateMonitor` callback with models that have no learning rate schedulers. Please see documentation for `configure_optimizers` method.\n",
            "  rank_zero_warn(\n",
            "Epoch 0:  92%|█████████▏| 66/72 [01:31<00:08,  1.39s/it, loss=0.328, v_num=wg5y]\n",
            "Validating: 0it [00:00, ?it/s]\u001b[A\n",
            "Validating:   0%|                                         | 0/6 [00:00<?, ?it/s]\u001b[A\n",
            "Epoch 0:  99%|█████████▊| 71/72 [01:43<00:01,  1.46s/it, loss=0.328, v_num=wg5y]\u001b[A\n",
            "Epoch 0: 100%|██████████| 72/72 [01:44<00:00,  1.45s/it, loss=0.328, v_num=wg5y]\u001b[A\n",
            "Epoch 1:  43%|████▎     | 31/72 [00:42<00:55,  1.36s/it, loss=0.287, v_num=wg5y]\u001b[A^C\n",
            "/home/vitek/anaconda3/envs/ravaen_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:897: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...\n",
            "  rank_zero_warn('Detected KeyboardInterrupt, attempting graceful shutdown...')\n"
          ]
        }
      ],
      "source": [
        "# ===== Parameters to adjust =====\n",
        "epochs = 100\n",
        "dataset_root_folder = \"<where we downloaded the data>/train_minisubset\"\n",
        "dataset=\"alpha_multiscene_tiny\" # for the demo, for the full training dataset we would use: dataset=\"alpha_multiscene\"\n",
        "\n",
        "name=\"VAE_128small\" # note \"small\" uses these settings > module.model_cls_args.latent_dim=128 module.model_cls_args.extra_depth_on_scale=0 module.model_cls_args.hidden_channels=[16,32,64]\n",
        "\n",
        "# ===== Parameters to keep the same ======\n",
        "training=\"simple_vae\"\n",
        "module=\"deeper_vae\"\n",
        "\n",
        "# ========================================\n",
        "\n",
        "!python3 -m scripts.train_model +dataset=$dataset ++dataset.root_folder=\"{dataset_root_folder}\" \\\n",
        "         +normalisation=log_scale +channels=high_res +training=$training +module=$module +project=train_VAE_128small +name=\"{name}\" \\\n",
        "         module.model_cls_args.latent_dim=128 module.model_cls_args.extra_depth_on_scale=0 module.model_cls_args.hidden_channels=[16,32,64] \\\n",
        "         training.epochs=$epochs\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cbYzpJwXRKFz"
      },
      "source": [
        "### More advanced settings:\n",
        "\n",
        "See the possible options using --help and then looking at the individual configuration files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qgxa-F3zRKF0",
        "outputId": "e7314858-d5d3-4c2d-9ab5-dd76538031e7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "train_model is powered by Hydra.\r\n",
            "\r\n",
            "== Configuration groups ==\r\n",
            "Compose your configuration from those groups (group=option)\r\n",
            "\r\n",
            "channels: all, high_res, high_res_phisat2overlap, rgb, rgb_nir, rgb_nir_b11, rgb_nir_b11_b12_landsat, rgb_nir_b12\r\n",
            "dataset: alpha_multiscene, alpha_multiscene_tiny, alpha_singlescene, dataloader_test, eval, fire, fires, floods_evaluation, hurricanes, landslides, landslides_2, oilspills, preliminary, preliminary_da, preliminary_multiscene, preliminary_sequential, preliminary_sequential_bigger, preliminary_sequential_bigger_9k, preliminary_sequential_bigger_multiEval, preliminary_sequential_bigger_multiEval_Germany, samples_for_gui, temporal_analysis, volcanos\r\n",
            "evaluation: ae_base, ae_fewer, vae_base, vae_da, vae_da_8px, vae_fewer, vae_paper\r\n",
            "module: deeper_ae, deeper_ae_bigger_latent, deeper_vae, grx, simple_ae, simple_ae_with_linear, simple_vae\r\n",
            "normalisation: log_scale, none\r\n",
            "training: da, simple_ae, simple_vae\r\n",
            "transform: eval_da, eval_da_8px, eval_nda, eval_nda_8px, none, random, random_1px, random_4px, random_6px, simple\r\n",
            "\r\n",
            "\r\n",
            "== Config ==\r\n",
            "Override anything in the config (foo.bar=value)\r\n",
            "\r\n",
            "entity: mlpayloads\r\n",
            "log_dir: /home/vitek/fdl_tmp/results\r\n",
            "cache_dir: /home/vitek/fdl_tmp/cache\r\n",
            "\r\n",
            "\r\n",
            "Powered by Hydra (https://hydra.cc)\r\n",
            "Use --hydra-help to view Hydra specific help\r\n",
            "\r\n",
            "\r\n"
          ]
        }
      ],
      "source": [
        "!python3 -m scripts.train_model --help"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s_cDsLxHRKF1",
        "outputId": "21b4ec6b-3ca7-4382-cf57-c2f3adc98ec3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "---\r\n",
            "gpus: -1\r\n",
            "epochs: 400\r\n",
            "grad_batches: 1\r\n",
            "distr_backend: 'dp'\r\n",
            "use_amp: true # ... true = 16 precision / false = 32 precision\r\n",
            "\r\n",
            "# The check_val_every_n_epoch and val_check_interval settings overlap, see:\r\n",
            "#     https://github.com/PyTorchLightning/pytorch-lightning/issues/6385\r\n",
            "val_check_interval: 0.2  # either in to check after that many batches or float to check that fraction of epoch\r\n",
            "check_val_every_n_epoch: 1 \r\n",
            "\r\n",
            "fast_dev_run: false\r\n",
            "\r\n",
            "num_workers: 16\r\n",
            "\r\n",
            "batch_size_train: 256\r\n",
            "batch_size_valid: 256\r\n",
            "batch_size_test: 256\r\n",
            "\r\n",
            "lr: 0.001\r\n",
            "weight_decay: 0.0\r\n",
            "# scheduler_gamma: 0.95\r\n",
            "\r\n",
            "# auto_batch_size: 'binsearch'\r\n",
            "#auto_lr: 'lr'\r\n"
          ]
        }
      ],
      "source": [
        "# to see the detiled options for \"training: da, simple_ae, simple_vae\"\n",
        "!cat config/training/simple_vae.yaml\n",
        "# for example we would then set epochs with adding this to the main command:\n",
        "# training.epochs=1"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.6"
    },
    "colab": {
      "name": "training_demo.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}