3_POSIX_Bash.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8369f35a-17cc-46ff-b1f0-dc8c5439100c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# EMBO Practical Course <br/>\"Advanced methods in bioimage analysis\"\n",
    "\n",
    "***\n",
    "\n",
    "Homepage: https://www.embl.org/about/info/course-and-conference-office/events/bia23-01/\n",
    "\n",
    "***\n",
    "\n",
    "## Day 2 - Session 1: Image Data Management - 10:15 to 11:15 \"Get set\"\n",
    "\n",
    "\n",
    "<table style=\"table { position: relative;  display: inline-block; } img {  position: absolute;  left: 0;  right: 0;  width: auto;  height: 100%;  object-fit: cover;  object-position: center;}\">\n",
    "    <tr>\n",
    "        <td style=\"vertical-align: top\">\n",
    "            <h3>Introduction</h3>\n",
    "            <p>\n",
    "                In this notebook, we'll look at issues arising from having many files\n",
    "                (<b>and many file formats!</b>b>) on your file servers. A quick intro into working\n",
    "                with POSIX file systems will be followed by tools for converting data into\n",
    "                a common file-format: OME-TIFF.\n",
    "            </p>\n",
    "            <p>\n",
    "                The two main goals are: (1) make sure you have the basics you might need\n",
    "                for the rest of the course and (2) encourage you to explore these tools\n",
    "                further on your own to speed up your day-to-day activities.\n",
    "            </p>\n",
    "            <p>\n",
    "                Outline:\n",
    "                <ol start=\"3\">\n",
    "                    <li>Bash & Scripting\n",
    "                        <ol type=\"a\">\n",
    "                            <li>Bash basics</li>\n",
    "                            <li>Our data directory</li>\n",
    "                            <li>Extras: time permitting\n",
    "                                <ul>\n",
    "                                    <li>Permissions</li>\n",
    "                                    <li>Simple scripting</li>\n",
    "                                    <li>shasum</li>\n",
    "                                    <li>h5tools</li>\n",
    "                                </ul>\n",
    "                            </li>\n",
    "                        </ol>\n",
    "                    </li>\n",
    "                        <li>bftools\n",
    "                        <ol type=\"a\">\n",
    "                            <li>showinf, bfconvert, etc.</li>\n",
    "                            <li>Working with OME-TIFF</li>\n",
    "                            <li>aicsimageio (time-permitting)</li>\n",
    "                        </ol>\n",
    "                    </li>\n",
    "                </ol>\n",
    "            </p>\n",
    "        </td>\n",
    "        <td>\n",
    "            <center>\n",
    "                <img src=\"images/falk/clara-shares-300dpi.png\"/>\n",
    "            </center>\n",
    "            <center>\n",
    "                <small>\n",
    "                    <a href=\"https://github.com/zarr-developers/zarr-illustrations-falk-2022#clara-shares\">\"Clara shares\"</a>\n",
    "                    by Henning Falk, ©2022 NumFOCUS, is used under a CC BY 4.0 license. Modifications to this photo include cropping.\n",
    "                </small>\n",
    "            </center>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6c051f1-39d3-47d8-8a71-9f3796a9d6e8",
   "metadata": {},
   "source": [
    "### How to get started after the workshop?\n",
    "\n",
    "[Binder](https://mybinder.org/) is a service for running notebooks online which can often be useful. In order to be be compatible with Binder, there is a folder `binder/` which contains the necessary files for configurig [this repo](https://github.com/ome/EMBO-Practical-Course-2023). The `environment.yml` defines a [Conda](https://docs.conda.io/en/latest/) environment which you can install yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "34e2ebef-1dbc-4e1c-8673-fbfb227b5910",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apt.txt  environment.yml\n"
     ]
    }
   ],
   "source": [
    "ls binder/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3b0b6df5-515e-4a9c-a072-53c51bb76fe2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "channels:\n",
      "  - ome\n",
      "  - conda-forge\n",
      "  - joshmoore\n",
      "  - defaults\n",
      "dependencies:\n",
      "  - python ~= 3.9.0\n",
      "  - notebook\n",
      "  - nb_conda_kernels\n",
      "  - napari\n",
      "  - napari-aicsimageio\n",
      "  - bioformats_jar\n",
      "  - ipython\n",
      "  - wheel\n",
      "  - requests\n",
      "  - h5utils\n",
      "  - s3fs\n",
      "  - scikit-image\n",
      "  - scipy\n",
      "  - xarray\n",
      "  - zarr >= 2.4.0\n",
      "  - bftools\n",
      "  - bioformats2raw\n",
      "  - raw2ometiff\n",
      "  - omero-py\n",
      "  - go-mc\n",
      "  - pv\n",
      "  - pip\n",
      "  - pip:\n",
      "      - PyQT5~=5.13.1\n",
      "      - ome-zarr\n",
      "      - awscli\n",
      "      - omero-cli-zarr\n",
      "      - napari-ome-zarr\n",
      "      - xmlschema\n",
      "      - bash_kernel\n"
     ]
    }
   ],
   "source": [
    "cat binder/environment.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0de9b7f9-1e66-43c7-a5c4-e99e759a17ea",
   "metadata": {},
   "source": [
    "You can perform those actions on your own system _without_ a Jupyter notebook. For that, you will need to install the correct tools like `bioformats2raw`. Instructions are available under https://github.com/ome/EMBO-Practical-Course-2023/blob/main/README.md\n",
    " \n",
    "*** "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dcd8544-a71e-4b9a-9ebc-75bcdc8e5021",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## 3a. Bash basics\n",
    "\n",
    "If you are managing files on a server, it will often be easier to use the command-line once you learn it. And if you are using a command-line, it's likely going to be Bash. There are a large number of useful commands, and it will take time to become familiar with them all, <b>but it is worth your time.</b>\n",
    "\n",
    "Often, the shorter the name the more useful it is. Let's take `ls` for \"list\". To learn more about a tool, you can use the `man` tool, or any many cases by passing the `-h` or `--help` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9f7bd03e-464a-42c3-88a4-556318571e54",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man: can't set the locale; make sure $LC_* and $LANG are correct\n",
      "LS(1)                            User Commands                           LS(1)\n",
      "\n",
      "\n",
      "\n",
      "NAME\n",
      "       ls - list directory contents\n",
      "\n",
      "SYNOPSIS\n",
      "       ls [OPTION]... [FILE]...\n",
      "\n",
      "DESCRIPTION\n",
      "       List  information  about  the FILEs (the current directory by default).\n",
      "       Sort entries alphabetically if none of -cftuvSUX nor --sort  is  speci-\n",
      "       fied.\n",
      "\n",
      "       Mandatory  arguments  to  long  options are mandatory for short options\n",
      "       too.\n",
      "\n",
      "       -a, --all\n",
      "              do not ignore entries starting with .\n",
      "\n",
      "       -A, --almost-all\n",
      "              do not list implied . and ..\n",
      "\n",
      "       --author\n",
      "              with -l, print the author of each file\n",
      "\n",
      "       -b, --escape\n",
      "              print C-style escapes for nongraphic characters\n",
      "\n",
      "       --block-size=SIZE\n",
      "              scale sizes by SIZE before printing them; e.g., '--block-size=M'\n",
      "              prints sizes in units of 1,048,576 bytes; see SIZE format below\n",
      "\n",
      "       -B, --ignore-backups\n",
      "              do not list implied entries ending with ~\n",
      "\n",
      "       -c     with -lt: sort by, and show, ctime (time of last modification of\n",
      "              file status information); with -l: show ctime and sort by  name;\n",
      "              otherwise: sort by ctime, newest first\n",
      "\n",
      "       -C     list entries by columns\n",
      "\n",
      "       --color[=WHEN]\n",
      "              colorize  the  output;  WHEN can be 'never', 'auto', or 'always'\n",
      "              (the default); more info below\n",
      "\n",
      "       -d, --directory\n",
      "              list directories themselves, not their contents\n",
      "\n",
      "       -D, --dired\n",
      "              generate output designed for Emacs' dired mode\n",
      "\n",
      "       -f     do not sort, enable -aU, disable -ls --color\n",
      "\n",
      "       -F, --classify\n",
      "              append indicator (one of */=>@|) to entries\n",
      "\n",
      "       --file-type\n",
      "              likewise, except do not append '*'\n",
      "\n",
      "       --format=WORD\n",
      "              across -x, commas -m, horizontal -x, long -l, single-column  -1,\n",
      "              verbose -l, vertical -C\n",
      "\n",
      "       --full-time\n",
      "              like -l --time-style=full-iso\n",
      "\n",
      "       -g     like -l, but do not list owner\n",
      "\n",
      "       --group-directories-first\n",
      "              group directories before files;\n",
      "\n",
      "              can   be  augmented  with  a  --sort  option,  but  any  use  of\n",
      "              --sort=none (-U) disables grouping\n",
      "\n",
      "       -G, --no-group\n",
      "              in a long listing, don't print group names\n",
      "\n",
      "       -h, --human-readable\n",
      "              with -l, print sizes in human readable format (e.g., 1K 234M 2G)\n",
      "\n",
      "       --si   likewise, but use powers of 1000 not 1024\n",
      "\n",
      "       -H, --dereference-command-line\n",
      "              follow symbolic links listed on the command line\n",
      "\n",
      "       --dereference-command-line-symlink-to-dir\n",
      "              follow each command line symbolic link\n",
      "\n",
      "              that points to a directory\n",
      "\n",
      "       --hide=PATTERN\n",
      "              do not list implied entries matching shell  PATTERN  (overridden\n",
      "              by -a or -A)\n",
      "\n",
      "       --indicator-style=WORD\n",
      "              append indicator with style WORD to entry names: none (default),\n",
      "              slash (-p), file-type (--file-type), classify (-F)\n",
      "\n",
      "       -i, --inode\n",
      "              print the index number of each file\n",
      "\n",
      "       -I, --ignore=PATTERN\n",
      "              do not list implied entries matching shell PATTERN\n",
      "\n",
      "       -k, --kibibytes\n",
      "              default to 1024-byte blocks for disk usage\n",
      "\n",
      "       -l     use a long listing format\n",
      "\n",
      "       -L, --dereference\n",
      "              when showing file information for a symbolic link, show informa-\n",
      "              tion  for  the file the link references rather than for the link\n",
      "              itself\n",
      "\n",
      "       -m     fill width with a comma separated list of entries\n",
      "\n",
      "       -n, --numeric-uid-gid\n",
      "              like -l, but list numeric user and group IDs\n",
      "\n",
      "       -N, --literal\n",
      "              print raw entry names (don't treat e.g. control characters  spe-\n",
      "              cially)\n",
      "\n",
      "       -o     like -l, but do not list group information\n",
      "\n",
      "       -p, --indicator-style=slash\n",
      "              append / indicator to directories\n",
      "\n",
      "       -q, --hide-control-chars\n",
      "              print ? instead of nongraphic characters\n",
      "\n",
      "       --show-control-chars\n",
      "              show nongraphic characters as-is (the default, unless program is\n",
      "              'ls' and output is a terminal)\n",
      "\n",
      "       -Q, --quote-name\n",
      "              enclose entry names in double quotes\n",
      "\n",
      "       --quoting-style=WORD\n",
      "              use quoting style WORD for entry names: literal, locale,  shell,\n",
      "              shell-always, c, escape\n",
      "\n",
      "       -r, --reverse\n",
      "              reverse order while sorting\n",
      "\n",
      "       -R, --recursive\n",
      "              list subdirectories recursively\n",
      "\n",
      "       -s, --size\n",
      "              print the allocated size of each file, in blocks\n",
      "\n",
      "       -S     sort by file size\n",
      "\n",
      "       --sort=WORD\n",
      "              sort  by  WORD instead of name: none (-U), size (-S), time (-t),\n",
      "              version (-v), extension (-X)\n",
      "\n",
      "       --time=WORD\n",
      "              with -l, show time as WORD instead of default modification time:\n",
      "              atime or access or use (-u) ctime or status (-c); also use spec-\n",
      "              ified time as sort key if --sort=time\n",
      "\n",
      "       --time-style=STYLE\n",
      "              with -l, show times using style STYLE: full-iso, long-iso,  iso,\n",
      "              locale,  or  +FORMAT;  FORMAT  is interpreted like in 'date'; if\n",
      "              FORMAT  is  FORMAT1<newline>FORMAT2,  then  FORMAT1  applies  to\n",
      "              non-recent  files  and FORMAT2 to recent files; if STYLE is pre-\n",
      "              fixed with 'posix-', STYLE takes effect only outside  the  POSIX\n",
      "              locale\n",
      "\n",
      "       -t     sort by modification time, newest first\n",
      "\n",
      "       -T, --tabsize=COLS\n",
      "              assume tab stops at each COLS instead of 8\n",
      "\n",
      "       -u     with  -lt:  sort by, and show, access time; with -l: show access\n",
      "              time and sort by name; otherwise: sort by access time\n",
      "\n",
      "       -U     do not sort; list entries in directory order\n",
      "\n",
      "       -v     natural sort of (version) numbers within text\n",
      "\n",
      "       -w, --width=COLS\n",
      "              assume screen width instead of current value\n",
      "\n",
      "       -x     list entries by lines instead of by columns\n",
      "\n",
      "       -X     sort alphabetically by entry extension\n",
      "\n",
      "       -1     list one file per line\n",
      "\n",
      "       SELinux options:\n",
      "\n",
      "       --lcontext\n",
      "              Display security context.   Enable -l. Lines  will  probably  be\n",
      "              too wide for most displays.\n",
      "\n",
      "       -Z, --context\n",
      "              Display  security context so it fits on most displays.  Displays\n",
      "              only mode, user, group, security context and file name.\n",
      "\n",
      "       --scontext\n",
      "              Display only security context and file name.\n",
      "\n",
      "       --help display this help and exit\n",
      "\n",
      "       --version\n",
      "              output version information and exit\n",
      "\n",
      "       SIZE is an integer and optional unit (example:  10M  is  10*1024*1024).\n",
      "       Units  are K, M, G, T, P, E, Z, Y (powers of 1024) or KB, MB, ... (pow-\n",
      "       ers of 1000).\n",
      "\n",
      "       Using color to distinguish file types is disabled both by  default  and\n",
      "       with  --color=never.  With --color=auto, ls emits color codes only when\n",
      "       standard output is connected to a terminal.  The LS_COLORS  environment\n",
      "       variable can change the settings.  Use the dircolors command to set it.\n",
      "\n",
      "   Exit status:\n",
      "       0      if OK,\n",
      "\n",
      "       1      if minor problems (e.g., cannot access subdirectory),\n",
      "\n",
      "       2      if serious trouble (e.g., cannot access command-line argument).\n",
      "\n",
      "       GNU  coreutils  online  help:  <http://www.gnu.org/software/coreutils/>\n",
      "       Report ls translation bugs to <http://translationproject.org/team/>\n",
      "\n",
      "AUTHOR\n",
      "       Written by Richard M. Stallman and David MacKenzie.\n",
      "\n",
      "COPYRIGHT\n",
      "       Copyright (C) 2013 Free Software Foundation, Inc.  License GPLv3+:  GNU\n",
      "       GPL version 3 or later <http://gnu.org/licenses/gpl.html>.\n",
      "       This  is  free  software:  you  are free to change and redistribute it.\n",
      "       There is NO WARRANTY, to the extent permitted by law.\n",
      "\n",
      "SEE ALSO\n",
      "       The full documentation for ls is maintained as a  Texinfo  manual.   If\n",
      "       the  info and ls programs are properly installed at your site, the com-\n",
      "       mand\n",
      "\n",
      "              info coreutils 'ls invocation'\n",
      "\n",
      "       should give you access to the complete manual.\n",
      "\n",
      "\n",
      "\n",
      "GNU coreutils 8.22               November 2020                           LS(1)\n"
     ]
    }
   ],
   "source": [
    "man ls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50f22249-8de5-45ab-8008-20a972303796",
   "metadata": {},
   "source": [
    "****\n",
    "\n",
    "I bring up `ls` first because it can be used to find other commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ccebbfb1-275c-4391-a8af-5da4162fe861",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lrwxrwxrwx. 1 root root     7 Oct 30  2020 \u001b[0m\u001b[38;5;51m/bin\u001b[0m -> \u001b[38;5;27musr/bin\u001b[0m\n",
      "drwxr-xr-x. 2 root root   131 Aug 22 13:16 \u001b[38;5;27m/usr/local/bin\u001b[0m\n",
      "dr-xr-xr-x. 2 root root 61440 Aug 25 06:33 \u001b[38;5;27m/usr/bin/\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -ltrad /bin /usr/bin/ /usr/local/bin"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f335d3a3-34c8-4073-a8f1-87d60bbedbd0",
   "metadata": {},
   "source": [
    "The \"$PATH\" (an environment variable) defines *all* of the places that commands can be found. (Don't worry. You don't need to do anything with all of these)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bf643a5f-83dd-456c-b0b7-02034b11b225",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/munge-0.5.12/bin:/opt/slurm-21.08.8/bin:/opt/slurm-21.08.8/sbin:/opt/nhc-1.4.2/sbin:/scratch/bioimagecourse2023/session1/mambaforge/envs/embo/bin:/scratch/bioimagecourse2023/session1/mambaforge/condabin:/opt/munge-0.5.12/bin:/opt/slurm-21.08.8/bin:/opt/slurm-21.08.8/sbin:/opt/nhc-1.4.2/sbin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/nextflow:/scratch/bioimagecourse2023/session1:/opt/nextflow\n"
     ]
    }
   ],
   "source": [
    "echo $PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd52d478-18af-4bc1-b991-98efe8328205",
   "metadata": {},
   "source": [
    "These are the directories where you will find many of them. For example, the command `bash` is one of them:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "255f084a-9997-47d5-9fb8-ca051b76ae58",
   "metadata": {},
   "source": [
    "You can use the `tr` command to \"translate\" characters, here from \":\" to newlines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "14898917-32f0-48b1-a3da-deea1f3353c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/munge-0.5.12/bin\n",
      "/opt/slurm-21.08.8/bin\n",
      "/opt/slurm-21.08.8/sbin\n",
      "/opt/nhc-1.4.2/sbin\n",
      "/scratch/bioimagecourse2023/session1/mambaforge/envs/embo/bin\n",
      "/scratch/bioimagecourse2023/session1/mambaforge/condabin\n",
      "/opt/munge-0.5.12/bin\n",
      "/opt/slurm-21.08.8/bin\n",
      "/opt/slurm-21.08.8/sbin\n",
      "/opt/nhc-1.4.2/sbin\n",
      "/usr/local/bin\n",
      "/usr/bin\n",
      "/usr/local/sbin\n",
      "/usr/sbin\n",
      "/opt/nextflow\n",
      "/scratch/bioimagecourse2023/session1\n",
      "/opt/nextflow\n"
     ]
    }
   ],
   "source": [
    "echo $PATH | tr : '\\n'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aed869c-b9c9-465f-a01d-c5724c95ae4b",
   "metadata": {},
   "source": [
    "There are many more environment variables. You can see all of them by running \"set\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f08ec9f5-b286-4a8b-bdfc-58c6fdc4bf4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ABRT_DEBUG_LOG=/dev/null\n",
      "BASH=/usr/bin/bash\n",
      "BASHOPTS=checkwinsize:cmdhist:expand_aliases:extglob:extquote:force_fignore:histappend:interactive_comments:progcomp:promptvars:sourcepath\n",
      "BASH_ALIASES=()\n",
      "BASH_ARGC=()\n",
      "BASH_ARGV=()\n",
      "BASH_CMDS=()\n",
      "BASH_COMPLETION_COMPAT_DIR=/etc/bash_completion.d\n",
      "BASH_ENV=/usr/share/lmod/lmod/init/bash\n",
      "BASH_LINENO=()\n"
     ]
    }
   ],
   "source": [
    "set | head -n 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e135160e-a971-4faf-bd75-306aee57c61b",
   "metadata": {},
   "source": [
    "`which` is a command that tells you where a command lives. `/bin/bash` is the command that gets run when you login. It's your \"shell\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "03a47295-9aee-4bc3-8400-2b6c0ccd3da6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/bin/bash\n"
     ]
    }
   ],
   "source": [
    "which bash"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "342594c9-681e-4f9b-b51e-059be69a134e",
   "metadata": {},
   "source": [
    "There's also an environment variable that tells you what your shell is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f94217ec-c93d-459d-9e59-34d4ef87fd14",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash\n"
     ]
    }
   ],
   "source": [
    "echo $SHELL"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f95369aa-8c37-4c33-91bb-db001310b01f",
   "metadata": {},
   "source": [
    "That *shell* (or interpreter) holds the state of everything you are doing, including any variables you might set. (This is also why Jupyter can be confusing.)\n",
    "\n",
    "Any time you start a new shell, you will need to make sure all of those variables are reset. For example, most of our notebooks will make use of the variable `YOURNAME`, e.g., to create directories, etc. You can find out your login name with the `whoami` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cc13f438-4e8c-4bdc-bab2-976d946939a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "josh_openmicroscopy\n"
     ]
    }
   ],
   "source": [
    "whoami"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "668d2bb9-c15a-465f-bcfa-4c427dd64bdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "YOURNAME=$(whoami)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e1de7ba4-de1d-4199-8c1c-4ae81693ef6e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "josh_openmicroscopy\n"
     ]
    }
   ],
   "source": [
    "echo $YOURNAME"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f37c418-5d63-4b8d-b972-7310658b2e6a",
   "metadata": {},
   "source": [
    "Now we have the variable we want set."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8dd1ca4-7590-4484-b4e6-fd10ac9645e3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## 3c. Our data directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73ebb17c-d62b-46d3-a1bf-fb0edd31b346",
   "metadata": {},
   "source": [
    "Now with this out of the way, we can start to work with the data directories that are available on BAND.\n",
    "\n",
    "`pwd` prints the directory that the current process is in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4b67eee0-1222-4b54-999a-7f0ebedd0426",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/scratch/bioimagecourse2023/session1/EMBO-Practical-Course-2023\n"
     ]
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "874edf9d-34d2-4f49-a781-042f9e9e3edc",
   "metadata": {},
   "source": [
    "We want to move to another directory, for that we use the `cd` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2ff1acb9-82c9-450f-a561-d62e68282350",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd /scratch/bioimagecourse2023/session1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d25ced95-cf5e-4e4c-96b3-fa4f3bd03fe5",
   "metadata": {},
   "source": [
    "Now we are in our session directory. Feel free to look around using `ls`. For example the flags `-ltra` mean: \"show me a long listing of the files in reverse order by time and even show me the weird files starting with `.`\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "27ee9b79-ee85-41e5-b42a-75cb4e727e59",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 22917\n",
      "-r--r-----  1 josh_openmicroscopy users 23433941 Aug 29 10:44 \u001b[0m\u001b[38;5;9mOMEZarrReader-0.3.1-jar-with-dependencies.jar\u001b[0m\n",
      "-r-xr-xr-x  1 josh_openmicroscopy users       78 Aug 29 11:13 \u001b[38;5;34mnotebook.sh\u001b[0m\n",
      "dr-xr-xr-x 19 josh_openmicroscopy users     4096 Aug 29 14:53 \u001b[38;5;27mmambaforge\u001b[0m\n",
      "-r--r--r--  1 josh_openmicroscopy users      751 Aug 29 14:56 .mambarc\n",
      "-r--r--r--  1 josh_openmicroscopy users       38 Aug 31 09:17 .gitignore\n",
      "-r--r--r--  1 josh_openmicroscopy users      676 Aug 31 10:44 README.md\n",
      "dr-xr-xr-x  6 josh_openmicroscopy users     4096 Aug 31 10:48 \u001b[38;5;27mdata\u001b[0m\n",
      "dr-xr-xr-x  2 josh_openmicroscopy users     4096 Aug 31 14:12 \u001b[38;5;27mbin\u001b[0m\n",
      "-r--r--r--  1 josh_openmicroscopy users      197 Aug 31 14:25 setup.sh\n",
      "dr-xr-xr-x  7 josh_openmicroscopy users     4096 Aug 31 14:25 \u001b[38;5;27m.\u001b[0m\n",
      "dr-xr-xr-x  8 josh_openmicroscopy users     4096 Sep  4 15:10 \u001b[38;5;27m.git\u001b[0m\n",
      "drwxr-xr-x  3 josh_openmicroscopy users     4096 Sep  5 09:22 \u001b[38;5;27m..\u001b[0m\n",
      "drwxr-xr-x  8 josh_openmicroscopy users     4096 Sep 10 15:12 \u001b[38;5;27mEMBO-Practical-Course-2023\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -ltra"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f628c4bc-28ad-4bfa-b17b-a50eab7f5056",
   "metadata": {},
   "source": [
    "There are many other flags you can use. \"-S\" flag means sort by size instead of time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8a2c1979-2527-4ec5-bc79-97e766a85302",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 17\n",
      "-r--r--r-- 1 josh_openmicroscopy users    0 Aug 31 09:59 a.fake\n",
      "-r--r--r-- 1 josh_openmicroscopy users  617 Aug 31 10:47 README.md\n",
      "dr-xr-xr-x 3 josh_openmicroscopy users 4096 Aug 31 10:46 \u001b[0m\u001b[38;5;27mTIFF\u001b[0m\n",
      "dr-xr-sr-x 2 josh_openmicroscopy users 4096 Mar 15  2018 \u001b[38;5;27mSVS\u001b[0m\n",
      "dr-xr-xr-x 3 josh_openmicroscopy users 4096 Aug 22 14:56 \u001b[38;5;27mHCS\u001b[0m\n",
      "dr-xr-xr-x 2 josh_openmicroscopy users 4096 Mar 28  2018 \u001b[38;5;27mDV\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -lSr data/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7ed6ba9-1e3f-446a-a9b3-c62de7d4437b",
   "metadata": {},
   "source": [
    "Another important tool for understanding the size of your data is `du`, for \"disk usage\". The `-sh` flags \"say show me only a (s)ummary of the data and put it in (h)uman-readable form\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2f9aa050-0d2d-4270-bb67-e14531b0705d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13G\tdata/\n"
     ]
    }
   ],
   "source": [
    "\\du -sh data/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9561d93-7ee3-4d2b-b742-3b7f2d7a3550",
   "metadata": {},
   "source": [
    "(The `\\` in front of the `du` command means \"ignore any aliases that are being used\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c035e52-7555-45c8-a32c-6d67aef35f27",
   "metadata": {},
   "source": [
    "Now we want to make a directory for you to do all of your work in:\n",
    " * `mkdir -p`  means \"create all parents\" (but also: \"don't fail if it also exists\" which is useful for Jupyter notebooks!)\n",
    " * `test -e data ||` means \"if the file or directory `data` does not exist, then create it by \"linking\" it into your own directory. This doesn't make a copy but does make it easier for you to find things."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "446b7d36-6bec-45f7-8849-4c5e4b4a8ed8",
   "metadata": {},
   "source": [
    "### ln (symlinks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5fc481cd-dae9-493c-ae7c-c3fb6b49b457",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir -p /scratch/${YOURNAME}/session1\n",
    "cd /scratch/${YOURNAME}/session1\n",
    "test -e data || ln -s /scratch/bioimagecourse2023/session1/data data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9486abcc-242a-48c6-b74c-5a7821d85026",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 274\n",
      "lrwxrwxrwx 1 josh_openmicroscopy users     41 Aug 31 14:18 \u001b[0m\u001b[38;5;51mdata\u001b[0m -> \u001b[38;5;27m/scratch/bioimagecourse2023/session1/data\u001b[0m\n",
      "drwxr-xr-x 4 josh_openmicroscopy users   4096 Aug 31 14:50 \u001b[38;5;27ma.ome.zarr\u001b[0m\n",
      "drwxr-xr-x 3 josh_openmicroscopy users   4096 Sep 10 15:08 \u001b[38;5;27m..\u001b[0m\n",
      "drwx------ 3 josh_openmicroscopy users   4096 Sep 10 15:13 \u001b[38;5;27m.\u001b[0m\n",
      "-rw-r--r-- 1 josh_openmicroscopy users 267691 Sep 10 15:13 \u001b[38;5;13ma.ome.tiff\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -ltra /scratch/${YOURNAME}/session1/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d792552-3f63-44ff-8b2c-3d097229c646",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "`find` is a very powerful tool that we can use to explore directories full of lots of files. (In this case, we've added `tail` to only show the last time lines of all the output)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8030ddf2-9b74-433f-8efb-8fdaa400adb1",
   "metadata": {},
   "source": [
    "### find"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2bd5e2ab-7182-45da-bb23-e24d610f4a3c",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/HCS/Operetta/59549/59549__2013-04-03T09_44_09-Measurement1/Images\n",
      "data/HCS/Operetta/59549/59549__2013-04-03T09_44_09-Measurement1/Images/Index.ref.xml\n",
      "data/HCS/Operetta/59549/59549__2013-04-03T09_44_09-Measurement1/Assaylayout\n",
      "data/HCS/Operetta/59549/59549__2013-04-03T09_44_09-Measurement1/Assaylayout/Unnamed.xml\n",
      "data/SVS\n",
      "data/SVS/77928.svs\n",
      "data/SVS/77917.svs\n",
      "data/SVS/COPYING\n",
      "data/SVS/readme.txt\n",
      "data/a.fake\n"
     ]
    }
   ],
   "source": [
    "find data/ | tail -n 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cd3837e-6734-4a5a-a782-52898247e9bf",
   "metadata": {},
   "source": [
    "The `-name` flag can be used to match patterns on the filename. You can use `-iname` if you want to ignore the case of the files. The `wc` command counts words, lines, etc. The \"-l\" says print how many lines, i.e. how many TIFF files were found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e304cc36-6277-4ca8-a11c-fcc373e6c420",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2571\n"
     ]
    }
   ],
   "source": [
    "find data/ -name \"*.tiff\" | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "645d3c51-790e-4c15-acfc-95c6e5773c51",
   "metadata": {},
   "source": [
    "You can also run a command on every file you found. The `file` command says what type of a file the operating system *assumes* a file to be. (It's not always right.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "94248236-38e6-41b0-8ddb-654dfcac4cce",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/SVS/77928.svs: TIFF image data, little-endian\n",
      "data/SVS/77917.svs: TIFF image data, little-endian\n",
      "data/SVS/COPYING: UTF-8 Unicode text\n",
      "data/SVS/readme.txt: ASCII text\n"
     ]
    }
   ],
   "source": [
    "find data/SVS/ -type f -exec file {} \\;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "660d08e4-698e-4886-9e04-98fccda490c5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Excercise 1:\n",
    "### Use the tools above to find some data you want to use and symlink data into your own folder for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f709974e-a12d-46e5-86bb-324e8ba00642",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Do something here or on the command-line. If you are having issues, ask."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15316f2f-6f69-4884-96b1-b8f45fac43e1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## 3d. Extras (time permitting)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79012f09-f709-4ffb-a588-8a6713a64eff",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Permissions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f318ee52-cfe9-48eb-a7e5-f386556328f8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "In the directory listings above, e.g.:\n",
    "\n",
    "```\n",
    "drwxr-xr-x   936 root     wheel  29952 Jul 11 10:56 /usr/bin/\n",
    "drwxr-xr-x@   39 root     wheel   1248 Jul 11 10:56 /bin\n",
    "drwxr-xr-x  2086 jamoore  admin  66752 Aug 20 18:35 /usr/local/bin\n",
    "```\n",
    "\n",
    "the info here is **critical**:\n",
    "\n",
    "```\n",
    "PERMISSIONS ---- USER     GROUP   SIZE MODIFIED     NAME\n",
    "```\n",
    "\n",
    "Figuring out your user and group is pretty easy. `whoami` from above is your user. `id` can tell you your groups:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "be3d8817-16c7-446f-876a-8b92c015777b",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "uid=1558(josh_openmicroscopy) gid=1102(users) groups=1102(users)\n"
     ]
    }
   ],
   "source": [
    "id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29119aa0-ff84-4fb8-9797-0dea36ed3a53",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Together we can start to piece together whether or not you can read or edit a file. From https://www.grymoire.com/Unix/Permissions.html :\n",
    "\n",
    "```\n",
    "Three sections after the \"d\" for directory marker:\n",
    "+------------+------+-------+\n",
    "| Permission | Octal| Field |\n",
    "+------------+------+-------+\n",
    "| rwx------  | 700  | User  |\n",
    "| ---rwx---  | 070  | Group |\n",
    "| ------rwx  | 007  | Other |\n",
    "+------------+------+-------+\n",
    "\n",
    "Each section can be in one of 8 states:\n",
    "+-----+---+--------------------------+\n",
    "| rwx | 7 | Read, write and execute  |\n",
    "| rw- | 6 | Read, write              |\n",
    "| r-x | 5 | Read, and execute        |\n",
    "| r-- | 4 | Read,                    |\n",
    "| -wx | 3 | Write and execute        |\n",
    "| -w- | 2 | Write                    |\n",
    "| --x | 1 | Execute                  |\n",
    "| --- | 0 | no permissions           |\n",
    "+------------------------------------+\n",
    "\n",
    "Examples:\n",
    "+------------------------+-----------+\n",
    "| chmod u=rwx,g=rwx,o=rx | chmod 775 | For world executables files\n",
    "| chmod u=rwx,g=rx,o=    | chmod 750 | For executables by group only\n",
    "| chmod u=rw,g=r,o=r     | chmod 644 | For world readable files\n",
    "| chmod u=rw,g=r,o=      | chmod 640 | For group readable files\n",
    "| chmod u=rw,go=         | chmod 600 | For private readable files\n",
    "| chmod u=rwx,go=        | chmod 700 | For private executables\n",
    "+------------------------+-----------+\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6028683-9e12-4b18-a53c-fa73b9a95de1",
   "metadata": {},
   "source": [
    "This is *especially* important when using a shared folder like `/scratch`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "557179f6-7ecb-4304-a995-ab9dbdc10b81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 12\n",
      "drwxrwxrwx 69 root                root  4096 Sep 10 15:08 \u001b[0m\u001b[48;5;10;38;5;21m..\u001b[0m\n",
      "drwxr-xr-x  3 josh_openmicroscopy users 4096 Sep 10 15:08 \u001b[38;5;27m.\u001b[0m\n",
      "drwx------  3 josh_openmicroscopy users 4096 Sep 10 15:13 \u001b[38;5;27msession1\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -ltra /scratch/${YOURNAME}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ab3c3140-796c-4716-949a-4d9c7aed39dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "chmod go-rx /scratch/${YOURNAME}/session1/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a0af16f8-2311-40f5-a7c1-6adecf79e2f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 12\n",
      "drwxrwxrwx 69 root                root  4096 Sep 10 15:08 \u001b[0m\u001b[48;5;10;38;5;21m..\u001b[0m\n",
      "drwxr-xr-x  3 josh_openmicroscopy users 4096 Sep 10 15:08 \u001b[38;5;27m.\u001b[0m\n",
      "drwx------  3 josh_openmicroscopy users 4096 Sep 10 15:13 \u001b[38;5;27msession1\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -ltra /scratch/${YOURNAME}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65dab158-d85d-4c9e-8c2c-3c87fda5126b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### shasum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "59fb41a3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8ef95b369e7f23a8d46327098c70430d66a03b6f  data/HCS/Operetta/59549/b8a3e30d-258c-46f1-b8fe-a96fb51b9af6/r12c01-1974092126.tiff.gz\n",
      "c7e9dc169f48836492a1a0e338c074751bc1bcaf  data/HCS/Operetta/59549/b8a3e30d-258c-46f1-b8fe-a96fb51b9af6/r15c01-0062694384.tiff.gz\n",
      "364497ce13411f8f9c6dcf1d145eeb0dee3b24c9  data/HCS/Operetta/59549/b8a3e30d-258c-46f1-b8fe-a96fb51b9af6/r16c01-1169526669.tiff.gz\n",
      "3dcbe901190f6ece57b5ea246937fdd6feb53f8a  data/HCS/Operetta/59549/59549__2013-04-03T09_44_09-Measurement1/Images/Index.ref.xml\n",
      "4025ebf6563b60d82182dc75e6b957e0251e6242  data/HCS/Operetta/59549/59549__2013-04-03T09_44_09-Measurement1/Assaylayout/Unnamed.xml\n",
      "337fbe356624c0288a48f87b537e0d0b74d1b315  data/SVS/77928.svs\n",
      "03e0322f8f46ba058b02ce0b5ab4c203df9a6a73  data/SVS/77917.svs\n",
      "1167f0e28fe2db01e38e883aaf1e749fb09f9ceb  data/SVS/COPYING\n",
      "faf783f348878c99f05ccf40e5ba78901a0e3149  data/SVS/readme.txt\n",
      "da39a3ee5e6b4b0d3255bfef95601890afd80709  data/a.fake\n"
     ]
    }
   ],
   "source": [
    "for x in $(find data/ -type f | tail -n 10);\n",
    "do\n",
    "    sha1sum ${x}\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "936cebd7-a820-428d-ad96-069f8d2802bf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### h5tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "bbaa5b56-f724-4912-a233-5d721909af4b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: h5ls [OPTIONS] file[/OBJECT] [file[/[OBJECT]...]\n",
      "  OPTIONS\n",
      "   -h, -?, --help  Print a usage message and exit\n",
      "   -a, --address   Print raw data address.  If dataset is contiguous, address\n",
      "                   is offset in file of beginning of raw data. If chunked,\n",
      "                   returned list of addresses indicates offset of each chunk.\n",
      "                   Must be used with -v, --verbose option.\n",
      "                   Provides no information for non-dataset objects.\n",
      "   -d, --data      Print the values of datasets\n",
      "   --enable-error-stack\n",
      "                   Prints messages from the HDF5 error stack as they occur.\n",
      "   --follow-symlinks\n",
      "                   Follow symbolic links (soft links and external links)\n",
      "                   to display target object information.\n",
      "                   Without this option, h5ls identifies a symbolic link\n",
      "                   as a soft link or external link and prints the value\n",
      "                   assigned to the symbolic link; it does not provide any\n",
      "                   information regarding the target object or determine\n",
      "                   whether the link is a dangling link.\n",
      "   --no-dangling-links\n",
      "                   Must be used with --follow-symlinks option;\n",
      "                   otherwise, h5ls shows error message and returns an exit\n",
      "                   code of 1.\n",
      "                   Check for any symbolic links (soft links or external links)\n",
      "                   that do not resolve to an existing object (dataset, group,\n",
      "                   or named datatype).\n",
      "                   If any dangling link is found, this situation is treated\n",
      "                   as an error and h5ls returns an exit code of 1.\n",
      "   -f, --full      Print full path names instead of base names\n",
      "   -g, --group     Show information about a group, not its contents\n",
      "   -l, --label     Label members of compound datasets\n",
      "   -r, --recursive List all groups recursively, avoiding cycles\n",
      "   -s, --string    Print 1-byte integer datasets as ASCII\n",
      "   -S, --simple    Use a machine-readable output format\n",
      "   -wN, --width=N  Set the number of columns of output\n",
      "   -v, --verbose   Generate more verbose output\n",
      "   -V, --version   Print version number and exit\n",
      "   --vfd=DRIVER    Use the specified virtual file driver\n",
      "   -x, --hexdump   Show raw data in hexadecimal format\n",
      "   --s3-cred=C     Supply S3 authentication information to \"ros3\" vfd.\n",
      "                   Accepts tuple of \"(<aws-region>,<access-id>,<access-key>)\".\n",
      "                   If absent or C->\"(,,)\", defaults to no-authentication.\n",
      "                   Has no effect if vfd flag not set to \"ros3\".\n",
      "   --hdfs-attrs=A  Supply configuration information to Hadoop VFD.\n",
      "                   Accepts tuple of (<namenode name>,<namenode port>,\n",
      "                   ...<kerberos cache path>,<username>,<buffer size>)\n",
      "                   If absent or A == '(,,,,)', all default values are used.\n",
      "                   Has no effect if vfd flag is not 'hdfs'.\n",
      "   --vol-value     Value (ID) of the VOL connector to use for opening the\n",
      "                   HDF5 file specified\n",
      "   --vol-name      Name of the VOL connector to use for opening the\n",
      "                   HDF5 file specified\n",
      "   --vol-info      VOL-specific info to pass to the VOL connector used for\n",
      "                   opening the HDF5 file specified\n",
      "   --vfd-value     Value (ID) of the VFL driver to use for opening the\n",
      "                   HDF5 file specified\n",
      "   --vfd-name      Name of the VFL driver to use for opening the\n",
      "                   HDF5 file specified\n",
      "   --vfd-info      VFD-specific info to pass to the VFL driver used for\n",
      "                   opening the HDF5 file specified\n",
      "\n",
      "  file/OBJECT\n",
      "    Each object consists of an HDF5 file name optionally followed by a\n",
      "    slash and an object name within the file (if no object is specified\n",
      "    within the file then the contents of the root group are displayed).\n",
      "    The file name may include a printf(3C) integer format such as\n",
      "    \"%05d\" to open a file family.\n",
      "\n",
      "  Deprecated Options\n",
      "    The following options have been removed in HDF5 1.12. Use the indicated\n",
      "    replacement option in all work.\n",
      "\n",
      "   -E or --external   Follow external links.\n",
      "                      Replaced by --follow-symlinks.\n",
      "   -e, --errors       Show all HDF5 error reporting\n",
      "                      Replaced by --enable-error-stack.\n"
     ]
    }
   ],
   "source": [
    "h5ls -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9b0ac65-f1a1-41d8-8f4d-8946492bf36c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Learning more\n",
    "\n",
    "This is obviously just a starting point. Perhaps check out a [Carpentry lesson](https://software-carpentry.org/lessons/) like [The Unix Shell](https://swcarpentry.github.io/shell-novice/) for more information. I can highly recommend them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7af9229-a6ce-4072-82c9-778ccbb8abf1",
   "metadata": {},
   "source": [
    "<hr/>\n",
    "\n",
    "## Half-time\n",
    "\n",
    "<hr/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "030e9b93-2683-4848-8ee1-e8da0e8360b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "##\n",
    "## Setup & Sanity checks\n",
    "##\n",
    "\n",
    "YOURNAME=$(whoami)\n",
    "WORKDIR=/scratch/${YOURNAME}/session1\n",
    "test -e ${WORKDIR} || {\n",
    "    echo Please run the steps above first.\n",
    "    exit 1\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8662a965-5294-46fb-899c-0e5b3b933d9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd /scratch/${YOURNAME}/session1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8fd589d-cd08-45d9-9eee-7475852b9d3b",
   "metadata": {},
   "source": [
    "## 4a. bftools: showinf, bfconvert, etc.\n",
    "\n",
    "bftools is a package of tools that can be downloaded from https://downloads.openmicroscopy.org/bio-formats/latest/artifacts/ or [conda](https://anaconda.org/ome/bftools) for working with bioimaging files from the command-line (`conda install -c ome bftools`).\n",
    "\n",
    "One of the tools, `formatlist` simply lists all the supported file formats with their associated file endings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f3a0f17d-c1f6-4e0d-96b7-159112cf2023",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "File pattern: can read (pattern)\n",
      "Zip: can read (zip)\n",
      "Animated PNG: can read, can write, can write multiple (png)\n",
      "JPEG: can read, can write (jpg, jpeg, jpe)\n",
      "SlideBook 7 SLD (native): can read (sldy)\n",
      "Portable Any Map: can read (pbm, pgm, ppm)\n",
      "Flexible Image Transport System: can read (fits, fts)\n",
      "PCX: can read (pcx)\n",
      "Graphics Interchange Format: can read (gif)\n",
      "Windows Bitmap: can read (bmp)\n",
      "IPLab: can read (ipl)\n",
      "IVision: can read (ipm)\n",
      "RCPNL: can read (rcpnl)\n",
      "Deltavision: can read (dv, r3d, r3d_d3d, dv.log, r3d.log)\n",
      "Medical Research Council: can read (mrc, st, ali, map, rec, mrcs)\n",
      "Gatan Digital Micrograph: can read (dm3, dm4)\n",
      "Gatan DM2: can read (dm2)\n",
      "Bitplane Imaris: can read (ims)\n",
      "Openlab RAW: can read (raw)\n",
      "OME-XML: can read, can write, can write multiple (ome, ome.xml)\n",
      "Leica Image File Format: can read (lif)\n",
      "Audio Video Interleave: can read, can write, can write multiple (avi)\n",
      "PICT: can read (pict, pct)\n",
      "SPCImage Data: can read (sdt)\n",
      "SPC FIFO Data: can read (spc, set)\n",
      "Encapsulated PostScript: can read, can write (eps, epsi, ps)\n",
      "Olympus Slidebook: can read (sld, spl)\n",
      "Alicona AL3D: can read (al3d)\n",
      "Multiple-image Network Graphics: can read (mng)\n",
      "Khoros XV: can read (xv)\n",
      "Visitech XYS: can read (xys, html)\n",
      "Laboratory Imaging: can read (lim)\n",
      "Adobe Photoshop: can read (psd)\n",
      "InCell 1000/2000: can read (xdce, xml, tiff, tif, xlog)\n",
      "Li-Cor L2D: can read (l2d, scn, tif)\n",
      "FEI/Philips: can read (img)\n",
      "Hamamatsu Aquacosmos: can read (naf)\n",
      "MINC MRI: can read (mnc)\n",
      "QuickTime: can read, can write, can write multiple (mov)\n",
      "Minolta MRW: can read (mrw)\n",
      "TillVision: can read (vws, pst, inf)\n",
      "ARF: can read (arf)\n",
      "Cellomics C01: can read (c01, dib)\n",
      "LI-FLIM: can read (fli)\n",
      "Truevision Targa: can read (tga)\n",
      "Oxford Instruments: can read (top)\n",
      "VG SAM: can read (dti)\n",
      "Hamamatsu HIS: can read (his)\n",
      "WA Technology TOP: can read (wat)\n",
      "Seiko: can read (xqd, xqf)\n",
      "TopoMetrix: can read (tfr, ffr, zfr, zfp, 2fl)\n",
      "UBM: can read (pr3)\n",
      "Quesant AFM: can read (afm)\n",
      "Bio-Rad GEL: can read (1sc)\n",
      "RHK Technologies: can read (sm2, sm3)\n",
      "Molecular Imaging: can read (stp)\n",
      "CellWorx: can read (pnl, htd, log)\n",
      "MetaXpress TIFF: can read (htd, tif)\n",
      "ECAT7: can read (v)\n",
      "Varian FDF: can read (fdf)\n",
      "AIM: can read (aim)\n",
      "InCell 3000: can read (frm)\n",
      "SPIDER: can read (spi)\n",
      "Volocity Library: can read (mvd2, aisf, aiix, dat, atsf)\n",
      "IMAGIC: can read (hed, img)\n",
      "Hamamatsu VMS: can read (vms)\n",
      "CellSens VSI: can read (vsi, ets)\n",
      "INR: can read (inr)\n",
      "Kodak Molecular Imaging: can read (bip)\n",
      "Volocity Library Clipping: can read (acff)\n",
      "Zeiss CZI: can read (czi)\n",
      "Andor SIF: can read (sif)\n",
      "Hamamatsu NDPIS: can read (ndpis)\n",
      "POV-Ray: can read (df3)\n",
      "IMOD: can read (mod)\n",
      "Simulated data: can read (fake)\n",
      "Aperio AFI: can read (afi)\n",
      "Lavision Imspector: can read (msr)\n",
      "Bio-Rad SCN: can read (scn)\n",
      "Zeiss LMS: can read (lms)\n",
      "PicoQuant Bin: can read (bin)\n",
      "FlowSight: can read (cif)\n",
      "Perkin-Elmer Nuance IM3: can read (im3)\n",
      "I2I: can read (i2i)\n",
      "Princeton Instruments SPE: can read (spe)\n",
      "Olympus OIR: can read (oir)\n",
      "KLB: can read (klb)\n",
      "MicroCT: can read (vff)\n",
      "Leica Object Format: can read (lof)\n",
      "Extended leica file: can read (xlef)\n",
      "Olympus .omp2info: can read (omp2info)\n",
      "JEOL: can read (dat, img, par)\n",
      "NIfTI: can read (nii, img, hdr, nii.gz)\n",
      "Analyze 7.5: can read (img, hdr)\n",
      "Olympus APL: can read (apl, tnb, mtb, tif)\n",
      "NRRD: can read (nrrd, nhdr)\n",
      "Image Cytometry Standard: can read, can write, can write multiple (ics, ids)\n",
      "PerkinElmer: can read (ano, cfg, csv, htm, rec, tim, zpo, tif)\n",
      "Amira: can read (am, amiramesh, grey, hx, labels)\n",
      "Olympus ScanR: can read (dat, xml, tif)\n",
      "BD Pathway: can read (exp, tif)\n",
      "Unisoku STM: can read (hdr, dat)\n",
      "Perkin Elmer Densitometer: can read (hdr, img)\n",
      "Fuji LAS 3000: can read (img, inf)\n",
      "PerkinElmer Operetta: can read (tif, tiff, xml)\n",
      "Inveon: can read (hdr)\n",
      "CellVoyager: can read (tif, xml)\n",
      "PerkinElmer Columbus: can read (xml)\n",
      "Yokogawa CV7000: can read (wpi)\n",
      "Bio-Rad PIC: can read (pic, xml, raw)\n",
      "Olympus FV1000: can read (oib, oif, pty, lut)\n",
      "Zeiss Vision Image (ZVI): can read (zvi)\n",
      "Image-Pro Workspace: can read (ipw)\n",
      "JPEG-2000: can read, can write (jp2, j2k, jpf)\n",
      "JPX: can read (jpx)\n",
      "Nikon ND2: can read (nd2, jp2)\n",
      "Compix Simple-PCI: can read (cxd)\n",
      "Bitplane Imaris 5.5 (HDF): can read (ims)\n",
      "CellH5 (HDF): can read (ch5)\n",
      "Veeco: can read (hdf)\n",
      "Tecan Spark Cyto: can read (db)\n",
      "Zeiss Laser-Scanning Microscopy: can read (lsm, mdb)\n",
      "Image-Pro Sequence: can read (seq, ips)\n",
      "Amersham Biosciences GEL: can read (gel)\n",
      "Bitplane Imaris 3 (TIFF): can read (ims)\n",
      "Evotec Flex: can read (flex, mea, res)\n",
      "Aperio SVS: can read (svs)\n",
      "Imacon: can read (fff)\n",
      "LEO: can read (sxm, tif, tiff)\n",
      "JPK Instruments: can read (jpk)\n",
      "Hamamatsu NDPI: can read (ndpi)\n",
      "PCO-RAW: can read (pcoraw, rec)\n",
      "Ventana .bif: can read (bif)\n",
      "OME-TIFF: can read, can write, can write multiple (ome.tiff, ome.tif, ome.tf2, ome.tf8, ome.btf, companion.ome)\n",
      "Pyramid TIFF: can read (tif, tiff)\n",
      "MIAS: can read (tif, tiff, txt)\n",
      "Leica TCS TIFF: can read (tif, tiff, xml)\n",
      "Leica: can read (lei, tif, tiff, raw)\n",
      "Nikon NEF: can read (nef, tif, tiff)\n",
      "Olympus Fluoview/ABD TIFF: can read (tif, tiff)\n",
      "Prairie TIFF: can read (tif, tiff, cfg, env, xml)\n",
      "Metamorph STK: can read (stk, nd, scan, tif, tiff)\n",
      "Micro-Manager: can read (tif, tiff, txt, xml)\n",
      "Improvision TIFF: can read (tif, tiff)\n",
      "Metamorph TIFF: can read (tif, tiff)\n",
      "Nikon TIFF: can read (tif, tiff)\n",
      "Mikroscan TIFF: can read (tif, tiff)\n",
      "Adobe Photoshop TIFF: can read (tif, tiff)\n",
      "FEI TIFF: can read (tif, tiff)\n",
      "SimplePCI TIFF: can read (tif, tiff)\n",
      "Nikon Elements TIFF: can read (tif, tiff)\n",
      "Trestle: can read (tif)\n",
      "Olympus SIS TIFF: can read (tif, tiff)\n",
      "DNG: can read (cr2, crw, jpg, thm, wav, tif, tiff)\n",
      "Zeiss AxioVision TIFF: can read (tif, xml)\n",
      "Leica SCN: can read (scn)\n",
      "PerkinElmer Vectra/QPTIFF: can read (tiff, tif, qptiff)\n",
      "Slidebook TIFF: can read (tif, tiff)\n",
      "Ionpath MIBI: can read (tif, tiff)\n",
      "DICOM: can read, can write, can write multiple (dic, dcm, dicom, jp2, j2ki, j2kr, raw, ima)\n",
      "Hitachi: can read (txt)\n",
      "Tagged Image File Format: can read, can write, can write multiple (tif, tiff, tf2, tf8, btf)\n",
      "Text: can read (txt, csv)\n",
      "Burleigh: can read (img)\n",
      "Openlab LIFF: can read (liff)\n",
      "SM Camera: can read ()\n",
      "SBIG: can read ()\n",
      "NOAA-HRD Gridded Data Format: can read ()\n",
      "Bruker: can read ()\n",
      "Canon RAW: can read (cr2, crw, jpg, thm, wav)\n",
      "OBF: can read (obf, msr)\n",
      "BDV: can read (xml, h5)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "formatlist"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e7d19ab-ca5a-4fe6-a960-0a43a1fd3c73",
   "metadata": {},
   "source": [
    "We've got several files from aa few of these types that we can look at."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "8d0f09e9-22f0-490d-a8e5-2c7e72af4396",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking file format [Simulated data]\n",
      "Initializing reader\n",
      "FakeReader initializing data/a.fake\n",
      "Initialization took 0.157s\n",
      "\n",
      "Reading core metadata\n",
      "filename = data/a.fake\n",
      "Used files = [/mnt/gluster2/scratch/josh_openmicroscopy/session1/data/a.fake]\n",
      "Series count = 1\n",
      "Series #0 :\n",
      "\tImage count = 1\n",
      "\tRGB = false (1) \n",
      "\tInterleaved = false\n",
      "\tIndexed = false (true color)\n",
      "\tWidth = 512\n",
      "\tHeight = 512\n",
      "\tSizeZ = 1\n",
      "\tSizeT = 1\n",
      "\tSizeC = 1\n",
      "\tTile size = 512 x 512\n",
      "\tThumbnail size = 128 x 128\n",
      "\tEndianness = intel (little)\n",
      "\tDimension order = XYZCT (certain)\n",
      "\tPixel type = uint8\n",
      "\tValid bits per pixel = 8\n",
      "\tMetadata complete = true\n",
      "\tThumbnail series = false\n",
      "\t-----\n",
      "\tPlane #0 <=> Z 0, C 0, T 0\n",
      "\n",
      "\n",
      "Reading global metadata\n",
      "\n",
      "Reading metadata\n"
     ]
    }
   ],
   "source": [
    "showinf -nopix data/a.fake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "fd520130-b8d6-4612-8535-26f91caddf4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
      "<OME xmlns=\"http://www.openmicroscopy.org/Schemas/OME/2016-06\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd\">\n",
      "   <Image ID=\"Image:0\" Name=\"a\">\n",
      "      <Pixels BigEndian=\"false\" DimensionOrder=\"XYZCT\" ID=\"Pixels:0\" Interleaved=\"false\" SignificantBits=\"8\" SizeC=\"1\" SizeT=\"1\" SizeX=\"512\" SizeY=\"512\" SizeZ=\"1\" Type=\"uint8\">\n",
      "         <Channel ID=\"Channel:0:0\" SamplesPerPixel=\"1\">\n",
      "            <LightPath/>\n",
      "         </Channel>\n",
      "         <MetadataOnly/>\n",
      "      </Pixels>\n",
      "   </Image>\n",
      "</OME>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "showinf -nopix -omexml-only data/a.fake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "a8b579ae-54cb-46af-bfb5-f505c303432d",
   "metadata": {},
   "outputs": [],
   "source": [
    "test -e a.ome.tiff && rm -rf a.ome.tiff || echo Not yet created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "d698eaae-a24d-4bbb-85cb-df8b9895bce9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/a.fake\n",
      "FakeReader initializing data/a.fake\n",
      "[Simulated data] -> a.ome.tiff [OME-TIFF]\n",
      "\tConverted 1/1 planes (100%)\n",
      "[done]\n",
      "3.046s elapsed (13.0+1573.0ms per plane, 1269ms overhead)\n"
     ]
    }
   ],
   "source": [
    "bfconvert data/a.fake a.ome.tiff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "0b9c852a-3ff4-4b73-bcc1-e87ce9ee79a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<?xml version=\"1.0\" encoding=\"UTF-8\"?><!-- Warning: this comment is an OME-XML metadata block, which contains crucial dimensional parameters and other important metadata. Please edit cautiously (if at all), and back up the original data before doing so. For more information, see the OME-TIFF web site: https://docs.openmicroscopy.org/latest/ome-model/ome-tiff/. --><OME xmlns=\"http://www.openmicroscopy.org/Schemas/OME/2016-06\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" Creator=\"OME Bio-Formats 7.0.0\" UUID=\"urn:uuid:1c096a89-e1d7-4b61-9666-e9625a4c1b07\" xsi:schemaLocation=\"http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd\"><Image ID=\"Image:0\" Name=\"a\"><Pixels BigEndian=\"false\" DimensionOrder=\"XYZCT\" ID=\"Pixels:0\" Interleaved=\"false\" SignificantBits=\"8\" SizeC=\"1\" SizeT=\"1\" SizeX=\"512\" SizeY=\"512\" SizeZ=\"1\" Type=\"uint8\"><Channel ID=\"Channel:0:0\" SamplesPerPixel=\"1\"><LightPath/></Channel><TiffData FirstC=\"0\" FirstT=\"0\" FirstZ=\"0\" IFD=\"0\" PlaneCount=\"1\"><UUID FileName=\"a.ome.tiff\">urn:uuid:1c096a89-e1d7-4b61-9666-e9625a4c1b07</UUID></TiffData></Pixels></Image></OME>\n"
     ]
    }
   ],
   "source": [
    "tiffcomment a.ome.tiff"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17e83c4a-68c1-4467-a65f-d58d2ae0839a",
   "metadata": {},
   "source": [
    "### Pattern files (many files)\n",
    "\n",
    "Pattern files are a Bio-Formats functionality that lets you write short patterns like below which group together other files along different dimensions. You can read more under https://bio-formats.readthedocs.io/en/stable/formats/pattern-file.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "f1eda27e-3768-4e68-aa2d-41e380a5b05e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01_POS218_Z<00>_T<00>_<D,F,R>.TIF\n"
     ]
    }
   ],
   "source": [
    "cat data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/POS218.pattern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "79671812-717c-43ba-b2ff-2d827a04288b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking file format [File pattern]\n",
      "Initializing reader\n",
      "FilePatternReader initializing data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/POS218.pattern\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "Reading IFDs\n",
      "Populating metadata\n",
      "Checking comment style\n",
      "Populating OME metadata\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "Reading IFDs\n",
      "Populating metadata\n",
      "Checking comment style\n",
      "Populating OME metadata\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "Initialization took 2.202s\n",
      "\n",
      "Reading core metadata\n",
      "filename = /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/POS218.pattern\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_F.TIF\n",
      "Reading IFDs\n",
      "Populating metadata\n",
      "Checking comment style\n",
      "Populating OME metadata\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_R.TIF\n",
      "Reading IFDs\n",
      "Populating metadata\n",
      "Checking comment style\n",
      "Populating OME metadata\n",
      "Used files:\n",
      "\t/mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/POS218.pattern\n",
      "\t/mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "\t/mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_F.TIF\n",
      "\t/mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_R.TIF\n",
      "Series count = 1\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "Reading IFDs\n",
      "Populating metadata\n",
      "Checking comment style\n",
      "Populating OME metadata\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "TiffDelegateReader initializing /mnt/gluster2/scratch/josh_openmicroscopy/session1/data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/01_POS218_Z00_T00_D.TIF\n",
      "Series #0 :\n",
      "\tImage count = 3\n",
      "\tRGB = false (1) \n",
      "\tInterleaved = false\n",
      "\tIndexed = true (false color, 8-bit LUT: 3 x 256)\n",
      "\tWidth = 1000\n",
      "\tHeight = 1006\n",
      "\tSizeZ = 1\n",
      "\tSizeT = 1\n",
      "\tSizeC = 3\n",
      "\tTile size = 1000 x 8\n",
      "\tThumbnail size = 127 x 128\n",
      "\tEndianness = intel (little)\n",
      "\tDimension order = XYCZT (uncertain)\n",
      "\tPixel type = uint8\n",
      "\tValid bits per pixel = 8\n",
      "\tMetadata complete = true\n",
      "\tThumbnail series = false\n",
      "\t-----\n",
      "\tPlane #0 <=> Z 0, C 0, T 0\n",
      "\tPlane #1 <=> Z 0, C 1, T 0\n",
      "\tPlane #2 <=> Z 0, C 2, T 0\n",
      "\n",
      "\n",
      "Reading global metadata\n",
      "BitsPerSample: 8\n",
      "Compression: Uncompressed\n",
      "ImageLength: 1006\n",
      "ImageWidth: 1000\n",
      "MetaDataPhotometricInterpretation: Monochrome\n",
      "MetaMorph: no\n",
      "NumberOfChannels: 3\n",
      "Orientation: 1st row - top; 1st column - left\n",
      "PhotometricInterpretation: Palette\n",
      "PlanarConfiguration: Chunky\n",
      "ResolutionUnit: Inch\n",
      "SamplesPerPixel: 1\n",
      "XResolution: 150.0\n",
      "YResolution: 150.0\n",
      "\n",
      "Reading metadata\n"
     ]
    }
   ],
   "source": [
    "showinf -nopix data/TIFF/cellprofiler/fruit-fly-cells/POS218.pattern/POS218.pattern"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2db466d-519a-4386-b002-660ee7797357",
   "metadata": {},
   "source": [
    "### Series (many images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "81cce918-9345-4a06-9070-8e361fca1677",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing reader\n",
      "FakeReader initializing a&series=2.fake\n",
      "Initialization took 0.166s\n",
      "\n",
      "Reading core metadata\n",
      "filename = a&series=2.fake\n",
      "Used files = [/mnt/gluster2/scratch/josh_openmicroscopy/session1/a&series=2.fake]\n",
      "Series count = 2\n",
      "Series #0 :\n",
      "\tImage count = 1\n",
      "\tRGB = false (1) \n",
      "\tInterleaved = false\n",
      "\tIndexed = false (true color)\n",
      "\tWidth = 512\n",
      "\tHeight = 512\n",
      "\tSizeZ = 1\n",
      "\tSizeT = 1\n",
      "\tSizeC = 1\n",
      "\tTile size = 512 x 512\n",
      "\tThumbnail size = 128 x 128\n",
      "\tEndianness = intel (little)\n",
      "\tDimension order = XYZCT (certain)\n",
      "\tPixel type = uint8\n",
      "\tValid bits per pixel = 8\n",
      "\tMetadata complete = true\n",
      "\tThumbnail series = false\n",
      "\t-----\n",
      "\tPlane #0 <=> Z 0, C 0, T 0\n",
      "\n",
      "Series #1 :\n",
      "\tImage count = 1\n",
      "\tRGB = false (1) \n",
      "\tInterleaved = false\n",
      "\tIndexed = false (true color)\n",
      "\tWidth = 512\n",
      "\tHeight = 512\n",
      "\tSizeZ = 1\n",
      "\tSizeT = 1\n",
      "\tSizeC = 1\n",
      "\tTile size = 512 x 512\n",
      "\tThumbnail size = 128 x 128\n",
      "\tEndianness = intel (little)\n",
      "\tDimension order = XYZCT (certain)\n",
      "\tPixel type = uint8\n",
      "\tValid bits per pixel = 8\n",
      "\tMetadata complete = true\n",
      "\tThumbnail series = false\n",
      "\t-----\n",
      "\tPlane #0 <=> Z 0, C 0, T 0\n",
      "\n",
      "\n",
      "Reading global metadata\n",
      "\n",
      "Reading series #0 metadata\n"
     ]
    }
   ],
   "source": [
    "showinf -nopix 'a&series=2.fake'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90c393f6-98bf-412f-954f-b354349d5c1b",
   "metadata": {},
   "source": [
    "### In bioimaging, anything is possible\n",
    "\n",
    "You can see a graphical explanation of the \"many-to-many\" problem in https://downloads.openmicroscopy.org/presentations/2013/fs-workshop-paris/#/7/1\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"images/many-to-many/one-to-one.png\"/>\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"images/many-to-many/one-to-many.png\"/>\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"images/many-to-many/many-to-many.png\"/>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97256625-31f7-4d88-8faa-12b8d11df986",
   "metadata": {},
   "source": [
    "## Excercise 2:\n",
    "### Use the tools above to find inspect the data you chose, and convert it to OME-TIFF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "3d2aa171-0191-428a-8dc0-06a5fea0ccb2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "outputs": [],
   "source": [
    "## Do something here or on the command-line. If you are having issues, ask."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1a3d89a-e7f3-4180-b506-69865c12db9b",
   "metadata": {},
   "source": [
    "## License\n",
    "Copyright (C) 2023 German BioImaging. All Rights Reserved.\n",
    "This program is free software; you can redistribute it and/or modify it\n",
    "under the terms of the GNU General Public License as published by the\n",
    "Free Software Foundation; either version 2 of the License, or\n",
    "(at your option) any later version.\n",
    "This program is distributed in the hope that it will be useful, but\n",
    "WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY\n",
    "or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for\n",
    "more details. You should have received a copy of the GNU General\n",
    "Public License along with this program; if not, write to the\n",
    "Free Software Foundation,\n",
    "Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}