Skip to content

III. Quickstart for analysts scripting

Jump to bottom
carlosuc3m edited this page Sep 13, 2023 · 2 revisions

Quickstart for analysts/scripting

This section (~15 min read) explains how to use JDLL in a Jython script. Jython is an implementation of Python in Java, thus it allows calling Java methods and classes in a Pythonic way. With Jython, Java methods and classes can be called in a script as Python methods.

Scripting is powerfull and usefull because it allows creating processing routines in a simple manner that otherwise would require a full software for them. This is the reason why many software applications such as Icy, Fiji or Napari have a script editor. Scripts can perform complex tasks reducing the need of developing specific plugins in each platform. Scripts are also mostly compatible with every software application that supports the language they are writen in.

The examples for JDLL use Jython because it supports Java and uses Python syntax. Python is one of the mostwidely used programming languages and also one of the easiers to learn. Python has a big community and many Open Source libraries that can be used in Jython too. However, note that Jython at the moment only supports Python2. The support for Python 3 is still in development. A good tutorial on how to create Jython scripts can be found here.

In addition, Jython scripts can be then used in softwares such as Icy or Fiji to create Image Processing tasks together with the tools they have available.

The following subsections provide essential instructions to use JDLL in Jyhton scripts:

Note that the JDLL Wiki contains an extensive documentation of the JDLL API. The API methods can be used with Jython. Consider going over the API to get a more in-depth idea of the scripting possibilities of JDLL.

0. Setting up JDLL

Download the dependency and include it in your project. Depending on the application, dependencies are installed differently. For example, in Icy, click on the Plugins tab, then Setup>Online plugin. In the window that opens look for the name JDLL within the plugin list, click on it and then click on Install on the middle right of the window.

In Fiji, the JDLL library and its dependencies are shipped from the deepImageJ update site. In order to install JDLL in Fiji, add the deepImageJ update site to the Fiji updater and update. The URL for the deepImageJ update site is: https://sites.imagej.net/DeepImageJ/

Once installed, JDLL should be ready to be used.

If the software application JDLL wants to be used in does not support automatic installation of libraries, download JDLL and its dependencies and locate them in the corresponding directory. The download links are:

1. Downloading a model (optional)

If a model from the supported by JDLL is already available you can skip this step. Note that for Tensorflow the models need to be saved in the SavedModel format.

If no model is available, a good starting point is downloading any of the models of the Bioimage.io respository. The download can be done manually or using JDLL. Bioimag.io is seamlessly integrated into JDLL, offering multiple methods to effortlessly mange and use its models. An example Jython script to download a Bioimage.io model can be found here.

Here is an emaple of how JDLL can be used to download any Bioimage.io model, in this case the B. Sutilist bacteria segmentation - Widefield microscopy - 2D UNet model.

from io.bioimage.modelrunner.bioimageio import BioimageioRepo

# Name of the model of interest, note that each model is unique. The names are case sensitive.
modelName = "B. Sutilist bacteria segmentation - Widefield microscopy - 2D UNet"
# Directory where the model folder will be downloaded
modelsDir = "/path/to/wanted/model/directory"

# First create an instance of the Bioimage.io repository
br = BioimageioRepo.connect()
try:
	br.downloadByName(modelName, modelsDir)
	print("Great success!")
except Exception as e:
	# If the download is interrumpted or any of the model files cannot be downloaded
	# and exception will be thrown
	print("Error downloading the model: ", str(e))

Output:

Great success!

More information on how to download Bioimage.io models can be found here. Note that all the methods decribed in JAva can be used in a pythonic way with Jytho. The static ones do not need the instantiation of a class and the non-static ones do require it.

2. Installing DL engines

JDLL is installed empty. Several models might require different Deep Learning framework versions, each of them consuming considerable amounts of disk space. In order to make JDLL setup light and fast JDLL is installed without default DL engines. The user can then get the Dl engines that they want depending on their needs.

JDLL provides the needed methods to install the wanted engines in an easy manner. Following the above example, find below some code that can be used to install a DL engine. As it can be observed the model that was downloaded supports Tensorflow 2 and Keras weights. Keras is not supported so in order to load and run the model, Tensorflow weights need to be installed.

from io.bioimage.modelrunner.engine.installation import EngineInstall

framework = "tensorflow"
version = "2.11.0"
cpu = True
gpu = True

enginesDir = "/path/to/wanted/engines/dir"
installed = EngineInstall.installEngineWithArgsInDir(framework, version, cpu, gpu, enginesDir)
if (installed):
	print("Great success!")
else:
	print("Error installing")

Output:

Great success!

As previously mentioned, JDLL integrates deeply with Bioimage.io models. An easier way to install the engines needed for Bioimage.io models is shown in the code below.

In the example it is shown how simply providing the name of the model of interest, JDLL will know which engines to install.

from io.bioimage.modelrunner.engine.installation import EngineInstall

modelName = "B. Sutilist bacteria segmentation - Widefield microscopy - 2D UNet"
modelsDir = "/path/to/wanted/models/dir"
installed =  EngineInstall.installEnginesForModelByNameinDir(modelName, modelsDir)
if (installed):
	print("Great success!")
else:
	print("Error installing")

Output:

Great success!

The Wiki covers extensively engine installation (here and here). In addtion JDLL also includes methods to manage the engines and know: the information about each engine, which engines are supported and which engines have already been installed Again, not that the methods explained in the Wiki can easily be used in Jython scripts.

In addition, a more detailed Jython script with an example on how to install an engine can be found here.

3. Creating the tensors

Once the model and the engine are already installed it is the moment to start the process of running the model on the tensors. In this section, creation of the tensors will be explained.

JDLL tensors are agnostic to the DL framework to be used, they are always creted in the same way. JDLL manages internally the conversion of the agnostic tensor into the framework specific tensor once the model is going to be run. The unified method of creating tensors facilitates the integration of every supported DL framework into any software.

JDLL tensors use ImgLib2 to store the tensor information. In practice, JDLL tensors are just wrappers of ImgLib2 RandomAccessibleIntervals that contain all the data needed to convert them back and forth into the framework specific tensors.

The example below will show how to create the input and output tensors required to run the example model. As per its rdf.yaml file, the model has one input named input_1, with bxyc axes (explanation here) and a required shape of[1, 512, 512, 1]. The ouptut of the model is named conv2d_19 with with bxyc axes and fixed shape [1, 512, 512, 3].

from io.bioimage.modelrunner.tensor import Tensor

from net.imglib2.img.array import ArrayImgFactory
from net.imglib2.type.numeric.real import FloatType

imgFactory = ArrayImgFactory( FloatType() )
img1 = imgFactory.create( [1, 512, 512, 1] )
# Create the input tensor with the nameand axes given by the rdf.yaml file
# and add it to the list of input tensors
inpTensor = Tensor.build("input_1", "bxyc", img1)

# Ouput tensors can be created empty, if the output shape is not known.
# Note that this method does not preallocate memory for the output tensor
outputEmptyTensor = Tensor.buildEmptyTensor("conv2d_19", "bxyc")

# Or ouptut tensors can also be built blank, to pre-allocate memory
# if the shape and data type are known.
outputBlankTensor = Tensor.buildBlankTensor("conv2d_19",
			"bxyc", [1, 512, 512, 3], FloatType())

More information about tensors can be found in the JDLL wiki.

Another option is to create the tensors directly from images are open on the software. The next example shows how an ImagePlus opened by Fiji can be converted into a tensor.

import os
from io.bioimage.modelrunner.tensor import Tensor
from ij import IJ
from net.imglib2.img.display.imagej import ImageJFunctions
from net.imglib2.view import Views

# Path to the model whose sample image is going to be displayed
model_name = "B. Sutilist bacteria segmentation - Widefield microscopy - 2D UNet"
models_dir = "/path/to/wanted/models/dir"
model_of_interest_path = os.path.join(models_dir, model_name)

#Open the image and show it
imp = IJ.openImage(os.path.join(model_of_interest_path, "sample_input_0.tif"))
imp.show()

# Convert the image into a float32 ImgLib2 image.
# Note that as a 2D image the dimensions are just "xy"
wrapImg = ImageJFunctions.convertFloat(imp)
# Convert to the required axes order "bxyc"
# Permute from "xy" to "yx"
wrapImg = Views.permute(wrapImg, 0, 1)
# Add one dimension to "yxb", from (512, 512) to (512, 512, 1)
wrapImg = Views.addDimension(wrapImg, 0, 0
# Permute from "yxb" and (512, 512, 1) to "bxy" and (1, 512, 512)
wrapImg = Views.permute(wrapImg, 0, 2)
# Add one dimension to get "bxyc", from (1, 512, 512) to (1, 512, 512, 1)
wrapImg = Views.addDimension(wrapImg, 0, 0)

# Build the corresponding tensor
inputTensor = Tensor.build("input_1", "bxyc", wrapImg)

In order to get the orginal image back from the tensor after all the permutations and dimensions added:

from net.imglib2.img.display.imagej import ImageJFunctions
from net.imglib2.view import Views

# Convert from "bxyc" (1, 512, 512, 1) into "xy" (512, 512)
wrapImg = Views.dropSingletonDimensions(wrapImg)
ImageJFunctions.show(wrapImg)

A complete example that contains tensor creation from Fiji ImagePlus images can be found here.

4. Loading the model

Before making inference with a model, it needs to be loaded. Similar to tensor creation JDLL provides an unified way to load models from any DL framework.

Loading a model implies first defining which engine is going to be used for the model and then loading the model. The engine is defined as an instance of the class io.bioimage.modelrunner.engine.EngineInfo. The engine used to load a model can be either the exact same version wanted, or a compatible one. Using the exact same version guarantees that the model loading and inference are going to be smooth but implies that many more engines will have to be installed.

An example of defining the EngineInfo instance needed to load a model is shown below. The engine required in this code is required to be the exact engine wanted.

Note that enginesDir is the directory where the wanted engines have been installed. Click here and look at the example in the redirected section.

from io.bioimage.modelrunner.engine import EngineInfo

framework = "tensorflow"
version = "2.11.0"
cpu = True
gpu = True
enginesDir = "/path/to/wanted/engines/dir"

enigneInfoExact = EngineInfo.defineDLEngine(framework, version, cpu, gpu, enginesDir)

In order to require a compatible engine, not the exact one:

from io.bioimage.modelrunner.engine import EngineInfo

framework = "tensorflow"
version = "2.11.0"
cpu = True
gpu = True
enginesDir = "/path/to/wanted/engines/dir"

enigneInfoCompat = EngineInfo.defineCompatibleDLEngine(framework, version, cpu, gpu, enginesDir)

The developer acknowledges that the class io.bioimage.modelrunner.engine.EngineInfo can be difficult to understand. This is why the wiki contains a detailed sections trying to explain it in an understandable manner with several examples.

Once the EngineInfo instance has been created, loading the model is easy. The only parameters needed now are the path to the model folder and the path to the source file. The model folder is the folder that contains the .pb file in Tensorflow, the .pt file in Pytorch and the .onnx file in Onnx. The source file is not needed for Tensorflow, is the path to the .pt file in Pytorch and the path to the .onnx file in Onnx.

Then with the arguments modelFolder, modelSource and the previously created enigneInfoExact or enigneInfoCompat an instance of io.bioimage.modelrunner.model.Model can be created. Tha object then can be loaded and run inference.

from io.bioimage.modelrunner.model import Model

# Path to the example model folder
modelFolder = "path/to/models/dir/B. Sutilist bacteria segmentation - Widefield microscopy - 2D UNet"
modelSource = None # Not needed in Tensorflow

model = Model.create(modelFolder, modelSource, enigneInfoCompat)
model.loadModel()

print("Great sucess!")

Output:

Great sucess!

JDLL tight integration with Bioimage.io models makes loading them easier. To load a Bioimage.io model it is not necessary to create the EngineInfo object.

For Bioimage.io models, loading is reduced to the next example:

from io.bioimage.modelrunner.model import Model

# Path to the Bioimage.io model folder
modelFolder = "path/to/models/dir/B. Sutilist bacteria segmentation - Widefield microscopy - 2D UNet"
enginesDir = "/path/to/wanted/engines/dir"

bioimageioModel = Model.createBioimageioModel(modelFolder, enginesDir)
bioimageioModel.load()

print("Great sucess!")

Output:

Great sucess!

More information about loading models and models in general can be found in this Wiki page.

In addition, a complete example showing how to load a Bioimage.io model can be found here.

5. Running the model

Once the model has been loaded and the input and output tensors have been created. Running the model is simple. The input tensors should be added to a List<?> in the same order the model expects. Same for the ouptuts in another List.

inputList = []
outputputList = []

inputList.append(inputTensor)
outputputList.append(outputEmptyTensor)
print("Ouptut tensor is empty: ",  outputEmptyTensor.isEmpty())

model.runModel(inputList, outputputList)
print("Ouptut tensor after inference is empty: ", outputEmptyTensor.isEmpty())

Output:

Ouptut tensor is empty: true
Ouptut tensor after inference is empty: false

6. Closing the model and the tensors

Models and tensors need to be closed to be released and free the memory that they were using

model.close()
inputTensor.close()
outputBlankTensor.close()
outputEmptyTensor.close()

Scripting examples

Clone this wiki locally