Skip to content

Commit

Permalink
Explore Neuroglancer URIs (#7416)
Browse files Browse the repository at this point in the history
  • Loading branch information
frcroth authored Nov 8, 2023
1 parent c346d04 commit 1ba905a
Show file tree
Hide file tree
Showing 4 changed files with 108 additions and 4 deletions.
3 changes: 2 additions & 1 deletion CHANGELOG.unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
### Added
- Zarr datasets can now be directly uploaded to WEBKNOSSOS. [#7397](https://github.com/scalableminds/webknossos/pull/7397)
- Added support for reading uint24 rgb layers in datasets with zarr2/zarr3/n5/neuroglancerPrecomputed format, as used for voxelytics predictions. [#7413](https://github.com/scalableminds/webknossos/pull/7413)
Added a filter to the Task List->Stats column to quickly filter for tasks with "Prending", "In-Progress" or "Finished" instances. [#7430](https://github.com/scalableminds/webknossos/pull/7430)
- Adding a remote dataset can now be done by providing a Neuroglancer URI. [#7416](https://github.com/scalableminds/webknossos/pull/7416)
- Added a filter to the Task List->Stats column to quickly filter for tasks with "Prending", "In-Progress" or "Finished" instances. [#7430](https://github.com/scalableminds/webknossos/pull/7430)

### Changed
- An appropriate error is returned when requesting an API version that is higher that the current version. [#7424](https://github.com/scalableminds/webknossos/pull/7424)
Expand Down
4 changes: 3 additions & 1 deletion app/models/dataset/explore/ExploreRemoteLayerService.scala
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import com.scalableminds.webknossos.datastore.explore.{
ExploreLayerService,
N5ArrayExplorer,
N5MultiscalesExplorer,
NeuroglancerUriExplorer,
NgffExplorer,
PrecomputedExplorer,
RemoteLayerExplorer,
Expand Down Expand Up @@ -130,7 +131,8 @@ class ExploreRemoteLayerService @Inject()(credentialService: CredentialService,
new N5ArrayExplorer,
new N5MultiscalesExplorer,
new PrecomputedExplorer,
new Zarr3ArrayExplorer
new Zarr3ArrayExplorer,
new NeuroglancerUriExplorer(dataVaultService, exploreLayerService, ec)
)
)
} yield layersWithVoxelSizes
Expand Down
7 changes: 5 additions & 2 deletions docs/datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,13 @@ With other converters, you may need to add the layers separately.
- a URL or domain/collection identifier to locate the dataset on the remote service (supported protocols are HTTPS, Amazon S3 and Google Cloud Storage).
- authentication credentials for accessing the resources on the remote service (optional)
4. Click the *Add Layer* button
5. WEBKNOSSOS will automatically try to infer as many dataset properties (voxel size, bounding box, etc) as possible and preview a [WEBKNOSSOS `datasource` configuration](./data_formats.md#dataset-metadata-specification) for your to review.
5. WEBKNOSSOS will automatically try to infer as many dataset properties (voxel size, bounding box, etc.) as possible and preview a [WEBKNOSSOS `datasource` configuration](./data_formats.md#dataset-metadata-specification) for your to review.
Consider setting the dataset `name` property and double-check all other properties for correctness.
6. Click `Import` to finish

WEBKNOSSOS can also import URIs from [Neuroglancer](https://github.com/google/neuroglancer). When viewing a dataset on a Neuroglancer instance in the browser,
copy the URI from the address bar and paste it into the *Add Remote Dataset* tab to view the dataset in WEBKNOSSOS.

WEBKNOSSOS will NOT download or copy datasets in full from these third-party data providers.
Rather, any data viewed in WEBKNOSSOS will be streamed read-only from the remote source.
These remote datasets will not count against your storage quota on WEBKNOSSOS.
Expand All @@ -82,7 +85,7 @@ On self-hosted instances, large datasets can be efficiently imported by placing
* Go to the [dataset view on the dashboard](./dashboard.md)
* Use the refresh button on the dashboard or wait for WEBKNOSSOS to detect the dataset (up to 10min)

Typically WEBKNOSSOS can infer all the required metadata for a dataset automatically and import datasets automatically on refresh. In some cases, you will need to manually import a dataset and provide more information:
Typically, WEBKNOSSOS can infer all the required metadata for a dataset automatically and import datasets automatically on refresh. In some cases, you will need to manually import a dataset and provide more information:

* On the dashboard, click *Import* for your new dataset
* Provide the requested properties, such as *scale*. It is also recommended to set the *largestSegmentId*. See the section on [configuring datasets](#configuring-datasets) below for more detailed explanations of these parameters.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
package com.scalableminds.webknossos.datastore.explore

import com.scalableminds.util.geometry.{Vec3Double, Vec3Int}
import com.scalableminds.util.tools.Fox
import com.scalableminds.webknossos.datastore.dataformats.n5.{N5DataLayer, N5SegmentationLayer}
import com.scalableminds.webknossos.datastore.dataformats.precomputed.{
PrecomputedDataLayer,
PrecomputedSegmentationLayer
}
import com.scalableminds.webknossos.datastore.dataformats.zarr.{ZarrDataLayer, ZarrSegmentationLayer}
import com.scalableminds.webknossos.datastore.dataformats.zarr3.{Zarr3DataLayer, Zarr3SegmentationLayer}
import com.scalableminds.webknossos.datastore.datavault.VaultPath
import com.scalableminds.webknossos.datastore.models.datasource.LayerViewConfiguration.LayerViewConfiguration
import com.scalableminds.webknossos.datastore.models.datasource.{DataLayer, LayerViewConfiguration}
import com.scalableminds.webknossos.datastore.storage.{DataVaultService, RemoteSourceDescriptor}
import play.api.libs.json._

import java.net.URI
import javax.inject.Inject
import scala.concurrent.ExecutionContext

class NeuroglancerUriExplorer @Inject()(dataVaultService: DataVaultService,
exploreLayerService: ExploreLayerService,
implicit val ec: ExecutionContext)
extends RemoteLayerExplorer {
override def name: String = "Neuroglancer URI Explorer"

override def explore(remotePath: VaultPath, credentialId: Option[String]): Fox[List[(DataLayer, Vec3Double)]] =
for {
_ <- Fox.successful(())
spec <- Json
.parse(remotePath.toUri.getFragment.drop(1))
.validate[JsObject]
.toFox ?~> "Did not find JSON object in URI"
layerSpecs <- (spec \ "layers").validate[JsArray].toFox
_ <- Fox.bool2Fox(credentialId.isEmpty) ~> "Neuroglancer URI Explorer does not support credentials"
exploredLayers = layerSpecs.value.map(exploreNeuroglancerLayer).toList
layerLists <- Fox.combined(exploredLayers)
layers = layerLists.flatten
renamedLayers = exploreLayerService.makeLayerNamesUnique(layers.map(_._1))
} yield renamedLayers.zip(layers.map(_._2))

private def exploreNeuroglancerLayer(layerSpec: JsValue): Fox[List[(DataLayer, Vec3Double)]] =
for {
_ <- Fox.successful(())
obj <- layerSpec.validate[JsObject].toFox
source <- (obj \ "source").validate[JsString].toFox
layerType = new URI(source.value).getScheme
sourceURI = new URI(source.value.substring(f"$layerType://".length))
name <- (obj \ "name").validate[JsString].toFox
remoteSourceDescriptor = RemoteSourceDescriptor(sourceURI, None)
remotePath <- dataVaultService.getVaultPath(remoteSourceDescriptor) ?~> "dataVault.setup.failed"
viewConfiguration = getViewConfig(obj)
layer <- exploreLayer(layerType, remotePath, name.value)
layerWithViewConfiguration <- assignViewConfiguration(layer, viewConfiguration)
} yield layerWithViewConfiguration

private def exploreLayer(layerType: String, remotePath: VaultPath, name: String): Fox[List[(DataLayer, Vec3Double)]] =
layerType match {
case "n5" =>
Fox.firstSuccess(
Seq(new N5ArrayExplorer().explore(remotePath, None), new N5MultiscalesExplorer().explore(remotePath, None)))
case "precomputed" => new PrecomputedExplorer().explore(remotePath, None)
case "zarr" | "zarr2" =>
Fox.firstSuccess(
Seq(new NgffExplorer().explore(remotePath, None),
new ZarrArrayExplorer(Vec3Int.ones, ec).explore(remotePath, None)))
case "zarr3" => new Zarr3ArrayExplorer().explore(remotePath, None)
case _ => Fox.failure(f"Can not explore layer of $layerType type")
}

private def getViewConfig(layerSpec: JsObject): LayerViewConfiguration = {
val opacity = (layerSpec \ "opacity").validate[Double].getOrElse(1.0)
val intensityRange = (layerSpec \ "shaderControls" \ "normalized" \ "range").validate[JsArray].asOpt
val options = Seq("alpha" -> JsNumber(opacity * 100)) ++ intensityRange.map("intensityRange" -> _)
options.toMap
}

private def assignViewConfiguration(
value: List[(DataLayer, Vec3Double)],
configuration: LayerViewConfiguration.LayerViewConfiguration): Fox[List[(DataLayer, Vec3Double)]] =
for {
_ <- Fox.successful(())
layers = value.map(_._1)
layersWithViewConfigs = layers.map {
case l: ZarrDataLayer => l.copy(defaultViewConfiguration = Some(configuration))
case l: ZarrSegmentationLayer => l.copy(defaultViewConfiguration = Some(configuration))
case l: N5DataLayer => l.copy(defaultViewConfiguration = Some(configuration))
case l: N5SegmentationLayer => l.copy(defaultViewConfiguration = Some(configuration))
case l: PrecomputedDataLayer => l.copy(defaultViewConfiguration = Some(configuration))
case l: PrecomputedSegmentationLayer => l.copy(defaultViewConfiguration = Some(configuration))
case l: Zarr3DataLayer => l.copy(defaultViewConfiguration = Some(configuration))
case l: Zarr3SegmentationLayer => l.copy(defaultViewConfiguration = Some(configuration))

}
} yield layersWithViewConfigs.zip(value.map(_._2))

}

0 comments on commit 1ba905a

Please sign in to comment.