Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow Exploring Datasets on Local File System #7389

Merged
merged 7 commits into from
Oct 16, 2023
Merged

Allow Exploring Datasets on Local File System #7389

merged 7 commits into from
Oct 16, 2023

Conversation

fm3
Copy link
Member

@fm3 fm3 commented Oct 16, 2023 •
edited
Loading

  • Unifies FileSystemDataVault with other DataVaults, removes FileSystemVaultPath. VaultPaths now use file:// uris with absolute paths for the local file system.
  • Allows exploring datasets in the local file system, in directories listed in the new config key datastore.localFolderWhitelist
  • Note that this introduces file:// uris also in the resulting datasource-properties.jsons. Those will not be sensibly explorable by the WebknossosZarrExplorer but I guess that isn’t a use case anyway

Steps to test:

  • Add a directory to config setting datastore.localFolderWhitelist
  • Put dataset directory in there (or into its children/descendents)
  • Explore that dataset in the Add Datasets view, prepend file:// to the absolute path (so there are then three consecutive slashes)
  • Should work
  • Should not work is the directory is not in the whitelist
  • Other datasets with mag locators (everything that isn’t wkw) should still work (with relative path + with remote path)

TODOs:

  • Unify FileSystemDataVault with other DataVaults, remove FileSystemVaultPath
  • Allow exploring locally
  • Add config option to whitelist directories in which webknossos may read
    • Use in datastore when reading from data vault
    • Use in webknossos when exploring
  • Test that this does not break existing datasource-properties.jsons
    • with local relative path
    • with remote path

Issues:

  • Updated changelog
  • Needs datastore update after deployment

@fm3 fm3 self-assigned this Oct 16, 2023
fm3
fm3 commented Oct 16, 2023
View reviewed changes
app/models/dataset/credential/CredentialService.scala
@@ -43,6 +43,8 @@ class CredentialService @Inject()(credentialDAO: CredentialDAO) {
secret <- credentialSecret
secretJson <- tryo(Json.parse(secret)).toOption
} yield GoogleServiceAccountCredential(uri.toString, secretJson, userId.toString, organizationId.toString)
case _ =>
None
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was where the match error originated in case of an unknown uri scheme. That shouldn’t happen.

fm3
fm3 commented Oct 16, 2023
View reviewed changes
...nossos-datastore/app/com/scalableminds/webknossos/datastore/dataformats/BucketProvider.scala
@@ -15,19 +14,19 @@ trait BucketProvider extends FoxImplicits with LazyLogging {
def remoteSourceDescriptorServiceOpt: Option[RemoteSourceDescriptorService]

// To be defined in subclass.
def loadFromUnderlying(readInstruction: DataReadInstruction)(implicit ec: ExecutionContext): Fox[DataCubeHandle] =
def openShardOrArrayHandle(readInstruction: DataReadInstruction)(implicit ec: ExecutionContext): Fox[DataCubeHandle] =
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just some renamings here

@fm3 fm3 marked this pull request as ready for review October 16, 2023 11:41
@fm3 fm3 requested a review from normanrz October 16, 2023 11:41
@fm3 fm3 changed the title WIP: Allow Exploring Datasets on Local File System Allow Exploring Datasets on Local File System Oct 16, 2023
@fm3 fm3 enabled auto-merge (squash) October 16, 2023 11:51
@fm3 fm3 merged commit 0fb478f into master Oct 16, 2023
2 checks passed
@fm3 fm3 deleted the explore-from-file-system branch October 16, 2023 12:02
@fm3 fm3 mentioned this pull request Oct 16, 2023
Closed
@fm3 fm3 mentioned this pull request May 2, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@normanrz normanrz normanrz approved these changes

Assignees

@fm3 fm3

Labels
backend new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

ExploreAndAdd route: support datasets on local file system
2 participants
@fm3 @normanrz