Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(app): experimental azure blob storage support #1374

Merged
merged 6 commits into from
Jan 10, 2023

Conversation

olevski
Copy link
Member

@olevski olevski commented Dec 30, 2022

closes #1354

This can be merged and released. Azure blob support is disabled by default. We can simply enable it where we want it as long as the cluster already has the azure blob csi driver.

There is also some very light refactoring for the following:

  • Handle mounting more than one cloud storage type.
  • Use different classes for Jupyter servers on creation requests and on listing requests. This is useful because when creating a server we need to a bunch of checks that we do not really care about when simply listing an existing server.

/deploy #persist

@olevski olevski temporarily deployed to renku-ci-nb-1374 December 30, 2022 15:05 — with GitHub Actions Inactive
@RenkuBot
Copy link
Contributor

You can access the deployment of this PR at https://renku-ci-nb-1374.dev.renku.ch

@olevski olevski temporarily deployed to renku-ci-nb-1374 January 3, 2023 14:55 — with GitHub Actions Inactive
@olevski olevski temporarily deployed to renku-ci-nb-1374 January 3, 2023 17:51 — with GitHub Actions Inactive
@olevski olevski temporarily deployed to renku-ci-nb-1374 January 5, 2023 21:41 — with GitHub Actions Inactive
renku_notebooks/api/notebooks.py Outdated Show resolved Hide resolved
renku_notebooks/api/classes/cloud_storage/existing.py Outdated Show resolved Hide resolved
mount_folder=config.cloud_storage.mount_folder,
read_only=config.cloud_storage.azure_blob.read_only,
)
else:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we check that s3 is enabled here via config.cloud_storage.s3.enabled?

Also we should fail is both s3.enabled=True and azure_blob.enabled=True in the values file somewhere (I didn't see code for this)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Panaetius for now there is no problem if both s3 and azure blob storage are enabled. And I foresee that some deployments will want to use both. Especially those running on Azure. But even non-Azure deployments can use both s3 and azure provided that they have installed the azure blob csi drivers in their k8s cluster.

The way I recognize azure blob storage is if the access key is not provided by the secret key is provided. That is the only way. It is not perfect but it should work to start. Then we can adapt the ui and have proper indication what storage type is being used. But for now we are jumping onto the same api for the s3 storage.

When the user mounts a azure blob container then the endpoint is still the azure endpoint, the bucket is the azure blob container name and the secret key is the azure blob credentials. And the public key is left blank.

So I will add the checks for what is enabled and fail if s3 stuff is provided but s3 is disabled. But we can definitely support both azure blob and s3.

@olevski olevski temporarily deployed to renku-ci-nb-1374 January 9, 2023 16:54 — with GitHub Actions Inactive
@olevski olevski requested a review from Panaetius January 9, 2023 16:54
@olevski olevski deployed to renku-ci-nb-1374 January 10, 2023 05:55 — with GitHub Actions Active
@olevski olevski merged commit 4cbcb14 into master Jan 10, 2023
@olevski olevski deleted the feat-azure-blob-storage branch January 10, 2023 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

support azure blob storage on the azure cloud PoC
3 participants