feature: minio support in fiab #163
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In fiab, mlflow is configured without an object store. Therefore, artifacts (e.g., model) are stored locally within a container. Once training is over, then the artifacts get lost. With support for minio (an open-source object store), now artifacts are saved in an object store. Also, minio's endpoint is exposed via ingress so that workers outside a cluster can save artifacts. Note that tls for minio is not enabled because selfsigned cert used in fiab is likely to make mlflow's client complain about a verification issue. A caveat is that no persistent volume is configured in this minio setup, artifacts will get lost if minio pod crashes or terminates. The support for minio is for reference so that users can configure minio or aws s3 correctly for production.
Codecov Report
@@ Coverage Diff @@
## main #163 +/- ##
=======================================
Coverage 21.29% 21.29%
=======================================
Files 34 34
Lines 1503 1503
=======================================
Hits 320 320
Misses 1173 1173
Partials 10 10 Continue to review full report at Codecov.
|
GaoxiangLuo
reviewed
Jun 30, 2022
There was a problem hiding this comment.
I can see a new pod named minio initialized. While running with MNIST example, the artifact full path starts with s3://mlruns/XX, which is as expected. If I login to the minio pod, where are the artifacts stored? Also, are we supposed to be able to open http://minio.flame.test?
|
@GaoxiangLuo you should be able to download artifacts from mlflow UI. |
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In fiab, mlflow is configured without an object store. Therefore,
artifacts (e.g., model) are stored locally within a container. Once
training is over, then the artifacts get lost. With support for
minio (an open-source object store), now artifacts are saved in an
object store. Also, minio's endpoint is exposed via ingress so that
workers outside a cluster can save artifacts. Note that tls for minio
is not enabled because selfsigned cert used in fiab is likely to make
mlflow's client complain about a verification issue.
A caveat is that no persistent volume is configured in this minio
setup, artifacts will get lost if minio pod crashes or terminates. The
support for minio is for reference so that users can configure minio
or aws s3 correctly for production.