Add support for Databricks File System backend

[maddiedawson]

Description of changes:

Add support for uploading data to and downloading data from Databricks File System (DBFS)

Issue #, if available:

STR-128

Merge Checklist:

Put an x without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

• I have read the contributor guidelines
• This is a documentation change or typo fix. If so, skip the rest of this checklist.
• I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the MosaicML team.
• I have updated any necessary documentation, including README and API docs (if appropriate).

Tests

• I ran pre-commit on my change. (check out the pre-commit section of prerequisites)
• I have added tests that prove my fix is effective or that my feature works (if appropriate).
• I ran the tests locally to make sure it pass. (check out testing)
• I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes.

[karan6181]

Thanks for doing this. Few minor comments:

• How does the dbfs authentication happens? I think it might be related to the UC Volume PR that you have created ?
• Can you also add the similar functionality in upload.py which basically uploads the shard files in UC during dataset conversion? You can look at PR 311 and 265 for reference.
• Can you also add a unit test for the same in test_download.py and test_upload.py ?
• Can you also update the documentation with UC Volume section on how user would provide a credentials ?

Thanks!

[maddiedawson]

Thanks for doing this. Few minor comments:

• How does the dbfs authentication happens? I think it might be related to the UC Volume PR that you have created ?

I added some details to the docs; users will set DATABRICKS_HOST and DATABRICKS_TOKEN.

• Can you also add the similar functionality in upload.py which basically uploads the shard files in UC during dataset conversion? You can look at PR 311 and 265 for reference.

Done

• Can you also add a unit test for the same in test_download.py and test_upload.py ?

Done

• Can you also update the documentation with UC Volume section on how user would provide a credentials ?

Done

Thanks!

[karan6181]

For CI test failure, you can change this line to "[all]" which will install all the extra dependency packages.

[maddiedawson]

For CI test failure, you can change this line to "[all]" which will install all the extra dependency packages.

Done

[karan6181]

For the lint failure, Can you please add the below lines in this file ? I have verified it and now the lint will pass.

# Disable checks for missing imports, as a conditional install of streaming will not include them
# Any incorrect imports will be discovered through test cases
reportMissingImports="none"

[maddiedawson]

For the lint failure, Can you please add the below lines in this file ? I have verified it and now the lint will pass.

# Disable checks for missing imports, as a conditional install of streaming will not include them
# Any incorrect imports will be discovered through test cases
reportMissingImports="none"

Done