Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adapt model to load 512x512 images from s3 bucket #85

Merged
merged 7 commits into from
Dec 11, 2023
Merged

Adapt model to load 512x512 images from s3 bucket #85

merged 7 commits into from
Dec 11, 2023

Commits on Dec 11, 2023

  1. 🔧 Increase image_size from 256 to 512, patch_size from 32 to 64 

    Increase the chip image size from 256 to 512 pixels, and the patch size from 32 to 64 pixels. Updated the unit test and an assert statement, and fixed a typo.
    @weiji14
    weiji14 committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    9ae91c0 View commit details
    Browse the repository at this point in the history

  2. 👽 Get YYYY-MM-DD from GeoTIFF tag instead of filename 

    Obtaining the YYYY-MM-DD date from the GeoTIFF's tag metadata, instead of parsing it from the filename, thanks to the change at 426aa06/#72.
    @weiji14
    weiji14 committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    b8c3e97 View commit details
    Browse the repository at this point in the history

  3. ✨ Allow GeoTIFFDataModule to get GeoTIFF data from an s3 bucket 

    New feature to allow passing in a URL to an s3 bucket, and loading the GeoTIFF data from there directly. Added a unit test that checks that this works to list a GeoTIFF file from s3://copernicus-dem-30m/. Also improved the docstring and type hint of the setup() function's 'stage' parameter.
    @weiji14
    weiji14 committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    f288eb8 View commit details
    Browse the repository at this point in the history

  4. 🐛 Add sharding filter before loading GeoTIFF data to torch.Tensor 

    Need to do this so that the data loading is distributed to the workers, otherwise each worker is doing duplicated work. Also set num_workers to 1 in test_geotiffdatapipemodule to get a consistent result.
    @weiji14
    weiji14 committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    60e7205 View commit details
    Browse the repository at this point in the history

  5. 🙈 Gitignore checkpoints in nested folders 

    Ensure that *.ckpt files in sub-folders are ignored too.
    @weiji14
    weiji14 committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    5659021 View commit details
    Browse the repository at this point in the history

  6. ⚡ Set float32 matmul precision to medium 

    Prevents messages like `You are using a CUDA device ('NVIDIA A10G') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance.`
    @weiji14
    weiji14 committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    82ec6f5 View commit details
    Browse the repository at this point in the history

  7. 📝 Mention in main README.md that data_path can be an s3 bucket 

    Just casually documenting in the main README.md on how one can directly generate embeddings from GeoTIFF files stored in an s3 bucket instead of locally.
    @weiji14
    weiji14 committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    7aeb431 View commit details
    Browse the repository at this point in the history