Batch setup

[yellowcap]

This is the version that we used to run our first deployment at scale.

What changes

Main pipeline logic changes:

• Only concatenate S2, DEM, and S1 arrays for each tile during tililng. This saves lots of memory as S2 can be kept in Uint16.
• Use indexing on the arrays instead of the .sel function. Indexing is almost instant, and sel is slow for large arrays.
• Make the target bucket name an argument

Organisational changes:

• Add info about how to run the pipeline on batch.
• Add mgrs sample data as geopackage.

[weiji14]

@yellowcap or @lillythomas, can you double check the Sentinel-1 pixel values? I've been getting NaN loss values, and tracked it down to the Sentinel-1 VV and VH bands being all NaNs. Sentinel-2 and DEM bands are ok. Sample code:

import os
import matplotlib.pyplot as plt
import rioxarray
import xarray as xr

os.environ["AWS_PROFILE"] = "xxx"

da = rioxarray.open_rasterio(
    filename="s3://clay-tiles-01/01/56HKH/2022-06-18/claytile-56HKH-2022-06-18-01-1.tif"
)
da = da.compute()

# %%
# Plot different bands
da.sel(band=1).plot.imshow()  # Blue
plt.show()
da.sel(band=10).plot.imshow()  # SWIR2
plt.show()
da.sel(band=11).plot.imshow()  # VH
plt.show()
da.sel(band=12).plot.imshow()  # VV
plt.show()
da.sel(band=13).plot.imshow()  # DEM
plt.show()

[weiji14]

Tried a few other tiles in other locations like MGRS 29SMD over Lisbon, and it seems like the DEM can come up with NaNs as well (in places over the ocean?). So maybe check through all those bands.

[yellowcap]

Good catch! Looks like the nodata filter is missing checks on S1 and DEM. That should be easy to correct.

[weiji14]

Ok, looks like Lilly will be fixing this in #60.

[yellowcap]

converted to issue for tracking #68

[weiji14]

When saving the TIF files, can the counter part of the filename be zero-padded to a consistent length? The previous batch had filenames like claytile-......-1.tif, claytile-......-2.tif, ... claytile-......-20.tif, claytile-......-300.tif. Having a consistent length with numbers like 001, 002, 020, 300 would make sorting easier, and allow us to parse out the date using a reverse index (e.g. filepath[-21:-11]).

[weiji14]

Alternatively, if we can store the actual timestamp in the metadata of the GeoTIFF, that would be even better!

[yellowcap]

#69
#70

[weiji14]

Should this geopackage be stored on git? Or just on the s3 bucket?

[yellowcap]

#71

[weiji14]

At about L102 on the filename, @mattpaul had some suggestions at #35 (comment) to:

1. Use underscore as the delimiter between MGRS/DATE/VERSION/COUNTER
2. Preface the VERSION with a v (e.g. v0)

Also mentioned at #44 (comment) that we could drop the hyphen in the date, i.e. YYYY-MM-DD becomes YYYYMMDD, but that didn't seem to have been applied in the initial batch.

[yellowcap]

ok thanks will take that into consideration before next run.

[yellowcap]

Converted remaining questions to issues and will merge now for the sake of moving fast.