downloading only the cloud mask

[daviddemeij]

Is there any possibility to only download the cloud masks from the Maja processed files? This would save a lot of downloading since we are planning to use this as a ground truth source of our own single-image cloud mask (so we don't need to process sequences to get a good cloud mask or for areas where we don't have an SRTM).

[olivierhagolle]

Hi David,
sorry for the late reply, posting a message 2 days before Christmas is not a good idea, as far as I am concerned ;)

I do not know how to do it, but I know it is possible to unzip a zip file through https, with some python commands. The guys in Sinergise did it to download all our cloud masks to use them as learning references for their own one.

If you find a way, I am interested
Olivier

[chauvenne]

@daviddemeij Did you find a way to achieve this?

[daviddemeij]

No I did not, in the end, we just downloaded the whole file unzipped and disregarded the other bands.

[chauvenne]

I found some way to do this. I first tried using GDAL /vsizip/ and /vsicurl/ but did not manage to make it work.

I then tried using Python requests module. I make a first request with the following header: headers = { "Authorization": "Bearer $YOUR_THEIA_TOKEN", "User-Agent": "python-requests" }
and with stream=True so that we do not start the download.

With the response, you can get the zip size and then fetch the "Central-Directory", giving you the full structure of it by downloading only a few bytes. See here on how to do that: https://stackoverflow.com/a/54222447 (this is for S3, but can easily be adapted to requests).

Once you have the structure, you can find the bytes range of the file you want, add this range in the headers and re-do the request. You finally save the obtained bytes into a zip file and fix it using zip -FF.