Quality of inference from the saved model

[Valavanca]

Hi Rémi,

I'm trying to use the saved model to make predictions on my Sentinel-2 images. However, I'm getting a poor result compared to the examples in the description. I'm unsure if the problem is with my input data or if I'm missing something for the inference.

Could you please give me some advice? Here are the steps I've taken so far:

1. Download bands from https://scihub.copernicus.eu/
2. Combine bands into one tif file

   otbcli_ConcatenateImages \
   -il bands/L1C_035225_20220321T091339_B04.jp2 \
       bands/L1C_035225_20220321T091339_B03.jp2 \
       bands/L1C_035225_20220321T091339_B02.jp2 \
       bands/L1C_035225_20220321T091339_B08.jp2 \
   -out T035225_4328.tif
   

3. Crop a region with QGIS
4. Run inference

   python ./code/sr.py \
   --savedmodel sr4rs_sentinel2_bands4328_france2020_savedmodel \
   --input data/crop_T035225_4328.tif \
   --output data/output/sr_T035225_4328.tif
   

Here is an example of my log and the result I'm getting:

Log:

(INFO) ReadImageInfo: 
Image general information:
        Number of bands : 4
        Data type : float
        No data flags : Not found
        Start index :  [0,0]
        Size :  [1025,816]
        Origin :  [300255,5.614e+06]
        Spacing :  [10,-10]
        Estimated ground spacing (in meters): [9.96869,9.99536]

Image acquisition information:
        Sensor : 
        Image projection : PROJCS["WGS 84 / UTM zone 36N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",33],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32636"]]

Image default RGB composition:
        [R, G, B] = [0,1,2]

Ground control points information:
        Number of GCPs = 0
        GCPs projection = 

INFO     Using encoding 5
INFO     Receptive field: 160, Expression field: 512

Result:

Any help you can provide would be greatly appreciated.

Thanks in advance!

[remicres]

Hi @Valavanca ,

Your settings are fine, I mean you are doing things the right way.

I suspect that this come from your L1C images.
Indeed the model has been trained from L2A images. Maybe the pixels values differs largely enough to harm the network.

Could you try with one L2A product, over the same area?

[Valavanca]

Unfortunately, it still didn't work. I tried to run it again on L2A_T35UQS_A026345_20220323T085859_2022-03-23 region and the result was similar.
Could you please share a train L2A image for reference?

[remicres]

You can test with any S2 image from Microsoft Planetary Computer, or Theia land data center.
Microsoft use Sen2cor calibration, Theia uses Maja, which are two completely different processors.
Both work fine up to my knowledge.
Maybe you are using GEE? we had an unresolved issue with their images I believe (#46 ) however I dont think image are the problem.

I don't have specific image to provide, sorry for that.
In the meantime, could you kindly provide:

• A screenshot of the input original S2 image
• The subset of the original S2 image corresponding to your ROI, so I can take a look
  Thanks

[Valavanca]

Thank you for your quick response. I used two options for downloading: copernicus.eu directly and a plugin for QGIS. I have not noticed the calibration options, but I will try to check it.

• Original preview: L2A_T35UQS_A026345_20220323T085859_2022-03-23
  
• ROI: Link
• ROI output: Link

[remicres]

Thanks.
I didn't notice anything on the image.
For now I don't have an explanation.

• It could be the limit of the model (we have trained it mostly over France mainland) however the output is really really ugly
• I have to check with the original image because it looks like there is some kind of thing aerosol, maybe that doesn't help neither.
  I will keep you updated

[dockoS]

hello @remicres
I encounter a problem with the quality of the super-resolution inference.
I download an image in Senegal and infer it with the pre-trained model but I have a result that is dark.

input : L2 sentinel image 16bits

output: link
Is it normal?
Where the problem can be?

[remicres]

@Valavanca I tried on L2A images acquired during the summer over the ROI, but I got the same outcome, i.e. ugly images with wrong radiometry.
@dockoS maybe you encounter the same issue.

My guess is that it is a limitation of the current model, which has been trained from Spot-6/7 images ranging from may until november, and in France Mainland area.
It could me worth mentioning somewhere in the repository what is the theoretical validity domain for the current model, and the future ones.

The ideal model would be trained from scenes scattered all across the world, all seasons.

[dockoS]

Thank you for your quick response.

An idea for me to have a good result will be to fine-tune the model in Senegal Data.
Thank you for your great work

[Valavanca]

Many thanks for your time. It helped to orient where to go next.

[remicres]

The best would be to fine tune the model from a few samples over your interest area.
It is quite straightforward to do fine tuning in TF v1, but hat is not explicitely implemented in sr4rs for now (#25 )

[dockoS]

Thanks for your feedback. Le ven. 23 déc. 2022 à 12:36, Rémi Cresson ***@***.***> a écrit :

…

The best would be to fine tune the model from a few samples over your interest area. It is quite straightforward to do fine tuning in TF v1, but hat is not explicitely implemented in sr4rs for now (#25 <#25> ) — Reply to this email directly, view it on GitHub <#57 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMBOK4QYQXKCWF6IYVEO2L3WOWME3ANCNFSM6AAAAAATEQQX7Y> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[NourSoltani]

@remicres
I have a question if you may Mr Rémi.
Are you thinking in providing a newer and updated version of the model trained on a bigger and wider data?

[remicres]

Hi @NourSoltani

I would like to. Unfortunately I don't have Spot-6/7 images all around the world. Only over France Mainland (we have a partnership with Airbus Defense & Space for french territories only)

[NourSoltani]

@remicres Oh, I understand..
Thank you for responding. I'm gonna test it on North Africa and see.
And as I understand, the pre-trained weights presented in here are the last version?!
https://nextcloud.inrae.fr/s/boabW9yCjdpLPGX

[remicres]

Yes it is.

If somebody from a remote sensing company give us nice images all around the globe we could train a worldwide model ;)

[NourSoltani]

Well.. yep..
We hope so :)