DeS3_Deshadow (AAAI'2024)

Introduction

This is an implementation of DeS3: Attention-driven Self and Soft Shadow Removal using ViT Similarity and Color Convergence

git clone https://github.com/jinyeying/DeS3_Deshadow.git
cd DeS3_Deshadow/

1. Datasets

1. SRD Train|BaiduPan, Test. Shadow Masks

2. AISTD|ISTD+ [link]

3. LRSS: Soft Shadow Dataset [link]
   The LRSS dataset contains 134 shadow images (62 pairs of shadow and shadow-free images).
   We use 34 pairs for testing and 100 shadow images for training. For shadow-free training images, 28 from LRSS and 72 randomly selected from the USR dataset.
   

   [Dropbox]	[BaiduPan (code:t9c7)]

4. USR: Unpaired Shadow Removal Dataset [link]

5. UCF, UIUC: Self Shadow [link]

2. SRD Dataset Results:

[Dropbox]	[BaiduPan(code:blk7)]

SRD Dataset Evaluation

1. set the paths of the shadow removal result and the dataset in evaluation/demo_SRD_RMSE.m and then run it.

demo_SRD_RMSE.m

Get the RMSE from Table 1 in the main paper on the SRD (size: 256x256):

Method	Training	Shadow	Non-Shadow	ALL
DeS3	Paired	5.88	2.83	3.72

2. set the paths of the shadow removal result and the dataset in evaluation/evaluate_SRD_PSNR_SSIM.m and then run it.

evaluate_SRD_PSNR_SSIM.m

Get the PSNR & SSIM from Table 1 in the main paper on the SRD (size: 256x256):

		PSNR	PSNR	PSNR	SSIM	SSIM	SSIM
Method	Training	Shadow	Non-Shadow	ALL	Shadow	Non-Shadow	ALL
DeS3	Paired	37.45	38.12	34.11	0.984	0.988	0.968

3. AISTD Dataset Results:

[Dropbox]	[BaiduPan(code:blk7)]

AISTD Dataset Train, Test

1. modify the path in

   DeS3_Deshadow/datasets/aistdshadow.py

   Line 30 in c294476

   filelist='/home1/yeying/DeS3_Deshadow/aistdtest.txt',

   DeS3_Deshadow/configs/AISTDshadow.yml

   Line 6 in c294476

   data_dir: "/home1/yeying/WeatherDiffusion_shadow/data/AISTD/"

2. download the AISTD checkpoint [Dropbox] | [BaiduPan(code:aistd)]

CUDA_VISIBLE_DEVICES=1,2 python train_aistd.py --config 'AISTDshadow.yml' --resume '/home1/yeying/DeS3_Deshadow/ckpts/AISTDShadow_ddpm.pth.tar'

CUDA_VISIBLE_DEVICES=1 python eval_aistd.py --config 'AISTDshadow.yml' --resume '/home1/yeying/DeS3_Deshadow/ckpts/AISTDShadow_ddpm.pth.tar'

AISTD Dataset Evaluation

1. set the paths of the shadow removal result and the dataset in evaluation/demo_AISTD_RMSE.m and then run it.

demo_AISTD_RMSE.m

Get the RMSE on the AISTD (size: 256x256):

Method	Training	Shadow	Non-Shadow	ALL
DeS3	Paired	6.56	3.40	3.94

2. set the paths of the shadow removal result and the dataset in evaluation/evaluate_AISTD_PSNR_SSIM.m and then run it.

evaluate_AISTD_PSNR_SSIM.m

Get the PSNR & SSIM on the ISTD (size: 256x256):

		PSNR	PSNR	PSNR	SSIM	SSIM	SSIM
Method	Training	Shadow	Non-Shadow	ALL	Shadow	Non-Shadow	ALL
DeS3	Paired	36.49	34.70	31.38	0.989	0.972	0.958

Acknowledgments

Code is implemented based WeatherDiffusion, we would like to thank them.

License

The code and models in this repository are licensed under the MIT License for academic and other non-commercial uses.
For commercial use of the code and models, separate commercial licensing is available. Please contact:

• Jonathan Tan (jonathan_tano@nus.edu.sg)

Citations

If this work is useful for your research, please cite our paper.

@inproceedings{jin2024des3,
  title={DeS3: Adaptive Attention-Driven Self and Soft Shadow Removal Using ViT Similarity},
  author={Jin, Yeying and Ye, Wei and Yang, Wenhan and Yuan, Yuan and Tan, Robby T},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={3},
  pages={2634--2642},
  year={2024}
}