Skip to content
/ MoNCE Public

[CVPR 2022] Modulated Contrast for Versatile Image Synthesis

License

Notifications You must be signed in to change notification settings

fnzhan/MoNCE

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

47 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Modulated Contrast for Versatile Image Synthesis

Teaser

Fangneng Zhan, Jiahui Zhang, Yingchen Yu, Rongliang Wu, Shijian Lu
Nanyang Technological University, Singapore

MoNCE outperforms Perceptual Loss and PatchNCE Loss.

Abstract: This paper presents MoNCE, a versatile metric that introduces image contrast to learn a calibrated metric for the perception of multifaceted inter-image distances. Unlike vanilla contrast which indiscriminately pushes negative samples from the anchor regardless of their similarity, we propose to re-weight the pushing force of negative samples adaptively according to their similarity to the anchor, which facilitates the contrastive learning from informative negative samples. Since multiple patch-level contrastive objectives are involved in image distance measurement, we introduce optimal transport in MoNCE to modulate the pushing force of negative samples collaboratively across multiple contrastive objectives.

Structure

Prerequisites

  • Linux or macOS
  • Python3, PyTorch
  • CPU or NVIDIA GPU + CUDA CuDNN

Installation

Please follow the guidance in SPADE and CUT.

Inference Using Pretrained Model

Unpaired Image Translation (CUT):

The pretrained model on Cityscapes, Horse2Zebra, Winter2Summer can be downloaded from Google Drive. Put them into CUT_MoNCE/checkpoints and run the command

cd CUT_MoNCE
bash test_cityscapes.sh

Paired Image Translation (SPADE):

The pretrained model on ADE20K, CelebA-HQ (semantic), DeepFashion can be downloaded from Google Drive. Put them into SPADE_MoNCE/checkpoints and run the command

cd SPADE_MoNCE
bash test_ade20k.sh

Training

Unpaired Image Translation (CUT):

Run the command

cd CUT_MoNCE
bash train_cityscapes.sh

Paired Image Translation (SPADE):

Run the command

cd SPADE_MoNCE
bash train_ade20k.sh

Citation

If you use this code for your research, please cite our papers.

@inproceedings{zhan2022modulated,
  title={Modulated contrast for versatile image synthesis},
  author={Zhan, Fangneng and Zhang, Jiahui and Yu, Yingchen and Wu, Rongliang and Lu, Shijian},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={18280--18290},
  year={2022}
}

Acknowledgments

This code borrows heavily from CUT and SPADE.

About

[CVPR 2022] Modulated Contrast for Versatile Image Synthesis

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published