Skip to content

[CVPR 2021] Adaptive Consistency Regularization for Semi-Supervised Transfer Learning

License

Notifications You must be signed in to change notification settings

SHI-Labs/Semi-Supervised-Transfer-Learning

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

44 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Adaptive Consistency Regularization for Semi-Supervised Transfer Learning

This repository is for Adaptive Knowledge Consistency and Adaptive Representation Consistency introduced in the following paper:

Abulikemu Abuduweili, Xingjian Li, Humphrey Shi, Cheng-Zhong Xu, and Dejing Dou, Adaptive Consistency Regularization for Semi-Supervised Transfer Learning, CVPR 2021.

Contents

  1. Introduction
  2. Tasks
  3. Citation

Introduction

In this work, we consider semi-supervised learning and transfer learning jointly, leading to a more practical and competitive paradigm that can utilize both powerful pre-trained models from source domain as well as labeled/unlabeled data in the target domain. To better exploit the value of both pre-trained weights and unlabeled target examples, we introduce adaptive consistency regularization that consists of two complementary components: Adaptive Knowledge Consistency (AKC) on the (labeled and unlabeled) examples between the source and target model, and Adaptive Representation Consistency (ARC) on the target model between labeled and unlabeled examples. Examples involved in the consistency regularization are adaptively selected according to their potential contributions (measured by prediction entropy) to the target task. Moreover, our algorithm is orthogonal to existing methods and thus able to gain additional improvements on top of the existing semi-supervised learning methods.

model

Tasks

Semi-supervised transfer learning

AKC and ARC regularization terms could be combined with other semi-supervised learning methods, like MixMatch and FixMatch. By utilizing AKC and ARC regularization techniques in MixMatch, the performance increased notably.

Results on CUB-200-2011 dataset:
cub200

The actual sample selected ratio in ARC and AKC is shown in Figure 2 on CUB-200-2011 dataset experiment. As can be seen, the sample selected ratio for ARC is gradually increasing. Which can be regarded as a kind of curriculum learning.
sample_ratio

Supervised transfer learning

Both AKC and ARC improve the performance of standard transfer learning.
supervised

Effectiveness of transfer learning in semi-supervised setting

In previous works, the effectiveness of transfer learning in semi-supervised settings was underestimated. With the Imprinting technique and proper training strategy, transfer learning could lead to a noticeable improvement, especially when labeled examples are insufficient.

Results of SSL methods with and without transfer learning on CIFAR-10:
tf_ssl

Citation

If you find the code helpful in your resarch or work, please cite the following papers.

@inproceedings{abuduweili2021adaptive,
  title={Adaptive Consistency Regularization for Semi-Supervised Transfer Learning},
  author={Abuduweili, Abulikemu and Li, Xingjian and Shi, Humphrey and Xu, Cheng-Zhong and Dou, Dejing},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={6923--6932},
  year={2021}
}