Skip to content

[CVPR 2023] Official repository of paper titled "Fine-tuned CLIP models are efficient video learners".

License

Notifications You must be signed in to change notification settings

muzairkhattak/ViFi-CLIP

Repository files navigation

Fine-tuned CLIP models are efficient video learners [CVPR 2023]

Fine-tuned CLIP models are efficient video learners
Hanoona Rasheed*, Muhammad Uzair Khattak*, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan

*Equally contributing first authors

Website paper video slides Jupyter Notebook

Official implementation of the paper "Fine-tuned CLIP models are efficient video learners".


🚀 News

  • (Nov 24, 2023)
    • Interactive notebook released. Inference with ViFi-CLIP on custom videos without significant installation dependencies!
  • (Feb 28, 2023)
    • Paper accepted at CVPR 2023 🎉
  • (Dec 6, 2022)
    • Training and evaluation codes for ViFi-CLIP, along with pretrained models are released.

Highlights

main figure

This work explores the capability of a simple baseline called ViFi-CLIP (Video Fine-tuned CLIP) for adapting image pretrained CLIP to video domain. The figure compares the zero-shot performance of vanilla CLIP and several of its variants adapted for videos (trained on Kinetics-400, evaluated on UCF-101 and HMDB-51). The t-SNE visualizations of video-embeddings obtained from ViFi-CLIP (4th col.) are compared with embeddings from vanilla CLIP (1st col.), individually tuned CLIP text (2nd col.) and image encoder (3rd col.) on videos, and recent state-of-the-art work, XCLIP (last col.) (∆ represents difference over XCLIP). The embeddings of ViFi-CLIP are better separable, indicating that a simple fine-tuning of CLIP is sufficient to learn suitable video-specific inductive biases, and can perform competitive to more complex approaches having dedicated components designed to model temporal information in videos.

Abstract: Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Since training on a similar scale for videos is infeasible, recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this pursuit, new parametric modules are added to learn temporal information and inter-frame relationships which require meticulous design efforts. Furthermore, when the resulting models are learned on videos , they tend to overfit on the given task distribution and lack in generalization aspect. This begs the following question: How to effectively transfer image-level CLIP representations to videos? In this work, we show that a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the domain gap from images to videos. Our qualitative analysis illustrates that the frame-level processing from CLIP image-encoder followed by feature pooling and similarity matching with corresponding text embeddings helps in implicitly modeling the temporal cues within ViFi-CLIP. Such fine-tuning helps the model to focus on scene dynamics, moving objects and inter-object relationships. For low-data regimes where full fine-tuning is not viable, we propose a `bridge and prompt' approach that first uses fine-tuning to bridge the domain gap and then learns prompts on language and vision side to adapt CLIP representations. We extensively evaluate this simple yet strong baseline on zero-shot, base-to-novel generalization, few-shot and fully supervised settings across five video benchmarks.

Main Contributions

  1. ViFi-CLIP: We formulate and show the significance of an often neglected but simple baseline for transferring image-based CLIP model to video domain. ViFi-CLIP (Video Fine-tuned CLIP) shows that a simple fine-tuning of CLIP is sufficient to learn suitable video-specific inductive biases, and can perform competitive to more complex approaches having dedicated components designed to model temporal information in videos.
  2. Base-to-novel generalization benchmark: We introduce base-to-novel generalization benchmark for video-domain for evaluating the generalization ability of models for video action recognition.
  3. Bridge and Prompt approach: We show the effectiveness of our proposed ‘bridge and prompt’ approach to first bridge the modality gap through fine-tuning followed by prompt learning in both visual and language branches of the CLIP model for low-data regimes.

Model Zoo

NOTE: All models in our experiments below uses publicly available ViT/B-16 based CLIP model. The trained model weights against each experiment is provided in tables below.

Zero-shot results

All models are trained on Kinetics-400 and then evaluated directly on downstream datasets.

Name (configs) Input HMDB-51 UCF-101 Kinetics-600 Model
CLIP image-FT 32x224 49.0 72.9 62.2 link
CLIP text-FT 32x224 48.5 69.8 68.5 link
ViFi-CLIP 32x224 51.3 76.8 71.2 link

Base-to-novel generalization results

Here, we divide each dataset into base and novel classes. All models are trained on base classes and evaluated on both base and novel classes. Results are averaged over 3 seeds for each experiment.

Kinetics-400

Name (configs) Input Base Acc. Novel Acc. HM Model
CLIP image-FT 32x224 72.9 58.0 64.6 seed1/seed2/seed3
CLIP text-FT 32x224 73.4 59.7 65.8 seed1/seed2/seed3
ViFi-CLIP 32x224 76.4 61.1 67.9 seed1/seed2/seed3

HMDB-51

Name (configs) Input Base Acc. Novel Acc. HM Model
CLIP image-FT 32x224 62.6 47.5 54.0 seed1/seed2/seed3
CLIP text-FT 32x224 70.0 51.2 59.1 seed1/seed2/seed3
ViFi-CLIP 32x224 73.8 53.3 61.9 seed1/seed2/seed3

UCF-101

Name (configs) Input Base Acc. Novel Acc. HM Model
CLIP image-FT 32x224 86.4 65.3 74.4 seed1/seed2/seed3
CLIP text-FT 32x224 90.9 67.4 77.4 seed1/seed2/seed3
ViFi-CLIP 32x224 92.9 67.7 78.3 seed1/seed2/seed3

SSv2

Name (configs) Input Base Acc. Novel Acc. HM Model
CLIP image-FT 32x224 9.2 8.5 8.8 seed1/seed2/seed3
CLIP text-FT 32x224 12.4 9.5 10.8 seed1/seed2/seed3
ViFi-CLIP 32x224 16.2 12.1 13.9 seed1/seed2/seed3

VL Prompting approach: Base-to-Novel

ViFi-CLIP is first trained on K400 and then vision and language prompts are further fine-tuned on the downstream datasets.

Dataset (configs) Input Base Acc. Novel Acc. HM Model
HMDB-51 32x224 77.1 54.9 64.1 seed1/seed2/seed3
UCF-101 32x224 95.9 74.1 83.6 seed1/seed2/seed3
SSv2 32x224 15.8 11.5 13.3 seed1/seed2/seed3

Few-shot results

Below table shows few-shot results of ViFi-CLIP for K=2, 4, 8 and 16.

Name (configs) Dataset K (shots) Input Top-1 Acc. Model
ViFi-CLIP HMDB-51 2 32x224 57.2 link
ViFi-CLIP HMDB-51 4 32x224 62.7 link
ViFi-CLIP HMDB-51 8 32x224 64.5 link
ViFi-CLIP HMDB-51 16 32x224 66.8 link
ViFi-CLIP UCF-101 2 32x224 80.7 link
ViFi-CLIP UCF-101 4 32x224 85.1 link
ViFi-CLIP UCF-101 8 32x224 90.0 link
ViFi-CLIP UCF-101 16 32x224 92.7 link
ViFi-CLIP SSv2 2 32x224 6.2 link
ViFi-CLIP SSv2 4 32x224 7.4 link
ViFi-CLIP SSv2 8 32x224 8.5 link
ViFi-CLIP SSv2 16 32x224 12.4 link

NOTE: Few-shot results for other CLIP Fine-tuned variants are presented in our main paper (Table 3). Model weights for other variants are provided here.

VL Prompting approach: Few-shot

ViFi-CLIP is first trained on K400 and then vision and language prompts are further fine-tuned on the downstream datasets in few-shot manner.

Dataset (configs) Input K=2 K=4 K=8 K=16 Model
HMDB-51 32x224 63.0 65.1 69.6 72.0 K=2/K=4/K=8/K=16
UCF-101 32x224 91.0 93.7 95.0 96.4 K=2/K=4/K=8/K=16
SSv2 32x224 6.7 7.9 10.2 13.5 K=2/K=4/K=8/K=16

Fully-supervised results on Kinetics-400

Name (configs) FLOPS(G) Input Top-1 Acc. Top-5 Acc. Model
CLIP image-FT 281 16x224 82.8 96.2 link
CLIP text-FT 281 16x224 73.1 91.2 link
ViFi-CLIP 281 16x224 83.9 96.3 link

Installation

For installation and other package requirements, please follow the instructions detailed in INSTALL.md.

Data preparation

Please follow the instructions at DATASETS.md to prepare all datasets.

Training

For all experiments shown in above tables, we provide config files in configs folder. For example, to train ViFi-CLIP (tunes both image and text encoder) on Kinetics-400, run the following command:

python -m torch.distributed.launch --nproc_per_node=8 \ 
main.py -cfg configs/fully_supervised/k400/16_16_vifi_clip.yaml --output /PATH/TO/OUTPUT 

Note:

  • We recommend keeping the total batch size as mentioned in respective config files. Please use --accumulation-steps to maintain the total batch size. Specifically, here the effective total batch size is 8(GPUs_NUM) x 4(TRAIN.BATCH_SIZE) x 16(TRAIN.ACCUMULATION_STEPS) = 512.
  • After setting up the datasets as instructed DATASETS.md, only argument in the config file that should be specified is data path. All other settings in config files are pre-set.

For detailed training instructions for all experimental setup, please refer to TRAIN.md.

Evaluating models

To evaluate a model, please use a suitable config and corresponding model weights. For example, to evaluate ViFi-CLIP with 16 frames on Kinetics-400, run the command below:

python -m torch.distributed.launch --nproc_per_node=8 main.py \
-cfg configs/fully_supervised/k400/16_16_vifi_clip.yaml --output /PATH/TO/OUTPUT \
--only_test --resume /PATH/TO/CKPT --opts TEST.NUM_CLIP 4 TEST.NUM_CROP 3

Contact

If you have any questions, please create an issue on this repository or contact at uzair.khattak@mbzuai.ac.ae or hanoona.bangalath@mbzuai.ac.ae .

Citation

If you use our approach (code, model or dataset splits) in your research, please consider citing:

@inproceedings{hanoonavificlip,
    title={Finetuned CLIP models are efficient video learners},
    author={Rasheed, Hanoona and khattak, Muhammad Uzair and Maaz, Muhammad and Khan, Salman and Khan, Fahad Shahbaz},
    booktitle={The IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2023}
}

Acknowledgements

Our code is based on XCLIP's repository. We sincerely thank the authors for releasing their code. If you use our model and code, please consider citing XCLIP as well.