Skip to content

filby89/NTUA-BEEU-eccv2020

Repository files navigation

NTUA-BEEU-ECCV

Winning solution for the BEEU (First International Workshop on Bodily Expressed Emotion Understanding) challenge organized at ECCV2020. Please read the accompanied paper for more details.

[Update] For an updated version please check this extension achieving significantly better results with lighter models.

Preparation

Training

Train an RGB Temporal Segment Network on BoLD dataset:

python train_tsn.py -c config_tsn.json --modality "RGB" -b 32 --lr 1e-3 --arch resnet101 --workers 4 --num_segments 3 --exp_name "rgb tsn" -d 0,1,2,3

Add context branch:

python train_tsn.py -c config_tsn.json --modality "RGB" -b 32 --lr 1e-3 --arch resnet101 --workers 4 --num_segments 3 --exp_name "rgb with context tsn" -d 0,1,2,3 --context

Add visual embedding loss:

python train_tsn.py -c config_tsn.json --modality "RGB" -b 32 --lr 1e-3 --arch resnet101 --workers 4 --num_segments 3 --exp_name "rgb with context tsn" -d 0,1,2,3 --context --embed

Change modality to Flow:

python train_tsn.py -c config_tsn.json --modality "Flow" -b 32 --lr 1e-3 --arch resnet101 --workers 4 --num_segments 3 --exp_name "rgb tsn" -d 0,1,2,3

Pretrained Models

We also offer weights of an RGB with context model with 0.2213 validation ERS and a Flow model with 0.2157 validation ERS. Their fusion achieves an ERS of 0.2613 on the test set. You can download the pretrained models here. An example on how to use them is shown in test_tsn.py script:

python test_tsn.py --modality "RGB" --arch resnet101 --workers 4 --context python test_tsn.py --modality "Flow" --arch resnet101 --workers 4

Citation

If you use this code for your research, consider citing our paper.

@inproceedings{NTUA_BEEU,
  title={Emotion Understanding in Videos Through Body, Context, and Visual-Semantic Embedding Loss},
  author={Filntisis, Panagiotis Paraskevas and Efthymiou, Niki and Potamianos, Gerasimos and Maragos, Petros},
  booktitle={ECCV Workshop on Bodily Expressed Emotion Understanding},
  year={2020}
}

Acknowlegements

Contact

For questions feel free to open an issue.

About

Code for the BEEU challenge winning paper.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages