Skip to content
/ Vi-ST Public

The official source code of paper in ECCV 2024. "Aligning Neuronal Coding of Dynamic Visual Scenes with Foundation Vision Models"

Notifications You must be signed in to change notification settings

wurining/Vi-ST

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Aligning Neuronal Coding of Dynamic Visual Scenes with Foundation Vision Models (ECCV 2024)

Model Architecture

Setup

Due to the dataset size, we had to only include the code part and the checkpoint. The dataset is not included in this repository. Please find the dataset in:

Onken, Arno et al. (2017). Data from: Using matrix and tensor factorizations for the single-trial analysis of population spike trains [Dataset]. Dryad. https://doi.org/10.5061/dryad.4ch10

# Install the required packages
pip3 install torch torchvision torchaudio lightning torchmetrics zarr
python -m pip install tslearn

# run train loop example
PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256 \
PYTHONPATH=. python src/trainer/Trainer.py -m \
task_name=eccv model.featrue_key=dinov2_feats_0 \
model=eccv dataset.cross_val_movie=True dataset.movie_name=movie01,movie03 tags='["eccv"]'

# the predictions will be saved in the following directory
# Mov1 -> Mov2
checkpoints/0/checkpoints
# Mov2 -> Mov1
checkpoints/1/checkpoints

BibTeX

@misc{Wu2024ViST,
      title={Aligning Neuronal Coding of Dynamic Visual Scenes with Foundation Vision Models}, 
      author={Rining Wu and Feixiang Zhou and Ziwei Yin and Jian K. Liu},
      year={2024},
      eprint={2407.10737},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.10737}, 
}

Acknowledgments

We thank Shanshan Jia, Zhile Yang, Zerui Yang and Jing Peng for the highly valuable discussions.

About

The official source code of paper in ECCV 2024. "Aligning Neuronal Coding of Dynamic Visual Scenes with Foundation Vision Models"

Topics

Resources

Stars

Watchers

Forks

Languages