[Bug] BEiT performance

[djaniak]

Branch

main branch (mmpretrain version)

Describe the bug

Pretrained BEiT model from checkpoint beit_beit-base-p16_8xb256-amp-coslr-300e_in1k does not work properly. Doing a linear probing on representation from this model (cls token or mean) gives almost random accuracy. I believe there may be something wrong with the checkpoint since the same code for BEiTv2 model (model = get_model('beitv2_beit-base-p16_8xb256-amp-coslr-300e_in1k', pretrained=True)) works fine and I can train the linear classifier on top of the frozen representation.

from mmpretrain import get_model

model = get_model('beit_beit-base-p16_8xb256-amp-coslr-300e_in1k', pretrained=True)
inputs = torch.rand(1, 3, 224, 224)
out = model(inputs)
print(type(out))
# To extract features.
feats = model.extract_feat(inputs)
print(type(feats))

Environment

{'sys.platform': 'linux',
 'Python': '3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]',
 'CUDA available': True,
 'MUSA available': False,
 'numpy_random_seed': 2147483648,
 'GPU 0,1,2,3,4': 'NVIDIA TITAN RTX',
 'CUDA_HOME': '/opt/cuda',
 'NVCC': 'Cuda compilation tools, release 12.2, V12.2.91',
 'GCC': 'gcc (GCC) 13.2.1 20230801',
 'PyTorch': '2.2.2+cu121',
 'TorchVision': '0.17.2+cu121',
 'OpenCV': '4.9.0',
 'MMEngine': '0.10.4',
 'MMCV': '2.2.0',
 'MMPreTrain': '1.2.0+6811d71'}

Other information

No response