InContextLearning*Dataset Default padding sides hardcoded?

[MFajcik]

Hi, I was wondering regarding your code here.

composer/composer/datasets/in_context_learning_evaluation.py

Line 655 in a7cad7c

inp, continuation_span = _make_padded_input(context_enc, continuation_enc, self.max_seq_len,

Why do you assume right padding (for InContextLearningMultipleChoiceTaskDataset problem, but also some others)?

1. Shouldn't the padding_side be derived from the tokenizer?
2. Assuming right padding breaks some models (Mistral is unusable).

Thanks for information.

[dakinggg]

Hey @MFajcik, sorry for the delayed response. tokenizers have a default padding side set, but models should all be compatible with different padding sides (unless they explicitly error out). Generally speaking, we use right padding by default (for training, single forward passes, etc), and left padding for generation (necessary for the auto regressive generation and kv cache to work out). Mistral should work fine (we've run it). You may need to update to the latest transformers version. If you have an issue there, please send a full repro. Thanks!