num_input_tokens_seen included the pad tokens if sample padding strategy used

[thincal]

System Info

latest transformers

Who can help?

@muellerzr @pacman100

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

If training sample is batched with padding strategy, the num_input_tokens_seen will be calculated including the pad tokens, this is not expected.

Expected behavior

1. Excluding the pad tokens during the calculation of num_input_tokens_seen, a suggested fix as bellow:

inputs_device = "cuda" if self.args.distributed_state.backend == "nccl" else "cpu"

self.state.num_input_tokens_seen += torch.sum(
    self.accelerator.gather(
        torch.tensor(torch.sum(inputs[main_input_name] != self.tokenizer.pad_token_id).item(), device=inputs_device, dtype=torch.int64)
    )
).item()

2. And if the nccl backend used but input_ids stay in cpu it will also cause problem, this is also fixed by deciding the device type according to backend.

inputs_device = "cuda" if self.args.distributed_state.backend == "nccl" else "cpu"

[thincal]

@pacman100 could help review with this issue reported ? thanks!

[amyeroberts]

Gentle ping @muellerzr @pacman100

[amyeroberts]

cc @muellerzr @SunMarc

[SunMarc]

Hi @thincal, thanks for raising the issue and sorry for the delay ! This seems like a good idea to remove the padded tokens and change the device depending on the backend ! Would you like to open a PR ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.