Attempt to fix a very occasional hang in datasets map/filter

[dakinggg]

Inspired by comments on this datasets issue huggingface/datasets#3172, I'm trying out removing lambdas.

Also adds some extra safety to first load and cache the tokenizer on local rank 0.

Test run: no-lambdas-1-nSBWMV
Loss still the same

Also opened an issue on datasets: huggingface/datasets#3172

[subercui]

Hi, has this issue been resolved already? I encounters the same message as in huggingface/datasets#6393 . Running with datasets 2.16.0 and llm-foundry recently built from source

[dakinggg]

If you're on llm foundry latest, it generally works fine. In very rare scenarios I have still hit the hang reported in that issue to huggingface datasets. If you're completely blocked, I'd recommend converting your dataset to our Streaming format and using that code path instead.