Axolotl test suite fills up /tmp and does not clear space once completed

[fpreiss]

Please check that this issue hasn't been reported before.

• I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

Running pytest axolotl/tests multiple times on the same machine should consistently give the same output.

Current behaviour

As files are left behind in /tmp, at some point (for me running pytest axolotl/tests twice). The system runs out of storage in /tmp and multiple tests fail with OSError: [Errno 28] No space left on device. See the following directory overview:

$ dust /tmp
Did not have permissions for all directories
1.4G     ┌── checkpoint-10      │                ░█ │   4%
1.4G     ├── checkpoint-20      │                ░█ │   4%
2.8G   ┌─┴ tmptprwyfva          │                ██ │   9%
3.2G   ├── tmpg_r4o387          │                ██ │  10%
1.4G   │ ┌── checkpoint-10      │                ░█ │   4%
1.4G   │ ├── checkpoint-20      │                ░█ │   4%
3.2G   ├─┴ tmpi9lgife8          │                ██ │  10%
1.4G   │ ┌── checkpoint-10      │                ░█ │   4%
1.4G   │ ├── checkpoint-20      │                ░█ │   4%
3.2G   ├─┴ tmpxtjofffg          │                ██ │  10%
2.6G   │ ┌── pytorch_model.bin  │             ░░░██ │   8%
2.6G   │ │ ┌── pytorch_model.bin│             ░▒▒██ │   8%
2.6G   │ │ ├── optimizer.pt     │             ░▒▒██ │   8%
5.3G   │ ├─┴ checkpoint-79      │             ░████ │  17%
7.9G   ├─┴ tmpu3z6b6u7          │             █████ │  25%
2.6G   │ ┌── pytorch_model.bin  │             ░░░██ │   8%
2.6G   │ │ ┌── pytorch_model.bin│             ░▒▒██ │   8%
2.6G   │ │ ├── optimizer.pt     │             ░▒▒██ │   8%
5.3G   │ ├─┴ checkpoint-177     │             ░████ │  17%
7.9G   ├─┴ tmp3_8rxmvb          │             █████ │  25%
 31G ┌─┴ tmp                    │██████████████████ │ 100%

Steps to reproduce

pytest axolotl/tests
pytest axolotl/tests

Config yaml

No response

Possible solution

No response

Which Operating Systems are you using?

• Linux
• macOS
• Windows

Python Version

3.10

axolotl branch-commit

6459ac7

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this bug has not been reported yet.
• I am using the latest version of axolotl.
• I have provided enough information for the maintainers to reproduce and diagnose the issue.

[winglian]

My hunch is we need to cleanuo the output dir after each e2e test. https://github.com/OpenAccess-AI-Collective/axolotl/blob/main/tests/e2e/test_fused_llama.py

[winglian]

https://chat.openai.com/share/efbf525f-83c4-46f0-8352-9a62282c7350