Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up the logs, bump datasets and transformers #804

Merged
merged 42 commits into from
Dec 15, 2023

Conversation

dakinggg
Copy link
Collaborator

@dakinggg dakinggg commented Dec 14, 2023

The noisiest logs were coming from an old datasets version. Bumping the datasets version ended up requiring bumping the transformers version due to a dependency quadrangle. So this PR is including the version bumps too (and associated workarounds).

  • Run regression tests against this PR

llama2 finetuning, before and after, with and without flash attention enabled
Screenshot 2023-12-15 at 1 37 06 PM
Screenshot 2023-12-15 at 1 36 19 PM
Screenshot 2023-12-15 at 1 36 43 PM

  • Verify that model saving and deployment handles safetensors (turned on by default in latest transformers)
  • Run a before and after for the logs (before: clean-logs-baseline-EJoND1, after: clean-logs-local-after-slljhI)

Related transformers issues:

@dakinggg dakinggg changed the title Clean up the logs a bit Clean up the logs, bump datasets and transformers Dec 14, 2023
@irenedea
Copy link
Contributor

irenedea commented Dec 14, 2023

Can you also update streaming to 0.7.2? Actually we can do a separate PR for streaming and composer updates, wdyt? Though, I was thinking it might be nice to do it all at once for regression tests.

llmfoundry/utils/logging_utils.py Show resolved Hide resolved
scripts/train/train.py Outdated Show resolved Hide resolved
tests/models/test_model.py Show resolved Hide resolved
llmfoundry/models/hf/hf_causal_lm.py Show resolved Hide resolved
tests/models/layers/test_huggingface_flash.py Show resolved Hide resolved
llmfoundry/__init__.py Show resolved Hide resolved
@dakinggg dakinggg marked this pull request as ready for review December 15, 2023 22:37
@dakinggg dakinggg requested a review from a team as a code owner December 15, 2023 22:37
Copy link
Contributor

@irenedea irenedea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! had a suggestion to use a context manager for filtering cases that require clean up

@dakinggg dakinggg enabled auto-merge (squash) December 15, 2023 23:26
@dakinggg dakinggg merged commit 06b9a1f into mosaicml:main Dec 15, 2023
10 checks passed
@dakinggg dakinggg mentioned this pull request Dec 18, 2023
1 task
@dakinggg dakinggg deleted the clean-logs branch February 10, 2024 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants