Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Spellcheck #345

Merged
merged 44 commits into from
Aug 23, 2024
Merged

feat: Spellcheck #345

merged 44 commits into from
Aug 23, 2024

Conversation

jeremyarancio
Copy link
Collaborator

What

  • LLMs with QLoRA development.

Description

  • LLM fine-tuning on the Spellcheck
  • Re-evaluation of Foundational models on the benchmark
  • Normalization evaluation algorithm to not consider some types of errors
  • Add data processing pipeline

Part of

  • Spellcheck

@jeremyarancio
feat(spellcheck): ⚡ Add feature: push to Argilla from an HF dataset
4ad41de
@jeremyarancio
fix(spellcheck): 🐛 Add fixes to T5 script
128e056
@jeremyarancio
fix(spellcheck): 🐛 Add guardrail to prevent compiuting metrics with e…
3217a89 
…mpty strings
@jeremyarancio
feat(spellcheck): 🎨 Training pipeline using Metaflow
8bd97f2
@jeremyarancio
feat(spellcheck): 🎨 LLM QLoRA TRL training script - Mistral - 7B - In…
a858bc8 
…struct
@jeremyarancio
perf(spellcheck): 🧪 Normalize evaluation algorithm
634b050 
"flavour" -> "flavor" - "ï" -> "i" - "â" -> "a" - "oe"
@jeremyarancio
feat(spellcheck): 🎨 Implement LLM training with Sagemaker & Metaflow
e16c6a2 
Scripts are customed to handle training in the cloud using Sagemaker Training Jobs
@jeremyarancio
feat(spellcheck): ⚡ Mistral 7b instruct v3 trained
2f6f75b
@jeremyarancio
feat(spellcheck): 🎨 Update guidelines: accents
db2780c
@jeremyarancio
refactor(spellcheck): ✨ Update Logging to consider script and src cod…
a6bc377 
…e for defining the level
@jeremyarancio
feat(spellcheck): ✨ Dataset processing methods & pipeline created
94b17d7
@jeremyarancio
build(spellcheck): ✨ Dataset processing (oe, percentage alignment): v3.1
8a3e9eb
@jeremyarancio
feat(spellcheck): ♻️ Training Mistral-7B-Instruct: instruction + unid…
fd0460b 
…ecode normalization + scheduler linear
@jeremyarancio
Delete previous training lllm dag
1559740
@jeremyarancio
feat(spellcheck): ⚡ Add eval normalization: remove "\n"
a612752
@jeremyarancio
refactor(spellcheck): 👷 Foundational LLMs re-evaluated on the benchmark
cb393c6 
Prompt was intentionally overfitted on the benchmark to create later the synthetic training dataset. Examples from benchmark are removed from the prompt.
@jeremyarancio
Modify overfitted prompt
edce9c5
@jeremyarancio
Merge branch 'develop' into spellcheck
1645a83
@jeremyarancio jeremyarancio self-assigned this Jul 8, 2024
@jeremyarancio
Update get_logger for Metaflow logging
cdcc825
@jeremyarancio
feat(spellcheck): ✨ Double the benchmark size: extraction and push to…
7cf82a9 
… Argilla pipeline (WIP)
@jeremyarancio
docs(spellcheck): 📝 Document benchmark generation pipeline
e28a470
@jeremyarancio
feat(spellcheck): 🐛 Remove legacy metadata in Argilla
b6b5bf8
@jeremyarancio
refactor(spellcheck): 🚧 Refactor training pipeline (WIP)
a4f295a
@jeremyarancio
refactor(spellcheck): 🚧 Refactor training script (WIP)
c6aaee5
@jeremyarancio
refactor(spellcheck): 🚧 Refactor training pipeline
4401966
@jeremyarancio
chore(spellcheck): ✨ Update Python from 3.9 to 3.10
e870a24
@jeremyarancio
refactor(spellcheck): ⚡ LLM training pipeline refactored
08a097a
@jeremyarancio
feat(spellcheck): ✨ Pretraining before fine-tuning (WIP)
b563be1
@jeremyarancio
feat(spellcheck): 🚑 Pretraining + Finetuning Mistral-7B
f0d924d
@jeremyarancio
refactor(spellcheck): ✨ Refactor training
ac4f496
@jeremyarancio
feat(spellcheck): 🚧 Batch processing (WIP)
1600a65
@teolemon teolemon changed the title Spellcheck feat: Spellcheck Aug 6, 2024
@raphael0202 raphael0202 merged commit 24adbb2 into develop Aug 23, 2024
1 of 2 checks passed
@raphael0202 raphael0202 deleted the spellcheck branch August 23, 2024 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raphael0202 raphael0202 raphael0202 approved these changes

Assignees

@jeremyarancio jeremyarancio

Labels
spellcheck
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jeremyarancio @raphael0202