Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Baal 2.0] All in one Experiment Object #288

Closed
Dref360 opened this issue May 11, 2024 · 0 comments · Fixed by #296
Closed

[Baal 2.0] All in one Experiment Object #288

Dref360 opened this issue May 11, 2024 · 0 comments · Fixed by #296
Assignees
Labels
enhancement New feature or request

Comments

@Dref360
Copy link
Member

Dref360 commented May 11, 2024

The active loop is pretty repetitive across experiments and is error prone as we know.

People forget to reset weights, log metrics, and check uncertainty.
ActiveLearningLoop has extended its stay imo and we should deprecate it in favor of this new object.
Baal would accommodate ModelWrapper and transformers.Trainer, but should make it easy to support new libraries as always.

So an experiment would now look like this:

wrapper = ModelWrapper(...)
wrapper.add_metric(...)

al_dataset = ActiveLearningDataset(...)
criterion = LabellingBudgetCriterion(...)


experiment = ActiveExperiment(trainer=wrapper, al_dataset=al_dataset, heuristic=EPIG(), query_size=100, criterion=criterion)

results = experiment.start()

And that's it! Optionally wrapper could be a transformers.Trainer.

To me this would give a bit of youth to our library that is now well over 5 years old :)
We could deprecate ActiveLearningLoop in 2.0 and introduce Criterion, Experiment and EPIG for a massive release.

@Dref360 Dref360 added the enhancement New feature or request label May 11, 2024
@Dref360 Dref360 changed the title All in one Experiment Object [Baal 2.0] All in one Experiment Object May 11, 2024
@Dref360 Dref360 pinned this issue May 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants