Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-3097: fix multitask model training #3101

Merged
merged 6 commits into from
Feb 14, 2023

Conversation

helpmefindaname
Copy link
Collaborator

@helpmefindaname helpmefindaname commented Feb 13, 2023

This PR fixes #3097 by introducing an evaluate_all parameter. That way, users can use multitask_model.evaluate(corpus.test, gold_label_type="Task_0", evaluate_all=False) to evaluate the first task. If evaluate_all is set to true, the value of gold_label_type will be ignored and every task will be evaluated on their respective label type.

While doing this implementation I also noticed, that the evaluation currently evaluates sentences that have multiple tasks annotated to be evaluated for only one random task. -> now the evaluation uses all sentences for all tasks they are assigned to as it should be expected.

I also added the possibilities to compute the loss of a sentence on all assigned tasks during training. Hence Training Knowledge Graph Construction (NER + Relation Extraction + NEL) with a shared transformer embedding, would lead to only slightly increased training time compared to NER only, while leaveraging information of all 3 tasks at any time.

I found a bug where the embeddigns where not embedded right, as the identify_dynamic_embeddings only looks at the first sentence of the batch which could be unenbedded (e.g. for relation extraction or NEL but has no NER labels). Now the logic searches the whole batch and continues searching until it finds a batch that contains ANY embedding.

Benedikt Fuchs added 6 commits February 13, 2023 13:30
… the same sentence to the transformers embedding twice in a single training pass, shouldn't produce different results.
…gs if the first sentence was not embedded.In general it searches for the first batch that contains embeddings
@alanakbik
Copy link
Collaborator

@helpmefindaname thanks a lot for fixing and improving this!

@alanakbik alanakbik merged commit f364e0e into master Feb 14, 2023
@alanakbik alanakbik deleted the bg-3097/fix_multi_task_training branch February 14, 2023 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Multitask evaluation (and therefore training) fails on current master.
2 participants