[Bug]: UnboundLocalError: local variable 'has_unknown_label' referenced before assignment

[anna-shopova]

Describe the bug

After completing the first training epoch and attempting to evaluate on the dev data, I encounter an error. This issue arises when I set both add_unk=True and add_dev_test=True for the corpus.make_label_dictionary() method.

To Reproduce

import flair
from flair.data import Corpus
from flair.datasets import ColumnCorpus
from flair.embeddings import TransformerWordEmbeddings
from flair.models import SpanClassifier
from flair.trainers import ModelTrainer


corpus: Corpus = ColumnCorpus(data_folder, columns,
                              train_file='train.txt',
                              test_file='test.txt',
                              dev_file='dev.txt',
                              document_separator_token="-DOCSTART-")

label_dictionary = corpus.make_label_dictionary('nel', add_unk=True)
# label_dictionary = corpus.make_label_dictionary('nel', add_dev_test=True)

corpus.filter_empty_sentences()
corpus.filter_long_sentences(800)

embeddings = TransformerWordEmbeddings(
    model='xlm-roberta-base',
    layers="-1",
    subtoken_pooling="first",
    fine_tune=True,
    use_context=True
)

tagger = SpanClassifier(
    embeddings=embeddings,
    label_dictionary=label_dictionary
)

trainer = ModelTrainer(tagger, corpus)

trainer.train('./results/',
              learning_rate=5.0e-5,
              optimizer=torch.optim.AdamW,
              mini_batch_size=1,
              train_with_dev=False,
              shuffle=False
              )

Expected behavior

The model should evaluate without any errors after the training epoch.

Logs and Stack traces

File /flair/trainers/trainer.py:200, in ModelTrainer.train(self, base_path, anneal_factor, patience, min_learning_rate, initial_extra_patience, anneal_with_restarts, learning_rate, decoder_learning_rate, mini_batch_size, eval_batch_size, mini_batch_chunk_size, max_epochs, optimizer, train_with_dev, train_with_test, reduce_transformer_vocab, main_evaluation_metric, monitor_test, monitor_train_sample, use_final_model_for_eval, gold_label_dictionary_for_eval, exclude_labels, sampler, shuffle, shuffle_first_epoch, embeddings_storage_mode, epoch, save_final_model, save_optimizer_state, save_model_each_k_epochs, create_file_logs, create_loss_file, write_weights, plugins, attach_default_scheduler, **kwargs)
    189 for var in [
    190     "self",
    191     "anneal_factor",
   (...)
    197     "kwargs",
    198 ]:
    199     local_variables.pop(var)
--> 200 return self.train_custom(**local_variables, **kwargs)

File /flair/trainers/trainer.py:680, in ModelTrainer.train_custom(self, base_path, learning_rate, decoder_learning_rate, mini_batch_size, eval_batch_size, mini_batch_chunk_size, max_epochs, optimizer, train_with_dev, train_with_test, max_grad_norm, reduce_transformer_vocab, main_evaluation_metric, monitor_test, monitor_train_sample, use_final_model_for_eval, gold_label_dictionary_for_eval, exclude_labels, sampler, shuffle, shuffle_first_epoch, embeddings_storage_mode, epoch, save_final_model, save_optimizer_state, save_model_each_k_epochs, create_file_logs, create_loss_file, write_weights, use_amp, plugins, **kwargs)
    677 validation_scores: tuple
    679 for evaluation_split, evaluation_split_data in evaluation_splits.items():
--> 680     eval_result = self.model.evaluate(
    681         evaluation_split_data,
    682         out_path=base_path / f"{evaluation_split}.tsv",
    683         mini_batch_size=eval_batch_size,
    684         exclude_labels=exclude_labels,
    685         main_evaluation_metric=main_evaluation_metric,
    686         gold_label_dictionary=gold_label_dictionary_for_eval,
    687         embedding_storage_mode=embeddings_storage_mode,
    688         gold_label_type=self.model.label_type,
    689         gold_label_dictionary_for_eval=gold_label_dictionary_for_eval,
    690     )
    692     # log results
    693     log.info(
    694         f"{evaluation_split.upper()} : loss {eval_result.loss}"
    695         f" - {main_evaluation_metric[1]}"
    696         f" ({main_evaluation_metric[0]})"
    697         f"  {round(eval_result.main_score, 4)}"
    698     )

File /flair/nn/model.py:284, in Classifier.evaluate(self, data_points, gold_label_type, out_path, embedding_storage_mode, mini_batch_size, main_evaluation_metric, exclude_labels, gold_label_dictionary, return_loss, **kwargs)
    281     datapoint.remove_labels("predicted")
    283 # predict for batch
--> 284 loss_and_count = self.predict(
    285     batch,
    286     embedding_storage_mode=embedding_storage_mode,
    287     mini_batch_size=mini_batch_size,
    288     label_name="predicted",
    289     return_loss=return_loss,
    290 )
    292 if return_loss:
    293     if isinstance(loss_and_count, tuple):

File /flair/nn/model.py:911, in DefaultClassifier.predict(self, sentences, mini_batch_size, return_probabilities_for_all_classes, verbose, label_name, return_loss, embedding_storage_mode)
    908     self._post_process_batch_after_prediction(batch, label_name)
    910 if return_loss:
--> 911     if has_unknown_label:
    912         log.info(
    913             "During evaluation, encountered labels that are not in the label_dictionary:"
    914             "Evaluation loss is computed without them."
    915         )
    916     return overall_loss, label_count

UnboundLocalError: local variable 'has_unknown_label' referenced before assignment

Screenshots

No response

Additional Context

No response

Environment

Versions:

Flair

0.13.0

Pytorch

2.1.0

Transformers

4.34.1

GPU

False

[helpmefindaname]

Hi @anna-shopova
Thank you for this bug report,
I added a fix and I am talking to @alanakbik about releasing Flair 0.13.1 to include all those bugfixes