datasets: add possibility to use custom preprocessing function for HIPE-2022

[stefan-it]

Hi,

this PR adds the possibility to use a custom preprocessing function for the HIPE-2022.

This would e.g. allow to use custom de-hyphenation or label normalization extensions defined in the preprocessing function.

[stefan-it]

When the dataset already exists, and an own preprocessing is used afterwards, please make sure, that the old base path is deleted.

The following code shows how to use an own preprocessing function:

from flair.datasets import NER_HIPE_2022

from pathlib import Path

def own_prepare_corpus(
    file_in: Path, file_out: Path, eos_marker: str, document_separator: str, add_document_separator: bool
):
    with open(file_in, "rt") as f_p:
        lines = f_p.readlines()

    with open(file_out, "wt") as f_out:
        # Add missing newline after header
        f_out.write(lines[0] + "\n")

        for line in lines[1:]:
            if line.startswith(" \t"):
                # Workaround for empty tokens
                continue

            line = line.strip()

            # Add "real" document marker
            if add_document_separator and line.startswith(document_separator):
                f_out.write("-DOCSTART- O\n\n")

            f_out.write(line + "\n")

            if eos_marker in line:
                f_out.write("\n")
    
    print("Own function used, awesome!!!")

corpus = NER_HIPE_2022(dataset_name="ajmc", language="en", preproc_fn=own_prepare_corpus)

[stefan-it]

Here's a real-world example that performs de-hyphenation of the HIPE-2020, NewsEye and SONAR datasets using an own preprocessing function:

https://github.com/dbmdz/clef-hipe/blob/main/experiments/clef-hipe-2022/utils.py

[alanakbik]

Thanks for adding this @stefan-it!