This folder contains raw datasets and processing scripts for each language/PII feature. Datasets are either:
- scraped from an open web interface
- acquired through an API request / database copy
- sourced from various open source repositories with compatible licenses
The final datasets are processed and aggregated by generate_dataset.py. The result is stored in
aggregate/{language-code}/ds_full.json
and exported to the npm package language models:
- Dutch aggregate and npm language data
Features are implemented as Python modules, this allows features to depend on other features (e.g. when filtering out
brand-names from medicine names), this can be useful when preprocessing of one or more dataset(s) is necessary, but only
storage of the original dataset is desirable. Each module has a NAME
variable which will be used as the feature name,
as well as a get_wordlists()
function, which returns the word lists for the feature as a dictionary of dictionaries.
The main
word list contains text, which if matched, could indicate the presence of the feature. Other word lists can
be added to provide more certainty or contextual hints which might indicate presence or absence of the feature.
generate_dataset.py
- aggregates all features / datasets and exports the files to the correct locations{language-code}
- sub folder for a language- feature_sets - folder containing all "feature modules"
__init__.py
- topmost module, contains all "feature modules" in a listfeature_name/
- folder containing feature module and raw datasets__init__.py
- "feature module", containing feature dataset loading and preprocessingraw/
- raw datasets (such as a database / scripts for aggregation or hand-made lists)
- feature_templates - contains benchmark and mapping data
- feature_sets - folder containing all "feature modules"
{
"name": "pii_dataset_nl",
"version": 0,
"wordlists": {
"medicine_names": {
"main": [
"aafact",
"abacavir",
"abacavir accord",
"abacavir hexal",
...
],
...
},
...
}
}
The names, URLs, and licenses of the various open source repositories contained in this folder are listed below. The selected contents of these repositories have been included as a copy, instead of as submodules. This removes the direct dependency on the remote, which might change or be removed. Some datasets contained in these repositories have been mined from open databases or by scraping a web interface. If you are an owner of one of the datasets which is listed below and have an objection to it's use in this software, feel free to open an issue.
Lists of names are generally not copyrightable, however, since work might have gone in to either compiling or scraping this information, the licenses for these repositories are also stated and linked for further reference.