Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE-REQUEST] Bias measurements for token-classification supertask. #103

Open
AJDERS opened this issue Feb 24, 2023 · 0 comments
Open
Labels
enhancement New feature or request new-dataset

Comments

@AJDERS
Copy link
Collaborator

AJDERS commented Feb 24, 2023

Is your feature request related to a problem? Please describe.
Introduce a new metric bias and corresponding evaluation-dataset which should quantify the intrinsic bias of different token-classification-models. The different demographical features for which bias should be quantified are (at least):

  • Gender
    • Male
    • Female
    • Transgender
    • Non-binary
  • Ethnicity
    • Caucasian
    • Non-caucasian
  • Religion
    • Christianity
    • Islam
    • Judaism
  • Sexuality
    • Heterosexuality
    • Homosexuality
    • Bisexuality

Describe the solution you'd like
This could be a synthetically produced dataset created from an already existing dataset. As an example for religion the dataset could include a set of sentences which includes institutions, names and places which has some kind of religious context which are then varied, and an indicator for which demographic the sentence belongs to, i.e.

("Islam", "Hun er præst i den lokale kirke"),
("Christianity", "Hun er imam i den lokale moske")
("Judaism", "Hun er rabbiner i den lokale synagoge")
...

This is largely an open question, so give it a go!

@AJDERS AJDERS added enhancement New feature or request new-dataset labels Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request new-dataset
Projects
None yet
Development

No branches or pull requests

1 participant