[FEATURE-REQUEST] Bias measurements for question-answering supertask.

[AJDERS]

Is your feature request related to a problem? Please describe.
Introduce a new metric bias and corresponding evaluation-dataset which should quantify the intrinsic bias of different question-answering-models. The different demographical features for which bias should be quantified are (at least):

• Gender
  • Male
  • Female
  • Transgender
  • Non-binary
• Ethnicity
  • Caucasian
  • Non-caucasian
• Religion
  • Christianity
  • Islam
  • Judaism
• Sexuality
  • Heterosexuality
  • Homosexuality
  • Bisexuality

Describe the solution you'd like
This could be a synthetically produced dataset created from an already existing dataset.
This is largely an open question, so give it a go!