FactScoreLite is an implementation of the FactScore metric, designed for detailed accuracy assessment in text generation. This package builds upon the framework provided by the original FactScore repository, which is no longer maintained and contains outdated functions.
Our development aims to address these shortcomings by updating the code and ensuring compatibility with current technologies. This makes FactScoreLite a reliable tool for anyone looking to evaluate textual accuracy in a minimal set up.
Since the project is using OpenAI APIs, make sure that you have set up the API key before running any code. For instructions refer to OpenAI documentation.
Make sure you have Python version is >= 3.10.
You can install this package using pip:
pip install factscorelite
or you can install it directly by cloning and installing:
git clone https://github.com/armingh2000/FactScoreLite.git
cd FactSoreLite
pip install .
The package contains three main components:
- AtomicFactGenerator: generating facts for a text.
# atomic_facts.py
class AtomicFactGenerator:
def run(self, text: str) -> list:
- FactScorer: scoring facts of a text based on a knowledge source.
# fact_scorer.py
class FactScorer:
def get_score(self, facts: list, knowledge_source: str) -> list
- FactScore:
- Generating the facts for a text.
- Scoring the facts based on a knowledge source.
- Dumping the results and GPT outputs to a local json file.
# factscore.py
class FactScore:
def get_factscore(
self,
generations: list,
knowledge_sources: list,
) -> tuple
To extract facts of a text and score them based on the input knowledge source and dump the results:
from FactScoreLite import FactScore
# generations = a list of texts you want to calculate FactScore for
# knowledge_sources = a list of texts that the generations were created from
scores, init_scores = FactScore.get_factscore(generations, knowledge_sources)
To only extract the facts from a text (without scoring/dumping):
from FactScoreLite import AtomicFactGenerator
facts = AtomicFactGenerator.run(text)
To only score the facts of a generation according to a knowledge source (wihtout dumping):
from FactScoreLite import FactScorer
scores = FactScorer.get_scores(facts, knowledge_sources)
To instruct GPT on how to break each sentence into facts, we have included examples (demonstrations, i.e., demons) that is contained in the prompt. These demons are currently for the vehicle domain. However, you might want to create your own domain specific demons. To do this, you can use GPT to create demons based on your requirements. We prompted GPT with instructions on how to generate the demons required for the vehicle domain. However, you can alter it based on your needs.
Once you have your own demons.json file, you can include it in the program by setting the correct config:
import FactScoreLite
FactScoreLite.configs.atomic_facts_demons_path = "/path/to/your/json/file"
# rest of your code
The prompt used for extracting facts from a sentence:
# atomic_facts.py
Please breakdown the following sentence into independent facts:
Sentence:
demon1_sentence
Independent Facts:
- demon1_fact1
- demon1_fact2
- demon1_fact3
Sentence:
demon2_sentence
Independent Facts:
- demon2_fact1
- demon2_fact2
Sentence:
target_sentence
Independent Facts:
We also use example demonstrations for scoring instructions prompt. The file contains one positive and multiple negative examples. In each prompt, the positive example in addition to a randomly selected negative prompt is added so that GPT performs better and more accurately. The file also contains reasons for each assignment; However, they are not used in the prompt generation but is a good way of improving the accuracy of GPT on scoring in the future.
You can also set your own domain-specific examples for the run by running the following:
import FactScoreLite
FactScoreLite.configs.fact_scorer_demons_path = "/path/to/your/json/file"
# rest of your code
The following prompt template is used to instruct GPT for scoring facts:
# fact_scorer.py
Instruction:
Only consider the statement true if it can be directly verified by the information in the context. If the information in the statement cannot be found in the context or differs from it, label it as false.
Context:
knw 1
Statement:
fact 1 True or False?
Output:
True
Context:
knw 2
Statement:
fact 2 True or False?
Output:
False
Context:
target_knowledge_source
Statement:
target_fact True or False?
Output:
If you want to change the source code for your use cases, you can check whether the change conflicts with other parts of the projcet by simply running the tests:
FactScoreLite/
pytest
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
We use SemVer for versioning. For the versions available, see CHANGELOG.md.
This project is licensed under the MIT License - see the LICENSE.md file for details