Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support OpenFinData #896

Merged
merged 3 commits into from
Feb 29, 2024
Merged

[Feature] Support OpenFinData #896

merged 3 commits into from
Feb 29, 2024

Conversation

Skyfall-xzz
Copy link
Contributor

@Skyfall-xzz Skyfall-xzz commented Feb 20, 2024

Support OpenFinData Dataset.

Motivation

Add more available datasets will make Opencompass more easy to use thus obtain broader impact.

Modification

Change reading and processing configs for OpenFindata. Modifications are mostly in configs/datasets and opencompass/datasets folders.

Use cases

In evaluation scripts, add OpenFinData dataset as other datasets by using
from .datasets.OepnFinData.OpenFinData_gen import OpenFinData_datasets

Example Results

dataset                             version    metric    mode      qwen-14b-chat-hf    internlm2-chat-7b-hf
----------------------------------  ---------  --------  ------  ------------------  ----------------------
OpenFinData-emotion_identification  b64193     accuracy  gen                  85.33                   78.67
OpenFinData-entity_disambiguation   b64193     accuracy  gen                  52                      68
OpenFinData-financial_facts         b64193     accuracy  gen                  70.67                   46.67
OpenFinData-data_inspection         a846b7     accuracy  gen                  53.33                   51.67
OpenFinData-financial_terminology   a846b7     accuracy  gen                  84                      73.33
OpenFinData-metric_calculation      a846b7     accuracy  gen                  55.71                   68.57
OpenFinData-value_extraction        a846b7     accuracy  gen                  84.29                   71.43
OpenFinData-intent_understanding    f0bd9e     accuracy  gen                  88                      86.67
OpenFinData-entity_recognition      81aeeb     accuracy  gen                  68                      84

Checklist

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues.
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
  • The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
  • CLA has been signed and all committers have signed the CLA in this PR.

Copy link
Collaborator

@bittersweet1999 bittersweet1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bittersweet1999 bittersweet1999 marked this pull request as ready for review February 28, 2024 03:21
@tonysy
Copy link
Collaborator

tonysy commented Feb 28, 2024

Please add a README.md for introducing this dataset(include the data preparation, example command and results)
For example: https://github.com/open-compass/OpenCompass/blob/main/configs/datasets/IFEval/IFEval.md

@tonysy tonysy merged commit 4c45a71 into open-compass:main Feb 29, 2024
9 checks passed
Leymore pushed a commit that referenced this pull request Apr 23, 2024
* [Feature] Support OpenFinData

* add README for OpenFinData

* update README
liuyaox pushed a commit to liuyaox/opencompass that referenced this pull request Jun 26, 2024
* [Feature] Support OpenFinData

* add README for OpenFinData

* update README
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants