Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the bug of incorrent result in qini for multiple models #520

Merged
merged 1 commit into from
Jun 24, 2022

Conversation

enzoliao
Copy link
Contributor

Proposed changes

Similar to #273 , the get_qini function re-uses the same df during evaluation for multiple models.
Therefore, result of the second model could be influenced by the first model.

This PR will fix the issue #519 by copying a new sorted_df in evaluation for each model.

Types of changes

What types of changes does your code introduce to CausalML?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the CONTRIBUTING doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in downstream modules

Further comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc. This PR template is adopted from appium.

@CLAassistant
Copy link

CLAassistant commented Jun 13, 2022

CLA assistant check
All committers have signed the CLA.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


enzoliao seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ppstacy
Copy link
Collaborator

ppstacy commented Jun 14, 2022

Hi @enzoliao thanks for submitting the PR! LGTM, wondering do you think if you can add the test code for this or we could open an issue to address it separately as well? T

@jeongyoonlee jeongyoonlee self-requested a review June 14, 2022 07:48
Copy link
Collaborator

@jeongyoonlee jeongyoonlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for catching this.

@enzoliao
Copy link
Contributor Author

Hi @enzoliao thanks for submitting the PR! LGTM, wondering do you think if you can add the test code for this or we could open an issue to address it separately as well? T

Hi @ppstacy . Thank you for your reviewing and it's okay for me to add some testing code. I'm wondering where should I put test code. Is it okay to add a new test_metrics.py in tests?

@enzoliao
Copy link
Contributor Author

Just add a simple test here. It will make sure that qini_score won't give different scores when calling with other models.
Probably it's better for me to open a new PR for adding tests after this one is merged?

@ppstacy
Copy link
Collaborator

ppstacy commented Jun 24, 2022

Hi @enzoliao thanks for preparing the test code. Yeah we can merge this one first.

@ppstacy ppstacy closed this Jun 24, 2022
@ppstacy ppstacy reopened this Jun 24, 2022
@ppstacy ppstacy merged commit 6437fb8 into uber:master Jun 24, 2022
@enzoliao enzoliao mentioned this pull request Jun 25, 2022
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants