Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create span question from SDK #4617

Merged

Conversation

frascuchon
Copy link
Member

@frascuchon frascuchon commented Feb 29, 2024

Description

This PR introduces support for configuring feedback datasets with span questions. A new SpanQuestion class is defined for that. The creation workflow would be as follows:

dataset = rg.FeedbackDataset(
   fields = [rg.TextField(name="text"),...],
   questions=[rg.SpanQuestion(name="spans", title="This is the Span question", labels=["person", "org"])],
)

## You can also create label options with a different text that will shown in the annotation UI
rg.SpanQuestion(
   name="spans", 
   title="This is the Span question", 
   labels=[
      rg.SpanLabelOption(value="person", text="Person"), 
      rg.SpanLabelOption(
         value="org", 
         text="Organization", 
         description="This is a description for label org which could be useful as contextual help from the UI"
      )
   ]
)

Closes #4616

Type of change

(Please delete options that are not relevant. Remember to title the PR according to the type of change)

  • New feature (non-breaking change which adds functionality)
  • Refactor (change restructuring the codebase without changing functionality)
  • Improvement (change adding some improvement to an existing functionality)

How Has This Been Tested

(Please describe the tests that you ran to verify your changes. And ideally, reference tests)

Tested locally using a running instance of the Argilla server.

Checklist

  • I added relevant documentation
  • I followed the style guidelines of this project
  • I did a self-review of my code
  • I made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I filled out the contributor form (see text above)
  • I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Feb 29, 2024
@dosubot dosubot bot added area: python sdk Indicates that an issue or pull request is related to the Python SDK language: python Pull requests or issues that update Python code severity: minor Indicates that the issue isn't urgent or blocking team: backend Indicates that the issue or pull request is owned by the backend team type: enhancement Indicates new feature requests labels Feb 29, 2024
@frascuchon
Copy link
Member Author

@nataliaElv I've included you as a reviewer to validate the creation workflow. Let me know if you miss something.

@burtenshaw I've included you to look at the development flow for the current SDK.

Copy link

The URL of the deployed environment for this PR is https://argilla-quickstart-pr-4617-ki24f765kq-no.a.run.app

Copy link

codecov bot commented Feb 29, 2024

Codecov Report

Attention: Patch coverage is 97.87234% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 89.06%. Comparing base (254f719) to head (7d6bb0c).

Files Patch % Lines
src/argilla/client/feedback/schemas/enums.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@                       Coverage Diff                        @@
##           feat/span-questions-support    #4617       +/-   ##
================================================================
+ Coverage                        45.26%   89.06%   +43.80%     
================================================================
  Files                              190      190               
  Lines                            11671    11700       +29     
================================================================
+ Hits                              5283    10421     +5138     
+ Misses                            6388     1279     -5109     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@frascuchon frascuchon changed the title [WIP] feat: create span question from SDK feat: create span question from SDK Mar 1, 2024
@frascuchon frascuchon requested a review from jfcalvo March 1, 2024 09:41
@frascuchon frascuchon force-pushed the feat/create-span-question-from-sdk branch from 35f6392 to ceded58 Compare March 1, 2024 10:05
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Mar 1, 2024
@jfcalvo jfcalvo self-requested a review March 1, 2024 11:00
@jfcalvo
Copy link
Member

jfcalvo commented Mar 1, 2024

Looks like there is a failing integration test related with span and a dataset deletion (probably because the dataset was not created well?)

@frascuchon frascuchon merged commit 2e84de7 into feat/span-questions-support Mar 4, 2024
16 checks passed
@frascuchon frascuchon deleted the feat/create-span-question-from-sdk branch March 4, 2024 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: python sdk Indicates that an issue or pull request is related to the Python SDK language: python Pull requests or issues that update Python code lgtm This PR has been approved by a maintainer severity: minor Indicates that the issue isn't urgent or blocking size:L This PR changes 100-499 lines, ignoring generated files. team: backend Indicates that the issue or pull request is owned by the backend team type: enhancement Indicates new feature requests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants