[Feature] Add Cleanlab's Trustworthiness Score

[AshishSardana]

Cleanlab provides trustworthiness score that can be used for guardrailing tasks. Read more about how this score is calculated here.

This PR aims to add support for a 3rd party API i.e. Cleanlab Studio's API, to enable users to use trustworthiness score for using guardrails on output.

[AshishSardana]

Hi @drazvan, I'd like to add brief documentation about trustworthiness score and configurable parameter in the rail (threshold on least_trustworthiness_score).
Would it go here?

Also, how do I add tests? Do you store the keys on the CI server to be able to run the tests?

[drazvan]

@AshishSardana : Yes, you can add it in the library documentation. We are in the process of refactoring the structure a bit, but we'll change this as well once that's finalized.

[nelsonauner]

Please take this review with a grain of salt. My comments are mainly to make our documentation more clear and succinct. I also try to facilitate usage of Cleanlab by using phrases like "It's easy to get started by doing X, Y and Z" instead of "It is required to do X if you want to use cleanlab"

I have some comments that are really more a preference, which I've prefaced with "Nit"

[AshishSardana]

I'm not sure how the commit history got duplicated with my last a7517e3 commit.

No new changes are introduced in these duplicate commits.

[AshishSardana]

Hi @drazvan, the PR is ready for review. Could you please give it a look?

[AshishSardana]

Hi @Pouyanpi , I've resolved the conflicts.

[AshishSardana]

Hi @Pouyanpi, please feel free to ask any clarifying questions!

[drazvan]

@AshishSardana, I'm just reviewing this, apologies for the delay. The code looks good, however, when I try to run it, I'm running into an error coming from the CleanLab python client.
Steps to reproduce:

1. Edit the config.yml file for the examples/bots/abc bot:

rails:
#  input:
#    flows:
#      - self check input

  output:
    flows:
      - cleanlab trustworthiness

(comment the self-check input and replace the self-check output with cleanlab trustworthiness).

2. Start the chat CLI nemoguardrails chat --config=examples/bots/abc --verbose.

3. Say "hi"

Relevant error:

Traceback (most recent call last):
  File ".../nemoguardrails/nemoguardrails/actions/action_dispatcher.py", line 197, in execute_action
    result = await result
  File ".../nemoguardrails/nemoguardrails/library/cleanlab/actions.py", line 48, in call_cleanlab_api
    trustworthiness_score = await cleanlab_tlm.get_trustworthiness_score_async(
  File ".../nemoguardrails/venv/lib/python3.10/site-packages/cleanlab_studio/studio/trustworthy_language_model.py", line 559, in get_trustworthiness_score_async
    assert isinstance(prompt, Sequence) and isinstance(processed_response, Sequence)
AssertionError

From what I can see, the processed_response is actually a dict like {"response": "..."}.

Let me know if you're able to reproduce

[AshishSardana]

@drazvan I was able to reproduce this error. We've pushed a fix to the cleanlab-studio package that you can fetch by pip install -U cleanlab-studio which resolves this error. Could you try?

[drazvan]

I was able to test and looks good 👍. Latency was a bit high in my tests, ~5s, but I guess that will be improved.