Show OSS models (and tracking) in LiteLLM application

[joshreini1]

Items to add to release announcement:

• Show TruLens tracking of OSS models in core LLM app

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[daniel-huang-1230]

Looks great to me!

Could we also add a link to https://litellm.vercel.app/docs/completion/token_usage#5-completion_cost perhaps on the top in the markdown description so that users can refer to the model cost setup and configurability within LiteLLM and down the road see them in TruLens dashboard?

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

<sup><sub>🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!</sub></sup>

[joshreini1]

Looks great to me!

Could we also add a link to https://litellm.vercel.app/docs/completion/token_usage#5-completion_cost perhaps on the top in the markdown description so that users can refer to the model cost setup and configurability within LiteLLM and down the road see them in TruLens dashboard?

We already use litellm.completion_cost in the background and get the cost tracking, part of what I'm trying to show in this change. Adding the leaderboard output to show this. Would it still be useful to link to the litellm cost references?

Here's the line reference:

trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/litellm.py

Line 61 in 46d362e

setattr(self.cost, "cost", completion_cost(response))

[daniel-huang-1230]

Looks great to me!
Could we also add a link to https://litellm.vercel.app/docs/completion/token_usage#5-completion_cost perhaps on the top in the markdown description so that users can refer to the model cost setup and configurability within LiteLLM and down the road see them in TruLens dashboard?

We already use litellm.completion_cost in the background and get the cost tracking, part of what I'm trying to show in this change. Adding the leaderboard output to show this. Would it still be useful to link to the litellm cost references?

Here's the line reference:

trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/litellm.py

Line 61 in 46d362e

setattr(self.cost, "cost", completion_cost(response))

Cool! yes the added cell output looks good to me and should be plenty clear for the user hopefully.
I'm aware the usage of completion_cost and also the motivation for this change 👍 (Originally was thinking about adding the link just because I've encountered user recently on Github asking about cost-tracking with LiteLLM)

[joshreini1]

Looks great to me!
Could we also add a link to https://litellm.vercel.app/docs/completion/token_usage#5-completion_cost perhaps on the top in the markdown description so that users can refer to the model cost setup and configurability within LiteLLM and down the road see them in TruLens dashboard?

We already use litellm.completion_cost in the background and get the cost tracking, part of what I'm trying to show in this change. Adding the leaderboard output to show this. Would it still be useful to link to the litellm cost references?
Here's the line reference:

trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/litellm.py

Line 61 in 46d362e

setattr(self.cost, "cost", completion_cost(response))

Cool! yes the added cell output looks good to me and should be plenty clear for the user hopefully. I'm aware the usage of completion_cost and also the motivation for this change 👍 (Originally was thinking about adding the link just because I've encountered user recently on Github asking about cost-tracking with LiteLLM)

Ahh - I see the gh issue where they're looking to track a model not in this list. Added the link to this list in the header to make this clear!