Simple Dashboard Launcher UI (To launch old dashboards)

[Nanthagopal-Eswaran]

Items to add to release announcement:

• Heading: 🚀 TruLens Dashboard Launcher

It provides a simple UI, where user can select sqlite files from previous runs and launch the TruLens Dashboard. In this way, users don't have to worry about logging the results for later and just use the trulens provided stremlit UI any time.

🕹️ Usage
Run the following command:

poetry run trulens-eval-dashboard

Features

1. Simple UI to quickly select different sqlite files and launch trulens dashboards.
   

2. Multiple dashboards can be viewed by giving different port numbers.

Open Multiple Dashboards

Currently, the tool can open only one dashboard at a time. To open multiple dashboards, quick work around is to launch this tool multiple times.

Other details that are good to know but need not be announced:

🌱 Improvements [For Future]

1. Open multiple dashboards from single window.
2. Allow sharing the streamlit dashboard with others using sharable link.
3. Build an exe to avoid the need of installing python and poetry. Also to make the tool more portable.
4. UI/UX Improvements

[Nanthagopal-Eswaran]

We have been internally using this tool since we need to open multiple dashboards side by side and compare. Thought this might be a common need for all trulens users. So sharing it here.

I know this might not be very attractive UI but the need is true.

[joshreini1]

@Nanthagopal-Eswaran would love to understand the need more here. Why do you need to compare multiple dashboards rather than logging the different apps to the same sqlite db and thus comparing the apps in the same dashboard?

[Nanthagopal-Eswaran]

@Nanthagopal-Eswaran would love to understand the need more here. Why do you need to compare multiple dashboards rather than logging the different apps to the same sqlite db and thus comparing the apps in the same dashboard?

Hi @joshreini1,

There are two main reasons,

Evaluating the changes in different versions of app
I get that we can use the same db and add. But this is an ideal case right. What if we want to compare the tests executed by different engineers / teams or if we want to automate these tests through github actions or azdo pipelines and share the db alone through mail.

Sharing the report / Re-open the report
Since it is a streamlit UI, we always need to use small snippet of python code to load previous results. I initially thought of an exe which I can share with my stakeholders, so that, they don't have to install python and dependencies to open the reports. Instead they can launch this tool and open the report.

[joshreini1]

Thanks @Nanthagopal-Eswaran - I hope you don't mind if I drill down a bit more :)

Evaluating the changes in different versions of app

I get that we can use the same db and add. But this is an ideal case right. What if we want to compare the tests executed by different engineers / teams or if we want to automate these tests through github actions or azdo pipelines and share the db alone through mail.

Comparing tests executed by different engineers/teams/automation would be better supported by using a shared database(s) to store the results than tracking a bunch of different sqlite dbs. Adding a shared database for TruLens to log to only requires passing a database URL compatible with SQLAlchemy (docs). Does this seem reasonable?

Sharing the report / Re-open the report
Since it is a streamlit UI, we always need to use small snippet of python code to load previous results. I initially thought of an exe which I can share with my stakeholders, so that, they don't have to install python and dependencies to open the reports. Instead they can launch this tool and open the report.

This use case seems reasonable, however I'm not sure it makes sense to support it directly in the package; it might be better as an internal tool. I would suggest that hosting the dashboard might be easier here (for you and the stakeholder), via an ec2 or similar.

[Nanthagopal-Eswaran]

@joshreini1, Thanks for your insights.

For first point, I get your point. We have to try how practical it is though. I am more worried about the data getting lost / corrupted by mistake and we have to spend huge cost to regenerate all the previous reports as they are in single location now. FYR, it takes more than $40 to execute one test run in our case.

And for the second point, yes it would be good to have this as a separate tool. Feel free to skip this PR. But please have an internal discussion and add this as a separate repo if really needed.

But this conversation actually made me realize the main problem here.
you can see the problem with reports being streamlit apps right?.
As a developer, I can use this tool to view previous results whenever I want. But to Stakeholders, they might not need full details. Is there a way to export the report as standalone html file (similar to pytest-html reports)? it doesn't have to have lot of features and details, atleast the leaderboard alone. This would also be helpful if we want to have automated tests and send the leaderboard alone in automated mail.

I quickly went through streamlit repo and found this issue - This clearly shows the important of standalone html reports - streamlit/streamlit#611

[joshreini1]

Thanks @Nanthagopal-Eswaran - definitely understand and agree with your points on the importance of standalone reports. I'll continue to discuss with the team and get back to you once we've got a plan here.

BTW - one additional workaround might be to use tru.get_leaderboard() in a notebook and export that to HTML.