insertdataMango.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MongoDB Atlas Quickstart\n",
    "\n",
    "[MongoDB Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) is part of the MongoDB platform that enables MongoDB customers to build intelligent applications powered by semantic search over any type of data. Atlas Vector Search allows you to integrate your operational database and vector search in a single, unified, fully managed platform with full vector database capabilities.\n",
    "\n",
    "You can integrate TruLens with your application built on Atlas Vector Search to leverage observability and measure improvements in your application's search capabilities.\n",
    "\n",
    "This tutorial will walk you through the process of setting up TruLens with MongoDB Atlas Vector Search and Llama-Index as the orchestrator.\n",
    "\n",
    "Even better, you'll learn how to use metadata filters to create specialized query engines and leverage a router to choose the most appropriate query engine based on the query.\n",
    "\n",
    "See [MongoDB Atlas/LlamaIndex Quickstart](https://www.mongodb.com/docs/atlas/atlas-vector-search/ai-integrations/llamaindex/) for more details.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/expositional/vector-dbs/mongodb_atlas/atlas_quickstart.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Requirement already satisfied: trulens-eval in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (0.28.2)\n",
      "Requirement already satisfied: llama-index in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (0.9.34)\n",
      "Requirement already satisfied: llama-index-vector-stores-mongodb in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (0.1.4)\n",
      "Requirement already satisfied: llama-index-embeddings-openai in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (0.1.9)\n",
      "Requirement already satisfied: pymongo in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (4.7.0)\n",
      "Requirement already satisfied: ipython>=8.12.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (8.24.0)\n",
      "Collecting ipywidgets>=8.0.6\n",
      "  Downloading ipywidgets-8.1.2-py3-none-any.whl.metadata (2.4 kB)\n",
      "Requirement already satisfied: numpy>=1.23.5 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (1.26.4)\n",
      "Requirement already satisfied: frozendict>=2.3.8 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (2.4.2)\n",
      "Requirement already satisfied: munch>=3.0.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (4.0.0)\n",
      "Requirement already satisfied: dill>=0.3.7 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (0.3.8)\n",
      "Requirement already satisfied: tqdm>=4.66.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (4.66.2)\n",
      "Requirement already satisfied: nltk>=3.8.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (3.8.1)\n",
      "Requirement already satisfied: requests>=2.31.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (2.31.0)\n",
      "Requirement already satisfied: nest-asyncio>=1.5.8 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (1.6.0)\n",
      "Requirement already satisfied: typing-extensions>=4.9.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (4.11.0)\n",
      "Requirement already satisfied: psutil>=5.9.8 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (5.9.8)\n",
      "Requirement already satisfied: pip>=24.0 in c:\\program files\\windowsapps\\pythonsoftwarefoundation.python.3.12_3.12.1008.0_x64__qbz5n2kfra8p0\\lib\\site-packages (from trulens-eval) (24.0)\n",
      "Requirement already satisfied: packaging>=23.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (23.2)\n",
      "Requirement already satisfied: python-dotenv>=1.0.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (1.0.1)\n",
      "Requirement already satisfied: pydantic<3,>=2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (2.7.1)\n",
      "Requirement already satisfied: merkle-json>=1.0.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (1.0.0)\n",
      "Requirement already satisfied: langchain>=0.1.14 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (0.1.16)\n",
      "Requirement already satisfied: langchain-core>=0.1.6 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (0.1.46)\n",
      "Requirement already satisfied: typing-inspect>=0.8.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (0.9.0)\n",
      "Requirement already satisfied: millify>=0.1.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (0.1.1)\n",
      "Requirement already satisfied: humanize>=4.6.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (4.9.0)\n",
      "Requirement already satisfied: streamlit>=1.32.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (1.34.0)\n",
      "Requirement already satisfied: streamlit-aggrid==0.3.4 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (0.3.4)\n",
      "Requirement already satisfied: streamlit-extras>=0.4.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (0.4.2)\n",
      "Requirement already satisfied: streamlit-pills>=0.3.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (0.3.0)\n",
      "Requirement already satisfied: rich>=13.6.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (13.7.1)\n",
      "Requirement already satisfied: sqlalchemy>=2.0.19 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (2.0.29)\n",
      "Requirement already satisfied: alembic>=1.11.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trulens-eval) (1.13.1)\n",
      "Requirement already satisfied: pandas>=1.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-aggrid==0.3.4->trulens-eval) (2.2.2)\n",
      "Requirement already satisfied: python-decouple<4.0,>=3.6 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-aggrid==0.3.4->trulens-eval) (3.8)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (3.9.5)\n",
      "Requirement already satisfied: beautifulsoup4<5.0.0,>=4.12.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (4.12.3)\n",
      "Requirement already satisfied: dataclasses-json in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (0.6.4)\n",
      "Requirement already satisfied: deprecated>=1.2.9.3 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (1.2.14)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (2024.3.1)\n",
      "Requirement already satisfied: httpx in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (0.27.0)\n",
      "Requirement already satisfied: networkx>=3.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (3.3)\n",
      "Requirement already satisfied: openai>=1.1.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (1.23.6)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (8.2.3)\n",
      "Requirement already satisfied: tiktoken>=0.3.3 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index) (0.6.0)\n",
      "Requirement already satisfied: llama-index-core<0.11.0,>=0.10.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index-vector-stores-mongodb) (0.10.34)\n",
      "Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from pymongo) (2.6.1)\n",
      "Requirement already satisfied: decorator in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipython>=8.12.0) (5.1.1)\n",
      "Requirement already satisfied: jedi>=0.16 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipython>=8.12.0) (0.19.1)\n",
      "Requirement already satisfied: matplotlib-inline in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipython>=8.12.0) (0.1.7)\n",
      "Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipython>=8.12.0) (3.0.43)\n",
      "Requirement already satisfied: pygments>=2.4.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipython>=8.12.0) (2.17.2)\n",
      "Requirement already satisfied: stack-data in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipython>=8.12.0) (0.6.3)\n",
      "Requirement already satisfied: traitlets>=5.13.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipython>=8.12.0) (5.14.3)\n",
      "Requirement already satisfied: colorama in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipython>=8.12.0) (0.4.6)\n",
      "Requirement already satisfied: comm>=0.1.3 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ipywidgets>=8.0.6) (0.2.2)\n",
      "Collecting widgetsnbextension~=4.0.10 (from ipywidgets>=8.0.6)\n",
      "  Downloading widgetsnbextension-4.0.10-py3-none-any.whl.metadata (1.6 kB)\n",
      "Collecting jupyterlab-widgets~=3.0.10 (from ipywidgets>=8.0.6)\n",
      "  Downloading jupyterlab_widgets-3.0.10-py3-none-any.whl.metadata (4.1 kB)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (23.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (6.0.5)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (1.9.4)\n",
      "Requirement already satisfied: Mako in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from alembic>=1.11.2->trulens-eval) (1.3.3)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from beautifulsoup4<5.0.0,>=4.12.2->llama-index) (2.5)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from deprecated>=1.2.9.3->llama-index) (1.16.0)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.3 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from jedi>=0.16->ipython>=8.12.0) (0.8.4)\n",
      "Requirement already satisfied: PyYAML>=5.3 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from langchain>=0.1.14->trulens-eval) (6.0.1)\n",
      "Requirement already satisfied: jsonpatch<2.0,>=1.33 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from langchain>=0.1.14->trulens-eval) (1.33)\n",
      "Requirement already satisfied: langchain-community<0.1,>=0.0.32 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from langchain>=0.1.14->trulens-eval) (0.0.34)\n",
      "Requirement already satisfied: langchain-text-splitters<0.1,>=0.0.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from langchain>=0.1.14->trulens-eval) (0.0.1)\n",
      "Requirement already satisfied: langsmith<0.2.0,>=0.1.17 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from langchain>=0.1.14->trulens-eval) (0.1.51)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from dataclasses-json->llama-index) (3.21.1)\n",
      "Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-vector-stores-mongodb) (1.0.8)\n",
      "Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-vector-stores-mongodb) (0.1.19)\n",
      "Requirement already satisfied: pillow>=9.0.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-vector-stores-mongodb) (10.3.0)\n",
      "Requirement already satisfied: click in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from nltk>=3.8.1->trulens-eval) (8.1.7)\n",
      "Requirement already satisfied: joblib in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from nltk>=3.8.1->trulens-eval) (1.4.0)\n",
      "Requirement already satisfied: regex>=2021.8.3 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from nltk>=3.8.1->trulens-eval) (2024.4.16)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from openai>=1.1.0->llama-index) (4.3.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from openai>=1.1.0->llama-index) (1.9.0)\n",
      "Requirement already satisfied: sniffio in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from openai>=1.1.0->llama-index) (1.3.1)\n",
      "Requirement already satisfied: certifi in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from httpx->llama-index) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from httpx->llama-index) (1.0.5)\n",
      "Requirement already satisfied: idna in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from httpx->llama-index) (3.7)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from httpcore==1.*->httpx->llama-index) (0.14.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from pandas>=1.2->streamlit-aggrid==0.3.4->trulens-eval) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from pandas>=1.2->streamlit-aggrid==0.3.4->trulens-eval) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from pandas>=1.2->streamlit-aggrid==0.3.4->trulens-eval) (2024.1)\n",
      "Requirement already satisfied: wcwidth in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=8.12.0) (0.2.13)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from pydantic<3,>=2->trulens-eval) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.18.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from pydantic<3,>=2->trulens-eval) (2.18.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests>=2.31.0->trulens-eval) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests>=2.31.0->trulens-eval) (2.2.1)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from rich>=13.6.0->trulens-eval) (3.0.0)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sqlalchemy>=2.0.19->trulens-eval) (3.0.3)\n",
      "Requirement already satisfied: altair<6,>=4.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (5.3.0)\n",
      "Requirement already satisfied: blinker<2,>=1.0.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (1.8.0)\n",
      "Requirement already satisfied: cachetools<6,>=4.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (5.3.3)\n",
      "Requirement already satisfied: protobuf<5,>=3.20 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (4.25.3)\n",
      "Requirement already satisfied: pyarrow>=7.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (16.0.0)\n",
      "Requirement already satisfied: toml<2,>=0.10.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (0.10.2)\n",
      "Requirement already satisfied: gitpython!=3.1.19,<4,>=3.0.7 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (3.1.43)\n",
      "Requirement already satisfied: pydeck<1,>=0.8.0b4 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (0.9.0)\n",
      "Requirement already satisfied: tornado<7,>=6.0.3 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (6.4)\n",
      "Requirement already satisfied: watchdog>=2.1.5 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit>=1.32.2->trulens-eval) (4.0.0)\n",
      "Requirement already satisfied: entrypoints>=0.4 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.4)\n",
      "Requirement already satisfied: htbuilder>=0.6.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.6.2)\n",
      "Requirement already satisfied: markdownlit>=0.0.5 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.0.7)\n",
      "Requirement already satisfied: prometheus-client>=0.14.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.20.0)\n",
      "Requirement already satisfied: st-annotated-text>=3.0.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (4.0.1)\n",
      "Requirement already satisfied: streamlit-camera-input-live>=0.2.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.2.0)\n",
      "Requirement already satisfied: streamlit-card>=0.0.4 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (1.0.0)\n",
      "Requirement already satisfied: streamlit-embedcode>=0.1.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.1.2)\n",
      "Requirement already satisfied: streamlit-faker>=0.0.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.0.3)\n",
      "Requirement already satisfied: streamlit-image-coordinates<0.2.0,>=0.1.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.1.6)\n",
      "Requirement already satisfied: streamlit-keyup>=0.1.9 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.2.4)\n",
      "Requirement already satisfied: streamlit-toggle-switch>=1.0.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (1.0.2)\n",
      "Requirement already satisfied: streamlit-vertical-slider>=2.5.5 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (2.5.5)\n",
      "Requirement already satisfied: validators>=0.20.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-extras>=0.4.0->trulens-eval) (0.28.1)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from typing-inspect>=0.8.0->trulens-eval) (1.0.0)\n",
      "Requirement already satisfied: executing>=1.2.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from stack-data->ipython>=8.12.0) (2.0.1)\n",
      "Requirement already satisfied: asttokens>=2.1.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from stack-data->ipython>=8.12.0) (2.4.1)\n",
      "Requirement already satisfied: pure-eval in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from stack-data->ipython>=8.12.0) (0.2.2)\n",
      "Requirement already satisfied: jinja2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from altair<6,>=4.0->streamlit>=1.32.2->trulens-eval) (3.1.3)\n",
      "Requirement already satisfied: jsonschema>=3.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from altair<6,>=4.0->streamlit>=1.32.2->trulens-eval) (4.22.0)\n",
      "Requirement already satisfied: toolz in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from altair<6,>=4.0->streamlit>=1.32.2->trulens-eval) (0.12.1)\n",
      "Requirement already satisfied: six>=1.12.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from asttokens>=2.1.0->stack-data->ipython>=8.12.0) (1.16.0)\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from gitpython!=3.1.19,<4,>=3.0.7->streamlit>=1.32.2->trulens-eval) (4.0.11)\n",
      "Requirement already satisfied: more-itertools in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from htbuilder>=0.6.2->streamlit-extras>=0.4.0->trulens-eval) (10.2.0)\n",
      "Requirement already satisfied: jsonpointer>=1.9 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from jsonpatch<2.0,>=1.33->langchain>=0.1.14->trulens-eval) (2.4)\n",
      "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from langsmith<0.2.0,>=0.1.17->langchain>=0.1.14->trulens-eval) (3.10.1)\n",
      "Requirement already satisfied: mdurl~=0.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from markdown-it-py>=2.2.0->rich>=13.6.0->trulens-eval) (0.1.2)\n",
      "Requirement already satisfied: markdown in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from markdownlit>=0.0.5->streamlit-extras>=0.4.0->trulens-eval) (3.6)\n",
      "Requirement already satisfied: lxml in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from markdownlit>=0.0.5->streamlit-extras>=0.4.0->trulens-eval) (5.2.1)\n",
      "Requirement already satisfied: favicon in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from markdownlit>=0.0.5->streamlit-extras>=0.4.0->trulens-eval) (0.7.0)\n",
      "Requirement already satisfied: pymdown-extensions in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from markdownlit>=0.0.5->streamlit-extras>=0.4.0->trulens-eval) (10.8.1)\n",
      "Requirement already satisfied: faker in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-faker>=0.0.2->streamlit-extras>=0.4.0->trulens-eval) (25.0.1)\n",
      "Requirement already satisfied: matplotlib in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from streamlit-faker>=0.0.2->streamlit-extras>=0.4.0->trulens-eval) (3.8.4)\n",
      "Requirement already satisfied: MarkupSafe>=0.9.2 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from Mako->alembic>=1.11.2->trulens-eval) (2.1.5)\n",
      "Requirement already satisfied: smmap<6,>=3.0.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from gitdb<5,>=4.0.1->gitpython!=3.1.19,<4,>=3.0.7->streamlit>=1.32.2->trulens-eval) (5.0.1)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=1.32.2->trulens-eval) (2023.12.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=1.32.2->trulens-eval) (0.35.1)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=1.32.2->trulens-eval) (0.18.0)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.4.0->trulens-eval) (1.2.1)\n",
      "Requirement already satisfied: cycler>=0.10 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.4.0->trulens-eval) (0.12.1)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.4.0->trulens-eval) (4.51.0)\n",
      "Requirement already satisfied: kiwisolver>=1.3.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.4.0->trulens-eval) (1.4.5)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in c:\\users\\msaad\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.4.0->trulens-eval) (3.1.2)\n",
      "Downloading ipywidgets-8.1.2-py3-none-any.whl (139 kB)\n",
      "   ---------------------------------------- 0.0/139.4 kB ? eta -:--:--\n",
      "   -- ------------------------------------- 10.2/139.4 kB ? eta -:--:--\n",
      "   -------- ------------------------------ 30.7/139.4 kB 660.6 kB/s eta 0:00:01\n",
      "   -------------------------------- ------- 112.6/139.4 kB 1.1 MB/s eta 0:00:01\n",
      "   ---------------------------------------- 139.4/139.4 kB 1.2 MB/s eta 0:00:00\n",
      "Downloading jupyterlab_widgets-3.0.10-py3-none-any.whl (215 kB)\n",
      "   ---------------------------------------- 0.0/215.0 kB ? eta -:--:--\n",
      "   -------------------- ------------------- 112.6/215.0 kB 3.3 MB/s eta 0:00:01\n",
      "   ---------------------------------------- 215.0/215.0 kB 2.2 MB/s eta 0:00:00\n",
      "Downloading widgetsnbextension-4.0.10-py3-none-any.whl (2.3 MB)\n",
      "   ---------------------------------------- 0.0/2.3 MB ? eta -:--:--\n",
      "   -- ------------------------------------- 0.1/2.3 MB 4.2 MB/s eta 0:00:01\n",
      "   -- ------------------------------------- 0.2/2.3 MB 2.0 MB/s eta 0:00:02\n",
      "   ------ --------------------------------- 0.4/2.3 MB 3.0 MB/s eta 0:00:01\n",
      "   ------- -------------------------------- 0.4/2.3 MB 2.8 MB/s eta 0:00:01\n",
      "   ---------- ----------------------------- 0.6/2.3 MB 2.7 MB/s eta 0:00:01\n",
      "   ------------ --------------------------- 0.7/2.3 MB 2.7 MB/s eta 0:00:01\n",
      "   -------------- ------------------------- 0.8/2.3 MB 2.6 MB/s eta 0:00:01\n",
      "   ---------------- ----------------------- 0.9/2.3 MB 2.6 MB/s eta 0:00:01\n",
      "   ------------------ --------------------- 1.0/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   -------------------- ------------------- 1.2/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   ---------------------- ----------------- 1.3/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   ------------------------ --------------- 1.4/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   ------------------------- -------------- 1.5/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   ---------------------------- ----------- 1.6/2.3 MB 2.4 MB/s eta 0:00:01\n",
      "   ----------------------------- ---------- 1.7/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   ------------------------------- -------- 1.8/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   --------------------------------- ------ 1.9/2.3 MB 2.4 MB/s eta 0:00:01\n",
      "   ----------------------------------- ---- 2.0/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   ------------------------------------- -- 2.2/2.3 MB 2.5 MB/s eta 0:00:01\n",
      "   ---------------------------------------  2.3/2.3 MB 2.4 MB/s eta 0:00:01\n",
      "   ---------------------------------------- 2.3/2.3 MB 2.4 MB/s eta 0:00:00\n",
      "Installing collected packages: widgetsnbextension, jupyterlab-widgets, ipywidgets\n",
      "Successfully installed ipywidgets-8.1.2 jupyterlab-widgets-3.0.10 widgetsnbextension-4.0.10\n"
     ]
    }
   ],
   "source": [
    "!pip install trulens-eval llama-index llama-index-vector-stores-mongodb llama-index-embeddings-openai pymongo \"ipython>=8.12.0\" \"ipywidgets>=8.0.6\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import TruLens and start the dashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using legacy llama_index version 0.9.34. Consider upgrading to 0.10.0 or later.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🦑 Tru initialized with db url sqlite:///default.sqlite .\n",
      "🛑 Secret keys may be written to the database. See the `database_redact_keys` option of Tru` to prevent this.\n",
      "Starting dashboard ...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20cbb87b24e74a78ad0cb83b924d7d2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Accordion(children=(VBox(children=(VBox(children=(Label(value='STDOUT'), Output())), VBox(children=(Label(valu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dashboard started at http://192.168.1.11:8501 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Popen: returncode: None args: ['streamlit', 'run', '--server.headless=True'...>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from trulens_eval import Tru\n",
    "\n",
    "tru = Tru()\n",
    "\n",
    "tru.reset_database()\n",
    "\n",
    "tru.run_dashboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set imports, keys and llama-index settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass, os, pymongo, pprint\n",
    "from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, StorageContext\n",
    "from llama_index.core.settings import Settings\n",
    "from llama_index.core.retrievers import VectorIndexRetriever\n",
    "from llama_index.core.vector_stores import MetadataFilter, MetadataFilters, ExactMatchFilter, FilterOperator\n",
    "from llama_index.core.query_engine import RetrieverQueryEngine\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "from llama_index.llms.openai import OpenAI\n",
    "from llama_index.vector_stores.mongodb import MongoDBAtlasVectorSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# importing necessary functions from dotenv library\n",
    "from dotenv import load_dotenv, dotenv_values \n",
    "# loading variables from .env file\n",
    "load_dotenv() \n",
    "OPENAI_API_KEY=os.getenv(\"OPENAI_API_KEY\")\n",
    "mongo_key=os.getenv(\"mongo_key\")\n",
    "mongo_pass=os.getenv(\"mongo_pass\")\n",
    "from urllib.parse import quote_plus\n",
    "\n",
    "username = quote_plus(mongo_key)\n",
    "password = quote_plus(mongo_pass)\n",
    "ATLAS_CONNECTION_STRING = 'mongodb+srv://' + username + ':' + password + '@datacluster.hjhs3xb.mongodb.net/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "Settings.llm = OpenAI()\n",
    "Settings.embed_model = OpenAIEmbedding(model=\"text-embedding-ada-002\")\n",
    "Settings.chunk_size = 100\n",
    "Settings.chunk_overlap = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sample data\n",
    "\n",
    "Here we'll load two PDFs: one for Atlas best practices and one textbook on database essentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from llama_index import download_loader\n",
    "PDFReader = download_loader(\"PDFReader\")\n",
    "loader = PDFReader()\n",
    "\n",
    "# Load HNSW PDF from LFS\n",
    "documents = loader.load_data(file=Path(\"docs/Seeratul Mustafa Abridged.pdf\"))\n",
    "\n",
    "documents1 = loader.load_data(file=Path(\"docs/Ar-Raheeq_Al-Makhtum_(The_Sealed_Nectar)_English_(www.TheChoice.one)_text.pdf\"))\n",
    "documents2 = loader.load_data(file=Path(\"docs/Seerat Ibn e Hisham - English Translation (1st Edition)_text.pdf\"))\n",
    "documents3 = loader.load_data(file=Path(\"docs/Seerat Ibn e Hisham - English Translation (2nd Edition)_text.pdf\"))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc ID: 2b62cee6-ebd8-4cfc-a2ab-6428008683f0\n",
      "Text: اَ    it    04    04   Better than you no eye has ever seen\n",
      "More beautiful than you no woman has given birth to   You have been\n",
      "created free from any defect  As if you were created like how you\n",
      "desired\n",
      "Seeratul Mustafa    \n"
     ]
    }
   ],
   "source": [
    "print(documents[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "all_documents = documents + documents1 + documents2 + documents3\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a vector store\n",
    "\n",
    "Next you need to create an Atlas Vector Search Index.\n",
    "\n",
    "When you do so, use the following in the json editor:\n",
    "\n",
    "```\n",
    "{\n",
    "  \"fields\": [\n",
    "    {\n",
    "      \"numDimensions\": 1536,\n",
    "      \"path\": \"embedding\",\n",
    "      \"similarity\": \"cosine\",\n",
    "      \"type\": \"vector\"\n",
    "    },\n",
    "    {\n",
    "      \"path\": \"metadata.file_name\",\n",
    "      \"type\": \"filter\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to your Atlas cluster\n",
    "mongodb_client = pymongo.MongoClient(ATLAS_CONNECTION_STRING)\n",
    "\n",
    "# Instantiate the vector store\n",
    "atlas_vector_search = MongoDBAtlasVectorSearch(\n",
    "    mongodb_client,\n",
    "    db_name = \"books\",\n",
    "    collection_name = \"seerahbooks\",\n",
    "    index_name = \"vector_index\"\n",
    ")\n",
    "vector_store_context = StorageContext.from_defaults(vector_store=atlas_vector_search)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def clean_up_text(content: str) -> str:\n",
    "    \"\"\"\n",
    "    Remove unwanted characters and patterns in text input.\n",
    "\n",
    "    :param content: Text input.\n",
    "    \n",
    "    :return: Cleaned version of original text input.\n",
    "    \"\"\"\n",
    "\n",
    "    # Fix hyphenated words broken by newline\n",
    "    content = re.sub(r'(\\w+)-\\n(\\w+)', r'\\1\\2', content)\n",
    "\n",
    "    # Remove specific unwanted patterns and characters\n",
    "    unwanted_patterns = [\n",
    "        \"\\\\n\", \"  —\", \"——————————\", \"—————————\", \"—————\",\n",
    "        r'\\\\u[\\dA-Fa-f]{4}', r'\\uf075', r'\\uf0b7'\n",
    "    ]\n",
    "    for pattern in unwanted_patterns:\n",
    "        content = re.sub(pattern, \"\", content)\n",
    "\n",
    "    # Fix improperly spaced hyphenated words and normalize whitespace\n",
    "    content = re.sub(r'(\\w)\\s*-\\s*(\\w)', r'\\1-\\2', content)\n",
    "    content = re.sub(r'\\s+', ' ', content)\n",
    "\n",
    "    return content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' اَ it 04 04 Better than you no eye has ever seen More beautiful than you no woman has given birth to You have been created free from any defect As if you were created like how you desired Seeratul Mustafa \\uf048\\uf020'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Call function\n",
    "cleaned_docs = []\n",
    "for d in documents: \n",
    "    cleaned_text = clean_up_text(d.text)\n",
    "    d.text = cleaned_text\n",
    "    # add meta data according to the book\n",
    "    metadata_additions = {\"title\": \"Seeratul Mustafa Abridged\"}\n",
    "    d.metadata.update(metadata_additions) \n",
    "    cleaned_docs.append(d)\n",
    "\n",
    "# Inspect output\n",
    "cleaned_docs[0].get_content()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "for d in documents1: \n",
    "    cleaned_text = clean_up_text(d.text)\n",
    "    d.text = cleaned_text\n",
    "    # add meta data according to the book\n",
    "    metadata_additions = {\"title\": \"AR-RAHEEQ al-makhtum\"}\n",
    "    d.metadata.update(metadata_additions) \n",
    "    cleaned_docs.append(d)\n",
    "for d in documents2: \n",
    "    cleaned_text = clean_up_text(d.text)\n",
    "    d.text = cleaned_text\n",
    "    # add meta data according to the book\n",
    "    metadata_additions = {\"title\": \"Seerat Ibn e Hisham\"}\n",
    "    d.metadata.update(metadata_additions) \n",
    "    cleaned_docs.append(d)\n",
    "for d in documents3: \n",
    "    cleaned_text = clean_up_text(d.text)\n",
    "    d.text = cleaned_text\n",
    "    # add meta data according to the book\n",
    "    metadata_additions = {\"title\": \"Seerat Ibn e Hisham\"}\n",
    "    d.metadata.update(metadata_additions) \n",
    "    cleaned_docs.append(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.chunk_size = 1024\n",
    "\n",
    "# Local settings\n",
    "from llama_index.core.node_parser import SentenceSplitter\n",
    "# load both documents into the vector store\n",
    "vector_store_index = VectorStoreIndex.from_documents(\n",
    "   all_documents, storage_context=vector_store_context, show_progress=True, transformations=[SentenceSplitter(chunk_size=1024)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup basic RAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = vector_store_index.as_query_engine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add feedback functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from trulens_eval.feedback.provider import OpenAI\n",
    "from trulens_eval import Feedback\n",
    "import numpy as np\n",
    "\n",
    "# Initialize provider class\n",
    "provider = OpenAI()\n",
    "\n",
    "# select context to be used in feedback. the location of context is app specific.\n",
    "from trulens_eval.app import App\n",
    "context = App.select_context(query_engine)\n",
    "\n",
    "from trulens_eval.feedback import Groundedness\n",
    "grounded = Groundedness(groundedness_provider=OpenAI())\n",
    "# Define a groundedness feedback function\n",
    "f_groundedness = (\n",
    "    Feedback(grounded.groundedness_measure_with_cot_reasons, name = \"Groundedness\")\n",
    "    .on(context.collect()) # collect context chunks into a list\n",
    "    .on_output()\n",
    "    .aggregate(grounded.grounded_statements_aggregator)\n",
    ")\n",
    "\n",
    "# Question/answer relevance between overall question and answer.\n",
    "f_answer_relevance = (\n",
    "    Feedback(provider.relevance_with_cot_reasons, name = \"Answer Relevance\")\n",
    "    .on_input_output()\n",
    ")\n",
    "# Context relevance between question and each context chunk.\n",
    "f_context_relevance = (\n",
    "    Feedback(provider.context_relevance_with_cot_reasons, name = \"Context Relevance\")\n",
    "    .on_input()\n",
    "    .on(context)\n",
    "    .aggregate(np.mean)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from trulens_eval import TruLlama\n",
    "tru_query_engine_recorder = TruLlama(query_engine,\n",
    "    app_id='Basic RAG',\n",
    "    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write test cases\n",
    "\n",
    "Let's write a few test queries to test the ability of our RAG to answer questions on both documents in the vector store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from trulens_eval.generate_test_set import GenerateTestSet\n",
    "\n",
    "test_set = {'MongoDB Atlas': [\n",
    "                \"How do you secure MongoDB Atlas?\",\n",
    "                \"How can Time to Live (TTL) be used to expire data in MongoDB Atlas?\",\n",
    "                \"What is vector search index in Mongo Atlas?\",\n",
    "                \"How does MongoDB Atlas different from relational DB in terms of data modeling\"],\n",
    "            'Database Essentials': [\n",
    "                \"What is the impact of interleaving transactions in database operations?\",\n",
    "                \"What is vector search index? how is it related to semantic search?\"\n",
    "                ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alternatively, we can generate test set automatically\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test = GenerateTestSet(app_callable = query_engine.query)\n",
    "# Generate the test set of a specified breadth and depth without examples automatically\n",
    "# test_set = test.generate_test_set(test_breadth = 3, test_depth = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get testing!\n",
    "\n",
    "Our test set is made up of 2 topics (test breadth), each with 2-3 questions (test depth).\n",
    "\n",
    "We can store the topic as record level metadata and then test queries from each topic, using `tru_query_engine_recorder` as a context manager."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tru_query_engine_recorder as recording:\n",
    "    for category in test_set:\n",
    "        recording.record_metadata=dict(prompt_category=category)\n",
    "        test_prompts = test_set[category]\n",
    "        for test_prompt in test_prompts:\n",
    "            response = query_engine.query(test_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check evaluation results\n",
    "\n",
    "Evaluation results can be viewed in the TruLens dashboard (started at the top of the notebook) or directly in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tru.get_leaderboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps if we use metadata filters to create specialized query engines, we can improve the search results and thus, the overall evaluation results.\n",
    "\n",
    "But it may be clunky to have two separate query engines - then we have to decide which one to use!\n",
    "\n",
    "Instead, let's use a router query engine to choose the query engine based on the query.\n",
    "\n",
    "## Router Query Engine + Metadata Filters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify metadata filters\n",
    "metadata_filters_db_essentials = MetadataFilters(\n",
    "   filters=[ExactMatchFilter(key=\"metadata.file_name\", value=\"DBEssential-2021.pdf\")]\n",
    ")\n",
    "metadata_filters_atlas = MetadataFilters(\n",
    "   filters=[ExactMatchFilter(key=\"metadata.file_name\", value=\"atlas_best_practices.pdf\")]\n",
    ")\n",
    "\n",
    "metadata_filters_databrick = MetadataFilters(\n",
    "   filters=[ExactMatchFilter(key=\"metadata.file_name\", value=\"DataBrick_vector_search.pdf\")]\n",
    ")\n",
    "# Instantiate Atlas Vector Search as a retriever for each set of filters\n",
    "vector_store_retriever_db_essentials = VectorIndexRetriever(index=vector_store_index, filters=metadata_filters_db_essentials, similarity_top_k=5)\n",
    "vector_store_retriever_atlas = VectorIndexRetriever(index=vector_store_index, filters=metadata_filters_atlas, similarity_top_k=5)\n",
    "vector_store_retriever_databrick = VectorIndexRetriever(index=vector_store_index, filters=metadata_filters_databrick, similarity_top_k=5)\n",
    "# Pass the retrievers into the query engines\n",
    "query_engine_with_filters_db_essentials = RetrieverQueryEngine(retriever=vector_store_retriever_db_essentials)\n",
    "query_engine_with_filters_atlas = RetrieverQueryEngine(retriever=vector_store_retriever_atlas)\n",
    "query_engine_with_filters_databrick = RetrieverQueryEngine(retriever=vector_store_retriever_databrick)\n",
    "\n",
    "from llama_index.core.tools import QueryEngineTool\n",
    "\n",
    "# Set up the two distinct tools (query engines)\n",
    "\n",
    "essentials_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=query_engine_with_filters_db_essentials,\n",
    "    description=(\n",
    "        \"Useful for retrieving context about database essentials\"\n",
    "    ),\n",
    ")\n",
    "\n",
    "atlas_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=query_engine_with_filters_atlas,\n",
    "    description=(\n",
    "        \"Useful for retrieving context about MongoDB Atlas\"\n",
    "    ),\n",
    ")\n",
    "\n",
    "databrick_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=query_engine_with_filters_databrick,\n",
    "    description = (\n",
    "        \"Useful for retrieving context about Databrick's course on Vector Databases and Search\"\n",
    "    )\n",
    ")\n",
    "\n",
    "# Create the router query engine\n",
    "\n",
    "from llama_index.core.query_engine import RouterQueryEngine\n",
    "from llama_index.core.selectors import LLMSingleSelector, LLMMultiSelector\n",
    "from llama_index.core.selectors import (\n",
    "    PydanticMultiSelector,\n",
    "    PydanticSingleSelector,\n",
    ")\n",
    "\n",
    "\n",
    "router_query_engine = RouterQueryEngine(\n",
    "    selector=PydanticSingleSelector.from_defaults(),\n",
    "    query_engine_tools=[\n",
    "        essentials_tool,\n",
    "        atlas_tool,\n",
    "        databrick_tool\n",
    "    ],\n",
    ")\n",
    "\n",
    "from trulens_eval import TruLlama\n",
    "tru_query_engine_recorder_with_router = TruLlama(router_query_engine,\n",
    "    app_id='Router Query Engine + Filters v2',\n",
    "    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tru_query_engine_recorder_with_router as recording:\n",
    "    for category in test_set:\n",
    "        recording.record_metadata=dict(prompt_category=category)\n",
    "        test_prompts = test_set[category]\n",
    "        for test_prompt in test_prompts:\n",
    "            response = router_query_engine.query(test_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tru.get_leaderboard()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "trulens_dev_empty",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}