Skip to content

Commit

Permalink
Fix comments (#122)
Browse files Browse the repository at this point in the history
Co-authored-by: Waqas Javed <waqasjaved@microsoft.com>
  • Loading branch information
w-javed and Waqas Javed committed Aug 5, 2024
1 parent 62573ac commit 212bfba
Showing 1 changed file with 17 additions and 10 deletions.
27 changes: 17 additions & 10 deletions scenarios/evaluate-model-endpoints/evaluate-models-target.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"\n",
"This tutorial provides a step-by-step guide on how to evaluate prompts against variety of model endpoints deployed on Azure AI Platform or non Azure AI platforms. \n",
"\n",
"This guide uses Python Class as a application target to evaluate results generated by LLM models against provided prompts. \n",
"This guide uses Python Class as an application target which is passed to Evaluate API provided by PromptFlow SDK to evaluate results generated by LLM models against provided prompts. \n",
"\n",
"This tutorial uses the following Azure AI services:\n",
"\n",
Expand Down Expand Up @@ -64,12 +64,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Target function\n",
"We will use a `ModelEndpoints` application target to get answers from multip model endpoints against provided prompts (questions). We will use `evaluate` API to evaluate `ModelEndpoints` applicaton\n",
"## Target Application\n",
"\n",
"`ModelEndpoints` class needs following list of model endpoints and their authentication keys.\n",
"We will use Evaluate API provided by Prompt Flow SDK. It requires a target Application or python Function, which handles a call to LLMs and retrieve responses. \n",
"\n",
"For simplicity, we have provided endpoints and keys in the `env_var` variable and passed into Application Target class `ModelEndpoints` in init() function."
"In the notebook, we will use an Application Target `ModelEndpoints` to get answers from multiple model endpoints against provided question aka prompts. \n",
"\n",
"This application target requires list of model endpoints and their authentication keys. For simplicity, we have provided them in the `env_var` variable which is passed into init() function of `ModelEndpoints`."
]
},
{
Expand Down Expand Up @@ -103,7 +104,7 @@
"metadata": {},
"source": [
"\n",
"Please provide Azure AI Project details so that traces and eval results are pushing in the project. "
"Please provide Azure AI Project details so that traces and eval results are pushing in the project in Azure AI Studio."
]
},
{
Expand All @@ -121,7 +122,7 @@
"source": [
"## Data\n",
"\n",
"Following code reads Json file \"data.jsonl\" which contains inputs to the Application Target __call__ function. It provides question, context and grouth truth for evaluators."
"Following code reads Json file \"data.jsonl\" which contains inputs to the Application Target function. It provides question, context and ground truth on each line. "
]
},
{
Expand Down Expand Up @@ -164,11 +165,17 @@
"source": [
"## Run the evaluation\n",
"\n",
"Following code runs Evaluate API and uses Content Safety, Relevance and Coherence Evaluator to evaluate results from different models.\n",
"The Following code runs Evaluate API and uses Content Safety, Relevance and Coherence Evaluator to evaluate results from different models.\n",
"\n",
"The following are the few parameters required by Evaluate API. \n",
"\n",
"+ Data file (Prompts): It represents data file 'data.jsonl' in JSON format. Each line contains question, context and ground truth for evaluators. \n",
"\n",
"+ Application Target: It is name of python class which can route the calls to specific model endpoints using model name in conditional logic. \n",
"\n",
"Test data is provided in json file 'data.jsonl' for App \n",
"+ Model Name: It is an identifier of model so that custom code in the App Target class can identify the model type and call respective LLM model using endpoint URL and auth key. \n",
"\n",
"Application Target uses the questions to call specific endpoints and retrive answer from response to evaluate using Evaluate API from Promoptflow SDK. "
"+ Evaluators: List of evaluators is provided, to evaluate given prompts (questions) as input and output (answers) from LLM models. "
]
},
{
Expand Down

0 comments on commit 212bfba

Please sign in to comment.