Skip to content

Latest commit

 

History

History
55 lines (33 loc) · 2.82 KB

how-to-use-pipelines-prompt-flow.md

File metadata and controls

55 lines (33 loc) · 2.82 KB
Raw
title titleSuffix description services ms.author author ms.reviewer ms.service ms.subservice ms.date ms.topic ms.custom
Use Azure Machine Learning pipelines with no code to construct RAG pipelines (preview)
Azure Machine Learning
Set up Azure Machine Learning pipelines to run Prompt Flow models (preview)
machine-learning
balapv
balapv
ssalgado
machine-learning
core
06/20/2024
how-to
prompt

Use Azure Machine Learning pipelines with no code to construct RAG pipelines (preview)

This article offers you examples on how to create an RAG pipeline. For advanced scenarios, you can build your own custom Azure Machine Learning pipelines from code (typically notebooks) that allows you granular control of the RAG workflow. Azure Machine Learning provides several in-built pipeline components for data chunking, embeddings generation, test data creation, automatic prompt generation, prompt evaluation. These components can be used as per your needs using notebooks. You can even use the Vector Index created in Azure Machine Learning in LangChain.

[!INCLUDE machine-learning-preview-generic-disclaimer]

Prerequisites

  • An Azure subscription. If you don't have an Azure subscription, create a free account.

  • Access to Azure OpenAI.

  • Enable prompt flow in your Azure Machine Learning workspace

In your Azure Machine Learning workspace, you can enable prompt flow by turn-on Build AI solutions with Prompt flow in the Manage preview features panel.

Prompt Flow pipeline notebook sample repository

Azure Machine Learning offers notebook tutorials for several use cases with prompt flow pipelines.

QA Data Generation

QA Data Generation can be used to get the best prompt for RAG and to evaluation metrics for RAG. This notebook shows you how to create a QA dataset from your data (Git repo).

Test Data Generation and Auto Prompt

Use vector indexes to build a retrieval augmented generation model and to evaluate prompt flow on a test dataset.

Create a FAISS based Vector Index

Set up an Azure Machine Learning Pipeline to pull a Git Repo, process the data into chunks, embed the chunks and create a langchain compatible FAISS Vector Index.

Next steps

How to create vector index in Azure Machine Learning prompt flow (preview)

Use Vector Stores with Azure Machine Learning (preview)