The NVIDIA RTX™ AI Toolkit is a suite of tools and SDKs for Windows application developers to accelerate model customization, optimization, and deployment of AI models into applications running on Windows PC for RTX — across both cloud and PC.
Follow the RTX AI Toolkit fine-tuning workflow with this tutorial - LLaMaA3-8B QLoRA
NVIDIA RTX AI Toolkit Launch Blog
NVIDIA RTX AI Toolkit includes 2 primary phases: Model Customization and Model Deployment. Each phase is tailored to guide you through the necessary steps to effectively customize and deploy your AI models.
Currently, we support an end-to-end workflow for customizing LLMs using PEFT (Parameter Efficient Fine-Tuning) techniques such as LoRA and(Low-Rank Adaptation of Large Language Models) and QLoRA on your RTX PC and deploying using NVIDIA TensorRT-LLM, ONNX-Runtime, llama.cpp, or as NIM endpoints in the cloud.
1. Model Customization - TUTORIAL:
The model customization tutorial walks you through launching AI Workbench, using the LlamaFactory GUI to do QLoRa fine-tuning, and exporting the quantized model. Optionally, we provide Jupyter notebooks for quantizing finetuned models for deployment with TensorRT-LLM.
2. Model Deployment - START HERE
There are two paths to deploy AI models: On device, or in cloud. Models deployed to device can achieve lower latency and don't require calls to the cloud at runtime, but have certain hardware requirements. Models deployed to the cloud can support an application running on any hardware, but have an ongoing operating cost. Different applications will do either, or both. The RTX AI Toolkit provides tools for both paths, and we provide instructions in the tutorial for deploying across on device and cloud environments.
NVIDIA AI Inference Manager (AIM) SDK offers developers a unified interface to orchestrate deployment of AI models across devices using multiple inference backends - from cloud to local PC execution environments. This is currently available to certain early access customers, apply now to get access.
Quantized (on-device) inference: For on-device inferencing the we below inferencing paths are supported:
| Platform | LoRA Adapter | Merged checkpoint |
|---|---|---|
| TensorRT-LLM | ✅ | |
| llama.cpp | ✅ | |
| ONNX Runtime - DML | ✅ |
FP16 (cloud) inference: For cloud deployments the following inferencing paths are supported:
| Platform | LoRA Adapter | Merged checkpoint |
|---|---|---|
| vLLM | ✅ | ✅ |
| NIMs | ✅ |
- AI Workbench LLaMa-Factory Project
- ChatRTX - Reference RAG Demo
- OpenAI Compatible Web Server
- Projects built by the community
Please file Issues on GitHub.
This repository is licensed under the Apache-2.0 License.
