Effortlessly deploy a hardware-passthrough (VFIO) setup for Virtual Machines (VMs) on a Linux desktop. Includes quality-of-life features that you didn't think you need!
- 1. Why?
- 2. Related Projects
- 3. Documentation
- 4. Host Requirements
- 5. Download
- 6. Usage
- 7. Features
- 8. Filenames and Pathnames Modified by deploy-VFIO
- 9. Graphics Hardware (GPUs)
- 10. Disclaimer
- 11. Contact
- 12. References
-
Separation of Concerns: Independently operate your workstation, gaming, and school Operating Systems (OS), as Virtual Machines (VMs), under one Host machine.
-
No Need for a Server
- Keep your Host's desktop experience intact; turns your Host into a Type-2 Hypervisor.
- Servers like Microsoft Hyper-V, Oracle VM, and Proxmox Linux are considered Type-1 or bare-metal Hypervisors.
-
Securely run a an OS
- Limited access to real hardware means greater security.
- The benefits extend to untrusted or legacy OSes.
-
Ease of use: support for automation by use of the Command Line Interface (CLI).
-
PCI Passthrough: prioritize real hardware over emulation.
-
Quality of Life: utilize multiple common-sense features that are known to experienced users.
-
Your Host OS is supported.
Note: For even greater security, use me_cleaner alongside a VFIO setup.
Project | Codeberg | GitHub |
---|---|---|
Deploy VFIO | link | link |
Auto X.Org | link | link |
Generate Evdev | link | link |
Guest Machine Guide | link | link |
Libvirt Hooks | link | link |
Power State Virtual Machine Manager | link | link |
- What is VFIO?
- VFIO Discussion and Support
- Hardware-Passthrough Guide
- Virtual Machine XML Format Guide
Linux Distributions | Tested | Supported |
---|---|---|
Arch | No | N/A |
Debian1 | Yes | 11, 12 |
Gentoo | No | N/A |
Red Hat Enterprise | No | N/A |
SUSE | No | N/A |
Required software packages (for this script):
xmlstarlet
- To install packages:
- Debian Linux:
sudo apt install -y xmlstarlet
Other requirements:
GRUB
to execute command lines at boot (if chosen).systemd
for system services.
The following firmware options are supported and enabled (motherboard and CPU):
- IOMMU
- For AMD machines:
AMD-Vi
- For Intel machines:
VT-d
- ARM (
SMMU
) and other CPU architectures are not explicitly supported by this script.
- For AMD machines:
-
Download the
.zip
file:- Viewing from the top of the repository's (current) webpage, click the
drop-down icon:
···
on Codeberg.<> Code
on GitHub.
- Click
Download ZIP
and save. - Open the
.zip
file, then extract its contents.
- Viewing from the top of the repository's (current) webpage, click the
drop-down icon:
-
Clone the repository:
- Open a Command Line Interface (CLI).
- Open a console emulator (for Debian systems: Konsole).
- Open a existing console: press
CTRL
+ALT
+F2
,F3
,F4
,F5
, orF6
.- To return to the desktop, press
CTRL
+ALT
+F7
. F1
is reserved for debug output of the Linux kernel.F7
is reserved for video output of the desktop environment.F8
and above are unused.
- To return to the desktop, press
- Change your directory to your home folder or anywhere safe:
cd ~
- Clone the repository:
git clone https://www.codeberg.org/portellam/deploy-VFIO
git clone https://www.github.com/portellam/deploy-VFIO
- Open a Command Line Interface (CLI).
-
Open the CLI (see Download).
-
Go to the directory of where the cloned/extracted repository folder is:
cd name_of_parent_folder/deploy-VFIO/
-
Make the installer script file executable:
chmod +x installer.bash
- Do not make any other script files executable. The installer will perform this action.
- Do not make any non-script file executable. This is not necessary and potentially dangerous.
- From the project folder, execute:
sudo bash installer.bash
-h, --help Print this help and exit. -i, --install Install deploy-VFIO to system. -u, --uninstall Uninstall deploy-VFIO from system.
- The installer will place all script files in
/usr/local/bin
. - The installer will place all configuration/text files in
/usr/local/etc
.
- The installer will place all script files in
- From anywhere, execute:
sudo bash deploy-VFIO
- The CLI's shell (bash) should recognize that the script file is located in
/usr/local/bin
.
- The CLI's shell (bash) should recognize that the script file is located in
-h, --help Print this help and exit.
-q, --quiet Reduce verbosity; print only relevant questions and
status statements.
-u, --undo Undo changes (restore files) if script has exited early
or unexpectedly.
--ignore-distro Ignore distribution check for Debian or Ubuntu system.
Specify the database to reference before parsing IOMMU groups:
--xml [filename] Cross-reference XML file. First-time, export if VFIO is
not setup. Consecutive-times, imports if VFIO is setup.
[filename] Reference specific file.
--no-xml Skips prompt.
Specify the IOMMU groups to parse:
-p, --parse [groups] Parse given IOMMU groups (delimited by comma).
all Select all IOMMU groups.
no-vga Select all IOMMU groups without VGA devices.
[x] Select IOMMU group.
[x-y] Select IOMMU groups.
Example:
--parse no-vga,14 Select IOMMU group 14 and all non-VGA groups.
--parse 1,14-16 Select IOMMU groups 1, 14, 15, and 16.
Pre-setup:
-c, --cpu Allocate CPU.
-e, --evdev Setup a virtual KVM switch.
-h, --hugepages [ARGS] Create static hugepages (pages greater than 4 KiB) to
allocate RAM for Guest(s).
--skip-pre-setup Skip execution.
--uninstall-pre-setup Undo all changes made by pre-setup.
Hugepages:
2M, 1G Hugepage size (2 MiB or 1 GiB).
[x] Amount of Hugepages (maximum amount is total memory
subtracted by 4 GiB).
Example:
--hugepages 1G 16 1 GiB hugepage 16 == 16 GiB allocated to hugepages.
--hugepages 2M 8192 2 MiB hugepage 8912 == 16 GiB allocated to hugepages.
VFIO setup:
-m, --multiboot [ARGS] Create multiple VFIO setups with corresponding GRUB
menu entries. Specify default GRUB menu entry by VGA IOMMU group ID.
-s, --static [ARGS] Single VFIO setup. Specify method of setup.
--skip-vfio-setup Skip execution.
--uninstall-vfio-setup Undo an existing VFIO setup.
Multiboot VFIO:
[x] The ID of the valid excluded VGA IOMMU group.
default Default menu entry excludes VFIO setup.
first Prefer the first valid excluded VGA IOMMU group.
last Prefer the last valid excluded VGA IOMMU group.
Static VFIO:
file Append output to system configuration files.
grub Append output to GRUB; single GRUB menu entry.
-
Isolate CPU
- Statically isolate Host CPU threads before allocating to Guest(s).
- Reduces Host overhead, and improves both Host and Guest performance.
- If installed, the dynamic Libvirt hook will skip its execution, to preserve the Static isolation.
-
Huge pages
- Statically isolate Host memory (RAM) into Huge memory pages before allocating to Guest(s).
- Reduces Host overhead, and improves both Host and Guest performance.
- Eliminates the need to defragment RAM before allocation to Guest(s) (as in the case with transparent hugepages).
-
Virtual Keyboard Video Mouse (KVM) switch
- Create a virtual KVM switch by Evdev (Event Devices).
- Allow a user to swap a group of Input devices (as a whole) between active Guest(s) and Host.
- Use a defined macro.
- Default macro:
L-CTRL
+R-CTRL
. - Change the macro for each VM in the XML configuration.
- Default macro:
- Implementation is known as Generate Evdev.
- Note: Using guest PCI USB alone is good. Using both implementations is better.
-
Multi-boot VFIO Setup
-
Create multiple VFIO setups with corresponding GRUB menu entries. More flexibility.
-
Select a GRUB menu entry with a VGA device excluded from VFIO.
- Default menu entry is without VFIO setup.
-
Best for systems with two or more PCI VGA devices, without an integrated VGA device (iGPU).
-
Ad: For best results, use Auto X.Org.
-
-
Static VFIO Setup
- Specify method:
- Append output to GRUB (single GRUB menu entry).
- Append output to system configuration files.
- Less flexible than either setups.
- Best for systems with one or more PCI VGA device(s) and one integrated VGA device (iGPU).
- Specify method:
-
Dynamic VFIO Setup (To be implemented in a future release)
- Use Libvirt hooks to bind or unbind devices at Guest(s) start or stop.
- Most responsibility; best for more experienced users.
- Most flexibility; Libvirt hooks allow Host to allocate and release resources dynamically.
- For an existing script of similar scope, you may try the project VFIO-Tools.
-
Auto X.Org system service to find and set a valid Host boot VGA device for X.Org.
-
Guest Audio Capture
- Create an audio loopback to output on the Host audio device Line-Out.
- Listen on Host audio device Line-In (from Guest PCI Audio device Line-Out).
- Useful for systems with multiple audio devices.
- For virtual implementation, see Virtual Audio Capture.
-
Libvirt Hooks
- Invoke hooks or scripts for all or individual Guests.
- Switch display input (video output) at Guest start.
- Dynamically allocate CPU cores and prioritize CPU scheduler.
- Libvirt-nosleep: per Guest system service, to prevent Host sleep while Guest is active.
-
RAM as Compressed Swapfile/partition
-
Virtual Audio Capture
- Setup a virtual audio driver for Windows that provides a discrete audio device and passthrough from a Guest to Host.
- Passthrough audio by direct-memory-access (DMA).
- Passthrough audio by a virtual Local Area Network (LAN) device.
- Implementation is known as Scream.
-
Virtual Video Capture
- Setup DMA of a PCI VGA device output (video and audio) from a Guest to Host.
- Implementation is known as Looking Glass.
- Disclaimer: Only supported for Guests running Windows 7 and later (Windows NT 6.1+).
/etc/apparmor.d/local/abstractions/libvirt-qemu
/etc/libvirt/qemu.conf
/etc/default/grub
/etc/grub.d/proxifiedScripts/custom
/etc/initramfs-tools/modules
/etc/modprobe.d/pci-blacklists.conf
/etc/modprobe.d/vfio.conf
/etc/modules
/etc/libvirt/hooks/
/usr/local/bin/
/etc/systemd/system/
/usr/local/bin/deploy-VFIO
/usr/local/etc/deploy-VFIO.d/
Note: Unfortunately, most or all GPUs without UEFI firmware (BIOS-only) are not currently compatible with VFIO. If your GPU does work, please share your success with this project and/or the VFIO community.
lspci -nnk | grep --extended-regexp --ignore-case "vga|graphics"
Example:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070] [10de:2484] (rev a1)
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] [1002:6719]
Some VGA devices, such as NVIDIA, may not be recognizable in a VM, as the video BIOS or VBIOS is tainted at Host OS start-up. This is usually the case if a given VGA device is used for the Host BIOS/UEFI booting process. To remedy this, you must obtain a clean copy of the VBIOS. You may review either NVIDIA-vBIOS-VFIO-Patcher, or the ArchWiki.
Note: Vendors that are unlisted are more than likely legacy graphics hardware. Therefore, it is safe to assume such hardware is BIOS-only.
Vendor | Model | Firmware |
---|---|---|
3dfx | any | BIOS |
AMD | Radeon HD 7750,7970 and newer | UEFI |
AMD | Radeon HD 7000-series and older | BIOS |
ATI | any | BIOS |
Intel | Gen5 (HD Graphics) and newer | UEFI |
Intel | Gen4 and older | BIOS |
NVIDIA | GeForce GTX 700 to 1000-series | UEFI |
NVIDIA | GeForce GTX 600-series and older | BIOS |
For emulating graphics hardware on old, legacy operating systems (such as Windows 98), try SoftGPU. Modern CPUs are more than powerful enough to emulate such hardware. Fortunately, this implementation is not software rendering. You may use whichever graphics API which is supported by the emulated GPU (example: Glide for 3dfx).
Typically, Linux is compatible with GPUs dating back to the end of the 20th century. However, GPU-compatibility may vary among different Linux distributions. Please review the documentation of your Linux distribution.
Legacy Windows version | Vendor | Model |
---|---|---|
XP / NT 4 and older | AMD | Radeon HD 7970 and older |
XP / NT 4 and older | NVIDIA | GeForce GTX 900-series |
7 / NT 6.1 | AMD | Radeon R9 200-series and newer |
7 / NT 6.1 | NVIDIA | GeForce GTX 3000-series and older |
Use at your own risk. Please review your system's specifications and resources.
Did you encounter a bug? Do you need help? Please visit the Issues page (Codeberg, GitHub).
portellam/audio-loopback. Codeberg. Accessed June 18, 2024. https://codeberg.org/portellam/audio-loopback.
portellam/audio-loopback. GitHub. Accessed June 18, 2024. https://github.com/portellam/audio-loopback.
portellam/auto-xorg. Codeberg. Accessed June 18, 2024. https://codeberg.org/portellam/auto-xorg.
portellam/auto-xorg. GitHub. Accessed June 18, 2024. https://github.com/portellam/auto-xorg.
portellam/generate-evdev. Codeberg. Accessed June 17, 2024. https://codeberg.org/portellam/generate-evdev.
portellam/generate-evdev. GitHub. Accessed June 17, 2024. https://github.com/portellam/generate-evdev.
Graphics card compatibility for Final Cut Pro, Motion, and Compressor. Apple Support. October 31, 2023. Accessed June 18, 2024. https://support.apple.com/en-us/102734.
Hugepages. Debian Wiki. Accessed June 17, 2024. https://wiki.debian.org/Hugepages.
Huge memory pages. PCI passthrough via OVMF - ArchWiki. Accessed June 14, 2024. https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Huge_memory_pages.
portellam/libvirt-hooks. Codeberg. Accessed June 18, 2024. https://codeberg.org/portellam/libvirt-hooks.
portellam/libvirt-hooks. GitHub. Accessed June 18, 2024. https://github.com/portellam/libvirt-hooks.
Looking Glass. Looking Glass. Accessed June 17, 2024. https://looking-glass.io/
LZ4/LZ4. GitHub. Accessed June 17, 2024. https://github.com/lz4/lz4.
corna/me_cleaner. GitHub. Accessed June 17, 2024. https://github.com/corna/me_cleaner.
dt-zero/me_cleaner. GitHub. Accessed June 17, 2024. https://github.com/dt-zero/me_cleaner.
Matoking/NVIDIA-vBIOS-VFIO-Patcher. GitHub. Accessed June 18, 2024. https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher.
4.5 Passing Keyboard/Mouse via Evdev. PCI passthrough via OVMF - ArchWiki. Accessed June 14, 2024. https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF.
PCI passthrough via OVMF. ArchWiki. Accessed June 14, 2024. https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF.
VFIO Discussion and Support. Reddit. Accessed June 14, 2024. https://www.reddit.com/r/VFIO/.
tonymacx86 - Will my Nvidia Graphics Card work with macOS ? List of Desktop Cards with Native Support. Archive.org. Accessed June 18, 2024. https://web.archive.org/web/20230926193339/https://www.tonymacx86.com/threads/will-my-nvidia-graphics-card-work-with-macos-list-of-desktop-cards-with-native-support.283700/.
duncanthrax/scream. GitHub. Accessed June 17, 2024. https://github.com/duncanthrax/scream.
JHRobotics/SoftGPU. GitHub. Accessed June 17, 2024. https://github.com/JHRobotics/SoftGPU.
Using Scream Over LAN. Looking Glass. Accessed June 17, 2024. https://looking-glass.io/wiki/Using_Scream_over_LAN.
Type 1 vs. Type 2 hypervisors. IBM. Accessed June 18, 2024. https://www.ibm.com/topics/hypervisors.
VFIO - ‘Virtual Function I/O’ - The Linux Kernel Documentation. The linux kernel. Accessed June 14, 2024. https://www.kernel.org/doc/html/latest/driver-api/vfio.html.
Virtualization technology. OpenSUSE Leap 15.5. Accessed June 18, 2024. https://doc.opensuse.org/documentation/leap/virtualization/html/book-virtualization/chap-virtualization-introduction.html.
PassthroughPOST/VFIO-Tools. GitHub. Accessed June 18, 2024. https://github.com/PassthroughPOST/VFIO-Tools.
Video Graphics Array. Wikipedia. August 18, 2002. Accessed June 18, 2024. https://en.wikipedia.org/wiki/Video_Graphics_Array.
XML Design Format. GitHub - libvirt/libvirt. Accessed June 18, 2024. https://github.com/libvirt/libvirt/blob/master/docs/formatdomain.rst.
XML Design Format: Input Devices. GitHub - libvirt/libvirt.Accessed June 18, 2024. https://github.com/libvirt/libvirt/blob/master/docs/formatdomain.rst.
foundObjects/zram-swap. GitHub. Accessed June 17, 2024. https://github.com/foundObjects/zram-swap.
New zram tuning benchmarks. Reddit. Accessed June 17, 2024. https://web.archive.org/web/20220201101923/https://old.reddit.com/r/Fedora/comments/mzun99/new_zram_tuning_benchmarks/.
zramswap. Arch Linux User Repository. Accessed June 17, 2024. https://aur.archlinux.org/packages/zramswap.