[viogpu3d] Virtio GPU 3D acceleration for windows #943
Conversation
|
@max8rr8 Thank for a very impressive work you've done. Nice work! |
viogpu/viogpu3d/BUILDING.md
Outdated
| 5. Build and install mesa binaries: `ninja install` | ||
| 6. Go to `viogpu` directory of this repository | ||
| 7. Run: `.\build_AllNoSdv.bat` | ||
| 8. Compiled drivers will be available in `viogpu\viogpu3d\objfre_win10_amd64\amd64\viogpu3d` |
There was a problem hiding this comment.
Hi @max8rr8,
This instruction is not clear to me. Can you please describe it more with directory examples? What dependencies should be installed and how configured? (Example: https://github.com/virtio-win/kvm-guest-drivers-windows/wiki/Building-the-drivers-using-Windows-11-21H2-EWDK).
I looked into the mesa compilation guide and it looks completely different.
meson setup builddir/
meson compile -C builddir/
sudo meson install -C builddir/
Several questions:
- What is
%MESA_PREFIX%and where it defined? - Can we use any precompiled MESA for Windows (for example https://fdossena.com/?p=mesa/index.frag)
There was a problem hiding this comment.
What is %MESA_PREFIX% and where it defined?
%MESA_PREFIX% environment variable is set during build process, it points to directory where mesa installs its files.
Can we use any precompiled MESA for Windows
I don't think so, as pointed out in mesa MR when building user-mode driver we have to build it with specific mesa flags to build only virgl driver (to avoid conflicts)
There was a problem hiding this comment.
Building instruction
NOTE 1: this is valid for now, until changes are not in upstream repositories
NOTE 2: this assumes that all build dependencies(meson, WDK, ninja, etc...) are installed
Part 1: Virglrenderer
On host machine it is required that patched version of virglrenderer is used.
- Acquire source code
git clone --branch viogpu_win https://gitlab.freedesktop.org/max8rr8/virglrenderer && cd virglrenderer - Create install directory
mkdir installand build directory:mkdir build && cd build - Configure build
meson --prefix=$(pwd)/../install(we set prefix to install libvirglrenderer not globally but to previously created dir install) - Compile and install
ninja install
Now ensure that qemu loads libvirglrenderer from install directory, this can be done by settingLD_LIBRARY_PATHto something like/some/path/to/starter_dir/virglrenderer/install
Part 2: Build mesa
Now inside virtual machine with build tools installed create working directory, then inside it (this assumes use of Powershell):
- Create mesa prefix dir
mkdir mesa_prefixand set envMESA_PREFIXto its path:$env:MESA_PREFIX="$PWD\mesa_prefix" - Get patched mesa source code
git clone --depth 10 --branch viogpu_win https://gitlab.freedesktop.org/max8rr8/mesaand then cd into itcd mesa - Create build directory
mkdir build && cd build - Configure build
meson .. --prefix=$env:MESA_PREFIX -Dgallium-drivers=virgl -Dgallium-d3d10umd=true -Dgallium-wgl-dll-name=viogpu_wgl -Dgallium-d3d10-dll-name=viogpu_d3d10 -Db_vscrt=mt, build options explained:
--prefix=$env:MESA_PREFIXset installation path to dir created in step 1-Dgallium-drivers=virglbuild only virgl driver-Dgallium-d3d10umd=truebuild DirectX 10 user-mode driver (opengl one is build by default)-Dgallium-d3d10-dll-name=viogpu_d3d10name of generated d3d10 dll toviogpu_d3d10.dll-Dgallium-wgl-dll-name=viogpu_wglname of generated wgl dll toviogpu_wgl.dll-Db_vscrt=mtuse static c runtime (see this comment)
- Build and install (to mesa prefix):
ninja install
Part 3: Build driver
Now that mesa is build and installed into %MESA_PREFIX% viogpu3d will be built (in case %MESA_PREFIX is not set viogpu3d inf generation is skipped)
- Acquire source code
git clone --branch viogpu_win https://github.com/max8rr8/kvm-guest-drivers-windowsand cd into itcd kvm-guest-drivers-windows - Go to viogpu
cd viogpu - (optional, but very useful) setup test code signning from visual studio
- Call build
.\build_AllNoSdv.bat
Part 4: Installation
Now copy kvm-guest-drivers-windows\viogpu\viogpu3d\objfre_win10_amd64\amd64\viogpu3d to target VM and install it.
EDIT: Added gallium-windows-dll-name to mesa parameters.
EDIT2: More changes related to dll naming in mesa parameters
|
Changes:
|
|
Changes:
Pre-built driver provided in description was updated with these changes. |
|
I've tested the driver on win10, and got a black screen. But if change the guest os to ubuntu(same qemu command line), glxgears performs well. (glxinfo shows that the renderer is NVIDIA gpu) envguest os: host os: qemu command line-display egl-headless,rendernode=/dev/dri/card1 -device virtio-vga-gl -trace enable="virtio_gpu*" -D qemu.log virglrenderer.loggl_version 46 - core profile enabled
GLSL feature level 460
vrend_check_no_error: context error reported 3 "" Unknown 1282
context 3 failed to dispatch DRAW_VBO: 22
vrend_decode_ctx_submit_cmd: context error reported 3 "" Illegal command buffer 786440
GLSL feature level 460
vrend_check_no_error: context error reported 5 "" Unknown 1282
context 5 failed to dispatch DRAW_VBO: 22
vrend_decode_ctx_submit_cmd: context error reported 5 "" Illegal command buffer 786440
GLSL feature level 460
GLSL feature level 460
GLSL feature level 460
GLSL feature level 460
GLSL feature level 460
vrend_check_no_error: context error reported 9 "" Unknown 1282
context 9 failed to dispatch DRAW_VBO: 22qemu_strace.logvirtio_gpu_features virgl 0
virtio_gpu_features virgl 0
virtio_gpu_cmd_get_edid scanout 0
virtio_gpu_cmd_get_display_info
virtio_gpu_cmd_ctx_create ctx 0x1, name
virtio_gpu_cmd_ctx_create ctx 0x2, name
virtio_gpu_cmd_res_create_3d res 0x1, fmt 0x1, w 1280, h 1024, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x2, res 0x1
virtio_gpu_cmd_res_back_attach res 0x1
virtio_gpu_cmd_res_create_3d res 0x2, fmt 0x1, w 1280, h 1024, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x2, res 0x2
virtio_gpu_cmd_res_create_2d res 0x3, fmt 0x2, w 1280, h 1024
virtio_gpu_cmd_res_back_attach res 0x3
virtio_gpu_cmd_set_scanout id 0, res 0x1, w 1280, h 1024, x 0, y 0
virtio_gpu_cmd_res_flush res 0x1, w 1280, h 1024, x 0, y 0
virtio_gpu_cmd_set_scanout id 0, res 0x1, w 1280, h 1024, x 0, y 0
virtio_gpu_cmd_res_flush res 0x1, w 1280, h 1024, x 0, y 0
virtio_gpu_cmd_ctx_create ctx 0x3, name
virtio_gpu_cmd_ctx_destroy ctx 0x3
virtio_gpu_cmd_ctx_create ctx 0x3, name
virtio_gpu_cmd_res_create_3d res 0x4, fmt 0xb1, w 48, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x4
virtio_gpu_cmd_res_create_3d res 0x5, fmt 0xb1, w 4000, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x5
virtio_gpu_cmd_res_create_3d res 0x6, fmt 0xb1, w 16, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x6
virtio_gpu_cmd_res_create_3d res 0x7, fmt 0xb1, w 48, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x7
virtio_gpu_cmd_res_create_3d res 0x8, fmt 0xb1, w 240012, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x8
virtio_gpu_cmd_res_create_3d res 0x9, fmt 0xb1, w 102400, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x9
virtio_gpu_cmd_res_create_3d res 0xa, fmt 0xb1, w 144, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xa
virtio_gpu_cmd_res_create_3d res 0xb, fmt 0xb1, w 160000, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xb
virtio_gpu_cmd_res_create_3d res 0xc, fmt 0xb1, w 16000, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xc
virtio_gpu_cmd_res_create_3d res 0xd, fmt 0xb1, w 240000, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xd
virtio_gpu_cmd_res_create_3d res 0xe, fmt 0xb1, w 192, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xe
virtio_gpu_cmd_res_create_3d res 0xf, fmt 0xb1, w 16, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xf
virtio_gpu_cmd_res_create_3d res 0x10, fmt 0xb1, w 272, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x10
virtio_gpu_cmd_res_create_3d res 0x11, fmt 0xb1, w 240, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x11
virtio_gpu_cmd_res_create_3d res 0x12, fmt 0xb1, w 272, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x12
virtio_gpu_cmd_res_create_3d res 0x13, fmt 0x1, w 50, h 50, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x13
virtio_gpu_cmd_ctx_create ctx 0x4, name
virtio_gpu_cmd_res_back_attach res 0x13
virtio_gpu_cmd_res_back_attach res 0x4
virtio_gpu_cmd_res_back_attach res 0x5
virtio_gpu_cmd_res_back_attach res 0x6
virtio_gpu_cmd_res_back_attach res 0xa
virtio_gpu_cmd_res_back_attach res 0xe
virtio_gpu_cmd_res_back_attach res 0xf
virtio_gpu_cmd_res_back_attach res 0x10
virtio_gpu_cmd_res_back_attach res 0x11
virtio_gpu_cmd_res_back_attach res 0x12
virtio_gpu_cmd_ctx_submit ctx 0x3, size 74220 |
It seems to be an bug in virglrenderer on nvidia related to GL_PRIMITIVE_RESTART_NV. You can try to apply this diff to fix it, but it's a bit hacky solution. |
|
What versions of windows does it go down to? |
|
I tried the pre-built driver on Windows 10 22H2 and it only showed a black screen before hard-locking and resetting. Also tested the same config with Ubuntu which had working virgl. Using |
|
Can we validate that this works with Windows 7 too? It's a fairly common virtualization guest for playing older games, and it's still WDDM class. |
Current driver theoretically supports windows 8.1, but it is built for and tested only on Windows 10 22H2. But it is important to note that support for blob resources which are required to improve performance and support vulkan will require using WDDM 2 which lift minimum windows version to 10.
I doubt it will work with current code, it might be possible to adapt it for windows 7, but i do not have interest in doing that (though wouldn't mind if someone else will adapt code). Plus at some point driver will have to use WDDM 2 which will splitting codebases or more likely require either dropping support for older OS.
Nothing seems wrong in attached |
|
That's great @max8rr8!
For the retro direction, above was mentioned Win7 and in addition I think such virgl/venus GPU will be very useful also for:
|
|
I compiled and tried it, but I got the BSOD First, I compiled QEMU in MSYS2 like this: I compiled virglrenderer (upstream) like your comment above, then replaced the dll in QEMU dir with the newly built one. Then I compiled mesa with VS (MinGW GCC prints a bunch of errors related to the WDK headers, so I gave up): It outputs to my root folder ( I signed the output in Then I booted up the VM, enabled test signing and then install the driver. The screen instantly went black and Windows crashes. WinDBG, qemu logs and VM command line attached. I tried using serial kernel debug but it just hangs forever at boot so I analyzed the dump file instead. |
|
Additionally, I tried booting Linux Mint 20.2 live cd but QEMU itself crashed on some heap corruption issue. I replaced the original MSYS2 virglrenderer dll which fixed it. The Windows 10 guest BSOD remains. |
|
Hi @RedGreenBlue09, it seems that kernel crash happened during driver unloading ( You should try to attach windbg to running vm (try to use network debugging instead of serial, it works for me). Additionally when connecting to vm from windbg enable break on connection and run following command This windbg command should add backtrace logging to all errors happening in dxgkrnl. After you ran that command continue kernel execution with windbg command |
I will test again later. Also, if I use VGA + virtio-gpu-gl setup the driver don't load and it have error 49 in device manager. The error message is something like "Windows unloaded the driver because it has reported problems" |
|
... Network debug also hangs like serial. Interestingly that doesn't happen with ReactOS. |
|
I have no luck with live kernel debug. I tried using accel tcg, official qemu, none of these helps. If Edit: Even ditching OVMF for SeaBIOS, it still freezes. |
|
@max8rr8 Okay, I'm sorry for the rant. It doesn't hang forever, just 30 minutes :((. I tried your command and got an error: |
|
After reloading symbols, the error is gone. Here is the new log. |
It depends on the driver quality. Some commercial solutions, like Parallels virtual machines, are offering the near-native performance. Most of the open source implementations are grossly unoptimized, however, and therefore are CPU bound. This driver is one of those. It could be used for the demo purposes but it's impractical for anything else. Unless you're developer who wishes to solve these issues there is no point of wasting your time on it. Just to make it clear: this driver is nowhere near the production level of stability and neither it is able to provide the reasonable performance. If you're casual user then it's a good idea to look somewhere else instead of trying to compile it. :) |
It can provide full or nearly full performance, just not in this particular case. There is also a matter of security. Some of developers are getting so obsessed with reaching a maximum possible performance that nothing good comes out of their work. Their drivers are essentially becoming the backdoors into a host process memory. Rendering the virtualization effectively useless. |
If you need an open source solution then VirGL is an answer to your question. The "GPU partitioning" is essentially a reinvention of the VirGL and other similar approaches. As long as you're limiting yourself to OpenGL api, there is no problem with hardware acceleration on Linux or Windows guests with Qemu. If you need something else, like Direct3D, then the translation layer is needed. As far as I can get it, the discussed project is providing this very translation layer. Or you can forget about all this and resort to the proprietary solutions. Like VMWare, for instance. They're providing the accelerated 3D graphics support for Linux and Windows guests too. It's glitchy sometimes but it works nevertheless. |
Thank you, I didn't understand these things before. I just used it on Hyper-V before and felt it was okay, so I thought about whether we could reuse Microsoft's solution without having to rewrite a GPU driver. |
VirGL is not the answer, or at least not the long-term solution. In addition to GPU partitioning (which uses hardware SR-IOV), there is also virtio-GPU native contexts, which is analogous to Windows GPU Paravirtualization. Both proxy the hardware-dependent kernel driver API, which is both more secure and faster than proxying a userspace API such as OpenGL or Vulkan. Qubes OS will be using virtio-GPU native contexts for opt-in GPU acceleration. They are currently in production with Qualcomm GPUs, and experimental for AMD, Intel, and Apple GPUs. |
|
Don't ask for the perfect solution when a reasonable one is near being
merged.
I just want to run any app which requires Windows with HW-acceeleration at
acceptable speeds and without graphic glitches.
|
well, that just went from "making and maintaining one driver" to "making and maintaining a driver for every gpu the host can run on" virgl has issues for sure, every solution has issues, but lets not pretend like virtgpu-native-context is some sliver bullet here. It's not a "real alternative" to solutions like virgl or virgl-venus. It has better performance, but it's scope is far more narrow. |
The guest kernel driver doesn’t care which GPU is in use, and Mesa supports every reasonably modern GPU for which Linux has an upstreamed driver.
In my use-case (Qubes OS), virgl and Venus were both considered and rejected due to security concerns. Their attack surface is far too high because they run Mesa on the host with untrusted input. |
Just point, what your link explicitly says what SR-IOV needed and article lists very few supported hardware all of which is not regular desktop hardware. Very limited. E.g. another useless feautre.
Great if it really works, still it is opposite from what documentation says. Looks like bug (if it works on hardware which is should not do that). |
It's just locked away in firmware, which is flashable. You're bypassing some intended market segmentation though, but it's really intended to deter cloud gaming / cloud GPU compute from offering segmented consumer cards. There is some precedent with NvFBC (another firmware locked feature) being supported by Looking Glass, but it's not a solution for everyone, not even for all Nvidia users. |
IT IS JUST? Flash your flash drive, please. It is locked officially by all gpu manufactures. No one (99%+) will flash own gpus for this. At least peoples who just buy modern gaming card with 1000$+ in own home station will not flash it. So it is not "a just". Yours technical aspect is clear, and we all known what it is software (firmware) limitation. My point was what it is not about any GPU at all as stated in message. Message put has been pure lie.
I even not nVidia user and never will be (more over, i prefer integrated). I tried intel initiative from start many years ago and then they just drop mine cpu at all. So, all of this stuff... should rely on things on which developers might rely: on self,not on hardware feats. Anyway, whole topic is about proper windows driver allowing proper communication with host, in my current understanding everything discussed in this message above - really did not matters |
|
Tho I do wonder how virtio-gpu native context, which functions via DRM contexts, can work on a windows guest? would the guest kernel driver be a translation layer between Linux' DRM and Windows' WDDM? Not to mention the userspace support. |
I don't think it's a bug, I also did not flash the firmware as described below. Microsoft needs to make this feature work on any GPU, and if I remember correctly, this was originally developed for Windows Sandbox, and this feature was intended for ordinary users. I mention GPU partitioning here because Windows has such a solution that requires no additional development of a new driver. I just want to know if it is possible to reuse Microsoft's solution. |
Please calm down. You asked a question, I provided an explanation. "Just" refers to it not actually being a physical hardware limitation. This is not a support forum and I'm not suggesting you do anything. Just that if you've seen GPU segmentation on Windows without paravirtualization that someone either has a Pro card or flashed their BIOS. This includes some large YouTube channels.
I think you're talking about paravirtualization in Windows (as mentioned already here). That was developed for GPU access in WSL, but it's pretty specific to Hyper-V and either a Windows guest or special drivers shipped in WSL. Not sure about the current status of AMD/ROCm there, but that feature launched NVIDIA only on WSL. |
The way I see this being implemented is:
|
According to the information I collected, there is no problem at all on AMD GPU, and it can even be used on Qualcomm GPU (in fact, it should be independent of what kind of GPU you use). I think if we could do something similar on Linux, we might be able to replace the entire virtio-gpu technology (or just for Windows guests). |
virtio-GPU native contexts is that something. |
yes, but you still need to port every single userland driver, and every single userland driver will need to support whatever interface needs to be done for getting enablement on windows. It's not a small task, implementing venus on windows would be much easier then porting and maintaining nvk, radv, anv, turnip, etc. |
My understanding is that only the virtio-GPU interface needs to be ported. The hardware-dependent code is (or at least should be) OS-agnostic. |
|
To be clear: getting Venus working is still useful in and of itself, and it might be able to run the shader compiler in a Xen stubdomain. |
|
Would it be possible to compile this for Windows 7? There, it's even more needed than Windows 10 as it would use the Basic theme. For me, at least, it's D water. |
|
Great work here. Hope the momentum continues. I tried building the driver with partial success and then found the prebuilt driver in this thread. However the prebuilt one got me a black screen as it has with others. But then I discovered something about using the original gpu driver that may be worth sharing. I am on a Dell precision 7730 running Linux Mint. My VM is obtained by booting into my physical Windows 11 partition. However when I click Update Driver -> Browse my computer -> Let me pick..., it shows "Red Hat Virtio GPU DOD Controller as a choice of available drivers. I select that and then click Next and then Windows shudders as it is wont to do, and the display resolution becomes perfectly fine with the message "Windows has successfully updated your drivers." The driver details still show viogpudo.sys. 3D accel seems to be better too than when I used QXL. All without the viogpu3d driver. When I shut down and restart the VM it may or may not start with the poor resolution problem. If it does I repeat the steps above. Best - Ram |
|
FYI I rebased this PR on latest master and made few improvements. But it needs more work since my build was crashing. https://github.com/davispuh/kvm-guest-drivers-windows/commits/viogpu3d/ |
If you can, would it be possible to add Windows 7 support? For me, W7 just sucks with no Aero. You just lose the good of it. |
|
Would it be possible to provide a pre-built driver to test with? The original link is dead. Thanks! |
It might be possible but you need someone who cares about Win7 to implement it.
It needs more work as my build just BSOD'ed on boot so there isn't really anything to test yet :D |
|
This, together with the @winapps-org project could allow many more opportunities, thanks for keeping this around :P |
|
@davispuh I think that there's just two steps to enabling it:
|
Hello! This series of changes spanning across multiple repositories introduce support for 3d accelerated virtiogpu windows guests.
How to test
NOTE: This driver does have some rendering glitches and might crash. Try at your own risk.
0. Create qemu windows VM with VirtIO GPU with 3d acceleration enabled. It is highly recommended to use "disposable" virtual machine to test, loss of data might occur.
viogpu_winKnown issues
FIXED:
Frames displayed on screen are lagging behindFIXED:
D3d10 clearing color is not supportedFIXED:
D3d10 applications using DXGI_SWAP_EFFECT_DISCARD and DXGI_SWAP_EFFECT_SEQUENTIAL are not displayed.Rendering glitches in WinUI3 apps.
There are some rendering glitches in apps based on WinUI3 (maybe other apps too), best way to see them is to install WinUI3 Gallery from microsoft store and navigate around it. Haven't yet invistigated.
Vscode (possibly other electron apps) does not render
Black window. Requires implementation of PIPE_QUERY_TIMESTAMP_DISJOINT in virglrenderer.
No preemption
Kernel-mode driver does not implement preemption, and i am very confised about how to implement it in WDDM. VioGpu3D disables preemption systemwide to workaround lack of preemption implementation, but this is not ideal. Would appreciate some help.
Siblings