hvpp is a lightweight Intel x64/VT-x hypervisor written in C++ focused primarily on virtualization of already running operating system.
Although several open-source research hypervisors aimed at simplicity already exist, in my opinion this field is still somewhat unexplored and needs more open-source projects. This can especially help those who have just started exploring virtualization technologies and are looking for small/reference projects. If you're one of them, my bets are that you're really disappointed right now, because all you've got are barely dozen of (great!) projects and huge pile of Intel Manual pages.
C++ has been chosen as a language for this project because of two reasons:
- The virtualization architecture can be better expressed in OOP concepts (with such objects as VCPU, EPT).
- I didn't find other projects which would use modern C++17 features, except for bareflank. Although bareflank is compilable under Windows, Linux and UEFI, on Windows, it uses cygwin to cross-compile the hypervisor. Since the hypervisor is a self contained ELF binary, the Windows kernel is missing the debug symbols for the hypervisor, which prevents easy debugging with WinDbg.
Even though this project is primarily developed for Windows, I've decided to not use traditional Windows Driver naming
convention (aka DrvCamelCase
). Instead, traditional C++ snake_case
is used. The reason is that hypervisor is very
"stand-alone" and doesn't depend on many OS functions. Therefore I decided to treat it as a regular C++ project.
If you want to challenge yourself in low-level programming, my advice would be to try and write a simple hypervisor. During the process you'll get invaluable knowledge and experience and you'll probably discover many new things. For instance here's a selection of some things I've learned thanks to writing this project:
- what are MTRR good for
- NT APIs like
MmGetPhysicalMemoryRanges()
(and\REGISTRY\MACHINE\HARDWARE\RESOURCEMAP
)KeIpiGenericCall
- which can be used instead of undocumentedKeGenericCallDpc
(but at the cost of higher IRQL)
- that TraceLogging API (built upon ETW) exist
- how Meltdown vulnerability has been mitigated in Windows
- that VMWare uses not only I/O port 0x5658 for backdoor, but also port 0x5659 and it uses almost all GP registers for both input and output parameters
- that bugs in other hypervisors exist (and found a way how to fix them)
- that VirtualBox uses ring 1 for software-based virtualization (i.e. when VT-x/AMD-V isn't available)
- how guest memory has been virtualized before EPT (by so called Shadow-paging, trapping
page-faults and
mov cr3, <source>
instructions in the guest)
Also - as obvious as it might sound - I'd like to point out that if you decide to write your own VT-x hypervisor, you'll NEED Intel® 64 and IA-32 architectures software developer’s manual combined volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4. So download the PDF - together with Adobe Acrobat Reader - because trust me, you don't want to read and navigate through 5000 pages with browser's built-in PDF reader.
- EPT with identity mapping with usage of 2MB pages for the first 512GB of physical memory (see ept.cpp). This results in faster translations of the memory. It also means splitting particular 2MB pages into 4kb pages might be desired if EPT hooking is required. This process is actually not complicated at all and this repository includes example on how to achieve that.
- Simple pass-through VM-exit handler, which can handle:
- exceptions or NMIs
CPUID
,(WB)INVD
,INVLPG
,RDTSC(P)
,MOV CR
,MOV DR
,IN/OUT
,RDMSR
,WRMSR
,SGDT
,SIDT
,LGDT
,LIDT
,SLDT
,STR
,LLDT
,LTR
,XSETBV
andINVPCID
instructionsVMCALL
instruction (used for termination of hvpp)VMCLEAR
,VMLAUNCH
,VMRESUME
,VMPTRLD
,VMPTRST
,VMREAD
,VMWRITE
,VMFUNC
,VMXOFF
,VMXON
,INVEPT
andINVVPID
instructions raise #UD (invalid opcode exception)
- Ability to run in VMWare (tested even with KVM - I advise to turn off Hyper-V Enlightenments, as it can cause conflicts). VMWare's nested virtualization makes development and debugging of hypervisors much easier.
- Simple custom memory manager (see mm.cpp). The need for custom memory manager emerges from the
fact that you should think twice before calling any kernel function from VM-exit handler, because many of them can be
called at IRQL <= DISPATCH_LEVEL (such as
ExAllocatePoolWithTag
). But in VM-exit handler, interrupts are disabled and your IRQL is effectively HIGH_LEVEL. - Detailed code comments, which should explain what the code does and why - sometimes with direct references to Intel Manual for further reading.
- TraceLogging API (which builds on ETW) - the main benefit is it can be used for really high
frequency logging (10'000+ per second) and it can be used from any IRQL - which makes it a perfect candidate even
for logging in VM-exit handlers.
- hvpp already includes tracing VM-exit handler (see vmexit_stats.cpp).
- Various reimplemented classes and functions - such as bitmaps and spinlocks - to avoid calling kernel functions.
- Included simple application (hvppctrl) which should demonstrate
CPUID
instruction interception, hiding hooks in user-mode applications via EPT and communication with hvpp viaVMCALL
Note: hvpp is compiled as a static library, which is linked with the hvppdrv project.
- Bootstrap of the driver (hvpp, driver.cpp)
- preallocate enough memory and initialize the hvpp memory manager
- initialize the logger
- Bootstrap of the hypervisor (hvppdrv, main.cpp)
- create vmexit_handler instance
- Start the hypervisor with provided VM-exit handler:
hypervisor::start(vmexit_handler& handler)
- initialize virtual cpu (VCPU) for each logical processor
- assign provided
vmexit_handler
instance to each VCPU - launch all VCPUs via IPI (inter-processor interrupt):
vcpu_t::start()
- setup VMXON region and VMCS:
vcpu_t::vmx_enter()
vmexit_handler::setup()
is called, which allows anyone to initialize the VM-exit handler and/or modify the VMCS before the launch (seevmexit_custom_handler::setup()
in hvppdrv, vmexit_custom.cpp)
- setup VMXON region and VMCS:
- Hypervisor is now running and handling VM-exits via provided VM-exit handler
- Stop the hypervisor:
hypervisor::stop()
- destroy each VCPU via IPI:
vcpu_t::stop()
vmexit_handler::teardown()
is called and switches into VMX mode (vmexit_passthrough_handler::teardown()
does it byVMCALL
instruction)- in VMX mode,
vcpu_t::vmx_leave()
is called - it leaves VMX mode withVMXOFF
instruction
- destroy each VCPU via IPI:
Compile hvpp using Visual Studio 2017. Solution file is included. The only required dependency is WDK.
You can run hvpp on Windows 7 or higher. Windows 10 is recommended though, because it supports TraceLogging.
Enable Test-Signing boot configuration option (note that you'll need administrative privileges to use
bcdedit
and sc
commands):
bcdedit /set testsigning on
Register driver with Service Control Manager (yes, it's important to leave these spaces):
sc create hvpp type= kernel binPath= "C:\full\path\to\hvppdrv.sys"
Now you should restart your computer for testsigning to take effect, otherwise you'll be unable to start the driver.
But before you do, you might want to prepare DebugView from SysInternals and
traceview.exe tool from the WDK (note that traceview
will work properly only on Windows 10).
After restart, launch DebugView
and TraceView
. In TraceView
:
- go to
File -> Create New Log Session
, click onAdd Provider
- pick
Manually Entered Control GUID or Hashed Name
- paste
916fcd3e-673b-4610-aaba-0b71e28acd40
(arbitrarily chosen, see lib/win32/tracelog.cpp) - click
OK
- pick
- in the next dialog, leave the
Source Of WPP Format Information
set toAuto
- click
OK
- click
- after that, click
Next
, which will bring you toLog Session Options
dialog- in
Log Session Name
editbox you can give this logging session any name you like, e.g.HvppSession
or you can leave it as it is - if you desire to analyze this logging session when it's stopped (e.g. with Windows Performance Analyzer)
you can optionally enable
Log Trace Event Data To File
, which saves whole logging session into an.ETL
file - click
Finish
- in
TraceView
is now set-up and ready to show tracelogs from hvpp. You can launch hvpp
now:
sc start hvpp
hvpp now performs various checks and enters VMX mode if they pass. In case of success you should see message
hvpp started
in the DebugView
.
Run hvppctrl:
hvppctrl.exe
-
hvppctrl performs
CPUID
instruction withEAX = 0x70707668 ('hvpp')
which hvpp should intercept and return stringhello from hvpp
in EAX, EBX, ECX and EDX registers (see vmexit_custom.cpp). hvppctrl should print this string. -
hvppctrl tries to "stealthily" hook
ntdll!ZwClose
function using EPT. The exact process is described further below. -
hvppctrl performs IOCTL, which should instruct hvpp to set one-time breakpoint when
IN/OUT
instruction manipulating with port0x64
(keyboard) is executed.
- locates
ZwClose
function in ntdll.dll- disassembles first 16 bytes of this function and prints them
- printed instructions should indicate that this function is NOT hooked yet
- calls this function (with
NULL
parameter, this function call will most likely fail with some NTSTATUS error code, which it ignores) - prints value of
HookCallCount
and it's expected value (explained below)
- disassembles first 16 bytes of this function and prints them
- hooks
ntdll!ZwClose
fuction using Detours- disassembles first 16 bytes of this function and prints them
- printed instructions should now indicate that the function IS hooked (by
jmp
being first instruction)
- printed instructions should now indicate that the function IS hooked (by
- calls this function (with
NULL
parameter)- instead of original function, the hook function will be called - on each call, it increments variable
HookCallCount
- instead of original function, the hook function will be called - on each call, it increments variable
- prints value of
HookCallCount
and it's expected value - it should be 1 now, as the hooked function has been called for the first time now
- disassembles first 16 bytes of this function and prints them
- calls hvpp by
VMCALL
instruction andRCX = 0xc1
(arbitrarily chosen),RDX = AddressOfReadPage
andR8 = AddressOfExecutePage
- this instructs hvpp to hide the hook- disassembles first 16 bytes of this function and prints them
- printed instructions should now indicate that the function hook is HIDDEN (by showing original content of
memory - no
jmp
)
- printed instructions should now indicate that the function hook is HIDDEN (by showing original content of
memory - no
- calls this function (with
NULL
parameter)- despite what the memory returned when we read it, the hook function will be called again and the
HookCallCount
will be incremented again
- despite what the memory returned when we read it, the hook function will be called again and the
- prints value of
HookCallCount
and it's expected value - it should be 2
- disassembles first 16 bytes of this function and prints them
- calls hvpp by
VMCALL
instruction andRCX = 0xc2
(arbitrarily chosen) - this instructs hvpp to unhide the hook- disassembles first 16 bytes of this function and prints them
- printed instructions should now indicate that the function hook is NOT HIDDEN and it should show
jmp
as a first instruction again
- printed instructions should now indicate that the function hook is NOT HIDDEN and it should show
- calls this function (with
NULL
parameter)- because the function is still hooked, the hook function will be called and
HookCallCount
will be incremented again
- because the function is still hooked, the hook function will be called and
- prints value of
HookCallCount
and it's expected value - it should be 3
- disassembles first 16 bytes of this function and prints them
- unhooks
ntdll!ZwClose
function- disassembles first 16 bytes of this function and prints them
- printed instructions should now indicate that the function is NOT hooked - it should show the same content as when this function wasn't hooked
- calls this function (with
NULL
parameter)- original function will be called, therefore the
HookCallCount
should not be incremented now
- original function will be called, therefore the
- prints value of
HookCallCount
and it's expected value - it should be still 3
- disassembles first 16 bytes of this function and prints them
At the same time you should see tracelog messages in the TraceView
- they are generated on each VMCALL
and on each
EPT Violation.
When you decide you want to turn off the hvpp, just execute:
sc stop hvpp
- hvpp is designed to virtualize already running OS - i.e. it's not capable of running multiple guests like VMWare or VirtualBox. It also lacks support for any nested VMX operations.
- hvpp is designed to run only on 64bit Intel processors, which support VT-x and EPT. This makes the code more simple and less bloated.
- hvpp is designed to run only on Windows - future work might focus on Linux.
- hvpp currently doesn't exit VMX mode on sleep or hibernate (S3 and S4 power states) - Intel Manual says we should do so - this is known limitation.
This software is open-source under the MIT license. See the LICENSE.txt file in this repository.
Detours is licensed under MIT license (a copy of the license is included here).
udis86 is licensed under the terms of the 2-clause "Simplified BSD License" (a copy of the license is included here).
SimpleVisor: https://github.com/ionescu007/SimpleVisor
HyperPlatform: https://github.com/tandasat/HyperPlatform
HyperBone: https://github.com/DarthTon/HyperBone
Bareflank: https://github.com/Bareflank/hypervisor
ksm: https://github.com/asamy/ksm
MoRE: https://github.com/ainfosec/MoRE
hyperdbg: https://github.com/rmusser01/hyperdbg
virtdbg: https://github.com/upring/virtdbg
BluePill: https://invisiblethingslab.com/resources/bh07/nbp-0.32-public.zip
Phrack #69: http://www.phrack.org/issues/69/15.html
NOVA Microhypervisor: https://github.com/udosteinberg/NOVA
Finally, I'd especially like to suggest reading interesting writings from Satoshi Tanda (github, twitter):
- http://tandasat.github.io/HyperPlatform/
- http://tandasat.github.io/HyperPlatform/userdocument/index.html
- http://standa-note.blogspot.com/2015/08/writing-hypervisor-for-kernel-mode-code.html
And notes from LordNoteworthy (github, twitter):
- https://github.com/LordNoteworthy/cpu-internals
- https://github.com/LordNoteworthy/windows-internals
If you find this project interesting, you can buy me a coffee
BTC 3GwZMNGvLCZMi7mjL8K6iyj6qGbhkVMNMF
LTC MQn5YC7bZd4KSsaj8snSg4TetmdKDkeCYk