Hyper-V synthetic timers (hv-stimer) requires Hypver-V clocksources (hv-time)

[deargle]

I'm on debian testing (well, actually, Kali rolling, which uses debian testing), so libvirt 6.4.0-2, https://packages.debian.org/source/bullseye/libvirt

deargle@localhost:~$ qemu-system-x86_64 --version
QEMU emulator version 5.0.0 (Debian 1:5.0-6)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

==> default: Starting domain.
There was an error talking to Libvirt. The error message is shown
below:                                                                     
                                                                           
Call to virDomainCreateWithFlags failed: internal error: qemu unexpectedly closed the monitor: 2020-08-01T15:54:13.704083Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]                                                           
2020-08-01T15:54:13.704121Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]   
2020-08-01T15:54:13.704160Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-ept-execonly [bit 0]                   
2020-08-01T15:54:13.704196Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-eptad [bit 21]                         
2020-08-01T15:54:13.704236Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(491H).vmx-eptp-switching [bit 0]                 
Hyper-V synthetic timers (hv-stimer) requires Hyper-V clocksources (hv-time)                                                                          
2020-08-01T15:54:13.704956Z qemu-system-x86_64: kvm_init_vcpu failed: Function not implemented 

Alas, hv-time is not supported:

Error while creating domain: Error saving the server: Call to virDomainDefineXML failed: unsupported configuration: unsupported HyperV Enlightenment feature: time

Setting hv-stimer state to off allows boot.

v.hyperv_feature :name => 'stimer', :state => 'off'

IIRC part of qemu more strict checks to see if requested enlightenment features are actually usable, which I guess stimer isn't without time?

[abbbi]

One likely issue is that the cpu on the host system does not support the features. Currently, the vagrant-libvirt plugin does not check wether if the host cpu has the needed featureset, allways enabling to use these features (most recent cpus have, though) if set in the vagrantfile. So the only workaround is to disable the feature if running the boxes on systems with older hardware.

You could script this in your vagrantfile, checking the cpus's vmx features and only enable the feature if they are present.

Maybe you also need to enable those options in your BIOS.

Another reason could be that your libvirt version is way newer than the linux kernel on the system, trying to use features your kernel does not yet support, but your hardware would.

[deargle]

Interesting, I'm running libvirt on a new GCP instance (https://cloud.google.com/compute/docs/instances/enable-nested-virtualization-vm-instances#restrictions), so it's not a case of older hardware, and vmx for the bios is not something I can fine-tune adjust except for just enabling nested vmx overall for my instannce. And the stimer enlightenment has been enabled for a long time in the base box vagrantfile for this repo and has run without error, suggesting it's not a case of a too-new libvirt, yeah?

[abbbi]

Not sure if related, but others seem to have such issues too, especially on GCP/Azure. see:

https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1882774

It could also be related to the default cpu emulation model that is picked up by libvirt, maybe it helps you to set another emulated cpu type, using the domain specific cpu settings by vagrant-libvirt: https://github.com/vagrant-libvirt/vagrant-libvirt#domain-specific-options

[ruzickap]

@deargle Is there any chance to try it on the real hardware? I'm not sure if it's a problem of the software or if the issues in "hardware" Azure/GCP (some limitation of public clouds).

I added these "hyperv_feature" they should speed up running Windows:

packer-templates/Vagrantfile-windows.template

Lines 14 to 19 in 4aefc7a

	# Enable Hyper-V enlightments: https://blog.wikichoon.com/2014/07/enabling-hyper-v-enlightenments-with-kvm.html
	libvirt.hyperv_feature :name => 'relaxed', :state => 'on'
	libvirt.hyperv_feature :name => 'stimer', :state => 'on'
	libvirt.hyperv_feature :name => 'synic', :state => 'on'
	libvirt.hyperv_feature :name => 'vapic', :state => 'on'
	libvirt.hyperv_feature :name => 'vpindex', :state => 'on'

.

I'm running VMs with these parameters with older laptop without any problems (but I'm using Ubuntu 20.04).

I can try to install Debian testing (bullseye) myself and five it a try, but this will take some time...

[wrbbz]

I'm trying this on my machine. And the result is the same:

vagrant up --provider=libvirt
Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'peru/windows-10-enterprise-x64-eval' version '20200801.01' is up to date...
==> default: Starting domain.
There was an error talking to Libvirt. The error message is shown
below:

Call to virDomainCreateWithFlags failed: internal error: process exited while connecting to monitor: 2020-08-04T11:58:00.501164Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-08-04T11:58:00.501200Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
Hyper-V synthetic timers (hv-stimer) requires Hyper-V clocksources (hv-time)
2020-08-04T11:58:00.502748Z qemu-system-x86_64: kvm_init_vcpu failed: Function not implemented

I'm on Arch Linux.

• libvirt: 6.5.0-1
• qemu-system-x86_64: 5.0.0

Log of lscpu:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              2
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           158
Model name:                      Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
Stepping:                        9
CPU MHz:                         4500.049
CPU max MHz:                     4500.0000
CPU min MHz:                     800.0000
BogoMIPS:                        8403.00
Virtualization:                  VT-x
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        1 MiB
L3 cache:                        8 MiB
NUMA node0 CPU(s):               0-7
Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Vulnerable: No microcode
Vulnerability Tsx async abort:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe sysc
                                 all nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pcl
                                 mulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx
                                 f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsba
                                 se tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsave
                                 s dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

[ruzickap]

Hello guys.

I was looking at it and seems the problem is in the vagrant-libvirt, which doesn't support the "clock" values - PR for this is still opened: https://github.com/vagrant-libvirt/vagrant-libvirt/pull/1047/files and the issue vagrant-libvirt/vagrant-libvirt#640 as well.

Anyway here is something to read about it:

• https://unix.stackexchange.com/questions/534155/win10-guests-high-kvm-host-cpu-usage
• https://bugzilla.redhat.com/show_bug.cgi?id=1738244
• https://bugzilla.redhat.com/show_bug.cgi?id=1816670

Few commands how to reproduce it and fix it (Debian Testing):

$ mkdir windows-10-enterprise-x64-eval
$ cd windows-10-enterprise-x64-eval
$ vagrant init peru/windows-10-enterprise-x64-eval
$ VAGRANT_DEFAULT_PROVIDER=libvirt vagrant up --no-destroy-on-error
...
...
==> default: Creating shared folders metadata...
==> default: Starting domain.
There was an error talking to Libvirt. The error message is shown
below:

Call to virDomainCreateWithFlags failed: internal error: qemu unexpectedly closed the monitor: 2020-08-05T07:29:04.474443Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-register [bit 8]
2020-08-05T07:29:04.474490Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9]
2020-08-05T07:29:04.474495Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7]
2020-08-05T07:29:04.474498Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-08-05T07:29:04.474501Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
Hyper-V synthetic timers (hv-stimer) requires Hyper-V clocksources (hv-time)
2020-08-05T07:29:04.474893Z qemu-system-x86_64: kvm_init_vcpu failed: Function not implemented

You will get the known error ^^^

You should be able to see the VM using this command:

$ virsh -c qemu:///system list --all
 Id   Name                                     State
---------------------------------------------------------
 -    windows-10-enterprise-x64-eval_default   shut off

Edit the VM definition using the virsh -c qemu:///system edit windows-10-enterprise-x64-eval_default
command and replace <clock offset='utc'/> by:

  <clock offset='utc'>
    <timer name='hypervclock' present='yes'/>
  </clock>

Then try tu run the vagrant up again:

VAGRANT_DEFAULT_PROVIDER=libvirt vagrant up

The VM should be successfully started.

Can you please try it please?
If it work for you I'll create an issue in vagrant-libvirt or update the older ones.

Thank you

[ruzickap]

What about disabling the stimer as a workaround for some time?

I created PR for that #159

Let me know if you have any objections or ideas how this can be done better.

[wrbbz]

@ruzickap Hello. Sorry for late reply!

I've tried your solution on two different machines (both with ArchLinux).

On one of them there were no error after this workaround.
But another one failing with timeout on an attempt to obtain IP address.

I'm sure that it is not connected with this issue.

Thank you!

[ruzickap]

Thank you...

I'll do more tests and re-generate boxes without stimer.
I'll keep this open for some time until I find some better solution (ideally with help of vagrant-libvirt).

[deargle]

Yes, it works for me too on gcp nested virt when I disable stimer. I havent tried the manual clock edit yet though.

…

On Thu, Aug 6, 2020, 8:33 AM Petr Ruzicka ***@***.***> wrote: Thank you... I'll do more tests and re-generate boxes without stimer. I'll keep this open for some time until I find some better solution (ideally with help of vagrant-libvirt). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI6Y7J42HIUTUMXFNFYPTDR7K5LTANCNFSM4PR36YHA> .

[ruzickap]

All images are without the stimer now.
Thanks for reporting it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.