=== Full CPUID Control for bhyve

Contact: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>

==== Introduction

Operating system kernels as well as certain user space libraries and programs often need to know what CPU model they are running on and which architectural features are available for use.
On x86-based platforms, the CPUID instruction can be used to query the CPU for all of this information.

In a virtual machine environment, not all of the information reported by the host CPU should be made available to the VM guest, some other CPU information such as for system topology needs to be modified to match the VM configuration, and then there is some additional information that needs to be reported that no real CPU supports, such as the hypervisor identification.

Thus as part of its normal operation as a hypervisor, bhyve emulates the x86 CPUID instruction for its VM guests.
At this time, most of the CPUID information from the host CPU is passed to the VM guest with some features masked out, and it also reports synthetic information about the hypervisor, its capabilities, and its configuration.
None of this is presently explicitly configurable in bhyve.

==== Motivation

Being in full control of the CPUID information emulated for the VM guest has a variety of important use cases:

===== VM Migration

In order to migrate a running or suspended VM between different VM hosts, usually both involved VM hosts have to be running on the same or at least feature-compatible host CPU families, models, and steppings.
Even slight differences in CPU model and stepping may cause a difference in the featureset available.

Being in control of the CPUID information for the guest VM allows the operator to restrict the VMs to use a common subset of the CPU features shared by all VM hosts the VM is possibly going to run on, which could also be based on an abstract x86 architecture level, rather than a real CPU model.

This is currently supported by QEMU/KVM on Linux and equivalent support in bhyve is a key step towards supporting fully-featured VM live migration.

===== Fixed CPU Model

Some software checks the CPU family/model/stepping and also the CPU brand string for software compatibility reasons.
A change in those parameters due to migration of a VM or a hardware change in the VM host would be detected and could be considered a system change, potentially disabling a proprietary software license.
Being able to use a pre-defined CPU model, be it based on a real CPU or an architectural subset, would avoid this problem entirely.

===== Hypervisor Signature Modification

Some software such as the Nvidia drivers check the hypervisor signature to match certain known values.
Bhyve identifies itself as "bhyve bhyve   ", while KVM reports "KVMKVMKVM\0\0\0".
While implementing a command line switch to change the hypervisor signature of bhyve on a per-VM basis would be possible without full CPUID control, an implementation of this feature utilizing full CPUID control would be more natural and lead to a cleaner design.

===== Feature Masking & Architectural Downgrade / Upgrade

Being able to easily downgrade or upgrade the CPU model, or mask individual feature bits can provide benefits to operating system and application development and testing.

==== Current state of CPUID control for bhyve

A basic implementation of the vmm kernel part for CPUID control has been available in bhyve on illumos for a number of years now.
This code was written by Oxide for use with their Propolis hypervisor, which is based on the kernel components of FreeBSD/bhyve that have been ported to illumos without using any of bhyve's userspace code.

Like all bhyve improvements made on illumos, the vmm kernel code for CPUID control is dual-licensed CDDL and BSD.
It has been ported to bhyve on FreeBSD under the BSD license.
As a proof of concept, new config options for CPUID control have been implemented in the bhyve userspace, showing the potential of the full CPUID control on bhyve.

==== Project Overview

Ongoing work on this project aims to integrate the existing proof-of-concept work into FreeBSD, and to add the following usability features:

- a user-friendly configuration method to override individual bits, parts or even whole CPUID functions as needed, while keeping the rest of the host CPUID information or a pre-defined CPUID configuration
- a user-friendly configuration method for the hypervisor signature reported by bhyve
- a set of predefined CPUID configurations based on common x86 architecture levels, perhaps also including a set of CPUID data for a few real CPU models, and a user-friendly configuration method to choose one for a VM

Sponsor: The FreeBSD Foundation
