=== Intel Volume Management Device driver update

Links: +
link:https://cgit.freebsd.org/src/commit/?id=7af4475a6e31202a865b1dd3727018659b44470f[main branch commit] URL: https://cgit.freebsd.org/src/commit/?id=7af4475a6e31202a865b1dd3727018659b44470f[https://cgit.freebsd.org/src/commit/?id=7af4475a6e31202a865b1dd3727018659b44470f]

Contact: Alexander Motin <mav@FreeBSD.org>

Intel VMD is used by Intel's VROC (Virtual RAID on chip) to isolate parts of
PCI tree, including NVMe devices, behind one or several VMD devices.
Each VMD device has 3 memory windows to access config space and memory of its
child devices, plus some number (I saw 19 and 33) of MSI-X interrupt vectors
to forward their MSI and MSI-X interrupts through.

The vmd(4) driver was first introduced in FreeBSD 13.0, but was found to have
a number of problems addressed in this update:
 - The VMD device was made to look more like a regular PCI-to-PCI bridge,
implementing the regular pcib(4) interface and using the standard pci(4) bus
driver on top.
 - Memory and bus numbers resource management was rewritten to properly handle
resource requests using memory windows of the VMD device.
 - Interrupt handling was completely rewritten to distribute child interrupts
among available VMD MSI-X vectors, setting them up to be handled via standard
OS mechanisms with minimal overhead instead of custom dispatching via taskqueue.
Due to the limited number of vectors and routing abilities of VMD, children are
limited to only one MSI or three MSI-X vectors per device to avoid sharing.
The limits can be tuned, depending on the number of VMD vectors and children.
To improve the flexibility the nvme(4) driver was made to work with a limited
number of vectors, starting from just one MSI/MSI-X if needed.

Sponsor: iXsystems, Inc.