|
|
|
## VFIO from scratch
|
|
|
|
|
|
|
|
by https://www.reddit.com/user/Chapo_Rouge
|
|
|
|
|
|
|
|
*The goal of this document is to provide an up-to-date (July 2016) way to setup a VGA
|
|
|
|
passthrough on QEMU/KVM with the help of VFIO on any Linux host.*
|
|
|
|
|
|
|
|
*This guide focus on Intel CPU and Nvidia GPU as I don't own another type of hardware, nevertheless, a good part of it is hardware agnostic.*
|
|
|
|
|
|
|
|
*This guide use the most recent KVM VGA passthrough method which is OVMF + vfio-pci (compared to Seabios + pci-stub) and requires a GPU with EFI ROM*
|
|
|
|
|
|
|
|
#### Hardware prerequisites :
|
|
|
|
|
|
|
|
+ VT-x & VT-d enabled hardware (CPU + motherboard)
|
|
|
|
+ 2 GPU
|
|
|
|
+ (optional) coffee or cold beers
|
|
|
|
|
|
|
|
#### Software used :
|
|
|
|
|
|
|
|
Gentoo amd64, Linux kernel 4.6.x
|
|
|
|
QEMU 2.5.1 + KVM
|
|
|
|
virt-manager & libvirt
|
|
|
|
OVMF (if using Windows >= 8.0)
|
|
|
|
|
|
|
|
### Part 1 : Host system setup
|
|
|
|
|
|
|
|
#### Kernel configuration
|
|
|
|
todo
|
|
|
|
|
|
|
|
CONFIG_VFIO_IOMMU_TYPE1=m
|
|
|
|
CONFIG_VFIO=m
|
|
|
|
CONFIG_VFIO_PCI=m
|
|
|
|
CONFIG_INTEL_IOMMU=y
|
|
|
|
CONFIG_IOMMU_HELPER=y
|
|
|
|
CONFIG_VFIO=m
|
|
|
|
CONFIG_IOMMU_API=y
|
|
|
|
CONFIG_IOMMU_SUPPORT=y
|
|
|
|
CONFIG_KVM_VFIO=y
|
|
|
|
|
|
|
|
#### Kernel boot options
|
|
|
|
|
|
|
|
edit `/etc/default/grub` and add to GRUB_CMDLINE_LINUX
|
|
|
|
|
|
|
|
intel_iommu=on
|
|
|
|
|
|
|
|
> not always needed ?
|
|
|
|
|
|
|
|
Then regenerate grub2 menu with
|
|
|
|
|
|
|
|
grub2-mkconfig -o /boot/grub/grub.cfg
|
|
|
|
|
|
|
|
or add the kernel option at boot time.
|
|
|
|
|
|
|
|
#### Check IOMMU groups
|
|
|
|
|
|
|
|
> This command should report the various IOMMU groups from your machine (An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine.)
|
|
|
|
|
|
|
|
> if there's nothing it means IOMMU is not properly enabled.
|
|
|
|
|
|
|
|
#!/bin/bash
|
|
|
|
|
|
|
|
for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);
|
|
|
|
do echo "IOMMU group $(basename "$iommu_group")";
|
|
|
|
for device in $(ls -1 "$iommu_group"/devices/);
|
|
|
|
do echo -n $'\t'; lspci -nns "$device";
|
|
|
|
done;
|
|
|
|
done
|
|
|
|
|
|
|
|
#### Isolating the GPU with vfio-pci
|
|
|
|
|
|
|
|
Get your vendor-id :
|
|
|
|
|
|
|
|
lspci | grep -i vga
|
|
|
|
|
|
|
|
> note down the first number, it is the slot number i.e 01:00.0
|
|
|
|
|
|
|
|
lspci -nns 01:00.0
|
|
|
|
|
|
|
|
> Note down the last number between "[]", this is the vendor-id, i.e 10de:1380
|
|
|
|
|
|
|
|
lspci -nnk -d vendor-id
|
|
|
|
|
|
|
|
Edit `/etc/modprobe.d/vfio.conf` with the vendor-id from your GPU you want to isolate gathered from the previous command, in this __example__, the vendor-id is __10de:13c2__ for the GPU and __10de:0fbb__ for the audio
|
|
|
|
|
|
|
|
options vfio-pci ids=10de:13c2,10de:0fbb
|
|
|
|
|
|
|
|
Add these modules to `/etc/conf.d/modules` (Gentoo/OpenRC specific)
|
|
|
|
|
|
|
|
modules="vfio vfio-pci vfio_iommu_type1 vfio_virqfd"
|
|
|
|
|
|
|
|
module loading at boot is enabled by
|
|
|
|
|
|
|
|
rc-update add modules boot
|
|
|
|
|
|
|
|
Reboot
|
|
|
|
|
|
|
|
#### Check that your GPU is correctly isolated :
|
|
|
|
|
|
|
|
$ dmesg | grep -i vfio
|
|
|
|
[ 0.329224] VFIO - User Level meta-driver version: 0.3
|
|
|
|
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000 <-- Good
|
|
|
|
|
|
|
|
$ lspci -nnk -d <VENDOR_ID>
|
|
|
|
Kernel driver in use: vfio-pci <-- Good
|
|
|
|
|
|
|
|
Same check for your sound card
|
|
|
|
|
|
|
|
If lspci -nnk -d reports a "Kernel driver in use" which is not vfio-pci, there's something wrong.
|
|
|
|
|
|
|
|
You can blacklist module loading by settings up the blacklist in /etc/modprobe.d/blacklist.conf
|
|
|
|
|
|
|
|
blacklist snd_hda_intel
|
|
|
|
|
|
|
|
Also check IOMMU & DMAR, output should contains :
|
|
|
|
|
|
|
|
dmesg | grep "IOMMU\|DMAR"
|
|
|
|
[ 0.000000] ACPI: DMAR 0x00000000BD9373C0 000080 (v01 INTEL HSW 00000001 INTL 00000001)
|
|
|
|
[ 0.019360] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
|
|
|
|
[ 0.019362] IOAPIC id 8 under DRHD base 0xfed90000 IOMMU 0
|
|
|
|
[ 0.292166] DMAR: No ATSR found
|
|
|
|
[ 0.292235] IOMMU: dmar0 using Queued invalidation
|
|
|
|
[ 0.292237] IOMMU: Setting RMRR:
|
|
|
|
[ 0.292246] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd8a6000 - 0xbd8b2fff]
|
|
|
|
[ 0.292301] IOMMU: Prepare 0-16MiB unity mapping for LPC
|
|
|
|
[ 0.292307] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
|
|
|
|
|
|
|
|
#### Kernel modules (.ko)
|
|
|
|
|
|
|
|
Make sure these modules are present when you run lsmod
|
|
|
|
|
|
|
|
$ lsmod
|
|
|
|
[...]
|
|
|
|
vfio (?)
|
|
|
|
vfio_iommu_type1 (?)
|
|
|
|
kvm
|
|
|
|
kvm_intel
|
|
|
|
vfio-pci
|
|
|
|
intel_iommu (?)
|
|
|
|
[...]
|
|
|
|
|
|
|
|
#### QEMU / KVM / libvirt
|
|
|
|
|
|
|
|
Below example is for Gentoo & OpenRC, adjust for your distro.
|
|
|
|
|
|
|
|
# emerge --ask qemu virt-manager libvirt git
|
|
|
|
# rc-update add libvirt-guests
|
|
|
|
# service libvirt-guests start
|
|
|
|
# service libvirtd start
|
|
|
|
|
|
|
|
#### OVMF setup
|
|
|
|
|
|
|
|
Get OVMF via https://www.kraxel.org/repos/jenkins/edk2/ and choose the
|
|
|
|
|
|
|
|
`edk2.git-ovmf-x64-[...] RPM.`
|
|
|
|
|
|
|
|
Unpack it and copy `OVMF-pure-efi.fd` and `OVMF_VARS-pure-efi.fd` to /usr/share/ovmf/x64/, creating the directory if needed.
|
|
|
|
|
|
|
|
Add this line to /etc/libvirt/qemu.conf :
|
|
|
|
|
|
|
|
nvram = [
|
|
|
|
"/usr/share/ovmf/x64/ovmf_x64.bin:/usr/share/ovmf/x64/ovmf_vars_x64.bin"
|
|
|
|
]
|
|
|
|
|
|
|
|
Create a new machine with virt-manager and check "edit settings" before finishing the setup, change firmware from BIOS to UEFI (or to "Custom: /usr/share/ovmf/x64/ovmf_x64.bin") then boot the VM.
|
|
|
|
|
|
|
|
> If you're dropped to an EFI shell, make sure the ISO you wish to boot is correct, try with a recent Ubuntu release for instance.
|
|
|
|
|
|
|
|
#### Networking (Poor man's)
|
|
|
|
|
|
|
|
emerge bridge-utils
|
|
|
|
|
|
|
|
Create bridge :
|
|
|
|
|
|
|
|
brctl addbr br0
|
|
|
|
brctl addif br0 eth0
|
|
|
|
dhcpcd br0
|
|
|
|
|
|
|
|
Delete bridge :
|
|
|
|
|
|
|
|
ifconfig br0 down
|
|
|
|
brctl delif br0 eth0
|
|
|
|
brctl delbr br0
|
|
|
|
|
|
|
|
Add a network card on guest, attached to br0
|
|
|
|
|
|
|
|
#### Unhinge QEMU (needed ?) :
|
|
|
|
|
|
|
|
Ref : http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm
|
|
|
|
|
|
|
|
Because QEMU normally runs sandboxed, we need to 'unhinge' it and give it root privileges so it can control hardware.
|
|
|
|
|
|
|
|
In /etc/libvirt/qemu.conf, add:
|
|
|
|
|
|
|
|
user = "root"
|
|
|
|
group = "root"
|
|
|
|
clear_emulator_capabilities = 0
|
|
|
|
|
|
|
|
#### Common patches explained
|
|
|
|
|
|
|
|
__vgaarb__ aka __i915__ :
|
|
|
|
|
|
|
|
`x-vga=on`
|
|
|
|
If OVMF : vgaarb is not needed
|
|
|
|
If SeaBIOS : Requires vgaarb
|
|
|
|
|
|
|
|
__ACS Override__
|
|
|
|
|
|
|
|
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson ACS override patch.
|
|
|
|
|
|
|
|
### Part 2 : Guest VM setup
|
|
|
|
|
|
|
|
The XML configuration of the VM is located at `/etc/libvirt/qemu`
|
|
|
|
|
|
|
|
#### Possible bugs :
|
|
|
|
|
|
|
|
+ Nvidia Code 43 :
|
|
|
|
|
|
|
|
Recent nvidia drivers block the driver if it runs on top of an hypervisor, the workaround is to hide KVM :
|
|
|
|
|
|
|
|
via cli :
|
|
|
|
|
|
|
|
-cpu [type],kvm=off
|
|
|
|
|
|
|
|
via libvirt :
|
|
|
|
|
|
|
|
> edit your VM XML definition with virsh edit <VM_NAME>
|
|
|
|
|
|
|
|
<domain type='kvm'>
|
|
|
|
...
|
|
|
|
<features>
|
|
|
|
<kvm>
|
|
|
|
<hidden state='on'/>
|
|
|
|
</kvm>
|
|
|
|
...
|
|
|
|
</features>
|
|
|
|
</domain>
|
|
|
|
|
|
|
|
|
|
|
|
### Part 3 : Optimizations
|
|
|
|
|
|
|
|
Chipset : 440FX is ok
|
|
|
|
|
|
|
|
### Appendices : Biblography
|
|
|
|
|
|
|
|
#### Online resources :
|
|
|
|
|
|
|
|
+ https://www.reddit.com/r/VFIO
|
|
|
|
+ https://www.redhat.com/archives/vfio-users/
|
|
|
|
+ https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF |
|
|
\ No newline at end of file |
