2021-01-21 05:14:27 +00:00
## **Table Of Contents**
* **[IOMMU Setup](#enable--verify-iommu)**
* **[Installing Packages](#install-required-tools)**
* **[Enabling Services](#enable-required-services)**
* **[Guest Setup](#setup-guest-os)**
* **[Attching PCI Devices](#attaching-pci-devices)**
* **[Libvirt Hooks](#libvirt-hooks)**
* **[Keyboard/Mouse Passthrough](#keyboardmouse-passthrough)**
* **[Video Card Virtualisation Detection](#video-card-driver-virtualisation-detection)**
* **[Audio Passthrough](#audio-passthrough)**
* **[GPU vBIOS Patching](#vbios-patching)**
### **Enable & Verify IOMMU**
2021-08-27 13:47:54 +00:00
***BIOS Settings*** \
2023-01-24 11:13:16 +00:00
Enable ** *Intel VT-d*** or ** *AMD-Vi*** in BIOS settings. If these options are not present, it is likely that your hardware does not support IOMMU.
2021-08-27 13:47:54 +00:00
2023-01-24 11:13:16 +00:00
Disable ** *Resizable BAR Support*** in BIOS settings.
2023-01-16 01:03:33 +00:00
Cards that support Resizable BAR can cause problems with black screens following driver load if Resizable BAR is enabled in UEFI/BIOS. There doesn't seem to be a large performance penalty for disabling it, so turn it off for now until ReBAR support is available for KVM.
2020-12-02 03:51:31 +00:00
***Set the kernel paramater depending on your CPU.*** \
For GRUB user, edit grub configuration.
2020-12-01 18:24:52 +00:00
| /etc/default/grub |
| ----- |
2020-12-02 03:51:31 +00:00
| `GRUB_CMDLINE_LINUX_DEFAULT="... intel_iommu=on iommu=pt ..."` |
2020-12-01 18:24:52 +00:00
| OR |
2020-12-02 03:51:31 +00:00
| `GRUB_CMDLINE_LINUX_DEFAULT="... amd_iommu=on iommu=pt ..."` |
2020-12-02 04:46:04 +00:00
2020-12-02 03:51:31 +00:00
***Generate grub.cfg***
2020-12-01 18:24:52 +00:00
```sh
grub-mkconfig -o /boot/grub/grub.cfg
```
2020-12-02 03:51:31 +00:00
Reboot your system for the changes to take effect.
2020-10-27 07:00:38 +00:00
2022-07-20 08:00:04 +00:00
***To verify IOMMU, run the following command.***
2020-12-01 18:24:52 +00:00
```sh
2022-07-20 08:00:04 +00:00
dmesg | grep IOMMU
```
2023-01-24 11:13:16 +00:00
The output should include the message `Intel-IOMMU: enabled` for Intel CPUs or `AMD-Vi: AMD IOMMUv2 loaded and initialized` for AMD CPUs.
2022-07-20 08:00:04 +00:00
2023-01-24 11:13:16 +00:00
To view the IOMMU groups and attached devices, run the following script:
2021-08-27 13:47:54 +00:00
```sh
#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V` ; do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
```
2023-01-24 11:13:16 +00:00
When using passthrough, it is necessary to pass every device in the group that includes your GPU. \
2021-08-27 13:47:54 +00:00
You can avoid having to pass everything by using [ACS override patch ](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Bypassing_the_IOMMU_groups_(ACS_override_patch )).
2020-12-01 18:24:52 +00:00
### **Install required tools**
2020-10-27 07:00:38 +00:00
< details >
2020-12-01 18:24:52 +00:00
< summary > < b > Gentoo Linux< / b > < / summary >
2023-01-24 11:13:16 +00:00
2020-10-27 07:00:38 +00:00
```sh
2020-12-01 18:24:52 +00:00
emerge -av qemu virt-manager libvirt ebtables dnsmasq
2020-10-27 07:00:38 +00:00
```
< / details >
< details >
2020-12-01 18:24:52 +00:00
< summary > < b > Arch Linux< / b > < / summary >
2020-10-27 07:00:38 +00:00
```sh
2020-12-01 18:24:52 +00:00
pacman -S qemu libvirt edk2-ovmf virt-manager dnsmasq ebtables
2020-10-27 07:00:38 +00:00
```
< / details >
< details >
2020-12-01 18:24:52 +00:00
< summary > < b > Fedora< / b > < / summary >
2020-10-27 07:00:38 +00:00
```sh
2020-12-01 18:24:52 +00:00
dnf install @virtualization
2020-10-27 07:00:38 +00:00
```
< / details >
2021-01-21 05:14:27 +00:00
< details >
< summary > < b > Ubuntu< / b > < / summary >
```sh
2021-03-15 03:31:10 +00:00
apt install qemu-kvm qemu-utils libvirt-daemon-system libvirt-clients bridge-utils virt-manager ovmf
2021-01-21 05:14:27 +00:00
```
< / details >
2020-12-01 18:24:52 +00:00
### **Enable required services**
2020-10-27 07:00:38 +00:00
< details >
2020-12-01 18:24:52 +00:00
< summary > < b > SystemD< / b > < / summary >
2020-10-27 07:00:38 +00:00
```sh
2020-12-01 18:24:52 +00:00
systemctl enable --now libvirtd
2020-10-27 07:00:38 +00:00
```
< / details >
< details >
2020-12-01 18:24:52 +00:00
< summary > < b > OpenRC< / b > < / summary >
2020-10-27 07:00:38 +00:00
```sh
2020-12-01 18:24:52 +00:00
rc-update add libvirtd default
rc-service libvirtd start
2020-10-27 07:00:38 +00:00
```
< / details >
2020-12-02 03:51:31 +00:00
Sometimes, you might need to start default network manually.
2020-12-01 18:24:52 +00:00
```sh
virsh net-start default
virsh net-autostart default
2020-10-27 07:00:38 +00:00
```
2020-12-01 18:24:52 +00:00
### **Setup Guest OS**
2020-12-02 03:51:31 +00:00
***NOTE: You should replace win10 with your VM's name where applicable*** \
2021-01-21 05:14:27 +00:00
You should add your user to ** *libvirt*** group to be able to run VM without root. And, ** *input*** and ** *kvm*** group for passing input devices.
```sh
usermod -aG kvm,input,libvirt username
```
2020-12-02 03:51:31 +00:00
Download [virtio ](https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso ) driver. \
Launch ** *virt-manager*** and create a new virtual machine. Select ** *Customize before install*** on Final Step. \
In ** *Overview*** section, set ** *Chipset*** to ** *Q35***, and ** *Firmware*** to ** *UEFI*** \
In ** *CPUs*** section, set ** *CPU model*** to ** *host-passthrough***, and ** *CPU Topology*** to whatever fits your system. \
For ** *SATA*** disk of VM, set ** *Disk Bus*** to ** *virtio***. \
In ** *NIC*** section, set ** *Device Model*** to ** *virtio*** \
Add Hardware > CDROM: virtio-win.iso \
Now, ** *Begin Installation***. Windows can't detect the ** *virtio disk***, so you need to ** *Load Driver*** and select ** *virtio-iso/amd64/win10*** when prompted. \
2021-03-15 03:31:10 +00:00
After successful installation of Windows, install virtio drivers from virtio CDROM. You can then remove virtio iso.
2020-12-01 18:24:52 +00:00
### **Attaching PCI devices**
2021-01-20 17:21:57 +00:00
Remove Channel Spice, Display Spice, Video QXL, Sound ich* and other unnecessary devices. \
2021-01-17 12:18:18 +00:00
Now, click on ** *Add Hardware***, select ** *PCI Devices*** and add the PCI Host devices for your GPU's VGA and HDMI Audio.
2020-12-02 04:37:16 +00:00
### **Libvirt Hooks**
2020-12-02 03:51:31 +00:00
Libvirt hooks automate the process of running specific tasks during VM state change. \
More info at: [PassthroughPost ](https://passthroughpo.st/simple-per-vm-libvirt-hooks-with-the-vfio-tools-hook-helper/ )
2021-08-27 13:47:54 +00:00
**Note**: Comment Unbind/rebind EFI framebuffer line from start and stop script if you're using AMD 6000 series cards, thanks to [cdgriffith ](https://github.com/cdgriffith ).
2021-08-27 13:53:20 +00:00
Also, move the line to unload AMD kernal module below detaching devices from host. These might also apply to older AMD cards.
2021-08-27 13:47:54 +00:00
2020-12-02 03:51:31 +00:00
< details >
< summary > < b > Create Libvirt Hook< / b > < / summary >
```sh
mkdir /etc/libvirt/hooks
touch /etc/libvirt/hooks/qemu
chmod +x /etc/libvirt/hooks/qemu
```
< table >
< tr >
< th >
/etc/libvirt/hooks/qemu
< / th >
< / tr >
< tr >
< td >
```sh
#!/bin/bash
GUEST_NAME="$1"
HOOK_NAME="$2"
STATE_NAME="$3"
MISC="${@:4}"
BASEDIR="$(dirname $0)"
HOOKPATH="$BASEDIR/qemu.d/$GUEST_NAME/$HOOK_NAME/$STATE_NAME"
set -e # If a script exits with an error, we should as well.
if [ -f "$HOOKPATH" ]; then
eval \""$HOOKPATH"\" "$@"
elif [ -d "$HOOKPATH" ]; then
while read file; do
eval \""$file"\" "$@"
done < < < "$(find -L "$HOOKPATH" -maxdepth 1 -type f -executable -print;)"
fi
```
< / td >
< / tr >
< / table >
< / details >
< details >
< summary > < b > Create Start Script< / b > < / summary >
2021-08-27 13:47:54 +00:00
2020-12-02 03:51:31 +00:00
```sh
mkdir -p /etc/libvirt/hooks/qemu.d/win10/prepare/begin
touch /etc/libvirt/hooks/qemu.d/win10/prepare/begin/start.sh
chmod +x /etc/libvirt/hooks/qemu.d/win10/prepare/begin/start.sh
```
< table >
< tr >
< th >
/etc/libvirt/hooks/qemu.d/win10/prepare/begin/start.sh
< / th >
< / tr >
< tr >
< td >
```sh
#!/bin/bash
set -x
# Stop display manager
systemctl stop display-manager
2021-10-24 12:25:17 +00:00
# Unbind VTconsoles: might not be needed
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
2020-12-02 03:51:31 +00:00
# Unbind EFI Framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
# Unload NVIDIA kernel modules
modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia
# Unload AMD kernel module
# modprobe -r amdgpu
# Detach GPU devices from host
# Use your GPU and HDMI Audio PCI host device
virsh nodedev-detach pci_0000_01_00_0
virsh nodedev-detach pci_0000_01_00_1
# Load vfio module
modprobe vfio-pci
```
< / td >
< / tr >
< / table >
< / details >
< details >
< summary > < b > Create Stop Script< / b > < / summary >
```sh
mkdir -p /etc/libvirt/hooks/qemu.d/win10/release/end
touch /etc/libvirt/hooks/qemu.d/win10/release/end/stop.sh
chmod +x /etc/libvirt/hooks/qemu.d/win10/release/end/stop.sh
```
< table >
< tr >
< th >
/etc/libvirt/hooks/qemu.d/win10/release/end/stop.sh
< / th >
< / tr >
< tr >
< td >
```sh
#!/bin/bash
set -x
# Attach GPU devices to host
# Use your GPU and HDMI Audio PCI host device
virsh nodedev-reattach pci_0000_01_00_0
virsh nodedev-reattach pci_0000_01_00_1
2022-09-15 19:06:58 +00:00
# Unload vfio module
modprobe -r vfio-pci
# Load AMD kernel module
#modprobe amdgpu
2020-12-02 03:51:31 +00:00
# Rebind framebuffer to host
echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind
# Load NVIDIA kernel modules
modprobe nvidia_drm
modprobe nvidia_modeset
modprobe nvidia_uvm
modprobe nvidia
2021-10-24 12:25:17 +00:00
# Bind VTconsoles: might not be needed
echo 1 > /sys/class/vtconsole/vtcon0/bind
echo 1 > /sys/class/vtconsole/vtcon1/bind
2020-12-02 03:51:31 +00:00
# Restart Display Manager
systemctl start display-manager
2022-09-15 19:06:58 +00:00
```
2020-12-02 03:51:31 +00:00
< / td >
< / tr >
< / table >
< / details >
2020-12-01 18:31:47 +00:00
2020-12-02 04:37:16 +00:00
### **Keyboard/Mouse Passthrough**
2021-09-02 06:14:47 +00:00
In order to be able to use keyboard/mouse in the VM, you can either passthrough the USB Host device or use Evdev passthrough.
Using USB Host Device is simple, \
***Add Hardware*** > ** *USB Host Device***, add your keyboard and mouse device.
For Evdev passthrough, follow these steps: \
Modify libvirt configuration of your VM. \
**Note**: Save only after adding keyboard and mouse devices or the changes gets lost. \
Change first line to:
2020-12-02 04:27:40 +00:00
< table >
< tr >
< th >
virsh edit win10
< / th >
< / tr >
< tr >
< td >
```xml
< domain type = 'kvm' xmlns:qemu = 'http://libvirt.org/schemas/domain/qemu/1.0' >
```
< / td >
< / tr >
< / table >
2020-12-02 04:46:04 +00:00
Find your keyboard and mouse devices in ** */dev/input/by-id***. You'd generally use the devices ending with ** *event-kbd*** and ** *event-mouse***. And the devices in your configuration right before closing ** *`</ domain > `*** tag. \
2020-12-02 04:27:40 +00:00
Replace ** *MOUSE_NAME*** and ** *KEYBOARD_NAME*** with your device id.
< table >
< tr >
< th >
virsh edit win10
< / th >
< / tr >
< tr >
< td >
```xml
...
< qemu:commandline >
< qemu:arg value = '-object' / >
< qemu:arg value = 'input-linux,id=mouse1,evdev=/dev/input/by-id/MOUSE_NAME' / >
< qemu:arg value = '-object' / >
< qemu:arg value = 'input-linux,id=kbd1,evdev=/dev/input/by-id/KEYBOARD_NAME,grab_all=on,repeat=on' / >
< / qemu:commandline >
< / domain >
```
< / td >
< / tr >
< / table >
You need to include these devices in your qemu config.
< table >
< tr >
< th >
/etc/libvirt/qemu.conf
< / th >
< / tr >
< tr >
< td >
```sh
...
user = "YOUR_USERNAME"
group = "kvm"
...
cgroup_device_acl = [
"/dev/input/by-id/KEYBOARD_NAME",
"/dev/input/by-id/MOUSE_NAME",
"/dev/null", "/dev/full", "/dev/zero",
"/dev/random", "/dev/urandom",
"/dev/ptmx", "/dev/kvm", "/dev/kqemu",
"/dev/rtc","/dev/hpet", "/dev/sev"
]
...
```
< / td >
< / tr >
< / table >
2020-12-02 04:46:04 +00:00
Also, switch from PS/2 devices to virtio devices. Add the devices inside ** *`< devices > `*** block
2020-12-02 04:27:40 +00:00
< table >
< tr >
< th >
virsh edit win10
< / th >
< / tr >
< tr >
< td >
```xml
...
< devices >
...
< input type = 'mouse' bus = 'virtio' / >
< input type = 'keyboard' bus = 'virtio' / >
...
< / devices >
...
```
< / td >
< / tr >
< / table >
### **Audio Passthrough**
2021-01-21 05:14:27 +00:00
VM's audio can be routed to the host. You need ** *Pulseaudio***. It's hit or miss. \
You can also use [Scream ](https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Passing_VM_audio_to_host_via_Scream ) instead of Pulseaudio. \
2020-12-02 04:27:40 +00:00
Modify the libvirt configuration of your VM.
< table >
< tr >
< th >
virsh edit win10
< / th >
< / tr >
< tr >
< td >
```xml
...
< qemu:commandline >
...
< qemu:arg value = "-device" / >
< qemu:arg value = "ich9-intel-hda,bus=pcie.0,addr=0x1b" / >
< qemu:arg value = "-device" / >
< qemu:arg value = "hda-micro,audiodev=hda" / >
< qemu:arg value = "-audiodev" / >
< qemu:arg value = "pa,id=hda,server=/run/user/1000/pulse/native" / >
< / qemu:commandline >
< / devices >
```
< / td >
< / tr >
< / table >
2020-12-02 04:37:16 +00:00
### **Video card driver virtualisation detection**
2020-12-02 04:27:40 +00:00
Video Card drivers refuse to run in Virtual Machine, so you need to spoof Hyper-V Vendor ID.
< table >
< tr >
< th >
virsh edit win10
< / th >
< / tr >
< tr >
< td >
```xml
...
< features >
...
< hyperv >
...
< vendor_id state = 'on' value = 'whatever' / >
...
< / hyperv >
...
< / features >
...
```
< / td >
< / tr >
< / table >
2020-12-02 04:37:16 +00:00
NVIDIA guest drivers also require hiding the KVM CPU leaf:
2020-12-02 04:27:40 +00:00
< table >
< tr >
< th >
virsh edit win10
< / th >
< / tr >
< tr >
< td >
```xml
...
< features >
...
< kvm >
< hidden state = 'on' / >
< / kvm >
...
< / features >
...
```
< / td >
< / tr >
< / table >
2021-01-21 05:14:27 +00:00
### **vBIOS Patching**
2023-01-24 11:13:16 +00:00
***NOTE: You are only making changes on dumped ROM file. Your hardware is safe.*** \
While most of the GPU can be passed with stock vBIOS, some GPU requires vBIOS patching to passthrough. \
2021-01-21 05:14:27 +00:00
In order to patch vBIOS, you need to first dump the GPU vBIOS from your system. \
If you have Windows installed, you can use [GPU-Z ](https://www.techpowerup.com/gpuz ) to dump vBIOS. \
2023-01-24 11:13:16 +00:00
To dump vBIOS on Linux, you can use following command (replace PCI id with yours):
2020-12-02 04:31:30 +00:00
```sh
2021-01-21 05:14:27 +00:00
echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom
cat /sys/bus/pci/devices/0000:01:00.0/rom > path/to/dump/vbios.rom
echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom
2020-12-02 04:31:30 +00:00
```
2023-01-24 11:13:16 +00:00
If you're not in root shell, you should use the above commands with sudo as:
```sh
echo 1 | sudo tee /sys/bus/pci/devices/0000:01:00.0/rom
sudo cat /sys/bus/pci/devices/0000:01:00.0/rom > path/to/dump/vbios.rom
echo 0 | sudo tee /sys/bus/pci/devices/0000:01:00.0/rom
```
2021-01-21 05:14:27 +00:00
To patch vBIOS, you need to use Hex Editor (eg., [Okteta ](https://utils.kde.org/projects/okteta )) and trim unnecessary header. \
For NVIDIA GPU, using hex editor, search string “VIDEO”, and remove everything before HEX value 55. \
2023-01-24 11:13:16 +00:00
I'm not sure about AMD, but the process should be similar.
2021-01-21 05:14:27 +00:00
To use patched vBIOS, edit VM's configuration to include patched vBIOS inside ** *hostdev*** block of VGA
< table >
< tr >
< th >
virsh edit win10
< / th >
< / tr >
< tr >
< td >
```xml
...
< hostdev mode = 'subsystem' type = 'pci' managed = 'yes' >
< source >
...
< / source >
< rom file = '/home/me/patched.rom' / >
...
< / hostdev >
...
```
< / td >
< / tr >
< / table >
2020-12-02 04:31:30 +00:00
2020-12-02 03:51:31 +00:00
### **See Also**
2021-10-24 12:25:17 +00:00
> [Single GPU Passthrough Troubleshooting](https://docs.google.com/document/d/17Wh9_5HPqAx8HHk-p2bGlR0E-65TplkG18jvM98I7V8)<br/>
2020-10-27 07:06:44 +00:00
> [Single GPU Passthrough by joeknock90](https://github.com/joeknock90/Single-GPU-Passthrough)<br/>
> [Single GPU Passthrough by YuriAlek](https://gitlab.com/YuriAlek/vfio)<br/>
2021-10-24 12:25:17 +00:00
> [Single GPU Passthrough by wabulu](https://github.com/wabulu/Single-GPU-passthrough-amd-nvidia)<br/>
2020-10-27 07:06:44 +00:00
> [ArchLinux PCI Passthrough](https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF)<br/>
2020-11-13 11:46:53 +00:00
> [Gentoo GPU Passthrough](https://wiki.gentoo.org/wiki/GPU_passthrough_with_libvirt_qemu_kvm)<br/>