Single GPU passthrough with QEMU and VFIO

Windows 10 1709

Table of Contents

1. What this does
2. What you need
3. Submit your own configuration
4. My system
5. vBIOS
6. Configure
7. Known problems
8. TODO

What this does

In one command it kills X, frees the GPU from drivers and console, detaches the GPU from the host, starts the VM with the GPU, waits until the VM is off, reattaches the GPU to the host and starts lightdm.

What you need

• An IOMMU enabled motherboard. Check your motherboard manual for an option in BIOS to enable IOMMU.
• CPU support for AMD-v/VT-x and AMD-Vi/VT-d (AMD/Intel).
• Virtualization support enabled on BIOS; check your motherboard manual if you need help.
• One GPU that supports UEFI and its BIOS. All GPUs from 2012 and later should support this, some may have issues. If the GPU does not support UEFI you may be able to make it work, but you won't see anything in the screen until the drivers inside Windows kick in.
• QEMU, OVMF UEFI and VIRTIO drivers for Windows. If you need to install, refer to the Install guide
• [Optional] HDD only for Windows
• [Recommended] Another computer to login remotely with ssh for convenience, at least until you have everything working.

Submit your own configuration

Create a pull merge with [a file explaining how you got it working](Hardware configurations/README.md)

My system

                                                [Hardware]
                                                 CPU: AMD Ryzen 5 2600
                                         Motherboard: Gigabyte AB350M-Gaming 3 rev1.1
                                    Motherboard BIOS: F23d
                                                 RAM: 16GB
                                                 GPU: Gigabyte Nvidia GeForce GTX 770
                                           GPU model: GV-N770OC-2GD
                                            GPU BIOS: 80.04.C3.00.0F
                                        GPU codename: GK104

                                                [Software]
                                        Linux Distro: ArchLinux
                                        Linux Kernel: 4.17.14 vanilla
                                       Nvidia divers: 396.51-1
                                        QEMU version: 2.12.1-1
                                        OVMF version: r24021

                                                 [Guests]
                                             Windows: Windows 10 Pro 1709 x64
                                               MacOS: MacOS High Sierra 10.13.3

vBIOS

I experienced some weird things when doing this on the display, like a corruption of the image, it may be my GPU. If you encounter anything, a reboot solved my problems.

Method 1 - Linux

It did not work for me, the ROM is 59KiB and it should be around 162KiB. It may work for you.

1. Execute scripts/iommu.sh to get the BUS ID for the GPU. Looks like 0000:06:00.0.
2. Edit scripts/extract-vbios-linux.sh to your convenience. Change videobusid=. [Optional] Change also the location where the vBIOS will be save VBIOS=.
3. Make the script executable with chmod +x scripts/extract-vbios-linux.sh.
4. Link the service to systemd: ln -s scripts/qemu@.service /usr/lib/systemd/system/.
5. Execute the systemd unit with sudo systemctl start qemu@extract-vbios-linux.service. You can also do it over ssh. The screen will turn dark for a while. The extracted ROM will be in the root directory /root/vBIOS.rom

From here you are alone, I don't know how to proceed. Maybe you need to edit, maybe don't.

Method 2 - nvflash in Linux

1. Download nvflash https://www.techpowerup.com/download/nvidia-nvflash/. Do not install from AUR; the package it's broken.
2. Unzip it as /root/nvflash_linux with # unzip nvflash_5.414.0_linux.zip -d /root/.
3. Execute scripts/iommu.sh to get the BUS ID for the GPU. Looks like 0000:06:00.0.
4. Edit scripts/extract-vbios-nvflash.sh. Change the variables videobusid with your GPU BUS ID; NVFLASH if you changed the location of the executable; and VBIOS if you want the ROM in other path.
5. Link the service to systemd: ln -s scripts/qemu@.service /usr/lib/systemd/system/.
6. Execute the systemd unit with sudo systemctl start qemu@extract-vbios-nvflash.service. You can also do it over ssh. The screen will turn dark for a while. The extracted ROM will be in the root directory /root/vBIOS.rom
7. Edit the vBIOS

Method 3 - Windows

Source. You can download the bios from techpowerup.com; if you do so, skip to edit the vBIOS.

1. Download and install GPU-Z.
2. Open GPU-Z and backup the GPU BIOS. Right next to the Bios Version; in my case 80.04.C3.00.0F, there is an icon for backup. A file named GK104.rom will be created [Your file name may vary].
3. Edit the vBIOS

Edit the VBIOS

1. Open the vBIOS ROM (vBIOS.rom) in the HEX editor.
2. [After a bunch of 00 there is a 55 or U in HEX, delete everything before the 55](/Screenshots/Hex vBIOS.png), and save. I strongly recommend not to overwrite the original ROM.

Configure

1. Clone this repository

$ git clone https://gitlab.com/YuriAlek/vfio.git

2. [Optional] Download virtio drivers. If you do not, modify scripts/windows.sh.

$ wget -o virtio-win.iso "https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso"

4. Get the iommu groups needed for the VM (GPU, GPU audio and USB controller). They look like 06:00.0

$ chmod +x scripts/iommu.sh
$ scripts/iommu.sh
-------------------------
# GPU
IOMMU group 13
	  06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
	  06:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
# USB 3.0 Controller
IOMMU group 16
	  07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
# SATA Controller
IOMMU group 18
	  08:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

5. [Optional] Create the image for the VM. Only if not using a physical hard drive. You can edit the path, the size and the format. Check man qemu-img for more information.

$ qemu-img create -f raw /path/to/image/windows.raw 60G

6. Edit the config in scripts/config.sh to convenience. If you use systemd to start the VM you have to edit EnvironmentFile in qemu@.service to point to your config file. Variables you may have to edit:

7. PCI devices. IOMMU_GPU; IOMMU_USB.

8. User.

9. Location of HDD/IMG, ISO, vBIOS and OVMF image.

10. The Desktop Environment, Display Manager, Window Manager, etc. lightdm/i3.

11. QEMU options like RAM and CPU cores.

12. Kernel modules.

13. Other things like add a command to kill PulseAudio pulseaudio -k and another, at the end of the script, to start it again pulseaudio --start.

14. The network options -device virtio-net-pci,netdev=n1 -netdev user,id=n1.

15. Swap virtio for sata as the HDD interface.

16. Start the VM

# scripts/windows-install.sh

8. When installing Windows, in the section Where do you want to install Windows? there will be no hard drives to install to; to fix it:

9. Load driver

10. Browse

11. CD Drive (E:) virtio-win-0.1.1

12. vioscsi

13. w10

14. amd64

15. ok

16. Load driver Red Hat VirtIO SCSI pass-through controller (E:\vioscsi\w10\amd64\vioscsi.inf)

17. Next

18. Select the Unallocated Space

19. Proceed as normal.

20. Let Windows find the drivers for the GPU (if Windows has network) or download the updated ones from NVIDIA.

21. Once installed Windows, run the VM with:

# scripts/windows.sh

For the sake of convenience

# ln -s scripts/qemu@.service /usr/lib/systemd/system/
# ln -s scripts/qemu-mac@.service /usr/lib/systemd/system/

$ alias windows="sudo systemctl start qemu@windows.service"
$ alias macos="sudo systemctl start qemu-mac@macos-hs.service"

To start the Windows VM

$ windows

To start the MacOS VM

$ macos

For MacOS

Check this amazing guide for creating the MacOS install image and Clover.

Known problems

Race condition

There is something somewhere that makes it crash. That's why there is so many sleep

MacOS does not like USB hubs, therefore anything connected to a hub will be ignored by MacOS

Sometimes works, sometimes does not

Sometimes the GPU will not have correct drivers inside Windows, even when yesterday was working just fine; Windows may install them.

Sometimes the QEMU command will just fail and the command continues and start X again.

Sometimes the QEMU command does not exit after shutting down the VM.

Windows version

Windows 10 Pro 1709 works for me, but 1803 does not (may be the UEFI). I have heard that the 1803 version comes with a Spectre patch and the performance is bad. The Spectre patch can be disabled.

TODO

• Unbind GPU without virsh.
• Update macos script.
• Run QEMU as user.
• Try if is necessary to edit /etc/mkinitcpio.conf. No need to load the kernel modules at boot.
• Extract the vBIOS in Linux.
• Install guide.
• Network.
• IOMMU guide.
• Audio.
• Troubleshooting guide.
• How to edit the windows.sh script.
• Fix the race condition.
• Create scripts for install and use (Without DVD images).
• Improve the script with multiple options for HDD/IMG; network; PCI devices, etc.
• ACS Patch (Does not work for me).
• CPU pinning and RAM HugePages.
• Not kill X, shifter & xpra may be the solution. Also uswsusp (userspace software suspend). Source