vfio-guide-2/README.md

214 lines
9.5 KiB
Markdown
Raw Normal View History

2018-08-10 17:02:31 +00:00
# Single GPU passthrough with QEMU and VFIO
<!-- AKA Passthrough VGA on first slot -->
2018-08-10 17:42:57 +00:00
2018-08-10 18:25:20 +00:00
![Windows 10 1709](/Screenshots/Windows 10 QEMU single GPU info.png)*Windows 10 1709*
2018-08-10 18:20:06 +00:00
2018-08-17 12:58:40 +00:00
## Table of Contents
2018-08-10 17:29:45 +00:00
1. [What this does](#what-this-does)
2. [What you need](#what-you-need)
3. [My system](#my-system)
2018-08-18 09:52:06 +00:00
4. [vBIOS](#vbios)
2018-08-18 09:48:25 +00:00
5. [Configure](#configure)
6. [Known problems](#known-problems)
7. [TODO](#todo)
2018-08-10 17:21:08 +00:00
2018-08-10 17:02:31 +00:00
## What this does
In one command it kills X, frees the GPU from drivers and console, detaches the GPU from the host, starts the VM with the GPU, waits until the VM is off, reattaches the GPU to the host and starts lightdm.
2018-08-06 10:21:29 +00:00
2018-08-10 17:02:31 +00:00
## What you need
* An IOMMU enabled motherboard. Check your motherboard manual.
* CPU support for AMD-v/VT-x and AMD-Vi/VT-d (AMD/Intel). And virtualization support enabled on BIOS.
2018-08-17 12:58:40 +00:00
* One GPU that supports UEFI and its BIOS. All GPUs from 2012 and later should support this. If the GPU does not support UEFI you may be able to make it work, but you wont see anything in the screen until the drivers inside Windows kick in.
2018-08-10 17:02:31 +00:00
* QEMU, OVMF UEFI and VIRTIO drivers for Windows.
* [Optional] HDD only for Windows
2018-08-18 09:48:25 +00:00
* [Recommended] Another computer to login remotely with `ssh` for convenience, at least until you have everything working.
2018-08-10 17:02:31 +00:00
## My system
```
2018-08-10 18:32:15 +00:00
[Hardware]
CPU: AMD Ryzen 5 2600
Motherboard: Gigabyte AB350M-Gaming 3 rev1.1
Motherboard BIOS: F23d
RAM: 16GB
GPU: Gigabyte Nvidia GeForce GTX 770
GPU model: GV-N770OC-2GD
GPU BIOS: 80.04.C3.00.0F
GPU codename: GK104
[Software]
Linux Distro: ArchLinux
2018-08-17 12:58:40 +00:00
Linux Kernel: 4.17.14 vanilla
Nvidia divers: 396.51-1
QEMU version: 2.12.1-1
2018-08-10 18:32:15 +00:00
OVMF version: r24021
[Guests]
Windows: Windows 10 Pro 1709 x64
2018-08-10 18:38:45 +00:00
MacOS: MacOS High Sierra 10.13.3
2018-08-10 17:02:31 +00:00
```
2018-08-18 09:48:25 +00:00
## vBIOS
I experienced some weird things when doing this on the display, like a corruption of the image, but seems to work fine. If you encounter anything, a reboot solved my problems.
2018-08-18 09:52:06 +00:00
### Method 1 - Linux
2018-08-18 09:48:25 +00:00
It did not work for me, the ROM is 59KiB and it should be around 162KiB. It may work for you.
1. Execute `scripts/iommu.sh` as **root** to get the BUS ID for the GPU. Looks like `0000:06:00.0`.
2. Edit `scripts/extract-vbios-linux.sh` to your convenience. Change `videobusid=`. [Optional] Change also the location where the vBIOS will be save `VBIOS=`.
3. Make the script executable with `chmod +x scripts/extract-vbios-linux.sh`
4. Link the service to systemd: `ln -s scripts/qemu@.service /usr/lib/systemd/system/`
5. Execute the systemd unit with `sudo systemctl start qemu@extract-vbios-linux.service`. You can also do it over `ssh`. The extracted ROM will be in the root directory `/root/vBIOS.rom`
From here you are alone, I don't know how to proceed. Maybe you need to edit, maybe don't.
2018-08-18 09:52:06 +00:00
### Method 2 - nvflash in Linux
2018-08-18 09:48:25 +00:00
1. Download nvflash https://www.techpowerup.com/download/nvidia-nvflash/. Do not install from AUR; the package it's broken.
2. Unzip it as `/root/nvflash_linux` with `# unzip nvflash_5.414.0_linux.zip -d /root/`
2018-08-18 09:52:06 +00:00
3. Execute `scripts/iommu.sh` as **root** to get the BUS ID for the GPU. Looks like `0000:06:00.0`.
4. Edit `scripts/extract-vbios-nvflash.sh`. Change `videobusid` with your GPU; `NVFLASH=` if you changed the location of the executable; and `VBIOS=` if you want the ROM in other place.
5. Link the service to systemd: `ln -s scripts/qemu@.service /usr/lib/systemd/system/`
6. Execute the systemd unit with `sudo systemctl start qemu@extract-vbios-nvflash.service`. You can also do it over `ssh`. The extracted ROM will be in the root directory `/root/vBIOS.rom`
7. [Edit the vBIOS](#edit-the-vbios)
2018-08-18 09:48:25 +00:00
2018-08-18 09:52:06 +00:00
### Method 3 - Windows
2018-08-18 10:10:31 +00:00
[Source](https://www.youtube.com/watch?v=1IP-h9IKof0). [You can download the bios from techpowerup.com](https://www.techpowerup.com/vgabios/); if you do so, [skip to edit the vBIOS](#edit-the-vbios).
1. [Download and install GPU-Z](https://www.techpowerup.com/gpuz/).
2. [Open GPU-Z and backup the GPU BIOS](/Screenshots/vBIOS.png). Right next to the `Bios Version`; in my case `80.04.C3.00.0F`, there is an icon for backup. A file named `GK104.rom` will be created [Your file name may vary].
3. [Edit the vBIOS](#edit-the-vbios)
2018-08-18 09:48:25 +00:00
### Edit the VBIOS
1. Open the vBIOS ROM (`vBIOS.rom`) in the HEX editor.
2. [After a bunch of `00` there is a `55` or `U` in HEX, delete everything before the `55`](/Screenshots/Hex vBIOS.png), and save. I strongly recommend not to overwrite the original ROM.
2018-08-10 17:02:31 +00:00
## Configure
1. Clone this repository
```bash
2018-08-17 12:58:40 +00:00
$ git clone https://gitlab.com/YuriAlek/vfio.git
2018-08-10 17:02:31 +00:00
```
2018-08-17 12:58:40 +00:00
2. [Optional] [Download virtio drivers](https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers). If you do not, modify the `windows.sh` script.
2018-08-10 17:02:31 +00:00
```
2018-08-17 12:58:40 +00:00
$ wget -o virtio-win.iso "https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso"
2018-08-10 17:02:31 +00:00
```
2018-08-17 12:58:40 +00:00
4. Get the iommu groups needed for the VM (GPU, GPU audio and USB controller)
```
$ chmod +x scripts/iommu.sh
$ scripts/iommu.sh
2018-08-10 17:42:57 +00:00
-------------------------
2018-08-10 17:02:31 +00:00
# GPU
IOMMU group 13
2018-08-10 17:42:57 +00:00
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
06:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
2018-08-10 17:02:31 +00:00
# USB 3.0 Controller
IOMMU group 16
2018-08-10 17:42:57 +00:00
07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
2018-08-10 17:02:31 +00:00
# SATA Controller
IOMMU group 18
2018-08-10 17:42:57 +00:00
08:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
2018-08-10 17:02:31 +00:00
```
2018-08-17 12:58:40 +00:00
5. [Optional] Create the image for the VM. Only if not using a physical hard drive.
```bash
$ qemu-img create -f raw windows.raw 60G
2018-08-10 17:02:31 +00:00
```
2018-08-17 12:58:40 +00:00
6. Edit the script `windows-install.sh` **and** `windows.sh` to convenience. Things you may have to edit:
2018-08-10 17:02:31 +00:00
1. PCI devices
2. User
2018-08-10 17:42:57 +00:00
3. Location of HDD, ISO, vBIOS and OVMF image
4. The Desktop Environment, Display Manager, Window Manager, etc.
5. QEMU options like RAM and CPU
2018-08-10 17:02:31 +00:00
6. Kernel modules
2018-08-17 12:58:40 +00:00
7. Start the VM
2018-08-10 17:02:31 +00:00
```
2018-08-17 12:58:40 +00:00
# scripts/windows-install.sh
2018-08-10 17:02:31 +00:00
```
2018-08-17 12:58:40 +00:00
8. When installing Windows, in the section `Where do you want to install Windows?` there will be no hard drives to install to; to fix it:
2018-08-10 17:02:31 +00:00
1. Load driver
2. Browse
3. CD Drive (E:) virtio-win-0.1.1
4. vioscsi
5. w10
6. amd64
7. ok
8. Load driver `Red Hat VirtIO SCSI pass-through controller (E:\vioscsi\w10\amd64\vioscsi.inf)`
9. Next
10. Select the `Unallocated Space`
11. Proceed as normal.
2018-08-10 18:38:45 +00:00
12. Let Windows find the drivers for the GPU (if Windows has network) or [download the updated ones from NVIDIA](https://www.nvidia.com/Download/index.aspx?lang=en-us).
2018-08-10 17:02:31 +00:00
2018-08-17 12:58:40 +00:00
9. Once installed Windows, run the VM with:
2018-08-10 17:02:31 +00:00
```
2018-08-17 12:58:40 +00:00
# scripts/windows.sh
2018-08-10 17:02:31 +00:00
```
2018-08-10 17:21:08 +00:00
### For the sake of convenience
2018-08-17 12:58:40 +00:00
```
# ln -s scripts/qemu@.service /usr/lib/systemd/system/
2018-08-10 17:29:45 +00:00
```
2018-08-10 17:42:57 +00:00
```bash
2018-08-17 12:58:40 +00:00
$ alias windows="sudo systemctl start qemu@windows.service"
$ alias macos="sudo systemctl start qemu@macos-hs.service"
2018-08-10 17:42:57 +00:00
```
To start the Windows VM
```
2018-08-17 12:58:40 +00:00
$ windows
2018-08-10 17:42:57 +00:00
```
To start the MacOS VM
2018-08-10 17:29:45 +00:00
```
2018-08-17 12:58:40 +00:00
$ macos
2018-08-10 17:29:45 +00:00
```
2018-08-10 17:02:31 +00:00
2018-08-18 09:48:25 +00:00
### For MacOS
[Check this amazing guide](https://github.com/kholia/OSX-KVM) for creating the MacOS install image and Clover.
2018-08-10 17:02:31 +00:00
## Known problems
### Race condition
2018-08-10 17:15:39 +00:00
There is something somewhere that makes it crash. That's why there is so many `sleep`
2018-08-10 17:02:31 +00:00
2018-08-10 17:21:08 +00:00
### MacOS does not like USB hubs, therefore anything connected to a hub will be ignored by MacOS
2018-08-10 17:15:39 +00:00
2018-08-17 12:58:40 +00:00
### Sometimes works, sometimes does not
2018-08-18 09:48:25 +00:00
Sometimes the GPU will not have correct drivers inside Windows, even when yesterday was working just fine; Windows may install them.
2018-08-17 12:58:40 +00:00
Sometimes the QEMU command will just fail and the command continues and start X again.
Sometimes the QEMU command does not exit after shutting down the VM.
### Windows version
Windows 10 Pro 1709 works for me, but 1803 does not (may be the UEFI). I have heard that the 1803 version comes with a Spectre patch and the performance is pathetic.
2018-08-10 17:15:39 +00:00
## TODO
- [x] Unbind GPU without `virsh`
2018-08-10 18:38:45 +00:00
- [x] Update macos script
2018-08-18 09:48:25 +00:00
- [x] Run QEMU as user.
- [x] Try if is necessary to edit `/etc/mkinitcpio.conf`. No need to load the kernel modules at boot.
2018-08-10 17:15:39 +00:00
- [ ] Network
2018-08-10 17:29:45 +00:00
- [ ] Audio
2018-08-17 12:58:40 +00:00
- [ ] IOMMU guide
- [ ] Troubleshooting guide
- [ ] Extract the vBIOS in Linux guide
2018-08-10 17:29:45 +00:00
- [ ] How to edit the `windows.sh` script
- [ ] Fix the race condition
- [ ] Create scripts for install and use (Without DVD images)
2018-08-18 09:48:25 +00:00
- [ ] ACS Patch (Does not work for me).
- [ ] Install guide
2018-08-10 17:15:39 +00:00
- [ ] ???
2018-08-10 18:20:06 +00:00
2018-08-10 18:42:26 +00:00
<!--
And you must supply QEMU with the Full GPU's ROM extracted extracted using a tool called "nvagetbios" , which you can find in a package called "envytools"
-->
2018-08-10 18:20:06 +00:00
<!-- Links -->
[Xen Wiki]: https://wiki.xen.org/wiki/VTd_HowTo
[IOMMU Hardware]:https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware
[archwiki-url]: https://wiki.archlinux.org/index.php/Main_page
[qemu_archwiki]: https://wiki.archlinux.org/index.php/QEMU
[kvm-archwiki]: https://wiki.archlinux.org/index.php/KVM
[pci_passthrough-archwiki]: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
[libvirt_archwiki]: https://wiki.archlinux.org/index.php/Libvirt