GPU PCI Pass Through

From Sea of Fate
Jump to navigationJump to search

Introduction

Wit the the successful GPU PCI Pass Through of the Nvidia GPU, Walnut, is able to run jellyfish quite well. However, I would prefer to have a Linux host on the Home Lab with the GPU mainly because I already have a Windows 11 desktop with a powerful GPU so it will be better to have this the small Nvida on a Linux host and within a Home Lab enviroment. I am getting the AI Gemmini to generate the instructions for the wiki so they may be a bit different to the usual form of my notes

Updated information

Any changes to the GPU passthough need to be noted here so that We have a complete history of the GPU passthrough

Addition of 2.5 Gb p/s NIC

The addition of a 2.5 Gb p/s NIC and managed switch to the homelab detailed in Network Configuration caused the GPU to change its PCIe bus information because the NIC grouped the all of the PCIes together and while they were being separated the onboard NIC somehow changed to ID 6 and the GPU changed to ID 7. So when Walnut booted Proxmox passed the NIC through to Walnut and as walnut was expecting it to be a GPU it failed to boot and as the onboard NIC is used to manage the Proxmox PVE the connection to the WebGUI was lost. The resulting lesson learned is that any changes to PCIe or GPU will have unforeseen effects and to check the configuration carefully.

NVIDIA RTX 5060 PCI Passthrough on Proxmox VE

This guide details the steps for successfully passing through an NVIDIA RTX 5060 consumer GPU to a Virtual Machine (VM) on Proxmox VE. This process requires careful configuration to ensure the host releases the GPU and the guest can utilize it.

Note: This guide assumes you are running Proxmox VE 8.x with a 6.8 series kernel and an AMD CPU (Ryzen platform). Specific commands and file contents reflect the issues encountered and resolved in your specific setup.

1. Host Preparation (Proxmox VE - `pear`)

This phase configures the Proxmox host to release the GPU to the `vfio-pci` driver.

1.1. Enable IOMMU and Essential Kernel Parameters

Edit the GRUB configuration file to enable IOMMU (I/O Memory Management Unit) and add crucial parameters for passthrough.

  1. ssh root@your_proxmox_host_ip
nano /etc/default/grub

Locate the line starting with `GRUB_CMDLINE_LINUX_DEFAULT=` and ensure it contains the following parameters: 

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
  • (Optional, if experiencing IOMMU grouping issues): `pcie_acs_override=downstream,multifunction`
  • (Optional, if experiencing persistent framebuffer issues): `video=efifb:off nomodeset`

Save and exit the file.

1.2. Update GRUB and Initramfs

Apply the GRUB changes and ensure the initial ramdisk (initramfs) is updated to include the necessary modules and configurations for passthrough. 

update-grub
update-initramfs -u -k all

1.3. Add VFIO Modules for Early Loading

Ensure the VFIO modules (which claim PCI devices for passthrough) are loaded early in the boot process. 

nano /etc/modules

Add the following lines to the end of the file: 

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Save and exit.

1.4. Force VFIO to Bind to the NVIDIA GPU's PCI ID

This is critical to prevent the host's NVIDIA drivers (like `nouveau` or `nvidia`) from claiming the GPU.

  1. Identify your GPU's PCI ID (Vendor ID:Device ID)
lspci -n -s 06:00.0
  • Example output: 06:00.0 0300: 10de:2d05 (rev a1)
  • Note: 10de is NVIDIA's Vendor ID, 2d05 is the Device ID for the VGA function.
  • Also identify the GPU's Audio Device PCI ID (e.g., 06:00.1 0403: 10de:22eb)
nano /etc/modprobe.d/vfio.conf

(Create this file if it doesn't exist)

Add this line, replacing `10de:2d05` and `10de:22eb` with your GPU's exact Vendor:Device IDs for both the VGA and Audio functions: 

options vfio-pci ids=10de:2d05,10de:22eb

Save and exit.

1.5. Aggressively Blacklist Conflicting Host Drivers

This ensures `nouveau` and `nvidia` drivers don't load on the host, allowing VFIO to claim the GPU.

root@pear:~# nano /etc/modprobe.d/blacklist-nouveau.conf

Add/ensure these lines are present:

blacklist nouveau options nouveau modeset=0

Save and exit.

1.6. Final Host Reboot

This reboot is crucial for all the host-level kernel, module, and GRUB changes to take effect.

root@pear:~# reboot

2. Verify GPU Binding on Host

After the host reboots, verify that the `vfio-pci` driver has successfully claimed your NVIDIA GPU.

root@pear:~# lspci -k | grep -i "nvidia" -A 3

Expected Output: You should see `Kernel driver in use: vfio-pci` for both your GPU (06:00.0) and its audio device (06:00.1). If `nouveau` or `nvidia` are still listed as "Kernel driver in use", the host is still controlling the GPU, and passthrough will fail.

3. Configure VM for Passthrough (Windows 11 - `walnut` or Ubuntu Desktop - `lychee`)

This phase configures the Virtual Machine to receive the passed-through GPU.

3.1. Shut Down Current GPU VM (if switching)

If the GPU is currently passed through to another VM eg Walnut , shut down that VM gracefully and remove the PCI Device from its hardware configuration in the Proxmox web GUI.

3.2. Set VM Display to None

To prevent conflicts with the passed-through GPU and to ensure the VM uses the physical GPU for display, set its virtual display adapter to "None".

  • In the Proxmox web GUI: Select your target VM (e.g., `walnut` or `lychee`).
  • Go to the Hardware tab.
  • Find the Display device.
  • Click "Edit" and change its "Graphics card" to None.
  • Click "OK".

3.3. Add PCI Device (NVIDIA GPU) to VM

Now, add the physical NVIDIA GPU to your VM's hardware configuration.

  • In the Proxmox web GUI: Select your target VM.
  • Go to the Hardware tab.
  • Click Add -> PCI Device.
  • In the dropdown, select your NVIDIA RTX 5060 (it should appear in the list with both functions, e.g., `06:00.0 / 06:00.1`).
  • Check All Functions.
  • Check Rom-Bar (often required for consumer NVIDIA cards).
  • Check PCI-Express (requires QEMU `machine: q35` in VM config).
  • Check Primary GPU (often necessary, especially for Windows guests).
  • Click "Add".

3.4. (Optional) Pass Through a USB Controller for Native Input

For native, low-latency keyboard and mouse input (especially crucial for Windows guests with passthrough), pass through a dedicated USB controller from your host.

  1. Identify a suitable USB controller on your Proxmox host

root@pear:~# lspci -v | grep -i usb

  1. Choose one not critical for host operation (e.g., 08:00.3)
  2. Get its Vendor:Device ID: lspci -n -s 08:00.3
  3. Example: 1022:43ea
  • Add an options line to `/etc/modprobe.d/vfio-usb.conf` (create if needed):
 `options vfio-pci ids=1022:43ea`
  • Update initramfs: `sudo update-initramfs -u -k all`
  • Reboot the Proxmox host.
  • In Proxmox web GUI, for the VM, Add -> PCI Device -> select the USB controller -> check All Functions, Rom-Bar.
  • Physically plug your keyboard and mouse into the USB ports controlled by this passed-through controller.

3.5. Adjust VM Config File (if needed for Code 43 on Windows)

For Windows guests with consumer NVIDIA GPUs, a "Code 43" error in Device Manager is common. This often requires adding specific QEMU arguments.

  • Edit `/etc/pve/qemu-server/<vmid>.conf` (replace `<vmid>` with your VM's ID, e.g., 307 for `walnut`).
  • Ensure the `machine` type is `q35`.
  • Add these lines (or modify existing `cpu:` and `args:`):

machine: q35 cpu: host,hidden=1,flags=+pcid args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NVIDIA,kvm=off'

  • Reboot the Proxmox host after making changes to this file.

4. Start VM and Install Guest Drivers

4.1. Start the VM

Start your VM from the Proxmox web GUI. It should now boot.

4.2. Install NVIDIA Drivers in Guest VM

  • For Windows 11 (`walnut`):
   * Log into Windows.
   * Download the latest NVIDIA GeForce drivers for your RTX 5060 from NVIDIA's website.
   * Run the installer.
   * After installation, reboot the VM (gracefully from within Windows).
   * Verify in Device Manager that the NVIDIA GPU is listed without errors.
  • For Ubuntu Desktop (`lychee`):
   * Log into `lychee` (via NoMachine if already set up, or temporary VNC if basic display driver is working).
   * Install necessary build tools and headers: `sudo apt update && sudo apt install linux-headers-$(uname -r) build-essential dkms`
   * Download the NVIDIA driver `.run` file for Linux from NVIDIA's website.
   * Install the driver: `sudo sh NVIDIA-Linux-x86_64-XXX.YY.run --dkms --no-cc-version-check`
   * Generate Xorg configuration: `sudo nvidia-xconfig`
   * Reboot the VM.

5. Remote Access to VM (Important for GPU Passthrough)

Once GPU passthrough is active, the Proxmox web GUI console for the VM will no longer display the VM's desktop. You'll need to use a remote desktop solution to interact with the VM.

  • For Windows VMs (like `walnut`): Use RDP (Remote Desktop Protocol) from another PC.
   * Ensure RDP is enabled within the Windows VM (Settings -> System -> Remote Desktop).
   * Configure RDP client settings (on your local desktop) for optimal performance (e.g., lower visual quality settings, lower resolution, higher color depth if bandwidth allows) to reduce input lag and blur.
  • For Linux Desktop VMs (like `lychee`): Use NoMachine (highly recommended for performance), or VNC, or SSH with X11 forwarding.
   * NoMachine: Offers superior performance over VNC due to its efficient compression. Install the NoMachine server on `lychee` and the client on your remote PC. Remember to open port 4000/TCP in `lychee`'s firewall and pfSense (if connecting directly), or ideally, connect via your VPN.
   * VNC: (e.g., TigerVNC, x11vnc). Provides a full desktop. Less performant than NoMachine. Always tunnel over SSH for security or ensure your VPN protects the connection.
   * SSH with X11 Forwarding: Best for running individual graphical applications. You connect via SSH and launch the app from the command line, and its window appears on your local desktop. Less suitable for a full desktop experience.
Retrieved from "https://wiki.seaoffate.net/index.php?title=GPU_PCI_Pass_Through&oldid=662"