Switch GPU instance to VM with PCI passthrough #57

Merged

AlexCaswen merged 2 commits from gpu-vm-passthrough into main

2026-05-10 05:47:19 +00:00

AlexCaswen commented

2026-05-10 05:45:29 +00:00

(Migrated from gitlab.com)

Why

IncusOS gpu-support doesn't bundle NVIDIA kernel modules or userspace (incus-os #992). Container GPU passthrough (nvidia.runtime=true) has nothing to work with — no /dev/nvidia* nodes, no nvidia-smi, no libcuda.

VMs bypass this entirely: raw PCI passthrough gives the VM direct hardware access. The VM loads its own NVIDIA driver.

Changes

nixos-image.tf

Add incus_image.nixos_unstable_vm (VM variant of NixOS/unstable)

gpu.tf

type: container → virtual-machine
image: container image → VM image
profiles: ["default"] only (nixos-container profile is container-specific)
nvidia.runtime: removed (VM loads its own driver)
gpu.pci: set to 0000:01:00.0 (specific PCI passthrough)
Boot config set directly (not from profile)

Plan

Terraform will destroy the existing container and create a new VM. The gpu-data volume on m3-trade-pool persists.

Post-merge

The VM will boot with raw GPU access. Install NVIDIA driver inside the VM's NixOS config to get nvidia-smi and CUDA working.

## Why IncusOS gpu-support doesn't bundle NVIDIA kernel modules or userspace (incus-os #992). Container GPU passthrough (`nvidia.runtime=true`) has nothing to work with — no `/dev/nvidia*` nodes, no nvidia-smi, no libcuda. VMs bypass this entirely: raw PCI passthrough gives the VM direct hardware access. The VM loads its own NVIDIA driver. ## Changes ### nixos-image.tf - Add `incus_image.nixos_unstable_vm` (VM variant of NixOS/unstable) ### gpu.tf - `type`: `container` → `virtual-machine` - `image`: container image → VM image - `profiles`: `["default"]` only (nixos-container profile is container-specific) - `nvidia.runtime`: removed (VM loads its own driver) - `gpu.pci`: set to `0000:01:00.0` (specific PCI passthrough) - Boot config set directly (not from profile) ## Plan Terraform will destroy the existing container and create a new VM. The `gpu-data` volume on `m3-trade-pool` persists. ## Post-merge The VM will boot with raw GPU access. Install NVIDIA driver inside the VM's NixOS config to get nvidia-smi and CUDA working.

AlexCaswen (Migrated from gitlab.com) approved these changes

2026-05-10 05:45:29 +00:00

AlexCaswen commented

2026-05-10 05:46:41 +00:00

(Migrated from gitlab.com)

assigned to @AlexCaswen