Configure Proxmox 9 with a GPU for a vLLM VM

Run your own language models on your own terms, in your own data centre – with no cloud lock-in. This guide walks you through, step by step, how to pass an NVIDIA GPU through to a virtual machine on Proxmox VE 9 using IOMMU/VFIO and run vLLM inside it as a high-performance, OpenAI-compatible LLM server.

Architecture & idea

Why a VM at all? Because clean separation matters. The Proxmox host stays lean and only handles virtualisation, storage and networking. The GPU is fully detached from it and handed over to a single VM. That VM holds the entire AI stack – NVIDIA driver, Docker and the vLLM container – isolated, versioned and ready to back up or migrate as one unit.

vLLM is an inference engine that does two things exceptionally well: it keeps GPU memory tightly packed with PagedAttention and it bundles many concurrent requests using continuous batching. To the outside world it speaks the OpenAI API – your existing tools, SDKs and RAG pipelines work without any rewrite.

Requirements

For reliable passthrough you need hardware that supports I/O-MMU virtualisation – standard today on server and most desktop platforms:

• CPU: Intel with VT-x and VT-d or AMD with AMD-V and AMD-Vi (IOMMU).
• Motherboard/BIOS: IOMMU support and, ideally, clean IOMMU groups.
• GPU: a professional NVIDIA GPU – pro-grade RTX (e.g. RTX 4000/5000/6000 Ada) or data-centre (L4, L40S, A100, H100) – with enough VRAM for your model.
• Software: Proxmox VE 9.0, freshly installed and updated.

Start by checking the version and CPU capabilities directly on the host:

# Show the Proxmox version and kernel
pveversion -v | head -n 3

# CPU vendor and virtualisation flags
lscpu | grep -E "Virtualization|Vendor ID|Model name"

1.Prepare BIOS/UEFI

In the server's firmware setup, enable the following (names vary by vendor):

• Intel VT-x / AMD SVM Mode – CPU virtualisation.
• Intel VT-d / AMD IOMMU – the prerequisite for passthrough.
• Above 4G Decoding – required for modern GPUs with large BARs.
• Re-Size BAR Support – if available, for better performance.
• Primary Display – set to the iGPU/"onboard" if possible, so the dedicated GPU stays free.

2.Enable IOMMU in the kernel

Depending on the install, Proxmox VE 9 uses one of two bootloaders: systemd-boot (with a ZFS root or UEFI) or GRUB. One command tells you which is active:

proxmox-boot-tool status

Option A – systemd-boot

Add the IOMMU parameters to the kernel command line in /etc/kernel/cmdline (all on one line) and rewrite the boot config:

# Intel CPU – append to the existing line:
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

# AMD CPU instead:  amd_iommu=on iommu=pt

proxmox-boot-tool refresh

Option B – GRUB

Edit /etc/default/grub, add the parameters to GRUB_CMDLINE_LINUX_DEFAULT and update GRUB:

# Intel:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# AMD:
# GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

update-grub
reboot

3.Load the VFIO modules

Register the VFIO kernel modules so they're ready at boot. On kernel 6.14, vfio_pci pulls in the rest automatically – but listing them explicitly does no harm:

vfio
vfio_iommu_type1
vfio_pci

4.Verify IOMMU & understand groups

After the reboot, IOMMU should be active. Verify it in the kernel log:

dmesg | grep -e DMAR -e IOMMU
# Expect, among others:  DMAR: IOMMU enabled

dmesg | grep 'remapping'
# Expect:  DMAR-IR: Enabled IRQ remapping  (Intel)
#          AMD-Vi: Interrupt remapping enabled  (AMD)

The key to clean passthrough is the IOMMU groups. A group is the smallest unit that can be isolated and passed through. A GPU usually consists of two functions – the graphics part and the HDMI audio part – that belong together. This small script lists every group:

#!/bin/bash
# Print all IOMMU groups and the PCI devices they contain
for g in /sys/kernel/iommu_groups/*/devices/*; do
    n=${g%%/devices/*}; n=${n##*/}
    printf 'IOMMU group %s: ' "$n"
    lspci -nns "${g##*/}"
done | sort -V

5.Bind the GPU to vfio-pci

First, find the exact device IDs. The form [10de:2204] is the vendor/device ID we'll hand to vfio-pci next:

lspci -nn | grep -Ei "nvidia|vga|audio"
# Example output:
# 01:00.0 VGA compatible controller [0300]: NVIDIA ... [10de:2204]
# 01:00.1 Audio device          [0403]: NVIDIA ... [10de:1aef]

Now bind both IDs to vfio-pci and use softdep to make sure vfio-pci grabs the device before any NVIDIA driver:

# Bind the GPU (VGA) + audio function firmly to vfio-pci
options vfio-pci ids=10de:2204,10de:1aef

# vfio-pci must load before the NVIDIA drivers
softdep nouveau  pre: vfio-pci
softdep nvidia   pre: vfio-pci
softdep nvidiafb pre: vfio-pci
softdep drm      pre: vfio-pci

Also banish the host drivers with a blacklist, so the Proxmox host never touches the card:

blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist nvidia_drm

update-initramfs -u -k all
reboot

After the reboot, confirm that vfio-pci – not nouveau or nvidia – actually holds the card:

lspci -nnk -s 01:00.0
# Kernel driver in use: vfio-pci   <-- exactly what you want

6.Create the VM (q35 + OVMF)

For PCIe passthrough the VM needs the q35 machine type and OVMF (UEFI) rather than SeaBIOS. You can create the VM in the web UI – here's the reproducible CLI path:

# New VM with the q35 chipset and UEFI/OVMF
qm create 100 \
  --name vllm-gpu \
  --machine q35 \
  --bios ovmf \
  --cpu host \
  --cores 8 --sockets 1 \
  --memory 32768 \
  --scsihw virtio-scsi-single \
  --net0 virtio,bridge=vmbr0 \
  --ostype l26

# Add the EFI disk (for UEFI variables) and the system disk
qm set 100 --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0
qm set 100 --scsi0 local-lvm:80,ssd=1,discard=on

# Attach the Ubuntu Server ISO as a CD-ROM and set the boot order
qm set 100 --ide2 local:iso/ubuntu-24.04-live-server-amd64.iso,media=cdrom
qm set 100 --boot 'order=scsi0;ide2'

7.Pass the GPU through

Now assign the GPU to the VM. Give only the device address without the function number (0000:01:00) – that passes both functions (.0 and .1) through together:

qm set 100 --hostpci0 0000:01:00,pcie=1,x-vga=1
# pcie=1   -> attach as a native PCIe device (requires q35)
# x-vga=1  -> use the GPU as the VM's primary display

The result lands in the VM config /etc/pve/qemu-server/100.conf and looks roughly like this:

machine: q35
bios: ovmf
cpu: host
hostpci0: 0000:01:00,pcie=1,x-vga=1
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=0,size=4M
scsi0: local-lvm:vm-100-disk-1,discard=on,size=80G,ssd=1

8.Guest: install the NVIDIA driver

Install Ubuntu 24.04 LTS in the VM. After the first boot, the GPU should be visible inside the guest:

# Did the GPU arrive in the guest?
lspci | grep -i nvidia

# Build tools and DKMS for the driver
sudo apt update && sudo apt install -y build-essential dkms

# Pick and install the recommended NVIDIA driver automatically
sudo ubuntu-drivers install
# or explicitly:  sudo apt install -y nvidia-driver-570

sudo reboot

After the reboot, nvidia-smi must show the card cleanly – the moment of truth:

nvidia-smi
# Shows driver version, GPU model, VRAM and running processes.

9.Docker & container toolkit

We run vLLM as the official container – it bundles CUDA, PyTorch and every dependency and keeps the guest clean. For that we install Docker and the NVIDIA Container Toolkit, which passes the GPU into the container:

# Install Docker
curl -fsSL https://get.docker.com | sh

# NVIDIA Container Toolkit – add the package repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit

# Point Docker at the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Test: is the GPU visible inside a container?
docker run --rm --gpus all ubuntu nvidia-smi

10.Start the vLLM container

A single docker run command starts the OpenAI-compatible server: it loads the model, sets up the HTTP API and reserves GPU memory. --gpus all passes the GPU in and --ipc=host gives vLLM enough shared memory:

docker run -d --name vllm --gpus all --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e "HF_TOKEN=your-hf-token" \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name llama3.1-8b \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --api-key "your-secret-key"

If you have multiple GPUs, vLLM shards a large model across the cards with tensor parallelism:

# Shard the model across, say, 2 GPUs
docker run -d --name vllm --gpus all --ipc=host \
  -p 8000:8000 -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-32B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 --max-model-len 16384

11.Run it with Docker Compose

For continuous operation we cast the container into a docker-compose.yml – with automatic restart, fixed configuration and clean logs:

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    restart: unless-stopped
    ports:
      - "8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - HF_TOKEN=your-hf-token
    ipc: host
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --served-model-name llama3.1-8b
      --gpu-memory-utilization 0.90 --max-model-len 8192
      --api-key your-secret-key
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

docker compose up -d
docker compose logs -f vllm

12.Test it

The endpoint behaves like the OpenAI API. A quick test with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{
    "model": "llama3.1-8b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain PagedAttention in one sentence."}
    ]
  }'

And the same request with the official OpenAI Python SDK – only the base_url changes:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-key",
)

resp = client.chat.completions.create(
    model="llama3.1-8b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Key vLLM parameters

Performance tuning

• Maximise the KV cache: set --gpu-memory-utilization as high as stays stable – more cache means more throughput under concurrent load.
• Keep the context realistic: an unnecessarily high --max-model-len reserves cache that then can't be used for concurrency.
• Quantisation: FP8 or AWQ/GPTQ roughly halve the memory footprint of large models with little quality loss.
• CPU pinning & NUMA: set CPU affinity in the VM config and keep the GPU on the matching NUMA node – this noticeably lowers latency.
• Hugepages: for large models, hugepages reduce the guest's memory overhead.
• Pre-load the model: the first start downloads the model from the hub – after that it sits in the mounted Hugging Face cache volume and starts fast.

Troubleshooting

Security & operations

A local LLM server is a production service – treat it like one. Three ground rules:

• Isolate the network: never expose port 8000 to the open internet. Run vLLM behind a reverse proxy (TLS, authentication, rate limiting) in a segregated network segment.
• Protect access: always set an --api-key and issue a separate key per application.
• Leverage data sovereignty: the decisive advantage – no prompt and no document leaves your premises. That's precisely what makes this approach GDPR-compliant and auditable.

Frequently asked questions

Sources

External sources, as of August 2026 (open in a new tab):

• Proxmox – Proxmox VE 9.0 (Debian 13, kernel 6.14, QEMU 10)
• Proxmox VE Wiki – PCI(e) passthrough (IOMMU/VFIO)
• vLLM – installation on NVIDIA GPUs
• vLLM – running with Docker (vllm/vllm-openai)
• NVIDIA – Container Toolkit: installation guide

This article is for general information. Commands, device IDs and version numbers are examples and must be adapted to your hardware. As of August 2026, Proxmox VE 9.0.