Passing GPUs into Processing Containers from Python

The dense matching job runs at CPU speed inside the container and at GPU speed outside it. The host has a working GPU, the image was built with CUDA support, and the job reports no error — it simply takes eleven times longer.

There are two independent things that must be true for a container to use a GPU, they fail separately, and from the application’s point of view they look identical. This page covers requesting the device correctly from Python, distinguishing the two failures, and sharing one GPU between concurrent workers without them destroying each other. It extends containerising photogrammetry workers with Docker.

Two layers, one symptom

Device passthrough puts the GPU device nodes and the host’s driver libraries into the container. Without it the container has no GPU at all, and any library that probes for one finds nothing.

Library support is a property of the image. An OpenCV built without CUDA reports zero CUDA devices even on a machine with four, because it has no code to talk to them.

Both produce “no GPU available” from the application. The first is fixed by a flag on the run; the second requires a different image. Establishing which is present takes two commands and saves an afternoon.

The two independent requirements for GPU use in a container A two-by-two matrix of device passthrough against library support. With neither, the application sees no GPU and runs on the CPU. With passthrough but no library support, nvidia-smi works inside the container while the application still reports zero devices, which is the confusing case. With library support but no passthrough, the library is present and finds nothing. With both, the GPU is usable. Each cell names the remedy: a run flag, a different image, or both. no passthrough passthrough enabled no CUDA build CUDA build nothing works nvidia-smi: not found application: 0 devices fix: flag and image the confusing case nvidia-smi: works application: 0 devices fix: rebuild the image library present, no device nvidia-smi: not found application: 0 devices fix: add the run flag GPU usable both layers present record the driver version in the run manifest

Figure 1 — Four states, three of which report the same thing to the application.

Minimal reproducible solution

import docker


def run_gpu_worker(image: str, command: list[str], *, mounts: list,
                   device_ids: list[str] | None = None,
                   memory_limit: str = "24g") -> dict:
    """Run a container with specific GPUs made available.

    device_ids selects which GPUs; None requests all of them. Selecting
    explicitly is what makes concurrent workers possible — two jobs that both
    request all devices will contend for the same memory and fail in ways
    that look like application bugs.
    """
    client = docker.from_env()
    request = docker.types.DeviceRequest(
        device_ids=device_ids if device_ids else None,
        count=-1 if not device_ids else 0,
        capabilities=[["gpu", "compute", "utility"]])

    container = client.containers.run(
        image, command=command, mounts=mounts, detach=True,
        device_requests=[request], mem_limit=memory_limit,
        shm_size="4g",
        environment={"NVIDIA_VISIBLE_DEVICES": ",".join(device_ids) if device_ids else "all"})
    status = container.wait()
    logs = container.logs(tail=300).decode("utf-8", "replace")
    container.remove()
    return {"exit_code": status["StatusCode"], "logs": logs,
            "devices": device_ids or "all"}

Probing both layers

import subprocess


def gpu_diagnostics(image: str) -> dict:
    """Distinguish a passthrough failure from a library failure."""
    findings = {}

    smi = subprocess.run(["docker", "run", "--rm", "--gpus", "all", image,
                          "nvidia-smi", "--query-gpu=name,memory.total,driver_version",
                          "--format=csv,noheader"],
                         capture_output=True, text=True)
    findings["passthrough"] = smi.returncode == 0
    findings["devices"] = smi.stdout.strip().splitlines() if smi.returncode == 0 else []
    if smi.returncode != 0:
        findings["passthrough_error"] = smi.stderr.strip()[:200]

    lib = subprocess.run(
        ["docker", "run", "--rm", "--gpus", "all", image, "python3", "-c",
         "import cv2, json; "
         "print(json.dumps({'opencv': cv2.__version__, "
         "'cuda_devices': cv2.cuda.getCudaEnabledDeviceCount()}))"],
        capture_output=True, text=True)
    findings["library_ok"] = lib.returncode == 0 and '"cuda_devices": 0' not in lib.stdout
    findings["library_output"] = (lib.stdout or lib.stderr).strip()[:200]

    if not findings["passthrough"]:
        findings["diagnosis"] = "device not passed through — install or fix the container toolkit"
    elif not findings["library_ok"]:
        findings["diagnosis"] = "device present, library built without CUDA — rebuild the image"
    else:
        findings["diagnosis"] = "GPU is usable from the application"
    return findings

Running this once when a new image or a new host enters service turns the most common support question on a GPU fleet into a one-line answer.

Verifying a GPU is genuinely usable inside the container A four-stage check. Stage one confirms the device is visible, since a container started without the runtime flag sees no device at all. Stage two confirms the driver and runtime versions are compatible, because a container built against a newer runtime than the host driver supports fails at the first allocation rather than at start. Stage three confirms memory can actually be allocated, which is where a device already fully used by another container reveals itself. Stage four runs a small representative kernel and compares its result against the CPU path, which is the only stage that proves the device computes correctly rather than merely responding. 1. visible the runtime flag was passed at all 2. compatible container runtime against host driver version 3. allocatable memory free, or already taken by another container 4. correct a small kernel against the CPU path Only stage 4 distinguishes a working device from one that merely answers.

Figure 3 — Four checks, and the first three are all passed by a broken setup.

Edge-case matrix

Situation Symptom Handling
Container toolkit not installed nvidia-smi not found in container Install it on the host
Image lacks CUDA build nvidia-smi works, app sees zero Rebuild or use a CUDA image
Driver older than the image’s CUDA Cryptic initialisation error Driver must be at least as new
Two workers, one GPU Out-of-memory in both Assign device ids explicitly
Multi-GPU host, job uses one Wasted capacity Partition by device id
--gpus all with MIG enabled Unexpected device list Reference MIG instances explicitly
Container memory limit too low Killed during transfer GPU work still needs host memory
shm_size at the default Multi-process matcher crashes Raise it; unrelated to the GPU but coincident

The driver-version row causes the most confusing failures. A CUDA runtime inside the image requires a host driver at least as new; an older driver produces an initialisation error that names neither version, and the fix is on the host rather than in the image.

def driver_compatible(driver_version: str, required_major: int) -> dict:
    """Is the host driver new enough for the image's CUDA runtime?"""
    try:
        major = int(str(driver_version).split(".")[0])
    except (ValueError, IndexError):
        return {"ok": False, "note": f"cannot parse driver version {driver_version!r}"}
    return {"ok": major >= required_major, "driver_major": major,
            "required_major": required_major,
            "note": ("driver is new enough" if major >= required_major else
                     "host driver is older than the image's CUDA runtime — "
                     "upgrade the host, not the image")}

Sharing one GPU between workers

A single GPU can serve several workers, and doing it naively is worse than not doing it: two jobs that each allocate most of the device memory both fail partway through, having each wasted the time spent up to that point.

import itertools


class GpuPool:
    """Hand out GPU device ids so concurrent workers do not contend.

    A simple exclusive assignment is right for photogrammetry: dense matching
    fills the device memory, so two jobs on one GPU is not a throughput gain,
    it is two failures.
    """

    def __init__(self, device_ids: list[str]):
        self._free = list(device_ids)
        self._in_use: dict[str, str] = {}

    def acquire(self, job_id: str) -> str | None:
        if not self._free:
            return None
        device = self._free.pop(0)
        self._in_use[device] = job_id
        return device

    def release(self, device: str) -> None:
        self._in_use.pop(device, None)
        if device not in self._free:
            self._free.append(device)

    def status(self) -> dict:
        return {"free": list(self._free), "in_use": dict(self._in_use)}

Exclusive assignment rather than fractional sharing is the right default here, and it is worth being explicit about why: the memory-bound stages of a photogrammetry pipeline use whatever the device has, so time-slicing two of them produces two slower jobs and a high chance that both exhaust memory.

Exclusive GPU assignment against unmanaged sharing Two timelines on a host with two GPUs and four queued jobs. With exclusive assignment, two jobs run concurrently on one GPU each and the remaining two follow, all four completing in about fifty minutes. With unmanaged sharing, all four jobs start at once on both devices, each allocates most of the memory, and three of the four fail with out-of-memory after twenty minutes of wasted work, leaving one to complete. A note records that unmanaged sharing is slower and less reliable than a queue. exclusive assignment — all four complete job 1 · gpu 0 job 2 · gpu 1 job 3 · gpu 0 job 4 · gpu 1 50 min unmanaged sharing — three fail job 1 · OOM job 2 · OOM job 3 · OOM job 4 · completes, slowly 90 min A queue is faster and more reliable than contention, on memory-bound work.

Figure 2 — Why a pool with exclusive assignment beats letting jobs compete.

Recording what the GPU contributed

A GPU worker should report what it actually did, for two reasons: to prove the acceleration happened, and to give the scheduler something to plan with.

The proof matters because the fallback path is silent by design. A pipeline that detects no CUDA device and continues on the CPU is behaving correctly, and on a busy fleet nobody notices that every job has been falling back for three weeks until somebody looks at the timings.

import time


def record_gpu_use(stage: str, *, used_gpu: bool, device: str | None,
                   elapsed_s: float, points: int) -> dict:
    """Per-stage record of whether the GPU was used and what it bought."""
    return {"stage": stage, "used_gpu": used_gpu, "device": device,
            "elapsed_s": round(elapsed_s, 1),
            "points_per_second": round(points / max(elapsed_s, 1e-9)),
            "note": ("GPU path" if used_gpu else
                     "CPU fallback — expect roughly an order of magnitude slower")}


def fallback_rate(records: list[dict]) -> dict:
    """How often the fleet has been silently falling back to the CPU."""
    gpu_stages = [r for r in records if r["stage"] in {"dense_matching", "depth_maps"}]
    if not gpu_stages:
        return {"note": "no GPU-eligible stages in this window"}
    fell_back = sum(1 for r in gpu_stages if not r["used_gpu"])
    return {"stages": len(gpu_stages), "fell_back": fell_back,
            "fallback_rate": fell_back / len(gpu_stages),
            "alert": fell_back / len(gpu_stages) > 0.05}

An alert on a fallback rate above a few percent is the monitoring that catches a driver upgrade, a toolkit removal or an image rebuild without CUDA — all of which are invisible from the outputs and obvious from the timings.

When to escalate

  • The driver cannot be upgraded on the host. Build the image against an older CUDA runtime. The constraint is real and the image is the flexible side.
  • A job needs more GPU memory than any single device has. Tile the work, or use a device with more memory. Splitting across devices is not automatic for most photogrammetry libraries.
  • Performance is poor with the GPU correctly attached. The bottleneck may be host-to-device transfer rather than compute, which is a pipeline-shape problem covered in fixing out-of-memory on GPU during dense matching.

← Containerising Photogrammetry Workers with Docker