Detecting CUDA Availability and Falling Back to CPU

The pipeline runs on a workstation with a GPU, on a laptop without one, and on a rented instance where the GPU is sometimes already occupied by the job before yours. The same code has to do the right thing on all three, and the failure that keeps recurring is a startup ImportError or an AttributeError on a host that was never expected to have a device — a crash on the machine where the correct behaviour was simply “run slower”.

This page is a detection routine that distinguishes the four states a host can be in, returns a decision rather than an exception, and reports which decision it made loudly enough that nobody later wonders why a run took four hours.

Why availability is not a boolean

“Is CUDA available?” collapses four genuinely different situations into one answer, and three of them need different handling.

No CUDA build. The installed OpenCV wheel has no cv2.cuda module. Touching it raises AttributeError, not ImportError, because cv2 imported fine — which is why try: import cv2.cuda does not catch it.

CUDA build, no device. The module exists and getCudaEnabledDeviceCount() returns zero. Nothing is wrong; this is a CPU host running a GPU-capable build, which is the normal state of a CI runner.

Device present, driver mismatch. The count is positive but every device call fails, because the runtime the library was built against is newer than the installed driver. This one is the trap: detection succeeds and use fails, potentially an hour later.

Device present, insufficient free memory. Everything works and there is not enough room. Starting anyway produces an allocation failure part-way through, having already spent the time to get there.

Four host states behind one question Four host configurations arranged as cards. The first has no CUDA build, so touching the module raises an attribute error rather than an import error. The second has a CUDA build but no device, which is the normal state of a continuous-integration runner and is not an error. The third has a device but a driver older than the runtime, so detection succeeds and the first real device call fails, possibly much later. The fourth has a working device whose free memory is insufficient, so a run started anyway fails part-way through having already spent the time. Each card names the check that distinguishes it and the correct action. no CUDA build cv2 imports fine; cv2.cuda does not exist raises AttributeError, not ImportError check: hasattr(cv2, "cuda") action: host path normal, not a fault build, no device module present, device count is zero the usual state of a CI runner check: device count action: host path normal, not a fault driver mismatch count is positive; every device call fails detection succeeds, use fails an hour later check: a tiny probe op action: host path + warn a real misconfiguration device busy works, but free VRAM is below the floor starting anyway fails part-way through check: free memory action: host path or wait transient

Figure 1 — Four states, two of which are perfectly normal and two of which are faults. A boolean answer cannot distinguish “this host has no GPU” from “this host has a broken GPU”, and only one of those deserves a warning.

Minimal reproducible solution

The routine below returns a small record describing the decision and why it was made. It never raises, and every branch is reachable on some real host.

from dataclasses import dataclass
from typing import Literal

Backend = Literal["cuda", "cpu"]


@dataclass(frozen=True)
class Decision:
    backend: Backend
    reason: str
    device_index: int | None = None
    free_mb: float | None = None
    warn: bool = False          # True only for genuine misconfiguration


def choose_backend(min_free_mb: float = 4096) -> Decision:
    """Pick a backend and explain the choice. Never raises."""
    try:
        import cv2
    except ImportError:
        return Decision("cpu", "opencv not importable", warn=True)

    if not hasattr(cv2, "cuda"):
        return Decision("cpu", "opencv built without CUDA")

    try:
        count = cv2.cuda.getCudaEnabledDeviceCount()
    except cv2.error as exc:
        return Decision("cpu", f"cuda enumeration failed: {exc}", warn=True)

    if count < 1:
        return Decision("cpu", "no CUDA device present")

    # A device exists. Prove it works with the smallest possible operation
    # before committing an hour of matching to it.
    try:
        import numpy as np
        probe = cv2.cuda_GpuMat()
        probe.upload(np.zeros((8, 8), dtype=np.uint8))
        _ = probe.download()
    except cv2.error as exc:
        return Decision("cpu", f"device present but unusable: {exc}", warn=True)

    free_mb, index = _largest_free_device()
    if free_mb is None:
        return Decision("cuda", "device usable; free memory unknown", 0)
    if free_mb < min_free_mb:
        return Decision("cpu", f"only {free_mb:.0f} MB free on device {index}",
                        index, free_mb)
    return Decision("cuda", "device usable", index, free_mb)


def _largest_free_device() -> tuple[float | None, int | None]:
    """Free MB on the emptiest device, or (None, None) if NVML is unavailable."""
    try:
        import pynvml
        pynvml.nvmlInit()
    except Exception:
        return None, None
    try:
        best_mb, best_i = -1.0, None
        for i in range(pynvml.nvmlDeviceGetCount()):
            h = pynvml.nvmlDeviceGetHandleByIndex(i)
            mb = pynvml.nvmlDeviceGetMemoryInfo(h).free / (1024 ** 2)
            if mb > best_mb:
                best_mb, best_i = mb, i
        return (best_mb, best_i) if best_i is not None else (None, None)
    finally:
        pynvml.nvmlShutdown()

The probe operation is the part worth defending. Uploading an 8 × 8 array and downloading it costs microseconds and exercises exactly the path that a driver mismatch breaks. Without it, detection reports a working GPU on a host where the first real call will fail — and that first real call happens after rectification, which is not where anyone wants to discover a driver problem.

Edge-case matrix

Host state hasattr(cv2, "cuda") Device count Probe Decision
CPU-only wheel False — — cpu, no warning
CUDA wheel, CPU host True 0 — cpu, no warning
Driver older than runtime True ≥ 1 raises cpu, warn
Device held by another job True ≥ 1 ok cpu, no warning
CUDA_VISIBLE_DEVICES="" True 0 — cpu, no warning
NVML absent, device works True ≥ 1 ok cuda, memory unknown
Multiple devices, one free True ≥ 2 ok cuda on the emptiest
Container without device passthrough True 0 — cpu, no warning

The CUDA_VISIBLE_DEVICES="" row is worth internalising: setting that variable is how a scheduler tells a process to stay off the GPU, and the correct response is a silent CPU run, not a warning. Treating it as a fault produces alarming logs on exactly the hosts where everything is working as intended.

Verification snippet

The valuable test is that no host configuration makes the routine raise, and that a forced CPU run and a GPU run agree about which pixels have depth.

import os
import subprocess
import sys


def test_no_configuration_raises() -> None:
    """Run detection under each simulated host state in a subprocess."""
    cases = {
        "normal": {},
        "hidden-devices": {"CUDA_VISIBLE_DEVICES": ""},
        "bogus-device": {"CUDA_VISIBLE_DEVICES": "99"},
    }
    for name, extra in cases.items():
        env = {**os.environ, **extra}
        proc = subprocess.run(
            [sys.executable, "-c",
             "from backend import choose_backend; d = choose_backend();"
             " print(d.backend, d.warn, d.reason)"],
            env=env, capture_output=True, text=True,
        )
        assert proc.returncode == 0, f"{name}: detection raised\n{proc.stderr}"
        backend, warn, *_ = proc.stdout.split()
        assert backend in {"cuda", "cpu"}, f"{name}: bad backend {backend!r}"
        if name == "hidden-devices":
            assert warn == "False", "hiding devices is not a misconfiguration"

Running the cases in subprocesses rather than in-process is deliberate: CUDA reads its environment once at initialisation, so mutating os.environ inside a running interpreter does not simulate anything. Only a fresh process actually tests the case.

What the decision looks like in a log, and why it belongs there Two log excerpts for the same job. The first records only that the run started and finished, so a four-hour duration is unexplained and indistinguishable from a slow dataset. The second records the backend decision and its reason at startup — device unusable because the driver is older than the runtime — so the four-hour duration is immediately attributable and the fix is named. A note observes that the line costs nothing and is the difference between an investigation and a glance. without the decision line 10:02 start quarry-north 10:02 dense matching… 14:11 done 4 h — slow data, or slow path? with the decision line 10:02 start quarry-north 10:02 backend=cpu WARN device present but unusable (driver) 14:11 done 4 h — and the cause is on the line above One line at startup, emitted whether or not a device was found. Logging only on success is what makes a silently degraded fleet possible in the first place.

Figure 2 — The decision is worth a log line on every run, including the runs where nothing is wrong. A fleet that logs only failures cannot tell you that half its hosts have quietly been on the slow path for a month.

When to escalate

  • The probe succeeds and real matching still fails with an initialisation error. The probe exercises upload and download but not the compute kernels, some of which are compiled on first use. Extend the probe to construct the stereo matcher and run it on a tiny synthetic pair — a few milliseconds, and it covers kernel compilation.
  • Detection reports cuda and the run is no faster. The device is being used but starved, which is a host-side bottleneck rather than a detection problem; the diagnosis is in GPU acceleration for dense matching in Python.
  • Free memory reads as plentiful and allocation still fails. Another process allocated between the check and the attempt. This is inherent to a shared device; handle it by falling back for that pair rather than by tightening the threshold, which only narrows the race without closing it.

One organisational note. The decision function belongs in exactly one place in the codebase, called once at startup, with the result passed down rather than re-derived. Detection scattered across modules produces the situation where rectification took the device path and matching took the host path in the same run — internally consistent at each call site and incoherent as a whole. A single call, one record, threaded through, is what makes the backend a property of the run rather than of whichever function happened to ask.

← GPU Acceleration for Dense Matching in Python

Cost of discovering a driver mismatch, with and without a probe Two timelines for a job on a host whose driver is older than the CUDA runtime. Without a probe, detection reports a working device, the pipeline loads imagery, rectifies it, and fails at the first dense-matching call roughly forty minutes in, having produced nothing. With a probe, an eight by eight upload and download at startup fails within milliseconds, the run takes the host path from the beginning, and completes. A note gives the cost of the probe as microseconds against forty wasted minutes. no probe — failure at first real use detect load + rectify first device call → fail ≈ 40 min, nothing produced with an 8 × 8 probe probe fails host path, start to finish completes The probe costs microseconds and is the only thing that moves this failure to startup. Enumeration alone cannot detect a driver mismatch, because enumeration does not touch the driver's compute path. This is the same principle as validating a control file before queueing: pay the cheap check to avoid the expensive discovery.

Figure 3 — Why detection includes a trivial device operation. Enumeration answers “is a device listed?”; only an actual upload answers “can this process use it?”, and the difference is forty minutes.