Containerising Photogrammetry Workers with Docker

A photogrammetry pipeline depends on a geospatial stack that is famously difficult to install: GDAL and PROJ with matching data files, OpenCV built with the right options, PDAL linked against the same GDAL, and a reconstruction engine with its own opinions about all of them. Getting it working on one machine takes an afternoon. Getting the same result on a second machine six months later takes considerably longer, because one of those components moved.

Containers solve the reproducibility problem and introduce four of their own: image size, data access, device access, and file ownership. Each is straightforward once seen and each costs a day the first time.

This page covers building a worker image that reproduces a result, mounting survey data without copying it, passing a GPU through, and the permissions trap that makes container output unusable. It supports the orchestration in orchestrating photogrammetry jobs with Python schedulers.

Audience and prerequisites. Python 3.10+, Docker 24+, and a pipeline that currently runs on a developer machine. Familiarity with setting up OpenDroneMap with Python is assumed.

# Prerequisites

# Conceptual architecture

A worker image has three layers with different rates of change, and separating them is what keeps rebuilds fast.

The base layer carries the geospatial stack: GDAL, PROJ, GEOS, PDAL, OpenCV. It changes rarely, takes a long time to build, and should be pinned hard and rebuilt deliberately.

The application layer carries the pipeline’s own Python dependencies and code. It changes often and builds in seconds provided the base layer is cached.

The runtime configuration — data mounts, environment, the job definition — is not in the image at all. An image that contains a path to a survey is an image that works for one job.

Figure 1 — Three layers, three rates of change. Mixing them is what makes a twenty-minute rebuild happen on every commit.

# Step 1: Pin the base layer by digest

A tag is a moving reference. gdal:ubuntu-small-3.8.0 can be rebuilt by its publisher, and a pipeline that pulls it a year later may get different binaries with the same name. A digest cannot move.

# Pinned by digest, not by tag: a tag can be repointed, a digest cannot.
FROM ghcr.io/osgeo/gdal@sha256:0f2a1c2b3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8

ENV DEBIAN_FRONTEND=noninteractive \
    PROJ_NETWORK=OFF \
    GDAL_CACHEMAX=512

# PROJ grids baked in, so a job never depends on a network fetch at runtime.
COPY proj-data/ /usr/share/proj/

RUN apt-get update && apt-get install -y --no-install-recommends \
        python3-pip pdal libpdal-dev \
    && rm -rf /var/lib/apt/lists/*

# A non-root user whose ids are supplied at build time — see the permissions
# section; running as root is what makes container output unusable on the host.
ARG UID=1000
ARG GID=1000
RUN groupadd -g ${GID} worker && useradd -u ${UID} -g ${GID} -m worker
USER worker

PROJ_NETWORK=OFF with the grids copied in is the setting that makes coordinate transformations reproducible. With network fetching enabled, a transformation’s accuracy depends on whether a grid file happened to download, which means the same pipeline can produce different coordinates on two machines — the fault described in fixing GDAL PROJ database context errors.

# Step 2: Mount data rather than copying it

A survey is hundreds of gigabytes. Copying it into an image is impossible; copying it into a container at start-up is merely slow. Bind mounts give the container the host’s files directly.

import docker
from pathlib import Path


def run_worker(image: str, survey_dir: str, output_dir: str,
               *, command: list[str], gpus: bool = False) -> dict:
    """Run a processing job with the survey bind-mounted read-only.

    Read-only on the input is not a formality: a reconstruction engine that
    writes intermediate files into the image directory will fill the survey
    storage and leave artefacts that confuse the next run.
    """
    client = docker.from_env()
    mounts = [
        docker.types.Mount(target="/data/input", source=str(Path(survey_dir).resolve()),
                           type="bind", read_only=True),
        docker.types.Mount(target="/data/output", source=str(Path(output_dir).resolve()),
                           type="bind", read_only=False),
    ]
    kwargs = {}
    if gpus:
        kwargs["device_requests"] = [
            docker.types.DeviceRequest(count=-1, capabilities=[["gpu"]])]

    container = client.containers.run(
        image, command=command, mounts=mounts, detach=True,
        user=f"{Path(output_dir).stat().st_uid}:{Path(output_dir).stat().st_gid}",
        **kwargs)
    result = container.wait()
    logs = container.logs(tail=200).decode("utf-8", "replace")
    container.remove()
    return {"exit_code": result["StatusCode"], "logs": logs}

Deriving the container’s user from the output directory’s ownership is the fix for the permissions trap, and it is the single most useful line in this page. A container running as root writes files owned by root, which the host user then cannot delete without elevation — a small annoyance that becomes a large one on a fleet.

# Step 3: Pass a GPU through, and prove it arrived

import subprocess


def gpu_available_in_image(image: str) -> dict:
    """Confirm the container can actually see and use a GPU.

    Two separate things can fail: the device may not be passed through, and
    the libraries inside the image may not be built with CUDA support. The
    first is a runtime flag and the second is a rebuild, so distinguishing
    them saves a lot of time.
    """
    device = subprocess.run(
        ["docker", "run", "--rm", "--gpus", "all", image, "nvidia-smi", "-L"],
        capture_output=True, text=True)
    if device.returncode != 0:
        return {"device_visible": False,
                "note": "the GPU was not passed through — check the container toolkit"}

    cuda = subprocess.run(
        ["docker", "run", "--rm", "--gpus", "all", image, "python3", "-c",
         "import cv2; print(cv2.cuda.getCudaEnabledDeviceCount())"],
        capture_output=True, text=True)
    count = cuda.stdout.strip()
    return {"device_visible": True,
            "cuda_devices_seen_by_opencv": count,
            "note": ("the image's OpenCV is built without CUDA" if count == "0"
                     else "GPU is usable from the application")}

The two-stage check matters because the symptoms are identical from the application’s point of view. The remedies are not: one is a flag on docker run and the other is a different image. The downstream consequences are covered in detecting CUDA availability and falling back to CPU.

# Step 4: Size the worker so it cannot take the host down

A reconstruction that exhausts memory on a bare machine kills the machine; the same reconstruction in a container with no limit kills the machine and every other container on it. Setting limits is not an optimisation, it is a containment requirement.

import docker
import psutil


def worker_limits(image_points: int, *, host_fraction: float = 0.8,
                  bytes_per_point: int = 64) -> dict:
    """Memory and CPU limits for one worker, from the job's own size.

    Reserving a fraction of the host rather than all of it leaves room for
    the orchestrator, the filesystem cache and a second worker — and the
    filesystem cache matters more than it looks on a job reading terabytes.
    """
    total = psutil.virtual_memory().total
    need = int(image_points * bytes_per_point * 2.5)
    limit = min(need, int(total * host_fraction))
    if need > limit:
        raise MemoryError(
            f"this job needs about {need / 1e9:.1f} GB and the host allows "
            f"{limit / 1e9:.1f} GB — tile the input or use a larger machine")
    return {"mem_limit": limit, "memswap_limit": limit,   # no swap: fail fast
            "nano_cpus": int(psutil.cpu_count() * 1e9 * host_fraction)}

Setting memswap_limit equal to mem_limit disables swap for the container, which sounds harsh and is correct. A photogrammetry worker that starts swapping does not recover; it runs a hundred times slower and holds its resources for hours. Failing immediately is better than degrading invisibly, and it is the same argument made for admission control in orchestrating photogrammetry jobs with Python schedulers.

# Step 5: Keep the image small enough to move

A geospatial base image is easily five gigabytes, and on a fleet that pulls it on every node the size is a real operational cost. Three measures cut it substantially without giving anything up.

Use a slim base. The difference between a full GDAL image and a small one is largely optional drivers, most of which a drone pipeline never touches.

Combine and clean in one layer. Each RUN creates a layer, and deleting a file in a later layer does not reclaim its space — the bytes are still in the image. Package lists and build caches must be removed in the same instruction that created them.

Use a multi-stage build for anything compiled. Build tools, headers and source trees belong in a stage that is discarded; only the resulting binaries are copied forward.

# Build stage: compilers and headers, discarded afterwards.
FROM ghcr.io/osgeo/gdal@sha256:0f2a1c2b3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8 AS build
RUN apt-get update && apt-get install -y --no-install-recommends build-essential cmake     && rm -rf /var/lib/apt/lists/*
COPY src/ /src/
RUN cmake -S /src -B /build && cmake --build /build --target install

# Runtime stage: only the artefacts.
FROM ghcr.io/osgeo/gdal@sha256:0f2a1c2b3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8
COPY --from=build /usr/local/lib/ /usr/local/lib/
COPY --from=build /usr/local/bin/ /usr/local/bin/

On a typical worker these three measures take an image from around five gigabytes to under three, which on a twenty-node fleet pulling a new build weekly is a material saving in time as well as bandwidth.

# Step 6: Record the image with the run

The whole reproducibility argument collapses if a result cannot be tied back to the image that produced it. The tie must be by digest, because a tag can point somewhere else tomorrow.

import json
import subprocess
from datetime import date


def run_provenance(image: str, job: dict) -> dict:
    """Everything needed to re-run this job identically."""
    digest = subprocess.run(
        ["docker", "inspect", "--format", "{{index .RepoDigests 0}}", image],
        capture_output=True, text=True, check=True).stdout.strip()
    return {"run_date": date.today().isoformat(),
            "image": image, "image_digest": digest,
            "versions": image_versions(image),
            "job": {k: job[k] for k in ("survey_id", "parameters", "input_digest")
                    if k in job}}

Two conventions make this durable on a fleet. Never run a worker from a tag in production — resolve the tag to a digest at dispatch and run that. And keep old images: a registry that garbage-collects last year’s builds has made last year’s results unreproducible, whatever the manifest says.

# What containers do not fix

Containerisation is often sold as solving reproducibility outright, and it solves a specific part of it. Being clear about the remainder avoids a false sense of security.

Hardware differences remain. A reconstruction that uses a GPU produces slightly different results on different GPU generations, because floating-point reduction orders differ. The same image on two machines is not bit-identical output, and for a photogrammetry pipeline that is usually fine — but it is not nothing, and it should be stated rather than assumed away.

Thread counts change results. Many parallel algorithms sum in whatever order threads finish, so a sixteen-core host and a four-core host produce answers that differ in the last few digits. Where bit-identical output matters, the thread count belongs in the job definition alongside everything else.

Mounted configuration leaks in. An image is only reproducible to the extent that nothing outside it influences the run. A mounted config file, an environment variable set by the orchestrator, or a PROJ grid on a host path all break the property quietly.

Data is not versioned by the image. The same image over different input produces different output, obviously — but “different input” includes a survey directory that somebody added images to. Hashing the input set into the run record is what closes that gap.

# Parameter deep-dive

--shm-size deserves the attention it rarely gets. Docker’s default shared-memory size is 64 MB, and multi-process matchers that communicate through shared memory fail on it in ways that look like unrelated crashes.

# Verification and output inspection

An image that reproduces a result is one whose component versions are known, so the image should be able to report them.

import json
import subprocess


VERSION_PROBE = """
import json, sys
out = {"python": sys.version.split()[0]}
try:
    from osgeo import gdal
    out["gdal"] = gdal.__version__
except Exception as exc:
    out["gdal"] = f"unavailable: {exc}"
try:
    import pyproj
    out["proj"] = pyproj.proj_version_str
    out["proj_data"] = pyproj.datadir.get_data_dir()
except Exception as exc:
    out["proj"] = f"unavailable: {exc}"
try:
    import cv2
    out["opencv"] = cv2.__version__
except Exception as exc:
    out["opencv"] = f"unavailable: {exc}"
print(json.dumps(out))
"""


def image_versions(image: str) -> dict:
    """Record the exact stack inside an image, for the run manifest."""
    result = subprocess.run(["docker", "run", "--rm", image, "python3", "-c",
                             VERSION_PROBE], capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(result.stderr[:400])
    versions = json.loads(result.stdout)
    digest = subprocess.run(
        ["docker", "inspect", "--format", "{{index .RepoDigests 0}}", image],
        capture_output=True, text=True).stdout.strip()
    return {"image_digest": digest, **versions}

Storing that alongside every job’s output is what makes a result reproducible in practice. A year later, “which GDAL produced this” is a lookup rather than an archaeology exercise.

Figure 3 — Four benefits, three of which come from discipline rather than from Docker.

Figure 4 — Five stages, and the last two are the ones usually added after the first outage.

# Troubleshooting

Output files are owned by root and cannot be deleted. The container ran as root. Set the user to the host’s uid and gid.

Coordinate transformations differ between machines. PROJ grid availability differs. Bake the grids in and set PROJ_NETWORK=OFF.

A matcher crashes with no useful error. Shared memory exhausted at the 64 MB default. Raise --shm-size.

The GPU is invisible inside the container. Either the device was not passed through or the libraries lack CUDA. The two-stage probe distinguishes them.

A rebuild takes twenty minutes for a one-line code change. The application layer is below the base layer in the Dockerfile, so the cache is invalidated. Reorder.

The same image produces different results on two hosts. Something is being read from outside the image: a mounted configuration, an environment variable, or a PROJ grid. Compare the version probe output from both, then compare the environment.

A worker is killed and the orchestrator reports nothing useful. The container hit its memory limit and was terminated by the kernel rather than by the application. Container exit code 137 says so, and the limit is the thing to look at rather than the code.

# Related

• Pinning GDAL, PROJ and OpenCV Versions Reproducibly
• Passing GPUs into Processing Containers from Python
• Mounting Large Datasets Without Copying Them
• Fixing Permission Denied on Container Output Directories
• Orchestrating Photogrammetry Jobs with Python Schedulers

← Core Photogrammetry Fundamentals for Python Pipelines