Mounting Large Datasets Without Copying Them

A containerised reconstruction has been running for forty minutes and the host’s root filesystem is full. The survey is on a separate 20 TB volume with plenty of space; the container’s writable layer, which lives on the root filesystem, is not.

Container storage defaults are designed for application code, not for terabytes of imagery, and a photogrammetry job writes far more than it reads. This page covers getting survey data into a container without copying it, keeping intermediates off the wrong volume, and the storage characteristics that decide whether a job takes two hours or six. It extends containerising photogrammetry workers with Docker.

# Four places data can live, and what each costs

Inside the image. Baked in at build time. Appropriate for reference data measured in megabytes — PROJ grids, a camera database — and never for a survey.

The container’s writable layer. Everything a container writes that is not on a mount. It lives on the Docker storage driver’s filesystem, is slow because of copy-on-write, and disappears when the container is removed.

A bind mount. A host directory mapped into the container. No copy, host filesystem performance, and the data outlives the container.

A volume. Docker-managed storage. Useful for state a container owns and awkward for data that other tools also need to reach.

For a photogrammetry worker the answer is almost always bind mounts for input, output and intermediates, with nothing of consequence in the writable layer.

Figure 1 — Four locations, and the one that causes the outage.

# Minimal reproducible solution

import shutil
from pathlib import Path

import docker


def build_mounts(survey_dir: str, output_dir: str, scratch_dir: str) -> list:
    """Input read-only, output and scratch read-write, nothing in the image.

    Declaring an explicit scratch mount is the part that prevents the
    full-disk failure: a reconstruction writes tens of gigabytes of
    intermediates, and unless a mount claims that path they land in the
    container's writable layer on the host root filesystem.
    """
    for path in (survey_dir, output_dir, scratch_dir):
        Path(path).mkdir(parents=True, exist_ok=True)

    return [
        docker.types.Mount("/data/input", str(Path(survey_dir).resolve()),
                           type="bind", read_only=True),
        docker.types.Mount("/data/output", str(Path(output_dir).resolve()),
                           type="bind", read_only=False),
        docker.types.Mount("/data/scratch", str(Path(scratch_dir).resolve()),
                           type="bind", read_only=False),
    ]


def check_capacity(scratch_dir: str, image_count: int,
                   *, gb_per_thousand_images: float = 45.0) -> dict:
    """Refuse to start a job that will not fit on the scratch volume."""
    need = image_count / 1000.0 * gb_per_thousand_images
    free = shutil.disk_usage(scratch_dir).free / 1e9
    if need > free * 0.9:
        raise OSError(f"this job needs about {need:.0f} GB of scratch and "
                      f"{free:.0f} GB is free on {scratch_dir}")
    return {"needed_gb": round(need), "free_gb": round(free)}

Checking capacity before dispatch rather than discovering it forty minutes in is worth the four lines. A job that fails on a full disk has usually taken a worker out of the pool as well, because the host’s root filesystem filling affects everything on it.

# Storage that is fast enough

Bind mounts give host filesystem performance, which raises the question of whether the host’s filesystem is fast enough. Photogrammetry’s access pattern is unusual: a long sequential read of the imagery, then a very large amount of random read and write on intermediates.

The consequence is that network storage behaves very differently for the two phases. Reading a survey over a 10 Gbit link is fine — it is sequential and the link is the limit. Running the reconstruction’s intermediates over the same link is not, because the latency on small random operations dominates and a job can take three times as long.

import time
from pathlib import Path


def measure_scratch(path: str, *, size_mb: int = 512) -> dict:
    """Crude but decisive throughput and latency test on a candidate scratch path.

    A sequential figure alone is misleading: network storage can be fast
    sequentially and unusable for the random small operations a reconstruction
    generates. The small-file test is the one that predicts the job.
    """
    target = Path(path) / ".scratch_probe"
    target.mkdir(parents=True, exist_ok=True)
    block = b"\0" * (1 << 20)

    start = time.perf_counter()
    with (target / "seq.bin").open("wb") as fh:
        for _ in range(size_mb):
            fh.write(block)
        fh.flush()
    seq = size_mb / (time.perf_counter() - start)

    start = time.perf_counter()
    for i in range(2000):
        (target / f"small_{i}").write_bytes(b"x" * 4096)
    small = 2000 / (time.perf_counter() - start)

    for f in target.iterdir():
        f.unlink()
    target.rmdir()

    return {"sequential_mb_s": round(seq), "small_files_per_s": round(small),
            "suitable_for_scratch": small > 800,
            "note": ("adequate for reconstruction intermediates" if small > 800 else
                     "small-file rate is too low — use local storage for scratch")}

Figure 3 — For a four-hundred-gigabyte survey only one column is available.

# Edge-case matrix

The SELinux row is worth knowing because the error is indistinguishable from an ordinary permission problem. On a host with SELinux enforcing, a bind mount is inaccessible to the container until it is relabelled, and the :z suffix does that.

# Verification snippet

import subprocess


def verify_mounts(container_id: str, expected: dict[str, bool]) -> dict:
    """Confirm each mount is present and has the expected writability.

    `expected` maps a container path to whether it should be writable. A
    read-only input mount that is actually writable is as much a fault as a
    missing one, because it lets a job modify the survey.
    """
    problems = []
    for path, should_write in expected.items():
        probe = subprocess.run(
            ["docker", "exec", container_id, "sh", "-c",
             f"test -d {path} && touch {path}/.wtest 2>/dev/null && "
             f"rm -f {path}/.wtest && echo rw || echo ro"],
            capture_output=True, text=True)
        state = probe.stdout.strip()
        if state not in {"rw", "ro"}:
            problems.append(f"{path} is not mounted")
        elif (state == "rw") != should_write:
            problems.append(f"{path} is {state} but should be "
                            f"{'rw' if should_write else 'ro'}")
    return {"checked": list(expected), "problems": problems, "ok": not problems}

Figure 2 — Three times the duration from one mount decision.

# A layout that works on a fleet

Naming conventions sound trivial and decide how much of a pipeline has to know about storage. One that has held up well is three mounts with fixed container paths and variable host paths.

/data/input is always the survey, always read-only. /data/output is always where the deliverables go. /data/scratch is always node-local and always disposable. The application never sees a host path, so the same image runs on a workstation, a build agent and a cloud node with no change.

The scheduler supplies the host side, and the one rule it must enforce is that scratch is node-local. Everything else can be network storage; scratch cannot, and making that a property of the scheduler rather than of each job definition is what stops it being forgotten.

Cleaning scratch is the scheduler’s job too. A worker that leaves its intermediates behind will fill its own disk within a few jobs, and a cleanup that runs on completion misses the case that matters — the job that failed.

# When to escalate

• There is no local storage on the worker. Either provision some or accept the slowdown explicitly in the scheduling estimates. Pretending the network scratch is equivalent produces timing estimates that are wrong by a factor of three.
• The survey lives somewhere the container cannot reach. Copy it, or move the worker. An object-store survey needs a fetch step with its own capacity check.
• Intermediates must be retained for debugging. Mount them to a path with a retention policy rather than leaving them in the writable layer, where they vanish with the container.

# Related

• Containerising Photogrammetry Workers with Docker
• Fixing Permission Denied on Container Output Directories
• Best Practices for Storing Raw UAV Datasets

← Containerising Photogrammetry Workers with Docker