Writing a Manifest-Driven Batch Runner for ODM

The script that processes tonight’s surveys usually starts as a for loop over a directory listing and grows a flag at a time: skip the ones already done, except re-do that one, use a smaller split for the big site, and don’t touch the one the client is still reviewing. Six months later nobody can say what it will do without running it. This page replaces that with a manifest — a declarative file naming every survey and its settings — and a runner whose only job is to reconcile that manifest against what already exists on disk, print the plan, and then execute it.

The reconciliation model is what makes it different from a loop. The manifest describes the desired state; the runner computes the difference against the actual state and acts only on the difference. That is the same shape as any infrastructure tool, and it buys the same property: running it twice in a row is safe, and running it with --plan tells you exactly what a real run would do.

# Why a loop over a directory stops working

Three pressures break the ad-hoc script, and they arrive together.

Per-survey settings. A 3,000-image corridor and a 400-image quarry do not want the same --split, and a survey with control does not want the same georeferencing arguments as one without. Encoding that as if statements against filename patterns works until a filename does not match the pattern.

Partial completion. Half the surveys finished last night. A loop either re-runs them, which wastes the night, or skips anything with an output directory, which silently skips the ones that failed halfway — the case discussed in retrying failed photogrammetry jobs idempotently.

Auditability. When a client asks why their orthomosaic is at 5 cm rather than 3 cm, the answer must be a line in a file with a commit history, not an argument someone typed.

Figure 1 — The runner computes a plan and then executes it. Because the plan is the same object either way, --plan is not an approximation of what will happen; it is what will happen.

# Minimal reproducible solution

The manifest is a plain file with defaults and per-survey overrides. TOML reads well for this and is in the standard library from Python 3.11.

# surveys.toml — the desired state of the whole batch.
[defaults]
target_epsg      = 25832
ortho_resolution = 0.03      # metres
split            = 500
engine_digest    = "sha256:6f1c…"   # pinned, never ':latest'

[[survey]]
name       = "quarry-north"
images     = "/srv/raw/2026-08-04/quarry-north"
gcp        = "/srv/control/quarry-north.txt"

[[survey]]
name       = "corridor-a"
images     = "/srv/raw/2026-08-05/corridor-a"
split      = 900             # long thin block, larger submodels
ortho_resolution = 0.05

[[survey]]
name       = "dam-face"
images     = "/srv/raw/2026-08-06/dam-face"
gcp        = "/srv/control/dam-face.txt"
ortho_resolution = 0.02
enabled    = false           # client review in progress — do not touch

The runner loads it, merges defaults, and produces one action per survey. Notice that the only branching is over state, not over filenames.

import tomllib
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class Action:
    name: str
    verb: str            # skip | build | rebuild | resume | disabled
    reason: str
    from_stage: str | None = None


def load_manifest(path: Path) -> list[dict]:
    doc = tomllib.loads(path.read_text())
    defaults = doc.get("defaults", {})
    return [{**defaults, **s} for s in doc.get("survey", [])]


def plan(manifest: list[dict], projects_root: Path) -> list[Action]:
    """Reconcile desired settings against what is on disk. No side effects."""
    from .retries import rewind_project, _is_complete   # from the retry guide

    actions: list[Action] = []
    for spec in manifest:
        name = spec["name"]
        proj = projects_root / name

        if not spec.get("enabled", True):
            actions.append(Action(name, "disabled", "enabled = false"))
            continue

        ortho = proj / "odm_orthophoto/odm_orthophoto.tif"
        if not proj.exists():
            actions.append(Action(name, "build", "no project directory"))
        elif not _is_complete(ortho):
            stage = rewind_project(proj, dry_run=True)
            actions.append(Action(name, "resume", "incomplete output", stage))
        elif _settings_of(proj) != _settings_key(spec):
            actions.append(Action(name, "rebuild", "settings differ from manifest"))
        else:
            actions.append(Action(name, "skip", "up to date"))
    return actions

The comparison in the third branch is what makes the manifest authoritative rather than advisory. _settings_key(spec) builds a small dictionary of the settings that affect output; _settings_of(proj) reads the same dictionary back from the run manifest the previous execution wrote into the project directory. If they differ, the product on disk was not built from what the manifest now says, and the runner rebuilds it.

import json
from pathlib import Path

OUTPUT_AFFECTING = ("target_epsg", "ortho_resolution", "split", "engine_digest", "gcp")


def _settings_key(spec: dict) -> dict:
    return {k: spec.get(k) for k in OUTPUT_AFFECTING}


def _settings_of(project_dir: Path) -> dict | None:
    """Read back what the previous run recorded. Absent means 'unknown'."""
    f = project_dir / "run_manifest.json"
    if not f.exists():
        return None                       # unknown provenance → rebuild
    return {k: json.loads(f.read_text()).get(k) for k in OUTPUT_AFFECTING}

Returning None for a project with no recorded provenance, and treating that as a mismatch, is deliberate: a directory whose origin is unknown is not evidence that the work was done under the current settings.

# Edge-case matrix

Two of those rows are policy rather than mechanism. The runner never deletes a project that has fallen out of the manifest: reporting it is useful, acting on it is how a batch tool eats a deliverable. And validation errors fail the whole plan before any survey runs, because discovering a bad path at 03:00 after four hours of work is strictly worse than discovering it at 23:00 before any.

# Verify the plan before you run it

The plan is data, so it can be asserted on. Two checks are worth running every night before execution.

def assert_plan_sane(actions: list[Action], manifest: list[dict]) -> None:
    """Cheap guards against a plan that would waste or destroy a night."""
    names = [s["name"] for s in manifest]
    assert len(names) == len(set(names)), "duplicate survey names in the manifest"

    rebuilds = [a for a in actions if a.verb == "rebuild"]
    if len(rebuilds) > len(actions) * 0.5:
        raise SystemExit(
            f"{len(rebuilds)}/{len(actions)} surveys would rebuild — "
            "a shared default probably changed; re-run with --plan and confirm"
        )

    for a in actions:
        assert a.verb in {"skip", "build", "rebuild", "resume", "disabled"}
        assert a.reason, f"{a.name}: action without a reason"

The rebuild threshold is the useful one. Changing a value under [defaults] — the engine digest, say — legitimately invalidates every survey, and that is sometimes exactly what you want. But it is also what an accidental edit looks like, and the difference between the two is a human confirming it. Failing loudly at that point costs a minute; not failing costs the night.

# The night's actual invocation: plan, eyeball, then run.
python -m batch_runner --manifest surveys.toml --plan
python -m batch_runner --manifest surveys.toml --run --jobs 2

Figure 2 — Reconciliation needs something to reconcile against. The run manifest is the only record of what the product on disk was actually built from, and its absence is correctly treated as a mismatch.

# When to escalate

• The plan says rebuild for a survey nobody touched. The recorded settings and the manifest disagree on a field that does not actually affect output — a path written as absolute in one place and relative in the other is the usual culprit. Normalise before comparing rather than removing the field from OUTPUT_AFFECTING.
• resume keeps being chosen and the survey never completes. The job is failing at the same stage every time, so the runner correctly resumes and correctly fails. This is a job problem, not a runner problem; the exit code will say whether it is a resource failure or a deterministic one, as set out in resolving ODM exit code 1 and 137 OOM.
• The batch runs longer than the window every night despite skipping most surveys. The runner is scheduling correctly and the hardware is the constraint; the memory-gated queue in orchestrating photogrammetry jobs with Python schedulers will overlap the cheap stages, but past that point the answer is fewer surveys per night or more hardware.

# Related

• Orchestrating photogrammetry jobs with Python schedulers
• Retrying failed photogrammetry jobs idempotently
• Structuring drone imagery for batch processing

← Orchestrating Photogrammetry Jobs with Python Schedulers

Figure 3 — The whole value of a plan-then-run design is the ordering. Validation, planning and sanity checks all complete in seconds, so the one expensive step is the last thing that happens and the easiest to withhold.