Retrying Failed Photogrammetry Jobs Idempotently

The symptom is unusually unpleasant: a job that failed once, was retried automatically, reported success, and produced an orthomosaic with a visible seam through the middle or a point cloud missing a strip. Nothing errored. The retry did not fail — it succeeded against a project directory that still held partial output from the attempt before it, and the engine, finding those files present, skipped the stages that would have replaced them. This page is about making a retry mean “run this job again” rather than “resume whatever state was left behind”.

The orchestration layer that decides when to retry is covered in orchestrating photogrammetry jobs with Python schedulers. This page is about the operation itself.

# Why a naive retry corrupts the output

Reconstruction engines are deliberately resumable. Each stage checks whether its output already exists and skips itself if so — the behaviour that makes resuming a long run cheap. That check asks whether a file is present, not whether it is complete, and not whether it was produced under the same configuration.

Three distinct problems follow from that.

A truncated output looks finished. A process killed while writing a dense point cloud leaves a file on disk. The next attempt sees it, skips the stage, and every subsequent stage consumes a cloud that is missing its tail. The result is geometrically self-consistent and wrong, which is the worst combination available.

A configuration change is ignored. Retry a failed job with --split reduced from 800 to 400, and any stage whose output already exists keeps the results computed at 800. The run reports the new configuration and delivers a mixture of both.

Stale intermediates outlive their inputs. If the retry follows a fix to the input imagery — a corrupted frame removed, EXIF repaired — the stages that already ran used the old inputs. Nothing revalidates them.

None of the three announces itself. The engine’s log records the stages it skipped, which reads as efficiency.

Figure 1 — A killed job leaves a directory that looks resumable and is not. Everything from the interrupted stage onward has to go, because everything after it was derived from the file that was being written.

# Minimal reproducible solution

The retry has three jobs: decide whether retrying is appropriate at all, truncate the project state back to a known-good boundary, and record what it did. The first is a classification, the second is a deletion, and the third is what makes the whole thing auditable.

import shutil
from pathlib import Path

# Stage outputs in execution order. Deleting stage N means deleting N..end.
STAGE_OUTPUTS = [
    ("dataset", "images.json"),
    ("opensfm", "opensfm"),
    ("openmvs", "opensfm/undistorted/openmvs"),
    ("odm_filterpoints", "odm_filterpoints"),
    ("odm_meshing", "odm_meshing"),
    ("mvs_texturing", "odm_texturing"),
    ("odm_georeferencing", "odm_georeferencing"),
    ("odm_orthophoto", "odm_orthophoto"),
]


def _is_complete(path: Path) -> bool:
    """A stage output counts as complete only if it opens, not if it exists."""
    if not path.exists():
        return False
    if path.is_dir():
        return any(path.iterdir())
    if path.suffix in {".tif", ".tiff"}:
        import rasterio                       # openability is the real test
        try:
            with rasterio.open(path) as ds:
                return ds.count > 0
        except Exception:
            return False
    return path.stat().st_size > 0


def rewind_project(project_dir: Path, from_stage: str | None = None) -> str:
    """Delete the first incomplete stage and everything after it.

    Returns the stage the next attempt will start from. With from_stage given,
    rewinds to that stage explicitly — used when the inputs or configuration
    changed and earlier outputs are stale even though they are complete.
    """
    names = [n for n, _ in STAGE_OUTPUTS]
    if from_stage is not None:
        start = names.index(from_stage)
    else:
        start = len(names)
        for i, (_, rel) in enumerate(STAGE_OUTPUTS):
            if not _is_complete(project_dir / rel):
                start = i
                break

    for _, rel in STAGE_OUTPUTS[start:]:
        target = project_dir / rel
        if target.is_dir():
            shutil.rmtree(target, ignore_errors=True)
        elif target.exists():
            target.unlink()

    return names[start] if start < len(names) else "complete"

The from_stage parameter is the part that is easy to omit and expensive to omit. When a retry follows a configuration change or an input fix, the completed stages are complete but wrong, and only the caller knows which stage the change invalidates. A lower --split invalidates from opensfm; a repaired image file invalidates from dataset; a changed orthophoto resolution invalidates only odm_orthophoto.

# Which stage a configuration change invalidates. Anything not listed is
# assumed to invalidate everything, because guessing low is the unsafe error.
INVALIDATES_FROM = {
    "images_dir": "dataset",
    "gcp_file": "odm_georeferencing",
    "target_epsg": "odm_georeferencing",
    "split": "opensfm",
    "pc_quality": "openmvs",
    "ortho_resolution_cm": "odm_orthophoto",
    "engine_digest": "dataset",
}


def rewind_for_change(project_dir: Path, changed_fields: set[str]) -> str:
    names = [n for n, _ in STAGE_OUTPUTS]
    stages = [INVALIDATES_FROM.get(f, "dataset") for f in changed_fields]
    earliest = min(stages, key=names.index) if stages else None
    return rewind_project(project_dir, from_stage=earliest)

Defaulting an unknown field to dataset is deliberate. A wrong guess in that direction costs a full re-run; a wrong guess in the other direction ships a mixed-configuration product that nobody can later explain.

# Edge-case matrix

The last row is the one that turns an annoyance into corruption. Rewinding is destructive, so it must happen under the same exclusive lock that guards the run itself — otherwise a second worker can begin deleting stage outputs while the first is writing them.

Figure 2 — Where the critical section begins is not a detail. A lock that starts at the engine call leaves the deletion outside it, which is the one operation that can destroy another worker’s work.

# Verification snippet

The property to assert is stronger than “it ran”: a retried job must produce what a fresh job would have produced. That is checkable directly, and it is worth doing once per pipeline change rather than never.

import hashlib
from pathlib import Path


def raster_fingerprint(path: Path) -> str:
    """Hash the pixels and the georeference, ignoring irrelevant metadata."""
    import rasterio
    h = hashlib.sha256()
    with rasterio.open(path) as ds:
        h.update(str(ds.crs).encode())
        h.update(str(tuple(round(v, 9) for v in ds.transform)).encode())
        for _, window in ds.block_windows(1):
            h.update(ds.read(window=window).tobytes())
    return h.hexdigest()


def assert_retry_is_idempotent(fresh_dir: Path, retried_dir: Path) -> None:
    """A retried run must match a clean run, pixel for pixel."""
    ortho = "odm_orthophoto/odm_orthophoto.tif"
    a = raster_fingerprint(fresh_dir / ortho)
    b = raster_fingerprint(retried_dir / ortho)
    assert a == b, (
        "retry produced a different orthomosaic than a clean run — "
        "the rewind did not remove everything it should have"
    )

Run it against a job deliberately killed at each stage boundary in turn. If the retry is correct, every one of those kills produces the same final fingerprint as an uninterrupted run. If any of them differs, the rewind list is missing a stage output — and the stage it is missing is named by which kill point diverges.

Note the exclusions in raster_fingerprint: creation timestamps and software tags differ between runs and are not part of the result. Including them turns a useful check into one that always fails.

Figure 3 — Killing the job at each boundary in turn converts “is the retry correct?” into a table with one row per stage. A gap in the rewind list shows up as exactly one failing row, and that row identifies it.

# When to escalate

• The fingerprints differ at one specific stage boundary and nowhere else. The rewind list is incomplete for that stage — it writes something outside the directory the list names, typically a cache or a log the next stage reads. Find it by diffing the two project directories after a divergent retry, and add it to STAGE_OUTPUTS.
• Retries succeed but each one produces a slightly different orthomosaic. The non-determinism is inside the engine, not in the retry logic — usually thread-count-dependent summation order, as discussed in the reproducibility section of the fundamentals overview. Pin the worker count before concluding the retry is at fault.
• Every retry rewinds to dataset and the fleet never makes progress. The completeness check is rejecting a valid output, most often because the file is a format _is_complete does not know how to open and falls through to the size test on a directory. Add the format rather than loosening the check.

# Related

• Orchestrating photogrammetry jobs with Python schedulers
• Resolving ODM exit code 1 and 137 OOM
• Writing a manifest-driven batch runner for ODM

← Orchestrating Photogrammetry Jobs with Python Schedulers