Versioning Deliverables with Checksums and Manifests

Four months after delivery, a client says the volumes do not match the drawings. Somebody reprocessed that survey in March after a geoid correction, and nobody can now say with certainty which orthomosaic the drawings were traced from — because the file was overwritten at the same path and the only date on it is the upload timestamp.

The whole dispute is avoidable with two conventions that cost nothing to adopt: never overwrite, and record a content digest for everything shipped. This page sets them out, completing publishing rasters to STAC and object storage.

# Immutable keys, explicit supersession

The rule is one sentence: a key, once written, always returns the same bytes. A reprocessed deliverable goes to a new key, and the relationship between old and new is recorded rather than implied by a filename.

This costs storage, which is the objection raised every time. A superseded orthomosaic moved to an archive tier costs a small fraction of a penny per gigabyte per month — against which the cost of one unresolvable dispute is not a close comparison.

What the rule buys is that every URL ever handed out keeps meaning what it meant. A map saved in September still renders September’s pixels in March, and when somebody asks why the March version differs, the answer is a diff between two things that both still exist.

Figure 1 — The same reprocessing, two policies.

# Minimal reproducible solution

The deliverable manifest is the record of what was shipped, and it is what a dispute is resolved against.

import hashlib
from datetime import datetime, timezone
from pathlib import Path


MANIFEST_VERSION = "1.0"


def digest_file(path: str, *, block: int = 1 << 20) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as handle:
        for chunk in iter(lambda: handle.read(block), b""):
            h.update(chunk)
    return h.hexdigest()


def deliverable_manifest(*, survey_id: str, version: str, files: dict[str, str],
                         supersedes: str | None, reason: str | None,
                         inputs: dict) -> dict:
    """The shipping record for one version of one survey's deliverables.

    `supersedes` and `reason` together are what make a version history
    readable. A chain of versions with no stated reason for any of them is an
    audit trail that records that something changed and not what or why, which
    is the part anybody actually needs.
    """
    if supersedes and not reason:
        raise ValueError("a superseding version must state why it exists")

    return {
        "manifest_version": MANIFEST_VERSION,
        "survey_id": survey_id,
        "version": version,
        "created_at": datetime.now(timezone.utc).isoformat(),
        "supersedes": supersedes,
        "reason": reason,
        "inputs": inputs,
        "files": {
            name: {
                "sha256": digest_file(path),
                "bytes": Path(path).stat().st_size,
            }
            for name, path in files.items()
        },
    }

The reason requirement is deliberately enforced in code. A version chain where every entry says only that it superseded the last is barely better than an overwrite, because the question that arrives later is always why, and reconstructing it from memory four months on is guesswork presented as fact.

# What counts as a version

Not every reprocessing produces a new deliverable version, and treating every re-run as one fills a bucket with near-identical copies. The test is whether a consumer’s work could change.

A new version when the pixels or the geometry change: a different datum, corrected control, a re-flown line, a changed processing parameter that moves values.

Not a new version when only non-substantive things change: recompression at the same values, added overviews, a corrected typo in a metadata field. These are amendments to the same version, and the digest changing is expected — which is exactly why the manifest records a digest per file rather than one per version.

Figure 2 — The test is whether somebody’s answer could change.

# Linking versions in the catalogue

STAC has a standard way to express this, and using it means a client discovers the newest version without knowing the naming convention. The version extension adds a version property and predecessor-version / successor-version links, plus a deprecated flag on superseded items.

import pystac


def link_versions(old: pystac.Item, new: pystac.Item, *,
                  old_version: str, new_version: str) -> None:
    """Record supersession in both directions and deprecate the old item.

    Linking both ways matters: a client that arrives at the old item by a saved
    URL needs to find the new one, and a client at the new item needs to know
    what it replaced in order to explain a difference to somebody.
    """
    ext = "https://stac-extensions.github.io/version/v1.2.0/schema.json"
    for item in (old, new):
        if ext not in item.stac_extensions:
            item.stac_extensions.append(ext)

    old.properties["version"] = old_version
    new.properties["version"] = new_version
    old.properties["deprecated"] = True

    old.add_link(pystac.Link("successor-version", new,
                             media_type=pystac.MediaType.JSON))
    new.add_link(pystac.Link("predecessor-version", old,
                             media_type=pystac.MediaType.JSON))

Deprecating rather than deleting is the important half. A client that follows a saved link lands on something that still renders, sees the deprecation flag, and can follow the successor link when it chooses to — which is a considerably better experience than a broken URL and considerably safer than silently receiving different pixels.

Figure 3 — Four stages that belong inside the run, not after it.

# Edge-case matrix

# Verification snippet

def audit_version_chain(manifests: list[dict]) -> dict:
    """Check a survey's version history is complete and consistent.

    Three failures are worth catching automatically: a chain with a gap, where
    a manifest supersedes a version nobody recorded; a fork, where two versions
    claim the same predecessor; and a digest repeated across versions, which
    means an identical file was shipped as though it had changed.
    """
    by_version = {m["version"]: m for m in manifests}
    problems = []

    superseded = [m["supersedes"] for m in manifests if m["supersedes"]]
    for target in superseded:
        if target not in by_version:
            problems.append(f"version {target} is superseded but not recorded")

    for target in set(superseded):
        claimants = [m["version"] for m in manifests if m["supersedes"] == target]
        if len(claimants) > 1:
            problems.append(f"{target} superseded by several: {claimants}")

    seen: dict[str, str] = {}
    for m in manifests:
        for name, entry in m["files"].items():
            key = f"{name}:{entry['sha256']}"
            if key in seen and seen[key] != m["version"]:
                problems.append(
                    f"{name} identical in {seen[key]} and {m['version']}")
            seen[key] = m["version"]

    roots = [m["version"] for m in manifests if not m["supersedes"]]
    if len(roots) != 1:
        problems.append(f"expected exactly one original version, found {roots}")

    return {"versions": len(manifests), "ok": not problems,
            "problems": problems}

The duplicate-digest check earns its place surprisingly often. It catches a reprocessing run that produced a byte-identical output — which means the change that motivated it did not reach the deliverable, and the new version is a lie by omission rather than by statement.

# When to escalate

• Storage cost is genuinely a constraint. Move superseded versions to an archive tier rather than deleting them. Retrieval in hours is acceptable for something nobody is actively using and infinitely better than absence.
• A version was overwritten before the policy existed. Record what is known, mark the gap explicitly, and do not reconstruct a digest from a copy. A stated unknown is worth more than a plausible fabrication.
• Two teams disagree about what was delivered. This is precisely what the manifests exist for, and the resolution takes minutes rather than an afternoon of recollection. Compare digests against the recorded values; whichever file matches is the one that shipped, and the question stops being a matter of opinion.

# Related

• Publishing Rasters to STAC and Object Storage
• Uploading COGs to Object Storage with Python
• Reporting Accuracy in a Machine-Readable Manifest

← Publishing Rasters to STAC and Object Storage