Reporting Accuracy in a Machine-Readable Manifest
A survey’s accuracy is computed, written into a report, and effectively lost. Six months later, the questions that matter — has accuracy drifted, which camera produces the best results, did that firmware update change anything — require somebody to open forty PDFs and retype numbers.
The remedy is to emit the accuracy as structured data alongside the human-readable report, from the same computation. This page covers what that manifest should contain, how to keep it stable across a programme, and what becomes possible once it exists. It completes checkpoint-based accuracy validation in Python.
What belongs in the manifest
Four groups of fields, and the temptation is to include only the first.
The statistics — horizontal and vertical RMSE, bias, scatter, maximum, checkpoint count, confidence level. These are what the report quotes.
The residuals themselves, per checkpoint. They allow every statistic to be recomputed, which turns a disagreement into a traceable difference of method rather than an impasse.
The method — how the split was made, the seed, the confidence multiplier, whether blunders were excluded and which. Without it, two manifests are not comparable.
The inputs — the reconstruction’s parameters, the camera, the control set, the datum, the software versions. These are what makes a trend across a season interpretable rather than merely visible.
Figure 1 — Four groups. Most reports carry the top one.
Minimal reproducible solution
from datetime import date, datetime, timezone
import json
from pathlib import Path
MANIFEST_VERSION = "1.2"
def accuracy_manifest(*, survey_id: str, flight_date: date, statistics: dict,
residuals: list[dict], method: dict, inputs: dict) -> dict:
"""The structured record that accompanies every accuracy statement.
A version field on the manifest itself is what lets the schema evolve
without breaking a programme's history: a reader can handle both the
old shape and the new one, which a schemaless document cannot support.
"""
return {
"manifest_version": MANIFEST_VERSION,
"survey_id": survey_id,
"flight_date": flight_date.isoformat(),
"generated_at": datetime.now(timezone.utc).isoformat(),
"statistics": statistics,
"residuals": residuals,
"method": method,
"inputs": inputs,
}
def write_manifest(manifest: dict, path: str) -> str:
"""Write it next to the deliverables, sorted so diffs are readable."""
out = Path(path)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(manifest, indent=2, sort_keys=True, default=str))
return str(out)
Sorting the keys is a small choice with a real benefit: two manifests from consecutive months diff cleanly, so a change in method or inputs is visible at a glance rather than buried in reordered fields.
Keeping it comparable across a programme
A manifest is only useful if this month’s can be compared with last month’s, which requires the fields to mean the same thing. Two mechanisms hold that.
REQUIRED_FIELDS = {
"statistics": ["horizontal", "vertical", "checkpoints", "confidence"],
"method": ["split", "seed", "confidence_multiplier", "excluded"],
"inputs": ["camera_serial", "control_set_id", "vertical_datum",
"software_versions", "intrinsics_policy"],
}
def validate_manifest(manifest: dict) -> dict:
"""Refuse a manifest that cannot be compared with the rest of the series."""
problems = []
if manifest.get("manifest_version") != MANIFEST_VERSION:
problems.append(f"version {manifest.get('manifest_version')} — "
f"expected {MANIFEST_VERSION}")
for group, fields in REQUIRED_FIELDS.items():
present = manifest.get(group, {})
missing = [f for f in fields if f not in present]
if missing:
problems.append(f"{group} is missing {missing}")
if not manifest.get("residuals"):
problems.append("no per-checkpoint residuals — statistics cannot be recomputed")
return {"valid": not problems, "problems": problems}
def comparable(a: dict, b: dict) -> dict:
"""Are two manifests methodologically comparable?"""
differences = []
for field in ("confidence_multiplier", "split"):
if a["method"].get(field) != b["method"].get(field):
differences.append(f"method.{field}: {a['method'].get(field)} vs "
f"{b['method'].get(field)}")
for field in ("vertical_datum", "intrinsics_policy"):
if a["inputs"].get(field) != b["inputs"].get(field):
differences.append(f"inputs.{field}: {a['inputs'].get(field)} vs "
f"{b['inputs'].get(field)}")
return {"comparable": not differences, "differences": differences}
The comparable check is what turns a trend from an observation into evidence. Two accuracy figures that differ because the method changed are not a trend, and a series that silently mixes them will be interpreted as one.
What becomes possible once it exists
A season of manifests is a dataset, and three questions become one-line queries.
Which camera performs best? Group by camera serial and compare distributions. The answer is occasionally surprising and always more convincing than an impression.
Did that change matter? Split the series at the date of a firmware update, a new geoid model or a processing change and compare the two halves.
Is anything drifting? Fit a trend to horizontal RMSE and to vertical bias separately; the second is the one worth alerting on, because a drifting bias is a datum problem developing.
import pandas as pd
def manifests_to_frame(manifests: list[dict]) -> pd.DataFrame:
"""Flatten a programme's manifests into one queryable table."""
rows = []
for m in manifests:
rows.append({
"survey_id": m["survey_id"],
"flight_date": pd.to_datetime(m["flight_date"]),
"checkpoints": m["statistics"]["checkpoints"],
"h_rmse": m["statistics"]["horizontal"]["rmse_m"],
"v_rmse": m["statistics"]["vertical"]["rmse_m"],
"v_bias": m["statistics"]["vertical"].get("bias_m"),
"camera": m["inputs"]["camera_serial"],
"datum": m["inputs"]["vertical_datum"],
"intrinsics": m["inputs"]["intrinsics_policy"],
"software": m["inputs"]["software_versions"].get("reconstruction"),
})
return pd.DataFrame(rows).sort_values("flight_date")
Figure 3 — Two audiences, one source.
Edge-case matrix
| Situation | Effect | Handling |
|---|---|---|
| Statistics only, no residuals | Nothing recomputable | Include per-checkpoint rows |
| No method recorded | Manifests not comparable | Record split, seed and multiplier |
| Schema changed mid-season | Series breaks | Version the manifest |
| Blunders excluded silently | Statistics not reproducible | Record which and why |
| Manifest in a PDF | Not machine-readable | JSON alongside the report |
| Keys unordered | Diffs unreadable | Sort on write |
| Coordinates in the residuals | Possible confidentiality issue | Offsets only, where required |
| Manifest not stored with the data | Drifts from the deliverable | Write it next to the products |
The confidentiality row comes up on defence and utility sites, where publishing checkpoint coordinates is not acceptable. Residual components without absolute positions preserve every statistic on this page while revealing nothing about where the points are.
Verification snippet
import numpy as np
def recompute_from_manifest(manifest: dict, *, tol: float = 1e-6) -> dict:
"""Recompute the quoted statistics from the stored residuals.
A manifest whose statistics cannot be derived from its own residuals is
internally inconsistent, which usually means one of the two was edited by
hand after generation.
"""
r = np.array([[x["d_east"], x["d_north"], x["d_height"]]
for x in manifest["residuals"]], dtype=float)
h = float(np.sqrt(np.mean(np.linalg.norm(r[:, :2], axis=1) ** 2)))
v = float(np.sqrt(np.mean(r[:, 2] ** 2)))
quoted_h = manifest["statistics"]["horizontal"]["rmse_m"]
quoted_v = manifest["statistics"]["vertical"]["rmse_m"]
problems = []
if abs(h - quoted_h) > tol:
problems.append(f"horizontal RMSE recomputes to {h:.6f}, quoted {quoted_h:.6f}")
if abs(v - quoted_v) > tol:
problems.append(f"vertical RMSE recomputes to {v:.6f}, quoted {quoted_v:.6f}")
if len(r) != manifest["statistics"]["checkpoints"]:
problems.append("checkpoint count disagrees with the residual rows")
return {"consistent": not problems, "problems": problems}
Figure 2 — What a season of manifests makes answerable that a season of reports does not.
Where the manifest should live
A manifest that sits in a different place from the data it describes will eventually describe different data. Three placement rules keep the two together.
Write it next to the deliverables, in the same directory as the orthomosaic and the DEM, not in a separate reports folder. When somebody copies the products to a client’s storage, the accuracy record travels with them rather than being left behind — and a deliverable that arrives without its manifest is then visibly incomplete, which is exactly the signal you want.
Name it deterministically, accuracy.json or <survey_id>.accuracy.json, never with a timestamp in the filename. A timestamped name produces four near-identical files after four reruns and no way to tell which one the report quoted. If earlier runs matter, keep them in a history/ subdirectory with the generation timestamp inside the file, where the generated_at field already records it.
Treat it as generated, never edited. The moment a manifest is opened in an editor to adjust a number, the verification on this page stops meaning anything, because the residuals and the statistics can be brought into agreement by hand. If a figure is wrong, the fix belongs in the computation that produced it, and the manifest is regenerated. A read-only file permission on write is a cheap way to make that habit stick.
When to escalate
- The manifest and the report disagree. One was edited by hand. Regenerate both from the same computation and remove the manual step.
- The schema needs to change. Bump the version and keep a reader for the old shape. A programme’s history is worth more than a tidy schema.
- Checkpoint coordinates cannot be published. Store residual components only; every statistic survives and nothing locational is revealed.