Best Practices for Storing Raw UAV Datasets

You re-open a survey folder three weeks after the flight, launch the reconstruction, and the run dies with Not enough images — or worse, it finishes but the orthomosaic is visibly warped, with seams and doubled rooftops. The imagery looks fine in a thumbnail viewer, so the failure feels random. It is not. The dataset was stored on cloud-synced or consumer media that silently stripped EXIF, locked files during indexing, or flipped a single bit during transfer, and nothing in the pipeline checked. This page fixes the root layer: a deterministic, integrity-verified storage layout for raw imagery, telemetry, RTK/PPK corrections, and calibration files, with a single Python ingest routine that refuses to let a corrupted dataset reach the expensive stages.

# Why raw storage quietly breaks photogrammetry pipelines

Raw UAV imagery is the immutable source of truth for every downstream product — tie-point matching, bundle adjustment, point-cloud densification, and the final orthomosaic. Once the structure-from-motion engine has consumed it, there is no recovering geometry the storage layer threw away. Three storage-side failure modes account for the overwhelming majority of “the data looks fine but the reconstruction is wrong” tickets:

1. Silent EXIF loss. Cloud-sync clients (Dropbox, OneDrive, Google Drive) and some image-management tools rewrite JPEGs to generate previews, dropping the GPSLatitude, GPSLongitude, GPSAltitude, and Make/Model tags. OpenDroneMap then skips those frames or falls back to GPS-free relative reconstruction, which drifts. The fix at parse time is covered in how to validate EXIF GPS data before processing, but prevention starts with never storing active datasets where a sync daemon can touch them.
2. Bit-level corruption during offload. SD-card reads, USB hubs, and SMB/NFS mounts occasionally flip bytes. A corrupted JPEG either fails to decode (a hard error you at least notice) or decodes with garbage in a tile region, injecting phantom features that derail matching. Without a checksum captured at the source, you cannot tell a good copy from a bad one.
3. Non-deterministic paths. Flat dumps, OS-dependent separators, and dynamically named folders break the relative-path parsing that batch processors rely on, producing FileNotFoundError or, on locked files, PermissionError mid-run. A predictable tree is a hard dependency of any batch processing structure, not a tidiness preference.

The canonical layout that defeats all three is a project-centric, ISO 8601 date-stamped hierarchy with explicit role separation:

/project_root/
├── 2025-05-12_site_alpha/
│   ├── raw_imagery/
│   │   ├── flight_01/
│   │   │   ├── DJI_0001.JPG
│   │   │   └── DJI_0001.XMP
│   │   └── flight_02/
│   ├── telemetry/
│   │   ├── flight_01.csv
│   │   └── rtk_corrections.pos
│   ├── calibration/
│   │   └── camera_lens_profile.json
│   └── metadata/
│       └── exif_manifest.csv

Names use ISO 8601 dates (YYYY-MM-DD), flight identifiers (flight_XX), and zero-padded frame numbers (DJI_0001.JPG) so a glob sorts in capture order and no path exceeds the Windows 260-character limit.

Figure 2 — The tiering is a claim about reversibility, not about disks. Anything that can be regenerated is allowed to be destroyed; the one tier that cannot be regenerated is the one nothing is ever permitted to write to.

# Minimal reproducible solution: a verified ingest

The routine below is the whole fix in under 60 lines. It computes a SHA-256 for every image at the source before any copy, materializes the canonical tree, and writes a manifest. Verification (next section) replays the same hashes against the destination, so a flipped bit is caught before the dataset is ever queued for reconstruction.

import hashlib
import shutil
from pathlib import Path

# Directories every project must contain; ingest creates them deterministically.
REQUIRED_DIRS = ("raw_imagery", "telemetry", "calibration", "metadata")
IMAGE_EXTS = {".jpg", ".jpeg", ".tif", ".tiff", ".dng"}


def _sha256(path: Path, chunk: int = 1 << 16) -> str:
    """Stream a file through SHA-256 so large frames never load fully into RAM."""
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()


def ingest_flight(source: Path, project: Path, flight_id: str) -> Path:
    """Copy one flight into the canonical tree and emit a SHA-256 manifest.

    `source`  — the SD-card flight directory (read-only).
    `project` — e.g. project_root / "2025-05-12_site_alpha".
    Returns the manifest path; raises on a structurally empty source.
    """
    for sub in REQUIRED_DIRS:                       # idempotent: safe to re-run.
        (project / sub).mkdir(parents=True, exist_ok=True)

    dest = project / "raw_imagery" / flight_id
    dest.mkdir(parents=True, exist_ok=True)
    images = sorted(p for p in source.iterdir() if p.suffix.lower() in IMAGE_EXTS)
    if not images:
        raise ValueError(f"No imagery found in {source} — wrong card or path?")

    manifest = project / "metadata" / f"{flight_id}.sha256"
    with manifest.open("w") as mf:
        for img in images:
            digest = _sha256(img)                   # hash BEFORE the copy.
            shutil.copy2(img, dest / img.name)      # copy2 preserves mtime + EXIF.
            mf.write(f"{digest}  {flight_id}/{img.name}\n")
    return manifest

shutil.copy2 is deliberate: a plain copy discards the file modification time that some PPK tooling correlates against, and any re-encoding step would mutate the EXIF block. The hash is taken from the original bytes, so it certifies the source, not the copy.

# Edge-case matrix

Storage ingest meets messier inputs than a clean SD card. Handle each explicitly rather than letting it surface as a confusing failure three stages later.

Figure 3 — Three different hops, three different truncations, one check that covers all of them. Comparing file counts between card and archive is the reassurance that misses every case here.

# Verify the copy before anything reads it

Ingest is only trustworthy if you assert on it. The check below replays the manifest against the destination tree; a single mismatch raises, halting the pipeline before a corrupted frame reaches the flight overlap validation routine or the reconstruction engine.

def verify_manifest(manifest: Path, project: Path) -> int:
    """Re-hash every listed file; raise on the first mismatch. Returns count verified."""
    raw_root = project / "raw_imagery"
    verified = 0
    with manifest.open() as mf:
        for line in mf:
            if not line.strip():
                continue
            expected, rel = line.split(maxsplit=1)
            target = raw_root / rel.strip()
            if not target.exists():
                raise FileNotFoundError(f"Missing after copy: {rel.strip()}")
            if _sha256(target) != expected:
                raise RuntimeError(f"CHECKSUM MISMATCH (corrupt copy): {rel.strip()}")
            verified += 1
    assert verified > 0, "Empty manifest — ingest wrote nothing."
    print(f"Integrity OK: {verified} frames match their source hashes.")
    return verified

For network transfers, rsync -avz --checksum --progress followed by sha256sum -c manifest.sha256 --status enforces the same guarantee at the shell level, and tar --zstd produces a lossless archive once a dataset is verified and ready for cold storage.

# When to escalate

This storage-layer fix is necessary but not always sufficient. Escalate to the parent workflow when:

• Verification passes but reconstruction still warps the orthomosaic. The bytes are intact, so the problem is upstream geometry, not storage — re-run the flight overlap validation routine to confirm the capture actually met its forward and side overlap budget.
• Manifests verify but ODM skips frames. EXIF was lost before ingest (in-camera or during a prior copy), not in storage. Move to how to validate EXIF GPS data before processing and reinject corrections from the telemetry logs.
• Storage is sound but the run is I/O-bound or OOMs. This is a processing-configuration concern; tune cache and streaming in setting up OpenDroneMap with Python rather than touching the dataset.

# Related

• Calculating Optimal Flight Overlap for Python Processing
• Structuring Drone Imagery for Batch Processing
• How to Validate EXIF GPS Data Before Processing

← Calculating Optimal Flight Overlap for Python Processing