Structuring Drone Imagery for Batch Processing

This page solves a concrete engineering scenario: you have folders of UAV imagery from one or more missions, and before a single reconstruction job runs you must reorganise those frames into a deterministic, validated, CRS-safe layout that a photogrammetry engine can ingest without guesswork. The moment a team moves from one-off flights to enterprise-scale orthomosaic generation, ad-hoc folder structures become the dominant source of failure — duplicate filenames collide across blocks, calibration shots leak into the reconstruction queue, frames with missing GPS silently abort dense matching, and mismatched coordinate systems corrupt the sparse point cloud. The remedy is a script-driven structuring stage that imposes a predictable schema, gates every frame on its metadata, and writes a single manifest that becomes the authoritative input for everything downstream. These principles build directly on Core Photogrammetry Fundamentals for Python Pipelines, where input organisation is treated as a hard pipeline dependency rather than housekeeping.

Audience and prerequisites. This guide targets Python 3.10+ on a 64-bit OS (Linux or Windows) with at least 8 GB RAM; the routines stream per-block, so a survey laptop is sufficient even for tens of thousands of frames. You should be comfortable with pathlib, generators, and basic coordinate-reference-system concepts. All distance and partition math is performed in a projected, metre-based CRS — never in raw latitude/longitude. Before structuring, the flight overlap validation routine should already have confirmed that the captured frames satisfy the overlap budget, so this stage focuses purely on layout, metadata integrity, and spatial partitioning.

# Prerequisites

Install the following libraries before running any snippet on this page. Versions are the minimum tested against Python 3.10+.

No GDAL build is required for structuring itself — CRS handling is delegated to pyproj, which ships self-contained PROJ data wheels. The standard-library pathlib, json, and logging modules cover directory traversal, manifest serialisation, and audit logging.

# Conceptual architecture

Structuring sits between acquisition and reconstruction. It consumes raw, manufacturer-formatted imagery and emits two things: a canonical on-disk tree, and a manifest that records the validated, projected geometry of every frame that survived the gates. Failed frames never reach the expensive stages — they are quarantined with a machine-readable reason so the operator can decide between reflight, manual repair, or permanent exclusion. The output manifest then drives setting up OpenDroneMap with Python, and the per-frame projected coordinates reuse the same CRS contract described in managing coordinate reference systems in GDAL.

A deterministic schema organises projects by mission ID, then by sensor payload, then by chronological capture block. Within each block, frames are renamed with a zero-padded index so filesystem sort order is identical on Linux and Windows. Partitioning into processing blocks is driven by geographic neighbourhoods — bounding boxes or flight-line indices — never by arbitrary file counts, so bundle adjustment always receives contiguous spatial regions rather than randomly sliced fragments.

# Step 1: Build the canonical directory tree

The first step imposes the mission/sensor/block schema and renames frames into a stable, zero-padded sequence. pathlib traverses the source tree; the rename is index-driven so two cameras can never produce a colliding IMG_0001.JPG.

import logging
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger(__name__)

# Project layout: <root>/<mission>/<sensor>/<block>/<frames>
PROJECT_ROOT = Path("/data/uav_missions/mission_2025_07A")
PAD_WIDTH = 4  # zero-padding for the sequential index (IMG_0001.JPG)


def canonicalize_block(block_dir: Path, sensor_tag: str) -> int:
    """Rename frames in a block to a stable, zero-padded, sensor-prefixed sequence.

    Returns the number of frames renamed. Renaming is two-pass (to a temp
    suffix, then to final) so it is safe even when the new names overlap the old.
    """
    frames = sorted(p for p in block_dir.iterdir()
                    if p.suffix.upper() in {".JPG", ".JPEG", ".TIF", ".TIFF"})
    # Pass 1: move everything aside so no source name clashes with a target name.
    staged = []
    for frame in frames:
        tmp = frame.with_name(frame.name + ".staging")
        frame.rename(tmp)
        staged.append(tmp)
    # Pass 2: assign deterministic final names.
    renamed = 0
    for index, tmp in enumerate(staged, start=1):
        final = block_dir / f"{sensor_tag}_{index:0{PAD_WIDTH}d}{tmp.suffixes[0]}"
        tmp.rename(final)
        renamed += 1
    logger.info("Canonicalized %d frames in %s", renamed, block_dir.name)
    return renamed

Renaming in two passes (stage, then finalise) avoids the classic destructive overwrite where a new name collides with a not-yet-moved source. The sensor prefix keeps RGB, multispectral, and thermal frames distinguishable once they share a manifest.

# Step 2: Parse and gate EXIF metadata

Geotagged frames carry GPS, altitude, and timestamp tags that are notoriously inconsistent across manufacturers. The gate below extracts the essentials and rejects anything malformed before it can crash dense matching. Multi-payload acquisitions add band-alignment and timestamp-synchronisation rules covered in handling mixed sensor data in photogrammetry pipelines; here we enforce only the universal minimum.

from pathlib import Path
from typing import Optional
import exifread


def _dms_to_decimal(dms_tag, ref: str) -> float:
    """Convert an EXIF degrees/minutes/seconds ratio tag to signed decimal degrees."""
    deg, minute, sec = (r.num / r.den for r in dms_tag.values)
    decimal = deg + minute / 60 + sec / 3600
    return -decimal if ref in {"S", "W"} else decimal


def parse_frame_exif(img_path: Path) -> Optional[dict]:
    """Return GPS/altitude/timestamp for a frame, or None if any required tag is absent."""
    with open(img_path, "rb") as fh:
        tags = exifread.process_file(fh, details=False)

    required = ("GPS GPSLatitude", "GPS GPSLatitudeRef",
                "GPS GPSLongitude", "GPS GPSLongitudeRef", "GPS GPSAltitude")
    if any(tags.get(key) is None for key in required):
        return None  # caller quarantines with reason "missing_gps"

    alt = tags["GPS GPSAltitude"].values[0]
    return {
        "filename": img_path.name,
        "lat": _dms_to_decimal(tags["GPS GPSLatitude"], str(tags["GPS GPSLatitudeRef"])),
        "lon": _dms_to_decimal(tags["GPS GPSLongitude"], str(tags["GPS GPSLongitudeRef"])),
        "alt_m": alt.num / alt.den,
        "timestamp": str(tags.get("EXIF DateTimeOriginal", "")),
    }

Reading rational tags through r.num / r.den rather than float(str(tag)) avoids locale and rounding bugs that surface on cameras storing high-precision GPS fixes. Returning None keeps the failure explicit so the orchestrator can log a reason rather than crashing on a partially populated frame.

# Step 3: Validate and project coordinates

Many consumer drones embed WGS84 (EPSG:4326) coordinates, while survey workflows demand a projected, metre-based CRS for valid distance and area math. This step performs geospatial sanity checks, then transforms to the project EPSG so the manifest stores both the original and projected geometry.

from typing import Optional
from pyproj import CRS, Transformer

TARGET_CRS = "EPSG:32633"   # UTM Zone 33N — set per project centroid
MIN_ALTITUDE_M, MAX_ALTITUDE_M = 0.0, 500.0
MAX_HDOP = 2.5              # reject fixes with high horizontal dilution of precision

# Build the transformer once: WGS84 (lon/lat) -> projected metric CRS.
_WGS84 = CRS.from_epsg(4326)
_TARGET = CRS.from_user_input(TARGET_CRS)
_TO_TARGET = Transformer.from_crs(_WGS84, _TARGET, always_xy=True)


def project_fix(lat: float, lon: float, alt_m: float) -> Optional[dict]:
    """Validate a geographic fix and project it to the target CRS, or return None."""
    if not (-90 <= lat <= 90) or not (-180 <= lon <= 180):
        return None
    if not (MIN_ALTITUDE_M <= alt_m <= MAX_ALTITUDE_M):
        return None
    easting, northing = _TO_TARGET.transform(lon, lat)  # always_xy -> (x, y)
    return {"easting": easting, "northing": northing, "altitude_m": alt_m,
            "crs": TARGET_CRS}

Constructing the Transformer once at module scope — not inside the per-frame loop — is the single biggest performance win when projecting tens of thousands of fixes; rebuilding it each call dominates runtime. Keep always_xy=True and pass coordinates as (lon, lat) so axis order never silently swaps eastings and northings.

# Step 4: Partition frames into spatial processing blocks

Bundle adjustment converges fastest when each job covers a contiguous region. Rather than slice by file count, group frames into a grid keyed on projected coordinates so every block is a spatial neighbourhood. shapely is optional here; integer grid-cell keys are sufficient and cheap.

from collections import defaultdict

BLOCK_SIZE_M = 250.0  # grid cell edge length in metres


def assign_blocks(records: list[dict]) -> dict[str, list[dict]]:
    """Group projected records into square spatial blocks for independent jobs."""
    blocks: dict[str, list[dict]] = defaultdict(list)
    for rec in records:
        proj = rec["projected_coords"]
        col = int(proj["easting"] // BLOCK_SIZE_M)
        row = int(proj["northing"] // BLOCK_SIZE_M)
        blocks[f"block_{col:+05d}_{row:+05d}"].append(rec)
    # Drop undersized blocks that cannot reconstruct on their own; merge upstream instead.
    return {key: frames for key, frames in blocks.items() if len(frames) >= 3}

A grid edge close to the flight-line spacing keeps each block dense enough for structure-from-motion while bounding per-job memory. Blocks with fewer than three frames are discarded here because they cannot triangulate independently — the orchestrator can widen BLOCK_SIZE_M or merge stragglers into a neighbour.

# Step 5: Generate the processing manifest

The orchestrator ties the previous steps together: it walks the canonical tree, gates each frame, quarantines failures with a reason, partitions survivors, and writes a single processing_manifest.json that is the authoritative downstream input.

import json
from pathlib import Path

REJECTED_DIR = PROJECT_ROOT / "rejected"
MANIFEST_PATH = PROJECT_ROOT / "processing_manifest.json"


def _quarantine(frame: Path, reason: str) -> None:
    REJECTED_DIR.mkdir(parents=True, exist_ok=True)
    frame.rename(REJECTED_DIR / frame.name)
    (REJECTED_DIR / f"{frame.stem}.reason.json").write_text(
        json.dumps({"frame": frame.name, "reason": reason}, indent=2))


def build_manifest(root: Path) -> dict:
    """Validate every frame under the canonical tree and emit the manifest."""
    records: list[dict] = []
    for frame in sorted(root.rglob("*.JPG")):
        exif = parse_frame_exif(frame)
        if exif is None:
            _quarantine(frame, "missing_or_malformed_gps")
            continue
        projected = project_fix(exif["lat"], exif["lon"], exif["alt_m"])
        if projected is None:
            _quarantine(frame, "coordinate_or_altitude_out_of_bounds")
            continue
        records.append({
            "filename": exif["filename"],
            "absolute_path": str(frame.resolve()),
            "file_size_bytes": frame.stat().st_size,
            "gps_wgs84": {"lat": exif["lat"], "lon": exif["lon"], "alt_m": exif["alt_m"]},
            "projected_coords": projected,
            "capture_time": exif["timestamp"],
        })

    manifest = {
        "pipeline_version": "1.3.0",
        "target_crs": TARGET_CRS,
        "blocks": assign_blocks(records),
        "total_valid_frames": len(records),
    }
    MANIFEST_PATH.write_text(json.dumps(manifest, indent=2))
    logger.info("Manifest written: %s (%d valid frames)", MANIFEST_PATH, len(records))
    return manifest


if __name__ == "__main__":
    if not PROJECT_ROOT.exists():
        raise FileNotFoundError(f"Project root not found: {PROJECT_ROOT}")
    build_manifest(PROJECT_ROOT)

Quarantining writes a sidecar .reason.json next to each rejected frame so a failed mission is auditable without rerunning the pipeline. The manifest stores blocks directly, so the photogrammetry engine can submit one job per spatial block in parallel.

# Parameter deep-dive

Every configurable knob, its type and default, the valid range, and how it trades output quality against runtime or memory.

# Verification and output inspection

Never assume the manifest is correct because the script exited zero. Assert structural and geospatial invariants before any downstream job runs.

import json
from pyproj import CRS

manifest = json.loads(MANIFEST_PATH.read_text())

# 1. Manifest exists and is non-empty.
assert manifest["total_valid_frames"] > 0, "no frames survived the gates"

# 2. Every block holds at least the triangulation minimum.
assert all(len(frames) >= 3 for frames in manifest["blocks"].values())

# 3. Declared CRS is genuinely projected and metre-based.
crs = CRS.from_user_input(manifest["target_crs"])
assert crs.is_projected, "manifest CRS must be projected, not geographic"
assert crs.axis_info[0].unit_name == "metre"

# 4. Every absolute_path actually resolves on disk.
from pathlib import Path
for frames in manifest["blocks"].values():
    for rec in frames:
        assert Path(rec["absolute_path"]).exists(), rec["filename"]

# 5. Projected coordinates fall inside the CRS area of use (sanity bound).
for frames in manifest["blocks"].values():
    for rec in frames:
        e, n = rec["projected_coords"]["easting"], rec["projected_coords"]["northing"]
        assert 100_000 <= e <= 900_000, f"easting out of UTM band: {rec['filename']}"
print("manifest verified:", manifest["total_valid_frames"], "frames")

Checking crs.is_projected and the metre unit catches the most damaging silent failure — a manifest that still carries raw degrees, which would make every downstream distance meaningless. The easting band assertion flags a wrong UTM zone before bundle adjustment wastes hours on misplaced cameras.

# Troubleshooting

canonicalize_block raised FileExistsError halfway through and left .staging files behind. How do I recover? The two-pass rename was interrupted. Re-run the function: pass 1 only stages files whose suffix is in the image set, and leftover .staging files keep their original suffix inside the name, so finalise them by renaming any *.staging back and rerunning. Always run structuring on a copy of the raw data until the layout is trusted.

parse_frame_exif returns None for frames that clearly have GPS in the camera app. Why? The manufacturer wrote GPS into a maker-note or XMP block rather than the standard EXIF GPS IFD that exifread reads. Confirm with exifread.process_file(..., details=True) and, if the tags live elsewhere, extract them with an XMP parser before the gate rather than loosening the required-tag check.

Distances and block boundaries look an order of magnitude wrong. What happened? You projected with the wrong UTM zone or swapped latitude and longitude. Keep always_xy=True, pass (lon, lat), and verify TARGET_CRS against the survey centroid — the verification easting-band assertion will catch this before reconstruction.

Valid frames are being quarantined as coordinate_or_altitude_out_of_bounds over hilly terrain. MAX_ALTITUDE_M is referenced to launch height but your decoded GPS altitude is ellipsoidal or geoid-referenced, so genuine frames exceed the bound. Widen the altitude window or convert altitudes to a consistent vertical datum before gating.

The manifest is huge and slow to write on a 40,000-frame survey. Serialising every record into one JSON object holds the whole structure in memory. Stream per-block manifests to newline-delimited JSON files keyed by block id, and keep only block summaries in the top-level manifest.

Two sensors produced colliding filenames after structuring. You renamed both payloads with the same sensor_tag. Give each payload a distinct prefix in handling mixed sensor data in photogrammetry pipelines so RGB, multispectral, and thermal frames never share an index namespace.

# Related

• Calculating Optimal Flight Overlap for Python Processing
• Managing Coordinate Reference Systems in GDAL
• Setting Up OpenDroneMap with Python
• Handling Mixed Sensor Data in Photogrammetry Pipelines

← Core Photogrammetry Fundamentals for Python Pipelines