Handling Mixed Sensor Data in Photogrammetry Pipelines

You feed a single reconstruction job imagery from two drones — or one drone carrying an RGB, multispectral, and thermal payload — and the run dies during sparse alignment. The log shows Reconstruction failed, a sparse cloud that splits into two disconnected components, or a bundle adjuster that bows the model into a dome. The frames look fine in a viewer, so the failure feels random. It is not: the moment imagery from heterogeneous cameras, focal lengths, or staggered flight campaigns enters the same Structure-from-Motion (SfM) solver without explicit harmonization, the geometric and radiometric divergence between sensors breaks feature matching and destabilizes the optimization. This page gives you the intrinsics-consistency gate that catches the problem before ingest, the harmonization steps that make the frames matchable, and a routing layer that isolates a payload rather than letting it poison the whole block. It is the mixed-sensor edge case of structuring drone imagery for batch processing, where every frame is gated on its metadata before a job is queued.

# Why mixed sensors break SfM alignment

SfM assumes a consistent pinhole camera model per image group: a stable focal length, a known principal point, and a predictable sensor footprint. When you mix payloads, three of those assumptions break at once. First, focal lengths and sensor dimensions differ between cameras, so a bundle adjuster that refines a single shared intrinsic set is fed contradictory geometry and either diverges or bends the surface into the classic bowl/dome artifact. Second, the cameras fly at different altitudes or carry different lenses, so their ground sampling distance (GSD) diverges — and once the per-pixel scale between two frames differs by more than a few percent, SIFT-class descriptors lose the scale invariance they rely on and matching collapses to a handful of inliers. Third, RGB, thermal, and multispectral sensors record entirely different radiometry, so a descriptor computed on a thermal frame has almost nothing in common with one computed on the RGB frame of the same scene.

The dangerous part is that none of this announces itself. EXIF is frequently incomplete or silently wrong — a missing focal-plane resolution tag, a mislabeled sensor, a focal length that drifts 4% across a “single” camera profile because two firmware revisions reported it differently. Because batch processing structure feeds whatever it finds straight into the engine, a single mislabeled payload corrupts camera-pose estimation long before any visible artifact appears. The fix is a deterministic validation gate that groups frames by true camera intrinsics, flags divergence, and refuses to let an inconsistent profile reach the solver. The georeferencing side of the same problem — frames arriving in mismatched coordinate systems — is handled separately by managing coordinate reference systems in GDAL.

# Minimal reproducible fix

The highest-leverage change is a pre-ingest gate that parses EXIF, groups frames by their real camera profile, and rejects any profile whose focal length, resolution, or derived sensor size is internally inconsistent. Focal length and the focal-plane tags live in the Exif sub-IFD (0x8769), not the 0th IFD — reading them from the wrong directory is itself a common source of phantom “missing” intrinsics.

import logging
from pathlib import Path
from PIL import Image

logging.basicConfig(level=logging.INFO, format="[%(levelname)s] %(message)s")

# focal-plane resolution unit tag -> millimetres per unit
_UNIT_MM = {2: 25.4, 3: 10.0, 4: 1.0, 5: 0.001}


def profile_images(image_dir: str, max_focal_variance: float = 0.03) -> dict:
    """Group frames by camera and flag intrinsics that will break bundle adjustment."""
    profiles: dict[str, dict] = {}
    for img_path in Path(image_dir).glob("*.[Jj][Pp][Gg]"):
        with Image.open(img_path) as img:
            exif = img.getexif()
            sub = exif.get_ifd(0x8769)            # Exif sub-IFD holds the real intrinsics
            make = str(exif.get(271, "Unknown")).strip()
            model = str(exif.get(272, "Unknown")).strip()
            focal_mm = float(sub.get(37386, 0))   # FocalLength, already in mm
            width, height = img.size

            unit = _UNIT_MM.get(sub.get(41488, 2), 25.4)
            fp_x, fp_y = float(sub.get(41486, 0)), float(sub.get(41487, 0))
            sensor_w = (width / fp_x) * unit if fp_x else 0.0   # derive sensor size

        key = f"{make}_{model}"
        p = profiles.setdefault(key, {"focal": [], "res": set(), "sensor_w": set()})
        p["focal"].append(focal_mm)
        p["res"].add((width, height))
        p["sensor_w"].add(round(sensor_w, 2))

    for cam, p in profiles.items():
        mean = sum(p["focal"]) / len(p["focal"]) if p["focal"] else 0.0
        var = (max(p["focal"]) - min(p["focal"])) / mean if mean else 0.0
        if var > max_focal_variance:
            logging.warning("%s: focal variance %.2f%% > %.0f%% — lock intrinsics, do not refine",
                            cam, var * 100, max_focal_variance * 100)
        if len(p["res"]) > 1:
            logging.warning("%s: mixed resolutions %s — resample to a common GSD", cam, p["res"])
        if len(p["sensor_w"]) > 1:
            logging.warning("%s: inconsistent sensor width %s — verify calibration sheet", cam, p["sensor_w"])
    return profiles

A profile that trips the focal-variance warning must be reconstructed with intrinsics locked (refinement disabled) rather than refined per image, and a profile that mixes resolutions must be resampled to a shared GSD before ingest. Both actions are covered next.

# Harmonizing GSD and radiometry across payloads

Once the gate has grouped frames, bring every group onto a common pixel footprint. The target GSD for a frame is set by the capture geometry:

$$\text{GSD}_{\text{cm/px}} = \frac{S_w \cdot H \cdot 100}{f \cdot I_w}$$

where $S_w$ is sensor width (mm), $H$ is flight altitude (m), $f$ is focal length (mm), and $I_w$ is image width (px). When two payloads’ computed GSD differ by more than ±5%, resample the coarser imagery to the finer footprint before matching. GDAL handles the resample deterministically:

# Resample to a unified 2.5 cm/px footprint with tiled, compressed output
gdalwarp -tr 0.025 0.025 -r bilinear -co COMPRESS=LZW -co TILED=YES \
         -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 \
         input_mixed_sensor.tif output_harmonized.tif

Radiometry needs its own pass. RGB and thermal frames will never share descriptors directly, so do not try to match across spectra — match within each spectrum and fuse later. Where you must align a thermal or NIR band to RGB for fusion, estimate an affine warp with cv2.findHomography(..., cv2.RANSAC, maxIters=5000) on co-registered control points rather than relying on the SfM solver to bridge the gap. The working thresholds:

• GSD: abs(gsd_current - gsd_target) / gsd_target > 0.05 triggers a resample.
• Radiometric offset within a spectrum: a mean channel delta above ~15 DN warrants histogram matching before extraction.
• Cross-spectrum alignment: affine homography on shared control points, never direct descriptor matching.

# Tuning cross-sensor feature matching and bundle adjustment

Even after harmonization, descriptors carry more noise across sensors than within one. The feature detection stage needs a higher noise floor and a relaxed ratio, and the solver must not refine focal length when EXIF variance is high. These COLMAP flags hold a mixed block together:

Driving COLMAP through its CLI keeps the flags identical to the table above and avoids pycolmap option objects whose names change between releases:

import subprocess
from pathlib import Path

def run_locked_sfm(image_dir: str) -> None:
    """Run COLMAP sparse reconstruction with locked intrinsics for mixed-sensor stability."""
    image_path = Path(image_dir)
    db_path = image_path / "database.db"
    sparse_path = image_path / "sparse"
    sparse_path.mkdir(exist_ok=True)

    subprocess.run([
        "colmap", "feature_extractor",
        "--database_path", str(db_path),
        "--image_path", str(image_path),
        "--SiftExtraction.peak_threshold", "0.003",
        "--SiftExtraction.max_num_features", "12000",
    ], check=True)

    subprocess.run([
        "colmap", "exhaustive_matcher",
        "--database_path", str(db_path),
        "--SiftMatching.max_ratio", "0.85",
    ], check=True)

    subprocess.run([
        "colmap", "mapper",
        "--database_path", str(db_path),
        "--image_path", str(image_path),
        "--output_path", str(sparse_path),
        "--Mapper.ba_refine_focal_length", "0",
        "--Mapper.filter_max_reproj_error", "4.0",
    ], check=True)

# Deterministic fallback routing

A mixed block will sometimes refuse to register no matter how the matcher is tuned, and an automated run must degrade gracefully instead of aborting. Route on the alignment report: when inlier counts or reprojection error breach their bounds, fall back to sequential pairwise alignment with ground control point injection, and if that still fails, isolate the offending payload and process it on its own rather than letting it fragment the whole reconstruction.

Figure 1 — PipelineRouter decision logic: alignment proceeds only when inlier counts and reprojection error stay within bounds; otherwise the controller degrades to pairwise alignment, GCP injection, or payload isolation.

class PipelineRouter:
    def __init__(self, min_inliers: int = 30, max_reproj_error: float = 2.0):
        self.min_inliers = min_inliers
        self.max_reproj_error = max_reproj_error

    def evaluate_alignment(self, match_report: dict) -> str:
        inliers = match_report.get("num_inliers", 0)
        mean_error = match_report.get("mean_reproj_error", 0.0)
        if inliers < self.min_inliers or mean_error > self.max_reproj_error:
            logging.warning("Alignment threshold breached; routing to fallback strategy.")
            return "fallback_pairwise"
        return "proceed_bundle_adjustment"

    def execute_fallback(self, strategy: str, image_subset: list) -> None:
        if strategy == "fallback_pairwise":
            logging.info("Sequential pairwise alignment with GCP injection on %d frames.", len(image_subset))
            # e.g. cv2.estimateAffinePartial2D, or OpenDroneMap --force-gcp
        elif strategy == "proceed_bundle_adjustment":
            logging.info("Proceeding to dense reconstruction.")

# Edge-case matrix

The validation gate must handle malformed metadata, not just the happy path. These are the input variants that recur in real mixed-sensor blocks and the behavior the gate should enforce:

# Verifying the fix

Re-run the affected directory through the gate and assert that every camera profile is internally consistent before any job is queued. Turning the warnings into a hard assertion converts a silent reconstruction failure into a fast, local error:

def assert_profiles_consistent(image_dir: str, max_focal_variance: float = 0.03) -> bool:
    profiles = profile_images(image_dir, max_focal_variance)
    for cam, p in profiles.items():
        mean = sum(p["focal"]) / len(p["focal"]) if p["focal"] else 0.0
        var = (max(p["focal"]) - min(p["focal"])) / mean if mean else 0.0
        assert var <= max_focal_variance, f"{cam}: focal variance {var:.2%} — lock intrinsics or split profile"
        assert len(p["res"]) == 1, f"{cam}: mixed resolutions {p['res']} — resample before ingest"
        assert len(p["sensor_w"]) == 1, f"{cam}: inconsistent sensor width {p['sensor_w']}"
    logging.info("All %d camera profile(s) consistent — safe to queue.", len(profiles))
    return True

# Expect: one clean profile per distinct camera, and no AssertionError.

If the assertion passes and the reconstruction that previously fragmented now registers as a single connected component, the metadata divergence was the fix.

# When to escalate

This page fixes per-sensor intrinsics, harmonization, and routing. Escalate when the failure is no longer about which sensor produced the frame:

• Profiles are consistent but the model still bows or diverges. The problem has moved into the optimization itself — tune the solver as covered in optimizing bundle adjustment with Python rather than retuning the gate.
• Fallback routing keeps choosing GCP injection. If pairwise alignment only succeeds with control points, the block needs a real ground-control workflow before reconstruction; wire it through setting up OpenDroneMap with Python and its --force-gcp path.
• Every payload reconstructs alone but they will not georeference together. The remaining divergence is spatial-reference, not sensor — resolve it with managing coordinate reference systems in GDAL before fusing the outputs.

# Related

• Structuring Drone Imagery for Batch Processing
• Managing Coordinate Reference Systems in GDAL
• Setting Up OpenDroneMap with Python
• Optimizing Bundle Adjustment with Python

← Structuring Drone Imagery for Batch Processing