Matching Thermal and RGB Orthomosaics

The thermal orthomosaic and the RGB orthomosaic of the same site do not line up. A roof that is crisp in the RGB is offset by two metres in the thermal, and the offset varies across the survey — a metre here, three metres there, in different directions.

The instinct is to warp one onto the other. That fixes the appearance and not the product: a thermal mosaic built from a failed reconstruction has internal geometric errors that no global warp removes, so aligning it at one point misaligns it at another.

The robust approach is to stop reconstructing from thermal imagery at all. This page covers solving the geometry from the RGB camera, measuring the fixed transform between the two sensors, and projecting thermal frames through the RGB camera model. It is the geometric strategy referenced throughout thermal orthomosaic processing in Python.

Why thermal reconstruction fails

Structure-from-motion needs distinctive, repeatable features across many frames. Thermal imagery of a landscape offers few of them.

The resolution is low — 640 × 512 is common, against 20 megapixels for an RGB camera on the same airframe — so there are two orders of magnitude fewer pixels to find features in. The contrast is driven by temperature differences, which are smooth over most natural surfaces; a field that is rich in visual texture is nearly featureless thermally. And what contrast there is moves: shadows shift, surfaces warm unevenly, and a feature matched between two frames taken eight minutes apart may not be in the same place because the thermal scene itself changed.

The result is reconstructions that converge on a subset of frames, break into submodels, or produce a camera solution with large residuals — exactly the failure modes described in troubleshooting alignment and matching failures, but reached far more easily.

Two strategies for placing thermal frames Two pipelines compared. In the first, thermal frames are reconstructed independently, producing a weak camera solution and a mosaic with internal geometric errors of one to three metres that vary across the site. In the second, the RGB frames are reconstructed to centimetre accuracy and the thermal frames are projected through the resulting camera poses using a fixed rigid transform measured once, giving the thermal product the RGB survey's geometry. A note states that warping the finished mosaics together cannot fix the first case because its errors are internal rather than global. reconstruct from thermal thermal frames weak solution submodels, large residuals mosaic with internal errors of 1–3 m varying across the site — no global warp fixes it project through the RGB solution RGB frames strong solution centimetre accuracy thermal projected through the same poses via one rigid transform, measured once The thermal product inherits the RGB survey's geometry. And the transform is a property of the rig, so it is measured once and reused.

Figure 1 — Two pipelines. Only one of them has a geometry worth the radiometry it carries.

Measuring the rigid transform once

The two cameras are bolted to the same frame, so the transform between them is fixed. It is measured from a calibration flight over a target visible in both — which in practice means a target with both visual and thermal contrast, such as a metal plate on grass, or a tray of water.

import cv2
import numpy as np


def solve_extrinsic(rgb_points: np.ndarray, thermal_points: np.ndarray,
                    rgb_K: np.ndarray, thermal_K: np.ndarray,
                    dist_rgb=None, dist_thermal=None) -> dict:
    """Rigid transform from the RGB camera frame to the thermal camera frame.

    Uses correspondences on a target seen by both. Because the cameras are
    rigidly mounted, one good calibration flight serves indefinitely — but it
    must be re-measured whenever either camera is removed or serviced.
    """
    ok, rvec, tvec, inliers = cv2.solvePnPRansac(
        rgb_points.astype(np.float32).reshape(-1, 1, 3),
        thermal_points.astype(np.float32).reshape(-1, 1, 2),
        thermal_K, dist_thermal, flags=cv2.SOLVEPNP_ITERATIVE,
        reprojectionError=2.0, confidence=0.999)
    if not ok:
        raise ValueError("could not solve the camera-to-camera transform")

    R, _ = cv2.Rodrigues(rvec)
    T = np.eye(4)
    T[:3, :3] = R
    T[:3, 3] = tvec.ravel()

    projected, _ = cv2.projectPoints(rgb_points.astype(np.float32), rvec, tvec,
                                     thermal_K, dist_thermal)
    resid = np.linalg.norm(projected.reshape(-1, 2) - thermal_points, axis=1)
    return {"transform": T,
            "baseline_m": float(np.linalg.norm(tvec)),
            "inliers": int(len(inliers) if inliers is not None else 0),
            "median_residual_px": float(np.median(resid))}

A residual above about two thermal pixels means the correspondences or the thermal intrinsics are poor, and the transform should not be trusted. A baseline far from the physical separation of the two cameras on the rig — which is measurable with a ruler — is a second and very effective sanity check.

Projecting the thermal frames

import numpy as np


def thermal_pose_from_rgb(rgb_pose: np.ndarray, extrinsic: np.ndarray) -> np.ndarray:
    """Thermal camera pose, given the RGB pose the reconstruction solved.

    The reconstruction solved where the RGB camera was for every frame. The
    thermal camera was, by construction, at a fixed offset from it, so its
    pose is the composition — no thermal feature matching required.
    """
    return rgb_pose @ np.linalg.inv(extrinsic)


def time_align(rgb_times: np.ndarray, thermal_times: np.ndarray,
               *, max_gap_s: float = 0.5) -> list[tuple[int, int]]:
    """Pair each thermal frame with the RGB frame taken closest in time.

    The two cameras rarely trigger simultaneously, and at 8 m/s a half-second
    offset is four metres of ground. Frames that cannot be paired within the
    tolerance are excluded rather than paired approximately.
    """
    pairs = []
    for j, t in enumerate(thermal_times):
        i = int(np.argmin(np.abs(rgb_times - t)))
        if abs(rgb_times[i] - t) <= max_gap_s:
            pairs.append((i, j))
    return pairs

Time alignment is where this approach most often goes wrong in practice. The two cameras have independent clocks, and a drift of even a second between them puts every thermal frame several metres from where the pipeline thinks it is. Synchronising the clocks before the flight, or recording a shared event both cameras see, is worth the small effort.

Three obstacles to aligning a thermal mosaic with an RGB one Three rows. Resolution differs by roughly an order of magnitude, so a feature matcher tuned for RGB finds almost nothing in the thermal image and the match must be driven from geometry rather than from texture. Feature appearance differs fundamentally, because what is visible in thermal is a temperature pattern that need not correspond to any visible edge — a shaded wall and a sunlit one differ thermally and may look identical in RGB. Capture time differs, so even a perfect geometric alignment pairs a thermal pixel with an RGB pixel of a different moment, which matters wherever anything moved. resolution an order of magnitude apart — drive the match from geometry, not texture feature appearance a thermal edge need not be a visible edge, and often is not capture time even a perfect alignment pairs different moments wherever anything moved Align through the shared camera geometry and the surface model, not through image matching.

Figure 3 — Three reasons image matching is the wrong tool here.

Edge-case matrix

Situation Handling
No RGB camera flown Reconstruct from thermal and accept the geometry, or re-fly
Cameras triggered independently Pair on timestamps; exclude unpaired frames
Clock drift between cameras Fit an offset from a shared event; re-check each flight
Camera removed and remounted Re-measure the extrinsic; it has changed
Thermal intrinsics unknown Calibrate from a thermal checkerboard before anything else
Target with no thermal contrast Use a metal plate or a water tray, not a printed board
Thermal frames at a different rate Expected; pair by time and use every thermal frame once
Large terrain relief Project through the reconstructed surface, not a plane

The last row matters on any site with structures. Projecting a thermal frame onto a flat plane places roofs at ground elevation, displacing them by the building height times the tangent of the view angle — several metres at the edge of a frame. Projecting through the reconstructed surface removes it.

Verification snippet

import cv2
import numpy as np


def check_alignment(thermal_mosaic: np.ndarray, rgb_mosaic: np.ndarray,
                    *, sample_windows: int = 40, window_px: int = 128) -> dict:
    """Measure residual offsets between the two mosaics at many locations.

    A single global offset means the extrinsic is wrong; offsets that vary
    across the site mean the geometry is wrong, which a warp cannot fix.
    """
    h, w = thermal_mosaic.shape[:2]
    rng = np.random.default_rng(0)
    offsets = []
    for _ in range(sample_windows):
        y = int(rng.integers(0, max(h - window_px, 1)))
        x = int(rng.integers(0, max(w - window_px, 1)))
        a = np.nan_to_num(thermal_mosaic[y:y + window_px, x:x + window_px]).astype(np.float32)
        b = np.nan_to_num(rgb_mosaic[y:y + window_px, x:x + window_px]).astype(np.float32)
        if a.std() < 1e-3 or b.std() < 1e-3:
            continue
        shift = cv2.phaseCorrelate((a - a.mean()) / a.std(), (b - b.mean()) / b.std())[0]
        offsets.append(shift)

    arr = np.array(offsets)
    if arr.size < 8:
        return {"note": "too few textured windows to measure"}
    return {"median_offset_px": arr.mean(axis=0).tolist(),
            "offset_spread_px": float(np.std(np.linalg.norm(arr, axis=1))),
            "systematic": bool(np.linalg.norm(arr.mean(axis=0)) > 2 * np.std(arr)),
            "note": ("a consistent offset — the extrinsic is wrong"
                     if np.linalg.norm(arr.mean(axis=0)) > 2 * np.std(arr)
                     else "offsets vary across the site — the geometry is wrong")}

Distinguishing a consistent offset from a varying one is the whole diagnostic. A consistent offset is a transform error and is fixable in seconds; a varying one means the thermal geometry is internally wrong and the projection approach is needed.

Residual offsets across the site, for two kinds of fault Two plan views with arrows showing the residual offset between thermal and RGB at sampled locations. On the left, every arrow points the same direction with the same length, indicating a constant offset caused by a wrong camera-to-camera transform, which is fixable. On the right, the arrows vary in direction and length across the site, indicating that the thermal geometry itself is wrong, which no global warp can correct. A note gives the test as whether the offsets are consistent. consistent — transform error varying — geometry error fix the extrinsic and re-project reconstruct from RGB and project the thermal One picture separates a five-minute fix from a reprocessing job. And it needs nothing beyond the two mosaics already in hand.

Figure 2 — The diagnostic. Consistency of the offsets is the whole question.

When to escalate

  • No RGB camera was flown. Either accept the thermal geometry with its errors stated, or re-fly. There is no processing fix for a reconstruction that had insufficient features.
  • The extrinsic changes between flights. A camera is not rigidly mounted, or is being removed between jobs. That is a hardware issue and every flight until it is resolved needs its own calibration.
  • Alignment is good and features still look displaced on tall structures. The projection used a plane rather than the reconstructed surface. Re-project through the surface model.

← Thermal Orthomosaic Processing in Python