Removing Moving Objects and Ghosting from Mosaics

A haul road with a lorry on it, photographed twice ninety seconds apart, is two different scenes. The mosaic has to choose, and its default choices produce the two defects clients notice fastest: a half-vehicle, where a hard cut runs through the lorry and takes its cab from one image and nothing from the other, and a ghost, where feathering averages the lorry with the empty road and leaves it translucent.

Both are avoidable, and the detection needed to avoid them is cheap. This page covers it, as part of orthomosaic radiometry and seamline control.

Detection from the overlap disagreement

The signature of a moving object is that two well-registered, well-balanced images disagree strongly over a compact area while agreeing everywhere around it. That is a much easier thing to detect than the object itself, and it needs no object recognition at all.

Three properties separate a moving object from ordinary noise and from tall-structure parallax:

  • Magnitude. The disagreement is large — a dark vehicle on light tarmac is most of the dynamic range, not a few digital numbers.
  • Compactness. It occupies a connected region of roughly vehicle size, not scattered pixels.
  • Asymmetry with height. Parallax on a tall building also produces disagreement, but it produces it as a displaced copy — the same object in two places — whereas a moving vehicle usually appears in one image and not the other.
Moving object and tall-structure parallax distinguished in the overlap difference Two panels showing what the difference between two overlapping images looks like. The left panel, labelled moving object, shows a single compact high-difference region, with the note that the object is present in one image and absent in the other. The right panel, labelled tall structure parallax, shows two compact high-difference regions of similar size and shape a short distance apart, with the note that the same object appears in two places because the two views see it leaning differently. A closing note states that a paired signature means parallax and calls for excluding the footprint, while a single signature means movement and calls for choosing one source. moving object one compact region — present in one image, absent in the other tall-structure parallax two similar regions nearby — the same object seen leaning two ways Paired: exclude the footprint. Single: choose one source for the whole object.

Figure 1 — The difference image answers the question without recognising anything.

Minimal reproducible solution

import numpy as np
from scipy import ndimage


def detect_transients(img_a: np.ndarray, img_b: np.ndarray,
                      overlap: np.ndarray, *,
                      threshold_sigma: float = 4.0,
                      min_area_px: int = 400,
                      max_area_px: int = 40000) -> tuple[np.ndarray, list[dict]]:
    """Find compact regions where two balanced images disagree strongly.

    The threshold is set from the robust spread of the difference inside the
    overlap rather than from a fixed value, so the same call works on a bright
    quarry and a dark woodland without retuning.
    """
    diff = np.abs(img_a.astype("float32") - img_b.astype("float32"))
    inside = diff[overlap]
    if inside.size == 0:
        return np.zeros_like(overlap), []

    centre = float(np.median(inside))
    scale = 1.4826 * float(np.median(np.abs(inside - centre))) or 1.0
    candidate = overlap & (diff > centre + threshold_sigma * scale)

    candidate = ndimage.binary_closing(candidate, iterations=3)
    labels, n = ndimage.label(candidate)

    mask = np.zeros_like(candidate)
    found = []
    for index in range(1, n + 1):
        region = labels == index
        area = int(region.sum())
        if not (min_area_px <= area <= max_area_px):
            continue                      # too small to matter, too big to be a vehicle
        rows, cols = np.nonzero(region)
        found.append({
            "area_px": area,
            "centroid": (float(rows.mean()), float(cols.mean())),
            "mean_difference": float(diff[region].mean()),
        })
        mask |= region
    return mask, found

The area bounds carry real meaning. Below the lower bound, the detection is noise or a small registration error. Above the upper bound, a whole region disagrees, which is a shadow or a balance failure rather than an object — treating that as a moving vehicle and routing around it produces a seamline that takes a long detour for no reason.

Choosing a source rather than blending

Once a transient region is known, the rule is simple and absolute: every pixel of that region comes from one image. Never average, never cut through it.

Which image? Two criteria, applied in order.

Prefer the image without the object. An empty road is almost always the better deliverable — the vehicle was not a feature of the site, it was passing through. Comparing each candidate against the surrounding local median identifies which image contains the anomaly.

Failing that, prefer the more central view. If both images contain the object in different positions, take the one where it sits closer to the frame centre, where lean and occlusion are least.

def choose_source(img_a: np.ndarray, img_b: np.ndarray,
                  region: np.ndarray, surround: np.ndarray) -> str:
    """Which image should supply an entire transient region?

    The image whose values inside the region look most like the surrounding
    ground is the one without the object in it — no recognition required, just
    a comparison against the neighbourhood.
    """
    context = float(np.median(np.concatenate([
        img_a[surround].ravel(), img_b[surround].ravel()])))
    deviation_a = abs(float(np.median(img_a[region])) - context)
    deviation_b = abs(float(np.median(img_b[region])) - context)
    return "a" if deviation_a <= deviation_b else "b"


def region_surround(region: np.ndarray, *, width: int = 20) -> np.ndarray:
    """A ring of clean ground around a region, for the comparison above."""
    grown = ndimage.binary_dilation(region, iterations=width)
    return grown & ~ndimage.binary_dilation(region, iterations=4)

Feeding the transient mask into the seamline cost surface as an exclusion — the same mechanism used for tall buildings — makes the routing honour this automatically, without a special case in the mosaicking code.

Three ways a mosaic can handle a vehicle in the overlap Three outcomes for the same lorry lying in the overlap between two images. Cutting through it produces a half-vehicle, with the cab from one image and empty road from the other. Feathering across it produces a translucent ghost, the vehicle averaged with the empty road. Routing the seamline around it and taking the whole region from the image without the vehicle produces clean empty road, which is the correct deliverable because the vehicle was passing through rather than being a feature of the site. cut through it half a vehicle feather across it translucent ghost route around it clean empty road Whole regions come from one image. Never average across a transient, never cut through one. Feed the transient mask to the seamline search as an exclusion and the routing honours this automatically, with no special case in the mosaicking code.

Figure 2 — Three handlings, one of them correct.

Separating parallax from movement in practice

The paired signature described above is worth detecting explicitly, because the two cases call for opposite responses and confusing them is expensive in both directions. Treating a building as a moving vehicle sends the seamline on a long detour around a structure it could have taken whole from one image; treating a moving vehicle as a building leaves it ghosted.

The test is a cross-correlation between the two candidate regions. Parallax produces near-identical content offset by a short distance along the line joining the two camera positions; movement produces two regions that look nothing alike, because one of them is empty road.

def looks_like_parallax(img_a: np.ndarray, img_b: np.ndarray,
                        region_1: np.ndarray, region_2: np.ndarray,
                        *, similarity_threshold: float = 0.7) -> bool:
    """Do two nearby high-difference regions hold the same content?

    Comparing the patches directly is enough. A building seen from two angles
    correlates strongly with itself; a lorry and the empty road it later
    vacated do not correlate at all.
    """
    def patch(img, region):
        rows, cols = np.nonzero(region)
        sub = img[rows.min():rows.max() + 1, cols.min():cols.max() + 1]
        return (sub - sub.mean()) / (sub.std() or 1.0)

    p1, p2 = patch(img_a, region_1), patch(img_b, region_2)
    height = min(p1.shape[0], p2.shape[0])
    width = min(p1.shape[1], p2.shape[1])
    if height < 8 or width < 8:
        return False
    correlation = float((p1[:height, :width] * p2[:height, :width]).mean())
    return correlation > similarity_threshold

There is a cheaper shortcut that works well where a surface model already exists: threshold the height raster a metre or two above the local ground plane and treat everything above it as a permanent exclusion. Buildings, masts and mature trees are caught by construction, and anything the transient detector then flags outside those footprints is very likely to be moving. Most pipelines that already produce a DSM should use this rather than the correlation test, and keep the correlation test for the sites where no surface model is available at mosaicking time.

What the detector should and should not be asked to catch Three rows. Firmly in scope are vehicles and plant that appear in one image and not another, which are compact, high-contrast and exactly what the area and magnitude bounds are tuned for. Marginal are pedestrians and small equipment, which fall near the lower area bound and will be caught inconsistently, and which are usually small enough that a seamline crossing one is not worth a detour. Out of scope are water, wind-moved vegetation and the shadow edge of a passing cloud, all of which disagree between frames continuously and will flood the detector unless masked beforehand. firmly in scope vehicles and plant — compact, high contrast, what the bounds are tuned for marginal pedestrians and small equipment — near the lower bound, caught inconsistently out of scope water, moving vegetation, cloud shadow edges — mask these before detecting Flooding the detector and raising the threshold is how the vehicles stop being caught too.

Figure 3 — Masking the third row is what keeps the first row working.

Edge-case matrix

Situation Effect Handling
Feathering over a vehicle Translucent ghost Exclusion mask, single source
Hard cut through a vehicle Half a vehicle Route the seam around it
Every frame has the vehicle Cannot remove it Take the most central view
Parallax mistaken for movement Long pointless detours Check for a paired signature
Whole region flagged Shadow or balance failure Cap the region area
Water in the overlap Constant disagreement Mask water before detection
Crop moving in wind Broad low-level difference Below threshold; leave it
Vehicle spans both images’ centres No clean choice Accept it, or re-fly

Water deserves its own note. A lake or a wet quarry floor disagrees between every pair of frames, because the specular reflection depends on view direction. Left unmasked it floods the transient detector with regions, and the seamline search then routes around the entire waterbody, which is rarely what anyone wants.

Verification snippet

def audit_transients(mosaic_mask: np.ndarray, transient_mask: np.ndarray,
                     seam_mask: np.ndarray) -> dict:
    """Confirm no seamline crosses a detected transient in the final mosaic."""
    crossings = ndimage.label(seam_mask & transient_mask)[1]
    covered = float((transient_mask & mosaic_mask).sum()) / max(transient_mask.sum(), 1)
    return {
        "transient_pixels": int(transient_mask.sum()),
        "seam_crossings": int(crossings),
        "coverage": covered,
        "ok": crossings == 0 and covered > 0.99,
    }

A non-zero crossing count means the exclusion cost was not high enough to deflect the path, usually because a transient sits where the overlap is too narrow to route around. That is a flight-planning finding as much as a processing one.

When to escalate

  • A vehicle appears in every overlapping frame. It was parked, not passing. Either accept it in the deliverable or arrange for the site to be cleared before the next flight.
  • The detector floods with regions. Water, dense moving vegetation or a failed balance. Fix the cause; do not raise the threshold until the symptoms stop.
  • Active plant across the whole site. A working quarry during shift hours will never mosaic cleanly. Fly at shift change.

← Orthomosaic Radiometry and Seamline Control