Deduplicating Overlapping Flights in a Dataset

A project folder holds 1,840 images where the flight plan called for 1,200. Somewhere in there is a card copied twice, a partial re-flight after a battery change, and a deliberate second pass at a lower altitude over the area of interest. Feeding all of it to the reconstruction wastes hours; deleting the wrong third of it costs the survey.

The three cases look identical in a file listing and are cleanly separable from the metadata. This page separates them and keeps the frames that improve the block.

Three kinds of duplicate

Byte-identical copies. The same file present twice under different names, from a card copied to two locations or an interrupted transfer retried. These are pure waste: they add processing time and, worse, add a second observation of every feature at exactly the same camera position, which is a degenerate configuration the solver has to work around.

Re-flights of the same ground. The aircraft covered the same strip twice, usually after an interruption. The frames are different photographs of the same place, taken minutes apart under near-identical conditions. Keeping both adds redundancy the reconstruction can use, and adds processing time proportional to the duplication.

Deliberate multi-resolution coverage. A second pass at a different altitude, or an oblique pass over a structure. These are not duplicates at all: the differing viewpoint is exactly what improves the reconstruction of the area they cover, and removing them removes the reason they were flown.

Three kinds of overlap and what to do with each Three cases of frames covering the same ground. Byte-identical copies share a checksum and a camera position, contribute nothing, and should be removed. Re-flights share the ground but have different positions and timestamps, contribute genuine redundancy, and should be thinned rather than deleted. Multi-resolution passes share the ground with a different altitude or viewing angle, contribute the viewpoint diversity they were flown for, and must be kept in full. Each case names the metadata field that identifies it: checksum, timestamp gap, and altitude or attitude difference. byte-identical copy same checksum same camera position same timestamp contributes nothing and creates a degenerate zero-baseline pair remove re-flight different checksum nearby position minutes apart genuine redundancy useful up to a point; costs time beyond it thin, do not delete multi-resolution pass different altitude or different attitude possibly a different day the reason it was flown viewpoint diversity is what improves the block keep in full Three cases, three metadata fields, three different actions — and identical appearances in a file listing.

Figure 1 — The classification. A deduplication that treats all three the same either wastes hours or deletes the pass the survey was flown for.

Minimal reproducible solution

Classify against the manifest built during ingest, using checksum, position, timestamp and altitude in that order.

from collections import defaultdict
from dataclasses import dataclass

import numpy as np


@dataclass(frozen=True)
class Frame:
    path: str
    sha256: str
    easting: float
    northing: float
    height_m: float
    timestamp_s: float


def classify_overlaps(frames: list[Frame], position_tol_m: float = 3.0,
                      time_gap_s: float = 120.0,
                      height_tol_m: float = 15.0) -> dict[str, list]:
    """Group frames into exact duplicates, re-flights, and distinct passes."""
    by_hash: dict[str, list[Frame]] = defaultdict(list)
    for f in frames:
        by_hash[f.sha256].append(f)
    exact = [group for group in by_hash.values() if len(group) > 1]

    unique = [group[0] for group in by_hash.values()]
    xy = np.array([[f.easting, f.northing] for f in unique])

    reflights: list[tuple[Frame, Frame]] = []
    distinct: list[tuple[Frame, Frame]] = []
    for i in range(len(unique)):
        d = np.linalg.norm(xy - xy[i], axis=1)
        for j in np.where((d < position_tol_m) & (np.arange(len(unique)) > i))[0]:
            a, b = unique[i], unique[j]
            if abs(a.height_m - b.height_m) > height_tol_m:
                distinct.append((a, b))          # different altitude: keep both
            elif abs(a.timestamp_s - b.timestamp_s) > time_gap_s:
                reflights.append((a, b))          # same place, later: redundancy
            # else: adjacent frames of one pass — not an overlap at all
    return {"exact": exact, "reflights": reflights, "distinct": distinct}

The order of the tests is what makes the classification correct. Height is checked before time, because a second pass at a different altitude flown ten minutes later is a multi-resolution pass and not a re-flight — testing time first would misclassify it and thin away exactly the frames that were flown deliberately.

Removing exact duplicates is unconditional; thinning re-flights is a judgement about how much redundancy is worth its processing cost.

def thin_reflights(frames: list[Frame], reflight_pairs: list, keep_ratio: float = 0.5):
    """Drop a fraction of the later frames in each re-flown region.

    Keeps the earlier frame of each pair preferentially, since a re-flight
    after an interruption is often flown in worse light. Returns the paths to
    exclude rather than deleting anything.
    """
    later = {b.path for _, b in reflight_pairs}
    ordered = sorted(later)
    drop = set(ordered[::max(1, int(1 / max(1 - keep_ratio, 1e-6)))])
    return drop

Returning paths to exclude rather than deleting files is deliberate. The frames stay in the immutable archive; only the working set and the manifest change, so a decision that turns out to be wrong is reversible by regenerating the manifest.

Edge-case matrix

Situation Signature Action
Card copied twice Identical checksums Remove one, unconditionally
Interrupted transfer retried Identical checksums, different names Remove one
Re-flight after a battery change Same place, > 2 min apart, same height Thin
Second pass at a lower altitude Same place, height differs Keep both
Oblique pass over a structure Same place, attitude differs Keep both
Two surveys months apart Same place, days apart Separate projects, not duplicates
Adjacent frames in one strip Same place, seconds apart Not an overlap
Same ground, different sensor Same place, different camera model Keep; route separately

The last row connects to a different decision entirely: frames from two payloads over the same ground are not duplicates and should not be merged into one reconstruction either, which is the routing question in handling mixed sensor data in photogrammetry pipelines.

Verify the fix worked

Deduplication must not open a coverage gap, which is the one way it can do real harm.

import numpy as np


def assert_coverage_preserved(before: list[Frame], after: list[Frame],
                              footprint_radius_m: float,
                              min_multiplicity: int = 3) -> None:
    """Every point still covered by at least `min_multiplicity` frames."""
    xy_before = np.array([[f.easting, f.northing] for f in before])
    xy_after = np.array([[f.easting, f.northing] for f in after])

    # Sample the area on a grid at half the footprint radius.
    step = footprint_radius_m / 2
    xs = np.arange(xy_before[:, 0].min(), xy_before[:, 0].max() + step, step)
    ys = np.arange(xy_before[:, 1].min(), xy_before[:, 1].max() + step, step)
    gx, gy = np.meshgrid(xs, ys)
    pts = np.column_stack([gx.ravel(), gy.ravel()])

    def multiplicity(cams):
        d = np.linalg.norm(pts[:, None, :] - cams[None, :, :], axis=2)
        return (d <= footprint_radius_m).sum(axis=1)

    m_before, m_after = multiplicity(xy_before), multiplicity(xy_after)
    covered = m_before >= min_multiplicity          # only where it was covered
    lost = int(np.sum(covered & (m_after < min_multiplicity)))
    assert lost == 0, (
        f"{lost} sample points fell below {min_multiplicity} views after "
        "deduplication — the thinning opened a coverage gap")

Evaluating only where coverage was adequate before is what keeps this honest: an area the original flight never covered properly is not a regression caused by thinning, and including it would make the assertion fail for a reason deduplication cannot fix.

Thinning uniformly against thinning by multiplicity Two thinning strategies over a survey with an unevenly re-flown region. Dropping every second frame of the re-flight uniformly reduces multiplicity everywhere, including a strip that was already at the minimum, leaving that strip below three views. Dropping frames only where multiplicity exceeds a target leaves the marginal strip untouched and removes frames only from the doubly covered core. A note observes that both remove the same number of frames and only one of them preserves coverage. uniform thinning marginal strip — 3 views re-flown core — 7 views strip drops to 2 views core drops to 4 — still ample thinning by multiplicity marginal strip — untouched re-flown core — thinned to 4 strip stays at 3 views same number of frames removed Both strategies remove the same count; only the second one knows where the redundancy actually was.

Figure 2 — Where to take the frames from. Uniform thinning is simpler and removes redundancy from the one place that had none to spare.

Where this belongs in the pipeline

Deduplication is an ingest concern, not a processing one. By the time frames reach the reconstruction they have already been copied, checksummed and indexed, and every duplicate has cost storage and transfer bandwidth as well as the compute it is about to cost.

Running the classification as the last step of ingest — after the manifest is built and before any block is planned — has three benefits. The checksums it needs are already computed, since the verified ingest computes them anyway. The positions it needs are already validated. And the decision is recorded in the manifest rather than applied as a deletion, so a project reprocessed a year later reproduces the same working set rather than a differently-thinned one.

Record the counts, too: how many frames were exact duplicates, how many were thinned, and how many distinct passes were identified. Those three numbers turn “the project has 1,840 images and the plan called for 1,200” into an explained figure rather than a discrepancy somebody will re-investigate on the next flight of the same site.

When to escalate

  • Exact duplicates exist across two archive locations. Deduplicating the working set is right; deduplicating the archive is not, because a second copy is a backup. Resolve the working set and leave the immutable tier alone, as set out in best practices for storing raw UAV datasets.
  • Removing duplicates does not reduce processing time. The duplicates were not the constraint. Check whether the pair-selection stage was already excluding them, in which case they cost storage rather than compute.
  • Two passes were flown at the same altitude on different days. These are separate epochs rather than a re-flight, and merging them into one reconstruction averages any real change between the two. Process them separately and compare.

← Structuring Drone Imagery for Batch Processing

Why an exact duplicate is worse than a wasted file Two image pairs shown with their triangulation geometry. A normal pair has a baseline between the two camera positions, so rays from the two images intersect at a well-defined angle and the depth of the ground point is well determined. A duplicated frame has zero baseline, so the two rays are collinear and the intersection is undefined at any depth. A note explains that the matcher happily produces perfect correspondences between the two copies, which then contribute no geometric information while consuming matching time and appearing in the statistics as excellent matches. normal pair — real baseline baseline depth well determined duplicated frame — zero baseline two cameras, one position rays collinear — depth undefined The matcher reports perfect correspondences, and they carry no geometric information at all. So a duplicate does not merely waste time; it inflates the match statistics that would otherwise flag a weak block.

Figure 3 — The second cost of an exact duplicate. It contributes a pair with a perfect inlier ratio and no baseline, which flatters exactly the statistic used to judge whether the block is well connected.