Masking Sky and Water Before Feature Detection

A coastal survey reconstructs the land correctly and puts a scatter of points twenty metres below the sea surface. An oblique flight over a reservoir produces camera poses that wander. In both cases the detector found plenty of keypoints on the water, matched a good fraction of them, and the geometry it inferred from them is nonsense — because the features it matched were sun glint and wave crests, which move between exposures.

This page excludes those regions before detection, using masks derived from what the surfaces physically do rather than from a colour range that a different day’s light will break.

Why water and sky are not merely low-texture

Sand is hard to match; water is impossible to match correctly, and the distinction changes the remedy. A sand flat has real, stationary structure that a lowered contrast threshold recovers. Water has plenty of high-contrast structure and it is not attached to the ground: a wave crest photographed two seconds apart is a different wave, and specular glint moves with the observer rather than with the scene.

That produces matches which pass the ratio test — the descriptors genuinely are similar — and then pass geometric verification often enough to matter, because a set of moving points can be consistent with some camera geometry. The result is a small population of confident, wrong correspondences distributed over the water area, pulling the solution toward a geometry that explains them.

Sky behaves the same way for a different reason: on an oblique or corridor flight, cloud edges are strong features at effectively infinite distance, so they constrain rotation and provide no scale, and a solver that treats them as ground points produces a badly conditioned block.

A stationary feature and a moving one, two seconds apart Two frames of the same scene taken two seconds apart, showing a shoreline with land above and water below. On the land, a rock produces a keypoint at the same ground position in both frames, so the correspondence is correct and the ray intersection is well conditioned. On the water, a sun-glint highlight appears in both frames at positions that correspond to no fixed ground point, because the glint follows the observer; the descriptors match, the correspondence is accepted, and the triangulated point lands below the water surface at an arbitrary depth. A note states that this is a correct match of an incorrect assumption, which is why filtering after matching does not remove it. frame at t land water rock glint frame at t + 2 s same rock different wave Both correspondences pass the ratio test. One of them describes a ground point and the other does not. This is a correct match of an incorrect assumption, which is why post-hoc outlier filtering does not reliably remove it.

Figure 1 — The failure is not a bad match. It is a good match between two things that are not the same point in the world, and the descriptor has no way to know.

Minimal reproducible solution

The robust mask comes from properties water and sky have and land does not, combined so that no single cue has to carry the decision.

import cv2
import numpy as np


def water_sky_mask(bgr: np.ndarray, horizon_frac: float | None = None) -> np.ndarray:
    """Return a uint8 mask: 255 where detection is allowed, 0 where excluded.

    Three cues, combined by voting, so a single unusual condition — flat light,
    turbid water, a white roof — cannot decide the outcome alone.
    """
    hsv = cv2.cvtColor(bgr, cv2.COLOR_BGR2HSV)
    h, s, v = cv2.split(hsv)

    # 1. Saturation: water and sky are far less saturated than vegetation or
    #    bare earth, and this holds across most illumination.
    low_sat = s < 60

    # 2. Local texture energy: computed over a window large enough that wave
    #    texture averages out but a field boundary does not.
    gray = cv2.cvtColor(bgr, cv2.COLOR_BGR2GRAY).astype(np.float32)
    mean = cv2.blur(gray, (31, 31))
    energy = cv2.blur((gray - mean) ** 2, (31, 31))
    smooth = energy < np.percentile(energy, 25)

    # 3. Blue dominance: both water and sky sit toward the blue end.
    b, g_, r = cv2.split(bgr.astype(np.int16))
    bluish = (b - r) > 12

    votes = low_sat.astype(np.uint8) + smooth.astype(np.uint8) + bluish.astype(np.uint8)
    excluded = votes >= 2

    if horizon_frac is not None:          # oblique frame: sky is above the horizon
        cut = int(bgr.shape[0] * horizon_frac)
        excluded[:cut, :] = True

    mask = np.where(excluded, 0, 255).astype(np.uint8)
    # Close small holes so isolated boats or rocks do not fragment the mask.
    kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (25, 25))
    return cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)

The voting arrangement is what makes this survive a real dataset. A colour threshold alone fails on turbid brown water; a texture threshold alone excludes fresh asphalt; a blue-channel test alone excludes a blue roof. Requiring two of three keeps each cue’s failure isolated.

OpenCV detectors accept the mask directly, which is the whole point — excluding at detection rather than filtering afterwards means the descriptors are never computed and the matcher never sees them.

def detect_masked(gray: np.ndarray, mask: np.ndarray, detector) -> tuple:
    """Detect only where the mask permits. Excluded regions cost nothing."""
    return detector.detectAndCompute(gray, mask)

Edge-case matrix

Scene Cue that fails alone Voting result
Clear blue water none Excluded correctly
Turbid brown water blue dominance Excluded on saturation + texture
Fresh asphalt texture Kept — saturation and blue disagree
Blue metal roof blue dominance Kept — texture and saturation disagree
Water with strong glint texture Excluded on saturation + blue
Snow saturation, texture Excluded — usually correct, sometimes not
Shallow water over sand blue, saturation Kept — often correct, features are real
Overcast sky, oblique blue dominance Excluded on saturation + texture

The snow row is the honest limitation: snow votes like water on two of three cues and is genuinely excluded. On a winter survey that is frequently the right outcome and occasionally not, and the correct response is an explicit per-project override rather than weakening the cue set for everyone.

The shallow-water row is the opposite case: over sand in clear water the bottom really is visible and stationary, the features are real, and keeping them is correct — which the voting happens to get right because the seabed is textured and reasonably saturated.

Verification snippet

The check is that masking removed matches rather than merely pixels, and that the removed matches were the bad ones.

import numpy as np


def assert_mask_improves_geometry(unmasked_inliers: np.ndarray,
                                  masked_inliers: np.ndarray,
                                  water_polygon_test) -> None:
    """Masking should remove correspondences over water and keep the rest."""
    over_water_before = int(sum(water_polygon_test(p) for p in unmasked_inliers))
    over_water_after = int(sum(water_polygon_test(p) for p in masked_inliers))
    on_land_before = len(unmasked_inliers) - over_water_before
    on_land_after = len(masked_inliers) - over_water_after

    assert over_water_after < over_water_before * 0.1, (
        f"{over_water_after} of {over_water_before} water correspondences "
        "survived — the mask is not covering the water")
    assert on_land_after >= on_land_before * 0.95, (
        f"land correspondences fell from {on_land_before} to {on_land_after} — "
        "the mask is over-eager and is excluding usable ground")

Asserting both directions is what stops the mask being tuned into a blunt instrument. A mask that excludes half the frame will certainly remove every water correspondence, and it will also remove the shoreline detail that ties the land block together across the bay.

Two failure directions for a mask Three versions of the same coastal frame. In the first, no mask is applied and correspondences are found on both land and water, including a dense set over the water that is wrong. In the second, an over-eager mask excludes the water and a wide strip of the shoreline, removing the very correspondences that connect the two sides of the bay. In the third, a correctly sized mask follows the waterline and keeps the shoreline detail, removing the water correspondences and none of the land ones. A note states that both failure directions are quantifiable, so the mask can be tuned against numbers rather than appearance. no mask water correspondences kept over-eager mask shoreline detail lost too correct mask land kept, water removed Both directions are measurable, so the mask can be tuned against correspondence counts rather than against how it looks.

Figure 2 — The two ways to get a mask wrong. Only comparing counts on both sides of the waterline distinguishes an effective mask from an aggressive one.

When to escalate

  • The mask is correct and the reconstruction still places points under water. Some correspondences were formed before masking — check that the mask is passed to detectAndCompute and not applied afterwards, since a mask used only at match time still allows the descriptors to exist and be matched from the other frame’s side.
  • A coastal block splits at the waterline. Masking removed the water and the two land areas were only connected through it. This is a flight-planning issue rather than a masking one: the survey needs a strip that crosses the water at a point where both shores are visible in the same frame.
  • Snow-covered ground is being excluded. The voting cannot distinguish snow from water on two of its three cues. Add an explicit per-project override that disables the saturation cue for winter surveys, rather than lowering the vote threshold, which would weaken the mask everywhere.

One last operational note. Store the mask alongside the frame in the out-of-core feature store, rather than recomputing it when the matching stage runs. Masks are cheap to compute once and awkward to reproduce later — the cue thresholds may have moved, and a mask that differs between the detection run and a subsequent diagnostic run makes the diagnostic worthless. It also gives the export stage something to clip against, so the same waterline that excluded features from the reconstruction can exclude fabricated elevations from the delivered surface.

← Feature Detection Algorithms for Drone Imagery

Three cues, and what each one alone gets wrong A table of three mask cues against four surfaces. Low saturation excludes water and sky correctly but also excludes fresh asphalt. Low texture energy excludes water and asphalt but also excludes a smooth roof. Blue dominance excludes water and sky but also excludes a blue metal roof. Requiring two of the three votes excludes only water and sky, because each surface that a single cue mistakes is rescued by the other two disagreeing. A note observes that snow is the exception, voting like water on two cues. low saturation low texture blue dominant ≥ 2 votes water sky fresh asphalt blue metal roof snow yesyesyesexcluded yesyesyesexcluded yesyesnoexcluded noyesyesexcluded yesyesnoexcluded Asphalt and a blue roof are excluded here too — which is why the texture window must be large enough to keep road markings and roof edges.

Figure 3 — The cue table, including its honest failures. Two of the surfaces below the fold are excluded by the voting as written, which is why the window sizes matter as much as the vote count.