Porting an OpenCV Matcher into an OpenSfM Pipeline

A prototype in OpenCV matches a difficult dataset — a repetitive orchard, a low-texture quarry — noticeably better than the reconstruction engine’s own matcher. The obvious next step is to use it for the real job, and the obvious obstacle is that the engine expects features and matches in its own files, with its own conventions.

The port is not difficult and it has three traps that produce a reconstruction which runs, converges and is wrong: a coordinate convention that differs from OpenCV’s, an index convention in the match records, and a feature ordering that the engine relies on. This page covers the conversion and the tests that catch each trap. It extends the comparison in OpenCV vs OpenSfM for feature matching.

The three conventions that differ

Coordinate normalisation. OpenCV keypoints are in pixels with the origin at the top-left corner. OpenSfM stores features in a normalised frame centred on the image, scaled by the larger dimension, so a pixel coordinate must be converted rather than written directly. Writing pixels into a normalised field produces features that all sit far outside the image, and the engine quietly finds no matches.

Index basis. Match records reference features by index into each image’s feature array. If the arrays are written in a different order from the one the matcher used, every match points at the wrong feature — and because the indices are all valid, nothing errors.

Feature ordering. Some pipelines assume features are sorted by scale or response. An arbitrary order is usually accepted and occasionally interacts with downsampling steps that keep “the first N”.

Pixel coordinates against the normalised convention Two coordinate frames over the same image. In the OpenCV pixel frame the origin is the top-left corner, x increases right and y increases down, and a point at the image centre has coordinates two thousand by fifteen hundred. In the normalised frame the origin is the image centre and both axes are divided by the larger image dimension, so the same point is at zero, zero and the image corners are at plus or minus zero point five in the long axis. A note records that writing pixel values into the normalised field puts every feature far outside the image and yields silently zero matches. OpenCV pixels normalised +x +y (2000, 1500) origin at the top-left corner +x −y (0, 0) origin at the centre, scaled by the long axis Pixels written into the normalised field land hundreds of image-widths away. The engine finds no matches and reports no error, which is the worst combination.

Figure 1 — The conversion, and what skipping it looks like.

Minimal reproducible solution

import numpy as np


def to_normalised(keypoints_xy: np.ndarray, width: int, height: int) -> np.ndarray:
    """Convert OpenCV pixel coordinates to the centred, scaled convention.

    The scale factor is the larger image dimension for both axes, which
    preserves the aspect ratio — dividing each axis by its own dimension
    would distort the features and produce a subtly wrong reconstruction
    rather than an obviously broken one.
    """
    size = float(max(width, height))
    centred = keypoints_xy.astype(np.float64) - np.array([width / 2.0, height / 2.0])
    return centred / size


def to_pixels(normalised_xy: np.ndarray, width: int, height: int) -> np.ndarray:
    """The inverse, for verification."""
    size = float(max(width, height))
    return normalised_xy * size + np.array([width / 2.0, height / 2.0])

Using the larger dimension for both axes rather than each axis’s own is the subtle half of this. Both conventions are self-consistent; only one matches what the engine expects, and the wrong one produces a reconstruction that converges to a slightly stretched solution — which will pass every internal check and fail against ground control.

Writing the feature and match files

import gzip
import pickle
from pathlib import Path

import numpy as np


def write_features(path: str, keypoints, descriptors, colors,
                   width: int, height: int) -> dict:
    """Write features in the engine's expected structure.

    Order is fixed here and must not change afterwards, because the match
    records reference these rows by index. Any later sort invalidates every
    match file that was written against this one.
    """
    xy = np.array([kp.pt for kp in keypoints], dtype=np.float64)
    scale = np.array([kp.size / max(width, height) for kp in keypoints], dtype=np.float64)
    angle = np.array([np.radians(kp.angle) for kp in keypoints], dtype=np.float64)

    points = np.column_stack([to_normalised(xy, width, height), scale, angle])
    payload = {"points": points.astype(np.float32),
               "descriptors": np.asarray(descriptors, dtype=np.float32),
               "colors": np.asarray(colors, dtype=np.uint8)}

    Path(path).parent.mkdir(parents=True, exist_ok=True)
    with gzip.open(path, "wb") as fh:
        pickle.dump(payload, fh)
    return {"features": len(points), "path": path}


def write_matches(path: str, matches_by_image: dict[str, np.ndarray]) -> dict:
    """Write match index pairs for one image against its candidates."""
    payload = {name: np.asarray(pairs, dtype=np.int32)
               for name, pairs in matches_by_image.items()}
    Path(path).parent.mkdir(parents=True, exist_ok=True)
    with gzip.open(path, "wb") as fh:
        pickle.dump(payload, fh)
    return {"images_matched": len(payload),
            "total_matches": int(sum(len(v) for v in payload.values()))}

Verifying the port before trusting it

Three tests catch all three traps, and they run in seconds on a handful of images.

import numpy as np


def verify_coordinates(normalised: np.ndarray) -> dict:
    """Normalised features must lie within about half a unit of the origin."""
    radius = np.linalg.norm(normalised[:, :2], axis=1)
    outside = float((radius > 0.75).mean())
    return {"max_radius": float(radius.max()), "fraction_outside": outside,
            "ok": outside < 0.001,
            "note": ("coordinates look normalised" if outside < 0.001 else
                     "features lie far outside the image — pixels were written "
                     "into a normalised field")}


def verify_match_indices(matches: np.ndarray, n_features_a: int,
                         n_features_b: int) -> dict:
    """Match indices must reference existing features in both images."""
    if len(matches) == 0:
        return {"ok": True, "note": "no matches to check"}
    bad_a = int((matches[:, 0] >= n_features_a).sum())
    bad_b = int((matches[:, 1] >= n_features_b).sum())
    duplicates = len(matches) - len({tuple(m) for m in matches.tolist()})
    return {"out_of_range_a": bad_a, "out_of_range_b": bad_b,
            "duplicates": duplicates,
            "ok": bad_a == 0 and bad_b == 0 and duplicates == 0}


def verify_round_trip(keypoints_xy: np.ndarray, width: int, height: int,
                      *, tol_px: float = 1e-6) -> dict:
    """Normalising and un-normalising must return the original pixels."""
    back = to_pixels(to_normalised(keypoints_xy, width, height), width, height)
    error = float(np.abs(back - keypoints_xy).max())
    return {"max_error_px": error, "ok": error < tol_px}

The index check is the one that catches the most damaging trap. Out-of-range indices produce an error the engine reports; in-range but wrong indices do not, and the round-trip test on coordinates plus a visual check of a few matched pairs is what exposes them.

What has to be reconciled when an OpenCV matcher meets a pipeline Four rows. Coordinate conventions differ: an image-pixel origin and a normalised camera coordinate are both common, and a mismatch produces matches that look correct and reconstruct into nonsense. Feature identity must be stable, because the pipeline refers to features by index across stages and a matcher that re-detects per pair breaks those references. The match file format must carry what the pipeline expects, including any confidence or inlier flag it uses downstream. And the camera model must agree, since a matcher assuming a pinhole camera and a pipeline applying distortion will disagree about where a point is by exactly the distortion. coordinate conventions pixel origin versus normalised camera — plausible matches, nonsense model stable feature identity the pipeline refers to features by index across stages match file format including any confidence or inlier flag used downstream the camera model a pinhole matcher and a distorting pipeline disagree by the distortion Three of the four fail silently. Only the last produces an error anybody notices.

Figure 3 — Four contracts, mostly implicit.

Edge-case matrix

Situation Symptom Handling
Pixels in a normalised field Zero matches, no error Convert; check the radius
Per-axis normalisation Slightly stretched reconstruction Scale both axes by the larger dimension
Features re-sorted after writing Matches point at wrong features Fix the order before writing anything
Descriptor dtype mismatch Engine errors or silently misreads Write float32 for float descriptors
Binary descriptors May not be supported Check before porting an ORB prototype
Angle in degrees Orientation ignored or wrong Convert to radians
Scale in pixels Feature scales misinterpreted Normalise by the same factor
Colors absent Point cloud renders grey Sample the image at each keypoint

The angle and scale rows are easy to miss because nothing fails: the reconstruction runs with orientation information that is wrong by a factor of 57, and the effect is a modest loss of matching quality that looks like the imagery being difficult.

Deciding whether the port is worth it

Before porting, it is worth confirming that the prototype’s advantage survives contact with the full pipeline. A matcher that finds more raw matches does not necessarily produce a better reconstruction — the engine’s own matcher may be discarding matches its geometric verification would reject anyway.

def compare_reconstructions(a: dict, b: dict) -> dict:
    """Compare two pipelines on what matters, not on match counts.

    More matches is not the goal. More images reconstructed, more tie points
    surviving the adjustment and better checkpoint accuracy are.
    """
    return {
        "images_reconstructed": (a["images"], b["images"]),
        "tie_points": (a["tie_points"], b["tie_points"]),
        "mean_track_length": (a["mean_track_length"], b["mean_track_length"]),
        "checkpoint_rmse_m": (a["checkpoint_rmse_m"], b["checkpoint_rmse_m"]),
        "better": ("custom" if (b["images"] >= a["images"]
                                and b["checkpoint_rmse_m"] <= a["checkpoint_rmse_m"])
                   else "stock"),
        "note": "raw match count is not in this comparison, deliberately",
    }

Mean track length is the quiet indicator worth watching. A matcher producing many pairwise matches that do not chain into long tracks is finding correspondences that the adjustment cannot use, and the reconstruction will be no stronger for them.

Match count against reconstruction quality for two matchers Four measures compared between a stock matcher and a ported custom one on a repetitive orchard dataset. The custom matcher finds two point one times as many raw matches. It reconstructs ninety-eight percent of images against eighty-one percent. Its mean track length is four point six against three point one. Its checkpoint accuracy is three point one centimetres against five point eight. A note states that the raw match count is the least informative of the four and is the one usually quoted. raw matches — the least informative measure 2.1× more images reconstructed 81 % 98 % checkpoint accuracy — lower is better 5.8 cm 3.1 cm The port is justified here — but by the bottom two rows, not the top one.

Figure 2 — The comparison that decides whether a port is worth maintaining.

When to escalate

  • The port produces zero matches. Check the coordinate convention first; it accounts for most instances and produces exactly this symptom.
  • The reconstruction is stretched. Per-axis normalisation. Scale both axes by the larger dimension.
  • The custom matcher wins on matches and loses on checkpoints. It is finding correspondences the adjustment cannot use. Compare mean track length before investigating further.

← OpenCV vs OpenSfM for Feature Matching