Splitting Control and Checkpoints Without Bias

Twelve control points, four held out at random as checkpoints, and all four landed in the western half of the site. The adjustment is now unconstrained in the west and the accuracy measurement says nothing about the east. The split was random, which felt fair, and it produced a result that is biased in both directions at once.

Random splitting works when the sample is large. Survey control sets are small, and over a dozen points a clustered draw is not unlikely — it is the most common single outcome. This page covers splitting deliberately, guaranteeing coverage on both sides, and the alternative when there are too few points to split at all. It supports checkpoint-based accuracy validation in Python.

What a bad split costs

Two failures follow from clustering, and they are not symmetric.

Clustered checkpoints measure accuracy where they are and say nothing elsewhere. The survey’s accuracy statement is then a statement about one part of a site, which is misleading rather than merely incomplete.

Clustered control leaves part of the survey unconstrained, and the deformation modes that constraint would have suppressed — doming, tilt — reappear there. The checkpoints then correctly report poor accuracy in that area, and the cause is the split rather than the survey.

Both are avoidable by making coverage a requirement of the split rather than an outcome of it.

A clustered random split against a spatially stratified one Two plan views of the same twelve control points over a site. In the random split, all four checkpoints landed in the western half, leaving the west unconstrained and the east unmeasured. In the stratified split, the site is divided into four cells and one checkpoint is drawn from each, so both sets span the site and every cell retains at least two control points. A note records that over a dozen points a clustered draw is the most common single outcome of a random split. random split stratified split unconstrained unmeasured both sets span the site; every cell keeps control Over a dozen points, the clustered draw is the most common single outcome. Which is why coverage should be a requirement of the split, not an outcome of it.

Figure 1 — The same points, two splits, and two different surveys.

Minimal reproducible solution

import numpy as np


def stratified_split(positions_xy: np.ndarray, *, checkpoint_fraction: float = 0.3,
                     min_control_per_cell: int = 1, seed: int = 0) -> dict:
    """Split so both sets span the site and no cell loses its control.

    Cells are formed from quantiles rather than a regular grid, so an
    irregular site or an uneven point distribution still yields cells with
    comparable populations.
    """
    xy = np.asarray(positions_xy, dtype=float)
    rng = np.random.default_rng(seed)
    n = len(xy)
    if n < 6:
        return {"note": "too few points to split; use cross-validation instead"}

    cells = max(int(np.floor(np.sqrt(n / 3))), 1)
    x_edges = np.quantile(xy[:, 0], np.linspace(0, 1, cells + 1))[1:-1]
    y_edges = np.quantile(xy[:, 1], np.linspace(0, 1, cells + 1))[1:-1]
    key = (np.digitize(xy[:, 0], x_edges) * cells
           + np.digitize(xy[:, 1], y_edges))

    control, checkpoints = [], []
    for cell in np.unique(key):
        members = np.flatnonzero(key == cell)
        rng.shuffle(members)
        take = int(round(len(members) * checkpoint_fraction))
        take = min(take, max(len(members) - min_control_per_cell, 0))
        checkpoints.extend(members[:take].tolist())
        control.extend(members[take:].tolist())

    return {"control": sorted(control), "checkpoints": sorted(checkpoints),
            "cells": int(len(np.unique(key))), "seed": seed,
            "control_count": len(control), "checkpoint_count": len(checkpoints)}

Recording the seed is what makes a split reproducible, and reproducibility matters here because the accuracy statement depends on it. A report quoting an accuracy from an unrecorded split cannot be re-derived, which is exactly the property the checkpoint exercise was meant to provide.

Measuring the quality of a split

import numpy as np
from scipy.spatial import ConvexHull


def split_quality(positions_xy: np.ndarray, control: list[int],
                  checkpoints: list[int]) -> dict:
    """Do both sets cover the site, and does control surround the checkpoints?

    The second question matters: a checkpoint outside the convex hull of the
    control is measuring an extrapolation rather than the survey, and its
    residual will be worse for reasons that are not the survey's fault.
    """
    xy = np.asarray(positions_xy, dtype=float)
    site_hull = ConvexHull(xy)
    site_area = float(site_hull.volume)

    def coverage(indices):
        pts = xy[indices]
        if len(pts) < 3:
            return 0.0
        return float(ConvexHull(pts).volume / max(site_area, 1e-9))

    control_hull = ConvexHull(xy[control]) if len(control) >= 3 else None
    extrapolated = 0
    if control_hull is not None:
        from matplotlib.path import Path
        path = Path(xy[control][control_hull.vertices])
        extrapolated = int((~path.contains_points(xy[checkpoints])).sum())

    return {"control_coverage": coverage(control),
            "checkpoint_coverage": coverage(checkpoints),
            "checkpoints_outside_control_hull": extrapolated,
            "balanced": coverage(control) > 0.7 and coverage(checkpoints) > 0.4,
            "note": ("both sets span the site" if coverage(checkpoints) > 0.4 else
                     "checkpoints do not span the site — the accuracy statement "
                     "applies only where they are")}
Three splits that look reasonable and are not Three rows. Splitting by whichever points fitted worst, and promoting them to control, guarantees that the remaining checkpoints are the ones the adjustment was always going to fit well, producing an accuracy figure that is a selection effect. Splitting geographically, with all control on one half of the site, leaves the checkpoints extrapolating beyond the control, which measures extrapolation rather than the survey. Splitting after the adjustment has already seen every point measures nothing at all, because no point is independent of the fit any longer. by which fitted worst the remaining checkpoints were always going to fit well geographically, one half each the checkpoints extrapolate — that is what gets measured after the adjustment ran no point is independent of the fit any longer All three produce a defensible-sounding number and none of them is a test.

Figure 3 — Three splits, none of which measures the survey.

Edge-case matrix

Situation Effect Handling
Random split, few points Clustered sets Stratify spatially
Checkpoints outside the control hull Measuring extrapolation Keep control on the perimeter
Corridor site Cells degenerate to a line Stratify along the corridor only
One point in a large cell Cell loses control or checkpoint Merge sparse cells
Fewer than six points Split not viable Cross-validate instead
Unrecorded seed Accuracy not reproducible Record it in the manifest
Points at very different accuracies Weak points weight the statistic Split by quality as well as position
Control required at specific points Split not free Constrain those to the control set

The corridor row deserves a note. On a linear site the two-dimensional stratification produces degenerate cells, and the right adaptation is to stratify along the corridor’s length only — which preserves the property that matters, that control and checkpoints alternate along its length rather than clustering at one end.

When points are too few to split

Below about eight points, holding any out weakens the adjustment more than the measurement is worth. Leave-one-out cross-validation gives an accuracy estimate without a permanent split, at the cost of one adjustment run per point.

import numpy as np


def leave_one_out(points: list[dict], run_adjustment) -> dict:
    """Estimate accuracy by holding out each point in turn.

    `run_adjustment` takes the control subset and returns the predicted
    position of the held-out point. Every point contributes to a residual
    while every adjustment keeps all but one constraint, which is the best
    available trade when points are scarce.
    """
    residuals = []
    for i in range(len(points)):
        control = [p for j, p in enumerate(points) if j != i]
        predicted = run_adjustment(control, points[i])
        actual = np.array([points[i]["e"], points[i]["n"], points[i]["h"]])
        residuals.append(predicted - actual)

    r = np.asarray(residuals)
    return {"points": len(points),
            "horizontal_rmse_m": float(np.sqrt(np.mean(
                np.linalg.norm(r[:, :2], axis=1) ** 2))),
            "vertical_rmse_m": float(np.sqrt(np.mean(r[:, 2] ** 2))),
            "note": "leave-one-out; each estimate used all but one constraint"}

Cross-validation slightly overestimates the error of the final product, because each fold had one fewer constraint than the delivered adjustment. That is the conservative direction and is the right one for an accuracy claim.

Choosing between a split and cross-validation by point count A scale of available control points from five to forty, divided into three regions. Below eight points, a split leaves too little control and leave-one-out cross-validation is preferred despite costing one adjustment run per point. Between eight and fifteen, a stratified split with a smaller checkpoint fraction of about a quarter is workable. Above fifteen, a standard stratified split at thirty percent is straightforward and cross-validation is unnecessary. A note records that cross-validation slightly overestimates the delivered error, which is the conservative direction. 5 – 8 points leave-one-out one run per point 8 – 15 points stratified split, 25 % smaller checkpoint share 15+ points stratified split, 30 % straightforward available control points Cross-validation slightly overestimates the delivered error. Which is the conservative direction and the right one for a claim.

Figure 2 — Which method suits which point count.

Verification snippet

import numpy as np


def split_stability(positions_xy: np.ndarray, residual_fn,
                    *, trials: int = 20) -> dict:
    """How much does the accuracy estimate depend on which split was drawn?

    A large spread across seeds means the estimate is a property of the split
    rather than of the survey, which is a signal that there are too few
    points rather than that the survey is variable.
    """
    estimates = []
    for seed in range(trials):
        split = stratified_split(positions_xy, seed=seed)
        if "note" in split:
            continue
        estimates.append(residual_fn(split["control"], split["checkpoints"]))

    e = np.asarray(estimates, dtype=float)
    if e.size < 5:
        return {"note": "not enough successful splits to assess stability"}
    return {"trials": int(e.size), "mean_m": float(e.mean()),
            "spread_m": float(e.max() - e.min()),
            "relative_spread": float((e.max() - e.min()) / max(e.mean(), 1e-9)),
            "stable": float((e.max() - e.min()) / max(e.mean(), 1e-9)) < 0.5}

Planning the field work around the split

The split is easier when the field campaign anticipated it. Two habits at planning time remove most of the difficulty.

Collect more points than the adjustment strictly needs. A survey that needs five control points and collects twelve can hold out four comfortably; one that collects exactly five cannot hold out any. The marginal cost of an extra point is minutes of field time, and it is the cheapest accuracy assurance available.

Place them with the split in mind: perimeter points that will stay as control, and interior points that can go either way. That arrangement guarantees the control hull contains the checkpoints without constraining which interior points end up where, and it satisfies the interior-control requirement that suppresses doming at the same time.

A field sheet that marks each point as “must be control”, “prefer checkpoint” or “either” costs nothing to produce and makes the processing decision mechanical.

When to escalate

  • The estimate varies by more than half across seeds. There are too few points for a stable measurement. Collect more, or cross-validate and say so.
  • Specific points must be control for contractual reasons. Constrain them and stratify the rest; the split is less balanced and still better than random.
  • Checkpoints consistently fall outside the control hull. The control is not on the perimeter. That is a field-planning correction rather than a splitting one.

← Checkpoint-Based Accuracy Validation in Python