Validating Classification Against Manual Samples

Classification results are almost always accepted on sight. Somebody colours the cloud by class, rotates it, decides the buildings look like buildings, and the run proceeds. That check is real but narrow: it detects gross failures — everything in one class, a roof shaped like a hill — and is blind to the two errors that actually reach a client. A uniform bias in the ground class produces a terrain model that is smooth, plausible and wrong. A modest confusion rate between vegetation and buildings produces footprints that are fine everywhere the reviewer happened to look.

This page covers the sampling design and the statistics that turn “it looks right” into a number, and it is the check referenced throughout classifying point clouds with PDAL and Python.

Why sighted inspection fails on exactly the errors that matter

A rendering shows the shape of a surface. Shape is a function of the spread of errors around their own mean — the roughness — and vegetation misclassification barely changes that. A ground class that consistently sits on top of 25 cm of grass is as smooth as one sitting on the soil; it is simply 25 cm higher. There is no visual cue whatsoever, because there is nothing in the frame to compare against.

The same logic applies to class confusion. If four percent of tree points are labelled building, they form scattered fragments rather than a visible block, and a reviewer rotating the cloud sees exactly what they expect. The error surfaces later, when somebody dissolves the building class into polygons and gets three hundred spurious footprints.

Both errors are detectable with very few samples, because both are mean effects rather than tail effects, and means converge quickly.

How the uncertainty in a measured bias shrinks with sample count A curve showing the ninety-five percent confidence half-width on an estimated ground bias against the number of validation samples, for a per-sample scatter of eight centimetres. At ten samples the half-width is about five centimetres, at thirty it is about three, at sixty about two, and at two hundred about one. A marked band shows the region below three centimetres where the estimate is tight enough to act on. A note states that thirty well-placed samples are enough to detect the biases that matter, which is roughly an hour of work. tight enough to act on 5 15 30 60 120 200 validation samples bias uncertainty 30 samples ≈ ±3 cm Diminishing returns set in early: the hundredth sample buys a tenth of what the tenth bought.

Figure 1 — The economics of validation. Thirty samples resolve the errors that matter; nobody needs three hundred.

Minimal reproducible solution

The sample set must be placed by a rule rather than by hand, because hand-placed samples land where the technician expects to find problems, and that is a biased estimator of the site. Stratify by class and by an independent covariate — slope works well — then draw at random inside each stratum.

import json
import numpy as np
import pdal


def draw_validation_samples(las_path: str, per_stratum: int = 8,
                            seed: int = 0) -> list[dict]:
    """Stratified random sample locations for manual classification checking.

    Strata are (class, slope band). Hand-picked samples cluster where the
    technician suspects trouble, which measures the technician rather than the
    classification; a seeded random draw inside each stratum does not.
    """
    pipe = pdal.Pipeline(json.dumps({"pipeline": [
        las_path,
        {"type": "filters.hag_nn"},
        {"type": "filters.covariancefeatures", "knn": 24,
         "feature_set": "Dimensionality"},
    ]}))
    pipe.execute()
    a = pipe.arrays[0]

    rng = np.random.default_rng(seed)
    slope_band = np.digitize(1.0 - a["Planarity"], [0.05, 0.2, 0.5])
    samples = []
    for cls in np.unique(a["Classification"]):
        for band in range(4):
            idx = np.flatnonzero((a["Classification"] == cls) & (slope_band == band))
            if idx.size == 0:
                continue
            take = rng.choice(idx, size=min(per_stratum, idx.size), replace=False)
            for i in take:
                samples.append({
                    "x": float(a["X"][i]), "y": float(a["Y"][i]),
                    "z": float(a["Z"][i]),
                    "assigned_class": int(a["Classification"][i]),
                    "slope_band": int(band),
                    "true_class": None,       # filled in by the reviewer
                })
    return samples

Leaving true_class as None and having a reviewer fill it in from the imagery — without seeing assigned_class — is what keeps the exercise honest. A reviewer shown the machine’s answer agrees with it far more often than one who is not.

From the completed sheet, two numbers come out: a confusion matrix for the class labels, and a height bias for the ground class.

import numpy as np


def confusion_and_bias(samples: list[dict],
                       measured_ground_z: dict[int, float]) -> dict:
    """Confusion matrix over all classes, plus ground-class height bias.

    `measured_ground_z` maps a sample index to a surveyed ground height where
    one exists — checkpoints, or a total-station shot. Absent that, the bias
    term is skipped and only the label agreement is reported.
    """
    labels = sorted({s["assigned_class"] for s in samples}
                    | {s["true_class"] for s in samples if s["true_class"]})
    index = {c: i for i, c in enumerate(labels)}
    m = np.zeros((len(labels), len(labels)), dtype=int)
    for s in samples:
        if s["true_class"] is None:
            continue
        m[index[s["true_class"]], index[s["assigned_class"]]] += 1

    per_class = {}
    for c, i in index.items():
        tp = m[i, i]
        per_class[int(c)] = {
            "recall": float(tp / m[i].sum()) if m[i].sum() else float("nan"),
            "precision": float(tp / m[:, i].sum()) if m[:, i].sum() else float("nan"),
        }

    resid = [samples[i]["z"] - z for i, z in measured_ground_z.items()
             if samples[i]["assigned_class"] == 2]
    r = np.asarray(resid, dtype=float)
    return {
        "labels": [int(c) for c in labels],
        "matrix": m.tolist(),
        "per_class": per_class,
        "overall_accuracy": float(np.trace(m) / m.sum()) if m.sum() else float("nan"),
        "ground_bias_m": float(np.mean(r)) if r.size else None,
        "ground_rmse_m": float(np.sqrt(np.mean(r ** 2))) if r.size else None,
        "ground_n": int(r.size),
    }

Edge-case matrix

Sampling variant What it measures Verdict
Hand-picked “interesting” points The reviewer’s suspicions Not a validation
Uniform random over the whole cloud Mostly ground, since ground dominates Rare classes never sampled
Stratified by class Every class, including rare ones Correct; weight results by class share
Stratified by class and slope Class and terrain interaction Best; catches slope-dependent failures
Reviewer sees the assigned class Agreement, not accuracy Blind the reviewer
Samples drawn from one flight line That line’s geometry Spread across the survey
Fewer than ~10 per class Nothing usable Increase or drop the class from the claim
Checkpoints reused as samples Ground bias only Fine for bias; not for label accuracy

The last row is worth taking advantage of. Any project doing checkpoint-based accuracy validation already has surveyed ground points, and they are exactly the truth data the ground-bias term needs — at no extra field cost.

A worked confusion matrix and the two claims it supports A four-by-four confusion matrix of true class against assigned class for ground, low vegetation, high vegetation and building. The diagonal holds the large counts: thirty ground, eighteen low vegetation, twenty-six high vegetation and twenty-two building. Off-diagonal cells show six true low vegetation points assigned to ground, four true high vegetation points assigned to building, and two true building points assigned to high vegetation. Annotations read off ground recall of eighty-three percent and building precision of eighty-five percent, and note that the six low vegetation points assigned to ground are the source of a positive height bias. assigned class → true class ↓ ground low veg high veg building ground low veg high veg building 30000 61800 00264 00222 ground recall 83 % · building precision 85 % · overall 88 % The six low-vegetation points called ground are the height bias, in a single cell.

Figure 2 — One matrix, two findings. The off-diagonal cell in the second row is both a label error and the cause of a systematic elevation offset.

Verification snippet

The validation itself needs a gate, or it becomes a number somebody records and nobody reads.

def gate_classification(report: dict, *, min_ground_recall: float = 0.90,
                        max_ground_bias_m: float = 0.08,
                        min_overall: float = 0.85) -> None:
    """Fail the run on a classification that does not meet the stated standard."""
    g = report["per_class"].get(2, {})
    if g.get("recall", 0) < min_ground_recall:
        raise ValueError(
            f"ground recall {g.get('recall'):.2f} below {min_ground_recall} — "
            "terrain products would inherit the gaps")
    bias = report.get("ground_bias_m")
    if bias is not None and abs(bias) > max_ground_bias_m:
        raise ValueError(
            f"ground bias {bias:+.3f} m exceeds {max_ground_bias_m} m — "
            "every elevation in the deliverable is offset by this amount")
    if report["overall_accuracy"] < min_overall:
        raise ValueError(f"overall accuracy {report['overall_accuracy']:.2f} below standard")

Stating the bias tolerance in metres rather than as a percentage is deliberate: a client’s tolerance is a distance, and converting it into a classification metric is the pipeline’s job, not theirs.

Where the validation gate sits in the run A linear sequence with a gate. Classification produces a classified cloud, which passes into a stratified sample draw and a blind review. The review result enters a gate that tests ground recall, ground bias and overall accuracy. A pass releases the cloud to volume, change detection and surface products. A failure returns to parameter tuning, and the loop is recorded in the run manifest so the number of attempts is visible. A note states that without the gate the validation becomes a number nobody reads. classified cloud stratified draw seeded, per class blind review labels hidden gate recall · bias · overall retune parameters attempt count recorded release to products The loop is recorded, so "we tried eleven parameter sets" is visible rather than folklore. A validation without a gate is a number in a log file.

Figure 3 — Validation as a stage with an exit condition, not a report appended after the fact.

Keeping the validation cheap enough to actually run

A check that takes a day is a check that gets skipped under deadline, and a skipped check is worse than no check because the pipeline still claims to have one. Three practices keep the cost low enough that it happens on every job.

Reuse the review effort across runs. The sample locations are coordinates, and the true labels attached to them do not change when a parameter does. Store the reviewed sheet keyed by coordinate, and a re-run after a parameter change is a re-scoring rather than a re-review — seconds instead of an hour. Only new strata, or a materially different flight, need fresh labelling.

import json
from pathlib import Path


def rescore_against_cached_truth(samples: list[dict], truth_cache: str,
                                 tol: float = 0.05) -> list[dict]:
    """Attach previously reviewed labels to a freshly classified cloud.

    Matching on rounded coordinates is enough: the samples are drawn from the
    same survey, so a point either exists at that location or the sample is
    new and needs a reviewer.
    """
    cache = {tuple(k.split(",")): v
             for k, v in json.loads(Path(truth_cache).read_text()).items()}
    out = []
    for s in samples:
        key = (f"{s['x']:.2f}", f"{s['y']:.2f}")
        s = dict(s, true_class=cache.get(key))
        out.append(s)
    return out

Review in the imagery, not in the cloud. Deciding whether a point is building or vegetation is far faster looking at the orthomosaic at that coordinate than rotating a three-dimensional view. A reviewer can label sixty points in twenty minutes from a tiled image viewer; the same work in a point-cloud viewer takes an hour and produces worse labels, because occlusion makes the three-dimensional view genuinely harder to read.

Stop at the precision you need. The confidence interval on the bias estimate narrows with the square root of the sample count, so the marginal value of sample sixty-one is small. Fixing the count at thirty per class, and only increasing it when a result lands near a gate threshold, keeps the effort proportional to the decision being made.

A fourth practice is organisational rather than technical: record the validation result in the same manifest as the classification parameters, so a run’s claimed accuracy travels with the file rather than living in a spreadsheet that diverges from it. The same manifest discipline is described for job orchestration in writing a manifest-driven batch runner for ODM.

When to escalate

  • Ground recall is high and the bias persists. The classifier is finding ground correctly and the ground itself is offset — that is a vertical datum problem, handled in geoid models and vertical datum automation, not a classification one.
  • Accuracy varies strongly between strata. A classification that works on flat ground and fails on slopes needs terrain-adaptive parameters, not a different threshold.
  • No surveyed ground exists anywhere on site. Label accuracy can still be measured; height bias cannot. Say so explicitly in the deliverable rather than implying an accuracy that was never tested.

← Classifying Point Clouds with PDAL and Python