Benchmarking PDAL Pipelines Against laspy Loops

Somebody times a read in both libraries, gets 4.1 seconds against 11.8, and the pipeline architecture is decided. A week later the same test on a different machine gives the opposite ordering, and nobody can explain it.

Point-cloud benchmarks are unusually easy to get wrong, because almost all of the time in a realistic operation goes to decompression and file I/O rather than to the library’s own logic — and both of those are dominated by effects that have nothing to do with the code being compared. This page covers measuring honestly, and the three regimes where the answer genuinely differs, extending the comparison in PDAL vs laspy for point cloud automation.

Four ways a point-cloud benchmark lies

The page cache. The first read of a 4 GB LAZ pulls it from disk; the second reads it from RAM. Whichever library runs second wins by a factor that has nothing to do with either. Every measurement must either run warm — after a discarded warm-up pass — or drop caches between runs, and the two choices measure different things.

Unrepresentative files. A benchmark on a ten-million-point tile says little about behaviour on a four-hundred-million-point survey, because the regime changes: at the small size everything fits in memory and at the large size it does not, and the library that wins is the one that never had to swap.

Different work. laspy.read() decompresses every dimension in the file. A PDAL pipeline may only materialise the dimensions its stages need. Comparing them directly compares different amounts of work, and the fix is to name the dimensions explicitly on both sides.

Peak memory ignored. A run that completes in 40 seconds using 30 GB is not faster than one taking 90 seconds in 3 GB, if the production machine has 16 GB. Wall time alone is the wrong metric for this comparison.

The same comparison measured four ways, giving four orderings Four paired bars comparing two libraries under different measurement conditions. Cold cache shows both slow and nearly equal. Warm cache shows both much faster with a clear ordering. Full dimensions versus named dimensions reverses the ordering, because one library was materialising less data. Including peak memory reverses it again on a sixteen gigabyte machine, where the faster run would not have completed at all. A note states that none of the four measurements is wrong, and that each answers a different question. cold cache warm cache named dims peak memory OOM AB AB AB AB Four measurements, three different winners, no mistakes. Each condition answers a different question; the benchmark must state which one it asked.

Figure 1 — Why point-cloud benchmarks are so often contradicted by the next person to run one.

Minimal reproducible solution

import gc
import json
import time
import tracemalloc
from statistics import median


def bench(fn, *, repeats: int = 5, warmup: int = 1) -> dict:
    """Time and peak-memory a callable, discarding warm-up runs.

    tracemalloc measures Python allocations only, which is the honest number
    for laspy and an undercount for PDAL, whose work happens in C++. Peak RSS
    from the OS is the fair comparison; tracemalloc is a useful supplement.
    """
    for _ in range(warmup):
        fn()
    gc.collect()

    times = []
    tracemalloc.start()
    for _ in range(repeats):
        gc.collect()
        start = time.perf_counter()
        fn()
        times.append(time.perf_counter() - start)
    _, py_peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()

    return {"median_s": median(times), "min_s": min(times),
            "spread_s": max(times) - min(times),
            "python_peak_mb": py_peak / 1e6}


def peak_rss_mb() -> float:
    """Process peak resident set size, which includes the C++ side."""
    import resource
    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.0

Reporting the spread alongside the median is what makes a benchmark trustworthy. A spread larger than the difference between the two candidates means the measurement cannot distinguish them, and the correct conclusion is “no measurable difference” rather than whichever number came out lower.

Making the two sides do the same work

import json

import laspy
import numpy as np
import pdal

DIMS = ["X", "Y", "Z", "Classification"]


def laspy_read(path: str) -> np.ndarray:
    """Read only the dimensions under test."""
    las = laspy.read(path)
    return np.column_stack([las.x, las.y, las.z,
                            las.classification]).astype(np.float64)


def pdal_read(path: str) -> np.ndarray:
    """Read the same dimensions through a minimal pipeline."""
    pipe = pdal.Pipeline(json.dumps({"pipeline": [path]}))
    pipe.execute()
    a = pipe.arrays[0]
    return np.column_stack([a[d] for d in DIMS]).astype(np.float64)


def assert_equivalent(path: str) -> None:
    """A benchmark of two functions that return different things is meaningless."""
    a, b = laspy_read(path), pdal_read(path)
    assert a.shape == b.shape, f"different point counts: {a.shape} vs {b.shape}"
    order_a = np.lexsort((a[:, 2], a[:, 1], a[:, 0]))
    order_b = np.lexsort((b[:, 2], b[:, 1], b[:, 0]))
    assert np.allclose(a[order_a], b[order_b], atol=1e-6), "results differ"

The equivalence assertion is not optional. A benchmark where one side quietly returns fewer dimensions, or drops withheld points, is measuring two different operations — and this is the single most common flaw in published comparisons.

What an honest point cloud benchmark has to control Four controls are listed. Page cache state must be equalised, because a second run of the same file reads from memory and appears several times faster than the first for reasons unrelated to the library. Output must be verified identical, since a pipeline that skips work is not faster, it is doing less. Peak memory must be recorded alongside wall time, because the streaming and in-memory approaches trade exactly these against each other. The dataset must be representative in both size and density, as a result from a two-million-point sample rarely survives contact with a four-hundred-million-point survey. page cache state a second read comes from memory and looks several times faster identical output a pipeline that skips work is not faster, it is doing less peak memory alongside time streaming and in-memory trade exactly these against each other a representative dataset a two-million-point sample rarely survives a real survey Uncontrolled, any of the four produces a result that reverses on the real workload.

Figure 3 — Four controls, each of which alone can invert the conclusion.

Edge-case matrix

Benchmark condition What it measures When it is the right question
Cold cache, single run Disk throughput First-touch latency on a batch server
Warm cache, repeated Library and decompression cost Iterative development, repeated access
File larger than RAM Whether the job completes at all Production sizing
Named dimensions both sides Like-for-like read cost Any fair comparison
All dimensions both sides Full materialisation cost Archival conversion work
Single-threaded Per-core efficiency Shared, oversubscribed machines
All cores Wall time for one job Dedicated processing nodes
Peak RSS recorded Feasibility on the target machine Always

The single-threaded row deserves attention on shared infrastructure. PDAL’s filters take a threads option and will happily consume every core, which looks excellent in a benchmark on an idle machine and degrades every other job on a busy one. Measuring at the concurrency production actually has is the only measurement that predicts production.

Verification snippet

def report(results: dict[str, dict], threshold: float = 0.15) -> str:
    """Turn two benchmark results into an honest one-line conclusion."""
    (name_a, a), (name_b, b) = results.items()
    ratio = a["median_s"] / b["median_s"]
    noise = (a["spread_s"] + b["spread_s"]) / (a["median_s"] + b["median_s"])

    if noise > abs(1 - ratio):
        return (f"no measurable difference: run-to-run spread ({noise:.0%}) exceeds "
                f"the gap between {name_a} and {name_b}")
    faster, slower = (name_b, name_a) if ratio > 1 else (name_a, name_b)
    return (f"{faster} is {max(ratio, 1/ratio):.2f}× faster than {slower} "
            f"(spread {noise:.0%}); peak RSS "
            f"{a['python_peak_mb']:.0f} vs {b['python_peak_mb']:.0f} MB")
Three regimes where the faster library changes Three panels covering the regimes. For plain reading, both libraries land within about twenty percent of each other because the time is spent in shared decompression code. For standard filters such as outlier removal or ground classification, PDAL is roughly fifteen times faster because its implementations are tuned and a hand-rolled equivalent is not. For bespoke per-point arithmetic, laspy is about three times faster because the alternative pays Python costs plus marshalling overhead. A note states the regime should be identified before the benchmark, not after. plain reading both in the same decompression code within 20 % chunk size matters more standard filters outlier, SMRF, HAG tuned for a decade PDAL ≈ 15× not a language difference bespoke arithmetic one NumPy expression vs a Python filter stage laspy ≈ 3× marshalling overhead Identify the regime first. The benchmark then confirms rather than decides. A single number for "which is faster" is a category error — there are three answers.

Figure 2 — The three regimes, and roughly what to expect in each before measuring anything.

Reporting a benchmark somebody else can trust

A benchmark result travels further than the machine it was run on, so it should carry enough context to be re-evaluated. Five items are enough: the library versions, the file used (point count, dimensions, compression), the machine (cores, RAM, storage type), the cache condition, and the concurrency. Without them, a number is folklore within a month.

import platform

import laspy


def benchmark_context(path: str) -> dict:
    """Everything a reader needs to judge whether a result applies to them."""
    import pdal
    with laspy.open(path) as fh:
        h = fh.header
    return {
        "laspy": laspy.__version__,
        "pdal_python": pdal.__version__,
        "file": {"points": int(h.point_count),
                 "point_format": h.point_format.id,
                 "compressed": str(path).endswith(".laz")},
        "machine": {"cpu": platform.processor(),
                    "cores": __import__("os").cpu_count(),
                    "python": platform.python_version()},
        "cache": "warm",          # state it explicitly; the reader cannot infer it
        "concurrency": 1,
    }

Two conventions make the results actually useful. Record the benchmark alongside the decision it informed, so that when the decision is revisited the evidence is there. And re-run it when a major version of either library lands — PDAL’s filter implementations and laspy’s LAZ backends both change materially between releases, and a two-year-old benchmark is a statement about software nobody is running any more.

What to do with the result

A benchmark’s job is to close a question, and closing it means writing down what was decided as well as what was measured. Three outcomes are common and each has a different follow-up.

A clear winner in the regime that matters. Adopt it for that stage and move on. Do not generalise the result to other stages — the regimes really are different, and a library that wins the filtering benchmark may lose the header benchmark by a larger factor.

No measurable difference. This is the most common outcome for plain reading, and it is a useful one: it means the choice can be made on maintainability, packaging or team familiarity with no performance cost. Record that explicitly, so the next person does not re-run the same benchmark hoping for a different answer.

A surprising result. Investigate before acting. Surprises in this area are usually measurement artefacts — a cache effect, a dimension mismatch, or one side quietly running multi-threaded. A result that contradicts the regimes above is more likely to be a flawed benchmark than a discovery.

When to escalate

  • The two libraries disagree on the result, not just the time. Stop benchmarking and find out why. A difference in point count or coordinate values is a correctness problem and makes the timing irrelevant.
  • Performance is the deciding factor in an architecture choice. It usually should not be. Streaming capability, reproducibility and maintainability outlast a factor of two, and the comparison page covers the axes that persist.
  • A job is slow and neither library is the bottleneck. Check storage throughput before rewriting anything; on network-mounted data, both libraries are waiting on the same pipe.

← PDAL vs laspy for Point Cloud Automation