Caching PROJ Grids for Offline Processing

The same survey processed on two machines produces coordinates eleven centimetres apart. Both ran the same code, the same library versions and the same input. One machine had downloaded a transformation grid at some point; the other had not, and PROJ quietly used a lower-accuracy path.

Modern PROJ can fetch grids on demand over the network, which is convenient for interactive work and corrosive for a processing pipeline: it makes a numerical result depend on network state. This page covers taking control of that — determining which grids a project needs, fetching them once, pinning them into the processing environment, and proving afterwards that the expected ones were used. It supports the reproducibility discipline in geoid models and vertical datum automation.

# Why on-demand fetching is a problem for a pipeline

Three properties make it unsuitable for production, and none of them is a criticism of the feature itself.

The result depends on network state. A machine behind a proxy, a worker on an isolated network, or a transient outage produces a different answer rather than an error.

The cache is invisible. Grids land in a user cache directory whose contents differ between machines and between users on one machine, so “it works on my machine” is literally true and unhelpful.

Grids are revised. A national agency publishing an improved grid changes the answer for everybody who fetches afterwards, which is an accuracy improvement and a reproducibility failure at the same time.

Figure 1 — The failure is not that a machine is wrong; it is that they disagree and none of them says so.

# Determining which grids a project needs

A project needs the grids for the transformations it performs, which is a short list that can be enumerated from the CRS pairs in use.

from pyproj import CRS
from pyproj.transformer import TransformerGroup


def required_grids(crs_pairs: list[tuple[str, str]]) -> dict:
    """Enumerate every grid the project's transformations can use.

    Both available and unavailable grids are listed, because the point is to
    fetch the missing ones rather than to work around them.
    """
    needed: dict[str, dict] = {}
    for source_def, target_def in crs_pairs:
        source = CRS.from_user_input(source_def)
        target = CRS.from_user_input(target_def)
        group = TransformerGroup(source, target, always_xy=True)
        for operation in list(group.transformers) + list(group.unavailable_operations):
            for grid in getattr(operation, "grids", []):
                needed[grid.short_name] = {
                    "full_name": grid.full_name,
                    "url": grid.url,
                    "available": grid.available,
                    "direct_download": grid.direct_download,
                    "open_license": grid.open_license,
                }
    missing = {k: v for k, v in needed.items() if not v["available"]}
    return {"grids": needed, "missing": missing,
            "restricted": {k: v for k, v in needed.items() if not v["open_license"]}}

The restricted list matters. Some national grids are not openly licensed and cannot be redistributed inside an image, so a project depending on them needs them obtained and installed separately — which is a licensing conversation rather than a technical one, and better had at the start of a project than at delivery.

# Fetching and pinning them

import hashlib
import shutil
import subprocess
from pathlib import Path


def sync_grids(target_dir: str, *, area_of_interest: dict | None = None) -> dict:
    """Download the grids a project needs into a directory under its control.

    projsync fetches into the PROJ data directory; copying into a
    project-controlled directory afterwards is what makes the set
    reproducible, because a user cache is neither versioned nor shared.
    """
    cmd = ["projsync", "--target-dir", str(target_dir), "--all"]
    if area_of_interest:
        cmd = ["projsync", "--target-dir", str(target_dir),
               "--bbox", "{west},{south},{east},{north}".format(**area_of_interest)]
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(f"projsync failed: {result.stderr[-400:]}")

    files = sorted(p for p in Path(target_dir).rglob("*") if p.is_file())
    digest = hashlib.sha256()
    for path in files:
        digest.update(path.name.encode())
        digest.update(str(path.stat().st_size).encode())

    return {"directory": str(target_dir), "files": len(files),
            "inventory_digest": digest.hexdigest()[:16],
            "total_mb": round(sum(p.stat().st_size for p in files) / 1e6, 1)}

Using a bounding box rather than fetching everything is worth doing: the complete grid set is tens of gigabytes and a project needs the few hundred megabytes covering its own region.

# Disabling the network in the processing environment

import os


def lock_proj_environment(grid_dir: str) -> dict:
    """Point PROJ at a fixed grid directory and forbid network access.

    Setting both is necessary. PROJ_DATA alone leaves network fetching
    enabled, so a missing grid is still fetched and the directory is not the
    complete story of what was used.
    """
    os.environ["PROJ_DATA"] = str(grid_dir)
    os.environ["PROJ_NETWORK"] = "OFF"
    os.environ.pop("PROJ_LIB", None)          # legacy variable; can override PROJ_DATA

    import pyproj
    pyproj.network.set_network_enabled(False)
    return {"PROJ_DATA": os.environ["PROJ_DATA"],
            "PROJ_NETWORK": os.environ["PROJ_NETWORK"],
            "pyproj_data_dir": pyproj.datadir.get_data_dir(),
            "network_enabled": pyproj.network.is_network_enabled()}

Removing PROJ_LIB is the detail that catches people. It is the older variable name, several tools still set it, and where both exist the behaviour depends on the PROJ version — so a carefully set PROJ_DATA can be silently overridden by an inherited PROJ_LIB from a parent process.

Figure 3 — The cost of on-demand is a wrong answer, not a slow one.

# Edge-case matrix

# Verification snippet

import hashlib
from pathlib import Path

import pyproj


def verify_grid_environment(expected_digest: str, grid_dir: str) -> dict:
    """Confirm the running environment has exactly the expected grid set."""
    files = sorted(p for p in Path(grid_dir).rglob("*") if p.is_file())
    digest = hashlib.sha256()
    for path in files:
        digest.update(path.name.encode())
        digest.update(str(path.stat().st_size).encode())
    actual = digest.hexdigest()[:16]

    problems = []
    if actual != expected_digest:
        problems.append(f"grid inventory digest is {actual}, expected {expected_digest}")
    if pyproj.network.is_network_enabled():
        problems.append("PROJ network fetching is enabled — results may depend on it")
    if Path(pyproj.datadir.get_data_dir()) != Path(grid_dir):
        problems.append(f"pyproj is using {pyproj.datadir.get_data_dir()}, "
                        f"not {grid_dir}")

    return {"inventory_digest": actual, "files": len(files),
            "problems": problems, "ok": not problems}

Running that at worker start-up rather than at first use is the right placement. A worker with the wrong grid set should refuse to start, not process a survey and produce coordinates nobody can reproduce.

Figure 2 — Three strategies, and why the narrowest is the best one for a pipeline.

# Updating the pinned set

A pinned grid set is deliberately frozen, which means updating it is an event rather than a background process. Three steps make the update safe.

Fetch the new set into a separate directory and record its digest. Re-process a reference survey with both sets and compare the coordinates, which gives a concrete number for what the update changed. And record the change in the project history with that number, so a discontinuity in a monitoring series has a documented cause rather than appearing as unexplained movement.

Most grid updates move nothing measurable, and the ones that do are exactly the ones worth telling a client about before they notice. Keeping the previous set alongside the new one for an overlap period also makes it possible to reproduce an older deliverable exactly, which is occasionally required and impossible once the old grids are gone.

# When to escalate

• A required grid cannot be redistributed. Install it on the workers separately and verify its presence at start-up. The digest check handles the verification either way.
• Two regions need incompatible grid versions. Process them separately and state the model used for each; merging products converted with different models introduces a step at the boundary.
• The digest changes and nobody knows why. Something is fetching. Check that network fetching is off in every process, including any subprocess that inherits a different environment.

# Related

• Geoid Models and Vertical Datum Automation
• Applying a Geoid Grid with pyproj Transformers
• Pinning GDAL, PROJ and OpenCV Versions Reproducibly

← Geoid Models and Vertical Datum Automation