Publishing Rasters to STAC and Object Storage

The processing finished, the orthomosaic validated as a Cloud-Optimized GeoTIFF, and the file is sitting in a directory named after the site with a date in it. Somebody now has to find it in eight months, know what datum it is in, and open it from a web map without downloading four gigabytes.

That last mile — catalogue metadata, object storage layout, and a versioning scheme that survives reprocessing — is what this page covers. It is the final stage of DEM/DSM generation and raster export automation, immediately downstream of exporting Cloud-Optimized GeoTIFFs with rasterio.

Audience and prerequisites. Python 3.10+, valid COGs to publish, and credentials for an S3-compatible object store. No STAC API server is required — a static catalogue of JSON files served over HTTP is a complete, useful implementation.

# Prerequisites

# Conceptual architecture

Three things have to be true before a raster is genuinely published, and each is independently easy to get wrong.

The bytes are reachable. The file is in object storage, at a stable key, with permissions that allow the intended readers to issue HTTP range requests against it. A COG served without range request support is just a large download.

The metadata is findable. A STAC item describes what the raster is: its footprint, its datetime, its coordinate reference system, its bands, and a link to the bytes. A catalogue of these is searchable by space and time without opening a single raster.

The identity is stable. Reprocessing produces a new file that is meant to replace an old one. Without a versioning convention, either the old URL silently starts returning different pixels — which breaks every saved map — or the new version is never found.

Figure 1 — Three requirements, each independently easy to miss.

# What a STAC item actually is

A STAC item is a GeoJSON Feature with a fixed set of extra fields. That is the whole specification at the level that matters for publishing survey rasters: a footprint geometry, a bounding box, a datetime, a properties object, and an assets object mapping names to URLs.

The value is not in the format, which is unremarkable. It is that a directory of these files is a spatial-temporal index that any client can read without a database, and that the field names are shared, so a tool written for satellite imagery works unchanged on a drone survey.

Two extensions carry most of the useful survey detail. The projection extension records the EPSG code, the affine transform and the raster shape, which lets a client decide whether a raster is worth opening before opening it. The raster extension records per-band data type, nodata value and statistics, which lets a viewer choose a sensible stretch without reading any pixels.

import pystac
from datetime import datetime, timezone


def survey_item(*, item_id: str, footprint: dict, bbox: list[float],
                captured: datetime, cog_url: str, epsg: int,
                gsd_m: float, survey_id: str) -> pystac.Item:
    """A STAC item for one survey raster.

    `gsd` and the projection EPSG are the two fields that make a catalogue
    genuinely useful to somebody else later: the first answers "is this
    detailed enough for what I need", the second answers "will this line up
    with my data", and both answers arrive without downloading anything.
    """
    item = pystac.Item(
        id=item_id,
        geometry=footprint,
        bbox=bbox,
        datetime=captured.astimezone(timezone.utc),
        properties={"gsd": gsd_m, "survey:id": survey_id},
    )
    item.properties["proj:epsg"] = epsg
    item.add_asset(
        "data",
        pystac.Asset(
            href=cog_url,
            media_type=pystac.MediaType.COG,
            roles=["data"],
            title="Orthomosaic",
        ),
    )
    return item

# Minimal reproducible solution

Deriving the geometry from the raster rather than typing it is the difference between a catalogue that is trustworthy and one that is decorative.

import rasterio
from rasterio.warp import transform_bounds
from shapely.geometry import box, mapping


def footprint_from_raster(path: str) -> dict:
    """Footprint and bbox in WGS84, as STAC requires, derived from the file.

    STAC mandates geographic coordinates for `geometry` and `bbox` regardless
    of the raster's own projection — a catalogue is searched in one frame or it
    cannot be searched at all. The dataset's native CRS goes in proj:epsg.
    """
    with rasterio.open(path) as src:
        native = src.bounds
        epsg = src.crs.to_epsg()
        west, south, east, north = transform_bounds(
            src.crs, "EPSG:4326", *native, densify_pts=21)
        gsd = abs(src.transform.a)

    geom = box(west, south, east, north)
    return {
        "geometry": mapping(geom),
        "bbox": [west, south, east, north],
        "epsg": epsg,
        "gsd_m": float(gsd),
    }

The densify_pts argument matters more than it looks. Transforming only the four corners of a projected bounding box produces a geographic box that is systematically too small, because the projected rectangle’s edges bow outward in geographic coordinates. On a site a few hundred metres across the error is negligible; on a large corridor survey it is enough to make a spatial search miss the very item it is looking for.

# Object storage layout

The key structure is a decision made once and regretted for years if made badly. Three principles.

Put the survey identity in the path, not only in the filename. A key like sites/hawkridge/2026-09-17/ortho.tif sorts and lists usefully. A key like hawkridge_ortho_20260917_v3_final_FINAL.tif in a flat bucket does not.

Never overwrite. A reprocessed orthomosaic goes to a new key. The old one stays until somebody deliberately removes it. This is what makes a saved web map keep working and a reported problem still reproducible.

Keep the item next to the asset. The STAC item’s own JSON lives in the same prefix as the raster it describes. Copying a prefix then moves a complete, self-describing unit.

Figure 2 — A layout that survives reprocessing.

# Parameter deep-dive

Multipart threshold and part size. Uploads above a few hundred megabytes should be multipart. A part size of 16–64 MB balances retry cost against request count; below 8 MB the request overhead dominates on large rasters.

Content-Type. A COG served as application/octet-stream works, but image/tiff; application=geotiff; profile=cloud-optimized tells intermediaries and browsers what they have. Some CDN configurations make caching decisions on it.

Cache-Control. Because keys are never overwritten, deliverables can be given a very long max-age. The catalogue JSON is the opposite: it changes when items are added, and wants a short max-age or explicit invalidation.

CORS. A browser-based map reading a COG by range request needs the bucket to expose Range in Access-Control-Allow-Headers and Content-Range and Content-Length in Access-Control-Expose-Headers. Omitting these produces a failure that looks like a corrupt file.

Storage class. Archive tiers have retrieval latency measured in hours. They are correct for superseded versions and completely wrong for anything a web map points at.

# Collections, and why a flat catalogue stops working

A single item is easy. A programme with four hundred of them needs an organising layer, and STAC provides exactly one: the collection. A collection groups items that share a definition — the same sensor, the same processing chain, the same licence — and carries the fields that would otherwise be repeated on every item.

Two things belong on the collection rather than on each item. The extent, a spatial and temporal envelope covering everything inside it, which lets a client discard an entire collection without reading a single item. And the summaries, which describe the range of values the items take: the ground sample distances present, the platforms used, the EPSG codes in play.

import pystac
from datetime import datetime, timezone


def survey_collection(*, collection_id: str, title: str,
                      items: list[pystac.Item]) -> pystac.Collection:
    """Build a collection from the items it will contain.

    Deriving the extent from the items rather than declaring it by hand is the
    only way it stays true. A hand-written extent is correct on the day it is
    written and quietly wrong from the next survey onward, which is worse than
    having none at all — a client trusts it and skips the collection.
    """
    boxes = [item.bbox for item in items]
    spatial = [
        min(b[0] for b in boxes), min(b[1] for b in boxes),
        max(b[2] for b in boxes), max(b[3] for b in boxes),
    ]
    times = [item.datetime for item in items]

    collection = pystac.Collection(
        id=collection_id,
        description=title,
        extent=pystac.Extent(
            spatial=pystac.SpatialExtent([spatial]),
            temporal=pystac.TemporalExtent([[min(times), max(times)]]),
        ),
        license="proprietary",
    )
    collection.summaries = pystac.Summaries({
        "gsd": sorted({round(i.properties["gsd"], 3) for i in items}),
        "proj:epsg": sorted({i.properties["proj:epsg"] for i in items}),
    })
    for item in items:
        collection.add_item(item)
    return collection

Partitioning is the decision that follows. A collection per site works when the programme is site-oriented and each site is revisited — it makes the temporal series obvious and keeps each collection small. A collection per product type works when consumers want “every orthomosaic” rather than “everything about Hawkridge”. Most survey programmes end up with collections by product type and use the survey:id property to group by site, because a consumer asking for all orthomosaics at 3 cm or better across every site is the query that arrives most often.

The link structure is what makes a static catalogue traversable without a server. Every item carries a parent link to its collection and a root link to the top of the catalogue; every collection carries child links to its items. A client that fetches the root can walk the whole tree, and a broken link is the one failure that makes a published item unreachable while every byte of it is present and correct.

# Publishing as part of the pipeline, not after it

The most common way a catalogue decays is that publishing is a separate manual step done at the end of a job, by somebody who has already mentally finished. Items get skipped, fields get typed rather than derived, and within a season the catalogue describes some of the data some of the time.

Making publication the last stage of the processing run fixes this structurally. The run already knows the datum, the ground sample distance, the flight datetime and the survey identifier — they are its own inputs — so deriving the item costs nothing and cannot disagree with reality.

def publish_outputs(run: dict, uploader, catalogue_root: str) -> dict:
    """Publish every raster a processing run produced, as one unit.

    The run either publishes completely or not at all. A partially published
    survey — pixels uploaded, item missing — is the state that produces files
    nobody can find, and it is far easier to avoid than to detect later.
    """
    published, problems = [], []
    prefix = f"sites/{run['site']}/{run['capture_date']}"

    try:
        assets = {}
        for name, path in run["rasters"].items():
            key = f"{prefix}/{name}.tif"
            assets[name] = uploader.upload(path, key)

        derived = footprint_from_raster(next(iter(run["rasters"].values())))
        item = survey_item(
            item_id=f"{run['site']}-{run['capture_date']}",
            footprint=derived["geometry"],
            bbox=derived["bbox"],
            captured=run["captured_at"],
            cog_url=assets[run["primary"]],
            epsg=derived["epsg"],
            gsd_m=derived["gsd_m"],
            survey_id=run["survey_id"],
        )
        for name, href in assets.items():
            if name != run["primary"]:
                item.add_asset(name, pystac.Asset(
                    href=href, media_type=pystac.MediaType.COG, roles=["data"]))

        uploader.upload_json(item.to_dict(), f"{prefix}/item.json")
        published.append(item.id)
    except Exception as exc:                      # noqa: BLE001 — reported, not swallowed
        problems.append(f"{prefix}: {exc}")

    return {"published": published, "problems": problems,
            "complete": not problems}

Two details in that function are worth stating plainly. The item is built from footprint_from_raster against a file that has already been uploaded, so the metadata describes the published bytes rather than a local copy that may differ. And the exception is recorded rather than swallowed: a publication step that fails silently produces exactly the half-published state it was meant to prevent.

# Edge-case matrix

# Verification snippet

import requests


def verify_published(item_url: str) -> dict:
    """Prove a published item is genuinely usable, not merely present.

    Three independent things are checked because each fails on its own: the
    item parses, the asset responds to a range request, and the bytes returned
    are a TIFF rather than an error page served with a 200 status.
    """
    problems = []
    item = requests.get(item_url, timeout=30).json()

    assets = item.get("assets", {})
    if "data" not in assets:
        problems.append("no asset with role data")
        return {"ok": False, "problems": problems}

    href = assets["data"]["href"]
    head = requests.head(href, timeout=30)
    if head.headers.get("Accept-Ranges") != "bytes":
        problems.append("asset does not advertise range request support")

    part = requests.get(href, headers={"Range": "bytes=0-1023"}, timeout=30)
    if part.status_code != 206:
        problems.append(f"range request returned {part.status_code}, expected 206")
    elif part.content[:2] not in (b"II", b"MM"):
        problems.append("first bytes are not a TIFF header")

    for field in ("proj:epsg", "gsd"):
        if field not in item.get("properties", {}):
            problems.append(f"item is missing {field}")

    return {"ok": not problems, "problems": problems, "asset": href}

Figure 3 — Publication as a sequence, with verification as part of it.

# FAQ

# Do I need a STAC API server?

No. A static catalogue — a directory of item JSON files with a collection JSON linking them, served over ordinary HTTP — is a complete STAC implementation and is what most survey programmes should build. An API server adds server-side search, which matters at thousands of items and is overhead below a few hundred.

# Should the DEM and the orthomosaic be one item or two?

One item with two assets, when they came from the same flight and share a footprint and datetime. That is exactly what the assets object is for, and splitting them doubles the catalogue while halving the information in each entry.

# How does this relate to the accuracy manifest?

They are complementary and belong in the same prefix. The STAC item says where and when; the accuracy manifest says how good. A consumer choosing between two overlapping surveys needs both.

# Can an existing directory of rasters be catalogued retrospectively?

Yes, and it is usually worth doing. Footprint, bounding box, ground sample distance and EPSG all come from the files themselves, so the only field that genuinely needs external knowledge is the capture datetime — and that is normally recoverable from the source imagery’s EXIF or from the job records. A backfill script that walks a bucket, derives what it can and reports the items it could not date is a few hours’ work and turns an archive into something searchable.

# What about very large collections?

Beyond a few thousand items, listing a prefix becomes slow and a static catalogue’s link structure needs care — partition by year or by site rather than keeping a single flat collection. The item format does not change; only the organisation above it does.

# Guides in this topic

• Writing a STAC Item for an Orthomosaic
• Uploading COGs to Object Storage with Python
• Serving COG Range Requests Without a Tile Server
• Versioning Deliverables with Checksums and Manifests

# Related

• Exporting Cloud-Optimized GeoTIFFs with Rasterio
• Tiling and Serving Orthomosaics as XYZ Tiles
• Orthomosaic Radiometry and Seamline Control

← DEM/DSM Generation & Raster Export Automation