Uploading COGs to Object Storage with Python

A four-gigabyte orthomosaic uploaded over a site connection is a twenty-minute operation that fails at minute eighteen more often than anybody expects. The difference between a pipeline that copes and one that does not is entirely in how the upload is structured — and the structure that copes is also the one that verifies what arrived.

This page covers multipart uploads, the headers that matter, and the integrity check that catches a truncated transfer before a STAC item is written against it. It belongs to publishing rasters to STAC and object storage.

# Why multipart is not optional

A single-request upload of a large file has three properties that make it unsuitable for survey deliverables: it cannot resume, it cannot parallelise, and many endpoints reject it outright above five gigabytes.

Multipart splits the object into parts uploaded independently. A failed part is retried on its own rather than restarting the transfer. Parts upload concurrently, which on a connection with any latency is where most of the speed comes from. And the completion step is atomic — the object does not exist at its key until every part has arrived, so a consumer never sees a half-written raster.

Figure 1 — The failure behaviour is the reason, not the speed.

# Minimal reproducible solution

boto3’s transfer configuration handles the mechanics; the value of wrapping it is in the headers and the verification around it.

import hashlib
from pathlib import Path

import boto3
from boto3.s3.transfer import TransferConfig
from botocore.config import Config

COG_TYPE = "image/tiff; application=geotiff; profile=cloud-optimized"

TRANSFER = TransferConfig(
    multipart_threshold=64 * 1024 * 1024,   # anything above 64 MB goes multipart
    multipart_chunksize=32 * 1024 * 1024,   # 32 MB parts: few enough requests,
    max_concurrency=8,                      # small enough to retry cheaply
    use_threads=True,
)


def storage_client(endpoint_url: str | None = None):
    """An S3 client with retry behaviour suited to a site connection.

    The adaptive retry mode backs off on throttling rather than hammering, and
    ten attempts is not excessive when a single part failure would otherwise
    strand a twenty-minute upload.
    """
    return boto3.client(
        "s3",
        endpoint_url=endpoint_url,
        config=Config(retries={"max_attempts": 10, "mode": "adaptive"}),
    )


def upload_cog(client, path: str, bucket: str, key: str, *,
               cache_seconds: int = 31536000) -> dict:
    """Upload one COG with the headers a web client needs.

    A year-long max-age is safe here only because keys are never overwritten —
    a reprocessed raster goes to a new key. Under an overwrite policy this same
    header makes stale pixels unevictable for a year.
    """
    digest = sha256_of(path)
    client.upload_file(
        Filename=path,
        Bucket=bucket,
        Key=key,
        ExtraArgs={
            "ContentType": COG_TYPE,
            "CacheControl": f"public, max-age={cache_seconds}, immutable",
            "Metadata": {"sha256": digest},
        },
        Config=TRANSFER,
    )
    return {"key": key, "sha256": digest, "bytes": Path(path).stat().st_size}


def sha256_of(path: str, *, block: int = 1 << 20) -> str:
    """Digest the local file in blocks, so memory does not scale with size."""
    h = hashlib.sha256()
    with open(path, "rb") as handle:
        for chunk in iter(lambda: handle.read(block), b""):
            h.update(chunk)
    return h.hexdigest()

Storing the digest in the object’s metadata is what makes verification possible later without keeping a separate record. It travels with the object, survives copies between buckets, and costs nothing.

# The headers that matter

Content-Type. image/tiff; application=geotiff; profile=cloud-optimized is precise and understood by tooling. The generic application/octet-stream works for a direct read and loses information every intermediary could have used.

Cache-Control. Long and immutable for deliverables, short for catalogue JSON. The asymmetry is the point: pixels never change at a given key, metadata does.

Metadata. The digest, and usefully the survey identifier. Object metadata is returned by a HEAD request, so a consumer can check both without transferring a byte of content.

Content-Encoding. Leave it unset. A COG is already compressed internally, and gzipping it at the transport layer defeats range requests entirely — the server can no longer serve an arbitrary byte range without decompressing the whole object.

The last of these causes a specific and confusing failure: a COG that works when downloaded whole and fails in every web client, because range requests against a transport-compressed object return the wrong bytes.

Figure 2 — Four headers, one of which breaks everything if set.

# Tuning part size and concurrency

The two transfer parameters interact, and the usual instinct — make the parts bigger — is the wrong lever on the connections survey work actually runs over.

Throughput on a single stream is bounded by the bandwidth-delay product: how much data can be in flight before an acknowledgement returns. On a link with 80 ms round-trip latency, one stream saturates at a few tens of megabits regardless of the available bandwidth. Adding streams multiplies that; making each stream’s chunks larger does not.

def suggest_transfer_config(file_bytes: int, rtt_ms: float,
                            link_mbps: float) -> TransferConfig:
    """Pick part size and concurrency from the connection, not from habit.

    Concurrency is derived from how many streams it takes to fill the
    bandwidth-delay product; part size is then chosen so the object divides
    into enough parts to keep every stream busy, without dropping below the
    size where request overhead starts to dominate.
    """
    per_stream_mbps = max(1.0, 8 * 64 * 1024 / max(rtt_ms, 1.0) / 1000)
    concurrency = int(min(16, max(4, link_mbps / per_stream_mbps)))

    target_parts = concurrency * 4
    part = file_bytes // max(target_parts, 1)
    part = min(64 * 1024 * 1024, max(16 * 1024 * 1024, part))

    return TransferConfig(
        multipart_threshold=64 * 1024 * 1024,
        multipart_chunksize=int(part),
        max_concurrency=concurrency,
        use_threads=True,
    )

Two limits bound the useful range. Below 16 MB the per-request overhead becomes a measurable fraction of the transfer on a high-latency link. Above 64 MB a failed part costs more to retry than it saved, and on a genuinely unreliable connection the retry cost is the dominant term in total upload time rather than an edge case.

Concurrency has its own ceiling, and it is usually local: eight to sixteen streams will saturate most site connections, and beyond that the threads contend for the same bandwidth while multiplying the chance that one of them trips a throttling response. If throughput plateaus as concurrency rises, the bottleneck has moved and more streams will make it worse.

Figure 3 — Three failures, one of which is visible without looking.

# Edge-case matrix

The abandoned-upload row is a real and invisible cost. A multipart upload that fails and is never completed or aborted leaves its parts billable indefinitely; a bucket lifecycle rule aborting incomplete uploads after seven days costs nothing to add and prevents a bill that nobody can account for.

# Verification snippet

def verify_upload(client, bucket: str, key: str, *,
                  expected_sha256: str, expected_bytes: int) -> dict:
    """Confirm what arrived matches what was sent, without downloading it.

    A HEAD returns the size and the metadata digest, which together catch the
    two failures that matter: a truncated transfer and a completed upload of
    the wrong file. Both are cheap to detect now and expensive to discover
    after a STAC item has been published against them.
    """
    head = client.head_object(Bucket=bucket, Key=key)
    problems = []

    if head["ContentLength"] != expected_bytes:
        problems.append(
            f"size {head['ContentLength']} != expected {expected_bytes}")

    stored = head.get("Metadata", {}).get("sha256")
    if stored is None:
        problems.append("no sha256 in object metadata")
    elif stored != expected_sha256:
        problems.append("digest mismatch — the object is not the file sent")

    if head.get("ContentEncoding"):
        problems.append(
            f"Content-Encoding is {head['ContentEncoding']} — ranges will break")

    if not head.get("ContentType", "").startswith("image/tiff"):
        problems.append(f"unexpected Content-Type {head.get('ContentType')}")

    return {"ok": not problems, "problems": problems,
            "etag": head.get("ETag")}

Note that the ETag is deliberately not used as the integrity check. On a multipart upload it is a digest of digests, not of the content, so it cannot be compared against a local file’s hash — a mismatch tells you nothing and a match is coincidence.

# When to escalate

• Uploads fail consistently at the same size. Suspect an intermediary with a request-size limit rather than the connection. Reducing the part size often resolves it immediately.
• The digest mismatches reproducibly. The local file is being modified during upload, usually because the processing run has not actually finished writing it. Add a completion barrier.
• Throughput is far below the link capacity. Raise concurrency before raising part size; on a high-latency link the limit is almost always the number of requests in flight.

# Related

• Publishing Rasters to STAC and Object Storage
• Serving COG Range Requests Without a Tile Server
• Versioning Deliverables with Checksums and Manifests

← Publishing Rasters to STAC and Object Storage