Decoding GDAL Cannot Allocate Memory on Export

You launch a final export — a gdalwarp reprojection of a 40,000 × 35,000 pixel orthomosaic, a gdal_translate to a distributable GeoTIFF, or a rasterio write of a merged DEM — and the process either aborts with ERROR 1: Cannot allocate memory or simply vanishes with exit status -9 and no message at all. The input is nowhere near your total disk size, the machine has 16 GB of RAM, and a smaller test tile exported fine minutes earlier. The symptom is misleading: GDAL is not leaking memory, and the file is not corrupt. It is being asked to hold an entire raster plane in RAM at once, and on a large ortho that single allocation exceeds what the process — or the kernel — will grant. This page decodes exactly which allocation blows up, and gives you a windowed, cache-capped export that keeps peak resident memory flat regardless of raster size.

# Why GDAL runs out of memory on a large export

A raster’s in-memory footprint is not its file size. Compression, tiling, and overviews shrink what lands on disk, but an operation that materialises a full band computes its cost from the uncompressed pixel grid. For a single-band Float32 DEM the whole-plane allocation is width × height × 4 bytes; for a 3-band Byte ortho it is width × height × 3. The 40,000 × 35,000 ortho above therefore needs $40000 \times 35000 \times 3 = 4.2\ \text{GB}$ for one interleaved read, and a matching output buffer, before any warp scratch space. When a single malloc of that size fails, GDAL surfaces CPLError as ERROR 1: Cannot allocate memory; when the allocation succeeds on paper but the kernel cannot back it with physical pages, the OOM-killer sends SIGKILL and you see exit -9 with no GDAL message.

Three specific decisions push an export from “streams fine” to “allocates everything”:

• Whole-raster reads. Calling ds.read() with no window, or a legacy tool that processes a full band at once, allocates the entire uncompressed plane. This is the dominant cause.
• An oversized GDAL_CACHEMAX. GDAL’s block cache defaults to a percentage of RAM in some builds. On a machine shared with a photogrammetry engine, a cache sized at 40% of 16 GB competes directly with the reconstruction process, and the sum OOMs.
• Untiled (striped) input. A striped GeoTIFF forces GDAL to read whole scanlines. Windowed access on a striped file still pulls full rows into the cache, so “windowing” a striped raster does not bound memory the way it does on a tiled one.

A fourth, quieter cause is the missing BIGTIFF flag: a standard GeoTIFF is capped at 4 GB, and a large uncompressed output either fails to create or triggers workarounds that inflate memory. Understanding that the footprint is driven by the uncompressed grid — not the file — is what makes the fix obvious. The broader menu of export failures this sits within is catalogued in the troubleshooting DEM and raster export failures guide.

# Minimal reproducible solution

The fix is to never hold a full plane in RAM. The routine below copies any raster to a tiled, BIGTIFF output by iterating the source’s native block windows and reading/writing one block at a time, with the GDAL cache pinned to a small fixed budget. Peak memory is then roughly one block plus the cache, not the whole raster. It is under 60 lines and depends only on rasterio.

import os
import rasterio
from rasterio.windows import Window

# Pin the cache to a fixed MB budget BEFORE opening any dataset. A fixed
# value (not a percentage) is what keeps a shared machine from OOMing.
os.environ["GDAL_CACHEMAX"] = "256"          # megabytes


def block_copy(src_path: str, dst_path: str, blocksize: int = 512) -> None:
    """Copy a raster block-by-block into a tiled, BIGTIFF GeoTIFF."""
    with rasterio.open(src_path) as src:
        profile = src.profile.copy()
        profile.update(
            driver="GTiff",
            tiled=True,                       # tiled output = windowed reads later
            blockxsize=blocksize,
            blockysize=blocksize,
            compress="deflate",               # shrinks disk, not the read buffer
            bigtiff="YES",                    # lift the 4 GB GeoTIFF ceiling
            num_threads="all_cpus",
        )
        with rasterio.open(dst_path, "w", **profile) as dst:
            # Iterate the SOURCE's block windows so each read is one tile.
            for _, window in src.block_windows(1):
                for band in range(1, src.count + 1):
                    data = src.read(band, window=window)   # ~blocksize² pixels
                    dst.write(data, band, window=window)

The load-bearing detail is src.block_windows(1): it yields the raster’s own tiling, so each src.read(..., window=window) pulls exactly one on-disk tile rather than a reconstructed region that might span many. If the source is striped rather than tiled, block_windows yields full-width strips — still bounded in height, but you should re-tile early (this routine does, on the output) so downstream steps get true 512 × 512 blocks.

For a reprojection rather than a straight copy, use gdalwarp with explicit memory bounds instead of letting it default:

# -wm bounds warp working memory (MB); creation options force tiled + BIGTIFF.
gdalwarp -t_srs EPSG:32633 \
  -wm 512 --config GDAL_CACHEMAX 256 \
  -co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 \
  -co COMPRESS=DEFLATE -co BIGTIFF=YES -co NUM_THREADS=ALL_CPUS \
  -multi ortho_in.tif ortho_utm.tif

-wm 512 caps warp scratch memory and --config GDAL_CACHEMAX 256 caps the block cache; together they bound the two allocations that actually blow up. -multi parallelises across the tiled output without raising per-tile memory. Once the export is memory-safe, the choice between driving it from the CLI or from Python is a real tradeoff covered in GDAL vs rasterio for DEM and orthomosaic export.

# Edge-case matrix

These are the input variants that turn a “safe” windowed export back into an OOM, and how to handle each.

The striped-source row is the one that most often re-breaks a “fixed” pipeline: windowing a striped raster does not bound memory the way it does on a tiled one, because the smallest readable unit is a full-width strip.

# Verification snippet

Prove two things after the export: the process peak resident memory stayed within budget, and the output exists and is valid. The block below reads back the result windowed (never whole-plane) and asserts the structural properties that make it memory-safe for the next consumer, then reports peak RSS via resource.

import resource
import rasterio


def verify_export(path: str, max_peak_mb: float = 1024.0) -> None:
    """Assert the output is valid, tiled, and that peak RSS stayed bounded."""
    with rasterio.open(path) as ds:
        assert ds.width > 0 and ds.height > 0, "output has zero size"
        # Tiled output is the precondition for memory-safe downstream reads.
        assert ds.block_shapes[0][1] != ds.width, "output is striped, not tiled"
        # Windowed read of the first block proves pixels are readable cheaply.
        _, first = next(ds.block_windows(1))
        sample = ds.read(1, window=first)
        assert sample.size > 0, "first block is empty"

    # ru_maxrss is KB on Linux, bytes on macOS; normalise to MB for Linux.
    peak_mb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.0
    assert peak_mb < max_peak_mb, f"peak RSS {peak_mb:.0f} MB exceeded budget"
    print(f"export OK: tiled, readable, peak RSS {peak_mb:.0f} MB")

Run this in the same process as the export so ru_maxrss reflects the export’s real high-water mark. A peak that tracks the whole-raster size means a full-plane read slipped back in somewhere; a peak near one block plus the cache means the windowing held.

# When to escalate

The windowed export bounds raster memory, but if the blow-up is upstream of the writer, escalate to the stage that actually owns the data:

• The OOM is in point-cloud rasterization, not the GeoTIFF write. If pdal or the dense-matching stage is what dies, the raster fix cannot help — stream the cloud instead, following streaming LAS tiles to avoid out-of-memory kills and the broader memory management for large point clouds guide.
• The export succeeds but the result is empty or black, not an OOM. That is a different fault entirely; return to the troubleshooting DEM and raster export failures guide and work the metadata-inspection step.
• Memory is bounded but the write is now hours long. The bottleneck has moved from RAM to I/O or compression; tune block size, compression level, and overview strategy in the exporting Cloud-Optimized GeoTIFFs with rasterio guide.

# Related

• Troubleshooting DEM and Raster Export Failures
• GDAL vs Rasterio for DEM and Orthomosaic Export
• Streaming LAS Tiles to Avoid Out-of-Memory Kills

← Troubleshooting DEM and Raster Export Failures