Resolving ODM Exit Code 1 and 137 (Out of Memory)

An unattended OpenDroneMap (ODM) run dies twelve hours in, the container is gone, and subprocess hands you back a bare integer: 137. Sometimes it is 1 instead, thrown during densification or meshing with a truncated stack trace. Both leave the same forensic vacuum — no orthophoto, no obvious message, just a code — and the two demand opposite fixes, so guessing wastes another twelve-hour run. Exit 137 is the kernel out-of-memory (OOM) killer sending SIGKILL to a container that exceeded its memory cap; exit 1 is a pipeline error inside ODM, frequently a stage that ran out of a resource other than the memory ceiling or hit a genuine data fault. This page shows how to read those codes correctly, inspect the actual memory budget, and tune the flags that bound peak RAM so the next run finishes. It is the memory-failure deep dive for setting up OpenDroneMap with Python.

# Why ODM containers die with 137 and 1

The exit code is the number 128 + signal for killed processes, so 137 is 128 + 9 — SIGKILL, which on Linux is overwhelmingly the OOM killer reclaiming memory when a container’s working set breaches its cgroup limit. ODM’s memory demand is dominated by two stages: OpenMVS densification (which holds depth maps and the growing dense cloud in RAM) and meshing (which builds a global surface). Both scale super-linearly with --pc-quality and with the number of images per submodel, so a block that reconstructs fine at medium can be OOM-killed at high on the same hardware. Exit 137 therefore almost always means “the memory cap was too low for the quality and block size,” and its four levers are the Docker --memory limit, ODM’s --max-concurrency (more parallel workers multiply peak RAM), --feature-quality/--pc-quality, and the block size you feed a single submodel.

Exit 1 is different. It is ODM’s own non-zero return from a stage that failed for a reason the process could name — a genuinely broken input, an alignment stage that produced no reconstruction, or a resource exhaustion that raised a Python exception rather than triggering the kernel killer (disk-space exhaustion on the scratch volume is a common one). The trap is treating a 1 as an OOM and throwing more memory at it, which changes nothing because memory was never the constraint. The fix always starts with reading the code correctly, then the log tail, and only then tuning.

Figure 1 — The exit code answers a question the log cannot: whether the process failed or was killed. Those two have disjoint causes and disjoint fixes, and the code is available before any log is read.

# Minimal reproducible solution

The wrapper below runs ODM, classifies the exit code deterministically, and — for a 137 — re-runs once with memory-bounding flags derived from the live RAM budget instead of guessing. It sets --memory and --memory-swap to the same value so an over-budget run fails fast rather than thrashing on swap for hours, which is what turns an OOM into a wall-clock catastrophe.

import subprocess
import logging
import psutil
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
GIB = 1024 ** 3

def build_cmd(datasets: Path, project: str, mem_gb: int,
              conservative: bool) -> list[str]:
    """docker run vector; conservative mode bounds peak RAM for a 137 retry."""
    cmd = [
        "docker", "run", "--rm",
        f"--memory={mem_gb}g", f"--memory-swap={mem_gb}g",   # equal -> no swap
        "-v", f"{datasets}:/datasets",
        "opendronemap/odm:3.5.4", "--project-path", "/datasets", project,
        "--dsm", "--orthophoto-resolution", "2.0",
    ]
    if conservative:
        cmd += [
            "--pc-quality", "medium",       # densification is the RAM peak
            "--feature-quality", "medium",
            "--max-concurrency", "4",       # fewer parallel workers = less RAM
            "--split", "400", "--split-overlap", "100",  # bound submodel size
        ]
    return cmd

def run_odm(datasets: Path, project: str, timeout_s: int = 86_400) -> Path:
    """Run ODM, classify the exit code, and retry a 137 with bounded memory."""
    budget = int(psutil.virtual_memory().available / GIB) - 2   # leave headroom
    for conservative in (False, True):
        cmd = build_cmd(datasets, project, budget, conservative)
        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout_s)
        if proc.returncode == 0:
            return datasets / project / "odm_orthophoto" / "odm_orthophoto.tif"
        tail = (proc.stderr or proc.stdout)[-1500:]
        if proc.returncode == 137:
            logging.error("Exit 137 (OOM kill). Retrying with bounded memory.\n%s", tail)
            continue                         # retry loop injects conservative flags
        # Exit 1 (or anything else) is NOT an OOM; do not throw memory at it.
        raise RuntimeError(f"ODM exit {proc.returncode} (not OOM):\n{tail}")
    raise RuntimeError("ODM OOM-killed even with conservative flags; split the area.")

The design choice that matters is the branch on proc.returncode: only a 137 triggers the memory-bounded retry, while a 1 raises immediately with the log tail so you diagnose the real fault instead of wasting a second run. The conservative flag set lowers --pc-quality (the single biggest densification RAM driver), caps --max-concurrency so fewer worker threads hold depth maps at once, and enables --split so no submodel is large enough to breach the cap. When even the conservative pass is OOM-killed, the block itself is too big for the machine and must be split geographically — the same submodel strategy detailed in setting up OpenDroneMap with Python.

Figure 2 — “The machine has 64 GB free” is compatible with all three of these limits being exceeded. Each boundary is set in a different place, and only the innermost one is invisible to every host-level monitor.

# Edge-case matrix

The exit code, read together with the stage that failed, points at the correct lever. Confusing 137, 1, and 139 is the most common reason a fix does nothing.

Figure 3 — Every one of these reduces peak memory, and they are not equivalent. The cheapest lever costs only time; the one most often reached for first costs resolution the client paid for.

# Verification snippet

After a run, verify the reason it succeeded or failed rather than trusting the bare integer — assert on the exit code, confirm the OOM signature in the log when you expect it, and check the orthophoto materialized.

from pathlib import Path

def classify_exit(returncode: int, log_tail: str) -> str:
    """Turn a bare exit code plus log tail into an actionable class."""
    if returncode == 0:
        return "success"
    if returncode == 137:
        return "oom_kill"          # 128 + 9 (SIGKILL): bound memory / split
    if returncode == 139:
        return "segfault"          # 128 + 11: native crash, not memory
    if returncode == 1 and "No space left on device" in log_tail:
        return "disk_full"         # exit 1 masquerading as a memory problem
    if returncode == 1:
        return "pipeline_error"    # read the tail; not an OOM
    return f"unknown_{returncode}"

def verify_success(ortho: Path, returncode: int) -> None:
    assert returncode == 0, f"ODM did not exit clean: {classify_exit(returncode, '')}"
    assert ortho.exists() and ortho.stat().st_size > 0, "orthophoto missing/empty"
    print(f"verified {ortho.name}")

The disk_full branch is worth its own line: a scratch volume filling during densification produces an exit 1 that looks nothing like an OOM, and no amount of memory tuning fixes it — you free disk instead. Classifying before acting is the whole point.

# When to escalate

Escalate from this page when memory tuning has hit its floor or the code is not actually an OOM:

• Conservative flags still OOM-kill the run. The block exceeds what the machine can hold at any quality. Split it geographically and merge submodels, following the memory-aware wrapper in setting up OpenDroneMap with Python, or move densification to the streaming approach in memory management for large point clouds.
• The kill happens outside ODM, on the point-cloud stage of your own pipeline. That is a downstream RAM problem; the tile-streaming technique in streaming LAS tiles to avoid out-of-memory kills keeps peak memory flat.
• You get exit 1, not 137, with a divergence or “no reconstruction” message. That is not a memory failure at all — it is a reconstruction-quality failure covered by fixing OpenSfM bundle adjustment divergence.

# Related

• Setting Up OpenDroneMap with Python
• Fixing OpenSfM Bundle Adjustment Divergence
• Memory Management for Large Point Clouds

← Setting Up OpenDroneMap with Python