Choosing Between NodeODM Workers and Local Runs

Two architectures process the same survey with the same engine and produce the same outputs. One runs a long-lived service that accepts jobs over HTTP, queues them and reports progress; the other invokes a container per job and watches the exit code. They differ in almost everything except the result.

The choice is usually made by whichever was set up first, and it is worth making deliberately, because switching later means rewriting the orchestration layer. This page sets out what each costs and where each fits, extending the interface comparison in pyODM vs direct ODM CLI for automation.

# What each architecture actually is

A processing service is a long-running container exposing a job API. A client uploads imagery, receives a task id, polls for progress and downloads results. It owns its own queue, reports percentage progress, and survives the client disconnecting.

A direct run launches a container per job with the imagery bind-mounted, waits for it to exit, and reads the results from a mounted directory. There is no upload, no queue and no progress beyond the log.

The differences that matter are not about the engine. They are about where the queue lives, whether the imagery moves, and what happens when something fails.

Figure 1 — Where the survey goes, which is the difference that decides most deployments.

# Minimal reproducible solution

A single interface over both keeps the decision reversible, which is worth more than picking correctly the first time.

from abc import ABC, abstractmethod
from pathlib import Path


class Backend(ABC):
    """One interface, so the architecture choice is a configuration value."""

    @abstractmethod
    def submit(self, survey_dir: str, output_dir: str, options: dict) -> str: ...

    @abstractmethod
    def poll(self, job_id: str) -> dict: ...


class ServiceBackend(Backend):
    """Submits to a long-running processing node over HTTP."""

    def __init__(self, host: str, port: int = 3000):
        from pyodm import Node
        self._node = Node(host, port)
        self._tasks: dict[str, object] = {}

    def submit(self, survey_dir: str, output_dir: str, options: dict) -> str:
        images = sorted(str(p) for p in Path(survey_dir).glob("*.[jJ][pP][gG]"))
        task = self._node.create_task(images, options)
        self._tasks[task.uuid] = (task, output_dir)
        return task.uuid

    def poll(self, job_id: str) -> dict:
        task, output_dir = self._tasks[job_id]
        info = task.info()
        if str(info.status).endswith("COMPLETED"):
            task.download_assets(output_dir)
        return {"status": str(info.status), "progress": info.progress,
                "message": getattr(info, "last_error", "")}


class DirectBackend(Backend):
    """Launches a container per job with the imagery mounted in place."""

    def __init__(self, image: str):
        import docker
        self._client = docker.from_env()
        self._image = image
        self._containers: dict[str, object] = {}

    def submit(self, survey_dir: str, output_dir: str, options: dict) -> str:
        import docker
        args = []
        for key, value in options.items():
            args += [f"--{key}", str(value)]
        container = self._client.containers.run(
            self._image, command=["--project-path", "/data", "job", *args],
            mounts=[docker.types.Mount("/data/job/images", str(Path(survey_dir).resolve()),
                                       type="bind", read_only=True),
                    docker.types.Mount("/data/job/odm_orthophoto",
                                       str(Path(output_dir).resolve()), type="bind")],
            detach=True)
        self._containers[container.id] = container
        return container.id

    def poll(self, job_id: str) -> dict:
        container = self._containers[job_id]
        container.reload()
        state = container.attrs["State"]
        return {"status": state["Status"],
                "progress": None,                    # no progress without log parsing
                "message": state.get("Error", "")}

The asymmetry in poll is the honest part: a service reports a percentage and a direct run does not, unless somebody parses the engine’s log output. That is the single largest operational difference between the two.

Figure 3 — The crossover is fleet size, not preference.

# Edge-case matrix

The last row is the deciding one for many deployments. A service can accept work from anywhere because it takes the imagery with the request; a direct run needs the worker and the storage to see the same filesystem, which on a single site is trivial and across sites is not.

# Verification snippet

import time


def compare_backends(backend_a: Backend, backend_b: Backend,
                     survey_dir: str, out_a: str, out_b: str,
                     options: dict) -> dict:
    """Run the same survey both ways and compare timings and outputs.

    Worth doing once on representative data: the transfer overhead is easy to
    estimate and the queueing and start-up costs are not, and on small jobs
    they dominate.
    """
    results = {}
    for name, backend, out in (("a", backend_a, out_a), ("b", backend_b, out_b)):
        start = time.perf_counter()
        job = backend.submit(survey_dir, out, options)
        while True:
            state = backend.poll(job)
            if state["status"].lower() in {"completed", "exited", "failed"}:
                break
            time.sleep(10)
        results[name] = {"elapsed_s": round(time.perf_counter() - start),
                         "final_status": state["status"]}
    ratio = results["a"]["elapsed_s"] / max(results["b"]["elapsed_s"], 1)
    return {**results, "ratio": round(ratio, 2)}

Figure 2 — Where the overhead bites, which is on the jobs a monitoring programme runs most.

# Failure handling, which differs more than the happy path

Both architectures process a successful job equally well. They diverge when something goes wrong, and that divergence is worth understanding before a fleet depends on either.

A service reports failure as a status and an error message on the task. That is convenient and lossy: the message is whatever the engine surfaced, and the working directory that would explain it lives inside the node, where the orchestrator cannot reach it. Recovering the detail means logging into the node, and on a node that has since started three other tasks, the directory may be gone.

A direct run fails as a non-zero exit code with the container’s full log available and, crucially, the scratch mount still on disk. Every intermediate the engine wrote is there to look at, which turns most diagnoses into a file listing.

def capture_failure(backend: str, job_id: str, *, scratch_dir: str | None,
                    logs: str, exit_code: int | None) -> dict:
    """What can be preserved from a failed job, by architecture."""
    record = {"backend": backend, "job_id": job_id, "exit_code": exit_code,
              "log_tail": logs[-4000:]}
    if backend == "direct" and scratch_dir:
        from pathlib import Path
        record["intermediates"] = sorted(
            str(p.relative_to(scratch_dir)) for p in Path(scratch_dir).rglob("*")
            if p.is_file())[:200]
        record["note"] = "scratch retained; stage outputs are inspectable"
    else:
        record["note"] = ("service task failed; the working directory is inside the "
                          "node and may not survive the next task")
    return record

Retaining scratch on failure and clearing it on success is a small scheduler rule that makes the direct architecture substantially easier to operate, and it has no equivalent on the service side without access to the node.

# Which to choose

Choose a service when work arrives from machines that do not share storage with the workers, when built-in progress reporting matters to a user interface, or when the operational simplicity of one long-running component outweighs the transfer cost.

Choose direct runs when the imagery already sits on storage the workers can see, when precise per-job resource limits matter, or when the orchestration layer already has a queue — which, on any fleet running the discipline in orchestrating photogrammetry jobs with Python schedulers, it does.

The hybrid that works for larger operations is direct runs for scheduled bulk processing and a service for interactive or externally submitted work, behind the same interface so the choice is a routing decision rather than an architecture.

# When to escalate

• Transfer time dominates and a service is required. Put the node next to the storage, so the upload is a local copy rather than a network transfer.
• Progress reporting is needed from direct runs. Parse the engine’s stage output into a progress estimate; it is coarse but adequate for a user interface.
• Failures are hard to diagnose either way. The common remedy is keeping the scratch directory after a failure, which direct runs make trivial and a service usually does not.
• Both architectures are in use and results disagree. Compare the engine versions before anything else; two deployments drift apart the moment one of them is updated independently of the other.

# Related

• pyODM vs Direct ODM CLI for Automation
• Resolving pyODM Connection Refused to NodeODM
• Containerising Photogrammetry Workers with Docker

← pyODM vs Direct ODM CLI for Automation