Resolving PyODM Connection Refused to NodeODM

Node("localhost", 3000) raises NodeConnectionError, the container is listed as running, and curl from your shell returns a perfectly good JSON info document. The disagreement is real and it is almost always about which network namespace the two ends are in, or about when — the service is up but not yet listening.

This page separates the failure modes by their symptom at the socket level, because “cannot connect” covers three distinct conditions that no amount of retrying will fix if you have the wrong one.

Three failures wearing one message

Connection refused means the TCP SYN reached a host and nothing was listening on that port. The host is reachable; the service is not there. Causes: the container is not running, the port is not published, or the port is published on a different interface than the one being dialled.

Connection reset or a hang followed by a timeout means something accepted and then dropped, or nothing answered at all. Causes: a firewall dropping packets silently, the wrong host entirely, or a service still initialising and not yet accepting.

HTTP-level errors — a 404 or an unexpected payload — mean you connected to something, and it is not NodeODM. Usually another service on the same port, or a proxy in front of it.

The distinction matters because only the third condition tells you the network is fine. The first two are network problems and retrying them without changing anything is a way of waiting.

Where a connection dies, and what each stage rules out A connection attempt drawn as four sequential stages: name resolution, TCP handshake, HTTP request, and the NodeODM info response. Beside each stage is the error it produces when it fails and what that failure rules out. A resolution failure names the host; a refused handshake means the host is reachable and nothing is listening; a timeout means packets are being dropped; an unexpected HTTP payload means a different service answered. A note observes that only the last of the four proves the network path is sound, so the first three cannot be fixed by retrying. resolve host DNS or /etc/hosts TCP handshake port 3000 HTTP GET /info NodeODM answers version, task slots name not resolved wrong hostname, or no service DNS in the container network refused / timeout refused: reachable, nothing listening timeout: dropped wrong service 404 or unexpected payload — the network path is fine Only the rightmost failure proves the path works. The other two are not waiting problems. A retry loop around a refused connection is a way of failing more slowly.

Figure 1 — Reading the failure by where it occurred. The socket error is more informative than the exception PyODM raises around it, and it is available.

Minimal reproducible solution

Probe the layers separately, in order, and stop at the first one that fails.

import socket
import urllib.request
import json


def probe(host: str, port: int, timeout: float = 3.0) -> str:
    """Return a diagnosis string naming the first layer that failed."""
    try:
        addrs = socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
    except socket.gaierror as exc:
        return f"resolve: {host} did not resolve ({exc})"

    family, socktype, proto, _, sockaddr = addrs[0]
    with socket.socket(family, socktype, proto) as sock:
        sock.settimeout(timeout)
        try:
            sock.connect(sockaddr)
        except ConnectionRefusedError:
            return (f"tcp: {sockaddr[0]}:{port} refused — host is reachable, "
                    "nothing is listening on that port")
        except socket.timeout:
            return (f"tcp: {sockaddr[0]}:{port} timed out — packets are being "
                    "dropped; check firewalls and the network the container is on")
        except OSError as exc:
            return f"tcp: {exc}"

    try:
        with urllib.request.urlopen(
                f"http://{host}:{port}/info", timeout=timeout) as resp:
            body = json.loads(resp.read())
    except Exception as exc:
        return f"http: connected, but /info failed ({exc})"

    if "version" not in body:
        return f"http: something answered on {port} and it is not NodeODM: {body!r}"
    return f"ok: NodeODM {body['version']}, {body.get('maxParallelTasks', '?')} slots"

Each branch names both the fact and its implication, which is what turns a support conversation into a fix. “Refused” and “timed out” look similar in a stack trace and mean opposite things about the network in between.

Edge-case matrix

Situation Probe says Fix
Container not started tcp: refused Start it; check the exit code of the previous run
Port not published tcp: refused Add -p 3000:3000 to the run command
Published on 127.0.0.1 only refused from another host Publish on 0.0.0.0 or use the right interface
Client inside another container resolve fails Use the service name on a shared network, not localhost
Service still initialising refused then works Wait for readiness, do not fix anything
Firewall between hosts tcp: timed out Open the port; retrying will not
Reverse proxy in front http: unexpected payload Point at the service, or fix the proxy route
Two NodeODM instances ok but tasks vanish You are submitting to one and polling another

The row that catches almost everyone running the client in a container is the fourth: localhost inside a container is that container, not the host and not a sibling. On a shared user-defined network the address is the other container’s service name; from a container to the host it is the platform’s host gateway alias.

What "localhost" means from three vantage points Three network positions and the address each must use to reach a NodeODM container. From a shell on the host, localhost with the published port works. From inside a sibling container on the same user-defined network, localhost refers to that container itself and the correct address is the NodeODM service name on the internal port. From inside a container with no shared network, neither works and the container must be attached to the network first. A note states that the published-port mapping applies only to traffic arriving from the host side. NodeODM listening on 3000 published as host 3000 shell on the host localhost:3000 works via the published port sibling container localhost is itself use nodeodm:3000 internal port, not published container off the network neither address resolves attach it to the network no port mapping helps The published port maps host traffic only; container-to-container traffic uses the internal port.

Figure 2 — Three vantage points, three correct addresses. A published port is not a global address, which is why a client that works from a shell fails from a worker container.

Verification snippet

The right shape for a client is not a retry loop but a bounded readiness wait, because the overwhelmingly common transient case is a service that will be up in a few seconds and is not yet.

import time


def wait_for_node(host: str, port: int, timeout_s: float = 90.0,
                  interval_s: float = 2.0):
    """Block until NodeODM answers, or fail with the last real diagnosis.

    Retries only the conditions that can resolve on their own. A wrong host
    or a wrong service is reported immediately rather than waited on.
    """
    deadline = time.monotonic() + timeout_s
    last = ""
    while time.monotonic() < deadline:
        last = probe(host, port)
        if last.startswith("ok:"):
            from pyodm import Node
            return Node(host, port)
        if last.startswith(("resolve:", "http: something answered")):
            raise RuntimeError(f"not transient — {last}")
        time.sleep(interval_s)
    raise TimeoutError(f"NodeODM not ready after {timeout_s:.0f}s — {last}")

Distinguishing transient from permanent inside the wait is the whole point. A refused connection during the first seconds of a container’s life is expected and resolves; an unresolvable hostname never will, and waiting ninety seconds to say so wastes ninety seconds on every misconfigured deployment.

Two operational notes are worth adding to any deployment that runs this unattended. Log the resolved address and port alongside the diagnosis on every attempt, not only on failure — a client that silently connected to a stale node is otherwise indistinguishable from one that connected correctly. And treat the node’s reported version as part of the run manifest: a fleet where two nodes drift to different engine versions produces two subtly different reconstructions from identical inputs, and the connection layer is the only place that difference is visible before the outputs are compared.

When to escalate

  • The probe reports ok and task submission still fails. The connection is sound and the problem is elsewhere — most often the payload: an image set larger than the node’s configured upload limit, which surfaces as a reset mid-upload rather than as a connection failure.
  • It works, then stops after a few hours. The node has hit its task limit and is refusing new work rather than refusing connections. Poll /info for available slots and treat exhaustion as backpressure, which is what the scheduler’s admission control exists to handle.
  • Tasks are submitted successfully and never appear. Two nodes are in play — a stale one from an earlier container and the current one — and the client is round-robining. Pin the address rather than relying on a name that resolves to more than one container.

A final note on where this check belongs. Running the probe once at worker startup, rather than at the first submission, means a misconfigured deployment fails before it has accepted any work — which is the difference between one clear error at boot and a queue full of jobs that each failed after being claimed. The scheduler’s health check and this probe are the same call, and wiring them together costs nothing beyond deciding to do it.

Where several nodes are available, probing all of them at startup and recording which answered also gives the scheduler a real capacity figure rather than an assumed one, which is the input its admission control needs and the number most fleets never actually measure.

← Setting Up OpenDroneMap with Python

A bounded readiness wait against a blind retry loop Two client behaviours against the same service startup. The blind retry loop retries every failure identically, so a service that will never appear because of a wrong hostname consumes the full timeout before reporting. The bounded readiness wait classifies each probe: a refused connection is retried because it resolves on its own once the service binds, while an unresolvable hostname or an unexpected service is reported immediately. A timeline shows the service becoming ready after eight seconds, at which point both succeed, and a second timeline shows a misconfigured host where only the classifying client fails fast. service becomes ready at 8 s — both succeed refused, retrying connected either client hostname is wrong — only one client says so blind retry retrying an unresolvable name for the full 90 s classifying wait fails "not transient — host did not resolve" Retry what can resolve itself; report what cannot. The classification costs one branch and turns a ninety-second silence into an immediate, actionable message.

Figure 3 — The wait that is worth writing. Most connection failures at startup are genuinely transient; the ones that are not deserve to fail in a second rather than in a minute and a half.