Fixing Permission Denied on Container Output Directories

A reconstruction completes, writes 80 GB of results, and the operator cannot delete them: every file is owned by root. Or the reverse — the container starts, runs for a minute and exits with Permission denied trying to create its output directory on a mount that is plainly writable from the host.

Both are the same mismatch seen from two directions. A container process has a numeric user id, the host filesystem enforces ownership by numeric id, and nothing reconciles them unless somebody arranges it. This page covers arranging it, for both the running worker and the files it leaves behind. It completes the operational detail in containerising photogrammetry workers with Docker.

# Why the ids do not line up

A container image defines a user — often root, sometimes an application user created at build time with an arbitrary id such as 1000 or 999. That id is what the kernel sees when the process writes to a bind-mounted host directory, and the host filesystem has no notion of “inside the container”.

Three consequences follow. A container running as root writes root-owned files, which a non-root host user cannot delete. A container running as uid 999 cannot write to a directory owned by uid 1000 unless the permissions allow it. And a chown applied afterwards needs elevation, which is precisely what the setup was trying to avoid.

The fix is to run the container as the id that owns the output directory, supplied at run time rather than baked into the image — because the right id depends on the host, and an image should not.

Figure 1 — Three ids, three outcomes, one comparison in the kernel.

# Minimal reproducible solution

import os
from pathlib import Path

import docker


def run_as_owner(image: str, command: list[str], *, output_dir: str,
                 mounts: list) -> dict:
    """Run a container as the user that owns the output directory.

    Reading the ids from the directory rather than from the calling process
    handles the case where a scheduler runs as a service account while the
    output belongs to a project group — which is the usual arrangement on a
    shared machine.
    """
    out = Path(output_dir)
    out.mkdir(parents=True, exist_ok=True)
    stat = out.stat()

    client = docker.from_env()
    container = client.containers.run(
        image, command=command, mounts=mounts, detach=True,
        user=f"{stat.st_uid}:{stat.st_gid}",
        group_add=[str(g) for g in os.getgroups()[:16]],
        environment={"HOME": "/tmp"})
    status = container.wait()
    logs = container.logs(tail=200).decode("utf-8", "replace")
    container.remove()
    return {"exit_code": status["StatusCode"], "ran_as": f"{stat.st_uid}:{stat.st_gid}",
            "logs": logs}

Setting HOME=/tmp is a small but necessary detail. A container run as an arbitrary uid has no entry in the image’s password file, so HOME is unset and libraries that write caches into it — matplotlib, pip, several GDAL configurations — fail with errors that name a cache directory rather than a permission problem.

# Making the image tolerate any uid

An image that only works for one uid is fragile. Building it so any numeric id can run it removes the coupling entirely.

# Create the application directories with group write, and make the group
# root (gid 0), which every container user belongs to regardless of uid.
RUN mkdir -p /app /data/scratch \
    && chgrp -R 0 /app /data/scratch \
    && chmod -R g+rwX /app /data/scratch

# No USER directive pinning a numeric id; the run supplies it.
WORKDIR /app
ENV HOME=/tmp \
    XDG_CACHE_HOME=/tmp/.cache \
    MPLCONFIGDIR=/tmp/.mpl

Assigning group ownership to gid 0 is the pattern used by platforms that run containers as arbitrary uids. Every user is in group 0 inside a container, so group-writable directories are writable by whichever id the run specifies, without the image knowing anything about it.

Figure 3 — Three fixes, two of which are still correct a year later.

# Edge-case matrix

The Docker-created-directory row is the sharpest edge. If a bind mount’s host path does not exist, Docker creates it as root, and a subsequent run as a normal user then fails on a directory that the pipeline itself caused. Creating mount paths explicitly on the host before the run removes it.

from pathlib import Path


def prepare_mount_paths(paths: list[str], *, uid: int, gid: int,
                        mode: int = 0o775) -> None:
    """Create mount targets on the host with the right owner before running.

    Letting Docker create a missing mount path produces a root-owned
    directory that the container then cannot write to — a failure the
    pipeline caused itself.
    """
    import os
    for p in paths:
        path = Path(p)
        path.mkdir(parents=True, exist_ok=True)
        try:
            os.chown(path, uid, gid)
        except PermissionError:
            if path.stat().st_uid != uid:
                raise PermissionError(
                    f"{path} is owned by uid {path.stat().st_uid}, not {uid}, "
                    "and cannot be changed — create it as the right user")
        path.chmod(mode)

# Verification snippet

import subprocess
from pathlib import Path


def verify_output_ownership(output_dir: str, *, expect_uid: int,
                            expect_gid: int, sample: int = 200) -> dict:
    """After a run, confirm the results are owned by the right account."""
    files = list(Path(output_dir).rglob("*"))[:sample]
    wrong = [str(f) for f in files
             if f.is_file() and (f.stat().st_uid != expect_uid
                                 or f.stat().st_gid != expect_gid)]
    return {"checked": len(files), "wrong_owner": len(wrong),
            "examples": wrong[:5],
            "ok": not wrong,
            "note": ("output is owned correctly" if not wrong else
                     "some output is owned by another account — the container user "
                     "did not match the output directory")}

Running this immediately after a job, rather than when somebody tries to delete the results a week later, is the difference between a one-line fix and an operator with a permissions problem and no context.

Figure 2 — Why the fix belongs at run time rather than afterwards.

# Why not simply run everything as root

Running containers as root makes all of this go away, which is why it is so common, and it is worth being explicit about what it costs beyond the undeletable output.

A root process in a container that escapes its isolation is root on the host. The escape routes are rarer than they used to be and they are not zero, and a photogrammetry worker processing files supplied by a client is exactly the kind of workload that parses untrusted input with large C++ libraries.

Root also removes a useful safety property. A worker running as an ordinary user cannot modify the survey it was given even if a bug tries to, because the mount is read-only and the process lacks the privilege to override it. As root, only the mount flag stands between a buggy pipeline and the original imagery.

The practical position is that running as a normal user costs the few lines on this page and removes both problems, so there is no case for root in a processing worker. Where a base image insists on it — some vendor images do — that is a reason to look for a different base rather than to accept it.

# When to escalate

• The output lives on a network filesystem with squashed ownership. The export options decide the answer, not the container. Check them before debugging anything locally.
• A tool inside the image insists on writing to a fixed path. Mount a writable directory at that path rather than running as root to satisfy it.
• Rootless Docker is in use. Ids are remapped between the container and the host, so the numbers do not match directly. Verify from the host side and use the remapped values.

# Related

• Containerising Photogrammetry Workers with Docker
• Mounting Large Datasets Without Copying Them
• Fixing “ODM Cannot Find Images” in the Project Path

← Containerising Photogrammetry Workers with Docker