Skip to content
Peppy

Containers

Containers package a node and all of its dependencies into a single, self-contained image. A containerized node runs identically regardless of what is installed on the host: no more “works on my machine” issues.

Use containers when you need:

  • Portability: ship a node to another machine without worrying about system dependencies.
  • Reproducibility: guarantee the same runtime environment every time.
  • Isolation: prevent conflicts between nodes that need different versions of the same library.

Peppy uses Apptainer as its container runtime. On macOS, Apptainer runs transparently inside a Lima virtual machine; no extra setup is needed.

Setup (Linux)

Section titled “Setup (Linux)”

On Linux, Apptainer uses unprivileged user namespaces which may require a one-time system configuration. The installer handles this automatically, but if you skipped it or installed Peppy manually you can run:

Terminal window
peppy container setup

This configures the following (prompting for sudo when needed):

  1. uidmap package: installs newuidmap (required for fakeroot mode).
  2. AppArmor profile (on systems that restrict unprivileged user namespaces via AppArmor): installs a profile that allows Apptainer to create user namespaces and loads it into the kernel. When the profile already exists but is not loaded, it is reloaded without being rewritten.

To check the current state without making any changes:

Terminal window
peppy container status

This prints a pass/fail summary of each prerequisite and exits with code 0 (all pass) or 1 (something needs fixing).

Initializing a container node

Section titled “Initializing a container node”

Pass the --container flag to peppy node init:

Terminal window
peppy node init --toolchain uv --container my_node

This generates the same project scaffolding as a regular node, plus an apptainer.def file that describes how the container image is built.

The peppy.json5 configuration

Section titled “The peppy.json5 configuration”

A container node includes a container block inside its execution section instead of the usual build_cmd and run_cmd fields. The two are mutually exclusive: a node is either a container node or a process node, never both.

peppy.json5
{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
    },
  }
}

The def_file field points to the Apptainer definition file relative to the node root. You can rename or relocate it as long as def_file matches.

Compare this with a standard process node, which defines build_cmd and run_cmd instead:

peppy.json5 (process node)
{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "python",
    build_cmd: ["uv", "sync", "--no-editable"],
    run_cmd: ["./.venv/bin/python", "-m", "my_node"]
  }
}

Container nodes don’t need build_cmd or run_cmd; the definition file takes care of both building and running the node.

The apptainer.def file

Section titled “The apptainer.def file”

The generated definition file is a standard Apptainer definition file. Here is what peppy node init --container generates:

apptainer.def
Bootstrap: docker
From: peppybot/python-uv-base:latest


%labels
    Name my_node
    Version v1


%environment
    export PATH="/opt/my_node/.venv/bin:$PATH"


%files
    . /opt/my_node


%post
    set -eux
    export DEBIAN_FRONTEND=noninteractive


    export UV_PYTHON_INSTALL_DIR=/opt/uv-python


    cd /opt/my_node
    uv python install
    uv sync --no-editable


%runscript
    cd /opt/my_node
    exec ./.venv/bin/python -m my_node

Each section serves a specific purpose:

SectionPurpose
Bootstrap / FromBase image to build from (Ubuntu 24.04 by default)
%labelsMetadata embedded in the image
%environmentEnvironment variables set when the container runs
%filesCopies the node source into the image at /opt/<node_name>
%postBuild steps: install system packages, toolchains, and compile the node
%runscriptEntry point executed when the container starts

Adding a container node

Section titled “Adding a container node”

Adding a container node works the same as a regular node: first stage it, then build.

Terminal window
peppy node add ./my_node
peppy node build my_node:v1

You can also combine both steps with peppy node add ./my_node --build (shorthand -b). If you’ve just edited peppy.json5, add --sync/-s as well, e.g. peppy node add ./my_node -sb to sync, add, and build in one shot.

Under the hood, Peppy runs apptainer build during the build phase to produce a .sif (Singularity Image Format) file. This replaces the build_cmd step used by process nodes; the entire build happens inside the container according to the %post section of the definition file.

The resulting .sif file is stored under ~/.peppy/built_nodes/<name>_<tag>/, named after the fingerprint of the staged sources (see Build caching), and is ready to be started.

Starting a container node

Section titled “Starting a container node”

Starting a container node also uses the same command:

Terminal window
peppy node run my_node:v1

Peppy runs the .sif image with apptainer run and a clean environment: the daemon’s own environment is not inherited. The instance’s declared env_vars (minus HOME, which is dropped) and PEPPY_RUNTIME_CONFIG are passed in explicitly, and the image’s %environment section applies as usual.

The container executes the %runscript section, which runs the compiled binary (Rust) or the Python module entry point.

Mounting host directories

Section titled “Mounting host directories”

By default, a container is isolated from the host filesystem. Use mount_paths to bind-mount host directories into the running container, useful for sharing datasets, persisting output, or exposing device files.

Add a mount_paths array to the container block inside execution in peppy.json5:

peppy.json5
{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "/data/models:/opt/models:ro",
        "/tmp/my_node_output:/output:rw"
      ]
    },
  }
}

Each entry follows the format host_path:container_path[:options]:

FormatExampleBehaviour
host_path"/data/models"Mounted at the same path inside the container
host_path:container_path"/data/models:/opt/models"Mounted at a different path inside the container
host_path:container_path:options"/data/models:/opt/models:ro"Mounted with explicit options (ro for read-only, rw for read-write)

host_path may start with ~, which expands to the home directory of the user running the daemon on the machine where the instance runs. In a federated launch, each machine expands ~ against its own home, so a single manifest works across machines with different usernames ("~/.cache/my_node:/cache:rw"). The ~user form is not supported and is rejected while the launch is being validated, before any stack is replaced. Only the host side is expanded: a ~ in container_path is passed to the container verbatim.

A source under /dev with no remap (no destination, or a destination equal to the source) is passed through rather than bound: host devices reach the container through Apptainer’s device mount, and only a remapped device bind emits an explicit --bind.

A missing mount source is created on the host as a directory before the container starts, and a warning names each auto-created path, so a typo in a file bind shows up as an unexpectedly created empty directory. Sources under the host runtime trees (/dev, /proc, /run, /sys) are used as-is and never created.

Using parameters in mount paths

Section titled “Using parameters in mount paths”

Mount paths can reference runtime parameters using the ${parameters:<path>} syntax. This lets each node instance mount a different host path based on its configuration.

peppy.json5
{
  peppy_schema: "node/v1",
  manifest: {
    name: "uvc_camera",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "rust",
    parameters: {
      device_path: "string",
    },
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "${parameters:device_path}:/dev/video0:rw"
      ]
    },
  },
}

When the node runs, ${parameters:device_path} is replaced with the actual value provided in the deployment configuration. For example, if the instance supplies device_path: "/dev/video2", the resulting bind mount is /dev/video2:/dev/video0:rw.

For nested parameters, use dot notation:

peppy.json5 (nested example)
{
  // ...
  execution: {
    // ...
    parameters: {
      video: {
        device_path: "string",
        frame_rate: "u16",
      },
    },
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "${parameters:video.device_path}:/dev/video0:rw"
      ]
    },
  },
  // ...
}

Build caching

Section titled “Build caching”

Build artifact cache

Section titled “Build artifact cache”

Every peppy node build, including the ones peppy stack launch and peppy stack build run for each deployment, fingerprints the staged node tree before building: the node sources, peppy.json5, the def file, the generated .peppy/ code and the vendored peppylib, mixed with the peppy version, the host platform and the artifact kind. The artifact is stored under that fingerprint:

~/.peppy/built_nodes/<name>_<tag>/<fingerprint>.sif       container node
~/.peppy/built_nodes/<name>_<tag>/<fingerprint>.tar.zst   process node

When the fingerprint already has an artifact on disk, the build reuses it instead of running apptainer build (or build_cmd for a process node), and the build log carries one line:

Reusing cached build of my_node:v1 (fingerprint 3f9a1c2e5d7b8a4f) at ~/.peppy/built_nodes/my_node_v1/3f9a1c2e5d7b8a4f.sif

The cache is a plain directory tree keyed by content, so it survives daemon restarts and reboots, and a disk prepared on one machine carries its built nodes to every machine that boots from it with the same peppy version, OS and architecture. A process node’s archive also embeds what the host toolchain produced, which the fingerprint does not cover, so those machines need a compatible build environment as well. Each node identity keeps exactly one artifact: a build whose fingerprint differs replaces the previous one.

Pass --rebuild to peppy node build, peppy stack launch or peppy stack build to build from the staged sources even when a matching artifact exists; the result replaces it. Use it when something outside the fingerprint changed:

  • A floating base image tag (From: peppybot/rust-cargo-base:latest) that now resolves to a different image. Pin the image with @sha256:<digest> in the def file to make base image changes part of the fingerprint.
  • For a process node, the host toolchain or the environment build_cmd runs with.

--rebuild is independent of --force, which cancels an in-flight build of the same node.

Build caches inside container builds

Section titled “Build caches inside container builds”

Rust and Python container builds reuse a shared cache across builds, without any per-node configuration. The daemon bind mounts ~/.peppy/cache/container_build into the build at /peppy-cache and points the %post toolchain at it.

A Rust container build gets:

  • The crates.io registry in cargo-home/, via CARGO_HOME, so dependencies are downloaded once instead of on every build.
  • sccache caches rustc invocations in sccache-cache/, via RUSTC_WRAPPER and SCCACHE_DIR, so unchanged crates are not recompiled. The executable ships inside peppybot/rust-cargo-base, so nothing needs to be installed on the host, on Linux or macOS; the wrapper is activated only for defs that bootstrap from that image.

A Python container build gets:

  • The packages uv downloads and builds in uv-cache/, via UV_CACHE_DIR, so a rebuild installs every package it already downloaded from the cache.
  • The interpreter archives uv python install downloads in uv-python/, via UV_PYTHON_CACHE_DIR, so each Python version is downloaded once.
  • UV_LINK_MODE=copy: uv copies the files of the packages it installs from its cache into the image. The cache is a bind mount, where uv cannot hard-link into the image, and a copy keeps every file of the image independent of the host cache.

Both get PEPPY_DOWNLOAD_CACHE, which names /peppy-cache/downloads, a directory for the files a %post downloads. No tool reads it on its own: a def that pins a download by its sha256 looks the file up there under that checksum before it downloads it, and stores what it downloaded there once the checksum matches. A rebuild then reaches the network only for a pin it has not seen yet.

A def tests for PEPPY_DOWNLOAD_CACHE, not for /peppy-cache: the variable is set only when the cache is bind mounted, while /peppy-cache also exists as a plain directory of the base image, where anything written ends up in the built image. Because the directory is shared, a def checks a cached file against its pin before it uses it, and stores a file under a temporary name before it renames it, so concurrent builds never read a partial file:

Terminal window
# fetch_pinned URL SHA256 DEST
fetch_pinned() {
    local cached="${PEPPY_DOWNLOAD_CACHE:-}/$2" part
    if [ -n "${PEPPY_DOWNLOAD_CACHE:-}" ] && [ -f "$cached" ] \
        && echo "$2  $cached" | sha256sum -c --status; then
        cp "$cached" "$3"
        return
    fi
    curl -fsSL --retry 3 --retry-all-errors -o "$3" "$1"
    echo "$2  $3" | sha256sum -c -
    [ -n "${PEPPY_DOWNLOAD_CACHE:-}" ] || return 0
    part=$(mktemp "$PEPPY_DOWNLOAD_CACHE/.$2.XXXXXX") \
        && cp "$3" "$part" && mv -f "$part" "$cached" \
        || { rm -f "$part"; echo "warning: $1 not stored in the download cache"; }
}

Both also get PEPPY_NODE_BUILD_CACHE, which names /peppy-cache/nodes/<name>_<tag>: a directory of the node being built, which no build of another node uses. Peppy never reads it. A def keeps there what its next build can use, such as a part of the image that it builds again only when the inputs of that part change, stored under a hash of those inputs. A build of the node can start while an earlier one still ends (after a --force build, say), so a def stores a file there under a temporary name and then renames it, as with the downloads. Nothing prunes the directory: a def that stores a new entry removes the entries it no longer needs.

Caching is best effort and never fails a build: when the main directory of the cache of a build (cargo-home/ or uv-cache/) is unavailable, the build runs exactly as it would without caching, and when another part is unavailable, the build runs without that part only. Set PEPPY_NO_CONTAINER_BUILD_CACHE=1 in the daemon’s environment to disable caching entirely. The daemon also skips the cache for one build when the def file or apptainer_build_extra_args mention one of the markers of the build’s language:

LanguageMarkers
Rustrustup (installing it in %post is incompatible with the CARGO_HOME override), CARGO_HOME, RUSTC_WRAPPER, SCCACHE_DIR, SCCACHE_SERVER_PORT
PythonUV_CACHE_DIR, UV_PYTHON_CACHE_DIR, UV_LINK_MODE, --link-mode (the flag of uv overrides UV_LINK_MODE)

A Python def that installs rustup, to build a Rust extension for example, keeps its uv cache: the Python cache sets no cargo variable.

After each Python container build that uses the cache and succeeds, the daemon prunes uv-cache/ with uv cache prune, run with the uv of the image that the build wrote. Each build stages the local projects of the node again (the node itself, and .peppy/libs/peppylib with its native extension), so uv builds them again and keeps every earlier build of them, which no later build uses. The prune removes these. It also removes each file or directory at the top of uv-cache/ that is not a cache directory of uv, and each archive in archive-v0/ that no cache entry names; other files in the cache directories of uv stay. A line of the build output gives the result, for example Container build cache: uv cache pruned: Removed 15 files (6.8MiB).

A failed prune does not fail the build. A build that fails or is cancelled does not prune. A build prunes only when no other Python container build uses the cache; else a later build that succeeds prunes it. A Python build that starts while another build prunes waits until the prune ends, and a line of the build output says so: Container build cache: waiting for another build to prune the uv cache.

The prune runs the first uv on the PATH of the image that is not in a Python virtual environment, as %post does not see the virtual environment that a %environment puts on PATH. There is no prune when the image has no such uv, and the daemon stops a prune that runs for 30 seconds. uv also removes the entries of a cache layout that is not its own: nodes whose images have uv versions with different cache layouts remove each other’s entries.

PyPI mirror for Python builds

Section titled “PyPI mirror for Python builds”

Python builds download their packages from PyPI. When PyPI is slow from the host, set pypi_mirror in the daemon configuration to the packages base URL of a PyPI mirror, and restart the daemon:

pypi_mirror: "https://pypi.tuna.tsinghua.edu.cn/packages/",

Before a Python node builds, a container node or a process node alike, the daemon reads each uv.lock in the staged copy of the node. It collects the files that the lock pins by sha256 for packages from a registry, at a URL below https://files.pythonhosted.org/packages/. It asks the mirror for each of these files with a HEAD request, 8 at a time, and puts the mirror in the URL of each file that the mirror has with the size the lock records. Every other file keeps its PyPI URL, so a file that the mirror does not have yet still comes from PyPI.

Only the staged copy changes: the node sources stay as they are, and the build artifact cache keys the build by the staged tree as it was before the change, so a build finds the same artifact with and without a mirror. The image or the archive of the node holds the rewritten uv.lock.

A mirror that you do not control is safe: uv checks each file it downloads against the sha256 in the lock, and the lock comes from the repository of the node. A mirror can make a build slow or make it fail, but it cannot change what the build installs. uv accepts the rewritten lock with uv sync, uv sync --locked and uv sync --frozen, and the uv cache stays valid when you change or remove the mirror. Packages from other sources (a direct URL, git, a path), registry files on another host (such as the files that download.pytorch.org serves itself), and files that no lock pins do not use the mirror. Files that no lock pins include the build requirements uv installs to build the node and its .peppy/libs projects (such as hatchling and setuptools), the Python interpreter, GitHub release assets, apt packages and base image layers.

The presence check uses the proxy that ALL_PROXY, HTTPS_PROXY or HTTP_PROXY names in the environment of the daemon (HTTP or SOCKS), and skips the hosts in NO_PROXY, as uv does.

Each build that uses the mirror shows one line per uv.lock, which names the lock, as the build may not read every lock of the node (the lock of a test project, for example):

PyPI mirror: pypi.tuna.tsinghua.edu.cn; uv.lock: 868 of 870 locked files from the mirror, 2 from files.pythonhosted.org

A check that takes more than a few seconds also shows how many files it checked. When the mirror does not answer, when a uv.lock cannot be read or written, or when the check cannot run, the build shows a warning and downloads the files concerned from PyPI. A mirror that answers the HEAD request for a file but then fails to send the file makes uv fail the build: build again, or set pypi_mirror back to null.

A packages base URL is the URL under which a mirror keeps the files of PyPI with the path layout of https://files.pythonhosted.org/packages/: the path of a file after the packages base URL is the same on both. It is not the index URL of the mirror, which ends with /simple/. Some mirrors and their packages base URL:

MirrorLocationPackages base URL
OSSONetherlandshttps://pypi.osso.nl/packages/
YandexRussiahttps://mirror.yandex.ru/pypi/web/packages/
Tsinghua University (TUNA)Chinahttps://pypi.tuna.tsinghua.edu.cn/packages/
USTCChinahttps://mirrors.ustc.edu.cn/pypi/packages/
Nanjing UniversityChinahttps://mirror.nju.edu.cn/pypi/web/packages/
SJTUChinahttps://mirror.sjtu.edu.cn/pypi-packages/
Huawei CloudCDNhttps://repo.huaweicloud.com/repository/pypi/packages/

To check that a mirror has a file, take the URL of the file from your uv.lock, replace https://files.pythonhosted.org/packages/ with the packages base URL of the mirror, and send a HEAD request that follows redirects. The mirror has the file when the last answer is 200 with a Content-Length equal to the size in the lock, and does not come from files.pythonhosted.org: some mirrors redirect every file to PyPI, and the daemon then keeps the PyPI URL:

Terminal window
curl -sIL https://pypi.tuna.tsinghua.edu.cn/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl

A mirror can be hours behind PyPI, and PyPI is fast from most places, so set a mirror only when PyPI is slow from the host.

Extra runtime arguments

Section titled “Extra runtime arguments”

You can pass additional command-line arguments directly to Apptainer or Lima using apptainer_build_extra_args, apptainer_run_extra_args, and lima_shell_extra_args in the container block inside execution.

peppy.json5
{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
      apptainer_build_extra_args: ["--no-setgroups"],
      apptainer_run_extra_args: ["--no-setgroups"],
    },
  }
}
FieldPurpose
apptainer_build_extra_argsExtra flags appended to apptainer build (e.g., ["--no-setgroups"])
apptainer_run_extra_argsExtra flags appended to apptainer run (e.g., ["--no-setgroups"])
lima_shell_extra_argsExtra flags passed to limactl shell on macOS (ignored on Linux)

All fields are optional and default to an empty list when omitted.

Peppy packs every image it builds with zstd at level 3 (--mksquashfs-args "-comp zstd -Xcompression-level 3"), which packs a node image about ten times faster than Apptainer’s default gzip, at the same size. The extra build arguments come after that flag, and Apptainer reads the last --mksquashfs-args, so apptainer_build_extra_args: ["--mksquashfs-args", "-comp xz"] packs the image with xz.

A node that renders or computes on the host’s NVIDIA GPU passes apptainer_run_extra_args: ["--nv"]: Apptainer then binds the host’s driver libraries (CUDA, EGL, GLX, NVML) into the container at start. Apptainer’s own list of these libraries covers NVIDIA’s desktop driver. The L4T driver of a Jetson board loads more libraries, and the set changes from one L4T release to the next. Each L4T release lists them in its container manifest, the *.csv files in /etc/nvidia-container-runtime/host-files-for-container.d. Before it starts a --nv node, peppy adds every shared library of that manifest to the list of the bundled Apptainer. So the same node configuration renders and computes through the GPU on a desktop and on a Jetson.

macOS support

Section titled “macOS support”

On macOS, Apptainer is not natively available. Peppy bundles a Lima virtual machine that runs Apptainer inside a lightweight Linux guest. This is handled transparently and requires no additional installation or configuration; peppy node commands work the same way on macOS and Linux, with one exception: mount sources under the host runtime trees (/dev, /proc, /run, /sys) are refused, since a device plugged into the Mac cannot be forwarded into the Linux guest.

Customizing the definition file

Section titled “Customizing the definition file”

The generated apptainer.def is a starting point. You can modify it freely to fit your needs. Common customizations include:

Adding system dependencies

Section titled “Adding system dependencies”

Add packages to the %post section:

%post
    apt-get update
    apt-get install -y --no-install-recommends \
    libopencv-dev libudev-dev
    rm -rf /var/lib/apt/lists/*

Changing the base image

Section titled “Changing the base image”

Swap the From line to use a different base:

Bootstrap: docker
From: nvidia/cuda:12.4.0-devel-ubuntu24.04

Using a pre-built base image for faster builds

Section titled “Using a pre-built base image for faster builds”

Every peppy node add runs the full %post section from scratch, installing system packages, toolchains, and compiling dependencies each time. For nodes with heavy dependencies this can be slow.

You can speed things up by baking those slow steps into a custom Docker image and using it as your base. The first build pays the cost once; every subsequent node add starts from the cached image and only rebuilds your application code.

  1. Create a Dockerfile with the dependencies your node needs:

    Dockerfile
    FROM ubuntu:24.04
    

    RUN set -eux \
        && export DEBIAN_FRONTEND=noninteractive \
        && apt-get update \
        && apt-get install -y --no-install-recommends \
            ca-certificates curl python3 python3-venv \
        && rm -rf /var/lib/apt/lists/* \
        && curl -LsSf https://astral.sh/uv/install.sh | sh

  2. Build and push the image to a registry your build machine can reach:

    Terminal window
    docker build -t my-registry/my_node-base:latest .
    docker push my-registry/my_node-base:latest

  3. Point your apptainer.def at the new image and remove the steps that are already baked in:

    apptainer.def
    Bootstrap: docker
    From: my-registry/my_node-base:latest
    

    %labels
        Name my_node
        Version v1
    

    %environment
        export PATH="/root/.local/bin:/opt/my_node/.venv/bin:$PATH"
    

    %files
        . /opt/my_node
    

    %post
        set -eux
        cd /opt/my_node
        uv sync --no-editable
    

    %runscript
        cd /opt/my_node
        exec ./.venv/bin/python -m my_node

Now peppy node add only runs the application-specific build steps; package installation and toolchain setup are already in the base image.

Pulling from a private registry

Section titled “Pulling from a private registry”

On Linux, a build reads registry credentials from the same file Apptainer reads: $APPTAINER_AUTH_FILE when it is set, else ~/.apptainer/docker-config.json (written by apptainer registry login) when it exists, else ~/.docker/config.json (written by docker login). Log in to the registry that holds your base image with either tool before you build.

Peppy gives Apptainer a copy of that file without the https://index.docker.io/v1/access-token and https://index.docker.io/v1/refresh-token entries that Docker’s browser login adds. These entries hold Docker’s own session tokens, not registry passwords, and Apptainer would otherwise use one of them at random in place of your Docker Hub credential. Peppy writes the copy before each build, so a new docker login applies to the next build without a daemon restart. If Peppy cannot read or parse the file, the build fails and names the file.

On macOS, builds run inside the Lima guest, so they use the credential files in the guest user’s home directory.

Adding environment variables

Section titled “Adding environment variables”

Add variables to the %environment section so they are available at runtime:

%environment
    export PATH="/opt/my_node/.venv/bin:$PATH"
    export PYTHONUNBUFFERED=1