Containers

Containers package a node and all of its dependencies into a single, self-contained image. A containerized node runs identically regardless of what is installed on the host: no more “works on my machine” issues.

Use containers when you need:

• Portability: ship a node to another machine without worrying about system dependencies.
• Reproducibility: guarantee the same runtime environment every time.
• Isolation: prevent conflicts between nodes that need different versions of the same library.

Peppy uses Apptainer as its container runtime. On macOS, Apptainer runs transparently inside a Lima virtual machine; no extra setup is needed.

Setup (Linux)

On Linux, Apptainer uses unprivileged user namespaces which may require a one-time system configuration. The installer handles this automatically, but if you skipped it or installed Peppy manually you can run:

peppy container setup

This configures the following (prompting for sudo when needed):

1. uidmap package: installs newuidmap (required for fakeroot mode).
2. AppArmor profile (on systems that restrict unprivileged user namespaces via AppArmor): installs a profile that allows Apptainer to create user namespaces and loads it into the kernel. When the profile already exists but is not loaded, it is reloaded without being rewritten.

To check the current state without making any changes:

peppy container status

This prints a pass/fail summary of each prerequisite and exits with code 0 (all pass) or 1 (something needs fixing).

Note

On macOS, no setup is needed; containers run inside a Lima VM which handles permissions transparently.

Initializing a container node

Pass the --container flag to peppy node init:

Python

peppy node init --toolchain uv --container my_node

Rust

peppy node init --toolchain cargo --container my_node

This generates the same project scaffolding as a regular node, plus an apptainer.def file that describes how the container image is built.

The peppy.json5 configuration

A container node includes a container block inside its execution section instead of the usual build_cmd and run_cmd fields. The two are mutually exclusive: a node is either a container node or a process node, never both.

Python

peppy.json5

{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
    },
  }
}

Rust

peppy.json5

{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "rust",
    container: {
      def_file: "apptainer.def",
    },
  }
}

The def_file field points to the Apptainer definition file relative to the node root. You can rename or relocate it as long as def_file matches.

Compare this with a standard process node, which defines build_cmd and run_cmd instead:

Python

peppy.json5 (process node)

{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "python",
    build_cmd: ["uv", "sync", "--no-editable"],
    run_cmd: ["./.venv/bin/python", "-m", "my_node"]
  }
}

Rust

peppy.json5 (process node)

{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "rust",
    build_cmd: ["cargo", "build", "--release"],
    run_cmd: ["./target/release/my_node"]
  }
}

Container nodes don’t need build_cmd or run_cmd; the definition file takes care of both building and running the node.

The apptainer.def file

The generated definition file is a standard Apptainer definition file. Here is what peppy node init --container generates:

Python

apptainer.def

Bootstrap: docker
From: peppybot/python-uv-base:latest

%labels
    Name my_node
    Version v1

%environment
    export PATH="/opt/my_node/.venv/bin:$PATH"

%files
    . /opt/my_node

%post
    set -eux
    export DEBIAN_FRONTEND=noninteractive

    export UV_PYTHON_INSTALL_DIR=/opt/uv-python

    cd /opt/my_node
    uv python install
    uv sync --no-editable

%runscript
    cd /opt/my_node
    exec ./.venv/bin/python -m my_node

Rust

apptainer.def

Bootstrap: docker
From: peppybot/rust-cargo-base:latest

%labels
    Name my_node
    Version v1

%files
    . /opt/my_node

%post
    set -eux

    cd /opt/my_node
    cargo build --release

%runscript
    cd /opt/my_node
    exec ./target/release/my_node

Each section serves a specific purpose:

Section	Purpose
Bootstrap / From	Base image to build from (Ubuntu 24.04 by default)
%labels	Metadata embedded in the image
%environment	Environment variables set when the container runs
%files	Copies the node source into the image at /opt/<node_name>
%post	Build steps: install system packages, toolchains, and compile the node
%runscript	Entry point executed when the container starts

Note

The %files section copies the entire node directory into the container. Peppy automatically copies (rather than symlinks) internal libraries like peppylib so that they are fully available inside the image.

Caution

Keep whatever the build produces out of the container user’s home directory. At run time apptainer mounts the home of the user running the daemon over the image’s own home, so anything the build left there is hidden. That is what UV_PYTHON_INSTALL_DIR above is for: uv installs managed interpreters under $HOME, so without it the interpreter lands in /root and a daemon running as root hides it, leaving the node to fail at startup with ./.venv/bin/python: not found.

Adding a container node

Adding a container node works the same as a regular node: first stage it, then build.

peppy node add ./my_node
peppy node build my_node:v1

You can also combine both steps with peppy node add ./my_node --build (shorthand -b). If you’ve just edited peppy.json5, add --sync/-s as well, e.g. peppy node add ./my_node -sb to sync, add, and build in one shot.

Under the hood, Peppy runs apptainer build during the build phase to produce a .sif (Singularity Image Format) file. This replaces the build_cmd step used by process nodes; the entire build happens inside the container according to the %post section of the definition file.

The resulting .sif file is stored under ~/.peppy/built_nodes/<name>_<tag>/, named after the fingerprint of the staged sources (see Build caching), and is ready to be started.

Note

Container builds can take longer than regular builds because they install system packages and toolchains from scratch.

Starting a container node

Starting a container node also uses the same command:

peppy node run my_node:v1

Peppy runs the .sif image with apptainer run and a clean environment: the daemon’s own environment is not inherited. The instance’s declared env_vars (minus HOME, which is dropped) and PEPPY_RUNTIME_CONFIG are passed in explicitly, and the image’s %environment section applies as usual.

The container executes the %runscript section, which runs the compiled binary (Rust) or the Python module entry point.

Mounting host directories

By default, a container is isolated from the host filesystem. Use mount_paths to bind-mount host directories into the running container, useful for sharing datasets, persisting output, or exposing device files.

Add a mount_paths array to the container block inside execution in peppy.json5:

peppy.json5

{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "/data/models:/opt/models:ro",
        "/tmp/my_node_output:/output:rw"
      ]
    },
  }
}

Each entry follows the format host_path:container_path[:options]:

Format	Example	Behaviour
host_path	"/data/models"	Mounted at the same path inside the container
host_path:container_path	"/data/models:/opt/models"	Mounted at a different path inside the container
host_path:container_path:options	"/data/models:/opt/models:ro"	Mounted with explicit options (ro for read-only, rw for read-write)

host_path may start with ~, which expands to the home directory of the user running the daemon on the machine where the instance runs. In a federated launch, each machine expands ~ against its own home, so a single manifest works across machines with different usernames ("~/.cache/my_node:/cache:rw"). The ~user form is not supported and is rejected while the launch is being validated, before any stack is replaced. Only the host side is expanded: a ~ in container_path is passed to the container verbatim.

A source under /dev with no remap (no destination, or a destination equal to the source) is passed through rather than bound: host devices reach the container through Apptainer’s device mount, and only a remapped device bind emits an explicit --bind.

A missing mount source is created on the host as a directory before the container starts, and a warning names each auto-created path, so a typo in a file bind shows up as an unexpectedly created empty directory. Sources under the host runtime trees (/dev, /proc, /run, /sys) are used as-is and never created.

Note

Top-level system directories such as /, /tmp, /var, /etc, /dev, /usr, /home, /opt, /bin, and /sbin cannot be used as mount sources, and on macOS neither can their /private/* equivalents (/private/tmp, /private/var, /private/etc). Subdirectories of these paths are fine; for example, /tmp/my_app_data is allowed but /tmp is not.

Note

On macOS, Peppy automatically configures the Lima VM to make mounted paths accessible inside the guest, including paths outside your home directory. Sources under the host runtime trees (/dev, /proc, /run, /sys) are the exception: a device plugged into the Mac cannot be forwarded into the Linux guest, so such a bind is refused with an error naming the path; run that node on a Linux daemon instead.

Registering a new path with the VM restarts it, which stops every container node running on that machine. peppy stack launch prepares a machine’s mount paths before it starts anything, so a launch never pays that cost in stopped nodes: the restart, when needed, happens before anything is running. A peppy node run that would introduce a new path while container nodes are already running is refused instead, naming the instances it would have stopped: stop them first, or launch them together.

Using parameters in mount paths

Mount paths can reference runtime parameters using the ${parameters:<path>} syntax. This lets each node instance mount a different host path based on its configuration.

peppy.json5

{
  peppy_schema: "node/v1",
  manifest: {
    name: "uvc_camera",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "rust",
    parameters: {
      device_path: "string",
    },
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "${parameters:device_path}:/dev/video0:rw"
      ]
    },
  },
}

When the node runs, ${parameters:device_path} is replaced with the actual value provided in the deployment configuration. For example, if the instance supplies device_path: "/dev/video2", the resulting bind mount is /dev/video2:/dev/video0:rw.

For nested parameters, use dot notation:

peppy.json5 (nested example)

{
  // ...
  execution: {
    // ...
    parameters: {
      video: {
        device_path: "string",
        frame_rate: "u16",
      },
    },
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "${parameters:video.device_path}:/dev/video0:rw"
      ]
    },
  },
  // ...
}

Note

Only parameters of type "string" can be referenced in mount paths. Numeric or object parameters are rejected at parse time.

Note

A literal blocked source (e.g., /tmp) is rejected at parse time. A source containing ${parameters:...} is checked at runtime instead, against the resolved path.

Build caching

Build artifact cache

Every peppy node build, including the ones peppy stack launch runs for each deployment, fingerprints the staged node tree before building: the node sources, peppy.json5, the def file, the generated .peppy/ code and the vendored peppylib, mixed with the peppy version, the host platform and the artifact kind. The artifact is stored under that fingerprint:

~/.peppy/built_nodes/<name>_<tag>/<fingerprint>.sif       container node
~/.peppy/built_nodes/<name>_<tag>/<fingerprint>.tar.zst   process node

When the fingerprint already has an artifact on disk, the build reuses it instead of running apptainer build (or build_cmd for a process node), and the build log carries one line:

Reusing cached build of my_node:v1 (fingerprint 3f9a1c2e5d7b8a4f) at ~/.peppy/built_nodes/my_node_v1/3f9a1c2e5d7b8a4f.sif

The cache is a plain directory tree keyed by content, so it survives daemon restarts and reboots, and a disk prepared on one machine carries its built nodes to every machine that boots from it with the same peppy version, OS and architecture. A process node’s archive also embeds what the host toolchain produced, which the fingerprint does not cover, so those machines need a compatible build environment as well. Each node identity keeps exactly one artifact: a build whose fingerprint differs replaces the previous one.

Pass --rebuild to peppy node build or peppy stack launch to build from the staged sources even when a matching artifact exists; the result replaces it. Use it when something outside the fingerprint changed:

• A floating base image tag (From: peppybot/rust-cargo-base:latest) that now resolves to a different image. Pin the image with @sha256:<digest> in the def file to make base image changes part of the fingerprint.
• For a process node, the host toolchain or the environment build_cmd runs with.

--rebuild is independent of --force, which cancels an in-flight build of the same node.

Compile caches inside container builds

Rust container builds reuse a shared cache across builds, without any per-node configuration. The daemon bind mounts ~/.peppy/cache/container_build into the build at /peppy-cache and points the %post toolchain at it:

• The crates.io registry persists via CARGO_HOME, so dependencies are downloaded once instead of on every build.
• sccache caches rustc invocations via RUSTC_WRAPPER, so unchanged crates are not recompiled. The executable ships inside peppybot/rust-cargo-base, so nothing needs to be installed on the host, on Linux or macOS; the wrapper is activated only for defs that bootstrap from that image.

Caution

Caching requires a /peppy-cache directory in the base image: build-time binds cannot create their destination. The peppy base images (peppybot/rust-cargo-base, peppybot/python-uv-base) ship it, and the daemon enables caching automatically when a def file bootstraps from one of them. Any other base image must create the directory itself (RUN mkdir /peppy-cache in its Dockerfile) and the def file must mention /peppy-cache to opt in; such builds get registry caching, and sccache additionally requires the image to provide the executable on its PATH.

Caching is best effort: if any part of it is unavailable, the build runs exactly as it would without caching. Set PEPPY_NO_CONTAINER_BUILD_CACHE=1 in the daemon’s environment to disable caching entirely. When the def file mentions rustup (installing it in %post is incompatible with the CARGO_HOME override) or sets one of the managed variables itself, caching is skipped automatically for that build.

Note

The cache lives at ~/.peppy/cache/container_build and is safe to delete at any time; the next build recreates it. The sccache half prunes itself to sccache’s size limit (10 GiB by default), while cargo-home/ grows with every dependency version ever built and is only reclaimed by deleting it. Because the directory is shared and writable, anything a %post writes there persists into later builds on the same machine, the same trust the build itself already runs with. On macOS, builds run inside the Lima VM: the cache directory is bind mounted through it and the environment rides the guest command, so container builds cache the same way as on Linux.

Extra runtime arguments

You can pass additional command-line arguments directly to Apptainer or Lima using apptainer_build_extra_args, apptainer_run_extra_args, and lima_shell_extra_args in the container block inside execution.

peppy.json5

{
  peppy_schema: "node/v1",
  manifest: {
    name: "my_node",
    tag: "v1",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
      apptainer_build_extra_args: ["--no-setgroups"],
      apptainer_run_extra_args: ["--no-setgroups"],
    },
  }
}

Field	Purpose
apptainer_build_extra_args	Extra flags appended to apptainer build (e.g., ["--no-setgroups"])
apptainer_run_extra_args	Extra flags appended to apptainer run (e.g., ["--no-setgroups"])
lima_shell_extra_args	Extra flags passed to limactl shell on macOS (ignored on Linux)

All fields are optional and default to an empty list when omitted.

Note

These arguments are passed verbatim; no validation is performed beyond basic string checks. Incorrect flags will cause Apptainer or Lima to fail at build or start time.

macOS support

On macOS, Apptainer is not natively available. Peppy bundles a Lima virtual machine that runs Apptainer inside a lightweight Linux guest. This is handled transparently and requires no additional installation or configuration; peppy node commands work the same way on macOS and Linux, with one exception: mount sources under the host runtime trees (/dev, /proc, /run, /sys) are refused, since a device plugged into the Mac cannot be forwarded into the Linux guest.

Customizing the definition file

The generated apptainer.def is a starting point. You can modify it freely to fit your needs. Common customizations include:

Adding system dependencies

Add packages to the %post section:

%post
    apt-get update
    apt-get install -y --no-install-recommends \
    libopencv-dev libudev-dev
    rm -rf /var/lib/apt/lists/*

Changing the base image

Swap the From line to use a different base:

Bootstrap: docker
From: nvidia/cuda:12.4.0-devel-ubuntu24.04

Using a pre-built base image for faster builds

Every peppy node add runs the full %post section from scratch, installing system packages, toolchains, and compiling dependencies each time. For nodes with heavy dependencies this can be slow.

You can speed things up by baking those slow steps into a custom Docker image and using it as your base. The first build pays the cost once; every subsequent node add starts from the cached image and only rebuilds your application code.

1. Create a Dockerfile with the dependencies your node needs:

   Python

   Dockerfile

   FROM ubuntu:24.04
   
   RUN set -eux \
       && export DEBIAN_FRONTEND=noninteractive \
       && apt-get update \
       && apt-get install -y --no-install-recommends \
           ca-certificates curl python3 python3-venv \
       && rm -rf /var/lib/apt/lists/* \
       && curl -LsSf https://astral.sh/uv/install.sh | sh

   Rust

   Dockerfile

   FROM ubuntu:24.04
   
   RUN set -eux \
       && export DEBIAN_FRONTEND=noninteractive \
       && apt-get update \
       && apt-get install -y --no-install-recommends \
           ca-certificates curl build-essential pkg-config \
       && rm -rf /var/lib/apt/lists/* \
       && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

2. Build and push the image to a registry your build machine can reach:

   docker build -t my-registry/my_node-base:latest .
   docker push my-registry/my_node-base:latest

3. Point your apptainer.def at the new image and remove the steps that are already baked in:

   Python

   apptainer.def

   Bootstrap: docker
   From: my-registry/my_node-base:latest
   
   %labels
       Name my_node
       Version v1
   
   %environment
       export PATH="/root/.local/bin:/opt/my_node/.venv/bin:$PATH"
   
   %files
       . /opt/my_node
   
   %post
       set -eux
       cd /opt/my_node
       uv sync --no-editable
   
   %runscript
       cd /opt/my_node
       exec ./.venv/bin/python -m my_node

   Rust

   apptainer.def

   Bootstrap: docker
   From: my-registry/my_node-base:latest
   
   %labels
       Name my_node
       Version v1
   
   %environment
       export PATH="/root/.cargo/bin:$PATH"
   
   %files
       . /opt/my_node
   
   %post
       set -eux
       cd /opt/my_node
       cargo build --release
   
   %runscript
       cd /opt/my_node
       exec ./target/release/my_node

Now peppy node add only runs the application-specific build steps; package installation and toolchain setup are already in the base image.

Tip

If multiple nodes share the same system dependencies, a single base image can serve all of them. Update the base image when dependencies change and tag it with a version so builds stay reproducible.

Adding environment variables

Add variables to the %environment section so they are available at runtime:

Python

%environment
    export PATH="/opt/my_node/.venv/bin:$PATH"
    export PYTHONUNBUFFERED=1

Rust

%environment
    export PATH="/root/.cargo/bin:$PATH"
    export RUST_LOG=info