Containers

Containers package a node and all of its dependencies into a single, self-contained image. A containerized node runs identically regardless of what is installed on the host — no more “works on my machine” issues.

Use containers when you need:

• Portability — ship a node to another machine without worrying about system dependencies.
• Reproducibility — guarantee the same runtime environment every time.
• Isolation — prevent conflicts between nodes that need different versions of the same library.

PeppyOS uses Apptainer as its container runtime. On macOS, Apptainer runs transparently inside a Lima virtual machine — no extra setup is needed.

Setup (Linux)

On Linux, Apptainer uses unprivileged user namespaces which may require a one-time system configuration. The installer handles this automatically, but if you skipped it or installed peppy manually you can run:

peppy container setup

This configures the following (prompting for sudo when needed):

1. uidmap package — installs newuidmap (required for fakeroot mode).
2. AppArmor profile (Ubuntu 24.04+ only) — installs a profile that allows Apptainer to create user namespaces.

To check the current state without making any changes:

peppy container status

This prints a pass/fail summary of each prerequisite and exits with code 0 (all pass) or 1 (something needs fixing).

Note

On macOS, no setup is needed — containers run inside a Lima VM which handles permissions transparently.

Initializing a container node

Pass the --container flag to peppy node init:

Python

peppy node init --toolchain uv --container my_node

Rust

peppy node init --toolchain cargo --container my_node

This generates the same project scaffolding as a regular node, plus an apptainer.def file that describes how the container image is built.

The peppy.json5 configuration

A container node includes a container block inside its execution section instead of the usual build_cmd and run_cmd fields. The two are mutually exclusive — a node is either a container node or a process node, never both.

Python

peppy.json5

{
  schema_version: 1,
  manifest: {
    name: "my_node",
    tag: "0.1.0",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
    },
  }
}

Rust

peppy.json5

{
  schema_version: 1,
  manifest: {
    name: "my_node",
    tag: "0.1.0",
  },
  interfaces: {},
  execution: {
    language: "rust",
    container: {
      def_file: "apptainer.def",
    },
  }
}

The def_file field points to the Apptainer definition file relative to the node root. You can rename or relocate it as long as def_file matches.

Compare this with a standard process node, which defines build_cmd and run_cmd instead:

Python

peppy.json5 (process node)

{
  schema_version: 1,
  manifest: {
    name: "my_node",
    tag: "0.1.0",
  },
  interfaces: {},
  execution: {
    language: "python",
    build_cmd: ["uv", "sync", "--no-editable"],
    run_cmd: ["./.venv/bin/python", "-m", "my_node"]
  }
}

Rust

peppy.json5 (process node)

{
  schema_version: 1,
  manifest: {
    name: "my_node",
    tag: "0.1.0",
  },
  interfaces: {},
  execution: {
    language: "rust",
    build_cmd: ["cargo", "build", "--release"],
    run_cmd: ["./target/release/my_node"]
  }
}

Container nodes don’t need build_cmd or run_cmd — the definition file takes care of both building and running the node.

The apptainer.def file

The generated definition file is a standard Apptainer definition file. Here is what peppy node init --container generates:

Python

apptainer.def

Bootstrap: docker
From: tuatini/peppy-python-uv-base

%labels
Name my_node
Version 0.1.0

%environment
export PATH="/opt/my_node/.venv/bin:$PATH"

%files
. /opt/my_node

%post
set -eux

cd /opt/my_node
uv sync --no-editable

%runscript
cd /opt/my_node
exec ./.venv/bin/python -m my_node

Rust

apptainer.def

Bootstrap: docker
From: tuatini/peppy-rust-cargo-base

%labels
Name my_node
Version 0.1.0

%files
. /opt/my_node

%post
set -eux

cd /opt/my_node
cargo build --release

%runscript
cd /opt/my_node
exec ./target/release/my_node

Each section serves a specific purpose:

Section	Purpose
Bootstrap / From	Base image to build from (Ubuntu 24.04 by default)
%labels	Metadata embedded in the image
%environment	Environment variables set when the container runs
%files	Copies the node source into the image at /opt/<node_name>
%post	Build steps — install system packages, toolchains, and compile the node
%runscript	Entry point executed when the container starts

Note

The %files section copies the entire node directory into the container. PeppyOS automatically copies (rather than symlinks) internal libraries like peppylib so that they are fully available inside the image.

Adding a container node

Adding a container node works the same as a regular node — first stage it, then build:

peppy node add ./my_node
peppy node build my_node:0.1.0

You can also combine both steps with peppy node add ./my_node --build (shorthand -b). If you’ve just edited peppy.json5, add --sync/-s as well — e.g. peppy node add ./my_node -sb to sync, add, and build in one shot.

Under the hood, PeppyOS runs apptainer build during the build phase to produce a .sif (Singularity Image Format) file. This replaces the build_cmd step used by process nodes — the entire build happens inside the container according to the %post section of the definition file.

The resulting .sif file is stored in PeppyOS’s internal storage and is ready to be started.

Note

Container builds can take longer than regular builds because they install system packages and toolchains from scratch. Subsequent builds reuse Docker layer caches when possible.

Starting a container node

Starting a container node also uses the same command:

peppy node run my_node

PeppyOS runs the .sif image with apptainer run. Environment variables such as PEPPY_RUNTIME_CONFIG are passed into the container automatically — you don’t need to configure anything beyond what a regular node requires.

The container executes the %runscript section, which runs the compiled binary (Rust) or the Python module entry point.

Mounting host directories

By default, a container is isolated from the host filesystem. Use mount_paths to bind-mount host directories into the running container — useful for sharing datasets, persisting output, or exposing device files.

Add a mount_paths array to the container block inside execution in peppy.json5:

peppy.json5

{
  schema_version: 1,
  manifest: {
    name: "my_node",
    tag: "0.1.0",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "/data/models:/opt/models:ro",
        "/tmp/my_node_output:/output:rw"
      ]
    },
  }
}

Each entry follows the format host_path:container_path[:options]:

Format	Example	Behaviour
host_path	"/data/models"	Mounted at the same path inside the container
host_path:container_path	"/data/models:/opt/models"	Mounted at a different path inside the container
host_path:container_path:options	"/data/models:/opt/models:ro"	Mounted with explicit options (ro for read-only, rw for read-write)

PeppyOS creates any missing parent directories on the host automatically before starting the container.

Note

Top-level system directories such as /, /tmp, /var, /etc, /dev, /usr, /home, /opt, /bin, and /sbin cannot be used as mount sources. Subdirectories of these paths are fine — for example, /tmp/my_app_data is allowed but /tmp is not.

Note

On macOS, PeppyOS automatically configures the Lima VM to make mounted paths accessible inside the guest. No extra setup is needed — paths outside your home directory are handled transparently.

Using parameters in mount paths

Mount paths can reference runtime parameters using the ${parameters:<path>} syntax. This lets each node instance mount a different host path based on its configuration.

peppy.json5

{
  schema_version: 1,
  manifest: {
    name: "uvc_camera",
    tag: "0.1.0",
  },
  interfaces: {},
  execution: {
    language: "rust",
    parameters: {
      device_path: "string",
    },
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "${parameters:device_path}:/dev/video0:rw"
      ]
    },
  },
}

When the node runs, ${parameters:device_path} is replaced with the actual value provided in the deployment configuration. For example, if the instance supplies device_path: "/dev/video2", the resulting bind mount is /dev/video2:/dev/video0:rw.

For nested parameters, use dot notation:

peppy.json5 (nested example)

{
  // ...
  execution: {
    // ...
    parameters: {
      video: {
        device_path: "string",
        frame_rate: "u16",
      },
    },
    container: {
      def_file: "apptainer.def",
      mount_paths: [
        "${parameters:video.device_path}:/dev/video0:rw"
      ]
    },
  },
  // ...
}

Note

Only parameters of type "string" can be referenced in mount paths. Numeric or object parameters will be rejected at parse time.

Note

Blocked system directory validation (e.g., rejecting /tmp as a mount source) is applied to the resolved path at runtime, not at parse time.

Extra runtime arguments

You can pass additional command-line arguments directly to Apptainer or Lima using apptainer_build_extra_args, apptainer_run_extra_args, and lima_shell_extra_args in the container block inside execution.

peppy.json5

{
  schema_version: 1,
  manifest: {
    name: "my_node",
    tag: "0.1.0",
  },
  interfaces: {},
  execution: {
    language: "python",
    container: {
      def_file: "apptainer.def",
      apptainer_build_extra_args: ["--no-setgroups"],
      apptainer_run_extra_args: ["--no-setgroups"],
    },
  }
}

Field	Purpose
apptainer_build_extra_args	Extra flags appended to apptainer build (e.g., ["--no-setgroups"])
apptainer_run_extra_args	Extra flags appended to apptainer run (e.g., ["--no-setgroups"])
lima_shell_extra_args	Extra flags passed to limactl shell on macOS (ignored on Linux)

All fields are optional and default to an empty list when omitted.

Note

These arguments are passed verbatim — no validation is performed beyond basic string checks. Incorrect flags will cause Apptainer or Lima to fail at build or start time.

macOS support

On macOS, Apptainer is not natively available. PeppyOS bundles a Lima virtual machine that runs Apptainer inside a lightweight Linux guest. This is handled transparently — all peppy node commands work identically on macOS and Linux. No additional installation or configuration is required.

Customizing the definition file

The generated apptainer.def is a starting point. You can modify it freely to fit your needs. Common customizations include:

Adding system dependencies

Add packages to the %post section:

%post
    apt-get update
    apt-get install -y --no-install-recommends \
        libopencv-dev libudev-dev
    rm -rf /var/lib/apt/lists/*

Changing the base image

Swap the From line to use a different base:

Bootstrap: docker
From: nvidia/cuda:12.4.0-devel-ubuntu24.04

Using a pre-built base image for faster builds

Every peppy node add runs the full %post section from scratch — installing system packages, toolchains, and compiling dependencies each time. For nodes with heavy dependencies this can be slow.

You can speed things up by baking those slow steps into a custom Docker image and using it as your base. The first build pays the cost once; every subsequent node add starts from the cached image and only rebuilds your application code.

1. Create a Dockerfile with the dependencies your node needs:

Python

Dockerfile

FROM ubuntu:24.04

RUN set -eux \
    && export DEBIAN_FRONTEND=noninteractive \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
        ca-certificates curl python3 python3-venv \
    && rm -rf /var/lib/apt/lists/* \
    && curl -LsSf https://astral.sh/uv/install.sh | sh

Rust

Dockerfile

FROM ubuntu:24.04

RUN set -eux \
    && export DEBIAN_FRONTEND=noninteractive \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
        ca-certificates curl build-essential pkg-config \
    && rm -rf /var/lib/apt/lists/* \
    && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

2. Build and push the image to a registry your build machine can reach:

   docker build -t my-registry/my_node-base:latest .
   docker push my-registry/my_node-base:latest

3. Point your apptainer.def at the new image and remove the steps that are already baked in:

Python

apptainer.def

Bootstrap: docker
From: my-registry/my_node-base:latest

%labels
    Name my_node
    Version 0.1.0

%environment
    export PATH="/root/.local/bin:/opt/my_node/.venv/bin:$PATH"

%files
    . /opt/my_node

%post
    set -eux
    cd /opt/my_node
    uv sync --no-editable

%runscript
    cd /opt/my_node
    exec ./.venv/bin/python -m my_node

Rust

apptainer.def

Bootstrap: docker
From: my-registry/my_node-base:latest

%labels
    Name my_node
    Version 0.1.0

%environment
    export PATH="/root/.cargo/bin:$PATH"

%files
    . /opt/my_node

%post
    set -eux
    cd /opt/my_node
    cargo build --release

%runscript
    cd /opt/my_node
    exec ./target/release/my_node

Now peppy node add only runs the application-specific build steps — package installation and toolchain setup are already in the base image.

Tip

If multiple nodes share the same system dependencies, a single base image can serve all of them. Update the base image when dependencies change and tag it with a version so builds stay reproducible.

Adding environment variables

Add variables to the %environment section so they are available at runtime:

%environment
    export PATH="/root/.cargo/bin:$PATH"
    export RUST_LOG=info