Skip to content

Publishing nodes

A container node builds into a single Apptainer .sif image. Peppy can push that image to an OCI registry (GitHub Container Registry, Docker Hub, or a private registry) so other machines with that oci repository can download the image instead of rebuilding it from source. This saves minutes per build, especially on smaller devices like a Raspberry Pi.

Only container nodes can be published. Process nodes (build_cmd / run_cmd) always build locally.

  1. Choose one OCI repository to store your published nodes, such as oci://ghcr.io/acme/peppy-nodes. One repository holds nodes for all Peppy versions because each artifact gets its own tag (see Registry layout).

  2. On the publishing machine, set publish_repository in ~/.peppy/conf/peppy_config.json5 and restart the daemon:

    publish_repository: "oci://ghcr.io/acme/peppy-nodes",
  3. Log in to the registry on every publishing machine and on any machine that pulls from a private repository (see Registry credentials).

  4. On machines that should pull prebuilt images, add the OCI repository with a lower id than your source repositories (--top) and refresh:

    Terminal window
    peppy repo add oci://ghcr.io/acme/peppy-nodes --top
    peppy repo refresh

Because artifacts are tied to a specific Peppy release version, only release builds of Peppy can publish or pull OCI node artifacts. Non-release builds reject both operations with an explanation.

Peppy reads registry credentials from Apptainer’s auth file chain: $APPTAINER_AUTH_FILE if set, otherwise ~/.apptainer/docker-config.json (from apptainer registry login) if it exists, otherwise ~/.docker/config.json (from docker login). Log in with either tool:

Terminal window
docker login ghcr.io
apptainer registry login --username <user> docker://ghcr.io

For a given registry, Peppy resolves credentials in this order:

  1. The helper named in credHelpers for that registry.
  2. The default helper named in credsStore.
  3. The registry’s entry under auths (username/password, token, or identitytoken).

Credential helpers run as docker-credential-<name> on the daemon’s PATH. Without credentials, Peppy sends unauthenticated requests (which works for pulling from public repositories). The auth file is re-read before every pull and push, so new logins take effect without restarting the daemon.

Pushing requires write permission (for GitHub Container Registry, a personal access token with the write:packages scope).

Terminal window
peppy node publish <name>:<tag> [--platforms <platform>,...] [--idle-timeout <secs>] [--max-timeout <secs>] [--force]

The node must already be added to the stack (peppy node add <name>:<tag>). The daemon builds the node for the host platform (just like peppy node build) and pushes the .sif to publish_repository under the tag <name>.<tag>.<peppy version>.linux-<arch> (for example, camera.v1.v0.32.0.linux-amd64).

FlagEffect
--platformsAdditional comma-separated platforms to build and push alongside the host platform (linux/amd64, linux/arm64). See Build for another architecture.
--idle-timeoutPer-platform build idle timeout in seconds (same as peppy node build).
--max-timeoutPer-platform maximum build duration in seconds. Platforms build sequentially, so total time is up to this value times the platform count.
--forceCancel any running build of the node and start publishing.

Before building anything, peppy node publish runs these preflight checks in order. The first failure aborts immediately with a fix message:

  1. publish_repository is configured on the daemon.
  2. Peppy is a release build.
  3. The node is a container node (process nodes are refused with a link to the container guide).
  4. The node did not come from an oci repository (no local source exists to build).
  5. Every --platforms entry is linux/amd64 or linux/arm64 (validated locally by the CLI).
  6. The registry accepts a push using this machine’s credentials.
  7. Cross-architecture prerequisites are present for any non-host platform in --platforms: QEMU system emulator and UEFI firmware are installed, Peppy has Lima and Apptainer for that architecture, the apptainer.def base image supports that platform, and (for Python nodes) Peppy includes the peppylib native extension for that platform. Release builds bundle Lima, Apptainer, and peppylib for all supported Linux platforms.
  8. Any existing tag in the registry for a target platform has the same source digest as this node. Tags are immutable and never overwritten.

The source digest is the SHA-256 of the node’s staged tree (excluding vendored peppylib for Python nodes), so identical sources produce the same digest on every machine and platform. The command prints it (e.g., Source digest of `camera:v1`: sha256:9c1e...).

  • Same digest already published: The platform is skipped and reported as already published.
  • Different digest on the same tag: The publish fails and prints both digests. Bump the node tag (e.g., v2) or publish under a new Peppy version (each Peppy release uses distinct tags).

The command streams build output plus hashing/push progress (every 1% and at least once every 10 seconds while transferring), then prints one summary line per platform:

linux/amd64: pushed camera.v1.v0.32.0.linux-amd64 (sha256:4b7f...)
linux/arm64: already published as camera.v1.v0.32.0.linux-arm64
Published node camera:v1

If any requested platform fails to publish, the command exits with an error reporting how many platforms failed.

Pass --platforms to build for the other Linux architecture alongside the host’s (--platforms linux/arm64 on amd64, or --platforms linux/amd64 on arm64). Listing the host’s own platform does not build it twice.

Cross-architecture builds run inside a persistent Lima VM using QEMU emulation (peppy-aarch64 for linux/arm64, peppy-x86_64 for linux/amd64) on both Linux and macOS. Peppy bundles Lima and Apptainer for both architectures, but you must install QEMU and UEFI firmware on the host:

Operating systemFor linux/arm64 on an amd64 machineFor linux/amd64 on an arm64 machine
macOSbrew install qemubrew install qemu
Debian, Ubuntusudo apt install qemu-system-arm qemu-efi-aarch64sudo apt install qemu-system-x86 ovmf
Fedorasudo dnf install qemu-system-aarch64-core edk2-aarch64sudo dnf install qemu-system-x86-core edk2-ovmf
Arch Linuxsudo pacman -S qemu-system-aarch64 edk2-aarch64sudo pacman -S qemu-system-x86 edk2-ovmf

Emulated builds are significantly slower than native builds. Each platform builds from its own copy of the staged tree with per-architecture build caches. Your definition file’s base image must support the target platform (peppybot/rust-cargo-base and peppybot/python-uv-base support both).

Terminal window
peppy stack build <launcher> --publish [--platforms <platform>,...]

Each machine in the stack builds its assigned container nodes and pushes their .sif images to the invoking daemon’s publish_repository, authenticating with each machine’s own registry credentials. Passing --platforms (which requires --publish) builds and pushes those additional platforms on each machine as well.

Before any build starts, the invoking daemon runs checks 1, 2, and 6 locally, and every other participating machine verifies registry push access during reservation. Any failure there aborts the whole command and names the machine. Per-node/per-platform checks do not abort the rest of the stack: skipped items are logged in the final report while other nodes continue:

Publish report:
camera:v1@cn-atlas linux/amd64: pushed camera.v1.v0.32.0.linux-amd64 (sha256:4b7f...)
camera:v1@cn-atlas linux/arm64: not published: the base image docker.io/acme/camera-base:1.2 of apptainer.def has no linux/arm64
planner:v2@cn-atlas: not published: process node
arm_driver:v1@cn-pi: not published: comes from the oci repository oci://ghcr.io/acme/peppy-nodes

If any container node platform fails to publish, the command finishes printing the report and exits with code 4 (failed goal). Process nodes and nodes sourced from an oci repository appear in the report without causing an error.

One OCI repository holds all published nodes. Each artifact represents one (node, Peppy version, platform) tuple:

  • Tag: <node name>.<node tag>.<peppy version>.linux-<arch> (amd64 or arm64). Registries limit tags to 128 characters, so nodes whose tag exceeds 128 characters cannot be published.
  • Manifest: OCI image manifest with artifact type application/vnd.peppy.node.v1 and empty config application/vnd.oci.empty.v1+json.
  • Layer 1 (.sif image): Media type application/vnd.sylabs.sif.layer.v1.sif.
  • Layer 2 (peppy.json5 manifest): Media type application/vnd.peppy.node.manifest.v1+json5. peppy repo refresh reads this layer alone without downloading the .sif.
  • Annotations: bot.peppy.node.name, bot.peppy.node.tag, bot.peppy.version, bot.peppy.platform, and bot.peppy.source-digest. Nodes sourced from a git repository also include bot.peppy.source.url, bot.peppy.source.commit, and bot.peppy.source.path.

Peppy never deletes or overwrites tags. Remove obsolete Peppy versions using your registry’s own tools.