Launch files

Launch files let you start multiple nodes simultaneously, including those with dependencies on each other. With a single launch file, you or your team can recreate an entire node environment using just one command.

Running the example

Start by cloning this repository:

1. Clone the repository:

   git clone https://github.com/Peppy-bot/launchers-hub

2. Navigate into the directory:

   cd launchers-hub

In the examples/ directory, you’ll find a launcher file for each language. Opening it reveals the following structure:

Python

examples/python_robot.json5

Rust

examples/rust_robot.json5

1. The file is identified by peppy_schema: "launcher/v1" at the root: peppy uses this to distinguish launcher files from node peppy.json5 files (which use "node/v1")
2. All deployments are defined in the deployments array
3. Each deployment requires a source (the node to run, or the MCP exposures to serve) and an instances array (how many copies to start, and how each one is configured). Every entry needs an instance_id; arguments, env_vars, framework, core_node, and links are optional. See Instance attributes for what each one does

The source attribute references a node from your configured repositories by name and tag. Specify them as separate fields, or use the combined name: "<name>:<tag>" shorthand. The node is resolved against your local nodes cache at ~/.peppy/cache/nodes.json5. A source can instead list MCP exposures to serve, as shown below.

If a node you want to deploy lives outside your configured repositories, add its location first: register a local directory with peppy repo add /path/to/my/nodes, or a git repository with peppy repo add https://github.com/org/repo.git. A repository states what it publishes through a committed peppy_repository.json5, so run peppy repo index /path/to/my/nodes to write it, and re-run it after adding, moving, or renaming a node, since the refresh reads the index rather than the filesystem. Then peppy repo refresh makes the nodes resolvable (see Repositories).

Note

The source resolves nodes from the locally cached index of the machine the launch is submitted to. Run peppy repo refresh on that machine before launching to ensure its cache is up to date. In a federated launch it is the only machine whose cache matters: it resolves every deployment and its dependencies once and ships the result to the other machines as pinned content, so their caches need no refresh and cannot change what the launch runs.

If a node or tag is missing from the cache, the launch fails with a “not found” error. If the cache instead contains the same name:tag twice from one repository, the launch fails with a distinct ambiguity error naming both manifests, never “not found”, since the node is present rather than absent. The refusal happens while sources are being resolved, before anything is added, built, or run, and it covers transitive dependencies too: an ambiguity two levels down is exactly as fatal as one at the top. See Using a repository-indexed node.

// Repository source (long form)
{
  source: {
    name: "openarm01_controller",
    tag: "v1",
  },
  instances: [
    { instance_id: "the_nervous_system" }
  ]
}

// Repository source (combined shorthand, equivalent to the example above)
{
  source: {
    name: "openarm01_controller:v1",
  },
  instances: [
    { instance_id: "the_nervous_system" }
  ]
}

Serving MCP exposures

A deployment whose source lists exposures runs the MCP server built into peppy for them instead of a node. Each entry is an MCP exposure as "<name>:<tag>", resolved against ~/.peppy/cache/mcp_exposures.json5, and the contracts the exposures reference resolve through the contracts cache. The daemon derives the server’s node from those documents, so the deployment behaves like any other: its instances take arguments (the server’s single port), links for the contract slots the exposures’ targets become, and core_node for placement. The list must not be empty or repeat an exposure, and two deployments cannot list the same set.

// Exposure source: the built-in MCP server serving two exposures on one port
{
  source: {
    exposures: ["camera_and_recording:v1", "arm_status:v1"],
  },
  instances: [
    {
      instance_id: "mcp_server",
      arguments: { port: 8900 },
      links: { front_camera: "front_cam_inst", recorder: "recorder_inst", arm: "arm_inst" },
    }
  ]
}

Caution

A git ref pinned in repositories.json5 and the node tag declared in peppy.json5 are independent values, and they do not share the same format: git refs can contain dots (e.g. v0.1.0), but node tags must start with an ASCII letter and forbid dots (e.g. v1, donut). This means a repository could be pinned to the git ref v0.1.0 while peppy.json5 specifies tag: "v1". Keep them coordinated so a checkout of a given git ref lines up with the tag your launcher expects.

From within the cloned repository, execute the launcher:

Python

peppy stack launch ./examples/python_robot.json5

Rust

peppy stack launch ./examples/rust_robot.json5

Peppy automatically inspects the name and tag of each node, verifies that all dependencies are met, and constructs the node stack in the correct order.

Launching from a repository

Launchers discovered by peppy repo refresh (see Repositories) can be invoked by bare name, no path required:

peppy stack launch openarm01_sim_teleop

An argument that contains a path separator or ends in .json5 is resolved as a filesystem path: ./my_launcher.json5, /abs/my_launcher.json5, my_launcher.json5. A bare name (no separator, no extension) is a repository launcher and is never looked up on disk, whatever the current directory holds: peppy stack launch my_launcher resolves through the repository cache at ~/.peppy/cache/launchers.json5 even when a my_launcher.json5 sits next to the caller. Run peppy repo refresh first on the machine you submit the launch to; the launcher document, like every repository source, resolves on that machine alone.

If the name resolves neither to a file nor to a cached launcher, the launch fails with an explicit “not found” error. If one repository provides two launcher files with that name, the launch fails with a distinct ambiguity error naming both: launchers are keyed by filename, so two identically named .json5 files anywhere in one repository contest the same name even in different directories.

Caution

Running the peppy stack launch or peppy stack build command clears the existing node stack and stops all running instances

A launch ends with the log file of every node it added, built and started, then the URLs the started instances serve, and the completion line. The URLs come from the nodes that declare execution.endpoints: the daemon hosting each instance expands the socket the node announced into one URL per address of its machine, loopback first, and the core node name after the @ says which machine that is. Pages an operator opens in a browser and endpoints an MCP client attaches to are printed under separate headings, Web pages: before MCP endpoints:, and a heading appears only when it has an entry. peppy stack join prints the same blocks for the instances the copy started.

Node log files:
  ...
Web pages:
  alpha_commander_inst (openarm_web_commander:v1) @cn-sweet-edison
    panel   http://127.0.0.1:8765
            http://100.123.58.116:8765
  simulation_inst (waldo:v1) @cn-sweet-edison
    viewer  https://127.0.0.1:8080/
            https://100.123.58.116:8080/
MCP endpoints:
  robot_control_inst (mcp_robot_control_v1:builtin) @cn-sweet-edison
    robot_control_v1  http://127.0.0.1:8900/robot_control/v1/mcp
  world_control_inst (mcp_simulation_v1:builtin) @cn-sweet-edison
    simulation_v1     http://127.0.0.1:8902/simulation/v1/mcp
Launch completed successfully

Verify that the node stack was configured correctly:

$ peppy stack list
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Core node: cn-adoring-zebra (host: robot-host)                                                                                                                             │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Slice of launch launch-brave-otter (coordinator: cn-adoring-zebra)                                                                                                         │
│                                                                                                                                                                            │
│ Node stack                                                                                                                                                                 │
│                                                                                                                                                                            │
│ ┌──────────────────────────────────────┬───────┬───────────┬────────────────────────────────────────────────────────────────────────────┐                                  │
│ │ NODE                                 │ STAGE │ INSTANCES │ PATH                                                                       │                                  │
│ ├──────────────────────────────────────┼───────┼───────────┼────────────────────────────────────────────────────────────────────────────┤                                  │
│ │ cn-adoring-zebra:v0.10.0             │ Root  │ 1 running │ ~/workspace/peppy                                                          │                                  │
│ │ fake_openarm01_controller:v1         │ Ready │ 1 running │ ~/.peppy/built_nodes/fake_openarm01_controller_v1/b81d0e4c72a5f936.tar.zst │                                  │
│ │ fake_robot_brain:v1                  │ Ready │ 1 running │ ~/.peppy/built_nodes/fake_robot_brain_v1/6c2e9f0a1d47b385.tar.zst          │                                  │
│ │ fake_uvc_camera:v1                   │ Ready │ 2 running │ ~/.peppy/built_nodes/fake_uvc_camera_v1/e57a3b91c0d64f28.tar.zst           │                                  │
│ │ fake_video_reconstruction:v1         │ Ready │ 1 running │ ~/.peppy/built_nodes/fake_video_reconstruction_v1/19d4c7e2a8b03f65.tar.zst │                                  │
│ └──────────────────────────────────────┴───────┴───────────┴────────────────────────────────────────────────────────────────────────────┘                                  │
│                                                                                                                                                                            │
│ Instance bindings                                                                                                                                                          │
│                                                                                                                                                                            │
│ ┌──────────────────────────────────────┬──────────────────────────────┬─────────┬─────────┬──────────────────────────────────────────────────────────────────────────────┐ │
│ │ NODE                                 │ INSTANCE                     │ STATUS  │ HEALTH  │ BINDINGS                                                                     │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ cn-adoring-zebra:v0.10.0             │ cn-adoring-zebra             │ running │ healthy │ (none)                                                                       │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ fake_openarm01_controller:v1         │ vibrant-keller-2310          │ running │ healthy │ (none)                                                                       │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ fake_robot_brain:v1                  │ zealous-bohr-5512            │ running │ healthy │ fake_openarm01_controller → vibrant-keller-2310@cn-adoring-zebra             │ │
│ │                                      │                              │         │         │ fake_uvc_camera → mystic-galois-7701@cn-adoring-zebra                        │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ fake_uvc_camera:v1                   │ mystic-galois-7701           │ running │ healthy │ (none)                                                                       │ │
│ │                                      │ brave-noether-3098           │ running │ healthy │ (none)                                                                       │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ fake_video_reconstruction:v1         │ gallant-curie-4417           │ running │ healthy │ fake_uvc_camera → brave-noether-3098@cn-adoring-zebra                        │ │
│ └──────────────────────────────────────┴──────────────────────────────┴─────────┴─────────┴──────────────────────────────────────────────────────────────────────────────┘ │
│                                                                                                                                                                            │
│ Dependencies                                                                                                                                                               │
│   fake_robot_brain:v1 ➔ fake_openarm01_controller:v1                                                                                                                       │
│   fake_robot_brain:v1 ➔ fake_uvc_camera:v1                                                                                                                                 │
│   fake_video_reconstruction:v1 ➔ fake_uvc_camera:v1                                                                                                                        │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

The output confirms that all nodes have been added to the stack along with their dependency relationships. When an instance serves an endpoint, an Instance endpoints table follows the bindings, one row per URL with the endpoint’s KIND and LABEL. The Instance bindings table breaks that down per instance: the consumers (fake_robot_brain, fake_video_reconstruction) list the depends_on slots they resolved, while the producers (fake_openarm01_controller, fake_uvc_camera) and the core node declare no slots and show (none). A resolved binding shows the chosen producer(s) as instance_id@core_node, ordered by link ID; a multi-cardinality slot lists every bound member in plan order, and an empty slot, whether a zero_or_more slot bound to nothing or a zero_or_one slot declared vacant, shows (empty set). Every declared producer slot (depends_on.nodes / depends_on.contracts) is either bound or, where its manifest declares cardinality: "zero_or_one", declared vacant; only a zero_or_more slot may be left out entirely, and a launcher that leaves any other one out is rejected at validation (see dependency cardinality). Pairing and observer slots follow the same shape: a participant slot is paired and an observer slot is linked, or, where its manifest declares it cardinality: "zero_or_one", declared vacant, unless its cardinality is zero_or_more, where omitting it IS its empty set. The launch also prints a Slice of launch ... line above the tables naming the launch that produced this stack and its coordinator.

Instance attributes

Every entry in an instances array configures one running copy of the deployment’s node. Two instances of the same node are independent: they get their own arguments, their own environment, and their own bindings.

peppy_launcher.json5

{
  source: { name: "uvc_camera:v1" },
  instances: [
    {
      instance_id: "camera_front",
      arguments: { device: { path: "/dev/video0" }, frame_rate: 30 },
      env_vars: { GST_DEBUG: "2" },
      framework: { clock: "simulation" },
    },
    {
      instance_id: "camera_rear",
      arguments: { device: { path: "/dev/video1" }, frame_rate: 15 },
    },
  ],
}

instance_id (required)

The name this copy runs under. It labels the instance in peppy stack list, in its log file, and in every links value that points at it.

Allowed characters are letters, digits, _ and -. The id must be unique across the whole stack, not just within its own deployment: two deployments cannot both declare camera_front, and no instance may take the core node’s own id. A collision is reported before anything starts.

arguments (optional)

Values for the parameters the node declares in its execution.parameters block, written with the same nesting as that schema. See the Parameters reference for the full syntax.

A parameter that declares a $default may be omitted, and the daemon fills it in before the node starts. A required parameter left without a value fails the launch as that instance starts, naming the leaf it is missing (Missing required parameters: device.serial). Unknown keys and type mismatches are caught by the node itself as it boots, so they surface as that instance failing to come up.

A container node can also reference these values in its mount_paths with ${parameters:<path>}, which is how two instances of one node mount two different host devices.

env_vars (optional)

Environment variables for this instance’s process.

peppy stack launch forwards a filtered copy of your shell environment to the instances it starts on the machine you launch from, so a node there already sees your PATH and the rest. The filter drops PWD, OLDPWD, TMPDIR, TEMP, TMP, PYTHONPATH, and PYTHONHOME, plus any name that is not a shell identifier, any value with whitespace or shell metacharacters, and the injection-prone names listed below. PYTHONPATH and PYTHONHOME are stripped so a caller shell that exports them cannot displace a container node’s vendored peppylib/peppygen path; a node that genuinely needs either sets it through its own env_vars. An instance placed on another machine receives only its declared env_vars. env_vars is layered on top of the forwarded set for what differs per instance (a device path, a board serial, a log level), and a name declared here replaces the forwarded value:

{
  source: { name: "esp32_board:v1" },
  instances: [
    { instance_id: "esp32_left",  env_vars: { ESP32_DEVICE: "/dev/ttyUSB0" } },
    { instance_id: "esp32_right", env_vars: { ESP32_DEVICE: "/dev/ttyUSB1" } },
  ],
}

Entries are checked when the launcher file is read, before the running stack is touched:

• Names must be shell identifiers: a letter or _ first, then letters, digits and _. ESP32_DEVICE is fine, esp32-device is not.
• Values may hold letters, digits and _ - . / : , = @ + %. Whitespace, quotes and shell metacharacters (;, $, `, |, and friends) are rejected, so a value like "--flag other" has to reach the node another way, typically as an argument.
• Injection-prone names are rejected outright: the dynamic-linker and shell hooks (LD_PRELOAD, LD_LIBRARY_PATH, DYLD_INSERT_LIBRARIES, BASH_ENV, IFS, and the rest of that family).

Note

The name and value rules exist because a container node receives its environment through Apptainer, which forwards each variable under the APPTAINERENV_ prefix into a script the container sources at startup, where the shell expands the value. A name that is not a shell identifier would not parse as an assignment, and a value carrying $, a backtick or a quote would be expanded or run inside the container rather than arriving as written. The same rule applies to process nodes so that adding a container block to a node later cannot break the launchers that already deploy it.

env_vars reach the instance when it starts. Fetching and building a node happen once for the node, not once per instance, so a variable your build needs (a registry token, a toolchain path) belongs in the shell you run peppy stack launch or peppy stack build from, where it is forwarded to build steps on that machine too. A build on another machine sees none of your caller environment: your environment describes your machine, so it stays there.

framework (optional)

Framework knobs owned by peppylib rather than by the node author, so they mean the same thing for every node. The block holds one knob:

• clock: the timeline this instance reads, naming wall or a domain the launcher declares under framework.clocks. An instance that omits it reads wall time. An instance a declaration names as a domain’s publisher carries no clock of its own, since that declaration already assigns it one. See System clock.

An unknown key inside framework fails the launch, so a typo surfaces where it is written.

core_node (optional)

The machine this instance runs on, named by one of the placeholders the launcher declares under core_nodes. An instance that omits it runs on the daemon you launched from. See Running one launcher across several machines.

links (optional)

One map filling every kind of depends_on slot the node declares, keyed by the slot’s link_id. A value is interpreted against the slot it names:

• a producer slot (depends_on.nodes / depends_on.contracts) takes the producer instance to bind: a scalar for a one or zero_or_one slot, an array for a multi-cardinality slot;
• a pairing participant slot takes the peer instance(s) to pair with, each "<peer>" or "<peer>/<peer_link_id>" to disambiguate which of the peer’s slots to use: a scalar for a one or zero_or_one slot, an array for a multi-cardinality slot, one pair per entry;
• an observer slot takes the source instance(s) to observe, each "<source>" or "<source>/<source_link_id>": a scalar for a one slot, an array for a multi-cardinality slot, in the order the node reads them back.

A slot that starts unresolved on purpose says so here too, as { vacant: "<why>" }. That value binds nothing and covers the slot, so the fate of every declared slot is one entry in this one map and a slot you forgot is an absence rather than a value. The reason is required, is trimmed, and travels with the instance, so peppy stack launch reports it back when the node starts.

Vacancy is a two-sided contract, and the node owns one side of it. A slot is vacatable only where the node’s own manifest declares it emptiable, with cardinality: "zero_or_one" on the depends_on.pairings, depends_on.pairing_observers, depends_on.nodes or depends_on.contracts entry. On a slot the manifest declares required, a vacancy is rejected and the error names the manifest key that would lift the bar. Every other slot is rejected with the spelling it does have: a zero_or_more slot, producer or observer, writes its empty set as [] (or omits the key), and a one_or_more slot of either kind has no empty state at all. The manifest says “may be empty”, the launcher says “is empty here, because”, and forgetting the slot entirely stays an error whichever the manifest says.

One instance filling all three kinds, plus a slot left deliberately empty:

{
  source: { name: "session_recorder:v1" },
  instances: [
    {
      instance_id: "rec_1",
      links: {
        // Producer slots: `storage` is `one`, `cameras` is multi-cardinality.
        storage: "disk_1",
        cameras: ["cam_left", "cam_right"],
        // A participant slot names the peer to pair with, with
        // `/<peer_link_id>` when the peer declares more than one slot.
        arm: "arm_1/limb",
        // A scalar observer slot names a single source instance to observe.
        // The multi cardinalities take these same values in an array.
        watch: "ctrl_1",
        // A slot this deployment leaves empty on purpose, and why. The
        // manifest declares `gripper` `cardinality: "zero_or_one"`, which is
        // what makes this legal.
        gripper: { vacant: "bench rig: no gripper on this frame" },
      },
    },
  ],
}

Every target must name an instance_id declared somewhere in the same launcher.

Two instances of the same node can choose different fates for the same slot, which is why the decision is written per instance rather than in the node’s manifest (whose cardinality: "zero_or_one" on this slot only says that some deployment may make it):

instances: [
  {
    instance_id: "backbone_governed",
    links: { leader_left_arm: "commander_inst/left_arm" },
  },
  {
    instance_id: "backbone_monitor",
    links: { leader_left_arm: { vacant: "monitor rig: nothing commands this backbone" } },
  },
]

Clock domains

A launcher declares the timelines its instances read in a root-level framework block, beside deployments:

peppy_launcher.json5

{
  peppy_schema: "launcher/v1",
  framework: {
    clocks: {
      // One instance supplies this timeline, and naming it here is what
      // makes `simulation_inst` the domain's publisher.
      simulation: { publisher: "simulation_inst" },
      // A name for the time every machine already keeps.
      physical_robot: "wall",
    },
  },
  deployments: [
    {
      source: { name: "waldo", tag: "v1" },
      instances: [{ instance_id: "simulation_inst" }],
    },
    {
      source: { name: "openarm_sim_arm", tag: "v1" },
      instances: [{ instance_id: "arm_inst", framework: { clock: "simulation" } }],
    },
  ],
}

A declaration is { publisher: "<instance_id>" } for a timeline one instance supplies, or "wall" for a name the time every machine already keeps answers to. wall itself is built in and cannot be declared. Every other instance names the domain it reads with framework.clock.

The launcher’s declarations and those of every fragment the selection pulls in are merged: identical declarations agree, two different declarations of one name are refused naming both documents, and an unselected fragment contributes nothing. A copy binds the domains the stack declares, naming them as the stack does.

Every producer binding, pairing and observation joins two instances that read one clock, so a launcher wiring an instance on one domain to an instance on another is refused before anything starts. See System clock.

Reusing built nodes across launches

Every node build a launch performs goes through the build artifact cache: when a deployment’s staged sources match an earlier build byte for byte, and that build ran with the same peppy version, OS, architecture and artifact kind, the launch reuses its artifact. Launching the same launcher again, after a daemon restart or on another machine booted from the same disk, therefore spends seconds on the build phase instead of rebuilding every node. Each reuse is reported in the launch output and in that node’s build log:

Reusing cached build of waldo:v1 (fingerprint 3f9a1c2e5d7b8a4f) at ~/.peppy/built_nodes/waldo_v1/3f9a1c2e5d7b8a4f.sif

peppy stack launch --rebuild and peppy stack build --rebuild build every node of the launch from its staged sources even when a matching artifact exists, on the coordinator and on every peer; use it when something the fingerprint does not cover changed, such as the image behind a floating base image tag.

Building without starting

peppy stack build runs a launch up to its builds. It replaces the node stack as a launch does, then adds and builds each node the launch would run on the machine the launch places it on, and starts no instance. It takes every argument peppy stack launch takes, so a later peppy stack launch with the same launcher, placement, --with words and --join copies starts from these builds.

Build a launcher on a disk, save the disk as an image, and every machine booted from that image launches without building:

peppy stack build openarm_simulation --with mujoco
# Save the disk as an image and boot each machine from it, then on each one:
peppy stack launch openarm_simulation --with mujoco

On Linux the daemon derives its default core node name from the machine id the disk stores, read from /var/lib/dbus/machine-id, else /etc/machine-id. Machines booted from one image therefore derive the same name, and a daemon whose name is already live on its router or federation refuses to start. Give each machine its own core_node_name; a name set on the imaged disk is shared the same way.

After a build, peppy stack list shows each node at stage Ready with 0 instances, under a Slice of launch <id> line naming the build that replaced this machine’s stack. A build records no launcher, so peppy stack join is refused until peppy stack launch starts the stack. A machine booted from the image starts with an empty stack: its builds wait under ~/.peppy/built_nodes/, and its launch names each one it reuses with a Reusing cached build of line.

Launch timeouts

A launch adds, builds and starts its nodes one phase at a time, and each phase runs under an idle budget: the clock resets whenever the phase produces output (a git or http download’s progress, a build_cmd’s lines, a starting node’s lines), and the phase fails when it goes quiet for longer than its budget. The budgets are per phase and per node:

Flag	Phase	Default
--node-add-idle-timeout-secs	fetching a node’s source and generating its bindings	600
--node-build-idle-timeout-secs	running its build_cmd, or downloading and assembling its image; bytes landing on disk and CPU time the build burns count as output too	180
--node-run-idle-timeout-secs	starting an instance, until it signals ready	600

Every build runs with CARGO_TERM_PROGRESS_WHEN=always and CARGO_TERM_PROGRESS_WIDTH=80, in a build_cmd and in a container def’s %post alike. Without them, cargo prints a line only when a crate download completes when its output is not a terminal, and until then the download writes nothing to disk and burns no CPU, so a large crate on a slow connection reads as an idle build. With these settings cargo reprints its progress line each time the bytes left to download change, and each reprint counts as output. A download that receives nothing changes nothing, so it still runs out the budget. If you set either variable yourself, a build_cmd gets your value; a def sets its own by exporting it in %post.

--max-timeout-secs adds a budget for the whole launch on top of the idle ones; without it, only the idle budgets apply.

When a budget runs out the launch fails, names the phase and, for an idle budget, the flag that raises it, and clears the stack as any failed launch does. What the interrupted phase leaves behind depends on the phase:

• An interrupted add never publishes its node. The daemon generates a node’s bindings into the node’s staged copy on a worker thread of its own, so a generation that is already running when the add’s budget runs out finishes on that thread, and the staged copy is removed once it does; a queued generation whose add is gone is skipped. Either way the node never enters the stack and nothing of it is built.
• An interrupted build has its build_cmd or image build killed.
• An interrupted run has its half-started instance torn down before the failure is reported.

peppy stack build takes the same flags and ends after the build phase, so --node-run-idle-timeout-secs bounds nothing in a build. peppy stack join takes the same flags and runs its copy’s nodes through the same phases, --max-timeout-secs bounding the whole join. A join whose budget runs out fails as any failed join does: it stops what it started and removes the nodes it added, leaving the stack it joined running.

Only the node’s own output resets a phase’s clock; the launcher’s progress lines do not.

Running one launcher across several machines

A launcher can span machines. It declares named placeholders under core_nodes, each instance says which one it runs on via core_node, and peppy stack launch --place <core-node-link>@<core-node> binds those placeholders to real hardware at launch time. Producer links, pairings, and observations all work across the boundary, and consumers still start after their producers. See Federation.

Every placeholder the launcher declares must be bound with --place; a launch that leaves one unwired is refused. The escape hatch is peppy stack launch --local, which ignores the launcher’s placements and runs every instance on the daemon you launched from. --local and --place are mutually exclusive.

Swapping parts at launch time

A launcher file can describe a family of stacks whose members differ in which implementation fills a slot: a real robot or a simulator, a browser panel or a headset, with or without a recorder. The launcher declares component axes, deployments says which option of each it starts, and the launch command swaps one with --with. A launcher without components is a plain flat stack, and that stays the ordinary way to write a one-off.

The whole feature fits in one file. Here is a viewer stack with two swappable cameras:

viewer.json5

// One file, two swappable cameras: the viewer stack declares a `camera`
// axis whose options are written inline, so composition needs no second
// file. `peppy stack launch ./viewer.json5` runs the camera the file
// deploys (`uvc`); the same launcher with `--with=zed` swaps it.
{
  peppy_schema: "launcher/v1",

  components: [
    {
      name: "camera",
      provides: ["cam_inst"],
      options: {
        uvc: {
          deployments: [
            {
              source: { name: "uvc_camera_linux", tag: "v1" },
              instances: [
                {
                  instance_id: "cam_inst",
                  arguments: {
                    device_path: "/dev/video0",
                    video: {
                      frame_rate: 30,
                      resolution: { width: 1280, height: 720 },
                      camera_encoding: "mjpeg",
                      topic_encoding: "rgb8",
                    },
                  },
                },
              ],
            },
          ],
        },
        zed: {
          deployments: [
            {
              source: { name: "zed_camera", tag: "v1" },
              instances: [
                {
                  instance_id: "cam_inst",
                  arguments: {
                    resolution: "hd720",
                    frame_rate: 30,
                    depth: { min_depth_m: 0.4, downscale: 2, block_size: 3 },
                  },
                },
              ],
            },
          ],
        },
      },
    },
  ],

  deployments: [
    {
      source: { name: "web_viewer", tag: "v1" },
      instances: [{ instance_id: "viewer_inst", links: { camera: "cam_inst" } }],
    },
    { camera: "uvc" },
  ],
}

peppy stack launch ./viewer.json5              # runs the camera the file deploys (uvc)
peppy stack launch ./viewer.json5 --with=zed   # runs the ZED camera instead

Components declare what may run; deployments says what runs. An axis entry has four fields:

Field	Meaning
name	The axis’s name, unique in the launcher. Used in --with axis=option, in deployments, and in adjustment guards.
options	The alternatives that fill this axis, at least one. Each option is a fragment: a body using the same deployments grammar the launcher’s own deployments use, written inline, or a path string naming its own file, or the object { fragments: ["fragments/relays.json5", "fragments/mujoco.json5"], ... }, which is how an option names more than one file and how, in a launcher, it carries the adjustments that run when the option is selected.
cardinality	How many selections of the axis a stack allows. one (the default): exactly one option runs. zero_or_one: at most one. zero_or_more: any number, each a named copy. one_or_more: the same copies, with at least one started by every launch and stack remove refusing the last one.
provides	Instance IDs every option defines, so a link written against one is valid whichever option is selected. On an axis that runs as copies the IDs are checked as written, before a copy’s name prefixes them.

A deployments entry deploys a node, { source, instances }, or a component’s option, { <component>: "<option>" }:

• On a one axis the entry names the option that runs, ids as written, and --with swaps it for another option of that axis. A one axis the file does not deploy must be named with --with.
• A zero_or_one axis is never deployed by the file, since nothing on the command line could switch it off. --with turns it on. A feature that is on by default and switchable is a one axis with a none option (its body can be empty), deployed by the file and swapped off with --with none.
• On an axis that runs as copies the entry lists the ones that run, each under instances with a name, or lists none and says how every copy of the option is set up; see copies. --with cannot name one, since a copy needs a name; --join OPTION:NAME starts one with the launch.

Each --with entry is either option (looked up across every axis in reach) or axis=option. A bare word matching options on more than one axis is refused, naming the axes, so axis=option says which you mean. An axis takes one option; repeating it, or naming it once bare and once as axis=option, selects it once. On stack launch and stack build, NAME.option or NAME.axis=option selects the own axis of a copy the file deploys, over the file’s with.

A launch of a launcher with components echoes the full resolution before anything runs, marking the axes a deployments entry filled:

components: simulation=waldo (from file)  robot_control=(off)  scene_commander=(off)
copy alpha: robot=openarm_sim  hardware_version=v2 (from file)  robot_commander=web_commander (from file)  recorder=(off)  camera_rig=none (from file)  brain=(off)

Passing --with to a launcher that declares no components is refused.

Adjustments

A family of stacks often needs more than swapped deployments: selecting the recorder binds it into whichever commander runs, and selecting the headset flips a mode argument on the backbone. An adjustment names one instance and one change to it, and every change names its operation. A fragment writes only the instances it deploys, so the connections between fragments are the launcher’s, written under the option that names the fragment:

components: [
  { name: "simulation", cardinality: "zero_or_one", provides: ["simulation_inst"],
    options: { mujoco: "simulation/fragments/mujoco.json5", isaac_sim: "simulation/fragments/isaac_sim.json5",
               waldo: "simulation/fragments/waldo.json5" } },
  { name: "recorder", cardinality: "zero_or_one", provides: ["recorder_inst"],
    options: {
      lerobot_recorder: {
        // An inline body, beside the launcher's writes for it.
        deployments: [{ source: { name: "lerobot_recorder", tag: "v1" },
                        instances: [{ instance_id: "recorder_inst" }] }],
        adjustments: [
          // Runs when this option is selected; the guard adds a condition
          // of its own, see below.
          { target: "recorder_inst", when: { simulation: "isaac_sim" },
            set_arguments: { robot_type: "openarm_v2_isaac" } },
        ],
      },
    } },
  { name: "robot_commander", provides: ["commander_inst"],
    options: {
      xr_commander: {
        fragments: ["robot_commanders/fragments/xr_commander.json5"],
        adjustments: [
          {
            target: "backbone_inst",              // an instance the launcher's deployments define
            set_arguments: { upstream_mode: "pose" },
            set_links: { collision_ctrl: { vacant: "the headset never produces governor_control" } },
            unset_links: ["leader_left_arm_pose"],
          },
        ],
      },
    } },
],

The launcher’s own top-level adjustments list takes the same entries, guarded on the launcher’s axes, which is how one launcher specializes what every option wrote.

Operation	Effect
set_arguments	Replaces the target’s value for each named argument key, creating it when absent. Nested objects replace wholesale.
set_framework	Replaces the target’s framework knobs, today its clock. How the launcher, under the option that selects a simulated robot, binds the commander and recorder that robot shares with its physical twin to the clock domain this selection runs on.
set_links	Replaces the target’s whole entry for each named slot, creating it when absent. The value uses the ordinary links grammar, including vacancies.
add_links	Appends targets to each named slot’s array binding, creating the binding when the slot has no entry. Refused when the slot holds a scalar or a vacancy: replacing is set_links’s job. Appending is what lets independent options each contribute bindings to one multi-cardinality slot, and what lets a copy that joins add its own instances to a running instance’s set.
unset_links	Drops each named slot’s entry entirely, returning the slot to absent.

A fragment’s own entries run first, in the order its files are read, then the launcher’s: the options’ entries in the order components declares their axes, then the top-level list. The order is what decides a field two entries both write, the later one winning, so a launcher’s entry overrides the fragment’s and its top-level list overrides its options’.

Three kinds of entry run in copies, their target resolved inside each copy:

• an entry under an option of an axis that runs as copies runs in every copy of that option;
• an entry under an option of an axis that fills once, a one or a zero_or_one, or at the top level, guarded on an axis that runs as copies, runs in every copy of the options its guard names;
• an entry whose target only the options of axes that run as copies define runs in every copy of every such axis.

One routing is ambiguous, and it is refused when the launcher is read: an entry that names no axis that runs as copies, neither through the option it sits under nor in its guard, whose target an option of such an axis defines as well as an option that fills once on an axis other than the entry’s own, when the entry’s own option does not define the target: an entry writing an instance its own option defines is that option’s write. The refusal names both, because either reading is a legal launcher: write the entry under the option that fills once to write that option’s instance, or guard it on the axis that runs as copies to write each copy’s.

An adjustment whose target the current selection does not define is skipped, not refused: a top-level write to recorder_inst does not run when the recorder axis is off. A target nothing in the entry’s reach defines is a dead reference and fails when the launcher is read. From the top level, or under an option that fills once, an entry reaches the launcher’s deployments, its own option and every option of the other axes, those that run as copies included; under an option that runs as copies it reaches that option’s instances and the stack’s, never another copy’s.

An option’s adjustments run when that option is selected, so the option is their guard. when is for conditions existence cannot express, mostly shape: two leaders may both provide commander_inst, but only one of them has camera slots. A top-level guard names the launcher’s axes; an option’s names the launcher’s other axes and the option’s own. Each axis accepts an option name or a nonempty list of option names: lists match any listed option (OR), and every named axis must match (AND).

A copy’s selection holds the launcher’s axes and the copied option’s own. So an entry that reaches the copies from an option that fills once, or from the top level, is guarded on the launcher’s axes alone, and one guarded on that option’s own axis is refused when the launcher is read. An entry names one axis that runs as copies, through the option it sits under or its guard, and targets an instance those copies or the stack define; naming two, or targeting anything else, is refused when the launcher is read.

For example, the openarm option binds physical-driver alerts for either commander that consumes alerts:

when: { robot_commander: ["web_commander", "xr_commander"] }

This guard matches both, and the MCP commander does not match. The same scalar-or-list syntax applies inside constraint when, requires, and forbids conditions.

The ownership rule of thumb: if a value is something the operator genuinely decides, make it another option on the axis, visible in the command line. If it is a consequence of a choice already made, write the adjustment under that option, where it needs no guard: it runs exactly when the option is selected and skips when its target is absent.

Constraints

Adjustments change what a selection builds; a constraint refuses selections nobody should run. Some members of a family are stacks no operator should be able to name: the ZED camera that films only the physical rig must not launch against the simulator. Those members flatten cleanly, so neither provides (which only promises ids exist) nor an adjustment’s when guard (which only decides whether one adjustment applies inside a legal selection) can say it. A constraint can, and it only ever refuses: a violating selection is rejected before anything is pinned or started. It never picks an option to satisfy itself.

constraints: [
  // With the ZED camera selected, the robot must be the real one.
  { when: { camera: "zed" }, requires: [{ robot: "real" }],
    reason: "the ZED films the physical rig" },

  // The whole combination is refused, no guard needed.
  { forbids: [{ robot: "mujoco", camera: "zed" }],
    reason: "the stereo camera pairs with the real rig only" },
]

Field	Meaning
when	The guard: the selections this constraint speaks about, in the same axis: options grammar as an adjustment’s when: every named axis holds one of its listed options (an AND across axes, an OR within one). Omit it to speak about every selection.
requires	What the guarded selections must also contain: a list of alternatives, at least one of which must match wholly. Each alternative is an axis: options map where every axis must match (an AND); listing several alternatives is an OR.
forbids	What the guarded selections must not contain: a list of axis: options maps, and matching any one of them (each an AND of its axes) refuses the selection outright.
reason	Why the refused selections must not run. Required, and quoted verbatim in the refusal: peppy can render what was required or forbidden, but only the author knows what the dead combination would have done.

A constraint states at least one of requires and forbids; with neither it refuses nothing, and the document is refused when it is read. The rest of the shape is held to the same standard: when and every entry name at least one axis, an entry may not repeat an axis the guard already fixes (under the guard that axis is decided before anything is selected), the same map may not be listed twice in one list, and one map may not appear in both lists.

Matching treats an unfilled axis as matching no entry, on either side: a guard on camera: "zed" stays quiet when the camera axis is off, and a requires alternative naming that option is satisfied only by selecting it, never by leaving the axis unfilled. That asymmetry is the tool for optional features: requires is how an option demands company, while forbids is the only way to say an optional option never launches with another one (nothing requirable means “that axis stays off”). Where the axis is required, prefer requires of the wanted option instead: requires fails closed for options added later, while a forbids list names today’s options and stays silent about tomorrow’s.

The refusal happens at selection time and names the document that declared the constraint, the full resolution (with its (from file) markers showing the axes the operator never named), what was required or forbidden, and the author’s reason. When several constraints overlap, the first declared one refuses, so the author controls which reason the operator reads.

A launcher’s constraints speak in the launcher’s axis and option names; a fragment’s speak in its own axes and the launcher’s, which is where a rule about an option’s own axes belongs. peppy repo index --check holds them to leaving a usable family behind: a constraint that refuses no selection, an option no legal selection can pick, a bare launch that violates a constraint, and a file’s copy whose own constraints refuse it under every admitted selection (so what the file deploys can never start: loosen a constraint or change the copy’s with) are each reported when the launcher is indexed. A file copy is held only to the stacks its constraints admit: a selection those constraints refuse by design, as a join of it would be, is not flagged.

Fragments: splitting a launcher across files

An option’s body can live in its own file, called a fragment. This is an organizational choice, never a requirement: a launcher whose options are all inline is self-contained. Extract a fragment once a second launcher wants the same body, so the body is stated once.

A fragment file carries the launcher_fragment/v1 tag:

fragments/zed_camera.json5

// The `zed` option of a viewer launcher, extracted into its own file: the
// body is the same `deployments` grammar the launcher itself uses, and the
// fragment tag is what keeps a fragment from ever being mistaken for a
// launchable stack. Referenced from a launcher as
// `zed: "fragments/zed_camera.json5"`.
{
  peppy_schema: "launcher_fragment/v1",

  deployments: [
    {
      source: { name: "zed_camera", tag: "v1" },
      instances: [
        {
          instance_id: "cam_inst",
          arguments: {
            resolution: "hd720",
            frame_rate: 30,
            depth: { min_depth_m: 0.4, downscale: 2, block_size: 3 },
          },
        },
      ],
    },
  ],
}

The launcher references it by path, and the option value becomes that path string:

options: {
  uvc: { deployments: [ /* inline, as above */ ] },
  zed: "fragments/zed_camera.json5",
}

Fragment paths are relative to the declaring document’s directory: the launcher’s for its options, the fragment file’s for the options of its own axes. Inside a Peppy repository, the nearest ancestor containing a valid peppy_repository.json5 sets the boundary: openarm/fragments/openarm_sim.json5 can include ../../robot_commanders/fragments/xr_commander.json5. Parent segments and resolved symlinks must stay within that repository. Standalone launchers use their own directory as the boundary. Absolute paths are rejected. Repository names and filesystem paths use the same resolution rules.

The object form of an option names its files under fragments, read in order, and may carry a body of its own beside them; two options share one body by naming the same file:

// In the launcher, whose `adjustments` under an option are its own writes.
options: {
  mujoco: { fragments: ["fragments/relays.json5", "fragments/mujoco.json5"] },
  isaac_sim: { fragments: ["fragments/relays.json5", "fragments/isaac_sim.json5"],
               adjustments: [
                 { target: "viewer_inst", set_arguments: { renderer: "path_traced" } },
               ] },
}

An option declared inside a fragment, or inside an option body, carries no adjustments of its own: the writes a fragment makes are its top-level adjustments, which name the instances it deploys, those of an option whose body is written inside it included, and everything else is the launcher’s, under the option that selects the fragment.

The parts of one option declare its axes between them, each axis once, and deploy each of those axes once. Besides deployments, components, constraints, adjustments and framework, a fragment may declare core_nodes: additional placement placeholders its instances need, unioned with the launcher’s own, so selecting the option makes the matching --place bindings required exactly as if the launcher declared them. An entry in a fragment’s adjustments naming an instance the fragment does not deploy is refused when the fragment is read, naming what it does deploy. The connections between fragments are the launcher’s, written under the option, as above.

A fragment is not a stack. The repository indexer never lists a fragment as a launchable launcher, it needs no entry in peppy_repository.json5, and it cannot collide with a launcher name. peppy repo index --check validates every fragment a launcher references and flattens every legal selection of its axes, so a broken combination is refused on the branch that introduced it.

An option’s own axes

An option’s fragment can declare components of its own: the axes that belong to that option. An OpenArm robot fragment declares its robot commander and its recorder, and deploys the browser panel unless told otherwise:

fragments/openarm_sim.json5

{
  peppy_schema: "launcher_fragment/v1",
  deployments: [
    { source: { name: "openarm_backbone", tag: "v1" }, instances: [{ instance_id: "backbone_inst" }] },
    { hardware_version: "v2" },
    { robot_commander: "web_commander" },
    { camera_rig: "none" },
  ],
  components: [
    // The hardware version the simulation stands: each option deploys the
    // initializer that names its model, inline.
    { name: "hardware_version", provides: ["init_inst"],
      options: {
        v1: { deployments: [{ source: { name: "robot_initializer", tag: "v1" },
                              instances: [{ instance_id: "init_inst", framework: { clock: "simulation" },
                                            arguments: { model: "openarm_v1" }, links: { simulation: "simulation_inst" } }] }] },
        v2: { deployments: [{ source: { name: "robot_initializer", tag: "v1" },
                              instances: [{ instance_id: "init_inst", framework: { clock: "simulation" },
                                            arguments: { model: "openarm_v2" }, links: { simulation: "simulation_inst" } }] }] },
      } },
    { name: "robot_commander",
      options: { web_commander: "web_commander.json5", xr_commander: "../../robot_commanders/fragments/xr_commander.json5",
                 mcp_commander: "../../robot_commanders/fragments/mcp_commander.json5", ker_commander: "ker_commander.json5" } },
    { name: "recorder", cardinality: "zero_or_one", provides: ["recorder_inst"],
      options: { lerobot_recorder: "lerobot_recorder.json5" } },
    { name: "camera_rig",
      options: { none: "../../common/fragments/none.json5", cameras_sim: "cameras_sim.json5" } },
    { name: "brain", cardinality: "zero_or_one", provides: ["brain_inst"],
      options: { ai_brain: "ai_brain.json5" } },
  ],
}

A fragment’s axes are one or zero_or_one; how many robots run is the launcher’s robot axis. They are in reach whenever the fragment runs once: on a launcher whose robot axis is one, --with xr_commander reaches into the robot’s fragment, and --with robot_commander=xr_commander says so explicitly. A fragment’s adjustments and constraints speak in its own axis names and the launcher’s, so the “cameras need a consumer” rule and the recorder’s trigger rule live in the robot fragment beside the axes they speak about, and the MCP commander’s requirement names the launcher’s robot_control axis. A launcher’s own adjustments and constraints speak in the launcher’s axis names. A fragment two levels below the launcher declares no axes: an option of a fragment’s axis is a plain fragment.

Copies

An axis with cardinality zero_or_more or one_or_more runs as copies: named instances of one of its options, exactly as a node runs as named instances. The launcher’s deployments list the copies a launch starts, peppy stack join adds one to a running stack, and peppy stack remove takes one out. Every instance a copy’s fragments define is minted as NAME_<id>, so any number of copies run side by side under one launcher. The <id> a fragment writes is the instance’s name inside the copy, and a per-robot MCP surface publishes it: it is the value a client passes to pick which of a robot’s cameras or limbs a call addresses, so renaming an instance in a fragment renames it for every client of that surface.

The openarm_simulation launcher holds the simulation as a one axis, and the robot as copies:

components: [
  { name: "simulation", provides: ["simulation_inst"],
    options: { mujoco: "...", isaac_sim: "...", waldo: "..." } },
  { name: "robot", cardinality: "zero_or_more",
    options: { openarm_sim: "...", so101_sim: "..." } },
],
deployments: [
  { simulation: "waldo" },
  { robot: "openarm_sim", instances: [
      { instance_id: "alpha", arguments: { commander_inst: { http_port: 8765 } } },
  ] },
],

The same axis written cardinality: "one_or_more" is the same copies with a floor under them: a launch starts at least one, and stack remove refuses the last one, so every stack of the launcher runs a robot. The copy can come from deployments, as the entry above does, or from --join OPTION:NAME on the launch, so a launcher that deploys none makes the choice of robot part of the command line. A launch that would start none is refused, naming the axis and both ways to fill it.

{ name: "robot", cardinality: "one_or_more",
  options: { openarm_sim: "...", so101_sim: "..." } },

peppy stack launch openarm_simulation                               # the copy the file deploys
peppy stack launch openarm_simulation --join openarm_sim:alpha      # when the file deploys none

A copy entry has four fields: instance_id, the copy’s name; with, the selection of the option’s own axes, an axis left out taking what the fragment deploys; arguments, overrides of the copy’s instances’ arguments, keyed by the id written in the fragment and then by argument; and adjustments, the fragment verbs over the copy’s instances. The option entry itself takes with and arguments too, shared by every copy of the option, listed or joined, and a copy’s own values win per axis and per argument:

{ robot: "openarm_sim", with: { robot_commander: "xr_commander" }, instances: [
    { instance_id: "alpha" },
    { instance_id: "bravo", with: { hardware_version: "v1" }, arguments: { commander_inst: { https_port: 4444 } } },
] },

An entry and a copy also take adjustments, the same verbs the launcher’s adjustments use (set_arguments, set_framework, set_links, add_links, unset_links) with an optional when over the launcher’s axes that fill once and the copy’s own, targeting the copy’s instances by the ids written in the fragment. They run after the launcher’s adjustments, the entry’s before the copy’s, and the copy’s arguments last. On stack launch and stack build, a --with word NAME.axis=option or NAME.option selects one copy’s own axis for that launch, over the file’s with, whether the file lists that copy or --join names it:

peppy stack launch openarm_simulation --with mujoco,alpha.xr_commander

An entry may list no copy. It then says how every copy of the option is set up, and a launch starts none of them; --join OPTION:NAME on stack launch, once per copy and comma-separated as --with is, starts a copy with the launch: a copy of the entry under that name, composed, validated and started with the launch, and failing with it. The words that address a listed copy address it too, --with NAME.option and --place NAME@CORE_NODE:

deployments: [
  { simulation: "waldo" },
  { robot: "so101_sim", with: { robot_commander: "mcp_commander", camera_rig: "cameras_sim" } },
],

peppy stack launch simulation_mcp                                                    # the simulation and the copies the file lists
peppy stack launch simulation_mcp --join so101_sim:charlie                           # one more, as the entry sets it up
peppy stack launch simulation_mcp --join so101_sim:charlie,so101_sim:delta --with delta.xr_commander

A copy named on the command line is composed as a join. --join lays it over the configuration the file states and over the copies named before it, as stack join lays a copy over the running stack, so what a launch accepts a later stack join accepts too, and a robot you can name at launch is a robot you can also join later. A name the launch already starts, listed or joined, is refused.

One robot, two robots, many:

# The simulation and the copies the file deploys.
peppy stack launch openarm_simulation

# The same copies in MuJoCo.
peppy stack launch openarm_simulation --with mujoco

# Another robot in place of the first, on another machine, then its removal.
peppy stack remove alpha
peppy stack join openarm_sim:bravo --with xr_commander --place jetson-2
peppy stack list
peppy stack remove bravo

# A stack with no robot yet, then robots as they arrive.
peppy stack launch physical
peppy stack join openarm:alpha --place jetson-1
peppy stack join openarm:bravo --with v1,xr_commander --place jetson-2

# The same two robots, a v2 and a v1, named at launch.
peppy stack launch physical --join openarm:alpha --join openarm:bravo --with bravo.hardware_version=v1,bravo.xr_commander --place alpha@jetson-1 --place bravo@jetson-2

stack join OPTION:NAME names the option to copy and the copy’s name, the one spelling --join takes too. Its --with words select the copied option’s own axes. Every instance the copy’s fragments define receives the name as a prefix: alpha_left_arm_inst, alpha_backbone_inst, alpha_commander_inst. Every link naming one of them follows it under the prefix, including array links and pairing targets such as alpha_backbone_inst/leader_left_arm; links to stack instances such as simulation_inst keep their IDs. The name is also the copy’s core node link: --place alpha@jetson-1 at launch and --place jetson-1 at join run the whole copy there, and a copy with no placement runs on the coordinator.

A joined copy starts from the same settings as a copy the file lists. Weakest first: what the option’s fragment deploys, then the with, arguments and adjustments of the launcher’s entry for that option, then the join’s --with words and --set-arguments. A word wins on its own axis and the entry’s other axes stay selected, so the option’s constraints judge the two together, as they do for a launch word NAME.option. Under the xr_commander entry above, a plain peppy stack join openarm_sim:charlie is another headset robot, and peppy stack remove alpha followed by peppy stack join openarm_sim:alpha brings back the copy the file deployed. One entry speaks for every copy of its option, so a launcher deploys each option of an axis that runs as copies in one entry, and a second entry naming the same option is refused.

--set-arguments INSTANCE.ARGUMENT=JSON5 on stack join overrides one argument of one of the copy’s instances, named as the fragment writes it, the way a copy entry’s arguments does in the file. It applies after every adjustment, over the option entry’s arguments, once per argument on the command line. Numbers must be finite; quote string values:

peppy stack join openarm:charlie --with xr_commander \
  --set-arguments backbone_inst.max_ee_velocity_m_s=0.2 \
  --set-arguments commander_inst.https_port=4444

A launch snapshots its launcher and fragment files, and every join composes against that snapshot; launch again to load edits. Constraints see the stack’s selection and the copy’s together, so mcp_commander on an openarm_simulation stack launched without robot_control is refused at join with the commander’s reason. The launcher’s adjustments under a copy’s option may write to a stack instance, and each field they write is what the copy needs of it: the copies the file lists must agree on every field they both write, and a joined copy, whether --join names it at launch or stack join adds it later, may write to a stack instance only what already runs there, with two exceptions: its own instances may pair into the slots the instance declares vacant (the pairing instance’s links name the slot, and the launcher’s adjustments under the copied option release the vacancy with unset_links on the instance), and they may join the one_or_more or zero_or_more sets it declares, below. A join that would change a running instance any other way is refused, naming the instance and each field that differs. That is why the rendered camera rig, which turns the simulation’s rendering on, is selected in the file at launch.

A stack instance that watches or consumes every robot declares the slot as a set, and the launcher adds each robot under its option. The stack runs the monitor:

deployments: [
  { source: { name: "fleet_monitor", tag: "v1" },
    instances: [{ instance_id: "monitor_inst" }] },
],

where the monitor’s own manifest declares the slot as a set:

depends_on: {
  nodes: [
    { name: "openarm_backbone", tag: "v1", link_id: "robots", cardinality: "zero_or_more" },
  ],
},

and the launcher adds each copy’s backbone to that robots slot under the robot option:

options: {
  openarm: {
    fragments: ["openarm/fragments/openarm.json5"],
    adjustments: [
      { target: "monitor_inst", add_links: { robots: ["backbone_inst"] } },
    ],
  },
},

peppy stack launch monitored          # monitor_inst reads robots as []
peppy stack join openarm:alpha        # [alpha_backbone_inst]
peppy stack join openarm:bravo        # [alpha_backbone_inst, bravo_backbone_inst]
peppy stack remove alpha              # [bravo_backbone_inst]

The members follow the order the copies joined, and a copy that rejoins under the same name is a member once, at the end. A member a join adds behaves like one the launch writes: it is subscribed like any other, keeps its position, and an observed member reads as not live while its source is down. A join adds only the copy’s own instances, and only to a one_or_more or zero_or_more producer or observer slot of the instance’s manifest; naming an instance the stack runs, a one or zero_or_one slot, or a pairing slot is refused at join, naming the slot and the fix. A join reserves the machine of every instance whose set it grows, wherever that instance runs, and is refused when that machine is not live on the federation or no longer runs the instance, naming the slot, the machine and the fix. A copy the launcher file deploys at launch may add a stack instance to a set, beside or in place of its own; removing that copy takes the member out again, and rejoining it by name is refused, so such a copy is launched with the stack.

One simulation per stack, and each simulated robot joins it as a copy that brings its own model: the openarm_simulation launcher ships one copy, an OpenArm v2, --join OPTION:NAME names another at launch and stack join adds more, of any model the simulation’s catalog carries, as many as the machine can run. A physical robot reads wall time and a simulated one the simulation’s clock, so the physical robots have a launcher of their own, physical.

stack remove NAME stops the copy’s instances in reverse dependency order, takes the members it added out of the running sets, and takes the copy out of the stack; every instance whose set shrank returns the smaller set at its accessor’s next call. A set on a machine that is not live keeps the members it holds, and the removal says so; when the machine returns with the instance still running, the next join or removal changing that slot delivers it whole, and when it returns without it, launch again. Two floors refuse a removal: one that would leave a one_or_more slot with no member is refused, naming the slot, and one that would take the last copy of a one_or_more axis is refused, naming the copy and the axis. A copy another copy links to is refused until that copy goes first, and a copy a stack instance links to any other way stays until the next launch. Removal leaves the stack’s own instances running: removing a simulated robot takes it out of the simulation, which stays up for the next join, and stack reset stops everything. A copy that released a stack instance’s vacancy to pair into that slot restores the vacancy when it goes, so the slot stands vacant again as the launch declared it, the mirror of pairing into a vacancy above. A failed join stops what it started, removes the nodes it added, and reports the failure; when that cleanup itself fails the copy stays in stack list and the message says to retry stack remove.

stack list names each copy, the option it copies, its host, its selection, its instances and the set members it added, one row per slot under Set members: (monitor_inst.links.robots → alpha_backbone_inst, the target by the id the stack runs it under; the bindings table adds its machine), and its bindings table names the copy beside each member a copy brought (robots → alpha_backbone_inst@cn-robot [copy alpha]); stack list --json carries the same under copies, the members under set_members, and each member of a slot’s slot_bindings under producer, with a copy key on a member a copy brought, naming the copy and the id its fragment wrote for the instance. The node holding the set reads the same fact through its bound_members() accessor. Preview a launch and the copies it starts without starting anything, which composes it and runs the link rules a launch runs, leaving what only a running stack can answer (whether the instance whose set grows is running, and whether its machine is live) to the join itself:

peppy stack resolve openarm_simulation --with mujoco
peppy stack resolve physical --join openarm:alpha --join openarm:bravo --with bravo.hardware_version=v1,bravo.xr_commander

Join and remove run on the coordinator (--core-node NAME reaches it from another machine), serialize with launches and with each other, and reserve only the machines they touch: an unrelated host that is offline blocks neither. A reset cancels the operation in flight, waits for its cleanup, and empties the stack. A restarted coordinator has no launcher to compose against; launch again before joining.

peppy repo index --check composes every legal selection the way a launch does: the stack under every selection of its axes, then every copy an axis that runs as copies can add, as a launch beside the stack and as a join onto it; an entry listing no copy is composed once under the option’s own name. An axis the file leaves for the command line, a one axis for --with or a one_or_more axis for --join, has no bare launch, and the check is silent about it.

Auditing a composed launch

Composition changes instances from a distance, so it ships with an audit command:

peppy stack resolve physical --join so101:alpha --with alpha.xr_commander,alpha.lerobot_recorder

stack resolve needs no running stack. It prints the flattened launcher/v1 document to stdout and a resolution report to stderr. An excerpt:

components: robot_control=(off)
copy alpha: robot=so101  robot_commander=xr_commander  recorder=lerobot_recorder  camera_rig=(off)
adjustments applied:
  alpha_backbone_inst.arguments.upstream_mode: "joints" -> "pose"  (so101/fragments/so101.json5)
  alpha_commander_inst.links.recorder: + alpha_recorder_inst  (robot_commanders/fragments/xr_commander.json5)
  alpha_commander_inst.links.right_arm: (absent) -> "alpha_backbone_inst/leader_pose"  (physical.json5, option `robot.so101`)
  alpha_recorder_inst.arguments.robot_type: (absent) -> "so101"  (physical.json5, option `robot.so101`)
adjustments skipped:
  alpha_commander_inst: guard camera_rig=cameras robot_commander=xr_commander is not met  (physical.json5, option `robot.so101`)
alpha_backbone_inst: clock `wall`
alpha_commander_inst: clock `wall`
alpha_recorder_inst: clock `wall`
link rules hold: slot keys, vacancies, pairing coverage, observation sources and clock agreement checked over 5 node manifest(s)

Every applied adjustment is listed with the value it replaced and where it was written, the fragment’s own entries under the fragment’s path and the launcher’s under the option that carries them, and every skip says why. Because stdout is a plain launcher, it doubles as the escape hatch: flatten, hand-edit, launch the flat file. A launcher that deploys copies resolves with them, one copy NAME: line each, and --join OPTION:NAME resolves one more, repeatable and comma-separated, each joined onto the plan the ones before it left, exactly as stack launch composes them. A copy --join names takes its own axes from --with NAME.option. The resulting flat launcher declares each copy’s name in core_nodes; launch that file with --place NAME@CORE_NODE or --local.

The report ends by holding the flat plan to the launch-time link rules that need no daemon. stack resolve reads this machine’s nodes cache at ~/.peppy/cache/nodes.json5, loads each deployed node’s manifest, and checks slot-key legality, vacancies, and pairing coverage across them; the link rules hold line names how many manifests it checked. A node published by a git repository is read out of the checkout the caches already hold, peppy repo refresh handing every clone it makes to the checkout cache, so the nodes and launchers a hub publishes are checked against each other without the command ever reaching the network. The contract-binding rules stay launch-only, because satisfying them can involve the root node that only a running daemon knows. When the plan breaks a rule a launch would reject, say a zero_or_one pairing slot left neither paired nor vacant, stack resolve fails with that launch’s own error instead of surfacing it minutes later at launch. When a manifest cannot be read locally the check is skipped rather than run on a partial picture: the report says link rules not checked, names the missing manifest, and advises peppy repo refresh, which is what fills both the nodes cache and the checkouts the git-backed manifests are read from.