Launch files

Launch files let you start multiple nodes simultaneously, including those with dependencies on each other. With a single launch file, you or your team can recreate an entire node environment using just one command.

Running the example

Start by cloning this repository:

1. Clone the repository:

   git clone https://github.com/Peppy-bot/launchers-hub

2. Navigate into the directory:

   cd launchers-hub

In the examples/ directory, you’ll find a launcher file for each language. Opening it reveals the following structure:

Python

examples/python_robot.json5

Rust

examples/rust_robot.json5

1. The file is identified by peppy_schema: "launcher/v1" at the root: peppy uses this to distinguish launcher files from node peppy.json5 files (which use "node/v1")
2. All deployments are defined in the deployments array
3. Each deployment requires a source (the node to run, or the MCP exposures to serve) and an instances array (how many copies to start, and how each one is configured). Every entry needs an instance_id; arguments, env_vars, framework, core_node, and links are optional. See Instance attributes for what each one does

The source attribute references a node from your configured repositories by name and tag. Specify them as separate fields, or use the combined name: "<name>:<tag>" shorthand. The node is resolved against your local nodes cache at ~/.peppy/cache/nodes.json5. A source can instead list MCP exposures to serve, as shown below.

If a node you want to deploy lives outside your configured repositories, add its location first: register a local directory with peppy repo add /path/to/my/nodes, or a git repository with peppy repo add https://github.com/org/repo.git. A repository states what it publishes through a committed peppy_repository.json5, so run peppy repo index /path/to/my/nodes to write it, and re-run it after adding, moving, or renaming a node, since the refresh reads the index rather than the filesystem. Then peppy repo refresh makes the nodes resolvable (see Repositories).

Note

The source resolves nodes from the locally cached index of the machine the launch is submitted to. Run peppy repo refresh on that machine before launching to ensure its cache is up to date. In a federated launch it is the only machine whose cache matters: it resolves every deployment and its dependencies once and ships the result to the other machines as pinned content, so their caches need no refresh and cannot change what the launch runs.

If a node or tag is missing from the cache, the launch fails with a “not found” error. If the cache instead contains the same name:tag twice from one repository, the launch fails with a distinct ambiguity error naming both manifests, never “not found”, since the node is present rather than absent. The refusal happens while sources are being resolved, before anything is added, built, or run, and it covers transitive dependencies too: an ambiguity two levels down is exactly as fatal as one at the top. See Using a repository-indexed node.

// Repository source (long form)
{
  source: {
    name: "openarm01_controller",
    tag: "v1",
  },
  instances: [
    { instance_id: "the_nervous_system" }
  ]
}

// Repository source (combined shorthand, equivalent to the example above)
{
  source: {
    name: "openarm01_controller:v1",
  },
  instances: [
    { instance_id: "the_nervous_system" }
  ]
}

Serving MCP exposures

A deployment whose source lists exposures runs the MCP server built into peppy for them instead of a node. Each entry is an MCP exposure as "<name>:<tag>", resolved against ~/.peppy/cache/mcp_exposures.json5, and the contracts the exposures reference resolve through the contracts cache. The daemon derives the server’s node from those documents, so the deployment behaves like any other: its instances take arguments (the server’s single port), links for the contract slots the exposures’ contract targets become, and core_node for placement. A daemon target takes no link: the instance gives it a scope in daemon_scopes. The list must not be empty or repeat an exposure, and two deployments cannot list the same set.

// Exposure source: the built-in MCP server serving two exposures on one port
{
  source: {
    exposures: ["camera_and_recording:v1", "arm_status:v1"],
  },
  instances: [
    {
      instance_id: "mcp_server",
      arguments: { port: 8900 },
      links: { front_camera: "front_cam_inst", recorder: "recorder_inst", arm: "arm_inst" },
    }
  ]
}

Caution

A git ref pinned in repositories.json5 and the node tag declared in peppy.json5 are independent values, and they do not share the same format: git refs can contain dots (e.g. v0.1.0), but node tags must start with an ASCII letter and forbid dots (e.g. v1, donut). This means a repository could be pinned to the git ref v0.1.0 while peppy.json5 specifies tag: "v1". Keep them coordinated so a checkout of a given git ref lines up with the tag your launcher expects.

From within the cloned repository, execute the launcher:

Python

peppy stack launch ./examples/python_robot.json5

Rust

peppy stack launch ./examples/rust_robot.json5

Peppy automatically inspects the name and tag of each node, verifies that all dependencies are met, and constructs the node stack in the correct order.

Launching from a repository

Launchers discovered by peppy repo refresh (see Repositories) can be invoked by bare name, no path required:

peppy stack launch openarm01_sim_teleop

An argument that contains a path separator or ends in .json5 is resolved as a filesystem path: ./my_launcher.json5, /abs/my_launcher.json5, my_launcher.json5. A bare name (no separator, no extension) is a repository launcher and is never looked up on disk, whatever the current directory holds: peppy stack launch my_launcher resolves through the repository cache at ~/.peppy/cache/launchers.json5 even when a my_launcher.json5 sits next to the caller. Run peppy repo refresh first on the machine you submit the launch to; the launcher document, like every repository source, resolves on that machine alone.

If the name resolves neither to a file nor to a cached launcher, the launch fails with an explicit “not found” error. If one repository provides two launcher files with that name, the launch fails with a distinct ambiguity error naming both: launchers are keyed by filename, so two identically named .json5 files anywhere in one repository contest the same name even in different directories.

Caution

Running the peppy stack launch or peppy stack build command clears the existing node stack and stops all running instances

A launch ends with the log file of every node it added, built and started, then the URLs the started instances serve, and the completion line. The URLs come from the nodes that declare execution.endpoints: the daemon hosting each instance expands the socket the node announced into one URL per address of its machine, loopback first, and the core node name after the @ says which machine that is. Pages an operator opens in a browser and endpoints an MCP client attaches to are printed under separate headings, Web pages: before MCP endpoints:, and a heading appears only when it has an entry. peppy stack join prints the same blocks for the instances the copy started.

Node log files:
  ...
Web pages:
  alpha_commander_inst (openarm_web_commander:v1) @cn-sweet-edison
    panel   http://127.0.0.1:8765
            http://100.123.58.116:8765
  simulation_inst (waldo:v1) @cn-sweet-edison
    viewer  https://127.0.0.1:8080/
            https://100.123.58.116:8080/
MCP endpoints:
  robot_control_inst (mcp_robot_control_v1:builtin) @cn-sweet-edison
    robot_control_v1  http://127.0.0.1:8900/robot_control/v1/mcp
  world_control_inst (mcp_simulation_v1:builtin) @cn-sweet-edison
    simulation_v1     http://127.0.0.1:8902/simulation/v1/mcp
Launch completed successfully

Verify that the node stack was configured correctly:

$ peppy stack list
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Core node: cn-adoring-zebra (host: robot-host)                                                                                                                                                │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Slice of launch launch-brave-otter (coordinator: cn-adoring-zebra)                                                                                                                            │
│                                                                                                                                                                                               │
│ Node stack                                                                                                                                                                                    │
│                                                                                                                                                                                               │
│ ┌──────────────────────────────────────┬───────┬───────────┬────────────────────────────────────────────────────────────────────────────┐                                                     │
│ │ NODE                                 │ STAGE │ INSTANCES │ PATH                                                                       │                                                     │
│ ├──────────────────────────────────────┼───────┼───────────┼────────────────────────────────────────────────────────────────────────────┤                                                     │
│ │ cn-adoring-zebra:v0.10.0             │ Root  │ 1 running │ ~/workspace/peppy                                                          │                                                     │
│ │ fake_openarm01_controller:v1         │ Ready │ 1 running │ ~/.peppy/built_nodes/fake_openarm01_controller_v1/b81d0e4c72a5f936.tar.zst │                                                     │
│ │ fake_robot_brain:v1                  │ Ready │ 1 running │ ~/.peppy/built_nodes/fake_robot_brain_v1/6c2e9f0a1d47b385.tar.zst          │                                                     │
│ │ fake_uvc_camera:v1                   │ Ready │ 2 running │ ~/.peppy/built_nodes/fake_uvc_camera_v1/e57a3b91c0d64f28.tar.zst           │                                                     │
│ │ fake_video_reconstruction:v1         │ Ready │ 1 running │ ~/.peppy/built_nodes/fake_video_reconstruction_v1/19d4c7e2a8b03f65.tar.zst │                                                     │
│ └──────────────────────────────────────┴───────┴───────────┴────────────────────────────────────────────────────────────────────────────┘                                                     │
│                                                                                                                                                                                               │
│ Instance bindings                                                                                                                                                                             │
│                                                                                                                                                                                               │
│ ┌──────────────────────────────────────┬──────────────────────────────┬─────────┬─────────┬──────────┬───────┬──────────────────────────────────────────────────────────────────────────────┐ │
│ │ NODE                                 │ INSTANCE                     │ STATUS  │ HEALTH  │ RESTARTS │ CLOCK │ BINDINGS                                                                     │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────┼───────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ cn-adoring-zebra:v0.10.0             │ cn-adoring-zebra             │ running │ healthy │ 0        │ wall  │ (none)                                                                       │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────┼───────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ fake_openarm01_controller:v1         │ vibrant-keller-2310          │ running │ healthy │ 0        │ wall  │ (none)                                                                       │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────┼───────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ fake_robot_brain:v1                  │ zealous-bohr-5512            │ running │ healthy │ 0        │ wall  │ fake_openarm01_controller → vibrant-keller-2310@cn-adoring-zebra             │ │
│ │                                      │                              │         │         │          │       │ fake_uvc_camera → mystic-galois-7701@cn-adoring-zebra                        │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────┼───────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ fake_uvc_camera:v1                   │ mystic-galois-7701           │ running │ healthy │ 0        │ wall  │ (none)                                                                       │ │
│ │                                      │ brave-noether-3098           │ running │ healthy │ 0        │ wall  │ (none)                                                                       │ │
│ ├──────────────────────────────────────┼──────────────────────────────┼─────────┼─────────┼──────────┼───────┼──────────────────────────────────────────────────────────────────────────────┤ │
│ │ fake_video_reconstruction:v1         │ gallant-curie-4417           │ running │ healthy │ 0        │ wall  │ fake_uvc_camera → brave-noether-3098@cn-adoring-zebra                        │ │
│ └──────────────────────────────────────┴──────────────────────────────┴─────────┴─────────┴──────────┴───────┴──────────────────────────────────────────────────────────────────────────────┘ │
│                                                                                                                                                                                               │
│ Dependencies                                                                                                                                                                                  │
│   fake_robot_brain:v1 ➔ fake_openarm01_controller:v1                                                                                                                                          │
│   fake_robot_brain:v1 ➔ fake_uvc_camera:v1                                                                                                                                                    │
│   fake_video_reconstruction:v1 ➔ fake_uvc_camera:v1                                                                                                                                           │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

The output confirms that all nodes have been added to the stack along with their dependency relationships. When an instance serves an endpoint, an Instance endpoints table follows the bindings, one row per URL with the endpoint’s KIND and LABEL. The Instance bindings table breaks that down per instance: the consumers (fake_robot_brain, fake_video_reconstruction) list the depends_on slots they resolved, while the producers (fake_openarm01_controller, fake_uvc_camera) and the core node declare no slots and show (none). A resolved binding shows the chosen producer(s) as instance_id@core_node, ordered by link ID; a multi-cardinality slot lists every bound member in plan order, and an empty slot, whether a zero_or_more slot bound to nothing or a zero_or_one slot declared vacant, shows (empty set). Every declared producer slot (depends_on.nodes / depends_on.contracts) is either bound or, where its manifest declares cardinality: "zero_or_one", declared vacant; only a zero_or_more slot may be left out entirely, and a launcher that leaves any other one out is rejected at validation (see dependency cardinality). Pairing and observer slots follow the same shape: a participant slot is paired and an observer slot is linked, or, where its manifest declares it cardinality: "zero_or_one", declared vacant, unless its cardinality is zero_or_more, where omitting it IS its empty set. The launch also prints a Slice of launch ... line above the tables naming the launch that produced this stack and its coordinator.

Instance attributes

Every entry in an instances array configures one running copy of the deployment’s node. Two instances of the same node are independent: they get their own arguments, their own environment, and their own bindings.

peppy_launcher.json5

{
  source: { name: "uvc_camera:v1" },
  instances: [
    {
      instance_id: "camera_front",
      arguments: { device: { path: "/dev/video0" }, frame_rate: 30 },
      env_vars: { GST_DEBUG: "2" },
      framework: { clock: "simulation" },
    },
    {
      instance_id: "camera_rear",
      arguments: { device: { path: "/dev/video1" }, frame_rate: 15 },
    },
  ],
}

instance_id (required)

The name this copy runs under. It labels the instance in peppy stack list, in its log file, and in every links value that points at it.

Allowed characters are letters, digits, _ and -. The id must be unique across the whole stack, not just within its own deployment: two deployments cannot both declare camera_front, and no instance may take the core node’s own id. A collision is reported before anything starts.

arguments (optional)

Values for the parameters the node declares in its execution.parameters block, written with the same nesting as that schema. See the Parameters reference for the full syntax.

A parameter that declares a $default may be omitted, and the daemon fills it in before the node starts. A required parameter left without a value fails the launch as that instance starts, naming the leaf it is missing (Missing required parameters: device.serial). Unknown keys and type mismatches are caught by the node itself as it boots, so they surface as that instance failing to come up.

A container node can also reference these values in its mount_paths with ${parameters:<path>}, which is how two instances of one node mount two different host devices.

env_vars (optional)

Environment variables for this instance’s process.

peppy stack launch forwards a filtered copy of your shell environment to the instances it starts on the machine you launch from, so a node there already sees your PATH and the rest. The filter drops PWD, OLDPWD, TMPDIR, TEMP, TMP, PYTHONPATH, and PYTHONHOME, plus any name that is not a shell identifier, any value with whitespace or shell metacharacters, and the injection-prone names listed below. PYTHONPATH and PYTHONHOME are stripped so a caller shell that exports them cannot displace a container node’s vendored peppylib/peppygen path; a node that genuinely needs either sets it through its own env_vars. An instance placed on another machine receives only its declared env_vars. env_vars is layered on top of the forwarded set for what differs per instance (a device path, a board serial, a log level), and a name declared here replaces the forwarded value:

{
  source: { name: "esp32_board:v1" },
  instances: [
    { instance_id: "esp32_left",  env_vars: { ESP32_DEVICE: "/dev/ttyUSB0" } },
    { instance_id: "esp32_right", env_vars: { ESP32_DEVICE: "/dev/ttyUSB1" } },
  ],
}

Entries are checked when the launcher file is read, before the running stack is touched:

• Names must be shell identifiers: a letter or _ first, then letters, digits and _. ESP32_DEVICE is fine, esp32-device is not.
• Values may hold letters, digits and _ - . / : , = @ + %. Whitespace, quotes and shell metacharacters (;, $, `, |, and friends) are rejected, so a value like "--flag other" has to reach the node another way, typically as an argument.
• Injection-prone names are rejected outright: the dynamic-linker and shell hooks (LD_PRELOAD, LD_LIBRARY_PATH, DYLD_INSERT_LIBRARIES, BASH_ENV, IFS, and the rest of that family).

Note

The name and value rules exist because a container node receives its environment through Apptainer, which forwards each variable under the APPTAINERENV_ prefix into a script the container sources at startup, where the shell expands the value. A name that is not a shell identifier would not parse as an assignment, and a value carrying $, a backtick or a quote would be expanded or run inside the container rather than arriving as written. The same rule applies to process nodes so that adding a container block to a node later cannot break the launchers that already deploy it.

env_vars reach the instance when it starts. Fetching and building a node happen once for the node, not once per instance, so a variable your build needs (a registry token, a toolchain path) belongs in the shell you run peppy stack launch or peppy stack build from, where it is forwarded to build steps on that machine too. A build on another machine sees none of your caller environment: your environment describes your machine, so it stays there.

framework (optional)

Framework knobs peppy reads the same way for every node. The block holds two knobs:

• clock: the timeline this instance reads, naming wall or a domain the launcher declares under framework.clocks. An instance that omits it reads wall time. An instance a declaration names as a domain’s publisher carries no clock of its own, since that declaration already assigns it one. See System clock.
• restart: what the daemon does when this instance’s process exits on its own or stops answering health checks, never, on_failure or always, or { policy, max_restarts }. It wins over the node’s own framework.restart and the launcher root’s. See Restart policy.

An unknown key inside framework fails the launch, so a typo surfaces where it is written.

core_node (optional)

The machine this instance runs on, named by one of the placeholders the launcher declares under core_nodes. An instance that omits it runs on the daemon you launched from. See Running one launcher across several machines.

links (optional)

One map filling every kind of depends_on slot the node declares, keyed by the slot’s link_id. A value is interpreted against the slot it names:

• a producer slot (depends_on.nodes / depends_on.contracts) takes the producer instance to bind: a scalar for a one or zero_or_one slot, an array for a multi-cardinality slot;
• a pairing participant slot takes the peer instance(s) to pair with, each "<peer>" or "<peer>/<peer_link_id>" to disambiguate which of the peer’s slots to use: a scalar for a one or zero_or_one slot, an array for a multi-cardinality slot, one pair per entry;
• an observer slot takes the source instance(s) to observe, each "<source>" or "<source>/<source_link_id>": a scalar for a one slot, an array for a multi-cardinality slot, in the order the node reads them back.

A slot that starts unresolved on purpose says so here too, as { vacant: "<why>" }. That value binds nothing and covers the slot, so the fate of every declared slot is one entry in this one map and a slot you forgot is an absence rather than a value. The reason is required, is trimmed, and travels with the instance, so peppy stack launch reports it back when the node starts.

Vacancy is a two-sided contract, and the node owns one side of it. A slot is vacatable only where the node’s own manifest declares it emptiable, with cardinality: "zero_or_one" on the depends_on.pairings, depends_on.pairing_observers, depends_on.nodes or depends_on.contracts entry. On a slot the manifest declares required, a vacancy is rejected and the error names the manifest key that would lift the bar. Every other slot is rejected with the spelling it does have: a zero_or_more slot, producer or observer, writes its empty set as [] (or omits the key), and a one_or_more slot of either kind has no empty state at all. The manifest says “may be empty”, the launcher says “is empty here, because”, and forgetting the slot entirely stays an error whichever the manifest says.

One instance filling all three kinds, plus a slot left deliberately empty:

{
  source: { name: "session_recorder:v1" },
  instances: [
    {
      instance_id: "rec_1",
      links: {
        // Producer slots: `storage` is `one`, `cameras` is multi-cardinality.
        storage: "disk_1",
        cameras: ["cam_left", "cam_right"],
        // A participant slot names the peer to pair with, with
        // `/<peer_link_id>` when the peer declares more than one slot.
        arm: "arm_1/limb",
        // A scalar observer slot names a single source instance to observe.
        // The multi cardinalities take these same values in an array.
        watch: "ctrl_1",
        // A slot this deployment leaves empty on purpose, and why. The
        // manifest declares `gripper` `cardinality: "zero_or_one"`, which is
        // what makes this legal.
        gripper: { vacant: "bench rig: no gripper on this frame" },
      },
    },
  ],
}

Every target must name an instance_id declared somewhere in the same launcher.

Two instances of the same node can choose different fates for the same slot, which is why the decision is written per instance rather than in the node’s manifest (whose cardinality: "zero_or_one" on this slot only says that some deployment may make it):

instances: [
  {
    instance_id: "backbone_governed",
    links: { leader_left_arm: "commander_inst/left_arm" },
  },
  {
    instance_id: "backbone_monitor",
    links: { leader_left_arm: { vacant: "monitor rig: nothing commands this backbone" } },
  },
]

daemon_scopes (optional)

The scope of each daemon target of the instance, keyed by the target’s name: what a client of the target may do through the daemon interface it names. An instance has the daemon targets of its deployment: those of the exposures of an exposure deployment (see daemon targets of an exposure), or those that the manifest of a node lists in depends_on.daemon, each named by its link_id (see daemon targets of a node). Only an instance with a daemon target takes daemon_scopes. A daemon target takes a scope and no link.

mcp_framework_controls.json5

// Deploying an exposure with a daemon target: the robots run as copies, and
// the framework's endpoint lets a client add and remove them.
//
//   peppy stack launch mcp_framework_controls
{
  peppy_schema: "launcher/v1",
  components: [
    {
      name: "robot",
      cardinality: "zero_or_more",
      options: {
        // Its fragment declares the own axis `brain`, whose option
        // `ai_brain_vla` deploys the brain's instance.
        openarm_sim: "robots/fragments/openarm_sim.json5",
        so101_sim: "robots/fragments/so101_sim.json5",
      },
    },
  ],
  deployments: [
    {
      source: { exposures: ["framework_controls:v1"] },
      instances: [
        {
          instance_id: "framework_controls_inst",
          arguments: { port: 8903 },
          // The daemon targets take no link. Each takes a scope, keyed by
          // the target's name. The instance has no `core_node`: an instance
          // that serves a daemon target runs on the coordinator.
          daemon_scopes: {
            // The members of fleet:v1 that the endpoint calls: a client
            // lists the fleet and halts the stack. Without `clear_halt`, the
            // endpoint serves no fleet.clear_halt, and the halt ends with
            // `peppy fleet clear-halt`.
            fleet: { members: ["list", "halt"] },
            stack: {
              max_copies: 4,
              options: [
                {
                  option: "openarm_sim",
                  // The robot model a robot of the option is, which a
                  // client reads before it adds one.
                  model: "openarm_v2",
                  description: "A simulated OpenArm v2 standing on the floor: two arms with grippers",
                  // A feature selects one option of one own axis of the
                  // option, here `brain=ai_brain_vla`. A client adds a robot
                  // with it, and names the machine of its part.
                  features: [
                    {
                      feature: "ai_brain",
                      axis: "brain",
                      option: "ai_brain_vla",
                      description: "An AI brain that finds items by words, then grabs and places them",
                    },
                  ],
                },
                {
                  option: "so101_sim",
                  model: "so101",
                  description: "A simulated SO-101 clamped on the edge of a table: one arm with a gripper",
                },
              ],
            },
          },
        },
      ],
    },
  ],
}

{ vacant: "<why>" } leaves a daemon target unserved: the server of an exposure serves none of its entries, and the daemon refuses each call of a node through it. The reason is required. For example, a launcher whose robots no client may add or remove leaves the stack target of its framework endpoint vacant.

When the launcher is read, a scope is a free value, because only the resolution of the deployments knows which targets are daemon targets. A launcher that writes its options in its own file and its endpoint in a fragment gives the scope with a top-level set_daemon_scopes adjustment instead. A scope belongs to the stack: a copy writes none.

The launch, each join and peppy stack resolve parse every scope into the type of its interface before anything starts. These checks read the launcher, the exposure documents and the node manifests only, so peppy stack resolve runs those of the exposure deployments also when the nodes cache is empty. It reads the manifest of a node from the nodes cache, and its report names each node deployment whose manifest this machine’s caches do not hold, whose checks did not run. A launch refuses a node whose daemon target names an interface, or a tag of it, that this peppy does not serve. The checks refuse:

• daemon_scopes on an instance that has no daemon target;
• a scope key that is not a daemon target of the instance; the refusal says so when the key is a target or a slot that takes links;
• a daemon target with no scope;
• a scope that does not parse into the type of its interface, for example a stack_copies scope with no option, with an option that names no model, with max_copies 0, with features: [], or with two features of one option on the same axis, or a fleet scope with no member or with a member named twice;
• an instance with a daemon target that does not run on the coordinator: one that declares core_node, or one of a copy;
• a links key that names a daemon target: the refusal says that the target takes a scope;
• an instance with two daemon targets of one interface: a call names no target, so a call of an interface goes with the one scope of that interface that the instance has.

The launch and peppy stack resolve also hold each scope against the launcher and the stack selection, and refuse:

• a stack_copies scope that names an option that is not an option of an axis that runs as copies, with the menu of the options a join can add; in a launcher that declares no such axis, a flat launcher included, every stack_copies scope;
• a feature of a stack_copies scope that selects an axis that is not an own axis of its option, an option that its axis does not have, or an option that brings no instance;
• an option of a stack_copies scope whose copy does not compose as a join over the bare stack with one of its feature sets (no feature, each feature, each combination), or whose copy with no feature already holds one of its features. The refusal names the option and the features of the set.

Each of these three checks runs only when the ones before it hold (see the scope checks). A join holds each scope to the first list only: the launcher and the stack selection of a launch do not change, so a join cannot change the verdict of the second. The refusals of every instance are reported at once.

An instance reads its scopes from its runtime configuration. A node with a daemon target that starts without these checks, for example with peppy node run, has no scope. The server reads its scopes from the runtime configuration of its instance when it starts. A server that starts without these checks, for example with peppy node run of its node, and finds a daemon target with no scope or a scope that does not parse, serves nothing and stops, and its run log says which target lacks a scope and how to give one (see Deploy it).

Clock domains

A launcher declares the timelines its instances read in a root-level framework block, beside deployments:

peppy_launcher.json5

{
  peppy_schema: "launcher/v1",
  framework: {
    clocks: {
      // One instance supplies this timeline, and naming it here is what
      // makes `simulation_inst` the domain's publisher.
      simulation: { publisher: "simulation_inst" },
      // A name for the time every machine already keeps.
      physical_robot: "wall",
    },
  },
  deployments: [
    {
      source: { name: "waldo", tag: "v1" },
      instances: [{ instance_id: "simulation_inst" }],
    },
    {
      source: { name: "openarm_sim_arm", tag: "v1" },
      instances: [{ instance_id: "arm_inst", framework: { clock: "simulation" } }],
    },
  ],
}

A declaration is { publisher: "<instance_id>" } for a timeline one instance supplies, or "wall" for a name the time every machine already keeps answers to. wall itself is built in and cannot be declared. Every other instance names the domain it reads with framework.clock.

The launcher’s declarations and those of every fragment the selection pulls in are merged: identical declarations agree, two different declarations of one name are refused naming both documents, and an unselected fragment contributes nothing. A copy binds the domains the stack declares, naming them as the stack does.

Every producer binding, pairing and observation joins two instances that read one clock, so a launcher wiring an instance on one domain to an instance on another is refused before anything starts. See System clock.

The launcher’s framework also takes restart, the policy of every instance whose entry and node declare none; a fragment writes a policy on its instance entries or through an adjustment’s set_framework, never at its root.

Restart policy

When a node process exits on its own, the daemon records the exit in the stack log and moves the instance by its restart policy: under never to finished (exit code 0) or failed (any other code, or a signal), where it stays; under on_failure to restarting after a failed exit (any code but 0, a signal, or the health kill below); under always to restarting after any exit, a clean one included. Under on_failure or always the daemon also kills an instance that stops answering health checks, and its policy treats that exit as a failure; the rule and its outage guard are in Instance health and lifecycle. The most specific level wins:

1. the instance: framework.restart on its launcher entry, set_framework.restart in an adjustment, or --restart on peppy node run or peppy node add --run (with --max-restarts for a cap);
2. the node: framework.restart in its peppy.json5;
3. the launcher root: framework.restart at the root of the launcher;
4. the default: on_failure.

A node declares its own policy in peppy.json5, under the same framework key the launcher uses:

peppy.json5

// A node that declares its own restart policy: an instance of it is
// started again after a failed exit, whatever launcher deploys it, unless a
// launcher entry, an adjustment or `--restart` writes one.
{
  peppy_schema: "node/v1",
  manifest: { name: "wrist_camera", tag: "v1" },
  interfaces: {},
  execution: {
    language: "rust",
    build_cmd: ["cargo", "build", "--release"],
    run_cmd: ["./target/release/wrist_camera"],
  },
  framework: { restart: "on_failure" },
}

A launcher declares a policy at its root and an instance entry its own. An adjustment under an option writes the same key on its target with set_framework: { restart: "never" }, as the adjustments table shows. In the launcher below, the root’s always covers the rear camera, the front camera’s entry writes its own, and the wrist camera takes the on_failure of the node above:

cameras.json5

// A launcher whose root policy is `always`: an instance is started again
// after any exit, which is what the rear camera gets. The front camera's
// entry writes its own, `on_failure` capped at five attempts in a row, and
// the wrist camera takes the `on_failure` its node declares.
{
  peppy_schema: "launcher/v1",
  framework: { restart: "always" },
  deployments: [
    {
      source: { name: "uvc_camera_linux", tag: "v1" },
      instances: [
        {
          instance_id: "front_cam_inst",
          framework: { restart: { policy: "on_failure", max_restarts: 5 } },
          arguments: {
            device_path: "/dev/video0",
            video: {
              frame_rate: 30,
              resolution: { width: 1280, height: 720 },
              camera_encoding: "mjpeg",
              topic_encoding: "rgb8",
            },
          },
        },
        {
          instance_id: "rear_cam_inst",
          arguments: {
            device_path: "/dev/video2",
            video: {
              frame_rate: 30,
              resolution: { width: 1280, height: 720 },
              camera_encoding: "mjpeg",
              topic_encoding: "rgb8",
            },
          },
        },
      ],
    },
    {
      source: { name: "wrist_camera", tag: "v1" },
      instances: [{ instance_id: "wrist_cam_inst" }],
    },
  ],
}

The value is a string, restart: "on_failure", or an object, restart: { policy: "on_failure", max_restarts: 5 }, where max_restarts caps the restart attempts in a row, failed starts included, and the next exit or failed start past them leaves the instance failed; a level writes the value whole, so a cap written at one level never combines with a policy from another. max_restarts needs a policy that restarts, on_failure or always; with never it is refused.

The policy applies once an instance has been running: a first start that fails still fails the launch or the peppy node run. A restart is a new process under the same id, with the arguments, environment, clock, core node and slots the instance had. It pairs and observes again, and the consumers bound to it receive from it again with no further action. When the previous process crashed, its shutdown hooks did not run, so write the node’s setup to bring its hardware to a known state before it enables anything.

The daemon waits between the exit and the next start. The first wait is one second; each further exit or failed start doubles it, up to one minute, and from then on every attempt waits a minute. The new process clears the same startup health gate as a first start. An exit after a run of two minutes or more starts the backoff over: its wait is one second and its attempt is 1 again. A run the health monitor ended counts only up to the last health check the process answered, so a node that hangs again after each start keeps backing off and reaches its max_restarts. The exit or failed start past max_restarts attempts in a row leaves the instance failed, and the stack log says `max_restarts` reached.

An instance that publishes a clock domain takes its policy like any other and keeps its domain when it restarts; System clock says what the instances reading it see. A fragment or option body that writes framework.restart at its root is refused when a selection composes it, naming the launcher root.

While the daemon waits, the instance shows restarting in peppy stack list, whose RESTARTS column counts the times a restarted process reached running; peppy node info adds the attempt, the last exit and the next start. The stack log records a restarting line with the attempt and the wait, a restarted line once the new process is running, a start failed line when the new process did not reach running (the run log holds the whole failure), a restart refused line when the daemon refused to start the process again (for example a clock domain whose publisher, on the same machine, has ended), and the `max_restarts` reached line. Once this machine’s nodes cache is filled (peppy repo refresh), peppy stack resolve prints each instance’s policy with its origin on its clock line. Each process writes its own run log, <instance_id>_<timestamp>.log, as add and build logs are named: a crash’s output stays in its file, a restart’s file opens with a line naming the attempt and the previous file, and peppy node info lists the latest one.

For a single instance, --restart and --max-restarts on peppy node run set the policy:

peppy node run wrist_camera:v1 -i wrist_cam_inst --restart on_failure --max-restarts 5

A crash of that instance and its restart read in the stack log as:

[2026-10-07T14:02:11.204] Instance 'wrist_cam_inst' of node 'wrist_camera:v1' failed: signal 11 (SIGSEGV), core dumped
[2026-10-07T14:02:11.209] Instance 'wrist_cam_inst' of node 'wrist_camera:v1' restarting in 1 s (attempt 1)
[2026-10-07T14:02:13.487] Instance 'wrist_cam_inst' of node 'wrist_camera:v1' restarted (attempt 1)

peppy node stop, peppy node remove, peppy node add over its node, peppy stack reset, peppy stack remove, peppy stack launch of the stack again and a daemon shutdown end an instance for good and cancel a waiting restart; peppy node stop also clears a finished, failed or restarting instance, which frees its id, and ends the start of an instance its restart is starting again. While a restart holds its daemon, from its start through the pairs its new process asks for again, a peppy stack launch, build, join or removal on that daemon answers busy, naming the instance and the stop that ends its restart. A start waits ten minutes for a node that never signals ready before it fails, and each attempt holds the daemon for its own start, so an instance whose every start hangs keeps its daemon busy, between its waits, until that stop or its max_restarts ends the restarts. peppy node remove holds restarts off while it runs: an instance that exits on the removal’s shutdown request ends there whatever its policy, and one whose restart falls due waits, so a removal that succeeds removes both. An instance starting when the removal reaches it, or running a process that reached running after the removal began, keeps its node; the removal’s error names it, and peppy node remove <name>:<tag> --stop-instances run again once none is starting removes it. peppy stack join refuses a copy whose adjustment changes the policy of an instance the stack runs, and a join that grows a restarting instance’s slot.

Starting a failed instance again

An instance whose restarts ended, with a restart refused or `max_restarts` reached line in the stack log, stays failed, as does one that failed under never, and one that finished stays finished. Either starts again by how it was started: an instance peppy stack resolve of the launcher lists, with the launch’s --with and --join words, was started by the launch; a member on the Instances: line under a Copy line of peppy stack list that the resolve does not list belongs to a copy peppy stack join added; any other instance was started by peppy node run or peppy node add --run.

• peppy node run or peppy node add --run: peppy node stop <instance-id> clears it and frees its id; then start it with peppy node run <name>:<tag> and the same arguments, links and flags (a peppy node add of its node again stops and clears every other instance of the node, and one from git or a URL does so without asking). The stop makes the instances paired with it forget those pairs. A pair the instance made itself, by its own --link, is already among those flags. A peer that made a pair with it by --link, or asked for one again at its own restart, names the pair in its run log under ~/.peppy/logs/run/ as paired: CORE/PEER_INSTANCE:PEER_LINK ⇌ CORE/INSTANCE:LINK_ID, and grep -l '⇌ [^ ]*/<instance-id>:' ~/.peppy/logs/run/*.log lists those logs; for each pair they name that the new run’s --link flags do not already cover, add --link LINK_ID@PEER_INSTANCE[/PEER_LINK] (see Pairing), in place of a --vacant-link on the same slot.
• A launch: peppy stack launch on the coordinator, with the launcher, placement, --with words and --join copies it was launched with. A launch replaces the whole stack, which ends the copies peppy stack join added and the instances peppy node run started: note the Copy lines of peppy stack list first, and join again each copy the launch does not bring back.
• A member of a copy peppy stack join added: read the copy’s Copy line in peppy stack list first, since the removal drops it. Then, on the coordinator or with --core-node <coordinator>, run peppy stack remove NAME, which clears the copy’s instances, and peppy stack join OPTION:NAME@CORE_NODE with the line’s option and machine, its axis=option words as --with, and the --set-arguments it was joined with. Where the removal is refused, as for the last copy of a one_or_more axis, launch the stack again.

For arm_1 of the Pairing example, which ctrl_1 paired with --link arm@arm_1, ctrl_1’s run log holds paired: cn-adoring-zebra/ctrl_1:arm ⇌ cn-adoring-zebra/arm_1:controller, and these commands start arm_1 again with that pair:

peppy node stop arm_1
peppy node run --instance-id arm_1 robot_arm:v1 --link controller@ctrl_1

For the copy bravo, whose backbone reached its max_restarts, peppy stack list on the coordinator shows:

Copy bravo of openarm on jetson-2: hardware_version=v1,robot_commander=ker_commander
  Instances: bravo_backbone_inst, bravo_camera_inst

and these commands start it again as it was joined:

peppy stack remove bravo
peppy stack join openarm:bravo@jetson-2 --with hardware_version=v1,robot_commander=ker_commander

A restart refused line names its cause. When the cause is a clock domain whose publisher ended, the line also says what publishes the name now. For a reader a launch or peppy stack join started, a launch of the stack starts the reader and its publisher again; join again, after the launch, each copy peppy stack join added. For one peppy node run started, stop it and run it again with its --clock, after starting a publisher with --publish-clock when nothing publishes the name.

Reusing built nodes across launches

Every node build a launch performs goes through the build artifact cache: when a deployment’s staged sources match an earlier build byte for byte, and that build ran with the same peppy version, OS, architecture and artifact kind, the launch reuses its artifact. Launching the same launcher again, after a daemon restart or on another machine booted from the same disk, therefore spends seconds on the build phase instead of rebuilding every node. Each reuse is reported in the launch output and in that node’s build log:

Reusing cached build of waldo:v1 (fingerprint 3f9a1c2e5d7b8a4f) at ~/.peppy/built_nodes/waldo_v1/3f9a1c2e5d7b8a4f.sif

peppy stack launch --rebuild and peppy stack build --rebuild build every node of the launch from its staged sources even when a matching artifact exists, on the coordinator and on every peer; use it when something the fingerprint does not cover changed, such as the image behind a floating base image tag.

Building without starting

peppy stack build runs a launch up to its builds. It replaces the node stack as a launch does, then adds and builds each node the launch would run on the machine the launch places it on, and each joinable node on this machine, and starts no instance. It takes every argument peppy stack launch takes, so a later peppy stack launch with the same launcher, placement, --with words and --join copies starts from these builds, and so do the joins of its scoped options.

Build a launcher on a disk, save the disk as an image, and every machine booted from that image launches without building:

peppy stack build openarm_simulation --with mujoco
# Save the disk as an image and boot each machine from it, then on each one:
peppy stack launch openarm_simulation --with mujoco

On Linux the daemon derives its default core node name from the machine id the disk stores, read from /var/lib/dbus/machine-id, else /etc/machine-id. Machines booted from one image therefore derive the same name, and a daemon whose name is already live on its router or federation refuses to start. Give each machine its own core_node_name; a name set on the imaged disk is shared the same way.

After a build, peppy stack list shows each node at stage Ready with 0 instances, under a Slice of launch <id> line naming the build that replaced this machine’s stack. A build records no launcher, so peppy stack join is refused until peppy stack launch starts the stack. A machine booted from the image starts with an empty stack: its builds wait under ~/.peppy/built_nodes/, and its launch names each one it reuses with a Reusing cached build of line.

Building the nodes a join brings

A launcher whose MCP endpoint serves a stack_copies daemon target lets a client add robots after the launch. The launch builds their nodes first, so that a join on the coordinator adds and starts nodes and builds none.

The launch checks of a stack_copies scope compose one trial copy of each feature set of each scoped option: no feature, each feature, and each combination (see Scopes). The nodes of these copies are the joinable nodes of the launch: every node that a join of a scoped option can bring. With two scopes, the joinable nodes are those of both. The built-in MCP server of a copy builds nothing, so it is no joinable node.

The launch resolves the joinable nodes with its own deployments and pins their contract and pairing documents with its own, all before it changes a machine. In its add-and-build phase, it adds and builds each joinable node on the coordinator, after its own nodes there and in the same build slots, so with a long build of the launch, the builds of the joinable nodes run beside it. A node that the launch adds on the coordinator anyway is added and built once. A node that the launch places only on another machine is added and built on the coordinator too, because a join can run it there. --place and --local move no joinable node: a joinable node is for the coordinator, where the endpoint that adds the robots runs. The launch starts no instance of a joinable node, and it starts its own instances only once every build has ended. peppy stack list then shows each joinable node at stage Ready with 0 instances. The add line of a joinable node names the scoped options that bring it:

Adding so101_backbone:v1 on `cn-desk` for stack joins of `so101_sim`

A later join of a scoped option on the coordinator, through the MCP join tool or with peppy stack join and the words of its features, finds each node it brings built: it adds the node again, and the build reuses the artifact in a few seconds. A join runs the resolution that the launch recorded for each joinable node, so it stages the trees that the launch built, also after a peppy repo refresh; launch again to run newer node content. A part that a join places on another machine builds on that machine when the join runs, and so does a node that a --with word selects and that no feature of the scope selects. --set-arguments selects no node, so it never causes a build.

A joinable node that does not resolve refuses the launch before any machine changes, so the stack that runs keeps running. A joinable node that does not build fails the launch, which clears the stack, as the failure of any node of the launch does. Each refusal names the node, the scoped options that bring it, and --no-prebuild. A trial copy that does not compose refuses the launch, as the launch checks of the scope do.

--no-prebuild on peppy stack launch and peppy stack build makes the launch resolve and build only the nodes it runs. A join then resolves the nodes of its robot against the repositories as they are when it runs, and builds them on each of their machines, so you change a robot’s node, run peppy repo refresh and join it again with no new launch. --rebuild builds the joinable nodes again too. A launcher with no stack_copies scope has no joinable node, and neither has a selection that deploys no instance with such a scope.

Launch timeouts

A launch adds, builds and starts its nodes one phase at a time, and each phase runs under an idle budget: the clock resets whenever the phase produces output (a git or http download’s progress, a build_cmd’s lines, a starting node’s lines, the ready signal of a starting instance), and the phase fails when it goes quiet for longer than its budget. The budgets are per phase and per node:

Flag	Phase	Default
--node-add-idle-timeout-secs	fetching a node’s source and generating its bindings	600
--node-build-idle-timeout-secs	running its build_cmd, or downloading and assembling its image; bytes landing on disk and CPU time the build burns count as output too	180
--node-run-idle-timeout-secs	starting an instance, until its setup ends and the daemon records it as running	600

These defaults are the only ones: the daemon has no budgets of its own, and every launch, build and join carries the budgets of its caller. When no output arrives, the command waits for the largest of the three budgets plus 60 s before it stops, so the daemon’s error for the silent phase comes first. A command that stops waiting sends no cancel, but it exits with code 1 and its process ends: the daemon then sees that the sender of the goal is gone and cancels the change, as for a command whose process goes away.

A join over MCP, through the join tool of a stack_copies daemon target, sends these same defaults, with no overall budget. Give such a tool a progress window of the largest of the three budgets plus the same 60 s, 660 s for the defaults, so that the daemon’s error for a silent phase comes first.

Every build runs with CARGO_TERM_PROGRESS_WHEN=always and CARGO_TERM_PROGRESS_WIDTH=80, in a build_cmd and in a container def’s %post alike. Without them, cargo prints a line only when a crate download completes when its output is not a terminal, and until then the download writes nothing to disk and burns no CPU, so a large crate on a slow connection reads as an idle build. With these settings cargo reprints its progress line each time the bytes left to download change, and each reprint counts as output. A download that receives nothing changes nothing, so it still runs out the budget. If you set either variable yourself, a build_cmd gets your value; a def sets its own by exporting it in %post.

--max-timeout-secs adds a budget for the whole launch on top of the idle ones; without it, only the idle budgets apply.

When a budget runs out the launch fails, names the phase and, for an idle budget, the flag that raises it, and clears the stack as any failed launch does. What the interrupted phase leaves behind depends on the phase:

• An interrupted add never publishes its node. The daemon generates a node’s bindings into the node’s staged copy on a worker thread of its own, so a generation that is already running when the add’s budget runs out finishes on that thread, and the staged copy is removed once it does; a queued generation whose add is gone is skipped. Either way the node never enters the stack and nothing of it is built.
• An interrupted build has its build_cmd or image build killed.
• An interrupted run has its half-started instance torn down before the failure is reported. An instance that has not answered ready has its process group killed at once; an instance that answered ready gets the shutdown request and the grace first, so its shutdown hooks run (see How an instance starts).

peppy stack build takes the same flags and ends after the build phase, so --node-run-idle-timeout-secs bounds nothing in a build. peppy stack join takes the same flags and runs its copy’s nodes through the same phases, --max-timeout-secs bounding the whole join. A join whose budget runs out fails as any failed join does: it stops what it started and removes the nodes it added, leaving the stack it joined running.

Only the node’s own output and, in the run phase, the ready signal of the instance reset a phase’s clock; the launcher’s progress lines do not. Inside the run phase, the daemon also waits for each setup for the setup budget of its node, at most 540 seconds, which is less than the default run idle budget.

How a stack command ends

peppy stack launch, build, join and remove each send the daemon one goal and follow it to its end. The daemon refuses a goal that it cannot start, for example while another stack change runs, and the command prints the refusal with its failure kind:

Error: Launch refused (busy): a stack launch, build, join, removal or reset, or a coordinator's change to this daemon's slice, is in progress on this daemon; run this again once it ends

A goal that fails ends the command with its failure kind, the reason, and the launch log, after the log files of the nodes:

Error: Launch failed (attempt_failed): failed to build node arm:v1: build_cmd `cargo build --release` failed with status exit status: 101. Log file: /home/ubuntu/.peppy/logs/launch/launch_20261023_101500_123.log

Kind	What it says
rejected	The launcher, a word of the command or a value in it is not valid, or it names something that is not there
busy	Other work holds the stack of a machine: another stack change runs, a node command or a restart runs, a peppy stack reset runs, or another launch holds the machine. The same command can succeed once that work ends
precondition_failed	The stack or a machine does not allow the change: an instance or a copy is in the way, or the stack has no launcher
resource_unavailable	A machine of the change is not live, or did not answer
timeout	A budget ran out
attempt_failed	A step failed, for example a build or the start of an instance

busy is the kind of the daemon itself. The other kinds are kinds of peppy (see How a goal ends). A federated launch fails with busy when each machine that refused it is busy.

The command exits with code 3 when the goal is refused, and 4 when it fails. Each end of a goal has an exit status of its own (see Exit statuses of the other commands).

The command keeps up to 65536 lines of the change’s output that it did not show yet. A command that falls further behind, for example on a terminal that shows lines slower than a build prints them, cancels the change, says The command fell behind the output of the launch by more than 65536 unread messages and cancelled it., and exits with code 1.

Stopping a stack command with Ctrl+C

The first Ctrl+C sends the daemon a cancel of the goal, and the command says so:

Cancelling the launch; press Ctrl-C again to stop waiting for its end

The daemon stops the change at the step it reached, and ends it as a failure at that step ends it: a launch or a build clears the stack, a join is undone, and a removal keeps its copy, with the instances that it did not stop yet. A join or a removal reads the cancel before it changes a machine, so a cancel that comes early leaves every stack as it was. A step that runs on another machine gets the cancel too. The command follows the change to that end, prints how it ended, and exits with the status of that end, 5 for a cancel. The rollback can be silent for longer than the idle budgets, so after a Ctrl+C only --max-timeout-secs bounds the wait:

Error: Launch was cancelled by its caller. Log file: /home/ubuntu/.peppy/logs/launch/launch_20261023_101500_123.log

A change that ended before the cancel reached it keeps its end, and the command says so: Launch completed before the cancel took effect., with the exit status 0. A change that failed for a cause of its own, for example a build that failed before the cancel reached it, ends failed with its kind and the exit status 4, also when the cancel arrives during its rollback. A second Ctrl+C stops the wait for the end: the command waits only until the cancel reaches the daemon, which goes on with it, and exits with code 130. A third Ctrl+C ends the command at once.

A command whose process goes away, for example when its terminal closes or the process is killed, cancels its goal too: the daemon sees that the sender of the goal is gone, and cancels the goal one minute later. The launch log then ends with cancelled: its sender is gone. The minute outlasts a restart of the messaging router by its watchdog, during which the daemon cannot see the commands that still run, so a change goes on through a router restart or a short loss of the link between two machines. Each peppy process is a sender of its own, so the goals of other peppy processes go on.

A peppy stack reset cancels the change that runs: the change ends, Launch was cancelled by a stack reset., and the command exits with code 5, as for any cancel.

Running one launcher across several machines

A launcher can span machines. It declares named placeholders under core_nodes, each instance says which one it runs on via core_node, and peppy stack launch --place <core-node-link>@<core-node> binds those placeholders to real hardware at launch time. Producer links, pairings, and observations all work across the boundary, and consumers still start after their producers. See Federation.

Every placeholder the launcher declares must be bound with --place; a launch that leaves one unwired is refused. The escape hatch is peppy stack launch --local, which ignores the launcher’s placements and runs every instance on the daemon you launched from. --local and --place are mutually exclusive, and under --local no --join and no --with word names a machine. A copy runs where it is named, and each part of it where the word that selects it says.

Swapping parts at launch time

A launcher file can describe a family of stacks whose members differ in which implementation fills a slot: a real robot or a simulator, a browser panel or a headset, with or without a recorder. The launcher declares component axes, deployments says which option of each it starts, and the launch command swaps one with --with. A launcher without components is a plain flat stack, and that stays the ordinary way to write a one-off.

The whole feature fits in one file. Here is a viewer stack with two swappable cameras:

viewer.json5

// One file, two swappable cameras: the viewer stack declares a `camera`
// axis whose options are written inline, so composition needs no second
// file. `peppy stack launch ./viewer.json5` runs the camera the file
// deploys (`uvc`); the same launcher with `--with=zed` swaps it.
{
  peppy_schema: "launcher/v1",

  components: [
    {
      name: "camera",
      provides: ["cam_inst"],
      options: {
        uvc: {
          deployments: [
            {
              source: { name: "uvc_camera_linux", tag: "v1" },
              instances: [
                {
                  instance_id: "cam_inst",
                  arguments: {
                    device_path: "/dev/video0",
                    video: {
                      frame_rate: 30,
                      resolution: { width: 1280, height: 720 },
                      camera_encoding: "mjpeg",
                      topic_encoding: "rgb8",
                    },
                  },
                },
              ],
            },
          ],
        },
        zed: {
          deployments: [
            {
              source: { name: "zed_camera", tag: "v1" },
              instances: [
                {
                  instance_id: "cam_inst",
                  arguments: {
                    resolution: "hd720",
                    frame_rate: 30,
                    depth: { min_depth_m: 0.4, downscale: 2, block_size: 3 },
                  },
                },
              ],
            },
          ],
        },
      },
    },
  ],

  deployments: [
    {
      source: { name: "web_viewer", tag: "v1" },
      instances: [{ instance_id: "viewer_inst", links: { camera: "cam_inst" } }],
    },
    { camera: "uvc" },
  ],
}

peppy stack launch ./viewer.json5              # runs the camera the file deploys (uvc)
peppy stack launch ./viewer.json5 --with=zed   # runs the ZED camera instead

Components declare what may run; deployments says what runs. An axis entry has four fields:

Field	Meaning
name	The axis’s name, unique in the launcher. Used in --with axis=option, in deployments, and in adjustment guards.
options	The alternatives that fill this axis, at least one. Each option is a fragment: a body using the same deployments grammar the launcher’s own deployments use, written inline, or a path string naming its own file, or the object { fragments: ["fragments/relays.json5", "fragments/mujoco.json5"], ... }, which is how an option names more than one file and how, in a launcher, it carries the adjustments that run when the option is selected.
cardinality	How many selections of the axis a stack allows. one (the default): exactly one option runs. zero_or_one: at most one. zero_or_more: any number, each a named copy. one_or_more: the same copies, with at least one started by every launch and stack remove refusing the last one.
provides	Instance IDs every option defines, so a link written against one is valid whichever option is selected. On an axis that runs as copies the IDs are checked as written, before a copy’s name prefixes them.

A deployments entry deploys a node, { source, instances }, or a component’s option, { <component>: "<option>" }:

• On a one axis the entry names the option that runs, ids as written, and --with swaps it for another option of that axis. A one axis the file does not deploy must be named with --with.
• A zero_or_one axis is never deployed by the file, since nothing on the command line could switch it off. --with turns it on. A feature that is on by default and switchable is a one axis with a none option (its body can be empty), deployed by the file and swapped off with --with none.
• On an axis that runs as copies the entry lists the ones that run, each under instances with a name, or lists none and says how every copy of the option is set up; see copies. --with cannot name one, since a copy needs a name; --join OPTION:NAME starts one with the launch.

Each --with entry is either option (looked up across every axis in reach) or axis=option. A bare word matching options on more than one axis is refused, naming the axes, so axis=option says which you mean. An axis takes one option; repeating it, or naming it once bare and once as axis=option, selects it once. On stack launch and stack build, NAME.option or NAME.axis=option selects the own axis of a copy the file deploys, over the file’s with.

A launch of a launcher with components echoes the full resolution before anything runs, marking the axes a deployments entry filled:

components: simulation=waldo (from file)  robot_control=(off)  scene_commander=(off)
copy alpha: robot=openarm_sim  hardware_version=v2 (from file)  robot_commander=web_commander (from file)  recorder=(off)  camera_rig=none (from file)  brain=(off)

Passing --with to a launcher that declares no components is refused.

Adjustments

A family of stacks often needs more than swapped deployments: selecting the recorder binds it into whichever commander runs, and selecting the KER leader releases the backbone’s joint sockets. An adjustment names one instance and one change to it, and every change names its operation. A fragment writes only the instances it deploys, so the connections between fragments are the launcher’s, written under the option that names the fragment:

components: [
  { name: "simulation", cardinality: "zero_or_one", provides: ["simulation_inst"],
    options: { mujoco: "simulation/fragments/mujoco.json5", isaac_sim: "simulation/fragments/isaac_sim.json5",
               waldo: "simulation/fragments/waldo.json5" } },
  { name: "recorder", cardinality: "zero_or_one", provides: ["recorder_inst"],
    options: {
      lerobot_recorder: {
        // An inline body, beside the launcher's writes for it.
        deployments: [{ source: { name: "lerobot_recorder", tag: "v1" },
                        instances: [{ instance_id: "recorder_inst" }] }],
        adjustments: [
          // Runs when this option is selected; the guard adds a condition
          // of its own, see below.
          { target: "recorder_inst", when: { simulation: "isaac_sim" },
            set_arguments: { robot_type: "openarm_v2_isaac" } },
        ],
      },
    } },
  { name: "robot_commander", provides: ["commander_inst"],
    options: {
      ker_commander: {
        fragments: ["openarm/fragments/ker_commander.json5"],
        adjustments: [
          {
            target: "backbone_inst",              // an instance the launcher's deployments define
            set_arguments: { upstream_mode: "joints" },
            set_links: { collision_ctrl: { vacant: "the KER never produces governor_control" } },
            unset_links: ["leader_left_arm"],
          },
        ],
      },
    } },
],

The launcher’s own top-level adjustments list takes the same entries, guarded on the launcher’s axes, which is how one launcher specializes what every option wrote.

Operation	Effect
set_arguments	Replaces the target’s value for each named argument key, creating it when absent. Nested objects replace wholesale.
set_framework	Replaces the target’s value for each named framework knob, clock or restart; a knob the adjustment omits keeps its value. How the launcher, under the option that selects a simulated robot, binds the commander and recorder that robot shares with its physical twin to the clock domain this selection runs on.
set_links	Replaces the target’s whole entry for each named slot, creating it when absent. The value uses the ordinary links grammar, including vacancies.
add_links	Appends targets to each named slot’s array binding, creating the binding when the slot has no entry. Refused when the slot holds a scalar or a vacancy: replacing is set_links’s job. Appending is what lets independent options each contribute bindings to one multi-cardinality slot, and what lets a copy that joins add its own instances to a running instance’s set.
unset_links	Drops each named slot’s entry entirely, returning the slot to absent.
set_daemon_scopes	Replaces the scope of each named daemon target, creating it when absent. No operation removes a scope. Refused in the composition of a copy, because a scope belongs to the stack: write it at the top level, under an option of an axis that does not run as copies, or in the instance’s own daemon_scopes.

A fragment’s own entries run first, in the order its files are read, then the launcher’s: the options’ entries in the order components declares their axes, then the top-level list. The order is what decides a field two entries both write, the later one winning, so a launcher’s entry overrides the fragment’s and its top-level list overrides its options’.

Three kinds of entry run in copies, their target resolved inside each copy:

• an entry under an option of an axis that runs as copies runs in every copy of that option;
• an entry under an option of an axis that fills once, a one or a zero_or_one, or at the top level, guarded on an axis that runs as copies, runs in every copy of the options its guard names;
• an entry whose target only the options of axes that run as copies define runs in every copy of every such axis.

One routing is ambiguous, and it is refused when the launcher is read: an entry that names no axis that runs as copies, neither through the option it sits under nor in its guard, whose target an option of such an axis defines as well as an option that fills once on an axis other than the entry’s own, when the entry’s own option does not define the target: an entry writing an instance its own option defines is that option’s write. The refusal names both, because either reading is a legal launcher: write the entry under the option that fills once to write that option’s instance, or guard it on the axis that runs as copies to write each copy’s.

An adjustment whose target the current selection does not define is skipped, not refused: a top-level write to recorder_inst does not run when the recorder axis is off. A target nothing in the entry’s reach defines is a dead reference and fails when the launcher is read. From the top level, or under an option that fills once, an entry reaches the launcher’s deployments, its own option and every option of the other axes, those that run as copies included; under an option that runs as copies it reaches that option’s instances and the stack’s, never another copy’s.

An option’s adjustments run when that option is selected, so the option is their guard. when is for conditions existence cannot express, mostly shape: two leaders may both provide commander_inst, but only the browser panel has motor health and alert slots to bind. A top-level guard names the launcher’s axes; an option’s names the launcher’s other axes and the option’s own. Each axis accepts an option name or a nonempty list of option names: lists match any listed option (OR), and every named axis must match (AND).

A copy’s selection holds the launcher’s axes and the copied option’s own. So an entry that reaches the copies from an option that fills once, or from the top level, is guarded on the launcher’s axes alone, and one guarded on that option’s own axis is refused when the launcher is read. An entry names one axis that runs as copies, through the option it sits under or its guard, and targets an instance those copies or the stack define; naming two, or targeting anything else, is refused when the launcher is read.

For example, a guard on the two simulations that model no camera response:

when: { simulation: ["mujoco", "isaac_sim"] }

This guard matches either, and Waldo does not match. The same scalar-or-list syntax applies inside constraint when, requires, and forbids conditions.

The ownership rule of thumb: if a value is something the operator genuinely decides, make it another option on the axis, visible in the command line. If it is a consequence of a choice already made, write the adjustment under that option, where it needs no guard: it runs exactly when the option is selected and skips when its target is absent.

Constraints

Adjustments change what a selection builds; a constraint refuses selections nobody should run. Some members of a family are stacks no operator should be able to name: the ZED camera that films only the physical rig must not launch against the simulator. Those members flatten cleanly, so neither provides (which only promises ids exist) nor an adjustment’s when guard (which only decides whether one adjustment applies inside a legal selection) can say it. A constraint can, and it only ever refuses: a violating selection is rejected before anything is pinned or started. It never picks an option to satisfy itself.

constraints: [
  // With the ZED camera selected, the robot must be the real one.
  { when: { camera: "zed" }, requires: [{ robot: "real" }],
    reason: "the ZED films the physical rig" },

  // The whole combination is refused, no guard needed.
  { forbids: [{ robot: "mujoco", camera: "zed" }],
    reason: "the stereo camera pairs with the real rig only" },
]

Field	Meaning
when	The guard: the selections this constraint speaks about, in the same axis: options grammar as an adjustment’s when: every named axis holds one of its listed options (an AND across axes, an OR within one). Omit it to speak about every selection.
requires	What the guarded selections must also contain: a list of alternatives, at least one of which must match wholly. Each alternative is an axis: options map where every axis must match (an AND); listing several alternatives is an OR.
forbids	What the guarded selections must not contain: a list of axis: options maps, and matching any one of them (each an AND of its axes) refuses the selection outright.
reason	Why the refused selections must not run. Required, and quoted verbatim in the refusal: peppy can render what was required or forbidden, but only the author knows what the dead combination would have done.

A constraint states at least one of requires and forbids; with neither it refuses nothing, and the document is refused when it is read. The rest of the shape is held to the same standard: when and every entry name at least one axis, an entry may not repeat an axis the guard already fixes (under the guard that axis is decided before anything is selected), the same map may not be listed twice in one list, and one map may not appear in both lists.

Matching treats an unfilled axis as matching no entry, on either side: a guard on camera: "zed" stays quiet when the camera axis is off, and a requires alternative naming that option is satisfied only by selecting it, never by leaving the axis unfilled. That asymmetry is the tool for optional features: requires is how an option demands company, while forbids is the only way to say an optional option never launches with another one (nothing requirable means “that axis stays off”). Where the axis is required, prefer requires of the wanted option instead: requires fails closed for options added later, while a forbids list names today’s options and stays silent about tomorrow’s.

The refusal happens at selection time and names the document that declared the constraint, the full resolution (with its (from file) markers showing the axes the operator never named), what was required or forbidden, and the author’s reason. When several constraints overlap, the first declared one refuses, so the author controls which reason the operator reads.

A launcher’s constraints speak in the launcher’s axis and option names; a fragment’s speak in its own axes and the launcher’s, which is where a rule about an option’s own axes belongs. peppy repo index --check holds them to leaving a usable family behind: a constraint that refuses no selection, an option no legal selection can pick, a bare launch that violates a constraint, and a file’s copy whose own constraints refuse it under every admitted selection (so what the file deploys can never start: loosen a constraint or change the copy’s with) are each reported when the launcher is indexed. A file copy is held only to the stacks its constraints admit: a selection those constraints refuse by design, as a join of it would be, is not flagged.

Fragments: splitting a launcher across files

An option’s body can live in its own file, called a fragment. This is an organizational choice, never a requirement: a launcher whose options are all inline is self-contained. Extract a fragment once a second launcher wants the same body, so the body is stated once.

A fragment file carries the launcher_fragment/v1 tag:

fragments/zed_camera.json5

// The `zed` option of a viewer launcher, extracted into its own file: the
// body is the same `deployments` grammar the launcher itself uses, and the
// fragment tag is what keeps a fragment from ever being mistaken for a
// launchable stack. Referenced from a launcher as
// `zed: "fragments/zed_camera.json5"`.
{
  peppy_schema: "launcher_fragment/v1",

  deployments: [
    {
      source: { name: "zed_camera", tag: "v1" },
      instances: [
        {
          instance_id: "cam_inst",
          arguments: {
            resolution: "hd720",
            frame_rate: 30,
            depth: { min_depth_m: 0.4, downscale: 2, block_size: 3 },
          },
        },
      ],
    },
  ],
}

The launcher references it by path, and the option value becomes that path string:

options: {
  uvc: { deployments: [ /* inline, as above */ ] },
  zed: "fragments/zed_camera.json5",
}

Fragment paths are relative to the declaring document’s directory: the launcher’s for its options, the fragment file’s for the options of its own axes. Inside a Peppy repository, the nearest ancestor containing a valid peppy_repository.json5 sets the boundary: so101/fragments/so101.json5 can include ../../common/fragments/none.json5. Parent segments and resolved symlinks must stay within that repository. Standalone launchers use their own directory as the boundary. Absolute paths are rejected. Repository names and filesystem paths use the same resolution rules.

The object form of an option names its files under fragments, read in order, and may carry a body of its own beside them; two options share one body by naming the same file:

// In the launcher, whose `adjustments` under an option are its own writes.
options: {
  mujoco: { fragments: ["fragments/relays.json5", "fragments/mujoco.json5"] },
  isaac_sim: { fragments: ["fragments/relays.json5", "fragments/isaac_sim.json5"],
               adjustments: [
                 { target: "viewer_inst", set_arguments: { renderer: "path_traced" } },
               ] },
}

An option declared inside a fragment, or inside an option body, carries no adjustments of its own: the writes a fragment makes are its top-level adjustments, which name the instances it deploys, those of an option whose body is written inside it included, and everything else is the launcher’s, under the option that selects the fragment. A fragment’s framework declares clocks only.

The parts of one option declare its axes between them, each axis once, and deploy each of those axes once. Besides deployments, components, constraints, adjustments and framework, a fragment may declare core_nodes: additional placement placeholders its instances need, unioned with the launcher’s own, so selecting the option makes the matching --place bindings required exactly as if the launcher declared them. An entry in a fragment’s adjustments naming an instance the fragment does not deploy is refused when the fragment is read, naming what it does deploy. The connections between fragments are the launcher’s, written under the option, as above.

A fragment is not a stack. The repository indexer never lists a fragment as a launchable launcher, it needs no entry in peppy_repository.json5, and it cannot collide with a launcher name. peppy repo index --check validates every fragment a launcher references and flattens every legal selection of its axes, so a broken combination is refused on the branch that introduced it.

An option’s own axes

An option’s fragment can declare components of its own: the axes that belong to that option. An OpenArm robot fragment declares its robot commander and its recorder, and deploys the browser panel unless told otherwise:

fragments/openarm_sim.json5

{
  peppy_schema: "launcher_fragment/v1",
  deployments: [
    { source: { name: "openarm_backbone", tag: "v1" }, instances: [{ instance_id: "backbone_inst" }] },
    { hardware_version: "v2" },
    { robot_commander: "web_commander" },
    { camera_rig: "none" },
  ],
  components: [
    // The hardware version the simulation stands: each option deploys the
    // initializer that names its model, inline.
    { name: "hardware_version", provides: ["init_inst"],
      options: {
        v1: { deployments: [{ source: { name: "robot_initializer", tag: "v1" },
                              instances: [{ instance_id: "init_inst", framework: { clock: "simulation" },
                                            arguments: { model: "openarm_v1" }, links: { simulation: "simulation_inst" } }] }] },
        v2: { deployments: [{ source: { name: "robot_initializer", tag: "v1" },
                              instances: [{ instance_id: "init_inst", framework: { clock: "simulation" },
                                            arguments: { model: "openarm_v2" }, links: { simulation: "simulation_inst" } }] }] },
      } },
    // The robot commander of the robot: a node that drives it outside the
    // leases and the halt.
    { name: "robot_commander", role: "robot_commander",
      options: { web_commander: "web_commander.json5", none: "../../common/fragments/none.json5",
                 ker_commander: "ker_commander.json5" } },
    { name: "recorder", cardinality: "zero_or_one", provides: ["recorder_inst"],
      options: { lerobot_recorder: "lerobot_recorder.json5" } },
    { name: "camera_rig",
      options: { none: "../../common/fragments/none.json5", cameras_sim: "cameras_sim.json5" } },
    { name: "brain", cardinality: "zero_or_one", provides: ["brain_inst"],
      options: { ai_brain_vla: "ai_brain_vla.json5" } },
  ],
}

role: "robot_commander" marks the robot commander axis of the option: the option of a copy on that axis is the robot commander of the robot, a node that drives its own robot outside the leases and the halt, such as a leader arm, the KER or the browser panel. The fleet listing gives it as the robot_commander of the robot, and the option none, or an option that marks no axis, gives none. An option marks one axis at most, and an axis that runs as copies takes no role.

A fragment’s axes are one or zero_or_one; how many robots run is the launcher’s robot axis. They are in reach whenever the fragment runs once: on a launcher whose robot axis is one, --with ker_commander reaches into the robot’s fragment, and --with robot_commander=ker_commander says so explicitly. A fragment’s adjustments and constraints speak in its own axis names and the launcher’s, so the “cameras need a consumer” rule and the recorder’s trigger rule live in the robot fragment beside the axes they speak about, and the camera rule and the recorder rule name the launcher’s headset axis among the consumers that satisfy them. A launcher’s own adjustments and constraints speak in the launcher’s axis names. A fragment two levels below the launcher declares no axes: an option of a fragment’s axis is a plain fragment.

Copies

An axis with cardinality zero_or_more or one_or_more runs as copies: named instances of one of its options, exactly as a node runs as named instances. The launcher’s deployments list the copies a launch starts, peppy stack join adds one to a running stack, and peppy stack remove takes one out. Every instance a copy’s fragments define is minted as NAME_<id>, so any number of copies run side by side under one launcher. The <id> a fragment writes is the instance’s name inside the copy, and a per-robot MCP surface publishes it: it is the value a client passes to pick which of a robot’s cameras or limbs a call addresses, so renaming an instance in a fragment renames it for every client of that surface.

The openarm_simulation launcher holds the simulation as a one axis, and the robot as copies:

components: [
  { name: "simulation", provides: ["simulation_inst"],
    options: { mujoco: "...", isaac_sim: "...", waldo: "..." } },
  { name: "robot", cardinality: "zero_or_more",
    options: { openarm_sim: "...", so101_sim: "..." } },
],
deployments: [
  { simulation: "waldo" },
  { robot: "openarm_sim", instances: [
      { instance_id: "alpha", arguments: { commander_inst: { http_port: 8765 } } },
  ] },
],

The same axis written cardinality: "one_or_more" is the same copies with a floor under them: a launch starts at least one, and stack remove refuses the last one, so every stack of the launcher runs a robot. The copy can come from deployments, as the entry above does, or from --join OPTION:NAME on the launch, so a launcher that deploys none makes the choice of robot part of the command line. A launch that would start none is refused, naming the axis and both ways to fill it.

{ name: "robot", cardinality: "one_or_more",
  options: { openarm_sim: "...", so101_sim: "..." } },

peppy stack launch openarm_simulation                               # the copy the file deploys
peppy stack launch openarm_simulation --join openarm_sim:alpha      # when the file deploys none

A copy entry has four fields: instance_id, the copy’s name; with, the selection of the option’s own axes, an axis left out taking what the fragment deploys; arguments, overrides of the copy’s instances’ arguments, keyed by the id written in the fragment and then by argument; and adjustments, the fragment verbs over the copy’s instances. The option entry itself takes with and arguments too, shared by every copy of the option, listed or joined, and a copy’s own values win per axis and per argument:

{ robot: "openarm_sim", with: { robot_commander: "web_commander" }, instances: [
    { instance_id: "alpha" },
    { instance_id: "bravo", with: { hardware_version: "v1" }, arguments: { commander_inst: { http_port: 8766 } } },
] },

An entry and a copy also take adjustments, the same verbs the launcher’s adjustments use (set_arguments, set_framework, set_links, add_links, unset_links) with an optional when over the launcher’s axes that fill once and the copy’s own, targeting the copy’s instances by the ids written in the fragment. They run after the launcher’s adjustments, the entry’s before the copy’s, and the copy’s arguments last. On stack launch and stack build, a --with word NAME.axis=option or NAME.option selects one copy’s own axis for that launch, over the file’s with, whether the file lists that copy or --join names it:

peppy stack launch openarm_simulation --with mujoco,alpha.ker_commander

An entry may list no copy. It then says how every copy of the option is set up, and a launch starts none of them; --join OPTION:NAME on stack launch, once per copy and comma-separated as --with is, starts a copy with the launch: a copy of the entry under that name, composed, validated and started with the launch, and failing with it. The words that address a listed copy address it too, --with NAME.option, and it runs where its --join OPTION:NAME@CORE_NODE names:

deployments: [
  { simulation: "waldo" },
  { robot: "so101_sim", with: { robot_commander: "none", camera_rig: "cameras_sim" } },
],

peppy stack launch simulation_mcp                                                    # the simulation and the copies the file lists
peppy stack launch simulation_mcp --join so101_sim:charlie                           # one more, as the entry sets it up
peppy stack launch simulation_mcp --join so101_sim:charlie,so101_sim:delta --with delta.so101_leader

A copy named on the command line is composed as a join. --join lays it over the configuration the file states and over the copies named before it, as stack join lays a copy over the running stack, so what a launch accepts a later stack join accepts too, and a robot you can name at launch is a robot you can also join later. A name the launch already starts, listed or joined, is refused.

One robot, two robots, many:

# The simulation and the copies the file deploys.
peppy stack launch openarm_simulation

# The same copies in MuJoCo, with the stack's headset.
peppy stack launch openarm_simulation --with mujoco,xr_commander

# Another robot in place of the first, on another machine, led by the headset, then its removal.
peppy stack remove alpha
peppy stack join openarm_sim:bravo@jetson-2 --with robot_commander=none
peppy stack list
peppy stack remove bravo

# A stack with no robot yet, then robots as they arrive.
peppy stack launch physical --with xr_commander
peppy stack join openarm:alpha@jetson-1 --with v2
peppy stack join openarm:bravo@jetson-2 --with v1,robot_commander=none

# The same two robots, a v2 and a v1, named at launch.
peppy stack launch physical --join openarm:alpha@jetson-1 --join openarm:bravo@jetson-2 --with xr_commander,bravo.hardware_version=v1,bravo.robot_commander=none

stack join OPTION:NAME names the option to copy and the copy’s name, the one spelling --join takes too. Its --with words select the copied option’s own axes. Every instance the copy’s fragments define receives the name as a prefix: alpha_left_arm_inst, alpha_backbone_inst, alpha_commander_inst. Every link naming one of them follows it under the prefix, including array links and pairing targets such as alpha_backbone_inst/leader_left_arm; links to stack instances such as simulation_inst keep their IDs. The name is also the copy’s core node link.

A copy runs where it is named: OPTION:NAME@CORE_NODE on stack join and on --join at launch, and --place NAME@CORE_NODE for a copy the launcher file lists. With no machine, the copy runs on the coordinator. The copy’s machine runs its base, the instances of the copied option’s own fragments. The instances that an own axis of the copy brings, those of the option the copy selects on that axis, are a part of the copy, and a part runs where the word that selects it says: a selection word takes a machine after @, option@CORE_NODE or axis=option@CORE_NODE, and at launch NAME.option@CORE_NODE or NAME.axis=option@CORE_NODE. self names the coordinator, after the @ of a copy as of a word. A word with no machine, and an own axis that no word names, run with the base. To place a part that the launcher entry or the fragment selects, write its option again with a machine: --with cameras_sim@cn-gpu-box. When every instance of a copy comes from parts that run elsewhere, the copy’s machine runs nothing, and stack list still shows it as the copy’s machine. Links, pairings and observations between the base and a part, and between two parts, cross machines as they do between any two instances of a federated launch.

# An OpenArm on jetson-1, its brain on a machine with a GPU.
peppy stack join openarm_sim:delta@jetson-1 --with ai_brain_vla@cn-gpu-box

# The same robot named at launch, and a listed copy's camera rig on the GPU machine.
peppy stack launch openarm_simulation --join openarm_sim:delta@jetson-1 --with delta.ai_brain_vla@cn-gpu-box,alpha.cameras_sim@cn-gpu-box

A launcher file names no machine for a copy or its parts: the with of an entry or a copy takes option names, which hold no @; a copy entry or an instance of a copy’s fragments that declares core_node, and a copy’s fragment that declares core_nodes, are refused.

These words are refused, each with a text that says what to write instead:

• a placed word whose option brings no instance, such as --with robot_commander=none@cn-gpu-box: drop the @;
• two words on one own axis whose machines differ as written: ai_brain_vla and ai_brain_vla@cn-gpu-box differ, and so do @self and @ with the coordinator’s name, because peppy stack resolve has no coordinator to compare with. The same word written twice is one word;
• a bare option that two own axes have, with a machine or without one: axis=option@CORE_NODE names the axis;
• at launch, a machine on a word of a stack axis, such as --with mujoco@cn-gpu-box: a stack instance runs where its core_node link says, --place LINK@CORE_NODE;
• at launch, --place NAME@CORE_NODE where NAME is a copy that --join adds: place a joined copy where you join it, --join OPTION:NAME@CORE_NODE;
• under --local, a --with word that names a machine, @self included, and a --join copy that names a machine other than self: --local runs the whole launch on this machine;
• on stack join, a word NAME.option, which selects an own axis of a copy on stack launch only.

A joined copy starts from the same settings as a copy the file lists. Weakest first: what the option’s fragment deploys, then the with, arguments and adjustments of the launcher’s entry for that option, then the join’s --with words and --set-arguments. A word wins on its own axis and the entry’s other axes stay selected, so the option’s constraints judge the two together, as they do for a launch word NAME.option. Under the web_commander entry above, a plain peppy stack join openarm_sim:charlie is another robot under its own browser panel, and peppy stack remove alpha followed by peppy stack join openarm_sim:alpha brings back the copy the file deployed. One entry speaks for every copy of its option, so a launcher deploys each option of an axis that runs as copies in one entry, and a second entry naming the same option is refused.

--set-arguments INSTANCE.ARGUMENT=JSON5 on stack join overrides one argument of one of the copy’s instances, named as the fragment writes it, the way a copy entry’s arguments does in the file. It applies after every adjustment, over the option entry’s arguments, once per argument on the command line. Numbers must be finite; quote string values:

peppy stack join openarm:charlie --with web_commander \
  --set-arguments backbone_inst.max_ee_velocity_m_s=0.2 \
  --set-arguments commander_inst.http_port=8766

A launch snapshots its launcher and fragment files, and every join composes against that snapshot; launch again to load edits. A join runs each node source that the launch or an earlier join resolved with the pins resolved then, however many joins and removals came between, and resolves only the sources that are new to the stack. Where the join places a node, it reuses the node when that node already runs instances there, and adds and builds it otherwise. A launch with a stack_copies scope resolves and builds the nodes of the robots that its scope offers before any join (see Building the nodes a join brings). Constraints judge the stack’s selection and the copy’s together: on an openarm_simulation stack launched without robot_control or the headset, a copy that selects cameras_sim but no recorder or brain is refused with the rig’s reason. The launcher’s adjustments under a copy’s option may write to a stack instance, and each field they write is what the copy needs of it: the copies the file lists must agree on every field they both write, and a joined copy, whether --join names it at launch or stack join adds it later, may write to a stack instance only what already runs there, with two exceptions: its own instances may pair into the slots the instance declares vacant (the pairing instance’s links name the slot, and the launcher’s adjustments under the copied option release the vacancy with unset_links on the instance), and they may join the one_or_more or zero_or_more sets it declares, below. A join that would change a running instance any other way is refused, naming the instance and each field that differs. That is why the rendered camera rig, which turns the simulation’s rendering on, is selected in the file at launch.

A copy’s composition writes no daemon scope, on its own instances or on a stack instance: a scope written through a copy would exist only while that copy runs. A set_daemon_scopes that the composition of a copy applies (an adjustment under the copied option, a top-level adjustment guarded on the copy’s axis, or the copy’s own adjustments) is refused at launch, at join and by peppy repo index --check, naming the copy, its option, the instance and the target. A copy entry that declares daemon_scopes is refused when the launcher is read.

A stack instance that watches or consumes every robot declares the slot as a set, and the launcher adds each robot under its option. The stack runs the monitor:

deployments: [
  { source: { name: "fleet_monitor", tag: "v1" },
    instances: [{ instance_id: "monitor_inst" }] },
],

where the monitor’s own manifest declares the slot as a set:

depends_on: {
  nodes: [
    { name: "openarm_backbone", tag: "v1", link_id: "robots", cardinality: "zero_or_more" },
  ],
},

and the launcher adds each copy’s backbone to that robots slot under the robot option:

options: {
  openarm: {
    fragments: ["openarm/fragments/openarm.json5"],
    adjustments: [
      { target: "monitor_inst", add_links: { robots: ["backbone_inst"] } },
    ],
  },
},

peppy stack launch monitored          # monitor_inst reads robots as []
peppy stack join openarm:alpha        # [alpha_backbone_inst]
peppy stack join openarm:bravo        # [alpha_backbone_inst, bravo_backbone_inst]
peppy stack remove alpha              # [bravo_backbone_inst]

The members follow the order the copies joined, and a copy that rejoins under the same name is a member once, at the end. A member a join adds behaves like one the launch writes: it is subscribed like any other, keeps its position, and an observed member reads as not live while its source is down. A join adds only the copy’s own instances, and only to a one_or_more or zero_or_more producer or observer slot of the instance’s manifest; naming an instance the stack runs, a one or zero_or_one slot, or a pairing slot is refused at join, naming the slot and the fix. A join reserves the machine of every instance whose set it grows, wherever that instance runs, and is refused when that machine is not live on the federation or no longer runs the instance, naming the slot, the machine and the fix. A copy the launcher file deploys at launch may add a stack instance to a set, beside or in place of its own; removing that copy takes the member out again, and rejoining it by name is refused, so such a copy is launched with the stack.

One simulation per stack, and each simulated robot joins it as a copy that brings its own model: the openarm_simulation launcher ships one copy, an OpenArm v2, --join OPTION:NAME names another at launch and stack join adds more, of any model the simulation’s catalog carries, as many as the machine can run. A physical robot reads wall time and a simulated one the simulation’s clock, so the physical robots have a launcher of their own, physical.

stack remove NAME stops the copy’s instances in reverse dependency order, takes the members it added out of the running sets, and takes the copy out of the stack; every instance whose set shrank returns the smaller set at its accessor’s next call. It stops them one at a time across all the machines of the copy, the last started first. Before it stops an instance, it prints a line that names the instance, such as Stopping instance `alpha_backbone_inst` of copy `alpha` , so a removal of many instances shows each step; an instance on another machine stops through one request of its own, after a line that names its machine: Stopping instance `alpha_brain_inst` of copy `alpha` on `cn-gpu-box` . Each machine of the copy is in one of three states:

• live and holding the slice of this launch: the copy’s instances there stop;
• live but holding no slice of this launch, because someone reset it: nothing stops there, and a line says so: `cn-gpu-box` holds no instance of this launch, so nothing stops there;
• not live on the federation: nothing stops there, and a warning names the copy’s instances that stay there until a peppy stack reset on that machine clears them once it returns, and every other instance of this launch that this reset also stops there, the parts of other copies and the stack’s own instances.

A source on a machine that is not live, whose watchers the removal changes, does not hold the removal: it keeps its watchers until that machine is reset. A stop that fails skips the later stops on its machine while the other machines still stop, and the removal then fails with every error and keeps the copy in stack list, so stack remove can run again. A set on a machine that is not live keeps the members it holds, and the removal says so; when the machine returns with the instance still running, the next join or removal changing that slot delivers it whole, and when it returns without it, launch again. Two floors refuse a removal: one that would leave a one_or_more slot with no member is refused, naming the slot, and one that would take the last copy of a one_or_more axis is refused, naming the copy and the axis. A copy another copy links to is refused until that copy goes first, and a copy a stack instance links to any other way stays until the next launch. Removal leaves the stack’s own instances running: removing a simulated robot takes it out of the simulation, which stays up for the next join, and stack reset stops everything. A copy that released a stack instance’s vacancy to pair into that slot restores the vacancy when it goes, so the slot stands vacant again as the launch declared it, the mirror of pairing into a vacancy above. A failed join stops what it started on the live machines, with the same line for each instance it stops, removes the nodes it added, and reports the failure; when that cleanup itself fails the copy stays in stack list and the message says to retry stack remove.

stack list names each copy on one line with the option it copies, its machine and its selection: Copy delta of openarm_sim on jetson-1: hardware_version=v2,robot_commander=none,camera_rig=cameras_sim,brain=ai_brain_vla@cn-gpu-box. The selection is each filled own axis, in the order the option declares them, as the word axis=option that selects it, with @machine after a part that runs on another machine than the copy’s, joined by ,. A listed copy is its own recipe: peppy stack join openarm_sim:delta@jetson-1 --with followed by the text after the colon adds the same copy on the same machines. A join ends with the same line, Copy `delta` of `openarm_sim` joined on `jetson-1`: hardware_version=v2,…. Under its line, stack list names the copy’s instances and the set members it added, one row per slot under Set members: (monitor_inst.links.robots → alpha_backbone_inst, the target by the id the stack runs it under; the bindings table adds its machine), and its bindings table names the copy beside each member a copy brought (robots → alpha_backbone_inst@cn-robot [copy alpha]). A join writes its copy into the stack only when it ends, so while a join or a removal runs, the coordinator’s panel names it on a line of its own: Copy bravo of so101_sim joins the stack now, or leaves the stack now for a removal. stack list --json carries the same under copies, each selection as { "axis", "option" } with a core_node key on a part that runs on another machine than the copy’s, and copy_change ({ "action": "join", "name": "bravo", "option": "so101_sim" }, or null), the members under set_members, and each member of a slot’s slot_bindings under producer, with a copy key on a member a copy brought, naming the copy and the id its fragment wrote for the instance. The node holding the set reads the same fact through its bound_members() accessor. Preview a launch and the copies it starts without starting anything, which composes it and runs the link rules a launch runs, leaving what only a running stack can answer (whether the instance whose set grows is running, and whether its machine is live) to the join itself:

peppy stack resolve openarm_simulation --with mujoco
peppy stack resolve physical --join openarm:alpha --join openarm:bravo --with xr_commander,bravo.hardware_version=v1,bravo.robot_commander=none

Join and remove run on the coordinator (--core-node NAME reaches it from another machine), serialize with launches and with each other, and reserve only the machines they touch: an unrelated host that is offline blocks neither. A reset cancels the operation in flight, waits for its cleanup, and empties the stack. A restarted coordinator has no launcher to compose against; launch again before joining.

peppy repo index --check composes every legal selection the way a launch does: the stack under every selection of its axes, then every copy an axis that runs as copies can add, as a launch beside the stack and as a join onto it; an entry listing no copy is composed once under the option’s own name. An axis the file leaves for the command line, a one axis for --with or a one_or_more axis for --join, has no bare launch, and the check is silent about it.

Auditing a composed launch

Composition changes instances from a distance, so it ships with an audit command:

peppy stack resolve physical --join so101:alpha --with xr_commander,alpha.robot_commander=none,alpha.lerobot_recorder

stack resolve needs no running stack. It prints the flattened launcher/v1 document to stdout and a resolution report to stderr. An excerpt:

components: robot_control=(off)  teleop=xr_commander
copy alpha: robot=so101  robot_commander=none  recorder=lerobot_recorder  camera_rig=(off)
adjustments applied:
  alpha_backbone_inst.arguments.upstream_mode: "joints" -> "pose"  (so101/fragments/so101.json5)
  alpha_backbone_inst.links.leader_pose: {"vacant":"upstream_mode joints: arm commands arrive on the joint slot"} -> "headset_inst/right_arm"  (so101/fragments/so101.json5)
  headset_inst.links.recorder: + alpha_recorder_inst  (physical.json5, option `robot.so101`, copy `alpha`)
  alpha_recorder_inst.arguments.robot_type: (absent) -> "so101"  (physical.json5, option `robot.so101`)
adjustments skipped:
  alpha_backbone_inst: guard robot_commander=so101_leader is not met  (so101/fragments/so101.json5)
  headset_inst: guard camera_rig=cameras teleop=xr_commander robot_commander=none is not met  (physical.json5, option `robot.so101`, copy `alpha`)
headset_inst: clock `wall`, restart on_failure (default)
alpha_backbone_inst: clock `wall`, restart on_failure (default)
alpha_recorder_inst: clock `wall`, restart on_failure (default)
link rules hold: slot keys, vacancies, pairing coverage, observation sources and clock agreement checked over 5 node manifest(s)

Every applied adjustment is listed with the value it replaced and where it was written, the fragment’s own entries under the fragment’s path and the launcher’s under the option that carries them, and every skip says why. Because stdout is a plain launcher, it doubles as the escape hatch: flatten, hand-edit, launch the flat file. The one exception is a launcher with a stack_copies scope: the flat file keeps the scope and declares no axis, so no option of the scope runs as copies in it, and its launch refuses the scope and says why. Launch such a launcher itself, or take the instance that serves the target out of the flat file before you launch it. A launcher that deploys copies resolves with them, one copy NAME: line each, and --join OPTION:NAME resolves one more, repeatable and comma-separated, each joined onto the plan the ones before it left, exactly as stack launch composes them. A copy --join names takes its own axes from --with NAME.option. A copy NAME: line writes @machine after each part that a word places, as the word wrote it: brain=ai_brain_vla@cn-gpu-box. stack resolve reads no daemon, so it prints @self as written, and its lines name no machine for a copy, which --join OPTION:NAME@CORE_NODE or --place NAME@CORE_NODE gives at launch. The words that a launch refuses, a part placed twice or a placed part that brings nothing, fail stack resolve too. The resulting flat launcher declares each copy’s name in core_nodes and places each copy whole on its name’s link, so a part that a word places runs with its copy when you launch the flat file; launch that file with --place NAME@CORE_NODE or --local.

The report then holds the flat plan to the checks of daemon_scopes. They read the exposure documents and the node manifests alone, so those of the exposure deployments run whatever the nodes and contracts caches hold, and a plan that breaks one fails with the launch’s own refusal. The daemon scopes hold line says how many daemon targets were scoped. When this machine’s caches do not hold an exposure document, or the manifest of a node, the report says daemon scopes not checked and names the deployment, whose checks it skips; the checks of every other deployment still run. A document or a manifest in the caches that a launch refuses, for example one that does not parse or that names an interface this peppy does not serve, fails stack resolve with that refusal.

The report ends by holding the flat plan to the launch-time link rules that need no daemon. stack resolve reads this machine’s nodes cache at ~/.peppy/cache/nodes.json5, loads each deployed node’s manifest, and checks slot-key legality, vacancies, and pairing coverage across them; the link rules hold line names how many manifests it checked. A node published by a git repository is read out of the checkout the caches already hold, peppy repo refresh handing every clone it makes to the checkout cache, so the nodes and launchers a hub publishes are checked against each other without the command ever reaching the network. The contract-binding rules stay launch-only, because satisfying them can involve the root node that only a running daemon knows. When the plan breaks a rule a launch would reject, say a zero_or_one pairing slot left neither paired nor vacant, stack resolve fails with that launch’s own error instead of surfacing it minutes later at launch. When a manifest cannot be read locally the check is skipped rather than run on a partial picture: the report says link rules not checked, names the missing manifest, and advises peppy repo refresh, which is what fills both the nodes cache and the checkouts the git-backed manifests are read from.