The node stack

The node stack is the central registry that manages all your nodes and their relationships in Peppy. Think of it as a directed graph where:

• Nodes are registered configurations that can have multiple running instances
• Edges represent explicit dependencies declared via depends_on in the node manifest

When you add a node to the stack, Peppy validates that its dependencies are satisfied: any topics, services, or actions your node relies on must be provided by other nodes already in the stack. The exception is a dependency whose slot admits an empty bound set (cardinality: "zero_or_one" or "zero_or_more"): it may be absent from the stack, its interfaces resolve through the repository cache so code generation still completes, and a dependency missing from both sources refuses the add; at launch, zero_or_one requires an explicit vacant binding, while zero_or_more may be omitted. This dependency validation prevents runtime errors and helps you understand how your nodes interconnect.

The stack always contains a core node and an instance of it at its root, which coordinates the lifecycle of all other nodes. You can query the stack to see which nodes depend on each other and manage node instances. The node stack is started as a daemon on your system and the peppy binary communicates with it to execute user actions.

Note

The peppy node lifecycle commands (add, build, run, stop, and remove) serialize with stack operations on the same daemon. Each is refused while a stack launch, stack build, stack join, stack remove, or stack reset holds the stack, and any one already in flight when a stack reset starts is cancelled once the reset has given it its cleanup window.

Adding hello_world to the stack

Inside your node directory, add it to the Peppy stack by running:

peppy node add .

This command registers the node with the core node by staging a snapshot of the source directory. After a successful add, the node enters the Added stage: its config is known to the daemon, but no runnable artifact exists yet (since build_cmd has not been run). You’ll produce that artifact in the next section with peppy node build.

The output of this command provides some useful information:

Adding node from /private/tmp/hello_world...
Log file: ~/.peppy/logs/add/hello_world_v1_20260121_224824_699.log
Added node hello_world:v1 to the node stack

The log file holds each step the daemon reported while adding the node, and its error in case the command fails. When a node is added to the node stack, a copy of the source is staged in a working directory under ~/.peppy/tmp/ to “snapshot” the node, and the node’s bindings are generated into that copy, never into the source directory. The add reports success once generation has finished and the node is in the stack; the artifact that peppy node build produces later lands under ~/.peppy/built_nodes/<name>_<tag>/, named after a fingerprint of the staged sources. The staged copy is removed together with the node, so a failed add or a peppy node remove leaves nothing behind under ~/.peppy/tmp/. No two copies of the same node with the same name + tag can exist at the same time in the node stack. Re-adding a node with the same name and tag overrides the one that was previously in the node stack. The overwrite is refused when the re-add changes the node’s interfaces while other nodes depend on it. If an instance of the node has a process or awaits a restart, peppy node add of a local directory or a name:tag asks for confirmation, naming each such instance with its state; the add stops and clears every tracked instance before overwriting, an exited one without asking; an add from git or a URL stops and clears them without asking. While one of them is starting, the add is refused and stops nothing.

Building hello_world

An added node has no runnable artifact until it’s built. Building runs the node’s build_cmd (for example uv sync for Python or cargo build --release for Rust) against the snapshot that peppy node add staged, unless an artifact built from that same snapshot is already cached (see below):

peppy node build hello_world:v1

The format is <node_name>:<tag> for a node that is already part of the node stack. While the build is running the node sits in the Building stage; once it completes, it moves to Ready and can be run.

Python

Caution

uv must be installed on your system. Without it, the build will fail with a No such file or directory error.

Note

Container builds on the peppy Python base image cache the packages and the Python interpreter archives uv downloads automatically, so a rebuild downloads neither again; see Build caching.

Rust

Caution

cargo must be installed on your system. Without it, the build will fail with a No such file or directory error.

Note

For faster Rust builds, install sccache. When Peppy detects sccache on your system PATH and you have not set RUSTC_WRAPPER yourself, it automatically sets RUSTC_WRAPPER=sccache for process-node builds. Container builds on the peppy base images cache the cargo registry and sccache artifacts automatically, no installation needed; see Build caching.

The output looks like:

Building node hello_world:v1...
Log file: ~/.peppy/logs/build/hello_world_v1_20260121_224901_312.log
Built node hello_world:v1. Artifact: ~/.peppy/built_nodes/hello_world_v1/3f9a1c2e5d7b8a4f.tar.zst

The artifact is a .tar.zst archive. When you start an instance with peppy node run, the daemon extracts it into a per-instance directory and executes run_cmd there.

The file name is the fingerprint of the staged sources, mixed with the peppy version, the host platform and the artifact kind. Building the node again from byte-identical sources with the same peppy version on the same platform reuses the artifact instead of running build_cmd, and the build log says so:

Reusing cached build of hello_world:v1 (fingerprint 3f9a1c2e5d7b8a4f) at ~/.peppy/built_nodes/hello_world_v1/3f9a1c2e5d7b8a4f.tar.zst

Pass peppy node build hello_world:v1 --rebuild to build from the sources anyway, for instance after a toolchain change on the host; the result replaces the cached artifact. See Build caching for what the fingerprint covers.

Starting your node

Start your node:

peppy node run hello_world:v1

This will run the run_cmd command from the peppy.json5 process configuration. The format is <node_name>:<tag> for a node that is already part of the node stack.

peppy node run also decides which clock the instance reads. --clock <DOMAIN> binds it to a domain that is already running, named as wall, a short name, or a qualified name@core_node when one name runs on several machines; --publish-clock <DOMAIN> declares a domain and starts this instance as its publisher. The two flags conflict, and an instance that names neither reads wall time:

peppy node run waldo:v1 -i simulation_inst --publish-clock simulation
peppy node run openarm_sim_arm:v1 -i arm_inst --clock simulation

--restart POLICY sets the instance’s restart policy, never, on_failure or always, over the node’s framework.restart; --max-restarts N caps the restart attempts in a row, failed starts included, and the next exit or failed start past them leaves the instance failed; it requires --restart on_failure or --restart always.

A node whose manifest declares execution.endpoints ends node run with the URLs its instance serves, one per address of the machine hosting it, loopback first:

Started node instance 'panel_inst' (pid: 12345)
Web pages:
  panel_inst (openarm_web_commander:v1) @cn-sweet-edison
    panel  http://127.0.0.1:8765
           http://100.123.58.116:8765

The node run command outputs the path to a log file where you can inspect the node’s output. For example:

cat /home/ubuntu/.peppy/logs/run/elegant-solomon-5423_20260217_101523_412.log
[2026-02-17T10:15:23.599] Executing run_cmd: uv run hello_world (working_dir: /home/ubuntu/.peppy/instances/elegant-solomon-5423)
[2026-02-17T10:15:23.837] [stdout] hello world count 1
[2026-02-17T10:15:26.848] [stdout] hello world count 2
[2026-02-17T10:15:29.852] [stdout] hello world count 3

How an instance starts

The daemon starts an instance in three steps. peppy node run, peppy stack launch and peppy stack join use the same steps.

1. Ready. The daemon runs the run_cmd (or the container) as the leader of a new process group, and waits up to 600 seconds for the ready signal of the instance. The node runtime (peppylib) answers ready when it is connected to messaging and listens for the shutdown request, before your setup function runs.
2. Setup. The daemon then waits for your setup function to return, for the setup budget of the node. A node declares its setup budget in its manifest as execution.setup_timeout_secs, a whole number of seconds from 1 to 540. A node that declares none gets the default setup budget, 15 seconds. When the setup does not return within the budget, the start fails. The message gives the budget and its source: the execution.setup_timeout_secs of the node, or the default setup budget, with the instruction to declare the key.
3. Endpoints. A node that declares execution.endpoints then reports the sockets it bound. The daemon waits 15 seconds for them.

peppy.json5

{
  // ...
  execution: {
    language: "rust",
    run_cmd: ["./target/release/arm_driver"],
    // The setup waits for the arm controller, which can take two minutes to boot.
    setup_timeout_secs: 180,
  },
}

Declare a setup budget when your setup waits for something outside the node, such as a device, a simulation or a remote service. Keep the budget as small as the work permits: the daemon finds a setup that hangs only at the end of its budget.

A node reads its setup budget with node_runner.setup_timeout() in Rust (a Duration) and node_runner.setup_timeout_secs() in Python (an int). The daemon writes the budget it applies into the runtime configuration of the instance. A node that runs standalone, or in its test harness, reads the budget of its manifest. A setup that waits for something can use the budget to stop with its own reason before the daemon stops the instance.

When the instance answers ready, the daemon sends a progress line. This line restarts the run idle budget of a launch (see Launch timeouts). The largest setup budget (540 seconds) is less than the default run idle budget (600 seconds), thus a setup that prints nothing gets the setup error, not the idle error.

When the daemon stops a start before it ends, the step of the start decides how:

• Before the ready signal, the node does not listen for the shutdown request. The daemon kills the whole process group of the instance at once: the node and every process that it started in its group.
• After the ready signal, the daemon stops the instance as peppy node stop stops a running instance. It sends the shutdown request, waits for the shutdown grace and the runtime teardown, and then kills the process group. The node runs its shutdown hooks, so a setup that holds a resource (a lock, a robot in a simulation) releases it. This applies when the setup outlasts its budget, when the endpoints step fails, when the run budget of a launch or a join runs out (--node-run-idle-timeout-secs, --max-timeout-secs), when peppy stack reset cancels the start, and when the command that started it cancels it (see When a node command ends early). A stack reset waits for this stop to end before it clears the stack.

A synchronous Python setup cannot see the shutdown request before it returns. When it does not return within the grace and the runtime teardown, the kill ends it. An async setup stops at its next await. The shutdown lifecycle guide gives the full sequence.

Chaining add, build, and run

Now that you’ve seen all three steps, peppy node add exposes flags that chain them for you:

• --build (-b) runs peppy node build immediately after the add succeeds.
• --run (-r) also spawns an instance once the build finishes, and implies --build.
• --sync (-s) runs peppy node sync before the add, so the snapshot picks up any edits you made to peppy.json5. Only valid for local filesystem sources; remote git/HTTP sources are synced server-side on fetch.

peppy node add -sb .    # sync, then add, then build
peppy node add -sr .    # sync, then add, then build, then run an instance

Tip

-sb and -sr are the shorthands you’ll reach for constantly during development. The typical edit-loop is: change peppy.json5 or source, hit peppy node add -sr ., watch the logs, iterate. Without -s, a stale .peppy/ bindings directory can make the daemon see an out-of-date interface; without -r, you have to follow every add and build with a separate peppy node run. Bundling them collapses the whole “I edited something, now run it” cycle to a single command.

Run-only options (trailing key=value arguments, --instance-id, --link KEY@VALUE, --vacant-link SLOT=<why>, --clock DOMAIN, --publish-clock DOMAIN, --restart POLICY and --max-restarts N) are only accepted alongside -r/--run; combining them with a plain peppy node add is rejected at parse time so an arg you cared about isn’t silently dropped. Keep key=value arguments after the source path (e.g. peppy node add -sr . frequency=30) so the source positional isn’t confused with the trailing args. --link mirrors the peppy node run --link flag and runs the same validator; see Bindings and routing.

When a node command ends early

peppy node add, build and run each send the daemon one goal and follow it to its end. The daemon refuses a goal that it cannot start, and a goal can fail. Either way the command prints the failure kind and the reason, a failure also the log file, and exits with code 3 for a refusal and 4 for a failure (see Exit statuses of the other commands):

Error: Node add refused (busy): action already in progress (times out in 590s), use `--force` to force adding the node
Error: Node build failed (attempt_failed): build_cmd `uv sync` failed with status exit status: 2. Log file: ~/.peppy/logs/build/hello_world_v1_20260121_224901_312.log

The daemon refuses a goal with the kind busy while other work holds the stack: another add, build or run of the same command, a stack change, a peppy stack reset, or a launch that holds the machine. The same command can succeed once that work ends. peppy node stop and peppy node remove get the same refusal while a stack change, a reset or a launch that holds the machine runs.

The first Ctrl+C sends the daemon a cancel of the goal. The daemon stops the work at its next step: an add publishes no node, a build kills its build_cmd and leaves the node Added, and a start stops its instance as How an instance starts describes. The command follows the goal to that end under --max-timeout alone, prints how it ended, and exits with the status of that end, 5 for a cancel:

Cancelling the node build; press Ctrl-C again to stop waiting for its end
Error: Node build was cancelled by its caller. Log file: ~/.peppy/logs/build/hello_world_v1_20260121_224901_312.log

A goal that ended before the cancel reached it keeps its end, and the command says so: Node build completed before the cancel took effect., with the exit status 0. A goal that failed for a cause of its own, for example a build that failed before the cancel reached it, ends failed with its kind and the exit status 4. A second Ctrl+C stops the wait for the end: the command waits only until the cancel reaches the daemon, which goes on with it, and exits with code 130. A third Ctrl+C ends the command at once. A command whose process goes away, for example when its terminal closes, cancels its goal too: the daemon sees that the sender of the goal is gone, and cancels the goal one minute later, as for a stack command.

peppy node add --run follows three goals, one after the other: the add, the build and the start. Between two of them, a Ctrl+C ends the command at once with code 130. A Ctrl+C during one of them stops the command when that goal ends: a goal that completes before the cancel takes effect also ends the command with code 130, and the goals after it do not start. The same applies to peppy node add --build, and to peppy node run when it builds the node before the start.

--force on peppy node add or peppy node build cancels the add or the build of the node that runs. That goal ends Node add was superseded by a forced node add. or Node build was superseded by a forced node build., its command exits with code 5, and the forced one starts once it ended.

--idle-timeout and --max-timeout bound the wait of the command. A command whose budget runs out stops waiting and exits with code 1. It sends no cancel, but its process ends, so the daemon cancels the goal, as for any command whose process goes away.

The command keeps up to 65536 lines of the goal’s output that it did not show yet. A command that falls further behind cancels the goal, says The command fell behind the output of the node build by more than 65536 unread messages and cancelled it., and exits with code 1.

Checking node status

View all added & running nodes:

peppy stack list

You should see something like:

$ peppy stack list
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Core node: cn-funny-salamander (host: robot-host)                                                                               │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Node stack                                                                                                                      │
│                                                                                                                                 │
│ ┌─────────────────────────────────────────┬───────┬───────────┬──────────────────────────────────────────────────────────────┐  │
│ │ NODE                                    │ STAGE │ INSTANCES │ PATH                                                         │  │
│ ├─────────────────────────────────────────┼───────┼───────────┼──────────────────────────────────────────────────────────────┤  │
│ │ cn-funny-salamander:v0.10.0             │ Root  │ 1 running │ ~/.peppy/bin                                                 │  │
│ │ hello_world:v1                          │ Ready │ 1 running │ ~/.peppy/built_nodes/hello_world_v1/3f9a1c2e5d7b8a4f.tar.zst │  │
│ └─────────────────────────────────────────┴───────┴───────────┴──────────────────────────────────────────────────────────────┘  │
│                                                                                                                                 │
│ Instance bindings                                                                                                               │
│                                                                                                                                 │
│ ┌─────────────────────────────────────────┬─────────────────────────────────┬─────────┬─────────┬──────────┬───────┬──────────┐ │
│ │ NODE                                    │ INSTANCE                        │ STATUS  │ HEALTH  │ RESTARTS │ CLOCK │ BINDINGS │ │
│ ├─────────────────────────────────────────┼─────────────────────────────────┼─────────┼─────────┼──────────┼───────┼──────────┤ │
│ │ cn-funny-salamander:v0.10.0             │ cn-funny-salamander             │ running │ healthy │ 0        │ wall  │ (none)   │ │
│ ├─────────────────────────────────────────┼─────────────────────────────────┼─────────┼─────────┼──────────┼───────┼──────────┤ │
│ │ hello_world:v1                          │ suspicious-swanson-5880         │ running │ healthy │ 0        │ wall  │ (none)   │ │
│ └─────────────────────────────────────────┴─────────────────────────────────┴─────────┴─────────┴──────────┴───────┴──────────┘ │
│                                                                                                                                 │
│ Dependencies                                                                                                                    │
│   (none)                                                                                                                        │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

With no arguments, stack list discovers every live core node in the same messaging namespace and prints one outer panel per distinct core-node name: the local name first, then the others by name. The panel header identifies the core node and its hostname, while the enclosing border groups its node stack, instance bindings, and dependencies, plus, when present, an Instance endpoints table (one row per HTTP endpoint an instance serves, such as an MCP exposure), an Instance pairings table, and launch-ownership lines (Slice of launch ..., and Reserved for launch ... naming the reset command that clears it). If multiple live daemons collide on one name, discovery deduplicates them into a single warning-bearing panel. Use peppy stack list --core-node <name> to filter the output to that one name. If a core node disappears between discovery and the stack query, the other panels still render, the failed panel contains an error line, and the command exits non-zero. The panels and their tables fit the terminal: the widest columns give way first, never below a header’s width, and over-long cells or lines wrap inside the borders; piped output is never wrapped. --json prints the same discovery as one JSON document instead: one entry per core node with its identity, launch and reservation, its copies and copy_change, and either the stack graph (nodes and edges) or the query error.

The STAGE column reports each node’s stage, its artifact-level lifecycle:

• Added: the node is registered and a snapshot of the source has been taken, but build_cmd has not been run yet, so there is no runnable artifact.
• Building: a build is in progress. A second concurrent node build on the same entity is rejected until this one finishes, unless it passes --force, which cancels and supersedes the in-progress build. A build whose staged sources match a cached artifact passes through this stage without running build_cmd; --rebuild makes it build anyway.
• Ready: the build artifact is on disk. The node can be run; it may currently have zero, one, or several instances.
• Root: the synthetic core-node entity. Always present, and always reports exactly one running instance (the daemon itself).

The INSTANCES column aggregates per-instance state (starting, running, restarting, and the terminal finished / failed) for the node, independently of its stage. To see individual instance IDs, use peppy node info.

Below the node table, the Instance bindings table expands the stack one level further: one row per instance, grouped by node, alongside the slot bindings that instance holds: what it resolved when it started, as joins and removals have since grown or shrunk them. A binding ties one of the consumer’s depends_on slots (identified by its link_id) to the producer instance(s) that feed it, rendered as link_id → producer (a multi-cardinality slot lists every bound member in plan order; an empty slot, whether a zero_or_more slot bound to nothing or a zero_or_one slot declared vacant, shows (empty set)). A daemon target takes no links entry: it shows the daemon that runs the instance, under its per-boot instance id. Every other declared producer slot is either bound or, where its manifest declares cardinality: "zero_or_one", declared vacant; only a zero_or_more slot may be left out of the launcher entirely, and launch rejects any other unfulfilled producer slot. (none) means the instance has no producer bindings: a node whose only slots are pairings or observations also shows (none) here, with its pairing slots rendered in the separate Instance pairings table. Only nodes that currently have at least one instance appear in this table. See Bindings and routing for how slots are declared and how --link pins them. In the listing above neither node declares a dependency, so every binding cell is (none).

Instance health and lifecycle

Beyond the bindings, the Instance bindings table reports four per-instance columns that follow an instance through its runtime lifecycle:

• STATUS is the instance’s lifecycle state. A freshly launched instance is starting while its process comes up and clears its startup health gate, then becomes running once it has started successfully. If its process later exits on its own, it moves to restarting when its restart policy starts it again, and is starting and running once more after the wait; a start that fails returns it to restarting, and the attempt past max_restarts, or a restart the daemon refuses, leaves it failed. Otherwise it moves to a terminal state: finished when it exited cleanly (a one-shot node that completed its work and shut itself down, see the shutdown lifecycle guide), or failed when it exited with an error code or was killed. A terminal instance has stopped for good. These are the same states peppy node info prints as [starting] / [running] / [restarting] / [finished] / [failed].
• HEALTH is whether a running instance answers the core node’s health checks: healthy or unhealthy. Every instance starts out healthy. An instance without a process (restarting, finished or failed) has no live health to report and shows -.
• RESTARTS counts the times a restarted process of the instance reached running.
• CLOCK is the clock the instance reads: wall, or a simulated domain as name@core_node (see the system clock guide).

Once an instance is running, the daemon keeps probing its health in the background (every 5 seconds, allowing 3 seconds per health check) and updates the HEALTH flag from the run of results:

• Three consecutive missed health checks mark the instance unhealthy. This is for a process that is alive but not answering: a wedged or deadlocked node, or one that has lost its messaging session. The process is still up; it just is not responding, and has not been for about twenty seconds.
• A single missed health check leaves the instance healthy. A node, its container, or the host stalling for a few seconds (a swapped-out VM, a saturated CPU) misses a check or two and then answers again; that is a blip, not a health change, and the daemon only records it at debug level.
• One successful health check marks an unhealthy instance healthy again.
• Under on_failure or always, an instance that has answered no health check for a minute is killed at its next missed check: SIGTERM, then SIGKILL of its process group once the shutdown grace and the runtime’s teardown have had their time (12 seconds at the default grace). The exit counts as a failure, so the policy restarts the instance unless max_restarts is reached, and the run counts only up to the last health check the process answered. A node that goes silent is killed 60 to 68 seconds after the last check it answered: the minute, then the check in flight, which takes up to 8 seconds (the 5-second wait and its own 3-second timeout); the node went quiet up to 8 seconds after that answer, while the next check was in flight, so the kill comes 52 to 68 seconds after it stopped answering. The minute runs from the last answered check whether later checks time out or fail at once, and it outlasts the 45.5 seconds a peppy-managed router listening on TCP or TLS takes at most to come back after one restart by its watchdog, plus the checks in flight when it dies and when it comes back (up to 11 seconds), so a router the watchdog heals never costs a node its process. While two or more instances other than the core node’s own run on this daemon and none of them is healthy, the daemon reads a messaging outage and holds the kill until one of them answers again; a lone silent instance is killed. The stack log records the kill, or the outage. Under never an unhealthy instance stays running.

A process that has actually exited is a case of its own, apart from an unhealthy one. The daemon watches each running node’s process directly, so when a node exits on its own (a one-shot node finishing, or a crash) the instance moves to restarting, finished or failed by its restart policy as soon as its process group is killed, and gets no health check in between. For a container on macOS that kill reaches into the Lima VM and takes up to 30 seconds.

Neither an unhealthy nor a terminal instance is automatically removed from the stack. Under never, an unhealthy running instance stays listed until it recovers on its own, so an outage (a lost messaging session that takes a while to come back) surfaces as an unhealthy window and the stack keeps every instance; under on_failure or always it is killed after a minute without an answered health check, outside a messaging outage, and its policy treats that exit as a failure. A finished or failed instance likewise stays listed, so you can see that it completed or died. Either is cleared when you stop it with peppy node stop (one instance) or peppy node remove (the whole node), or when the stack is relaunched. peppy node stop also clears a restarting instance and cancels its restart. Starting a failed instance again says how to start a failed or finished instance again.

Each lifecycle transition (a running instance going unhealthy or recovering, a process exiting on its own, as a finished or failed line whether or not a restart follows, an exit on peppy node remove’s shutdown request that lands before the removal claims the instance included, a health kill or the outage that holds it, and a restart scheduled, completed, failed to start, refused or capped by max_restarts) is appended to the daemon’s stack log at ~/.peppy/stack_log.log, giving you a timestamped record of when an instance failed, finished, or came back. With the OpenTelemetry log export on, the daemon also sends each line of this log, and of the launch, add, build and run logs, to the endpoint you configured.

Daemon shutdown and orphan prevention

Spawned nodes are child processes of the daemon, so Peppy takes care to ensure that no node (or any process a node itself spawned) is ever left running after the daemon goes away.

A node that exits on its own. Whatever a node forked into its process group is killed with the group the moment the node’s own process exits, whether the exit is a clean one, a crash, or the kill the health monitor sends, and so is the group of a start that fails. A child that moves itself to another process group or session is outside it.

Clean shutdown (Ctrl+C, systemctl stop, SIGTERM). Before it exits, the daemon tears down every spawned node: it asks each node to shut down cooperatively (in the node, this fires the cancellation token and runs its shutdown hooks; see the shutdown lifecycle guide), waits for each to exit cooperatively (the shared shutdown grace period for cleanup, shutdown_grace_secs, 5 seconds by default, plus the brief runtime teardown that follows), then force-kills the process group of anything still alive. This is immediate: it does not wait the daemon-liveness grace period described below, so stopping the daemon feels instant while still guaranteeing nothing is orphaned. It’s the same graceful-then-forced sequence peppy node stop runs for a single instance, which uses the same shutdown_grace_secs window.

Unclean daemon death (crash, OOM, SIGKILL). When the daemon dies without the chance to run that cleanup, it can’t tear its nodes down. On Linux the kernel ends the daemon’s zenohd the moment the daemon process dies, so until a daemon starts again the nodes have no router: in the router topology every message relays through it, and in the peer topology it carries cross-machine traffic and seeds discovery for new peers (see local messaging topology). On macOS a daemon run in a terminal leaves its zenohd running, and the next daemon start is refused with “Zenoh router port already in use” until that zenohd is stopped (pkill -f zenohd_config_<port>.json5, with the port from the message); under the installed peppy service, launchd (with AbandonProcessGroup unset) ends the processes in the daemon’s process group, zenohd included, and restarts the daemon.

The nodes themselves keep running. Each runs a watchdog that listens for the periodic heartbeat of the daemon that started it. Each daemon process draws an instance id when it starts, sends its heartbeat and holds a presence token under that id, and writes the id into the runtime config of every node it starts. The node shuts itself down when:

• it sees no heartbeat of its own daemon for the daemon grace period, daemon_grace_secs (180 seconds / 3 minutes by default). The heartbeats of other daemons do not count;
• another daemon of its core node sends a heartbeat, and its own daemon no longer holds its presence token: a new daemon runs. A new daemon starts with an empty stack, so no stack holds the node any more, and your next launch starts the nodes again.

So a restart of the daemon stops every node that outlived the old daemon as soon as the new daemon sends its first heartbeat, also when the old daemon died before the node received anything from it. This applies to a daemon run in a terminal, and to the installed peppy service on macOS, where launchd ends only the processes in the daemon’s process group. Under the installed peppy service on Linux, systemd ends every process of the unit when the daemon exits, nodes included, and then restarts the daemon on its own.

The grace period is deliberately generous so a brief daemon stall or a restart of the daemon’s messaging router does not tear your nodes down: the router restarts inside the same daemon process, which keeps its instance id and its presence token. A daemon of another core node, or a daemon on another messaging port, never stops your nodes. Neither does a second daemon that claims the same core node name while the daemon of your nodes runs: each daemon reports that name collision, and the nodes keep running.

Both grace periods are configured in ~/.peppy/conf/peppy_config.json5 (see Daemon configuration for the full file reference) and take effect after a daemon restart:

peppy_config.json5

{
  lifecycle: {
    // Seconds a node waits without a daemon heartbeat before it self-terminates
    // to avoid orphaning. Default: 180 (3 minutes). Minimum: 30.
    daemon_grace_secs: 180,

    // Seconds a clean shutdown and `peppy node stop` wait for a node to exit
    // cooperatively before force-killing it. Raise it for nodes that need longer
    // to park actuators or release hardware. Default: 5. Minimum: 1.
    shutdown_grace_secs: 5,
  },
}

Caution

daemon_grace_secs must be at least 30 seconds (a smaller value is rejected when the config is loaded): it has to comfortably exceed the heartbeat interval and a router restart, or a momentary daemon hiccup would trip every node’s watchdog. shutdown_grace_secs must be at least 1 second.

Inspecting a node with peppy node info

When you need more than the one-line view from stack list, peppy node info dumps the node’s stage, its currently-tracked instances with their individual states and last-known health, and the paths to its add log and per-instance run logs. It takes a <node_name>:<node_tag> reference to a node that has already been added to the stack; it doesn’t touch the filesystem or any git/http source, so run peppy node add first if the node isn’t in the stack yet:

peppy node info hello_world:v1

[INFO] Getting node info for hello_world:v1...

Node Information
==================================================

Name:      hello_world
Tag:       v1
Language:  Python
Build cmd:   uv sync
Run cmd: uv run hello_world

Node Stack Status
--------------------------------------------------
Stage:     Ready
Instances: 2 tracked
           - suspicious-swanson-5880  [running]  healthy  restart on_failure (default), 0 restarts
           - flamboyant-penrose-9622  [running]  healthy  restart on_failure (default), 0 restarts

Logs
--------------------------------------------------
hello_world:v1
  Add log: /home/ubuntu/.peppy/logs/add/hello_world_v1_20260416_101124_613.log
  Run logs:
    - suspicious-swanson-5880: /home/ubuntu/.peppy/logs/run/suspicious-swanson-5880_20260416_101131_207.log
    - flamboyant-penrose-9622: /home/ubuntu/.peppy/logs/run/flamboyant-penrose-9622_20260416_101131_318.log

Exposed Interfaces
--------------------------------------------------
Emitted Topics:
  - message_stream (qos: SensorData)

Integrity
--------------------------------------------------
Config SHA256: 67505086f8ac099d0861cb5dfa539b1c8c3c9354ad90ed7713a5147c7cb43342

This is usually the fastest way to answer “is my node built yet?” and “where is the log for this specific instance?”. The Add log path is what peppy node add writes to; a run log is what one process of an instance writes to, <instance_id>_<timestamp>.log, so a restart opens a new file and the listing names the latest one. Build logs live under ~/.peppy/logs/build/ and are surfaced by peppy node build directly when it runs.

Each instance line ends with its restart policy and the level that set it, then the restarts so far:

• The level is instance entry (the launcher’s entry, or an adjustment), --restart (the flag of peppy node run and peppy node add --run), peppy.json5 (the node’s own framework.restart), launcher root, or default (no level set one, so on_failure). A cap prints beside the policy as max_restarts N.
• A restarting instance adds the attempt it waits for and next start in N s, or next start due; it waits for the last exit's notices to other daemons, or for a stack launch, build, join or removal or a `peppy node remove` in progress on this daemon once the wait has passed and the restart has not begun.
• A failed instance adds attempt N refused: max_restarts M reached, failed starts included when the cap ended its restarts, or attempt N refused when the daemon refused to start it again (the stack log says why).
• Every instance that has exited adds its last exit, such as last exit: exit code 1, last exit: signal 9 (SIGKILL) or last exit: killed after 60 s without an answered health check, or why a start failed.

Exposed Interfaces summarizes the topics, services, and actions the node publishes per its peppy.json5; consumed interfaces get their own Consumed Interfaces section, and a Dependencies section lists the slots (link_id) they go through. A node with parameters gets a Parameters section too, and labels or a container block add a Labels: or Container: line under Node Information; each is omitted when empty, as in the listing above. Under Emitted Topics, a native topic (one whose shape the node declares in its own peppy.json5) states its QoS, such as message_stream above. A native topic that retains its newest messages for subscribers that join later also states its retention, for example - robot_mode (qos: Reliable, retention: latest 3). A contract-backed or pairing-backed topic names its slot instead of a QoS, for example - video_stream (via implements slot: cam). Its QoS comes from the contract or the pairing; a contract-backed topic takes its retention from the contract, and a pairing-backed topic is always live only. Config SHA256 is the fingerprint of the config that was fed through peppygen; it’s what peppy node add checks against to refuse a stale snapshot and tell you to run peppy node sync.

An instance of a node that declares execution.endpoints lists each endpoint under its instance line as label (kind) url, one line per URL:

Instances: 1 tracked
           - panel_inst  [running]  healthy  restart on_failure (default), 0 restarts
               panel (page)  http://127.0.0.1:8765
               panel (page)  http://100.123.58.116:8765

Calling a running instance

peppy node describe <instance> shows the topics, services and actions of a running instance with their message formats. peppy node call <instance> <member> calls one of its services or actions, and peppy node echo <instance> <topic> prints the messages of one of its topics, with no code of your own:

peppy node call camera_inst camera.set_brightness --set value=12

{"outcome":"completed","response":{"applied":true}}

See Calling a running instance for the names of the members, the flags that give a message, the output and the exit statuses.

Stopping your node

When you’re done, you can stop your node:

peppy node stop <instance-id>

This stops one instance of the hello_world:v1 node, but the node itself remains in the node stack. peppy node stop also clears a finished, failed or restarting instance, and ends the start of an instance its restart policy is starting again; a first start in flight is refused until it completes.

Stopping is a graceful-then-forced operation, and the command blocks until the instance’s process has actually exited:

1. The daemon asks the instance to shut down cooperatively and gives it a grace period (shutdown_grace_secs, 5 seconds by default; see Daemon shutdown and orphan prevention) for its cleanup to run. For a robot node this is its chance to park actuators, release hardware, and flush state. Inside the node, this ask fires the runtime’s cancellation token and then runs the node’s registered shutdown hooks; see Graceful shutdown for the quick-start and the shutdown lifecycle guide for the full node-side contract.
2. If the instance doesn’t exit cooperatively in time (the grace window plus the brief teardown its runtime needs afterward), the daemon force-kills its whole process group (the node and any child processes it spawned), so nothing is left running.

If the instance had to be force-killed, peppy node stop warns you instead of reporting a clean stop:

WARN Node instance 'hello_world-...' did not shut down gracefully within the grace period and was force-killed

Caution

A force-kill is abrupt: the node gets no chance to run its own shutdown logic. A node that must stop hardware cleanly should handle the cooperative shutdown signal and exit promptly within the grace period. The same graceful-then-forced sequence runs for every node when the daemon itself shuts down; see Daemon shutdown and orphan prevention.

Removing your node

To remove the node from the node stack, run:

peppy node remove hello_world:v1

A node that tracks instances is removed with --stop-instances (without it the command asks first). The removal asks each running instance to shut down and, once every request succeeds, waits for each process to exit until the force-kill deadline, the grace period plus the runtime’s teardown (12 seconds at the default grace); it force-kills the whole process group of one that has not exited, and returns once every process is gone, warning for each it force-killed. An instance whose shutdown service is unreachable gets no request (a container gets a SIGTERM) and is force-killed at the same deadline. A shutdown request that fails fails the removal, which then stops nothing: an instance that answered its request ends if it exits within the force-kill deadline, and peppy node stop ends the one that failed it. An instance starting when the removal reaches it, or running a process that reached running after the removal began, keeps the node, and the removal’s error names it.

If you run peppy stack list again, you’ll see that only the core node remains:

$ peppy stack list
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Core node: cn-funny-salamander (host: robot-host)                                                                               │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Node stack                                                                                                                      │
│                                                                                                                                 │
│ ┌─────────────────────────────────────────┬───────┬───────────┬──────────────┐                                                  │
│ │ NODE                                    │ STAGE │ INSTANCES │ PATH         │                                                  │
│ ├─────────────────────────────────────────┼───────┼───────────┼──────────────┤                                                  │
│ │ cn-funny-salamander:v0.10.0             │ Root  │ 1 running │ ~/.peppy/bin │                                                  │
│ └─────────────────────────────────────────┴───────┴───────────┴──────────────┘                                                  │
│                                                                                                                                 │
│ Instance bindings                                                                                                               │
│                                                                                                                                 │
│ ┌─────────────────────────────────────────┬─────────────────────────────────┬─────────┬─────────┬──────────┬───────┬──────────┐ │
│ │ NODE                                    │ INSTANCE                        │ STATUS  │ HEALTH  │ RESTARTS │ CLOCK │ BINDINGS │ │
│ ├─────────────────────────────────────────┼─────────────────────────────────┼─────────┼─────────┼──────────┼───────┼──────────┤ │
│ │ cn-funny-salamander:v0.10.0             │ cn-funny-salamander             │ running │ healthy │ 0        │ wall  │ (none)   │ │
│ └─────────────────────────────────────────┴─────────────────────────────────┴─────────┴─────────┴──────────┴───────┴──────────┘ │
│                                                                                                                                 │
│ Dependencies                                                                                                                    │
│   (none)                                                                                                                        │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Next steps

Now that you can manage nodes in the stack, you can:

• Add interfaces to communicate with other nodes
• Configure parameters for your node
• Start multiple nodes at once with launch files