The Node stack

The node stack is the central registry that manages all your nodes and their relationships in PeppyOS. Think of it as a directed graph where:

• Nodes are registered configurations that can have multiple running instances
• Edges represent explicit dependencies declared via depends_on in the node manifest

When you add a node to the stack, PeppyOS validates that all its dependencies are satisfied, ensuring that any topics, services, or actions your node relies on are provided by other nodes already in the stack. This dependency validation prevents runtime errors and helps you understand how your nodes interconnect.

The stack always contains a core node and an instance of it at its root, which coordinates the lifecycle of all other nodes. You can query the stack to see which nodes depend on each other, visualize the dependency graph, and manage node instances. The node stack is started as a daemon on your system and the peppy binary communicates with it to execute user actions.

Adding hello_world to the stack

Inside your node directory, add it to the PeppyOS stack by running:

peppy node add .

This command registers the node with the core node by staging a snapshot of the source directory. After a successful add, the node enters the Added stage: its config is known to the daemon, but no runnable artifact exists yet (since build_cmd has not been run). You’ll produce that artifact in the next section with peppy node build.

The output of this command provides some useful information:

Adding node from /private/tmp/hello_world...
Log file: ~/.peppy/logs/add/hello_world_v1_20260121_224824_699.log
Added node hello_world:v1 to the node stack

This gives you access to the log file for debugging in case the add command fails. When a node is added to the node stack, a copy is made in the peppy cache under ~/.peppy/built_nodes/ to “snapshot” the node based on its name and tag. No two copies of the same node with the same name + tag can exist at the same time in the node stack. Re-adding a node with the same name and tag overrides the one that was previously in the node stack, except if that node has dependencies, in which case the operation fails.

Building hello_world

An added node has no runnable artifact until it’s built. Building runs the node’s build_cmd (for example uv sync for Python or cargo build --release for Rust) against the snapshot that peppy node add staged:

peppy node build hello_world:v1

The format is <node_name>:<tag> for a node that is already part of the node stack. While the build is running the node sits in the Building stage; once it completes, it moves to Ready and can be run.

Python

Caution

uv must be installed on your system. Without it, the build will fail with a No such file or directory error.

Rust

Caution

cargo must be installed on your system. Without it, the build will fail with a No such file or directory error.

Note

For faster Rust builds, install sccache. When Peppy detects sccache on your system PATH, it automatically sets RUSTC_WRAPPER=sccache for Rust node builds, caching compilation artifacts across runs.

The output looks like:

Building node hello_world:v1...
Log file: ~/.peppy/logs/build/hello_world_v1_20260121_224901_312.log
Built node hello_world:v1. Artifact: ~/.peppy/built_nodes/hello_world_v1.tar.zst

The artifact is a .tar.zst archive. When you start an instance with peppy node run, the daemon extracts it into a per-instance directory and executes run_cmd there.

Starting your node

Start your node:

peppy node run hello_world:v1

This will run the run_cmd command from the peppy.json5 process configuration. The format is <node_name>:<tag> for a node that is already part of the node stack.

The node run command outputs the path to a log file where you can inspect the node’s output. For example:

cat /home/ubuntu/.peppy/logs/run/elegant-solomon-5423.log
[2026-02-17T10:15:23.599] Executing run_cmd: uv run hello_world (working_dir: /home/ubuntu/.peppy/instances/elegant-solomon-5423)
[2026-02-17T10:15:23.837] [stdout] hello world count 1
[2026-02-17T10:15:26.848] [stdout] hello world count 2
[2026-02-17T10:15:29.852] [stdout] hello world count 3

Chaining add, build, and run

Now that you’ve seen all three steps, peppy node add exposes flags that chain them for you:

• --build (-b) runs peppy node build immediately after the add succeeds.
• --run (-r) also spawns an instance once the build finishes, and implies --build.
• --sync (-s) runs peppy node sync before the add, so the snapshot picks up any edits you made to peppy.json5. Only valid for local filesystem sources; remote git/HTTP sources are synced server-side on fetch.

peppy node add -sb .    # sync, then add, then build
peppy node add -sr .    # sync, then add, then build, then run an instance

Tip

-sb and -sr are the shorthands you’ll reach for constantly during development. The typical edit-loop is: change peppy.json5 or source, hit peppy node add -sr ., watch the logs, iterate. Without -s, a stale .peppy/ bindings directory can make the daemon see an out-of-date interface; without -r, you have to follow every add and build with a separate peppy node run. Bundling them collapses the whole “I edited something, now run it” cycle to a single command.

Run-only options — trailing key=value arguments, --instance-id, and --bind KEY@VALUE — are only accepted alongside -r/--run; combining them with a plain peppy node add is rejected at parse time so an arg you cared about isn’t silently dropped. Keep key=value arguments after the source path (e.g. peppy node add -sr . frequency=30) so the source positional isn’t confused with the trailing args. --bind mirrors the peppy node run --bind flag and runs the same validator; see Bindings and routing.

Checking node status

View all added & running nodes:

peppy stack list

You should see something like:

$ peppy stack list
Node stack

┌─────────────────────────────────────────┬───────┬───────────┬─────────────────────────────────────────────┐
│ NODE                                    │ STAGE │ INSTANCES │ PATH                                        │
├─────────────────────────────────────────┼───────┼───────────┼─────────────────────────────────────────────┤
│ core-node-funny-chatterjee-6386:v0.10.0 │ Root  │ 1 running │ ~/.peppy/bin                                │
│ hello_world:v1                          │ Ready │ 1 running │ ~/.peppy/built_nodes/hello_world_v1.tar.zst │
└─────────────────────────────────────────┴───────┴───────────┴─────────────────────────────────────────────┘

Instance bindings

┌─────────────────────────────────────────┬─────────────────────────────────┬─────────┬─────────┬──────────┐
│ NODE                                    │ INSTANCE                        │ STATUS  │ HEALTH  │ BINDINGS │
├─────────────────────────────────────────┼─────────────────────────────────┼─────────┼─────────┼──────────┤
│ core-node-funny-chatterjee-6386:v0.10.0 │ core-node-funny-chatterjee-6386 │ running │ healthy │ (none)   │
├─────────────────────────────────────────┼─────────────────────────────────┼─────────┼─────────┼──────────┤
│ hello_world:v1                          │ suspicious-swanson-5880         │ running │ healthy │ (none)   │
└─────────────────────────────────────────┴─────────────────────────────────┴─────────┴─────────┴──────────┘

Dependencies
  (none)

The STAGE column reports each node’s stage, its artifact-level lifecycle:

• Added: the node is registered and a snapshot of the source has been taken, but build_cmd has not been run yet, so there is no runnable artifact.
• Building: a build is in progress. A second concurrent node build on the same entity is rejected until this one finishes.
• Ready: the build artifact is on disk. The node can be run; it may currently have zero, one, or several instances.
• Root: the synthetic core-node entity. Always present, and always reports exactly one running instance (the daemon itself).

The INSTANCES column aggregates per-instance state (starting, running) for the node, independently of its stage. To see individual instance IDs, use peppy node info.

Below the node table, the Instance bindings table expands the stack one level further: one row per instance, grouped by node, alongside the slot bindings that instance resolved when it started. A binding ties one of the consumer’s depends_on slots — identified by its link_id — to the producer instance(s) that feed it, rendered as link_id → producer. A from_any slot left unbound resolves to link_id → (any) (it accepts every producer of that slot), and an instance whose node declares no depends_on shows (none). Only nodes that currently have at least one instance appear in this table. See Bindings and routing for how slots are declared and how --bind pins them. In the listing above neither node declares a dependency, so every binding cell is (none).

Instance health and lifecycle

Beyond the bindings, the Instance bindings table reports two per-instance columns that follow an instance through its runtime lifecycle:

• STATUS is the instance’s lifecycle state. A freshly launched instance is starting while its process comes up and clears its startup health gate, then becomes running once it has started successfully. This is the same state peppy node info prints as [starting] / [running].
• HEALTH is the outcome of the core node’s most recent liveness probe against the instance: healthy or unhealthy. Every instance starts out healthy.

Once an instance is running, the daemon keeps probing its health in the background (every 5 seconds, allowing 3 seconds per probe) and updates the HEALTH flag from the result:

• A failed probe marks the instance unhealthy. This covers a crashed or wedged process, or one that has lost its messaging session.
• A later successful probe marks it healthy again.

A running instance is never removed from the stack because it is unhealthy. It stays listed, reported as unhealthy, until it either recovers on its own or you stop it explicitly with peppy node stop (one instance) or peppy node remove (the whole node). So a transient failure, such as a momentary loss of the messaging session, surfaces as a short unhealthy window rather than silently tearing instances out of your stack, and a process that has truly died stays visible so you can see that it died instead of wondering where it went.

Each health transition (a running instance going unhealthy, or recovering afterwards) is appended to the daemon’s stack log at ~/.peppy/stack_log.log, giving you a timestamped record of when an instance failed and when it came back.

Daemon shutdown and orphan prevention

Spawned nodes are child processes of the daemon, so peppy takes care to ensure that no node (or any process a node itself spawned) is ever left running after the daemon goes away.

Clean shutdown (Ctrl+C, systemctl stop, SIGTERM). Before it exits, the daemon tears down every spawned node: it asks each node to shut down cooperatively (in the node, this fires the cancellation token; see Graceful shutdown), waits a short shared shutdown grace period (shutdown_grace_secs, 3 seconds by default), then force-kills the process group of anything still alive. This is immediate: it does not wait the daemon-liveness grace period described below, so stopping the daemon feels instant while still guaranteeing nothing is orphaned. It’s the same graceful-then-forced sequence peppy node stop runs for a single instance, which uses the same shutdown_grace_secs window.

Unclean daemon death (crash, OOM, SIGKILL). When the daemon dies without the chance to run that cleanup, it can’t tear its nodes down. Instead, each spawned node runs a watchdog that listens for a periodic heartbeat from the daemon. If it sees no heartbeat for the daemon grace period, daemon_grace_secs (180 seconds / 3 minutes by default), the node shuts itself down rather than lingering as an orphan. The period is deliberately generous so a brief daemon blip or a quick restart doesn’t tear your nodes down: a node started in peer mode survives a daemon restart that completes within the window.

Both grace periods are configured in ~/.peppy/conf/peppy_config.json5 (see Daemon configuration for the full file reference) and take effect after a daemon restart:

peppy_config.json5

{
  lifecycle: {
    // Seconds a node waits without a daemon heartbeat before it self-terminates
    // to avoid orphaning. Default: 180 (3 minutes). Minimum: 30.
    daemon_grace_secs: 180,

    // Seconds a clean shutdown and `peppy node stop` wait for a node to exit
    // cooperatively before force-killing it. Raise it for nodes that need longer
    // to park actuators or release hardware. Default: 3. Minimum: 1.
    shutdown_grace_secs: 3,
  },
}

Caution

daemon_grace_secs must be at least 30 seconds (a smaller value is rejected when the config is loaded): it has to comfortably exceed the heartbeat interval and a router restart, or a momentary daemon hiccup would trip every node’s watchdog. shutdown_grace_secs must be at least 1 second.

Inspecting a node with peppy node info

When you need more than the one-line view from stack list, peppy node info dumps the node’s stage, its currently-tracked instances with their individual states and last-known health, and the paths to its add log and per-instance run logs. It takes a <node_name>:<node_tag> reference to a node that has already been added to the stack; it doesn’t touch the filesystem or any git/http source, so run peppy node add first if the node isn’t in the stack yet:

peppy node info hello_world:v1

[INFO] Getting node info for hello_world:v1...

Node Information
==================================================

Name:      hello_world
Tag:       v1
Language:  Python
Build cmd:   uv sync
Run cmd: uv run hello_world

Node Stack Status
--------------------------------------------------
Stage:     Ready
Instances: 2 tracked
           - suspicious-swanson-5880  [running]  healthy
           - flamboyant-penrose-9622  [running]  healthy

Logs
--------------------------------------------------
hello_world:v1
  Add log: /home/ubuntu/.peppy/logs/add/hello_world_v1_20260416_101124_613.log
  Run logs:
    - suspicious-swanson-5880: /home/ubuntu/.peppy/logs/run/suspicious-swanson-5880.log
    - flamboyant-penrose-9622: /home/ubuntu/.peppy/logs/run/flamboyant-penrose-9622.log

Exposed Interfaces
--------------------------------------------------
Emitted Topics:
  - message_stream (qos: SensorData)

Integrity
--------------------------------------------------
Config SHA256: 67505086f8ac099d0861cb5dfa539b1c8c3c9354ad90ed7713a5147c7cb43342

This is usually the fastest way to answer “is my node built yet?” and “where is the log for this specific instance?”. The Add log path is what peppy node add writes to; per-instance run logs are what peppy node run writes to. Build logs live under ~/.peppy/logs/build/ and are surfaced by peppy node build directly when it runs.

Exposed Interfaces summarizes the topics, services, and actions the node publishes or consumes per its peppy.json5. Config SHA256 is the fingerprint of the config that was fed through peppygen; it’s what peppy node add checks against to refuse a stale snapshot and tell you to run peppy node sync.

Stopping your node

When you’re done, you can stop your node:

peppy node stop <instance-id>

This stops one instance of the hello_world:v1 node, but the node itself still remains in the node stack.

Stopping is a graceful-then-forced operation, and the command blocks until the instance’s process has actually exited:

1. The daemon asks the instance to shut down cooperatively and gives it a grace period (shutdown_grace_secs, 3 seconds by default; see Daemon shutdown and orphan prevention) to stop cleanly. For a robot node this is its chance to park actuators, release hardware, and flush state. Inside the node, this ask arrives as the runtime’s cancellation token firing; see Graceful shutdown for how to handle it in Rust and Python.
2. If the instance doesn’t exit within that window, the daemon force-kills its whole process group (the node and any child processes it spawned), so nothing is left running.

If the instance had to be force-killed, peppy node stop warns you instead of reporting a clean stop:

WARN Node instance 'hello_world-...' did not shut down gracefully within the grace period and was force-killed

Caution

A force-kill is abrupt: the node gets no chance to run its own shutdown logic. A node that must stop hardware cleanly should handle the cooperative shutdown signal and exit promptly within the grace period. The same graceful-then-forced sequence runs for every node when the daemon itself shuts down; see Daemon shutdown and orphan prevention.

Removing your node

To remove the node from the node stack, run:

peppy node remove hello_world:v1

If you run peppy stack list again, you’ll see that only the core node remains:

$ peppy stack list
Node stack

┌─────────────────────────────────────────┬───────┬───────────┬──────────────┐
│ NODE                                    │ STAGE │ INSTANCES │ PATH         │
├─────────────────────────────────────────┼───────┼───────────┼──────────────┤
│ core-node-funny-chatterjee-6386:v0.10.0 │ Root  │ 1 running │ ~/.peppy/bin │
└─────────────────────────────────────────┴───────┴───────────┴──────────────┘

Instance bindings

┌─────────────────────────────────────────┬─────────────────────────────────┬─────────┬─────────┬──────────┐
│ NODE                                    │ INSTANCE                        │ STATUS  │ HEALTH  │ BINDINGS │
├─────────────────────────────────────────┼─────────────────────────────────┼─────────┼─────────┼──────────┤
│ core-node-funny-chatterjee-6386:v0.10.0 │ core-node-funny-chatterjee-6386 │ running │ healthy │ (none)   │
└─────────────────────────────────────────┴─────────────────────────────────┴─────────┴─────────┴──────────┘

Dependencies
  (none)

Next steps

Now that you’ve created your first node, you can:

• Add interfaces to communicate with other nodes
• Configure parameters for your node
• Learn about the node stack and dependencies