Core node functions

Every Peppy stack has a core node that tracks which nodes have been added, which instances are running, and a few facts about the host it is running on. peppylib exposes two helpers that let any node query the core node at runtime:

• info: a typed snapshot of the core node (uptime, hostname, node count, git version, container runtimes).
• stack.list: the live node graph, with the serving daemon’s hostname and every node’s owning core node, stage, and running instances.

info is a single top-level call; stack.list and the StackList type it returns live in the stack submodule. They are available from Rust as peppylib::info, peppylib::stack::list, and peppylib::stack::StackList, and from Python as from peppylib import info and from peppylib import stack (then stack.list, stack.StackList). They are not re-exported by peppygen; import them from peppylib directly.

Both take a NodeRunner, the same handle your node receives from NodeBuilder::new().run(...) (see the services guide for a typical setup).

A node also calls the daemon interfaces of peppy, such as the fleet listing and the leases of fleet:v1, through the daemon targets of its manifest, with the client that peppygen generates for them.

When to use these

• Runtime introspection of instances. A node can call stack.list to discover, at runtime, how many instances of a given dependency are currently up and what their instance IDs are. A router or load-balancer node, for example, can iterate graph.nodes, find the entry whose name matches its target, and read instances (keeping only those whose state is "running") to decide where to dispatch work.
• Health checks and dashboards. info returns uptime, hostname, node count, git version, and container runtime versions, enough to drive a status page or a liveness probe without scraping logs.
• Topology inspection. stack.list returns nodes and dependency edges as structured data, so tooling can filter, render, or analyze the graph without scraping CLI output.
• Tests and tooling. Integration tests or CLI utilities that need to assert “node X is up with N instances” or “edge A→B exists in the graph” can use the same helpers production code does.

info

info polls the core node’s info service and returns a typed response.

The response carries:

• uptime_secs: how long the core node has been up, in seconds.
• core_node_name, core_node_instance_id: identity of the core node.
• host_name: the machine hosting the core node.
• node_count: how many nodes are currently in the stack.
• git_version: the Peppy build the core node was compiled from.
• container_info.apptainer_version, container_info.lima_version: container runtime versions.
• messaging_port: the port the messaging layer is listening on.

The second argument is a timeout. Rust accepts anything that converts to Option<Duration>; Python accepts a float in seconds. Pass None (or omit it in Python) to use the default of 10 seconds.

Python

src/my_node/__main__.py

from peppygen import NodeBuilder, NodeRunner
from peppygen.parameters import Parameters
from peppylib import info


async def setup(_params: Parameters, node_runner: NodeRunner) -> None:
    response = await info(node_runner, 3.0)
    print(
        f"{response.core_node_name} on {response.host_name}: "
        f"{response.node_count} nodes, up {response.uptime_secs}s"
    )


def main():
    NodeBuilder().run(setup)


if __name__ == "__main__":
    main()

Rust

src/main.rs

use std::time::Duration;

use peppygen::{NodeBuilder, Parameters, Result};
use peppylib::info;

fn main() -> Result<()> {
    NodeBuilder::new().run(|_args: Parameters, node_runner| async move {
        let response = info(&node_runner, Duration::from_secs(3)).await?;
        println!(
            "{} on {}: {} nodes, up {}s",
            response.core_node_name,
            response.host_name,
            response.node_count,
            response.uptime_secs,
        );
        Ok(())
    })
}

stack.list

stack.list returns a StackList with seven fields:

• graph: the node graph, with every node’s metadata and its tracked instances.
• copies: the copies the serving daemon runs.
• copy_change: the join or the removal of a copy that holds the serving daemon’s stack now, with its action (join or remove), the copy’s name and its option; none while no such change runs. A copy that joins is in copies only once its join ends.
• shutdown_grace_secs: the serving daemon’s cooperative shutdown grace.
• core_node: the core-node name of the daemon that served this stack snapshot.
• instance_id: the serving daemon’s own generation instance id.
• host_name: the hostname of the daemon that served this stack snapshot.

Each node entry carries its required core_node owner plus its name, tag, config_path, optional artifact_path, stage (Added, Building, Ready, or Root), and its instances. Each instance has an instance_id, a state (starting, running, restarting while the daemon waits to start it again, or the terminal finished / failed), a healthy flag, its slot_bindings, and its pairing_slots. A router or load-balancer keeps only the running ones, which excludes both still-warming and already-exited instances. Its restart policy comes with it: restart, the policy as a launcher writes it ("on_failure", or { policy, max_restarts } with a cap); restart_origin, the level that set it (instance, flag, node, launcher_root or default); restart_count, the times a restarted process reached running; attempt, the count of restart attempts in a row that max_restarts caps (0 before the first exit the policy restarts; each such exit or failed start takes the next, and one after a run of two minutes or more takes attempt 1 again, where a run the health monitor ended counts only up to its last answered health check; at the cap it names the attempt refused; an exit peppy node remove asked for sets 0, and any other exit the policy does not restart leaves it as it was); last_exit, how the last start ended, a process’s exit or a start failure ({ code }, { signal: { number, name, core_dumped } }, { health_kill: { unanswered_secs } } or { message }, null before any exit); and running_since_ms and next_start_ms, milliseconds since the Unix epoch, null when there is none. The graph’s edges list the dependency relationships, with each edge pointing from one node entry to another. The list covers both direct depends_on.nodes dependencies and those resolved through contract implementation: a contract edge carries a via_contract of name:tag (the contract it was resolved through), while a direct edge has none.

Rust and Python expose the graph differently:

• In Rust, result.graph is a typed SerializedNodeGraph with nodes: Vec<SerializedNode> and edges: Vec<SerializedEdge>. Field access is direct (node.name, node.instances), and an instance’s restart fields sit on its restart_status (instance.restart_status.restart_count), whose policy reads through restart.policy() and restart.max_restarts().
• In Python, result.graph is a plain dict of the same shape: {"nodes": [...], "edges": [...]}. Access fields by key (node["name"], node["instances"][0]["state"]).

The example below uses stack.list to count the running instances of each node and print their IDs, a typical pattern for a router or load-balancer node that needs to dispatch work across instances of a dependency.

Python

src/my_node/__main__.py

from peppygen import NodeBuilder, NodeRunner
from peppygen.parameters import Parameters
from peppylib import stack


async def setup(_params: Parameters, node_runner: NodeRunner) -> None:
    result = await stack.list(node_runner, 3.0)

    print(f"served by {result.host_name}")

    for node in result.graph["nodes"]:
        running = [i for i in node["instances"] if i["state"] == "running"]
        print(
            f"{node['name']} on {node['core_node']} ({node['stage']}): "
            f"{len(running)} running instance(s)"
        )
        for instance in running:
            print(f"  - {instance['instance_id']}")


def main():
    NodeBuilder().run(setup)


if __name__ == "__main__":
    main()

Rust

src/main.rs

use std::time::Duration;

use peppygen::{NodeBuilder, Parameters, Result};
use peppylib::stack;

fn main() -> Result<()> {
    NodeBuilder::new().run(|_args: Parameters, node_runner| async move {
        let result = stack::list(&node_runner, Duration::from_secs(3)).await?;

        println!("served by {}", result.host_name);

        for node in &result.graph.nodes {
            println!(
                "{} on {} ({:?}): {} running instance(s)",
                node.name,
                node.core_node,
                node.stage,
                node.running_instance_ids().len(),
            );
            for instance_id in node.running_instance_ids() {
                println!("  - {instance_id}");
            }
        }
        Ok(())
    })
}

Looking up instances by (name, tag)

When you know which dependency you want, skip the manual iteration with running_instance_ids_by_node(name, tag), exposed on the StackList result in Python and on result.graph in Rust. The Rust signature is Result<Vec<&str>, NodeNotFound>; Python raises KeyError for the same condition. In both languages, an empty list (not an error) means the node is present but none of its instances is running: each is starting, restarting, or finished or failed for good. A router can back off while one starts or restarts, and read the states in the graph to tell a node that is coming back from one that is not.

Python

src/my_node/__main__.py

result = await stack.list(node_runner, 3.0)
try:
    ids = result.running_instance_ids_by_node("router_target", "v1")
except KeyError as e:
    print(e)
else:
    if ids:
        print(f"dispatch to {ids}")
    else:
        print("router_target:v1 is present but none of its instances is running")

Rust

src/main.rs

let result = stack::list(&node_runner, Duration::from_secs(3)).await?;
match result.graph.running_instance_ids_by_node("router_target", "v1") {
    Ok(ids) if !ids.is_empty() => println!("dispatch to {ids:?}"),
    Ok(_) => println!("router_target:v1 is present but none of its instances is running"),
    Err(err) => println!("{err}"),
}

Note

Both info and stack.list default to a 10-second timeout. Pass an explicit timeout when calling from a latency-sensitive path so a slow or unreachable core node does not stall your node.

Daemon targets

A node calls a daemon interface, an interface that the peppy daemon serves and no node implements, through a daemon target of its manifest. A daemon interface is a document in the contract format that is compiled into peppy. The interfaces peppy serves lists them. For example, fleet:v1 holds the fleet listing, the leases of the robots and the halt of the stack (see Robot leases and the halt).

Declaring a daemon target

manifest.depends_on.daemon lists the daemon targets of the node. Each entry names the interface by name and tag, and the slot of the target by link_id. The node lists the members that it calls in the consumes lists of its interfaces with that link_id, as it does for a contract slot. This lease holder lists the fleet, takes robots and holds a lease goal:

Python

lease_holder/peppy.json5

{
  peppy_schema: "node/v1",
  manifest: {
    name: "lease_holder",
    tag: "v1",
    depends_on: {
      daemon: [{ name: "fleet", tag: "v1", link_id: "fleet" }],
    },
  },
  interfaces: {
    services: {
      consumes: [
        { link_id: "fleet", name: "list" },
        { link_id: "fleet", name: "take" },
      ],
    },
    actions: {
      consumes: [{ link_id: "fleet", name: "lease" }],
    },
  },
  execution: {
    language: "python",
    build_cmd: ["uv", "sync"],
    run_cmd: ["uv", "run", "lease_holder"],
  },
}

Rust

lease_holder/peppy.json5

{
  peppy_schema: "node/v1",
  manifest: {
    name: "lease_holder",
    tag: "v1",
    depends_on: {
      daemon: [{ name: "fleet", tag: "v1", link_id: "fleet" }],
    },
  },
  interfaces: {
    services: {
      consumes: [
        { link_id: "fleet", name: "list" },
        { link_id: "fleet", name: "take" },
      ],
    },
    actions: {
      consumes: [{ link_id: "fleet", name: "lease" }],
    },
  },
  execution: {
    language: "rust",
    build_cmd: ["cargo", "build", "--release"],
    run_cmd: ["./target/release/lease_holder"],
  },
}

• Never pinned. An entry takes no sha256. The document of the interface ships with the peppy release that serves its tag, and peppy serves one tag of each interface.
• One producer. An entry takes no cardinality. The daemon that serves the interface is the one producer of the slot: the daemon of the coordinator of the launch that started the instance, or the daemon that runs an instance of no launch. The daemon binds the slot each time the instance starts, by a launch, a join, peppy node run or a restart, under the core node name and the per-boot instance id of that daemon. A launcher writes no links entry for a daemon target, and a launch refuses one.
• One target of each interface. A node names each interface once. A call names no target, so a call of an interface goes with the one scope of that interface that the instance has.
• One link_id namespace. The link_id of a daemon target shares one namespace with the other slots of depends_on and with implements.

The add and the sync of the node read the members of the interface from the document compiled into peppy, never from the contract cache. They refuse an entry that names an interface or a tag that this peppy does not serve, with the interfaces and the tags that it serves, and a consumed entry that names no member of the interface. The instance_documents service, which peppy node describe reads, gives the document of each daemon target of an instance after its pinned contract documents.

The client of a daemon target

The generator writes the client of each consumed member as for a contract slot, under the link_id of the target: consumed_services/<link_id>/<member> and consumed_actions/<link_id>/<member>. bound_producer() gives the daemon, and each call goes to the daemon as the provider of the interface. Each member gives the failure kinds of the interface that it lists, beside the kinds of peppy: take of fleet:v1 gives halted and leased.

Python

src/lease_holder/__main__.py

from peppygen.consumed_actions.fleet import lease
from peppygen.consumed_services.fleet import list as fleet_list
from peppygen.consumed_services.fleet import take


async def hold_leases(node_runner) -> None:
    daemon = fleet_list.bound_producer(node_runner)
    fleet = await fleet_list.poll(node_runner, daemon, 5.0)
    goal = await lease.ActionHandle.fire_goal(node_runner, daemon, 5.0)
    if goal.refusal is not None:
        print(f"the lease goal is refused: {goal.refusal.message}")
        return
    for robot in fleet.data.robots:
        try:
            await take.poll(node_runner, daemon, take.Request(robot=robot.name, name="a run"), 5.0)
        except take.Failure as failure:
            if failure.kind not in (take.FailureKind.LEASED, take.FailureKind.HALTED):
                raise
            print(failure.message)
    feedback = await goal.on_next_feedback_message()
    print(feedback.event, feedback.robot, feedback.cause)
    await goal.cancel_goal(5.0)

Rust

src/main.rs

use peppygen::consumed_actions::fleet::lease;
use peppygen::consumed_services::fleet::{list, take};
use std::time::Duration;

async fn hold_leases(node_runner: &peppygen::NodeRunner) -> peppygen::Result<()> {
    let daemon = list::bound_producer(node_runner);
    let fleet = list::poll(node_runner, daemon, Duration::from_secs(5)).await?;
    let mut goal = lease::ActionHandle::fire_goal(
        node_runner,
        daemon,
        Duration::from_secs(5),
        peppygen::DEFAULT_FEEDBACK_BOUND,
    )
    .await?;
    if let Some(refusal) = &goal.refusal {
        println!("the lease goal is refused: {}", refusal.message);
        return Ok(());
    }
    for robot in &fleet.data.robots {
        let request = take::Request::new(robot.name.clone(), "a run".to_owned());
        match take::poll(node_runner, daemon, Duration::from_secs(5), request).await {
            Ok(()) => println!("leased {}", robot.name),
            Err(error) => match take::Failure::of(&error) {
                Some(take::Failure {
                    kind: take::FailureKind::Leased | take::FailureKind::Halted,
                    message,
                }) => println!("{message}"),
                _ => return Err(error),
            },
        }
    }
    let feedback = goal.on_next_feedback_message().await?;
    println!("{} {} {}", feedback.event, feedback.robot, feedback.cause);
    goal.cancel_goal(Duration::from_secs(5)).await?;
    Ok(())
}

A long goal of an interface, such as changes and lease of fleet:v1, runs until its client cancels it. The handle of the goal keeps every feedback message until the client reads it, up to the feedback bound of fire_goal. A client that reads the feedback and falls behind by more messages than its bound cancels the goal, and its next read gives the loss as an error (see Receiving feedback).

Scoping a daemon target

The launcher gives each daemon target of an instance a scope in the daemon_scopes of the instance, keyed by the link_id of the target, or with a set_daemon_scopes adjustment on the instance. { vacant: "<why>" } leaves the target unserved. A fleet scope names the members of fleet:v1 that the client may call: members names at least one of list, contract, changes, lease, check_move, halt and clear_halt, each once, and lease brings take, release, offer, open and close.

peppy_launcher.json5

{
  peppy_schema: "launcher/v1",
  deployments: [
    {
      source: { name: "lease_holder", tag: "v1" },
      instances: [
        {
          instance_id: "lease_holder",
          daemon_scopes: { fleet: { members: ["list", "lease", "check_move"] } },
        },
      ],
    },
  ],
}

The launch, each join and peppy stack resolve hold each daemon target of a node to the checks of daemon scopes before anything starts. The instance runs on the coordinator, the machine of the daemon that runs the stack, outside every copy, so it declares no core_node, and the option of an axis that runs as copies does not deploy it. peppy node run of the node starts its instance with no scope: only a launcher gives scopes.