System clock

Every instance reads one clock for its whole lifetime, and its deployment decides which. By default that is its own machine’s OS wall clock; a launch can instead put it on a clock domain, a named timeline one instance supplies.

Wall time is fine on a single host. The trouble starts when two nodes need to compare timestamps. For example, matching a sensor reading to the controller command that triggered it, or merging logs from multiple machines into a single timeline. Their clocks drift independently, so “1.7 s ago” on one node no longer points to the same instant as “1.7 s ago” on another.

peppylib::clock exposes two helpers that let any node measure the offset between its local clock and the core node’s clock:

synchronize: a one-shot NTP-style request/response. Returns the offset and the observed round-trip delay.
subscribe: a long-lived subscription to the periodic clock topic the core node publishes. Each ClockTick carries the core node’s clock at emission.

They are available from Rust as peppylib::clock::synchronize and peppylib::clock::subscribe, and from Python as from peppylib.clock import synchronize, subscribe.

When to use each

	`synchronize`	`subscribe`
Pattern	Request/response	Topic subscription
Result	Offset and observed RTT	Snapshot of this instance’s clock
Staleness	Bounded by the RTT you just measured	~one one-way network delay (uncorrected)
Cost per use	One round trip	Passive (ticks arrive at 10 Hz)
Use when	You need a precise offset before stamping a recorded artifact	You want continuous correlation, e.g. driving a status display

If unsure, start with synchronize; it gives you both numbers and you can call it once at startup. Reach for subscribe when you need the steady drumbeat.

`synchronize`

synchronize performs the standard NTP 4-timestamp exchange against the core node’s clock service: the helper stamps t0 before sending, the core node stamps t1 on receive and t2 before responding, and the helper stamps t3 on receive. It returns a ClockSync with three fields:

offset_ns (i64 in Rust, int in Python): signed nanoseconds. local + offset_ns ≈ core_node. Negative means the local clock leads the core node.
round_trip_delay_ns (u64 in Rust, int in Python): the round-trip delay observed on this exchange.
raw: the wire response with server_recv_time (t1), server_send_time (t2), and the echoed client_send_time (t0), exposed for callers that want to do their own analysis.

Both sides of the exchange read OS wall time: the caller stamps t0/t3 from its own machine’s clock, and a daemon serves the clock service from its machine’s clock whatever its instances read. The measurement is therefore the same one in every domain, the offset between two machines’ wall clocks.

The second argument is a response timeout. Rust accepts Option<Duration>; Python accepts float | None in seconds. Pass None (or omit it in Python) to use the default of 10 seconds.

Python
Rust

import time

from peppygen import NodeBuilder, NodeRunner
from peppygen.parameters import Parameters
from peppylib.clock import synchronize


async def setup(_params: Parameters, node_runner: NodeRunner) -> None:
    sync = await synchronize(node_runner, response_timeout_secs=3.0)

    local_ns = time.time_ns()
    aligned_ns = local_ns + sync.offset_ns
    print(
        f"offset {sync.offset_ns} ns, RTT {sync.round_trip_delay_ns} ns; "
        f"local={local_ns}, aligned={aligned_ns}"
    )


def main():
    NodeBuilder().run(setup)


if __name__ == "__main__":
    main()

use std::time::{Duration, SystemTime, UNIX_EPOCH};

use peppygen::{NodeBuilder, Parameters, Result};
use peppylib::clock::synchronize;

fn main() -> Result<()> {
    NodeBuilder::new().run(|_args: Parameters, node_runner| async move {
        let sync = synchronize(&node_runner, Some(Duration::from_secs(3))).await?;

        let local_ns = SystemTime::now()
            .duration_since(UNIX_EPOCH)
            .expect("clock before unix epoch")
            .as_nanos() as i128;
        let aligned_ns = local_ns + sync.offset_ns as i128;

        println!(
            "offset {} ns, RTT {} ns; local={}, aligned={}",
            sync.offset_ns, sync.round_trip_delay_ns, local_ns, aligned_ns,
        );
        Ok(())
    })
}

`subscribe`

subscribe opens a long-lived subscription to the ticks of this instance’s own clock and returns a ClockSubscription. On wall time that is the bound core node’s clock topic; on a simulated domain it is that domain’s stream, whichever machine its publisher runs on. Call on_next_tick in a loop to receive each tick; it returns None when the subscription closes (typically because the producing side went away).

The core node publishes wall ticks at 10 Hz (every 100 ms) by default: high enough to correlate logs across nodes, low enough not to flood the bus. The rate is set on the core node side via CoreNodeArguments.clock_publish_interval. A simulated domain ticks whenever its publisher publishes.

Each ClockTick carries a single instant, read with time() in Rust and the time attribute in Python: nanoseconds on the clock it came from, stamped at emission. On wall time that is the Unix epoch; on a simulated domain it is whatever origin its publisher counts from. The value is never 0, which is the wire’s “no tick observed yet” sentinel. Because the tick is one-way, the value is stale by roughly one one-way network delay on receipt; there is no RTT correction. Use synchronize if you need bounded staleness.

Python
Rust

import asyncio

from peppygen import NodeBuilder, NodeRunner
from peppygen.parameters import Parameters
from peppylib.clock import subscribe


async def follow_clock(node_runner: NodeRunner) -> None:
    sub = await subscribe(node_runner)
    while True:
        tick = await sub.on_next_tick()
        if tick is None:
            # The producing side closed the subscription.
            return
        print(f"clock: {tick.time} ns")


async def setup(_params: Parameters, node_runner: NodeRunner) -> list[asyncio.Task]:
    return [asyncio.create_task(follow_clock(node_runner))]


def main():
    NodeBuilder().run(setup)


if __name__ == "__main__":
    main()

use peppygen::{NodeBuilder, Parameters, Result};
use peppylib::clock::subscribe;

fn main() -> Result<()> {
    NodeBuilder::new().run(|_args: Parameters, node_runner| async move {
        tokio::spawn(async move {
            let mut sub = subscribe(&node_runner)
                .await
                .expect("subscribe should succeed");

            while let Some(tick) = sub
                .on_next_tick()
                .await
                .expect("on_next_tick should not error")
            {
                println!("clock: {} ns", tick.time());
            }
            // Subscription closed; the producing side went away.
        });
        Ok(())
    })
}

Behavior notes

QoS profile is SensorData. Stale ticks are dropped on slow subscribers rather than back-pressuring the publisher: a clock value the network couldn’t deliver fresh is not worth delivering late.
synchronize is hardened against a misbehaving server. Offset and delay are computed in widened arithmetic and saturated when narrowed back to i64 / u64, so a peer returning extreme t1/t2 cannot wrap the result or flip its sign.
on_next_tick returning None is terminal. Treat it as “the producing side went away” and rebuild the subscription if you need to keep listening.
Default timeout for synchronize is 10 seconds. Pass an explicit timeout from latency-sensitive paths so a slow or unreachable core node does not stall your node.

Clock domains

A clock domain is one timeline with a name. wall is built in and is the time every machine already keeps. A simulated domain is supplied by one instance, which publishes the instants every other instance bound to it reads, so a simulator or a bag replay drives the same node binaries a physical robot drives.

One daemon hosts wall time and any number of simulated domains at once, and the launch decides which: running a simulation beside a physical robot is a matter of launching it.

Declaring a domain

A launcher, or a fragment it selects, names its domains in a root-level framework block:

// Two timelines in one launch: the simulation the robot runs in, and the
// wall clock the operator's panel reads. `peppy clock list` shows both once
// the launch is up.
{
  peppy_schema: "launcher/v1",

  // The clock domains this launch declares. `simulation` is supplied by one
  // instance, and naming that instance here is what makes it the publisher.
  // `operator_station` is a name for the time every machine already keeps.
  framework: {
    clocks: {
      simulation: { publisher: "simulation_inst" },
      operator_station: "wall",
    },
  },

  deployments: [
    {
      // The publisher carries no `framework.clock` of its own: the
      // declaration above already assigns it `simulation`.
      source: { name: "waldo", tag: "v1" },
      instances: [{ instance_id: "simulation_inst" }],
    },
    {
      source: { name: "openarm_sim_arm", tag: "v1" },
      instances: [
        {
          instance_id: "arm_inst",
          framework: { clock: "simulation" },
          links: { simulation: "simulation_inst" },
        },
      ],
    },
    {
      // Both ends of a binding read one clock, so the recorder that stamps
      // the arm's episodes runs on the simulation too.
      source: { name: "session_recorder", tag: "v1" },
      instances: [
        {
          instance_id: "recorder_inst",
          framework: { clock: "simulation" },
          links: { arm: "arm_inst" },
        },
      ],
    },
    {
      // The operator's panel reads wall time. It holds no clock-dependent
      // connection to the simulated instances.
      source: { name: "web_viewer", tag: "v1" },
      instances: [
        { instance_id: "viewer_inst", framework: { clock: "operator_station" } },
      ],
    },
  ],
}

A declaration takes one of two forms:

{ publisher: "<instance_id>" } is a simulated domain. Naming the instance is what assigns it the domain and the publisher role together, so the declaration is the one place that instance’s clock is written, and one instance supplies at most one domain. The publisher must be an instance the launch deploys.
"wall" is a name for the time every machine already keeps. Two such names are aliases of one timeline, which is how a launcher gives wall time a name that says what reads it.

wall itself cannot be declared: it is built in, and every instance can name it directly.

Binding an instance to a domain

Any instance in a deployments entry names the domain it reads:

{ instance_id: "backbone_inst", framework: { clock: "simulation" } },

An instance that omits framework.clock reads wall time. Naming a domain the launch does not declare is refused, naming the instance, the clock and the domains that do exist:

instance `backbone_inst` binds clock `sim`, which no `framework.clocks` entry declares. Declare it, or name one of `wall`, `operator_station`, `simulation`

An adjustment binds an instance a fragment deploys, written in the launcher under the option that selects the fragment, which is how one commander fragment serves a physical robot and a simulated one:

options: {
  openarm_sim: {
    fragments: ["openarm/fragments/openarm_sim.json5"],
    adjustments: [
      { target: "commander_inst", set_framework: { clock: "simulation" } },
    ],
  },
},

An instance the fragment deploys itself declares its own clock where it is written, so only the instances a launcher shares between a physical robot and a simulated one need this.

Composing declarations

A launch’s domains are the root launcher’s and every selected fragment’s, merged. Two documents may declare one domain, and identical declarations agree: a robot fragment and the simulation it runs in can both name the same timeline. Two different declarations of one name are refused, naming both documents. A fragment the selection leaves out contributes nothing.

Copies bind the domains the stack declares, naming them exactly as the stack does, and a copy’s adjustments write those bindings onto the instances it mints. A copy that also writes to a stack instance must write what already runs there, its clock included.

Reading the clock in a node

Hot-path code reads whichever clock the instance was bound to through the generated clock module: peppygen::clock in Rust, peppygen.clock in Python.

init(node_runner): async; a consumer opens its domain’s subscription here, so the first now_ns after a tick lands returns without setup latency. Idempotent per node: repeat calls for the same node are no-ops, and initializing a different node rebinds (how consecutive test-harness boots in one process each read their own clock). Call it once at the top of your setup function.
now_ns(): sync; reads the current time on this instance’s clock. A wall instance reads its machine’s OS clock. A consumer reads the last instant its domain published, and returns Err(PeppyError::ClockNotReady) (Rust) or raises RuntimeError (Python) until the first one lands. A publisher reads back the instant it last committed.

Python
Rust

from peppygen import NodeBuilder, NodeRunner, clock
from peppygen.parameters import Parameters


async def setup(_params: Parameters, node_runner: NodeRunner) -> None:
    # Idempotent. On a simulated domain this opens the subscription so
    # the background feeder is already running before `now_ns()` is
    # called below.
    await clock.init(node_runner)
    try:
        # A wall instance always returns its machine's OS time. A
        # consumer raises RuntimeError until its domain's first tick.
        now = clock.now_ns()
        print(f"now: {now} ns")
    except RuntimeError:
        # Reachable on a simulated domain before its first tick. Real
        # apps retry or wait on a tick.
        print("clock not ready yet")


def main():
    NodeBuilder().run(setup)


if __name__ == "__main__":
    main()

use peppygen::{NodeBuilder, NodeRunner, Parameters, Result};

fn main() -> Result<()> {
    NodeBuilder::new().run(|_args: Parameters, node_runner| async move {
        // Idempotent. On a simulated domain this opens the subscription
        // so the background feeder is already running before `now_ns()`
        // is called below.
        peppygen::clock::init(&node_runner).await?;
        match peppygen::clock::now_ns() {
            // A wall instance always lands here. A consumer reaches this
            // arm once its domain has published an instant.
            Ok(t) => println!("now: {t} ns"),
            // Reachable on a simulated domain before its first tick.
            // Real apps retry or wait on a tick.
            Err(e) => println!("clock not ready yet: {e}"),
        }
        Ok(())
    })
}

Prefer synchronize over the generated module when you need a bounded-staleness offset, since it observes a round trip and reports the RTT. subscribe is the right pick if you want every raw tick.

Publishing a domain

The instance a declaration names as publisher supplies its domain’s time. ClockPublisher::for_node returns Some (Rust) or the publisher (Python) exactly on that instance, and None everywhere else, so holding a publisher IS being one and a node that may or may not be a source branches on this one call. domain names the domain it supplies, and each publish commits the instant locally and sends one tick.

Python
Rust

import asyncio

from peppylib.clock import ClockPublisher


async def setup(_params, node_runner) -> None:
    # None unless this deployment named the instance a domain's
    # publisher. Build it once and keep the handle.
    engine.clock_publisher = await ClockPublisher.for_node(node_runner)
    if engine.clock_publisher is None:
        return
    print(f"supplying clock {engine.clock_publisher.domain}")


# Called from the engine's step callback. `publish` is async, so an engine
# running its physics on its own thread schedules it onto the node loop
# rather than awaiting it there.
def on_step(engine) -> None:
    if engine.clock_publisher is None:
        return
    loop.call_soon_threadsafe(
        lambda: asyncio.ensure_future(
            engine.clock_publisher.publish(engine.time_ns())
        )
    )

use peppylib::clock::ClockPublisher;

// None unless this deployment named the instance a domain's publisher.
// Build it once and keep the handle.
if let Some(publisher) = ClockPublisher::for_node(&node_runner).await? {
    println!("supplying clock {}", publisher.domain());
    // Then publish the engine's own clock from each step. Every instance
    // bound to the domain reads this instant, wherever it runs.
    loop {
        engine.step();
        publisher.publish(engine.time_ns()).await?;
    }
}

publish commits the instant before it sends it, so the publisher’s own now_ns() reports the instant it is publishing and can stamp the very data it emits for that step.

How often a publisher publishes is how finely its domain can be read: now_ns returns the last tick, so a publisher ticking at 100 Hz gives every reader a 10 ms grid. Publish on the cadence the readers compare against, not on the physics step, which is usually far finer than anything reads.

What makes two clocks the same

A domain is identified by three things: its name, the machine hosting its publisher, and an incarnation minted for each lifetime of that name. Written out, a domain reads simulation@cn-sim. Two domains are the same timeline only when all three agree, so a simulation on one machine and a simulation on another are two timelines, and relaunching under the same name is a third: anything still reading the previous lifetime keeps reading it, frozen at its last tick. You never type an incarnation; it is minted for you and shows up in peppy clock list --json.

Every connection reads one clock

Producer bindings, pairings and observations all carry timestamps one side stamps and the other interprets, so each of them joins two instances that read the same clock. Two aliases of wall time are one timeline and connect freely. Two simulated domains never are, however closely their instants happen to run.

A launch checks this before anything starts, and names both endpoints and both clocks:

`viewer_inst` reads clock wall and `arm_inst` reads simulation@cn-sim, so their binding `camera` would carry timestamps neither side can read on its own clock. Bind both instances to one domain, or give each domain its own instance of the node

peppy stack resolve runs the same check with no daemon, and a peppy node run that links a new instance to running ones is held to it too.

On the command line

peppy node run puts one instance on a clock, which is how a domain is driven outside a launcher:

# Start the simulation as a new domain's publisher.
peppy node run waldo:v1 -i simulation_inst --publish-clock simulation

# Start a consumer on it. `peppy clock list` shows what is running.
peppy node run openarm_sim_arm:v1 -i arm_inst --clock simulation

# Qualify the name when one name runs on several machines.
peppy node run openarm_sim_arm:v1 -i arm_inst --clock simulation@cn-sim

--clock takes wall, a short name, or a qualified name@core_node, and names a domain that is already running: an unknown name is refused, pointing at peppy clock list and at --publish-clock, and a short name that is ambiguous across machines is refused listing the qualified names to pick from. --publish-clock declares a domain and mints its lifetime on the daemon the command targets. The two flags conflict, and both are also accepted by peppy node add --run.

peppy clock list prints every domain the federation is running, gathered from the daemons that host them:

$ peppy clock list
┌───────────────────┬───────────┬─────────────────┬────────────────────────────────────┬───────┬───────────┐
│ CLOCK             │ SOURCE    │ PUBLISHER       │ OWNER                              │ READY │ CONSUMERS │
├───────────────────┼───────────┼─────────────────┼────────────────────────────────────┼───────┼───────────┤
│ wall              │ built-in  │ -               │ -                                  │ ready │ -         │
│ simulation@cn-sim │ simulated │ simulation_inst │ launch launch-brave-otter (cn-sim) │ ready │ 3         │
└───────────────────┴───────────┴─────────────────┴────────────────────────────────────┴───────┴───────────┘

OWNER is the launch that started the publisher, or peppy node run for a domain a command declared. READY reads waiting until the domain’s first tick reaches its host and ready afterwards, so a publisher that has yet to publish is visible beside the instance supplying it. CONSUMERS counts the instances reading the domain anywhere in the federation. --json prints the same listing with each domain’s name, core_node, incarnation, last_tick_ns and the instance@core_node of every consumer.

Two more commands report what each instance reads: peppy stack list carries a CLOCK column per instance, showing wall or the domain as name@core_node, and peppy stack resolve reports the clock of every instance of a plan before anything runs:

simulation_inst: publishes clock `simulation`
arm_inst: clock `simulation`
viewer_inst: clock `wall`

A launch names each domain and its publisher in its feedback as it starts:

instance `simulation_inst` publishes clock `simulation` (simulation@cn-sim)

Lifetime, replacement and removal

peppy stack launch mints a fresh lifetime for every domain it declares, so relaunching a stack under the same names produces new timelines, exactly as it produces new instances. peppy stack join follows the domains the stack is already publishing, and the machines supplying the clocks a copy reads take part in the join.

peppy stack remove takes a domain’s lifetime out with the instance that supplied it, and refuses while anything still reads it:

`alpha` supplies clock `simulation` to copies `bravo`; remove them first

`alpha` supplies clock `simulation` to `backbone_inst` and to copies `bravo`; run peppy stack reset, then launch without `alpha`: drop its deployments entry from the launcher, or leave it out of the copies you join

Stopping a publisher freezes its domain: the instances bound to it keep reading the last instant it committed. A domain is listed for as long as the instance supplying it is live, so a publisher that stops or crashes takes its domain out of peppy clock list while the instances bound to it hold that instant. Freeing the name is what lets a replacement publisher declare it again.

Domains across machines

A publisher and the instances reading it may sit on different machines. The domain’s ticks ride the clock topic of the machine hosting the publisher, under a key carrying the domain’s name and incarnation, and a consumer addresses that whole identity, so it reads the same timeline wherever the launch places it. Placement decides one thing about a domain: the machine in its identity, which is where its publisher landed. See Federation.