Observatory (chirpscope)

Normative — synced from the repo

This page mirrors docs/OBSERVATORY.md verbatim. In the lrc repo the docs are the spec: they move in the same commit as the behavior they describe.

chirpscope is meshtrack’s job (live command-and-monitoring for a mesh) reborn as a web app fed by chIRpChat’s own plumbing. This page is normative for the Wave 1 slice: the lrcd SSE event feed and its JSON schema. The full view set (topology graph, packet inspector, lanes/airtime dashboard, CHANSYNC/boot-id timeline, identity browser, federation/RTT view, OTA rollout view, trace replay) is Wave 3 — this wave ships the feed, a live raw event log with type filters, and labeled empty routes for those future views. See ROADMAP.md §WS-SCOPE for the program context.

Stance: no geolocation on the wire, ever. Node positions are operator-side local config only (project charter). No event, counter, or packet field defined here or anywhere in chIRpChat carries a coordinate. Any future “map view” places nodes from a local config file the operator maintains client-side, never from network state.

Feed endpoint

GET http://<host>:8463/

Default port 8463 (--sse-port, adjacent to the existing Prometheus /metrics on 8462). Server-Sent Events (SSE): a long-lived GET, response headers Content-Type: text/event-stream, then one event per data: <json>\n\n frame, indefinitely, until the client disconnects. No path or query parsing — any GET opens the same firehose (filtering happens client-side; see “Design decisions” below for why).

$ curl -N http://10.10.10.11:8463/
retry: 2000

data: {"v":1,"seq":1,"mono_ms":408800598,"kind":"registered","uid":"b158d1fdcc6391b4"}

data: {"v":1,"seq":2,"mono_ms":408800599,"kind":"packet_tx","type":"CHANCTL","flags":0,"ttl":7,"hops":0,"msgid":2268557940,"src":"b158d1fdcc6391b4","dst":null,"scope":{"chan":"9f86d081","rtr":9},"seq24":16777215,"payload_len":0,"via_tcp":false}

Event schema

Every event is a single-line JSON object. v is the schema version (1 today; bump on any breaking field change, additive fields do not bump it). seq is a per-process monotonic counter assigned at broadcast time — it is not the LPP msgid; it exists so a client can detect its own bounded buffer having dropped frames (a gap in seq on the wire is invisible to the client by construction, but sse.overflow_drop on /metrics reports it server-side). mono_ms is the daemon’s Clock::monotonic_ms() at the moment of observation, not wall-clock/unix time — this repo’s health data deliberately avoids depending on a disciplined clock (TELEMETRY.md privacy section mirrors this).

Envelope (every event)

Field	Type	Meaning
v	int	Schema version, currently 1
seq	int	Per-process monotonic sequence, assigned at broadcast
mono_ms	int	Monotonic observation timestamp
clock_kind	string	"mono" on this live feed; lrcsim traces carry "virtual" (LRCSIM.md §6). The replay/live discriminator: a viewer scrubs each stream against its own clock base and must never conflate the two
kind	string	One of the kinds below

kind: "packet_rx" \| "packet_tx" \| "packet_relay"

The LPP header summary only — never the decoded payload text. This mirrors TELEMETRY.md’s “no message text” rule: the observatory is a network health tool, not a message reader.

Field	Type	Meaning
type	string	LPP packet type name (CHANMSG, DM, BEACON, HELLO, WELCOME, BYE, ROTATE, CHECKPOINT, ADMIN, TELEM, RTRSYNC, LANEPROBE, MAILBOXSYNC, CHANCTL, CHANSYNC, ACK, PING, KEYREQ, KEYRESP, DCCCTL, DCCDATA, LANEGRANT, RACK)
flags	int	Raw LPP flags byte (packet.h flags::* bits)
ttl	int	v1 only. Time-to-live at observation; 0 for a v2 (has_erb) packet — v2 carries no wire hop counter at all (CREDIT_ROUTING.md deletes it rather than relocating it)
hops	int	v1 only. Hop count at observation; 0 for a v2 packet
has_erb	bool	True for a v2 packet (Edge Routing Block present); selects whether ttl/hops or credit/debit_units are meaningful for this event
credit	int	v2 only (Wave 2 AC wire event, CREDIT.md). The frame’s airtime-credit budget (kCreditUnitMs-sized units) on arrival at this node; 0 for a v1 packet
debit_units	int	v2 packet_relay only. The airtime debit this hop actually charged (credit_forward()’s CreditForwardResult::debit_units) when this node forwarded the frame; 0 for packet_rx/packet_tx events and for v1 packets
msgid	int	LPP MSGID (wraps; not a stable identifier across long time spans)
src	hex string	Sender UID, 16 hex chars (8 bytes)
dst	hex string | null	Recipient UID — present for group B (unicast) types, null otherwise
scope	object | null	{"chan": hex4, "rtr": int} — present for group A (channel) types, null otherwise
seq24	int | null	24-bit channel sequence number — present for group A types, null otherwise. 16777215 (0xFFFFFF) is the documented “not yet sequenced” sentinel (PROTOCOL.md), not an error
payload_len	int	Payload byte length (not the payload itself)
via_tcp	bool	Whether this copy is/was carried over TCP federation vs. RF
trail	hex string | null	v2 only, UNVERIFIED. The ERB TRAIL hop bytes as observed (one byte per hop — a UID first-byte, ambiguous by construction). The security contract demoted TRAIL to a diagnostic hint: it feeds no routing or trust decision anywhere in the system, and every UI surface must present it as unverified (chirpscope badges it). null when absent

Schema note (Wave 2, AC replaces TH): this is an additive rename, not a breaking change — v stays 1. Existing consumers that only read ttl/hops see unchanged behavior on v1 traffic and simply see 0 for those fields on v2 traffic instead of a meaningful TTL. New consumers (chirpscope’s packet inspector, lrcsim’s credit_debit trace record) read has_erb first to pick the correct field group. debit_units is populated only on a packet_relay event for an admitted v2 frame; it shares this one Node::Config::on_event seam with the live SSE feed rather than either side re-decoding the AC byte off the wire independently — a relay’s debit is computed once, in Node::handle_frame(), and observed once.

packet_rx fires once per accepted (post-dedup, post-signature-check) inbound frame, from either a peer link or a radio lane. packet_tx fires once per frame this node originates and sends (Node::send_frame / send_signed_frame). packet_relay fires when this node forwards someone else’s frame onward — TTL-guarded for v1 (path relay) or credit-guarded for v2 (credit_forward()’s gradient-descent + airtime-debit admission, CREDIT.md). A single wire packet can appear as packet_rx here and packet_relay a moment later if this node relays it — the two events share src/msgid but not seq (the feed’s own counter) or mono_ms.

Known Wave 1 gap: the fragmented target-aware unicast path used for large MAILBOXSYNC snapshots (Node::send_mailbox_sync_body’s fragment loop) does not yet fire packet_tx — only the two primary send_frame/send_signed_frame choke points are instrumented, which cover every other packet type. Extending coverage to that path is a small, isolated Wave 3 follow-up.

kind: "registered" \| "deregistered"

Local IRC session registration/deregistration — the closest analogue this codebase has to “HELLO/WELCOME” in user-facing terms (registration is an IRC-session concept here, not a wire packet type; HELLO/WELCOME on the wire are the router-registration handshake and already show up as packet_rx/packet_tx events with type: "HELLO" / type: "WELCOME").

Field	Type	Meaning
uid	hex string	The identity’s UID

registered fires from Node::session_registered (after mailbox delivery and join-restore). deregistered fires from Node::session_gone, after the node has already queued its outbound BYE packet (so you will see the BYE packet_tx event immediately before deregistered).

kind: "router_liveness"

A federation reachability transition — TCP link up/down or a RTRSYNC liveness record changing this node’s local view of a peer router. Fires only on actual state change (Node::apply_router_liveness’s changed flag), not on every liveness refresh, so a healthy steady-state link is silent here (see fed.liveness on /metrics for the raw refresh count).

Field	Type	Meaning
rtr	int	The router id whose reachability changed
reachable	bool	New reachability state
reason	string	Local diagnostic text (e.g. "tcp link down", "tcp link up") — operator-facing, not wire data

kind: "counters"

A periodic full counter-registry snapshot — the browser-reachable sibling of GET /metrics, multiplexed onto the same SSE connection so chirpscope never needs a second (CORS-exposed) endpoint. Emitted every --sse-counters-interval-ms (default 5000, 0 disables) while at least one SSE client is connected; an idle daemon pays nothing. The lanes/airtime compliance view charts region_*/tx_duty_defer, the timeline charts seq_gap/seq_backfill_*/seq_boot_flush, and the OTA rollout view charts the ota_* set, all as deltas between consecutive snapshots.

Field	Type	Meaning
counters	object	Every registry counter by its stable TELEMETRY.md name, e.g. {"rx_ok": 123, "ota_committed": 1, ...}

kind: "airtime"

A periodic per-node duty/compliance snapshot riding the counters cadence — the data source for the lanes dashboard’s remaining_duty_bp gauge. Values come from Node::region_tx_context() + region_remaining_duty_bp(); a radio-less lrcd truthfully reports enforcement-disabled full headroom, while an embedding that calls set_region_tx_context() (RF HAT, firmware profile) reports real numbers.

Field	Type	Meaning
region	string | null	Active region name; null = enforcement disabled on this node
freq_khz	int	Current TX carrier (resolves the sub-band)
preset	int	Airtime-currency preset
lane_index	int	AirtimeLedger lane key
duty_bp_cap	int	The resolved sub-band’s legal duty cap, basis points
remaining_duty_bp	int	Legal headroom remaining right now, basis points

Not yet events (lane occupancy)

The Wave 1 mission scoped lane occupancy changes into the feed. In this codebase, per-lane occupancy (lane.<n>.occupancy_pct etc.) is documented in TELEMETRY.md’s counter registry as a planned per-lane breakdown but is not yet wired to the flat counter registry or exported anywhere today — there is no existing “lane occupancy changed” call site to tap, discrete or polled. Rather than invent a synthetic threshold-crossing event ahead of the underlying metric existing, Wave 1 leaves lane occupancy out of the feed. Once WS-REGION/WS-SPECTRUM land the per-sub-band AirtimeLedger work the lanes/airtime dashboard view needs anyway, adding either a polled /metrics gauge or a discrete lane_occupancy SSE event kind (or both) is a well-scoped Wave 3 follow-up that can reuse this feed’s plumbing (Node::Config::on_event) without protocol changes.

Reconciling with lrcsim JSONL traces — the settled contract

One viewer, two sources: this live feed and lrcsim’s --trace JSONL output (LRCSIM.md §6) share the “one type-tagged JSON object per line” shape, and chirpscope’s replay view drives the same views from either. The reconciliation questions Wave 1 left open are settled as follows:

• clock_kind discriminator. Both streams carry mono_ms + clock_kind in the envelope: "mono" here (process-monotonic), "virtual" in traces (the scenario’s seeded virtual clock). The viewer keys its scrubber base on this field.
• seq. Live-feed-only (broadcast drop detection). Trace lines don’t carry it; the viewer treats it as optional.
• Kind namespace. Shared flat namespace. Kinds observable on a live daemon use bare names (packet_rx, packet_tx, packet_relay, registered, deregistered, router_liveness, counters, and the reserved shared lane); sim-only observations are sim_-prefixed (sim_drop, sim_node_state, sim_assert) and can never collide with a future live kind.
• Null-not-omitted. Both sides emit dst/scope/seq24 as explicit null when the packet’s type group doesn’t carry them.
• Packet type naming — known residual mismatch. This feed emits uppercase names ("CHANMSG"); traces emit the C++ enumerator verbatim ("ChanMsg"). Both are frozen surfaces for their own consumers (sweep tooling reads traces; this doc’s tables pin the feed), so the viewer normalizes case-insensitively (web/src/model/records.ts). If either side ever takes a breaking schema bump, aligning the case should ride along.
• Sim-only packet fields. Trace packet records additionally carry preset/freq_slot/snr_x10 (rx) and x/y scenario geometry. The viewer uses them when present (SNR-weighted topology edges, replay positioning); they never appear on the live feed — a real network carries no positions (charter), scenario geometry is synthetic simulation input.

Design decisions

SSE, not WebSocket. The feed is unidirectional — the browser never sends anything back on this connection — so a full-duplex protocol buys nothing here. SSE framing is two lines of text (data: <json>\n\n) trivially hand-rolled inside the existing poll()-loop Conn/outbuf model (see daemon/sse_feed.cpp’s sse_frame/sse_preamble), reusing the exact accept/read/write/close machinery already written for /metrics. A spec-compliant WebSocket needs a handshake (Sec-WebSocket-Accept computation, SHA-1 + base64 — a new dependency surface or a hand-rolled crypto util this repo doesn’t otherwise need), frame masking on the client-to-server direction, and an opcode state machine — real complexity for a direction this feed never uses. SSE also rides plain HTTP/1.1 keep-alive (works through any proxy/load balancer that already passes /metrics) and gives reconnection semantics for free via the retry: hint. The tradeoffs (server-to-client only, text-only, no binary framing) are both non-issues for a JSON event log. If a future wave needs true bidirectional control (e.g. live filter push-down to the daemon to cut bandwidth), that argues for a second, purpose-built endpoint rather than retrofitting this one.

Zero new daemon dependencies. daemon/sse_feed.h/.cc hand-roll JSON string building and SSE framing with the standard library only (AGENTS.md rule 6). This is a deliberate, bounded scope: fixed-shape objects with a known field set, not general JSON — a real JSON library would be overkill for emitting values whose shapes are pinned by tests/test_sse_feed.cpp.

Bounded per-client buffer, drop-and-count, never block. A single slow SSE client (a laptop that went to sleep with a browser tab open, a client on a congested link) must never stall the shared single-threaded poll() loop that also carries live IRC and federation traffic — this is the same non-negotiable as every other listener in lrcd. Each SseClient connection gets an SseClientBuffer (default cap 256 framed events, --sse-buffer-cap to override) that push()es at broadcast time and drain()s into the connection’s outbuf once the previous batch has fully written. Past capacity, push() returns false and the event is dropped for that client only (not the encode work, which is shared and happens once regardless of client count) — sse.overflow_drop counts it. The alternative (grow the buffer unbounded) risks unbounded daemon memory from one stuck browser tab; blocking the writer risks stalling every other listener on the same poll() loop. Drop-and-count is the same trade this codebase already makes for the dedup ring and the relay store-and-forward queue (path.store_drop).

Encode once, fan out to many. broadcast_node_event checks whether any SSE client is connected before paying the JSON-encoding cost at all, and encodes each event exactly once regardless of client count — the framed string is push()ed by reference-copy into each client’s buffer, not re-encoded per client.

Counters

See TELEMETRY.md’s counter registry: sse.clients, sse.events_sent, sse.overflow_drop.

The SPA (chirpscope views)

web/ is a standalone Vite + TypeScript + Preact project (its own toolchain, exempt from the no-new-deps rule per AGENTS.md rule 6; dependency rationale in the Wave 3 report and web/package.json). Wave 3 replaces the stub routes with working views, all driven by one record stream that is either the live SSE feed or a loaded lrcsim JSONL trace (clock_kind selects scrubber behavior):

• Event log — raw record stream with kind/type filters (Wave 1, retained).
• Topology — force-directed graph built incrementally from packet/ liveness records; SNR-weighted directed edges from trace snr_x10 (asymmetry renders as one-way edges); trace x/y geometry or an operator-local layout (localStorage + JSON import/export, never leaves the browser — charter) pins positions. ERB TRAIL overlay renders from an optional trail field flagged UNVERIFIED (the security contract demoted TRAIL to a diagnostic hint); neither source emits it yet.
• Packet inspector — filterable live LPP header decode; v1 (ttl/ hops) and v2 (has_erb/credit/debit_units) frames visually distinct.
• Lanes / airtime — compliance counters (region_*, tx_duty_defer, credit_reorig_lbt_limited) charted from counters snapshots; per-lane activity and sim_drop refusal reasons in replay.
• Timeline — swim-lanes for registration churn, router liveness, gap heal (seq_* counter deltas), and the OTA rollout funnel (ota_*).
• Replay — load a .jsonl trace (local file, nothing uploaded anywhere), time-scrub with play/pause/speed against the virtual clock.

npm ci && npm run build produces the static bundle; embedding it into lrcd as a served asset (so the daemon needs no separate web server in the field) remains open, tracked in ROADMAP.md.