Telemetry & field debugging

Design rule: every behavior the protocol exhibits must be observable from an IRC client. When this network is popular, the people debugging it will be on a hill with a phone, not at a desk with a JTAG probe.

Counter registry

core/include/lrc/telemetry.h defines a flat registry of u32 counters and gauges, grouped by subsystem, each with a stable short name:

rx.ok rx.crc rx.dedup rx.badver rx.badtag rx.badsig …
tx.ok tx.cad_busy tx.duty_defer tx.retry tx.fail tx.via_tcp_suppress tx.queue_drop …
region.unset_refusal region.dwell_clamped region.lbt_deferred region.lbt_admit region.out_of_band the region-compliance enforcement layer (core/include/lrc/region.h; docs/RADIO.md §Regulatory regions, “Enforcement”). All five are live at the two enforcement gates — Node::radio_tx_admitted() (core/src/node.cpp, every radio-bound frame) and the firmware radio loop’s region_tx_admitted() — plus firmware’s tx on console handler for the first: region.unset_refusal counts tx on requests refused because no region is configured (the region-before-tx-on gate); region.dwell_clamped counts TX refused for exceeding the region’s per-transmission dwell cap (US915/AU915/AS923 400 ms); region.lbt_deferred/region.lbt_admit count LbtResult outcomes in LBT-mandatory regions (Japan/Korea/China) — a persistent lbt_deferred-heavy ratio is the signal a caller should couple other budgets (e.g. credit re-origination) to instead of duty headroom there, since duty is not the limiting factor in an LBT-governed region; and region.out_of_band counts TX refused because the current frequency falls outside every sub-band the selected region defines. Duty refusals at the same gates count as the pre-existing tx.duty_defer.
region.mixed_beacon_ignored counts a heard BEACON whose region_id did not match this node’s own configured region (PROTOCOL.md §BEACON region id, beacon_region_accepted() in lanes.h) and was therefore ignored rather than merged into the accepted lane plan — the mixed-region fail-safe. A sustained nonzero rate at one site names a real regulatory border or a misconfigured neighbor router, not routine noise.
laneprobe.tx lane.retune lane.retune_fail lane.grant_tx lane.grant_rx router scheduler solicitations, firmware radio lane switches, and anchor-offload promotion grants sent/received …
lane.<n>.rx lane.<n>.tx lane.<n>.occupancy_pct lane.<n>.snr_avg …
path.discover path.fail path.repin …
seq.gap seq.backfill_req seq.backfill_served seq.boot_flush …
time.skew_drop history rows whose nonzero origin timestamp was hidden from IRC server-time because local time was undisciplined or outside tolerance …
telem.rx accepted neighbor rollups retained for /msg *lrc routers …
reg.join_restore joined channels restored for a local identity after IRC re-registration …
key.req key.resp key.cache key.reject key.rotate key.sign key.forget key.revoke key.unrevoke key.revoked_drop identity record lookup, response, cache acceptance, invalid/conflicting records, accepted signed CTK rotations, and local trust removal …
chkpt.tx chkpt.rx chkpt.mismatch chkpt.archive checkpoint emission, verified windows, signed hash mismatches, and local archive rows …
crash.report crash.clear retained postmortems printed to an operator and then cleared …
path.discover path.repin path.fail path.store path.forward path.store_drop path discovery, pinning, retry failure, and bounded relay store-and-forward …
mailbox.store mailbox.deliver mailbox.expire mailbox.drop mailbox.dedupe mailbox.tombstone mailbox.import mailbox.sync_tx mailbox.sync_rx mailbox.sync_drop mailbox.repair_tx mailbox.repair_rx mailbox.repair_drop …
pwr.wake.dio1 pwr.wake.timer pwr.sleep_ms …
store.log_flush store.kv_write store.reclaim store.horizon_s …
irc.sessions irc.lines irc.unknown_cmd irc.flood_drop …
chan.attest_drop chan.sig_drop …
chan.rx_seen chan.rx_unreach chan.rx_unstamped chan.rx_noview chan.rx_dup chan.rx_deliver the received-channel-message delivery funnel: every inbound ChanMsg that reaches handling, then the reasons it is dropped (router scope not reachable, unsequenced-and-not-ours-to-sequence, no matching local channel view, or a seq the local mirror already accepted) versus delivered to local IRC sessions. The gap between rx_seen and rx_deliver localizes why on-device chat is or isn’t surfacing without a debugger …
chan.tx_unstamped chan.seq_client the two halves of client-originated sequencing (PROTOCOL.md §Group A: “0xFFFFFF from a client means ‘not yet sequenced’”): a node that doesn’t sequence a scope (Config::sequences_scope, node.h) counts each unstamped channel message it originates in chan.tx_unstamped; the sequencing router counts each client message it stamps into its authoritative stream and rebroadcasts in chan.seq_client. A rising tx_unstamped on a client with a flat seq_client on its router (or receivers’ chan.rx_unstamped rising instead) localizes “my messages vanish” to the sequencing hop …
fed.links fed.sync fed.sync_delta fed.directory fed.directory_drop fed.record fed.tombstone fed.liveness fed.revoke fed.gossip_defer fed.auth_fail fed.rtt_sample fed.link_down …
sse.clients sse.events_sent sse.overflow_drop chirpscope event feed connects, framed events sent, and bounded-buffer drops …
ota.manifest_verified ota.manifest_rejected ota.staging_started ota.chunks_staged ota.resume_loaded ota.resume_persisted ota.hash_fail ota.staged_verified ota.armed ota.apply_recorded ota.health_pass ota.health_fail ota.committed ota.rolled_back ota.aborted ota.refused ota.offered ota.accepted ota.wave_promoted the OTA staging state machine (core/include/lrc/otastage.h, OTA.md) — every state transition and refusal, one counter each, so a rollout is watchable end-to-end from /metrics/stats without new plumbing …
wizard.started wizard.step_advanced wizard.step_invalid wizard.completed wizard.skipped the first-run wizard (core/include/lrc/wizard.h, USERGUIDE.md “First-run wizard”) — one node running through the flow bumps started once, step_advanced/step_invalid once per step() call depending on the result, and exactly one of completed/skipped when the flow ends, so onboarding funnel drop-off is visible without a survey.
fed_peering.pin_redeemed guided federation peering (core/include/lrc/fed_peering.h, USERGUIDE.md “Pinned lrcd peers and guided peering”) counts a --peering-pin-token that decoded successfully and let lrcd continue into its normal run loop. Both --print-peering-pin (pin generation) and a refused --peering-pin-token exit the process immediately (before any metrics listener starts) — there is no running daemon whose /metrics could ever report those two outcomes, so they intentionally have no counter; the printed pin/ fingerprint and the refusal reason string on stderr are each already the full observable signal for a one-shot CLI invocation.
whois.fingerprint_shown counts the phonetic-hex fingerprint numeric (WHOIS’s second 320 line, USERGUIDE.md “Verifying who you’re talking to”) actually rendering for a real UID, as opposed to being silently omitted for a non-standard extra line.

Counters are cheap (++ on a static array), never optional, and the registry is iterable — which is what makes all the surfaces below free. rx.badtag counts failed Tag8 checks, including missing or invalid chat tags from senders whose identity record is already cached. rx.badsig counts DMs dropped by the local require-sig policy because the frame did not carry a valid full signature from the cached sender identity. fed.links counts TCP peer link-up events, including accepted inbound links and successful outbound reconnects, after pinned peer authentication succeeds or legacy-compatible unpinned setup is accepted. fed.sync counts RTRSYNC frames emitted by the node. fed.sync_delta counts compact RTRSYNC frames that carry only the stream vector plus TTL-retained changed optional rows between full snapshots. fed.liveness counts applied router reachability transitions from local link state or RTRSYNC liveness records. fed.revoke counts newly applied trusted-peer RTRSYNC revoke-state rows, including both denylist rows and clear tombstones. fed.gossip_defer counts ready peer routers intentionally skipped by daemon bounded-fanout RTRSYNC anti-entropy passes; non-gossip traffic still uses the best ready link to every directly connected peer router. fed.auth_fail counts rejected TCP peer authentication attempts, unauthenticated legacy frames, and bad peer-frame MAC/sequence checks. fed.rtt_sample counts successful TCP peer keepalive RTT measurements. fed.link_down counts the moment the last ready TCP link to a peer router goes down and the router is marked unreachable locally. fed.directory_drop counts local evictions of remote online directory attachments when the node’s configured directory cap is exceeded. It is local memory pressure, not a BYE, tombstone, or router-liveness transition; it may still render local IRC presence cleanup for clients that had seen the attachment in a channel. telem.rx counts accepted remote TELEM rollups retained by the node. telem.aggregate counts accepted rollups folded into local aggregate buckets. mailbox.store counts offline DMs accepted into the recipient router’s local mailbox, mailbox.deliver counts reconnect flushes to IRC sessions, mailbox.expire counts TTL purges, and mailbox.drop counts oldest-record eviction when a per-UID mailbox limit is hit. mailbox.dedupe counts duplicate loaded mailbox snapshot rows dropped by (SRC, MSGID) before delivery. mailbox.tombstone counts live mailbox rows converted to deletion tombstones after delivery, local expiry, or tombstone-dominant snapshot normalization. mailbox.import counts externally supplied mailbox snapshots that changed the local merged mailbox set; replaying an already-covered snapshot does not move it. mailbox.sync_tx counts standby mailbox LCMB snapshots or row subsets emitted over target-aware authenticated peer links, mailbox.sync_rx counts accepted MAILBOXSYNC imports, and mailbox.sync_drop counts radio-ingress, malformed, unknown-recipient, encode, or over-fragment-limit drops. mailbox.repair_tx counts compact mailbox inventory and repair-request controls emitted over target-aware authenticated peer links. mailbox.repair_rx counts accepted inventory and request controls, and mailbox.repair_drop counts malformed, unknown-recipient, encode, or over-fragment-limit repair-control drops. reg.join_restore counts durable local channel memberships restored when an IRC client re-registers after a client or daemon restart. key.req counts local KEYREQ packets answered for an identity this node owns, key.resp counts KEYRESP records sent, key.cache counts verified identity records accepted into or restored into the bounded cache, and key.reject counts malformed records, same-epoch pubkey/CTK conflicts, and invalid/stale rotations. key.rotate counts signed higher-epoch CTK rotations emitted locally or accepted into the cache. key.sign counts local DMs emitted under the sign-dm full-signature policy. key.forget counts local cached-peer identity records or signature policies removed by key forget or as part of key revoke; key.revoke and key.unrevoke count generationed denylist state changes from local key commands, signed Admin identity revoke keys, or accepted RTRSYNC revoke-state rows. key.revoked_drop counts cache refreshes, inbound frames, and outbound DMs suppressed because the peer UID is locally revoked. chkpt.tx counts configured channel/DM checkpoints emitted by this node, chkpt.rx counts signed checkpoints verified against the retained canonical channel/DM window for a cached sender identity, chkpt.mismatch counts signed checkpoints whose hash did not match that window, and chkpt.archive counts valid signed checkpoint rows added to the bounded local archive, including unverified rows when the canonical window is unavailable. When lrcd --state-dir persists that bounded archive, store.kv_write counts the successful atomic write. It also counts successful firmware lrckv seed, identity-record, channel-record, safe Admin config, deferred cursor, joined-channel snapshot, and crash-report writes. store.log_flush counts successful append-only warm-log writes such as lrclog mailbox snapshots and channel-history records, and store.reclaim counts whole warm-log segments or cold K/V pages erased to make room. seq.boot_flush counts signed-WELCOME router incarnation changes that flush stale per-router channel mirrors and pending gap state. path.discover counts relay UID bytes appended to FLAGS.PATH discovery frames. path.repin counts successful path updates learned from reverse-path ACKs or signed WELCOME frames. path.fail counts stale pinned paths invalidated after ACK failure, retry-budget expiry, or reroute away from an unreachable pinned router. path.reroute counts pending path-pinned DMs demoted back to discovery because the pinned router became unreachable while another same-UID attachment was reachable. path.store counts path relay frames kept in the bounded local store-and-forward queue because no peer/RF egress was currently available. path.forward counts queued relay frames forwarded when an egress returned, and path.store_drop counts queued relay frames dropped on expiry or oldest entry eviction. dcc.offers counts remote CTCP DCC SEND offers converted to reliable DCCCTL OFFER controls. dcc.accepts counts local accept commands converted to matching reliable DCCCTL ACCEPT controls. dcc.listener_advert counts configured receiver-local CTCP DCC SEND adverts emitted to stock IRC clients with the gateway’s listener ip32/port. dcc.bytes counts DCCDATA payload bytes delivered to a local DccDataSink. dcc.resume counts RESUME bitmaps generated from a reassembler’s missing-chunk map, dcc.retransmit counts chunks selected by the sender-side RESUME replay scheduler, and dcc.reassembled counts completed local sidecar reassemblies. dcc.hash_fail counts completed reassemblies whose nonzero expected content hash did not match the staged bytes. dcc.closed counts inbound reliable DCCCTL CLOSE controls delivered to a local sidecar sink. dcc.lane_grant counts inbound reliable DCCCTL LANE_GRANT controls delivered to a local sidecar sink. The sidecar DccBurstGrantTracker owns grant lifetime/renewal/close state but does not add a counter beyond the delivered control counters. lrcd uses active grants to cap the file-backed sender pump by preset airtime budget, now issues and renews receiver-side grants for accepted local TCP sidecar transfers, and smoke-tests the local TCP byte-stream sidecar path by bridging a stock-client DCC source through DCCDATA into a receiver-local listener. Configured firmware lane-plan retunes report through lane.retune/lane.retune_fail; a DCC-specific burst RF adapter remains future gateway work. adm.ops counts accepted mutating ADMIN operations, including stored lane-plan updates plus callback-gated TX-off and reboot requests; adm.denied counts signed mutating ADMIN requests that were rejected by allowlist, key/value validation, unavailable quorum, unsupported stores, or unsupported callbacks. adm.quorum counts newly recorded pending approvals for a valid mutating ADMIN request that has not yet met its distinct-operator quorum; duplicate same-signer requests and final approvals do not increment it. sse.clients counts chirpscope SSE connections accepted (docs/OBSERVATORY.md). sse.events_sent counts node events successfully encoded and offered to at least one connected SSE client (encoding is skipped entirely when no client is connected). sse.overflow_drop counts events dropped for one specific client’s bounded per-connection buffer because that client had not drained its previous batch — the non-blocking guarantee that keeps a stalled browser tab from stalling the shared poll() loop. chan.attest_drop counts remote channel messages suppressed by +A because the sender had no learned router-attested attachment. chan.sig_drop counts remote channel messages suppressed by +S because the sender has no latest retained checkpoint archive row that is both verified and non-mismatching. tx.retry and tx.fail cover control-plane retry loops as well as payload delivery: online DM retry, DCCCTL retry, Admin telemetry query retry, HELLO same-frame retry, and exhausted retry budgets. tx.via_tcp_suppress counts TCP-marked transit frames that were eligible for further relay but intentionally not re-emitted on RF because this node was not the local egress. tx.queue_drop counts frames shed by the bounded TX queues under overload: priority-aware eviction in lrc::TxBoundedQueue (a PRIO frame preempting the oldest non-PRIO slot, or an age-expired entry reclaimed) and the per-peer outbuf high-water on a slow federation link. The seq/backfill + RTRSYNC machinery re-delivers anything dropped once capacity returns. laneprobe.tx counts successful LANEPROBE frame construction by the lane scheduler. It does not mean the LoRa transmit completed; tx.ok, tx.fail, and tx.cad_busy continue to report the physical TX outcome. lane.retune counts successful firmware RadioLib switches between anchor and configured non-anchor lane parameters. lane.retune_fail counts failed standby/frequency/bandwidth/SF/CR transitions; after a failure the firmware restarts receive and retries on the next runtime decision. lane.grant_tx counts LANEGRANT anchor-offload grants a router emits to a just-registered client it measured able to sustain a faster lane; lane.grant_rx counts grants a client accepts (addressed to it and session-key-authenticated). The client only retunes onto the granted lane when its local lane retune on RF opt-in is set; otherwise it records the grant but stays on the anchor. credit.relayed counts frames the credit-routing decision layer (core/include/lrc/credit.h, CREDIT.md) admitted for forward: gradient predicate satisfied and the airtime debit fit the frame’s remaining credit. credit.drop_progress counts frames dropped because relaying would not strictly lower router-distance (the gradient loop guard — not a hop count). credit.drop_exhausted counts frames dropped because the hop’s airtime debit exceeded the frame’s remaining credit (the resource bound biting). credit.reoriginate counts fresh re-origination budgets a router stamped crossing a scope boundary; see CREDIT.md for the live region-coupled scaling. credit.reorig_floor counts re-originations where the live signals (duty headroom exhausted, LBT choked, or evidence absent) fell to the minimum-viable floor budget — a climbing rate localizes “why is federation reach lean in this cell” to spectrum starvation rather than routing faults. credit.reorig_lbt_limited counts re-originations where LBT deferral pressure, not duty headroom, was the governing signal (the expected steady state in KR920/CN470/JP923 — the region.lbt_deferred-heavy case the region counters’ entry above anticipates); the caller maps CreditReoriginationResult::limiter onto these. Node::handle_frame()’s v2 relay branch and Node::credit_reorigination_stamp() are the landed wire-up (Wave 2 AC byte, PROTOCOL.md §The AC byte (v2)) that actually bumps these from live traffic; the live SSE/observatory feed and lrcsim’s trace recorder read the same decision through NodeEvent.credit/debit_units (docs/OBSERVATORY.md) rather than re-deriving it. rack.rx counts an inbound RACK (PROTOCOL.md §RACK) that parsed and transited dedup — regardless of whether it went on to authenticate. rack.rx_badtag counts a RACK this node could not verify: either it held no session key with the issuing router, or verification failed under every key it does hold. Per Rule C / Rule A.2, a rack.rx_badtag count is not necessarily hostile — it is also the ordinary shape of a keyless/foreign relay’s every RACK observation (shorten-only tier, never cancel), so a nonzero rate at a foreign-relay-heavy site is expected, not alarming, by itself.

ota.manifest_verified/ota.manifest_rejected count accept_verified_image() transitions on OtaStageMachine (core/include/lrc/otastage.h, OTA.md) — verification itself happens upstream in daemon/ota_manifest_verify.h, whose own refusal reasons (bad signature, board/profile mismatch, version floor) are logged there, not counted separately here; a rejected manifest never reaches the staging state machine at all. ota.staging_started counts successful begin_staging() calls (room/battery checks passed). ota.chunks_staged counts individual chunks written through the OtaChunkSink seam. ota.resume_loaded counts stages that picked up a persisted OtaResumeCheckpoint instead of starting from zero (OTA.md §4’s reboot-durable resume); ota.resume_persisted counts checkpoint flushes. ota.hash_fail counts whole-image SHA-256 mismatches — the corrupt-or-resumed-mismatched fallback that always returns to Idle and never arms. ota.staged_verified counts stages whose hash checked out. ota.armed counts admin-gated arm() transitions; ota.apply_recorded counts record_apply_attempted() (Wave-4 hardware would call this right after the real slot-swap reboot — today it only marks the decision state machine’s transition, see OTA.md §8). ota.health_pass/ota.health_fail count ota_health_gate_step() decisions before they roll into ota.committed/ota.rolled_back. ota.aborted counts explicit abort() calls (the admin ota abort/CLOSE path). ota.refused counts every OtaStageRefusal other than the specific ones above (wrong-state calls, out-of-range/size-mismatched chunks, sink write failures) — a rollout dashboard treats a nonzero, climbing ota.refused against an otherwise-quiet rollout as the first place to look. ota.refused also counts reserved-name (otastage) DCC offers a receiving node dropped at the OTA gate (no configured stage machine, machine not in Staging, or offer geometry disagreeing with the verified image — OTA.md §3). ota.accepted counts a receiving node auto-accepting an OTA stream at that same gate (alongside dcc.accepts); on the daemon it is also OtaFleetCoordinator’s (daemon/ota_fleet_coordinator.h) first-progress bookkeeping per node. ota.offered is the coordinator’s per-release count of nodes offered the active release. ota.wave_promoted counts ota.promote confirm admin actions taking effect (OTA.md §5).

Surfaces

IRC: /msg *lrc stats [subsystem] (human tables), /msg *lrc stats dump (complete key=value counter set, including zeroes), and /msg *lrc routers [rtN] for the latest retained remote TELEM rollup. /msg *lrc routers history [rtN] [count] lists retained remote rollups newest-first for field debugging. /msg *lrc routers aggregate [rtN] [count] lists bounded hourly aggregates derived from accepted remote rollups, so counter deltas survive raw history pruning. /msg *lrc admin rtN stats radio sends a signed ADMIN telemetry query and renders the current remote rollup when the signed reply arrives. Mutating ADMIN SET and reboot replies, including quorum-pending status, are rendered as service notices and counted by the local registry.
HTTP: GET http://10.97.83.1:8462/metrics on USB-net firmware, or the node’s LAN address in WiFi-client mode — Prometheus text format. Registry counters are exported as lrc_<counter>, and retained remote TELEM rollups are exported as lrc_remote_telem_* gauges with kind="latest" or kind="history", rtr, uid, and newest-first sample labels. Hourly aggregate buckets are exported as lrc_remote_telem_aggregate_* gauges with rtr, uid, and bucket_ms labels. Home routers graph themselves with a stock Grafana.
Serial/TUI: firmware lrc> status renders the compact local dashboard: radio RX/TX counters, queue depth, IRC sessions, channel memberships, local identities, router registrations, retained remote attachments, pending ACKs, remote TELEM rollups, USB/WiFi/peer status, ESP32 memory, and variant battery ADC data when already exposed. lrc> menu, lrc> stats, lrc> usb, lrc> fwupdate, lrc> crash, lrc> reboot, and lrc> say #chan text cover the current line shell. Heltec V3/V4 OLED builds mirror a compact version of the same status without retaining display-only history. lrcd provides the full TUI over telnet; firmware serial TUI binding is future work.
TELEM packets: routers can emit a fixed 48-byte rollup on the anchor lane every 15 min (config; off by default): queue depths, duty cycle, store horizon, battery, and a small counter snapshot. Nodes retain the latest accepted rollup for each heard router and render it with /msg *lrc routers [rtN]. lrcd --state-dir also keeps a bounded history of 96 samples per router, which is 24 h at the default 15 min cadence, and persists the latest view and bounded history in telem.rollups across daemon restart. Accepted rollups also fold into bounded local-hour aggregates stored separately in telem.agg; these carry sample counts plus accumulated tx_ok, tx_fail, and seq_gap deltas so the hourly change remains visible after the raw history ring prunes. If a remote counter decreases inside a bucket, the aggregate treats that as a remote reset and keeps counting from the post-reset sample. Signed remote fetch on demand is built through ADMIN, and signed mutating ADMIN can set telem.interval_ms, history display-skew guard, RTRSYNC cadence/delta retention, directory/mailbox bounds, relay store-and-forward bounds, checkpoint cadence, verification-window, audit-retention, Admin quorum/window values, and stored lane-plan intent, or invoke callback-gated reboot, when the caller is allowlisted and optional quorum passes. lrcd --state-dir stores those safe values in admin.cfg, and ESP32 firmware stores the same numeric safe-key slice in lrckv when the durable write succeeds. Signed lane.plan writes the separate laneplan record on targets that wire that store. Remote SET/quorum results are not added to the fixed 48-byte rollup; /metrics exports the bounded latest/history views and hourly aggregate buckets. The on-air payload is already fixed and test-covered.
SSE event feed: GET http://10.10.10.11:8463/ — chirpscope’s live structured event stream (packet rx/tx/relay, registration, router liveness), one JSON object per data: line. Full schema and design rationale in OBSERVATORY.md; this is the counter-registry surface’s sibling — the same lrc::NodeEvent hook that feeds it never carries message text, matching the privacy stance above.

Event log

A 256-entry ring (PSRAM) of structured events (state transitions, path re-pins, registration churn, checkpoint mismatches, store reclaims) with monotonic timestamps, dumped by log tail and included (last 32) in crash reports. Warm-tier rollups persist hourly aggregates so “what happened last night” survives a reboot.

Crash forensics

/msg *lrc crash (or lrc> crash) prints retained postmortem records and clears them after display. lrcd --state-dir exposes text records from crash/<n> files so platform writers can be tested end-to-end on POSIX. ESP32 firmware records abnormal reset reasons (panic, watchdog, brownout, unknown) into bounded lrckv crash slots during boot, including profile/role and heap details when the SDK exposes them, so a field node can report that it crashed instead of silently returning. Normal power-on, software reboot, external reset, and deep-sleep wake do not create crash reports.

Privacy

Telemetry contains radio/system health only: no message text, no UIDs of third parties, no location unless the operator explicitly set one. The TELEM rollup format is documented in PROTOCOL.md and frozen by tests, so the community can verify the claim by reading 48 bytes.