Testing

Normative — synced from the repo

This page mirrors docs/TESTING.md verbatim. In the lrc repo the docs are the spec: they move in the same commit as the behavior they describe.

LoRa Relay Chat is designed so that everything except the literal act of radiating RF power is testable on the host — no boards required. The docs are the spec, the tests are the law, and the whole-network behavior runs in CI in a few hundred milliseconds. This file documents the tiers, what each one covers, and — just as importantly — exactly what still needs a bench.

The tiers

Tier	Where	What it covers	Runs in CI?
Unit / pure policy	tests/test_*.cpp (crypto, packet, lanes, reg, routing_policy, return_path, peer_link, …)	One component in isolation, vector-driven	✅
VirtualRadio	tests/virtual_radio.h + test_virtual_radio.cpp	The simulated multi-lane PHY: lanes, spreading factor, SNR/range budgets, airtime/duty, lane contention	✅
ChaosNet	tests/test_chaos.cpp	N-node federation over a shared medium: loss/latency/dup/reorder, partitions, reboots, CHANSYNC gap healing	✅
lrcsim scenario testbed	sim/, tests/scenarios/*.scn, tests/test_sim_*.cpp	Whole-network scenarios with geometry, mobility, and fault injection; see LRCSIM.md	✅ (.github/workflows/scenarios.yml)
Daemon end-to-end	tests/smoke_lrcd.py	Real sockets, real lrcd instances on loopback, real IRC clients	✅
WS-SIM Mode B (multi-process)	sim/lrcbridge.cpp, tests/smoke_modeb.py	Real lrcd processes federating over simulated RF (UDP) through the same field model Mode A uses; --state-dir persistence across real SIGKILL, mixed TCP+RF topologies — see §Mode B below	✅
Firmware size-budget	tests/check_firmware_size.py ENV	PlatformIO build + flash-size parse against the 2 MB OTA slot budget — no hardware needed	✅
Fuzz (radio parse)	tests/fuzz/decode_fuzz.cpp	libFuzzer over decode() + decode_identity_record() — adversarial parse inputs (off by default, -DENABLE_FUZZ=ON)	✅ (CI)
RF bench	human-gated, AGENTS.md rule 5	TX power, regulatory airtime on actual silicon, antenna/range, board-level actuation	❌ — needs you

The first four are all driven by ./build/lrc_tests and python3 tests/smoke_lrcd.py from the build-and-verify loop in AGENTS.md.

The seam the whole thing hangs on

A Node exposes three emit seams and two ingest paths:

core/include/lrc/node.h

std::function<void(const uint8_t*, size_t)>              out_to_peers;   // TCP/federation
std::function<bool(RouterId, const uint8_t*, size_t)>    out_to_peer;    // directed federation
std::function<void(const uint8_t*, size_t)>              out_to_radio;   // RF lane  ← VirtualRadio
void on_peer_frame(const uint8_t*, size_t);                              // federation ingest
void on_radio_frame(const uint8_t*, size_t);                             // RF ingest ← VirtualRadio

On a board, the firmware’s SX1262 driver + the CAD-before-TX loop sit behind out_to_radio (firmware/src/main.cpp), and the radio ISR delivers to on_radio_frame. On the host, a function call replaces the air: ChaosNet wires out_to_peers to an in-process delivery queue, and VirtualRadio wires out_to_radio to a simulated PHY that applies the same lane/SF/airtime policy the firmware’s LaneSchedule/airtime_ms would. The Node under test is the same compiled code that ships in firmware.

What VirtualRadio faithfully replays

It is test-only code under tests/; it never ships in firmware. It uses the same shipped policy the firmware uses, so the two can’t drift:

• Lane scheduling — the caller tells it which lane a node is parked on; a frame is heard only by listeners on the same (preset, freq_slot).
• Spreading factor — a lane’s preset fixes one (SF, BW, CR). A frame is received only where the per-link SNR clears that preset’s probe floor (kPresets[].probe_snr_x10). This is the primitive behind “the slow preset reaches where the fast one can’t.”
• Airtime / duty cycle — every TX consumes airtime_ms(preset, bytes) from a per-node rolling budget; exhaustion defers exactly like the firmware’s CAD loop would. (When Phase 1’s AirtimeLedger lands in core, VirtualRadio delegates to it; the model stands alone today.)
• Lane contention — two TXs in the same slot on the same lane collide.
• Multi-radio listeners — a router parked on several lanes via LaneSchedule::assign_radios() hears each one concurrently.
• Geometry (optional) — (x, y, height) per node yields a log-distance SNR so “router on a hill reaches a hidden client on SF5, but the client only returns on SF10” is expressible without antennas.

What it deliberately does not model: crystal drift, capture effect, fading multipath, and radiated power — those only answer to a bench.

Mode B: real lrcd processes over simulated RF

lrcsim Mode A (above) proves in-process lrc::Node logic against a virtual clock. Mode B (docs/research/SCENARIO_TESTBED.md §2, docs/LRCSIM.md §8) is the complementary unix-native multi-process bed: REAL lrcd binaries, spawned as real OS processes, exchanging simulated RF frames through lrcbridge — so the things Mode A’s virtual clock and in-process Node calls can’t exercise get covered: the actual daemon binary and its poll() loop, --state-dir persistence across a real SIGKILL (not a C++ object destructor), and mixed TCP-peer + simulated-RF topologies. CI runs a small, bounded Mode B smoke set; the full scenario matrix stays Mode A’s job (fast, deterministic, seed-driven).

The seam: lrcd --rf-udp

lrc::Node exposes exactly one radio ingress/egress pair (docs/TESTING.md “The seam the whole thing hangs on”, above): out_to_radio / on_radio_frame. On a board, firmware’s SX1262 driver sits behind it; in Mode A, VirtualRadio/sim::Runner does. lrcd --rf-udp HOST:PORT wires the SAME seam to UDP datagrams exchanged with a lrcbridge process instead — no other daemon behavior changes, and no socket code was added to core/ (AGENTS.md rule 2: the daemon owns the socket, Node still only ever sees out_to_radio(bytes) / on_radio_frame(bytes)).

Because every frame Node emits (fresh TX, relay, RTRSYNC, CHANSYNC replay) already flows through emit_radio_frame() unconditionally (core/src/node.cpp), wiring this one seam is the ENTIRE daemon-side glue Mode B needs — a lrcd started with no --peer at all still fully federates (directory, channel chat, DMs, RTRSYNC) once two such processes share a lrcbridge. This was proven end-to-end while building this seam: two peer-less lrcd processes exchanged channel chat purely over --rf-udp, and a manually-inserted 100 km link correctly delivered nothing (the field model’s floor gate, not a stub).

Flags: --rf-udp HOST:PORT (the lrcbridge address; absent = historical behavior, out_to_radio unwired), --rf-node-id N (0..255 — this process’s stable wire identity; independent of --rtr, since an RF participant need not be a router), --rf-preset 0..7 and --rf-freq-slot N (this process’s starting lane; default anchor/0, matching every other lane-placement default in the codebase).

The datagram header

One UDP datagram is one on-air frame (matching the LoRa packet-radio abstraction the rest of the codebase assumes — one send() per transmission, no stream framing beyond UDP’s own datagram boundary):

u8  magic       0xC5 — rejects stray traffic before it's mistaken for a
                malformed LPP frame
u8  version     1 — bumped on any incompatible header change
u8  msg_type    0 = DATA, 1 = TUNE
u8  node_id     sender's --rf-node-id
u8  preset      sender's CURRENT lane preset (lrc::kPresets index, 0..7)
u8  freq_slot   sender's CURRENT frequency slot
u16 payload_len network byte order; 0 for TUNE (no body follows — the
                header above IS the tuning announcement)
[payload_len bytes]  opaque LPP frame bytes (DATA only)

A TUNE datagram is sent once at daemon startup so lrcbridge learns a listener’s lane even if it never transmits (a pure receiver still needs the bridge to know where it’s listening), and again whenever /msg *lrc rf retune (below) changes this process’s own lane mid-run. Every inbound datagram (DATA or TUNE) refreshes lrcbridge’s learned (source address, node_id, preset, freq_slot) for that sender — there’s no separate registration handshake, matching how a real radio medium doesn’t require a listener to “log in” before it can be heard.

Counters (docs/TELEMETRY.md): rfudp.tx / rfudp.rx / rfudp.drop / rfudp.tune_tx / rfudp.tune_rx / rfudp.retune / rfudp.retune_fail.

Mid-run lane retune: /msg *lrc rf retune

The single-radio frequency-agility story: one radio that retunes mid-run, reaching the daemon rather than staying a scenario-file-only lane placement. lrcd has no CLI/admin path over the wire for this (unlike admin rtN set ..., docs/USERGUIDE.md “Remote node administration”) because there is nothing to route: the --rf-udp seam this retunes is a daemon/main.cpp construct (RfUdpSeam) with no remote-router counterpart to sign a request for. Instead it’s a daemon-local *lrc service verb, authorized the same way IRC OPER power already is (Node::is_oper() — the connecting session’s IRC USERNAME must itself be a 16-hex-digit UID listed in --admin-operator, docs/USERGUIDE.md’s “one identity per IRC username” convention):

/msg *lrc rf retune <preset 0..7|PN> <freq_slot 0..255>

On success this does three things in order: (1) validates the preset id and frequency slot (rejecting anything parse_preset_span/parse_u8_span wouldn’t accept — a bare index, or P-prefixed like lane.plan’s own lane grammar, and 0..255 for the slot); (2) updates the daemon’s own announced tuning and re-sends the TUNE datagram to lrcbridge, exactly the one-shot-at-startup announcement above, just triggered again; (3) calls Node::set_relay_preset() so AC credit routing immediately prices relays at the new preset — deliberately NOT set_routing_distance_context(), which would also pin this node’s rdist (node.h’s own documented distinction: a lane retune is not a claim about hop distance, and pinning rdist on a retune-only path would fight the passive router-distance learner for every future embedder that only wants the airtime-pricing half). A node started without --rf-udp reports rf retune denied: unsupported; an unauthorized session reports rf retune denied: not an admin operator; a failed bridge send reports rf retune failed: ... and leaves the previous preset in effect (no partial state — set_relay_preset() is only called after the seam confirms the retune actually went out).

lrcctl radio (below) surfaces rfudp.tune_tx/rfudp.tune_rx and the new rfudp.retune/rfudp.retune_fail counters alongside the existing rfudp.* funnel, so an operator watching a running node sees a retune land without needing to grep /metrics by hand.

lrcbridge

sim/lrcbridge.cpp is the broker: a UDP server that, on each DATA datagram, decides which OTHER known peers hear it — reusing sim::Field (sim/field.h) exactly as Mode A’s sim::Runner does, so a Mode B run’s propagation is byte-for-byte the same model (not necessarily the same numbers, since Mode B runs on a real wall clock, not a virtual one) as Mode A’s. Single-tuner is the baseline truth: a listener hears a transmission only if its last-announced tuning exactly matches the sender’s (preset, freq_slot) AND the field-model SNR between them clears that preset’s probe floor (sim::Field::clears_floor — the same admission test VirtualRadio/ Mode A use).

cmake --build build -j --target lrcbridge
./build/lrcbridge --listen 127.0.0.1:9500 --config topology.scn \
    --control-udp 127.0.0.1:9501 --trace out.jsonl --verbose

The --config file is a SUBSET of the .scn grammar (sim/scn.h), parsed with the real parser (no forked grammar): seed (required by the parser), field pathloss/shadow/..., node <id> [x= y= | grid=] [z=] [tx_dbm=], and link <id>-><id> snr=<dB>|off. A node’s NAME in this file is the decimal wire node_id a peer’s --rf-node-id sends (e.g. node 1 x=0 y=0) — the only stable per-sender key available on the wire (a UDP source port isn’t: it can change across a daemon restart). Without --config, every link is an open link (clears every floor), matching sim::Field’s own default for a scenario node without geometry — so lrcbridge --listen H:P alone is a valid “just relay everything whose lane matches” bridge.

Runtime fault injection — the control channel (deliverable 2’s requirement) — accepts line-oriented commands over stdin (when the process is started without --control-udp, or always) or --control-udp datagrams, using node ids from the same config-file identity space:

partition <a> <b>     # cut the link both directions
heal <a> <b>          # restore it
snr <a> <b> <dB>      # pin a directional override (matches .scn `link`)
snr <a> <b> off       # pin "never hears" (matches .scn `snr=off`)
snr <a> <b> clear     # remove a live override, fall back to the config
                      # file's own field/link baseline

lrcbridge --trace

--trace FILE (or --trace - for stdout) emits one JSONL event per line using the SAME schema and sim::TraceWriter/sim::JsonlEvent machinery Mode A’s lrcsim --trace uses (sim/trace.h, docs/LRCSIM.md §6) — deliberately not a forked format, so chirpscope replay and any other trace tooling reads both modes’ output unmodified. Because Mode B has no virtual clock (SCENARIO_TESTBED.md §2), the envelope’s mono_ms is real wall-clock milliseconds since the bridge started (the writer still stamps clock_kind:"virtual" — that’s the schema’s own field/value, shared verbatim rather than forked for a real-time producer). The trace file is line-buffered so a SIGTERM/SIGKILL-style teardown (this process runs forever otherwise; smoke_modeb.py’s own cleanup convention) never loses the last few lines to libc’s full-buffering-for-a-regular-file default.

Every datagram this bridge receives emits exactly one of:

• packet_rx per listener that heard it — same fields as Mode A’s packet_rx (node, from, packet-header set, payload_len, snr_x10, preset, freq_slot).
• sim_drop per listener/attempt that didn’t, with reason one of lane_mismatch (tuning didn’t match), below_floor (SNR below the preset’s probe floor), partitioned (the control channel’s partition/snr ... off cut this link), or malformed (bad magic/ version/length — Mode B’s own addition to sim_drop’s reason vocabulary: Mode A never sees corrupt wire bytes, only this real UDP seam can).

./build/lrcbridge --listen 127.0.0.1:9500 --config topology.scn --trace out.jsonl

tests/smoke_modeb.py

Pattern-matches tests/smoke_lrcd.py (stdlib only, spawns the real lrcd/lrcbridge binaries, hard timeouts, cleans up every process). Five cases, all bounded well under CI’s 3-minute budget (~25 s total measured):

1. RF-only federation — two routers with NO --peer at all federate purely over lrcbridge: JOIN, channel chat, and a cross-node DM all converge. This proves the path smoke_lrcd.py’s TCP-only cases never touch — directory/registration/RTRSYNC riding on_radio_frame, not on_peer_frame.
2. SIGKILL persistence torture — one router is hard-killed (Popen.kill() → SIGKILL, not a graceful terminate()) mid-traffic and restarted with the SAME --state-dir: the survivor keeps serving its own local traffic uninterrupted (no TCP link-down machinery applies to an RF seam — there’s no keepalive/link-state concept, matching real radio silence), the restarted router comes back with a fresh boot-id and resumes sequencing without replaying anything twice, and post-restart traffic reconverges with no duplicate delivery on either side.
3. Partition / heal — lrcbridge’s partition/heal control verbs cut and restore the simulated link while both routers keep serving local traffic; a message sent during the partition is provably NOT delivered across it, and arrives once the link heals (RTRSYNC/relay catch-up), alongside immediate fresh two-way traffic post-heal.
4. Mid-run lane retune — /msg *lrc rf retune on both routers (single- tuner gating means retuning only one side would just silence the link, not prove a retune “works”) moves them from the anchor lane to REACH on a new freq_slot; asserts the TUNE datagram was genuinely re-announced (lrc_rfudp_tune_tx/lrc_rfudp_retune counters climb on both routers) and that fresh traffic still converges once both sides have retuned and settled.
5. lrcbridge --trace — runs a bridge with tracing on against a three-node topology (one node parked 500 km out, guaranteeing a below_floor drop), drives one delivered channel message, and asserts the trace file is valid JSONL containing at least one packet_rx and one sim_drop event with the documented fields.

Run: python3 tests/smoke_modeb.py [path-to-lrcd] [path-to-lrcbridge].

Known gaps

• No RF propagation between lrcbridge and Mode A’s assert grammar. Mode B’s trace is now the SAME schema/writer Mode A’s is (previous paragraph), but there is still no Mode B assert vocabulary — fault injection stays imperative (partition/heal/snr), and correctness is checked the same way smoke_lrcd.py checks it: scripted IRC clients and metrics, not a trace-driven pass/fail (chirpscope replay is the trace’s actual consumer, not a Mode B-native assertion pass).
• The bridge is O(n) per-recv over known peers, matching lrcsim’s own documented O(n²)-per-round tradeoff (LRCSIM.md §7) — fine at Mode B’s intended scale (a handful of processes proving persistence/process behavior, not a 500-node capacity run; that’s still Mode A’s job).

What still needs real hardware (and why)

Per AGENTS.md rule 5, RF safety is not reviewable by CI. The human-gated set shrinks to what can only be answered with a real antenna:

1. TX power and regulatory airtime on actual silicon. VirtualRadio proves the logic that decides when to TX and on which lane; the bench proves the radio retunes and radiates within the regulatory power/airtime budget (docs/HARDWARE.md, docs/POWER.md).
2. Asymmetric-return and RF-backbone actuation on real boards. The framing, auth, and lane-selection logic is host-testable; the physical retune and on-air timing need a multi-radio bench.
3. Anything touching firmware/variants/, RF-switch handling, TCXO voltage, or TX paths. Receive-only defaults; human sign-off recorded in the PR.

Everything else — the wire codec, crypto/identity, registration, transport recovery, lanes/airtime policy, the IRC gateway, the TUI, admin/oper, federation netsplit/catchup, DCC — is host-testable and runs in CI.