Scenario Testbed — `lrcsim`, a unix-native network laboratory

Status: DESIGN (forward-looking, not normative). Everything here is host-testable [HOST] work — no wire change, no RF. It extends the existing tiers in ../TESTING.md; it does not replace them.

1. Why a new tier

The current ladder — lrctest units → test_node.cpp in-process sims → ChaosNet fault injection → VirtualRadio multi-lane PHY → smoke_lrcd.py real sockets — answers “is this behavior correct?” It cannot yet answer “how does the network behave?”: What happens when a node moves across three cells? When the one router on the ridge dies at hour 6? When a hilltop node’s downlink reaches 200 nodes but its uplink hears only 5? Those are scenarios — topology + physics + time + faults — and they need a dedicated laboratory with a physical-layer model, mobility, and replayable traces.

lrcsim is that laboratory: a unix-native binary (macOS/Linux, CI-runnable) that composes the pieces the codebase already has — lrc::Node is pure logic, VirtualRadio already models lanes/SNR floors/capture — into whole-network scenario runs with assertions and trace output.

2. Two execution modes, one field model

Mode A — in-process (fast, deterministic, CI default). One process hosts N lrc::Node instances wired through the field model. Virtual clock (the Clock seam), seeded Rng: a scenario run is a pure function of (scenario file, seed). Thousands of sim-hours per CPU-minute; failures bisectable by seed.

Mode B — multi-process (the “unix-native binaries” bed). Real lrcd processes (and, later, firmware images under QEMU/Wokwi if that ever pays) joined by lrcbridge: a small broker that carries opaque radio frames over UDP and applies the same field model to decide who hears what, when, and how mangled. Mode B exercises what Mode A cannot: the actual daemon binary, its poll() loop, --state-dir persistence across real process kills (SIGKILL mid-write), TCP federation alongside simulated RF, and true wall-clock timing races. CI runs a small Mode-B smoke set; the full matrix is Mode A.

Status: BUILT (this section’s design realized without deviation from the shape above). lrcd --rf-udp wires Node::out_to_radio/on_radio_frame to UDP datagrams; sim/lrcbridge.cpp is the broker, reusing sim::Field exactly as planned (zero forked propagation logic); tests/smoke_modeb.py covers RF-only federation (no TCP --peer at all), the SIGKILL mid-traffic + --state-dir restart persistence torture, partition/heal reconvergence, a mid-run lane retune (below), and lrcbridge --trace producing valid JSONL with delivery and drop events. Normative reference: ../LRCSIM.md §8; day-to-day usage, the datagram header, the retune verb, and the trace schema: ../TESTING.md “Mode B”.

Both gaps this section used to defer are now closed:

Mid-run lane retune. lrcd has a *lrc service verb, rf retune <preset> <freq_slot> — daemon-local (there is no remote router to route a request to; the --rf-udp seam being retuned is a daemon/main.cpp construct), authorized the same way IRC oper power already is (Node::is_oper() against --admin-operator). It re-sends the TUNE datagram (the same mechanism the startup announcement uses, triggered again) and calls Node::set_relay_preset() so AC credit routing prices relays at the new preset without pinning this node’s learned router distance (set_routing_distance_context() is a different, unrelated setter — node.h documents the split). tests/smoke_modeb.py case (d) retunes both routers in a running pair and asserts traffic still converges on the new lane.
A Mode B trace vocabulary — the JSONL half, not the assert half. lrcbridge --trace FILE|- emits packet_rx/sim_drop events through the SAME sim::TraceWriter/sim::JsonlEvent (sim/trace.h) Mode A’s runner uses, so chirpscope replay and other trace tooling read both modes unmodified; sim_drop gains one Mode B-only reason, malformed, for corrupt wire bytes Mode A structurally never sees. tests/smoke_modeb.py case (e) asserts the trace file is valid JSONL with at least one delivery and one drop event carrying the documented fields. Mode B still has no assert grammar (see below) — the trace is a replay/tooling artifact here, not yet a Mode B-native pass/fail mechanism.

Deferred: firmware-under-QEMU/Wokwi (still just an acknowledged “if that ever pays” aspiration above, not attempted), and a Mode B assert vocabulary (Mode A’s assert grammar has no Mode B counterpart yet — correctness is checked the smoke_lrcd.py way, scripted IRC clients + metrics, now optionally cross-checked against the trace above, not a replay-and-assert pipeline).

The field model (shared library, used by both modes) extends VirtualRadio from per-pair SNR into geometry:

Geometry: each node has (x, y, z) meters, TX power dBm, antenna gain. Received SNR = link budget under log-distance path loss (exponent per-scenario, 2.7–3.5 urban/rural) + optional log-normal shadowing σ.
Asymmetry as a first-class citizen: per-node TX power and height make asymmetry emerge (the hilltop node is loud downhill but deaf to 20 mW handhelds), and per-pair directional overrides let a scenario pin exact pathologies. This is the single most important modeling requirement — the routing redesign lives or dies on asymmetric links.
PHY realism, modeled from measurement, not assumption: preset SNR floors (reuse VirtualRadio::preset_floor), co-SF collision unless capture (≥6 dB), imperfect inter-SF orthogonality (~−16 dB rejection matrix, not the “SFs are orthogonal” myth — matters for the channel-sharing workstream), duty-cycle enforcement via each node’s real AirtimeLedger.
Terrain (cheap version): optional per-pair obstruction penalties from a height map is deliberately out of scope v1; explicit overrides cover the hill scenario without writing a propagation engine.

3. Scenario files

Flat, diff-friendly, zero-dependency format (same spirit as lrctest.h — hand-rolled parser, no TOML library):

seed 42
field pathloss 3.0 shadow 2.0
node hill  role=router x=0 y=0 z=150 tx_dbm=27
node plain[1..200] role=client grid=1000x1000 tx_dbm=14
link hill->plain[*] snr=+8       # everyone hears the hill
link plain[6..200]->hill snr=off # the hill hears only plain[1..5]
link plain[1..5]->hill snr=-14   # …barely
at 60s  chanmsg from=plain[137] chan=#field text="can anyone hear me"
at 300s kill hill
at 420s revive hill              # reboot-with-persistence
mobility plain[42] waypoints (0,0)@0s (900,900)@600s
assert converged chan=#field by 900s
assert delivered from=plain[137] to=plain[1..200] ratio>=0.95
assert duty_legal all
assert no_dup_delivery all

Constructs: node groups with ranges/grids, directional link overrides, timed events (chanmsg, dm, join, kill, revive, partition, heal, skew), waypoint mobility, and an assertion vocabulary drawn from the chaos harness’s convergence invariants plus network-level metrics (delivery ratio, per-node duty legality, credit-drop counts, path-length distributions, anchor-lane occupancy).

4. Traces — the observatory’s food

Every run emits a JSONL event stream (tx, rx, drop{reason}, dedup, reg, lane, credit_debit, node_state) with virtual timestamps and positions. Consumers:

CI: assertion failures attach the trace as an artifact; a failure is reproducible from (scn, seed) alone.
The web observatory (CHIRPSCOPE workstream in ../ROADMAP.md) replays traces — scrub through a partition heal, watch the credit budget drain hop by hop, see the ERB return path diverge from the uplink. The same viewer that debugs a live network debugs a simulated one; one visualization pipeline, two sources.
Tuning sweeps: the credit-routing constants (routing-redesign/CREDIT_ROUTING.md §7) and contention-window calibration read distributions straight off traces.

5. The canonical scenario library

Checked into tests/scenarios/, run by CI, named in commit messages when they gate a change:

Scenario	What it proves
`hilltop_asym`	200-hear/5-heard asymmetric routing: registration, chat, and returns all survive gross asymmetry
`router_death`	critical-node failure: solo-scope degradation is documented-graceful; federated scopes fail over; revival heals via boot-id + CHANSYNC
`roam_walk`	a client walks across 3 cells: make-before-break roaming, mailbox flush, no dup/no gap
`partition_heal`	backbone splits 10 min under load; RTRSYNC anti-entropy converges without anchor-lane flooding
`metro_500`	500 clients / 6 routers: anchor occupancy stays under the cadence controller’s ceiling; promotion ladder drains the anchor
`eu868_duty_starve`	saturation under a 1% duty subband: ledgers never exceed budget; priority traffic still lands
`multi_sf_share`	co-channel multi-SF lanes under the −16 dB rejection matrix: measured (not assumed) concurrency win
`credit_depth`	delivery vs. diameter per preset; the empirical basis for the AC budget table
`reboot_storm`	rolling reboots with persistence: SEQ3/boot-id discipline holds; no gap storms

6. Build plan (agent-executable, in order)

sim/field.h — field model as pure library + unit tests (extends VirtualRadio patterns; no new deps).
sim/scn.h — scenario parser + tests (golden parse vectors).
lrcsim binary (Mode A): compose Nodes + field + events + assertions; port 3 existing chaos tests into .scn form to prove expressiveness.
Trace writer + assert vocabulary; wire the scenario library into CI (matrix job, artifacts on failure).
BUILT. lrcbridge + Mode B: UDP frame bus, spawn/kill real lrcd processes, port smoke_lrcd.py’s federation cases (tests/smoke_modeb.py case a); add the SIGKILL-mid-write persistence torture (case b) and a partition/heal reconvergence case (case c). See §2’s Mode B status note above and ../LRCSIM.md §8 for the shipped shape.
Nightly job: randomized seeds across the library; new-failure seeds get committed into a regression-seeds list (a failure found once is a test forever).

Non-goals: RF propagation research (this is a protocol laboratory — the field model needs to be adversarial and plausible, not geodetically true); replacing the chaos harness (it stays as the fast unit-grade tier); simulating silicon timing (Mode B + the bench cover that).