Remote OTA Update — signed, resumable, rollback-safe fleet updates

Research — non-normative

This page is synced from the repo’s docs/research/ tree: exploratory design work, forward-looking plans, or historical analysis. It may describe behavior that is not built or that shipped differently. The normative spec lives in the other docs sections.

Status: DESIGN (forward-looking, not normative). No wire change is required for the transport — it deliberately reuses the built DCC burst machinery — but the new signed-Admin keys and the release-key trust root are a security-surface event: update docs/IDENTITY.md and docs/SECURITY_AUDIT.md when this lands. Applying an image on hardware is RF-adjacent (a bricked router is a dead cell): the firmware-side apply path needs human sign-off per AGENTS.md rule 5 even though no TX behavior changes.

1. Scope and stance

Nodes with staging room — an SD card, ≥2 MB free PSRAM, or an otastage flash partition — can be updated remotely by an authenticated admin, with integrity, resume, and rollback guarantees. Nodes without staging room keep the local paths (USB flasher, pio run -t upload, the web flasher on the site). LoRa is a narrow pipe; this design is honest about it rather than pretending a mesh is a CDN.

Airtime reality. A ~1.4 MB image (the CI size gate pins the ceiling):

Path	raw throughput	pure transfer time	verdict
TCP backbone / WiFi	MBs	seconds	primary path for routers (profile B)
P7 burst lane (SF5/500, ~39 kbps)	~4.9 kB/s	~5 min	viable, near/strong links only
P6 burst lane (SF7/500, ~17 kbps)	~2.1 kB/s	~11 min	viable inside duty budget
P3 default client lane	~0.3 kB/s	~78 min	emergencies only
P0 anchor	—	—	never; the anchor is sacred

EU-style duty limits stretch wall-clock further (a 10% subband turns 11 min of airtime into ~2 h elapsed). The design treats LoRa OTA as a trickle with resume, not a download bar.

2. Trust model

• Release key — an Ed25519 keypair whose public half is baked into every firmware image at build time. It signs release manifests and nothing else. It is not an identity key, not an admin key; compromise of every node admin still cannot forge an image. Private half lives in CI release signing (GitHub environment-protected secret) — see the release-engineering workstream in ../ROADMAP.md.
• Admin authority — offering and applying an update are signed-Admin operations under the existing deny-by-default, optionally quorum-gated Admin slice. The release key authenticates what the bytes are; the admin path authorizes whether this node, now.
• Both are required. A valid manifest without an admin apply order does nothing; an admin order without a manifest-verified image does nothing.

3. Manifest

A small canonical structure, signed by the release key:

magic "LRCF" | manifest_ver u8 | fw_semver (maj u8, min u8, patch u8, build u32)
board_id u16 | profile u8 (A/B/C) | image_size u32 | image_sha256 [32]
min_running_semver (3 bytes) | flags u8 (CANARY, REQUIRES_STAGED_CONFIG…)
ed25519_sig [64]  // over all preceding bytes

board_id/profile prevent cross-flashing (a RAK image offered to a XIAO is rejected before a single chunk transfers). min_running_semver fences upgrade ordering when a release depends on a prior migration (e.g., a partition-map change).

4. Transport — reuse, don’t invent

The DCC stack already provides everything a firmware transfer needs, all host-tested: DccChunkScheduler (staged bytes → bounded chunks), DccReassembler (missing-chunk tracking, compact RESUME bitmaps), whole-file SHA-256 verification with CLOSE HashFail, LANE_GRANT burst leases with airtime budgets, and the lease state machine (DccBurstLeaseMachine). An OTA transfer is a DCC stream whose recipient sink writes to the staging area instead of a user file:

1. Admin issues signed ota.offer (manifest bytes + source: a backbone URL for TCP-attached nodes, or the offering node’s UID for LoRa DCC).
2. Node verifies manifest sig + board/profile/min-version, checks staging room and battery/power policy (POWER.md floor — never start a flash cycle a battery can’t finish), then ACCEPTs.
3. Bytes flow: TCP fetch when a backbone link exists, else DCC burst chunks under a LANE_GRANT lease. Chunks land in the staging area with the reassembler’s bitmap persisted periodically (lrckv otaresume record), so a reboot mid-transfer resumes from the bitmap instead of restarting — this is the “failed packet recovery” guarantee, and it is the existing RESUME machinery doing its normal job.
4. On completion: SHA-256 over the staged image must equal the manifest hash (the existing DCC verify path), else the stage is discarded and ota.hash_fail bumps.

5. Apply and rollback (ESP32 first)

• Partition maps gain dual OTA app slots (ota_0/ota_1) on 8 MB+ boards; apply = esp_ota_write from stage → set boot slot → callback-gated reboot (the exact pattern already built for signed-Admin reboot).
• Health gate before commit. The new image boots in “pending” state and must, within a window: mount its partitions, reach registered (or, for a router, serve one WELCOME), and record no crash slot. Only then does it mark itself valid (esp_ota_mark_app_valid_cancel_rollback). Otherwise the bootloader’s anti-rollback returns to the previous slot and the node reports ota.rolled_back — a bad push costs one reboot cycle, not a cell.
• TX safety is config, not code. tx on lives in NVS/lrckv, never in the image; a fresh image inherits the persisted radio state and the ship-OFF default otherwise. An OTA can therefore never enable TX.
• nRF52 (RAK4631): the Adafruit bootloader’s DFU + LittleFS staging is a later phase; investigate MCUboot bank-swap. Marked stretch — it is a new dependency conversation (AGENTS.md rule 6) and a different flash topology.

6. Fleet orchestration

• Canary → wave → fleet. ota.offer carries the CANARY flag for a configured UID set first; promotion to the fleet is a second signed order after the canaries pass their health gates. Wave size for LoRa cohorts is bounded by lane airtime (one transferring node per burst lane at a time — the lease machinery already serializes this naturally).
• Telemetry. ota.offered / accepted / chunks_rx / resume_tx / hash_fail / applied / rolled_back / health_pass counters ride the existing TELEM rollup and land in lrcd --state-dir history + Prometheus, so a rollout is watchable from the web observatory (CHIRPSCOPE in ../ROADMAP.md) without new plumbing.
• Operator UX. lrcctl ota status|offer|promote|abort wraps the signed Admin ops; the observatory renders rollout state per node.

7. Guardrails summary (the checklist reviewers hold this to)

1. Release-key signature verified before staging starts and re-verified as whole-image SHA-256 after staging completes.
2. Board/profile/min-version fences checked before any chunk is accepted.
3. Resume bitmap persisted; a reboot never restarts a transfer from zero.
4. Health-gated commit with automatic bootloader rollback.
5. TX state persisted outside the image; updates cannot enable radios.
6. Power floor honored before apply; anchor lane never carries image bytes.
7. Quorum-gateable admin ops; canary cohort before fleet; abort is a first-class signed op that tears down leases via the existing CLOSE.

8. Later (explicitly deferred)

• Delta updates (bsdiff/detools-style): huge airtime win, but the decompressor’s memory profile on-target and the new-dependency rule need their own review. Design the manifest flags bit now (done above), ship full images first.
• Peer-to-peer image seeding (a flashed node re-offers to its cell): powerful for far cells, but multiplies the trust/abort surface; needs the observed-delivery machinery from the routing redesign to mature first.