Messaging Patterns & Protocols¶
Status: Normative (Mandatory) · Implemented: Partial (since v0.1.0) — Critical Path (§1.1) is AS-IS; Redis / Interactive Path (§1.2, §3–§6) is AS-IS (since v0.2.0) Scope: System-wide inter-service communication
This document defines the communication standards within Silvasonic — how services discover work, exchange status, and receive commands.
1. Architectural Decision: Hybrid Communication¶
To satisfy Data Capture Integrity, the system employs a Hybrid Architecture that separates the critical recording/analysis path from the interactive UI path.
1.1. The Critical Path (Polling)¶
Philosophy: The Filesystem and Database are the Source of Truth. No message broker dependency.
- Recorder → Processor: Filesystem Polling. The Processor watches the Recorder's workspace directories for new audio files and indexes them into the
recordingstable. - Processor → Workers: Database Polling. Workers (BirdNET, BatDetect) independently poll the
recordingstable for unanalyzed files usingSELECT ... FOR UPDATE SKIP LOCKED. See ADR-0018.
[!TIP] The critical path has zero dependency on Redis. If Redis goes down, recording, ingestion, and analysis continue uninterrupted.
1.2. The Interactive Path (Redis, v0.2.0+)¶
Philosophy: Responsiveness for the User Interface. Best-effort, but reliable in practice (Redis is as stable as the database on this hardware).
- Service Heartbeats → UI Service: Every service publishes periodic heartbeats to Redis. The UI Service (currently Web-Mock) uses the Read + Subscribe Pattern for real-time display (see §4).
- Service Control → DB + Nudge: The UI Service writes desired state changes to the database (e.g.,
enabled=false). A simplePUBLISH silvasonic:nudge "reconcile"wakes the Controller to act immediately. See ADR-0017.
[!IMPORTANT] The interactive path is best-effort. If Redis is unavailable, the UI loses real-time status, but all critical operations (recording, analysis, upload) continue via the filesystem/DB path.
2. Service State: Desired vs. Actual¶
See ADR-0017 for the full decision.
| Dimension | Storage | Written By | Read By |
|---|---|---|---|
| Desired State (config) | system_services table (DB) |
Admin / Web-Interface | Controller |
| Actual State (runtime) | Redis: SET with TTL + PUBLISH (see §3) |
Each service (via SilvaService) |
Web-Interface |
3. Redis Usage — Minimal and Unified¶
Status: Implemented (since v0.2.0)
Redis serves exactly four purposes for Silvasonic:
| Mechanism | Redis Command | Purpose |
|---|---|---|
| Current Status (snapshot) | SET silvasonic:status:<id> <json> EX <TTL> |
Readable anytime. TTL auto-expires key if service stops (see DEFAULT_HEARTBEAT_TTL_S in heartbeat.py) |
| Live Updates (push) | PUBLISH silvasonic:status <json> |
Real-time notification for UI subscribers |
| Live Logs (push) | PUBLISH silvasonic:logs <json> |
Container log streaming for UI subscribers (ADR-0022) |
| State Reconciliation (nudge) | PUBLISH silvasonic:nudge "reconcile" |
Wake-up signal for the Controller (ADR-0017) |
No Redis Streams, no Consumer Groups, and no other channels beyond these four.
Instance ID Convention¶
Services are identified by a combination of service (type) and instance_id (unique instance):
| Service Type | instance_id |
Key Example |
|---|---|---|
| Tier 1 Singletons | = service name |
silvasonic:status:controller |
| Recorder (multi-instance) | = devices.name |
silvasonic:status:ultramic-01 |
| Uploader (multi-instance) | = storage_remotes.slug |
silvasonic:status:nextcloud-main |
| Tier 2 Singletons | = service name |
silvasonic:status:birdnet |
4. Read + Subscribe Pattern¶
The UI Service (currently Web-Mock) uses a two-step pattern to ensure no heartbeats are missed:
1. On page load: KEYS silvasonic:status:* → read all current statuses
2. Then: SUBSCRIBE silvasonic:status → receive live updates
This solves the inherent problem of Pub/Sub (fire-and-forget): if the UI subscribes after a heartbeat was published, it would miss the latest status. The SET with TTL ensures the current snapshot is always available.
5. Heartbeat Payload Schema¶
Status: Implemented (since v0.2.0)
Every service uses the same JSON schema, published by the SilvaService base class. The canonical schema definition is in ADR-0019 §2.4. All payloads MUST be valid JSON and validated via Pydantic models (ADR-0012).
6. Control Flow — State Reconciliation Pattern¶
Control is declarative (DB desired state), not imperative (HTTP commands). This follows the Kubernetes Operator Pattern:
┌──────────────────────────────────────────────────────────────────┐
│ State Reconciliation (DB + Nudge) │
│ │
│ 1. UI Service ──[DB Write]──► Database (desired state) │
│ 2. UI Service ──[PUBLISH silvasonic:nudge]──► Redis │
│ 3. Controller ──[wakes up, reads DB]──► reconcile() │
│ 4. Controller ──[podman-py]──► Podman ──► start/stop containers │
└──────────────────────────────────────────────────────────────────┘
The nudge is a simple wake-up signal — not a command. If the nudge is lost (Controller restarting), the reconciliation timer catches up. The DB desired state is never lost.
[!NOTE] Immutable services (Recorder, Workers, Processor) do not process runtime commands. To change their configuration, the Controller stops and restarts them with updated environment variables.
7. Communication Flow Overview¶
┌─────────────────────────────────────────────────────────────┐
│ CRITICAL PATH (Filesystem + DB — no Redis dependency) │
│ │
│ Recorder ──[WAV files]──► Processor ──[DB INSERT]──► DB │
│ │
│ BirdNET ──[SELECT FOR UPDATE SKIP LOCKED]──► DB │
│ BatDetect ──[SELECT FOR UPDATE SKIP LOCKED]──► DB │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ INTERACTIVE PATH (Redis + DB, v0.2.0+) │
│ │
│ All Services ──[heartbeat]──► Redis ──► UI Service (Web-Mock) │
│ UI Service ──[DB write + nudge]──► Controller ──► Podman │
└─────────────────────────────────────────────────────────────┘