SDR Service Architecture

Overview

The EAS Station™ uses a dual-service architecture for SDR (Software Defined Radio) operations to ensure reliable 24/7 operation required for life-safety systems. This document describes the architecture, components, and operational details.

Service naming: The two processes live at the repo root and run as the eas-station-sdr.service and eas-station-audio.service systemd units. They were historically referenced as sdr_service.py and audio_service.py; the actual files are sdr_hardware_service.py and eas_monitoring_service.py. A separate eas-station-eas.service (eas_service.py) used to run a second, independent EAS/SAME decoder against the same audio stream; it was retired as a redundant duplicate (see docs/reference/CHANGELOG.md) — eas_monitoring_service.py is now the sole decoder.

Architecture Diagram

Components

1. SDR Hardware Service (sdr_hardware_service.py)

Purpose: Dedicated service for SDR hardware operations only.

Responsibilities:

• SoapySDR device management (open, configure, stream)
• Dual-thread USB reading for jitter absorption
• IQ sample publishing to Redis
• Health metrics publishing
• Control command processing

Runtime Requirements (bare metal systemd):

• Runs as the eas-station-sdr.service unit (see systemd/eas-station-sdr.service).
• Direct USB access via the host's /dev/bus/usb tree (the service runs as a user in the plugdev group).
• RLIMIT_MEMLOCK lifted so SoapySDR can lock pages for high-rate IQ capture.
• Started by install.sh / update.sh; control via systemctl status|restart eas-station-sdr.service.

2. Ring Buffer (app_core/radio/ring_buffer.py)

Purpose: Lock-free buffer for USB jitter absorption.

Features:

• Single producer, single consumer (SPSC) design
• 1-second buffer capacity (configurable)
• Overflow/underflow detection
• Health statistics

Configuration:

# Buffer sizing
MIN_SIZE = 262144   # ~0.1s at 2.5 MHz
MAX_SIZE = 4194304  # ~1.6s at 2.5 MHz

# Default: 1 second of buffer
buffer_size = sample_rate * 1.0

3. Single-Thread Capture Loop (CURRENT IMPLEMENTATION)

Status: As of 2025-12-04, the dual-thread architecture code was removed as it was never activated.

Current Implementation: Single-threaded capture loop in _SoapySDRReceiver._capture_loop()

• Reads samples from USB
• Performs FFT for spectrum analysis
• Updates signal strength metrics
• Maintains ring buffer for USB jitter absorption
• Handles capture requests

Note: A dual-thread architecture was prototyped but never integrated. The mixin code was removed during refactoring. If needed in future, it can be re-implemented based on the single-thread foundation.

Previous Design (Not Implemented): 1. USB Reader Thread (Producer) - read from hardware 2. Processing Thread (Consumer) - FFT and analysis

4. Demod Service (services/demod/)

Purpose: FM/AM demodulation, split out of the audio service.

Why a separate process: demodulation used to run inline inside the audio service's own Redis-subscriber thread. A py-spy record --gil profile of the live process showed that thread dominating GIL-held time — several scipy.signal.oaconvolve FFT convolutions per IQ chunk (stereo pilot detection, RBDS extraction — see app_core/radio/demod/fm.py) were starving the audio service's three real-time Icecast feeder threads sharing the same interpreter, each of which needs to wake roughly every 50ms to keep its buffer fed. Moving the DSP work to its own OS process means it can never again share a GIL with anything real-time — the same reasoning that already split network/zigbee/gps/displays/gpio into their own processes (docs/architecture/SYSTEM_ARCHITECTURE.md).

Responsibilities:

• Subscribe to sdr:samples:{receiver_id} (one lightweight subscriber thread per receiver — parse JSON, hand off, nothing else)
• Own the actual FMDemodulator/AMDemodulator per receiver, each behind its own dedicated worker thread (mirrors app_core/radio/demod/rbds_worker.py's queue + single-consumer-thread pattern — a demodulator is stateful/order-dependent and must never be called from more than one thread or on out-of-order chunks)
• Publish demodulated PCM audio to demod:audio:{receiver_id} and decoder status (stereo lock, RBDS PS/PI/radiotext, ...) to demod:status:{receiver_id}
• Read per-receiver demod settings (modulation type, stereo, RBDS, de-emphasis) from the RadioReceiver table — the same source of truth this service's own hardware tuning already reads from

Runtime Requirements:

• Runs as the eas-station-demod.service systemd unit (see systemd/), peer of sdr/audio in eas-station.target (not a hardware.target member — see app_core/config/services.py's port-allocation comment)
• No USB access needed
• Redis + database connectivity (database only to read RadioReceiver rows — see services/demod/__main__.py::_discover_receiver_configs)
• No HTTP control API beyond /health on port 5106 — purely Redis-driven, same as the gpio subsystem

5. EAS Monitoring Service (eas_monitoring_service.py)

Purpose: Audio processing and EAS decoding.

Responsibilities:

• Subscribe to the demod service's demod:audio:{receiver_id} channel (app_core/audio/redis_sdr_adapter.py::RedisSDRSourceAdapter — a thin consumer now; it no longer touches a demodulator itself)
• EAS/SAME header detection
• Icecast streaming output
• Web audio streaming

Runtime Requirements:

• Runs as the eas-station-audio.service systemd unit (see systemd/).
• No USB access needed
• Redis connectivity only (publishes/subscribes on the channels listed below)

Redis Data Flow

Sample Publishing (SDR -> Demod)

Channel: sdr:samples:{receiver_id}
Format: JSON with zlib+base64 encoded samples

{
  "receiver_id": "noaa-1",
  "timestamp": 1701532800.123,
  "sample_count": 32768,
  "sample_rate": 2500000,
  "center_frequency": 162550000,
  "encoding": "zlib+base64",
  "samples": "<base64 encoded zlib compressed interleaved float32>"
}

Demodulated Audio (Demod -> Audio)

Channel: demod:audio:{receiver_id}
Format: binary -- 4-byte big-endian IQ sample rate + 4-byte big-endian
        center frequency + raw float32 PCM, already at the receiver's
        configured audio_sample_rate (no JSON/base64/zlib -- this hop is
        same-process-trusted and sits on the audio service's real-time
        feeder path, so it stays a single unpack call)

Key: demod:status:{receiver_id}  (SETEX, 10s TTL)
Value: pickle.dumps(DemodulatorStatus) -- stereo lock, RBDS PS/PI/
       radiotext/etc. See app_core/radio/demod/types.py::DemodulatorStatus.

Spectrum Data (Waterfall / Spectrum Scope)

Key: eas:spectrum:{receiver_id}    (RedisChannels.SPECTRUM_PREFIX, app_core/config/redis_config.py)
TTL: 5 seconds

{
  "identifier": "noaa-1",
  "spectrum": [0.1, 0.2, ...],  // Normalized 0-1
  "fft_size": 2048,
  "sample_rate": 2500000,
  "center_frequency": 162550000,
  "timestamp": 1701532800.123,
  "status": "available"
}

Read by /api/radio/spectrum/<id> (webapp/radio_settings/routes_signal.py), which feeds both the SDR Diagnostics page's Live Waterfall and Spectrum Scope views (they share one 500ms poll loop client-side). Falls back to a slower Redis command-queue round trip if this key has expired.

Historical Trend Archive (SDR Diagnostics "Historical Trends")

Key: sdr:trends:{receiver_id}        raw tier,  10s cadence, cap 360  (1h)
Key: sdr:trends:{receiver_id}:5m     rollup tier, 5min buckets, cap 2016 (7d)

# Each list entry (LPUSH'd newest-first) is one JSON sample/bucket:
{
  "t": 1701532800123,
  "signal_strength": -42.0,
  "locked": 1.0,                  // raw tier; rollup tier stores "locked_pct" (0-100) instead
  "sample_rate_ratio": 1.0,       // effective / configured sample rate
  "overflow_count": 0.0,          // delta since the previous sample (raw) or bucket sum (rollup)
  "underflow_count": 0.0
}

Sampled every 10s from publish_samples_and_metrics() (own wall-clock throttle, same discipline as the 100ms spectrum throttle) by app_core/radio/trends.py, modeled on services/gps/trends.py's bucket/rollup pattern but with 2 tiers instead of GPS's 4. Read by GET /api/radio/diagnostics/trends/<id>?window=1h|6h|24h|7d (webapp/radio_settings/routes_trends.py).

Health Metrics

Key: sdr:metrics
TTL: 30 seconds

{
  "service": "sdr_service",
  "timestamp": 1701532800.123,
  "pid": 12345,
  "receivers": {
    "noaa-1": {
      "running": true,
      "locked": true,
      "signal_strength": 0.42,
      "frequency_hz": 162550000,
      "sample_rate": 2500000,
      "ring_buffer": {
        "fill_percentage": 25.5,
        "overflow_count": 0
      }
    }
  }
}

Control Commands

Queue: sdr:commands (LPUSH/LPOP)

{
  "action": "restart",  // restart, stop, start
  "receiver_id": "noaa-1",
  "command_id": "cmd-12345"
}

Result: sdr:command_result:{command_id}
TTL: 30 seconds

{
  "command_id": "cmd-12345",
  "success": true,
  "message": "Receiver noaa-1 restarted"
}

Benefits of Separation

Fault Isolation

• SDR crashes don't affect audio processing
• Audio crashes don't affect SDR streaming
• Either service can be restarted independently

Security

• USB privileges isolated to the SDR service only
• Audio processing runs with minimal privileges
• Reduced attack surface

Performance

• USB reading never blocked by FFT or encoding
• Ring buffer absorbs USB latency jitter
• Processing can run on different CPU cores

Scalability

• SDR service can run on dedicated hardware
• Audio processing can be distributed
• Multiple audio consumers can subscribe

Configuration

Environment Variables

# Redis Connection
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0

# Database Connection
DATABASE_URL=postgresql+psycopg2://eas_station:<secure_password>@127.0.0.1:5432/alerts

# Application Config (defaults to /opt/eas-station/.env)
CONFIG_PATH=/opt/eas-station/.env

systemd Files

Hang Watchdog

eas-station-audio.service (and separately, eas-station-poller.service) run as Type=notify units with WatchdogSec= set. Restart=always already restarts a crashed process; the watchdog additionally catches a process that is still running but deadlocked (e.g. stuck in a blocking call with no exception raised). Both services send WATCHDOG=1 heartbeats from their main loop via app_utils/system/sd_notify.py — if the loop stops ticking, systemd kills and restarts the unit even though the process never crashed. No configuration is needed; this is transparent to normal operation and to journalctl -u eas-station-audio.service.

Troubleshooting

SDR Service Not Starting

1. Check USB device access:

   lsusb | grep -E "RTL|Airspy|Realtek"
   

2. Check SoapySDR detection:

   SoapySDRUtil --find
   

3. Check service logs:

   sudo journalctl -u eas-station-sdr -n 100 --no-pager
   

Buffer Overflows

Symptoms: "Ring buffer overflow" warnings in logs

Causes:

• Processing thread too slow
• Insufficient CPU
• High sample rate without adequate resources

Solutions:

• Reduce sample rate
• Increase ring buffer size
• Use faster hardware
• Check for CPU throttling

No Spectrum Data

1. Check Redis connectivity:

   redis-cli ping
   

2. Check spectrum key:

   redis-cli keys 'eas:spectrum:*'
   

3. Run the full diagnostics checklist (heartbeat, per-receiver health, ring-buffer drops, spectrum-cache freshness — also available from the web UI's "Run Full Diagnostics" button on /admin/radio/diagnostics):

   python3 scripts/diagnostics/check_sdr_status.py
   

Monitoring

Health Check Endpoints

The SDR service publishes a heartbeat to Redis:

# Check heartbeat
redis-cli get sdr:heartbeat

# Expected output:
{"timestamp": 1701532800.123, "pid": 12345, "receiver_count": 1}

Ring Buffer Statistics

# Check ring buffer health (logged periodically by the SDR service)
sudo journalctl -u eas-station-sdr -n 200 --no-pager | grep -i "ring buffer"

# Expected fields:
# fill_percentage: 25.5
# overflow_count: 0
# underflow_count: 0
# total_samples_written: 1234567890

Service Health

# Check all EAS Station services
systemctl list-units 'eas-station-*' --no-pager

# Check SDR service specifically
systemctl status eas-station-sdr

Performance Tuning

Buffer Sizes

For Airspy R2 at 2.5 MHz:

# USB read buffer: 50ms of samples
read_buffer = int(2_500_000 * 0.050)  # 125,000 samples

# Ring buffer: 1 second of samples  
ring_buffer = int(2_500_000 * 1.0)    # 2,500,000 samples

CPU Affinity

This is a bare-metal systemd deployment, not a container — pin CPUs via the unit's drop-in override rather than a compose file:

# /etc/systemd/system/eas-station-sdr.service.d/override.conf
[Service]
CPUAffinity=0 1

Apply with systemctl daemon-reload && systemctl restart eas-station-sdr.

Memory

Likewise, use MemoryMax= in a unit override instead of a container memory limit:

# /etc/systemd/system/eas-station-sdr.service.d/override.conf
[Service]
MemoryMax=1G

Icecast Streaming Architecture

Overview

After SDR samples are demodulated to PCM audio, they are streamed to Icecast for network distribution. The streaming pipeline uses FFmpeg to encode audio (MP3/OGG) and push to Icecast server.

CRITICAL: The FFmpeg -re flag behavior is source-dependent and must be configured correctly to prevent stalling or incorrect resampling.

FFmpeg -re Flag: Source-Specific Behavior

The -re flag in FFmpeg means "read input at native frame rate" and is designed for file playback simulation. Its use depends entirely on the audio source type:

SDR Sources (Live Hardware Capture)

DO NOT use -re flag

• Why: Audio is already captured in real-time by SDR hardware
• Problem if used: Creates fatal backpressure in the pipe buffer
  • FFmpeg throttles stdin reads to exactly real-time rate (e.g., 44.1kHz)
  • Audio chunks arrive faster than FFmpeg consumes them
  • Pipe buffer (64KB) fills up in <1 second
  • stdin.write() blocks, freezing the feed loop
  • Audio queue fills, stream stalls completely after 5-6 seconds
• Symptom: "Buffering..." message in player, never recovers
• Solution: Remove -re flag, let FFmpeg consume stdin as fast as available

Flow without -re (CORRECT for SDR):

SDR Hardware → IQ Samples → Demodulator → PCM Audio → 
  Feed Loop → FFmpeg stdin → Encoder → Icecast
  (no throttling, natural buffer pace)

HTTP/Stream Sources (Network Streams)

DO use -re flag

• Why: Remote streams need throttling for correct resampling
• Problem if omitted: FFmpeg processes too fast, resampling is incorrect
  • Network stream arrives at network speed (can be faster than real-time)
  • Without -re, FFmpeg decodes/resamples at maximum CPU speed
  • Timing relationships are lost, resampling produces wrong output
• Solution: Use -re flag to maintain proper timing

Flow with -re (CORRECT for HTTP streams):

HTTP Stream → FFmpeg (with -re) → Decode → Resample → 
  PCM Audio → Feed Loop → FFmpeg stdin → Encoder → Icecast
  (throttled to real-time, correct resampling)

Implementation

The conditional logic in app_core/audio/icecast_output.py:

def _start_ffmpeg(self) -> bool:
    # Determine if -re flag should be used based on source type
    use_re_flag = False
    source_type_name = type(self.audio_source).__name__
    
    # Network stream sources NEED -re flag
    if source_type_name in ('StreamSourceAdapter', 'IcecastIngestSource', 'HTTPIngestSource'):
        use_re_flag = True
        logger.debug(f"Using -re flag for {source_type_name} (network stream)")
    
    # SDR sources must NOT use -re flag
    elif 'SDR' in source_type_name or 'sdr' in source_type_name.lower():
        use_re_flag = False
        logger.debug(f"NOT using -re flag for {source_type_name} (live hardware)")
    
    # Fallback: check AudioSourceConfig.source_type enum
    elif hasattr(self.audio_source, 'config'):
        from .ingest import AudioSourceType
        config = self.audio_source.config
        if hasattr(config, 'source_type'):
            if config.source_type == AudioSourceType.SDR:
                use_re_flag = False
            elif config.source_type == AudioSourceType.STREAM:
                use_re_flag = True
    
    # Build FFmpeg command with conditional -re flag
    cmd = ['ffmpeg']
    if use_re_flag:
        cmd.append('-re')
    cmd.extend(['-f', 's16le', '-ar', str(sample_rate), ...])

Source Type Detection

Priority order:

1. Class name pattern matching:

   • StreamSourceAdapter → use -re
   • RedisSDRSourceAdapter → no -re
   • IcecastIngestSource → use -re
   • HTTPIngestSource → use -re

2. AudioSourceConfig.source_type enum:

   • AudioSourceType.STREAM → use -re
   • AudioSourceType.SDR → no -re

3. Default fallback: No -re (safer, prevents stalling)

Buffer Architecture

Understanding the buffer chain helps diagnose issues:

Troubleshooting Streaming Issues

Symptom: Stalling after 5-6 seconds

Diagnosis: -re flag on SDR source

# Check source type
grep "Using -re flag\|NOT using -re" /var/log/eas-station/audio-service.log

# Should see:
# "NOT using -re flag for RedisSDRSourceAdapter (live hardware)"

Fix: Ensure conditional logic detects SDR source correctly

Symptom: Incorrect resampling on HTTP streams

Diagnosis: Missing -re flag on network stream

# Check source type
grep "Using -re flag\|NOT using -re" /var/log/eas-station/audio-service.log

# Should see:
# "Using -re flag for StreamSourceAdapter (network stream)"

Fix: Ensure conditional logic detects stream source correctly

Symptom: Buffer overflow warnings

# Check buffer health
redis-cli GET "sdr:ring_buffer:{receiver_id}"

# Look for:
# "fill_percentage": >80%
# "overflow_count": >0

Causes:

• Downstream processing too slow
• Network congestion (Icecast streaming)
• CPU throttling

Solutions:

• Check network bandwidth
• Monitor CPU usage
• Reduce number of concurrent streams
• Increase buffer sizes

Performance Considerations

CPU Usage

• Without -re: FFmpeg encodes as fast as possible

  • Higher burst CPU usage
  • Lower average CPU (finishes encoding faster)
  • Better for SDR (no blocking)

• With -re: FFmpeg throttles to real-time

  • Steady CPU usage
  • Slightly higher average CPU
  • Required for HTTP streams (correct resampling)

Network Bandwidth

• Each Icecast stream: ~128kbps (MP3) or ~64-96kbps (OGG)
• Multiple SDR receivers = multiple streams
• Consider bandwidth limits on shared networks

Memory Usage

• Each IcecastStreamer: ~100MB peak
• Buffer memory: ~50MB per stream
• Monitor with: systemctl status eas-station-sdr (see MemoryCurrent) or ps -o rss,cmd -C python3

Best Practices

1. Always check logs for -re flag usage during startup
2. Test SDR streams for >60 seconds continuously
3. Verify HTTP stream audio quality after any changes
4. Monitor buffer health via Redis metrics
5. Use appropriate bitrates (128kbps for MP3, 96kbps for OGG)
6. Limit concurrent streams based on available resources

Related Files

• app_core/audio/icecast_output.py - FFmpeg streaming logic
• app_core/audio/redis_sdr_adapter.py - SDR demodulation
• app_core/audio/sources.py - HTTP stream sources
• app_core/audio/auto_streaming.py - Stream management
• docs/audio/AUDIO_MONITORING.md - Audio monitoring guide

Version History

• v2.42.5: Removed -re flag (broke HTTP streams)
• v2.42.6: Added conditional -re flag based on source type (current)