Changelog

All notable changes to this project are documented in this file. The format is based on Keep a Changelog.

[Unreleased]

• Nothing yet. Document changes here as they land; the next release cut moves them into a version heading.

[3.20.2] - 2026-09-19 - Add a "Re-sync Nginx Configuration" action to the Certbot admin page

Added

• A persistent "Re-sync Nginx Configuration" button on Settings -> Certbot, found missing while rolling out the v3.20.0 OCSP stapling fix to a production deployment: update.sh pulls new code and reloads nginx, but it doesn't re-run certificate installation -- the step that (re)writes /etc/nginx/snippets/ssl-letsencrypt.conf. The only existing UI path to re-run that step (the "Install Certificate Now" button) is conditionally rendered and only appears when a certificate has been obtained but never installed; once a certificate is already active -- the normal, common state -- there was no way through the UI to make it re-apply its nginx wiring, so an app update that adds a new SSL directive to the certbot-managed snippet (like ssl_trusted_certificate, needed for stapling) had no way to actually reach an existing installation without hand-editing nginx config. The new button reuses the same existing, idempotent /admin/api/certbot/install-certificate route -- no new certificate is requested, no Let's Encrypt/ACME call is made, it just rewrites the local nginx snippet for the currently-installed certificate and reloads nginx.

[3.20.1] - 2026-09-19 - Add HSTS preload flag to the nginx template

Added

• preload added to the Strict-Transport-Security header in both HTTPS server blocks of config/nginx-eas-station.conf (max-age=31536000; includeSubDomains -> max-age=31536000; includeSubDomains; preload), following up on the same Qualys SSL Labs report that prompted the OCSP stapling fix (v3.20.0). Deliberately deferred in that earlier change since preload is close to permanent once a domain is accepted onto browsers' built-in list -- removal can take months to propagate to already-shipped browsers. This header change alone does not enroll any domain; that's still a separate, manual, one-time step the domain's owner takes at https://hstspreload.org/ once this config is live and serving HTTPS-only on every subdomain.

[3.20.0] - 2026-09-18 - Add OCSP stapling support to the nginx template

Added

• OCSP stapling, prompted by a Qualys SSL Labs report on a production deployment (easstation.com) showing "OCSP stapling: No" as the one real gap in an otherwise A+ rating. config/nginx-eas-station.conf now sets ssl_stapling on; ssl_stapling_verify on; plus a resolver in both HTTPS server blocks (443 main site, 8081 pgweb proxy) -- nginx attaches the CA's cached, signed "still valid" response to the TLS handshake itself, so clients skip their own live OCSP query to the CA (faster handshake; the CA no longer sees every visitor's browsing habits via those individual per-visitor lookups).
• webapp/admin/certbot/install.py's nginx snippet writer now also writes ssl_trusted_certificate (from the issued cert's chain.pem, alongside the fullchain.pem/privkey.pem it already wrote) -- stapling verification needs the issuer chain to check the OCSP response against. Falls back to omitting the line (stapling silently stays inactive, nothing else breaks) if chain.pem is missing, since certbot always produces one but an unusual manually-imported cert might not.
• No-op and harmless for the default self-signed certificate (confirmed via nginx -t: a clean "ssl_stapling ignored, issuer certificate not found" warning, not an error) -- self-signed certs have no real CA-backed OCSP responder to staple, so stapling only actually activates once a real Let's Encrypt certificate is installed through the existing certbot flow.
• Investigated the same report's "fatal handshake" rows (a handful of pre-2014 clients: Windows XP-era IE/Chrome/Firefox, old Java, iOS 7/8 Safari) and the "HSTS preload not opted in" note -- both are the correct, intentional trade-off of the existing TLSv1.2/TLSv1.3-only configuration (re-enabling legacy protocols to satisfy those ~15-20-year-old clients would drop the rating well below A+ and reopen real vulnerabilities for effectively zero real-world traffic), and HSTS preload submission is a separate, much harder-to-reverse decision left for a deliberate future call rather than bundled into this fix. Neither needed a code change.

[3.19.2] - 2026-09-18 - Fix: dead "Service Heartbeat Status" card on the Uptime Monitoring page

Fixed

• The "Service Heartbeat Status" card (Settings -> Uptime Monitoring, right column) always showed "Not created" / "Never," regardless of whether Tickstem's per-service heartbeats were actually pinging successfully. Root cause: it read settings.service_heartbeat_id, settings.service_heartbeat_status, settings.last_service_heartbeat_at/_success/_error -- none of which exist on the TickstemSettings model, and none of which any route ever sets. Jinja2 silently renders a missing attribute as falsy rather than raising, so the card has shown this same dead placeholder state since it was added in #2557, unrelated to anything in this session's healthchecks.io work. The real, working per-service heartbeat data (both Tickstem's and healthchecks.io's) is already shown in the "Per-Service Heartbeats" table directly above this card -- with per-row status and last-ping timestamps that do reflect reality -- so removed the dead card rather than inventing a new single-value aggregate concept that has no well-defined meaning across N independently-scheduled per-service heartbeats.

[3.19.1] - 2026-09-18 - Fix: outbound API calls could hang for ~60s per request on a black-holed IPv6 path

Fixed

• healthchecks.io's per-service heartbeat bulk-create ("Monitor Selected Services") appeared to hang indefinitely instead of completing in under a second. Root-caused on the live deployment: the host has a valid global IPv6 address and default route, but that path is silently black-holed to multiple unrelated destinations (confirmed via ping6/curl -6 timeouts against healthchecks.io, Tickstem's API, Google's public DNS, and Cloudflare alike -- not specific to any one of them). socket.create_connection() tries getaddrinfo()'s results in order, and RFC 6724 sorts IPv6 first, so every outbound requests call hung for a full connect-timeout on the dead IPv6 address before falling back to the IPv4 address that actually works -- turning what should be a sub-second healthchecks.io API call into one that took about 60 seconds. The per-service bulk-create endpoint makes one such call per selected service (up to 12) in a single synchronous request, so the page's "Creating heartbeats..." status could sit for over ten minutes without any visible sign it was still working, rather than the seconds it should take.
• Added app_core.http_defaults.prefer_ipv4_for_outbound_requests(), called once at app.py import time, which makes requests/urllib3 skip AAAA lookups for the whole process. Verified against the live failure: the same healthchecks.io call went from ~60s to ~0.5s, both with and without gevent's socket monkey-patching (gunicorn's --worker-class gevent). This is a resilience fix independent of whatever is actually black-holing this host's IPv6 path (a router/upstream issue, not something in this codebase) -- every outbound integration this station makes works fine over IPv4 alone.

[3.19.0] - 2026-09-18 - Add healthchecks.io per-service heartbeat integration

Added

• healthchecks.io Management API v3 support (app_core/healthchecks_client.py), alongside the existing Tickstem integration, for outbound per-service dead-man's-switch monitoring. Requested: "Can we add healthchecks.io API support along with Tickstem?" One heartbeat check per app_core.config.get_eas_services() entry (the same 11 EAS subsystems + poller Tickstem's per-service heartbeats already cover), each pinged by the shared HeartbeatWorker background thread only while that specific systemd service is actually running — so a missed ping on healthchecks.io's side identifies exactly which subsystem stalled, not just "something is down." New HealthchecksSettings (encrypted account API key) and HealthchecksServiceHeartbeat (per-service check UUID/ping URL/interval/status) models, migration 20260918_healthchecks_settings.
• No new page. Per explicit direction ("No separate page though in the settings, update the tickstem page, maybe rename it") the integration has no standalone settings page or nav entry — webapp/admin/healthchecks.py is API-routes-only. The existing Tickstem settings page (webapp/admin/tickstem.py, templates/admin/tickstem.html) was renamed "Uptime Monitoring" and now renders both providers' settings and per-service heartbeat tables in one place, since both are the same kind of thing (outbound dead-man's-switch heartbeats) differing only in which third-party API they call.
• Bulk "create all" endpoint (/admin/healthchecks/service-heartbeats/create-all) stops immediately on an HTTP 403 (healthchecks.io's plan-quota-exhausted response) instead of retrying the same failure for every remaining service, and reports which services it did create before hitting the limit.
• tests/test_healthchecks_service_heartbeats.py — 15 tests covering the worker's active-and-due gating logic (mirroring the existing test_tickstem_service_heartbeats.py shape, since both providers share HeartbeatWorker._ping_one_service_heartbeat()), the API client's request/error-handling shape, and the bulk-create route's subset/quota-stop/missing-key behavior.

[3.18.3] - 2026-09-18 - Fix: duplicate security headers silently disabled HSTS enforcement

Fixed

• Strict-Transport-Security, X-Frame-Options, X-Content-Type-Options, and X-XSS-Protection were set in both app.py (Flask's after_request hook) and config/nginx-eas-station.conf (add_header ... always;), with nginx's add_header unaware of and not deduplicating against whatever the proxied Flask response already set. Confirmed live via curl -I against the running deployment: the HTTPS response carried two separate strict-transport-security header fields with different max-age values (63072000 from Flask, 31536000 from nginx). Per RFC 6797 §8.1, a browser that receives more than one Strict-Transport-Security header field is required to ignore all of them — so despite looking configured in two places, HSTS was not actually being enforced by any compliant browser. Removed the four duplicated headers from app.py; nginx — the actual TLS-terminating edge — is now the single source of truth for all of them. Content-Security-Policy stays in app.py since nginx never set it and its value is dynamic (depends on the configured Icecast origin, resolved per-request).
• Regression test (tests/test_security_headers.py) added to tests/known_failures.txt as a documented xfail, matching the existing test_support_smoke.py/test_upload_too_large_handler.py entries: the app_client fixture forces DATABASE_URL to sqlite unconditionally, and any real request through it hits db.create_all() against a JSONB column sqlite can't compile. Verified manually with live curl instead, the same way those existing entries document doing.

[3.18.2] - 2026-09-18 - Fix: missing dead-air around the pre/post MDC1200 chime burst

Fixed

• The 1-second dead-air gap that should bracket the pre-alert and post-alert chime/MDC1200 burst was only present on the inner side, not the outer side. EASAudioGenerator.build_files() and .build_manual_components() (app_utils/eas/generator.py) both play an optional chime — which can be an MDC1200 selective-calling packet when pre_alert_chime/post_alert_chime is set to mdc1200 — immediately before the first SAME header burst and immediately after the EOM sequence. The gap between that chime and the header/EOM it brackets already existed; the gap before the pre-chime burst (composite audio started the MDC1200 packet at sample 0) and after the post-chime burst (composite audio stopped dead at the end of the packet, no trailing silence at all) did not. Reported as "still not getting the second of dead air before the MDC1200 preceding the header... [and] the 1 second of dead air after the mdc1200 after the EOM." Added the missing leading and trailing 1-second silence in both methods, only when a chime is actually configured (unchanged, still zero added silence, when pre_alert_chime/post_alert_chime is none).

[3.18.1] - 2026-09-18 - Fix: alert detail map and per-service coverage percentages weren't actually alert-scoped

Two related bugs found from a user report ("the affected boundaries... is confusing and doesn't work the way it should" / "the percentages... make no sense") against a live alert detail page.

Fixed

• The alert detail map's boundary layer toggles ("Counties," "Fire Districts," "Villages," etc., labeled "Affected Boundaries" in the legend) weren't scoped to the alert at all. loadBoundaries() called /api/boundaries?type=X, the same unscoped endpoint the dashboard map uses, so toggling e.g. "Villages" drew every village in the configured service area — not just the ones this alert actually intersects (the sidebar's correct "22 of 126 affected" count comes from the same Intersection rows the map ignored). Added an optional alert_id query param to GET /api/boundaries that joins Intersection when present; the map now passes the current alert's ID. Every other caller (dashboard, admin boundary management) is unaffected since the param is optional.
• The per-service-type coverage percentages ("Villages: 27.3%", "Fire: 73.8%", ...) answered a different question than the county-level percentage (74.0%) displayed right next to them, with no indication of the difference. calculate_coverage_percentages() scoped each per-type percentage's denominator to only the boundaries already flagged as affected — i.e. "of the villages that got touched at all, how much of their own combined area is covered" — rather than "how much of the county's villages are covered," which is what the adjacent 74.0% figure (and the badge styling implying a severity grading) led a reader to expect. Rescoped the denominator to every boundary of that type within the configured county's polygon (resolved via the same county-boundary lookup the county-level percentage already used, moved earlier in the function so both blocks can share it), making the two percentages directly comparable. Falls back to the old, narrower scoping when the county polygon can't be resolved. Deliberately does not scope to "every boundary of that type this deployment has ever uploaded" — Boundary rows carry no county field and a deployment commonly holds a neighbouring county's fire districts/villages/etc. too, which is exactly the "misleadingly low percentage" bug tests/test_coverage_and_signature.py::TestCoverageCalculationLogic was written to prevent (fixed for the county-level figure previously; this fix brings the per-type figures in line with the same principle rather than reintroducing that bug at a different scope).

[3.18.0] - 2026-09-18 - Refactor: extract poller/cap_poller.py's CAP-geometry collaborator (Large File Refactor Plan, Phase 4c continuation)

The first slice of CAPPoller's 50 remaining stateful methods (the highest-risk item left in the plan) to actually land, using the 2e characterization-harness technique: pin behavior with tests written against the pre-extraction bound methods first, extract, then verify the same tests pass against the extracted free functions.

Changed

• Extracted 12 methods (plus the module-level _serialize_alert_for_sig helper and the MESSAGE_TYPE_PRIORITIES class constant) into poller/cap_geometry.py (664 lines): _parse_ipaws_xml_feed, _convert_cap_alert, _extract_cap_resources, _extract_area_details, _parse_cap_polygon, _parse_cap_circle, _approximate_circle_polygon, _message_type_priority, _alert_sort_key, _should_replace_alert, parse_cap_alert, _count_vertices. All 12 only ever touched self.logger (set once in __init__, never reassigned) or each other — never self.db_session or the poller's zone/SAME-code configuration — making them a genuinely low-risk collaborator, unlike the 50 methods that remain.
• logger is threaded through as an explicit parameter rather than a fresh per-module logging.getLogger(__name__), specifically to avoid the module-level-logger hazard docs/development/AGENTS.md documents from Phase 3e (a new logger here would silently rename every log record from poller.cap_poller to poller.cap_geometry). cap_poller.py's 4 remaining call sites pass self.logger explicitly.
• 59 characterization tests written against the pre-extraction bound methods first (tests/test_cap_geometry.py, superseding a since-deleted test_cap_geometry_characterization.py), 2 confirmed load-bearing via targeted mutation spot-checks (both caught immediately) before any code moved. Retargeted 3 existing test files (test_ipaws_event_code_extraction.py, test_cap_poller_batching.py, test_cap_poller_per_item_isolation.py) off the now-removed bound methods; one of those (test_parse_ipaws_xml_feed_one_malformed_alert_does_not_drop_the_others) needed a real monkeypatch retarget to intercept _parse_ipaws_xml_feed's internal same-module call to _convert_cap_alert, since an instance-attribute patch no longer has anything to intercept once both live in cap_geometry.py as free functions.
• cap_poller.py: 4800 → 4244 lines. CAPPoller itself is now ~3,183 of those lines across the remaining 38 methods — still the actual Phase 4c work, not started.
• Full suite green: 3426 passed (was 3367), 0 failures.

[3.17.0] - 2026-09-18 - Refactor: split app_core/eas_storage.py into a package (Large File Refactor Plan)

Profiling confirmed eas_storage.py (2,825 lines, 57 top-level functions) has the same pure-motion shape as system.py (4a) and eas.py (4b) — mostly independent functions across ~10 topics, not a god-class — unlike poller/cap_poller.py (4c). app_core/eas_storage/, sdr_hardware_service.py and eas_monitoring_service.py were the three remaining unprofiled files the plan flagged; this closes out the first of the three (the other two are dominated by a single ~900-1100-line function each and need the 2e characterization-harness technique, not this technique).

Changed

• Split into 14 modules by topic: audio_decode_log.py, file_cache.py, schema_migrations.py, backfill.py, delivery_records.py, delivery_trends.py, compliance_parsing.py, compliance_log.py, compliance_export.py, reports_common.py, reports_received_initiated.py, reports_summary.py, reports_export.py, precedence.py, plus a __init__.py shim re-exporting all 40 public names (and the format_local_datetime/utc_now pass-throughs some callers import from this module) exactly as before.
• All 69 top-level definitions (57 functions + 12 constants, plus the PrecedenceLevel/PRECEDENCE_AVAILABLE try/except import block) verified ast.dump()-identical to their originals — no normalization needed this time, since nothing moved out of a class (unlike 4a-ii/4c's self-stripping).
• Kept collect_compliance_log_entries and collect_compliance_dashboard_data in the same module (compliance_log.py) specifically because the latter calls the former as a same-module bare name — the same internal-call hazard shape as eas.py's build_same_header/clear_broadcast_active, caught this time by tracing the call graph before laying out modules rather than after.
• Dropped one confirmed-dead import (ORIGINATOR_DESCRIPTIONS from app_utils.eas, imported but never referenced anywhere in the original file).
• compliance_log.py lands at 405 lines, a negligible, deliberate overage over the 400-line guideline to keep the internal-call pair above together rather than split them across files.
• Full suite green: 3367 passed, 0 failures — identical pass count to before the split.

[3.16.1] - 2026-09-18 - Fix: lightweight Flask bootstrap for CLI/timer scripts (GitHub issue #2581)

Fixed

• scripts/ingest_security_perimeter_log.py booted the entire Flask app just to tail a log and insert a few DB rows. create_app() imports ~260 routes and initializes every subsystem (TTS, Icecast, hardware proxies, ...) regardless of what the caller needs, and this script runs every 2 minutes forever via security-perimeter-ingest.timer — a Nice=10/IOSchedulingClass=idle mitigation (see that unit's own comment, now updated) kept the ~6s-per-run CPU burst from starving the real-time SDR/demod/SAME-decode path, but the underlying cost was still paid every run. Measured 7.2s → 1.8s per invocation after the fix (639 registered routes → 1, Flask's own default static endpoint).

Added

• app_core/minimal_app.py: a create_minimal_app() helper that builds a bare Flask app bound only to the shared db extension (app_core.extensions.db) — no route registration, no subsystem init, no schema-migration sweep — for standalone scripts that only ever touch the ORM. Loads environment variables the same way app.py does (CONFIG_PATH or the default .env, both override=True), and sizes the SQLAlchemy engine pool for a one-shot process (pool_size=1) instead of app.py's two-gunicorn-worker sizing (pool_size=10). tests/test_minimal_app.py covers the DATABASE_URL-required contract, the zero-routes guarantee, and the sqlite/postgres engine-option split.
• Points scripts/ingest_security_perimeter_log.py, scripts/fix_admin_roles.py, and scripts/create_example_screens.py at the new bootstrap instead of app.py's create_app() — all three only ever needed db.session and ORM models, confirmed by tracing their imports (app_core.analytics.security_blocks, app_core.auth.roles, app_core.models) back to see none of them touch app.py.

[3.15.0] - 2026-09-18 - GitHub repo hygiene: fix the Release workflow, add SECURITY.md/CODEOWNERS/templates

Fixed

• .github/workflows/release.yml's "Validate release metadata" step was broken and had been since 2026-09-17, silently: it only ever installed bare pytest, not the project's own dependencies. tests/conftest.py's autouse _isolate_eas_stream_injection fixture (added that day, see the 3.10.3 entry above) imports app_core.audio.redis_commands for every test unconditionally, which needs Flask/SQLAlchemy/etc. importable even for tests/test_release_metadata.py's own dependency-light tests. Manually re-running the workflow (last run: 2026-08-31, v2.207.4 — three weeks and many version bumps stale) surfaced the failure immediately. Fixed by installing requirements.txt before running pytest, matching tests.yml's pattern; no service containers needed since nothing in this step does real DB/Redis I/O.
• Re-ran the Release workflow after the fix to catch main up to the current VERSION.

Added

• Branch protection on main: CodeQL, analyze, lint, and pytest (3.13) must all pass before a PR can merge; force-pushes and branch deletion are blocked; open PR conversations must be resolved. No required-review rule, so a maintainer (or an agent working under one) can still merge once checks are green.
• allow_auto_merge enabled at the repo level, and delete_branch_on_merge enabled — a PR can now be set to merge itself the moment checks pass, and its branch is deleted automatically afterward.
• Private vulnerability reporting enabled, with .github/SECURITY.md documenting how to use it, what's in scope, and the (rolling-release, latest-only) support policy.
• .github/CODEOWNERS: a single global rule for now (one-maintainer repo); add path-specific rules as that changes.
• .github/PULL_REQUEST_TEMPLATE.md: prompts for a summary, test plan, and the release/versioning checklist docs/development/AGENTS.md §9 already requires of every change.
• .github/ISSUE_TEMPLATE/: structured bug-report and feature-request forms (GitHub's YAML issue-forms format), plus a config.yml disabling blank issues and redirecting security reports to private vulnerability reporting instead.

[3.16.0] - 2026-09-18 - Refactor: extract poller/cap_poller.py's stateless methods (Large File Refactor Plan, Phase 4c partial)

Profiling poller/cap_poller.py (4,933 lines) confirmed it really is the god-class the plan originally assumed for Phase 4 — unlike system.py and eas.py, both of which turned out to be mostly independent functions once actually profiled. CAPPoller is ~3,978 of the file's lines across 59 methods.

Changed

• Extracted CAPPoller's 9 stateless methods (zero self references, 143 lines) into poller/cap_alert_parsing.py: _select_cap_info, _extract_cap_event_codes, _extract_cap_parameters, _summarise_geometry, _apply_cancellation_status, _validate_ugc_code, _normalize_same_code, _coords_equal, _safe_json_copy. Rewrote all 19 internal self.method(...) call sites to bare function calls. cap_poller.py: 4933 → 4800 lines — still far over the guidance; the other 50 methods (3,711 lines) are the actual remaining work, not started, and are now the highest-risk item left in this plan.
• All 9 moved functions verified ast.dump()-identical to their originals (after normalizing away self and one level of docstring indentation — the same two normalizations Phase 2d needed for GPSManager's stateless methods).
• Retargeted tests/test_ipaws_event_code_extraction.py's TestExtractCapEventCodes off a live CAPPoller instance (object.__new__(CAPPoller), built solely to reach the now-moved method) to call _extract_cap_event_codes directly.
• Full test suite green.

Fixed (tooling)

• ruff (not available in this environment by default) installed into a scratch venv for a real lint pass, catching two F821s that py_compile and even a real import of the new module had both stayed silent about: ET.Element/CAPAlert type hints relying on from __future__ import annotations to dodge real imports, which defers evaluation but doesn't exempt a name from needing to resolve. Fixed by importing ET the same way cap_poller.py itself does (get_element_tree_module()) and CAPAlert from app_core.models.

🔒 GitHub's PR-scoped CodeQL analysis flagged 3 "new" alerts on the app_utils/eas.py package PR that merged just before this one — a polynomial-regex pattern and a log-injection pattern in files that split out of eas.py, and a stack-trace-exposure pattern in webapp/admin/pending_alerts.py, a file that PR never touched at all. All three confirmed pre-existing (byte-identical or ast.dump()-identical to code already on main) rather than fixed — a security-flagged regex or log call is a behaviour change and doesn't belong in a pure-motion refactor commit. main has no branch protection requiring CodeQL to pass and already carries ~100 open alerts of the same rule categories elsewhere in the tree. Noted here for visibility, not silently dropped.

[3.14.0] - 2026-09-18 - Refactor: split app_utils/eas.py into a package (Large File Refactor Plan, Phase 4)

The single largest file in the tree at the time (4,246 lines, grown from the 3,848 lines the plan was originally scoped against). Unlike the plan's original note calling this "a very large class," the file had actually grown into 48 mostly-independent top-level functions plus two large classes (EASAudioGenerator, EASBroadcaster) — the 2a/2b pure-motion shape, not the god-class shape Phase 4 was expected to need.

Changed

• Split into 14 topic modules under app_utils/eas/: indicators.py (Redis-backed broadcast/incoming-alert state), config.py (load_eas_config), same_header_constants.py, same_header_decode.py, same_header_build.py, tts_normalize.py, tts_compose.py, tone_generation.py, chime.py (split out separately to stay under the line guidance), wav_io.py, broadcast_pid.py, audio_conversion.py, generator.py (EASAudioGenerator), broadcaster.py (EASBroadcaster). app_utils/eas.py is now a package; the old single-file module is gone, and app_utils/eas/__init__.py is the compatibility shim re-exporting every name external code ever imported from it.
• All 48 top-level functions/classes and all 32 module-level constants are ast.dump()-identical before and after — pure motion, not restructuring.
• 12 of 14 modules are within the 400-line guidance. generator.py (848, EASAudioGenerator) and broadcaster.py (489, EASBroadcaster) are known exceptions — each is one god-class; bringing them under the cap needs collaborators extracted from the class bodies, a behavioural change requiring its own characterization pass, tracked as a follow-up. config.py (408) is dominated by one 369-line function (load_eas_config) for the same reason.
• A confirmed internal-cross-call hazard was found and fixed before landing: EASBroadcaster.handle_alert() calls build_same_header() and clear_broadcast_active() as same-module bare names today; tests/test_gpio_centralized_keying.py monkeypatches both at the app_utils.eas module level expecting to intercept that internal call. Once EASBroadcaster moved to its own module importing those by value, the patch would have silently stopped reaching the call — confirmed by temporarily reverting the retarget and watching the test fail with KeyError: 'present' (the recording wrapper never got called). Retargeted to eas.broadcaster.build_same_header / eas.broadcaster.clear_broadcast_active.
• subprocess and time are re-exported from the package __init__ (import subprocess, import time) so tests/test_gpio_dump_broadcast.py's eas_module.subprocess.Popen patch and tests/test_airchain_fringe_cases.py's patch('app_utils.eas.time.sleep', ...) continue to resolve — both are shared singleton stdlib modules, so patching either one's attributes affects every module that imports it, regardless of which one the patch is aimed at.
• Verified against the full test suite: 3,362 passed, 282 skipped, 62 xfailed, 9 xpassed, 0 failures. 37 distinct names imported from app_utils.eas across the whole tree (production + tests) were enumerated first and confirmed to still resolve; every production consumer (app_core.audio.*, poller.cap_poller, scripts.*, services.gpio.alert_indicators, webapp.*) was imported directly to confirm the shim resolves.
• grep -n "__file__" across all 14 new modules is empty — no path-depth hazard.

[3.13.0] - 2026-09-18 - Refactor: split webapp/admin/certbot/routes_obtain_execute.py (Large File Refactor Plan, Phase 3e-ii)

obtain_certificate_execute was one 387-line try block — the one module Phase 3e left over the 400-line guidance, and with zero test coverage beforehand.

Changed

• Extracted pre-flight validation into obtain_validation.py (_validate_obtain_request, _check_certbot_installed) and the three certbot methods (standalone, nginx plugin, webroot) into obtain_methods.py. routes_obtain_execute.py is now 99 lines: parse the request, validate, check prerequisites, handle a staging→production cert switch, and dispatch to one method via a small lookup table.
• The route's public behavior, URL, and response shapes are unchanged.
• Added tests/test_certbot_obtain_execute.py: 28 characterization tests written and run green against the pre-refactor handler first (the module's first-ever test coverage), reaching the handler through __wrapped__ to bypass the permission decorator rather than standing up a full authenticated test client. A 20-mutation sweep across all three new/changed files confirmed the suite is discriminating — all caught after two isolation fixes (an assertion that matched raw pre-augmentation error text as readily as the augmented message, and a missing test for the webroot method's own permission-denied augmentation branch).
• Updated tests/test_certbot_package.py's size-guidance guard: routes_obtain_execute.py is no longer a known exception.

[3.12.0] - 2026-09-18 - Refactor: split app_utils/system/smart.py (Large File Refactor Plan, Phase 4a-ii)

Companion to the same phase's snapshot.py split. _collect_smart_health was one 425-line function: locate smartctl, then for each device build a query command, run it, validate/parse its JSON output, infer an overall health status from the exit code when smartctl's own verdict is absent, and populate ~35 result fields from the report.

Changed

• Extracted the per-concern pieces into their own modules: smart_command.py (locating smartctl, building its command line), smart_query.py (running smartctl, validating/parsing its output), smart_status.py (the exit-code health-inference fallback), smart_attributes.py (identity fields, SMART attributes, NVMe extended fields). smart.py is now 191 lines of per-device orchestration.
• _collect_smart_health's public signature and return shape are unchanged; still resolves from app_utils.system and is used unchanged by webapp/admin/api/routes_smart.py and scripts/diagnose_smart.sh.
• Added tests/test_smart_health_package.py: 24 characterization tests written and run green against the pre-refactor function first, covering smartctl discovery, every subprocess/output-validation failure mode, and field-extraction wiring (the existing tests/test_smart_health.py already covered the exit-code status inference in depth and needed only its subprocess.run/os.path patch targets retargeted to the modules that now call them). An 18-mutation sweep confirmed the suite is discriminating, including a follow-up test added directly against the newly-extracted _build_smartctl_command() for a branch (-n standby for ATA/SAT device types) that _detect_device_type() currently never actually produces, so the full _collect_smart_health() path can't reach it end to end.

[3.11.0] - 2026-09-18 - Refactor: split app_utils/system/snapshot.py (Large File Refactor Plan, Phase 4a-ii)

Pure-motion follow-up to Phase 4a (docs/development/LARGE_FILE_REFACTOR_PLAN.md). build_system_health_snapshot was one 480-line function collecting CPU, memory, disk, network, process-table, load-average and database figures inline before computing an overall status — the one piece of app_utils/system/ Phase 4a left over the 400-line guidance.

Changed

• Extracted the inline collection blocks into their own modules, each independently testable: cpu.py, memory.py, disk_usage.py, processes.py, loadavg.py, db_health.py, and a _collect_network_info() added to the existing network.py. The overall-status computation moved to a new status.py. snapshot.py is now 158 lines of orchestration calling these collectors and assembling the response dict — every module in app_utils/system/ is now within the 400-line guidance.
• build_system_health_snapshot's public signature and return shape are unchanged; app_utils.build_system_health_snapshot and app_utils.system.build_system_health_snapshot still resolve.
• Added tests/test_system_health_snapshot_package.py: 18 characterization tests written and run green against the pre-refactor function first (per the plan's ground rules), covering every branch through psutil, the process table and the database probe, with the twelve already-extracted sibling collectors (systemd, hardware, SMART, temperature, dependencies, GPS, RTC, clock sync, Raspberry-Pi health, OS details, shields badges, distro logo) stubbed so the tests exercise only the logic that moved. A 14-mutation sweep across the new modules confirmed the suite is discriminating (all caught after two isolation fixes — the initial CPU/DB critical-status test conflated the two triggers, and the disk-permission-error test couldn't distinguish "correctly skipped" from "silently fell back to /").

[3.10.4] - 2026-09-17 - Fix raw HTML tags rendering as visible text on the Alert Details page

Reported via a user-submitted PDF of the Alert Details page for a Van Wert County, OH Flood Warning (Database ID #1108): the Description and Instructions sections showed literal, visible text like </p><p class="mt-2 mb-0"> and <br> instead of real paragraph breaks — even though the alert's description column, verified directly against the database, is clean plain text with real \n\n breaks and contains no HTML at all.

Fixed

• Root cause: templates/alert_detail.html, audio_detail.html, and manual_eas_print.html spelled the "escape untrusted CAP text, then insert paragraph/line-break HTML" transform out inline as {{ text | e | replace('\n\n', '</p><p ...>') | replace(...) | safe }}. | e produces a Jinja/MarkupSafe Markup object; Jinja's |replace filter, given an already-Markup value, routes to Markup.replace() — which HTML-escapes its own replacement argument too, as part of the invariant that keeps a Markup value safe to pass around elsewhere. So the <p>/<br> tags each template meant to insert came out double-escaped: present in the string, but as the literal text <p ...> rather than a real tag, which a browser then displays as visible <p ...> text once the whole thing is marked | safe at the end.
• Moved the transform into Python (webapp/template_helpers.py): fixed the existing, previously-unused nl2br filter to escape input before substituting (building the finished HTML as a plain str, never touching Markup.replace(), then wrapping the result in Markup exactly once at the end), and added a new cap_paragraphs filter for the paragraph+bullet variant used by alert descriptions/instructions. All 5 call sites across the 3 templates now use one of these two filters instead of the inline chain.
• Added tests/test_cap_text_html_filters.py: unit tests for both filters (including the exact reported text) plus a structural regression test that scans every template for the | e | ... replace(...) shape and fails the build if it reappears.

[3.10.3] - 2026-09-17 - Fix CAP/IPAWS auto-forwarded alerts never reaching Icecast

A live Flash Flood Warning (Putnam County, OH, 2026-09-17 07:44 EDT) was auto-forwarded via IPAWS, logged "Auto-forwarded ... to air chain," and forwarded again minutes later by the forwarding catch-up sweep after eas-station-poller restarted mid-cycle — but a listener monitoring the stream live heard nothing either time.

Fixed

• Root cause: EASBroadcaster.handle_alert() (app_utils/eas.py) pushed generated broadcast audio into Icecast by calling app_core.audio.eas_stream_injector.inject_eas_audio() directly, in-process. That function is a no-op unless eas_stream_injector.set_controller() was called earlier in the same process — which only happens in eas_monitoring_service.py, the entry point of eas-station-audio.service. handle_alert() is also invoked from poller/cap_poller.py (the primary CAP/IPAWS ingest path, the gated-alert auto-release timer, and the forwarding catch-up sweep — all running as eas-station-poller.service) and from webapp/admin/pending_alerts.py's gated-alert "Approve" action (eas-station-web.service). In every one of those processes the injection call silently did nothing: the EASMessage DB row was still written and the alert still logged as forwarded, but no audio ever reached Icecast or local playback. The direct call was added in commit ff640d3 (2026-03-25), which — ironically — cited the PR that fixed this identical class of bug for Manual Send/RWT by routing through Redis, but added a new unguarded direct call for the auto-forward path instead of reusing that fix. The existing structural regression test (tests/test_broadcast_reaches_icecast_audit.py) is AST-based and only checks that a broadcast-trigger function calls something named inject_eas_audio(), so it could not detect that the call was a cross-process no-op.
• handle_alert() now checks eas_stream_injector.has_controller() and, when no controller is registered in the current process, falls back to the same cross-process Redis command Manual Send already uses (AudioCommandPublisher.inject_raw_eas_audio), asking eas-station-audio.service — the process that actually owns the live IcecastStreamer threads — to perform the injection. A failed fallback now logs an ERROR naming the problem instead of failing silently.
• The injection outcome is recorded on the EASMessage row (metadata_payload['icecast_injected']) so a transient failure (audio-service down, Redis unreachable at exactly that moment) isn't a permanent, silent loss. A new CAPPoller.retry_failed_icecast_injections() sweep, run every poll cycle alongside the existing forwarding catch-up sweep, re-sends any recent failed injection via the same resend command the EASMessage detail page's manual "Resend" button uses, up to 3 attempts, and raises a system-log ERROR if it still hasn't gone out after that.
• Added tests/test_eas_broadcaster_injection_fallback.py and tests/test_icecast_injection_retry_sweep.py.

Fixed (test infrastructure)

• While verifying this fix, a test run that exercised handle_alert()'s new fallback without mocking it sent 7 real inject_raw_eas_audio commands to the actual live eas-station-audio.service on the dev box, each one audibly injecting a short synthetic test WAV into the live Icecast streams. The existing REDIS_DB=15 test isolation (added after an earlier incident where the test suite keyed a live GPIO relay) did not protect against this: Redis PUBLISH/SUBSCRIBE are global across logical databases regardless of which DB a client has SELECTed, unlike key-value commands. Added an autouse fixture (tests/conftest.py::_isolate_eas_stream_injection) that stubs AudioCommandPublisher for every test by default, so no test can reach the real audio-command channel without explicitly opting in.

[3.10.2] - 2026-09-17 - Pin fsspec/s3fs to stop dependency-bump PRs from failing CI

A routine Dependabot boto3 patch bump (1.43.56 → 1.43.92) failed CI's dependency install step with pip: ResolutionImpossible, even though the bump touched nothing related to the reported conflict. Root cause: arm_pyart requires both fsspec and s3fs, but only loosely (fsspec>=2021.11.0), and neither was otherwise pinned in requirements.txt. Older s3fs releases hard-pin an exact matching fsspec version (e.g. s3fs==2025.3.1 requires fsspec==2025.3.1.*), so pip's resolver could wander into an old, mutually-incompatible s3fs/fsspec pair while re-resolving the whole file for an unrelated bump. Pinned both to 2026.7.0, the versions already verified working together in production, removing the ambiguity.

[3.10.1] - 2026-09-16 - Fix missing LED degree glyph and a fail2ban leader-lock race

A code review of the two preceding commits (the Icecast/fail2ban leader-lock fix and the LED screen editor WYSIWYG fix) turned up two follow-on bugs.

• Fixed a missing ° glyph in the LED screen editor's client-side font table: LED_GLYPH_5x7 in static/js/screen-editor.js omitted the degree symbol that scripts/dotmatrix_preview_font.py's server-side FONT_5x7 defines, so a legacy LED text line containing ° (e.g. a temperature message) rendered as a blank space on the live editing canvas while the "pixel-accurate" preview modal rendered the real glyph -- the exact WYSIWYG mismatch the prior commit set out to fix, reintroduced for one character.
• Fixed a non-atomic leader-lock renewal race in app_core/fail2ban_sync.py: _acquire_or_renew_leader_lock() renewed its Redis lease with a separate get() then expire() call, so the lock key could expire and be claimed by another worker in the gap between the two, letting the renewing worker wrongly believe it still held leadership and briefly run fail2ban-client/systemctl alongside the new leader in the same cycle. Renewal now uses a small atomic Lua script (_RENEW_LOCK_SCRIPT, run via EVAL) that checks ownership and extends the TTL in one round-trip.

[3.10.0] - 2026-09-16 - LED screen editor: honest WYSIWYG for the Alpha font/colour selection

An audit of the Alpha M-protocol driver and its "WYSIWYG" screen editor (prompted by suspicion the editor was sloppy even though the protocol driver itself checked out as solid and well-tested) found the editor's live canvas and the "pixel-accurate" preview modal both rendered every Alpha Font selection at the same fixed size -- picking a larger font changed nothing an operator could see. Tracing it further turned up a second, related class of bug: the legacy 4-line LED text mode reused the same free-form draggable-element canvas as OLED/VFD, which let an operator set several per-element fields (font size, alignment, invert, etc.) that were silently discarded and had zero effect on the real 4-line sign message, and building the saved message used raw element-insertion order rather than the order lines actually appeared top-to-bottom on the canvas.

• Font selection now actually changes the preview, both the always-visible editing canvas and the server-rendered modal: scripts/dotmatrix_preview_font.py gained blit_text_scaled(), which approximates a larger/smaller Alpha font by scaling the same shared 5x7 glyph shapes (declared font dimensions, e.g. FONT_32x16, aren't verified against real sign firmware -- see the existing protocol audit's §3.2 -- so this is a best-effort size approximation, not pixel-perfect). scripts/led_preview_render.py's render_led_preview() now stacks each line at its own scaled height (message-level or per-line font/colour override) instead of a fixed grid, and the client canvas mirrors it with a matching round-dot-plus-glow renderer (drawLedDotText()) instead of generic monospace text.
• Removed inert per-element fields (font size, align, max width, wrap, invert, allow-empty) from the properties panel when editing a legacy LED text line -- none of them ever reached render_led_screen()'s lines payload, so showing them let an operator "set" something that silently did nothing on real hardware.
• Fixed line order: buildTemplateData() now sorts LED text elements by Y position before building the lines array, so the order lines appear top-to-bottom in the editor is the order they're actually sent -- previously this used raw element-insertion order, which silently diverged from the visual layout as soon as a line was dragged past another.
• Fixed a stale-canvas-size bug on load: loadScreen() was setting the led-font dropdown's value after sizing the canvas and staggering line positions from it, so opening a saved screen briefly showed the wrong-sized canvas until the operator touched the font dropdown themselves.
• Added a 4-line cap (with a toast) when adding LED text elements, matching the sign's actual line limit instead of silently truncating on save with no UI feedback.
• Generalized _render_led_glow()'s single-colour dot rasterizer to a per-pixel colour callback so per-line colour overrides render correctly, and both existing LED preview call sites (render_led_preview, render_led_elements_preview) were updated to the new signature.
• No JS test harness exists in this repo, so the client-side change was verified by hand: a standalone static harness (the real templates/screen_editor.html markup + the actual screen-editor.js/utils.js) driven over the Chrome DevTools Protocol confirmed font selection resizes the canvas (120x32 at FONT_7x9 -> 720x64 at FONT_32x16), colour selection changes the glow colour, inert fields are hidden, the 4-line cap holds, and OLED/VFD/LED-graphics modes are all unaffected -- zero console errors or uncaught exceptions across the whole pass.
• 9 new Python regression tests across tests/test_display_preview_render.py (font-size-changes-the-render, per-line font/colour override, unknown-font fallback) and tests/test_dotmatrix_preview_font.py (scaled-blit unit tests, including a 1x-scale-matches-legacy-output equivalence check).

[3.9.3] - 2026-09-16 - Fix false-positive Icecast buffer warnings and duplicate fail2ban sync cycles

Found while reviewing service logs for real issues: two long-running, log-visible-but-benign-looking problems turned out to be genuine bugs once traced to their code.

• Icecast buffer health check always reported near-zero: IcecastStreamer._feed_loop read buffer_level = len(buffer) after buffer.popleft() in the same iteration -- since one chunk is read and one is popped per iteration, this measured the post-pop remainder (0-1) rather than the actual banked cushion, so the low-watermark check fired continuously regardless of real audio health. In production this logged ~120 "Icecast buffer running low ... 0/600 chunks" warnings per hour, per mount, on all four streams, non-stop -- pure log noise with zero diagnostic value, easy to mistake for a real audio problem (or to mask one). Fixed by snapshotting buffer_level before the pop, so it reflects the true depth of audio banked ahead of playout.
• fail2ban sync scheduler ran once per Gunicorn worker instead of once per app: app_core/fail2ban_sync.py's background thread is started at module import time in app.py, which every Gunicorn worker process imports independently (no --preload) -- so a 4-worker deployment ran 4 independent copies of the 60s sync loop, visible in the journal as bursts of privileged fail2ban-client/systemctl sudo calls every ~15-20s instead of once a minute. Fixed with the same Redis SETNX cross-worker leader-lock pattern app_core/rwt_scheduler.py already uses for its own per-worker duplication problem: a renewable lease (TTL = 3x the sync interval) so exactly one worker runs the cycle, and another takes over automatically if the leader dies; falls back to the historical every-worker behavior if Redis is unreachable rather than silently stopping the sync.
• Icecast local jitter buffer could never recover from a stall: once the pre-pop fix above started reporting real numbers, live monitoring caught a second, deeper bug it had been masking: IcecastStreamer._feed_loop reads and writes exactly one chunk per iteration, so a normal iteration nets zero change in buffer depth and a read timeout nets -1 -- there was no path that could ever net positive. That makes the buffer a one-way ratchet: any network stall on an internet-relay source drains it by exactly the chunks lost, permanently, since even a source that catches up afterward could only refill one chunk per iteration, matched chunk-for-chunk by that same iteration's single pop. Confirmed live: two internet-relay mounts (WNCI, ERN-LUC) ratcheted down in discrete steps (150 -> 149 -> 147 -> 146 chunks) over several hours and never recovered. Fixed by opportunistically draining any further chunks already queued (non-blocking) right after a successful read, so a post-stall backlog burst can add many chunks in one iteration while FFmpeg still only receives one -- letting the buffer actually climb back toward its target depth instead of only ever shrinking.
• 5 new regression tests: test_buffer_health_check_reflects_pre_pop_depth and test_buffer_recovers_after_a_stall_once_source_catches_up (both fail without their respective fixes, reproducing the exact production symptoms) plus three test_leader_lock_* tests covering single-leader election, lease takeover after expiry, and the Redis-unreachable fallback.

[3.9.2] - 2026-09-14 - Fix Identify Stations never completing a PS name

3.9.1's ring-buffer fix was necessary but not sufficient: re-tested live afterward with a diagnostic build, Identify Stations was still 0/9 -- but this time with real evidence of why. RBDS sync itself was working correctly (confirmed live: PI=5862, call sign WBKS resolved, sync confirmed ~1.8s in) -- the bug was in how the result was read afterward, and in how much time was budgeted.

• Root cause: the 8-character PS marquee text (ps_name) is assembled from 4 separate segments broadcast in different RDS groups over several seconds -- sync succeeding is necessary but not sufficient for a complete name. _run_bandscan_identify's default dwell_sec=2.5 left barely any runway after the ~1.8s sync itself, so ps_name was reliably still '' (incomplete, not decoded) when the dwell ended on every peak, every time -- including a station live-verified to sync fine every single pass.
• Fixed: call_sign (derived from the PI code carried in every synced group, available within ~1-2s of sync, no multi-group assembly needed) is now captured throughout the dwell as a fast, reliable fallback. The early-stop-on-decode path still only fires on a complete ps_name, so a lucky fast station still finishes quickly -- but a result that syncs without ever completing the marquee text now reports the call sign instead of silently reporting nothing.
• Raised the default dwell budget from 2.5s to 6.0s to give PS assembly more real runway on top of the call-sign fallback.
• webapp/radio_settings results list now shows ps_name || call_sign, preferring the fuller marquee text when it's actually complete.
• 1 new test (test_identify_falls_back_to_call_sign_when_ps_name_never_completes) plus updated coverage across the existing identify tests for the new call_sign field.
• This whole investigation (3.9.1 and 3.9.2 together) happened because the feature was tested against real hardware immediately after shipping, at the user's prompt, rather than trusting the unit tests' mocked demodulator -- see 3.9.1's own entry for why that specific bug class needed live testing to catch at all.

[3.9.1] - 2026-09-14 - Fix Identify Stations decoding zero RDS (ring-buffer race)

Live-tested against real hardware immediately after 3.9.0 shipped, at the user's prompt: Identify Stations decoded 0 of 9 real detected peaks -- including a station (93.9 MHz) the persistent live monitoring receiver decodes as "KISS" on every pass. This was a genuine functional bug, not bad luck.

• Root cause: _run_bandscan_identify (and _run_bandscan_sweep, and _auto_gain_calibrate before it) call receiver.get_samples() directly from their own thread, same as publish_samples_and_metrics()'s own live publisher thread does continuously for every running receiver. Both threads read from the same underlying ring buffer -- per that function's own existing comment, the publisher is normally the only thread draining it, and a second concurrent consumer fragments the stream between them. A coarse RMS level measurement (what the sweep and auto-gain need) tolerates that fine; RDS decode does not -- RBDSWorker's Costas/M&M loops need an unbroken, phase-continuous stream across many consecutive reads to ever reach lock, so every one of the 9 peaks silently failed to decode.
• Fixed: publish_samples_and_metrics() now skips reading (and publishing telemetry for) a receiver for the duration of any Bandscan sweep or Identify Stations pass against it (identifier in _state.active_bandscans), leaving the scan thread as the sole consumer of that receiver's samples for its duration -- consistent with, not a regression from, that receiver already being retuned away, muted, and dead-air-suppressed during the same window.
• Re-verified live after the fix (see commit/PR for the before/after real-hardware results).
• No test added for publish_samples_and_metrics() itself -- it has no existing test harness (an infinite while _state.running loop against live threading/hardware state) and this specific bug class (a cross-thread hardware race) is exactly the kind a mock cannot catch; live testing is what caught it, and is the right tool for verifying the fix too.

[3.9.0] - 2026-09-14 - Bandscan: Identify Stations (RDS station-name labeling of peaks)

Closes the second item explicitly deferred out of Bandscan's original 3.7.0 changelog entry: "RDS station labeling of detected peaks." A peak used to only ever show its bare frequency (e.g. "93.9"); this adds a second, explicit pass that retunes to each detected peak long enough to attempt an RDS PS (station name) decode.

• Why a second pass: RDS sync needs real dwell time -- RBDSWorker's own timing puts a decent signal's first PS decode at ~1 second end-to-end ("the same time a car radio takes"), unlike the sweep's near-instant per-channel level measurement. Sweeping 206 channels in ~150ms each can never decode RDS, so this can't be folded into the sweep itself.
• Added: a new "Identify Stations" button on SDR Diagnostics, shown once a sweep finishes with at least one detected peak. Its own confirm() states the added time (~2.5s per peak) and that the receiver still won't monitor for real alerts during it -- same risk framing as Bandscan itself.
• Added: sdr_hardware_service.py's _run_bandscan_identify, dispatched via a new bandscan_identify SDR-service action -- retunes to each target frequency and dwells (default 2.5s, stopping early the moment a PS name decodes) via a fresh, purpose-built FMDemodulator per frequency, so a stale lock from one peak can never leak into the next. Shares _state.active_bandscans (the sweep's own concurrency guard) and RedisChannels.BANDSCAN_ACTIVE_PREFIX (the audio-mute/dead-air-suppression flag shipped in 3.8.0) with the sweep -- either kind of scan blocks starting the other, and the existing Cancel button/endpoint works for both with no new code.
• Added: POST/GET /api/radio/bandscan/<id>/identify/{start,progress} (webapp/radio_settings/routes_bandscan.py) -- start validates the submitted peak frequencies fall within the Bandscan band before dispatching; progress is its own Redis key (BANDSCAN_IDENTIFY_PROGRESS_PREFIX) so a still-visible sweep result is never overwritten by an identify pass that follows it.
• Results render as a list below the chart (frequency, level, station name or "No RDS decoded") rather than on the canvas -- the peak-label collision/staggering logic tuned in 3.7.4 was sized for short frequency-only text, not an 8-character station name.
• 24 new tests across tests/test_bandscan.py (early-stop-on-decode, full-dwell-on-no-decode, per-frequency demodulator isolation, frequency restore on every exit path, shared mute flag, separate progress key) and new tests/test_bandscan_routes.py (frequency validation, permission gate, command dispatch, progress shape) -- there was no existing route-level test file for the other bandscan endpoints either.

[3.8.0] - 2026-09-14 - Bandscan: mute audio during the sweep

Closes the "audio muting during the scan (a real side effect, noted rather than silently ignored)" item explicitly deferred out of Bandscan's original 3.7.0 changelog entry. Until now a sweep's rapid channel-hopping across the whole FM band streamed live to Icecast and the archiver -- roughly 30-45 seconds of static/noise on air for a routine diagnostic action.

• Added: RedisChannels.BANDSCAN_ACTIVE_PREFIX (app_core/config/redis_config.py) -- a lightweight, short-TTL'd presence marker, separate from the existing (heavier, growing-JSON) BANDSCAN_PROGRESS_PREFIX key, set and refreshed every channel by sdr_hardware_service.py's _run_bandscan_sweep and explicitly deleted (not just left to expire) once the sweep ends, however it ends.
• Added: services/demod/worker.py's DemodWorker now checks this flag (cached ~0.5s to avoid a Redis round-trip on every ~32ms audio chunk) and publishes silence instead of the real demodulated audio for the sweep's duration -- the demodulator itself keeps running normally on the real swept IQ so its internal DSP/RBDS state isn't torn down and rebuilt around the mute.
• Added: a _dead_air_suppressed() hook on AudioSourceAdapter (app_core/audio/ingest.py, default False), overridden by RedisSDRSourceAdapter to check the same flag. Without this, the deliberate silence fed by the mute above would itself cross the dead-air monitor's duration threshold and raise a false alarm (tower light/buzzer) for a routine, operator-initiated diagnostic scan -- app_core/audio/silence.py's SilenceMonitor.process() call is skipped (not reset) while suppressed, so a real outage that was already accruing before the scan resumes counting from where it left off once the scan ends.
• 14 new tests across tests/test_bandscan.py, tests/test_demod_service.py, tests/test_redis_audio_pipeline.py, and tests/test_audio_ingest.py -- the active flag is set while running and explicitly cleared on every sweep exit path (completion/cancellation/error); published audio is verified as real vs. all-zero silence depending on the flag; the mute check is cached, not per-chunk; the dead-air hook is verified suppressed/not-suppressed/fail-safe-on-Redis-error, cached, and actually skips SilenceMonitor.process() when suppressed.

[3.7.4] - 2026-09-14 - Bandscan plot: fix a suppressed close peak and Y-axis label wrapping

Two more bugs reported immediately after 3.7.3 shipped, both from live use: a visually distinct peak sitting close (under ~0.5 MHz) to a taller one wasn't labelled, and the new Y-axis dBFS column wrapped its text onto two lines and overlapped the neighbouring gridline's own label.

• Fixed: PEAK_MIN_PIXEL_GAP (the minimum on-screen distance between two kept peak labels) was 30px (~1.2 MHz), wide enough to suppress a real, visually distinct second peak -- its own local minimum/notch clearly separating it from its neighbour -- just because it sat under 1.2 MHz from a taller one. Lowered to 12px (~0.5 MHz), with bandscanRender() now staggering a label's vertical position instead of dropping it outright when two kept peaks are close enough for their text to still overlap.
• Fixed: the new Y-axis column (added in 3.7.3) used the Bootstrap small class, which resolves against the page's rem base and was wide enough to wrap "-19 dBFS" onto two lines inside the 44px-wide column -- since each label is positioned assuming one line, the wrapped second lines overlapped the neighbouring gridline's own label into unreadable overlapping text. Switched to an explicit 10px monospace font (matching the plot's own peak-label font) with white-space: nowrap, and widened the column to 54px.
• No backend changes; tests/test_bandscan.py unaffected.

[3.7.3] - 2026-09-14 - Bandscan plot: fix peak detection and dBFS-label collisions

Two bugs reported immediately after 3.7.2 shipped: only 2 of the ~15 visually obvious peaks in a live scan got labelled, and the leftmost peak's label was drawn on top of (and unreadable against) the Y-axis dBFS text.

• Fixed peak detection: bandscanFindPeaks() used to compare every bump against one global noise-floor number (the sweep's own 25th-percentile level). On a live scan a single wide, deep fade between two strong stations dragged that whole-band percentile far below the ambient level on the rest of the band, so the effective threshold ended up above most of the other genuinely-present bumps -- only the two loudest peaks in the whole sweep qualified. Prominence is now measured against each peak's own local minimum (+/- 10 channels, ~1 MHz), so a modest local bump is still labelled even while some other stretch of the band is much louder or quieter.
• Fixed the Y-axis/peak-label collision: the dBFS scale is no longer drawn as text inside the canvas at all (where it could sit under the plotted line or a peak's own label). It now renders in a dedicated column beside the canvas (panel.yAxisCol), matching how the X-axis ticks already sit outside the plot below it.
• No backend changes; tests/test_bandscan.py unaffected.

[3.7.2] - 2026-09-14 - Bandscan plot: label detected peaks directly

• Added: the band plot now labels detected signal peaks in place, right next to the signal they belong to, instead of only labelling the shared X-axis ticks -- reading a station's frequency off the axis meant eyeballing an interpolation between two ticks that were often nowhere near it. bandscanFindPeaks() finds local maxima at least 6 dB above the sweep's own 25th-percentile noise floor, picks the strongest first, and keeps a candidate only when it's at least 34 canvas pixels from every peak already kept (capped at 12 labels) -- so a single station's skirt doesn't get one label per bin and labels never overlap.
• Axis ticks are unchanged and still shown for overall scale.

[3.7.1] - 2026-09-14 - Bandscan plot: axis labels and persistent results

Follow-up bug fix on the Bandscan sweep shipped in 3.7.0, reported immediately after use: the band plot had no readable Y axis and the finished result vanished almost as soon as the scan completed.

• Fixed: the band plot now draws horizontal dBFS gridlines with value labels, and the X axis shows several evenly-spaced round-MHz ticks (87.5/92/96/100/104/108 MHz) instead of only the two band edges -- previously there was no way to read a signal level or an arbitrary peak's frequency off the plot at all.
• Fixed: a finished scan's result no longer disappears within seconds. Root cause: templates/admin/radio_diagnostics.html's SDR Diagnostics page fully re-renders #receiversContainer (and every .bandscan-container inside it, which the template always renders style="display:none;") on its normal 15s auto-refresh; the frontend was only re-populating that container for receivers still actively polling (bandscanActiveReceivers), so as soon as a scan finished and polling stopped, the very next 15s refresh silently re-hid the result with nothing to redraw it. The frontend now keeps the last-known progress payload per receiver (bandscanLastPayload) and bandscanRebuildActive() redraws a finished scan's plot from it whenever the periodic refresh recreates the container, so the result stays visible until a new scan is started.
• No backend changes; tests/test_bandscan.py (5 tests, unaffected) still passes.

[3.7.0] - 2026-09-14 - SDR Diagnostics: Bandscan (full FM-band sweep)

Final item from the post-brochure FM-analyzer follow-up, and the one genuinely different from everything else in the set: every prior addition (MPX spectrum, oscilloscopes, deviation histogram, modulation power, stereo balance) is a passive view of whatever a receiver is already doing. Bandscan is active -- it retunes the receiver across the whole FM broadcast band to survey what's on the air, the way the reference hardware's own "Bandscan" view does.

• Added: a "Bandscan" button on SDR Diagnostics (styled and risk-tiered like the existing "Restart Receiver" button) that sweeps 87.5-108.0 MHz in 100 kHz steps (206 channels, ~30-45 seconds measured live), measuring signal level at each stop and rendering a live-filling band plot. A confirmation dialog states up front that the receiver will not monitor for real alerts until the scan finishes -- this is a deliberate, disruptive action, not a passive one.
• Added: bandscan_sweep/bandscan_cancel actions in sdr_hardware_service.py, running the sweep on its own background thread rather than inline in the shared SDR command queue -- a full sweep (minutes) would otherwise block every other receiver's tune/restart/diagnostics commands for its whole duration, unlike the existing auto_gain sweep's already-accepted ~90s worst case. The receiver is always retuned back to its assigned frequency when the scan ends, whether it completes normally, is cancelled, or hits an error -- the database's assigned frequency is never touched, only the live tuner, mirroring tune_frequency's own raw retune call without its DB-persistence step.
• Added: GET/POST /api/radio/bandscan/<id>/{start,progress,cancel} (webapp/radio_settings/routes_bandscan.py) -- start only waits for a fast "did it start" acknowledgment, not the whole sweep; the frontend polls progress (a transient Redis key, RedisChannels.BANDSCAN_PROGRESS_PREFIX) separately on its own ~1.5s cadence.
• tests/test_bandscan.py: 5 new tests against the sweep loop directly -- the channel list matches the expected band/step math exactly, and the original frequency is restored in all three ways the sweep can end (normal completion, mid-sweep cancellation, mid-sweep exception).
• Deliberately out of scope for this pass: RDS station labeling of detected peaks, audio muting during the scan (a real side effect, noted rather than silently ignored), and custom band/step configuration.

[3.6.0] - 2026-09-13 - SDR Diagnostics: modulation power, stereo balance, deviation histogram, MPX/audio oscilloscopes

Tier 1 of the post-brochure FM-analyzer follow-up: after reviewing the PIRA P275's full brochure and FM Scope software manual, five measurements were cheap enough to add in one pass because they only needed to plot or bin data the demod pipeline already computes every chunk (or one more same-cost RMS pass alongside it) -- no new capture mechanism, no new DSP concept.

• Added: modulation_power_dbr and stereo_balance_db on DemodulatorStatus (app_core/radio/demod/fm.py/types.py), computed every chunk exactly like the existing peak_deviation_hz/pilot_injection_hz/rds_injection_hz fields. Modulation power is RMS power of the composite MPX signal in dBr, referenced to a 75 kHz-full-scale sine (0 dBr = 100% modulation, the same convention the MPX spectrum's FFT already documents). Stereo balance is 20*log10(RMS(R)/RMS(L)), computed in the same kHz-deviation domain as the MPX oscilloscope below rather than normalized playback-audio loudness. Both thread through the existing Signal Quality history pipeline (redis_sdr_adapter.py -> AudioSourceMetrics -> routes_signal_quality.py -> two new Chart.js charts) with zero new infrastructure -- the same 4-hop pattern proven three times already in 3.4.0.
• Added: a frequency-deviation histogram (GET /api/audio/sources/<name>/signal_quality/deviation_histogram), binning peak_deviation_hz values already sitting in the Signal Quality history table over the selected 1h/6h/24h window, with a % of samples above 75 kHz compliance readout. Unlike the reference hardware (which needs a manual "Acquire" pass over a live N-minute capture), this is available instantly over any window already being logged continuously -- no separate acquisition step.
• Added: two new oscilloscope traces bundled into the already-shipped MPX Scope's existing payload/route (demod:mpx_spectrum:<id>, /api/radio/mpx_spectrum/<id>) rather than new keys -- an "MPX oscilloscope" (the raw composite multiplex waveform, not the FFT the panel already showed) and an "audio oscilloscope" (decoded L/R channels, only present once stereo is locked). Both captured on the same ~2 Hz gate the spectrum FFT already uses.
• tests/test_fm_deviation_injection_metrics.py: 5 new tests -- modulation power reads ~0 dBr at exactly the 75 kHz reference and ~-6 dBr at half scale, floors safely for a silent carrier, stereo balance matches a known 2:1 L/R ratio (~+6 dB) and defaults to 0.0 when mono.
• tests/test_mpx_spectrum.py: 4 new tests -- the MPX waveform is captured and length-capped alongside the spectrum, the audio-scope trace is present exactly when stereo is locked and absent (not stale) otherwise.
• tests/test_signal_quality_history_api.py: 6 new tests -- the two new history fields round-trip, the histogram bins known synthetic samples correctly, computes pct_above_limit correctly, handles an empty/unrelated window without erroring, and clamps an out-of-range reading into the last bin instead of raising.

[3.5.0] - 2026-09-13 - SDR Diagnostics: MPX (demodulated baseband) spectrum scope

Final piece of the FM-broadcast-analyzer feature set started in 3.3.0: a real demodulated-baseband spectrum showing the 19 kHz stereo pilot, 38 kHz stereo subcarrier, and 57 kHz RDS subcarrier as visible spectral features -- the actual "analyzer screen" look from the PIRA P275 reference screenshot that kicked off this whole set. The existing Spectrum Scope/Live Waterfall show the RF spectrum around the tuned carrier (pre-demodulation IQ); this is a second, genuinely new spectrum computed from the demodulator's own multiplex output, which cannot show this picture.

• Added: app_core/radio/spectrum.py -- compute_spectrum (RF, complex IQ, moved here verbatim from sdr_hardware_service.py with no behavior change) plus a new compute_real_spectrum (MPX, real-valued multiplex via numpy.fft.rfft, one-sided).
• Added: FMDemodulator.demodulate() (app_core/radio/demod/fm.py) computes the MPX spectrum every ~0.5s (2 Hz, rate-limited the same way DemodWorker._publish() already throttles its status key) from the raw discriminator output and exposes it via last_mpx_spectrum/last_mpx_spectrum_fresh.
• Added: services/demod/worker.py publishes a fresh MPX spectrum to a new dedicated Redis key (RedisChannels.MPX_SPECTRUM_PREFIX, 5s TTL) -- its own key, not folded into the existing status channel, matching how RF spectrum and audio already have their own channels distinct from status.
• Added: GET /api/radio/mpx_spectrum/<id> (webapp/radio_settings/routes_mpx_spectrum.py) -- returns status: "unavailable" (200) for AM/IQ-passthrough receivers or before the demod service has published one yet, rather than a 404/500.
• Added: "MPX Scope" toggle on the SDR Diagnostics page (Admin -> Radio -> Diagnostics), next to the existing Spectrum Scope button -- a fixed 0-100 kHz baseband view with dashed marker lines at 19/38/57 kHz, polled independently at ~2 Hz. Deliberately kept on its own simple MutationObserver-based refresh (mirroring radio_diagnostics_trends.js) rather than integrated into the RF Spectrum Scope/Live Waterfall's more complex shared zoom/pan/live-DOM-node-preservation system, to avoid introducing risk into that already-proven code.
• tests/test_mpx_spectrum.py: 5 new tests -- synthetic multiplex tones at exactly 19 kHz and 57 kHz land within one FFT bin of the expected peaks, one-sided output length, the too-few-samples and 0-1 bounding edge cases, and the 0.5s rate-limit gate.
• Live-verified on sdr-wbks (a real over-the-air FM station): the MPX Scope shows a sharp, correctly-positioned pilot spike right at the 19 kHz marker and visible elevated structure at the 38k/57k subcarrier markers.
• Also did a full authenticated click-through of the existing RF Spectrum Scope's zoom (button zoom in/out, mouse-wheel zoom, drag-to-pan, double-click reset, and persistence through the page's own ~1s auto-refresh, including a drag gesture spanning a live refresh) after a user report that zoom "doesn't work correctly" -- every interaction measured out exactly per the documented design (anchor-preserving zoom, precise pan-fraction math) with no reproducible defect found.

[3.4.0] - 2026-09-12 - SDR Diagnostics: peak deviation and pilot/RDS injection level (kHz)

Third and final planned step (after Signal Quality history in 3.3.0 and the button-audit fixes in 3.3.1) toward FM-broadcast-analyzer-style monitoring on the existing SDR stack, following a PIRA P275 hardware analyzer screenshot the user shared as reference. The one genuinely new piece of DSP in the set -- unlike stereo pilot strength / click rate, which only needed wiring an already-computed value through, these three are new measurements.

• Added: peak_deviation_hz, pilot_injection_hz, rds_injection_hz on DemodulatorStatus (app_core/radio/demod/types.py), computed in FMDemodulator.demodulate() (app_core/radio/demod/fm.py). All three are the inverse of the existing _audio_gain scale factor (raw discriminator output in radians/sample times sample_rate / (2*pi) is real Hz) -- not new calibration, just exposing a conversion the code already implicitly does for audio normalization. Peak deviation and pilot injection reuse values (multiplex, pilot_rms) already computed every chunk regardless; RDS injection needed one new FIR bandpass filter (55.6-58.4 kHz), built only when RBDS decoding is enabled so receivers that don't use RDS pay nothing extra.
• Threaded through the same pattern already proven for click_rate (PR #2621): app_core/audio/redis_sdr_adapter.py's metadata dict, webapp/admin/audio_ingest/routes_signal_quality.py's existing history endpoint (no new route), three more Chart.js charts in the already-shipped Signal Quality modal.
• Applied the axis-headroom lesson from 3.3.1: the two new percentage-like metrics use plain beginAtZero: true with no max/suggestedMax at all, rather than fighting Chart.js's auto-scale heuristics again -- real audio content makes these genuinely vary (confirmed live: peak deviation swung 80-92 kHz across a few seconds), so there's no constant-value-at-a-hard-ceiling failure mode to guard against here the way there was for a rock-solid 100% pilot lock.
• tests/test_fm_deviation_injection_metrics.py: 6 new tests synthesizing FM signals at known deviations (a pure tone at exactly 50 kHz peak deviation, a pilot at exactly 6.75 kHz peak, a 57 kHz tone at exactly 3.5 kHz peak) and asserting the computed values land close to the expected Hz/RMS values -- this is the one part of the feature where "the field threads through end to end" isn't sufficient coverage on its own, since the math itself is new.
• Live-verified on sdr-wbks (a real over-the-air FM station, not a synthetic signal): peak deviation ~80-92 kHz, pilot injection ~4.2-4.3 kHz, RDS injection ~3.3-3.4 kHz -- all in plausible real-world ranges (FCC full-scale reference is +/-75 kHz; typical healthy pilot/RDS injection is ~2-7.5 kHz depending on station configuration).
• tests/test_signal_quality_history_api.py: extended with a test covering the three new fields in the history response.

[3.3.1] - 2026-09-12 - SDR Diagnostics: fix Snapshot Waterfall crash and two invisible-line chart bugs

Found by systematically clicking through every control on the SDR Diagnostics page (logged in as a real admin) rather than trusting that "no console error" meant "actually works" -- three real, independent bugs surfaced this way.

• Fixed: "Snapshot Waterfall" returned a 500 on every click -- AttributeError: 'NpzFile' object has no attribute 'size'. routes_diagnostics_waterfall.py called np.load(path).size directly, but sdr_hardware_service.py actually writes captures as .npz (key 'iq'); np.load() on a .npz returns a lazy NpzFile container, not an array. routes_diagnostics_analyze.py already handled this correctly (checks the extension, opens the archive, extracts archive["iq"]) -- routes_diagnostics_waterfall.py had just never been exercised with this in mind. Fixed identically; also corrected the route's docstring, which still said .npy.
• Fixed: the "Lock %" chart in the per-receiver Historical Trends panel (static/js/radio_diagnostics_trends.js) rendered its line invisible for a receiver locked 100% of the window -- the common, healthy case. Same root cause as the Signal Quality pilot-strength chart fixed in 3.3.0, but worse: with the pilot chart, suggestedMax happened to work because that data had slight variance (99-100%); here the data is a literal constant 100.0 with zero variance, so suggestedMax: 100 computes max(100, 100) = 100 -- no headroom at all, confirmed live via the actual Chart.js instance. Re-fixed both charts (Lock % and pilot strength) with a hard max: 105, which holds regardless of data variance.
• Fixed: the "Sample Rate Ratio" chart auto-scaled to a nonsensical -1 to 1.5 range for a constant, real 0.25 ratio -- Chart.js has no sensible default for a zero-variance series with no configured bounds. sample_rate_ratio (effective/configured) is always in [0, 1] by construction; added { min: 0, suggestedMax: 1 }.
• tests/test_spectrum_frequency_axis.py: two new regression tests -- one reproducing the exact .npz/NpzFile.size bug and confirming the fixed extraction works, one guarding the source against a revert to the bare np.load(path).size pattern.

[3.3.0] - 2026-09-12 - SDR Diagnostics: Signal Quality history (stereo pilot + click rate)

• Added: a "Signal Quality" history modal on the SDR Diagnostics page (/admin/radio/diagnostics), next to the existing RBDS History button, charting stereo pilot lock strength and discriminator click rate (a multipath/impulse-noise indicator) over the last hour/6h/24h. First step toward FM-broadcast-analyzer-style monitoring (PIRA P275-style) built on the existing SDR receiver stack rather than dedicated hardware.
• app_core/audio/redis_sdr_adapter.py: click_rate/click_suppression_enabled (computed in the demodulator, app_core/radio/demod/fm.py) are now written into the audio source's metadata dict, mirroring the existing rf_signal_strength pattern. stereo_pilot_strength was already flowing through; this was the one missing field. Nothing else needed for persistence -- eas_monitoring_service.py's once/sec metrics snapshotter already dumps the whole metadata dict to the audio_source_metrics table.
• New endpoint GET /api/audio/sources/<name>/signal_quality/history (webapp/admin/audio_ingest/routes_signal_quality.py), copying routes_rbds.py's query/downsample pattern against the same table. Deliberately excludes RF signal strength: that's already charted from a different, hardware-level source in this page's existing "Historical Trends" panel (static/js/radio_diagnostics_trends.js, backed by app_core/radio/trends.py) -- a second, differently-sourced "signal strength" series next to it would invite "why don't these agree" confusion, not add information.
• tests/test_signal_quality_history_api.py: new tests (5) covering point extraction, the deliberate RF-signal-strength exclusion, skip-on-neither-field, unknown source/bad query param handling, and downsampling -- mirrors tests/test_rbds_history_api.py's existing coverage shape for the sibling endpoint.

[3.2.6] - 2026-09-12 - Fix AudioArchiver dropping chunks on every segment flush

• Fixed: AudioArchiver for sdr-wbks (and every other archived source) was dropping hundreds to thousands of audio chunks on the source's BroadcastQueue subscription around every segment boundary ("consuming slower than real time"), observed live climbing past 92,000 total dropped chunks. Root cause: _flush_segment() ran inline in _archive_loop(), and encoding a full segment via FFmpeg (measured live: ~14s wall time for a 600s 48kHz-stereo segment on this Raspberry Pi) blocked the loop from calling audio_queue.get_nowait() for that entire window -- so on every single flush, the upstream BroadcastQueue filled and started dropping chunks for this subscriber only.
• Fixed in app_core/audio/archiver.py: _start_flush_async() now snapshots and resets the in-memory chunk buffer synchronously, then hands the actual concatenate/encode/write/prune work to a background thread -- the archive loop keeps draining the queue in real time regardless of how long encoding takes. stop() now waits (bounded, 60s) for the loop thread, which itself waits for the final flush thread, so a shutdown can't cut off the last segment mid-write.
• tests/test_audio_archiver_async_flush.py: new regression test asserting the chunk buffer is reset (and can accept new audio) while a simulated slow encode is still running in the background, plus a no-op check when there's nothing to flush. Also manually end-to-end verified (real BroadcastQueue + real WAV encode) that segments still write correctly with intact audio and accurate stats.

[3.2.5] - 2026-09-12 - Fix stereo SDR audio doubling: slow Icecast playback and broken SAME/RMT decode

• Fixed: The live Icecast relay for stereo SDR receivers (e.g. sdr-wbks.mp3, WFM+RBDS) played back audibly slow -- deep, dragging pitch -- confirmed by measuring the running FFmpeg encoder's actual stdin byte rate (/proc/<pid>/io) against real elapsed time: it was reading raw PCM at ~1.95x the byte rate -ar 48000 -ac 2 calls for in real time, while services/demod/worker.py's Redis publish rate was independently verified to be exact real-time (bisected by subscribing to the demod:audio:<id> channel directly and comparing decoded sample count to wall-clock elapsed time).
• Root cause: services/demod/worker.py publishes stereo audio as np.column_stack((left, right)) -- a (frames, 2) array -- which .tobytes() flattens to interleaved L,R,L,R floats on the wire; the wire format carries no shape metadata, so RedisSDRSourceAdapter._unpack_audio_envelope() could only hand back a flat 1-D array. IcecastOutputStreamer._samples_to_pcm_bytes() (app_core/audio/icecast_output.py) saw ndim == 1 and treated every one of the resulting 2x-as-many flat values as an independent mono sample, then upmixed each to fake stereo -- doubling the frame count handed to FFmpeg per real second of audio and destroying the true L/R image in the process. FFmpeg still assumed config.sample_rate frames/sec, so the encoded stream represented ~2x the declared audio-duration per real second, which plays back at roughly half speed.
• Same root cause also corrupted SAME/EAS decoding on any stereo-enabled SDR receiver: AudioSourceAdapter._resample_for_eas() (app_core/audio/ingest.py) only downmixes stereo to mono when it receives a (frames, 2) array (if audio_chunk.ndim == 2: audio_chunk = audio_chunk.mean(axis=1)). Since the flat, mis-shaped array was always ndim == 1, that downmix never ran -- the interleaved L,R,L,R values were fed straight into the 48kHz-to-16kHz EAS resampler as if they were sequential mono samples, both scrambling the channels together and carrying the same ~2x time-base error into the SAME decoder's input. SAME/RMT header detection depends on precise AFSK tone frequencies at an exact 520.83 baud rate, so this would reliably have prevented the decoder from locking onto a real SAME preamble (Required Monthly Test or otherwise) on this receiver.
• Fixed in app_core/audio/redis_sdr_adapter.py: _redis_subscriber_loop() now reshapes the flat array to (-1, config.channels) immediately after unpacking, using the adapter's own known channel count, before it ever reaches the broadcast queue -- fixing it for every consumer (Icecast, the archiver, EAS monitoring's SAME decoder), not just the one mount that surfaced it.
• tests/test_redis_audio_pipeline.py::test_subscriber_loop_reshapes_stereo_audio_envelope: regression test publishing a real (frames, 2) stereo envelope (matching what fm.py's demodulator actually returns) and asserting the dequeued chunk keeps its (frames, 2) shape rather than coming back flat.

[3.2.4] - 2026-09-12 - Fix CAP poller hanging for minutes per cycle on broken IPv6

• Fixed: poller/cap_poller.py poll cycles were taking minutes instead of seconds. Root cause: this host's router advertises an IPv6 default route via RA, but has no working IPv6 upstream (SYN packets to public IPv6 addresses get no reply, confirmed via ss -tnp showing a stuck SYN-SENT socket and ping -6 to a public host timing out). apps.fema.gov and api.weather.gov are dual-stack, so Python's requests/urllib3 resolved and attempted each unreachable IPv6 address in turn, burning the full per-connection timeout before ever falling back to the working IPv4 address. Forced IPv4-only DNS resolution for the poller process via urllib3.util.connection.allowed_gai_family, which avoids the IPv6 attempts entirely regardless of what the network advertises.
• Investigated a companion report of "wrong rtlsdr sample rate" causing slowness: the active wbks receiver is configured for 1,024,000 Hz (a standard, driver-recommended RTL-SDR rate), decimated by design to 256,000 Hz effective for FM+RBDS processing (app_core/radio/decimation.py). eas-station-sdr.service logs show steady, drop-free throughput at the configured rate with buffer utilization under 2% -- no evidence of a sample-rate-driven slowdown in the capture pipeline itself. Not changed pending clarification of what symptom was observed.

[3.2.3] - 2026-09-11 - Close out the two items flagged after Phase 4

• Investigated, not a bug: data_management.html's generic boundary uploader offers a county type alongside electric/fire/school/etc; checked whether it duplicates /admin/county_boundaries. It writes to Boundary (table boundaries, arbitrary type string, map-overlay/intersection use) while /admin/county_boundaries writes to USCountyBoundary (table us_county_boundaries, FIPS/GEOID-keyed, Census TIGER/Line data for SAME-code alert matching) -- different tables, different purpose. Documented in docs/roadmap/SITE_REORGANIZATION.md; no code change.
• Fixed: CodeQL's py/command-line-injection alert on the System Upgrade ref handling (webapp/admin/maintenance/routes_operations.py). The regex-validate-then-re-derive pattern introduced for this in v3.2.0 didn't satisfy CodeQL when the validation lived behind a shared helper function (_validate_git_ref) -- its sanitizer recognition doesn't reliably trace a value re-derived inside a called function back to the caller. Inlined the same _GIT_REF_PATTERN.match() + match.group(0) re-derivation directly into both call sites (check_for_upgrade, run_one_click_upgrade); removed the now-unused helper.
• tests/test_upgrade_progress.py: added regression coverage for both call sites rejecting invalid refs (leading-dash flag injection, range expressions, shell metacharacters) before any subprocess call is made, plus new coverage for run_one_click_upgrade generally (previously untested) -- replaces the deleted tests/test_git_ref_validation.py, which tested the removed helper directly.

[3.2.2] - 2026-09-11 - Site reorganization Phase 4: remove duplicate Zone Catalog tab from Data Management

Completes a further pass of docs/roadmap/SITE_REORGANIZATION.md. data_management.html's "Zone Catalog" tab called the exact same /admin/zones/* endpoints (webapp/admin/zones.py) as the pre-existing, separately nav-registered /admin/zones page -- confirmed by grepping for every route string both templates' JS calls and finding exactly one backend implementation of each. An earlier extraction's docstring had claimed this was "confirmed not a duplicate" -- true then or not, it didn't hold up under inspection today.

Fixed

• templates/admin/data_management.html: removed the duplicate Zone Catalog tab and its now-unused zone-catalog.js include; added a link to the pre-existing /admin/zones page instead. The Boundaries + Manage tabs (a genuinely cohesive upload-then-browse/delete workflow) are untouched.
• webapp/admin/dashboard.py: corrected the data_management_page() docstring's stale "confirmed not a duplicate" claim.
• webapp/navigation/registry_settings.py: updated the Data Management NavItem description to drop the zone-catalog mention.
• Noted but not fixed (see the roadmap doc): data_management.html's generic boundary uploader also offers a county type, which may or may not overlap with /admin/county_boundaries' dedicated loader -- unconfirmed, flagged for later.
• tests/test_data_management_zone_dedup.py (2 tests): the tab and dead references are gone; the page links to /admin/zones; the remaining tabs are untouched.

[3.2.1] - 2026-09-11 - Site reorganization Phase 3: merge Custom Screens documentation into Help, fix a dead link

Completes docs/roadmap/SITE_REORGANIZATION.md's original scope. templates/screens.html's "Documentation" tab (template variables, data sources, LED/VFD JSON examples) was reference material for a different audience than the operational screen-management UI it shared a page with -- and it linked to /static/docs/guides/CUSTOM_DISPLAY_SCREENS.md, a file that has never existed in this repo. help.html already had a higher-level "Custom Display Screens" section with the same dead link.

Fixed

• templates/help.html: the "Custom Display Screens" accordion section now includes the Template Variables table, Available Data Sources list, and LED/VFD JSON examples that used to live only on /screens' Documentation tab. The dead CUSTOM_DISPLAY_SCREENS.md link is removed from both places it appeared (there was never a corresponding file to fix instead).
• templates/screens.html: removed the redundant "Documentation" tab/pane and its now-unused .doc-card CSS; added a "Documentation" button in the page header linking to /help.
• tests/test_screens_help_docs_merge.py (2 tests): /screens no longer embeds the removed tab or the dead link; /help has the merged reference content and the dead link is gone there too.

[3.2.0] - 2026-09-11 - Site reorganization Phase 2: System Upgrade gets its own page, duplicate boundary-recalc card removed

Continues docs/roadmap/SITE_REORGANIZATION.md. templates/admin/operations.html's "Alert Boundary Coverage" card turned out to be a pure duplicate of functionality already on the existing, already-linked /admin/intersections page (same two endpoints, plus that page has more tools besides) — removed rather than split. System Upgrade is on the same backend blueprint as DB Health/Backup but is a much bigger, riskier operation with its own live progress/log-streaming UI, confusing enough to deserve full-page attention.

Changed

• New page System Upgrade (/admin/system-upgrade, maintenance.system_upgrade_page), with a new NavItem under Settings → Data & Storage. operations.html links to it instead of embedding the upgrade wizard/progress UI.
• operations.html's "Alert Boundary Coverage" card now links to /admin/intersections instead of duplicating its recalculation buttons.
• operations.html keeps DB Health and the quick Backup trigger. (That Backup trigger is itself a near-duplicate of /admin/backups' own "Create New Backup" section, via a different API — flagged in the roadmap doc as a possible future consolidation, not addressed in this phase.)
• tests/test_system_upgrade_page.py (3 tests): permission gate, the new page renders the upgrade UI, the main page no longer embeds either removed panel.

[3.1.0] - 2026-09-11 - Site reorganization Phase 1: split Bad Actor Blocklist out of Application Settings

Continues the initiative from docs/roadmap/SITE_REORGANIZATION.md. The Bad Actor Blocklist panel is a fully independent subsystem -- own nginx geo-map config files, own systemd refresh timer, own JSON API (webapp/admin/bad_actors.py) -- that only shared a page with logging/storage/branding/password-policy settings for lack of a better home. None of its inputs were even part of the settings <form> it visually lived inside.

Added

• New page Settings → Bad Actor Blocklist (/admin/security/bad-actors/), on the existing bad_actors blueprint. templates/admin/application_settings.html links out to it instead of embedding the panel; Project Honeypot (a genuine two-field settings toggle saved through the normal settings form, not an independent subsystem) stays on the main page.
• New NavItem in registry_settings.py's "Security & Access" group.
• tests/test_bad_actor_blocklist_page.py (3 tests): permission gate, the new page renders the panel, the main page no longer does.

[3.0.1] - 2026-09-11 - Fix stale "not a public subscription service" wording on /sms-compliance

Fixed

• templates/sms_compliance.html: the top disclaimer still read "Messages are sent only to phone numbers explicitly configured by the system administrator... This is not a public subscription service" -- stale from before the self-serve /sms-opt-in double opt-in flow shipped (2.232.0), and directly contradicted by Section 2 further down the same page, which describes that public page and its QR code for signage. A carrier/Twilio reviewer reading the filed policy URL and then clicking through to /sms-opt-in would see the mismatch immediately. Reworded to describe both opt-in paths accurately.

[3.0.0] - 2026-09-11 - Site reorganization: split SMS out of Notifications

The first landing of a broader initiative to split pages that bundle multiple unrelated features into their own dedicated pages -- see docs/roadmap/SITE_REORGANIZATION.md for the full scope and rationale. templates/admin/notifications.html had grown to 1,029 lines (email + SMS + SNMP + Postfix, plus SMS's own Consent Records and Message Log audit trails); SMS is a fully separate feature from Email/SNMP and only shared the page for lack of a better home.

Changed

• Breaking (URL addition, not a removal): SMS notification settings, the opt-in QR/link callout, Consent Records, and the SMS Message Log moved from /admin/notifications/ to a new page, /admin/notifications/sms (Settings → SMS Notifications). /admin/notifications/ still exists and works -- it now shows Email + SNMP + Postfix only. Each page links to the other.
• webapp/admin/notifications.py: new sms_settings() view and update_sms_settings() POST route (/admin/notifications/sms/update), split out of notification_settings()/update_notification_settings(). Kept as genuinely separate routes rather than one shared form-with-defaults handler -- that handler blanks out any field it doesn't see in the posted form, so a page that only submits SMS fields would have silently cleared Email/SNMP settings (and vice versa) had the routes stayed shared.
• webapp/navigation/registry_settings.py: new "SMS Notifications" NavItem alongside the existing "Notifications" item.
• docs/guides/SMS_OPT_IN.md, docs/policies/SMS_MESSAGING.md, docs/guides/notifications.md, templates/sms_compliance.html: updated to reference the new page location.
• docs/roadmap/SITE_REORGANIZATION.md (new): the prioritized plan for splitting the next few pages that bundle unrelated features -- explicitly scoped to not duplicate docs/development/LARGE_FILE_REFACTOR_PLAN.md's already-tracked backend-module-split and frontend-JS-extraction work.
• tests/test_sms_settings_page.py (new, 4 tests): the two update routes stay disjoint, the new page renders SMS content, the main page no longer does.

[2.233.0] - 2026-09-11 - Searchable SMS message log

An operator asked whether there's a record of SMS messages actually sent, searchable by number. There wasn't -- Twilio sends only ever produced a logger.info() line, so the only way to find "did we text this number" was to grep journalctl for a service you'd have to guess.

Added

• app_core/_models_sms_log.py (SmsMessageLog, migration 20260911_add_sms_message_log): one row per outbound SMS send attempt, recorded right next to each Twilio call in app_core/notifications/sms.py (not at the call sites) so every current and future send path is covered automatically. Records the recipient number, message type (alert / verification / test), event code (for alerts), success/failure, the Twilio SID, and the error on failure. Recording is best-effort and never raises -- a logging hiccup can't be mistaken for a send failure. Verification codes themselves are never stored, only the outcome.
• Settings → Notifications → SMS Message Log: the last 200 send attempts, most recent first, with a search box that filters by recipient phone number (?sms_log_search=).
• tests/test_sms_message_log.py (7 tests): logging never raises even if the DB write fails, each of the three send paths (alert/verification/test) logs with the right type and outcome, and the admin search filters by number.

[2.232.1] - 2026-09-11 - Make the SMS opt-in page discoverable, and instance-safe

The self-serve opt-in page from 2.232.0 was live but not linked from anywhere a visitor or a Twilio reviewer would naturally look, and had no way to hand someone a printable/scannable link. An operator confirmed they run more than one EAS Station deployment, so anything added here has to derive its URL from the actual request, never a hardcoded host.

Added

• Settings → Notifications: a QR code (/admin/notifications/sms-optin-qr.png, gated behind system.configure) next to the existing "share this link" callout, for signage or printed material. Generated fresh per request from url_for(..., _external=True) -- never a fixed hostname -- so each deployment's QR code always points at itself.
• /sms-compliance now links directly to /sms-opt-in in its Opt-In/Consent Disclosure section (instead of only describing the admin-added path), and renders the consent disclosure language from the same CONSENT_TEXT constant /sms-opt-in itself uses, so the two pages can't drift out of sync.
• tests/test_sms_optin_qr.py (3 tests): permission gate, and that the URL encoded into the QR reflects the requesting host -- proving two different instances get two different, correct codes.

[2.232.0] - 2026-09-11 - Public double opt-in for SMS alerts

A carrier/Twilio A2P 10DLC campaign review needs a verifiable opt-in flow to point at. The previous design — an administrator adds a phone number and attests consent was obtained elsewhere (verbally, on paper) — gave a reviewer nothing to click through.

Added

• New public page /sms-opt-in: a visitor enters their own phone number, agrees to explicit TCPA-style consent language, and confirms by entering a one-time code texted to that number (double opt-in). Only then is the number added to the live SMS recipient list.
• app_core/_models_sms_optin.py (SmsOptInRequest, migration 20260911_add_sms_opt_in_requests): one row per opt-in attempt — a verbatim snapshot of the consent text shown, the submitter's IP, and a nullable verified_at that's the actual evidence of confirmed consent.
• Settings → Notifications → Consent Records: an admin-facing audit table of every verified sign-up, plus a link to share the opt-in page.
• Abuse protection: rate-limited per IP (reusing /login's LoginRateLimiter) and per phone number (a 60-second resend cooldown, so a bystander can't be used to spam a number they don't control), a 5-attempt cap on wrong confirmation codes, and hashed (never plaintext) codes at rest.
• docs/guides/SMS_OPT_IN.md, and docs/policies/SMS_MESSAGING.md updated to describe both opt-in paths (self-serve and the legacy admin-added one, which still exists for cases the self-serve flow can't cover).
• tests/test_sms_optin.py (14 tests): consent/phone validation, already-subscribed short-circuit, IP and per-phone rate limiting, and the confirm step's wrong-code/expired-code/lockout paths.

[2.231.1] - 2026-09-11 - Rate-limit the MFA verification step, no-store the enrollment QR code

A follow-up to the pgweb pentest finding: while reviewing MFA (prompted by "anything else we can improve for 2FA"), found the login-time TOTP/backup-code check had no rate limiting at all, unlike the password step right before it.

Fixed

• webapp/admin/auth.py: /mfa/verify (the code entry screen after a correct password) had no rate limiting -- /login's own lockout only covers the password step. An attacker who already had valid credentials for an MFA-enabled account (phished, leaked, stuffed) could try unlimited TOTP/backup-code guesses against this endpoint. Now reuses the same LoginRateLimiter class /login already uses (5 attempts / 15-minute lockout), under a separate mfa:-prefixed bucket per IP so MFA guesses and password guesses don't share or exhaust each other's attempt budget.
• webapp/routes_security.py: the MFA enrollment QR code image (/security/mfa/enroll/qr) had no cache headers. The image encodes the TOTP secret in the otpauth:// URI its pixels represent -- without Cache-Control: no-store, an intermediate proxy or the browser's disk cache could persist a copy of the secret past the enrollment session.
• New tests/test_mfa_verify_rate_limit.py: pins down the lockout threshold, that a locked-out request is rejected without even checking the code, that a successful verification clears the bucket, and that the MFA bucket is separate from the password-login bucket.

[2.231.0] - 2026-09-11 - Authenticated access gate for the pgweb database browser

A pentest against a live deployment (following the security-audit pass in 2.230.0) found pgweb -- an optional, operator-installed third-party PostgreSQL browser with no authentication of its own -- listening on 0.0.0.0:8081, with a firewall rule allowing the entire LAN subnet in. Anyone on that LAN got full, unauthenticated read/write SQL access to the production database, including administrator accounts; the nav registry even linked directly to the raw port with a description acknowledging the risk rather than closing it.

Added

• config/nginx-eas-station.conf: new listen 8081 server block that proxies to pgweb only after an auth_request subrequest confirms the caller has a logged-in session with the system.configure permission -- the same gate this app's other highest-sensitivity admin actions (e.g. downloading the TLS private key) already use. A denied request is redirected to /login instead of reaching pgweb.
• app.py: /api/internal/pgweb-auth-check, the endpoint that auth_request subrequest calls. Deliberately hand-written rather than using @require_permission -- that decorator's denial path redirects for a non-JSON request instead of returning a bare status, and nginx's auth_request module treats anything other than 2xx/401/403 as an upstream error (producing a 500 for the real client), not a denial.
• bin/eas-station-pgweb-launch.sh, systemd/eas-station-pgweb.service: corrected, repository-tracked versions of this box's ad-hoc setup, binding pgweb to 127.0.0.1 only (an internal port nginx proxies to) instead of 0.0.0.0.
• Settings → Data & Storage → Database Browser (pgweb) (webapp/admin/database_browser.py, /admin/database-browser/): shows whether pgweb is installed/running and links to the authenticated port. Replaces the nav registry's previous raw, hardcoded-IP link to the unauthenticated port directly.
• docs/guides/DATABASE_BROWSER.md: setup, removal, and the access-control design above.
• tests/test_database_browser.py: pins down the exact status codes nginx's auth_request contract needs (401 unauthenticated, 403 authenticated-without-permission, 200 authenticated-with-permission -- never a redirect) and the status page's rendering for both installed states.

Why this wasn't caught by the earlier audit

The five-area security audit that produced 2.230.0 covered this application's own code; pgweb is a third-party binary an operator installs outside that code entirely; nothing in the codebase's automated checks (tests, CodeQL, dependency scanning) has visibility into a manually-run process's bind address or an ad-hoc firewall rule. Found only by actually enumerating the live host's listening ports during a hands-on pentest of the deployed system.

[2.230.0] - 2026-09-11 - Security audit fixes: TLS key exposure, WebSocket auth, stored XSS, broken access control

A full security audit (injection/command execution, authN/authZ, file handling/SSRF, secrets/config, dependencies/shell scripts) turned up the issues below. Ranked by severity; deferred items are noted with why.

Fixed — Critical

• config/sudoers-eas-station, install.sh, update.sh, webapp/admin/certbot/paths.py: the certbot working directory (certbot_data/) was chmod -R 777'd, making the TLS private key world-readable and world-writable to any local user/process. Now 755 for the tree (still gives the web app read+traverse for its .exists()/directory-listing/openssl calls, none of which ever read a private key's bytes) plus privkey*.pem locked to 600 (root-only) specifically -- nothing in the codebase reads a key's content as the eas-station user, and nginx's master process reads it as root anyway. Ownership stays root:root unchanged, preserving the existing AppArmor renewal-compatibility fix.

Fixed — High

• app.py: the Socket.IO server had no connect-time authentication and defaulted cors_allowed_origins to '*' when unset -- any website (or any non-browser HTTP client, since CORS is browser-enforced only) could open a WebSocket to this server and receive live system logs, GPIO/audio/broadcast status, and alert summaries with zero credentials. Added a connect handler requiring a valid session, and stopped forcing the '*' CORS fallback.
• templates/alert_detail.html, templates/audio_detail.html, templates/manual_eas_print.html: CAP alert description/instruction/message_text rendered through |safe with no HTML escaping -- a spoofed/compromised NOAA/IPAWS feed could plant a stored-XSS payload that executes in any viewer's browser. Now escaped (|e) before the newline-to-HTML formatting that legitimately needs |safe.
• webapp/admin/dashboard.py: /api/admin/sessions (GET) had no permission decorator while its sibling endpoints (terminate/bulk-terminate) both require system.manage_users -- any authenticated user of any role could list every admin's username, IP, and session activity. Now requires system.manage_users, matching its siblings.
• webapp/admin/environment.py, templates/admin/environment.html: /admin/environment/download-ssl-key required only system.view_config (the same permission as read-only settings pages) instead of system.configure -- a read-only "viewer" role could exfiltrate the TLS private key. Escalated to system.configure; the download button is now hidden from accounts without it.

Fixed — Medium

• webapp/admin/maintenance/routes_operations.py: the ref/checkout request parameters reached git fetch/git checkout unvalidated -- a value like --upload-pack=<cmd> is a known git-argument-injection primitive. Both now validated against a strict branch/tag character-class before use.
• app_core/radio/demod/types.py, services/demod/worker.py, app_core/audio/redis_sdr_adapter.py: the demod worker's cross-process status handoff used pickle.dumps/pickle.loads over Redis -- an arbitrary-code-execution primitive the moment anything untrusted can write to that key. Replaced with plain JSON (demodulator_status_to_json_dict/_from_json_dict), base64-encoding the two bytes-typed RBDS fields and string-keying the two int-keyed dict fields JSON can't represent natively. Verified with a round-trip test before and after.
• webapp/admin/boundaries.py: upload_shapefile's shapefile_path form field accepted any filesystem path with only an .exists() check, unlike every sibling upload path in the same file -- an arbitrary-file-read primitive (pyshp error text can reveal whether an out-of-tree file exists and something about its shape) for an authenticated admin. Now resolved and containment-checked against the configured shapefile directory, mirroring webapp/routes_backups.py's resolve_backup_path().
• install.sh, update.sh, scripts/diagnose_502_504.sh, scripts/fix_website_504.sh, scripts/diagnose_smart.sh, scripts/update_bad_actors.sh: several root-run scripts wrote to predictable /tmp/* filenames (venv-creation.log, dep_check.txt, pip-install.log, test_smart.py, argon1v5.sh, bad-actors-nginx-test.log) -- a classic symlink race (CWE-377) where a local user can pre-plant that exact path pointing at an arbitrary file before the script runs. All switched to mktemp; fix_website_504.sh's documented curl -o /tmp/fix_website.sh ... && sudo bash usage instructions updated to the same pattern (a curl-then-sudo bash on a fixed path is also a race on the downloaded content, not just the log).

Fixed — Low

• app_core/_models_admin.py: the legacy SHA256-hash upgrade path compared hashes with plain == instead of a constant-time compare -- a timing side-channel, now hmac.compare_digest.
• webapp/admin/dashboard.py, webapp/admin/boundaries.py, webapp/public/logs.py: /admin/rbac, /admin/sessions, /admin/gpio/statistics, /admin/list_shapefiles, /logs, /logs/export.csv, /logs/export.pdf rendered for any authenticated user, relying only on the app-wide login gate rather than a specific permission like their sibling APIs already use. Now gated on system.view_users, system.manage_users, gpio.view, system.configure, and logs.view/logs.export respectively.

Deliberately deferred (documented, not fixed this pass)

• Sudoers wildcards (sed -i * /etc/nginx/sites-available/eas-station, ln -s * /etc/letsencrypt/live/*, cp /tmp/*.xml /etc/icecast2/icecast.xml): broader than the app's current hardcoded call sites need, but narrowing them risks silently breaking SSL certificate installation or Icecast config in a way that can't be verified without a live certbot/nginx pipeline. Not currently exploitable per the audit (no user input reaches these arguments today).
• CSP 'unsafe-inline' on script-src/style-src (app.py): removing it would require migrating every inline <script> across the template tree to a nonce-based CSP -- a large, high-blast-radius refactor out of scope for this pass. The stored-XSS root cause it would have provided defense-in-depth against is already fixed above.
• Icecast "test connection" SSRF-by-design: no host allow-list, but blocking private/internal targets would break the legitimate case of testing an Icecast server on the operator's own LAN. Already admin-gated (system.configure).
• Four systemd services running as root (certbot.service, eas-station-postal.service, bad-actors-update.service, eas-station-failure-recovery@.service): certbot.service genuinely needs root (matches the ownership design above); the other three write to root-owned system paths (/etc/nginx/, the Docker socket) that would need their own sudoers/script rework to safely demote, without a way to verify the result outside a live deployment.

[2.229.2] - 2026-09-10 - Dithered "desktop" backdrop behind the DOS-installer banner and completion screens

Added

• scripts/lib/ui.sh: ui_banner and show_celebration now center their box on a full-width, stippled two-tone "desktop" (a ▒ MEDIUM SHADE fill in the same grey-on-blue as the box border) instead of leaving plain terminal background on either side -- matching the floating-box-on-textured-backdrop look of the reference DOS installers (DOOM Setup, DOSBox config, Beneath a Steel Sky), rather than just a solid-blue box with black on both sides. Falls back to no margin/no dither on a terminal narrower than the box itself.
• This only applies to the two screens this file draws by hand. The live whiptail --gauge progress screen can't carry it: newt repaints its own root as a flat color fill on every redraw (confirmed by pre-filling the screen with the same dither pattern and watching whiptail's first paint wipe it), so dithering it would require replacing whiptail with a fully custom-drawn progress display.

[2.229.1] - 2026-09-10 - DOS-installer blue theme for every whiptail dialog, and a black-gutter box-art bug fix

Added

• scripts/lib/ui.sh: every whiptail dialog install.sh/update.sh show (--yesno, --msgbox, --gauge, --menu, ...) now renders in a solid Turbo-Vision-blue NEWT_COLORS theme, instead of newt's own default grey -- this is what actually ties the whole install experience together into one consistent DOS-installer look end to end, matching the hand-drawn banner/completion box screens rather than clashing with them. Respects a caller-supplied NEWT_COLORS if one is already set.
• The --gauge progress bar's filled portion is now green instead of red, which read as an error/danger color rather than progress.

Fixed

• _DOS_GREY used \033[0;37m, whose leading 0 resets ALL SGR attributes -- including the active blue background -- before setting the grey foreground. Inside the hand-drawn blue-background box screens (ui_banner, show_celebration), this silently knocked the background back to black for every box-rule character (╔═╗║╚╝) and every grey-colored label, leaving a black gutter around what should have been a solid blue box. Now \033[37m (foreground only).

[2.229.0] - 2026-09-10 - Progress bars now show elapsed/remaining time, and the alert-verification bar no longer freezes after 5 minutes

Added

• The alert-verification audio-decode progress overlay (templates/eas/alert_verification.html) and the one-click System Upgrade progress bar (templates/admin/operations.html) both now show elapsed time and a linear-extrapolation "About Nm Ns remaining" estimate, computed the same way in both places: elapsed / (percent / 100) - elapsed.
• webapp/routes/alert_verification/progress.py: ProgressTracker now tracks each operation's wall-clock started_at (recovered from the on-disk payload so a later phase's tracker doesn't reset the clock) and reports elapsed_seconds/eta_seconds alongside percent.
• webapp/admin/maintenance/routes_upgrade_progress.py: /admin/operations/upgrade/progress now also reports elapsed_seconds/eta_seconds, derived from systemd's ActiveEnterTimestampMonotonic property paired with /proc/uptime -- immune to timezone/NTP wall-clock issues that a parsed ActiveEnterTimestamp string would have.

Fixed

• The alert-verification progress overlay stopped polling for genuine completion after 5 minutes, freezing the bar at a fake 90% -- a real accuracy bug, since a run that legitimately took longer than 5 minutes could no longer show it had finished. Polling now keeps running indefinitely; only the status message degrades after 5 consecutive failed polls.
• Fixed a threading.Lock deadlock in ProgressTracker.update()/complete()/error(): the new elapsed/ETA calculation could recover started_at via the same-named, also-locking ProgressTracker.get(), which deadlocks if called from inside the already-held lock. The timing calculation now always runs before the lock is acquired.

[2.228.23] - 2026-09-10 - Share cards: same information across every aspect ratio, and instruction can no longer get crowded out

The narrow (landscape, FB/X/LinkedIn) share card and the wide cards (square/portrait/story, Instagram/TikTok/Snap) used to show genuinely different information for the same alert -- the narrow layout only drew HEADLINE/DESCRIPTION as a fallback when no threat-specific content rendered, and never drew AFFECTED AREAS or COVERAGE at all. All layouts must carry the same information regardless of width.

Fixed

• app_utils/image_export/render.py: HEADLINE, AFFECTED AREAS, DESCRIPTION and COVERAGE now render unconditionally in the narrow layout too, matching the wide layout's content set.
• Rendering an actual sample card with this fix exposed a second, related bug in both layouts: on a content-dense product (storm threat data + a long NWS headline), HEADLINE + AFFECTED AREAS + DESCRIPTION could fill the entire info panel before ever reaching INSTRUCTION, silently dropping "move to an interior room" off the bottom while less safety-critical narrative text survived. INSTRUCTION now draws immediately after the threat-summary section in both layouts, so space pressure can only clip the narrative sections (which already degrade gracefully everywhere else in this file), never the one thing on the card that tells someone what to physically do.

New tests in tests/test_image_export_broadcast_panels.py and tests/test_image_export_themes.py cover both the content-parity fix and the instruction-ordering fix, for both layout families, at the unit (drawer call order) and end-to-end (rendered pixel) level -- verified each new test actually fails against the pre-fix code before confirming it passes against the fix. Full test suite (3126 tests) passes.

[2.228.22] - 2026-09-10 - Cache the share-card radar overlay fetch

An audit of app_utils/image_export/ found _fetch_radar_overlay() (maps.py) doing an uncached synchronous WMS network fetch on every call, unlike its sibling basemap-tile fetcher in tiles.py, which already has a two-level (in-memory LRU + disk) cache. This isn't just an on-demand cost: generate_alert_image() runs on an automated path -- app_core/notifications/alert_image.py's build_alert_image() calls it for every notification email the CAP poller and audio monitoring services send for weather alerts. During a severe-weather outbreak, several warnings issued minutes apart, often with overlapping bounding boxes, frequently land in the same 5-minute WMS time bucket -- each previously triggered its own independent network fetch of what's very possibly the identical radar tile.

Fixed

• Added an in-memory LRU cache (_RADAR_CACHE, 32 entries) to _fetch_radar_overlay(), keyed on (tx_min, ty_min, tx_max, ty_max, z, canvas_w, canvas_h, scan_time). Deliberately no disk-backed second tier like tiles.py's -- unlike the fixed, immutable basemap tile grid, radar bboxes are per-alert-specific (far higher cardinality) and the key is already self-expiring (a new 5-minute time bucket naturally ages out prior entries), so a disk cache would only grow unbounded for one-off bboxes never fetched again. HTTP errors are never cached, so a transient WMS failure can't poison the cache for the next (retryable) request in the same time bucket.

New tests in tests/test_image_export_radar_overlay.py: a second call with identical params skips the HTTP fetch entirely; different bbox or different 5-minute time bucket are both cache misses; HTTP errors aren't cached. Added an autouse fixture clearing the now-module-level cache between tests -- without it, two pre-existing tests (..._returns_none_on_http_error, ..._returns_none_on_network_exception) would silently return a previous test's cached success instead of exercising their own mock; verified this by temporarily removing the fixture and confirming both failed exactly as expected before restoring it. Full tests/test_image_export_* suite (194 tests) passes.

[2.228.21] - 2026-09-10 - Fix a numpy/numba dependency conflict that took the whole site down mid-update

update.sh stops every eas-station service, updates the main venv, updates the SDR venv, then restarts everything. requirements.txt's numba pin was raised to >=0.67.0,<0.68.0 (needed for numpy>=2.5.2 -- every numba<0.65 release caps numpy at <2.4) but requirements-sdr.txt was never updated to match, still capped at <0.64.0. The main venv install succeeded; the SDR venv install then hit pip's ResolutionImpossible on numpy>=2.5.2 vs. numba<0.64.0's numpy<2.4 requirement, and update.sh exited without ever reaching its restart step -- the site was down until this was found and fixed by hand.

Fixed

• requirements-sdr.txt's numba pin now matches requirements.txt's (>=0.67.0,<0.68.0). Verified with a fresh dependency resolve: resolves cleanly to numpy-2.5.3, numba-0.67.0, llvmlite-0.49.0.

New tests/test_requirements_sdr_numba_numpy_sync.py: asserts the two files' numba pins stay identical and that requirements-sdr.txt's numpy floor is never below requirements.txt's pin. Verified it actually catches the original bug (fails against the pre-fix <0.64.0 pin, passes against the fix).

[2.228.20] - 2026-09-10 - Database audit: missing timestamp indexes, redundant Redis fetch, three N+1 query patterns

A four-way parallel audit (DB query patterns, background schedulers, Redis/connection pooling, Flask route handlers) turned up one critical, well-corroborated finding and several smaller ones.

Fixed

• system_log.timestamp had no index. app_core/_models_admin.py. SELECT * FROM system_log ORDER BY timestamp DESC LIMIT 20 cost 500ms-1s -- a parallel sequential scan across all 440k rows (810MB) plus a sort, just to fetch 20 rows. Hit every 10 seconds forever by websocket_push.py's _emit_logs_update (inside the persistent slow-loop session), plus the /logs page. Verified live: EXPLAIN (ANALYZE, BUFFERS) went from 1023.8ms to 0.055ms after adding the index -- roughly 18,600x. New migration 20260910_add_timestamp_indexes.
• Same gap on poll_history.timestamp (app_core/_models_polling.py), same migration -- 27-36ms per hit across several /logs-related routes and _emit_ipaws_status_update, smaller table but identical root cause.
• app_core/websocket_push.py: _emit_audio_sources_update and _emit_audio_health_update both run on the same 30s interval starting from the same zero offset, so they land on the same tick and each independently re-fetched the identical Redis metrics hash. Added a 1-second-TTL cache (_read_audio_metrics_cached) shared between them; the 4Hz fast-loop emit is untouched (wants every tick's freshest read).
• webapp/admin/maintenance/routes_import.py: the manual NOAA alert import endpoint issued one CAPAlert.query.filter_by(identifier=...).first() per feature in the response instead of one batched .filter(...in_(...)) lookup for the whole payload -- fine for a single alert, scales badly for a large historical backfill. Batched the existence check; preserved the original per-iteration behavior for a duplicate identifier appearing twice in one payload (must become insert-then-update, not a duplicate-key crash) by updating the lookup dict as each new row is inserted. New tests/test_import_alert_batching.py covers both the normal insert/update split and that regression case specifically.
• webapp/radio_settings/routes_diagnostics_status.py: the SDR metrics-to-UI conversion looked up each receiver's DB id with its own query inside the per-receiver loop (polled every 15s while the Radio Diagnostics page is open). Batched into one .filter(identifier.in_(...)) lookup before the loop.
• app_core/alert_purge.py's _delete_orphaned_messages (6-hourly auto-purge sweep) ran one query per candidate message id to check whether it was still referenced by a received_eas_alerts row. Batched into one query for the whole id list.

Full test suite (3113 tests) passes.

[2.228.19] - 2026-09-10 - Guard the remaining /dev/tty writes in scripts/lib/ui.sh's static-UI helpers

update.sh --non-interactive runs with no controlling tty when launched via systemd-run (the Admin -> Operations "System Upgrade" button's path, per bin/eas-station-run-update). Most of scripts/lib/ui.sh already guards its /dev/tty writes behind _UI_HAS_CONTROLLING_TTY (~15 call sites), but _dos_goto_row(), _tty(), _tty_raw(), ui_banner()'s plain-terminal branch, ui_progress_bar()'s plain-terminal branch, and ui_progress_end() were missed. A 2>/dev/null redirect on the printf only suppresses printf's own runtime stderr -- it does not catch bash's own failure to open /dev/tty in the first place (ENXIO, no controlling tty), which bash reports directly to the script's current stderr before printf ever runs.

Fixed

• Added the same _UI_HAS_CONTROLLING_TTY guard already used everywhere else in the file to the six call sites above. Verified with a real before/after run under setsid ... </dev/null (no controlling tty): the unfixed version throws 3 /dev/tty: No such device or address errors from a two-line smoke test; the fixed version is silent.

[2.228.18] - 2026-09-10 - Apply the proven glibc malloc-arena fix to eas-station-audio.service

system_metric_samples showed a week-long sawtooth: system memory climbing steadily after every eas-station-audio.service restart, then resetting on the next one. Correlating with process RSS directly: eas_monitoring_service.py grew from ~400 MB fresh to 3.44 GB RSS over 2.8 days of uptime, while swap sat chronically 47-100% full the whole time.

Fixed

• eas_monitoring_service.py never had services/common/bootstrap.py's init_runtime() applied — the exact fix already proven on eas-station-displays.service, which cut that service's RSS from 9.68 GB to 320 MB. glibc defaults to one malloc arena per thread (up to 8x on a Pi 5), and this is the most heavily threaded eas-station process (websocket push fast+slow loops, gated-alert scheduler, per-source audio pipelines, ffmpeg feeder threads) -- 33 threads were already running within 90 seconds of a fresh restart. Added init_runtime("audio") as the first statement in main(), before any thread spawns (arena caps only bind threads created afterward), and mirrored eas-station-displays.service's systemd env vars onto eas-station-audio.service: MALLOC_ARENA_MAX=2, MALLOC_TRIM_THRESHOLD_=131072, and MEMDIAG_DUMP_DIR=/var/log/eas-station (wires up the SIGUSR1/SIGUSR2 memdiag hooks too, previously entirely absent on this service, for diagnosing any residual growth without guessing).

New tests/test_eas_monitoring_service_glibc_tuning.py: asserts init_runtime("audio") is imported and called, that it precedes the first thread spawn in main(), and that the systemd unit pins the same malloc tuning + MEMDIAG_DUMP_DIR as the already-fixed Phase 4 units. Confirmed the module still imports cleanly (existing tests in tests/test_broadcast_metadata_reconcile.py, tests/test_dead_air_monitoring.py, tests/test_audio_metrics_snapshot_writer.py already import eas_monitoring_service directly -- 56 tests pass).

[2.228.17] - 2026-09-10 - Fix a long-lived idle-in-transaction connection leak in the WebSocket push service, compress rotated app logs, and close a retention-sweep coverage gap

A routine database health check (pg_stat_activity) found several connections stuck idle in transaction for hours, each frozen at a bare SELECT with no following COMMIT/ROLLBACK -- exactly the kind of open snapshot that pins Postgres's vacuum horizon and blocks dead-tuple cleanup database-wide.

Fixed

• app_core/websocket_push.py's slow push loop (_push_worker_slow) holds one shared SQLAlchemy session open for the life of the process (intentional, to avoid per-tick app-context overhead at high frequency -- see the _recover_db_session() docstring), but only rolled that session back inside each emit's except block. A purely successful pass -- the common case -- never committed, so every SELECT-only emit (e.g. _emit_pending_alerts_update's gated_alerts query, the count(CAPAlert.id) in the system-health snapshot) left its read transaction open indefinitely. Added one db.session.commit() per 1 Hz loop iteration, after all emits run, to close out the transaction on the success path too. The 4 Hz fast loop is untouched -- it's confirmed DB-free by inspection, matching its own docstring's claim.

Changed

• The rotated eas_station.log backups (app.py's RotatingFileHandler, 10 MB × 5 backups) are now gzip-compressed on rotation via a custom rotator/namer pair, instead of sitting on disk as plain text.
• The retention sweep (app_core/retention.py) now also prunes the *.json metadata sidecar that sdr_hardware_service.py writes next to every *.npy IQ capture -- previously only the .npy itself matched the prune pattern, so every capture left a small orphaned sidecar behind forever. Also added a new fixed-age (30 day, not exposed in retention_settings -- these are internal diagnostics, not a sized data category) sweep step for eas-memdiag-*.txt snapshots and eas-station-*-startup-error-*.log crash dumps in EAS_LOG_DIR, neither of which previously had any cleanup mechanism at all.

Updated tests/test_retention.py (test_sweep_prunes_iq_and_temp_files, test_one_failing_step_does_not_stop_the_rest) for the new diagnostic_files_removed summary field and the additional guarded prune_directory() calls. Full tests/test_retention.py suite (20 tests) passes.

[2.228.16] - 2026-09-09 - Audit and fix every pure-FIR lfilter slow-path instance in the codebase

Continuing the CPU hunt: profiling the demod service directly (ground-truth per-thread /proc census, then re-filtering the existing py-spy profile excluding known idle leaves -- same disciplined methodology as 2.228.14/15) found numpy.convolve again, this time inside RBDSWorker._process_rbds(). Finding the same bug a third time (after drivers.py in 2.228.14) was the signal to stop fixing these reactively, one profiler hit at a time, and instead grep the entire codebase for every .lfilter( call and classify each by whether its a coefficient is a true multi-tap IIR (fine, hits scipy's fast path) or a pure-FIR 1.0/[1.0] (the bug).

Fixed

• RBDSWorker._process_rbds()'s 54-60 kHz bandpass (app_core/radio/demod/rbds_worker.py, applied at the pre-decimation rate -- the highest-leverage stage in the RBDS pipeline) and its 2.4 kHz post-mix lowpass both called scipy.signal.lfilter(..., [1.0], ...) -- the same pure-FIR case that unconditionally takes scipy's slow np.apply_along_axis(...) -> np.convolve fallback fixed in drivers.py back in 2.228.14. The genuinely interesting part: the surrounding comments explain the developer had already diagnosed and fixed a real bug here -- switching from a plain per-chunk np.convolve (stateless, so every chunk boundary produced a transient that flooded the RBDS bit-sync with garbage) to lfilter with a persisted zi delay line. That fix was correct, but the developer had no way to know lfilter's fast C path only activates for a true IIR filter (len(a) > 1) -- for a pure-FIR filter it falls back internally to the exact same np.convolve routine being moved away from, just wrapped behind a state-carrying API. Both filters now use overlap-add via oaconvolve (FFT-based, and its convolution "tail" carries state across chunks exactly like zi did -- same technique as 2.228.14's fix and FMDemodulator._mono_audio_lowpass). The lowpass filter also drops its real/imag lfilter split entirely -- oaconvolve handles complex input natively.
• FMDemodulator.demodulate()'s RBDS early-decimation anti-alias filter (app_core/radio/demod/fm.py, the filter that decimates the multiplex down to the RBDS worker's intermediate rate when a receiver's raw SDR rate exceeds ~500 kHz) had the identical lfilter(..., 1.0, ...) pure-FIR pattern. Not currently exercised by the live wbks receiver (its 256 kHz effective rate needs no further RBDS-path decimation), but would hit full force the moment any higher-rate receiver -- including the disabled Airspy config already in this deployment's database, or an RTL-SDR bumped back toward 1 MHz+ -- is enabled. Fixed with the same overlap-add technique.
• scripts/rbds_diagnose.py (the offline RBDS troubleshooting tool operators run against a captured IQ file) had the same pattern in both its bandpass and post-mix lowpass. Not a live-service cost, but fixed for consistency and so the tool runs faster when someone actually needs it.
• Audited and confirmed correct, left unchanged: RBDSWorker._apply_interference_notch's scipy.signal.iirnotch-derived filter (a genuine multi-tap IIR, len(a) == 3) and FMDemodulator._apply_deemphasis's one-pole de-emphasis filter (a = [1.0, alpha - 1.0], len(a) == 2) -- both true IIR filters that correctly hit scipy's fast path already.

New tests: test_rbds_bandpass_oaconvolve_matches_lfilter_ground_truth, test_rbds_lowpass_oaconvolve_matches_lfilter_ground_truth, test_rbds_aa_filter_oaconvolve_matches_lfilter_ground_truth (numerical equivalence against each filter's mathematical ground truth -- a single lfilter call) and test_rbds_bandpass_chunked_matches_single_continuous_call (proves the tail-carry state stitches irregular real-world chunk boundaries seamlessly), all in tests/test_rbds_demodulation.py. Existing filter-behavior tests (test_rbds_post_mix_lowpass_rejects_stereo_sideband_artifact, frequency-response and stopband-rejection checks, not just call shape) continue to pass unchanged.

Live-verified on /opt before opening the PR: wbks decoding cleanly after the rbds_worker.py fix deployed -- 3439/3442 blocks OK (99.9%), full PS name and RadioText, only 2 sync-loss events total (both from the retune transient, not ongoing instability).

Live-verified on /opt before opening the PR, same discipline as 2.228.14/15.

[2.228.15] - 2026-09-09 - Batch the SDR publisher's per-chunk Redis calls into one round trip

Continuing the CPU hunt after 2.228.14: this time built a proper ground-truth measurement first, rather than trusting py-spy record's wall-clock sampling directly (a time.sleep() or blocked syscall shows up in a wall-clock profile proportional to how long it took, not how much CPU it used -- exactly the pitfall that produced the wrong diagnosis in 2.228.13). Reading utime+stime straight from /proc/<pid>/task/<tid>/stat over a fixed window gives real, unambiguous per-thread CPU consumption.

That census confirmed sdr_hardware_service.py's two hot threads (RTLSDRReceiver-wbks capture and SDR-Publisher) both do carry genuine CPU load (not just I/O wait) -- combined ~42.4% of a core. Re-filtering the existing py-spy profile to just the publisher thread's frames, and excluding its time.sleep() idle leaf this time, showed Redis client protocol overhead (command send + reply read + parse, repeated per call) at ~42% of that thread's real busy time -- more than zlib.compress alone.

Fixed

• publish_samples_and_metrics() (sdr_hardware_service.py) issued up to four separate synchronous Redis round trips per chunk -- publish() (IQ samples, every chunk), setex() (spectrum, rate-limited to every 100ms), and hset() + expire() (ring-buffer stats, every chunk, unlike the other two which are throttled). Each round trip pays the full redis-py call chain (execute_command -> _execute_command -> call_with_retry -> _send_command_parse_response -> parse_response -> read_response -> read_from_socket) even though none of these calls' return values were ever used. Now queues whichever of these are due each iteration onto one non-transactional redis_client.pipeline(transaction=False) and executes once. No behavior change -- same commands, same order, same effects, just one network round trip instead of up to four.

Live-verified before opening this PR, same discipline as 2.228.14: deployed directly to /opt, confirmed wbks still decoding correctly (stereo pilot locked, RBDS synced, real PS name/RadioText), confirmed the ring-buffer-stats and spectrum Redis keys still populate (proving the pipelined commands still execute), and re-ran the ground-truth per-thread /proc census: total CPU on sdr_hardware_service.py dropped from 42.4% to 38.9% of a core. Smaller than 2.228.14's win (this only removes redundant round-trip overhead, not a slow-path compute bug), but real and measured the same rigorous way.

[2.228.14] - 2026-09-09 - Correction: the real 2.228.13 fix is oaconvolve, not a real/imag lfilter split

2.228.13's diagnosis was wrong, caught by re-profiling live rather than trusting the theory. After deploying 2.228.13 to /opt and restarting eas-station-sdr.service, a fresh py-spy record showed numpy.convolve still dominant (29.7% of samples, barely down from 38.7%) -- top even showed the process running hotter (68.7% vs. the original ~60%).

Fixed

• Reading scipy's actual lfilter source (_signaltools.py) showed the real condition: the fast C path (_sigtools._linear_filter) only activates when len(a) > 1 -- a true IIR filter. This anti-alias filter is pure FIR (a=1.0), so any lfilter call on it -- real or complex input, split or not -- unconditionally takes the slow np.apply_along_axis(...) -> np.convolve fallback. The real/imag split in 2.228.13 ran two calls through the identical slow path instead of one, which is why CPU didn't meaningfully improve. app_core/radio/drivers.py's _capture_loop now replaces lfilter entirely with overlap-add via scipy.signal.oaconvolve (FFT-based, handles complex input natively, no split needed) -- the same technique FMDemodulator._mono_audio_lowpass already uses successfully. Benchmarked directly: oaconvolve on 1M complex64 samples with a 257-tap filter took ~46 ms vs. lfilter's >150 ms for just the real half alone. Confirmed live this time: a follow-up py-spy record after deploying shows numpy.convolve gone entirely from the profile (the only remaining FFT-related cost is legitimate oaconvolve work at ~8.4% combined), and sdr_hardware_service's live CPU dropped from ~60% to ~36%. Also fixes a latent double-counting bug present in the original pre-2.228.13 code (predates both attempts): it carried filter state (zi) across calls but also re-fed leftover unfiltered samples through that same state on the next call, filtering the boundary samples twice; the overlap-add tail-carry has no such issue since every input sample is consumed and filtered exactly once. New tests in tests/test_early_decimation.py: test_oaconvolve_hot_path_matches_single_complex_lfilter_call (numerical equivalence against the filter's mathematical ground truth) and test_chunked_hot_path_matches_single_continuous_call (proves the tail-carry/phase state stitches irregular real-world USB-read chunk boundaries seamlessly).

Process note for next time: verified live on /opt (file copied directly, service restarted, re-profiled) before opening this PR, rather than merging on passing unit tests alone and finding out after -- which is exactly what caught 2.228.13's wrong diagnosis in the first place.

[2.228.13] - 2026-09-09 - Fix scipy.signal.lfilter silently falling off its fast path on complex IQ

Correction (2.228.14): the fix below did not work. Live re-profiling after deployment showed numpy.convolve still dominant and CPU higher than before. The real/imag split assumption was wrong -- see 2.228.14 for the actual root cause and fix. Left here for the historical record of what shipped and why the diagnosis seemed right at the time.

While profiling sdr_hardware_service.py directly (ps/top showed it at ~60% CPU, higher than the demod worker itself) to look for more CPU wins after the RTL-SDR sample-rate investigation, a 15-second py-spy record found the single largest cost in the entire process: 38.7% of all samples (496/1283) were inside numpy.convolve, called via numpy.apply_along_axis from inside scipy.signal.lfilter.

Fixed

• _capture_loop's early-decimation anti-alias filter (app_core/radio/drivers.py) called scipy.signal.lfilter(self._early_decim_aa_filter, 1.0, to_decimate, zi=...) directly on the complex64 IQ stream. scipy.signal.lfilter's fast C implementation (sigtools' direct IIR/FIR routine) only handles real dtypes -- complex input silently falls back to a generic numpy.apply_along_axis(...) -> numpy.convolve path, an O(N·taps) direct-form convolution instead of the optimized routine, for every single USB read on every high-rate SDR receiver (Airspy, or any RTL-SDR run above 500 kHz). RBDSWorker had already discovered and worked around this exact scipy behavior (_apply_interference_notch and its 2.4 kHz post-mix lowpass both filter real and imaginary parts separately) but the fix was never applied to this call site. Now splits to_decimate into .real/.imag, filters each independently (two real-valued lfilter calls, each hitting the fast path) with separate zi delay-line state, and recombines as real_out + 1j*imag_out -- exact, not an approximation, since the filter coefficients are real and the two components are independent linear systems. New test_real_imag_split_matches_single_complex_lfilter_call in tests/test_early_decimation.py proves numerical equivalence (rtol=1e-5) against the old single-complex-call path on a multi-tone test signal; the existing test_alias_image_is_rejected and test_rbds_passband_is_flat tests (which exercise actual filter behavior, not just call shape) continue to pass unchanged.

Not yet measured live: the actual CPU delta on sdr_hardware_service.py after this deploys. Given the profile showed this call at 38.7% of total process time, a substantial drop is expected -- will confirm with a live py-spy record comparison after deployment rather than assume.

[2.228.12] - 2026-09-09 - Stop narrowing the analog IF filter when a WFM receiver's sample rate is set low for CPU

Follow-up investigation to 2.228.10/2.228.11's CPU work: since wbks (the live, actively-receiving wbks RTL-SDR) already decimates from its 1.024 MHz configured rate down to 256 kHz in software before the FM demod pipeline ever sees the signal (app_core/radio/decimation.py's EARLY_DECIM_TARGET_RATE), the obvious next question was: why not just configure the RTL-SDR to natively capture at ~250 kHz and skip the early-decimation FIR entirely? RTL-SDR hardware supports this directly (its low band is 225,001-300,000 Hz) and 250 kHz already clears every downstream FM-stereo/RBDS threshold in fm.py.

Tested live on wbks: it made things measurably worse. Over matched clean windows, the RBDS sync-loss rate roughly tripled (0.022/s at 1.024 MHz vs. 0.083/s at 250 kHz).

Fixed

• Root cause: _SoapySDRReceiver's device-open path (app_core/radio/drivers.py) called device.setBandwidth(SoapySDR.SOAPY_SDR_RX, channel, self.config.sample_rate) -- tying the tuner's analog IF filter bandwidth directly to the configured digital sample rate. At 1.024 MHz the analog filter stays wide open (far more than the ~120 kHz the FM multiplex needs), so all real filtering happens cleanly in software afterward. At 250 kHz that same line also narrows the analog RF filter down to ~250 kHz -- right in the neighborhood of the multiplex itself -- attenuating the pilot/L-R/RBDS subcarriers before they're even digitized, which no amount of downstream software decimation can recover. Added _SoapySDRReceiver.WFM_MULTIPLEX_MIN_BANDWIDTH_HZ = 300_000 (Carson's rule: full-deviation broadcast FM's occupied bandwidth is ~264 kHz) and floor the analog bandwidth request at that value whenever modulation_type is FM/WFM and stereo or RBDS is enabled -- narrowband receivers (e.g. NOAA NFM, no stereo/RBDS) are left untouched, since widening their analog filter would only admit more adjacent-channel noise for no benefit. New tests in tests/test_radio_drivers.py: test_wfm_stereo_floors_analog_bandwidth_below_multiplex_minimum, test_narrowband_sample_rate_above_floor_is_unaffected, test_non_wfm_low_sample_rate_bandwidth_not_floored.

Not yet re-tested: whether this fix actually closes the gap and makes 250 kHz native capture on RTL-SDR safe for real CPU savings. wbks is currently reverted to 1.024 MHz (confirmed stable) pending a fresh live A/B test of this fix before recommending the lower rate to anyone, in the UI or otherwise.

[2.228.11] - 2026-09-09 - Total-audit pass: two more redundant-computation fixes in the FM stereo path

Follow-up to 2.228.10 after being asked not to stop at a single finding: a full read-through of every file in app_core/radio/demod/ (fm.py, dsp.py, rbds_worker.py, rbds_decoder.py, kernels.py, am.py) looking for the same class of bug -- work computed once, then computed again for no reason. Two more confirmed in fm.py; the RBDS worker's apparent real/imag lfilter "duplication" turned out to be a different, already-documented pattern (a single complex lfilter call and two real-valued calls do the same total floating-point work either way -- not a finding) and its one diagnostic pilot_rms computation is already throttled to 1-in-100 calls.

Fixed

• FMDemodulator._decode_stereo() (app_core/radio/demod/fm.py) recomputed pilot_rms = np.sqrt(np.mean(pilot_filtered ** 2)) even after the previous fix started passing in pilot_filtered itself -- demodulate() already computes that identical mean+sqrt reduction over the same array to derive stereo_pilot_strength, before ever calling _decode_stereo. Now accepts pilot_rms as an optional parameter, passed through from demodulate(); only recomputed when omitted. New test_precomputed_pilot_rms_is_numerically_equivalent in tests/test_fm_stereo_decoder.py proves bit-for-bit equivalence.
• demodulate()'s stereo call site built stereo_sample_indices = np.arange(len(multiplex), dtype=np.float64) -- a full chunk-length float64 array, tens of MB/sec of allocation and fill at typical SDR rates -- on every stereo-locked chunk, purely to satisfy _decode_stereo's sample_indices parameter, which the method body has never read (its own docstring already said "unused; kept for backwards-compat", but nothing had removed the allocation at the call site). sample_indices is now optional (default None) and the call site no longer builds it. New test_decode_stereo_no_longer_requires_sample_indices confirms the method works without it.

Audited and found clean (no action needed): dsp.py's filter-design helpers and fast_decimate (each called once per purpose, no duplication); rbds_worker.py's two real/imag lfilter pairs (54-60 kHz bandpass and 2.4 kHz post-mix lowpass, both already running at the decimated rate); am.py (too small to have redundant work); kernels.py (JIT kernels, no wrapper-level duplication).

Not fixed here, same as noted in 2.228.10 and still deliberately out of scope for a "redundant computation" pass: _decode_stereo's remaining two oaconvolve calls and the polyphase resampler's einsum step still run at the SDR's full raw IQ rate rather than a decimated one. That is real further CPU to reclaim but is an algorithmic rate change, not a duplicate-work bug -- it needs its own dedicated, separately-validated pass.

[2.228.10] - 2026-09-09 - Stop recomputing the FM stereo pilot filter twice per chunk

Continuing the wbks CPU investigation (2.228.8/2.228.9): py-spy profiling found _decode_stereo's three oaconvolve FFT convolutions were the single largest category of the demod worker's on-CPU time once idle waits were excluded from the accounting.

Fixed

• FMDemodulator.demodulate() (app_core/radio/demod/fm.py) already computes the 19 kHz-bandpass-filtered multiplex once, to decide stereo_pilot_locked, before it ever calls _decode_stereo -- which then recomputed the identical oaconvolve(multiplex, self._pilot_filter, mode="same") call internally, same input, same filter, same result, a full third of its own FFT-convolution cost for zero benefit. _decode_stereo now accepts the already-computed value as an optional parameter and only recomputes it when a caller doesn't supply one (every direct-call test in tests/test_fm_stereo_decoder.py still exercises the original internal-compute path unchanged). New test_precomputed_pilot_filtered_is_numerically_equivalent proves the two paths produce bit-for-bit identical output. Full audio/demod/RBDS/FM/stereo/SDR test surface (467 tests) run clean.

Not fixed here: _decode_stereo's remaining two oaconvolve calls (LPR lowpass, DSB lowpass for L-R) and the polyphase resampler's einsum gather step still run at the SDR's full raw IQ rate. Real further reduction needs decimating before this stage, which the RBDS worker's own early-decimation path already validates is spectrally safe (RBDS needs up to 57 kHz, higher than stereo's 38 kHz) -- but wiring the stereo/pilot path through the same decimated signal is a larger, separate change deserving its own dedicated pass, not bolted onto this one.

Investigated why eas-station-displays.service was consuming 18%+ CPU at idle. Its log was spamming two distinct errors on every render cycle, continuously.

Fixed

• /api/gpio/status — the vfd_gpio_status default screen's data source, per the route's own docstring ("...with summary data for OLED") — was gated behind @require_permission('gpio.view') with no local-network exemption, so scripts.screen_renderer.ScreenRenderer's unauthenticated localhost requests 401'd on every single render cycle; confirmed live in the service's own logs. Added require_permission_or_local_network() (app_core/auth/roles.py, mirrors the existing require_permission_or_setup_mode pattern) so an anonymous local-network caller is let through -- the same case app.py's LOCAL_API_GET_PATHS already exempts from login app-wide -- while a signed-in session without gpio.view is still denied. /api/gpio/status registered in LOCAL_API_GET_PATHS to match.
• scripts/screen_renderer.py's evaluate_condition(): when the live value fails to parse as a number (exactly what happened above -- the 401 fed a non-numeric placeholder into a numeric condition), the code fell back to the original string for the live value but left the condition's configured expected value as whatever raw type its JSON stored, typically a bare int ({"value": 0}) -- mixing str and int on a >/</>=/<= comparison and raising TypeError, caught by the outer handler and logged on every cycle. Both sides now fall back to strings together.

New tests in tests/test_require_permission_or_local_network.py and tests/test_screen_renderer.py.

Investigated a reported missed Required Monthly Test on the wbks SDR receiver. eas-station-audio.service's "Icecast buffer running low for mount /sdr-wbks.mp3" warning -- the exact symptom commit 26b4eb9f (2026-08-26, "Split demod into its own process") was written to eliminate -- turned out to still be firing roughly every 30 seconds, continuously, unbroken from that commit's own era through today; a later Nice priority bump (2.228.1, PR #2579) didn't touch it either. Confirmed live via py-spy record on the actual running eas-station-demod.service process under real contention (system load 4.5-5.6 on 4 cores) rather than reasoning from code: of the worker thread's genuinely on-CPU time (a third of all samples were idle waits in the subscriber/queue threads and correctly excluded), publishing the per-chunk demod:status:<id> Redis key -- pickle.dumps() + base64 + SETEX, once per ~32ms IQ chunk (~31/s) -- was the single largest individual line, ahead of the actual FM stereo/RBDS DSP math.

Fixed

• RedisSDRSourceAdapter._get_remote_status() (app_core/audio/redis_sdr_adapter.py) already caches this key for 250ms because, per its own docstring, "_update_metrics() runs far more often than the status meaningfully changes" -- but DemodWorker._publish() (services/demod/worker.py) was still writing a fresh one on every single chunk, roughly 7-8x more often than any reader could ever consume. Added _STATUS_PUBLISH_INTERVAL_S = 0.2s throttle on the status write only; the audio publish() call right next to it -- the actual signal data -- is untouched and still fires every chunk. New tests in tests/test_demod_service.py covering both the throttling and that it resumes after the window elapses.

Not fixed here, flagged for dedicated follow-up rather than a rushed change to the live SAME-decode signal path: _decode_stereo's three oaconvolve FFT-convolution calls (app_core/radio/demod/fm.py), each against a 1025-tap filter (algorithmically the right choice at that length, not a bug), were the single largest category of on-CPU time once grouped -- and they run at the SDR's full raw IQ rate (e.g. 1.024 MHz for wbks) rather than a decimated rate, per an explicit CRITICAL FIX comment documenting a prior filter/sample-rate mismatch bug. Reordering that decimation could recover a large further reduction but touches the correctness of the live EAS decode path directly; it needs its own careful, validated pass, not something bolted onto this session's fix.

Verified live on a real multi-source deployment: GET /api/audio/now-playing (no ?source=) returned an empty (all-null) payload even though two other configured sources were actively streaming real songs with title/artist/ artwork metadata.

Fixed

• _default_public_source() (webapp/routes_now_playing.py) picked the first enabled, Icecast-published source ordered by priority descending -- but that priority field is source_manager.py's EAS/SAME failover preference (a hardware line kept reliable for alert monitoring can legitimately carry no song metadata at all), not a "worth showing the public" signal, and it's a different field from the one the actual SAME decoder (app_core/audio/eas_monitor_v3.py's UnifiedEASMonitorService) uses -- that class ignores priority entirely and watches every enabled source independently. Renamed to _default_public_candidate(); it now walks sources in priority order but returns the first one that actually has title or artist metadata right now, falling back to bare priority order only when nothing has metadata yet (e.g. right after startup), so the common single-station case is unaffected. New tests in tests/test_now_playing_api.py.

Changed

• Dependabot dependency bump (minor release, no CVE). Synced the three tech-stack badges (README.md x2, templates/partials/tech_stack_badges.html) and a stale requirements.txt comment that referenced the old pinned version -- Dependabot only ever touches the pin itself, not the badges or comments describing it.

[2.228.5] - 2026-09-09 - Restore Keep a Changelog category headers on every entry since Aug 27

The /version page's category badges ("Added (N)", "Fixed (N)", ...) come from app_utils/changelog_parser.py, which only recognizes real ### Added/### Fixed/### Changed H3 subheadings -- v2.193.8 (2026-08-26) was the last entry to use them. Every entry from v2.193.9 (2026-08-27) onward switched to flat prose with inline bold labels (- **Fixed**: ... as plain text, not a heading), so the parser found zero categorized sections for all ~90 of them and no badge ever rendered. Nobody course-corrected until now.

Fixed

• Restructured all 89 affected entries (v2.193.9 through v2.228.4) into proper ### Added/### Changed/### Fixed/### Removed/### Deprecated/### Security sections. 67 already had top-level bullets with inline labels (**New**:, **Fixed**:, **Changed**:, ...) that map deterministically to a category, grouped mechanically without rewording; unlabeled continuation bullets inherit the category of the immediately preceding labeled bullet in the same entry, matching how they were actually written (elaboration on the same change). The other 22 entries were pure prose with no bullets at all and were hand-restructured into itemized form. Verified via app_utils.changelog_parser.parse_changelog() directly: 571 entries parse cleanly, and only the pre-2026-08-26 entries this pass deliberately left alone still show zero categorized items.
• Going forward, new entries use ### Added/### Fixed/### Changed from the start instead of drifting back to flat prose.

Two commits landed on main with no VERSION bump and no CHANGELOG entry at all, discovered by scanning history for two different commits that both claim the same version number -- the signature of a bump that got clobbered when a second PR merged before the first one's version had moved. Both fixes have been live since their original commit date; this entry only catches the documentation up, same precedent as 2.227.0's own "(missed at merge time)" bump.

Changed

• Fixed (originally 2026-09-03, commit 287251ec): the live weather-alert video export route (/api/alerts/<id>/export-image.mp4) failed every time with "child watchers are only available on the default loop". ffmpeg's subprocess.run() was running inside _run_off_worker's gevent.get_hub().threadpool.spawn(), but gevent's cooperative subprocess handling needs a child watcher registered on the default event loop, which only exists on the process's original hub -- not the separate per-thread hub a threadpool worker gets. Split video_export.py into render_alert_video_frames() (CPU/network-bound Pillow work, safe on the threadpool) and encode_frames_to_mp4() (the ffmpeg subprocess call, which must run back on the request's own greenlet so gevent's cooperative subprocess.run() actually works); routes_alert_export_video.py now calls them separately instead of one function wrapped entirely in _run_off_worker.
• Fixed (originally 2026-08-26, commit 7aaa9b1e, #2471): the global broadcast overlay could get stuck open. Its local countdown reaching 0:00 never closed it -- the modal always waited for a broadcast_state_update push/poll to report active:false, and a backgrounded mobile tab can stall the WebSocket and miss that update entirely, leaving a live-looking abort button showing even after the broadcast had genuinely finished server-side. The overlay now closes when /api/broadcast/abort returns a 409 ("No broadcast is currently active"), and reconciles with the server on visibilitychange so a stale overlay self-corrects on tab focus.

[2.228.3] - 2026-09-09 - Bump lxml to 6.1.3

Changed

• Dependabot dependency bump (patch release, no CVE). Synced the three tech-stack badges (README.md x2, templates/partials/tech_stack_badges.html) that tests/test_tech_stack_badges.py checks against requirements.txt -- Dependabot only ever touches the pin, not the badges, so every dependency bump needs this same manual sync or the badge-drift test fails CI.

[2.228.2] - 2026-09-07 - Deprioritize the security-perimeter-ingest timer

Fixed

• Fixed: security-perimeter-ingest.service (new in 2.227.0, run every 2 minutes by security-perimeter-ingest.timer) boots the full Flask app via create_app() -- all ~260 routes, every subsystem -- just to tail the nginx log and insert a handful of rows. Measured at ~6s of near-single-core CPU per run on the bare-metal box, forever, every 2 minutes. That's the same create_app()-for-a-CLI-script pattern scripts/create_example_screens.py and scripts/fix_admin_roles.py use, which is harmless for an occasional by-hand admin task but becomes a recurring burst when applied to an automated timer -- one that competes with the CPU-contention-sensitive real-time SDR/demod/SAME-decode path (see 2.228.1's Nice=-3 fix below).
• Added Nice=10 and IOSchedulingClass=idle to security-perimeter-ingest.service so its periodic bursts always yield to the real-time services instead of contending with them. The proper fix -- a lightweight DB-only bootstrap instead of the full route-registering app factory -- is bigger scope; tracked for follow-up.

[2.228.1] - 2026-09-07 - Reduce dropped SDR audio chunks under CPU contention

Fixed

• Fixed: the wbks SDR receiver's SAME/EAS header decoder had produced zero alerts for two weeks (last success 2026-08-24) despite the receiver itself streaming samples normally and RDS still decoding -- while the two network-stream sources (ERN-LUC, WNCI) kept decoding alerts throughout, unaffected. Root cause: services.demod's own exit-stats log showed real dropped audio chunks (dropped=513) on a box running at a sustained load average of 3.5-4.0 on 4 cores; a chunk dropped during the ~1s SAME tone burst fails that header even though average throughput looks healthy. ERN-LUC/WNCI don't share this failure mode since they receive already-decoded PCM over the network instead of running the CPU-heavy SDR front end (filter/decimate 1.024 Msps IQ down to audio).
• app_core/radio/demod/rbds_worker.py: four RDS trace log lines were left at INFO instead of DEBUG -- one of them explicitly commented "for diagnostics only" -- producing ~845 log lines/minute (98% of the demod service's total log volume) for zero operational value. Downgraded to DEBUG. This alone did not measurably reduce CPU or the drop rate; kept as a legitimate cleanup, not the fix.
• eas-station-demod.service and eas-station-audio.service now run at Nice=-3 (systemd unit change), giving the real-time IQ-to-audio and SAME-decode path scheduling priority over less time-critical services when the box is under contention -- matching the existing Nice=-5 on eas-station-sdr.service, but one step lower since that service alone is servicing USB reads directly. Post-change, sdr-wbks's recurring Icecast buffer-underrun warnings (previously roughly one every 30s, continuously) dropped to zero in the most recent observation window.

[2.228.0] - 2026-09-09 - Public now-playing API (album art over Icecast, without Icecast)

Added

• New: GET /api/audio/now-playing -- a public, unauthenticated JSON endpoint (webapp/routes_now_playing.py) returning {source, stream_name, icecast_url, title, artist, album, artwork_url, length} for the station's public Icecast stream(s). Icecast/Shoutcast's in-stream metadata (ICY StreamTitle) is text-only -- there's no field for an image, so album art can never travel inside the audio stream to an external player (VLC, a phone app, a car radio, an embedded widget on another site). This is the standard workaround every real internet radio station uses: a small public "now playing" endpoint the player/widget polls alongside the raw audio. Optional ?source=<name> selects a stream in a multi-source deployment; omitted, it uses the first enabled source with a public Icecast mount.
• Deliberately a redacted view -- only display-safe fields. None of the machine-describing data the internal (session/local-network-gated) /api/audio/sources carries -- mount/server/port, bitrate, device params, priority -- is exposed, matching the existing public/local/private API tiers documented in app.py's PUBLIC_API_GET_PATHS/LOCAL_API_GET_PATHS.
• Refactored the ICY metadata field-extraction logic (title/artist/album/artwork_url/length parsing, XML/JSON attribute stripping, URL-decoding) out of IcecastStreamer._extract_metadata_fields into a standalone app_core/audio/now_playing_metadata.py so both the audio-service process (which still pushes StreamTitle updates to Icecast itself) and the webapp process (this new endpoint) share one implementation instead of two independently-drifting copies. IcecastStreamer._extract_metadata_fields is now a thin backward-compatible wrapper.
• Surfaced in the UI: /audio_monitoring now shows each public source's "Now Playing API" URL alongside its existing Icecast stream URL, so an operator can find and hand out the endpoint without reading code.
• New tests in tests/test_now_playing_api.py (default/named source selection, 404s for private/unknown/disabled sources, DB fallback when Redis has no fresh snapshot, the public-path registration guard) plus a regression check that the refactor didn't change IcecastStreamer's own extraction behavior (tests/test_icecast_metadata_url_decoding.py, unchanged, still passing).

[2.227.1] - 2026-09-09 - Fix narrow share-card info column (clipped EXPIRES time, missing content)

Fixed

• Fixed: the landscape share-card's narrow info column (app_utils/image_export/render.py, info panel < INFO_NARROW_MAX_W) only drew severe-thunderstorm-specific panels -- damage-tier callout, tornado tag, wind/hail stat boxes, storm motion. For any non-severe-weather CAP event (911/telephone outage notices, civil emergency messages, advisories with no convective threat data) every one of those was a no-op, so the card showed nothing but a bare EXPIRES time with a large empty column below it. The column now falls back to the same generic HEADLINE/DESCRIPTION text the wide-column layout always shows whenever none of the weather-specific panels rendered anything.
• Fixed: _draw_expires_block (app_utils/image_export/panels_broadcast.py) drew the absolute EXPIRES timestamp in a fixed 30px font with no width check against the column -- a stamp like "Sep 9 · 8:48 AM EDT" (342px) didn't fit the 284px-wide narrow column, and since that column sits only 8px from the canvas's right edge, the overflow ran past the image boundary and was hard-clipped (visible as "...8:48 AM E"). Now shrinks the value font to fit before drawing, matching the shrink-to-fit pattern already used for the header's event-name title.

[2.227.0] - 2026-09-04 - Perimeter defense: flood control, bad-actor blocklist, http:BL, Edge Defense analytics

Added

• New: nginx-level rate limiting (/api/ 20r/s, /login 5r/min, both limit_req) and a reject-before-the-app rule for WordPress/.env/.git/PHP-shell scanner paths -- this app is pure Python/Flask, so none of those paths are ever legitimate, and they previously fell through to a full 29KB /login page render on every scan hit.
• New: an updatable known-bad-actor IP blocklist sourced from Spamhaus DROP/EDROP (scripts/update_bad_actors.sh, refreshed daily via bad-actors-update.timer), merged with a hand-curated local list (config/bad-actors-local.conf), enforced by nginx before any proxy logic runs. Admin UI controls added on Application Settings ("Bad Actor Blocklist" panel, webapp/admin/bad_actors.py): enable/disable toggle, allowlist a false positive, trigger an immediate refresh -- previously only editable by hand over SSH.
• New: opt-in Project Honeypot http:BL reputation check on login attempts (app_core/auth/httpbl.py), auto-banning IPs flagged as harvesters/comment-spammers through the existing IPFilter blocklist (new IPFilterSource.HTTPBL). Configured via Application Settings (enabled flag + access key, both DB-backed rather than only .env); the key is write-only in the UI/API and never round-tripped in plaintext once saved.
• New: Security Center "Edge Defense" tab -- visibility into everything the protections above block before it ever reaches the app (none of it showed up in the Traffic tab, which only sees requests Flask actually handled). 24h counts by reason, top blocked IPs/paths, recent events, current blocklist size/state. Fed by a 2-minute systemd timer (security-perimeter-ingest.timer) tailing the nginx access log (rotation-safe checkpoint by inode+offset) into a new security_perimeter_events table; required adding the eas-station service user to the adm group (it couldn't read the nginx log at all before).
• Repeatable across deployments: install.sh and update.sh both seed the new nginx control files, enable the two new timers, and (update.sh) re-apply the nginx config diff/SSL-preservation and grant the new group membership on an existing installation, not just a fresh one.

Fixed

• Fixed: dashboard_status.js, health.js, and system_health.html kept retrying /api/eas-monitor/status, /api/system_status, and /api/system_health forever on 401, even though those endpoints are intentionally restricted to local-network/authenticated callers (app.py's LOCAL_API_GET_PATHS) -- every anonymous visitor's tab polled them indefinitely with no backoff. Each now stops retrying its gated endpoint after the first 401.

[2.226.0] - 2026-09-04 - API Dashboard

Added

• New: Reports -> Analytics -> API Dashboard (/api-dashboard) shows live request volume, latency (p50/p95/p99) and error rates for every /api/* route, broken out per route -- the usage companion to the existing static API Reference page, which documents routes but not how they're actually used. The Traffic Analytics dashboard only ever showed a single rolled-up "API hits" count; this is where that traffic gets broken out.
• Needed no new request-timing instrumentation: every request already flows through app.py's existing before_request/after_request hooks into WebRequestLog (async, buffered -- never a synchronous DB write on the request path). The one gap was that only the raw path was recorded, which would fragment a parameterized route like /api/alerts/<id> into one bucket per ID ever requested; WebRequestLog gained a nullable endpoint column (Flask's dotted view-function name) captured alongside it, in the same namespace compute_api_reference() already keys routes by, so usage data joins directly against each route's docstring/auth metadata.
• New app_core/analytics/api_stats.py (per-route counts, error rates, latency percentiles -- computed in Python rather than a database-side percentile_cont, since the same code needs to run on PostgreSQL in production and SQLite in tests) and webapp/routes_api_dashboard.py. Latency percentiles use nearest-rank over sorted per-route response times.
• New tests in tests/test_api_stats.py.

[2.225.3] - 2026-09-04 - Push CARTO Dark Matter map detail further

Changed

• Changed: the previous fix (2.225.1) made CARTO roads/labels survive the resize pipeline, but only just -- follow-up review against a live render wanted more headroom. TONE_PRESET_DARK_NATIVE's brightness lift raised from 2.1 to 3.0 and contrast from 1.25 to 1.4, confirmed against the same live alert render: place labels (city names, township names) and road structure are now clearly legible throughout the map inset, not just in isolated spots, while the map still reads as a dark-mode basemap rather than washing toward OSM's brightness.

[2.225.2] - 2026-09-04 - Fix Stream audio sources giving up permanently on HTTP 404

Fixed

• Fixed: a "Stream"-type audio source (app_core/audio/sources.py's StreamSourceAdapter) treated an HTTP 404 from its URL identically to 401/403 -- a permanent, unrecoverable error that stops the restart loop for good until someone manually restarts the source. That's wrong for the common case of a Stream source relaying another Icecast source client (e.g. SDRTrunk pushing to its own mount on this server's Icecast): whichever side isn't running yet when the other one starts gets a 404 and, previously, gave up forever even after both sides came up. 404 now keeps retrying on the normal backoff, same as any other transient failure; only 401/403 (genuinely bad credentials) still stop the loop.
• New tests in tests/test_stream_auth.py: test_stderr_pump_marks_404_as_error_but_not_fatal and test_restart_retries_on_404_instead_of_stopping.

[2.225.1] - 2026-09-04 - Fix washed-out CARTO Dark Matter map detail

Fixed

• Fixed: the CARTO Dark Matter basemap (Settings -> Map Tiles) rendered with no visible roads, place labels, or landcover -- just solid black under the radar/county overlays. Root cause: CARTO's own linework sits only ~50-65 (out of 255) above its near-black background, and that low-contrast signal didn't survive the card's Lanczos tile-resize plus radar-overlay compositing, unlike OSM's much higher-contrast tiles. TONE_PRESET_DARK_NATIVE (app_utils/image_export/map_style.py) previously left brightness/contrast at identity on the theory that a dark-native source needs no darkening; it now applies a brightness lift (1.0 -> 2.1) and a mild contrast lift (1.0 -> 1.25) so the linework survives downstream resizing, confirmed against a real fetched CARTO tile and a full share-card render of a live alert.
• New regression test test_tone_preset_dark_native_road_survives_the_map_inset_downscale in tests/test_image_export_map_style.py, built from the real ~50-65 contrast measured off a live CARTO tile; the existing preset test was renamed and re-asserted to expect a brightness lift instead of "stays close to source," since identity color ops turned out to be the actual bug.

[2.225.0] - 2026-09-04 - Broadcast-style landscape share card

Changed

• Changed: the landscape (1200×630) alert share card is now a map-dominant broadcast-style graphic, modeled on RyanHallYall/WeatherWise-style warning cards -- the radar map now fills ~75% of the canvas (up from ~50%), with a narrow callout column carrying a bold "DESTRUCTIVE DAMAGE EXPECTED" / "CONSIDERABLE DAMAGE THREAT" box (for the two elevated NWS Impact-Based-Warning tiers), a TORNADO POSSIBLE pill, a hero-sized EXPIRES time, stacked WIND GUST / HAIL SIZE stat tiles, a one-line storm-motion readout, and the safety-instruction block (now titled "WHAT TO DO"). Square/portrait/story cards are unchanged for now.
• This is a restyle, not new data: hail size, wind gust, tornado detection, and storm motion were already parsed (webapp/admin/api/display_data.py) and already rendered as gauge-style threat cards -- the new narrow column presents the same data as bold callouts/stat-boxes instead, since the wider gauge-card layout doesn't fit the narrower column. render.py switches between the two treatments based on the info panel's actual width (layout.INFO_NARROW_MAX_W), so a future wide-info layout keeps working unmodified.
• New app_utils/image_export/panels_broadcast.py (the narrow-column drawers) and a new _draw_stat_box primitive in drawing.py; app_utils/image_export/layout.py's landscape preset resized accordingly.
• New tests in tests/test_image_export_broadcast_panels.py.

[2.224.0] - 2026-09-04 - Optional CARTO Dark Matter basemap for the share card

Added

• New: Settings -> Map Tiles (/admin/map-tiles) lets an operator switch the alert share-card map inset from plain OpenStreetMap raster tiles (the zero-config default) to CARTO's Dark Matter style. OSM tiles are darkened/desaturated in post (tone_basemap()) since they're authored light and label-dense; CARTO's Dark Matter tiles are dark and minimal from the start, so the result reads noticeably cleaner. Requires a free CARTO API key (carto.com/basemaps/apikey, 5M tile requests/month, no approval queue) -- with no key configured, or if a tile fetch ever fails, rendering falls back to OpenStreetMap automatically, so an unconfigured or bad key can never break the map.
• New MapTileSettings model (carto_api_key encrypted at rest like every other stored credential in this project) and app_core/map_tile_settings.py accessor, following the same dual-path (Flask context / raw db_session) pattern this session's earlier fix (app_core/crypto.py's SECRET_KEY environment-variable fallback) made safe for the standalone CAP poller to use too.
• app_utils/image_export/tiles.py: tile cache keys (both the in-memory LRU and the on-disk cache) now carry the provider, so switching providers can never serve a tile that was cached under the other one for the same coordinate. map_style.py gained a TONE_PRESET_DARK_NATIVE preset (near-identity color ops) for pre-darkened sources, alongside the existing OSM-tuned defaults. The card's tile attribution is now computed once and reused for both the drawn text and the county-label keep-out box, instead of a hand-maintained pixel-width literal.
• New tests: tests/test_map_tile_settings.py (including a regression test reproducing the exact "no Flask app context" bug shape from the earlier app_core/crypto.py fix), plus new coverage in tests/test_image_export_themes.py and tests/test_image_export_map_style.py.

[2.222.0] - 2026-09-03 - Replace animated GIF alert export with MP4 video

Changed

• Changed: the animated share-card export for weather alerts (/api/alerts/<id>/export-image.mp4, reachable from the alert detail page's Export Social Image menu) is now an MP4 (H.264/yuv420p) instead of a GIF. Facebook and most other social platforms transcode an uploaded GIF into a silent looping MP4 on ingest anyway, so encoding straight to MP4 skips that lossy round-trip and sidesteps GIF's 256-colour palette entirely, which was producing visible banding/dithering on real radar reflectivity and multi-megabyte files for a ~10-frame loop.
• Same behavior otherwise: plays the radar from ~15 minutes before the alert was issued, then reveals the warning polygon only on the frame matching the real issuance time.
• app_utils/image_export/gif_export.py replaced by video_export.py -- reuses the same per-frame generate_alert_image() composition, then pipes the rendered PNG frames to ffmpeg (already a system dependency of this project) instead of Pillow's GIF encoder. webapp/admin/api/routes_alert_export_gif.py replaced by routes_alert_export_video.py.
• New tests in tests/test_image_export_video.py exercise the real ffmpeg binary (already a CI dependency for the audio-source tests) rather than mocking the encode step, including a regression test for scaled card sizes rounding to an odd pixel width/height, which yuv420p cannot encode without an explicit even-dimension filter.

[2.221.1] - 2026-09-03 - Fix encrypted settings unreadable outside a Flask app context

Fixed

• Bug fix: every EncryptedString-backed credential (TTS's Azure OpenAI key, Icecast source/admin passwords, SMTP password, Twilio auth token, SNMP community string, Tailscale auth key, Tickstem API key, admin MFA secrets) was silently unreadable from any process without an active Flask app context -- including the standalone CAP poller, which reads settings through its own sessionmaker() session with no Flask app ever pushed. Decrypting the column raised RuntimeError: Working outside of application context deep inside SQLAlchemy's row hydration (current_app.secret_key), which the poller's own error handling swallowed -- so a fully configured, enabled TTS provider was treated as unconfigured, and every forwarded alert went out tone-only with no spoken narration. The same silent failure applied to any other credential read the same way outside a request.
• Root-caused from a real production incident: alert #1057 (a Severe Thunderstorm Watch) was auto-forwarded with no voice narration despite Azure OpenAI TTS being enabled and fully configured in Settings -> TTS.
• app_core/crypto.py: _root_secret()/_fernet() now fall back to the SECRET_KEY environment variable when there's no Flask app context, rather than exclusively depending on current_app.secret_key. This derives the identical key those processes would get if a Flask app were pushed -- systemd's EnvironmentFile=/opt/eas-station/.env already puts SECRET_KEY in every service's environment, poller included.
• New regression tests in tests/test_secret_encryption.py reproduce the actual bug shape (a real EncryptedString column read via a raw sessionmaker() session with zero Flask app context, not a mock) -- the existing test_airchain_fringe_cases.py coverage for this code path used a MagicMock() session that never touched real column decryption, which is why it didn't catch this.

[2.221.0] - 2026-09-03 - Animated GIF export for weather alerts

Changed

• New: animated GIF share card for weather (category='Met') alerts, alongside the existing static PNG/WebP export -- /api/alerts/<id>/export-image.gif (ratio query param, same four aspect ratios as the PNG export). Reachable from the alert detail page's Export Social Image menu.
• The animation plays the radar in the ~15 minutes before the alert was issued, then reveals the warning polygon for the first time on the frame matching the alert's actual sent timestamp -- never earlier. A GIF can never imply a warning was active before it really was.
• app_utils/image_export/radar_loop.py: added RADAR_LOOP_LEADIN_MINUTES (15) and a show_polygon/issued flag threaded through build_radar_loop() and maps.py's _render_map(). The existing interactive Radar Loop viewer on the alert detail page picks up the same lead-in + polygon-reveal behavior automatically, since it's backed by the same function. radar_loop_hires.py's Level II loop is unaffected -- the lead-in window is opt-in per caller (_needed_timestamps(..., leadin_minutes=...)), not baked into the shared timestamp helper.
• New app_utils/image_export/gif_export.py: reuses generate_alert_image()'s full card composition once per radar-loop frame (only the map inset and polygon visibility change between frames), quantized against one shared colour palette so the static header/panels/footer don't flicker between frames.
• New tests: tests/test_image_export_gif.py, plus updates to tests/test_image_export_radar_loop.py for the lead-in window.

[2.220.0] - 2026-09-03 - Search Settings by field name and stored value

Changed

• New: search box on the Settings hub (/settings) that matches a setting's field label ("Stream Bitrate") and its currently-stored value ("128") -- not just the label of the settings page it lives on, which is all the existing Ctrl+K command palette could do. Deliberately scoped to the Settings page's own content, not a global header search bar.
• app_core/settings_search.py: builds the index by querying each mapped settings model's single row and humanizing its columns into labels (stream_bitrate -> "Stream Bitrate", with acronym fixups for SMTP/GPIO/API/TTS/etc.). Covers the ~14 settings pages backed by a simple single-row model (Icecast, TTS, Notifications, Hardware, Location & Alert Filtering, Poller, Heartbeat, Tickstem, Alert Gating, Tailscale, Certbot, Application Settings, EAS Encoder Settings) -- pages that are actions or record lists rather than field/value forms (Backups, RBAC, User Accounts, Environment Variables, the pgweb link) are intentionally left out.
• Security, verified by test and live: every field is checked against app_core.crypto's encrypted-column list and an isinstance(EncryptedString) check before inclusion, plus a manual blocklist for the one plaintext-but-secret-shaped column found (heartbeat_settings.ping_url, a bearer-token URL). A value from an encrypted column (Icecast/SMTP/SNMP/Tailscale/Tickstem credentials) can never appear in a search result, searchable or not, regardless of query.
• HardwareSettings (one ~60-column model shared by three pages) is routed by column prefix -- gpio_* to GPIO & Relays, zigbee_* to Zigbee, everything else to Hardware Settings -- rather than lumping every field under one page.
• Reuses the exact permission filtering webapp.navigation._flatten_settings_items() already does for the command palette, so a viewer without access to a given settings page never sees that page's fields in search either -- no separate permission logic to keep in sync.
• New tests in tests/test_settings_search.py.

[2.219.1] - 2026-09-03 - Every page now shows the navigation breadcrumb

Changed

• Root cause found and fixed: static/js/core/nav-enhance.js's breadcrumb/command-palette indexer only ever scanned the rendered navbar DOM. The Settings section (webapp/navigation/registry_settings.py, ~35 pages -- Icecast, NTP Server, GPIO, TTS, Backups, etc.) renders in the navbar as a single link (its items only ever appear as cards on the /settings hub page), so every one of those pages has been missing a breadcrumb since the feature existed, no matter how it was organized in the registry. Fixed at the root: webapp/navigation/__init__.py's inject_navigation() now also exposes a flat, already permission-filtered nav_settings_items list; templates/components/navbar.html embeds it as JSON; nav-enhance.js merges it into its index. One fix, all ~35 pages, no per-page registry duplication.
• Audited every page-rendering route in the app for breadcrumb coverage and fixed the remaining real gaps:
  • New registry entries for pages that had never been registered anywhere: Alert-Boundary Intersections, Zone Catalog (Monitor -> Alerts), GPIO Statistics/Interlocks/Pin Map (Monitor -> Station Hardware), SMS Compliance (Help -> About This System).
  • Hand-rolled a breadcrumb (matching the existing pattern in templates/alert_detail.html) on four dynamic per-record pages that can't take a static registry entry: Audio Detail, Received Alert Detail, Manual EAS Activation print view, and the OLED Screen editor (new + edit). Also added one to the global-search results page.
  • /admin and /search (no query) were confirmed as correctly breadcrumb-less: both just redirect (to /settings and / respectively) rather than rendering a page of their own.

[2.219.0] - 2026-09-03 - Consolidated Firewall settings page; fix Icecast auto-streaming after credential encryption

Fixed

• Fixed Icecast auto-streaming silently disabled since credential encryption shipped (2.218's predecessor, #2552): eas_monitoring_service.py builds its own minimal Flask app for the standalone audio/EAS process, and never set SECRET_KEY on it. app_core.crypto derives the encryption key for encrypted-at-rest settings (e.g. IcecastSettings.source_password) from current_app.secret_key, so reading Icecast settings from that process raised SECRET_KEY is not configured on every restart, was caught, and fell back to silently disabled -- with no loud, visible error and no periodic retry. All four Icecast streams had been down for hours before this was caught by a listener reporting they couldn't connect. Extracted the secret-key resolution app.py already had (env var, falling back to a shared persisted key file so every process agrees on the same key) into app_utils/secret_key.py and had eas_monitoring_service.py use it too.
• Fixed a related, pre-existing gap: tests/test_radio_audio_monitoring.py's DummyAdapter test double didn't implement is_quarantined() or set _start_time, so the real IngestController's background health-monitor thread (which the test registers the double into) threw and logged an AttributeError on every monitor cycle for that test's duration -- harmless (caught and isolated by design) but noisy.
• New: Firewall settings page (webapp/admin/firewall.py, /admin/firewall, Reports -> Security -> Firewall) -- one place for every host-firewall (UFW) rule the app manages, replacing a real gap where opening Icecast's port required a manual sudo ufw allow 8000/tcp nobody had documented as a required step, and an inline firewall-rule widget that had grown on the LAN NTP server's own settings page:
  • Host Firewall Baseline -- the existing Security Center UFW check (22/80/443, default-deny), with its one-click fix, now also surfaced here.
  • LAN NTP Server (UDP/123) -- the subnet-management UI moved off /admin/ntp-server (which now just shows status and links here); the underlying /admin/ntp-server/configure endpoint and its behavior are unchanged.
  • Icecast Streaming (TCP, configured port) -- new: list the subnet(s) allowed to reach the Icecast port, or 0.0.0.0/0 for a public stream, with detected-local-subnet suggestions (never applied automatically -- a cloud host's own interface subnet is usually the provider's range, not the operator's LAN).
  • Extracted app_core/network_info.py (detect_local_subnets(), validate_cidr()) out of webapp/admin/ntp_server.py so both features share one implementation instead of two copies drifting apart.
  • config/sudoers-eas-station gained the Icecast-port ufw allow/ufw delete allow entries (tagged eas-station-icecast, mirroring the existing NTP entries) -- run update.sh (or manually redeploy /etc/sudoers.d/eas-station from this file) on any existing installation, or the Icecast card's Apply button fails with a permission error.
  • New tests in tests/test_firewall_admin.py.
• Documented which ports need router port-forwarding for WAN access vs. which must stay closed, in a new "Router Port Forwarding" section of docs/troubleshooting/FIREWALL_REQUIREMENTS.md -- previously undocumented beyond a one-line troubleshooting note.
• Updated docs/compliance/FCC_26-38_EAS_CYBERSECURITY.md's network-segmentation section (47 CFR SS 11.35(d)(3)) to describe the new consolidated Firewall page as the platform's implementation of that control, and added it to templates/help.html / templates/about.html.

[2.218.1] - 2026-09-02 - Respect Tickstem's per-plan heartbeat quota when bulk-creating

Changed

• Confirmed live: Tickstem's free tier caps heartbeats at 5 total, well under the 12 critical services this box has. Bulk-creating all 12 at once burned through the quota, failed on the remaining ones with HTTP 402, and would have repeated the same failed attempts on every subsequent click since a failed create doesn't get remembered as "already tried."
• TickstemAPIError now carries status_code, so create_all_service_heartbeats() can stop the moment a 402 comes back instead of continuing to retry a request Tickstem has already said it won't honor for any of the remaining services.
• The bulk-create route accepts an optional service_names list, scoping the attempt to a specific subset instead of always going for every unmonitored critical service.
• /admin/tickstem now shows a checkbox per unmonitored service (with a "select all" convenience) instead of one blind "create everything" button, so a plan near its quota can choose which services matter most.
• New tests in tests/test_tickstem_service_heartbeats.py covering the service_names subset and the stop-on-402 behavior.

[2.218.0] - 2026-09-02 - Per-service Tickstem heartbeats for critical EAS Station subsystems

Changed

• New tickstem_service_heartbeats table and TickstemServiceHeartbeat model (app_core/_models_tickstem.py): one row per critical service from app_core.config.get_eas_services() (the 11 EAS subsystems plus the poller), each holding its own Tickstem heartbeat.
• Why per-service instead of one combined heartbeat: Tickstem's ping carries no payload, so a missed ping on one aggregate heartbeat can only ever mean "something's wrong" in the resulting alert. A heartbeat per service, each named on Tickstem's side (e.g. "EAS Station -- eas-station-poller.service"), means a missed ping names the exact subsystem that failed.
• app_core/tickstem_client.py gained create_heartbeat(), set_heartbeat_status(), and delete_heartbeat(), mirroring the existing Monitors API functions but against Tickstem's Heartbeats API -- fully outbound, no public URL needed (unlike the existing Monitors integration, which requires one).
• app_core/heartbeat_worker.py's background loop now also drives these: each row is pinged only when it's both due (its own interval_secs) and its matching systemd service is currently active, read from the same cached snapshot get_system_health() and the System Health page share. The loop moved from "sleep for the configured interval" to a fixed 60s tick with each signal checking its own last-ping timestamp, since multiple independently-scheduled heartbeats can no longer share one sleep duration.
• New admin UI on /admin/tickstem: a "Monitor N Remaining Services" button bulk-creates heartbeats (via the already-saved Tickstem API key) for every get_eas_services() entry that doesn't have one yet, plus a per-service table with individual pause/resume/delete.
• New tests in tests/test_tickstem_service_heartbeats.py: the due/active gating logic (the core of why this feature works), and the three new tickstem_client functions.

[2.217.1] - 2026-09-02 - NetBIOS hostname fallback, and fix chronyc's own resolution clashing with it

Fixed

• Added a NetBIOS (NBT-NS Node Status, UDP/137) fallback in _lookup_client_hostname() for when reverse DNS comes up empty -- the common case for a Windows PC on a LAN whose resolver has no PTR records for it, since Windows doesn't register itself in DNS by default. Hand-rolled the query/response (RFC 1002 wildcard-name encoding, minimal Node Status response parser) rather than adding a dependency for a two-message UDP protocol.
• Fixed a regression the hostname feature itself exposed once reverse DNS started working on a given deployment: chronyc clients' own first column does its own reverse-DNS resolution and truncates long names to fit a fixed-width text column, so once PTR records resolve, _client_summary()'s parser -- which expects that column to always be a numeric IP -- started reading truncated hostnames instead. Fixed by adding -n (raw IPs only) to the chronyc clients call, and updated the scoped sudoers entry in config/sudoers-eas-station to match the new exact command (/usr/bin/chronyc -n clients).
• Not a chrony bug or anything wrong with this feature in isolation -- it only ever showed up because this same session's earlier fix made PTR resolution actually work end-to-end for the first time, which is exactly the condition needed to expose it.

[2.217.0] - 2026-09-02 - Show hostnames in the LAN NTP Server's Recent Clients list

Changed

• webapp/admin/ntp_server.py's _client_summary() now attempts a reverse-DNS (PTR) lookup for each client IP via _reverse_dns(), capped at 1 second so a client with no PTR record -- normal for most phones, laptops, and IoT devices on a home LAN -- can't stall the whole list. templates/admin/ntp_server.html adds a "Hostname" column (both the server-rendered initial table and the JS-driven Refresh path), showing — when no record resolves.
• Wrapped the clients table in .table-responsive and added .text-break-anywhere to the IP/hostname cells while touching this template, per the mobile-friendly requirement in docs/development/AGENTS.md -- the new column made overflow at narrow viewports more likely.
• Whether this actually shows anything depends entirely on the deployment's DNS setup: a resolver that doesn't serve PTR records for RFC1918 addresses (e.g. a public DoH/DoT forwarder, which most resolv.confs on this kind of deployment end up pointing at) will show — for every client regardless of how well the feature works, since there's no PTR data to find. A home router that also acts as local DNS for its DHCP leases is the common case where this actually resolves something.

[2.216.3] - 2026-09-02 - Fix the LAN NTP Server's Recent Clients list always showing empty

Fixed

• A second, independent sandboxing bug in the same feature: chronyc clients (used to populate the "Recent Clients" list) connects to chronyd over a UNIX socket at /run/chrony/chronyd.sock, whose containing directory is drwx------, owned by _chrony. eas-station-web.service's CapabilityBoundingSet= caps what the sudo-escalated root process inside its sandbox can do, and it was missing CAP_DAC_OVERRIDE -- so that "root" can't traverse a directory it doesn't own. chronyc silently fell back to the legacy cmdmon protocol and got 501 Not authorised.
• This failed on every single page load, not just before a client's first sync: _client_summary() in webapp/admin/ntp_server.py treated any chronyc clients failure identically to a genuinely empty list ({"available": False, "clients": []}), and logged nothing, so the page always read "No non-local clients have queried this host yet" regardless of real client activity.
• Fix: added CAP_DAC_OVERRIDE to CapabilityBoundingSet= in systemd/eas-station-web.service. Confirmed via the same sandbox-reproduction method as 2.216.2: chronyc clients returns 501 Not authorised inside a transient unit mirroring the service's exact sandbox, and returns the real client list (three hosts, in this case) once the capability is added.
• Also added a logger.warning() in _client_summary()'s failure path so a future occurrence of this class of bug shows up in journalctl -u eas-station-web instead of silently rendering as "no clients yet" -- the UI copy itself is unchanged (a fresh server legitimately has zero clients, and that's not an error), only the previously-silent failure case now leaves a trace.

[2.216.2] - 2026-09-02 - Fix the real cause of the LAN NTP Server's "Read-only file system" write failure

Fixed

• Corrects the diagnosis in 2.216.1: the sudo tee write of /etc/chrony/conf.d/eas-station-ntp-server.conf was never transient. eas-station-web.service runs with ProtectSystem=strict, which bind-mounts the whole filesystem read-only inside that service's own mount namespace except for the paths listed in its ReadWritePaths=. /etc/chrony was never added to that list when the LAN NTP Server feature shipped in 2.216.0, so every write from inside the running service hits Read-only file system -- 100% of the time, not intermittently. The 2.216.1 write-up tested the path from an ordinary root shell, which sits outside the service's mount namespace and so is not sandboxed the same way; that made the write look "directly writable... afterward" when in fact the service itself could never write it. Confirmed by reproducing live: sudo journalctl -u eas-station-web showed the identical tee: ...: Read-only file system failure again at 16:30:01, a second and unrelated Apply click roughly 20 minutes after 2.216.1's commit claimed it had "self-resolved... no repeat since."
• Fix: added /etc/chrony to ReadWritePaths= in systemd/eas-station-web.service, alongside the existing /etc/nginx//etc/letsencrypt//etc/icecast2 entries this same service already needs write access to for other admin features.
• The retry-once logic added in 2.216.1 (_write_chrony_conf()) is left in place -- harmless now that the underlying write actually succeeds, and cheap insurance against a genuinely transient failure in the future -- but it is no longer the fix for this bug.
• Manually recovered the live host's config after reproducing the bug: the conf.d fragment had been briefly overwritten with placeholder content while confirming the failure was reproducible outside the sandbox, then restored to allow 192.168.8.0/24 / local stratum 10 and chrony restarted before this fix landed.

[2.216.1] - 2026-09-02 - Retry the LAN NTP Server's config write once on a transient failure

Changed

• Confirmed live: webapp/admin/ntp_server.py's sudo tee write of the chrony conf.d fragment hit a transient Read-only file system error for about 15 minutes right after this feature's first deploy, then self-resolved on its own with no code change and no repeat since. The exact trigger was never confirmed — it wasn't real filesystem corruption (no matching kernel/dmesg errors, and the target path is directly writable when tested from inside the service's own mount namespace afterward) — so this isn't a root-cause fix, but a short retry costs nothing on the common (successful) case and may ride out a similarly brief blip without the admin needing to notice the error and click Apply again themselves.
• Extracted the write into _write_chrony_conf(), which retries once after a 1-second pause on either a non-zero exit or a raised exception before giving up and surfacing the existing error message.
• New tests in tests/test_ntp_server.py: unit-level coverage of _write_chrony_conf() (succeeds without retrying, recovers after one failure, gives up after exhausting the retry, recovers from a raised exception the same as a bad return code) plus one integration-level test confirming the /configure route as a whole succeeds when the underlying write recovers on its retry.

[2.216.0] - 2026-09-02 - Add a LAN NTP Server admin page

Added

• chrony is installed and running on every deployment (it's the box's own time sync, and on GPS-HAT hardware the stratum-1 source), but by default it only ever acts as a client -- nothing in chrony.conf grants any subnet permission to query it, so a request from a LAN device is silently ignored. Which subnets should be trusted is inherently a per-deployment decision (a home LAN, an office VLAN, a Tailscale range, or nothing at all) with no correct default, so this needed to be admin-configured rather than something install.sh could set up once.
• New Settings -> Network -> NTP Server page (webapp/admin/ntp_server.py, templates/admin/ntp_server.html): list the subnet(s) allowed to query this host, enable/disable, and see recent clients with how long ago each last synced (parsed from chronyc clients). Deliberately stateless like webapp.admin.mail_server -- the chrony conf.d fragment on disk is the single source of truth, read back fresh on every status check rather than mirrored into a DB row that could drift from what's actually applied.
• The firewall side follows the same idempotent, tag-scoped reconciliation webapp.admin.security_checkup's UFW fix established: every rule this feature creates carries a fixed eas-station-ntp-server UFW comment, and only rules carrying that exact comment are ever added or removed through it -- an operator's own rules for other ports/services (Icecast's 8000, pgweb's 8081, etc.) are never inspected or touched. Disabling clears both the chrony config and every tagged firewall rule; nothing lingers.
• New sudoers entries (config/sudoers-eas-station) scoped to exactly what this needs: writing the conf.d fragment, restarting chrony, chronyc clients, and the two ufw ... comment eas-station-ntp-server entries. update.sh already re-syncs sudoers on every run, so this reaches every existing deployment on next update with no manual step.
• New tests/test_ntp_server.py (34 tests): CIDR validation/normalization, conf-file and UFW-rule parsing, the client-list parser (including the never-synced and localhost-exclusion cases), and the full configure route (enable/disable, dedup, validation errors, and that a disable only ever removes this feature's own tagged rules).

[2.215.3] - 2026-09-02 - Fix two more System Health dashboard accuracy bugs

Fixed

• SMART health falsely reported "passed" when smartctl never actually got any data. _collect_smart_health's exit-code fallback (used whenever smartctl's JSON has no smart_status block) only checked bits 3-7 of smartctl's exit code for "disk problem" bits, never bits 0-2 ("command line did not parse" / "device open failed" / "SMART command failed" -- i.e. no real data was ever retrieved at all). Found on a Vultr KVM instance: its virtio-blk-backed /dev/vda has no ATA/NVMe protocol to the underlying disk at all (true of virtio-blk generally, not specific to this app or provider), so every smartctl device-type probe returns exit code 2 with a mostly-empty but validly-parsing JSON report -- which sailed straight through "bits 3-7 clear -> passed" and got shown as a healthy drive despite smartctl never having successfully talked to anything. Now bits 0-2 short-circuit to overall_status: "unknown" with a real error message (smartctl's own JSON messages, or a generic exit-code explanation) -- the dashboard's existing (already-correct) "Unknown" badge + error-alert rendering picks this up with no template changes needed. Added tests/test_smart_health.py (8 tests: the execution-failure paths, plus regression coverage that the bits-3-7 "passed"/"failed" inference and the NVMe critical_warning path are unaffected).
• A failed systemd unit for a service retired from the codebase was completely invisible to the System Services panel. Found eas-station-eas.service (folded into -audio/-demod during the hardware subsystem split) sitting in systemctl --failed as a stale not-found/failed unit, killed by a stop timeout two weeks earlier -- update.sh never disables/removes units for services that get renamed or removed, so a box running since before such a change is left with a permanent stale failure record. app_utils/system/services.py's _collect_systemd_services only ever checked a fixed allowlist (EAS_SERVICES/POLLER_SERVICES in app_core/config/services.py), so a unit that fell off that list was never checked at all, no matter how broken. Added _collect_orphaned_failed_services: asks systemd directly (systemctl list-units --state=failed <prefix>-*) for any failed unit matching the service prefix regardless of whether the allowlist still knows its name, excluding template-instantiated units (e.g. eas-station-failure-recovery@<subsystem>.service) which are legitimate dynamic infrastructure, not retired services. Surfaces as a normal "EAS Station" category entry in the services list plus an actionable issue (includes the systemctl reset-failed command to clear it). Cleared the stale record on the deployment it was found on. Added tests/test_orphaned_services.py (6 tests, including a _collect_systemd_services integration test verifying the orphan flows through to summary/issues).

[2.215.2] - 2026-09-02 - Fix the one-click upgrade's live progress feed showing nothing useful

Fixed

• Admin -> Operations' "System Upgrade" progress panel (get_upgrade_progress in webapp/admin/maintenance/routes_operations.py) read exclusively from journalctl -u eas-station-update.service, but update.sh redirects its own stdout/stderr to /var/log/eas-update.log right after its root check (exec 1>>"$LOG_FILE" 2>&1) -- that redirect replaces the fd 1 the systemd unit handed the script, so none of update.sh's actual output (every echo_step/echo_info/... line, including the === UPDATE RESULT === marker the endpoint looks for) ever reached the journal. All the endpoint could see was sudo/PAM session noise from the commands update.sh runs, plus the unit's own bare start/stop lines -- confirmed against a real captured run, where the journal held nothing usable while the log file had the full step-by-step output including the final result marker.
• Added _tail_update_log() and made it the primary source for get_upgrade_progress; the journal is now only consulted for its unit-lifecycle lines (Failed with result / Deactivated successfully), kept as the fallback for a crash so early update.sh never got to write anything to its own log, per the existing _classify_upgrade_log_line logic. Only the most recent journal lifecycle line is used, so a stale entry from a previous run sitting in the same 500-line window can't override this run's own log-file content.
• Also fixed the /opt/eas-station/scripts/lib/ui.sh: line 866: /dev/tty: No such device or address noise visible in that same captured run's log: three of scripts/lib/ui.sh's TTY-write helpers (whiptail()'s wrapper, ui_gauge_stop(), and the cleanup_on_exit() trap) touched /dev/tty unconditionally instead of checking the existing _UI_HAS_CONTROLLING_TTY flag every other TTY write in the file already checks -- irrelevant interactively, but update.sh's one-click path runs via systemd-run with no controlling terminal at all, where 2>/dev/null on the same line does not suppress the error (confirmed empirically: bash reports a failed redirection to the current stderr before any later redirection on the same command line takes effect, regardless of what that later redirection points to). cleanup_on_exit() runs on every single script exit via its EXIT trap, so this fired on every non-interactive run, successful or not, adding an ugly stray line to what the UI now actually surfaces.
• Updated tests/test_upgrade_progress.py's endpoint tests to mock the new _tail_update_log() primary source instead of treating get_systemd_logs as authoritative, added direct tests for _tail_update_log() (missing file, ordering, max_lines truncation), and added a regression test for the stale-journal-line-vs-fresh-log-file ordering fix.

[2.215.1] - 2026-09-02 - Fix Checkup tab misreporting an active UFW firewall as inactive

Fixed

• The Checkup tab added in 2.215.0 (webapp/admin/security_checkup.py) reported "UFW is installed but not active" on a host where sudo ufw status verbose run interactively worked fine and UFW genuinely was active, default-deny-incoming, with the baseline ports allowed. Root cause: eas-station-web.service's CapabilityBoundingSet grants sudo enough to reach uid 0, but doesn't include CAP_NET_ADMIN, which iptables-nft needs just to read the ruleset via netlink -- so under the actual service (not an interactive shell, which isn't capability-bounded the same way) the command failed with Could not fetch rule set generation id: Permission denied (you must be root), and _run()'s nonzero-exit fallback silently parsed that failure as "inactive."
• Added CAP_NET_ADMIN to systemd/eas-station-web.service's CapabilityBoundingSet. Reproduced the failure and confirmed the fix with systemd-run transient units matching the service's exact capability set, rather than by trial-and-error on a live box -- this bug would otherwise reappear on every deployment using this unit file, not just the one it was found on.

[2.215.0] - 2026-09-02 - Security Center "Checkup" tab: detect and fix a missing host firewall

Fixed

• Found (on a real deployment, by hand) a host running with no firewall at all: install.sh only configures UFW automatically on a fresh install (v2.19.7+), and update.sh never re-runs that one-time provisioning — a deployment first installed before that version, or one where UFW was later removed, stays silently exposed through every subsequent application update. Also found that Fail2banSettings.enabled can be true in the database while the eas-station jail was never actually loaded — the enforcement toggle looked "on" while nothing was actually being mirrored to the host firewall.
• Added a new Checkup tab to Security Center (webapp/admin/security_checkup.py, /admin/security-checkup/status, /admin/security-checkup/fix-ufw): detects whether UFW is installed, active, default-deny-incoming, and has the baseline 22/80/443 rules, and reuses webapp.admin.fail2ban's already-accurate live jail state (it distinguishes the stored "enabled" flag from the real actuator_jail_loaded check) rather than duplicating that logic. A "Fix now" button reproduces install.sh's own baseline UFW setup as an idempotent, web-triggered action — no SSH required — without touching any rule an operator has added beyond that baseline (Icecast, pgweb, etc.).
• New sudoers entries (config/sudoers-eas-station) scoped to exactly the six commands the fix needs, following the same least-privilege pattern as every other privileged action in this file.
• New tests/test_security_checkup.py covers the UFW status parser against captured real output: inactive, properly baselined, active-but-missing-a-port, and the specific dangerous misconfiguration (active with a default-allow-incoming policy) this check exists to catch.

[2.214.0] - 2026-09-02 - Encrypt stored credentials at rest and pepper password hashes

Changed

• Every stored credential in the database was plaintext: Icecast source/admin passwords, the Azure OpenAI TTS key, SMTP password, Twilio auth token, the SNMP community string, the Tailscale pre-auth key, the Tickstem API key, and per-user TOTP (MFA) secrets. Found while investigating a related browser-exposure bug (2.213.2/2.213.3) and confirming passwords are salted (they are, via werkzeug's scrypt) -- these reversible secrets weren't, because hashing doesn't apply to a credential the app has to hand back to a third-party API.
• Added app_core/crypto.py: an EncryptedString SQLAlchemy column type (Fernet, keyed via HKDF-SHA256 derived from the app's SECRET_KEY -- no new required env var) that encrypts on write and decrypts on read transparently, so every existing read/write call site kept working unchanged. Legacy plaintext rows are tolerated (decrypted as-is) and get encrypted automatically on next save; a SECRET_KEY rotation fails closed (empty string, logged) instead of crashing.
• Applied it to all nine columns above. Added migration 20260902_encrypt_stored_secrets widening them from VARCHAR to TEXT, since Fernet ciphertext runs longer than the plaintext it replaces.
• Also found TailscaleSettings.auth_key had the same browser-exposure bug already fixed for the TTS key: pre-filled in plaintext in both the page template and the /api/tailscale/settings JSON response. Fixed the same way (masked in to_dict(), field left blank on page load, blank submission preserves the existing key) with one addition -- an explicit "clear the saved key" checkbox, since blank already had a meaning here (switch to browser-based login) distinct from "no change."
• Added a server-side password pepper (a second HKDF-derived secret, independent of anything in the database) to AdminUser.set_password/check_password and MFA backup-code hashing, so a stolen database dump alone isn't enough to brute-force credentials offline even with correct per-hash salts. Existing pre-pepper hashes/backup codes still verify via a fallback check and upgrade transparently in place on next successful use -- no forced password resets.
• New tests: tests/test_secret_encryption.py (encrypt/decrypt round-trip, legacy-plaintext tolerance, key-rotation failure mode, at-rest verification via a raw SQL read, peppered-hash verification, pre-pepper hash/backup-code upgrade paths).

[2.213.4] - 2026-09-02 - Add the standard page header to the documentation viewer

Added

• doc_viewer.html (the single-document reader behind /docs/<path> and the policy pages) rendered straight into a breadcrumb + card with no page header, unlike its sibling pages docs_index.html, docs/search.html, and docs/rbac_visual.html, which all use the standard components/page_header.html component. Added the same header, using the page's resolved title.

[2.213.3] - 2026-09-02 - Surface real playback errors on the TTS admin test player

Changed

• The Test TTS and Pronunciation Preview <audio> players had no error event handling at all: a genuine browser-side playback failure (bad decode, unsupported source, network error) rendered as the native control's bare "Error" label with zero diagnostic text, while the generated audio itself could be perfectly valid — confirmed by regenerating the same request server-side and validating the WAV with ffprobe/ffmpeg (clean pcm_s16le, 16kHz mono, decodes with no errors).
• Added a shared wireAudioErrorReporting() listener on both ttsAudioElement and previewAudioElement that reads the element's MediaError code and shows a concrete, human-readable message in a visible alert instead of leaving the user with an unexplained "Error" label.
• Also reset the new error panel at the start of each test run, alongside the existing audio-player visibility reset, so a stale error from a previous attempt can't linger.

[2.213.2] - 2026-09-01 - Fix incomplete hostname checks in the CAP poller's endpoint classification

Fixed

• CodeQL flagged poller/cap_poller.py:2044 (elif 'weather.gov' in endpoint.lower()) as "Incomplete URL substring sanitization" on PR #2549: a plain substring check matches a malicious or misconfigured endpoint like https://evil.example/weather.gov or https://weather.gov.evil.com, not just the real NOAA API.
• The same anti-pattern existed at five other call sites classifying self.cap_endpoints entries as NOAA/IPAWS/CUSTOM for logging and source-tagging (poll_and_process, get_poller_status, the startup endpoint log, the poll-summary log, and the zone-code-rebuild filter).
• Added _endpoint_host_matches(url, domain), which parses the URL with urllib.parse.urlparse and compares the actual hostname (exact match or subdomain) instead of doing a substring search, and switched all six call sites to use it.

[2.213.1] - 2026-09-01 - Isolate per-item failures in the CAP poller's processing loops

Changed

• The 2026-08-31 outage (fixed in 2.211.2) was caused by one unhandled data-shape variance in one alert's <references> field crashing the entire poll cycle, not just that alert — because poll_and_process()'s main per-alert loop had only one try around the whole cycle, not one per alert. That specific field bug was fixed, but the structural gap that let it take down every other alert in the batch was not.
• An investigation this session (prompted by a broader stability review) found four loops in poller/cap_poller.py with the same shape — one bad item's exception propagating out of the loop and aborting everything else in that batch/cycle — while confirming every other external-data ingestion point in the codebase (audio capture, GPS parsing, GPIO events, boundary uploads) already isolates per-item failures correctly.
• Wrapped each loop's per-item body in its own try/except that logs the offending item's identifier with a full traceback and moves on to the next item: poll_and_process()'s main per-alert loop (the one that caused the outage), fetch_cap_alerts()'s per-alert dedup/normalize loop, _parse_ipaws_xml_feed()'s per-<alert> XML conversion loop, and _process_cap_references_cancellation()'s per-reference loop (a single Cancel message can reference several prior alerts; one bad reference no longer blocks the others from being cancelled).
• Also added db_session.rollback() to the per-alert catch in the main loop: _process_cap_references_cancellation() mutates ORM objects and commits conditionally with no rollback path of its own, so a failure partway through one alert could otherwise leave dirty, uncommitted session state that cascades into the next alert's processing.
• New regression tests (tests/test_cap_poller_per_item_isolation.py) feed one malformed item alongside valid ones into each of the three functionally-testable loops and assert the valid items are still processed; the fourth (poll_and_process() itself, which needs a full Flask/DB context to exercise functionally) is guarded structurally, asserting the try/except-with-rollback wrapping is present around the exact loop that crashed in production.

[2.213.0] - 2026-09-01 - Reorganize the setup wizard into collapsible sections

Changed

• The setup wizard at /setup rendered all ~35 configuration fields as one flat, unbroken form with no section headers, no progress indication, and no visual grouping — despite the backend already modeling the config as 7 logical sections (WIZARD_SECTIONS in app_utils/setup_wizard.py). A first-time user got a wall of inputs before being told what any of it meant.
• templates/setup_wizard.html now renders those sections as a Bootstrap accordion with real headers, descriptions, and per-section field counts. Location Settings and EAS Broadcast — the two every install needs — are open by default; Core Settings, Audio Ingest, Icecast Streaming, Text-to-Speech, and Hardware Integration start collapsed. A section auto-expands regardless of category if one of its fields fails validation, so a resubmitted error can never end up hidden behind a collapsed header.
• Added a "Before you start" intro panel explaining what EAS Station is and what information to have ready, so the field wall isn't the first thing a new user sees.
• Fixed a real bug found while implementing this: only SECRET_KEY was blanked in the form when it already held a real, install.sh-configured value — the four POSTGRES_* credential fields (explicitly commented "NOT shown in wizard, managed by install.sh" in the code) were still rendered, unrecognized, prompting a user to re-enter database credentials they never chose. webapp/routes_setup.py's new _is_managed_field_present() generalizes the SECRET_KEY-only check to every system-managed field.
• That surfaced a deeper, separate bug: those four POSTGRES_* fields are dead weight. app.py reads DATABASE_URL directly and raises at startup if it's missing — there is no discrete-variable fallback in the running app (app_core/config/database.py::build_database_url() still supports one, but app.py never calls it), and install.sh/.env.example both only ever write DATABASE_URL. So on every current install, the four POSTGRES_* fields never held a value at all, and the wizard could never recognize them as configured no matter what. Replaced them with a single DATABASE_URL field (_validate_database_url(), matching the field the already-working Admin → Environment page manages), which now correctly shows "Already configured" like SECRET_KEY does.
• The same stale-variable assumption was live in a second place: the Diagnostics page's check_environment_config() checked POSTGRES_PASSWORD as a "critical" variable, which was therefore always reported "not set" on every current install — a permanent false-positive warning. Fixed to check DATABASE_URL's embedded password instead.
• Verified live via CDP screenshot against the actual running instance, both before and after this additional fix: intro panel and section badges render correctly, Location/EAS Broadcast open with real pre-filled data, all four optional sections collapsed, Core Settings expands on click and shows the new "Already configured" placeholder for both SECRET_KEY and DATABASE_URL. Checked at a 360px mobile viewport with no new horizontal overflow introduced.
• Documented the new section layout in docs/guides/SETUP_INSTRUCTIONS.md.

[2.212.1] - 2026-09-01 - Add friendly error pages for gateway-down and oversized-upload failures

Added

• Previously, if the Flask app (gunicorn, behind nginx on port 5000) was down or restarting — mid-deploy, crashed, or overloaded — nginx served its bare stock "502 Bad Gateway" page, which gives a visitor no information and no path forward. Same problem for an upload over the size limit: nginx's bare stock "413 Request Entity Too Large" page.
• Added static/errors/gateway-down.html, a self-contained page (no external assets besides the wordmark, which nginx also serves directly) explaining the web dashboard is temporarily unavailable, auto-rechecking every 15 seconds, and — importantly — reassuring the visitor that alert monitoring, the CAP poller, audio decoding, and GPIO/transmitter control are independent background services unaffected by the web app being down. Wired into config/nginx-eas-station.conf via error_page 502 503 504 pointing at an internal-only location block, so nginx serves it directly without proxying to the (unreachable) backend.
• Added static/errors/upload-too-large.html for oversized uploads, listing the size limit and practical next steps (lower bitrate, split shapefile components, or ask an admin to raise the limit).
• Discovered nginx's own error_page 413 doesn't render either custom page over HTTP/2 — a documented nginx limitation where the oversized-body check happens at the protocol/framing layer, bypassing error_page/location entirely, confirmed still present in nginx 1.26.3. Since HTTP/2 is nginx's default over HTTPS and what most browsers negotiate, this would have meant most real users never saw the custom page at all. Fixed at the right layer instead: app.py now sets MAX_CONTENT_LENGTH = 100 MB and a @app.errorhandler(413) that renders error.html (JSON for /api/*) — a normal application response is unaffected by the HTTP/2 quirk regardless of protocol, since it isn't an early protocol-level rejection. nginx's own client_max_body_size is raised to 110 MB as a hard backstop above the app's real limit, so the static upload-too-large.html page (which does still have the HTTP/2 limitation) is now only ever reached by a payload that exceeds even that backstop, over HTTP/1.1.
• Verified live: stopped eas-station-web.service and confirmed the gateway-down page renders at HTTP 502 (internal-only guard also confirmed via 404 on direct access); sent an oversized multipart upload and confirmed nginx's own 413 page over HTTP/2 vs. the custom page over forced HTTP/1.1, which is what led to the app-level fix above.

[2.212.0] - 2026-09-01 - Move the relay lead-in/lead-out from embedded audio silence to program-level GPIO timing

Changed

• 2.211.3 (below) added the relay lead-in as 1 second of silence embedded directly in the generated WAV. That was the wrong layer: resend/replay (/messages/<id>/resend) plays back the stored audio bytes from the original broadcast rather than regenerating them, so a message generated before this fix deployed could never retroactively show the lead-in — confirmed live by resending a pre-fix alert and finding no lead-in, since resend by design never re-runs audio generation. More broadly, baking transmitter-stabilization silence into the audio content meant every consumer of that audio (Icecast stream listeners, FCC-compliance exports, archived recordings) got artificial dead air mixed into the actual alert, permanently and unadjustably.
• The relay lead-in/lead-out is now purely a program-level GPIO timing concern: BROADCAST_LEAD_IN_SECONDS / BROADCAST_LEAD_OUT_SECONDS (both 1.0s, app_utils/eas.py) are applied as time.sleep() calls by every caller that drives the airchain — immediately after set_broadcast_active() and before real playout begins (lead-in), and immediately before clear_broadcast_active() after playout ends (lead-out) — in all four broadcast paths: EASBroadcaster.handle_alert() (automatic CAP-poller alerts and OTA-relay forwarding), the manual send route (webapp/eas/workflow.py), the RWT scheduler (app_core/rwt_scheduler.py, both the automated weekly test and the operator-triggered "Send Test RWT"), and the resend script (scripts/resend_eas_broadcast.py). Because resend replays whatever audio is stored, this also means resend now gets correct relay lead-in/lead-out timing for any stored message going forward, regardless of when that message's audio was generated.
• duration_seconds passed to set_broadcast_active() at each call site now includes both paddings so the Redis marker's TTL and the browser countdown overlay reflect the true on-air window; header_seconds/eom_seconds (the countdown's phase boundaries) are padded by the lead-in only, since they anchor to when real audio actually starts.
• Reverted the embedded-silence approach entirely: removed the lead-in silence added to EASAudioGenerator.build_files()'s 'same' segment, removed the pre-existing unconditional trailing-silence tail in both build_files() and build_manual_components() (the default case when no post-alert chime is configured), and removed build_manual_components()'s now-dead silence_before_header parameter along with its lead-in silence branch. No GPIO-subprocess code changes were needed — services/gpio/alert_indicators.py already keys/releases the relay purely off the broadcast_active marker's edges, so the callers controlling when that marker flips is exactly the right layer for this.

[2.211.3] - 2026-09-01 - Add missing lead-in silence before the SAME header on automatic broadcasts

Added

• EASAudioGenerator.build_files() — the path used for every automatic and forwarded alert (CAP poller auto-forward, OTA relay) — started the composite audio right on the first SAME header FSK bit when no pre-alert chime was configured, the default. The GPIO subprocess keys the relay off the broadcast_active marker, which tracks actual audio playback, so the transmitter got no lead time to come up and stabilize before the header burst — unlike the tail end, which already had a full second of trailing silence after the EOM (holding the relay a second past end-of-message), and unlike build_manual_components() (the manual-send/RWT path), which already had this same lead-in silence.
• Fixed by adding the same unconditional 1-second lead-in silence build_manual_components() already uses, folded into the 'same' audio segment so header_seconds (read by the caller that drives the countdown overlay) still measures true elapsed time from the start of the composite audio.
• No GPIO code changes were needed — the relay already keys and releases off actual audio playback duration via the broadcast_active marker's edges, so extending the audio symmetrically on both ends was sufficient to extend the relay hold symmetrically too.

[2.211.2] - 2026-09-01 - CRITICAL: fix a poller crash that stopped all alert ingestion for ~9 hours

Fixed

• The CAP poller crashed on every polling cycle starting 2026-08-31 ~20:54 EDT, silently dropping every alert fetched from NOAA/IPAWS for roughly 9 hours until diagnosed and fixed live. poller/cap_poller.py's (properties.get('references') or '').strip() assumed CAP's <references> field is always a string (sender,identifier,sent triples, space-separated, per CAP 1.2 §3.3.2.3), but api.weather.gov's JSON API represents the same field as a list of {identifier, sender, sent, @id} objects instead. The first "Update" message carrying that shape (a Heat Advisory) crashed AttributeError: 'list' object has no attribute 'strip' — and since this check runs unconditionally before the Cancel/Update type check, it took down the entire poll cycle, not just that one alert, for every cycle afterward.
• Fixed at the one shared point all three affected call sites already go through: parse_cap_reference_identifiers() now accepts either shape (the legacy CAP-string format from IPAWS, or api.weather.gov's list-of-objects), extracting identifiers correctly from both instead of assuming a string.
• Also added a full traceback (exc_info=True) to the poller's top-level exception log — the bare str(e) this incident originally logged took real production reproduction plus a temporary diagnostic change to pin down; a traceback would have shown the exact line immediately.
• Verified live against the real alert that was crashing every cycle: after the fix, the poller correctly processed and saved it (status: SUCCESS), restoring ingestion. Added regression coverage in tests/test_cap_references_cancellation.py for both shapes, including the exact real-world payload that crashed production, and confirmed all existing tests (CAP-string format, Update-supersession) still pass unchanged.

[2.211.1] - 2026-08-31 - Fix an 8x-slower-than-needed query on the /stats dashboard

Fixed

• Continuation of the same load-time investigation as 2.210.3/2.210.4: profiling every section of /stats's data pipeline individually found collect_polling_trend() alone taking 7.7 of the page's ~8.5 total seconds — every other section combined ran in well under a second.
• Root cause was ORM overhead, not a missing index or a slow query plan: poll_history rows carry a details JSON blob and an error_message Text column, and the function was fetching all columns for every row in the last 30 days (SELECT *, then .all()) via two separate, largely-overlapping queries (7-day and 30-day windows) when only timestamp/status/error_message/execution_time_ms are ever read. EXPLAIN ANALYZE showed the raw filtered scan itself takes under 100ms on this table's ~41K rows — the cost was fetching and fully hydrating tens of thousands of wide ORM objects nothing needed.
• webapp/public/stats_sections/polling.py's collect_polling_trend() now does one query with .with_entities(...) selecting only the four needed columns, and derives the 7-day subset from the 30-day result set in Python instead of querying twice.
• Verified live: collect_polling_trend() dropped from 7.7s to 0.91s (8.4x), and the full /stats page's data-build time dropped to 1.76s. Output values (rates, counts, p95) verified unchanged.
• Verified correctness with the real test suite (tests/test_public_stats_sections.py) run against a genuinely isolated scratch PostgreSQL database (created and dropped for this run only, matching CI's setup) rather than the live database, since that test file's fixtures wipe several tables between tests — all test_polling* tests pass; confirmed the only 3 failures elsewhere in the file (unrelated timezone-formatting assertions) are pre-existing and reproduce identically against the unmodified code.

[2.211.0] - 2026-08-31 - Add retention for system_log, shrink the audio metrics retention window

Added

• Found while auditing database size during the load-time investigation (2.210.3/2.210.4): system_log had no retention policy at all — confirmed by checking both app_core/retention.py's field list and alert_purge.py (which only ever writes audit entries to it, never prunes it). It had grown to 1M+ rows / 850+ MB with nothing capping it. Added system_log_max_age_days (default 90 days, matching the existing audio_alert_max_age_days precedent for operational logs) and wired SystemLog into RetentionScheduler's sweep.
• audio_metrics_max_age_days lowered from 30 to 3 (on both the model default and the already-persisted settings row, via migration). Nothing in the codebase reads raw audio_source_metrics samples older than a short troubleshooting window — the "latest value" and recent-trend endpoints only ever need current data, and app_core/analytics/aggregator.py already rolls raw samples into the separate, much smaller, permanent MetricSnapshot table for long-term history. 30 days of raw per-sample data (at ~288K rows/day) was pure bloat with nothing reading it once it aged past a few hours.
• New "System Log" field added to Settings → Application → Data Retention alongside the existing fields, following the same pattern (day-count input, help text explaining what it covers and doesn't).
• Verified live: migration applied cleanly, settings API round-trips the new field correctly (GET/PUT/GET), admin UI renders and saves it correctly.

[2.210.4] - 2026-08-31 - Fix a 20+ second query behind GET /api/audio/sources

Fixed

• Same load-time investigation as 2.210.3: /api/audio/sources had a 64-second worst case in the site's own request timing data, and the server log showed real psycopg2.errors.QueryCanceled: canceling statement due to statement timeout failures. Root cause: audio_source_metrics is an append-only time-series table (~3.3 rows/sec across all sources, 1.55M rows / 2.8GB at the time of this fix) and the "get the latest reading per source" helper backing this endpoint fetched every matching row, sorted them all in Postgres, and kept only the first one seen per source in Python -- confirmed via EXPLAIN ANALYZE at 21+ seconds with a 400+MB disk-spilled sort.
• Tried Postgres's native DISTINCT ON first (paired with a new composite index) since it's the idiomatic "top-1 per group" operator -- measured no better (17-18s) even with the index in place, because Postgres has no loose/skip-scan index strategy: DISTINCT ON with an IN list still has to walk every matching row before deduplicating. Replaced it with N separate ORDER BY timestamp DESC LIMIT 1 queries instead, one per source (the source list is small -- one row per configured hardware input) -- each one lands directly on an index and stops at the first match. Measured at ~0.15ms per source.
• The new composite index (source_name, timestamp DESC, added via 20260831_audio_metrics_latest_index, built CONCURRENTLY so it didn't lock out the audio service's continuous writes while building on 1.5M+ existing rows) turned out to matter for a real edge case caught during testing: one configured source had gone quiet days before the others, so a plain per-source lookup using only the existing timestamp index had to scan backward through everything every other source wrote since then before finding it. With the composite index, that same lookup is 0.14ms regardless of how stale a given source's data is.
• webapp/admin/audio_ingest/listing.py's _latest_metrics_by_source() now uses this per-source approach; all 14 existing tests in tests/test_audio_source_listing.py pass unchanged, confirming no behavioral regression.
• Verified live against the deployed app: the endpoint's real, uncached response time dropped from 20-64+ seconds to under a second.

[2.210.3] - 2026-08-31 - Fix site-wide request stalls caused by CPU-heavy work blocking a gevent worker

Fixed

• Root-caused via the site's own recorded request-timing data (WebRequestLog, not guesswork): /api/broadcast/state — a trivial Redis-backed status check polled from every page's status widget, called 13,880 times in the sample window — had a 230ms average but a 163-second worst case. The query behind it is fast (cap_alerts has under 1,000 rows; EXPLAIN ANALYZE showed 6.8ms). The real cause: gevent workers only yield control during I/O, and /alerts/<id>/export-image.png (the social-share image renderer) does 20-30 seconds of pure CPU work — Pillow composition, tile mosaicking — synchronously inside the request handler. With only 2 gunicorn workers, one in-flight image export could stall every other concurrent request routed to that worker, including completely unrelated ones like the broadcast-state poll.
• --workers 2 → 4 in systemd/eas-station-web.service (idle CPU headroom confirmed: 4 cores, only 2 in use). MemoryMax raised 1500M → 4000M to match — the existing 2 workers were already observed using ~956M+474M combined RSS, so doubling worker count without raising the cgroup limit would have hit the exact OOM-kill failure mode that MemoryMax was originally raised to avoid.
• Root-cause fix: webapp/admin/api/routes_alert_export.py's new _run_off_worker() runs the image renderer on a real OS thread via gevent's own threadpool instead of the request greenlet, so the worker's event loop stays free to serve other requests while it renders. generate_alert_image() already supported running outside a Flask context via an explicit db_session (used by the CAP poller's notification-email images) — reused that instead of inventing a new pattern, with a dedicated sessionmaker-backed session per render (the request's own db.session isn't safe to share across threads).
• Verified with a live concurrency test against the deployed app: fired the ~24s image export and three /api/broadcast/state polls at once — all three fast requests completed in under a second each while the image render was still in flight, and the returned PNG was pixel-identical to the pre-fix synchronous output.
• Added tests/test_alert_export_off_worker.py — regression coverage for the exact bug caught during manual testing of this fix: the first version looked up db.engine (which needs current_app) inside the threaded closure instead of on the calling greenlet, so it failed every real request with RuntimeError: Working outside of application context.

[2.210.2] - 2026-08-31 - Remove ambient debug logging from the browser console

Removed

• A repo-wide audit found 142 console.log/console.debug/console.info calls with no gating flag, firing unconditionally on normal page loads and user actions — page-init narration ("Interactive map page loaded", "Alert detail page scripts loading..."), per-request cache hit/miss chatter, and bare function-entry markers ("toggleRawData called") left over from active debugging. Removed 116 of them (16 in static/js/*.js, 100 across 10 templates) that were pure narration with no diagnostic value.
• One instance was more than noise: templates/admin/environment.html logged the signed-in admin's username and role to the console on every load of the environment-variables settings page — a minor but real thing to not be doing on a page that manages secrets.
• static/js/core/cache.js's cache hit/miss logging had a "dev only" gate (hostname === 'localhost') that was already broken for exactly this kind of self-hosted appliance, since the app is commonly accessed via http://localhost:5000 in normal use, not just development — removed rather than tightened, since the logging itself wasn't valuable.
• Kept every console.log/debug/info call that's a genuine diagnostic: anything inside a catch block logging a real caught error (still present in every file touched), and the two intentional opt-in developer tools this app ships (window.mapDebug.* on the dashboard map, and inspectBoundaries()'s report generator) — those only produce console output when a developer explicitly invokes them, not on ordinary page loads.
• Verified with a live browser console sweep (network cache disabled) across all 12 touched pages against the deployed app — every page loads completely silent, zero console output, zero new errors.

[2.210.1] - 2026-08-31 - Fix heading hierarchy accessibility warnings site-wide

Fixed

• static/js/accessibility-utils.js's setupHeadingHierarchy() flags any heading whose level skips more than one step deeper than the previous heading in DOM order (e.g. h2 straight to h5) — a real, longstanding gap in nearly every page, since card/widget headers throughout the app were written as bare h5/h6 regardless of the page's actual section depth. Fixed every skip across the template tree (82 templates, 1 shared JS component) rather than leaving it as a known issue.
• Two-pattern fix, chosen per heading: (1) renumber the tag to the correct sequential level relative to its context, adding a matching .hN class (h1, .h1/h2, .h2/etc. are already paired in static/css/base.css) so the visual size is unchanged even though the semantic level moved; (2) for headings that were really just small styled labels with no real document-outline meaning (a caption over a JSON blob, a stat tile's number label), demoted them the same way but the net effect is identical markup weight, just at a level that doesn't skip.
• templates/base.html's global footer (Quick Access/Resources/Legal & Info/System Status, previously h6) and its Display Units modal title were the single highest-leverage fix — both render on every page, so fixing them once cleared the same warning everywhere without touching per-page templates.
• Verified with a static heading-sequence scanner (expanding {% include %}s to check true rendered order, not just each file in isolation) and a live browser console sweep across 90 real routes against the deployed app — zero Heading hierarchy jump warnings remain anywhere, and no new console errors were introduced.

[2.210.0] - 2026-08-31 - Add boundary layer toggles to the alert detail map

Added

• The alert detail page's Alert Coverage Map fetched all 8 boundary types (counties, fire, ems, electric, townships, villages, telephone, school) on every load but only ever showed counties — the rest were invisible dead weight with no way to see them. Added a row of toggle switches (mirroring the dashboard's existing "Map Layers" panel in templates/index.html) below the map so any of the 8 can be shown on demand.
• Counties stay on by default — no change to the page's existing default appearance, just a way to opt into the others without editing the map itself.
• Layer color swatches reuse getBoundaryColor(), already defined on this page and shared with the boundary popups.

[2.209.0] - 2026-08-31 - Load incorporated city/village boundaries for the "affected municipalities" display

Changed

• New scripts/load_municipality_boundaries.py loads US Census incorporated-place (city/village) boundaries from the TIGER/Line cartographic "Places" file into the existing generic boundaries table (type villages, already a recognized, colored, grouped type in app_core/boundaries.py's BOUNDARY_TYPE_CONFIG). Once loaded, the alert detail page's existing boundary-intersection display — which already lists every boundary type an alert's polygon intersects — starts showing named cities and villages for free, with no new UI work.
• Deliberately scoped to the station's own coverage counties via RWTScheduleConfig.same_codes (not AlertFilterSettings.fips_codes, which carries non-geographic wildcard entries) so the boundaries table isn't bloated with the ~32,000-record national dataset. Unincorporated Census Designated Places (CDPs) are filtered out — this is meant to show real municipalities, not census-only place designations.
• The Places file has no per-record county field (a place isn't nested inside exactly one county the way a township is), so county scoping uses a real PostGIS ST_Intersects test against the already-loaded us_county_boundaries geometry rather than a FIPS-string compare.
• Verified end-to-end against the live database: loaded 99 real cities/villages across the station's 8-county coverage area, recalculated intersections for a real historical alert, and confirmed named cities (e.g. "Lima city", "Defiance city") and villages now appear in its stored intersections.
• Fixed a SAME-code/plain-FIPS format mismatch caught during testing: SAME codes are 6-digit PSSCCC (portion digit + state + county), while the Census shapefile's own STATEFP+COUNTYFP is 5-digit — comparing them directly matched nothing. Added tests/test_load_municipality_boundaries.py as a regression test for the normalization.

[2.208.0] - 2026-08-31 - Add a High-Resolution (Level II) radar loop with a velocity overlay

Added

• New High-Resolution Radar Loop card on the alert detail page, below the existing (Level III) Radar Loop card — an explicitly separate, distinctly-labeled feature, not a silent upgrade to it. maps.py's _render_map() used to always prefer a sharper Level II render when a site was in range, but that was reverted because the exported/looped image could then disagree pixel-for-pixel with the live "Radar (at time of alert)" toggle (different resolution, different color ramp). This reintroduces Level II behind its own opt-in radar_source='level2' parameter, used by exactly one caller, so the toggle, share-card, and standard loop are all unaffected.
• Adds a Reflectivity / Velocity selector — velocity is a Level II–only product (no Level III equivalent), useful for spotting rotation. radar_level2.py's render_frame()/_plot_ppi() now take a field parameter; velocity uses cmweather's NWSVel colormap over a ±32 m/s range (the practical base-velocity Nyquist limit) with its own VELOCITY_LEGEND. No de-aliasing is applied — a known limitation of the raw base product.
• Level II only reaches ~230km from a WSR-88D site, so radar_loop_hires.py checks coverage once per alert up front and reports a genuine coverage gap distinctly from "not a weather alert", rather than silently caching a radar-less frame that would look identical to a legitimate no-echo scan.
• render_frame() now returns (image, scan_time) — the matched volume's actual timestamp, not just the requested one — so the on-image "Radar H:MM" caption is accurate for Level II the same way it already was for Level III.
• Verified end-to-end against live NOAA data (not just mocks): both fields render correctly through radar_level2.render_frame() directly and through the full _render_map() composite (basemap + polygon + radar + legend).
• Added tests/test_image_export_radar_loop_hires.py (11 tests: eligibility, the coverage-gap short-circuit, caching, field-scoped cache isolation, render-failure handling).
• Corrected templates/help.html's Radar Loop documentation, which had drifted to describe the old (reverted) "loop prefers Level II automatically" behavior as current.

[2.207.4] - 2026-08-31 - Bump zigpy to 2.1, fixing a permit_joining rename it silently would have broken

Changed

• zigpy now floats >=2.1.0 (was >=0.60.0) at the maintainer's request, after closing Dependabot's version of this bump (#2524) pending verification — zigpy manages real Zigbee hardware pairing, which can't be exercised in CI.
• Found one real, confirmed break by inspecting the installed zigpy==2.1.0 + zigpy-znp==1.1.0 API directly (the version pair pip's resolver actually picks for these pins): ControllerApplication.permit_joining(duration) was renamed to .permit(time_s, node=None) — the old name doesn't exist at all on 2.1.0. services/zigbee/controller.py's permit_join()/close_join() (the pairing-mode open/close methods) called the old name; close_join()'s call is wrapped in a broad except Exception, so this would have failed the same silent way the pysnmp break below did.
• Verified the rest of this codebase's zigpy-facing API surface is unaffected: ControllerApplication.__init__, .add_listener, .startup, .shutdown, the device_joined/device_initialized listener callbacks, and the Device.ieee/.nwk/.model/.manufacturer attributes this code reads are all unchanged between 0.60.0 and 2.1.0.
• Added tests/test_zigbee_controller_permit.py — the first test coverage this file has ever had. It stubs the zigpy application object (no real coordinator hardware is available in CI) and asserts permit_join()/close_join() call .permit(), not the nonexistent .permit_joining().
• Real device pairing over a live coordinator has not been hardware-tested — this fix corrects a confirmed API break, but only physical testing can confirm end-to-end pairing behavior.

[2.207.3] - 2026-08-31 - Fix silently-broken SNMP compliance traps under pysnmp 7

Fixed

• A dependency bump already on main (pysnmp >=7.1.29, from a Dependabot PR merged earlier the same day) silently broke SNMP compliance trap sending. pysnmp 7 restructured hlapi into arch-specific, asyncio-native submodules and dropped the old flat pysnmp.hlapi module (CommunityData, SnmpEngine, sendNotification as a sync-flavored generator) this code imported from. Both HealthAlertWorker._send_snmp_traps() (app_core/system_health.py) and the admin "Test SNMP" button (webapp/admin/notifications.py) catch that import failure broadly and just log/return a warning — so this broke with no crash and, since there was no prior test coverage for SNMP trap sending at all, no test failure either. Traps would have silently stopped sending entirely.
• Both call sites now import from pysnmp.hlapi.v3arch.asyncio, call the now-async send_notification() via asyncio.run(), use add_varbinds (the renamed, non-deprecated method), and wrap the trap payload in an explicit OctetString (pysnmp 7 no longer auto-coerces a raw Python str varbind value). Each call's SnmpEngine is now explicitly closed via close_dispatcher() in a finally block — without it, every trap sent (including from the recurring background health-check interval) leaked a UDP dispatcher socket for the life of the process.
• Verified end-to-end, not just via import checks: sent a real trap over a real UDP socket to a local listener and confirmed the payload arrives, using an isolated venv with pysnmp==7.1.29 actually installed.
• Added tests/test_snmp_trap_pysnmp7.py — the first test coverage this code path has ever had.

[2.207.2] - 2026-08-31 - Bump numba to 0.67, lifting the previous llvmlite-size cap

Changed

• numba now floats >=0.67.0,<0.68.0 (was >=0.61.0,<0.64.0). The old cap existed specifically to avoid numba 0.64+'s heavier llvmlite dependency; verified that cost is real (llvmlite 0.49.0's aarch64 wheel is ~58MB) but accepted it deliberately as a one-time download rather than staying capped indefinitely.
• Verified end-to-end on the real target platform (aarch64) before merging: installed numba 0.67.0 into an isolated copy of the deployment venv (pulls llvmlite 0.49.0, keeps numpy at the already-pinned 2.3.5 — numba 0.67's own requirement, numpy<2.6, is looser than before), confirmed app_core/radio/demod/kernels.py JIT-compiles, and ran the full demod/RBDS test suite (93 tests) against it.
• Updated the Numba badges (README top badge, attribution table, footer partial) to the new range.

[2.207.1] - 2026-08-31 - Fix a startup crash from Flask-Caching 2.5.0's dropped CACHE_TYPE aliases

Fixed

• Bumped Flask-Caching to 2.5.0 (a Dependabot PR for this had failed CI: flask_caching.backends.redis no longer exists in that release). Cache._set_cache() builds an import path from CACHE_TYPE and imports it; 2.5.0 dropped the lowercase short aliases (redis, simple, filesystem, null) that flask_caching.backends used to expose, keeping only the actual class names (RedisCache, SimpleCache, FileSystemCache, NullCache). Passing the old alias straight through raised an ImportError from init_cache(), which runs unconditionally during app creation — this would have crashed the whole web service on boot, not just broken caching.
• app_core/cache.py now translates the lowercase alias to the class name right at the Flask-Caching boundary. Everything else — the CACHE_TYPE env var, the Settings → Environment dropdown, already-deployed .env files — keeps using the lowercase form; only the value actually handed to Flask-Caching changed.
• Added tests/test_app_cache_type_resolution.py covering all four aliases plus the "already a resolvable class name" passthrough case.

[2.207.0] - 2026-08-31 - Require Python 3.13 / Debian 13 (Trixie); drop 3.11/3.12 support

Changed

• Breaking for Debian 12 / Python 3.11 or 3.12 installs. The project now requires Python 3.13 and targets Debian 13 (Trixie) / Raspberry Pi OS (Trixie-based) only. Maintaining both floors was an ongoing tax: scipy was capped below 1.18 solely because that series drops 3.11, numba's compatible-numpy range and the audioop-lts marker both existed to branch on the interpreter version, and CI ran the whole suite twice per PR to catch drift between them.
• requirements.txt: unpinned scipy's 3.11-driven cap (now 1.18.1) and removed audioop-lts's python_version >= "3.13" marker (unconditional now that 3.13 is the floor).
• .github/workflows/tests.yml: CI matrix is now Python 3.13 only (was ['3.11', '3.13']); the lint job also moved off 3.11.
• .github/workflows/release.yml, .github/workflows/docs-pages.yml: bumped their own Python setup steps to 3.13 for consistency.
• pyproject.toml: ruff target-version is now py313; removed the now-dead audioop deprecation-warning filter (that warning only ever fired on <3.13, which no longer runs in CI).
• install.sh, sdr_hardware_service.py, scripts/fix_soapysdr_venv.sh: removed the "downgrade to Python 3.12" SoapySDR troubleshooting suggestion and the Python 3.10-3.12 site-packages fallback paths, since downgrading is no longer a supported workaround.
• Updated README.md's System Requirements table, docs/reference/ABOUT.md, docs/guides/HARDWARE_QUICKSTART.md, templates/help.html, and tests/README.md to state the new floor.

[2.206.0] - 2026-08-31 - Pin the one-click upgrade to a specific release, and fix the upgrade docs

Fixed

• The "Version to install" field on Admin → Operations' System Upgrade card is now a dropdown populated from the repository's actual release tags (via a new GET /admin/operations/upgrade/tags endpoint), instead of a blank text box requiring an operator to already know the exact tag spelling. "Track main (latest)" stays the default and behaves exactly as before; a "Custom branch, tag, or commit…" option keeps the free-text field available for anything not in the list.
• Rewrote docs/guides/one_button_upgrade.md, which described a Docker-image-based upgrade pipeline (kr8mer/eas-station:latest, a nonexistent .github/workflows/build.yml) this project has never used — EAS Station deploys bare-metal via install.sh/update.sh. It now accurately documents the real update.sh-via-systemd-unit mechanism.

[2.205.0] - 2026-08-31 - Bracket the weekly test with optional spoken announcements and a guaranteed lead-in silence

Changed

• The automated Required Weekly Test can now play optional station courtesy announcements before the SAME header ("This station is conducting a test of the Emergency Alert System...") and after the EOM ("This concludes this test..."), synthesized via the configured TTS provider and enabled/edited from the Weekly Test Automation page (/rwt-schedule). They play outside the encoded SAME/EOM burst, so they never affect RWT format compliance under 47 CFR §11.61(a)(1)(ii).
• The composite RWT audio now always opens with at least a second of true silence before the SAME header begins (previously the header started at t=0 whenever no pre-alert chime was configured), mirroring the second of silence that already follows the EOM before the air-chain returns to normal programming.
• EASAudioGenerator.build_manual_components() gained silence_before_header, lead_announcement_samples, and trail_announcement_samples parameters; RWTScheduleConfig gained pre_announcement_enabled/pre_announcement_text/post_announcement_enabled/post_announcement_text (migration 20260828_rwt_test_announcements).

[2.204.0] - 2026-08-31 - Show whether an upgrade is actually available before running one

The one-click "System Upgrade" button gave no way to tell whether there was anything to upgrade to -- clicking Start Upgrade was the only way to find out, and it always restarts every service even when the box is already current.

Added

• New GET /admin/operations/upgrade/check route: git fetchs the target branch (defaulting to whatever branch is currently checked out, usually main) and compares local HEAD against origin/<branch>, reporting the current and remote VERSION file contents and how many commits behind. git fetch only updates this checkout's own remote-tracking refs -- the same thing git status implicitly keeps current -- so it never touches the working tree, which is what makes it safe to run automatically on page load rather than waiting for a click.
• The Operations page now shows "Up to date (2.204.0)" or "Update available: 2.204.0 → 2.205.0 (7 commits behind main)" above the Start Upgrade button, with a "Check again" link that re-reads whatever's in the Git Branch/Tag field. Purely informational -- doesn't gate the button, since a specific tag or commit checkout isn't always comparable this way.

[2.203.7] - 2026-08-31 - Stop anonymous visitors' browsers from hitting a guaranteed-401 endpoint

Checked the browser console on every public page (the dashboard, /health, /style-guide, /about, /help) via a headless-Chromium CDP session. Every one of them showed two 401s for /admin/pending-alerts/api/list on every load, plus a repeat every 5s from its WebSocket-fallback poller.

Fixed

• navbar_scripts.html (included on every page via the navbar) fires this fetch to light the stack-light widget's blue "pending alerts" state. The endpoint requires login (@require_auth) then eas.view (@require_permission) on top of that; the existing .then(r.ok ? ... : null) handling already covers an authenticated-but-under-permissioned viewer gracefully (403 -> null, blue state just never lights), but nobody anticipated a completely anonymous visitor -- this dashboard has no login wall -- who gets a guaranteed 401 instead, every page load, every 5s poll. Gated both the initial fetch and the WebSocket fallback subscription behind current_user.is_authenticated (already available in the navbar's Jinja context). Anonymous visitors now skip the call entirely.

[2.203.6] - 2026-08-30 - Fix the actual highest-specificity rule controlling dashboard toggle switches

Fixed

• The app-wide .form-switch fix (2.203.5) didn't actually take effect on the dashboard: live-checked via the browser's own document.styleSheets (matching every CSS rule against the element in cascade order, not just reading computed style), the real winner was .layer-options .form-check.form-switch .form-check-input -- a page-local rule in templates/index.html with higher specificity (four classes) than the app-wide fix (two classes), still setting background-color: var(--light-color). That selector covers every switch in the Map Layers panel: Active/Historical Alerts, the severity filters, the event-type filters, and every boundary-layer toggle. Switched it to the same fixed #495057 used everywhere else tonight. Grepped every other template for a similarly-scoped override and found none.

[2.203.5] - 2026-08-30 - Sweep the rest of the app for the var(--text-muted)-as-fill bug

Grepped every background: var(--text-muted) / background-color: var(--text-muted) in the codebase (not just the dashboard) for the same failure mode fixed five times already tonight, and computed exact WCAG contrast for every hit that pairs it with text. Six more instances, all fixed the same way (a fixed colour instead of the variable):

Changed

• security_settings.html's .role-badge.viewer (bg --text-muted, text --bg-primary) failed in 17 of the 20 themes -- Blue measured 1.00:1, literally the same colour on both sides.
• alert_detail.html's .coverage-badge.bg-secondary (text --surface-color) and .scope-coverage-na (text --bg-color) failed in 5 and 9 themes respectively.
• displays_preview.html's .display-status.disabled was marginal (3.78:1) in 8 themes.
• static/css/styles.css's .form-switch .form-check-input unchecked track, paired with the switch knob's fixed white SVG circle, failed app-wide in 7 themes (1.33:1 in Lightning) -- every unchecked toggle switch on every page, not just one component.
• alert_detail.html also carried its own copy of the radar-legend chips fixed on the dashboard in 2.203.1 (rgb(...) badges missing an explicit color) -- same fix applied.

Also found, but left alone: static/css/styles.css already carries a large, deliberate comment block (~line 1131) documenting this exact var(--light-color)-without-paired-text pattern in .workflow-card > .card-header, .same-chip, .alert-summary-panel, .feature-group, .layout-preset-btn:hover and .rss-item-row, with an existing per-theme text-colour override. Unlike #date-filters (also on that list, and the actual bug fixed in 2.203.2 -- its label had a more specific rule fighting the override back to --text-muted), these five weren't verified live and weren't touched.

[2.203.4] - 2026-08-30 - Drop the fill on historical alerts -- semi-transparent fills stack

Severity colouring (2.203.3) fixed historical alerts all looking the same, but introduced a different problem visible immediately on a real 30-day window: a date range easily loads 100+ overlapping polygons (the same county gets re-alerted many times across a month), and semi-transparent fills stack. Even a faint 0.12 fillOpacity per shape compounded across dozens of overlapping ones into a solid, texture-free colour wash that hid the map underneath -- a worse failure than the flat-colour problem it was fixing.

Fixed

• displayHistoricalAlerts() now draws outline-only (fill: false, opacity: 0.75). Outline-only doesn't have the compounding failure mode a fill does; the dashed, severity-coloured stroke alone still distinguishes both alert type (colour) and historical-vs-active (dashed + reference pane, which sits under the hazard pane).

[2.203.3] - 2026-08-30 - Colour historical alerts by severity instead of one flat line

The previous pass (2.203.2) fixed historical alerts being invisible against the basemap, but left every one of them the same flat neutral line -- fixing invisible-to-visible didn't fix indistinguishable-from- each-other: a flood watch and a tornado warning looked identical, which throws away the one piece of information a reader actually wants when scanning a season of alerts.

Changed

• displayHistoricalAlerts() now resolves each alert's own colour via EASMap.severityColor() -- the same resolver hazardLayer() uses for active alerts -- instead of one fixed --map-reference-line value. Dashed stroke, lower opacity (0.65/0.12 vs. an active alert's full strength) and the reference pane (which sits under the hazard pane, so an active alert on the same spot always wins) still mark it as historical rather than happening now; only the hue was the problem.

[2.203.2] - 2026-08-30 - Fix the date-filter panel and historical-alert map lines being unreadable

Found from a user screenshot of the Lightning theme: the date-filter sub-panel's FROM/TO labels were nearly invisible, and every historical alert on the map rendered as a barely-visible pale line indistinguishable from the basemap.

Two more instances of the same root cause already fixed twice today elsewhere on this page (var(--text-muted) used somewhere it wasn't designed for):

Fixed

• #date-filters paired background: var(--light-color) with the label's color: var(--text-muted). That combination failed WCAG AA in all 20 themes, not just Lightning (1.24:1) -- Cosmo measured 2.45:1, Aurora 1.39:1, Charcoal 1.65:1, Slate 1.93:1, and the rest landed "marginal" in the 3.7-4.5 range. Switched to --surface-color / --text-secondary, the same pairing already verified safe everywhere else on this card.
• displayHistoricalAlerts() drew every historical alert polygon in getColorVar('--text-muted') at 0.6 opacity over the toned basemap -- near-white in several themes, so the shape all but disappeared into the map. Switched to --map-reference-line, the variable this same map already uses for county boundary lines, purpose-built and tuned for exactly this job (legible against the basemap in both light and dark map-tone modes, unlike --text-muted which has 20 unrelated values never designed to sit on top of map tiles).

[2.203.1] - 2026-08-30 - Fix unreadable radar-legend chips on the dashboard

The dashboard's Reflectivity (dBZ) legend (shown when the map's Radar toggle is on) has six colour chips; three of them (20/40/60+) never got an explicit color, so they fell back to .badge's default white text. That failed WCAG AA on two of them regardless of theme, since the colours are hardcoded RGB literals, not theme variables: the orange "40" chip measured 2.07:1, the green "20" chip 3.42:1 (AA needs 4.5:1).

Fixed

• No single ink works for all six -- the range runs from light green through saturated red and magenta. Set each chip's color individually to whichever of black/white actually clears 4.5:1 against its own background (computed, not eyeballed): #123 for 5/20/30/40, white for the red "50" chip, black for the magenta "60+" chip.

[2.203.0] - 2026-08-30 - Filter dashboard alerts by severity and event type

The dashboard map could only toggle Active vs. Historical alerts -- no way to narrow by severity or event type, even though the historical set alone spans 15+ event names (Flood Warning, Severe Thunderstorm Warning, routine Required Weekly Test, ...) and all 5 CAP severity levels.

Added

• A Severity checklist (fixed 5-item CAP set) and an Event Type checklist (populated from whatever's actually in the loaded active + historical data, with All/None shortcuts) to the Alert Types panel. Both filters apply to whichever alert layers are currently shown -- switching between Active and Historical, or reloading either one, never loses the selection, since displayAlerts()/displayHistoricalAlerts() filter from the same excludedSeverities/excludedEventTypes state and redraw from the already-cached data rather than re-fetching.

[2.202.2] - 2026-08-30 - Fix unreadable map-popup badges in several themes (worst in Lightning)

.historical-badge (the small "Historical" tag on a historical-alert map popup) used background: linear-gradient(45deg, var(--text-muted), #495057) with white text. --text-muted is tuned for text on a surface, not for carrying white text as a fill, and it's a near-white colour in several themes -- worst in Lightning (#d8e0f0), where the badge measured 1.33:1 contrast against its own white text (WCAG AA needs 4.5:1): effectively invisible. Cosmo, Dark, Coffee, Aurora, Charcoal and Slate had the same failure to varying degrees.

Fixed

• Switched to a fixed dark gradient (#495057 → #343a40, 8.2:1 / 11.5:1 with white) that doesn't depend on the theme. Fixed the same root cause in two related spots that shared it more marginally (dark text on var(--text-muted), 3.78:1 in eight themes against the 4.5:1 target): .severity-unknown and the "EXPIRED" alert-popup badge, both now #adb5bd background (8.4:1 with #1a1a1a text) instead of the theme variable.

[2.202.1] - 2026-08-30 - Fix the dashboard's Historical Alerts toggle

Fixed

• Checking the "Historical Alerts" layer checkbox only revealed the date pickers -- it never actually loaded or displayed anything. loadHistoricalAlerts() was wired to the "Apply Filter" button alone, so a user checking the box (the same gesture that immediately shows/hides Active Alerts) saw nothing happen until they noticed they also had to click Apply Filter below it. Checking the checkbox now calls loadHistoricalAlerts() immediately instead of waiting for a second click.
• Separately, active and historical alerts shared one Leaflet layer group (alertLayer). displayHistoricalAlerts() never cleared it before adding new polygons, so re-applying the date filter with a different range piled new alerts on top of old ones instead of replacing them; toggling "Active Alerts" off/on also wiped out historical polygons as a side effect of alertLayer.clearLayers(), since both lived in the same group. Historical alerts now get their own historicalLayer, cleared at the top of every displayHistoricalAlerts() call.

[2.202.0] - 2026-08-30 - Radar timestamp, card lift, bolded hazard numbers, and pill glow on the share card

Fifth polish pass on the social-share card, all four picked by the user from a shortlist:

Changed

• Radar "as of" timestamp (maps.py) — _fetch_radar_overlay() now returns (image, scan_time); _draw_radar_legend() shows "Radar 2:08 PM" under the dBZ legend so the card admits the reflectivity can be a few minutes older than the alert's own timestamps (the WMS-T service snaps to its own 5-minute cadence).
• Card lift (drawing.py's new _apply_card_lift()) — a thin, low-alpha inner shadow along the card's own rounded edge. A real drop shadow needs canvas space outside the card, which would change the finished PNG's pixel dimensions -- both the export UI's advertised sizes and the test suite assume those are exact -- so this stays inside the existing bounds instead, deliberately subtle since header/footer copy sits close to every edge.
• Bolded hazard numbers (panels_text.py's new _draw_emphasized_line()) — HAZARD/IMPACTS body text now bolds numeric magnitudes with units ("60 mph", 1.00") inline via a new small_bold font, so the actionable numbers are scannable without reading the full sentence.
• Pill glow (drawing.py's new _draw_pill_glow()) — the tier/severity badges (WARNING / SEVERE) get a soft blurred glow behind them, the same glow-behind-a-crisp-shape technique already used for the map's hazard polygon, so the single most important words on the card pop first.

[2.201.0] - 2026-08-30 - Section icons, a tornado confidence gauge, a storm ping, and a north arrow on the share card

Fourth polish pass on the social-share card:

Changed

• Section-header icons (icons.py's new _SECTION_ICON_FN, wired into drawing.py's _section_header()) — a small glyph (flag / pin / lines / triangle / ring / needle / bolt) next to HEADLINE, AFFECTED AREAS, DESCRIPTION, STORM THREATS, COVERAGE, STORM MOTION and ACTION, keyed off the exact title string so every call site picked it up with no signature changes.
• Tornado detection as a stepped gauge (panels.py) — tornado detection (possible/radar/observed) has no continuous magnitude like wind/hail, so it now gets the same gauge shape as a discrete 3-segment version (_draw_stepped_gauge) instead of the flat funnel icon; a bare "none" reading still falls back to the icon.
• Storm-position ping (storm_overlay.py) — two fading outward rings behind the "now" marker, reading as a live radar ping / location pulse even in a still image, instead of a plain disc.
• North arrow (maps.py's new _draw_north_arrow()) — small two-tone compass needle in the map's one previously-empty corner (upper-left), added to the county-label keep-out list like the other three chrome elements so nothing lands on top of it.

[2.200.1] - 2026-08-30 - Fix crowded threat-level/category labels on the share card

Fixed

• The threat-level line ("Radar") and the category label ("WIND") under each Storm Threats gauge sat only ~11px apart against ~13-15px-tall text, so they nearly touched. card_h bumped 108 → 118 and the category label's y is now derived from where the level line actually measures to (_th() + a real gap) instead of a second guessed constant -- closes the same gap for the icon-fallback path too (wind/hail with an unparsable gust/size).

[2.200.0] - 2026-08-30 - Add film grain, an expiration countdown badge, and threat gauges to the share card

Three additive polish passes on the social-share card, picked from user feedback after the polygon-joint and radar-opacity fixes:

Added

• Header film grain (weather_fx.py) — a subtle monochrome noise layer composited over the gradient before particles/text, so the header reads as a textured surface instead of a flat CSS-button gradient. Low alpha (14/255) by design; the goal is texture, not visible speckle.
• Expiration countdown badge (text.py's new _format_countdown(), wired into the footer in render.py) — "Expires in 42m" next to the Issued/Expires timestamps, colour-coded red/orange/slate by urgency (≤30 min / ≤2 h / normal), skipped entirely when it would crowd the brand credit on a narrow canvas.
• Storm-threat gauge meters (panels.py) — the Wind/Hail cards now plot their magnitude as a semicircular gauge arc (0-100 mph / 0-3") instead of a flat icon, with the value sitting inside the arc. Tornado detection has no continuous magnitude (radar/observed/possible), so it keeps the plain icon.

[2.199.2] - 2026-08-30 - Fix jagged stroke joints on the share-card alert polygon and county outlines

Fixed

• Every bend in the affected-area polygon (and the county reference outlines under it) showed a visible notch cut into the wide white+accent stroke -- PIL's ImageDraw.line() defaults to a hard miter join with no rounding, so a winding shape (a road corridor, an irregular county line) looked jagged at each vertex, worst on the crisp casing/core outline drawn on top. Added joint='curve' to the county-outline, glow, and casing/core line() calls in app_utils/image_export/maps.py so every vertex rounds smoothly instead.

[2.199.1] - 2026-08-30 - Raise radar overlay opacity so reflectivity actually reads at a glance

Changed

• _RADAR_OPACITY (app_utils/image_export/maps.py) and the matching radarLayer() default (static/js/core/map_theme.js) were both 0.45 -- legible over the basemap, but light/moderate reflectivity read as a faint haze on the social-share card and the in-app radar pane alike. Raised to 0.6, verified pixel-by-pixel on a live storm cell alongside 0.45 and a rejected 0.75 (which reproduced the washed-out-basemap problem a prior pass already hit at 0.65 and pulled back from). The two files share one constant by design, so both surfaces -- and the animated radar loop export, which reuses the same _render_map() call -- move together.

[2.199.0] - 2026-08-30 - Extend systemd watchdog coverage to every hardware subsystem and the web app

Only eas-station-audio, -demod, and -poller had Type=notify + WatchdogSec= -- Restart=always recovers a crashed process on every unit, but nothing caught a hung-but-still-alive one (a wedged I2C bus, a stuck GPIO ioctl, a deadlocked serial read) on -displays, -endec-feeds, -gpio, -gps, -network, -sdr, -zigbee, or -web.

Added

• Each of the six services/*/__main__.py split-hardware entry points plus sdr_hardware_service.py now call sd_notify("READY=1") once startup finishes and kick a Watchdog() (app_utils/system/sd_notify.py, the same helper eas_monitoring_service.py/cap_poller.py already used) from their existing ~1 Hz main loop; the matching unit files gained Type=notify/NotifyAccess=main/WatchdogSec=60.
• The web app runs under a multi-process gunicorn arbiter rather than a single Python loop, so it needed its own mechanism: a new gunicorn.conf.py (loaded via --config in eas-station-web.service) starts a background thread in the arbiter's when_ready hook that kicks the watchdog every 5s for as long as the arbiter's event loop is alive. This only covers an arbiter deadlock, not a single hung gevent worker -- that path is already gunicorn's own --timeout 300 (kills and respawns the worker).

[2.198.0] - 2026-08-28 - Make the one-click upgrade button run the real update.sh, with live progress

The one-click "System Upgrade" button (fixed in 2.196.3 to at least run something bare-metal-appropriate) still only replicated a thin slice of what update.sh actually does -- no pre-upgrade backup step, no service stop/start ordering, no system package updates, no systemd unit refresh, no nginx check, none of the migration-failure recovery path, and no step-by-step feedback beyond a final stdout/stderr dump once everything finished. Reimplementing all of that a second time in Python would have meant two upgrade scripts that inevitably drift apart, so the button now runs update.sh itself -- the exact script sudo bash update.sh runs at a terminal -- non-interactively.

update.sh gained --non-interactive, --skip-backup, and --checkout <ref> flags: every whiptail confirmation (the welcome dialog, "create a backup?", the migration-failure pause, the final success/error dialog) is skipped in favor of a safe default (a backup is taken by default, since there is no operator here to notice a silently-skipped one) instead of hanging on a keypress that will never come, while every echo_step/echo_info/echo_success/echo_warning/echo_error line still goes to the log exactly as it does interactively.

Getting there surfaced two real, previously-latent bugs in scripts/lib/ui.sh, neither ever exercised before because every prior invocation of install.sh/update.sh had a real controlling terminal attached:

Changed

• [ -w /dev/tty ], used throughout as the "is a real terminal available" guard, only checks the special device node's own permission bits (always broad, crw-rw-rw-) -- not whether this session actually has a controlling terminal to open it against. A fully detached session (like the one below) passes that check and then dies the moment something really writes to /dev/tty, with "No such device or address". Replaced every such guard with a real probe (attempt an actual open, once, cached in _UI_HAS_CONTROLLING_TTY).
• _ui_ensure_gauge returns 1 (an expected, ordinary result meaning "no whiptail gauge available, fall back to plain output") from a bare, unguarded call site in echo_step. Under update.sh's set -e, that ordinary "couldn't attach" result took the whole script down on its very first progress step -- every previous run had a terminal, so the gauge always attached and this had never fired.

The button itself now launches update.sh via a new bin/eas-station-run-update wrapper, as its own transient systemd unit (systemd-run --unit=eas-station-update --collect --no-block) rather than as a direct child of eas-station-web.service. That statement is load- bearing, not cosmetic: update.sh's own "Restarting Services" step restarts eas-station-web.service, and a direct child sits in that service's cgroup -- it would be killed by its own restart before ever reaching its final summary, the exact failure mode the previous one-click upgrade fix (2.196.3) still had. A sibling unit keeps running straight through that restart.

New GET /admin/operations/upgrade/progress (get_upgrade_progress in webapp/admin/maintenance/routes_operations.py) reads that unit's state back out for the Admin -> Operations page to poll, and is deliberately not backed by the existing in-memory _OPERATION_STATE dict, which resets the moment this very worker restarts partway through the upgrade it would be reporting on. It is also deliberately journal-first rather than systemctl show-first: manual testing against a real --collect unit showed it gets garbage-collected within a couple of seconds of exiting, success or failure alike, so systemctl show reliably answers "is it running right now" but not "how did it end" -- by the time anything polls, the unit routinely already looks exactly like one that never ran. The journal does not get cleaned up, so result detection reads update.sh's own === UPDATE RESULT: ... === marker (added to update.sh, printed unconditionally to the log right before the existing whiptail summary/error dialogs), falling back to systemd's own "Failed with result" / "Main process exited" / "Deactivated successfully" lines for the same unit if the script crashed before ever reaching its own summary.

The Operations page now shows a step counter ("Step 7 of 12: ..."), a progress bar, and a live color-coded log feed reading that endpoint every 3 seconds -- through the expected mid-upgrade disconnect and reconnect, not just a final dump after the fact. tools/inplace_upgrade.py, the Python reimplementation this replaces, is removed.

[2.197.0] - 2026-08-28 - Auto-configure a USB sound card as an audio source and output

Every Raspberry Pi board's onboard audio (the vc4hdmi* ALSA cards) is output-only -- there is no line-in or mic on the board, so it can never be an ingest source. A plugged-in USB DAC (e.g. an HS100B) previously required a manual trip through Admin -> Audio Ingest to become a source, and Admin -> EAS Settings to become the local alert-playback output, even though it was the only sensible device available.

Added

• app.py now runs _auto_configure_usb_audio_device() once per process start: when exactly one non-onboard ALSA card is present, it creates an enabled alsa-type AudioSourceConfigDB row (unless one already exists) and points EASSettings.audio_player at the same device (unless it has already been customized away from its aplay default). Zero or more than one external card is left alone as ambiguous. Documented in help.html.

[2.196.3] - 2026-08-28 - Fix the one-click upgrade button, remove unsupported Docker code paths, and fix Admin Operations navigation

Fixed

• The one-click "System Upgrade" button (Admin -> Operations) ran tools/inplace_upgrade.py, which only knew how to upgrade a Docker Compose deployment (docker compose pull/up/exec/restart) -- but EAS Station ships exclusively as a bare-metal systemd install (install.sh), and no docker-compose.yml exists in the repository. Every click of the button failed outright with "Neither 'docker compose' nor 'docker-compose' is available in PATH." The script now performs the actual bare-metal upgrade: git pull --ff-only, pip install --upgrade against this venv's requirements.txt, alembic upgrade head, then sudo systemctl restart eas-station.target (the same sudoers-granted command the Settings -> Environment "Restart All" button already uses). The now-meaningless "Compose File" field was removed from the Operations page and its route.
• tools/restore_backup.py had a live bug: any bare-metal deployment pointing at a non-localhost PostgreSQL host (a perfectly normal remote-database setup) was misrouted into running docker compose exec alerts-db psql ... against a container that was never going to exist, instead of connecting directly.
• The Admin Operations page (/admin/operations -- one-click backup, database optimization, alert-boundary recalculation, and the System Upgrade button fixed above) was filed in the navigation under Reports -> Analytics and labeled "Operations Report," which reads as a passive report rather than the maintenance/action page it actually is -- effectively making it undiscoverable. Moved it to Settings -> Data & Storage, next to Backups, and relabeled it "Admin Operations." Its route was also missing the system.configure permission check every sibling /admin/* route has (the page rendered for any logged-in user, though the backup/upgrade POST endpoints were already permission-gated) -- added. The page had no help.html documentation at all; added an entry.

Removed

• While auditing for other Docker assumptions: tools/create_backup.py carried a Docker/Podman volume backup path that was never actually invoked (backup_summary["volumes"] was always empty) -- removed. tools/restore_backup.py's "restore Docker volumes" step looked for volume-app-config.tar.gz / volume-certbot-conf.tar.gz archives that create_backup.py never produced -- also removed, and tools/validate_restore.py's post-restore guidance (which told bare-metal operators to run docker compose logs / docker compose restart) now prints the systemd/psql equivalents. tests/test_backup_restore.py::test_standby_config_exists no longer requires a docker-compose.standby.yml that was never shipped -- the standby doc it guards (examples/STANDBY_DEPLOYMENT.md) is already fully rsync/systemd based. Stale Docker mentions were also cleaned out of install.sh (dead rsync excludes for files that don't exist) and the installation/architecture docs; scripts/setup_postal.sh is untouched since the optional Postal mail server integration genuinely ships its own Docker-based install upstream.
• Removing those dead --no-volumes/--skip-volumes flags from create_backup.py/restore_backup.py broke three more call sites that still passed them: webapp/routes_backups.py (the full Backups page's create/restore actions), app_core/backup_scheduler.py (the in-process auto-backup scheduler, which defaulted include_volumes to False -- meaning every scheduled backup would have started failing), and the standalone tools/backup_scheduler.py cron/systemd-timer script. All three, plus the now-dead "Docker volumes" checkboxes in templates/admin/backups.html, are cleaned up to match.

[2.196.2] - 2026-08-28 - Fix radar mismatch between the alert page and exported cards; widen the Py-ART attribution card

The exported/shared alert card and the alert page's "Radar Loop" both rendered radar via _render_map(), which preferred a real per-site NEXRAD Level II decode (radar_level2.py, Py-ART) over the Level III WMS mosaic whenever a site was in range. The alert page's live "Radar (at time of alert)" toggle, however, is a static/js/core/map_theme.js Leaflet WMS tile layer that only ever shows the Level III mosaic. That meant the same alert could show two visually different radar products side by side on the same page (different resolution and color ramp -- IEM's own mosaic colors vs. cmweather's 15-band NWSRef), which read as a bug even though both were technically valid radar for the same storm.

Fixed

• _render_map() (app_utils/image_export/maps.py) now always uses the Level III WMS mosaic (_fetch_radar_overlay) -- the same request the live toggle makes -- so the live map, the Radar Loop, and every exported share card agree pixel-for-pixel. radar_level2.py's Level II decode/plot path (render_frame) is no longer called but is left in place (its REFLECTIVITY_LEGEND still backs the on-image legend) for a future pass that gives the live map a matching high-resolution option instead of silently diverging from it. Updated stale comments/docstrings in radar_loop.py that described the old Level-II-first behavior, and the Attribution page's Py-ART card to note the library is retained but not currently in the live overlay path.

Changed

• Widened the Attribution page's Py-ART/boto3/Cartopy/cmweather/Matplotlib card (.stack-item-wide in static/css/styles.css) to span two grid columns -- it credits five libraries with the longest description and license list on the page, and was visibly cramped at the same width as single-library cards.

[2.196.1] - 2026-08-27 - Fix Level II projection mismatch and coarse color banding

2.196.0's Level II renderer (pyart.map.grid_from_radars) turned out to smooth real gate detail into blurry blobs -- confirmed by requesting the same bbox at increasing pixel counts and finding no new detail appeared, the same test that had earlier confirmed the Level III mosaic's blockiness was its native resolution, not under-sampling. Replaced with RadarMapDisplay.plot_ppi_map, which draws each gate as its true azimuth/range quadrilateral via matplotlib pcolormesh -- no interpolation.

That swap surfaced two more real bugs, found by drawing the actual stored alert polygon directly onto a live radar image and checking pixel data rather than eyeballing screenshots:

Fixed

• Projection mismatch: plot_ppi_map was rendering in plain lat/lon (ccrs.PlateCarree), while the basemap tiles and hazard polygon it composites onto are Web Mercator -- different north-south scale at non-equatorial latitudes, misaligning the overlay. Switched to ccrs.epsg(3857) to match.
• Coarse color banding: the hand-rolled 6-color ramp was flattening real storm structure into solid blocks ("paint by numbers"). Replaced with cmweather's NWSRef colormap -- the ~15-band standard NWS reflectivity scale most radar apps use -- confirmed side-by-side against IEM's own archived radar image for the same storm at the same timestamp: same core position, same hook-like notch, same secondary cell.

Also added a light alpha-premultiplied Gaussian blur (_soften_beam_edges) at typical alert-polygon zoom, where a real beam-to-beam gap (confirmed genuine, not a rasterization artifact, via the same 4x-pixel-density test) can span 15-20 pixels and read as a hard picket-fence pattern -- softens the seams without blurring away real gradient structure the way the discarded grid_from_radars approach did.

New explicit dependencies (previously only transitive through arm_pyart, now imported directly): cartopy, cmweather, matplotlib. Attribution updated in about.html, attribution.html, and dependency_attribution.md.

[2.196.0] - 2026-08-27 - NEXRAD Level II radar for the weather-alert overlay

The radar overlay shipped in 2.195.0 used only NEXRAD Level III (IEM's WMS-T national mosaic, ~1km resolution) -- visibly blocky at close zoom, confirmed by requesting the same bbox at 256px through 2048px and finding the "chunk" size identical throughout: that's the mosaic's native resolution, not an under-sampling bug. app_utils/image_export/radar_level2.py (new) now tries raw Level II first -- the actual per-site volume scan (~250m gate resolution near the radar, the fine detail public radar apps show), falling back to the Level III mosaic wherever Level II isn't available (alert far outside every site's ~230km nominal range, no volume within tolerance of the requested time, a download/decode error). Both composite through the same _render_map pipeline and share one REFLECTIVITY_LEGEND color ramp, so a viewer can't tell which source produced a given frame from color alone.

Changed

• Site selection: nearest WSR-88D site by haversine distance, from the live NWS radar-stations API (cached 6h), capped at nominal base-reflectivity range -- returns nothing for a genuine coverage gap rather than guessing.
• Data source: NOAA's public Level II archive on AWS Open Data (unidata-nexrad-level2, no credentials needed), decoded with Py-ART. Verified live for both a fresh alert and one 8+ months old -- the archive goes back that far for the sites tested.
• Cost, accepted deliberately: arm_pyart pulls in a heavy transitive chain (cartopy, xarray, dask, pandas, matplotlib, netCDF4, h5py, shapely on top of the numpy/scipy/pyproj already required) -- a real departure from this project's normal minimal-dependency posture, with no lighter-weight NEXRAD Archive II decoder available in the Python ecosystem. A Level II frame costs ~15-30s (S3 download + decode + grid) versus one lightweight WMS GetMap call, so RADAR_LOOP_MAX_RENDER_PER_CALL dropped from 6 to 2 to keep a single request comfortably under Gunicorn's 300s worker timeout.
• Attribution added throughout (about.html, attribution.html, dependency_attribution.md, help.html): the AWS Open Data archive, the NWS radar-stations API, and Py-ART's own requested citation (Helmus & Collis, JORS 2016, doi:10.5334/jors.119).

[2.195.2] - 2026-08-27 - Radar overlay screenshot for the README tour

Added

• AGENTS.md's Documentation Requirements also call for a screenshot showing how to access a new feature, which 2.195.1 didn't add. Captured a real Alert Coverage Map with the radar toggle on (Tornado Warning over northwest Ohio, verified live against the deployed instance) and added it to the README's Screenshot Tour as docs/screenshots/radar-overlay.jpg.

[2.195.1] - 2026-08-27 - Document the radar overlay feature

Added

• 2.195.0 shipped the radar reflectivity toggle and Radar Loop card without touching templates/help.html or templates/about.html, missing the "Documentation Updates Required" step in docs/development/AGENTS.md. Added a description of both to the Help page's "Monitoring Live Alerts" section, and credited Iowa Environmental Mesonet (the data source) in about.html's acknowledgments, attribution.html's Data Sources table, and docs/reference/dependency_attribution.md.

[2.195.0] - 2026-08-27 - Radar reflectivity overlay for weather alerts

Weather (category='Met') alert pages and the dashboard map now have a "Radar (at time of alert)" toggle, pulling NEXRAD Level III base reflectivity (product N0Q) from Iowa Environmental Mesonet's public WMS-T mosaic — historical for past alerts, live for active ones. The alert detail page also gets a "Radar Loop" card: a lazily-generated, disk-cached sequence of frames spanning the alert's duration (5-minute cadence, capped at 36 frames / ~3 hours), so a completed alert can be replayed rather than only showing one frozen moment. The share-image renderer composites the same overlay onto weather alert graphics.

Both the client-side toggle (static/js/core/map_theme.js) and the server-side share/loop renderer (app_utils/image_export/maps.py) shared one bug: _RADAR_OPACITY at 0.65 blended mathematically correctly (traced pixel-by-pixel to confirm) but still read as a solid weather-radar image over a wide, intense cell, with the toned basemap and road detail essentially invisible underneath. Lowered to 0.45 in both places so the basemap stays legible.

Changed

• app_utils/image_export/radar_loop.py (new) — lazy, disk-cached frame generation; webapp/admin/api/routes_radar_loop.py (new) — GET /api/alerts/<id>/radar-loop.
• static/js/core/map_theme.js — new easRadar pane and EASMap.radarLayer().
• app_utils/image_export/maps.py — radar overlay + legend on the share-card renderer.
• templates/alert_detail.html, templates/index.html — toggle, legend, and (alert detail only) the Radar Loop player.

[2.194.1] - 2026-08-27 - Five independent "is this alert active" checks never excluded Cancelled

Following up on 2.194.0's CAP <references> fix: an alert correctly marked status='Cancelled' was still displayed as active on the dashboard's navbar stack light and the physical tower light. The 2.194.0 fix was necessary but not sufficient — it made the data correct, but at least five call sites across the codebase independently reimplemented "is this alert active" with their own ad-hoc expires > now check that predated app_core.alerts.get_active_alerts_query() (the canonical, already well-tested definition — see tests/test_alert_active_expired_partition.py) and never adopted its status.notin_(("Expired", "Cancelled")) / superseded_by_id.is_(None) exclusions:

Changed

• services/gpio/__main__.py::_make_active_alert_counter — drives the physical USB tower light.
• webapp/routes_monitoring.py::api_broadcast_state (/api/broadcast/state) — the page-load fallback for the navbar stack light.
• app_core/websocket_push.py::_emit_broadcast_state_update — the continuously-running WebSocket push that is the primary channel driving the stack light in an already-connected browser, and the dominant reason the light stayed lit even after the REST route was checked.
• app_core/websocket_push.py::_emit_alerts_update — feeds the active-alerts page and dashboard widgets.
• scripts/screen_manager.py::_has_active_alerts — the OLED/LED display's alert screen.

All five now delegate to get_active_alerts_query() instead of reimplementing it. Also fixed the public /alerts page and its PDF export (webapp/public/alerts_page/query.py, pdf_export.py), which excluded status == "Expired" but never "Cancelled" from the default view.

Manually corrected the one alert this actually happened to in production (confirmed cancelled via an independent PBS WARN report) since the CAP <references> Cancel message it depended on had already scrolled out of FEMA's IPAWS feed before today's fix existed — code changes alone can't retroactively catch a message that's no longer there to catch. Added tests/test_public_alerts_page_hides_cancelled.py.

[2.194.0] - 2026-08-27 - CAP <references>-based Cancel/Update linking, and a post-EAS silence gap

Investigated a report that a "Local Area Emergency" alert (OHDOT) shown as cancelled on PBS WARN was still displaying active on the dashboard. Root cause: per CAP 1.2 sec 3.3.2.3, a Cancel or Update message gets its own unique identifier and points back at the alert(s) it affects via <references> rather than reusing the original's — our poller never extracted or acted on <references> at all. Worse, a Cancel commonly carries no <info> block whatsoever (there's nothing left to describe), so it parsed as event="Unknown" with empty area codes and was silently dropped by the geographic-relevance filter before ever being looked at as a cancellation. Confirmed live: OHDOT's Cancel message reached our poller within minutes of being issued and was logged as "not specific enough" — the original alert then stayed "active" indefinitely with nothing to ever mark it otherwise.

Fixed

• Added poller.cap_poller.parse_cap_reference_identifiers() plus two new code paths: _process_cap_references_cancellation() intercepts a Cancel carrying <references> before the relevance filter (there's nothing else worth saving from it) and marks the referenced alert(s) Cancelled. _mark_cap_references_superseded() handles the CAP Update case — an Update does carry real content and still gets saved as its own alert normally, but previously nothing ever linked it back to the alert it updates unless that alert carried NWS VTEC identity (which a state DOT's IPAWS feed never does); now the referenced original is marked superseded_by_id, the same mechanism the VTEC chain already uses, so a stale original and its Update don't both show up as separate active alerts. Checked the CAP msgType enum for other exposure: Ack/Error are network-handshake types public feeds don't emit in practice, so they weren't specifically handled. Added tests/test_cap_references_cancellation.py and tests/test_cap_update_supersede.py.
• inject_eas_audio() released the air-chain gate the instant the last EAS sample was queued, so listeners heard the EOM tone cut directly into music/talk with zero break. POST_EAS_SILENCE_SECONDS (1.0s) is now queued as trailing silence before the gate clears, matching how a real station hands the air chain back to regular programming. Added tests/test_eas_stream_injector_trailing_silence.py.
• One more layer on the 2.193.10/2.193.11 ad-metadata work: even with those fixes, "resolve and play" on an ad in Song History almost always still failed — every VAST cache URL checked more than ~20 minutes after being logged already 404s. iHeartRadio's ad server (Triton) discards these per-impression cache entries within minutes; an operator browsing history later and clicking "resolve" is nearly always too late, and no amount of client-side fixing can resolve a link the ad network has already deleted. Moved the VAST-fetch/parse logic out of webapp/audio_archive/metadata.py into app_core/audio/vast_resolve.py (a Flask-free leaf module) so _handle_icy_metadata() (app_core/audio/sources.py) can resolve an ad tag immediately, on its own dedicated metadata thread, the moment the StreamTitle arrives — while the tag is still fresh — and store the underlying creative's durable CDN URL instead of the ephemeral VAST wrapper. That CDN file is a stable, reused asset, not a per-impression token, so "resolve and play" keeps working long after the original tag would have expired. webapp/audio_archive/metadata.py now just re-exports resolve_stream_url for the existing manual "resolve" API route. Added regression cases to tests/test_stream_metadata_parsing.py.

[2.193.11] - 2026-08-27 - Fix "Ad URL" title text not actually being clickable

Fixed

• Following up on 2.193.10's VAST namespace fix: after that fix landed, resolving an ad still appeared to do nothing when clicked, and the eas-station-web access log showed zero requests ever reaching /api/audio/archives/resolve-stream-url. Root cause was a second, independent bug: the Audio Archives Song History page renders "Ad URL" as the row's title (styled like a link, with an ad icon, right where a song title normally goes -- the obvious thing to click), but it was a plain <span> with no click handler. The actual working "resolve and play" button was a separate, tiny icon-only button off in the row's far-right action column, easy to miss and not visually connected to the "Ad URL" text at all. The title cell's "Ad URL" is now itself the clickable trigger.

[2.193.10] - 2026-08-27 - Fix VAST ad resolution never finding a playable MediaFile

Fixed

• Following up on 2.193.9's fix for iHeart ad metadata display: the Audio Archives Song History page's "resolve and play" button on an ad entry always reported "No playable audio found," even for VAST ad tags that contained a perfectly good audio/mpeg MediaFile. Root cause: resolve_stream_url() (webapp/audio_archive/metadata.py) searched for root.iter("MediaFile"), but real-world VAST responses (VAST 3.0+, which is standard — confirmed against live iHeartRadio/Triton ad-server responses) declare a default XML namespace on the <VAST> root element, so ElementTree parses every descendant's tag as {http://www.iab.com/VAST}MediaFile — the bare-string search silently matched nothing, regardless of whether the ad actually had playable audio. Fixed with a namespace-agnostic element search. Also now extracts AdTitle, AdSystem, and Duration from the VAST payload when present, and the player bar shows them instead of the generic "Ad URL" placeholder when available. Added tests/test_audio_archive_vast_resolve.py.

[2.193.9] - 2026-08-27 - Fix OLED init file-descriptor leak and iHeart ad-break metadata display

Investigated a report that eas-station-displays was consuming 8.9 GB RSS after ~4 days of uptime (every other service stayed under 1 GB over the same window).

Fixed

• Root cause: ArgonOLEDController.__init__ (app_core/oled.py) opens the I2C bus via smbus2.SMBus (a raw file descriptor with no __del__) before the ssd1306 handshake; on a host with no OLED physically attached, that handshake always fails, and the just-opened handle was never closed on the exception path. initialise_oled_display() retries this every 5 seconds indefinitely, so over ~4 days it leaked roughly 30,000 /dev/i2c-1 file descriptors (confirmed via /proc/<pid>/fd), which is what actually drove the RSS growth despite the existing glibc malloc-arena tuning. Fixed by closing the I2C handle before re-raising. Added tests/test_oled_init_fd_leak.py.
• iHeartRadio ad breaks send StreamTitle=adContext="<base64 VAST url>", which didn't match any of the known text=/title=/song=/artist= attribute patterns and wasn't recognized as a decodable base64 blob either (it's wrapped in an attribute, not a bare blob), so the raw, undecoded string was stored and displayed verbatim in the Audio Archives Song History page. _handle_icy_metadata (app_core/audio/sources.py) now scans quoted attributes for a base64 value that decodes to an http(s) URL and resolves it into the existing stream_url field, which the Song History UI already renders as a clickable "Ad URL" badge. Added a regression test to tests/test_stream_metadata_parsing.py.

[2.193.8] - 2026-08-26 - Documentation renders and links cleanly, and CI now enforces it

Auditing the published docs site (kr8mer.github.io/eas-station) for a reported footer whitespace issue turned up two real problems and a sizeable batch of broken links, none of which were previously caught by anything:

• GitHub Pages was misconfigured: repo settings still pointed at the legacy branch-build Pages source, which failed on every single push, running alongside (and redundant with) the custom mkdocs Actions workflow that actually publishes the site. Switched Pages' source to "GitHub Actions" via the API so the failing legacy builder stops firing.
• A stray, unpaired closing ``` fence in docs/development/AGENTS.md (line 973, no matching open) silently turned the ~1,100 lines after it into a mis-rendered code block on the live site — found by the new link checker below, since it corrupted markdown-fence-stripping enough that unrelated Python regex source lines started looking like broken links.
• 61 broken links (mkdocs build --strict WARNING-level) across 13 files: relative links from docs/*.md pages into source files (app_utils/eas.py, README.md, requirements.txt, etc.) that resolve fine in a raw repo checkout but 404 on the published site, since GitHub Pages only serves the docs/ tree. Converted to https://github.com/KR8MER/eas-station/blob/main/... URLs (matching the pattern already used elsewhere in these same docs), or tree/main/ for directory references.
• 13 broken in-page/cross-page anchors (INFO-level, not strict-mode-blocking but still dead links in practice) — mostly stale TOC entries that didn't track a heading rename, and GitHub/mkdocs's actual em-dash/&/emoji slugification (single hyphen, not double) being different from what was originally typed. Verified the fixed anchors resolve correctly by checking generated heading ids in the built HTML and click-testing several in a live browser.
• The "repeated footer" screenshot that prompted this investigation turned out to be a capture glitch from a flaky remote-browser connection, not a real bug — confirmed via direct DOM query (exactly one footer element, correctly positioned) rather than trusting the screenshot.

Added

• tests/test_docs_link_integrity.py: 4 tests enforcing no broken local links in docs/*.md and perfect mkdocs.yml nav coverage in both directions (every doc file reachable from nav; no nav entry points at a missing file). Fenced code blocks are stripped before link-scanning so example markdown syntax isn't mistaken for a real reference.
• .github/workflows/docs-pages.yml now runs mkdocs build --strict instead of a plain build, so a future broken link or bad anchor fails the deploy loudly instead of silently shipping.

Verified

• Live in Chromium: light and dark theme contrast/readability across prose, syntax-highlighted code blocks, and callouts — no issues found.
• mkdocs build --strict exits 0 (was 61 warnings + 13 info-level anchor mismatches).

[2.193.7] - 2026-08-26 - Remove the dormant direct-hardware SDRSourceAdapter

Pre-v3 punch-list nice-to-have (#6): app_core/audio/sources.py's SDRSourceAdapter (~430 lines) was the pre-separated-architecture way to read SDR audio directly via a local RadioManager, inline demodulation included -- the same "demod running inline in the audio-service process" smell that 2.193.0's services/demod split fixed for the adapter that's actually used. Confirmed genuinely unreachable: every SDR-backed AudioSourceConfigDB row is created exclusively by webapp/admin/audio_ingest/radio_sources.py, which always sets managed_by='radio'; eas_monitoring_service.py's loader short-circuits on that flag and constructs a RedisSDRSourceAdapter directly, never reaching the factory function that would have instantiated this class. The web UI's manual "Add Audio Source" form already excludes sdr as a selectable type for the same reason (its template has a comment: "SDR sources should be added via Settings > Radio page").

Removed

• SDRSourceAdapter and its exclusive RADIO_AVAILABLE/RadioManager import (app_core/audio/sources.py, 1855 → 1428 lines). The create_audio_source() factory's 'sdr' branch now raises a clear RuntimeError explaining SDR sources are managed via RedisSDRSourceAdapter instead of silently constructing dead code. RBDS_PROGRAM_TYPES (still imported by the live redis_sdr_adapter.py) is untouched.
• 2026-08-26 - Capacity and sizing documentation

Pre-v3 punch-list item: document how much CPU/RAM/thermal headroom a given SDR-receiver/EAS-monitor configuration actually needs, using a real labeled snapshot from a production reference deployment rather than a synthetic benchmark.

Added

• docs/hardware/CAPACITY_AND_SIZING.md: reference-deployment snapshot (Pi 5, load average, memory/swap, thermal, disk) plus a sizing table by configuration (web-only, +1 SDR receiver, +stream monitors, 2+ receivers). Linked from docs/README.md and docs/guides/HARDWARE_QUICKSTART.md's Requirements section, which previously covered only peripheral (GPIO/I2C) compatibility, not CPU/RAM planning.

[2.193.5] - 2026-08-26 - Orphaned SDR audio sources no longer thrash forever

Investigating the punch-list item on eas-unified-WNCI/ERN-LUC "Underrun" log noise turned up a live, unrelated bug: a deleted radio receiver's audio source (sdr-wxmon, from a receiver identifier no longer in the database) had been quarantine-retrying every ~30s-4min for hours in eas-station-audio.service, respawning its FFmpeg process and Icecast connection each cycle. Deleting a receiver sends one best-effort source_delete Redis command to tear the source down; if that single notification is ever missed (process briefly unreachable, a dropped pub/sub message) there was no other mechanism to remove it -- the adapter stayed registered in AudioIngestController and its stall supervisor just kept retrying a source that can never succeed (nothing publishes audio for a receiver id the database no longer has).

Added

• AudioCommandSubscriber.reconcile_orphaned_radio_sources() (app_core/audio/redis_commands.py): removes any RedisSDRSourceAdapter with no matching AudioSourceConfigDB(source_type='sdr', managed_by='radio') row, via the same EAS-monitor/Icecast/controller teardown the source_delete command already used (extracted into a shared _remove_source_everywhere() helper). Runs every 5 minutes from eas_monitoring_service.py's main loop as a self-healing safety net -- the one-shot Redis notification stays the primary removal path.

Investigated

• The WNCI/ERN-LUC "Underrun" warnings themselves are confirmed benign: the SAME-decoder tap's 0.1s read timeout is tight enough to time out waiting for the last chunk of a read even when its buffer is 1400+/1600 samples full, and AudioArchiver wrote continuous, gap-free files for both sources throughout the investigation window. Documented in a code comment at the warning site (app_core/audio/broadcast_adapter.py) so this isn't re-investigated from scratch later. A source that's actually starved logs buffer=0 on every read, as sdr-wxmon did above.

[2.193.4] - 2026-08-26 - Structural audit test: every airchain trigger must reach Icecast

Twice in the 2.193.x investigation, a broadcast-trigger function was found that keyed the GPIO relay and played audio locally but never pushed audio into the live Icecast stream queues at all (Manual Send in 2.193.1, and implicitly the same class of bug the 2.193.3 Audio Archive fix guards against). Both were only caught because a user noticed silence and asked pointed questions -- nothing in the test suite would have caught a future regression of the same shape.

Added

• tests/test_broadcast_reaches_icecast_audit.py: an AST-based structural regression test that walks every function under app_core, app_utils, webapp, and scripts calling set_broadcast_active() and asserts the same function also calls inject_eas_audio() or inject_raw_eas_audio(). app_core/rwt_scheduler.py::trigger_rwt_broadcast is exempted via a documented KNOWN_DELEGATING_TRIGGERS allowlist -- it sets the marker synchronously for instant overlay display, then delegates actual playout/injection to _drive_rwt_airchain() on a background thread, which the audit scans (and passes) on its own. A second test pins a minimum count of known broadcast-trigger functions so the audit can't silently go blind if set_broadcast_active() is renamed or moved.

[2.193.3] - 2026-08-26 - Resends now show up in Audio Archive

Audio Archive (/audio, "Browse and manage EAS broadcast recordings") lists one row per EASMessage. A resend replayed the original row in place instead of inserting a new one, so a retransmission was invisible there -- only the original generation event ever showed up, even though the resend keyed GPIO and injected real audio into the live air-chain (see 2.193.1). For a compliance log, a retransmission is itself a loggable event.

Added

• scripts/resend_eas_broadcast.py: each resend now clones the source EASMessage row (audio blobs, same_header, CAP-alert link, etc.) into a new row with a fresh created_at, tagged metadata_payload: {resend: true, resend_of_message_id, resent_by}. It sorts, filters, and downloads exactly like an original send on the Audio Archive page. Written unconditionally -- even a failed resend attempt -- matching the SystemLog/audit-ledger entries already written there for the same reason: a real event shouldn't silently vanish. Storage is duplicated per resend (there's no blob-dedup mechanism on this model); acceptable for a rare, human/scheduled-triggered action.

[2.193.1] - 2026-08-26 - Manual Send and RWT never reached the Icecast air-chain

A user manually sent a Required Weekly Test and heard nothing on any Icecast stream. Auditing every broadcast-trigger code path (every call site of set_broadcast_active()/play_broadcast_audio()) found that Manual Send and RWT (both the automated weekly test and the operator "Run Test Now" button/GPIO trigger) had never injected audio into Icecast at all -- only Resend and live auto-forward/OTA-relay did. Both paths keyed GPIO and played audio locally via audio_player_cmd (e.g. aplay), but nothing pushed the composite WAV into the live stream queues. The weekly compliance test had been airing nowhere a stream listener could hear it.

Also: a separate, smaller gap in the same audit -- only the live auto-forward path overrode each Icecast stream's "now playing" title with the alert text during a broadcast (app_core.audio.alert_metadata, an in-process-only singleton); every other path silently no-op'd if it tried the same call, since it doesn't share a process with the audio service's live IcecastStreamer objects.

Fixed

• app_core/audio/redis_commands.py: added inject_raw_eas_audio, a sibling of the existing inject_eas_audio (resend's mechanism) for callers that have composite WAV bytes in hand but no EASMessage row to reference by id -- Manual Send and RWT persist a ManualEASActivation, not an EASMessage. Same base64-in-JSON pattern abort_injected_audio already uses for its EOM burst.
• webapp/eas/workflow.py (Manual Send) and app_core/rwt_scheduler.py (_drive_rwt_airchain, shared by the automated weekly RWT, the "Run Test Now" button, and the GPIO RWT trigger) now call inject_raw_eas_audio right after keying the broadcast marker, mirroring exactly how scripts/resend_eas_broadcast.py already does it. Best-effort and non-fatal, same as every other injection call site -- a Redis/audio-service hiccup can never block relay keying or local playback.
• eas_monitoring_service.py: added _reconcile_broadcast_metadata(), polled from the existing ~4 Hz main loop, which mirrors the Redis broadcast-state marker's label onto every Icecast stream's title. Since every broadcast path already writes that marker (to key the GPIO relay and drive the countdown overlay), this covers Manual Send, RWT, Resend, and auto-forward automatically -- no per-caller wiring, unlike the old direct-call approach that only worked for auto-forward. The two direct calls in app_core/audio/auto_forward.py were removed as redundant.
• app_utils/eas.py (EASBroadcaster.handle_alert) and scripts/resend_eas_broadcast.py: both now prefer the original alert's headline over the generic event-type name for the broadcast label when one is on file, matching what auto-forward's now-removed direct call already did -- more specific text ("Severe Thunderstorm Warning issued until 2:15 PM by NWS Cleveland" vs. just "Severe Thunderstorm Warning") for both the countdown overlay and the new stream-metadata override.

[2.193.0] - 2026-08-26 - Split FM/AM demodulation into its own service

A user watched a resend's EAS audio get injected into all three configured Icecast mounts ("EAS stream injector: pushed 73.9s of EAS audio ... to source 'X' broadcast queue" -- confirmed in the logs) yet reported never having heard a clean one air. journalctl showed why: all three mounts hit "Icecast buffer running low" / "buffer completely empty -- Audio source starved" every 10-30 seconds, continuously, all day -- not just during EAS injections, interrupting normal programming too.

Profiling the live eas-station-audio.service with py-spy record --gil (non-blocking, 15s @ 30Hz) found the cause: the redis-sdr-* thread that receives IQ samples from eas-station-sdr.service dominated GIL-held samples. On every incoming IQ chunk it ran several scipy.signal.oaconvolve FFT convolutions inline (stereo pilot detection, RBDS extraction -- app_core/radio/demod/fm.py), starving the audio service's three real-time Icecast feeder threads sharing the same interpreter, each of which needs to wake roughly every 50ms to keep its buffer fed.

Added

• services/demod/: a new eas-station-demod.service subprocess that owns FM/AM demodulation, mirroring the queue + dedicated-worker-thread pattern app_core/radio/demod/rbds_worker.py already used for RBDS decoding (a demodulator is stateful/order-dependent and must never be called from more than one thread or on out-of-order chunks). Subscribes to sdr:samples:<receiver_id>, publishes demodulated PCM audio to demod:audio:<receiver_id> and decoder status (stereo lock, RBDS PS/PI/radiotext) to demod:status:<receiver_id>. Listens on port 5106, matching the existing per-subsystem-service pattern (docs/architecture/SDR_SERVICE_ARCHITECTURE.md, docs/troubleshooting/FIREWALL_REQUIREMENTS.md).

Fixed

• app_core/audio/redis_sdr_adapter.py: RedisSDRSourceAdapter no longer runs a demodulator inline -- it's now a thin consumer of the demod service's demod:audio:* channel, so the DSP work that was starving the Icecast feeders can never again share a GIL with them. RBDS/stereo metadata (the ~300-line block _update_metrics() builds for the UI) is unchanged; it now reads a pickled DemodulatorStatus snapshot from demod:status:* instead of a local demodulator reference.
• eas_monitoring_service.py: a numpy.float32 -> psycopg2 type-adapter failure (can't adapt type 'numpy.float32') was silently breaking every audio_source_metrics write, once a second, since the writer was added -- the table had never actually been populated in production. peak_level_db/rms_level_db/sample_rate/etc. are now coerced to native Python types before reaching SQLAlchemy. Found during the same investigation; confirmed via py-spy that it was not the cause of the Icecast starvation (a separate, real bug worth fixing anyway).

[2.192.1] - 2026-08-24 - Fix stale version claims in tech-stack shields and docs

A user asked "why is Redis showing version 7?" -- the footer badge and README had been advertising versions that no longer matched what's actually running, because install.sh doesn't pin these packages and nobody had gone back to re-check the shields since they were first written.

Fixed

• Redis: badge/README claimed 7.1; the running server is 8.0.2 (redis-cli INFO server, redis-server --version). install.sh installs redis-server via apt with no version pin, so the shield now reads Redis 8 and no longer implies a pin that doesn't exist.
• PostGIS: badge/README/Mermaid diagrams/architecture docs claimed 3.4; the actual extension is 3.5.2 (SELECT postgis_full_version()). install.sh only targets the major series (postgresql-${PG_MAJOR}-postgis-3), not a minor version.
• Nginx: badge said "Alpine", implying a Docker nginx:alpine image. The real deployment is a bare-metal apt-get install nginx package (confirmed via nginx -v -> 1.26.3); there is no Dockerfile or docker-compose anywhere in the project that uses nginx:alpine. Badge and attribution table now say "system package" / "system (apt)" instead.
• Chart.js: badge/README claimed a single version, 3.9.1. Two vendored copies are actually in concurrent use -- static/vendor/chartjs/chart-v3.min.js (v3.9.1, used by 5 dashboard templates) and chart.min.js (v4.4.0, used by system_health.html and repo_stats.html) -- an unconsolidated dual-version situation, not simple staleness. Badge and README now disclose both versions and both files.

Updated surfaces: templates/partials/tech_stack_badges.html, README.md (top badge row, Mermaid diagram, infrastructure prose, attribution table), docs/architecture/SYSTEM_ARCHITECTURE.md, docs/architecture/THEORY_OF_OPERATION.md, and docs/assets/diagrams/system-deployment-hardware.svg.

[2.192.0] - 2026-08-24 - Close the API reference's docstring gap, add structured param docs

Follow-up to 2.191.0's live /api-reference page. It shipped reporting 89% documentation coverage (231 of 259 /api/* routes) and only a one-line summary per route, even for endpoints that take a JSON body or query params -- enough to say "this endpoint exists" but not enough for someone integrating against it to know what to send.

Added

• Structured Body:/Query:/Path:/Returns: docstring sections, parsed by app_utils.api_reference._parse_docstring_sections() and rendered as a labeled block under each route's narrative on the reference page. Args:/Params: are also recognized, since ~100 existing route docstrings already used one of those loosely before this parser existed -- those routes get the richer rendering for free. A section header must be exactly Body: (etc.) on its own line; trailing qualifier text on the header line (Body (optional):) is not recognized and falls back into plain narrative text -- documented in docs/development/AGENTS.md's Flask Route Pattern, along with the section-header convention itself, so new routes follow it going forward.

Fixed

• The 28 routes with no docstring at all now have one -- audio archive management (12 routes), the LED sign controller (9), SDR receiver CRUD (4), the alert self-test harness, the diagnostics validation suite, and the radio monitoring snapshot. Coverage is now 259/259 (100%), verified against the live app after every change (SKIP_DB_INIT=1 avoids needing a real database just to read app.url_map and docstrings -- no HTTP request is made, so before_request's database initialization never runs).

Testing

• New tests for _parse_docstring_sections() (narrative/section splitting, header-order normalization, empty-section handling) and for compute_api_reference() exposing narrative/sections per route.
• Full local test suite: 2655 passed, 0 failed.
• Re-verified against the live app's real app.url_map after fixing the two docstrings that used the Body (optional): trailing-text mistake the new AGENTS.md warning now calls out -- caught only because the narrative/sections split was checked against real route data instead of trusting the docstrings were written correctly.

[2.191.0] - 2026-08-24 - Add a live REST API reference page

There was no backend API reference at all: docs/README.md pointed at docs/frontend/JAVASCRIPT_API.md and labelled it "REST API reference", but that file documents frontend JS globals (EASApi, EASWebSocket, ...), not the ~260 /api/* Flask routes, and its own banner already says large parts of that content are aspirational. Separately, docs/guides/API_KEY_MANAGEMENT.md describes a planned-but-unimplemented key-based auth system for external API access.

Added

• /api-reference (and its JSON sibling /api/api-reference) -- a live reference of every /api/* route, computed from app.url_map on each request rather than a hand-maintained file that immediately goes stale. Mirrors the /repo-stats idiom exactly (a page the codebase already criticizes its own predecessor for being "a committed HTML blob regenerated by a script"). Groups routes by module, shows HTTP methods, the view function's docstring (already present on ~89% of /api/* handlers), and its access requirement. Verified against the real running app (real Postgres + PostGIS, a real logged-in session): 259 routes, 231 documented, 182 access-gated, correctly listing itself.
• eas_auth_requirement introspection marker on every permission decorator (require_auth, require_role, require_permission, require_any_permission, require_all_permissions in app_core/auth/decorators.py / app_core/auth/roles.py, and require_permission_or_setup_mode in webapp/admin/environment.py) -- stamped onto the wrapped view function (survives functools.wraps, which only copies __doc__/__name__) so the reference page can show a route's real access requirement without re-deriving it from source text via regex, the same "trust the live app" principle app_utils/repo_stats already established for route counting.
• New webapp/navigation/registry.py entry (Analytics section) and tests/test_api_reference.py (16 tests): route filtering, docstring/ summary extraction, auth-marker introspection for all six decorators, nav registration, and template wiring.

Fixed

• docs/README.md's developer table no longer labels the JavaScript-globals doc as the REST API reference; it now points at the real live page and flags API_KEY_MANAGEMENT.md as not-yet-implemented design reference.
• Corrected a changelog authoring mistake from the 2.189.x/2.190.x entries above: 2.190.1 had been inserted between 2.189.1 and 2.190.0 instead of above both, and the repo-stats entry's heading had been overwritten to "2.189.1" (duplicating the resend-logging entry's version number) while keeping its original repo-stats content. Reordered newest-first and restored the repo-stats entry's correct heading, 2.189.0.

[2.190.1] - 2026-08-24 - Upgrade dependencies for security

Bumped every pinned dependency in requirements.txt to its latest available version (37 packages: Flask/Werkzeug/Flask-SocketIO/Flask-WTF/Flask-Caching/ Flask-Compress, SQLAlchemy/Alembic/GeoAlchemy2/psycopg2-binary/greenlet, requests/certifi/feedparser, redis/hiredis, gunicorn, pytz, psutil, pytest/ pytest-asyncio, lxml/Pillow/scipy, and others) — most notably certifi (CA bundle freshness, required for IPAWS TLS) and pytz (timezone database freshness, called out in its own comment as "CRITICAL FOR PROPER TIME DISPLAY").

Changed

• redis 7.1.0 → 8.1.0 and gunicorn 23.0.0 → 26.1.0 (both major version jumps) were checked against their actual release notes before bumping, not just pinned blindly: redis-py 8.0's headline change (RESP3 becomes the default wire protocol) explicitly preserves legacy RESP2 response shapes for existing code, and gunicorn 26's breaking change (dropping the eventlet worker class) doesn't apply here since this project runs --worker-class gevent, which gunicorn 26 still supports. Verified redis-py 8.1.0 directly against a real local Redis server (get/set/delete, decode_responses, pub/sub — the exact patterns app_utils/eas.py and app_core/audio/redis_commands.py use) rather than relying on mocked tests alone.
• numpy stays at 2.3.5, not bumped to the latest 2.5.2: numba carries a pre-existing, intentional <0.64.0 cap (documented reason -- 0.64+ pulls a new llvmlite requiring a 55 MB download, expensive on the Raspberry Pi hardware this runs on), and every numba release under that cap requires numpy<2.4. pip install -r requirements.txt outright refuses to resolve numpy==2.5.2 against that constraint -- this isn't a judgment call, it's a real, reproducible dependency conflict. 2.3.5 is the newest numpy release compatible with the existing numba pin.
• Synced every version number this changed in README.md's tech-stack badges and dependency-attribution tables and templates/partials/tech_stack_badges.html's footer badges, including several rows tests/test_tech_stack_badges.py doesn't cover (redis-py, hiredis, Flask-SocketIO, Flask-WTF, Flask-Caching, GeoAlchemy2, gpiozero, greenlet, orjson, psutil, psycopg2-binary, pyproj, pyshp, pytest, pytest-asyncio, pytz, requests, ujson, luma.oled) -- found via a full cross-reference sweep of requirements.txt pins against every README table row mentioning each package, not just the ones the existing test happened to check.

Testing

• Full local test suite against a venv built from the fully upgraded requirements.txt: 2634 passed, 267 skipped, 61 xfailed, 6 xpassed, 0 failed -- run twice (once mid-upgrade, once after the badge/table fixes) to confirm no regression crept in between.
• redis upgrade verified against a real, running local Redis server, not only mocks (see above).
• gunicorn's gevent worker class confirmed importable under 26.1.0.

[2.190.0] - 2026-08-24 - Hold to Abort Broadcast now stops injected Icecast audio too

"Hold to Abort Broadcast" (the web button and the physical GPIO Dump/Abort input) only ever stopped a local playback subprocess by PID. A broadcast's audio is also pushed directly into the live Icecast air-chain (eas_stream_injector.inject_eas_audio(), queued into each source's BroadcastQueue up front) -- a completely separate pipeline the local kill never touched, and the one that station operators and stream listeners actually hear. On a station with no local audio player configured at all (Icecast-only, a real supported deployment mode called out in the codebase's own comments), there was never a PID to find, so abort_current_broadcast() returned immediately ("nothing is currently playing") and the alert played out to completion regardless of how long the button was held.

Added

• abort_injected_audio() (app_core/audio/eas_stream_injector.py) -- purges every active source's queued EAS audio from its BroadcastQueue subscribers (using the existing clear_subscriber_queue()), releases the injection gate, and can optionally inject a replacement WAV (the EOM burst) immediately after, so stream listeners hear a compliant sign-off instead of a hard cut to dead air.
• New abort_injected_audio Redis command (app_core/audio/redis_commands.py): AudioCommandPublisher.abort_injected_audio(eom_wav=None) (base64-encodes the EOM bytes for the JSON-only command channel) and the matching AudioCommandSubscriber dispatch handler, mirroring the existing inject_eas_audio command's shape.

Fixed

• abort_current_broadcast() no longer treats "no local PID" as "nothing to abort." It now checks the broadcast-active marker too, and always attempts the Icecast purge (with the EOM burst injected into the stream) regardless of whether a local subprocess was ever running. The audit ledger's EAS_CANCELLATION entry now also records injected_audio_cleared (chunk count) alongside eom_sent.
• POST /api/broadcast/abort no longer 409s when there's no trackable PID. That case used to mean "couldn't actually abort anything"; it now correctly means "aborted via the Icecast purge instead of a local kill," since abort_current_broadcast() handles it.
• Extended tests/test_gpio_dump_broadcast.py and tests/test_broadcast_phase_and_web_abort.py to cover the no-PID/ Icecast-only abort path and the new injected_audio_cleared audit field; new tests/test_eas_abort_injected_audio.py covers the purge itself (multi-subscriber clearing, gate release, EOM re-injection) and the new Redis command's publisher/dispatcher wiring.

Real-hardware verification still needed

This closes the gap identified when 2.189.1 shipped, but per this project's standing practice for broadcast-control changes (see the 2.184.0/2.184.1 Dump/Abort Broadcast history below, where mocked unit tests alone missed two real gaps), the on-air behavior -- does an Icecast listener actually stop hearing the alert, and does the injected EOM burst actually play -- should be confirmed against a real running audio-service and Icecast mount before this is fully trusted in production.

[2.189.1] - 2026-08-24 - Fix resend logging and its dead local-playback config

Fixed

• A resend ("Resend on Air") was never actually reaching the tamper-evident audit ledger. Every other broadcast trigger (manual Send, RWT, live/ auto-forwarded alerts) writes an EAS_BROADCAST audit-log entry either via the SQLAlchemy after_insert listener on EASMessage (new transmissions) or an explicit AuditLogger.log() call (webapp/eas/workflow.py). A resend replays an existing EASMessage row rather than inserting a new one, so neither mechanism ever fired for it — the only record was a plain SystemLog row written at the very end of scripts/resend_eas_broadcast.py, and even that vanished with zero trace if anything raised earlier in the script, since its stdout/stderr are redirected to DEVNULL by the Flask route that launches it. The script now writes an explicit EAS_BROADCAST audit entry (tagged resend: true) unconditionally, and the whole playout body is wrapped so a failure is always recorded (as an ERROR-level SystemLog row and a success: false audit entry) instead of disappearing silently.
• A resend's local audio playback — and therefore its PID, which is what makes a broadcast abortable — was dead code on every real deployment. scripts/resend_eas_broadcast.py resolved its player command from an AUDIO_PLAYER_CMD app.config/env key that nothing else in the codebase ever sets; every other broadcast path resolves audio_player_cmd through load_eas_config() (the EAS_AUDIO_PLAYER env var or the EASSettings.audio_player DB column). Because of this, a resend never played through a locally configured player and never published a PID via play_broadcast_audio() — so on a station with a local player configured, the "Hold to Abort Broadcast" button had nothing to find and kill for a resent alert. Now reads the same load_eas_config() value as manual Send, RWT, and live/forwarded alerts.
• New tests/test_eas_resend_logging.py pins both fixes: audio_player_cmd comes from load_eas_config(), a successful resend writes both the SystemLog row and the EAS_BROADCAST audit entry, and a resend that raises mid-playout still writes both, flagged as a failure, and returns a non-zero exit code instead of looking like a silent success.

[2.189.0] - 2026-08-24 - Fix undercounted stats and add git history to Repo Stats

Fixed

• Blueprints undercounted on /repo-stats. count_components() only scanned app_core/ and webapp/ for = Blueprint(, missing the four production blueprints defined in services/*/api.py (zigbee, network, gps, displays) -- the same modules whose routes the source-scan fallback already counted, so the Routes and Blueprints tiles disagreed with each other. The scan now covers services/ too (37 -> 41).
• Scripts undercounted on /repo-stats. The Scripts tile only counted files under scripts/, missing every root-level operator entry point: install.sh, update.sh, uninstall.sh, diagnose.sh, app.py, wsgi.py, eas_monitoring_service.py, sdr_hardware_service.py. These now count too (73 -> 81).

Added

• Git history tile on /repo-stats -- commit count, distinct contributor count, and the most recent commit date, read via git log in app_utils/repo_stats/scanner.py:git_history(). Hidden automatically when the checkout isn't a git repository (matches the existing git ls-files vs. filesystem-walk fallback) rather than rendering misleading zeroes.

[2.188.1] - 2026-08-23 - Redirect /admin to Settings once an admin exists

Changed

• /admin now redirects to /settings for authenticated visitors instead of rendering a stats-overview stub. The route itself can't be removed -- app.py's auth gate exempts this exact endpoint from the login-wall specifically so a fresh install with zero admin users can reach the create-first-admin form without being able to log in yet (the admin_setup_mode allow-list in the before_request hook) -- but once an admin exists there was nothing left on the page worth a dedicated view for: the same Active Alerts / Total Boundaries counts already appear on the homepage. The first-time setup wizard (setup_mode) is unaffected and still renders normally.
• Removed the "Admin Dashboard" entry from Settings -> Configuration (href="/admin") -- it had become a circular link from inside Settings back to Settings itself.

[2.188.0] - 2026-08-23 - Retire the Admin Dashboard's tabbed interface (phase 5, final)

This completes the admin.html//settings consolidation started in 2.186.2 (phases 1-4: #2455, #2456, #2458, #2459). templates/admin.html goes from ~2099 lines to 135 -- a stats overview plus the first-time setup wizard, nothing else.

Added

• New /admin/location-settings page -- station jurisdiction, timezone, map defaults, and monitored counties/zones (SAME/FIPS + NOAA zone picker). Extracted verbatim (same pill-tab structure, same element IDs) from the Admin Dashboard's System tab.
• New /admin/eas-encoder-settings page -- SAME/EAS encoder configuration: station identity, pre/post-alert tone signaling (bell/beep/three-tone/QC-II/DTMF/MDC1200), event filtering, and dedup windows. Extracted verbatim from the Admin Dashboard's Broadcast tab, which had no other content -- Broadcast is now gone from admin.html entirely.
• Both new pages share static/js/admin/location-settings.js unmodified -- it already self-initializes three independently-guarded features (initLocationSettings, initZoneSearch, initEasSettings) on DOMContentLoaded, each a no-op when its target elements are absent, so loading the same file on two different pages was safe without any per-page glue code.

Removed

• The Admin Dashboard's entire tabbed interface. With System and Broadcast (the last two tabs) extracted, nothing was left in the tab shell -- removed it along with the now-fully-dead inline EAS_*/BOUNDARY_TYPE_CONFIG/boundaryCache/etc. globals block, #confirmationModal (nothing on the page calls showConfirmation() anymore), #operationStatus (dead since the Operations tab was removed in phase 2), and a "Keyboard Shortcuts" hint for a feature (setupKeyboardShortcuts()) that was already never actually invoked anywhere in the codebase. Replaced with a short pointer card to /settings. core.js, boundary-management.js, alert-management.js, zone-catalog.js, and location-settings.js are no longer loaded on admin.html -- only utilities.js (for showStatus()) and user-management.js (the first-time setup form) remain.
• Simplified dashboard.py's /admin route to match -- dropped all the location/EAS/boundary-stats context computation (location_settings, eas_event_codes, eas_fips_states, eas_recent_messages, boundary_stats, etc.) that admin.html no longer reads. It now only computes the four stat-card counts and setup_mode. Removed the now-unused imports this left behind (Dict/List, EASMessage, get_location_settings, build_county_forecast_zone_map, PRIMARY_ORIGINATORS, SAME_HEADER_FIELD_DESCRIPTIONS, manual_default_same_codes, ORIGINATOR_DESCRIPTIONS, P_DIGIT_MEANINGS).
• Updated tests/test_admin_dashboard_fixes.py's test_dashboard_uses_current_app, which pinned to two specific current_app.config.get(...) calls that were part of the now-removed EAS-message-stats computation. The regression it actually guards against (an undefined bare app variable) is still covered by the current_app.logger.* assertions, which remain.

[2.187.4] - 2026-08-23 - Fix broken boundary upload on Data Management page

Fixed

• GeoJSON/Shapefile upload forms on /admin/data-management did a plain HTML submission instead of the AJAX upload flow. #2459 extracted this page from the Admin Dashboard's Data tab but missed a script include: despite its name, static/js/admin/alert-management.js -- not boundary-management.js -- owns the #uploadForm/#shapefileUploadForm submit handlers (historical, not a naming mistake worth fixing right now). Added the missing <script src>.

[2.187.3] - 2026-08-23 - Extract Data Management from Admin Dashboard (phase 4 of consolidation)

Added

• New /admin/data-management page (templates/admin/data_management.html, dashboard.py:data_management_page()) -- boundary polygon upload (GeoJSON/Shapefile, any type: electric, fire, school, custom, ...), the NOAA zone catalog (.dbf upload/reload/search), and bulk manage/delete, extracted verbatim from the Admin Dashboard's Data tab. Confirmed this does NOT duplicate the already-registered /admin/county_boundaries -- that page manages NOAA county/zone reference lookup data (FIPS/SAME code resolution) via entirely different backend endpoints (/admin/county_boundaries/*) from this page's general boundary-polygon management (/admin/upload_boundaries, /admin/clear_boundaries/*, etc.).
• Extracted structurally verbatim rather than redesigned: same pill-tab sub-navigation, same element IDs, same static/js/admin/boundary-management.js and zone-catalog.js files -- both already expect these exact IDs and Bootstrap shown.bs.tab events, so relocating the markup as-is was the lowest-risk option.

Removed

• Admin Dashboard's Data tab. System is now the dashboard's first (default-active) tab.

  This completes the multi-phase admin.html//settings consolidation (phases 1-4: #2455, #2456, #2458, this one). Remaining in admin.html: System tab's Location Settings/Alert Filtering and Broadcast's EAS Encoder Settings -- both share one 1,475-line interactive JS module (FIPS pickers, live zone search, EAS tone/DTMF UI) and are the last piece.

[2.187.2] - 2026-08-23 - Extract Alert Management from Admin Dashboard (phase 3 of consolidation)

Added

• New /admin/alert-management page (templates/admin/alert_management.html, dashboard.py:alert_management_page()) -- the stored-alerts table (search/edit/delete) and mark/delete-expired operations that used to be the Admin Dashboard's "Alerts" sub-tab. No dedicated page existed for this, so it was extracted rather than deleted, reusing the exact same static/js/admin/alert-management.js and edit-alert modal markup (faithful move, not a rewrite, given how much interactive logic is in that file).

Fixed

• alert-management.js would have crashed on any page without the Data tab's boundary-upload form. Despite its name, this file also unconditionally wires up #uploadForm/#shapefileUploadForm submit handlers (document.getElementById(...).addEventListener(...) with no null check) -- historical, not something worth renaming/splitting right now. Guarded both, matching the defensive pattern already used elsewhere in the same file for #alerts-subtab. Required to load this file safely on the new standalone page; behavior on the Admin Dashboard's Data tab, where those elements do exist, is unchanged.

Removed

• Admin Dashboard's Alert Management sub-tab and its now-orphaned Edit Alert modal (nothing else in admin.html triggers it once the sub-tab and its "Edit" buttons are gone). alert-management.js itself stays loaded on admin.html -- it still runs the Data tab's boundary-upload handling.

  Remaining in admin.html: Data tab, System tab's Location/Alert Filtering, Broadcast's EAS Encoder Settings -- all three share one 1,475-line interactive JS module (FIPS pickers, zone search, EAS tone/DTMF UI) and are next.

[2.187.1] - 2026-08-23 - Remove duplicate navigation entries

Removed

• Hardware Settings and Zigbee were listed twice -- once under Monitor -> Station Hardware, again under Settings -> Hardware, same pages, two different icons/labels. Both are configuration screens, not live-status views, so Settings is their one home now; Monitor -> Station Hardware keeps only GPS & Time (a real live-status dashboard).
• Audio Ingestion (Monitor) duplicated Audio Streams (Settings) -- both pointed at /admin/audio-sources. Kept the Settings entry.
• Settings -> Hardware's entry now references endpoint="hardware.hardware_settings_page" instead of a hardcoded href="/admin/hardware" -- same URL, but matches how the rest of the registry references pages that can move, and keeps test_both_settings_pages_are_reachable_from_the_navigation (which checks for that specific endpoint reference) passing now that it's no longer duplicated under Monitor.
• Left Security Center duplicated on purpose -- also listed under both Monitor -> Diagnostics and Settings -> Security & Access with the same LOGS_VIEW permission gate. Removing the Settings copy left a LOGS_VIEW-only viewer with a Settings link that opens to zero cards (caught by test_settings_link_never_leads_to_an_empty_hub) -- it's the only Settings-hub item that permission level can see, so both copies stay.
• Left Security Settings/Security Policies duplicated on purpose -- one copy lives in the signed-in user's own account menu ("your password, MFA and active sessions"), the other in the Settings hub ("password policy, session limits and MFA settings"). Same URL, but a personal quick-link and a general-discoverability entry serve different purposes, similar to how most sites surface account security in both places.

[2.187.0] - 2026-08-23 - Remove Admin Dashboard's Operations and Security tabs (phase 2 of consolidation)

Added

• "Trigger Poll Now" and "Manual Alert Import" on Admin -> Alert Poller Settings (/admin/poller) -- these had no home other than the Admin Dashboard's Operations tab, so their forms and JS moved here rather than being deleted with the rest of that tab.
• New /admin/user-accounts page (templates/admin/user_accounts.html, dashboard.py:user_accounts_page()) extracting the Admin Dashboard's Security tab's "Users & Roles" sub-tab -- the full create/list/reset/ delete-user UI. This one was NOT a duplicate: the nav registry's "User Accounts" entry linked straight at /admin/users, which turned out to be a JSON-only API (dashboard.py's admin_users(), GET returns a user list, POST creates), not a page -- clicking that Settings link served a raw JSON blob. The Admin Dashboard's tab was the only working UI for user management, so it was extracted rather than deleted, and the registry entry fixed to point at the new page instead of the API.

Removed

• Admin Dashboard's entire Operations tab (CAP Alert Polling, One-Button Backup, One-Button Upgrade, Import Archived Alert, Database Optimization, Fix Intersections, Environment Config, Database Health, Alert Purge). Confirmed the backup/upgrade/DB-optimize/fix-intersections/DB-health buttons called the exact same backend endpoints (/admin/operations/backup, /admin/operations/upgrade, /admin/optimize_db, /admin/recalculate_intersections, /admin/check_db_health) as the already-registered /admin/operations page, which itself links onward to the more capable /admin/backups for full backup management -- these weren't just duplicate links, they were independent frontends for the same backend actions, and any future change to one would silently not apply to the other. Environment Config duplicated /admin/environment's own editor the same way; deleted its modal along with the two forms.

• static/js/admin/operations.js -- entirely dead code once the Operations tab was removed; nothing else loaded it.

• hardware-settings.js's redundant <script> include on admin.html -- found while auditing the other includes: nothing on the page calls any of its functions (it's the GPS sky-plot panel for the already-registered Hardware Settings page, which still loads it correctly on its own).

• Alert Purge was an orphaned page -- had no navigation entry anywhere, reachable only via this tab or a direct URL. Added to Settings -> Data & Storage, next to Backups, instead of just deleting its only link.

• Admin Dashboard's entire Security tab -- "Users & Roles" moved to the new /admin/user-accounts page above; "Local Authorities" was already a separate registered page, just linked from here too.

  This continues the admin.html//settings consolidation from 2.186.2. Remaining tabs (Data, System, Broadcast's EAS Encoder Settings form) still need the same treatment.

[2.186.2] - 2026-08-23 - Remove pure link-grid tabs from the Admin Dashboard

Removed

• Admin Dashboard's Services and Hardware tabs, and the "Broadcast Tools" and "Security Monitoring" link grids inside the Broadcast and Security tabs. All four were pure card-grids linking to other pages -- zero inline forms or JS actions -- and every single link duplicated a page already registered in the nav (webapp/navigation/registry.py / registry_settings.py), same class of drift that caused #2454. Verified each of the ~25 removed links against the registry before deleting (two, "EAS Workflow" and "RWT Schedule", are registered via endpoint= rather than a literal href= matching their admin.html URL, which is why an earlier plain-text grep missed them -- double-checked those against the actual endpoint names before confirming duplication).

  This is phase 1 of a larger admin.html//settings consolidation -- admin.html's remaining tabs (Data, System, Broadcast's encoder form, Security's user management, Operations) contain real inline functionality, including several that duplicate the implementation of a separately registered page (not just the link) -- e.g. its own user-CRUD UI, its own DB-optimize/recalculate-intersections buttons, its own backup trigger, and its own .env editor, each independent of the dedicated page for the same feature. Those need to be resolved (confirm equivalence, then delete one side) or extracted into their own registered pages, tab by tab, as separate follow-up work -- left as-is here since it changes application behavior, not just navigation.

Fixed

• Admin Dashboard's stat cards didn't match the rest of the app. static/css/admin.css carried a stale, plainer .stat-card/.stats-grid definition (flat primary/secondary gradient, 16px radius, no shimmer); static/css/styles.css owns the current "iOS 26 vibrant mesh gradient" version used everywhere else. Same selector, same specificity, and admin.css loads after styles.css (base.html's extra_css block), so the cascade tie silently went to the older rule -- the exact trap already documented and fixed once in this file for .nav-tabs .nav-link, just not caught here. Removed the stale duplicate so the Admin Dashboard picks up the same stat cards as every other page.

[2.186.1] - 2026-08-23 - Fix Tickstem/Heartbeat nav placement

Fixed

• Tickstem Uptime Monitor didn't appear on the Settings hub. #2453 added it (and a duplicate "Uptime Heartbeat" entry) to the navbar's Diagnostics section, on the mistaken belief that Heartbeat had no menu entry at all -- it did, in webapp/navigation/registry_settings.py (a separate module from registry.py that a plain grep missed), already surfaced correctly on /settings. Removed the duplicate/misplaced entries and added Tickstem next to Heartbeat in the Settings hub's Configuration group instead.

[2.186.0] - 2026-08-22 - Add Tickstem uptime-monitor integration

Added

• New Admin -> Tickstem Uptime Monitor page. Manages an inbound Tickstem uptime monitor (polling this box's public /health endpoint) directly from the admin UI via Tickstem's bearer-token Monitors API -- create, pause, resume, and delete the monitor, and view its recent check history, without leaving EAS Station. Distinct from the existing outbound dead-man's-switch Heartbeat page: that one pings out on a schedule (works with any healthchecks.io-style receiver, no API key); this one is Tickstem polling in, authenticated with an account API key (app_core/tickstem_client.py, TickstemSettings model/ tickstem_settings table). The account API key is never returned to the browser after being saved. Both the Heartbeat and new Tickstem pages were also added to the navigation registry (System Health group) -- the Heartbeat page previously had no menu entry at all.

[2.185.8] - 2026-08-22 - Widen img-src CSP and fix heartbeat ping method

Fixed

• Album art blocked by CSP on /audio-monitor -- station metadata (ICY StreamTitle) can point at any CDN the broadcaster happens to use (seen: image.iheart.com); the CSP's img-src only allowlisted three specific hosts, so art from any other station silently failed to load. img-src now allows any https: origin, since this only affects passive image loading, not script execution.
• Uptime heartbeat pings 401'd against some healthchecks.io-alternative services (e.g. Tickstem) -- send_heartbeat_ping() always sent GET; those services only accept POST on the ping route and (confusingly) reply 401 "missing authorization" to a GET, which looks like a credential problem but isn't one. Now sends POST, which healthchecks.io itself also accepts, so this is backward compatible.

[2.185.7] - 2026-08-22 - Fix broken audio streaming and several console errors site-wide

Fixed

• Icecast audio playback was silently blocked everywhere -- the CSP had no media-src allowance, so every <audio> element pointed at Icecast (a different origin/port than the Flask app) failed to load with no visible error beyond the browser console. media-src now includes the Icecast origin, resolved per-request from IcecastSettings (the host:port is user-configurable, so it can't be hardcoded).
• The /audio-monitor live players never rendered -- getIcecastStreamUrl() read metadata.icecast_mount, a field the API has never actually returned (it's metadata.icecast_stream_url). Derived the same-origin path the nginx proxy (location ~* ^/(?!eas-)([a-z0-9_-]+\.mp3)$, already correctly configured) expects from the field that does exist. Verified live: currentTime advancing during actual playback, not just a clean network response.
• /admin/audio-sources cards rendered correctly for ~30s, then flipped to "UNKNOWN" / "? Hz . ? ch" / frozen "-- dBFS" labels and stayed broken -- three compounding bugs: (1) the periodic WebSocket push (_emit_audio_sources_update) sends a deliberately thin payload with no config/metrics, but the frontend replaced the richer REST-fetched source objects wholesale instead of merging, discarding sample rate/channel/level data every ~30s; (2) that same push's id field was the numeric DB primary key while the REST endpoint's id is the source name, breaking the merge's DOM-element lookups once merging was fixed; (3) its type field assumed source_type was an Enum with a .value attribute when it's actually a plain string column, so type silently resolved to None -> "unknown" -> "UNKNOWN" on every source, every time. Fixed the merge to be a merge, and fixed the push to serialize id/type the same way the REST endpoint already does.
• Peak/RMS level labels were permanently stuck on their "-- dBFS" placeholder on any page not loading realtime-vu-meters.js (e.g. admin/audio_sources.html) -- the update guard skipped writing whenever the label's current text already contained "dBFS", which the static HTML placeholder always does, so the check could never tell "still the placeholder" from "already owned by the other script" apart. Replaced it with an explicit data-live-vu marker realtime-vu-meters.js sets only when it actually starts driving a label.
• bootstrap is not defined crashed /settings/stream-profiles on every load -- bootstrap.Modal.getOrCreateInstance(...) ran at the top of a synchronous inline <script>, but bootstrap.bundle.min.js loads via <script defer> and hadn't executed yet at that point in parsing. Deferred the modal instance creation into the page's own DOMContentLoaded handler.
• /audio/health/dashboard threw "Canvas is already in use" on a significant fraction of ordinary page loads (no theme switch involved) -- a race between theme.js's own DOMContentLoaded listener (registered earlier, since it loads in base.html) synchronously dispatching theme-changed as part of applying the saved theme, and this page's DOMContentLoaded handler creating the same charts unconditionally afterward. Made initializeCharts() idempotent (destroys any existing instances first) instead of relying on each call site to remember to.
• Removed two dead <script src="https://cdn.socket.io/..."> tags on /audio-monitor -- Socket.IO is already loaded globally from the vendored copy in base.html; these were both redundant and blocked outright by CSP (no external script-src allowance).
• Removed a premature typeof showLoading !== 'function' check on /admin/environment that logged a false-positive console error on every load (same "runs before the deferred script" pattern as the bootstrap fix above, just self-correcting rather than crashing).
• Added https://www.debian.org to img-src for the detected-OS logo on /system_health (a fixed, single, trusted domain, unlike the broadcast-metadata-sourced album art CSP gap on /audio-monitor, which is deliberately left as-is -- that content can point at literally any station's arbitrary third-party CDN, so it isn't safe to broadly allowlist).

[2.185.6] - 2026-08-22 - Fix modals trapped below the backdrop (unclickable, unscrollable)

Fixed

• Four modals rendered completely unusable -- visible, even appearing to scroll, but every click landed on the backdrop instead of the modal's own buttons, with no way to dismiss short of reloading the page: #trafficSettingsModal (Security Center's Traffic Collection Settings, reported live after the previous scrollable-modal fix still didn't resolve it), #envEditorModal and #editAlertModal (Admin Panel), and #purgeModal (Audio Archive). Root cause: static/css/styles.css's .page-shell > *:not(.orb):not(.modal) rule gives every direct child of <main class="page-shell"> its own stacking context (z-index: 1) so page content can't bleed through the sticky navbar; a .modal is excluded from that rule only when it is itself a direct child of .page-shell. All four were instead nested inside the page's .container-fluid wrapper (in #trafficSettingsModal's case, inside an {% include %}ed partial), inheriting that wrapper's stacking context and rendering trapped below Bootstrap's body-level backdrop (z-index: 1040) -- a pattern this codebase had already hit and fixed once before, on templates/admin/certbot.html's #logModal. Moved each modal's markup to be a direct child of {% block content %}, matching that existing fix. Verified live with real Playwright clicks (not just computed-style checks, which is how this shipped unnoticed the first time) confirming the actual close/cancel button -- not the backdrop -- receives the click at its real screen position while the modal is open.
• Added tests/test_modal_stacking.py: a static, {% include %}-resolving check over every page template that flags a .modal nested inside a non-modal, non-.orb wrapper within {% block content %}, with a documented allowlist for the one legitimate exception (#confirmationModal, relocated to <body> at runtime by static/js/admin/core.js before it can ever be shown). Catches this exact bug class in CI for any modal added or moved in the future.

[2.185.5] - 2026-08-22 - Theme readability audit: badges, invisible SAME codes, blank icons, off-screen dropdown

Fixed

• .status-badge.success/danger/warning/info rendered under WCAG AA (as low as ~2.1:1) on every theme because color: read straight from --success-color etc. instead of the AA-tuned --*-ink split .text-success and friends already use. Switched to the -ink variables and reduced each variant's background tint (20% down to 1-2%, depending on how much headroom that variant's -ink colour has) until all four variants clear 4.5:1 pixel-sampled across all 20 themes. Added .status-badge.* probes to scripts/diagnostics/check_theme_contrast.py so this can't silently regress again.
• SAME/FIPS code chips on the EAS broadcast workflow page (.same-chip code, e.g. "039003") were completely invisible on 7 dark themes (aurora, charcoal, lightning, midnight, nebula, obsidian, slate) -- measured at a 1.00 contrast ratio (text and background the literal same colour). Root cause: .same-chip's own dark-text override (added specifically for this "near-white chip on a dark theme" pattern) sets color on the chip container hoping to cascade to children, but the global code { color: var(--text-color) } rule is a direct declaration on the <code> element itself, and inheritance always loses to an explicit rule on the descendant regardless of the ancestor selector's specificity. Added a code-specific override alongside the existing .text-muted one in the same block, following the exact same pattern.
• Inactive tabs in the admin panel's tab bar (.nav-tabs .nav-link) read in a hard-to-see muted grey on several dark themes. static/css/admin.css had its own .nav-tabs .nav-link { color: var(--text-secondary) } rule that -- at identical specificity to static/css/styles.css's color: var(--text-color) rule for the same selector, added specifically "to ensure readability on all themes" -- won the cascade tie by loading second on admin pages, silently re-introducing the exact problem the other rule was written to fix. Removed the redundant, regressive declaration.
• The navbar's Help/Reports/Settings dropdown menus opened off the right edge of the viewport (confirmed clipped ~140px past a 1400px-wide viewport), because the shared nav-section dropdown template never got Bootstrap's dropdown-menu-end modifier -- already used correctly for the user-avatar dropdown at the far right, just missing here. Added it.
• ~150 icons across 6 pages (Security Center, Certbot/SSL admin, TTS settings, Audio Health Dashboard, both audio error pages) rendered as nothing at all -- bi bi-* (Bootstrap Icons) classes, but the project only ever loads Font Awesome (vendor/fontawesome); Bootstrap Icons' font was never included, so every bi-* class drew an empty glyph. Mapped all 37 distinct icon names used to their closest Font Awesome equivalent (e.g. bi-shield-lock -> fa-shield-halved, bi-arrow-clockwise -> fa-sync-alt) and replaced every occurrence, including ones built dynamically in inline JS template strings.

[2.185.4] - 2026-08-22 - Fix OLED-absent memory leak that repeatedly OOM-killed services

Fixed

• eas-station-displays.service leaked memory at ~17 MB/s whenever the Argon OLED panel was enabled in settings but not physically present (no ACK from I2C address 0x3C), triggering the kernel OOM-killer three times in under an hour (confirmed live: 13:02, 13:26, 13:51, each time after the process's RSS reached ~13 GB) and taking down whatever else the kernel picked alongside it. Root cause: scripts/screen_manager.py's main loop runs at 60 FPS, and several render paths (notably the scroll-animation tick, _update_active_oled_scroll()) fall back to oled_module.oled_controller or initialise_oled_display(logger) with no throttle of their own -- when the controller is absent, that calls initialise_oled_display() fresh on every single frame. Building the display driver allocates a width*height-sized pixel offset/bitmask table (luma.oled's ssd1306.__init__, ~100-300 KB for a typical 128x64+ panel) before it ever reaches the I2C handshake that fails, so each doomed attempt costs a real allocation, not a cheap syscall -- at ~60/s this outpaced glibc/pymalloc ever trimming freed memory back to the OS. (_ensure_oled_button_listener() in the same file already throttles its own retries to every 5s for exactly this "loop runs at 60 FPS" hazard -- the scroll/render paths just never got the same treatment.) initialise_oled_display() in app_core/oled.py now caches a failed probe for 5 seconds (module-level cooldown) before it will attempt the hardware handshake again, protecting every caller uniformly instead of requiring each call site to remember its own throttle. Verified live: retry frequency dropped from ~60/s to 1/5s and RSS stayed flat (~270-281 MB, stable) over a monitored 30s window after the fix, versus climbing ~52 MB every 3s before it.

[2.185.3] - 2026-08-22 - Fix live waterfall/scope zoom desync and unscrollable modals

Fixed

• The Live Waterfall / Spectrum Scope toggle buttons on Radio Diagnostics silently reverted to their "off" label ("Live Waterfall") every ~1s while the feed was still actively running, and any click-drag pan gesture on the waterfall/scope canvas longer than ~1s was silently cut short. Root cause: the receiver detail panel is fully re-rendered from scratch on every WebSocket radio_status_update push (roughly once a second), which re-emitted the toggle button with its static off-label (the server-side markup has no notion of the client-only liveWaterfallState) and handed the canvas a brand-new DOM node each time, silently abandoning any in-progress pointer-captured drag. updateReceiverDetails() in templates/admin/radio_diagnostics.html now moves the real .waterfall-container / .spectrum-scope-container DOM node (canvas bitmap, zoom/pan listeners and all) out before the re-render and back into the freshly-rendered placeholder afterward, and explicitly re-syncs the toggle labels to the actual running state. Verified live against a running SDR receiver: the toggle label now stays correct across dozens of refresh cycles, and a deliberate multi-second drag-to-pan now completes without the canvas being replaced mid-gesture.
• Modals with content taller than the viewport had no internal scrollbar and simply clipped off-screen, cutting off the Save/Cancel buttons in the footer (reported via the Security Center's Traffic Collection Settings modal, but the same gap existed repo-wide: 23 of the site's 35 .modal-dialog instances were missing Bootstrap's modal-dialog-scrollable modifier, which the project's own static/css/styles.css already has full, tested support for). Added modal-dialog-scrollable to every modal missing it across templates/base.html, templates/admin.html, templates/security_settings.html, templates/security/security_center.html, templates/security/_traffic_content.html, templates/screens.html, templates/screen_editor.html, templates/audio_history.html, templates/gpio_control.html, templates/docs/rbac_visual.html, and templates/admin/{local_authorities,tts_pronunciation,rbac_management,certbot,audio_archives,backups,network}.html. Header and footer now stay pinned in place with only the body scrolling internally, matching Bootstrap's intended behavior for tall modal content.

[2.185.2] - 2026-08-21 - Poller no longer runs a duplicate copy of the web app's background workers

Fixed

• eas-station-poller.service was accidentally running a second, independent copy of every one of the web app's background workers -- the uptime heartbeat ping, backup/retention/auto-purge schedulers, fail2ban sync, RWT scheduler, GPIO input listener, system metrics sampler, and web-traffic recorder. Root cause: poller/cap_poller.py does from app import (db, CAPAlert, ...) to reuse the web app's SQLAlchemy models instead of maintaining a second copy -- but that import executes app.py's entire module body as a side effect (first import wins), which is also where all of those workers get started, regardless of whether create_app() is ever called. Confirmed live with a py-spy stack dump on the running poller process: a thread literally named "HeartbeatWorker" inside cap_poller.py --continuous, sending a duplicate ping every 5 minutes alongside the two the web app's gunicorn workers already send. Since the poller is a long-running --continuous process that a systemctl restart eas-station-web never touches, this duplicate could also keep executing stale, pre-deploy bytecode for hours after a real fix shipped -- which is what made the 2.185.1 health-check User-Agent fix (see below) initially look incomplete: the two gunicorn workers were correctly fixed immediately, but the poller's duplicate kept sending the unfixed python-requests/2.32.5 default until it was separately restarted. poller/cap_poller.py now sets SKIP_DB_INIT=1 (the same flag alembic/env.py already uses for exactly this "import app.py for its models, not its side effects" situation) immediately before importing from app, so none of app.py's skip_background_services-gated workers start a second time. The poller's own dedicated schedulers (its polling loop, its gated-alert scheduler) are untouched -- verified via a fresh py-spy dump post-fix showing only the poller's own threads, and a live polling cycle completing normally immediately after restart.
• app.py's SKIP_DB_INIT comment updated to document that it's a general-purpose "don't start background services" flag for any script importing app.py for its models, not just an Alembic-specific one.

Added

• New tests/test_cap_poller_skip_db_init.py -- a structural regression test asserting the SKIP_DB_INIT setdefault appears in the source before the from app import (...) line that triggers app.py's execution, so a future edit can't silently reorder them back into the bug (a live-import test isn't used here deliberately: even in the fixed state, importing cap_poller.py has real side effects -- DB/Redis connections, thread pool creation -- unsuitable for a fast unit test).

[2.185.1] - 2026-08-21 - Outbound health checks now identify the station

Fixed

• Outbound "health check" requests were going out with the bare python-requests/x.x default User-Agent instead of identifying the station, unlike poller/cap_poller.py (which NOAA's Weather API terms require to carry a descriptive User-Agent with contact info). Affected: the uptime dead-man's-switch heartbeat ping (app_core/heartbeat_worker.py), the Icecast health probe feeding the dashboard/system-health status strip (app_core/system_health.py::collect_icecast_status()), and both Icecast admin routes (webapp/admin/icecast.py's test_connection() and get_status()). All four now send the same User-Agent the CAP poller uses, so a third-party monitor's or an Icecast server's request log shows "EAS Station" instead of an unidentified Python client.

Added

• New app_core/http_defaults.py::get_default_user_agent() -- single source of truth for the DB -> env-var -> hardcoded-default fallback chain that used to be duplicated inline in poller/cap_poller.py (and had already started drifting: five slightly different User-Agent strings were scattered across the codebase before this). cap_poller.py itself now calls the shared helper instead of inlining the same logic.
• New tests/test_http_defaults.py (6 tests): the DB/env/hardcoded fallback precedence (including a blank-DB-value edge case), that a DB lookup failure (no app context, DB unreachable) degrades gracefully instead of raising, and that the heartbeat ping actually sends the resolved value as its User-Agent header.

Testing

• Verified against a real local HTTP server: called the actual send_heartbeat_ping() function inside a real Flask app context and confirmed the request it sent carried User-Agent: EAS Station (+https://github.com/KR8MER/eas-station; support@easstation.com) -- this station's real DB-configured PollerSettings.noaa_user_agent value -- rather than the bare python-requests/2.32.5 reported before the fix.

[2.185.0] - 2026-08-21 - Broadcast overlay: phase indicator + web abort button

Added

• Broadcast phase indicator on the full-screen countdown overlay. The overlay now shows which phase a broadcast is in — Sending Header, Narration, or Sending EOM — instead of just a flat countdown. app_utils/eas.py::set_broadcast_active() gained two new optional parameters, header_seconds and eom_seconds (elapsed-time thresholds marking the end of the SAME header burst and the start of the EOM burst), which all four broadcast paths (RWT, manual Send, resend, live/forwarded alerts) now compute from the WAV segment durations they already have on hand and pass through. Pre-/post-alert chime duration (where configured) folds into the adjacent phase rather than getting a phase of its own, since chimes play immediately before the header or after the EOM. Both default to 0.0 ("unknown"), so a caller that doesn't supply them shows the old plain countdown rather than a wrong phase — no breaking change for any code that isn't touched by this release.
• "Hold to Abort Broadcast" button on the countdown overlay. The browser-side equivalent of holding a physical GPIO Dump/Abort Broadcast input for 3 seconds: a press-and-hold gesture on the on-screen button (fill-bar animation showing progress, releasing early cancels with no effect) calls the new POST /api/broadcast/abort route (webapp/routes/broadcast_control.py, gated on the eas.cancel permission — Admin and Operator roles have it by default), which calls the exact same abort_current_broadcast() the GPIO input dispatch loop uses: interrupts the in-progress message but always attempts the required EOM burst (47 CFR 11.61(a)) before releasing the relay, and writes an audit ledger entry with the logged-in operator's username (rather than the generic gpio-input operator a physical button records), so a compliance review can see who ended the broadcast from the web UI.
• New tests/test_broadcast_phase_and_web_abort.py (9 tests): phase breakpoints round-trip through set_broadcast_active()/ get_broadcast_state() (including the None-vs-omitted-kwarg edge case), and the abort route's permission gate, 409-when-idle case, 409-when-no- trackable-PID case (the state marker said "active" but nothing was published yet — a narrow timing window), 500-on-internal-failure case, and that the audit-facing operator identity comes from the session.

Testing

• Verified against real hardware in the lab: triggered a genuine RWT, confirmed header_seconds/eom_seconds appear correctly in /api/broadcast/state during playback (SAME burst ≈7.2s, EOM burst ≈3.2s for a 14.4s RWT, matching its no-narration FCC-mandated shape), then called the real POST /api/broadcast/abort route (via Flask's test client with a genuine authenticated admin session, not a direct function call) against the live broadcast and confirmed: the message was cut, the EOM burst audibly played from a second aplay invocation, the relay released only after that finished, and the audit ledger recorded username: 'admin' with eom_sent: true.

[2.184.1] - 2026-08-21 - Dump / Abort Broadcast: real-hardware fixes

Real-hardware verification of 2.184.0's Dump/Abort Broadcast action (triggering a genuine RWT and aborting it against real GPIO/audio hardware, per the project's "test on real hardware before merge" standard) surfaced two gaps that mocked unit tests couldn't have caught. Both are fixed here, in the same still-open PR, before merge.

Fixed

• _run_command()'s PID tracking only ever covered the live/auto-forwarded alert path (app_utils.eas.EASBroadcaster._play_audio()) — RWT broadcasts (app_core/rwt_scheduler.py::_drive_rwt_airchain()), manually "Sent" activations (webapp/eas/workflow.py), and resends (scripts/resend_eas_broadcast.py) each launched their own separate, untracked subprocess.run(...). A physical Dump/Abort press against any of those three — RWT is the most routinely-tested broadcast on the station — silently no-op'd: get_broadcast_pid() returned None even while aplay was audibly running. All three now call the same app_utils.eas.play_broadcast_audio() entry point _run_command() grew for this, so every broadcast, regardless of trigger, is abortable.
• A GPIO-forced abort could end a broadcast without ever sending the required End-Of-Message burst (47 CFR 11.61(a)) — the original implementation just killed the player process. abort_current_broadcast() now reads the isolated EOM tone-burst audio (published to Redis alongside the PID by the same play_broadcast_audio()/_run_command()) and plays it synchronously — via the same audio_player_cmd — before releasing the transmitter relay or writing the audit entry. The audit ledger's EAS_CANCELLATION entry now records eom_sent: true/false so an abort that genuinely could not send one (no audio player configured, playback failure) is visible in the compliance trail rather than silently missing. The relay is still always eventually released — a broadcast stuck on-air forever is its own violation — but only after the EOM attempt completes.
• The EOM audio Redis key needed base64 encoding. get_redis_client() returns a client configured with decode_responses=True (every other key this module publishes is text); raw WAV bytes are not valid UTF-8, so the first hardware test run showed the PID publishing correctly but get_broadcast_eom_audio() silently returning None on every read — the UTF-8 decode failure was swallowed by that function's own best-effort except Exception: return None. _publish_broadcast_pid() now base64-encodes the EOM payload; get_broadcast_eom_audio() decodes it back.
• Corrected an inaccurate claim in the 2.184.0 entry below: _run_command() was not already the chokepoint every broadcast path funneled through — see the first fix above.

Testing

• tests/test_gpio_dump_broadcast.py grew from 11 to 20 tests: EOM publish/clear round-trip through _run_command(), play_broadcast_audio() delegating correctly (including its new timeout param, replacing the three call sites' own subprocess.TimeoutExpired handling), abort_current_broadcast() always attempting the EOM burst before clear_broadcast_active()/the audit write (asserted via ordering checks inside the mocked EOM-play callback), and the fail-safe cases (no EOM audio published, EOM playback failure) still completing and flagging eom_sent: false rather than hanging the relay forever.
• Verified against real hardware in the lab: triggered a genuine RWT (audible, keyed GPIO pin 17, real aplay PID published), called abort_current_broadcast() against it mid-playback, and confirmed: message audio cut by SIGTERM, the isolated EOM burst audibly played from a second aplay invocation, the relay released only after that finished, and an audit_logs row written with eom_sent: true.

[2.184.0] - 2026-08-21 - GPIO input action: Dump / Abort Broadcast

Added

• New GPIO input action: Dump / Abort Broadcast — the fourth and final input action, completing the GPIO input framework. A physical button assigned this action forcibly stops whatever is currently on air: sends SIGTERM (escalating to SIGKILL after a 2s grace period if needed) to the playback subprocess, releases the transmitter relay via the same clear_broadcast_active() path a normal broadcast completion already uses, and writes an entry to the tamper-evident Ed25519-signed audit ledger (AuditAction.EAS_CANCELLATION) — an operator-forced abort of a life-safety broadcast is exactly the class of event that ledger exists for. No-ops (logged) when nothing is currently playing.
• Safety default: requires a sustained 3-second hold, not a tap (gpiozero's when_held, admin-configurable 1–10s per pin) — a momentary bump or contact bounce must never be able to abort a live EAS broadcast. This is the one input action that arms differently from the other three, which fire on a plain press.
• app_utils/eas.py::_run_command() — used by the live/auto-forwarded alert broadcast path — now publishes its playback subprocess's PID to Redis (eas:broadcast_pid) for the duration of the call and clears it on completion (success, launch failure, or exception mid-wait, via a finally). Behaviourally unchanged for every existing caller: still blocks until the command exits, still swallows and logs rather than raises. subprocess.Popen(...).wait() replaces the previous bare subprocess.run(...) — functionally identical, with the PID observable mid-flight by another process. Correction (2.184.1): at the time this entry was written, RWT/manual-Send/resend were believed to already funnel through this same function — real-hardware testing showed they did not, each running its own untracked subprocess.run(). See the 2.184.1 entry above for the fix.
• New app_core/audio/gpio_input_actions.py::abort_current_broadcast() — reads the published PID, drives the terminate/kill sequence, and writes the audit entry; never raises (runs inside the GPIO input-event dispatch loop).
• New Pin Map UI control: "Hold time to confirm (seconds)", shown only when Dump / Abort Broadcast is selected, wired through the existing input_hold_confirm_seconds field.
• New tests in tests/test_gpio_dump_broadcast.py (11 tests) — the heaviest coverage of the four input-action phases per the original design plan, given this is the one that touches the shared playback call path: _run_command() regression (still blocks, still swallows exceptions, PID published/cleared correctly including on failure), the SIGTERM→SIGKILL escalation, the no-op-when-idle case, the already-exited-process case, the audit entry's fields, and dispatch wiring. tests/test_gpio_input_watcher.py's hold-vs-press test no longer needs to temporarily unlock the action (it's genuinely implemented now), and its "not yet implemented" test was rewritten to validate the GPIO_INPUT_ACTION_IMPLEMENTED filter mechanism generically, since all four real actions are implemented as of this release.
• docs/architecture/THEORY_OF_OPERATION.md's Broadcast Orchestration section documents the abort path, per this project's standing rule that touching broadcast/verification logic keeps that doc in sync.

[2.183.0] - 2026-08-21 - GPIO input action: Acknowledge Dead Air

Added

• New GPIO input action: Acknowledge Dead Air. A physical button assigned this action silences the dead-air rack buzzer the same way the Acknowledge button on the Audio Health dashboard does -- the tower light stays lit until audio actually returns, matching standard alarm-panel behavior. Acknowledgement is station-wide (scoped to the current alarm episode, not a single source, since dead-air detection is per-source but the buzzer/light output is aggregated), so the input needs no source targeting.
• Refactor: the acknowledge logic moved out of the Flask route into a new shared function, app_core/audio/dead_air_alarm.py::acknowledge_dead_air() -- extracted so the GPIO input action and the web UI's Acknowledge button call the exact same Redis-backed logic instead of two copies drifting apart, the same pattern already used for Run RWT Now (trigger_rwt_broadcast()). webapp/admin/audio_ingest/routes_dead_air.py's audio_dead_air_acknowledge() route is now a thin wrapper with no behavior change (same status codes, same Redis keys/TTL).
• New app_core/audio/gpio_input_actions.py::acknowledge_dead_air_alarm() -- the GPIO-side wrapper, logging the outcome; never raises.
• New tests in tests/test_gpio_dead_air_ack.py (8 tests) covering the shared core function directly (active-alarm requirement, episode matching, un-acknowledge, no-Redis case) plus the GPIO wrapper and dispatch wiring. tests/test_dead_air_monitoring.py's existing source-level checks were updated to point at the new location of the extracted logic.

[2.182.0] - 2026-08-21 - GPIO input action: Forward Last Alert

Added

• The Forward Last Alert GPIO input action is now implemented (was a placeholder enum value since v2.181.0). A physical button assigned this action re-broadcasts whichever EAS message was most recently generated, the same way a message's "Resend" button does: same live-broadcast guard, same detached-subprocess launch (scripts/resend_eas_broadcast.py) so GPIO is never keyed from inside the web process.
• New app_core/audio/gpio_input_actions.py (forward_most_recent_alert()) -- finds the most recent EASMessage with stored audio via the blob-deferred query pattern already used elsewhere on that model, so the lookup never pulls a multi-megabyte audio blob into memory just to find an ID. No-ops (logged, never raises) when nothing broadcastable exists yet.
• New tests in tests/test_gpio_forward_last_alert.py (4 tests): correct message picked by recency, the broadcast-in-progress skip, the no-broadcastable-message no-op, and the dispatch wiring.

[2.181.0] - 2026-08-21 - GPIO input pins: framework + Run RWT Now

Added

• GPIO pins can now be configured as INPUTs, not just relay outputs (Admin → GPIO → Pin Map: select Input, then choose an action). The entire app_utils/gpio stack was output-only before this; a pin is read via gpiozero.Button (same library and debounce mechanism as the Argon OLED's front-panel button) by a new GPIOInputWatcher, running inside the eas-station-gpio subprocess. A press publishes an event on a new Redis channel (eas:gpio_input_events); the main web app subscribes and runs the actual action, since it — not the GPIO subprocess — owns things like the RWT scheduler.
• First implemented action: Run RWT Now — a physical button wired to an input pin triggers the same trigger_rwt_broadcast() call, with the same live-broadcast guard, as the existing manual "Run Test Now" button on the RWT Schedule page.
• The action enum ships complete (Forward Last Alert, Dump / Abort Broadcast are defined values) so later phases don't need another gpio_pin_map JSONB shape change, but only Run RWT Now is offered in the UI and does anything today — the others are inert placeholders.
• Validation at save time: an input pin cannot also carry an output behavior, and each action may only be assigned to one pin — both hard-blocked (webapp/admin/hardware.py's new validate_gpio_input_pin_config()), not just warned about.
• No migration required — direction/input_action/input_bounce_ms fields were added to the existing gpio_pin_map JSONB shape, the same way flash_partner_pin was added previously.
• New tests in tests/test_gpio_input_watcher.py (14 tests): button/queue wiring, debounce pass-through, the publish/subscribe event round trip, RWT dispatch (including the live-broadcast skip), config-loader parsing with back-compat defaults, and the save-time validation.

[2.180.0] - 2026-08-21 - Relay interlock groups (mutual exclusion)

Added

• Relay Interlock Groups (Admin → GPIO → Interlocks, /admin/gpio/interlocks): named groups of 2+ relay pins that must never be energized at the same time — the motivating case is two PTT lines that would key two transmitters into each other if both fired. By default, activating a pin while another group member is already active is refused (and logged as a failure in the GPIO Activation Log); an opt-in per-group "Force-release conflict" setting instead de-energizes the active relay first. Enforcement lives in GPIOController.activate() — the single chokepoint every activation passes through, whether manual or alert-driven — so the guarantee holds regardless of trigger path.
• A save-time warning (surfaced on both the Interlocks page and the GPIO Control page's diagnostics) flags interlock groups whose members share a broadcast-holding behavior (Transmitter PTT, Audio Playout, Duration of Alert, Audio Mute): during a real alert only one would ever actually key, and the interlock would silently refuse the other.
• New tables relay_interlock_groups / relay_interlock_members (migration 20260821_relay_interlocks). Like the rest of GPIO configuration, group definitions are read once at eas-station-gpio startup — changes require restarting the service.
• New tests in tests/test_gpio_relay_interlock.py, including a regression test for the interaction with GPIOBehaviorManager.start_alert()'s "key every configured pin" fallback (a refused pin must not spuriously trigger it).

[2.179.1] - 2026-08-21 - Fix false-positive GPIO Pin Map conflict on BCM 14

Fixed

• The GPIO Pin Map showed a conflict on BCM 14 whenever GPS or Zigbee was enabled on the Pi's default primary UART, even with the Argon OLED completely disabled and nothing else configured on the pin. BCM 14 is statically reserved for the Argon OLED module's wiring in app_utils/pi_pinout.py -- true regardless of whether the OLED feature is actually enabled -- but the pin map's conflict formula (_build_pin_entry() in webapp/routes/system_controls.py) treated that static label as a live claimant unconditionally. The equivalent oled_enabled gate already existed on the relay-config validation side (app_utils/gpio/config_loaders.py) but was never carried over to the pin map. The conflict-detection formula now only counts the Argon OLED reservation when oled_enabled is actually true, via a new standalone pin_reservation_is_active() helper (with regression coverage in tests/test_gpio_pin_map_reservations.py).
• The conflict banner also under-explained itself: it enumerated the dynamic hardware claims but never mentioned a fixed reservation (like the Argon OLED wiring) even when that reservation was the second claimant causing the conflict, so the message could say "claimed by more than one thing" while naming only one. The banner and the "Reserved & system pins" reference table now both surface the fixed reservation when it's part of the conflict, and the reference table marks an inactive reservation as "not active" so it's clear why a statically-reserved pin isn't blocking anything.

[2.179.0] - 2026-08-20 - Uptime diagnostics pack

Added

• systemd hang watchdog for eas-station-audio and eas-station-poller (Type=notify + WatchdogSec=). Previously Restart=always only caught process crashes -- a live-but-deadlocked process (e.g. stuck in a blocking call) kept running forever. Both services now send WATCHDOG=1 heartbeats from their main loops (app_utils/system/sd_notify.py); a hang now gets systemd to kill and restart the unit.
• Outbound dead-man's-switch heartbeat: every existing health check was inward-facing (a page someone has to look at). Settings → Uptime Heartbeat (/admin/heartbeat/) configures a periodic ping to an external monitor (e.g. healthchecks.io) that fires unconditionally on schedule, so total box/network failure -- not just degraded internal health -- raises an alarm on a channel outside this appliance.
• Scheduled backup-restore verification: Admin → Backups now has a "Backup Restore Verification" panel. tools/verify_backup_restore.py restores a backup's database dump into a throwaway scratch database, runs sanity checks, and always drops the scratch database afterward -- proving a backup actually restores, not just that it was created. Opt-in "Verify after each backup" toggle on the auto-backup schedule, plus a manual "Verify Latest Backup Now" button and history table.
• Clock/NTP drift monitoring: the system health snapshot (/system_health) now has a Clock Sync card, backed by a new consolidated collector (app_utils/system/clocksync.py). SAME timing and RWT scheduling both depend on correct system time; drift beyond 1 second or a desynchronized clock now raises a compliance alert.
• Combined NOAA+IPAWS feed-loss alarm: the existing poller-liveness check looked at the single most recent poll across all sources, so a live NOAA feed masked a dead IPAWS feed. An Alert Feeds card on /system_health now shows each feed's staleness independently, and an email/SNMP alert fires specifically when both feeds have stalled past the configurable threshold (Alert Poller settings → "Combined Feed-Loss Alert Threshold").

[2.178.0] - 2026-08-20 - Add a System Status strip to the main dashboard

Added

• The main dashboard (/) now shows a live System Status strip above the map: active alert count, EAS decoder monitor state (running/stopped, audio flowing/silent), overall app health (from /health) and the number of configured radio receivers. Previously the "Emergency Alert Dashboard" only showed the map — an operator had to know to navigate to Monitor → EAS Decoder Monitor or Diagnostics → Health Dashboard to see whether the decoder was actually listening. All four values come from endpoints that already existed and are already public (/api/alerts, /api/eas-monitor/status, /health); no new backend code was needed. Each tile links to the relevant detail page. New files: templates/components/dashboard_status_strip.html, static/js/pages/dashboard_status.js (self-refreshes every 30s, independent of the map's own 5-minute refresh cycle).

[2.177.2] - 2026-08-20 - Give the SDR capture service scheduling priority

Fixed

• The WBKS SDR receiver was logging ~1 SOAPY_SDR_OVERFLOW (buffer overflow) error every 15 minutes, around 96/day. Root cause isn't new -- app_core/radio/drivers.py already documents that doubling the RTL-SDR read buffer eliminated these errors but broke RBDS decoding far worse (sustained RBDS SYNC LOST), so the buffer size was deliberately kept small and the occasional overflow accepted as the lesser regression. What was never tried: eas-station-sdr.service ran at default process niceness, identical to gunicorn, ffmpeg, and everything else on the box, despite being the most latency-sensitive process on the system -- it has to service USB readStream() calls promptly enough to drain the driver's ring buffer before it overflows.
• Added Nice=-5 to systemd/eas-station-sdr.service so the CFS scheduler favors the capture loop under load, without touching the buffer/MTU trade-off already tuned for RBDS. No capability elevation required -- systemd applies the nice value before dropping to the eas-station user.

[2.177.1] - 2026-08-20 - Pin Map now catches GPS/Zigbee UART pin conflicts

Fixed

• The GPIO Pin Map's conflict detector (added in 2.177.0) only checked the GPS PPS pin, missing the far more common case: GPS and Zigbee both default to the Pi's primary hardware UART (/dev/serial0 for GPS, /dev/ttyAMA0 for Zigbee in HardwareSettings' own column defaults), which is permanently wired to BCM 14 (TXD0) / BCM 15 (RXD0). On this station specifically, GPS is enabled on /dev/serial0 while GPIO 15 also carries a relay playout behavior -- confirmed live with pinctrl (both pins reporting alt-function a4, the UART) -- meaning every EAS/RWT broadcast very likely interrupted GPS reception (and therefore chrony's GPS time source) for the length of the broadcast.
• _dynamic_hardware_reservations() now recognizes /dev/serial0, /dev/ttyS0, and the /dev/ttyAMA* family (covers the Pi 3/4 PL011 UART naming and the Pi 5 RP1 UART naming alike) as the primary UART and claims BCM 14/15 for whichever of GPS or Zigbee is configured to use it. The conflict check also now folds in the static Argon OLED reservations, so a dynamically-claimed pin that collides with fixed hardware wiring (not just another dynamic claim or a relay behavior) is flagged too.

[2.177.0] - 2026-08-20 - GPIO Pin Map now shows every pin actually in use

Added

• The GPIO Pin Map only knew about relay behaviors and the fixed Argon OLED wiring -- four real hardware features that claim a GPIO pin through a Hardware Settings field (the dead-air rack buzzer, the NeoPixel indicator strip, the GPS PPS timing input, and the Argon OLED's physical button) were invisible to the page, so their pins showed up as plain "available" GPIO even while genuinely in use. Worse, NeoPixel and GPS PPS both default to BCM 18 with no way for the page to warn about it.
• /admin/gpio/pin-map now computes these reservations live from HardwareSettings on every render (not hardcoded), routes claimed pins into the existing "Reserved & system pins" reference table alongside the static Argon reservations, and adds a red conflict banner plus per-pin conflict badges when two features -- or a hardware feature and a relay behavior -- claim the same pin.

Note

• This entry was dropped from the changelog when PR #2429 was merged shortly after PR #2428 -- both touched VERSION/CHANGELOG.md, and the merge conflict resolution kept #2428's 2.176.2 bump but discarded #2429's 2.177.0 entry even though the GPIO Pin Map code itself merged cleanly. This entry restores the record after the fact; no functional change.

[2.176.2] - 2026-08-20 - Stop the test suite from keying real GPIO hardware

Fixed

• Running pytest on a box that also hosts the live eas-station-gpio service could key a real relay. tests/conftest.py seeded a default DATABASE_URL for import-time isolation but never did the same for Redis, so app_core.config.redis_config fell back to localhost:6379 db 0 -- identical to production. Tests that exercise the RWT/airchain code path (test_airchain_fringe_cases.py, test_alert_gating.py, test_coverage_and_signature.py, test_ecig_compliance.py, test_forwarded_alert_audio_flow.py) call set_broadcast_active(), which writes the eas:broadcast_active marker straight into that shared Redis. The always-running eas-station-gpio service watches that exact key to key the physical relay, so it dutifully activated real hardware under fake alert identifiers like TEST-001 and TEST-RWT-001 -- dozens of times over several days, logged in the GPIO activation audit trail as genuine automatic broadcasts. tests/conftest.py now defaults REDIS_HOST, REDIS_PORT, and REDIS_DB (to a dedicated db 15) alongside the existing DATABASE_URL default, so the test suite can never again share a broadcast-state marker with a live relay-keying service.

[2.176.1] - 2026-08-20 - Fix certbot renewal: stale systemd path and never-clearing lock

Fixed

• Automatic certificate renewal was silently doing nothing. The deployed certbot.service still pointed at a pre-migration config directory (/home/user/eas-station/certbot_data/...) with no certificates in it, so the twice-daily renewal timer ran, always exited successfully, and never touched the real certificate — which then expired. Fixed the path in systemd/certbot.service to match where certbot_data/ actually lives.
• Renewal also failed whenever it did target the right certificate, because the stored renewal config used the standalone ACME authenticator (needs port 80), but nginx already holds port 80 in production and certbot renew never stops it. The live certificate has been re-obtained via the webroot method instead, which nginx already serves via its /.well-known/acme-challenge/ location block, switching the stored authenticator for future renewals.
• A stale .certbot.lock left by a crashed run blocked every later renewal attempt, including manual clicks in the admin UI, with "Another instance of Certbot is already running" — until the app process was restarted. _ensure_certbot_directories() only cleared locks at Flask app startup. Added clear_stale_locks() in webapp/admin/certbot/paths.py, called before each renew/obtain invocation, guarded by a pgrep check so a genuinely in-progress run isn't clobbered.

[2.176.0] - 2026-08-20 - Show originating equipment on the live received-alert page

Added

• The received-alert detail page now shows which ENDEC model produced the SAME transmission. app_utils/eas_demod.py's detect_endec_mode() (a port of EAS-Tools' encoder-fingerprinting method, voting on terminator-byte signatures, a leading-null quirk, and inter-burst gap timing) already computed and stored this for every decoded alert — live and manually uploaded — but the labeled "Originating Equipment" badge only ever rendered on the manual Audio Decoder upload tool. On the actual Received Alerts detail page the same value sat unlabeled inside a raw JSON dump. The badge is now on both pages, sharing one definition (templates/components/endec_badge.html) so the labels can't drift apart the way they would have as two copies.
• Documented in templates/help.html's Received Alerts section.

[2.175.4] - 2026-08-20 - Fix update.sh silently un-stamping alembic_version on every run

Fixed

• scripts/database/recover_split_location_settings.py was rewinding alembic_version back to 20260506_split_location_settings on every single run of update.sh, no matter how far ahead the database actually was. The script exists to heal one specific historical problem: a database where the May 6th 20260506_split_location_settings migration got half-applied by an old db.create_all() fallback, leaving the old location_settings columns dropped but alembic_version never advanced. Its detection was "the old columns are gone, and alembic_version isn't literally equal to 20260506_split_location_settings" -- but that condition is also true of a database that has moved on dozens of migrations past that point, since those columns were dropped once, by that migration, and stay dropped forever after. Every update stamped the version backward regardless, discovered live on this station: alembic current reported 20260506_split_location_settings immediately after an update that had just correctly carried the database to 20260818_dead_air_per_source, 47 migrations later.
• The actual schema was never affected -- confirmed directly against Postgres that every table/column from the 47 "missing" migrations (gated_alerts, admin_users.mfa_last_totp_counter, hardware_settings.dead_air_buzzer_enabled, etc.) was genuinely present. Only the version bookkeeping was wrong, which is still a real hazard: it misleads anyone reading alembic current, and an alembic upgrade head run while mis-stamped would attempt to replay every migration since May.
• Fixed by walking the actual Alembic migration graph (ScriptDirectory.walk_revisions) to ask "is the current revision this target, or downstream of it" instead of a plain string comparison. Also fixed an adjacent latent bug in the same stamp step: it used a bare UPDATE alembic_version SET version_num = ... with no INSERT fallback, which silently does nothing on a database whose alembic_version table exists but has no row yet.
• Live database restamped to the correct head (20260818_dead_air_per_source).

[2.175.3] - 2026-08-19 - Fix RBDS History never having any data, and its unreadable chart labels

Fixed

• RBDS History always showed "No stored RBDS snapshots in this window yet.", no matter how long a station had been running. The history API (/api/audio/sources/<name>/rbds/history) and the audio-health analytics aggregator both read the audio_source_metrics table on the documented assumption that the audio service persists a snapshot roughly once a second -- but nothing in production ever wrote to that table; only tests instantiated AudioSourceMetrics directly. eas_monitoring_service.py's main loop already publishes live metrics to Redis at 4 Hz for the VU meters/RSSI/RBDS UI; it now also snapshots each running source into audio_source_metrics at ~1 Hz, on its own background thread so a slow commit can never stall that 4 Hz loop. Covered by tests/test_audio_metrics_snapshot_writer.py.
• The RBDS History chart's time-axis labels rendered at a steep diagonal and were hard to read, especially on a phone. The default toLocaleTimeString() label (e.g. "10:09:34 AM") was too wide for Chart.js to keep horizontal at the configured tick count, so it fell back to auto-rotation. Axis labels now drop seconds ("10:09 AM"), the chart requests maxRotation: 0 so Chart.js skips ticks rather than tilting them, and full-precision timestamps (with seconds) still appear in the hover tooltip title.

[2.175.2] - 2026-08-19 - Fix modals with a form wrapper not scrolling on mobile

Fixed

• The Edit Audio Source modal (and every other scrollable modal that wraps its header/body/footer in a <form>, e.g. Add Audio Source, Radio Receiver, Stream Profile) was un-scrollable on mobile, cutting the form off partway through with no way to reach the fields below or the Save button. Bootstrap's scrollable-modal CSS gives .modal-body a bounded, scrollable height via flex: 1 1 auto -- but that only works when .modal-body is a direct child of the flex container .modal-content. These modals put a <form> in between so one submit handler can collect every field, and a plain <form> is not a flex container, so the flex chain broke silently: .modal-body's overflow-y: auto had no bounded height to scroll within, and .modal-content's own overflow: hidden just clipped the tail of the form instead. The Radio Receiver modal had been individually patched around this with a hardcoded inline max-height: calc(90vh - 130px) rather than the root cause being fixed. .modal-dialog-scrollable .modal-content > form is now itself a flex column filling the bounded height, so header/body/footer behave exactly as if there were no form wrapper at all -- fixing every modal with this pattern at once.

[2.175.1] - 2026-08-18 - Fix waterfall/scope zoom controls going inert after a refresh

Fixed

• The Live Waterfall's and Spectrum Scope's zoom controls looked normal but did nothing when clicked. updateReceiverDetails()'s periodic re-render (WebSocket-driven, roughly every 1s) preserved .waterfall-container and .spectrum-scope-container across the refresh by capturing their innerHTML and restoring it afterward -- but innerHTML serialization carries neither a <canvas>'s rendered bitmap nor any addEventListener bindings on cloned elements. The restored zoom buttons and canvas gesture handlers were visually identical but completely inert clones. This is the exact same defect class that made the Historical Trends charts "disappear almost immediately" (fixed for trends in #2414 by watching for DOM replacement and rebuilding instead of innerHTML-restoring), just never generalized to the waterfall/scope panels once they grew live zoom/pan gestures. Removed both from the innerHTML-preserve mechanism entirely -- .diagnostic-container (plain text/tables, no canvas, no listeners) is the only panel type that actually needs it. The waterfall and scope already detect their own DOM replacement and rebuild themselves with fresh listeners every 500ms via liveSpectrumTick(), so nothing else had to change.

[2.175.0] - 2026-08-18 - Make dead-air monitoring per-source

Changed

• Dead-air (silence) alarming moved from one station-wide policy to a per-source setting. Shipped in 2.174.0 as a single enabled flag and threshold set applied identically to every monitored source, it could not express "alarm on silence for this continuous broadcast monitor, but never for that state-relay source that's supposed to be silent except when relaying an actual alert." Confirmed live on this station: a state-relay source (ERN-LUC) sat in a false dead-air alarm for 39 minutes straight because the station-wide policy had no way to exempt it without also disabling alarming for every source that genuinely needed it.
• Detection policy (enabled, hold-off, silence level, open-carrier sensitivity) is now configured per source in that source's Add/Edit dialog on the Audio Sources page, stored alongside the source's other settings. The station-wide "Dead Air Monitoring" settings card on that page is now a live status readout only.
• The physical output side (rack buzzer GPIO pin, tower-light colour) is unchanged: one buzzer and one light for the whole station, configured on the Hardware page, same as before.
• /api/audio/dead-air/settings (GET/POST) is removed; /api/audio/dead-air/status and /api/audio/dead-air/acknowledge are unchanged.
• Database: the five station-wide detection columns (dead_air_enabled, dead_air_level_threshold_db, dead_air_detect_open_carrier, dead_air_flatness_threshold_pct, dead_air_duration_seconds) are dropped from hardware_settings; the output columns (dead_air_buzzer_enabled, dead_air_buzzer_gpio_pin, tower_light_silence_*) are untouched.
• Fixed 7 failing tests in tests/test_gpio_alert_indicators.py that read this station's live eas:dead_air Redis key instead of a mocked one, so they passed or failed based on whatever this station's real alarm state happened to be at test time.

[2.174.0] - 2026-08-18 - Retire the carrier-squelch feature

Removed

• "Carrier Squelch" is gone from receivers. It never did what its name and help text promised. _apply_squelch() gated on the RMS of the demodulated audio, not on carrier presence, so the gate was inverted against its own purpose. Measured against the real code path at the panel's own defaults (threshold -60 dBFS, open 200 ms, hang 900 ms):

  | Input | Level | Squelch | |---|---|---| | Off air, open-carrier hiss | -6 dBFS | passed through | | Off air, hiss | -20 dBFS | passed through | | Off air, weak hiss | -40 dBFS | passed through | | Truly dead feed (digital silence) | -240 dBFS | muted |

  It muted silence -- a no-op, since muting silence produces silence -- and passed hiss, the only thing worth muting. The panel promised to "automatically mute white noise when the carrier drops", and that is precisely the case it could not handle: an off-air FM receiver emits unsquelched noise tens of dB above any usable threshold. Its only working behaviour was therefore a no-op, so nothing functional is lost by removing it.

• "Raise alarm on carrier loss" is gone with it. It wrote a log line and a metadata flag consumed by a single status badge; it drove no GPIO, no tower light and no notification. Dead-air monitoring (added in 2.172.0) supersedes it properly: it detects the open-carrier case via spectral flatness rather than level, debounces it, and drives the tower light and rack alarm buzzer with an operator acknowledgement.

• Removed across the stack: the audio-path gate, the five radio_receivers columns (migration 20260818_retire_carrier_squelch, with a downgrade that recreates them), the ensure_radio_squelch_columns startup backfill, the service-config presets, the API payload validation and serialization, and the Edit Receiver form section with its status badges.

Changed

• The legacy instantaneous silence_detected metric took its thresholds from the squelch columns, which was a coincidence of naming rather than a real relationship. It now uses AudioSourceConfig's own defaults (-60 dBFS / 5 s), which are the same values the old fallbacks used, so behaviour is unchanged. The debounced dead-air alarm was never affected -- it reads its own station-wide policy.
• Renumbered the database-initialisation log steps in app.py, which were already inconsistent before this change (twelve labelled /15 and four /16, against sixteen actual steps). They now read [N/15] across the fifteen remaining steps.

[2.173.1] - 2026-08-18 - Code-review fixes for the dead-air relocation

Fixed

• Operator-supplied source names were interpolated into innerHTML. AudioSource.name is free text with no markup validation and reaches the browser through Redis, so a source named with a tag would have executed in another operator's session on both the Audio Health dashboard and the Audio Ingestion page. Both renderers now build text nodes; verified in Chromium with an <img src=x onerror=...> payload, which renders literally and does not fire.
• An active dead-air alarm was invisible to the people meant to watch for it. The navigation registry gates the Audio Health dashboard on the view-role trio (alerts.view / receivers.view / logs.view), but the status endpoint required system.configure -- so a radio watcher could open the dashboard while every poll returned 403 and the alarm banner silently never appeared. Status is now readable at the dashboard's own level; acknowledging stays restricted, and the response carries can_acknowledge so the UI hides that control rather than offering a button that would only fail.
• An acknowledgement could mute a later, unrelated outage. The ack was a bare flag with a 24 h TTL and could be written with no alarm active. The publisher now mints an episode id when the alarm goes active and drops it on recovery; the ack stores that id, the API refuses to acknowledge when nothing is wrong or when the operator's page is showing a stale episode, and the GPIO reader only stays silent for the episode that was actually acknowledged.
• A failed settings load could silently rewrite stored thresholds. The numeric inputs are populated by JavaScript rather than server-rendered, so an empty form posted Number('') == 0, which the API clamped to each field's minimum -- turning a 20 s hold-off into 1 s. Saving is now disabled until a load succeeds, with the failure surfaced in a dedicated error line kept separate from the live alarm readout so the two cannot overwrite each other.
• Completed the page-exclusivity tests, which checked fewer fields than they claimed: dead_air_detect_open_carrier was not asserted absent from Hardware, and only the buzzer pin was asserted absent from Audio Sources.

[2.173.0] - 2026-08-18 - Move dead-air settings to where the operator looks for them

Changed

• Dead-air monitoring is no longer configured entirely on the Hardware page. It shipped there in 2.172.0 because that is where the tower-light settings already were -- placement by implementation adjacency rather than by task. Every threshold it exposes is an audio quantity (dBFS, spectral flatness, a hold-off in seconds), and GPIO is only today's output: an email or SMS notifier added later would have had to read its detection policy out of the GPIO page. The controls now split along the seam that actually matters, detection policy versus output device:

  | Control | Page | |---|---| | Enable, hold-off, silence level, open-carrier sensitivity, detect-unmodulated-carrier | Audio Ingestion (/admin/audio-sources) | | Rack buzzer enable + GPIO pin, tower-light colour / enable / buzzer | Hardware Settings (/admin/hardware) | | Live alarm state + Acknowledge | Audio Health dashboard (/audio/health/dashboard) |

  All three cross-link to each other. This is the same separation the system already makes elsewhere: the alert pipeline decides what constitutes an alert, and hardware settings only decide what colour the light goes.

• Acknowledging moved off the settings form entirely. It is an operational act taken while an alarm is sounding, not configuration, so it now lives on the Audio Health dashboard as a banner that appears only when a source is actually silent -- naming the source, the reason and how long it has been dead. Acknowledging still silences the buzzer while leaving the indication up.

• No migration: the settings columns are unchanged, only which page edits them. Detection fields are no longer writable from the hardware form, so the two pages cannot fight over the same values.

Added

• webapp/admin/audio_ingest/routes_dead_air.py -- GET/POST /api/audio/dead-air/settings for the detection policy, plus /api/audio/dead-air/status and /acknowledge. The equivalents under /admin/hardware/dead-air/* are removed rather than left as duplicates.
• Navigation entries for two pages that never had them. Neither /admin/hardware nor /admin/audio-sources was in webapp/navigation/registry.py -- "Station Hardware" is a NavGroup label, not a link, so the hardware settings page could only be reached by typing the URL or via a card on the Admin panel. Both are now proper NavItems, which is the actual fix for "I could not find this".
• Regression tests pinning the split: detection fields on the audio page and absent from Hardware, output wiring the other way round, Acknowledge only on the dashboard, all three pages cross-linking, and both pages present in the navigation registry.

[2.172.1] - 2026-08-18 - Fix undefined logger in the dead-air buzzer pin resolver

Fixed

• _dead_air_buzzer_pin() in services/gpio/__main__.py referenced a module-level logger that does not exist -- that module resolves its logger inline via logging.getLogger(__name__) at each call site. Caught by ruff (F821) in CI. Not merely a lint nit: the NameError in the except clause would have escaped uncaught, so a failure to read the buzzer pin would have taken down GPIO service startup instead of falling back to "no buzzer configured".

[2.172.0] - 2026-08-18 - Dead-air monitoring with tower light and rack alarm buzzer

Added

• Dead-air (silence) monitoring for monitored audio sources, wired to the USB tower light and an optional GPIO rack alarm buzzer. Every other health check in the system watches process liveness -- service up, Redis reachable, SDR locked, buffers healthy -- and all of those can be green while the audio itself is dead. For an EAS monitoring station a silently dead assigned source means monitoring has stopped, and until now nothing said so.
• app_core/audio/silence.py -- the detector. Classifies on two independent axes, because a level threshold alone cannot do this job:
  • level: RMS below a floor (default -65 dBFS) catches true digital silence -- a stopped file, a muted feed, a dead stream.
  • flatness: spectral flatness (Wiener entropy) catches an unmodulated carrier. This is the axis that matters for an SDR. When an FM station leaves the air the receiver does not go quiet, it outputs unsquelched noise at full scale -- measured at -6 dBFS in testing, which every level-only detector in this codebase reports as "audio present". Noise spreads energy evenly and scores 0.40-0.57; speech and music concentrate it into harmonics and score below 0.01. The default threshold of 0.25 sits in a two-to-three order of magnitude gap, so it is not a delicate calibration.
• Timing reuses SilenceDetector in app_core/audio/metering.py rather than adding a fourth parallel implementation. That class already had duration debouncing, recovery-edge detection and an alert callback fan-out, and had been dead code since it was written -- exported but never instantiated outside tests. It now takes an optional explicit verdict so the flatness axis can drive the same state machine.
• Tower light: a new silence state in resolve_tower_state(), ranked with the existing fault tier -- above every alert indication, because an alert pipeline can look perfectly healthy while the source feeding it is dead. Deliberately not suppressed by quiet hours: an overnight schedule must never hide that monitoring has stopped. Configurable colour, optional tower buzzer, and can be switched off.
• Rack alarm buzzer on a configurable GPIO pin, with an operator acknowledgement. This does not reuse the existing _key_relay_on_edges path, which is edge-triggered off the broadcast marker and backed by a 300 s watchdog sized for an alert playout; dead air is a level condition that can persist for hours, so it gets its own pin held for as long as the condition lasts. Acknowledging silences the buzzer but leaves the tower light lit -- standard alarm-panel behaviour, since an acknowledgement means the fault was noticed, not fixed. The audio service clears the acknowledgement when audio returns, so the next outage sounds again instead of starting pre-silenced.
• Admin UI at Admin -> Hardware (existing "Station Hardware" page): enable toggle, hold-off, silence level, open-carrier sensitivity, buzzer pin, tower-light colour, and a live status readout with an Acknowledge button. Controls live in a new static/js/admin/dead-air.js rather than extending hardware-settings.js, which is already well past the size guidance.
• New endpoints GET /admin/hardware/dead-air/status and POST /admin/hardware/dead-air/acknowledge.
• Redis key eas:dead_air (30 s TTL) carries the aggregate state from the audio service to the GPIO service. A missing key reads as not alarming: absence means the feature is off or the publisher is gone, and neither should strand a rack buzzer on -- audio-service liveness already has its own monitoring.

Fixed

• SilenceDetector reported silence immediately when its first observed chunk was silent, bypassing the duration debounce entirely. On a service restart, where a source simply has not produced its first audio yet, that would have sounded the rack buzzer straight away. Alarm consumers now pass assume_prior_signal=True so the hold-off always applies.

Notes

• Dead-air thresholds are a station-wide policy, installed once via set_default_criteria() and re-read every 30 s by the source watchdog, so changes in the admin UI take effect without a service restart.
• The pre-existing AudioMetrics.silence_detected flag is unchanged. It is an instantaneous per-chunk comparison with no debounce -- it flips on every pause between words -- and continues to feed the analytics silence_detection_rate only. It is not what drives the alarm.

[2.171.1] - 2026-08-18 - Code-review fixes for the spectrum zoom

Fixed

• A null frequency axis rendered as 0.00000 MHz instead of blank. build_spectrum_payload() emits freq_min/freq_max as None when the centre frequency cannot be coerced, which serialises to JSON null -- and Number(null) is 0, which passes Number.isFinite. The guard in visibleFreqRange() therefore let it through and labelled both axis edges 0.00000 MHz across a zero-width span. It now rejects explicit nulls and degenerate (zero or inverted) ranges before converting. Older services that omit the keys entirely were never affected -- undefined becomes NaN, which the finite check already caught.
• The mouse wheel swallowed the page scroll at both zoom limits. preventDefault() ran before the zoom was evaluated, so at 1x every scroll-down and at the 64x cap every scroll-up cancelled the page scroll while doing nothing, leaving the operator unable to scroll past a full-width canvas. The default is now cancelled only when the gesture actually changes the zoom.
• The Spectrum Scope's peak-hold trace clipped against the top edge. The y-axis auto-scale computed its range from the live spectrum only, but scaleY() clamps to that range and the peak-hold line retains historical maxima above it. Peaks were pinned flat at y=0 instead of the axis scaling to show them. Both traces are now included in the range.

Changed

• Corrected an overstated claim in the 2.171.0 notes and the template's header comment. Zoom recovers the ~125 Hz bin resolution the unzoomed view discards -- 2048 bins over ~900 CSS pixels is ~2.3 bins per pixel (~5.7 on a phone), dropped rather than averaged by the nearest-neighbour upscale -- so the genuine gain is ~2-6x depending on panel width, not the "~16x" originally claimed. Past one bin per pixel, zoom magnifies for legibility without adding information.
• Pan repaints are coalesced to one per animation frame. Pointer events can outpace the display and each repaint recolours up to count x 200 bins, which matters on the Raspberry Pi hardware this runs on.
• The two status lines are built by shared waterfallStatusText() / scopeStatusText() helpers instead of being assembled separately in the poll tick and the zoom refresh, so the two paths cannot drift apart.
• Rebuilt zoom controls seed their readout and reset-button state from live zoom state. The panels rebuild roughly once a second, and a hardcoded 1x contradicted the already-zoomed canvas until the next poll.
• tests/test_spectrum_frequency_axis.py reads the template with an explicit UTF-8 encoding (the platform default is ASCII under a C/POSIX locale), asserts the two renderSpectrumAxis call sites individually rather than by a count that also matched the declaration, and matches the history-buffer invariant with a whitespace-tolerant regex.

[2.171.0] - 2026-08-18 - Zoom and pan on the live waterfall and spectrum scope

Added

• Zoom and pan on the Live Waterfall and Spectrum Scope (/admin/radio_diagnostics). The SDR service already publishes a 2048-bin FFT across the whole effective span -- ~125 Hz per bin at 256 kHz -- but both views downsampled that to roughly 900 CSS pixels: ~2.3 bins per pixel (~5.7 on a phone), dropped rather than averaged by the nearest-neighbour upscale, so a one-bin spur could miss the screen entirely. Zoom crops the published bins instead of retuning, so it costs no extra SDR work and no extra requests, and recovers the full ~125 Hz bin resolution the unzoomed view discards -- a genuine ~2-6x, depending on panel width. Past the point where one bin fills one pixel, further zoom magnifies for legibility without adding information; the cap and bin floor bound that so the UI never implies resolution the FFT lacks. Controls:
  • Zoom in / out / reset buttons in each panel header, with a live magnification readout.
  • Mouse wheel to zoom about the cursor, holding whatever is under the pointer still.
  • Click-drag (or touch-drag) to pan; double-click resets to full span.
  • Capped at 64x with a 32-bin floor, so the UI cannot pretend to resolution the FFT does not have. Going finer needs a narrower capture or a larger FFT server-side. Zoom is shared between the two views -- they render the same payload, so panning one and not the other would only be confusing -- and survives the WebSocket panel re-render that rebuilds the cards roughly once a second.
• The frequency axis and status line now describe the visible window rather than the full span: edge labels gain decimal places as you zoom (up to 5 dp on a few-kHz window), the status line reads e.g. 64.0 kHz of 256.0 kHz · 125 Hz/bin, and the centre label drops its "(carrier)" annotation and switches to the window centre once panning takes the tuned frequency off screen.

Changed

• The live waterfall's history buffer now stores raw per-bin power instead of finished RGBA pixels. Cropping a pixel buffer can only stretch what was already rasterised at full-span scale, so zoomed scrollback would have been a blur of upscaled blocks. Keeping the values means every repaint re-renders the entire 200-row history at the current zoom, so scrolling back through history at 16x shows real per-bin detail. The buffer also got smaller (2048 bins x 200 rows = 400 KB, a quarter of the RGBA cost), and colour mapping moved to a precomputed 256-entry lookup table so re-colouring the full history each frame stays cheap.
• The Spectrum Scope's y-axis auto-scale now follows the visible crop rather than the whole span, so zooming in on a weak feature is not defeated by a strong carrier elsewhere keeping ownership of the axis. Its peak-hold trace is indexed by absolute bin, so zooming or panning no longer discards peaks already accumulated off-screen.

[2.170.5] - 2026-08-18 - Fix live spectrum frequency axis labelled with the pre-decimation rate

Fixed

• The Live Waterfall and Spectrum Scope on /admin/radio_diagnostics labelled their frequency axis with the receiver's configured sample rate instead of the rate the displayed FFT is actually clocked at, drawing every signal several times wider than it is. Reported from a receiver tuned to an FM broadcast station at 93.900 MHz: the axis read 93.388-94.412 MHz, so a US FM channel -- 200 kHz wide by regulation -- appeared to occupy a full megahertz, which is physically impossible. High-rate SDRs are decimated in the USB callback before samples reach the ring buffer (Python cannot process multi-megahertz IQ in real time), so the samples every spectrum view FFTs are clocked at the post-decimation effective rate. On the reported receiver, 1.024 MHz configured decimates by 4 to 256 kHz, meaning the bins covered 256 kHz of RF while the axis claimed 1.024 MHz -- a 4x overstatement, and up to 10x on an Airspy at 2.5 MHz. The root cause was a missing field: sdr_hardware_service.py published sample_rate (correctly, the effective rate) but never freq_min/freq_max, so /api/radio/spectrum/<id> always hit its fallback of receiver.frequency_hz ± receiver.sample_rate / 2 -- the hardware rate off the RadioReceiver row. The service now publishes the axis explicitly via a new testable build_spectrum_payload(), alongside hardware_sample_rate and early_decim_factor; the route resolves the axis through a new _spectrum_axis() helper that prefers the published values and recomputes the effective span locally when an older sdr-service omits them. The same fallback in the command-queue path (which defaulted to 2.5 MHz for Airspy) is fixed identically.
• Capture requests asked the SDR service for decimation-factor times more samples than the requested duration. /api/radio/diagnostics/capture, /analyze and /waterfall each named a local effective_rate but assigned receiver_record.sample_rate -- the pre-decimation rate -- to it. Because the capture tap fires after early decimation, a 5 s request against an Airspy at 2.5 MHz (decim 10) queued 50 s worth of samples while the client waited only duration + 20 s, so longer captures timed out by construction. All three now convert duration at the effective rate.

Added

• app_core/radio/decimation.py -- the early-decimation threshold, factor and effective-rate math as one dependency-free module (no numpy, SciPy or SoapySDR imports) so the web layer can label a frequency axis with the same numbers the driver decimates by. app_core/radio/drivers.py now calls it instead of carrying its own copy, which is what let the two drift apart in the first place.
• The Live Waterfall and Spectrum Scope status lines now show the RF span the FFT covers, annotated with the decimation when it is active (e.g. 256.0 kHz span (1.024 MHz ÷ 4)), so an axis that disagrees with the hardware rate is visible on screen rather than silently wrong.
• tests/test_spectrum_frequency_axis.py -- regression coverage pinning the reported case (1.024 MHz configured must render a 256 kHz span, never 1.024 MHz), the recompute path for an older sdr-service, the service-payload/route round trip, and the capture duration conversion.

[2.170.4] - 2026-08-18 - Fix SDR Diagnostics charts vanishing and running narrower than the page

Fixed

• The per-receiver "Historical Trends" charts on /admin/radio_diagnostics disappeared almost immediately after being shown, and were never as wide as their container. The page's WebSocket-driven auto-refresh (renderReceivers(), firing roughly once a second) replaces the entire receiver-card tree, and a preservedPanels mechanism cloned each open panel's innerHTML across that replacement to survive it -- which works for the Run Diagnostic / Waterfall / Spectrum Scope panels, but not for the trends panel's Chart.js line charts: a <canvas> element's drawn pixels are never serialized into its HTML, and Chart.js's inline style="width:...px" (baked in at whatever moment the clone was taken) froze the chart at a stale size instead of tracking the container. Net effect: the chart flashed onto screen and then went blank on the very next refresh tick, and any width it did retain was a snapshot from whenever it was last captured rather than the current layout. templates/admin/radio_diagnostics.html now excludes .trends-container from the clone-and-restore list entirely, and static/js/radio_diagnostics_trends.js adds a MutationObserver on the receivers container (trendsRebuildActive()) that rebuilds and refetches any trends panel left open once the WebSocket refresh clobbers it, so the chart is always constructed fresh against the DOM's current width instead of replayed from a frozen snapshot.
• The whole SDR Diagnostics page -- including the Live Waterfall and Spectrum Scope canvases -- was narrower than it needed to be on any viewport at or above the 576px Bootstrap breakpoint. The page wrapped its content in <div class="container">, which caps at 540px/720px/ 960px/1140px/1320px regardless of the actual viewport width, instead of <div class="container-fluid">, which the comparable GNSS & Time Dashboard (templates/admin/gps_dashboard.html) already uses for the same reason: chart-heavy diagnostics pages should use the full width available rather than being centered in a fixed-width column. Both canvases already sized themselves to 100% of their parent, so this wasn't a canvas bug -- the parent itself was capped.

[2.170.3] - 2026-08-18 - Fix spectrum saturation behind the "still zoomed in" waterfall/scope

Fixed

• The Live Waterfall and Spectrum Scope both still looked "zoomed in" after 2.170.2's client-side auto-scale fix -- because the underlying data was actually saturated, not just badly rendered. sdr_hardware_service.py's compute_spectrum() converted the raw FFT magnitude to dB without dividing by the window's coherent gain (sum(window)) first, so for FFT_SIZE=2048 with a Hanning window, a full-scale tone's peak bin read around +54 dB raw -- far above the fixed SPECTRUM_DB_MIN=-80/SPECTRUM_DB_MAX=0 window the normalization assumed. Confirmed against a live capture: 1781 of 2048 bins (87%) clipped to the normalized maximum of 1.0, flattening both views into a saturated "brick" with no legible spectral shape. A single fixed dB window can't hold across every receiver/gain/antenna combination anyway, so the fix replaces it with per-capture normalization: stretch this capture's own dB min/max across 0-1, floored by a new MIN_SPECTRUM_DB_SPAN constant so a near-silent capture (no antenna, receiver just started) isn't auto-amplified into apparent noise. Added tests/test_sdr_spectrum_calibration.py (7 tests).

[2.170.2] - 2026-08-18 - Fix Spectrum Scope clipping and Historical Trends chart growth

Fixed

• Spectrum Scope drew directly against a fixed 0-1 range instead of auto-scaling to the current payload's actual min/max, unlike the sparkline strip on the same page which already did this correctly. Reported live: the trace looked "too zoomed in," with FFT edge-bin DC-offset spikes pegging the top of the canvas with no headroom while the real passband signal collapsed into a thin band near the bottom. scopeRender() now computes lo/hi from the current spectrum array (with an 8% margin) each tick, the same auto-scale approach drawSparkline() already uses, and the scope's status line now shows the current auto-scaled y-axis range so operators can see it's adaptive rather than a fixed scale.
• Historical Trends charts grew without bound ("kept scrolling"). Chart.js's responsive: true + maintainAspectRatio: false (used for all four trend charts) requires the canvas's parent element to have an explicit, bounded CSS height -- the canvas's own height="120" HTML attribute only sets its intrinsic pixel resolution and is ignored once responsive mode takes over. The charts sat directly in unconstrained .col-md-6 grid cells, so each one could resize upward indefinitely. Wrapped each chart canvas in a position:relative; height:220px; div, the pattern Chart.js's own docs require for this configuration.

[2.170.1] - 2026-08-18 - Fix a silently dropped IPAWS shelter-in-place alert

Fixed

• CAPPoller._convert_cap_alert() never extracted <eventCode> from IPAWS-sourced CAP XML. Reproduced against two live alerts from the same sender:

  • A county "Natural gas leak" shelter-in-place warning (severity Extreme, urgency Immediate, instruction "Shelter in place.") carried a correct <eventCode><valueName>SAME</valueName><value>SPW</value></eventCode> block in its raw CAP XML, but properties['eventCode'] was never populated, so the event code was unresolvable and the alert was silently rejected by the auto-forward allowlist ("Event 'UNKNOWN' is not in the configured forwarding allowlist") even though SPW itself was already on the allowlist. It was never broadcast.
  • Three days earlier, an alert whose event field read exactly like a routine "Severe Thunderstorm Warning" (matching headline, radar- indicated hail/wind description, "prepare to seek shelter" instruction) but whose eventCode explicitly said SAME=SPW was actually broadcast as ZCZC-CIV-SVR-039137+0130-2212336-KR8MER- (event code SVR) -- a shelter-in-place alert masquerading as a severe thunderstorm warning, because the missing eventCode meant name-based resolution took over.

  The IPAWS XML parser built <parameter>, geocode, and <resource> extraction from the CAP <info> block, but had no equivalent for <eventCode> at all. Added CAPPoller._extract_cap_event_codes() (mirrors the existing _extract_cap_parameters() pattern) and wired it into properties['eventCode'].

• app_utils.eas._collect_event_code_candidates() silently dropped a dict-shaped eventCode ({"SAME": ["SVR"], "NationalWeatherService": ["SVW"]} -- the shape both NWS's CAP-JSON feed and the now-fixed IPAWS XML parser produce). It only handled flat strings/lists, so resolution always fell back to matching the alert's plain-English event name instead of the source-supplied SAME code. Caught while investigating the alert above: NWS "Severe Thunderstorm Warning" VTEC continuation statements (eventCode.SAME=SVS, meaning "statement/update, not a fresh warning") were resolving to SVR by name instead of SVS by code. A separate, correct VTEC-action check already suppressed those from broadcast, so this hadn't caused a visible problem, but it's a real defect in the resolver itself.

• app_core.audio.auto_forward._resolve_event_code() was an independent, weaker duplicate of the same resolution logic, checking neither the eventCode block nor the event-code registry's aliases -- only an exact match against the CAP event name. This is the function that actually gated the forwarding-allowlist check for the dropped alert above; fixing the parser and the dict-handling bug wouldn't have been enough without also fixing this. Replaced its body to delegate to the shared resolver so the SAME-header-generation path and the allowlist-check path can never resolve an alert differently again.

• Added tests/test_ipaws_event_code_extraction.py (19 tests) covering all three fixes, including a replay of both real-world alerts.

[2.170.0] - 2026-08-18 - Strengthen SDR Diagnostics: spectrum scope, trend charts, full checklist

Fixed

• The Live Waterfall's fast path was silently broken. sdr_hardware_service.py computed and published a spectrum FFT to Redis every 100ms per receiver, but under its own ad-hoc key (sdr:spectrum:{id}) that never matched what webapp/radio_settings/routes_signal.py read (eas:spectrum:{id}, the key RedisChannels.SPECTRUM_PREFIX actually defines). Every spectrum poll — including every waterfall tick — missed the Redis cache and fell through to a slow ~5s command-queue round trip instead. Both sides now import RedisChannels.SPECTRUM_PREFIX from app_core/config/redis_config.py as the single source of truth.

Added

• Spectrum Scope, a classic frequency-vs-power line trace with a peak-hold overlay, next to the existing Live Waterfall on /admin/radio/diagnostics. Shares the waterfall's 500ms poll and Redis spectrum feed rather than opening a second one — running both at once for the same receiver costs no extra requests.
• Historical Trends: a new 2-tier Redis archive (app_core/radio/trends.py, modeled on services/gps/trends.py's bucket/rollup pattern — raw 10s samples capped at 1h, 5-minute rollups capped at 7d) tracks signal strength, lock percentage, sample-rate health, and ring-buffer overflow/underflow per receiver, sampled from sdr_hardware_service.py's existing publisher loop on its own 10s throttle. Served by GET /api/radio/diagnostics/trends/<id>?window=1h|6h|24h|7d (webapp/radio_settings/routes_trends.py) and rendered as four Chart.js line charts behind a new "Historical Trends" toggle, plus an always-on compact sparkline strip (signal strength, buffer drops) on every receiver card.
• "Run Full Diagnostics" button: a pass/warn/fail checklist (app_core/radio/diagnostics_report.py) covering the SDR service heartbeat, per-receiver running/locked/sample-flow status, ring-buffer drops, and spectrum-cache freshness, read from the live Redis state the separated eas-station-sdr.service process publishes. scripts/diagnostics/check_sdr_status.py is now a thin CLI wrapper around the same shared function (previously it read a local, almost-always-empty in-process RadioManager left over from the pre-separated-architecture design, so most of its checks had gone dead).

[2.169.2] - 2026-08-18 - Fix the EAS decoder stream and VU meters not following live audio

Fixed

• /api/eas/decoder-stream ("Listen to EAS Decoder Feed") produced zero audio bytes. Reproduced live: a 15+ second connection to the endpoint received no data at all, despite the server correctly subscribing to every running source and starting its ffmpeg encoder. eas_monitoring_service.py runs a continuous, numpy-heavy, CPU-bound monitor loop (SAME/FSK decode across every audio source) that holds Python's GIL for long stretches under Python's default 5ms switch interval — long enough to starve the decoder stream's I/O-bound ffmpeg-feeder thread of the scheduling it needs to keep the encoder fed in real time. Observed live: this process's CPU spiked well past 100% the moment a stream request came in. Added sys.setswitchinterval(0.001) near the top of the service so the interpreter hands off the GIL more often.
• VU meters didn't follow the actual playing audio. Three separate code paths in templates/audio_monitoring.html were all writing to the same peak-meter-*/rms-meter-* bar and peak-label-*/rms-label-* text elements: the current 60Hz realtime-vu-meters.js system (a Web Audio API tap on the actual <audio> element — the only one that reflects what's really audible), plus two older systems (updateLevelMetersFromSnapshot(), fed by a WebSocket push as often as every 250ms, and updateMetricsOnly(), fired every 30s) that wrote server-reported metrics computed from the same decode pipeline affected by the GIL-starvation bug above. The faster, more frequent WebSocket path in particular kept stomping the accurate live values with a slower/staler snapshot. Trimmed both older functions down to only the work that isn't already owned by the client-side tap — the PEAK/RMS cards and the RF RSSI meter, neither of which the Web Audio API can derive on its own — and removed the now-dead EMA-smoothing state and constants that existed solely to feed the redundant bar/label writes.
• Investigated general per-source live audio playback (item reported alongside the above): backend tested healthy on live verification (Icecast mounts serve real audio both directly and through nginx's existing proxy rule) — a prior nginx routing fix for exactly this symptom (/etc/nginx/sites-available/eas-station's per-source .mp3 mount proxy block) is already deployed and working. No further backend changes made for this one; if still reproducible after this update, it needs browser-side detail (which source, console error) to pin down further.

[2.169.1] - 2026-08-18 - Fix three efficiency problems found in a performance audit

Fixed

• SDR service polled Redis 10x/second forever for a queue that's empty 99.9%+ of the time. sdr_hardware_service.py's main loop called process_commands() (a non-blocking LPOP on sdr:commands) every 100ms regardless of whether any command was pending — SDR commands (device discovery, receiver start/stop) are rare, human-triggered admin actions, so this spent on the order of 864,000 wasted Redis round-trips per day. Switched to a blocking BLPOP with a short timeout so the process sleeps until a command actually arrives instead of polling.
• Audio service's metrics loop silently ran at half its documented rate. eas_monitoring_service.py's main loop is commented "publish metrics at 4 Hz" (metrics_interval = 0.25) but only woke up every 0.5s, so the 0.25s threshold could never be checked more often than every 0.5s — the loop actually ran at 2Hz, halving how fresh VU meters/RSSI/RBDS updates reaching the UI were. The 0.5s sleep was sized for a "check for commands every 500ms" step that no longer exists in this loop (removed in 2.168.3). Shortened the sleep to 0.1s so the 4Hz target is actually achievable.
• Two admin routes recalculated intersections with a per-boundary query loop instead of the already-existing batched query. /admin/ calculate_all_intersections re-fetched the entire boundaries table on every alert iteration and ran a separate PostGIS query per alert×boundary pair — with N alerts and M boundaries, N×M individual round-trips instead of N. /admin/calculate_intersections/<id> and /admin/ calculate_single_alert/<id> had the same one-query-per-boundary pattern for a single alert. All three now call calculate_alert_intersections() (app_core/alerts.py), the single-batched-query helper the other three routes in the same file already used — this was a duplicate, unmodernized implementation left behind, not a new algorithm.

[2.169.0] - 2026-08-17 - Fix broken VTEC chain links, add a lifecycle view to the alert trail

Fixed

• VTEC chain links silently dropped for alerts with an unparsed vtec_action. poller/cap_poller.py::_insert_new_alert() only called _mark_vtec_chain_superseded() when new_alert.vtec_action was both truthy and not 'NEW' — a follow-on product whose action code failed to parse (falsy) skipped linking entirely, with nothing to ever retry it. The guard now only excludes actual 'NEW' issuances.
• vtec_year came back NULL for any product whose VTEC segment carries the all-zeros "ongoing" sentinel (000000T0000Z) on both begin and end times — seen on every KIWX Flood Warning CON/EXT product in the live database. app_utils/vtec.py::extract_vtec_identity() only tried the VTEC times themselves; with neither usable, the chain key (office+phenomenon+significance+etn+year) is incomplete and _mark_vtec_chain_superseded() correctly refuses to link it — by design, but with nothing else to supply the missing year, the link was simply never made. Now falls back to the year of the CAP envelope's own sent timestamp when both VTEC times are the sentinel.
• Added a self-healing sweep the poller runs every poll cycle — repair_vtec_chain_gaps() — that finds any alert with complete VTEC identity, no superseded_by_id, and a newer sibling in the same event chain, and links it. Covers any future gap regardless of cause. It only ever fills a NULL; an existing link is never touched. Windowed to the last 30 days for cost.
• Audit against the live database found these bugs left roughly a third of cap_alerts VTEC chains with at least one broken link — alerts that should have been hidden as superseded on /alerts were instead lingering as if still current. A new migration, 20260817_repair_vtec_chain_gaps, re-derives vtec_year for the ~41 affected existing rows from their stored raw_json using the fixed parser, then backfills every superseded_by_id gap that newly makes linkable (plus any other pre-existing gap) using the same logic as the original 20260402_backfill_vtec_superseded_alerts migration.

Added

• Event Lifecycle panel on the alert trail page (/alerts/<identifier>/trail). Previously the trail page only showed the pipeline events for one specific CAP product (one NEW, EXT, CON, or CAN), so following a warning through its updates meant finding each update's row on /alerts one at a time. app_core/alert_trail.py now queries every product sharing the alert's VTEC identity directly (not by walking superseded_by_id, so it renders correctly even where that column has a gap) and the trail page shows them all as one clickable timeline, with the current state and the entry you're viewing both marked.

[2.168.3] - 2026-08-15 - Follow-up cleanup after the duplicate-decoder retirement

Fixed

• Orphaned Redis publish loop. eas_monitoring_service.py's _redis_publisher_monitor_loop() existed solely to feed audio to the now-deleted eas_service.py; it had zero subscribers left but kept running a thread every cycle regardless. Removed it along with app_core/audio/redis_audio_publisher.py, the module it exclusively depended on (confirmed via repo-wide grep — no other caller existed).
• No-op command stub. eas_monitoring_service.py's process_commands() was a leftover no-op from the same "separated architecture" migration (# Do nothing - all SDR hardware access is in sdr-service.py), still called every 250ms in the main loop for no effect. Removed the function and its call site.
• Wrong systemd unit name in three diagnostics scripts. scripts/collect_sdr_diagnostics.sh, scripts/diagnostics/check_sdr_status.py, and scripts/diagnostics/diagnose_audio_chain.py all referenced a unit named eas-station-sdr-hardware.service, which has never existed — the real unit is eas-station-sdr.service. Every SDR status/log check these tools printed has been silently returning "unit not found" since they were written.
• Stale references to the already-retired eas-station-hardware.service (superseded by eas-station-hardware.target in the Phase 4 hardware split, well before this cleanup) in .vscode/tasks.json and .vscode/VSCODE_SETUP.md, alongside a hardware_service.py file path that no longer exists in either doc.
• Added an explanatory comment to uninstall.sh's legacy eas-station-ipaws-poller.service/eas-station-noaa-poller.service stop lines (pre-unification poller units), matching the pattern already used for the other legacy-unit cleanup blocks in the same script.

[2.168.2] - 2026-08-15 - Retire the duplicate EAS decoder (eas_service.py)

Fixed

• Two independent processes were decoding the same live EAS/SAME audio. eas_service.py (eas-station-eas.service) ran its own EASMonitor subscribed to the same Redis audio stream eas_monitoring_service.py (eas-station-audio.service) already decodes via its "V3 Unified EAS Monitor" (app_core/audio/eas_monitor_v3.py). Git history shows this was leftover from a "3-tier separated architecture" experiment (2025-12-05) that the project's own maintainers reversed three days later (2025-12-08, "BREAKING: Clean service rename - remove confusion, fix architecture" — which merged EAS monitoring back into what became eas_monitoring_service.py). eas_service.py was never removed when that reversal happened, and both processes have been running in production ever since, with hacky heartbeat-based coordination logic just to stop them fighting over the same eas:metrics Redis key. create_fips_filtering_callback() forwards a FIPS-matched alert toward the broadcast pipeline before any storage-level dedup runs; since a SAME header repeats 3x over ~3 seconds and both processes decoded independently with no lock between them, there was a real (if narrow) double-broadcast race window. eas_monitoring_service.py is now the sole EAS decoder.
• Deleted eas_service.py, systemd/eas-station-eas.service, and app_core/audio/redis_audio_adapter.py (used only by the retired service); removed the unit from eas-station.target's Wants= list and from EAS_SERVICES in app_core/config/services.py. update.sh now actively stops/disables/removes the leftover unit on existing installs, mirroring the existing legacy-hardware-service retirement pattern.
• Corrected several docs that had drifted from reality around this area: app_core/audio/README_EAS_MONITORS.md didn't mention eas_monitor_v3.py at all; docs/audio/EAS_TEST_SIGNAL_PIPELINE.md documented a two-process pipeline that no longer exists; docs/architecture/SDR_SERVICE_ARCHITECTURE.md hedged with "eas-station-eas.service (or -audio.service)" instead of stating the actual single decoder plainly.

[2.168.1] - 2026-08-15 - Fix a DB session leak in the CAP poller's error handling

Fixed

• A single DB error during broadcast forwarding could wedge the poller's session, silently stalling all future alert processing. CAPPoller reuses one long-lived db_session across its entire process life. Three except blocks in poller/cap_poller.py — around auto_forward_cap_alert(), process_intersections(), and poll_and_process()'s own top-level catch-all — logged the failure but never called db_session.rollback(). A DB error there (a failed commit inside auto-forwarding, a PostGIS error during intersection calculation) left the session's transaction aborted, so every subsequent statement raised InFailedSqlTransaction until a manual service restart, even though the poller's own retry/backoff loop kept running. Added rollback() to all three, matching the pattern already used elsewhere in the same file; the already-committed alert row can't be lost since it's saved earlier in the same method, before any of these three blocks run.

[2.168.0] - 2026-08-15 - Require a shared-secret token on the hardware subsystem APIs

Added

• Shared-secret auth on the network/zigbee/gps/displays hardware services. Ports 5101-5104 bound 0.0.0.0 with no application-layer auth, so anything on the LAN that could reach them (a firewall misconfiguration, a non-bare-metal install, a future reverse-proxy slip) could rewrite WiFi credentials, the hostname, GPS wiring, or open the Zigbee pairing window with no credentials at all. app_core.config.get_hardware_service_token() derives a shared token from SECRET_KEY (already required, already loaded by every subsystem process) and services.common.bootstrap. install_service_auth() requires it via X-Hardware-Auth on every route but /health. The GPIO subsystem has no HTTP command surface beyond /health, so it's unaffected. docs/troubleshooting/FIREWALL_REQUIREMENTS.md documents this as a second line of defense, not a replacement for keeping those ports off the LAN.

Fixed

• OLED screen-editor preview used a hardcoded 128x64 panel size. render_oled_elements_preview() now reads the operator's actual configured panel dimensions from HardwareSettings, falling back to 128x64 only when no DB/app context is available (matches the client-side /screens canvas thumbnail fix below).
• OLED preview rendering could race under concurrent requests. The preview builder patched app_core.oled.i2c/ssd1306 — shared module globals — with unittest.mock.patch, which isn't safe when two preview requests overlap inside the same gunicorn/gevent worker; a losing greenlet could leave the real driver reference clobbered by a stale mock for the rest of the process's life. Replaced with a controller built via __new__ plus a _NullOLEDDevice stand-in that touches no shared state.
• /screens list-page cards showed raw, unresolved template syntax ({status.status}, {gps.lat}, etc.) instead of realistic sample values. The page now hydrates each card from the existing GET /api/screens/<id>/preview endpoint (the same resolution path the screen editor's live preview already used) instead of rendering the raw template_data directly.
• /screens OLED canvas thumbnails were hardcoded to 128x64 client-side, same as the server-side preview bug above — buildOledElementsPreview() now sizes the canvas from the hardware status bar's live OLED width/height instead of a fixed 128x64.

[2.167.0] - 2026-08-14 - Make alert-dedup windows configurable, add an audio-ingest confidence floor

Added

• Configurable cross-source deduplication windows. The two suppression windows in is_duplicate_broadcast() — 15 minutes for the FIPS-set-only match, and 24 hours (1440 minutes, previously a hardcoded HEADER_KEY_DEDUP_WINDOW_MINUTES constant) for the callsign-independent SAME-header-key match — are now EASSettings.cross_source_dedup_minutes and EASSettings.header_key_dedup_minutes, editable from Admin → EAS Settings under a new "Deduplication & Audio Detection" card. The old module constants remain as fallback defaults only.
• Minimum confidence floor for headerless audio detections. EASSettings.min_log_confidence_percent (0–100, default 0 = disabled) lets an operator stop low-confidence, no-decoded-header audio detections from being logged on the Received Audio Alerts page at all. The gate only applies when there is no decoded event code — a real decode (e.g. from a nationwide EAS relay network correctly reporting an alert outside this station's coverage area) is always stored regardless of confidence.
• New idempotent migration 20260814_eas_dedup_settings adds the three columns to eas_settings.

Fixed

• Reviewing a user-reported spike of "false decodes" on the Received Audio Alerts page traced most of the volume to a correctly-configured nationwide EAS relay stream legitimately decoding out-of-area alerts — not a bug. The remaining genuine noise (headerless, low-confidence detections) is now suppressible via min_log_confidence_percent instead of requiring a code change.

[2.166.0] - 2026-08-14 - Bring the screen editor UI up to date with the OLED/VFD/LED graphics engines

Added

• LED graphics/Dots mode in the screen editor. A "Message Type" toggle (Scrolling Text / Graphics) on the LED options panel switches the canvas to the sign's real 160×16 Picture File resolution and the element palette to the icon/text/rectangle/hline/vline/bar vocabulary render_led_elements() actually supports — previously the editor could only ever emit the legacy 4-line {lines, color, mode, speed, font} shape, so a custom LED graphics screen could not be built through the UI at all; only the 3 migration-seeded ones existed.
• compass and segments element types, and VFD added to icon/gauge/circle's available displays — the editor's element palette had drifted behind the backend all session: VFD's icon and gauge support, and both engines' compass, were invisible in the UI even though app_core/oled.py and scripts/vfd_controller.py already rendered them.
• Pixel-accurate live preview. POST /api/screens/preview renders a draft screen's actual template_data (unsaved edits included) through the same Pillow pipeline that drives real hardware — render_oled_ elements_preview() (new), render_vfd_elements_preview(), render_led_ elements_preview() / render_led_preview() — instead of the editor's own client-side canvas approximation, which the Preview button now only falls back to if the server render is unavailable.
• Icon picker now includes satellite, gps_pin, bolt (added to the shared icon set earlier this session but never added to the editor's ICON_NAMES list).
• The text element's Font Size dropdown is now display-aware: VFD and LED graphics mode only recognise a literal "large" value (their 14pt/15pt bold hero size) — offering OLED's 5-size list there let an operator "pick" a size with no effect on the real device.

Fixed

• _render_led_elements() (scripts/screen_renderer.py) silently dropped dotted_hline elements entirely (no matching case) instead of folding them to a plain hline the way render_vfd_screen() already does — hit as soon as the editor's shared H-Divider element (Dotted checkbox on) became reachable in LED graphics mode.

[2.165.0] - 2026-08-14 - Give the LED sign a graphics engine; seed its default screens for the first time

Added

• LED graphics/Dots mode. render_led_elements() (scripts/led_sign_controller.py) composes a 160×16 bitmap from text/icon/rectangle/hline/vline/bar elements — reusing the same _ICON_RENDERERS glyph set and max_width/overflow truncation the OLED/VFD engines use — and Alpha9120CController.render_frame() pushes it as one M-Protocol Picture File frame via send_dots_graphic(). The sign's real graphics capability existed at the protocol layer and was only reachable manually from the Dots tab; it's now wired into the screen-template/rotation system the way OLED and VFD's icon/gauge/bar vocabulary already was.
• 3 new LED graphics screens — led_status_graphic (clock, large time, small active-alert count; leads the rotation), led_alert_graphic (large active-alert count in red), and led_system_graphic (large CPU percentage plus MEM/DSK) — each a single hero row, since the 160×16 canvas is half the VFD's 32 rows and a quarter of the OLED's 64.
• LED sign Memory Configuration allocation. A file's type (TEXT/STRING/DOTS) is fixed when it's allocated in the sign's Memory Configuration table, and nothing in this codebase had ever allocated one — send_message()'s text file relied entirely on whatever the sign shipped with from the factory, almost certainly no DOTS file at all. Added Alpha9120CController.configure_graphics_memory() (confirm-gated, destructive — erases every message stored on the sign) and an Admin-only /api/led/configure_graphics_memory route plus a "Sign Memory" panel on the LED control page's Dots tab, so this is a deliberate one-time action against a specific physical sign, never something run automatically.
• LED elements preview — render_led_elements_preview() (services/displays/preview_render.py) renders graphics-mode screens as an authentic glowing amber/red/green dot-matrix PNG at the sign's native 160×16 resolution, reusing render_led_elements() so the preview can never drift from the real hardware output.

Fixed

• The LED sign's default screens were never actually seeded. scripts/create_example_screens.py defines 6 default LED screens and a led_default_rotation, but no Alembic migration had ever inserted them — production had zero rows at display_type='led'. This release seeds them for real, for the first time, the same way 20260814_upgrade_vfd_screens.py did for the VFD.
• LED rotation froze solid during an active alert — the same bug fixed on the VFD earlier today. led_default_rotation.skip_on_alert made _check_led_rotation() return before ever reaching the rotation loop whenever a CAP alert was active, freezing the sign on whatever was last drawn; the LED has no OLED-style alert-preemption path, so led_alert_summary (existing scrolling-text screen, unmodified here) had never actually been shown during a real alert. Seeded with skip_on_alert=false from the start.

[2.164.0] - 2026-08-14 - Show active alerts on the VFD; bring it to screen parity with the OLED

Added

• vfd_status — a new default VFD screen showing the time, date and active-alert count together; leads the rotation as the at-a-glance home screen.
• vfd_alert_status — a new default VFD screen showing the active CAP alert's event type and area, mirroring oled_alert_summary's fields (same /api/alerts data source).
• vfd_gpio_status, vfd_eas_decoder, vfd_audio_health, vfd_ipaws_poll_watch, vfd_receivers — VFD counterparts of the 5 OLED screens that had no VFD equivalent, bringing the VFD's default screen count to 11, matching the OLED. vfd_eas_decoder uses the gauge primitive for its health score — part of the shared graphics vocabulary since the last release, but not used on any VFD screen until now.
• Text truncation on the VFD (max_width/overflow, "trim" or "ellipsis") — ported from the OLED engine's _fit_text_to_width(). The VFD's render_vfd_elements() had no text clipping at all before this; a long alert event name or county list just drew straight past the 140px edge.
• A second, larger VFD font ("font": "large", 14pt bold, vs. the flat 7px base) — the OLED has a small/medium/large/xlarge size hierarchy that gives its screens real visual weight; the VFD had one size for every row on every screen. vfd_status (the clock), vfd_gpio_status (active relay count), vfd_eas_decoder and vfd_audio_health (health percentages), and vfd_ipaws_poll_watch (new-alert count) now each lead with one large hero value instead of a wall of same-sized text.

Fixed

• VFD rotation froze solid during an active alert. None of the VFD's default screens showed whether a CAP alert was active — not because no screen tried, but because vfd_default_rotation.skip_on_alert made _check_vfd_rotation() return before ever reaching the rotation loop whenever _has_active_alerts() was true, leaving the panel frozen on whatever was last drawn for the alert's whole duration. The OLED has its own dedicated scroll-text alert-preemption path for this; the VFD had no equivalent. skip_on_alert is now false for the VFD rotation, so it keeps cycling — including through the new vfd_alert_status screen — while gated pending alerts (a separate, still-preempting concept) are unaffected.
• vfd_system_meters' DSK row clipped 1px off the bottom of the 140×32 canvas (text drawn at y=26 with a 7px font bottoms out at y=33). Retimed the three meter rows to y=9/17/25.

[2.163.0] - 2026-08-14 - Fix OLED element collisions; give the VFD the same graphics engine as the OLED

Fixed

• OLED element collisions and clipping, found by rendering every default screen at real size (128x64) instead of eyeballing the template JSON:
  • The compass primitive's cardinal labels sat on top of its own tick marks and needle. Labels now render outside the ring, and the needle is shortened so it can never reach them; the GPS Status screen's compass and text columns were repositioned to fit.
  • AUDIO HEALTH and IPAWS POLLER's header banners overlapped their own right-aligned stat text at small font — the audio one duplicated the Score row already shown below (dropped), the poller's title was shortened (IPAWS POLLER → POLLER).
  • The GPIO Status header's "{count} active" overlapped its title; shortened to a bare count, matching every other screen's convention.
  • The Clock Face's IP address was squeezed into a ~60px column next to the date and truncated to "192.168." or worse; moved to a full-width footer row (the analog clock face shrank slightly to make room).
  • The satellite icon (rectangles + ellipse + diagonal line) was illegible below ~16px; redrawn as a bolder body-dot-with-spokes glyph that reads clearly at 9px.
• VFD custom screens were completely non-functional. Both render paths called vfd_controller.clear_display() — a method that does not exist on NoritakeVFDController (the real method is clear_screen()) — raising AttributeError on the very first command of every render. Underneath that, every text element also called draw_text() with its arguments in the wrong order ((text, x, y) instead of the real (x, y, text) signature), which would have silently garbled positioning even once the first bug was fixed. Both are pinned by regression tests that fail against the original code and pass against the fix.

Added

• The VFD display now has the same icon/gauge/compass/bar-chart engine as the OLED, plus a VFD-only segments primitive (a classic segmented LED VU-meter look — discrete lit/unlit blocks instead of one continuous fill). The Noritake GU140x32F-7000B already supported full bitmap pushes via draw_bitmap(), but the template system only ever drove it through discrete GU-7000 primitive commands (text/rectangle/line). scripts. vfd_controller.render_vfd_elements() is a new hardware-independent PIL renderer — the same pure function NoritakeVFDController.render_frame() uses to build the real bitmap and the web preview uses to render an accurate PNG — reusing the OLED's icon glyph library directly rather than a second copy of the drawing code. Switched the default font from 10px to 7px (matching the panel's native 5x7/7x10 sizes) since 10px only left room for ~2 text rows on the 32px canvas.
• The 3 default VFD screens were reflowed onto the new engine, and a 4th — GPS Status, with a heading compass — was added for parity with the OLED's flagship screen: System Meters (bar meters), Audio VU Meter (segmented meters), GPS Status (compass), Network Status (icon+banner). All 4 are seeded if missing rather than only updated if already present — some installs never had them (no VFD hardware attached at setup time).
• README screenshot gallery: pixel-accurate renders of all 11 OLED screens and all 3 VFD screens (authentic blue-green phosphor glow), generated directly from the driver code rather than mocked up.

[2.162.0] - 2026-08-14 - Add a GPS OLED screen and richer display graphics

Added

• New OLED render primitives: compass (an N/E/S/W heading dial with an optional needle — omit/null heading for a bare dial when there's no fix yet) and bars (a multi-value vertical bar chart, e.g. one bar per visible satellite's SNR — distinct from the existing single-meter bar). Both are implemented in app_core/oled.py's render_frame(), mirrored in scripts/screen_renderer.py for DisplayScreen template support (with live-data binding: heading resolves a {var.path} template to a float, values_source resolves a dot-path directly to a live array rather than a single scalar), and previewed in the /screens admin builder's canvas.
• Three new OLED icon glyphs: satellite, gps_pin, and bolt (lightning bolt, for GPIO/relay activity), added to the existing hand-drawn vector icon library alongside antenna/shield/warning/etc. bolt closes a gap where the /screens admin builder's preview already had a glyph mapped for it but no backend renderer existed.
• New GPS status OLED screen ("GPS Status") added to the default OLED rotation: a heading compass, fix-quality badge (NO FIX / 2D FIX / 3D FIX), lat/lon/altitude/speed readout, and a per-satellite signal-strength bar chart. Sourced from a new public-but-local-only /api/gps_status endpoint (webapp/admin/api/routes_system.py), which flattens the gps:status Redis key GPSManager already publishes — same tier as /api/system_status in the deny-by-default gate (app.py's LOCAL_API_GET_PATHS), since GPS coordinates describe the machine's physical location and must not be internet-public.
• Restyled the GPIO Status OLED screen to match every other default screen's icon+banner+divider design language. It was the one screen still on the legacy plain-text lines format, and its header used invert: true with no filled rectangle behind it — which renders as black-on-black, i.e. invisible.

Fixed

• A migration seeding a new screen into an existing screen_rotations row used a raw sa.text() UPDATE to write the JSONB screens column from a plain Python list of dicts — psycopg2.errors can't adapt type 'dict', since a text() bind parameter carries no column type information. Switched to the table's typed update() construct, which serialises JSONB correctly. Caught by running the migration against a real clone of the production database rather than relying on py_compile and the single-head check alone — this exact append-to-existing-rotation code path had never actually been exercised by a passing CI run before.

Added

• Pending Alerts now show on the USB tower/stack light. A non-empty gated-alerts review queue (GatedAlert rows with status="pending") now drives a new gate_pending tower-light state — configurable color (default blue, flashing) — ranked below an active/incoming alert (a live alert can never be visually buried) but above quiet hours (an operator action item is never silenced by a dark schedule). New settings: tower_light_gate_pending_enabled / tower_light_gate_pending_color in Admin → Hardware Settings.
• Website navbar stack light mirrors the same state, subscribing to the existing pending_alerts_update WebSocket push (with a polling fallback) so the on-page indicator and the physical tower light never drift apart.
• Pending Alerts now show on the OLED, LED sign, and VFD displays. A new "Pending Review" scene rotates in — showing the queue depth and the top few headlines/event codes — whenever the queue is non-empty, and disappears entirely at zero. Ranked below an active/incoming alert (which always preempts it) but above the normal screen rotation. The web preview/control pages (/displays) now also surface the pending count.

Fixed

• Audio service could silently stop reporting itself as healthy for hours, showing "Audio service metrics are unavailable" / "source failed to start" on the Live Audio page even though audio was still streaming. Root cause: AutoStreamingService._monitor_loop() (app_core/audio/auto_streaming.py) called remove_source() for a stopped source while still holding self._lock from an outer with block; remove_source() re-acquires that same non-reentrant threading.Lock(), so the monitor thread deadlocked against itself. Every other caller of get_status() — including the main loop that publishes the Redis heartbeat the web UI's health status is read from — then blocked forever on the same lock. Fixed by collecting the names to remove while holding the lock and calling remove_source() after releasing it, matching the pattern already used a few lines above for the EAS-ingest-stream cleanup step. Regression test: tests/test_auto_streaming_lock.py.

[2.160.0] - 2026-08-13 - Fix silently-broken automatic boundary/intersection calculation

Fixed

• Root cause found for "boundaries aren't calculated automatically, I have to run a command": every raw-SQL intersection query bound alert.geom directly as a parameter. psycopg2 has no adapter registered for geoalchemy2.elements.WKBElement — the type the ORM returns for any already-persisted geometry column — so this raised psycopg2.ProgrammingError: can't adapt type 'WKBElement' on every call, silently caught and logged as a non-fatal error (by design, so a PostGIS hiccup can never block saving or forwarding an alert). This affected:
  • poller/cap_poller.py::process_intersections() — the automatic, ingest-time calculation for every new/updated CAP alert.
  • poller/cap_poller.py::_has_geometry_changed() — silently defeated an optimization (every update was treated as "geometry changed").
  • app_core/alerts.py::calculate_alert_intersections() — used by the Fix Intersections / Recalculate All Intersections admin buttons (webapp/admin/intersections.py).
  • Confirmed against a real PostGIS database: identical failure with the pre-fix code, resolved by stringifying the geometry (str(alert.geom) — its EWKB hex encoding, which PostGIS parses natively as raw SQL text) before binding. The admin routes built on SQLAlchemy's expression language (func.ST_Intersects(alert.geom, ...) inside a db.session.query(...)) were never affected — SQLAlchemy applies the Geometry column's bind processor for those automatically, which is presumably the "command" that was working.

Added

• Boundaries now auto-recalculate on upload. Uploading a GeoJSON or shapefile boundary (webapp/admin/boundaries.py) now automatically recalculates intersections for active (unexpired) alerts, instead of requiring a manual "Fix Intersections" click afterward.
• Background retry sweep for intersection calculation, mirroring the existing forwarding-decision catch-up sweep: poller/cap_poller.py's poll loop now also retries intersection calculation (within a 24-hour window) for any recent alert that has geometry but no intersection rows yet — closing the "a transient failure was silently swallowed and never retried" gap now that the underlying calculation itself is fixed.

[2.159.0] - 2026-08-13 - Statistics page: remove fabricated metrics, add gated-alert stats

Added

• New Gated Alerts section on /stats: a headline "Alerts Gated" count, plus a Pending / Auto-Released / Approved Early / Cancelled breakdown, backed by a new webapp/public/stats_sections/gating.py section querying the gated_alerts table added in 2.158.0.

Removed

• "Days Online" stat card — it was never backed by real data. The number came from a per-browser localStorage value the user had to type in manually (defaulting to a hardcoded 2024-01-01 on first visit), not any server-side install or first-alert date. Removed the card, its JS, and the now-orphaned "System Start Date" settings panel that existed only to let someone manually correct the fake number.
• Dead lifecycle_timeline placeholder — computed on every page load and threaded all the way into the client-side statsData object, but never read by any chart or table ("a chart that is not wired up yet," per the original comment). Removed from both the Python aggregator and the template's JS payload.

Fixed

• "Reliability" card silently showed a fabricated 99% when there was no poller history to compute a real success rate from, rather than indicating the number wasn't real. Now shows an em dash with "No poll history yet" instead of a made-up figure.
• "Avg Broadcast Latency" showed a literal N/A card when there were no EAS broadcasts yet. The card is now hidden entirely in that case, matching how every other data-dependent card/section on the page already behaves.
• "System Performance" section header could float over an empty grid — its guard checked for the presence of the polling dict (always non-empty, even with zero poll history) rather than whether either chart inside it actually had anything to show. Guard now matches the inner per-chart conditions.
• Fixed the "EAS Received" card using an undefined yellow stat-card color class (silently falling back to unstyled default rendering).

[2.158.1] - 2026-08-13 - Fix missing CAP-side gated-alerts release scheduler

Fixed

• CAP-sourced gated alerts (2.158.0) could be held and manually approved, but never auto-released once their hold-off timer expired. The CAP poller (poller/cap_poller.py) got the GatedAlert/AlertGatingSettings model mirrors needed to read gate state, but the background sweep that actually releases expired holds was only wired up for the OTA path (app_core/gating_scheduler.py, running in the EAS-monitor service) — the CAP-side scheduler was never implemented.
  • Added _CapPollerGatedAlertScheduler to poller/cap_poller.py, started from CAPPoller.__init__. Since this process has no Flask app (unlike the OTA scheduler), each sweep opens its own plain SQLAlchemy session from a sessionmaker bound to the poller's own engine, and always queries explicitly via that session (session.query(GatedAlert)) rather than the Flask-SQLAlchemy Model.query shortcut, which requires an app context this process doesn't have.
  • Releases expired CAP-sourced holds by re-invoking auto_forward_cap_alert() — the same function used at ingest time — so the broadcast tail is never duplicated.
  • Added regression tests in tests/test_alert_gating.py covering the sweep's release path and its resilience to a single row's release failing without aborting the rest of the sweep.

[2.158.0] - 2026-08-13 - Gated alerts: hold-off timer with manual operator override

Added

• New optional feature: lower-priority CAP/OTA alerts can now be held for a configurable hold-off timer before broadcasting, instead of airing immediately. An operator watching the new Pending Alerts queue can approve a held alert early or cancel it outright; if nobody acts, it auto-releases and broadcasts normally once the timer expires. Disabled by default — enabling it changes nothing about which alerts broadcast, only when the lower-priority ones do.
  • Immediate urgency or Extreme severity alerts always bypass the gate and broadcast immediately, unchanged from today — gating never delays a Tornado Warning or similarly urgent product.
  • Applies to both ingest paths: the CAP poller (NOAA/IPAWS feed alerts) and the OTA relay (alerts decoded off the air).
  • New gated_alerts / alert_gating_settings tables and a GatedAlert state machine (pending → approved | cancelled | released) inserted into the existing auto_forward_cap_alert() / auto_forward_ota_alert() check chains in app_core/audio/auto_forward.py — release simply re-invokes the same function, so the broadcast tail (dedup, audio generation, GPIO, notifications) is never duplicated.
  • Background release schedulers run in the poller and EAS-monitor services (never the web app), sweeping for expired holds roughly every 15–30 seconds.
  • New admin pages: Settings → Alert Gating (enable/disable, hold-off duration) and Broadcast → Pending Alerts (live countdown, Approve/ Cancel, real-time via WebSocket push with automatic polling fallback).
  • New optional GPIO behavior, Gated Alert Pending, for a lamp or buzzer that stays active while the queue is non-empty — useful for an unattended studio. Assignable from the existing GPIO pin map.
  • See docs/guides/GATED_ALERTS.md for the full operator guide.

[2.157.1] - 2026-08-13 - Overlapping alerts on the dashboard map no longer blob together

Fixed

• The dashboard map drew every active alert in raw API-return order with identical styling, so overlapping alerts stacked into an undifferentiated blob. This is an everyday NWS pattern, not an edge case — e.g. a county-wide Winter Storm Watch with a small, more urgent Tornado Warning polygon inside it. With no z-ordering, which alert ended up on top (or completely hidden under a bigger, less urgent one) was arbitrary, and every alert additionally got its own white "casing" halo stroke — several of those overlapping is what actually produced the blob look, more than the fill colors themselves.
  • displayAlerts() in templates/index.html now sorts alerts before drawing: county-wide/broad alerts first (bottom), specific polygon alerts last (top); within each group, ascending by severity, so the single most specific + most severe alert on the map is always the one guaranteed visible on top.
  • Broad/county-wide alerts are restyled as context rather than a second subject: thinner outline (was actually thicker than specific alerts before this fix — backwards), lighter fill (0.3 → 0.15), and no white casing halo. Specific polygon alerts are unchanged (full casing, normal weight, 0.4 fill) so they clearly read as the primary subject.
  • Added EASMap.severityRank() to static/js/core/map_theme.js, exposing the same severity-ordering table refreshGlow() already used internally, so the dashboard's sort uses one source of truth instead of a second copy of that table that could drift from it.
  • Added an optional casing flag to EASMap.hazardLayer() (default true, unchanged for the single-alert pages — alert detail, VTEC trail — that already called it) so the dashboard can turn the halo off per-alert without a second code path.
• Verified the sort/styling logic against a realistic four-alert overlap scenario (county-wide Watch + county-wide Advisory + two specific polygons of different severities): confirmed the most specific + most severe alert always sorts last (drawn on top), county-wide alerts always sort first (drawn on the bottom), and each group still orders correctly by severity internally.

[2.157.0] - 2026-08-13 - Installer runs in a real full-screen dialog

Added

• install.sh/update.sh now drive an actual whiptail --gauge for the whole run — a real bordered full-screen TUI dialog, the same widget Debian-installer and Slackware's setup use, rather than plain text on the bare terminal (the 2.156.0 static screen). A persistent gauge is opened via a FIFO the first time echo_step fires (deliberately not from ui_banner — both scripts show whiptail --yesno confirmations between the banner and the first real step, and a gauge already holding the screen would fight those dialogs), fed live percent + single-line message updates for the rest of the run, and closed by show_celebration/cleanup_on_exit before either draws its own dialog. echo_step/echo_info/success/warning/error/ progress/header, the spinner, and ui_apt_install's per-package percentage all redirect into it automatically when it's active — no caller in install.sh/update.sh changed.
  • The gauge widget only renders one line of message text (confirmed directly against this whiptail build: sending a 3-line message shows only the first line) — a real constraint of the widget, not a bug here. Every caller collapses to one evolving status line as a result: the global percent comes from STEP_NUM/TOTAL_STEPS and stays put across sub-progress (apt/pip detail, the spinner) so it only advances at step boundaries, exactly like 2.156.0's fixed-row layout.
  • Falls back automatically, in order, when whiptail can't drive the show: 2.156.0's fixed-row static layout when whiptail is missing or /dev/tty isn't usable, then further to the pre-2.155.0 scrolling behavior under NO_COLOR or no real TTY at all.
  • Two real bugs this testing caught, both the kind that only show up when actually run, not read:
    1. A genuine deadlock: opening the FIFO write-only (exec 9>fifo) blocks until a reader connects, and whiptail's own read (<fifo) blocks symmetrically until a writer connects — with neither side timing out, a fixed open order against a backgrounded whiptail could hang forever. Fixed by opening read-write (exec 9<>fifo) instead, the standard idiom for driving a FIFO from a script without racing the other end — a read-write open never blocks regardless of which side connects first.
    2. exec 9<>fifo 2>/dev/null (and the matching close) looked like it only suppressed one command's stderr, but exec with no command applies every listed redirect permanently to the current shell — that line was silently discarding all subsequent stderr for the rest of the script, not just its own error path. Fixed by scoping the redirect to a brace group ({ exec 9<>fifo; } 2>/dev/null) instead of the bare exec, which reverts after the group instead of persisting. Caught both by tracing actual execution (set -x, isolating each piece down to a minimal reproduction) after a real run hung for the tool's full timeout with no diagnostic output — not by reading the first version of the code, which looked correct.

[2.156.0] - 2026-08-13 - Installer screen is now static, not scrolling

Added

• install.sh/update.sh now render on a genuinely static screen instead of a scrolling log — the DOS-installer restyle in 2.155.0 changed the colors and box art but not the interaction model; this changes the model itself. Contained entirely to scripts/lib/ui.sh, as before.
  • Banner stays fixed at the top. Below it, three rows are established once and redrawn in place via cursor positioning for the rest of the run: a global progress bar ([step/total] [bar] pct%), the current operation (what echo_step/echo_progress announce), and the last status message (echo_info/success/warning/error and the spinner) — only the latest message is visible on screen at a time; full history is unchanged, still in the log file.
  • ui_stream() (wraps pip install, apt-get install, alembic upgrade head) no longer tees the subprocess's raw output live to the screen — that was the single biggest source of scrolling. It now runs the command in the background with a spinner on the status row, still fully captured to the log. Ctrl+C is explicitly forwarded to the child process (kill, wait, then exit 130) so aborting still kills the real work instead of leaving it orphaned in the background — a background job doesn't inherit terminal process-group signal delivery the way the original foreground pipeline did, so this had to be added explicitly, not just inherited for free.
  • ui_apt_install()'s live per-package percentage now redraws the operation row instead of scrolling a \r-updated line.
  • Falls back to the previous scrolling append-only behavior whenever the static layout can't be established: before ui_banner has run, under NO_COLOR, or when there's no real TTY to position a cursor against.
• Verified with a real terminal emulator (pyte), not by eyeballing escape codes: fed a captured run (banner → 6 steps with mixed info/success/ warning/error calls → a spinner → a simulated failure → completion) into a simulated 90×30 screen and confirmed the progress bar, operation line, and status line each end at their correct final state with nothing else printed below them — i.e. actually static, not just visually plausible. Separately verified ui_stream's Ctrl+C handling end-to-end: sent a real SIGINT mid-run and confirmed the backgrounded child was killed (no orphaned process), the script exited 130, and the trap chain to cleanup_on_exit still fired.
• Known pre-existing gap, not introduced by this change: the scrolling fallback path in echo_step/echo_info/etc. doesn't check _ui_tty_supports_color before emitting color codes, so NO_COLOR=1 isn't fully honored in that fallback. Flagged, not fixed here — separate from what this change was about.

[2.155.0] - 2026-08-13 - Old-school DOS-style installer UI

Added

• Restyled install.sh/update.sh's terminal chrome to an old-school DOS-installer look — entirely contained to scripts/lib/ui.sh (confirmed nothing outside it references the color internals directly, so neither script needed a single line changed):
  • Flat 16-color EGA/VGA palette in place of the previous 256-color cyan-blue gradient.
  • Banner and completion card are now fully-closed double-line CP437 boxes (╔═╗║╚═╝) on a blue background — the two moments the script fully owns the screen (right after a clear, and at the very end); everywhere else keeps the terminal's natural background since real subprocess output (apt-get, pip, git, alembic) is interleaved there and can emit its own color resets.
  • Classic 4-frame ASCII spinner (| / - \) in place of the Unicode braille spinner.
  • Single-glyph status gutter (i info, * ok, ! warn, X error) in place of [INFO]/[ OK ]/[WARN]/[ERR!] bracket tags.
  • █/░ block-fill progress bars were already period-correct CP437 and are unchanged, just recolored.
• Two real bugs turned up by actually testing this instead of just reading the diff:
  1. The progress bar's filled and empty portions were built as one concatenated string, so despite two different color codes in the printf, the entire bar would have rendered a single color. Split into separate bar_filled/bar_empty variables.
  2. The first pass at the fully-closed boxes used printf '%*s' N '' | tr ' ' '═' to build the horizontal rule — tr silently mangles multi-byte UTF-8 fill characters even in a UTF-8 locale (a known GNU tr limitation, not a locale misconfiguration; reproduced in isolation). Replaced with the same bash-loop character-concatenation the block/shade progress bar already used safely. Verified by stripping ANSI codes from a real captured run and measuring every box line at exactly 72 characters.
• Verified via a real script-captured pty run (not just visual inspection of the source): bash -n on all three files, shellcheck shows zero new warnings beyond what the original file already had, the spinner was tested against a real backgrounded job with correct exit-code passthrough, and the NO_COLOR=1 plain-text fallback path (unchanged) still renders correctly.

[2.154.3] - 2026-08-13 - Poller stopped silently dropping VTEC-bearing alerts

Fixed

• A Numba disk-cache race could make the poller silently drop every NWS product carrying a VTEC string. poller/cap_poller.py tries from app import ..., CAPAlert, ... first and only falls back to its own standalone (non-Flask) CAPAlert model, defined inline in an except block, when that import fails for any reason. That fallback model was last updated before vtec_office, vtec_phenomenon, vtec_significance, vtec_etn, vtec_year, vtec_action, superseded_by_id and cancelled_at were added to the canonical model — 8 of 36 columns missing. Two call sites do unconditional attribute access on those columns: existing.vtec_office in _update_existing_alert() and new_alert.vtec_action in _insert_new_alert(). On the fallback model neither is a real instrumented column, so the access raised a bare AttributeError and "Error saving CAP alert" — for every affected alert, on every poll cycle, for as long as that poller process stayed in fallback mode. Confirmed in production logs: 4 distinct NWS warnings failed to save on 430+ separate poll cycles across two independent poller restarts (vtec_office AttributeError ×430, vtec_action ×1).
• Root cause of the import failure: _dll_drain_bits_numba() in app_utils/eas_demod.py was JIT-compiled with cache=True, persisting compiled machine code to a shared .nbi/.nbc file pair next to the source. Up to half a dozen independent OS processes (2 web workers, poller, EAS/SDR/audio services) import this module and can start within the same second on a service restart or reboot; Numba's on-disk cache isn't safe against that many processes racing to compile and write it on first import. One such race corrupted the cache with cannot cache function '_dll_drain_bits_numba': no locator available for file '.../eas_demod.py', which broke from app import ... for the poller process that hit it. Removed cache=True here and from the 6 equivalent @jit kernels in app_core/radio/demod/kernels.py (same documented pattern, same exposure) — each process still JIT-compiles independently at import time (a few tens of ms, once), it's just never persisted to a file multiple processes can race on. Verified bit-exact output unchanged (134 existing decode/demod tests pass) and both modules still report Numba JIT active.
• Hardened both CAPAlert VTEC attribute-access call sites in poller/cap_poller.py with getattr(..., None) as defense in depth, matching the style already used by _apply_cancellation_status() in the same file — a future schema drift degrades gracefully instead of crashing the whole alert save.
• Added tests/test_cap_alert_fallback_model_parity.py: statically parses both CAPAlert class definitions and fails the build the moment a column exists on one but not the other, so the next schema change can't reintroduce this gap silently. Removed the now-obsolete if not hasattr(cp.CAPAlert, "vtec_action") workaround from tests/test_forwarding_pipeline_guard.py, added when this exact gap was first noticed but never fixed at the source.

[2.154.2] - 2026-08-13 - Alert narration no longer sits ~15 dB below the tones

Fixed

• TTS/embedded/relay narration was mixed into broadcasts at whatever raw level the source happened to produce, with no leveling against the SAME header, attention tone and EOM tones — which are always generated at a fixed, controlled amplitude. EASAudioGenerator.build_files() (the automatic path used for real NOAA/IPAWS alerts) appended voice_samples straight into the composite. Measured against production audio: the attention tone sits at -9.1 dB mean/RMS; Azure OpenAI TTS narration measured -24.7 to -24.8 dB across four separate alerts, and IPAWS-embedded audio -21.9 dB — a consistent ~13-16 dB gap, easily described as "wildly different" volume between the tone and the voice message. Every source that can populate voice_samples (TTS synthesis, IPAWS-embedded audio, OTA relay capture) funnels through the same code path, so a single fix point covers all three.
• build_manual_components() (the manual/uploaded-narration broadcast path) already normalized narration to the tone's RMS via _normalize_audio_amplitude(); build_files() never called it. Added the same call to build_files()'s narration handling.
• Also hardened _normalize_audio_amplitude() itself: pure RMS matching against real TTS audio caused ~2.5% of samples to hard-clip, because speech has a much higher peak-to-RMS ratio (crest factor) than the sine-wave tones the function was modeled on. Added a peak-safety cap so the resulting gain never pushes the loudest sample in a clip past full scale; loudness falls a little short of the RMS target only for unusually peaky source audio, in exchange for zero clipping. This also improves the existing manual-narration path, which had the same latent risk.
• Added tests/test_narration_loudness_normalization.py: unit coverage for the peak-safety cap (verifies no clipping on high-crest-factor input, and that low-crest-factor input still reaches the target RMS), plus an integration test that stubs a quiet TTS engine and asserts the tone/ narration RMS gap in the generated composite stays under 6 dB instead of regressing to the ~15-16 dB gap this fix addresses.

[2.154.1] - 2026-08-13 - Static asset URLs stop double-appending the cache-bust param

Fixed

• Every url_for('static', ...) link double-appended ?v=. The @app.url_defaults hook in app_core/flask/url_defaults.py already adds ?v={{ static_asset_version }} to every static URL automatically, but 35 template references across 11 files (favicons, the new PWA manifest link, map.css, admin JS bundles, etc.) also manually appended the same suffix as a literal string, producing URLs like /static/manifest.json?v=2.154.0?v=2.154.0. Harmless in practice — Flask's static route ignores the malformed trailing query string and serves the file regardless — but wasteful and confusing in the rendered markup. Left templates/repo_stats/_content.html alone: it deliberately strips url_for's auto-appended ?v= (.split('?')[0]) and re-adds it manually because it builds a dynamic per-language logo path url_for can't express directly, so its manual suffix is the only one and is correct as-is.

[2.154.0] - 2026-08-13 - Installable as a home-screen app

Added

• PWA manifest for "Add to Home Screen." static/manifest.json (name, standalone display mode, theme/background colors, 192/512/maskable icons rendered from the existing eas-app-icon.svg) plus the matching <link rel="manifest"> and apple-mobile-web-app-* meta tags in base.html. Lets phones running EAS Station launch it full-screen from a home-screen icon instead of a browser tab — relevant for a system meant to be checked in the field during severe weather.
• Along the way, fixed apple-touch-icon pointing at the 2080×325 wordmark banner (meant for the favicon/header, not a home-screen tile) — iOS was squashing it into a square. Replaced with a proper 180×180 render of the square app icon.

[2.153.5] - 2026-08-12 - Alert Verification stops loading audio it never shows

Fixed

• The Alert Verification page loaded every audio blob in the display window just to show delivery status. collect_alert_delivery_records() queried whole EASMessage ORM rows, which drags along all six LargeBinary audio columns even though the page only reads same_header, created_at and metadata_payload. Measured on production: 235 messages averaging ~6.4 MB of audio each meant the default 30-day window pulled roughly 180 MB from Postgres on every page load — for bytes that were never rendered. Switched to EASMessage.without_audio(), which defers all six blob columns in SQL. Added tests/test_alert_verification_queries.py, which inspects the emitted SQL to pin this query shape and catch a regression back to EASMessage.query.

[2.153.4] - 2026-08-12 - Idle-in-transaction sessions get a timeout

Fixed

• idle_in_transaction_session_timeout was unlimited (0) in production. A live audit found several eas-station app sessions sitting idle in an open transaction for 20+ minutes on dashboard-style count queries. A long-held open transaction pins Postgres's vacuum cleanup horizon, which lines up with autovacuum never having completed a single run on the largest tables despite heavy dead-tuple accumulation. Added scripts/database/apply_postgres_tuning.sh, an idempotent script that applies a 5-minute timeout via ALTER SYSTEM + reload (no restart required), wired into both install.sh (fresh installs) and update.sh (existing deployments) so the fix persists instead of only living in a one-off manual ALTER SYSTEM call.

[2.153.3] - 2026-08-11 - Page-chrome clocks tell the truth

Fixed

• The navbar and footer clocks were hardcoded to Eastern. Every page renders a live clock from static/js/core/utils.js, and all three readouts passed timeZone: 'America/New_York' literally. The station timezone is configurable — it is stored in LocationSettings.timezone, edited from the Location settings form, and the setup wizard offers 22 US zones including Anchorage, Honolulu and Guam — so any station outside Eastern read a clock that was hours off.

  The clocks also pass timeZoneName: 'short', so the wrong zone was stated outright: a Honolulu station saw a time six hours ahead labelled "EDT" rather than an ambiguous number it might have questioned. Server-rendered timestamps had just been moved onto the configured zone in 2.153.1, which left the clock in the chrome disagreeing with the tables beneath it.

  base.html now stamps the configured zone onto <body data-timezone> via a new station_timezone() template global, and utils.js reads it. An unusable or missing value falls back to the browser's zone rather than a guess — toLocaleString throws a RangeError on a bad IANA name, which would have killed the one-second interval and frozen every clock on the page.

• System Health showed the wrong "Last Updated" value and a dead footer clock. system_health.html gave its "Last Updated" metric id="current-time", which base.html already uses for the footer clock. Both elements land in the same document and {% block content %} renders before the footer, so getElementById returned the metric.

  Two one-second timers then wrote to that one node: the global wall clock in utils.js and the page's own "10:06:43 (5s ago)" readout. The field alternated between the two meanings depending on which timer fired last, and the footer clock stayed on "Loading..." for as long as the page was open. Nothing errored — duplicate IDs are legal HTML and getElementById just picks the first match.

  The metric is now health-last-updated. This was the only ID collision against base.html in the template tree.

Added

• tests/test_client_clock_rendering.py fails the build if a template redefines an ID that base.html owns, or if the chrome clocks hardcode an IANA timezone again.

[2.153.2] - 2026-08-11 - Filter dropdowns no longer clipped by their card

Fixed

• Filter pull-downs were sliced off at the card's edge. On the Received Alerts page, opening the Audio Source or Event Type multi-select showed the Include/Exclude toggle and roughly one option — the rest of the menu was invisible and un-clickable, making the filters unusable for any event type past the first.

  The menus were not mis-positioned; they were clipped. .card in static/css/styles.css sets overflow: hidden so the ::before accent line, the ::after glow and card media stay inside the rounded corners, and that declaration clips every absolutely-positioned descendant that leaves the card box. A Bootstrap dropdown is exactly that: measured in a browser, the Event Type menu overhung the filter card by 188px, and a hit-test at the bottom of the menu landed on the results card underneath rather than on the menu. Nothing about the markup looked wrong, which is why the fault survived review — only a rendered page shows it.

  A card now stops clipping while a dropdown inside it is open, and is lifted above the cards that follow it. The lift is load-bearing rather than cosmetic: backdrop-filter makes each .card its own stacking context, and equal-z-index stacking contexts paint in document order, so the next card down the page would paint straight over the un-clipped menu. The chosen z-index: 3 clears sibling cards while staying far below the sticky navbar (1030), modals and toasts.

  The card's two decorations fall out of this: both exist only because the card crops them. The ::after glow is sized 200% and offset -50%, so unclipped it bleeds across neighbouring content; the ::before accent line is a 3px strip whose ends the card's 24px corner arc trims, and it cannot round itself to match — border-radius on a 3px-tall box is scaled down to a 3px radius, leaving coloured nubs sticking out past the corner. Both are hover-only flourishes sitting at opacity: 0 the rest of the time, so they are simply held at 0 while the card is unclipped.

  The rule is expressed as .card:has(.dropdown-menu.show), with static/js/core/utils.js mirroring it via a has-open-dropdown class toggled from Bootstrap's show.bs.dropdown / hidden.bs.dropdown events so the fix also holds where :has() is unsupported. Being a global rule, it covers every card-hosted dropdown in the app, not just the Received Alerts filters. Verified in a headless browser at 320, 360, 768 and 1024px wide in both the light and dark themes: the full menu is reachable and no element exceeds the viewport width. Regression coverage in tests/test_card_dropdown_clipping.py.

[2.153.1] - 2026-08-11 - Timestamps read in the station's local time

Fixed

• Alert times displayed UTC while claiming to be local. On the Received Alerts page an alert that arrived at 6:06 AM Eastern was listed as 2026-08-11 10:06:43 — four hours ahead of the operator's wall clock, with nothing on the row to say the number was UTC. The same defect ran through the log viewer, alert detail, search results, GPIO activation history, the display/VFD message tables, security settings and three admin settings pages.

  Every timestamp is persisted in UTC, and these templates called .strftime() straight on the column. That is a silent bug: strftime formats whatever zone the value is already in, so the page renders correct digits for the wrong zone and no error is raised anywhere. The FCC report builders in app_core/eas_storage.py were unaffected — they already went through format_local_datetime() — which is why the exported logs and the on-screen tables disagreed.

  app_utils/time.py gains to_location_time() and format_local(), wired up as the localtime and to_local Jinja filters, and all 24 affected template sites now use them. Naive columns (the settings tables default to datetime.utcnow) are treated as UTC so they convert identically to the timezone-aware ones. Rendered timestamps now carry a %Z suffix — EDT, EST — so the zone is stated rather than assumed, and the offset follows DST via the configured location timezone instead of a fixed value.

  Client-rendered tables were already correct: those endpoints serialize timezone-aware columns, so new Date(...).toLocaleString() converts to the browser's zone. The mismatch was confined to server-rendered markup.

Added

• tests/test_local_time_rendering.py — covers the DST and naive-column conversions, and fails the build if .strftime() is called on a stored timestamp in any template again.

[2.153.0] - 2026-08-10 - The in-app maps get the share card's treatment

Added

• A shared, theme-aware Leaflet skin — static/css/map.css plus static/js/core/map_theme.js (window.EASMap). Release 2.152.0 gave the exported share card a toned basemap and a hazard that reads as the subject; the maps inside the app kept rendering raw OpenStreetMap: pastel landcover, orange motorways and dozens of town labels at full saturation, dropped into a themed card and completely indifferent to which of the 20 themes was active. The alert polygon had to compete with the road network to be seen.

  The skin is the browser-side twin of app_utils/image_export/map_style.py and reuses its numbers. The basemap runs the same tone chain in the same load-bearing order — saturate(0.38) contrast(1.18) brightness(0.58) on dark themes, gentler on light ones — then blends toward the theme's own background, which is what makes the map belong to the current theme instead of merely sitting on top of it. A radial vignette fades the frame edges so the map stops looking like a screenshot pasted into the card, and zoom, scale, attribution and popups all read as chrome in the active palette.

  Alert geometry is drawn the way the share card draws it: a blurred glow behind a white casing stroke behind a severity-coloured core, over a semi-transparent fill, in a dedicated pane that always sits above the context boundaries no matter which finishes loading first. Boundaries move to their own pane underneath as quiet reference lines — still in their category colour, no longer competing.

  Overlay colours are resolved from the theme's --severity-* variables at draw time and re-resolved on the theme-changed event, so switching themes restyles a live map without a reload.

Changed

• Every Leaflet map in the app now goes through the skin: the alert detail coverage map, the dashboard's interactive map, the alert trail map, the county-boundary browser and the Security Center traffic map. The two browsing maps opt out of the vignette (EASMap.create(..., {vignette: false})), where edge darkening would only hide data.
• base.html stamps data-theme-mode in its anti-flash script. It was only set by theme.js on DOMContentLoaded, so anything keyed on it — the new basemap tone, the chart palettes — painted its light variant first and snapped to dark a moment later. tests/test_map_theme.py keeps the hardcoded dark-theme list in sync with theme.js.
• The Leaflet chrome moved out of styles.css (loaded on every page) into map.css (loaded only by pages with a map), and the duplicated .popup-title / .popup-detail rules that index.html and alert_detail.html each carried their own copy of were promoted with it — clearing two entries from tests/css_collisions_allowlist.txt.

Fixed

• Vector layers in a custom Leaflet pane rendered nothing. Leaflet ships a max-width: none reset for exactly this, but scopes it to .leaflet-overlay-pane; the mobile-overflow rule svg, canvas { max-width: 100% } in styles.css resolves against the pane, and a Leaflet pane is a 0x0 positioned box, so the overlay's width computed to zero. The same trap applies to CSS filter, whose region derives from the element's own box — a filter on a pane is silently discarded while devtools still reports it, which is why the tone chain lives on .leaflet-tile and the hazard glow on the pane's <svg>.

[2.152.0] - 2026-08-10 - The share-card map reads like a warning graphic

Changed

• The alert now fills the map instead of sitting in it. The renderer picked the integer OSM zoom at which the alert's bounding box fit inside 60% of the map slot, then cropped a fixed map_w x map_h window at that zoom. Two lossy steps stacked: the 60% fit factor, and flooring to an integer zoom, which can nearly halve the apparent size again. Together with 30%-per-side bbox padding, a county-scale flood advisory rendered with its polygon covering barely a tenth of the frame, adrift in unrelated geography.

  Zoom now decides tile detail only (_detail_zoom), and the crop frames the padded bbox itself, widened to the slot's aspect ratio and resampled (_crop_window). The subject fills the frame at any zoom. Padding dropped to 16% per side now that it is the entire margin rather than compounding with a fit factor.

  Because overlays are drawn on the tile canvas before that resample, every overlay dimension is now pre-divided by the resample factor — polygon strokes, the storm cone, arrow and its callout (which also switches to a proportionally larger font) all arrive at their intended size instead of shrinking.

• The basemap is toned down so the hazard is what reads. A raw OSM tile is the loudest thing on the card — pastel landcover, orange motorways, dozens of town labels, all at full saturation — and dropped into a dark card it reads as a bright rectangle pasted in from elsewhere. New map_style.tone_basemap() desaturates, lifts contrast, darkens and tints toward the card's slate before any overlay is drawn, so the alert polygon is the only saturated thing in frame (mean luminance on a typical mosaic: 223 → 93). apply_vignette() fades the edges so the inset blends into the card rather than ending in four hard borders.

• County names are back on the map, with collision avoidance. Labels were removed once because drawing one per boundary produced overlapping, unreadable text — but a map with no place names cannot answer "where is this?". map_style.place_labels() places them greedily in priority order (counties inside the alert first), skipping any label that would overlap an already-placed one, land on the scale bar or attribution, or clip the frame, and caps the total so a 40-county watch does not become a wall of text.

• Counties inside the alert are drawn brighter than the surrounding reference lines, the way NWS warning graphics separate "in the warning" from "here for context". _fetch_county_outlines() now asks PostGIS which counties intersect the alert geometry and returns an ST_PointOnSurface anchor for each label (point-on-surface, not centroid, so the anchor cannot fall outside a crescent-shaped or multi-part county).

Fixed

• The storm-motion callout no longer covers the arrow it labels. The pill was centred just past the arrow tip, so its own width laid it back over the arrow; it is now offset far enough that its near edge clears the tip.

Internal

• maps.py (904 lines after the changes above) was split: the storm-motion overlay moved to storm_overlay.py, the PostGIS lookups to map_data.py, and the new basemap treatment and label placement live in map_style.py. maps re-exports every moved name, so no caller changed.
• Removed _best_zoom(), dead once the map started framing its own crop. It is the one name the package __init__ no longer re-exports.

[2.151.0] - 2026-08-10 - Share cards keep the outline NWS wrote

Changed

• Share-card descriptions render as the outline NWS actually wrote, not a wall of text. Modern NWS products are written to a tagged bullet structure — WHAT is happening, WHERE, WHEN, what the IMPACTS are — but CAP delivers the whole thing as one free-text description. The renderer flattened it (collapse newlines, strip the leading asterisk), so a flood advisory landed on the card as nine unbroken lines with stray * and - glyphs sitting mid-sentence, and the reader had to parse the structure back out by eye.

  app_utils/image_export/nws_text.py recovers the segments and the DESCRIPTION panel lays them out as a label gutter plus wrapped prose. Both NWS conventions are recognised: the * WHAT... bullet form and the asterisk-free HAZARD... / SOURCE... / IMPACT... form used by severe thunderstorm and tornado warnings. The untagged lede that opens those warnings — "At 900 PM EDT, a severe thunderstorm was located near Lima, moving east at 40 mph", the most informative sentence in the product — is kept and rendered full width. Text carrying no recognisable outline, including shouted legacy products, still falls back to the paragraph path.

• Segments the card already shows elsewhere are dropped. WHERE is the AFFECTED AREAS county list written out as a sentence, WHEN restates the footer's expiry stamp, and the PRECAUTIONARY/PREPAREDNESS ACTIONS block is the CAP instruction field the card renders under its own ACTION banner. Each is suppressed only when the card is genuinely carrying that information, which is what buys the room for the hazard copy.

• Bare URLs no longer consume a line of copy. NWS descriptions end with links like http://www.weather.gov/safety/flood; printed on an image they cannot be clicked, wrap badly, and displace hazard text. Stripped from both the description and the instruction.

• areaDesc no longer repeats the state on every entry. NWS sends "Allen, OH; Defiance, OH; Henry, OH; Paulding, OH; Putnam, OH", which spends a wrapped row restating "OH" four more times than needed. Entries are grouped by state in first-appearance order and the code is factored out once per group: "Allen, Defiance, Henry, Paulding, Putnam (OH)". Multi-state lists group separately; zone and marine products that carry no state codes are passed through untouched.

Internal

• The prose section drawers (headline, description, action) moved from app_utils/image_export/panels.py — already past the module-size guidance — into a new sibling panels_text.py, next to the parser they depend on. panels re-exports every name, so existing imports are unchanged.
• Added _lighten() to the image-export palette so accent-coloured text stays legible on the dark card whatever hue a hazard family's theme resolves to.

[2.150.2] - 2026-08-10 - Stop a dead audio service from looking like stopped sources

Fixed

• A dead audio service rendered as three deliberately-stopped sources. Live Audio showed grey "Stopped" badges and "Start the source to listen" for every source while the EAS Continuous Monitor panel — reading the same Redis key — correctly reported "Audio service metrics are unavailable". Only one of those two statements was true, and it was not the one attached to the buttons the operator would reach for.

  In the separated deployment the web process builds an adapter for every audio_source_configs row but deliberately never starts one: _start_audio_sources_background() returns immediately because the audio service owns capture. Those adapters are placeholders that permanently report STOPPED with no error message. build_source_listing() preferred them over the service_dead signal, so _serialize_db_only's "Audio service is not running – source failed to start" branch was unreachable in production — the tests only reached it because they stub the controller empty.

  GET /api/audio/sources now ignores a non-running local adapter when the audio service is publishing no metrics at all, so an auto_start source reports error with an actionable message. Integrated deployments are unaffected: there the web process is itself the metrics publisher, so service_dead is never True and a running adapter stays authoritative.

• Audio commands reported success when nothing received them. Redis PUBLISH succeeds with zero subscribers, so with the audio service down every fire-and-forget command (eas_monitor_start, streaming_start, source_add/source_update) returned success: True and went nowhere. Pressing Start Monitor produced no error, no state change and no explanation. AudioCommandPublisher._publish_command() now checks the receiver count and fails with "The audio service is not running, so the command was not delivered." This also makes the wait-for-response commands (source_start/source_stop) fail immediately instead of burning their full 5-second timeout.

• POST /api/eas-monitor/control returned 500 for a down dependency. An undelivered command is now a 503 carrying a hint pointing at Settings → Services.

• The Start/Stop Monitor buttons gave no feedback on success. controlEASMonitor() showed a toast only on failure, which was indistinguishable from the command silently going nowhere. It now confirms acceptance and surfaces the server's hint on failure.

Added

• tests/test_audio_command_delivery.py — the receiver-count contract, both directions, for fire-and-forget and wait-for-response commands.
• Two cases in tests/test_audio_source_listing.py covering the production controller shape (placeholder adapters present) rather than the stubbed-empty controller the existing dead-service test relies on.

[2.150.1] - 2026-08-10 - Fix one-click backup and upgrade

Fixed

• One-click backup and one-click upgrade never ran. The worker that _start_background_operation spawns is a bare daemon thread, and Flask contexts are per-thread, so current_app is unavailable there. The very first statement in the worker was current_app.logger.info(...), which raised RuntimeError: Working outside of application context. The except handler then called current_app.logger.exception(...) and raised again, so message was never assigned. The net effect: the subprocess.run was never reached, and the UI showed a failed operation with an empty message.

  The caller already resolves current_app.logger inside the request context and passes it in as the logger argument — the worker simply was not using it. All four call sites now use the injected logger, and current_app is no longer imported by that module.

  This is pre-existing, not a regression from the Phase 3f split: the same four current_app.logger calls are in the single-file webapp/admin/maintenance.py, which is why the split was still 31/31 AST-identical. Found by CodeRabbit's review of #2355 and verified against the pre-split file.

Added

• tests/test_maintenance_package.py gains two guards. One runs a real background operation with no application context and asserts the subprocess actually executed — it fails against the pre-fix source, so it pins the behaviour rather than the spelling. The other asserts current_app does not appear in operations.py at all, since that is the one module here doing work off the request thread.

[2.150.0] - 2026-08-08 - Phase 3h: split the alert verification routes

Changed

• webapp/routes/alert_verification.py (1,668 lines) is now the webapp/routes/alert_verification/ package — 14 modules plus a 123-line __init__, all within the 400-line guidance. Phase 3 is complete for every file except app.py, which is assessed separately (3g).

  This was the harder shape: 687 of those lines were a single register(app, logger) with eight helpers and seven handlers nested inside it, so the split had to reproduce a closure, not just move text.

  | Module | Lines | Contents | | --- | ---: | --- | | errors.py | 26 | The self-test error type | | samples.py | 31 | The bundled sample recordings | | routes_export.py | 77 | The CSV export | | temp_audio.py | 77 | Decode an upload, then persist | | helpers.py | 89 | The capture-free helpers from register | | decode_serialization.py | 100 | A decode result across the thread boundary | | composite_audio.py | 124 | Stitching segments into one file | | routes_api.py | 141 | Progress, header decode, decode audio | | routes_self_test.py | 170 | The end-to-end self test | | audio_buffer.py | 174 | PCM extraction and caching | | routes_operations.py | 203 | Starting an async run | | routes_page.py | 269 | The verification dashboard | | progress.py | 289 | On-disk progress and result stores | | eas_detection.py | 316 | Locating every EAS burst in a file | | __init__.py | 123 | register, fanning out to the route modules |

• The closure was reproduced from symtable, not guessed. Four helpers capture nothing from register, so they were dedented to module scope in helpers.py — semantically identical, and their ast.dump is unchanged. The other four capture route_logger, repo_root or app, so each topic module keeps its own register(app, logger) that rebuilds exactly the locals its handlers need and nests them inside. Three modules needed only route_logger; emitting repo_root unconditionally left an unused local in each (F841).

Fixed

• Four mutable globals are deliberately not re-exported from the package. _progress_dir, _progress_lock, _result_dir and _result_lock are replaced with tmp_path by tests/test_alert_verification_async.py. This was verified rather than assumed: with the names re-exported and the patch aimed at the package, all six tests still pass — while writing to the real temp directory instead of tmp_path. The patch rebinds a copy; the classes keep reading the original. Absent from the package, the same call raises AttributeError naming the module, which is the whole point. The six patch sites across two test files now target alert_verification.progress, .helpers (get_location_settings) and .routes_self_test (AlertSelfTestHarness).

• One dead import dropped out: struct was imported and never used.

Added

• tests/test_alert_verification_package.py — 14 tests. The closure-specific ones inspect the handler through __wrapped__, since app.view_functions[...] returns the outermost require_auth decorator whose only free variable is the function it wraps.

• A flaky test that made every "suite green" claim unreliable. tests/test_8khz_stress_test.py generated its noise with the global unseeded random, then asserted a confidence threshold — so a decoder regression and an unlucky draw were indistinguishable. Measured at 2 failures in 12 runs of test_8khz_with_increasing_noise[0.2]. It now uses a seeded random.Random, and 15 consecutive runs pass.

  Seeding it surfaced something worth a look. Sweeping ten seeds across five noise levels, 6 of 50 combinations fail at 20-25% noise — and they fail with len(result.headers) == 0 while bit_confidence stays around 0.92. That is not gradual degradation: bit recovery still looks healthy while header framing drops out entirely. Whether the decoder should tolerate that noise is a signal-processing decision, so it is documented in the helper's docstring with a one-line reproducer rather than hidden by the seed.

Notes

• An import cycle was created and caught during the split. The call chain is _process_temp_audio_file → _detect_comprehensive_eas_segments → _build_composite_audio_segment. Grouping the two ends into one composite_audio module — which is what their names suggest — put eas_detection in the middle of a cycle. Split into composite_audio and temp_audio, and the generator now walks the emitted imports and fails on a cycle by name rather than leaving Python to raise a partially-initialised ImportError at startup. Group by the call graph, not by topic name.
• No __file__ hazard here, unlike 3e and 3f: repo_root is derived from app.root_path, so it does not shift with the module's depth.
• eas_detection.py (316) holds one 276-line function. Within the guidance as a module, but it is the same shape as the other single-function modules this plan has flagged; noted rather than tracked.
• Verification. 27 of 28 definitions (top-level and nested) are ast.dump()-identical. The one difference is register itself, which is deliberately restructured to fan out to the route modules. URL map unchanged at 549 rules, 0 differences, all 7 endpoints intact.

[2.149.0] - 2026-08-08 - Phase 3f: split the maintenance routes

Changed

• webapp/admin/maintenance.py (1,802 lines) is now the webapp/admin/maintenance/ package — 15 modules plus a 118-line __init__, every one within the 400-line guidance. All 31 top-level definitions moved verbatim: 31 of 31 are ast.dump()-identical.

  | Module | Lines | Contents | | --- | ---: | --- | | blueprint.py | 28 | maintenance_bp | | paths.py | 41 | repo_root — the tools scripts and .env live under it | | routes_poll.py | 53 | Out-of-band feed poll | | serialization.py | 65 | A CAP alert row for the admin views | | eas_settings.py | 85 | The singleton EAS settings row | | routes_env.py | 122 | The .env editor | | routes_operations.py | 140 | Operation status, backup, upgrade | | routes_database.py | 153 | DB health and optimize | | routes_expiry.py | 162 | Mark and clear expired alerts | | operations.py | 165 | Backup/upgrade progress state and its lock | | routes_location.py | 178 | Location settings, filtering, FIPS lookup | | routes_import.py | 249 | Manual single-alert import | | routes_eas_settings.py | 259 | The EAS settings page | | routes_alerts.py | 264 | Admin alert list and detail | | noaa.py | 271 | The api.weather.gov client | | __init__.py | 118 | register_maintenance_routes, __all__ |

Fixed

• repo_root would have silently moved, taking backup, upgrade and the .env editor with it. It is Path(__file__).resolve().parent.parent.parent — three hops to the repository root from webapp/admin/maintenance.py, one short from webapp/admin/maintenance/paths.py. It resolves tools/create_backup.py and tools/inplace_upgrade.py, is passed as their cwd, and locates the .env that Settings → Environment reads and writes. Unfixed, all three would point into webapp/: the backup would invoke a script that does not exist and the environment editor would edit a file nothing reads. This is the second consecutive phase to hit this hazard.

• get_operation_status is re-exported even though __all__ does not list it. app_core/websocket_push.py imports it by name to feed the admin operation-status push. The import sits inside a function whose caller logs and continues, so losing it would have been silent — precisely the shape of the AudioSourceConfigDB regression the audio_ingest split caused. The test derives the required names by AST-walking the tree rather than listing them, so it cannot drift.

Added

• tests/test_maintenance_package.py — 20 tests, the first this module has had. Beyond the structural guards it pins two things worth stating:

  • The blueprint takes no url_prefix — /admin is written into each route decorator, the opposite of the certbot blueprint next door, which is registered with url_prefix='/admin' and whose decorators must not repeat it. The two conventions sit in adjacent packages; the test records which is which so "harmonising" them is a deliberate act.
  • _OPERATION_STATE is mutated in place, never rebound. That is what makes a by-value import of it safe. A module that rebound it would give the status endpoint a private copy that never updates, and the test fails if one ever does.

  Both guards are mutation-checked: dropping a .parent fails 2 tests, and removing the get_operation_status re-export fails 1.

Notes

• limit is allow-listed but never sent upstream. build_noaa_alert_request accepts it and drops it; retrieve_noaa_alerts applies it client-side by slicing the response. So a manual import fetches the full NOAA result set and trims it locally. Left as-is — this is a motion commit — but now asserted by a test, so a future change that starts forwarding it is a visible decision rather than an accident.

[2.148.0] - 2026-08-08 - Phase 3e: split the Certbot routes

Changed

• webapp/admin/certbot.py (1,946 lines) is now the webapp/admin/certbot/ package — 14 modules plus a 105-line __init__. Like 3d this was ordinary top-level definitions, so all 22 moved verbatim.

  | Module | Lines | Contents | | --- | ---: | --- | | blueprint.py | 40 | certbot_bp and the domain/email patterns | | log.py | 43 | The one logger, named webapp.admin.certbot | | failures.py | 50 | certbot exit → operator-readable explanation | | routes_pages.py | 50 | The rendered settings page | | nginx.py | 93 | Is nginx up, and bring it up | | routes_status.py | 119 | Certificate status and the log tail | | routes_settings.py | 134 | Reading and writing stored settings | | staging.py | 149 | Detect and clear staging certificates | | paths.py | 157 | The writable certbot_data tree | | routes_actions.py | 228 | Auto-renewal, download, install | | routes_obtain.py | 273 | Obtain dry-run and the domain test | | routes_renew.py | 286 | Renew dry-run and the real run | | install.py | 293 | Install a certificate into nginx | | routes_obtain_execute.py | 449 | The real certificate run | | __init__.py | 105 | register_certbot_routes, side-effect imports |

  register_certbot_routes(app, logger) is unchanged, and __all__ still exports it alongside certbot_bp.

Fixed

• The certbot_data directory would have silently moved. CERTBOT_BASE_DIR is Path(__file__).parent.parent.parent / 'certbot_data' — three hops from webapp/admin/certbot.py to the repository root. From webapp/admin/certbot/paths.py that is one hop short, so it resolves to <repo>/webapp/certbot_data. Nothing raises: certbot would have built a fresh empty tree in the wrong place and every existing certificate would have looked like it had vanished. Fixed to four hops and pinned by a test that asserts the resolved value, plus a second test that rejects any new Path(__file__).parent.parent.parent added at the wrong depth.

• Log records keep the name webapp.admin.certbot. The single-file module had one logging.getLogger(__name__) behind all 92 call sites. Per-module loggers would have renamed those records to webapp.admin.certbot.routes_obtain and friends, breaking any log filter or journald grep keyed on the old name. There is one log.py holding getLogger(__package__) and every module imports it. A by-value import is safe here specifically because register_certbot_routes does not rebind the module logger — it only writes one line through the logger it is handed. (Phase 3a's audio_ingest package needed a fan-out instead precisely because its register did rebind.)

• Two dead imports dropped out: os and datetime.datetime were imported by the single-file module and used by none of it.

Added

• tests/test_certbot_package.py — the first tests this module has ever had. It had zero coverage before the split, which is worth stating plainly given it drives certificate issuance. Eleven tests covering the three failure modes that produce no error: the resolved certbot_data paths, the logger name, and routes lost from the URL map. Also the /admin prefix invariant the module docstring warns about, and that the domain and email patterns still discriminate — they gate every certificate request.

  All three guards are mutation-checked: dropping a .parent fails 2 tests, reverting the logger to __name__ fails 1, and removing a route module from the __init__ imports fails 2.

Notes

• routes_obtain_execute.py is 449 lines and knowingly over the guidance. obtain_certificate_execute is a single 387-line try block, so module-level splitting cannot shrink it — that needs collaborators extracted from the body, which is a behavioural refactor and needs the behaviour pinned first. On a module that had no coverage at all, that is its own piece of work; tracked as Phase 3e-ii. The size test names it as a known exception and fails if it is ever silently joined by another.
• Verification. 22 of 23 top-level definitions are ast.dump()-identical. The one difference is register_certbot_routes, retyped into the package __init__: its docstring lost the trailing whitespace on one blank line. That is the whole diff — "...app.\n \n Routes" became "...app.\n\n Routes". The URL map is unchanged at 549 rules, 0 differences, all 14 certbot endpoints intact, and the four CERTBOT_* paths resolve to exactly their pre-split values.
• Four pre-existing F541 warnings (f-strings with no placeholders) moved across verbatim rather than being cleaned up, to keep the diff pure motion.

[2.147.0] - 2026-08-08 - Phase 3d: split the admin API routes

Changed

• webapp/admin/api.py (2,105 lines) is now the webapp/admin/api/ package — 13 modules plus a 95-line __init__. Unlike the last two phases this file was 21 ordinary top-level definitions rather than handlers nested inside one giant register(), so every definition moved verbatim.

  | Module | Lines | Contents | | --- | ---: | --- | | blueprint.py | 45 | The shared api_bp | | hostinfo.py | 81 | Host CPU sample cache and primary-IP detection | | motion.py | 98 | The NWS storm-motion parameter parser | | county.py | 164 | The county-wide heuristic and location terms | | display_data.py | 322 | One alert flattened for the detail views | | routes_geometry.py | 176 | /api/alerts/<id>/geometry | | routes_alert_detail.py | 337 | /alerts/<id> | | routes_alert_export.py | 265 | PDF, social-share image, IPAWS audio | | routes_alerts_list.py | 321 | /api/alerts and /api/alerts/historical | | routes_boundaries.py | 141 | /api/boundaries | | routes_system.py | 281 | /api/system_status, /api/system_health | | routes_system_history.py | 136 | /api/system_health/history | | routes_smart.py | 165 | /api/smart_diag | | __init__.py | 95 | register_api_routes and the side-effect imports |

  All 14 modules are within the 400-line guidance. The entry point is unchanged — webapp/admin/__init__.py still does from .api import register_api_routes, and it is the only name the package re-exports.

• Import blocks are derived, not hand-written. Guessing them produced 127 ruff errors on the first attempt. The generator now narrows each of the original file's import statements to the names the module actually uses, with the free-variable set computed by symtable rather than by counting ast.Name nodes. Name counting is scope-blind and had already claimed a local variable desc inside _extract_alert_display_data as a use of from sqlalchemy import desc — the same bug Phase 4a hit with a parameter named text.

• Three dead imports dropped out of the split: flask.current_app, app_utils.vtec.extract_vtec_identity and optimized_parsing.json_dumps were imported by the single-file module and used by none of it.

Fixed

• tests/test_api_field_fixes.py and tests/test_detect_county_wide_false_positive.py read the API source as text and would have broken on the move. They now scan the package directory, which also means a future split cannot make them pass vacuously: an assertion against a missing file fails loudly, but one against a shim that no longer holds the code would quietly succeed.

Added

• tests/test_api_package.py — the structural guards for the split, all covering things that fail silently: a route module dropped from the __init__ imports vanishes from the URL map without raising; the blueprint's import_name shifts if blueprint.py passes its own __name__; and the CPU sample cache stops updating if any module imports it by value.

  It also exercises the real _detect_county_wide against a stub alert. The existing regression tests reimplemented the two heuristics locally and then grepped the source to check the original still matched — a copy of the logic cannot catch a change to the original. Both heuristics are mutation-checked: removing the _multi_county_list guard fails 2 cases, and dropping county_short from the county_and_state test fails 2 more.

  Writing those cases surfaced that state_code holds the two-letter postal code (OH), not the spelled-out state. The _multi_county_list guard counts ", <state_code>" occurrences against area descriptions NWS writes as "Allen, OH; Putnam, OH; Van Wert, OH", so a fixture using "ohio" counts zero and reinstates the false positive the guard exists to prevent.

Notes

• The blueprint's root_path moved from webapp/admin to webapp/admin/api. import_name is unchanged, but it now names a package rather than a module, so Flask derives a different directory. Nothing reads it — the blueprint sets neither template_folder nor static_folder and never calls open_resource — and a test pins that, so adding one later is a deliberate decision rather than a surprise.
• Verification. 21 of 21 top-level definitions are ast.dump()-identical before and after, every non-blank line of the original lands in exactly one module (the 11 that do not are the re-rendered import statements, the module docstring and the Blueprint(...) line), and the app's URL map is unchanged: 549 rules, 0 differences, verified against a worktree at the pre-split commit.

[2.146.1] - 2026-08-08 - Restore the audio-source WebSocket push

Fixed

• The live audio-source list stopped updating over WebSocket. The Phase 3a split turned webapp/admin/audio_ingest.py into a package, and the package __init__ re-exports only the names it means to. The single-file module had imported AudioSourceConfigDB at its top, so the model was incidentally importable from it — and app_core/websocket_push.py imported it from there in two places. Both broke:

  • _emit_audio_sources_update() imports at the top of the function, outside its try, so every call raised ImportError before doing any work. The Audio Monitoring page's source list never received a push.
  • _refresh_config_cache() imports inside a try/except Exception that logs at debug level, so the shared audio-source config cache silently stayed empty for the process's whole life.

  Neither surfaced an error a human would see — one was swallowed at debug, the other logged once per emit into a stream nobody reads. Both now import the model from app_core.models, which is where every other consumer (app_core/audio/eas_monitor.py, eas_monitoring_service.py, the diagnostics scripts) already gets it. A routes package was never the right home for a model import.

Changed

• tests/test_audio_ingest_package.py now derives the shim's required exports from the tree instead of a hand-written list. The existing PUBLIC_IMPORTS tuple is kept as an explicit floor, but it is the exact mechanism that let this through: AudioSourceConfigDB was never added to it, so the guard was green while the import dangled. The new test_every_absolute_import_of_the_package_resolves AST-walks every .py in the repository for from webapp.admin.audio_ingest import … and asserts each name still resolves. Confirmed discriminating — it fails with the pre-fix source and passes after.

[2.146.0] - 2026-08-08 - Phase 3c: split the radio settings routes

Changed

• webapp/routes_settings_radio.py (2,781 lines) is now the webapp/radio_settings/ package — 17 modules behind a 52-line shim. It was eight module-level helpers plus a 2,114-line register() with 26 route handlers nested inside it.

  Every handler closed over exactly app and route_logger, so each one moved verbatim into a topic module that keeps its own small register(app, route_logger) — no reindentation, no signature changes.

  | Module | Lines | Contents | | --- | ---: | --- | | deps.py | 129 | The seams a test injects fakes for, plus the capture constants | | sdr_client.py | 64 | _send_sdr_command | | serialization.py | 162 | A RadioReceiver row rendered to the API payload | | payload.py | 291 | _parse_receiver_payload — the whole write-side contract | | sync.py | 174 | Reconciling the database against the live RadioManager | | routes_pages.py | 66 | The two rendered pages | | routes_receivers.py | 259 | Receiver CRUD | | routes_receiver_control.py | 241 | Restart, audio-monitor wiring | | routes_devices.py | 265 | Discovery, capabilities, frequency validation | | routes_presets.py | 49 | Built-in tuning presets | | routes_signal.py | 355 | Waveform and spectrum | | routes_monitoring.py | 64 | Dashboard status and the diagnostics summary | | routes_diagnostics_status.py | 306 | Diagnostics status, SoapySDR error decoding | | routes_diagnostics_capture.py | 275 | IQ capture request and download | | routes_diagnostics_waterfall.py | 315 | The waterfall view | | routes_diagnostics_analyze.py | 386 | Capture analysis and the auto-gain sweep | | __init__.py | 105 | register, fanning out to the topic modules |

  All 17 modules are within the 400-line guidance.

• Test seams are now reachable in one place. The names the radio tests inject fakes for — get_redis_client, get_radio_manager, _log_radio_event, RADIO_CAPTURE_DIR and the other capture constants — live in deps.py and are called through the module. A by-value import would have snapshotted the real object, so a stub set in one place would have been silently ignored everywhere else.

Fixed

• Nothing user-visible. 34 of 34 moved definitions are ast.dump()-identical, every non-blank line of the original lands in exactly one module, and the app's URL map is unchanged: 549 rules, 0 differences, verified against a worktree at the pre-split commit.

Notes

• Three functions import get_redis_client from app_core.redis_client locally, shadowing the module-level import from app_core.extensions — a genuinely different object. Those call sites were left exactly as they were; rewriting them would have changed behaviour, not just location.
• tests/test_8khz_stress_test.py::test_8khz_with_increasing_noise[0.2] is flaky: it generates noise with an unseeded random.randint, so the 20 % case sits close to the confidence threshold and fails occasionally. Unrelated to this change; noted here because it can turn a full-suite run red.

[2.145.0] - 2026-08-08 - Phase 3b-ii: split the /alerts browse surface

Changed

• The 385-line alerts() handler and its ~175-line PDF export are now the webapp/public/alerts_page/ package. This completes Phase 3b-ii — every module in webapp/public/ is now within the size guidance.

  Unlike the previous two, this one is a pipeline, so the modules are stages:

  | Module | Lines | Contents | | --- | ---: | --- | | filters.py | 210 | AlertFilters, the sortable-column allow-list, request parsing and clamping | | query.py | 133 | Search, exact filters, date range, VTEC, visibility rules, sorting | | pagination.py | 101 | MockPagination and the paginate-with-fallback | | options.py | 88 | Filter dropdown values and the headline counts | | enrichment.py | 160 | Audio map, manual activations, lazy IPAWS backfill | | pdf_export.py | 150 | The export's query, blocks and filter summary | | __init__.py | 94 | build_alerts_page |

  webapp/public/alerts.py drops from 648 to 112 lines.

• Added tests/test_public_alerts_page.py (73 tests). Neither handler had coverage. The suite pins the input clamping (page, per_page, sort, direction are all attacker-controlled), the VTEC override, every filter, the pagination fallback including iter_pages() elision, and the PDF export's formatting and truncation.

Fixed

• Removed a dead per_page capture in the PDF export that had been flagged by ruff (F841) for some time. ruff check --select F,E9 is now clean across the whole of webapp/public/.

Notes

• The PDF export applies a strict subset of the page's filters. It honours search, status, severity, event, source and show_expired, but not the date range, the VTEC event chain or the superseded rule — so a PDF exported from a filtered page can contain rows the page was hiding. This is pre-existing behaviour, now documented in pdf_export.py and pinned by test_pdf_export_ignores_filters_the_page_supports. Unifying the two query builders is a behaviour change and needs its own commit.

[2.144.0] - 2026-08-08 - Phase 3b-ii: split the /stats dashboard

Changed

• The 645-line stats() handler is now the webapp/public/stats_sections/ package. The handler was seventeen try/except blocks in a row, each running a few queries, writing into a shared stats_data dict and declaring its own fallback so one failing query could not lose the whole dashboard. That pattern is now expressed once, as a StatsSection contract:

  | Module | Lines | Contents | | --- | ---: | --- | | common.py | 81 | The StatsSection contract and the runner | | alerts_overview.py | 215 | Headline counts, boundary/status/severity/event breakdowns, urgency, certainty, message types | | timeline.py | 237 | Hour/weekday/month/year buckets and the recent-alert feed | | coverage.py | 143 | Most-affected boundaries, alert durations, coverage overlap | | broadcast.py | 173 | Forwarding rate, manual activations, received alerts, broadcast latency, relay stats | | polling.py | 183 | Poller success rate, timings and trend | | __init__.py | 100 | The ordered pipeline and build_stats_data |

  webapp/public/stats.py drops from 693 to 50 lines. The pipeline order is declared explicitly in one place because it is load-bearing — three sections divide by total_alerts and must run after the counts.

• Added tests/test_public_stats_sections.py (32 tests). The handler had no coverage; the suite pins every derived rate, every time bucket, the per-section fallbacks, and the full set of keys stats.html indexes.

Fixed

• Nothing user-visible. The rendered payload was compared key-by-key against the pre-refactor handler across an empty and a populated database — 70 keys, zero differences — and the test suite was mutation-checked before (17/17 caught) and after (17/17 caught) the split.

Notes

• 29 of the handler's 31 trailing setdefault calls were dead. Every section already set its keys on both its success and its failure path, so those defaults could never fire. Only avg_durations and lifecycle_timeline had no producer; the new package keeps just those two and the section contract now guarantees the rest structurally.

[2.143.0] - 2026-08-08 - Phase 3b-ii: split the /logs query layer

Changed

• _load_logs_data (1,057 lines in a single function) is now the webapp/public/logs_sources/ package, nine modules behind a shared contract. This is the Phase 3b-ii follow-up recorded in docs/development/LARGE_FILE_REFACTOR_PLAN.md — the first of the three webapp/public/ modules that Phase 3b left over the size guidance.

  The function was a seventeen-way if/elif on log_type, each branch querying its own source and shaping rows into the generic log dict the template renders. Every branch is now a loader taking one LogQuery and returning one LogPage, so the dispatcher is a table lookup:

  | Module | Lines | Contents | | --- | ---: | --- | | common.py | 83 | The LogQuery / LogPage contract and the shared timestamp sort key | | database.py | 329 | System, polling, polling-debug, audio, audio-metrics, audio-health, GPIO | | eas.py | 304 | EAS messages, decoded audio, manual activations, received alerts | | audit.py | 130 | The audit trail and the compliance ledger | | reports.py | 146 | The six FCC report kinds and the report metadata envelope | | services.py | 90 | The systemd journal category | | aggregate.py | 112 | The "All Logs" merge, fault tolerance and truncation | | aggregate_collectors.py | 346 | The eleven per-category collectors the merge runs | | __init__.py | 81 | LOADERS and resolve_loader |

  webapp/public/logs_data.py drops from 1,116 to 80 lines and now only dispatches; MIN_LOGS_PER_CATEGORY is re-exported so the old import path still resolves. Adding a log category no longer means editing a 1,000-line function — it means writing a loader and registering it.

• Added tests/test_public_logs_data.py (79 tests). The loader previously had no test coverage at all. The suite asserts the complete returned triple for every log type — display name, every key of every row, and the report metadata — plus the per-branch level-derivation rules, the fallback strings for missing fields, the audit action filter running in SQL, the session rollback after a failing report builder, and the limit arithmetic.

Fixed

• Nothing user-visible. The split is behaviour-preserving: the loader's full output was compared across 115 scenarios (23 log types × 5 parameter combinations) against the pre-refactor code with zero differences, and the test suite was mutation-checked both before the refactor (15/15 caught) and after it (16/16 caught) to prove it would notice if that changed.

[2.142.0] - 2026-08-07 - Phase 4a: split the system-monitoring helpers

Changed

• app_utils/system.py (2,580 lines) is now the app_utils/system/ package, 16 focused modules behind a re-exporting __init__. This is the Phase 4 app_utils/system.py entry in docs/development/LARGE_FILE_REFACTOR_PLAN.md — the one the plan flagged as "mostly independent helpers — easier than it looks", and it was: 47 top-level definitions sharing one file, with a strictly acyclic internal dependency graph and no __file__-relative paths to shift.

  | Module | Lines | Contents | | --- | ---: | --- | | common.py | 102 | SystemHealth, _safe_read_text, _safe_int, _coerce_int, _to_bool, _is_valid_temperature | | dependencies.py | 96 | _collect_dependency_versions | | services.py | 146 | _collect_systemd_services | | badges.py | 189 | Shields.io badges and the distro logo URL | | osinfo.py | 131 | OS release details and virtualization detection | | network.py | 111 | Interface traffic counters, primary-interface selection | | device_tree.py | 110 | DEVICE_TREE_CANDIDATES and the device-tree readers | | block_devices.py | 163 | lsblk inventory and its simplifier | | hardware.py | 276 | CPU, USB and platform inventory | | disks.py | 99 | Disk enumeration and device-type detection | | smart.py | 429 | _collect_smart_health | | smart_fields.py | 202 | smartctl/NVMe field extraction | | temperature.py | 177 | Thermal sensor readings | | rtc.py | 127 | Real-time clock status | | subsystems.py | 147 | Hardware subsystem and GPS probes | | snapshot.py | 478 | build_system_health_snapshot |

  Pure motion, verified as such: 48 top-level definitions, 48 ast.dump() matches, zero differences, plus the standing assertion that every non-blank line of the original lands in exactly one module.

• DEVICE_TREE_CANDIDATES is deliberately not re-exported from the package. It is a mutable list that two tests replace with monkeypatch.setattr; re-exporting it would have turned both patches into silent no-ops, since device_tree resolves the name from its own globals. Leaving it out made both fail loudly with AttributeError, which is what pointed at the right patch target. Same for Path and psutil in the CPU-details test — all three now patch app_utils.system.<module>.

Fixed

• docs/guides/SMART_SETUP.md pointed at line ranges in the old monolith (app_utils/system.py lines 1276-1557) and at webapp/routes_public.py for the health-page route, which moved to webapp/public/pages.py in 2.141.0. Both now name modules and functions rather than line numbers.

Notes

• smart.py (429) and snapshot.py (478) are still over the 400-line guidance. Each is one function — _collect_smart_health is 396 lines and build_system_health_snapshot is 406 — so module-level splitting cannot shrink them; they need collaborator extraction with a characterization harness built first, as in Phase 2e / 3a-ii. Tracked as Phase 4a-ii.
• One pre-existing lint finding was left alone deliberately: smart.py has an unused device_name local (F841), present in the monolith. Fixing it is a behaviour-adjacent change and does not belong in a pure-motion commit.

[2.141.0] - 2026-08-07 - Phase 3b: split the public route surface

Changed

• webapp/routes_public.py (2,849 lines) is now the webapp/public/ package, one module per surface. This is Phase 3b of docs/development/LARGE_FILE_REFACTOR_PLAN.md.

  Unlike the audio_ingest split in 2.139.0, this module had exactly one top-level definition: its entire body was a single 2,779-line register(app, logger) with all 21 route handlers nested inside it. Nothing could be moved at module level.

  That made the split simpler rather than harder. Every handler closed over only app (for the @app.route decorator) and route_logger — verified by walking the AST for names resolving to register's scope, not assumed — so each surface keeps its own register(app, route_logger) and the handler bodies move verbatim, still nested inside a register, just a much smaller one. No reindentation, no rebinding, no signature changes.

  | New module | Lines | Contents | | --- | ---: | --- | | pages.py | 189 | /, /about, /help, /style-guide, /attribution, /support, /navigation, /terms, /privacy, /sms-compliance, /system_health, /audio-monitor | | sitemap.py | 115 | /sitemap.xml | | stats.py | 693 | /stats | | alerts.py | 648 | /alerts, /alerts/export.pdf | | logs_data.py | 1116 | _load_logs_data, via a build_logs_loader(route_logger) factory | | logs.py | 265 | /logs, /logs/export.csv, /logs/export.pdf |

  _load_logs_data is a helper the three /logs handlers share, not a route. Wrapping it in a factory rather than re-signaturing it to take route_logger explicitly keeps its 1,057-line body byte-identical — only the enclosing scope changed. logs.py receives it as a parameter named _load_logs_data, so its three call sites resolve unchanged.

  webapp/routes_public.py remains as a 31-line shim re-exporting register, so the route-module registry in webapp/__init__.py is untouched.

  Verified three independent ways, each confirmed discriminating before its result was trusted: 21/21 handlers ast.dump()-identical with every non-blank line placed exactly once (2,753 lines, 0 unplaced); the Flask URL map diffed against a worktree at the pre-split commit (549 rules, 0 differences, mutation-checked by deleting a register() call); and all 28 public surfaces fetched through the test client on both sides and hashed (28/28 identical, 28 distinct digests). Full suite: 2,045 passed.

  logs_data.py, stats.py and alerts.py remain over the 400-line guidance — each is dominated by one enormous function, which module-level splitting cannot shrink. Tracked as Phase 3b-ii.