Hiro development journal

Model-unreachable Telegram alerts removed

2026.07.20 at 18:08:41 America/Los_Angeles Published Machine-readable JSON

Executive summary

Stopped Hiro's health monitor from sending Telegram messages when the active local model endpoint is unreachable or returns an error. Model availability remains visible to health checks and local logs, but routine model downtime no longer interrupts the user through Telegram.

Suppressed both messages associated with a model-availability incident: the initial health alert and the later self-recovery success or failure notification. Other health, lab, lifecycle, and improvement notifications retain their existing Telegram behavior.

Restarted the API/dashboard service so the policy took effect immediately and validated the exact unavailable-model condition that had been generating the unwanted messages.

Work completed

Selective Telegram suppression

Completed

Added an optional telegram_alert policy flag to health issues while preserving the previous default of sending alerts.
Marked both LLM_PRIMARY_DOWN and LLM_PRIMARY_ERROR as Telegram-silent at the point where those issues are created.
Kept model health detection and local recovery behavior active so silencing Telegram does not hide the condition from Hiro's internal monitoring.
Propagated the same policy into model recovery reporting so a silent reachability incident cannot produce a follow-up self-fix Telegram message.

Regression coverage

Completed

Added coverage proving that a model-availability issue is returned by the monitor without calling the Telegram sender.
Added direct coverage proving an HTTP failure from the model endpoint is tagged Telegram-silent.
Added coverage proving model recovery follow-ups remain silent and an unrelated LAB_STUCK issue still sends normally.
Ran the new health-monitor tests together with the existing Telegram lifecycle tests.

Runtime activation

Completed

Identified the API/dashboard process as the owner of the background health-monitor loop.
Restarted the service with the same local runtime, host, port, and application target so the new policy was loaded immediately.
Confirmed the dashboard evaluation endpoint remained available after restart while the health endpoint accurately reported that no model endpoint was currently reachable.

Decisions and reasoning

Suppress model reachability notifications by issue policy instead of disabling Telegram globally, preserving useful non-model alerts.
Keep model health checks active rather than removing them; observability and local recovery are still useful even when Telegram escalation is not.
Suppress recovery-result messages when they originate from a silent model incident so the user does not receive an indirect version of the same unwanted alert.
Limit this change to notification behavior. The legacy checked-in model-profile fallback was observed but not changed as part of this focused session.

Validation and evidence

Check	Status	Result
Focused notification regression suite	passed	Six focused tests passed, covering model issue suppression, direct model HTTP failure tagging, silent recovery reporting, preservation of non-model alerts, and Telegram lifecycle gating.
Runtime service restart	passed	The API/dashboard service restarted successfully, loaded the updated monitor, and listened on its established local port without startup errors.
Dashboard availability after restart	passed	The evaluation overview endpoint returned HTTP 200 after restart. The health endpoint returned the expected unavailable status because no model endpoint was reachable.
Public journal production build	passed	The timestamp identity tests and full frontend production build passed with eight journal entries; generated validation passed the timestamp schema, same-day identity, alias, sitemap, Atom, noindex, and IndexNow-key checks.
Production journal retrieval	passed	The production retrieval validator completed 74 requests with zero failures, covering the timestamped HTML and JSON, same-day sibling, latest aliases, schema-version-2 index, Atom feed, sitemap, noindex metadata, and multiple user-agent profiles.

Current state

Model-unreachable and model-HTTP-error incidents are retained internally but do not send Telegram alerts.
Model self-recovery success or failure messages are also silent when they originate from those incidents.
Other health-monitor issues continue to use Telegram unless they explicitly opt out.
The restarted dashboard/API service is available; the local model endpoint was unavailable during validation.
The checked-in default model profile still references the legacy Gemma profile, even though current model work has moved to Qwen.
This timestamped journal entry is live and independently retrievable alongside the day's earlier entries.

Next steps

Observe the next model-down health interval and confirm no Telegram message is received.
Align the checked-in active model fallback and recovery launcher with the current Qwen deployment in a separate focused change.