Hiro development journal

Lifecycle controls and evaluation timeout-cascade diagnosis

2026.07.14 Detailed development update Machine-readable JSON

Executive summary

Hiro's local operating workflow was tightened so the model server and Telegram integration behave as one intentional lifecycle instead of unrelated background processes. Starting Hiro now enables Telegram notifications and starts the bot, while the new shutdown path disables notifications before stopping services.

The change also teaches Hiro's monitoring and notification layers to distinguish an intentional shutdown from a failure. That prevents avoidable alerts and automatic recovery behavior when the operator has deliberately taken the system offline.

Documentation and focused tests were added alongside the implementation. The code and test intent were inspected during this work session, but the targeted Python test could not be executed in the current tool environment because a project Python runtime with pytest was not available. Full runtime verification therefore remains an explicit follow-up.

A later controlled laboratory session validated the reliability branch with all 75 Python tests passing, then ran repeated public and repository-external held-out evaluations using only the local Gemma model. Four consecutive daytime repetitions converged on the same degraded result, establishing that the failure was systematic rather than sampling noise.

Detailed tracing identified a timeout cascade: Hiro's full agent request currently takes about 97 seconds, while each evaluator case is limited to 60 seconds. Cancellation then leaves the single model slot occupied for roughly 16 additional seconds, consuming the next case's budget and propagating failures through the suite. No behavioral candidate was promoted.

Work completed

Unified startup and shutdown behavior

Implemented

The existing model-server startup command now invokes a dedicated Telegram lifecycle helper after launching the local model.
The helper is idempotent: it reuses an already-running managed Telegram process instead of starting duplicates.
A companion shutdown command disables Telegram notifications first, stops the managed bot, records that downtime is intentional, and then stops local model-server processes.
Runtime marker and process files are kept in a local-only state directory that is excluded from version control.

Notification gating

Implemented

Central configuration now exposes explicit state for whether Telegram notifications are enabled and whether Hiro's servers were intentionally stopped.
Improvement-cycle summaries, regression alerts, and task-runner messages check the lifecycle state before attempting a Telegram delivery.
Disabling notifications is therefore a system-level state rather than a convention that each operator must remember manually.

Downtime-aware monitoring

Implemented

The local model health check returns without trying to restart the model when an intentional-stop marker is present.
The continuous health monitor similarly suppresses model-unavailable incidents during deliberate downtime.
Starting the system clears the intentional-stop state, restoring normal health checks and recovery behavior.

Operator documentation and tests

Added

The project README now describes the coupled startup behavior, the new shutdown command, and the reduced set of processes that must be started manually.
Focused tests cover the lifecycle marker semantics and confirm that notification delivery remains disabled when the enabling marker is absent.
The test implementation avoids making a real outbound Telegram request.

Controlled evaluation reliability study

Diagnosed; promotion rejected

A clean external worktree and local-only branch isolated the study from Hiro's uncommitted lifecycle work.
The local model identity, startup latency, warmup latency, external-suite fingerprint, single-writer discipline, and protected-file scope were checked before evaluation.
Three requested sequential repetitions produced identical held-out metrics: 12.5 percent pass rate, a weighted score of about 0.519, 24 invariant failures per run, and latency pinned near the 60-second case deadline.
Across those three repetitions, 6 of 48 held-out observations passed and 72 invariant failures were recorded. Combined with an earlier matching run, four consecutive repetitions reproduced the degraded state.
Earlier nighttime measurements had reached a 75 percent held-out pass rate, demonstrating that the current daytime failure was an operational reliability regression rather than a justified response-behavior change.

Timeout and cancellation tracing

Root cause confirmed

Per-case inspection showed that deterministic resolver responses completed in milliseconds while every model-backed case reached the 60-second deadline with an empty response.
A direct short model request completed in about 32.6 seconds. The exact Hiro agent path used a roughly 1,662-token prompt and completed correctly in about 97.0 seconds: approximately 82.8 seconds of prompt processing followed by 13.9 seconds for nine generated tokens.
A controlled cancellation reproduced delayed recovery. The client stopped at 60 seconds, but the model server retained its only slot for approximately 16 more seconds before releasing it.
The evidence supports a combined cause: the evaluator deadline is shorter than current end-to-end latency, and delayed single-slot cancellation causes subsequent requests to inherit queue delay.

Decisions and reasoning

Use explicit marker files for operator intent. Process availability alone cannot distinguish a crash from a deliberate shutdown, while persisted local intent can be read consistently by independent Hiro components.
Disable notifications before stopping other services. This ordering closes the window in which shutdown-related health failures could generate false alerts.
Manage only the Telegram process started by the lifecycle helper. The helper validates the saved process identity before stopping it, reducing the chance of affecting an unrelated process.
Keep lifecycle state outside version control. The markers describe one machine's current runtime state and are not project configuration.
Do not promote response behavior from the current measurements. Repeated failures were dominated by infrastructure timing and invariant regressions, not credible evidence of a behavioral improvement.
Stop repeating an unchanged evaluation once identical timeout-bound results become systematic. Further repetitions have diminishing diagnostic value until cancellation recovery and throughput gating are implemented.
Treat resolver false positives as a separate measured problem. They should be addressed only with a narrowly scoped hypothesis and matched validation after the evaluation harness is reliable.

Validation and evidence

Check	Status	Result
Implementation inspection	passed	Startup, shutdown, configuration, monitoring, notification, documentation, and test changes were reviewed together for consistent lifecycle semantics.
Secret and security-content screening	passed	This update contains no credentials, API values, private identifiers, or actionable descriptions of an unresolved security weakness.
Focused Python tests	not-run	The current tool environment did not expose a configured Python installation with pytest, so the new targeted tests remain to be run in Hiro's normal development runtime.
Live process-cycle exercise	not-run	A real start-stop-start exercise was not performed during this publication step because it would alter the operator's currently running local services.
Hiro full Python suite	passed	All 75 tests passed on the isolated reliability branch. The changed-file audit was limited to local model readiness code and its focused tests.
Repeated public and external held-out evaluations	passed-with-regression	The evaluator completed all requested sequential runs and wrote their append-only records successfully, but response quality was rejected because the three requested held-out repetitions each passed only 2 of 16 observations and recorded 24 invariant failures.
Extended core-agent latency trace	passed	The exact model-backed identity request completed correctly under an extended diagnostic deadline in about 97.0 seconds, proving the model was reachable and the 60-second evaluation deadline was insufficient under current throughput.
Cancellation recovery trace	passed	A controlled 60-second cancellation showed that the server's single slot remained occupied for approximately 16 seconds after client cancellation, confirming the cascade mechanism.

Current state

The lifecycle implementation and its focused tests are present in Hiro's current development working tree.
Telegram notification attempts are gated by explicit lifecycle state across the main notification paths touched by this change.
Intentional downtime is represented separately from an unexpected health failure.
Full automated and live operational verification is still pending.
The lifecycle implementation was subsequently exercised during controlled local sessions: notifications were disabled before model shutdown, no evaluator or model process remained afterward, and the dashboard was preserved.
Evaluation measurements are not currently suitable for behavioral promotion because model-backed requests can exceed their deadlines and contaminate following cases through delayed slot recovery.
No resolver, response-latency, baseline-pin, or promotion candidate was accepted from the diagnosed runs.

Next steps

Run the focused Telegram lifecycle tests in Hiro's configured Python environment.
Exercise the full startup, status, shutdown, and restart sequence on the development machine and confirm process and marker cleanup.
Run the broader integration suite to detect interactions with scheduling, monitoring, and improvement-cycle reporting.
Commit the verified Hiro lifecycle changes once the operational checks pass.
Add cancellation recovery that waits for the local model slot to become idle before advancing to another evaluation case.
Record per-case slot state and server timing so prompt processing, generation, queueing, and cancellation delays can be distinguished.
Add a readiness and throughput probe that refuses to start a measurement when predicted latency cannot fit within the evaluation deadline.
Calibrate the evaluation timeout or reduce evaluation prompt overhead without changing scoring, held-out content, safety policy, or grounding requirements.
Run matched baseline and candidate repetitions only after three consecutive stable baseline measurements are available.