Hiro development journal

Nightly evaluation connection preflight repaired and verified

2026.07.21 at 07:56:36 America/Los_Angeles Published Machine-readable JSON

Executive summary

The nightly evaluation connection failure was traced to a dependency gap: the preflight kept Hiro's API available but did not ensure the local Gemma inference server was available on port 8080, which the evaluator calls directly.

The nightly preflight now invokes the existing idempotent Gemma launcher, which validates model identity and warms the model, before it accepts an already-running API as sufficient readiness.

A real preflight confirmed the API and Gemma model endpoint. Today's fresh proposal-only evaluation then completed all 16 public RSI observations at 100 percent pass rate with zero invariant failures and no observation errors.

The connection defect is resolved for the tested path. Five of 22 lightweight regression probes still failed on separate functional expectations; those are reported as follow-up quality work, not connection failures.

Work completed

Connection-path diagnosis and repair

Completed

Confirmed that the prior failed evaluation had an available API process but no listening model endpoint before the repair investigation began.
Added the existing start_gemma_server.ps1 launcher to nightly_api_preflight.ps1 after the required Path/PATH normalization and before the API readiness short-circuit.
The preflight now validates that its model launcher exists, exposes it in validate-only output, and fails rather than permitting an evaluation when Gemma cannot become ready.
The model launcher is idempotent and verifies model identity plus a warm-up response, so the preflight does not create a duplicate model service when Gemma is already healthy.

Focused and live validation

Completed

Focused tests for nightly preflight, nightly evaluation, and the Gemma launcher passed: 8 passed.
A real preflight confirmed the configured Gemma endpoint exposed the expected model and the Evaluation Observatory API returned HTTP 200.
The initial manual evaluator invocation did not execute because the interactive console lacked UTF-8 output configuration. Retrying with the scheduled API's UTF-8 setting completed successfully; this is an invocation-environment issue, not an evaluation failure.
The successful manual evaluation did emit non-fatal Windows log-rotation warnings because another process held the shared main log open. The run completed, wrote its append-only results, and exited with code 0.

Today?s fresh nightly evaluation

Completed

Run nightly-2026-07-21-rsi-cycle-public-1-0-0-db468ad4f84c started at 2026-07-21T14:55:04Z and completed at 2026-07-21T14:55:12Z in new mode.
All 16 public RSI observations passed. The ledger reports zero invariant failures, 100 percent pass rate, weighted score 1.0, p95 observation latency about 766 milliseconds, and no observation errors.
The proposal-only cycle completed in 1.2 minutes, reviewed 100 response envelopes, and created approval-required proposals from organic-envelope evidence rather than from the now-passing synthetic evaluation.

Decisions and reasoning

Treat model availability as a required precondition for a nightly evaluation, not as an implication of API availability.
Reuse the checked-in Gemma launcher instead of adding a second startup mechanism so model identity, warm-up, and idempotency remain centralized.
Count the July 21 run as a valid completed night because the model was reachable, all observations completed, and the append-only ledger has a complete error-free result set.
Do not infer that all quality signals are healthy: five lightweight regression probes remain failing and require separate, evidence-based triage before any broader process adjustment.

Validation and evidence

Check	Status	Result
Focused Hiro test set	passed	tests/test_nightly_api_preflight.py, tests/test_nightly_evaluation.py, and tests/test_gemma_launcher.py: 8 passed.
Real model and API preflight	passed	The preflight found the expected Gemma model on port 8080 and the Evaluation Observatory API returned HTTP 200.
Fresh nightly public evaluation	passed	16 of 16 observations passed with zero invariant failures and no connection or model errors.
Lightweight regression probes	passed-with-functional-failures	22 probes completed; 17 passed and 5 failed. The failures are separate functional expectations, not connection failures.

Current state

The nightly preflight now verifies the model dependency before declaring readiness.
Today's valid completed nightly evaluation restores the comparable successful-night count to two when combined with the prior valid RSI night; the July 20 infrastructure failure remains excluded.
The process remains short of the minimum three valid nights and still needs recurring-infrastructure evidence plus retained regression-probe outcomes before expansion is recommended.
No automatic promotion or behavioral code change was applied from the generated proposals.

Next steps

Observe at least one more valid scheduled night for the minimum readiness threshold, preferably three more for the preferred threshold.
Preserve model/API preflight evidence with each nightly record and retain regression-probe pass/fail outcomes, not only a probe count.
Triage the five remaining lightweight regression-probe failures independently of the resolved connection issue.
Only after stable successful nights and probe evidence are available, seek approval for bounded expansion favoring broader diversity and adaptive testing before additional nightly repetitions.