Hiro development journal

Stopping an incident-specific repair loop for an architecture reset

2026.08.17 at 10:17:01 America/Los_Angeles Pipeline audit completed; Hiro paused before further candidate work Machine-readable JSON

Executive summary

A full-path audit repaired multiple real defects in candidate construction, evaluation reachability, runner ownership, causal comparison, and fresh-canary handling. The latest Hiro regression suite passed 672 tests in 187.60 seconds.

The target arithmetic proposal reached deterministic replay and corrected public and held-out evaluation, but an independent fresh canary returned a bare result that violated the response contract. The candidate was not promoted.

Subsequent model-generated revisions failed during isolated construction. Repeatedly adding arithmetic-specific instructions to rescue that one incident would overfit the harness and would not demonstrate a general self-improvement capability.

Hiro was therefore paused. The approval architecture is retained as useful infrastructure, while the candidate-construction strategy now requires a deliberate reset around generic primitives, explicit reachability evidence, portfolio proposals, and governed stop rules.

Work completed

End-to-end execution audit

Completed

Audited the path from active queue selection through reproduction, isolated construction, targeted contrast, public and held-out evaluation, integration, independent canaries, and final governance.
Corrected stale service-state observations by restarting the exact process that owned Hiro's listening ports and verifying the running revision rather than assuming disk changes were live.
Hardened Windows runner serialization, orphaned lease reclamation, bounded model context, callable-signature evidence, candidate test execution, and category-regression replication.
Required candidates to demonstrate user-visible behavior and to run fresh candidate-authored tests instead of relying only on static patch inspection.

Causal evaluation correction

Completed

Found that the isolated evaluator's default agent entrypoint bypassed the response boundary modified by the candidate, making the candidate causally unreachable even though the evaluation appeared to run normally.
Routed evaluated output through the actual response boundary, made the diagnostic comparison deterministic, and inferred the applicable response task type when production callers omitted it.
The corrected Stage 4 comparison produced identical baseline and candidate aggregate scores: 0.92917 on the public set and 0.90417 on the held-out set, with no category regression. This supported progression but did not establish a positive global effect.

Independent canary result

Completed with candidate failure

The retained incident reproduction passed after the candidate normalized the expected formatting boundary.
A fresh independent model execution returned only the numeric result. That output was numerically correct but failed the user-visible response contract, proving that the initial patch did not generalize to a second sample.
The canary correctly blocked promotion. The system now permits one bounded canary-informed revision while preserving the fresh evidence that motivated it.

Candidate revision autopsy

Stopped after non-convergent construction

The local proposal model attempted five isolated revisions intended to handle both the captured formatted answer and the fresh bare answer.
Attempts failed for distinct implementation reasons, including an undefined helper, a mismatched expected status value, and digit parsing that combined operands with the result rather than reconstructing a valid explanation.
The resulting record is artifact blocked, not idea rejected. No patch from these revisions entered the production tree and no autonomous promotion occurred.
Further arithmetic-specific builder instructions were halted because success under increasingly incident-specific coaching would not be evidence of a general repair capability.

Decisions and reasoning

Keep the generic safety and validity improvements: serialized ownership, bounded context, fresh test execution, response-contract guards, causal evaluator reachability, deterministic diagnostic comparison, replicated category checks, fresh canaries, and preserved canary evidence.
Stop changing general candidate-builder policy in response to one proposal. A platform change should require a recurring failure pattern across multiple unrelated incidents or an independently specified invariant.
Separate approval-system validity from candidate-synthesis capability. This run supplies evidence that the gates can stop a superficially successful but non-general patch; it does not supply evidence that the local model can reliably construct improvements.
Treat a failed fresh canary as a new hypothesis requiring explicit evidence, not as permission for unlimited silent refinement of the same candidate.
Pause Hiro before another candidate and conduct a bounded architecture reset rather than continuing prompt-level repairs.

Validation and evidence

Check	Status	Result
Latest Hiro regression suite	passed	672 tests passed in 187.60 seconds at revision 18efa74651040ef69ed852a7d8e3e149d058f632.
Corrected public and held-out comparison	passed as a non-regression check	Baseline and candidate scored 0.92917 on public cases and 0.90417 on held-out cases, with zero measured category regression. The result was neutral rather than evidence of broad improvement.
Captured incident replay	passed for the initial candidate	The retained deterministic arithmetic incident satisfied its response contract after candidate normalization.
Fresh independent canary	failed	An independent execution returned a bare numeric answer and failed the required user-visible response contract, correctly preventing promotion.
Subsequent candidate construction	artifact blocked	Five isolated model-generated repair attempts failed their targeted tests. The idea was not classified as disproven and no patch was promoted.
Continuous Hiro service	paused intentionally	The process owning Hiro's local service ports was stopped after the architecture concern was confirmed.

Current state

The arithmetic incident remains artifact blocked and has not been implemented or promoted.
The most recent candidate passed captured replay and neutral non-regression evaluation but failed an independent fresh canary.
Hiro's continuous service is paused; the local model service remains available.
The repository contains broad pipeline hardening, but the session did not demonstrate a successful autonomous improvement.

Next steps

Freeze the approval platform at a reviewed revision and distinguish future platform defects from ordinary candidate failures.
Introduce generic candidate-construction primitives such as bounded source transformations, contract-derived test scaffolds, and deterministic edit operations. Let the model select and compose primitives instead of authoring an unrestricted boundary patch.
Generate a small portfolio of independent candidate hypotheses and compare them under the same fixed gates rather than repeatedly modifying the harness to rescue one proposal.
Instrument candidate reachability and require evidence that changed functions or lines execute during targeted tests and fresh canaries before causal credit is assigned.
Add failure-taxonomy stop rules: an incident may trigger candidate revision, but platform policy changes require the same structural defect across several unrelated incidents.
Use the local model primarily for proposal generation and critique until measured construction reliability supports granting it broader patch-authoring responsibility.
Resume the continuous queue only after the reset has explicit success criteria, a fixed evaluation protocol, and a bounded experiment budget.