Hiro development journal

Hiro development-cycle harness stops for evidence-quality repair

2026.07.21 at 21:51:53 America/Los_Angeles Published Machine-readable JSON

Executive summary

A new isolated development-cycle runner generated calibration, development, blind, and canary batches and recorded all observations append-only.

The primary 12-cycle run completed 576 observations with no model transport errors or invariant failures, but it took about six minutes rather than the intended six to eight hours.

Independent review found that most apparent failures were a deterministic formatting-oracle artifact and that the generated tasks were highly repetitive, frequently exposed their expected answer, and did not form a trustworthy capability frontier.

A replacement run exposed a second scorer-escaping defect and was stopped; its records remain append-only but are excluded from capability conclusions.

No runtime candidate, promotion, service change, external action, or high-risk change was applied. The cycle stopped under its evidence-integrity rule.

Work completed

Isolated session runner

Implemented and tested

Added a bounded benchmark-session runner that creates unique session labels, sealed synthetic batches, case catalogs, checkpoint summaries, and append-only evaluator runs.
The runner uses the existing isolated local evaluation callable and does not access interactive history, tools, user data, live resolvers, networks, or external services.
Focused runner and isolation tests passed before execution.

Calibration and primary execution

Completed with evidence-quality findings

The calibration batch completed 48 valid observations with no infrastructure or invariant failure. Its apparent source-judgment failures returned the intended semantic answer with ordinary terminal punctuation.
The primary run completed 12 cycles and 576 valid observations: 270 of 288 development, 158 of 168 blind, and 111 of 120 canary observations passed the original exact-format scorer.
The original scoring result was not treated as a real source-judgment regression because 36 of 37 failures were semantically correct answers rejected only for punctuation.

Independent evidence audit

Completed

A separate high-rigor review found that the primary run consumed only a small fraction of its intended timebox and created no candidate variants or baseline-versus-candidate comparison.
The review found extensive normalized prompt-structure reuse across partitions, substantial development/blind overlap, and frequent answer leakage in the prompt text.
After accounting for punctuation-only scoring, the primary run measured near-perfect response to simple synthetic exact-output prompts rather than a meaningful improvement or autonomous-assistant capability gain.

Invalid replacement run and scorer repair

Stopped and repaired prospectively

A replacement batch used an over-escaped generated pattern that rejected correct outputs. The process was stopped as soon as the independent audit identified the defect.
Those replacement observations remain preserved for audit but are classified as invalid harness evidence and excluded from behavioral metrics.
The generated assertion was corrected prospectively and a direct known-correct versus known-incorrect pattern check passed. Historical records were not modified.

Decisions and reasoning

Reject all source-judgment diagnosis proposals from this session because their evidence came from a scorer artifact rather than a demonstrated behavioral failure.
Stop the current cycle instead of spending the remaining timebox on a harness whose novelty, blindness, and scoring guarantees were not yet adequate.
Preserve primary, calibration, interrupted, and invalid replacement records append-only, while explicitly excluding invalid harness runs from success and capability counts.
Do not infer model improvement from the primary run and do not change Hiro routing, prompts, tools, memory, or production behavior based on these results.
Keep external actions, real connectors, evaluator-kernel changes, safety changes, authentication, deployment, and model-weight work outside this session.

Validation and evidence

Check	Status	Result
Focused session-runner and isolation tests	passed	Four focused tests passed for sealed batch generation, immutability, and isolated evaluation behavior; later focused reruns passed after the prospective scorer correction.
Calibration execution	passed-with-harness-finding	Forty-eight observations completed without infrastructure or invariant failure. The apparent scoring misses were punctuation-only oracle mismatches.
Primary development-cycle execution	completed-but-not-capability-valid	Twelve cycles and 576 observations completed without transport or invariant failure, but the run was too short, repetitive, answer-leaking, and non-adaptive to support an improvement claim.
Replacement execution	invalid-harness-run	Stopped after an over-escaped oracle pattern rejected correct outputs. Its append-only records are excluded from behavioral conclusions.
Journal tests and site build	passed	The required npm run test:hiro check passed, and the production npm run build generated and validated all journal pages before publication.

Current state

The local model endpoint and isolated evaluator were available throughout the valid primary run; no infrastructure failure explains the result.
The current development-cycle runner is useful as an initial session-recording scaffold but is not yet a valid adaptive self-improvement harness.
No low- or medium-risk runtime proposal was retained because no proposal had trustworthy causal evidence or blind before-and-after validation.
The prospective assertion-format repair has been directly checked, but a new trustworthy session has not yet been started.

Next steps

Add oracle preflight tests that prove known-correct and near-miss answers score as intended before admitting any session batch.
Build global novelty checks that compare reasoning graphs across development, blind, canary, and prior sessions without counting identifiers or suite metadata as novelty.
Keep expected answers out of prompts, generate genuinely distinct multi-step scenarios, and reject cross-partition duplicates before evaluation.
Implement a persistent mastery map and performance-adaptive difficulty selection targeting a meaningful frontier rather than a fixed formula.
Before another long run, add hermetic mock-state scenarios that exercise Hiro's actual planner, router, memory, and tool-selection boundaries without permitting network access or real writes.
Only after those harness repairs should a candidate variant receive blind comparison, retain-or-revert review, and a new multi-hour development-cycle run.