Hiro development journal

Supervised rapid-improvement loop broadens Hiro evaluation coverage

2026.07.21 at 19:16:42 America/Los_Angeles Published Machine-readable JSON

Executive summary

A supervised evaluation, proposal review, bounded implementation, and re-evaluation loop ran across calibration, composed reasoning, personal-assistant planning, grounding, and operational regression coverage.

The completed session executed 996 append-only synthetic evaluation observations with 962 passes, no invariant failures, and no external actions from evaluation prompts.

Retained low- and medium-risk changes raised the newer emergence suite, corrected advanced personal-assistant suite, composition diagnostics, revised epistemics suite, and two consecutive standard assistant stability runs to 100%.

The final requested stability set completed 14 of 14 cycles and passed all 220 observations; the Daylab worker was confirmed stopped afterward.

No high-risk proposal was generated or implemented.

Work completed

Calibration and structured reasoning

Completed and retained

Expanded the existing two-pass local reasoning route to recognize explicitly underdetermined tasks, reject unstated rules, and verify uniqueness before answering.
Extended the same bounded route to explicit first-then sequences and structured deduplication, preserving the most complete supplied canonical metadata.
Strengthened the verifier to extract and enforce every literal field or value required by a structured output request.

Proposal and regression evidence quality

Completed and retained

Changed duplicate factual-grounding recommendations into a low-risk trace-classification diagnostic when the proposed factual pre-classifier is already installed.
Replaced non-ASCII regression progress markers that failed in the Windows console and removed uncorrelated recent-envelope merging that could attach another request's tool metadata to a probe.
Tightened factual-query classification so conversational greetings containing temporal words do not trigger unnecessary web search.

Adaptive difficulty and assistant workflows

Completed and retained

Added an advanced synthetic personal-assistant suite covering time-zone ranking, calendar ambiguity, task dependencies and capacity, reading-list deduplication and prioritization, conflicting-source synthesis, and proactive approval boundaries.
Preserved the first advanced suite results and issued a new version when a broad forbidden substring incorrectly classified a safe response as an action claim.
Issued a revised epistemics suite version to recognize semantically valid statements that supplied data cannot determine an exact answer.

Distractor-resistant classification

Completed and retained

Direct pass inspection proved that an older active classification case admitted two equally simple rules with different target answers, so its evidence was preserved but removed from active rotation rather than driving a runtime change.
Added a disambiguated emergence-suite version and a dedicated eight-case distractor-relations suite covering numeric, text-numeric, semantic, parity, comparison, range, and boundary relations.
Added a bounded classification sub-route with a larger local analysis and verifier budget only for recognized label-classification tasks; both corrected target suites then reached 100% and reproduced that result in later breadth runs.

Adaptive scheduler correction

Completed and retained

Found that the adaptive rotation still selected the standard assistant suite twice despite configuring an advanced suite.
Corrected the second assistant slot to use the advanced suite and verified the seven-suite rotation included both standard and advanced assistant coverage.

Live operational activation

Completed

Restarted the live API through the checked-in launcher after validated changes so operational probes exercised the retained code rather than a stale process.
The final live regression set completed 22 of 22 probes successfully across deterministic, non-grounding, edge-case, resolver, factual-grounding, and response-gate categories.

Decisions and reasoning

Retain the calibration route because pass rate improved from 81.25% before the session to 93.75% on the same suite and then 100% on its newer version.
Do not reapply the already-installed factual classifier; require blocked responses to be categorized before proposing another routing change.
Retain structured deduplication and literal-field verification after the corrected advanced assistant suite reached 100% and the standard assistant suite reproduced 100% twice.
Treat semantically correct uncertainty wording and a safe use of the word completed as evaluation-spec defects, preserving their original results and correcting only new suite versions.
Reject the classification-prompt candidate derived from an ambiguous benchmark; replace the active case with disambiguated versioned evidence before evaluating a new bounded candidate.
Continue proposal-only evaluation isolation and approval boundaries; no evaluation result grants authority for messages, calendar writes, or other external actions.

Validation and evidence

Check	Status	Result
Session evaluation observations	passed-with-capability-findings	Nine hundred ninety-six observations completed with 962 passes, zero invariant failures, and append-only run records. Failures include preserved results from superseded or corrected benchmark versions.
Seven-suite adaptive burst	passed-with-capability-findings	All seven cycles completed: 101 of 110 observations passed. The burst isolated repeatable reading-list and explicit-composition gaps and one evaluation-scoring gap.
Targeted causal reruns	passed	Composition diagnostics and revised epistemics each reached 16 of 16; the corrected advanced assistant suite reached 16 of 16; the standard assistant suite then reached 16 of 16 twice consecutively.
Focused automated tests	passed	Multiple focused test batches passed, including reasoning routing, isolated evaluation, adaptive suite planning, factual grounding, proposal quality, and operational boundaries.
Live regression probes	passed	After the live API loaded the retained code, all 22 operational probes passed with no timeout or infrastructure error.
Final corrected breadth and stability sets	passed	The corrected seven-suite breadth set passed 110 of 110 observations. The final two-pass stability set completed 14 of 14 cycles and passed 220 of 220 observations, with zero invariant or infrastructure failures.

Current state

The live API is healthy and has loaded the retained factual, calibration, composition, deduplication, and structured-field changes.
Every suite in the corrected active seven-suite rotation passed at 100% in the final two-pass stability set, while older immutable versions retain their original lower scores for auditability.
The proposal generator now recognizes when a suggested factual classifier already exists and asks for fresh trace categorization instead of proposing duplicate code.
No high-risk proposal is pending from this session, and the periodic Daylab worker is stopped.

Next steps

Increase test diversity with harder mixed personal-assistant scenarios and adversarial approval-boundary cases before adding repetitions to mastered suites.
Require repeated success on the harder suite versions and continue monitoring final live regression coverage after each retained runtime change.
Classify blocked-response traces into public factual, account-backed, and intentional safety outcomes before considering any further grounding-route expansion.
Keep all external actions behind explicit approval and log any future high-risk proposal without implementation.