Hiro development journal

Audit of Hiro's simulated personal-agent harness

2026.07.22 at 17:08:01 America/Los_Angeles Published Machine-readable JSON

Executive summary

The simulated personal-assistant evaluation layer is substantial but not yet a true stateful tool harness. Hiro has broad prompt-only coverage for calendar planning, tasks, reading lists, briefings, approvals, recovery choices, and productive autonomy, plus a sealed development-cycle generator and append-only evidence.

The missing center is executable state. Current isolated evaluations give Hiro a fictional snapshot in one prompt and score its text response; they do not let Hiro call deterministic mock calendar, mailbox, task, reading, and approval tools, mutate a world over multiple turns, encounter injected failures, or verify the final state.

The recommended next milestone is Harness v2: a deterministic world model, schema-compatible mock tool runtime, actual agent tool loop with dependency injection, trace and final-state oracles, and fresh multi-turn scenario generators. This should replace high-volume repetition as the primary development frontier.

Work completed

Static personal-assistant suites

Implemented and saturated

Versioned dry-run suites cover calendar cascades and recipient ambiguity, dependency-aware task planning, reading-edition deduplication, conflicting and untrusted briefing sources, proactive monitoring proposals, and high-impact approval boundaries.
The latest version contains eight cases and sixteen repeated observations per run. Recent version 1.0 and 2.2 runs repeatedly passed 16 of 16 with zero invariant failures.
These cases are useful policy and response-format canaries, but every relevant fact and usually the required response headings are supplied directly in a single prompt. They do not prove tool selection, argument construction, state mutation, recovery, or completion verification.

Continuous development-cycle generator

Operational but structurally shallow

The development-cycle runner creates development, blind, and canary partitions across eight families, writes immutable case files and catalogs, records hashes and structural signatures, and runs through the tool-free isolated evaluator.
The completed continuous session recorded 929 cycles and 44,592 valid observations with no infrastructure or invariant failures.
An independent normalized-prompt audit found only 2,820 normalized forms across those 44,592 records, a 93.68 percent repetition rate. Long-context distractor permutations account for 2,778 of those forms; most other families have only four to eight normalized templates, with individual forms repeated approximately 1,500 times.
The generator rejects duplicate signatures only inside one partition write. It does not maintain a session-wide or historical structural novelty set, so the continuous controller repeatedly regenerates mastered structures.

Older conversational simulation framework

Present but not suitable as the hermetic tool harness

The repository contains a SimulationEnvironment data class with mock weather, calendar, web, time, memory, and tool-budget fields, and a large scenario generator for calendar, email, memory, multi-step, and adversarial conversations.
Those environment fields are not passed into the agent or tool registry. The conversation runner calls the production run_turn path directly, and its transcript explicitly records an empty tool list because tool instrumentation is not implemented.
Consequently this path can reach live resolvers or user state, cannot prove that mock responses were used, does not compare before and after simulated state, and is not appropriate for safe stateful agent evaluation without refactoring.

Foundations available for Harness v2

Reusable

The evaluation schema already supports per-case context, expected and forbidden tools, recorded tool use, sources, resolvers, deterministic scoring, visibility, repetitions, and append-only run evidence.
The new typed SQLite model provides tasks, reading items, approvals, append-only action events, provenance, and supersession semantics that can inform the simulated state contract.
The real tool registry has a common BaseTool interface and JSON schemas. Harness v2 can implement schema-compatible in-memory tools instead of inventing a second agent protocol.
The isolated evaluator and sealed batch machinery remain valuable for text-only canaries, deterministic case admission, blind partitions, and ledger integration.

Decisions and reasoning

Treat the current personal-assistant suites as policy and reasoning canaries, not evidence that Hiro can execute personal-assistant workflows.
Do not extend the old SimulationEnvironment by monkeypatching live tools. Introduce an explicit tool-runtime dependency so a simulation cannot accidentally reach production services or personal data.
Make the primary oracle the tool trace and final world-state diff. Text-format assertions become secondary checks.
Start with read-only and local reversible operations, then test approval-gated simulated writes. Live calendar invitations, messages, and destructive operations remain outside the harness and require separate approval.
Replace continuous repetitions with global structural novelty tracking, mastery retirement, and frontier targeting near 65 to 85 percent on fresh cases.

Validation and evidence

Check	Status	Result
Repository architecture inspection	completed	Inspected the isolated evaluator, adaptive suite rotation, personal-assistant suites, development-cycle runner and tests, simulation environment, scenario generator, conversation runner, tool registry, evaluation schema, deterministic scorer, and typed memory contracts.
Recent personal-assistant ledger evidence	completed	Recent dry-run suite versions 1.0 and 2.2 repeatedly completed 16 observations at 100 percent pass with zero invariant failures. Earlier version progression moved from 81.25 percent to 93.75 percent to 100 percent as cases and assertions were corrected and behavior improved.
Continuous-session evidence audit	completed	The session summary reports 929 cycles, 44,592 valid observations, zero infrastructure failures, zero invariant failures, and a timebox stop. Direct case-file analysis measured 2,820 normalized prompt forms and a 93.68 percent normalized repetition rate.
Executable state and tool behavior	missing	No evaluated path currently supplies an isolated mutable calendar, mailbox, task list, reading list, and approval store to the real agent tool loop and then scores both its trace and final state. Recovery is represented as text describing a prior timeout rather than an injected tool failure.
Hiro runtime changes	not-run-by-design	This session is a read-only diagnosis and implementation plan. No Hiro code, configuration, service, schedule, evaluation case, ledger result, model, memory, or user data was changed.
Journal tests and site build	passed	The timestamped-entry unit test passed. The production build generated and validated 26 journal entries, verified timestamp identity, aliases, sitemap, Atom content, noindex metadata, and the IndexNow key, then completed TypeScript and Vite compilation.

Current state

Hiro is well tested at choosing or describing an action from a supplied fictional snapshot.
Hiro is not yet well tested at discovering state through tools, executing a multi-step plan, changing simulated state, recovering from tool failures, and verifying outcomes.
The benchmark and ledger foundations are strong enough to build Harness v2 without replacing the evaluation kernel.
High-volume continuous results should not be used as general agent-readiness evidence because most structures were repeated and the agent had no executable tool environment.

Next steps

Slice 1: define a versioned SimulationWorld containing clock, owners, calendars, mailboxes, tasks, reading items, approvals, permissions, and a deterministic failure script. Add snapshot hashing, clone, state diff, and append-only event trace.
Slice 2: implement schema-compatible in-memory tools for calendar reads and proposals, mailbox search and drafts, task and reading-list operations, approvals, and controlled failures. Require idempotency keys for writes and make unapproved operations return deterministic denials.
Slice 3: refactor the agent loop to accept a ToolRuntime dependency and run it with only simulated tools, an isolated owner, isolated memory database, no network, and no production resolvers. Capture every tool name, argument, result, retry, and state transition.
Slice 4: add a multi-turn scenario DSL and deterministic trace and final-state oracles. Cover conflict discovery, clarification, deduplication, approval acquisition, partial failure, retry, stale-state refresh, double-write prevention, cross-tool planning, and outcome verification.
Slice 5: replace local partition-only deduplication with session-wide and historical structural signatures, retire mastered templates, generate fresh state graphs, and use blind frontier performance rather than repetition count to select difficulty.
Promote in order: read-only discovery, local task and reading writes, draft-only email and calendar proposals, approval-gated simulated writes, and only later a separately approved live sandbox integration.