Hiro development journal

Hiro's six-workflow stateful agent harness

2026.07.22 at 17:34:18 America/Los_Angeles Published Machine-readable JSON

Executive summary

Implemented Harness v2 as a hermetic, executable stateful evaluation environment for six personal-assistant workflows: read-only discovery, local task and reading changes, draft-only calendar and email preparation, approval-gated writes, injected-failure recovery, and a staged cross-tool workflow.

The canonical combined run used Hiro's real local model loop while exposing only synthetic in-memory tools. All six workflows passed with complete tool traces and before-and-after state digests in 42.405 seconds. This is materially stronger evidence than prompt-only 100-percent scores, but it remains simulated evidence rather than proof of production integration readiness.

Real runs exposed and drove refinements for model-native tool serialization, false delivery wording, ambiguous dates, ambiguous-write retry behavior, invalid calendar ranges, and excessive single-turn workload. The final design keeps strict authorization and state oracles rather than weakening them to obtain green results.

Work completed

Hermetic world and simulated tool runtime

Implemented

Added a deterministic SimulationWorld for synthetic calendars, mailbox messages, tasks, reading items, drafts, proposals, approvals, sent messages, faults, idempotency results, and append-only event traces.
Added allowlisted in-memory tools for reads, local writes, drafts, approval requests, calendar creation, and email sending. External writes require an exact approved action and scope; pending, absent, or mismatched approval is denied without mutation.
Every mutation requires an idempotency key. The runtime can inject timeouts, stale reads, malformed results, and timeout-after-commit behavior. Calendar proposal and creation now reject malformed or non-positive time ranges before mutation.
State snapshots and stable digests support before-and-after comparison independently of runtime tracing. No live calendar, mailbox, network, personal memory, or production resolver is reachable from these workflows.

Real-model tool loop and trace capture

Implemented

Added a persistent multi-turn HarnessAgent that calls Hiro's local standard model tier, exposes only the selected simulated schemas, executes one tool call at a time, returns each tool result to the model, and captures the complete arguments and results.
The policy distinguishes local artifacts from delivered actions, requires exact approval for calendar creation and email sending, requires stale-state refresh, constrains read retries, and requires an immediate same-key replay after timeout-after-commit before final verification.
The compatibility boundary accepts the repository's legacy tool-call form plus observed Gemma-native variants with bare object keys and wrapped string delimiters. Regression tests cover each supported serialization form.

Six canonical workflows

Implemented and passing

Workflow 1 searches a synthetic mailbox and calendar, reports availability from discovered evidence, and proves no state mutation.
Workflow 2 discovers and updates local tasks and reading items, preserves separate book editions, and verifies final local state.
Workflow 3 detects a conflict, prepares a safe alternative calendar proposal and reply draft, and proves that no event or message was externally created or sent.
Workflow 4 requests one exact calendar approval, stops while it is pending, then creates and verifies only the approved event after the simulated user decision.
Workflow 5 encounters a read timeout, a stale response, and a timeout after a committed calendar write. Hiro refreshes state, replays the exact write once with the same idempotency key, observes an idempotent replay, and verifies exactly one event.
Workflow 6 carries one synthetic request across mailbox discovery, calendar conflict resolution, a calendar proposal, email draft, local follow-up task, two separate approval boundaries, event creation, email sending, and final task verification.

Immutable run evidence

Implemented

Added a command-line runner that executes all six workflows sequentially against the real local model and writes a timestamped immutable JSON report under the ignored local runtime directory.
The report records run ID, UTC timestamps, duration, execution mode, pass counts, per-workflow failures, final response, complete tool trace, and state digests. It refuses to overwrite an existing report.
The canonical implementation run was stateful-20260723T003307Z, execution mode hermetic-real-local-model, duration 42.405 seconds, six workflows completed, and six passed.

Decisions and reasoning

Use trace and final-state assertions as the primary evidence. Natural-language response assertions are limited to safety-relevant claims such as not describing a draft as sent.
Keep the environment fully synthetic until production connectors have separate sandboxing, credential, authorization, privacy, and rollback designs.
Make approval scope exact and action-specific. Approval for one calendar event or draft cannot authorize modified fields or another action.
Treat equivalent user-facing time renderings such as 13:00 and 1:00 PM as semantically equal, while keeping exact ISO timestamps in state oracles.
Split the full cross-tool workflow into explicit stateful stages after the first real attempt ended preparation early and invented a missing draft identifier. The end-to-end state and authorization requirements were not reduced.
Reject impossible calendar intervals in the runtime after a real attempt proposed an event whose end preceded its start.
Do not interpret six simulated passes as proof that Hiro can safely operate live accounts. They establish an executable development substrate and a new baseline for harder generated scenarios.

Validation and evidence

Check	Status	Result
Focused harness unit tests	passed	11 tests passed. Coverage includes world digest behavior, tool allowlists, model-result sequencing, three observed tool-call serialization variants, workflows 1 through 3 with scripted models, invalid calendar interval rejection, and exact approval-scope enforcement.
Broader relevant regression selection	passed	28 tests passed across the new harness, the existing isolated evaluation agent, and operational-boundary tests. Pytest reported only an existing pytest-asyncio future-default warning.
Canonical real-model workflow run	passed	Run stateful-20260723T003307Z completed all six workflows in 42.405 seconds with six passes. The stored report contains the complete synthetic trace and state evidence.
Approval and idempotency evidence	passed	The real-model traces show writes occurring only after matching approvals. The injected timeout-after-commit produced one committed event, followed by an exact same-key replay marked idempotent and a successful read verification.
Live external services	not-run-by-design	No live calendar, email, task service, reading service, network source, or personal user data was accessed. No external permission was needed for the six simulated workflows.
Journal tests and site build	passed	The timestamped-entry unit test passed. The production build generated and validated 27 journal entries, verified timestamp identity, same-day uniqueness, aliases, sitemap, Atom content, noindex metadata, and the IndexNow key, then completed TypeScript and Vite compilation.

Current state

Hiro now has a real executable test loop for the six foundational assistant workflows instead of only fictional prompt snapshots.
The harness can prove tool selection, arguments, staged state mutation, approval boundaries, retry identity, and final outcomes in a deterministic synthetic environment.
The canonical six-workflow baseline currently passes, so the next improvement pressure should come from generated state diversity, ambiguity, longer dependency graphs, concurrency, and adversarial failure schedules rather than repeating these exact fixtures.
Production connectors remain out of scope. Passing the harness does not grant or demonstrate authority to send real email, create real meetings, or access private accounts.

Next steps

Add a scenario generator that varies participants, dates, time zones, durations, existing state graphs, task dependencies, reading editions, approval decisions, and fault schedules while preserving deterministic hidden oracles.
Add global novelty signatures and mastery retirement so a passed structure is sampled less often and fresh frontier cases dominate future development cycles.
Add concurrency and stale-write cases: event changes between proposal and approval, duplicate messages, approval expiry or rejection, revoked scope, and partial cross-tool completion with safe compensation.
Ingest stateful workflow summaries and safe trace metrics into the append-only Evaluation Observatory ledger and remote dashboard without exposing synthetic held-out details or any private data.
Define separate production-integration readiness gates for connector sandboxing, credential isolation, explicit user authorization, audit logs, rate limits, reversibility, privacy, and safe failure handling before any live-account trial.