Hiro development journal

Generated adversarial stateful scenarios for Hiro

2026.07.22 at 18:13:18 America/Los_Angeles Published Machine-readable JSON

Executive summary

Expanded Hiro's hermetic stateful harness from six fixed workflows to seeded generated scenarios with topology-based novelty signatures. The generator now exercises concurrent calendar changes, approval scope drift, approval expiration, explicit rejection, partial success, rate limits, and permission denials.

A fresh real-local-model run completed 15 scenarios with 15 signatures not seen in the preceding calibration run. All 15 passed in 34.676 seconds: three cases in each of five adversarial families, including one each of timeout-after-commit, rate-limited, and permission-denied email behavior.

The original six canonical stateful workflows still passed after the stricter runtime changes, and 33 relevant regression tests passed. All scenarios remained synthetic and hermetic; no live account, personal data, external connector, or production write was accessible.

Work completed

Seeded scenario graphs and novelty control

Implemented

Added five adversarial scenario families generated from deterministic seeds. Fixtures vary participant, project, date, duration, time bucket, distractor-event count, intervention type, and failure mode.
Each case receives a structural signature derived from its topology rather than its literal prompt. The immutable runner reads signatures from prior adversarial reports and refuses to select a previously observed structure for the next set.
Reports record the seed range, prior-signature count, novel-signature count, family distribution, pass status, state digests, and complete simulated tool traces. Existing report files are never overwritten.

Concurrent and changing synthetic state

Implemented

The runtime can apply a deterministic hidden intervention immediately before a selected tool-call occurrence. Supported interventions currently insert a conflicting calendar event, update an approved draft, or expire an approval.
Calendar creation now checks the current world for overlap after authorization. A newly inserted hold causes a non-mutating calendar_conflict result containing the current conflicting event.
Email authorization is recalculated from the current draft. If the subject or recipients change after approval, the prior approval fails with approval_scope_mismatch and the message is not sent.

Approval lifecycle and adversarial failures

Implemented

The runtime now distinguishes required, rejected, expired, and scope-mismatched approvals. Hiro is instructed to stop rather than retry or seek replacement authority unless the user explicitly asks.
Generated rejection cases span two turns: Hiro requests exact approval and stops, the synthetic user rejects it, and Hiro must respect the decision without attempting the write or requesting again.
Generated failure cases cover timeout-after-commit with exact same-key replay and exactly-once state, plus non-retryable rate-limit and permission-denied results that must not be retried or mutate state.

Real-model calibration and refinement

Completed

The first generated set recorded 15 immutable scenarios and nominally passed 11. Three failures were an overly strict oracle requiring a redundant calendar refresh even though the conflict response already contained authoritative current details; the oracle was corrected without weakening the no-write outcome.
The fourth calibration failure exposed another bounded local-model serialization form for approval_request. The parser was extended only for that observed wrapper and extra trailing brace pattern, with a regression test.
A new set with unseen structural signatures then passed 15 of 15. The earlier calibration report remains immutable and is not rewritten as a passing run.

Decisions and reasoning

Count novelty by state and failure topology, not by changed names or prompt wording alone.
Apply deterministic interventions immediately before a tool call so the agent must respond to changed state rather than a prompt that merely describes a prior change.
Treat a conflict result containing the current conflicting event as sufficient authoritative evidence; an additional list read is optional and should not be required solely for ceremony.
Preserve exact approval scope across mutable state. An approval does not follow a changed draft, changed recipient, changed subject, moved event, or expired decision.
Keep non-retryable denials non-mutating and require a safe stop. Reserve automatic replay for explicitly ambiguous timeout-after-commit results and require the identical idempotency key.
Retain failed calibration evidence and run a genuinely fresh signature set after refinements rather than rerunning and overwriting the same cases.

Validation and evidence

Check	Status	Result
Adversarial generator and runtime tests	passed	16 focused harness tests passed, including 15 unique generated signatures, concurrent conflict insertion, approval expiration before write, exact scope enforcement, non-retryable failure behavior, and all observed tool-call serialization variants.
Fresh generated real-model set	passed	Run adversarial-20260723T011128Z completed 15 of 15 scenarios in 34.676 seconds. It started with 15 prior signatures and added 15 unseen signatures: three each for concurrent conflict, scope drift, approval expiry, approval rejection, and adversarial failure.
Failure-mode distribution	passed	The adversarial-failure family included one permission-denied case, one rate-limited case, and one timeout-after-commit partial-success case. All produced the required mutation and retry outcomes.
Canonical workflow regression	passed	The original six real-model stateful workflows passed 6 of 6 after the runtime and policy changes.
Broader relevant regression selection	passed	33 tests passed across the stateful harness, isolated evaluation agent, and operational-boundary tests. Pytest reported only the existing pytest-asyncio future-default warning.
Live services and user data	not-run-by-design	No live calendar, email, task system, personal memory, external network source, credential, or production resolver was exposed to the generated scenarios.
Journal tests and site build	passed	The timestamped-entry unit test passed. The production build generated and validated 28 journal entries, verified timestamp identity, same-day uniqueness, aliases, sitemap, Atom content, noindex metadata, and the IndexNow key, then completed TypeScript and Vite compilation.

Current state

Hiro now has fixed canonical workflows plus an append-only generated adversarial lane that avoids previously recorded structural signatures.
The harness can deterministically change synthetic state between observation and action, revoke effective authority, and distinguish ambiguous retryable outcomes from non-retryable denials.
The clean 15-case result is evidence for these specific simulated structures, not proof of broad production reliability or permission to operate live accounts.
The failed calibration report remains useful evidence of parser and oracle weaknesses and is excluded from the clean passing-set claim.

Next steps

Expand the topology grammar with dependency graphs, recurring calendar rules, ambiguous identities, time-zone transitions, duplicate and reordered messages, approval revocation, and partial cross-tool completion.
Add true interleaving schedules with deterministic virtual clocks and versioned resources so multiple actors can update the same object between read, approval, and commit.
Track mastery by family and topology, increase generation pressure around the 65-to-85-percent frontier, and retire structures that remain reliably mastered across fresh seeds.
Add a privacy-safe Observatory summary containing run metadata, family pass rates, failure categories, and novelty counts without publishing held-out state details or full traces.
Define separate live-sandbox gates for credentials, connector authorization, auditability, rate limits, revocation, rollback, and user confirmation before any production trial.