Hiro development journal

Defining Hiro's controlled path from synthetic evaluation to internet observation

2026.08.05 at 18:10:38 America/Los_Angeles Validated and published Machine-readable JSON

Executive summary

Confirmed that Hiro's current public evaluation lane consists of locally stored, synthetic, versioned test cases run through a tool-free local inference path; the word public distinguishes development-visible cases from secret held-out cases and does not mean live internet traffic.

Defined live-internet access as a separate evidence-collection lane rather than a replacement for deterministic evaluation, preserving reproducibility while allowing Hiro to discover real-world failures and changing information.

Recommended starting a read-only, allowlisted internet shadow probation after the repository has a clean pinned baseline and one complete offline autonomous candidate cycle has successfully exercised construction, evaluation, retention, dashboard review, and notification delivery.

The proposed first internet stage would collect sanitized snapshots and interaction traces for later replay in the sandbox, without credentials, posting, purchases, downloads that execute, production changes, or automatic promotion.

Work completed

Current evaluation provenance

Inspected

The nightly suite is configured at hiro/evals/public_cases/rsi_cycle_v1.jsonl and is loaded from the repository as bounded public development cases.
The isolated evaluation agent explicitly prohibits tools, browsing, memory access, external services, and remote-model fallback; prompts and fictional context are treated as complete.
Nightly failures are grouped by category and converted into diagnostic specifications, while a separately located held-out vault remains reserved for candidate validation.
Therefore current evaluation results measure repeatable behavior against cases authored and versioned for Hiro, not performance against uncontrolled live websites.

Real-world evidence architecture

Designed

Keep deterministic public and secret held-out suites as the promotion evidence lanes.
Add a distinct internet-observation lane that performs narrowly scoped read-only retrieval through an egress policy, records provenance and timestamps, strips active content, and stores immutable snapshots.
Turn observed failures or opportunities into replayable cases before candidate construction, so the candidate and baseline are compared against the same evidence.
Treat all web content as untrusted data and prevent it from changing Hiro's policy, instructions, credentials, schedules, repositories, or promotion decisions.

Activation prerequisites

Recommended gate

Establish a clean and pinned Hiro repository baseline; the autonomous sandbox currently fails closed when the source repository contains unrelated working-tree changes.
Complete at least one end-to-end offline sandbox cycle, including frozen evidence, benchmark visibility, and Telegram review notification, before adding another variable to the system.
Use an explicit domain and method allowlist, read-only GET/HEAD access, request and byte budgets, timeouts, content-type limits, private-network denial, redirect checks, and a durable emergency-stop control.
Run the first internet stage as a no-write shadow probation with captured evidence, independent audit logs, and no automatic candidate promotion.

Decisions and reasoning

Use the term public evaluation to mean developer-visible synthetic cases, not live-public-web evaluation.
Do not let mutable internet responses become direct pass/fail promotion gates; freeze and replay their relevant evidence first.
Separate internet discovery authority from code construction, evaluation, and promotion authority.
Begin with read-only allowlisted retrieval and expand domains or interaction types only after measured probation evidence supports it.
Retain the current no-automatic-promotion boundary and keep Stage 6 disabled during internet-access probation.

Validation and evidence

Check	Status	Result
Evaluation-path source inspection	passed	Confirmed the configured repository suite, local JSONL provider, tool-free isolated evaluation agent, held-out separation, deterministic failure clustering, and fail-closed sandbox prerequisites.
Hiro journal generation and frontend build	passed	Timestamped-entry unit tests passed; the generator produced and validated 67 journal pages, and the TypeScript and Vite production build completed successfully.

Current state

Hiro does not currently use the internet for the public evaluation lane.
Internet access has not been enabled by this session.
The autonomous sandbox remains isolated, requires a clean source baseline and an external held-out vault, and stops before integration.
Stage 6 remains disabled and the existing promotion boundary is unchanged.

Next steps

Resolve or intentionally commit the existing Hiro working-tree changes so the sandbox has a clean pinned baseline.
Run and review the first complete offline autonomous candidate cycle, including its Telegram notification.
Implement a read-only internet-observation adapter with strict egress, provenance, snapshotting, prompt-injection containment, budgets, and emergency-stop tests.
Conduct a time-bounded shadow probation before considering broader domains, authenticated access, or interactive actions.