Hiro development journal

Daylab evaluation isolation restored after benchmark routing finding

2026.07.21 at 18:10:19 America/Los_Angeles Published Machine-readable JSON

Executive summary

A completed seven-cycle Daylab evidence set revealed that synthetic personal-assistant prompts were entering the normal interactive resolver path, making those benchmark scores unsuitable for capability assessment.

Daylab was paused immediately. The evaluation harness now uses a dedicated local-only inference path with no tools, resolver access, user history, session persistence, or remote-model fallback.

A clean replacement evaluation completed 16 observations at an 87.5% pass rate with zero tool, source, resolver, or execution-error observations. Its only measured gap was the reading-list tag formatting case.

No external write action was enabled or performed. The contaminated observations are retained in the append-only ledger for audit but excluded from readiness conclusions.

Work completed

Synthetic evaluation isolation

Completed

Added a dedicated evaluation callable that sends only the synthetic system instruction and case prompt to the configured local model.
The callable deliberately bypasses the conversational agent, tool registry, live resolvers, user memory, session history, and remote fallback routing.
Both direct Daylab evidence collection and the proposal-only self-improvement evaluation phase now use this isolated callable.

Contaminated-result handling

Completed

Classified the affected personal-assistant observations as invalid for capability comparison because their execution path was not self-contained.
Preserved the existing append-only records rather than rewriting or deleting them, and excluded them from successful-run and readiness counts.
Ran a distinct, labeled replacement evaluation after the fix to establish a clean baseline.

Clean validation

Completed

The replacement synthetic personal-assistant run executed 16 observations with an 87.5% pass rate and no execution errors.
The recorded observations had zero tools, sources, and resolvers, confirming the intended isolation boundary.
The deterministic scorer identified two repeated misses in the reading-list tag-formatting scenario; the system produced a low-risk diagnosis proposal only, with no candidate code change.

Decisions and reasoning

Treat evaluation isolation as a prerequisite for interpreting assistant-workflow metrics, not as an optional benchmark convenience.
Do not use contaminated observations to claim progress, regressions, readiness, or successful autonomous behavior.
Continue to separate evidence collection and diagnosis from implementation: no evaluation proposal may alter Hiro without explicit user approval and later held-out validation.
Keep all target-workflow scenarios synthetic and approval-gated until the relevant capability has stable clean evidence.

Validation and evidence

Check	Status	Result
Focused isolation, Daylab, adaptive-planning, ledger, and dashboard tests	passed	Fifteen tests passed using the project runtime. An initial test attempt encountered a stale Windows temporary-directory permission condition; the same suite passed in an approved project-local temporary directory.
Clean personal-assistant evaluation	passed-with-capability-finding	Sixteen observations completed with an 87.5% pass rate, zero tool/source/resolver observations, and zero execution errors. The only repeated deterministic failure was a missing reading-list tag.
Safety review of the affected benchmark	passed	The faulty route was stopped before further periodic execution. No external write action was enabled or performed during the affected or replacement evaluation.

Current state

The personal-assistant benchmark has a clean, tool-free local evaluation path and a fresh replacement baseline.
The earlier affected observations remain visible as audit history but are not included in clean capability or readiness comparisons.
The periodic Daylab worker was resumed in this session. Its first isolated full proposal-only cycle completed in 1.4 minutes: 14 ordered-binding observations at a 57.1% pass rate, 22 regression probes, and zero tool, source, resolver, or execution-error observations.
Daylab may collect synthetic evidence and generate bounded diagnoses but cannot change code, configuration, integrations, data, schedules, or external services.

Next steps

Monitor the resumed periodic Daylab worker for the same zero-tool, zero-resolver boundary in each completed synthetic evaluation.
Repeat the clean personal-assistant suite across several distinct labeled runs before treating its current pass rate as stable.
Consider only the low-risk latent-binding diagnosis from the first resumed full cycle; do not approve the separate medium-risk factual-routing idea until it has fresh, targeted evaluation evidence.
If the reading-list formatting gap persists, present its bounded prompt or response-shaping diagnosis for approval; do not implement it automatically.
Continue to favor broader synthetic workflow diversity and adaptive gap selection over repeated narrow baseline runs.