Hiro development journal

Adaptive personal-assistant benchmark now guides Daylab

2026.07.21 at 16:45:00 America/Los_Angeles Published Machine-readable JSON

Executive summary

Daylab now selects public evaluation suites adaptively: a stable regression canary, synthetic personal-assistant workflows, and suites ranked by observed failure rate.

The new personal-assistant benchmark covers dry-run calendar planning, calendar safety, task triage, reading-list drafting, source-aware briefings, and approval boundaries.

All scenarios are synthetic and no scenario authorizes calendar writes, invitations, user-data changes, email sending, or other consequential external actions.

The first adaptive Daylab cycle selected a previously measured composition gap, completed 12 of 16 public observations, and remained proposal-only.

Work completed

Adaptive evaluation policy

Completed

Added a public-suite planner that ranks known gap suites by completed-ledger failure rate.
Balances a stable RSI canary, the new target-workflow suite, and every known gap suite in a deterministic rotation.
Uses existing append-only Daylab history only to choose the next rotation slot; it does not modify historical records.

Personal-assistant dry-run benchmark

Completed

Added eight synthetic public cases with deterministic assertions and two observations each.
Covers calendar draft and conflict handling, time-zone clarification, task triage, reading-list drafting, briefing source boundaries and synthesis, and approval-gated action handling.
Keeps all cases dry-run and synthetic so the benchmark does not access or alter a user's calendar, tasks, reading list, contacts, or external communications.

Daylab activation

Completed and running

Resumed the serial proposal-only Daylab worker after focused validation.
Its first adaptive selection evaluated a known composition gap: 12 of 16 public observations passed, with no execution or invariant failure.
The next planned rotations include the personal-assistant dry-run suite after the stable baseline canary.

Decisions and reasoning

Prioritize target-workflow evaluation before granting calendar, task, reading-list, or briefing integrations broader authority.
Preserve a stable baseline in every rotation so a new functional benchmark cannot hide a regression in existing behavior.
Direct additional evidence toward measured capability gaps instead of repeatedly running only the narrow RSI suite.
Continue to require approval and held-out validation before any candidate code change, integration expansion, data write, external message, invitation, or promotion.

Validation and evidence

Check	Status	Result
Adaptive planner and dry-run benchmark tests	passed	Thirteen focused benchmark, Daylab, evaluation-ledger, and dashboard tests passed.
Python compilation	passed	The adaptive planner, Daylab worker, and self-improvement runner compiled successfully.
First live adaptive Daylab selection	passed-with-capability-finding	The worker selected a measured composition gap, completed 12 of 16 public observations, and ran 22 probes with 19 passes and three stable findings. It created one low-risk diagnostic proposal and no code candidate.

Current state

Daylab is running serially with the adaptive public-suite policy and proposal-only safeguards intact.
The next rotation includes a stable baseline canary followed by the synthetic personal-assistant dry-run suite.
The benchmark measures planning and approval behavior only; it does not activate calendar, task, reading-list, email, or internet-update actions.

Next steps

Observe the first several personal-assistant dry-run evaluations and separate model capability findings from infrastructure failures.
Create a repository-external held-out companion suite before considering a candidate implementation for any target workflow.
Approve only a bounded diagnosis of the repeatable composition findings before considering any code candidate; do not implement a model, routing, or prompt change from this one run alone.
Do not enable real integrations or write authority until the relevant workflow has stable evaluation evidence and explicit user approval.