Hiro development journal

Plan to close Qwen stateful workflow gaps

2026.07.22 at 20:11:04 America/Los_Angeles Published Machine-readable JSON

Executive summary

Reviewed both Qwen 3.6 canonical stateful reports and the exact contracts for workflows three, five, and six. Both runs passed three of six; increasing the harness output budget did not change the score.

The failures have different causes and should not be addressed with one broad prompt change. Workflow three is primarily a synthetic search-tokenization artifact, workflow five combines overly literal timestamp scope matching with a real stale-read sequencing miss, and workflow six is a genuine multi-stage planning and context-management gap plus a missing invalid-scope guard.

The plan corrects harness semantics first, reruns the unchanged model, and only then introduces bounded agent-policy or context changes for failures that remain. No implementation was performed in this planning session.

Work completed

Evidence and telemetry gate

Planned

Record turn boundaries, raw tool-call parsing status, finish reason, visible and reasoning token counts, prompt size, and exact failure taxonomy without storing secrets or private data.
Run each failed workflow repeatedly at temperature zero before changing behavior to measure stability and distinguish deterministic defects from model variance.
Keep immutable pre-change reports and label every new report with profile and exact model identifier.

Workflow three: mailbox discovery and safe drafts

Planned

Make the synthetic mailbox search punctuation- and possessive-insensitive so a query such as Maya's Atlas kickoff matches sender Maya and subject Atlas kickoff time. This better reflects ordinary mailbox search semantics.
Add a generic one-step progressive broadening policy after an empty named-email search: remove punctuation and low-information words while preserving sender and subject anchors; do not hardcode Maya or Atlas.
Validate against distractor messages, renamed people and projects, empty results, and cases where broadening would become unsafe.

Workflow five: approval equivalence and recovery state machine

Planned

Canonicalize calendar start and end timestamps to instants before approval comparison so 13:00 Pacific and 20:00 UTC are equivalent, while continuing to reject any changed title, attendee, duration, or instant.
Preserve original and canonical scope representations in audit evidence. This is a semantic normalization, not a weakening of approval scope.
Make stale read results explicitly return refresh_required and enforce a fresh-read precondition before calendar mutation in scenarios that model stale state.
Retain the exact same idempotency key for the required one-time replay after timeout-after-commit, followed by final-state verification.

Workflow six: multi-stage planning and context control

Planned

Reject empty or structurally incomplete approval scopes at approval-request time. No pending approval should be created for an empty email or calendar scope.
Expose read-only inspection of local proposals and drafts so the agent can verify the artifact it intends to authorize instead of reconstructing scope from long conversation history.
Add deterministic context compaction between stages: retain user constraints, artifact identifiers, exact scopes, approvals, and tool outcomes while dropping redundant prose. Do not use a second model or summarize away authorization details.
Add a stage-completion check that prevents final prose when the requested local artifact or verification read has not occurred.

Decisions and reasoning

Correct harness artifacts before changing the Qwen prompt or routing behavior.
Prefer runtime safety guards and typed tool contracts over relying only on model instructions for approval and freshness boundaries.
Do not hardcode the current names, dates, projects, or expected tool sequence into the model prompt.
Treat live connector changes, production approval semantics, and external writes as a later high-risk decision requiring separate user approval.
Implement and evaluate one gap family at a time so attribution remains clear.

Validation and evidence

Check	Status	Result
Evidence review	completed	Both Qwen reports used qwen_qwen3.6-35b-a3b and passed three of six. Workflow-three search used a possessive query that the literal synthetic tokenizer could not match; workflow five used a UTC-equivalent time rejected by exact-string approval comparison and failed to refresh stale data; workflow six created an empty email approval scope and lost the intended draft-before-approval sequence.
Implementation and rerun	not-run-by-design	This session produced an approval-ready plan only. No harness, prompt, runtime, model, evaluation result, service, or schedule was changed.

Current state

Qwen 3.6 remains the active verified runtime.
Deterministic repository tests pass, but the latest real-model canonical capability result remains three of six workflows.
The failed reports remain valid evidence, although two failures contain harness-contract artifacts that must be corrected before capability conclusions are drawn.

Next steps

Implement low-risk telemetry, search normalization, timestamp canonicalization, and invalid-scope validation with unit tests.
Rerun the unchanged Qwen model after harness corrections and classify what remains.
Apply bounded progressive-search, stale-read state-machine, artifact-inspection, and deterministic context-compaction changes only where failures persist.
Require three consecutive six-of-six canonical runs, all existing adversarial cases, at least thirty fresh structural variants, zero unauthorized writes, and no deterministic-suite regression before declaring the gap set closed.