Hiro development journal

Why promotions stopped after the supervised successes

2026.08.22 at 17:38:41 America/Los_Angeles Diagnostic complete; corrective change not yet authorized in this session Machine-readable JSON

Executive summary

The promotion pipeline did not stop running after the two supervised successes. The live queue remained active, Qwen 3.8 27B remained healthy at a 16,384-token context window, and the circuit breaker was closed.

After the latest promotion, the queue recorded 22 investigations and 22 confirmed potentials. None reached paired evaluation: 11 were scheduled for another construction attempt and nine exhausted construction attempts as artifact_generation_blocked. One additional obsolete incident was active on its third attempt during inspection.

The common cause is a stale-data scheduling defect. High-priority interaction-audit incidents created before the full-prompt replay repair do not contain the synthetic prompt that holds required facts such as schedule, weather, transit, or conversation context. Their abbreviated user query refers to a supplied snapshot but does not include it. Candidate construction is consequently asked to synthesize required facts from unavailable evidence.

A second queue-policy defect amplifies the problem: artifact-blocked records are automatically reactivated whenever the active repository revision changes. The successful promotion changed the revision, so the scheduler began draining obsolete pre-fix incidents ahead of newer causally complete records. This creates sustained construction churn that looks like a stopped promotion system even though the workers are busy.

Work completed

Verify live service and queue health

Completed

Hiro was serving on ports 8001 and 8765 from revision af627600bfa3d497d6d9b27ab12426057cda093f.
Qwen 3.8 27B was resident on port 8080 with --ctx-size 16384, GPU layers enabled, and the expected qwen/qwen3.8-27b alias.
The queue reported 296 actionable records, one active candidate, five implemented records, 43 rejected records, nine artifact-blocked records, and a closed circuit breaker with zero consecutive infrastructure failures.

Measure activity after the latest promotion

Completed

The latest implementation event was event 3285 at 2026-08-22T22:11:28.849998+00:00 for incident-fe44ffbe49335e3aa0d5f30d.
Events after that implementation comprised 22 investigation_started, 22 potential_confirmed, 11 candidate_revision_scheduled, nine artifact_generation_blocked, five idea_queued, and no further candidate implementation or evaluation event.
Recent construction cycles continued at roughly four-to-six-minute intervals, demonstrating active work rather than a dead scheduler, expired lease, open circuit, or unavailable model.

Trace the repeated construction failures

Completed

All nine currently artifact-blocked records were created from older interaction-audit data and lacked replay_fixture.prompt. Their latest blocked revision was the current af627600 head, so they had been retried after the promotion and exhausted again.
The active incident, everyday-current-game, required Comets, September 12, 4:00 p.m., and North Field, but its stored replay supplied only the abbreviated question and a malformed model answer. The schedule containing those facts was absent.
Comparable failures appeared for weather, transit, travel, and follow-up cases. Candidate output either remained the malformed fallback, became empty, or partially reconstructed a fixture while omitting required facts.
Newer audit records created after the replay repair do preserve the full prompt. They were ranked behind at least fifteen older same-priority current-information and transit records because ties are ordered by oldest creation time.

Identify the scheduler amplification

Completed

ContinuousQueue.requeue_artifact_blocked retries one artifact-blocked record whenever its blocked_on_revision differs from the active revision.
The promotion and subsequent platform repair changed the active revision. That correctly made genuinely repairable artifacts eligible for reconsideration, but it did not check whether the originating replay was causally complete.
The lane-two audit bonus gives these obsolete records a priority near 112, so each resurrected record preempts lower-ranked research ideas and consumes as many as three Qwen construction attempts before returning to artifact_blocked.

Decisions and reasoning

Classify the current non-promotion interval as queue-input and retry-policy churn, not as model outage, pipeline idleness, evaluator rejection, or lack of candidate ideas.
Do not interpret artifact_generation_blocked as a negative judgment about idea merit; these candidates never reached the paired evaluator.
Treat replay completeness as an admission invariant: a synthetic case that refers to supplied evidence must carry the full prompt or structured evidence needed to reproduce it.
Do not automatically resurrect stale artifacts solely because any repository revision changed; reactivation must be tied to a relevant builder/harness compatibility change and valid current evidence.

Validation and evidence

Check	Status	Result
Live process and ports	passed	Hiro main.py owned ports 8001 and 8765; Qwen owned port 8080 with the expected 27B model and 16K context configuration.
Continuous queue health endpoint	passed	The endpoint responded and showed one active candidate, 296 actionable records, five implemented records, and a closed circuit breaker.
Post-promotion event audit	diagnostic finding	Twenty-two construction investigations occurred after the latest implementation, but zero candidates advanced to evaluation or promotion; nine exhausted as artifact-blocked and 11 received bounded retries.
Replay completeness audit	failed for stale records	All nine current artifact-blocked interaction-audit records lacked replay_fixture.prompt. Newer post-repair records contained the full prompt, confirming a stale-record boundary rather than a current capture failure.
Repository tests	not run	This session was read-only diagnosis; no Hiro source change was made and no claim of a new repository test pass is made.

Current state

The live system is active but spending its candidate capacity on obsolete high-priority audit incidents.
The inspected active candidate was incident-5ad4bf2c51f2cba013c9c740 on revision attempt three for everyday-current-game.
Nine older records had already returned to artifact_blocked on the current revision, while additional old prompt-incomplete records remained ahead of newer valid records in the ranked queue.
No evidence indicates that Qwen 3.8 27B, the queue lease, or the infrastructure circuit breaker caused the promotion pause.

Next steps

Invalidate or supersede queued prompt-incomplete audit records before candidate construction, while preserving their original audit events.
Regenerate current incidents from the canonical audit cases so their replay fixtures include the full prompt, evidence metadata, and contract.
Change artifact reactivation to require a relevant builder compatibility version plus a replay-completeness check, rather than any active Git revision change.
Add fairness or bounded-age handling so a large same-priority stale cohort cannot monopolize candidate throughput.
Run focused migration and scheduler tests, the full Hiro suite, five exact-revision qualification cycles, and then restart the live queue only after those checks pass.