Hiro development journal

Phase 3F-RS contains Qwen hangs but misses the consecutive-cycle target

2026.08.30 at 13:27:50 America/Los_Angeles Runtime-supervisor implementation, deterministic faults, 40-cycle bounded campaign, repository validation, and final runtime restoration complete Machine-readable JSON

Executive summary

Phase 3F-RS added a fail-closed process supervisor around the exact Qwen3.8-27B direct serving configuration qualified in Phase 3F-RT. It performed no new discovery, full-corpus extraction, claim-semantic change, reproduction, candidate construction, governor action, or promotion.

The supervisor distinguishes process, API, and inference health; records exact PID, creation time, executable, command, model/configuration hash, port owner, runtime instance, and slot state; and refuses termination whenever ownership is ambiguous.

A stalled request is detected from absent stream and decoded-token progress, cancelled first, and escalated to exact-worker replacement only when cancellation does not restore inference health. The identical model/configuration is reloaded and verified by bounded structured inference before admission reopens.

Every failed request attempt is immutable. At most one separately versioned retry is allowed after verified recovery; a second failure terminates explicitly rather than circulating.

All deterministic supervisor faults passed: owned worker exit during a real frozen-source request, forced inference-unhealthy replacement, and qualification-only reload failure with no retry loop.

Across the completed primary campaign and clean post-repair continuation, 40 two-source cycles produced 27 successful cycles and 13 bounded terminal failures. Eighty source requests created 105 immutable attempts: 67 valid completions and 38 real hard hangs. Twenty-five retries were issued and 12 succeeded.

The supervisor replaced 38 real campaign workers plus two deterministic live-fault workers. After every completed cycle, active model/runtime identity, slot state, and orphan state were checked. The final runtime is healthy with admission open, one idle slot, and no orphan serving worker.

The requested 10-consecutive-cycle acceptance target was not met. The best completed streak was eight cycles. A versioned continuation reached nine before exposing a redundant post-recovery health-probe defect; a clean rerun after the minimal repair crossed that boundary but still encountered model-level double-hang terminal failures.

The final disposition is MIXED BOTTLENECK. Runtime containment is strongly demonstrated, but the complete Phase 3F-RS acceptance criteria are not. The frozen 20-source Phase 3F-CE corpus remains blocked and was not rerun.

The bounded supervisor qualification was committed and pushed on the isolated qualification branch at 777193e. It was not merged or activated into Hiro production code.

Work completed

Exact process ownership and identical reload

Completed

Termination authority requires agreement among the supervisor-state PID and creation time, observed executable, observed full command hash, frozen runtime-config hash, the sole listener on the configured port, and the sole process matching the executable/model/port tuple.
If a previously verified process has already exited, replacement is allowed only when no listener and no matching worker remain. Any ambiguous ownership returns an explicit replacement failure without terminating anything.
The replacement uses the same llama.cpp CUDA 2.31.2 backend, Qwen3.8-27B Q4_K_M model and hash, 16,384-token context, full GPU offload, one slot, flash attention on, Jinja template, mmap loading, no multimodal projector, and unchanged batch settings.
The Windows child environment is normalized to one Path key and prepends the exact selected backend and CUDA vendor dependency directories. The process is launched without a visible console window.

Progress, cancellation, and inference health

Completed

Progress is recorded from two independent signals: streamed response events and increases in the sole slot's decoded-token count. Request ID, attempt ID, runtime instance, slot task ID, timing, chunk count, and decoded count are retained.
The preregistered threshold remains 30 seconds with neither progress signal. Wall-clock duration alone does not classify a request as stalled.
Cancellation is attempted first. The runner waits up to 10 seconds for the slot to return idle and requires bounded inference health before considering cancellation recovered.
Inference health requires exact process ownership, a valid health response, expected model identity, a tiny strict JSON inference returning the required value, and an idle slot afterward. A started process or HTTP success alone is insufficient.

Immutable attempts and bounded retry

Completed

Original and retry attempts are separate immutable JSON records with sidecar hashes. Records include source and request hashes, runtime identity, progress evidence, terminal reason, recovery actions, and forward/backward retry linkage.
Only infrastructure failures are retryable. Semantic/schema failures do not trigger runtime replacement or request retry.
Every source receives at most one retry after verified recovery. A hard hang or invalid runtime on that retry terminates as REQUEST_RETRY_FAILED; there is no third attempt.
The old serving PID and port owner are proven absent before a retry is admitted, so the original inference is physically unable to continue and duplicate work cannot survive replacement.

Deterministic fault qualification

Passed

During a controlled real frozen-source request, the supervisor terminated only the exactly owned serving worker. The failed attempt was preserved, a new runtime instance loaded identical Qwen, bounded inference passed, and the separately versioned retry succeeded.
A forced inference-unhealthy boundary closed admission, replaced the exact worker, restored the frozen model/configuration, and passed inference health.
A qualification-only reload failure exercised the real replacement state machine and terminated as MODEL_RELOAD_FAILED after one launch attempt, with no uncontrolled process or request retry loop.

Repeated real-runtime campaign

Acceptance target not met

The primary campaign completed 30 two-source cycles: 20 passed and 10 ended after a retry also hard-hung. Its closing streak reached eight consecutive successful cycles.
A cryptographically linked continuation preserved the immutable primary report and its eight-cycle streak. One continuation reached nine before a later harness defect stopped it; the failed attempt and boundary were preserved.
A clean continuation under the repaired harness completed cumulative cycles 31 through 40. Seven passed and three terminated after bounded retry failures. It did not produce a 10-cycle streak.
Across the accepted completed evidence there were 80 source requests, 105 immutable attempts, 67 valid completions, 38 real hard hangs, 25 retries, 12 successful retries, 27 successful cycles, and 13 terminal failed cycles.
The final worker passed exact runtime/model identity and orphan checks; its only slot was idle and admission remained open.

Decisions and reasoning

Keep the model, runtime, prompt, schema, source pair, stall threshold, cancellation grace, and one-retry policy frozen throughout qualification.
Treat process ownership as fail-closed and never use a broad process-name kill.
Count an automatically recovered hard hang as a successful cycle only when its separately versioned retry succeeds and final health/orphan checks pass.
Reset the consecutive-success count when a retry also fails; do not reinterpret a contained terminal failure as a successful workload cycle.
Use a versioned continuation linked by the immutable prior-report hash rather than rewriting completed campaign evidence.
Stop after the bounded campaign and continuation show that 10 consecutive cycles are not attained at a reasonable cost; do not extend retries or loosen thresholds to manufacture success.
Do not rerun the frozen 20-source corpus, resume discovery, build candidates, or touch promotion machinery.
Keep the supervisor qualification isolated on its branch; do not merge or activate it automatically.

Validation and evidence

Check	Status	Result
Focused supervisor, Phase 3F-RT, and Qwen launcher tests	passed	Fifteen focused tests passed in 1.29 seconds. Coverage includes exact frozen commands, normalized child environment, exact ownership and fail-closed ambiguity, runtime-state compatibility, terminal reload failure, immutable one-retry linkage, semantic non-retry behavior, frozen-source enforcement, and hard-hang classification.
Complete Hiro repository suite	passed	904 tests passed, one expected test skipped, and zero tests failed in 428.62 seconds. Six existing unknown-mark warnings were reported.
Deterministic worker-exit fault	passed	The owned process exited during a real request; old PID and port ownership were removed, identical Qwen reloaded, bounded inference passed, the retry succeeded, and later work remained runnable.
Deterministic inference-unhealthy fault	passed	Admission closed, only the exactly owned worker was replaced, identical model/configuration reloaded, and inference health passed.
Deterministic reload-failure fault	passed	The qualification-only process-boundary injection terminated explicitly as MODEL_RELOAD_FAILED after one launch attempt and zero request retries.
Repeated frozen two-source workload	mixed	Forty completed cycles yielded 27 passes and 13 explicit terminal failures. The best completed consecutive streak was eight rather than the required ten.
Final runtime and orphan health	passed	The expected model and exact owned process remained active, admission was open, the sole slot was idle, and no orphan matching serving worker existed.

Current state

Phase 3F-RS ends with disposition MIXED BOTTLENECK because containment worked but the 10-consecutive-cycle workload requirement did not.
The supervisor implementation and evidence documentation are pushed on codex/phase3fce-qualification at commit 777193e and are not merged or active production code.
The live Qwen process is restored to the frozen direct configuration and is healthy at session end. Hiro's production application revision was not changed by this qualification.
The exact frozen 20-source Phase 3F-CE corpus is not cleared and was not run.

Next steps

Decide separately whether the next bounded objective is reducing the unchanged model/runtime double-hang rate or revising the Phase 3F-RS acceptance question to distinguish containment reliability from model-output reliability.
If another runtime-supervisor campaign is authorized, preserve the same frozen semantics and report exact terminal failures; do not add retries or loosen progress thresholds merely to reach a streak.
Do not rerun the frozen 20-source Phase 3F-CE corpus until a future authorization explicitly resolves the unmet consecutive-cycle criterion.
Do not begin new discovery or meta-improvement from this result.