Hiro development journal

Phase 3F-CE isolates an intermittent Qwen runtime hard hang

Hiro qualification, repository tests, public journal tests, and production journal build complete Machine-readable JSON

Executive summary

Phase 3F-CE used the exact 20-source immutable corpus from the failed Phase 3F campaign. It did not run new discovery and did not reach relevance, feasibility, reproduction, candidate, governor, or promotion stages.

The original request path was established precisely: sequential non-streaming HTTPX calls to Qwen 3.8 27B, one server slot, strict JSON Schema output, temperature zero, reasoning disabled, a 3,072-token extraction ceiling, and a 300-second HTTP inactivity timeout with no separate wall-clock deadline.

The retained prompts were small: 685 to 1,011 input tokens. A clean control completed in 7.59 seconds, and a longer preserved extraction completed in 61.025 seconds. Ordinary latency and input size therefore do not explain the original uniform 20 timeouts.

The real failure is an intermittent hard generation hang. A preserved extraction stopped at 116 decoded tokens while continuing to consume CPU and GPU. The model health endpoint could remain healthy while the execution slot and later requests stopped completing. Client disconnect did not clear the hard-hung slot.

The old non-streaming transport allowed every later source request to accumulate behind the unavailable single slot. This explains how individually viable prompts produced 20 identical ReadTimeout outcomes.

A bounded streamed request contract, normalized terminal reasons, request/token/timing hashes, queue observation, cancellation verification, a frozen-corpus runner, and focused tests were implemented on an isolated qualification branch. The branch commit is 37ccb85380b5259a24c3ed7368ad810619261d6e.

The qualification stopped at the first unresolved runtime divergence after two source attempts: one success, one timeout/hard hang, and 18 intentionally not attempted. The instrumentation branch was not merged or activated. Production Hiro remained on revision 19ff77a5b09cf3b76712a4b5eef3d6617089fdd9.

The final disposition is PHASE 3F-CE NOT DEMONSTRATED — MODEL LATENCY/REQUEST CONTRACT FAILURE. A new fresh Phase 3F campaign is not justified until a stable runtime implementation passes the same two-source reproducer and then all 20 frozen sources.

Work completed

Original-contract reconstruction

Completed
  • Confirmed HTTPX as the client and the OpenAI-compatible chat-completions endpoint as the model boundary.
  • Confirmed the 300-second scalar HTTPX timeout applies to connect, read, write, and pool inactivity; there was no distinct total request budget.
  • Confirmed sequential corpus iteration and one Qwen server slot, so the claim caller itself did not issue a parallel batch.
  • Confirmed strict JSON Schema output, temperature zero, reasoning effort none, a 3,072-token extraction limit, and a 2,048-token validator limit.
  • The failed campaign duration was approximately 20 times the 300-second read limit, matching the preserved 20 ReadTimeout outcomes.

Input and healthy-runtime controls

Completed
  • Exact chat-template tokenization placed every preserved extraction prompt between 685 and 1,011 tokens; source text ranged from 1,040 to 2,158 characters.
  • The intended Qwen 3.8 27B model loaded with the expected 16,384-token context and one execution slot.
  • A simple strict structured-output health probe completed in 1.344 seconds before diagnostics and in 0.867 seconds after production restoration.
  • The first preserved extraction completed in 7.59 seconds with 783 input and 136 output tokens and a schema-valid response.
  • A longer preserved extraction completed in one clean control in 61.025 seconds with 1,011 input and 1,942 output tokens and a schema-valid response.

Hard-hang and abandoned-work diagnosis

Failed runtime boundary isolated
  • During qualification, the second preserved extraction reached 116 decoded tokens and then ceased token progress while a CPU core and approximately 48 percent GPU utilization remained active.
  • The execution slot remained marked processing after the client disconnected. In the hard-hung state, slot inspection could itself stop responding even though the health endpoint remained healthy.
  • An ordinary streamed cancellation control was different: a forced two-second cutoff after 28 chunks made the slot idle immediately and it remained idle.
  • The distinction proves that client timeout cannot be treated as server cancellation. Streaming handles ordinary cancellation, but cannot recover a server thread already wedged below the request layer.
  • The original non-streaming campaign had no queue admission check or post-timeout cancellation proof, so later requests repeatedly waited behind abandoned single-slot work.

Concurrency qualification

Completed
  • Two identical preserved requests took 9.855 seconds sequentially and 9.797 seconds with client concurrency two.
  • Concurrency two provided no throughput gain on the one-slot server.
  • The second concurrent request's time to first activity increased from approximately 0.64 seconds to 5.135 seconds.
  • The smallest reliable scheduling policy is therefore client concurrency one; increasing parallelism only adds queue latency and abandoned-work risk.

Bounded request instrumentation

Implemented but not production-qualified
  • Added an explicit policy covering input/output bounds, concurrency, connect/write/read/pool/queue/total budgets, bounded retries, streaming, cancellation verification, and accounting.
  • Added normalized terminal reasons for success, active timeout, model unavailability, queue timeout, schema invalidity, response invalidity, cancellation, and runtime failure.
  • Added exact prompt/config, request, response, and policy hashes plus start/end, prompt/output token, queue, first-activity, generation, model, and runtime telemetry.
  • Added pre-submit slot-idle checks and post-cancellation slot verification. Cleanup has an independently enforced overall deadline even when the slot endpoint itself hangs.
  • Added a Phase 3F-CE runner that verifies the exact frozen corpus hash and count, stops after the existing validator, and explicitly denies every downstream authority.

Minimal-repair experiments

No sufficient repair found
  • Prompt-cache disable and explicit slot reset did not reliably prevent the hard hang.
  • Moving strict-schema enforcement out of the server while retaining the unchanged schema for independent client validation did not reliably prevent the hard hang.
  • A documented CUDA-graph workaround for Qwen3.8 on RTX 5090-class hardware did not reliably prevent this local failure and was reverted.
  • Because the failure survived those bounded experiments, further timeout inflation, cache tuning, schema changes, or backend substitution would be speculative. The first-divergence rule required stopping.

Decisions and reasoning

Validation and evidence

CheckStatusResult
Focused claim-runtime and source-claim tests passed Eight focused tests passed in 0.57 seconds, including terminal taxonomy, policy bounds, schema checks, sequential admission, and a cancellation-cleanup deadline test.
Complete Hiro repository suite passed 892 tests passed, one expected test skipped, and zero tests failed in 454.66 seconds. Six existing unknown-mark warnings were reported.
Clean single-source extraction control passed The first preserved source completed in 7.59 seconds with 783 prompt tokens, 136 output tokens, and valid structured output.
Long preserved extraction control passed once The second preserved source completed in one clean control in 61.025 seconds with 1,011 prompt tokens, 1,942 output tokens, and valid structured output, proving the prompt is individually executable.
Frozen-corpus qualification failed at first unresolved divergence Twenty sources were received. Two extraction requests were attempted, one completed and one hard-hung/timed out; 18 were intentionally not attempted after the first unresolved divergence. Zero raw and zero validated claims were produced before the stop.
Production restoration passed The original Qwen runtime was restored, a strict structured probe returned HTTP 200 in 0.867 seconds, the execution slot was idle, and Hiro health reported connected Qwen with matching loaded and checkout revisions.
Public journal tests and production build passed npm run test:hiro passed. The first build generated and validated 186 entries, then stopped because the clean clone had no installed TypeScript toolchain. After npm ci installed the lockfile-defined packages, npm run build regenerated and validated all 186 entries, compiled TypeScript, and completed the Vite production bundle. Installation reported one existing high-severity dependency advisory outside this qualification's scope.

Current state

Next steps