Hiro development journal

Automatic incident intake gaps identified from live conversational failures

2026.08.11 at 20:19:38 America/Los_Angeles Diagnosed; repair not yet implemented Machine-readable JSON

Executive summary

A live conversation exposed three failures that were recorded by parts of Hiro's telemetry but did not become actionable continuous-improvement queue incidents.

The response-quality ledger contains the affected server responses, and the incident harvester advanced its watermark past them, but each row remained marked as not requiring repair.

The queue therefore correctly reports zero actionable incidents under its present rules, even though the user experienced an unnecessary evidence refusal, a client timeout, and an unusable directions response.

The diagnosis identifies a missing bridge between final-validator outcomes, client-visible failures, conversational follow-ups, and automatic incident intake. No Hiro code or runtime configuration was changed in this diagnostic session.

Work completed

Response-quality ledger audit

Completed

The affected preference question was logged with a blocked final-validator result caused by a grounding-required-without-evidence decision, but with repair_needed set to false.
The directions request was logged with six web-evidence records and a blocked malformed-response validator result, but it also retained repair_needed false.
The follow-up request completed on the server after the mobile client had already displayed a timeout. Its eventual server response was recorded as passing and did not represent the client-visible failure.
The incident harvester's watermark advanced through all three rows, confirming that they were reviewed and skipped rather than waiting to be processed.

Grounding classification diagnosis

Completed

The universal grounding classifier currently treats the standalone word 'which' as sufficient to require external evidence.
That rule is overly broad for ordinary preference and comparison questions, where Hiro can answer directly and state that the choice is subjective.
The final gate replaced an otherwise serviceable comparison with an instruction to check external information, creating an unnecessary follow-up path.

Client-timeout diagnosis

Completed

The web client aborts chat requests after sixty seconds and renders a local timeout message.
The affected server request completed approximately two seconds after that boundary, so the client and server ended with different outcomes.
Client aborts are not currently posted to an incident endpoint or written to the response-quality ledger, leaving the automatic queue blind to this visible failure mode.

Queue-intake diagnosis

Completed

The harvester re-runs the deterministic response-quality detector and queues only rows that it or the stored row marks as requiring repair.
A blocked universal validator result is currently metadata only and does not itself require repair.
Generic unusable fallback language is not included in the detector's universal failure patterns, so the directions failure was skipped despite an explicit blocked result.
The current queue snapshot contained no actionable incident corresponding to this conversation.

Decisions and reasoning

Do not claim that every wrong answer is automatically queued; the evidence demonstrates that these failures were not.
Keep response logging, deterministic replay, and the ranked queue architecture, but repair the missing signal-conversion boundaries rather than add another lab loop.
Treat final-validator blocks and client-visible transport failures as first-class incident evidence with bounded deduplication.
Narrow grounding classification by intent instead of using broad interrogative words as unconditional evidence requirements.
Preserve private conversation text locally; public reporting describes only the failure classes and system behavior.

Validation and evidence

Check	Status	Result
Response-quality database inspection	confirmed	Three consecutive live rows were present. The preference and directions rows had blocked validator metadata, while all three rows had repair_needed false.
Conversation timing inspection	confirmed	The follow-up request began roughly sixty-two seconds before its eventual server response, exceeding the web client's sixty-second abort boundary.
Incident-harvester watermark	confirmed	The stored watermark matched the newest affected quality-log row, proving the harvester had already scanned past the conversation.
Continuous queue snapshot	confirmed	The live queue reported zero actionable items and contained no new incident for these failures.
Repository mutation	not performed	This session was diagnostic only. No Hiro source file, policy, queue record, or runtime setting was changed.

Current state

Successful automatic intake exists for failure shapes recognized by the deterministic response-quality detector.
The present detector does not convert every final-validator block into a repair incident.
The mobile web client does not report its local timeout outcome back to Hiro.
Conversational follow-ups can lose the intended prior action and enter an unrelated resolver path.
Until these gaps are repaired, user reports remain necessary for failure classes that escape the existing detector.

Next steps

Convert blocked final-validator outcomes into repair-needed incidents except for explicitly expected, user-safe clarification states.
Add a small authenticated client-failure endpoint so timeout and transport errors are recorded with the request trace identifier and no private response content.
Replace unconditional interrogative-word grounding with intent-aware classification and add preference-comparison replay cases.
Add a directions and transit-planning contract that requires a concrete route, timing guidance, and current evidence when the date is relative.
Add follow-up continuity tests so a short confirmation continues the pending action instead of selecting an unrelated resolver.
Verify the repairs with positive and negative seeds, a full repository run, live mobile acceptance, and queue-intake confirmation.