Hiro development journal

Controlled Hiro reliability and emergence lab

2026.07.16 Detailed development update Machine-readable JSON

Executive summary

The launcher now uses the supported pinned LM Studio CLI, verifies Gemma 4 12B model identity and readiness, warms the model, records telemetry, and enables Telegram only after success.

Complete local validation passed all 20 response probes, all 10 tool/resolver probes, a 1.396-second real Hiro agent turn, 0.144-second cancellation recovery, and all 76 repository tests. No paid or external LLM API was used.

The smoke report is PARTIAL only because the Hiro HTTP/HTTPS application endpoints were not running. A separate non-fatal log-rotation file contention warning remains a follow-up.

Three consecutive unchanged public and repository-external held-out evaluation pairs were completed after the launcher repair. Every repetition produced 87.5 percent public and 87.5 percent held-out pass rates, zero invariant failures, and sub-second p95 latency, demonstrating that the previous timeout cascade is resolved under the current machine state.

All misses were deterministic resolver false positives rather than model-server failures: the public word-sort case was routed to Gmail and the held-out uncertainty case was routed to weather. No response change was created or promoted.

Two narrowly scoped resolver intent-boundary corrections eliminated the only repeatable failures from the stable three-run baseline. The matched candidate evaluation passed all 16 public and all 16 repository-external held-out observations with a 1.0 weighted score, zero invariant failures, and no latency regression.

A 20/80 program was designed to move Hiro from infrastructure stabilization into controlled testing for transferable and emergent capabilities. Twenty percent of each laboratory budget is reserved for eliminating measurement threats; eighty percent is reserved for an escalating capability ladder using novel, independently scored tasks.

Level 1 emergence testing began with three unchanged rule-induction and transfer baseline pairs. The new suites are deliberately harder than the operational smoke suite: mean public pass rate was 66.7 percent and mean repository-external held-out pass rate was 52.1 percent, with zero invariant failures and stable sub-second p95 latency.

The results show selective transfer rather than general mastery. Hiro was consistently strong on relational classification, analogy, explicit exception transfer, and distractor resistance, but consistently weak on multi-step numeric composition, position-based token extraction, and a harder held-out symbol transformation. Ambiguity calibration varied across repetitions.

Level 1 measurement was extended prospectively without rewriting version 1.0. Version 1.1 broadened calibration expression coverage, a structurally independent repository-external held-out family tested surface transfer, and a nine-case diagnostic isolated the composition bottleneck.

The diagnostic evidence rules out basic arithmetic, intermediate-state retention, single-operation inference, explicit natural-language composition, and output formatting as primary causes. Hiro failed both abstract operation-order controls and the latent two-step case, supporting a specific hypothesis that the weakness lies in binding inferred or named operations into the correct ordered sequence.

The Level 1.1 stability requirement was completed with two additional unchanged matched pairs. All three repetitions were quality-identical: 75 percent public, 62.5 percent held-out, calibration 1.0, zero invariant failures, public weighted score 0.9583, and held-out weighted score 0.9375.

A narrowly scoped ordered-operation-binding prompt candidate was implemented and tested against three stable pre-candidate baselines and three matched candidate repetitions. It was rejected and removed because it produced no public gain, no latent-binding gain, and only one non-repeatable held-out improvement.

The negative result is informative: a generic instruction to bind operations, preserve order, compute intermediates, and verify them does not change Gemma behavior reliably. The limitation requires a different mechanism than adding a high-level system-prompt reminder.

A materially different two-stage infer-then-execute candidate was tested. It consistently raised the counterbalanced public ordered-binding suite from 57.14 to 71.43 percent, but held-out gains were mixed and the broader Level 1.1 suite regressed from 75 to 62.5 percent. The candidate was rejected and removed under the no-regression gate.

A separate two-hour lab ran in C:\tmp\hiro-two-hour-lab-20260716-2120 on codex/two-hour-lab-20260716-2120, based on codex/rsi-first-cycle. The dirty main workspace and its Telegram lifecycle changes were inspected but not edited.

Sixteen local reliability commits were retained: Windows-safe shared log rotation, a single-writer lease, resumable run IDs, exact-model readiness preflight, bounded timeout and transport recovery, immutable case fingerprints, in-flight lease heartbeats, preflight-before-model lease ordering, explicit READY sentinels, invalid-work rejection before preflight, evaluation-memory isolation, internal-session filtering, and read-only recovery status.

Three behavior mechanisms remained proposal-only and were removed. A typed internal ledger regressed public quality and latency; broad dual solving helped named operations but hurt latent chains and latency; a narrowed named-operation consensus path failed to improve a fresh external family. No response behavior was promoted.

After reliability hardening, three public Level 1.1 confirmations were identical at 75 percent pass, 0.9583 weighted score, zero invariant failures, and 509-915 ms p95. Three frozen external named-consensus confirmations were identical at 50 percent, 0.9167, zero invariants, and 548-811 ms p95.

A non-promotable direct-local prompt diagnostic showed that Gemma retained 75 percent public performance while reaching 75 percent on the already-observed external family at roughly 309-333 ms p95. Prompt ablation then exposed cross-case contamination: normal evaluations were persisting synthetic conversations and injecting recent evaluation cases into later prompts.

Evaluation turns now read no personal or episodic memory, persist no conversations, and extract no facts. Internal evaluation, simulation, lab, and benchmark sessions are filtered from real-user cross-session context. A live run added zero conversation rows; 2,898 historical evaluation rows remain stored but are now excluded from retrieval and were not deleted.

The final isolated repository suite passed 114 tests. Five forced-timeout soak cycles recovered in 184-201 ms and every next control passed in 181-334 ms. A hard-killed run resumed to 2/2, and a competing evaluator was rejected before preflight in 601 ms with no run record.

Work completed

Launcher and readiness gate

Completed

Registered the existing 6.5 GiB GGUF with LM Studio by hard link, avoiding a duplicate copy.
Pinned the local CLI; configured localhost port 8080, maximum GPU offload, 4096-token context, and one inference slot.
Verified the served model, performed a warmup, recorded PID/model/timestamps/latencies, and gated Telegram startup on readiness.

Complete Gemma reliability validation

Completed

Cold startup took 7.437 seconds and warmup 0.210 seconds; the final lifecycle rerun took 7.366 and 0.146 seconds.
The smoke benchmark passed 30 automated model and tool checks with zero timeouts/errors and 0.71-second average latency.
The full Hiro agent path completed in 1.396 seconds. A cancelled long request released the single slot and the first follow-up succeeded after 0.144 seconds.
The complete Python suite passed 76 tests in 5.18 seconds; final focused lifecycle coverage passed 10 tests in 0.69 seconds.

Shutdown verification

Completed

Extended shutdown for LM Studio while preserving Telegram-first ordering.
Final state: Telegram stopped and disabled, no evaluator, no loaded model, port 8080 closed, dashboard untouched.
GPU use fell from approximately 8.65 GB loaded to approximately 1.15 GB after unload.

Three-run evaluation baseline

Completed

Each repetition ran 16 public observations followed by 16 repository-external held-out observations through one serial ledger writer. Both suites and their assertions remained unchanged.
Public pass rate was 87.5 percent in all three runs with weighted score 0.975, zero invariant failures, and p95 latencies of 931.6, 943.6, and 973.7 milliseconds.
Held-out pass rate was 87.5 percent in all three runs with weighted score 0.9792, zero invariant failures, and p95 latencies of 546.2, 565.1, and 568.1 milliseconds.
The code, configuration, prompt, toolset, and model hashes matched across all three repetitions. Repetition-specific labels were required because the CLI includes a fresh created_at timestamp in the otherwise immutable manifest.
Public run IDs: 727e269d-1308-4af4-8a1f-2b8f0153e83e, b3afb742-32ff-4a50-bb52-4c4d591cce00, f38a080f-1319-4c19-99cb-c9aa578e8ee6. Held-out run IDs: e75fb330-3347-49e8-855d-72052ffed442, 15f4541e-9b8b-4744-9a12-746b81ca162c, efcb7883-887a-4a31-959e-9eadfc7ab641.

Narrow resolver intent-boundary corrections

Completed

Gmail order routing now requires commerce or shipping context instead of treating every bare ORDER token as an email-search request. Normal requests such as where is my order, find the latest order email, and track my package remain supported.
Weather detection now uses complete-word matching for strong weather terms. Ambiguous adjectives such as hot, cold, warm, and cool require an outdoor, time, or measurement cue, preventing fragments inside unrelated words from triggering weather.
Added public and synthetic regression tests for the observed formatting-label failure, non-weather substring collisions, and preserved positive commerce/weather routes.
Allowed implementation scope was limited to core/gmail_resolver.py, core/weather_resolver.py, and tests/test_resolver_intent_boundaries.py. Evaluator code, assertions, public cases, held-out data, safety policy, grounding, credentials, environment files, and CI were unchanged.

Twenty-percent reliability track

Planned

Time budget: one unit of reliability work for every four units of capability testing. Reliability work may exceed that ratio only when a failed gate makes capability measurements invalid.
R1 fixes non-fatal log rollover contention so evaluation output remains readable and attributable. Exit gate: concurrent process ownership cannot flood stderr or lose evaluation records.
R2 fixes immutable-manifest repetition semantics so identical code, config, prompt, toolset, and model hashes can share one variant identity while individual run timestamps remain run metadata. Exit gate: three repetitions under one immutable manifest.
R3 adds explicit single-writer leasing and stale-lease recovery. Exit gate: a second evaluator is rejected, an interrupted owner can be recovered, and the ledger remains append-only.
R4 validates resumability by interrupting a run between observations and completing it without duplicate case records.
R5 adds preflight readiness and throughput gates. Exit gate: wrong model, occupied slot, insufficient latency headroom, and unhealthy endpoint all prevent measurement before the first case.
Reliability work cannot change evaluation scoring, assertions, held-out content, safety policy, grounding requirements, credentials, environment files, or CI.

Eighty-percent emergence-testing ladder

Planned

Level 1, rule induction and transfer: infer a latent rule from examples, explain it minimally, and apply it to structurally different unseen cases. Measure transfer accuracy, calibration, and sensitivity to irrelevant surface features.
Level 2, compositional generalization: combine two familiar skills in novel arrangements without a dedicated resolver. Use counterbalanced tasks to distinguish composition from memorized templates.
Level 3, self-diagnosis and recovery: inject incomplete evidence, misleading intermediate results, and recoverable tool failures. Score detection, revised plan quality, final correctness, and whether Hiro avoids repeating the failed approach.
Level 4, multi-tool agency: solve bounded objectives requiring tool selection, sequencing, evidence reconciliation, and stopping. Include decoy tools and conflicting sources; require grounded final answers and full traces.
Level 5, cross-session learning: provide a lesson in one session and test transfer later with new wording and a different domain. Measure retention, appropriate retrieval, non-interference, and resistance to false-memory prompts.
Level 6, hypothesis formation: present ambiguous observations and require multiple competing hypotheses, discriminating tests, evidence updates, and calibrated conclusions. Reward information gain rather than confident guessing.
Level 7, bounded self-improvement: Hiro may propose a narrowly scoped change and predicted effects, but an external evaluator runs untouched public and held-out tests. Changes remain proposal-only until independent gates pass.
Every level includes public development cases, repository-external held-out cases, adversarial variants, and negative controls where the learned behavior should not activate.

Program cadence and evidence gates

Planned

Use two-week cycles or equivalent work units: approximately one day/unit on reliability and four on emergence experiments. Unused reliability budget returns to emergence testing; it does not accumulate into speculative refactoring.
Each experiment starts with a written capability hypothesis, causal mechanism, allowed-file scope, predicted public and held-out effects, invariant list, latency budget, and falsification condition.
Use at least three unchanged baseline repetitions before candidate comparison. Candidate evidence requires matched public and external held-out runs, zero invariant regressions, no material latency regression, and improvement across task families rather than one prompt.
Advancement requires success on two independently generated held-out task sets and at least one delayed transfer test. A perfect score on a small suite is treated as suite mastery, not emergence.
Promotion remains proposal-only. No experiment may edit its evaluator, assertions, held-out vault, grounding requirements, or safety constraints.
Stop immediately for timeout cascades, slot contamination, ledger ambiguity, safety or grounding regression, held-out leakage, unexplained cross-run variance, or improvement confined to development cases.
Maintain a failure taxonomy separating model reasoning, resolver routing, tool execution, memory retrieval, orchestration, infrastructure, and evaluator defects. Only capability-level failures count toward emergence conclusions.
Primary program measures: unseen-family transfer, compositional success, recovery rate, calibration, evidence grounding, cross-session retention, negative-transfer rate, intervention efficiency, latency p95, and invariant failures.

Level 1 rule-induction and transfer baselines

Completed

Created eight versioned public cases and eight separately stored repository-external held-out cases. Families cover latent-rule induction, relational classification, symbolic transfer, two-step composition, irrelevant-feature resistance, exception transfer, analogy, and underdetermination. Each case ran twice per baseline.
Both suites validated without changing the evaluation kernel, scoring implementation, existing suites, safety policy, grounding rules, credentials, environment files, or CI. Focused evaluation tests passed 21 of 21 and the complete repository suite passed 80 of 80.
Baseline 1: public 62.5 percent, weighted 0.9375, p95 931.4 ms; held-out 43.75 percent, weighted 0.90625, p95 888.7 ms.
Baseline 2: public 75 percent, weighted 0.9583, p95 993.4 ms; held-out 62.5 percent, weighted 0.9375, p95 876.4 ms.
Baseline 3: public 62.5 percent, weighted 0.9375, p95 904.3 ms; held-out 50 percent, weighted 0.9167, p95 861.6 ms.
Public run IDs: 06e5e9a5-3088-4146-8f1c-68d286493653, bd203b2a-c064-4195-bebb-2f742b3fd3c9, bfee14d5-c556-4e2e-8b7b-c4e6bddf50c7. Held-out run IDs: b2950708-df59-4554-ac24-d3e3020b5453, 91064225-20bc-464a-a6da-2f5cfcfc85dd, 01768885-ab68-492c-a7c7-e83f3fb859f1.
Stable successes across all six observations per family included public and held-out relational classification, analogy transfer, and exception transfer. Public symbolic shifting and distractor resistance were also stable.
Stable failures included both public and held-out two-step composition, public position extraction, held-out token selection, and held-out symbolic shifting. The failed composition and symbol responses varied, indicating unstable inferred mechanisms rather than a fixed routing error.
Calibration scoring has known undercoverage: semantically appropriate responses such as indeterminate, undefined, or does not have a unique mapping did not always match the narrow accepted-expression regex. Existing results were preserved and not rescored after inspection.

Level 1.1, independent transfer, and composition diagnosis

Completed

Preserved all version 1.0 suites and ledger results. Added public and external held-out version 1.1 files with only prospective calibration-language expansion.
Created a second repository-external held-out family using different surface structures: length parity, token rotation, vowel counting, letter composition, shape distractors, case-preserving exceptions, access analogies, and underdetermination.
Created nine public composition diagnostics with two repetitions each, isolating single-operation inference, intermediate output, arithmetic execution, explicit composition, operation order, formatting, and the original latent composition.
All four suite files validated. Evaluation-kernel tests passed 21 of 21 and the complete repository suite passed 80 of 80.
The v1.1 matched diagnostic scored 75 percent public and 62.5 percent held-out with calibration 1.0 on both, zero invariant failures, and p95 latency 885.9/903.4 ms. Public run ef4ec653-e490-436b-8817-002dfa434ddf; held-out run dc616b8a-b43c-4a12-a776-4baff0d5036a.
The independent held-out family scored 37.5 percent, weighted 0.875, zero invariant failures, and p95 868.8 ms. Run dbe81239-cab8-4f93-b48c-ba76c9865a77. Rotation, distractor resistance, and exception transfer passed; length parity, vowel counting, and letter composition failed.
The independent analogy case was contaminated by Gmail routing because the prompt contained ticket; it is classified as resolver overreach rather than relational reasoning evidence. The calibration answer Not determined uniquely was semantically correct but still outside the prospective regex, showing that phrase-list scoring remains brittle.
The composition diagnostic scored 66.7 percent pass rate, weighted 0.9444, zero invariant failures, and p95 1097.3 ms. Run d4c60fd6-73e6-4927-b841-09882866f924.
Single-operation inference, intermediate state, arithmetic, explicit natural-language composition, and formatting each passed 100 percent. Both operation-order controls and the latent composition case passed 0 percent. This localizes the dominant failure to ordered operation binding rather than the component skills.

Three unchanged Level 1.1 baselines

Completed

The earlier v1.1 diagnostic pair was retained as baseline repetition 1 because it used the same unchanged public and external held-out suite versions, model, prompt, code, configuration, and toolset surfaces.
Repetition 1: public run ef4ec653-e490-436b-8817-002dfa434ddf scored 75 percent with p95 885.9 ms; held-out run dc616b8a-b43c-4a12-a776-4baff0d5036a scored 62.5 percent with p95 903.4 ms.
Repetition 2: public run 76013adc-3999-4dc2-8888-db1574b2f8b5 scored 75 percent with p95 1026.1 ms; held-out run 2bb4639d-ed63-4050-931d-1ee1780f2f3f scored 62.5 percent with p95 891.9 ms.
Repetition 3: public run e59c34e3-05c2-4f5a-a84a-59adbc579c2d scored 75 percent with p95 921.5 ms; held-out run b7d38fd0-59e9-4002-8047-a95de45709ab scored 62.5 percent with p95 926.5 ms.
Every category score was identical across all three repetitions. Calibration, distractor resistance, exception transfer, and relational transfer were 1.0 on both suites. Public structural transfer was 1.0; held-out structural transfer was 0.8333. Composition was 0.8333 and rule induction 0.9167 on both.
All 96 observations across the three public/held-out pairs completed with zero invariant failures. No candidate or behavior change was introduced between repetitions.

Rejected ordered-operation-binding candidate

Rejected and removed

Hypothesis: Hiro has the component skills but loses the binding between operation identity, sequence position, and intermediate state. A generic internal procedure might improve ordered composition without teaching evaluation answers.
Allowed behavior scope was only core/prompt_builder.py. The temporary candidate added one generic sentence instructing Hiro to bind each operation to its stated step, apply steps in order to the preceding intermediate, verify intermediates, and return the requested format. A bounded regression test asserted the instruction contained no evaluation terminology or suite identifiers.
Before implementation, created seven counterbalanced public cases and seven third-family external held-out cases, each repeated twice. They cover latent numeric/letter binding, both orders of noncommutative string and arithmetic operations, and single-step controls. Both suites validated; evaluation tests passed 21 of 21 and the full suite 80 of 80.
Three pre-candidate baselines were quality-identical. Public pass rate 57.14 percent, weighted 0.9286; held-out pass rate 28.57 percent, weighted 0.8810; zero invariant failures. Single-step controls were 1.0 while latent binding was 0.8333.
Baseline public runs: b61d9bac-8e18-4c64-b3db-f0923d49ee18, 681cd064-8cb8-41bb-938d-0a888c469a06, 8fe97d0a-ab4c-4767-b603-17d0f145903f. Baseline held-out runs: 4fc1331e-1692-40aa-b437-127553a42e7b, 548d8465-2f80-4aba-b56a-c1c9f9a1cb90, f9fbd4f7-2232-43c3-917a-90c78598f74a.
With the candidate, focused tests passed 23 of 23 and the full suite 81 of 81. Candidate run 1 left public unchanged at 57.14 percent and increased held-out to 42.86 percent. Runs 2 and 3 exactly matched baseline at 57.14/28.57 percent. Latent binding remained unchanged in every run and invariant failures remained zero.
Candidate public runs: 25920070-a3fb-4c92-a7ec-5714a91209e0, ed8ae109-534b-44e4-bc42-0bf6670dccf8, 8a249710-3b1d-4010-bb9a-841054b19d13. Candidate held-out runs: faeeaa1f-01bb-47c1-a7df-c28ada240627, f9bbb539-a11f-4709-b232-df34aac00d83, 0db56bfe-c3bb-46e6-a0de-6ec3b090113d.
The candidate failed its falsification gate because improvement was neither repeatable nor present in the target latent-binding category. The prompt change and temporary test were removed. The restored full suite passed 80 of 80.
No evaluator, assertion, held-out case, resolver, safety policy, grounding requirement, credential, environment file, or CI configuration was changed by the candidate. The reusable public counterbalanced suite was retained and committed locally.

Rejected two-stage ordered-operation candidate

Rejected and removed

Hypothesis: separating operation inference from execution into two local Gemma calls would preserve order and intermediate state better than a prose reminder. The first call produced a compact internal plan without the answer; the second executed that plan against the original request.
The generic activation scope required either multiple arrow demonstrations or two defined named operations plus an ordered application request. Focused tests passed 13 of 13 and the candidate full suite passed 87 of 87 before live evaluation.
Three public qualification runs were identical at 71.43 percent, weighted score 0.9524, zero invariant failures, and 100 percent single-step controls, improving the 57.14 percent baseline. Public run IDs: b7122ca0-4832-4ca2-81bd-539b0fcd9e74, 56cfcb2d-d43a-4ac3-ba9e-1108324dd656, 3d67c51e-5248-423f-a060-909534a0c4cd.
Because the public gate passed, three unchanged repository-external held-out runs were executed. Pass rates were 28.57, 42.86, and 42.86 percent versus three 28.57 percent baselines; weighted scores were 0.8810, 0.9048, and 0.9048; all had zero invariant failures and 100 percent controls. Held-out run IDs: e7a69f8b-fe4b-4f0f-9be8-ce70bee17627, eaf7ab66-b4fb-429a-be64-bc1d0020a498, 0264cb5c-c9cf-4e82-a808-18bdc21c5f52.
Broader regression gates were decisive. Core RSI remained perfect at 100 percent with zero invariants (dc10bc6d-969b-471e-8fbb-9044aabc2343), but Level 1.1 fell from the stable 75 percent baseline to 62.5 percent with p95 latency 2590.5 ms (e6f9f4e4-e426-4f05-bbd8-f1453e94a41b).
The broad arrow trigger intercepted ordinary single-rule induction examples, demonstrating negative transfer. The candidate was removed rather than tuned after held-out observation. The restored complete suite passed 80 of 80. No response behavior was promoted.
Gemma startup identified the intended model at PID 2624, with 7.521-second startup and 0.167-second warmup. Concurrent log rollover repeatedly encountered a non-fatal Windows file-lock error; evaluation completion and ledger records remained intact.

Evaluation reliability kernel hardening

Completed locally

Shared one rotating handler per log file and deferred Windows rollover retries instead of emitting repeated PermissionError tracebacks.
Added an atomic SQLite single-writer lease with process liveness, stale/dead recovery, in-flight heartbeats, process-bound owner identity, and acquisition before readiness traffic.
Added planned run IDs, resume, immutable SHA-256 case fingerprints, incomplete-run diagnostics, and exact skip semantics for already-recorded repetitions.
Added exact configured-model checks, model-list verification, explicit READY sentinel warmup, latency ceiling, local-standard-only routing, and a no-paid-tier evaluation guard.
Added bounded recovery probes after case deadlines and local HTTP transport failures. Recovery failure leaves an incomplete resumable run and always releases its owned lease.

Behavior candidates and no-regression gates

Rejected; code removed

Typed internal ledger candidate public runs were 25, 35, and 35 percent versus 40-50 percent baselines, with 3.4-6.1 second p95 latency. It failed the public gate, so the external suite was not run.
Broad dual solving reached 50 percent on all three public named-operation runs and 55-60 percent external, but latent-chain behavior worsened and p95 latency reached 6.1-9.6 seconds. The broad mechanism was removed.
The narrowed named-operation-only mechanism held public at 50 percent but remained exactly 50 percent on all three fresh external-family runs. It failed the generalization gate and was removed.
Candidate changes never touched evaluator scoring, assertions, held-out content, safety policy, grounding, credentials, environment files, or CI. No promotion decision was recorded as accepted.

Evaluation memory isolation

Completed locally

Diagnosed that evaluation sessions were saved into the normal conversations table and then selected by cross-session episodic retrieval, creating order-dependent leakage and contaminating real-user context.
Evaluation and other internal session types now suppress message persistence and fact extraction through a context-local guard. Evaluation prompts also skip personal memory, learned facts, history, and episodic context.
User-facing cross-session queries exclude eval:, sim_, lab_, and benchmark_ session prefixes, so the 2,898 historical evaluation rows are no longer retrieved. Existing data was not deleted.
Three isolated public runs scored 75, 68.75, and 75 percent; three isolated external runs scored 68.75, 62.5, and 62.5 percent. All had zero invariants and sub-505 ms p95, confirming cleaner but still sampling-variable capability measurement.

Live failure and recovery drills

Passed

Forced-timeout run timeout-recovery-local-only-20260716-2231 recovered in 0.140 seconds; the next READY control passed in 0.683 seconds. Five later soak runs recovered in 0.184-0.201 seconds with every next control passing.
Cross-process contention rejected blocked-contention-live-20260716-2340 in 601 ms before run_started or readiness traffic, then returned to zero lease rows.
Hard interruption left resume-live-soak-20260716-2345 started with zero results and a dead-owner lease. A 0.435-second slot probe recovered, resume completed both cases with zero invariants, and status returned no writer or incomplete run.
The wrong-model preflight validation created neither run evidence nor a lease. A separate live interruption previously resumed from one result to exactly 20 without duplication.

Prompt-load capability diagnostic

Diagnostic only; not promotable

Three direct minimal-prompt runs were identical at 75 percent public and 75 percent on the already-observed external family, weighted score 0.9583, zero invariants, and approximately 309-333 ms p95.
Hiro base-persona direct prompting held the external family at 75 percent at temperature zero; the full memory/tool prompt fell to 62.5 percent. Matched 0.7-temperature runs showed the same direction.
String-order failures remained under every prompt ablation, demonstrating a real model limitation. Full context added a formatting miss, and cross-case episodic contamination added an additional list-order miss.
Because these task families had already been observed and the direct path omitted real Hiro orchestration, the diagnostic is mechanism evidence only. It does not justify response promotion.

Decisions and reasoning

Use LM Studio CLI as the supported Windows runtime entry point; the bundled llama-server is a backend shim.
Keep controlled evaluation single-slot and serial.
Treat HTTP/HTTPS failures as missing application-server coverage, not Gemma failures.
Do not promote response behavior from this runtime test alone.
Accept these runs as a stable runtime baseline, not as authorization to promote response behavior.
Investigate the two resolver false positives with narrowly scoped hypotheses before proposing any behavior change.
Keep the candidate as a local proposal supported by matched evidence; do not auto-promote or deploy it.
Adopt the 20/80 allocation as a guardrail: reliability work validates measurements, while the majority of effort targets transferable capability rather than benchmark-specific patches.
Treat suite mastery and resolver correctness as prerequisites, not evidence of emergence.
Accept the three runs as an informative Level 1 baseline, but do not treat aggregate pass rate alone as the emergence measure; family-level transfer and variance are the primary evidence.
Do not create a behavior candidate until calibration undercoverage is addressed prospectively in a new suite version and the stable composition failures have a falsifiable mechanism hypothesis.
Adopt ordered operation binding as the leading composition-failure hypothesis: when multiple inferred or abstractly named transformations must be bound and sequenced, Hiro loses or substitutes the intended operations.
Do not use the independent held-out aggregate as a pure reasoning score because one case was resolver-contaminated and another exposed assertion undercoverage; preserve the run and use uncontaminated family evidence.
Accept Level 1.1 as a stable matched baseline. The exact repetition agreement satisfies the planned stability gate for forming - but not yet promoting - an ordered-operation-binding candidate.
Reject generic system-prompt reminders as the ordered-binding intervention. They do not produce stable transfer and should not be promoted.
Retain the ordered-binding hypothesis but revise the mechanism: future work should test structured intermediate representations or an explicit bounded reasoning scaffold rather than another prose reminder.
Reject the two-stage candidate because a stable public gain and partial held-out gain do not outweigh a measured Level 1.1 regression. Do not tune this tested mechanism against already-observed held-out results.
Retain reliability and evaluation-isolation changes locally for review; do not auto-promote, merge, push, deploy, or change production services.
Reject all three response mechanisms because public regression, latency, latent-chain weakness, or failure on a fresh external family violated the gates.
Treat the minimal-prompt result as evidence that prompt and memory relevance matter, not as a production-ready fast path.
Do not delete the 2,898 historical evaluation conversation rows automatically. Exclude them from retrieval now and require an explicit backup-and-cleanup decision later.
Keep production user-channel persistence behavior unchanged while suppressing internal evaluation, simulation, lab, and benchmark state.

Validation and evidence

Check	Status	Result
Startup and warmup	passed	Verified gemma-4-12b-it-qat-q4_0; startup 7.437 seconds, warmup 0.210 seconds.
Gemma smoke benchmark	partial	30 passed, zero model timeouts/errors, 0.71-second average. Two app-channel connections failed because the app server was off; two manual checks skipped.
Hiro agent and cancellation	passed	Agent turn 1.396 seconds; first recovery probe succeeded 0.144 seconds after cancellation.
Python suites	passed	76 complete-suite tests passed; 10 final focused tests passed.
Final process state	passed	Telegram disabled, evaluator absent, model unloaded, port 8080 closed, dashboard untouched.
Three consecutive evaluation pairs	passed	All three produced identical 87.5 percent public and held-out pass rates with zero invariant failures and stable sub-second p95 latency.
Manifest equivalence	passed	Code, config, prompt, toolset, and model hashes were identical across the three repetition labels.
Evaluation shutdown	passed	Telegram disabled, model unloaded, port 8080 closed, no evaluator process, dashboard untouched.
Focused resolver and integration tests	passed	21 tests passed in 0.80 seconds.
Complete Python suite after resolver changes	passed	80 tests passed in 5.01 seconds.
Matched public candidate evaluation	passed	16 of 16 observations passed, weighted score 1.0, zero invariant failures, p95 latency 924.1 milliseconds; run 2f5d56b1-53d3-420a-8557-01ca346cd4fa.
Matched held-out candidate evaluation	passed	16 of 16 observations passed, weighted score 1.0, zero invariant failures, p95 latency 545.3 milliseconds; run c4073d44-62b7-400f-9892-908e7f647b95.
Post-evaluation lifecycle	passed	Telegram disabled, Gemma unloaded, port 8080 closed, no evaluator process, dashboard untouched.
Level 1 suite validation	passed	Eight public and eight external held-out cases validated at version 1.0.0.
Level 1 three-run baseline	passed	All six component runs completed with zero invariant failures and sub-second p95 latency; public mean pass rate 66.7 percent, held-out mean 52.1 percent.
Final lifecycle state	passed	Gemma unloaded, Telegram disabled, port 8080 closed, no evaluation writer, dashboard PID 41900 remains running.
Level 1.1 matched diagnostic	passed	Public 75 percent and held-out 62.5 percent; calibration 1.0 on both; zero invariants.
Independent held-out transfer	diagnostic	37.5 percent overall with stable genuine failures plus one resolver-contaminated analogy and one semantically correct calibration false negative.
Composition mechanism isolation	passed	Component skills and explicit composition 100 percent; operation-order controls and latent composition 0 percent, supporting the ordered-binding hypothesis.
Final lifecycle state	passed	Gemma unloaded, Telegram disabled, port 8080 closed, no evaluator process, dashboard PID 41900 running.
Three-run Level 1.1 stability	passed	All three pairs were quality-identical at 75 percent public and 62.5 percent held-out, with calibration 1.0 and zero invariant failures.
Post-baseline process state	passed	Gemma unloaded, Telegram disabled, port 8080 closed, no evaluator process, dashboard PID 41900 running.
Ordered-binding pre-candidate stability	passed	Three identical baselines: public 57.14 percent, held-out 28.57 percent, zero invariants.
Ordered-binding candidate replication	failed	Public unchanged in all runs; held-out improved once but returned to baseline twice; latent binding never improved.
Rejected-candidate cleanup	passed	Candidate prompt and temporary test removed; complete suite restored to 80 of 80 passing.
Final process state	passed	Gemma unloaded, Telegram disabled, port 8080 closed, no evaluator process, dashboard PID 41900 running.
Two-stage public qualification	passed	Three identical 71.43 percent runs, weighted score 0.9524, zero invariants, and 100 percent single-step controls.
Two-stage held-out transfer	diagnostic	28.57, 42.86, and 42.86 percent; partial repeatable gain but not uniform across runs; zero invariants.
Broader no-regression gate	failed	Level 1.1 regressed from 75 to 62.5 percent, while core RSI remained 100 percent.
Two-stage cleanup	passed	Candidate and temporary tests removed; restored repository suite passed 80 of 80.
Final process state after two-stage test	passed	No evaluator, port 8080 closed, Telegram notifications disabled, dashboard PID 41900 listening on port 8001.
Final isolated Python suite	passed	114 tests passed in 4.99 seconds; git diff --check passed and the worktree was clean.
Public reliability confirmation	passed	final-public-r1/r2/r3-20260716: all 75 percent, weighted 0.9583, zero invariants, p95 544.7, 508.8, and 914.6 ms.
External reliability confirmation	passed	final-heldout-r1/r2/r3-20260716: all 50 percent, weighted 0.9167, zero invariants, p95 810.6, 547.7, and 755.4 ms.
Evaluation memory isolation	passed	Live held-out run added zero rows to the conversation database. Known synthetic prompts were absent from real-user episodic retrieval.
Timeout cascade soak	passed	Five forced deadlines recovered in 184-201 ms; all five immediately following controls passed in 181-334 ms.
Writer contention	passed	Second process rejected as busy in 601 ms, with zero blocked run_started rows and zero lease rows after owner exit.
Hard interruption and resume	passed	resume-live-soak-20260716-2345 recovered a dead lease, completed 2/2 with zero invariants, and left no incomplete run.
Scope audit	passed	No scoring, assertion, held-out, safety, credential, certificate, environment, CI, or grounding file changed.
All-run invariant audit	passed	64 completed lab runs were audited. Only seven deliberately forced timeout cases had invariants; no ordinary run had an unexpected invariant failure.
Idle-gap readiness soak	passed	Five exact READY probes passed at 172.5-474.0 ms.
Final lifecycle state	passed	Telegram disabled and absent; no evaluator; no writer or incomplete run; port 8080 closed; dashboard HTTP 200 on PID 41900.

Current state

Launcher and shutdown changes remain local with pre-existing Telegram lifecycle work; no Hiro commit or push was made.
Gemma is fast and reliable under the present low-contention machine state.
Model and Telegram are intentionally stopped; dashboard remains running.
A non-fatal PermissionError occurred when another process held logs/main.log during rollover.
Runtime readiness is established, but behavior promotion still requires repeatable evaluation baselines.
The three-run stability requirement is now satisfied for the repaired Gemma runtime.
Two repeatable resolver false positives remain: sort intent incorrectly triggers Gmail, and an unknowable held-out query incorrectly triggers weather.
The resolver-boundaries-20260716 candidate is perfect on the matched public and held-out pair, improving both pass rates from 87.5 to 100 percent without an invariant or latency regression.
The candidate remains local and unpromoted pending an explicit promotion decision.
The next Hiro phase is a controlled emergence program beginning at rule induction and compositional transfer while reliability fixes proceed under a capped twenty-percent budget.
Level 1 now has a non-ceiling baseline with reproducible strengths, reproducible weaknesses, and measurable cross-run variance.
No response candidate was created or promoted from these baseline observations.
Steps 1 through 3 are complete. Hiro now has a falsifiable ordered-operation-binding hypothesis rather than a generic reasoning diagnosis.
No reasoning behavior, prompt, resolver, evaluator implementation, scoring code, safety rule, or grounding requirement was changed in this diagnostic pass.
The originally requested steps 1 through 4 are now complete. Level 1.1 has three stable unchanged baselines plus independent held-out and mechanism-diagnostic evidence.
The first ordered-binding candidate is rejected, removed, and not promoted. All evidence remains in the append-only ledger.
The next mechanism should be materially different: a structured internal step representation or bounded two-stage execution, tested first on development-only counterbalanced cases.
The two-stage infer-then-execute mechanism is rejected and absent from the working tree. Its public gain was real, but it caused negative transfer on Level 1.1.
All new evaluation evidence remains in the append-only local ledger; no Hiro commit, push, deployment, or promotion occurred.
The isolated branch contains sixteen new local commits and is clean. It has not been pushed or merged.
The main Hiro workspace still contains the same pre-existing uncommitted Telegram lifecycle work and was not edited by this lab.
All response candidates are absent. Only reliability, diagnostics, the prospective public typed-operation suite, and evaluation-memory isolation remain.
The shared ledger has no active writer and no incomplete run. The dashboard remains on port 8001.
2,898 historical eval: conversation rows remain in the database but are excluded from all user-facing episodic retrieval.
Final process state: Telegram disabled and stopped, Gemma unloaded, port 8080 closed, no evaluator, no ledger lease, no incomplete run, and dashboard PID 41900 still serving HTTP 200.

Next steps

Review and selectively integrate the reliability commits before any further long capability session.
Decide explicitly whether to back up and remove the 2,898 historical synthetic conversation rows; do not perform destructive cleanup implicitly.
Design a production-capable memory-relevance or prompt-budget mechanism, then freeze a genuinely new external family before testing it. Do not tune against the observed named-consensus families.
Separate genuine string-operation binding limits from formatting and context-load effects with counterbalanced fresh tasks.
Continue proposal-only gates with at least three public and two independently generated external families, zero invariant regressions, and production-channel fidelity.