Local model tuning and Hiro self-improvement orchestration

Executive summary

Hiro's two local-model libraries were relocated from the space-constrained system volume to dedicated directories on E:. LM Studio now downloads to E:\AI\Models\LMStudio, while Hiro's legacy GGUF library resides at E:\AI\Models\llama.cpp.

The migration used copy-first activation. Every source and destination file was compared by relative path, byte length, and SHA-256 before the original paths were replaced by compatibility junctions. This preserved Hiro's existing pinned paths without changing application source.

LM Studio discovered four language models and one embedding model after migration. A real 6.50 GiB Gemma 4 12B load from E: completed, used the intended 4,096-token single-slot configuration, returned READY in 0.363 seconds, and unloaded cleanly.

Removing the verified C: backups increased measured free space from 20.83 GiB to 65.25 GiB, reclaiming 44.42 GiB. The main tradeoff is cold-load latency: the Gemma load took 3 minutes 13.95 seconds from E:, far slower than the historical system-volume startup.

A controlled follow-up compared the same 6.497 GiB Gemma file on the separate D: and E: USB hard drives. Two uncached reads averaged 104.16 MiB/s on D: versus 42.52 MiB/s on E:, making D: 2.45 times faster. A fresh-file LM Studio cold load took 69.67 seconds from D: versus 199.73 seconds from E:, a 65.1% reduction.

Once loaded, storage placement did not materially change inference: the first fixed response took 0.252 seconds from D: and 0.253 seconds from E:, with subsequent responses near 0.11 seconds on both. Hiro was restored to its original E: configuration after the benchmark; both temporary D: copies were deleted, and no model or server was left running.

After the benchmark established D: as the faster device, Hiro's complete active model estate was intentionally migrated from E: to D:. The move covered 19.392 GiB across five LM Studio files and 31.516 GiB across thirteen legacy files; all 50.908 GiB matched by relative path, length, and SHA-256 before activation.

LM Studio now downloads directly to D:\AI\Models\LMStudio, while the active legacy library is D:\AI\Models\llama.cpp. Direct compatibility junctions preserve both the prior C: and E: paths. A real Gemma load completed in 69.76 seconds, three READY responses passed, the E: backups were removed, and no model or server was left running.

Downloaded bartowski's recommended single-file Q4_K_M conversion of Qwen3.6-35B-A3B directly through LM Studio to D:. The 22,285,080,192-byte main GGUF matched the publisher's SHA-256 exactly, and LM Studio also obtained the 899,283,296-byte F16 vision projector.

Qwen3.6-35B-A3B Q4_K_M was evaluated in LM Studio with an 8,192-token context, one parallel slot, and full GPU offload. Estimate-only reported 22.51 GiB. The real load completed in 399.48 seconds; LM Studio reported a 21.59 GiB model footprint, while total GPU use increased from 6,307 MiB before loading to 29,996 MiB after loading.

The 120-case deterministic Hiro core suite completed without API errors at a raised 1,024-token generation ceiling. The primary run averaged 78.37%, with 94 of 120 cases meeting the 70% pass threshold: evidence obedience 29/30, no-evidence refusal 17/20, simulated tool routing 25/25, structured JSON 20/20, troubleshooting 0/15, and self-improvement specification 3/10.

The weak categories were dominated by reasoning-token starvation rather than inability to solve the tasks. LM Studio kept Qwen3.6 thinking enabled even when OpenAI-compatible template parameters and the native reasoning-off request were attempted. A repeated 120-case sweep reproduced the same runaway-reasoning pattern. Two representative failed cases succeeded when retried at 4,096 tokens, taking 8.49 and 8.99 seconds and consuming 1,144 and 1,240 hidden reasoning tokens respectively.

Qwen3.6 passed a real function-call request in 1.21 seconds, selecting the requested weather function with the correct Seattle argument. It also passed a near-full-window retrieval probe with 7,623 prompt tokens and 7,817 total tokens in 2.46 seconds, returning the exact final ledger marker.

Raw decode throughput is strong on the RTX 5090: a sustained sample produced 2,048 completion tokens in 12.02 seconds, or 170.32 tokens per second. However, 1,829 tokens, 89.3%, were hidden reasoning, and the requested long answer stopped after only 165 visible words. A synthetic vision card was read correctly in 3.10 seconds, identifying all three colored shapes and the text HIRO 8K.

Qwen3.6 is not being promoted to Hiro's default in its current runtime configuration. Its primary 78.37% suite score is below the historical Qwen 9B score of 79.96% and Gemma 4 score of 83.13%, despite the larger output allowance, and reasoning overhead breaks Hiro's compact-response economics. The model was unloaded, the API server was stopped, and total GPU use returned to 6,482 MiB.

Restored Hiro's local Evaluation Observatory at the previously used http://127.0.0.1:8765/benchmarks address. The failure was service availability, not lost dashboard code or evaluation data: no process was listening on port 8765, while the FastAPI route, static dashboard, and 1.5 MB evaluation ledger remained present.

The machine no longer had a registered system Python interpreter capable of running Hiro. Rebuilt the pinned project environment inside Hiro's ignored .runtime directory, leaving the dirty source worktree and all existing application changes untouched.

Started the existing FastAPI application as a standalone loopback-only service on port 8765. Live checks returned HTTP 200 for the dashboard, confirmed the Evaluation Observatory page title, and loaded the read-only overview API with 100 completed runs across 101 variants.

Left the local dashboard service running so benchmark history can be monitored without loading an LLM. The latest ledger entry remains the completed resume-live-soak run with a 1.0 weighted score.

Repeated Qwen3.6-35B-A3B Q4_K_M at 8,192-token context through Hiro's actual append-only Evaluation Observatory ledger, using one slot, full GPU offload, the frozen emergence-level1-public@1.1.0 suite, and the full run_turn agent path.

The valid 16-observation run scored 85.42% with a 37.5% strict pass rate, zero invariant failures, and 3,545.67 ms p95 latency. Three matched Gemma 4 12B ledger runs averaged 95.49%, 72.92%, and 379.91 ms respectively, so Qwen trailed by 10.07 score points and 35.42 pass-rate points while running 9.33 times slower at p95.

A first attempt was preserved as invalid audit evidence because restricted execution made Hiro's main conversation database read-only and every case failed before inference. The valid r2 run used normal Hiro host permissions; its zero invariant failures and recorded model responses distinguish the model result from the infrastructure-only r1 failure.

The matched ledger run reinforces the no-promotion decision. Qwen's forced hidden reasoning frequently exhausted Hiro's 512-token agent completion allowance, leaving partial reasoning fragments instead of the exact compact answers the suite requested. The confidence intervals overlap at this small sample size, so the ledger does not label the score difference statistically clear, but the quality and latency directions are both unfavorable.

After ingestion, the live loopback Evaluation Observatory reported 130 completed runs across 103 variants and identified qwen36-35b-a3b-8k-public-20260720-r2 as the latest run. Qwen was unloaded and the LM Studio API stopped; the dashboard remained available at http://127.0.0.1:8765/benchmarks.

A read-only architectural diagnosis mapped what would be required to move evaluation orchestration and self-improvement from an external ChatGPT-led development loop into Hiro. Hiro already has strong measurement primitives and proposal generation, but the trusted variant laboratory is intentionally operator-driven while the autonomous legacy loop lacks equivalent isolation, comparative baselines, and reliable rollback.

The central migration challenge is preserving an evaluator that the candidate cannot influence. Hiro may own scheduling, hypothesis formation, candidate construction, test execution, and evidence packaging, but evaluator definitions, held-out cases, safety policy, permission grants, and high-risk promotion authority must remain outside the mutable candidate boundary.

Changed Hiro's local router from a universal 512-token completion ceiling to model-profile-aware defaults. Gemma and the smaller Qwens retain 512 tokens, explicitly short calls retain their requested limits, and the new qwen35 profile receives 7,168 tokens on both non-streaming and streaming paths.

The larger Qwen budget fixed the observed reasoning starvation. A live Hiro preflight expanded the ordinary request to 4,096 tokens and returned visible READY in 1.41 seconds; a 4,096-token ledger run then scored 97.92% and passed 15 of 16 observations, versus 85.42% and 6 of 16 at 512 tokens.

The only 4,096-token failure passed as LABEL: GLIM in 7.63 seconds with 7,168-token headroom. The final 7,168-token run passed all 16 observations, scored 100%, achieved 1.0 in every category, and recorded zero invariant failures.

Qwen's perfect quality result exceeded the three-run Gemma mean by 4.51 score points and 27.08 pass-rate points, and exceeded the starved Qwen run by 14.58 score points and 62.5 pass-rate points. The improvement over starved Qwen is statistically clear under the ledger comparison; the smaller advantage over Gemma is not statistically clear at this sample size.

The quality gain has a substantial latency cost: the perfect run averaged 9.83 seconds, had a 4.82-second median, and reached 36.77 seconds at p95. That p95 is 96.8 times the matched Gemma mean, so the tuned Qwen profile is quality-first rather than a universal low-latency replacement.

After validation, Qwen was unloaded and LM Studio's API stopped. The loopback Evaluation Observatory remained online, reporting 132 completed runs across 105 variants with qwen36-35b-a3b-8k-public-7168-20260720-r4 as the latest result.

Defined a staged operating workflow for transferring self-improvement work from ChatGPT-assisted operation into Hiro. The sequence starts with observation, clustering, experiment specification, evaluation orchestration, and evidence packaging before expanding into isolated candidate construction and narrowly bounded automatic promotion.

The transition is organized around task-level delegation rather than an all-at-once autonomous loop. ChatGPT remains the reviewer and implementation fallback while Hiro proves reliability at each stage, with responsibility expanding after repeated cycles produce complete, reproducible, useful handoff packets.

Implemented cycle coordinator v0, the first operational step in transferring self-improvement work into Hiro. It consumes one structured ImprovementSpec, validates its evidence and planned scope, records an immutable cycle, runs or resumes paired public and external held-out baseline evaluations, pins the baseline, and writes review-ready JSON and Markdown evidence packets.

The coordinator intentionally stops before candidate construction or promotion. Twenty-five focused evaluation, variant-laboratory, CLI, and coordinator tests passed after redirecting pytest temporary files into Hiro's ignored runtime directory; the initial default-temp attempt was an infrastructure failure rather than a product-test failure.

Added a nightly synthetic evaluation harvest to Hiro's existing 22:00 self-improvement job so useful improvement evidence no longer depends on organic conversation volume. The first development suite runs eight versioned public cases with two repetitions each, for at most sixteen Hiro turns per night.

Each nightly run now uses deterministic scoring and the append-only evaluation ledger, resumes interrupted work, reuses an already-completed same-day run, compares against the previous matching run, and converts the highest-impact failure category into a cycle-coordinator-compatible ImprovementSpec.

The scheduler remains proposal-only: held-out cases, candidate construction, promotion, and live code changes are excluded. Twenty-eight focused evaluation, laboratory, coordinator, nightly-harvest, and scheduler tests passed.

Removed the running Hiro API as a manual prerequisite for the nightly evaluation. A registered Windows task now wakes at 21:55, checks the real Observatory endpoint, starts Hiro through the pinned project runtime only when needed, waits for readiness, and keeps the API process attached through a ten-hour overnight window.

Preserved one scheduler authority by serving HTTPS 8000, HTTP 8001, and the familiar loopback Evaluation Observatory at 8765 from the same Hiro process. Live verification showed all three listeners owned by one PID, while the scheduled preflight remained in the Running state.

Added a Nightly tab to the Evaluation Observatory. It reads append-only nightly ledger runs, displays pass rate, weighted score, failures, latency, failed cases, longitudinal history, and the deterministic improvement signal without exposing held-out cases or adding promotion controls.

Added a daily Codex readiness review for Hiro's nightly self-improvement process. The review runs each morning after the overnight window, classifies the latest result separately from infrastructure failures, compares valid nights longitudinally, and recommends bounded adjustments only after repeated reliable evidence.

The readiness policy requires at least three valid completed nights, with five preferred, before expanding the workload. It favors broader case diversity and adaptive testing over repeated execution of the same small suite, and reserves larger workloads for a separately reviewed weekly deep run.

A live pre-night check found that the API and Evaluation Observatory were available and the 22:00 schedule was enabled, but the local model endpoint was unavailable. Consequently, an evaluation recorded tonight without additional model startup would be classified as infrastructure evidence rather than a valid product-quality baseline.

Work completed

Decisions and reasoning

• Use two explicit E: destinations instead of merging unrelated directory layouts. LM Studio-managed models and Hiro's legacy files retain their existing organization and identifiers.
• Keep compatibility junctions even after updating LM Studio's direct download setting. Hiro contains pinned legacy paths, and junctions preserve behavior without coupling this operational migration to application-source edits.
• Copy and hash before activation rather than performing an unverified cross-volume move. Cross-volume failure must not leave Hiro with a partially populated live library.
• Keep C: backups until a real model loads and answers from E:. File equality proves copy integrity, while runtime validation proves the inference stack can consume the relocated model.
• Remove the large backups only after the real-load gate passes. The measured free-space result, not logical file totals, is the authoritative reclaimed-capacity value.
• Treat E: cold-load speed as a material limitation. The volume offers capacity, but the 3-minute-plus Gemma load means larger candidate startup times must be measured before operational adoption.
• Do not claim Hiro's launcher validation is sufficient when only the model-discovery endpoint responds. LM Studio can list installed models while none are loaded, so readiness must include loaded-state or completion evidence.
• Use direct uncached reads of the exact GGUF in addition to application load timing. This separates storage delivery from LM Studio initialization and Windows file-cache effects.
• Reject ambiguous application trials. A D:-routed load was not accepted until the preserved E: copy was outside LM Studio's scan tree and discovery returned exactly one Gemma candidate.
• Treat D: as the preferred hot-model volume based on the measured 2.45-times read advantage and 65.1% cold-load reduction. E: remains useful as higher-free-space bulk storage.
• Restore the permanent E: configuration after benchmarking because the request authorized measurement, not a second storage migration. Moving the active library to D: should be a separate intentional change.
• After receiving explicit authorization, move both the managed and legacy active libraries to D: rather than moving only Gemma. This makes performance consistent for every currently installed local model and consumes only about 51 GiB of D:'s available capacity.
• Keep separate D: directories for LM Studio-managed and legacy llama.cpp assets. Their layouts, identifiers, and consumers differ, and merging them would add operational risk without improving speed.
• Point the C: compatibility junctions directly to D: and retain E: junctions to D:. Direct C: links avoid an unnecessary hop, while E: links preserve scripts or tools that adopted the intermediate E: paths.
• Keep a small pre-D: LM Studio settings backup but remove the large E: model backups only after discovery, pinned-path checks, real loading, inference, unload, server shutdown, and final junction validation pass.
• Do not modify Hiro source paths during the storage move. Compatibility junctions preserve behavior while the model-path centralization work remains a separate, reviewable task.
• Use the base Qwen3.6-35B-A3B rather than an abliterated or instruction-altering derivative for the first trial. This keeps the evaluation anchored to the Qwen team's published model behavior.
• Select bartowski's Q4_K_M conversion because it is a standard, single-file, recommended default quant with a published SHA-256 and enough quality headroom while fitting the RTX 5090 with a practical context.
• Let LM Studio manage both the main GGUF and vision projector so discovery, multimodal pairing, identifiers, and future library maintenance remain native to the runtime.
• Download and verify now, but do not silently make the model Hiro's default. Resource estimation establishes feasibility; a controlled runtime and task-quality evaluation must precede routing changes.
• Evaluate Qwen3.6 at 8K rather than the earlier 4K estimate because 8K is the intended first operational context for Hiro and still fits the 32 GB GPU at one slot.
• Raise the suite ceiling from the historical 220 tokens to 1,024 for this checkpoint because separated hidden reasoning otherwise prevents many responses from reaching visible output. Record this as a comparability caveat rather than silently treating starved output as model quality.
• Do not compensate for the current reasoning configuration by making 4,096 output tokens Hiro's default. Representative tasks become answerable, but the latency and hidden-token overhead are excessive for routine compact responses.
• Keep Gemma 4 and Qwen 9B as the operational baselines until Qwen3.6 can run with a reliable reasoning-off chat template or runtime control and then pass the same compact-budget benchmark.
• Treat raw completion throughput and useful visible throughput as separate metrics for reasoning models. The measured 170.32 tokens per second is hardware throughput, not equivalent to 170 visible answer tokens per second.
• Restore the exact 127.0.0.1:8765 dashboard address as a standalone loopback service rather than changing Hiro's committed 8000/8001 application ports.
• Reconstruct the missing Python runtime under the already ignored .runtime directory. This restores execution without mixing generated environment files into the dirty source worktree.
• Use Hiro's existing FastAPI dashboard and evaluation-ledger API instead of creating a replacement static report, preserving all run, comparison, and lineage functionality.
• Leave the dashboard process running because the user requested an active tracking page; keep it independent from LM Studio so viewing results does not consume model VRAM.
• Use the existing Gemma 4 12B evaluation-isolation runs as the matched ledger baseline because they use the same suite and full Hiro agent path. The earlier Qwen 9B score remains useful broader context but is not represented by a directly compatible run in this ledger slice.
• Keep the infrastructure-only r1 record rather than deleting or rewriting append-only evidence. Exclude it explicitly from model comparison and use the completed r2 run as the valid candidate.
• Do not promote Qwen3.6 from this ledger result. It is materially worse on both compact-answer quality and latency under Hiro's current 512-token agent allowance, even though the small public suite does not make the score difference statistically clear.
• Treat reasoning control as the blocking experiment. A future rerun should keep the same frozen suite and ledger path while changing only a verified reasoning-off runtime or template, so any improvement is attributable.
• Treat self-improvement autonomy as an orchestration migration, not a transfer of unlimited authority. Hiro can own the repetitive research loop while protected evaluation assets and high-impact decisions remain independently governed.
• Retire or quarantine the legacy live-checkout mutation path before enabling autonomous candidate construction. Two competing pipelines with different safety and evidence semantics would make audit results ambiguous.
• Require candidate generation and execution to occur in disposable, resource-limited worktrees or equivalent sandboxes. Stable Hiro must supervise candidates; candidates must not supervise themselves or share writable evaluator state.
• Keep held-out cases, evaluator definitions, safety policy, permission boundaries, deployment credentials, and promotion policy outside candidate write scope. Automatic benchmark generation may add training and public-development cases, but it cannot manufacture its own promotion evidence.
• Use human approval as a permanent authority boundary for evaluator, safety, permissions, dependencies, schema migrations, deployment, and other high-blast-radius changes. A future automatic lane should be limited to predefined reversible scopes with proven rollback and health checks.
• Use a profile-aware budget instead of globally deleting completion limits. The stronger GPU makes Qwen's long reasoning affordable, but smaller models and intentionally compact tool summaries still benefit from bounded outputs.
• Set qwen35's 8K quality-first budget to 7,168 tokens after 4,096 passed 15/16 and an isolated 7,168-token retry fixed the remaining failure. This preserves a nominal prompt reserve while allowing the forced-reasoning GGUF to reach visible output.
• Treat the perfect r4 result as proof that output starvation, not core task capability, caused the earlier failure. Keep latency as a separate promotion dimension because the 36.77-second p95 is operationally significant.
• Do not silently change Hiro's active default profile as part of output tuning. The qwen35 profile is now supported and validated, while the final routing decision should explicitly balance perfect suite quality against cold-load and tail-latency costs.
• Transfer self-improvement responsibilities by task class, beginning with repetitive evidence handling and evaluation work rather than handing the complete cycle to Hiro at once.
• Make evaluation orchestration the first major autonomous responsibility after nightly triage because its outputs are structured and independently checkable.
• Limit each early cycle to one primary opportunity and one candidate so failures remain attributable and review packets stay compact.
• Keep ChatGPT as implementation fallback and reviewer during the first stages, then reduce its role after Hiro repeatedly produces complete experiment records and reproducible results.
• Use measured graduation criteria for each stage: complete artifacts, reproducible reruns, low duplicate or invalid proposal rates, zero worktree-scope violations, and successful rollback drills before authority expands.
• Treat automatic promotion as the final transition stage and initially restrict it to a small allowlist of reversible prompt, test, benchmark, or non-operational configuration changes.
• Use a distinct append-only ImprovementCycle record before candidate creation rather than fabricating a VariantExperiment with a nonexistent branch or worktree.
• Make cycle identity deterministic across the spec, suite contents, and baseline manifest so retrying the same work is idempotent and changing evidence creates a new cycle.
• Reject reuse of a baseline variant identifier when its immutable code, prompt, configuration, tool, model, or Git identity no longer matches; the operator must choose a new baseline identifier.
• Keep coordinator runtime packets outside version control under .hiro_runtime and write them atomically so an interrupted report write cannot leave a valid-looking partial packet.
• Stop coordinator v0 at baseline_ready. Candidate construction, candidate evaluation, promotion recommendation, merge, and deployment remain later transition stages.
• Summarize held-out evaluation outcomes in the evidence packet without copying held-out cases or assertions into the coordinator output.
• Use a synthetic public development suite nightly because it creates controlled, repeatable usage without consuming the held-out evidence needed to judge future candidates.
• Cap the first nightly harvest at eight cases and sixteen observations. This supplies broad evidence while reserving the remainder of the sixty-call self-improvement budget for conversation analysis, curiosity, journaling, and legacy probes.
• Use deterministic assertions rather than a model judge for nightly scoring. Hiro may generate the answers, but it does not get to reinterpret whether required formats, boundaries, tool restrictions, or content assertions passed.
• Create an improvement proposal only when deterministic failures exist. A perfect nightly run remains useful trend and stability evidence but does not create synthetic busywork.
• Generate a diagnostic, non-code-changing ImprovementSpec from the highest-impact cluster. The next transition stage can turn that diagnosis into allowed files and candidate tests after the shared cause is understood.
• Keep the existing lightweight probes temporarily as a separate compatibility signal. They are not treated as durable improvement evidence because their current integration still reports only a probe count.
• Run API preflight at 21:55 rather than replacing the 22:00 self-improvement scheduler. This separates process availability from evaluation orchestration and preserves the already-tested nightly path.
• Use the public /benchmarks page as readiness evidence. The existing /health endpoint includes model connectivity and may return 503 even when the API and scheduler are fully operational.
• Keep the API process attached to the scheduled task through the overnight window. A short-lived fire-and-forget launcher proved unreliable under Windows process-parent behavior.
• Serve the stable 8765 Observatory address from main.py instead of maintaining a second dashboard process. One process and one application startup lock avoid duplicate scheduler instances.
• Build the Nightly tab from the append-only evaluation ledger rather than scraping generated Markdown reports. Ledger data has explicit schemas, immutable run identity, exact case results, and durable historical comparisons.
• Keep the Nightly API and UI read-only. They expose public-development evidence and diagnostic proposals but do not add candidate creation, held-out access, promotion, or mutation controls.
• Use completed valid nights, not elapsed calendar nights, as the expansion gate. Model outages, API failures, interrupted cycles, and missing runs are operational evidence but do not establish evaluation stability.
• Require at least three valid completed nights and prefer five before changing workload size. This is long enough to expose basic scheduling, persistence, runtime, and proposal-quality problems without delaying small, reversible improvements indefinitely.
• Expand coverage before repetition count. New capability categories, multi-turn behavior, tool use, memory, and recovery scenarios provide more improvement information than repeatedly sampling the same deterministic prompts.
• Keep readiness review advisory. The monitor may recommend a bounded configuration or suite change, but code, schedule, evaluation, and authority changes still require explicit user approval.

Validation and evidence

Current state

• LM Studio's direct download location and managed library are D:\AI\Models\LMStudio.
• Hiro's active legacy GGUF library is D:\AI\Models\llama.cpp.
• Both former C: model paths are direct compatibility junctions to D:, so existing Hiro configuration continues to resolve without source edits.
• Both former E: library paths remain as compatibility junctions to D: for tools that adopted the intermediate storage layout.
• C: has 65.25 GiB free after verified backup cleanup.
• No local model or LM Studio API server was left running.
• D: has 875.34 GiB free after receiving the active model estate and the Qwen3.6 package; E: has 3,391.19 GiB free after its verified backups were removed.
• D: is the measured performance winner for active models: 104.16 MiB/s uncached reads and a 69.67-second Gemma cold load, versus 42.52 MiB/s and 199.73 seconds on E:.
• Drive placement does not materially affect steady inference after the model is resident on the GPU.
• The launcher can confuse model discovery with loaded readiness when LM Studio's server lists installed models. A future readiness fix should require loaded-state or successful completion evidence.
• Qwen3.6-35B-A3B Q4_K_M and its F16 vision projector remain installed in LM Studio's D: library, hash verified, indexed, evaluated at 8K, and currently unloaded.
• The Qwen3.6 8K single-slot configuration fits the RTX 5090 but leaves limited VRAM headroom under the active Windows desktop workload: total observed use was 29,996 of 32,607 MiB.
• Qwen3.6 is not Hiro's default. Its current LM Studio chat template forces heavy hidden reasoning, producing a 78.37% primary core score and severe compact-output budget starvation despite fast raw generation.
• Qwen3.6 passed actual function calling, a 7,623-token context probe, and multimodal image understanding, so it remains a promising candidate once reasoning mode can be controlled reliably.
• The local Evaluation Observatory is available at http://127.0.0.1:8765/benchmarks and its loopback-only service is intentionally left running for benchmark tracking.
• The dashboard runtime lives under the ignored .runtime directory and does not change Hiro's configured 8000/8001 application ports or load an LLM.
• The valid Evaluation Observatory candidate is qwen36-35b-a3b-8k-public-20260720-r2: 0.854167 weighted score, 0.375 pass rate, zero invariant failures, and 3,545.67 ms p95 on emergence-level1-public@1.1.0.
• The matched three-run Gemma 4 12B mean remains substantially better at 0.954861 weighted score, 0.729167 pass rate, and 379.91 ms p95; Qwen3.6 remains unpromoted.
• The append-only ledger also contains an explicitly invalid r1 infrastructure run caused by restricted database permissions; it must not be interpreted as model quality.
• Hiro's scheduled self-improvement v2 loop is proposal-only: it can observe operational evidence and create structured improvement ideas, but it does not construct or promote candidates.
• The trusted evaluation and variant laboratory has the correct beginnings of an autonomy boundary, including external held-out cases, append-only evidence, worktree isolation, fingerprints, and proposal-only gates, but operator sequencing still supplies the orchestration now performed through ChatGPT-assisted development.
• The legacy improvement loop is not a safe shortcut to full autonomy because it mutates the live checkout before evaluation, lacks a rigorous isolated baseline comparison, and does not guarantee automatic restoration after a failed experiment.
• Hiro's router now includes a persistent qwen35 profile at qwen36-35b-a3b with a 7,168-token ordinary completion budget on both streaming and non-streaming paths.
• Gemma, Qwen 4B, Qwen 9B, explicitly short calls, and unknown models retain their previous output-budget behavior.
• The final r4 Qwen result is perfect on emergence-level1-public@1.1.0: 1.0 weighted score, 1.0 pass rate, every category at 1.0, and zero invariant failures.
• The quality-first Qwen profile has a substantial tail-latency cost: 9.83-second mean and 36.77-second p95 in r4, plus a measured 10-minute 18-second cold load in this session.
• The live dashboard exposes 132 completed evaluation runs across 105 variants and keeps qwen36-35b-a3b-8k-public-7168-20260720-r4 as its latest entry.
• The practical transition can begin without granting Hiro merge or deployment authority: its first responsibilities are producing a single ranked opportunity, a falsifiable experiment specification, and a reproducible evaluation packet.
• ChatGPT remains useful during the transition as a reviewer, difficult-change implementer, and diagnosis fallback rather than as the routine operator of every evaluation command.
• The desired end state is a Hiro-operated research loop with review-by-exception: routine cycles are collected, run, and summarized internally, while ambiguous, high-impact, or failing cycles are escalated.
• Cycle coordinator v0 now exists in Hiro but is not scheduled automatically. It is an explicit operator command that may invoke the configured model when given one accepted ImprovementSpec and matched evaluation suites.
• The coordinator's tested endpoint is baseline_ready: a durable cycle, resumable public and held-out runs, a baseline pin, and compact review artifacts, with no candidate or promotion action.
• The transition workflow is now preserved in the project documentation rather than only in conversation history.
• Hiro's persisted lab schedule is enabled, starts at 22:00 local time, and selects self_improvement_v2 in proposal_only mode.
• When the Hiro API process is running, its startup hook launches the lab scheduler and the next nightly job will execute the public rsi-cycle suite before conversation-derived analysis.
• The nightly public run writes full per-observation evidence to hiro/evals/evaluations.sqlite, while the self-improvement report includes a compact summary and any generated diagnostic proposal.
• No held-out evaluation, real model nightly run, candidate worktree, patch, promotion decision, or live runtime code change was executed during this implementation session.
• The older lightweight regression-probe phase still runs after the new harvest and still exposes only a count; it has not yet been migrated to the durable ledger.
• The Windows task Hiro Nightly API Preflight is registered for 21:55 local time and is currently Running after a successful manual end-to-end verification.
• One live Hiro process currently serves HTTPS on 8000, HTTP on 8001, and the loopback Evaluation Observatory on 8765. Both /evaluations/api/nightly endpoints return HTTP 200.
• The Observatory Nightly tab is live at http://127.0.0.1:8765/benchmarks and currently shows an intentional zero-run state pending the first 22:00 nightly evaluation.
• The task uses an Interactive logon principal. It can wake the computer and run while the user session is present, but it is not configured with stored credentials for operation after a full sign-out.
• API availability is now automatically handled before the nightly window; local model availability remains a separate dependency and is reported through evaluation failures or the existing model-health mechanisms.
• The active Hiro nightly readiness review runs daily at 07:30 America/Los_Angeles and reports whether the latest night was valid, operationally invalid, incomplete, or absent.
• No nightly evaluation has completed yet. The readiness counter therefore remains at zero valid completed nights, and no workload expansion is currently justified.
• The current nominal cycle targets 45 minutes with a 60-minute configured limit, but sequential evaluation and probe timeout blocks can allow a severely degraded run to extend to approximately 75 to 80 minutes.
• The local model endpoint was unavailable during the latest live check. Unless it is started before the cycle, tonight's data should be treated as infrastructure failure evidence rather than Hiro capability evidence.

Next steps

• Use D: for new active model trials while reserving E: for archived models, datasets, and other capacity-oriented storage.
• Correct Hiro's Gemma launcher readiness check so an installed-but-unloaded model cannot satisfy the existing-server fast path.
• Record cold-load time, warm-load time, and steady-state inference separately for each model candidate.
• Keep the compatibility junctions until all Hiro model paths are intentionally centralized through configuration.
• Find a supported LM Studio runtime or model chat-template configuration that reliably disables Qwen3.6 reasoning for OpenAI-compatible requests, then repeat the 120-case suite at Hiro's historical 220-token ceiling.
• Update Hiro's benchmark reporting so visible answer tokens and hidden reasoning tokens are recorded separately for reasoning-capable checkpoints.
• If a reasoning-off rerun clears the compact-budget quality gate, add representative coding, repository-editing, and multi-turn tool tests before considering a routing change.
• Keep Qwen3.6 unloaded between experiments unless startup policy is intentionally changed; its measured D: load time was 6 minutes 39.48 seconds.
• If the Evaluation Observatory should survive reboots, add a reviewed launcher or scheduled startup entry for the loopback-only 8765 service instead of relying on an ad hoc process.
• Obtain a verified reasoning-off control for Qwen3.6, then rerun emergence-level1-public@1.1.0 through the same Evaluation Observatory path with the same 512-token agent ceiling and at least three repetitions.
• Add an explicit validity or exclusion annotation mechanism to the append-only ledger UI so infrastructure-only runs such as r1 remain auditable without appearing to be candidate model results.
• Use the Observatory for future model bake-offs whenever a frozen compatible suite and model-identity preflight are available; retain separate ad hoc probes for vision, tool calling, and long-context behavior until ledger suites cover them.
• Design one canonical self-improvement state machine that joins v2 observation and proposal generation to the isolated variant laboratory, and explicitly deprecate the legacy live-mutation path.
• Add a candidate-builder interface constrained by experiment-declared allowed paths, then run it inside the candidate worktree with fixed time, process, storage, model-call, and network budgets.
• Add an orchestrator that evaluates pinned stable and candidate manifests on the same public and external held-out suite versions, resumes interrupted runs, records infrastructure validity separately from model quality, and produces an independently verifiable evidence bundle.
• Before any automatic promotion lane, implement real transactional promotion and verified automatic rollback, post-deployment canaries, a kill switch outside Hiro's control, loop and budget limits, and durable alerts for stalled or repeatedly failing cycles.
• Measure evaluator reliability and contamination resistance over multiple generations before expanding authority. Start with Hiro generating proposals for review, then allow autonomous experiments, and only then consider automatic integration of a very small allowlisted class.
• Repeat the 7,168-token frozen-suite run at least twice more to measure stochastic stability and obtain a multi-run latency distribution before changing Hiro's default routing profile.
• Add visible-versus-reasoning token accounting to the Evaluation Observatory so output headroom and useful-answer efficiency can be compared directly across reasoning models.
• Evaluate a larger runtime context if routine Hiro prompts approach the nominal 1,024-token reserve left by the 7,168-token completion ceiling at 8K context.
• Decide whether qwen35 should be reserved for quality-critical requests or promoted more broadly after weighing its perfect suite score against 36.77-second p95 latency and long cold loads.
• Begin Stage 1 by standardizing the nightly output to one ranked opportunity plus a complete experiment specification and morning briefing; review ten cycles for usefulness, duplication, and evidence quality.
• In Stage 2, let Hiro pin or select a baseline, run matching public and held-out evaluations, resume interrupted runs, compare results, and produce the evidence packet without constructing a candidate.
• In Stage 3, connect approved experiment specifications to candidate worktree creation and allow Hiro to propose edits only inside declared paths, followed by syntax checks and targeted tests.
• In Stage 4, permit a bounded repair loop with a small attempt limit, then run the full stable-versus-candidate evaluation and prepare a promotion recommendation for review.
• After repeated successful reviewed promotions and rollback drills, define the first automatic-promotion allowlist and add post-promotion canary checks before expanding it.
• Track transition metrics separately from product-quality metrics: proposal acceptance rate, duplicate rate, invalid-run rate, human correction rate, time to evidence packet, scope violations, rerun reproducibility, and rollback success.
• Run coordinator v0 on a deliberately selected real ImprovementSpec and compatible public and external held-out suites when the configured local model is ready, using a new immutable baseline variant identifier.
• Review the first real evidence packet for completeness, useful failure classification, model and suite identity, and morning-briefing suitability before scheduling coordinator runs.
• Expose improvement-cycle status and evidence packet references in the read-only Evaluation Observatory after the first real coordinator cycle establishes the desired presentation.
• After repeated clean baseline packets, implement the approved-to-candidate transition that creates an external worktree and records a real VariantExperiment without adding merge authority.
• Allow the enabled 22:00 job to complete one real nightly public harvest and inspect its ledger report, runtime, pass-rate stability, and generated proposal quality.
• Add a morning analysis stage that reads the nightly report and enriches the top diagnostic signal into a falsifiable candidate specification with explicit allowed files and tests.
• Decide whether the legacy lightweight probes add distinct coverage; migrate useful cases into versioned suites and retire redundant count-only execution.
• Build suite rotation or staged expansion after several nights establish runtime and stability, while keeping a fixed anchor suite for longitudinal comparison.
• Add an operating-system wake or catch-up mechanism only if Hiro's API is not reliably running at 22:00; the current scheduler is process-resident and does not backfill a missed launch.
• Inspect the task state and first real Nightly tab entry after tonight's 22:00 evaluation, including whether the run was new, resumed, or reused.
• Confirm that the configured local model is available before 22:00 or add a separate bounded model preflight if unattended model startup is also desired.
• After the first run, verify that clicking its Nightly history row opens the expected per-case responses and deterministic scorer failures.
• Consider switching the Windows task to a non-interactive service principal only if overnight execution must continue after the user fully signs out.
• Add explicit startup-attempt and scheduler-launch events to the Observatory after several nights establish which operational details are useful rather than noisy.
• Let the morning readiness review classify and compare at least three valid completed runs, preferably five, before recommending any nightly workload change.
• When the valid-run gate is met, broaden the public suite with new capability categories before increasing repetitions, while retaining fixed anchor cases for longitudinal comparison.
• After nightly reliability is established, consider a separate weekly deep evaluation of approximately 100 to 200 observations rather than stretching every nightly run into a multi-hour job.
• Close the unattended local-model startup gap so the readiness monitor can distinguish real Hiro quality measurements from repeated connection failures.