{ "schemaVersion": 1, "date": "2026.07.20", "title": "Local model tuning and Hiro self-improvement orchestration", "publicationStatus": "Published", "executiveSummary": [ "Hiro's two local-model libraries were relocated from the space-constrained system volume to dedicated directories on E:. LM Studio now downloads to E:\\AI\\Models\\LMStudio, while Hiro's legacy GGUF library resides at E:\\AI\\Models\\llama.cpp.", "The migration used copy-first activation. Every source and destination file was compared by relative path, byte length, and SHA-256 before the original paths were replaced by compatibility junctions. This preserved Hiro's existing pinned paths without changing application source.", "LM Studio discovered four language models and one embedding model after migration. A real 6.50 GiB Gemma 4 12B load from E: completed, used the intended 4,096-token single-slot configuration, returned READY in 0.363 seconds, and unloaded cleanly.", "Removing the verified C: backups increased measured free space from 20.83 GiB to 65.25 GiB, reclaiming 44.42 GiB. The main tradeoff is cold-load latency: the Gemma load took 3 minutes 13.95 seconds from E:, far slower than the historical system-volume startup.", "A controlled follow-up compared the same 6.497 GiB Gemma file on the separate D: and E: USB hard drives. Two uncached reads averaged 104.16 MiB/s on D: versus 42.52 MiB/s on E:, making D: 2.45 times faster. A fresh-file LM Studio cold load took 69.67 seconds from D: versus 199.73 seconds from E:, a 65.1% reduction.", "Once loaded, storage placement did not materially change inference: the first fixed response took 0.252 seconds from D: and 0.253 seconds from E:, with subsequent responses near 0.11 seconds on both. Hiro was restored to its original E: configuration after the benchmark; both temporary D: copies were deleted, and no model or server was left running.", "After the benchmark established D: as the faster device, Hiro's complete active model estate was intentionally migrated from E: to D:. The move covered 19.392 GiB across five LM Studio files and 31.516 GiB across thirteen legacy files; all 50.908 GiB matched by relative path, length, and SHA-256 before activation.", "LM Studio now downloads directly to D:\\AI\\Models\\LMStudio, while the active legacy library is D:\\AI\\Models\\llama.cpp. Direct compatibility junctions preserve both the prior C: and E: paths. A real Gemma load completed in 69.76 seconds, three READY responses passed, the E: backups were removed, and no model or server was left running.", "Downloaded bartowski's recommended single-file Q4_K_M conversion of Qwen3.6-35B-A3B directly through LM Studio to D:. The 22,285,080,192-byte main GGUF matched the publisher's SHA-256 exactly, and LM Studio also obtained the 899,283,296-byte F16 vision projector.", "Qwen3.6-35B-A3B Q4_K_M was evaluated in LM Studio with an 8,192-token context, one parallel slot, and full GPU offload. Estimate-only reported 22.51 GiB. The real load completed in 399.48 seconds; LM Studio reported a 21.59 GiB model footprint, while total GPU use increased from 6,307 MiB before loading to 29,996 MiB after loading.", "The 120-case deterministic Hiro core suite completed without API errors at a raised 1,024-token generation ceiling. The primary run averaged 78.37%, with 94 of 120 cases meeting the 70% pass threshold: evidence obedience 29/30, no-evidence refusal 17/20, simulated tool routing 25/25, structured JSON 20/20, troubleshooting 0/15, and self-improvement specification 3/10.", "The weak categories were dominated by reasoning-token starvation rather than inability to solve the tasks. LM Studio kept Qwen3.6 thinking enabled even when OpenAI-compatible template parameters and the native reasoning-off request were attempted. A repeated 120-case sweep reproduced the same runaway-reasoning pattern. Two representative failed cases succeeded when retried at 4,096 tokens, taking 8.49 and 8.99 seconds and consuming 1,144 and 1,240 hidden reasoning tokens respectively.", "Qwen3.6 passed a real function-call request in 1.21 seconds, selecting the requested weather function with the correct Seattle argument. It also passed a near-full-window retrieval probe with 7,623 prompt tokens and 7,817 total tokens in 2.46 seconds, returning the exact final ledger marker.", "Raw decode throughput is strong on the RTX 5090: a sustained sample produced 2,048 completion tokens in 12.02 seconds, or 170.32 tokens per second. However, 1,829 tokens, 89.3%, were hidden reasoning, and the requested long answer stopped after only 165 visible words. A synthetic vision card was read correctly in 3.10 seconds, identifying all three colored shapes and the text HIRO 8K.", "Qwen3.6 is not being promoted to Hiro's default in its current runtime configuration. Its primary 78.37% suite score is below the historical Qwen 9B score of 79.96% and Gemma 4 score of 83.13%, despite the larger output allowance, and reasoning overhead breaks Hiro's compact-response economics. The model was unloaded, the API server was stopped, and total GPU use returned to 6,482 MiB.", "Restored Hiro's local Evaluation Observatory at the previously used http://127.0.0.1:8765/benchmarks address. The failure was service availability, not lost dashboard code or evaluation data: no process was listening on port 8765, while the FastAPI route, static dashboard, and 1.5 MB evaluation ledger remained present.", "The machine no longer had a registered system Python interpreter capable of running Hiro. Rebuilt the pinned project environment inside Hiro's ignored .runtime directory, leaving the dirty source worktree and all existing application changes untouched.", "Started the existing FastAPI application as a standalone loopback-only service on port 8765. Live checks returned HTTP 200 for the dashboard, confirmed the Evaluation Observatory page title, and loaded the read-only overview API with 100 completed runs across 101 variants.", "Left the local dashboard service running so benchmark history can be monitored without loading an LLM. The latest ledger entry remains the completed resume-live-soak run with a 1.0 weighted score.", "Repeated Qwen3.6-35B-A3B Q4_K_M at 8,192-token context through Hiro's actual append-only Evaluation Observatory ledger, using one slot, full GPU offload, the frozen emergence-level1-public@1.1.0 suite, and the full run_turn agent path.", "The valid 16-observation run scored 85.42% with a 37.5% strict pass rate, zero invariant failures, and 3,545.67 ms p95 latency. Three matched Gemma 4 12B ledger runs averaged 95.49%, 72.92%, and 379.91 ms respectively, so Qwen trailed by 10.07 score points and 35.42 pass-rate points while running 9.33 times slower at p95.", "A first attempt was preserved as invalid audit evidence because restricted execution made Hiro's main conversation database read-only and every case failed before inference. The valid r2 run used normal Hiro host permissions; its zero invariant failures and recorded model responses distinguish the model result from the infrastructure-only r1 failure.", "The matched ledger run reinforces the no-promotion decision. Qwen's forced hidden reasoning frequently exhausted Hiro's 512-token agent completion allowance, leaving partial reasoning fragments instead of the exact compact answers the suite requested. The confidence intervals overlap at this small sample size, so the ledger does not label the score difference statistically clear, but the quality and latency directions are both unfavorable.", "After ingestion, the live loopback Evaluation Observatory reported 130 completed runs across 103 variants and identified qwen36-35b-a3b-8k-public-20260720-r2 as the latest run. Qwen was unloaded and the LM Studio API stopped; the dashboard remained available at http://127.0.0.1:8765/benchmarks.", "A read-only architectural diagnosis mapped what would be required to move evaluation orchestration and self-improvement from an external ChatGPT-led development loop into Hiro. Hiro already has strong measurement primitives and proposal generation, but the trusted variant laboratory is intentionally operator-driven while the autonomous legacy loop lacks equivalent isolation, comparative baselines, and reliable rollback.", "The central migration challenge is preserving an evaluator that the candidate cannot influence. Hiro may own scheduling, hypothesis formation, candidate construction, test execution, and evidence packaging, but evaluator definitions, held-out cases, safety policy, permission grants, and high-risk promotion authority must remain outside the mutable candidate boundary.", "Changed Hiro's local router from a universal 512-token completion ceiling to model-profile-aware defaults. Gemma and the smaller Qwens retain 512 tokens, explicitly short calls retain their requested limits, and the new qwen35 profile receives 7,168 tokens on both non-streaming and streaming paths.", "The larger Qwen budget fixed the observed reasoning starvation. A live Hiro preflight expanded the ordinary request to 4,096 tokens and returned visible READY in 1.41 seconds; a 4,096-token ledger run then scored 97.92% and passed 15 of 16 observations, versus 85.42% and 6 of 16 at 512 tokens.", "The only 4,096-token failure passed as LABEL: GLIM in 7.63 seconds with 7,168-token headroom. The final 7,168-token run passed all 16 observations, scored 100%, achieved 1.0 in every category, and recorded zero invariant failures.", "Qwen's perfect quality result exceeded the three-run Gemma mean by 4.51 score points and 27.08 pass-rate points, and exceeded the starved Qwen run by 14.58 score points and 62.5 pass-rate points. The improvement over starved Qwen is statistically clear under the ledger comparison; the smaller advantage over Gemma is not statistically clear at this sample size.", "The quality gain has a substantial latency cost: the perfect run averaged 9.83 seconds, had a 4.82-second median, and reached 36.77 seconds at p95. That p95 is 96.8 times the matched Gemma mean, so the tuned Qwen profile is quality-first rather than a universal low-latency replacement.", "After validation, Qwen was unloaded and LM Studio's API stopped. The loopback Evaluation Observatory remained online, reporting 132 completed runs across 105 variants with qwen36-35b-a3b-8k-public-7168-20260720-r4 as the latest result.", "Defined a staged operating workflow for transferring self-improvement work from ChatGPT-assisted operation into Hiro. The sequence starts with observation, clustering, experiment specification, evaluation orchestration, and evidence packaging before expanding into isolated candidate construction and narrowly bounded automatic promotion.", "The transition is organized around task-level delegation rather than an all-at-once autonomous loop. ChatGPT remains the reviewer and implementation fallback while Hiro proves reliability at each stage, with responsibility expanding after repeated cycles produce complete, reproducible, useful handoff packets.", "Implemented cycle coordinator v0, the first operational step in transferring self-improvement work into Hiro. It consumes one structured ImprovementSpec, validates its evidence and planned scope, records an immutable cycle, runs or resumes paired public and external held-out baseline evaluations, pins the baseline, and writes review-ready JSON and Markdown evidence packets.", "The coordinator intentionally stops before candidate construction or promotion. Twenty-five focused evaluation, variant-laboratory, CLI, and coordinator tests passed after redirecting pytest temporary files into Hiro's ignored runtime directory; the initial default-temp attempt was an infrastructure failure rather than a product-test failure.", "Added a nightly synthetic evaluation harvest to Hiro's existing 22:00 self-improvement job so useful improvement evidence no longer depends on organic conversation volume. The first development suite runs eight versioned public cases with two repetitions each, for at most sixteen Hiro turns per night.", "Each nightly run now uses deterministic scoring and the append-only evaluation ledger, resumes interrupted work, reuses an already-completed same-day run, compares against the previous matching run, and converts the highest-impact failure category into a cycle-coordinator-compatible ImprovementSpec.", "The scheduler remains proposal-only: held-out cases, candidate construction, promotion, and live code changes are excluded. Twenty-eight focused evaluation, laboratory, coordinator, nightly-harvest, and scheduler tests passed.", "Removed the running Hiro API as a manual prerequisite for the nightly evaluation. A registered Windows task now wakes at 21:55, checks the real Observatory endpoint, starts Hiro through the pinned project runtime only when needed, waits for readiness, and keeps the API process attached through a ten-hour overnight window.", "Preserved one scheduler authority by serving HTTPS 8000, HTTP 8001, and the familiar loopback Evaluation Observatory at 8765 from the same Hiro process. Live verification showed all three listeners owned by one PID, while the scheduled preflight remained in the Running state.", "Added a Nightly tab to the Evaluation Observatory. It reads append-only nightly ledger runs, displays pass rate, weighted score, failures, latency, failed cases, longitudinal history, and the deterministic improvement signal without exposing held-out cases or adding promotion controls.", "Added a daily Codex readiness review for Hiro's nightly self-improvement process. The review runs each morning after the overnight window, classifies the latest result separately from infrastructure failures, compares valid nights longitudinally, and recommends bounded adjustments only after repeated reliable evidence.", "The readiness policy requires at least three valid completed nights, with five preferred, before expanding the workload. It favors broader case diversity and adaptive testing over repeated execution of the same small suite, and reserves larger workloads for a separately reviewed weekly deep run.", "A live pre-night check found that the API and Evaluation Observatory were available and the 22:00 schedule was enabled, but the local model endpoint was unavailable. Consequently, an evaluation recorded tonight without additional model startup would be classified as infrastructure evidence rather than a valid product-quality baseline." ], "workstreams": [ { "title": "Storage and process inventory", "status": "Completed", "details": [ "Confirmed that no LM Studio, llama.cpp, or Hiro model process and no relevant inference listener was active before copying began.", "Inventoried 19.392 GiB across five files in LM Studio's managed model directory.", "Inventoried 31.516 GiB across thirteen files in Hiro's legacy llama.cpp directory.", "Confirmed that E: was NTFS with approximately 3.39 TiB free and that the selected destination and backup paths did not already exist.", "Read LM Studio's active download setting without exposing credentials and confirmed that it still targeted the system-volume model directory." ] }, { "title": "Copy-first model migration", "status": "Completed", "details": [ "Created E:\\AI\\Models\\LMStudio and E:\\AI\\Models\\llama.cpp as the two durable destinations.", "Copied each library independently with retry-limited, metadata-preserving file transfer.", "For every file, compared source and destination relative paths, byte lengths, and SHA-256 hashes before activation.", "Kept the original directories unchanged throughout copying and hashing. A report-generation bug stopped the first attempt before any live path changed, demonstrating that the activation guard worked.", "Resumed safely from the completed first copy, repeated verification, and activated destinations only after both libraries matched." ] }, { "title": "Compatibility and LM Studio configuration", "status": "Completed", "details": [ "Replaced the former LM Studio model directory with a directory junction targeting E:\\AI\\Models\\LMStudio.", "Replaced the former legacy llama.cpp directory with a directory junction targeting E:\\AI\\Models\\llama.cpp.", "Updated LM Studio's downloadsFolder setting to the direct E: destination while preserving a small pre-migration settings backup.", "The compatibility junctions allow Hiro's existing launcher and benchmark registry paths to continue resolving without source edits.", "A settings-rewrite guard rejected an incorrect substring-count implementation after junction activation. The setting was then updated from a separately patched, JSON-validated copy; model files and junctions were unaffected." ] }, { "title": "Runtime and model-discovery validation", "status": "Completed", "details": [ "LM Studio indexed four local language models and one embedding model from the E: managed library, including the expected Gemma 4 12B QAT entry.", "LM Studio reported no loaded model before the real-load test.", "Hiro's pinned Qwen 9B and Gemma 4 paths both resolved through the legacy compatibility junction with their expected 7.134 GiB and 6.497 GiB sizes.", "Gemma 4 12B QAT loaded from E: with maximum GPU offload, a 4,096-token context, one parallel slot, and the expected identifier. The load completed in 3 minutes 13.95 seconds.", "During the loaded-state check, total GPU memory use was 12,844 MiB including Windows display and runtime allocations.", "A local chat-completions warmup returned READY in 0.363 seconds. Gemma then unloaded successfully, LM Studio reported an empty loaded-model list, and the temporary API server was stopped." ] }, { "title": "System-volume cleanup and final state", "status": "Completed", "details": [ "Preserved the original directories under reversible backup names until discovery, path resolution, real model load, inference, and unload had all passed.", "Validated that the original path objects were junctions to the exact E: destinations and that the backup objects were normal directories within the intended user profile.", "Removed only the two verified model-library backups after all runtime gates passed.", "Measured C: free space immediately before and after cleanup: 20.83 GiB before, 65.25 GiB after, for 44.42 GiB reclaimed.", "Left no model loaded and no LM Studio API server active." ] }, { "title": "Controlled E: versus D: Gemma benchmark", "status": "Completed", "details": [ "Mapped D: and E: to two separate healthy 6 TB WD Elements USB hard drives rather than two partitions on one device. D: had approximately 948 GiB free before the temporary copy.", "Copied the exact 6.497 GiB Gemma Q4_0 GGUF to a temporary D: benchmark directory with unbuffered I/O. Source and destination matched in byte length and SHA-256 before any LM Studio path changed.", "Read the complete GGUF twice per drive with Windows no-buffering and sequential-scan flags. E: measured 42.52 and 42.51 MiB/s; D: measured 104.56 and 103.76 MiB/s.", "Repeated the full LM Studio configuration on both drives: maximum GPU offload, 4,096-token context, one parallel slot, and the same model identifier. E: loaded in 199.73 seconds. A fresh, never-read D: copy loaded in 69.67 seconds.", "Excluded a preliminary D:-routed run because the preserved E: directory remained within LM Studio's scan tree and produced two matching candidates. The preserved directory was moved outside the scan tree before the accepted D: trial so exactly one model was discoverable.", "Measured a 5.25-second D: reload after the first D: load and classified it as warm-cache behavior rather than disk performance.", "The same three fixed READY prompts returned correctly on each drive. E: measured 0.253, 0.107, and 0.108 seconds; D: measured 0.252, 0.110, and 0.109 seconds.", "After the accepted D: load, its fresh source still matched the expected SHA-256. The original normal E: directory and LM Studio download setting were restored, 12.994 GiB of temporary D: copies was deleted, and final checks found no loaded model or running API server." ] }, { "title": "Permanent activation of D: for active models", "status": "Completed", "details": [ "Closed five LM Studio desktop processes after confirming that no model was loaded and no inference server was active, preventing library changes during migration.", "Inventoried 19.392 GiB across five managed LM Studio files and 31.516 GiB across thirteen legacy files. D: had 947.84 GiB free and both intended destination directories were absent.", "Copied the managed library in 465.93 seconds and the legacy library in 756.73 seconds with unbuffered, metadata-preserving transfers. The original E: directories and all live paths remained unchanged during copying.", "Hashed the full E: and D: estates concurrently. All eighteen relative paths, byte lengths, and SHA-256 values matched across 50.908 GiB with zero discrepancies; the integrity phase took 1,250.93 seconds.", "Renamed the original E: directories to reversible backup names, created E: compatibility junctions targeting D:, and repointed the two existing C: junctions directly to D: rather than chaining through E:.", "Patched and JSON-validated a copy of LM Studio's settings, preserved a pre-D: settings backup, and changed downloadsFolder to D:\\AI\\Models\\LMStudio.", "LM Studio discovered four language models and one embedding model from D:. Hiro's pinned Qwen 9B and Gemma paths resolved through the direct C: legacy junction with their expected byte sizes.", "Gemma loaded from the single D: candidate in 69.76 seconds with maximum GPU offload, a 4,096-token context, and one parallel slot. Three fixed prompts returned READY in 0.319, 0.110, and 0.095 seconds.", "Gemma unloaded and the API server stopped. After rechecking direct junction targets, D: file counts, and LM Studio's setting, removed the two original E: backup directories and measured 50.91 GiB reclaimed on E:.", "Final state: D: has 896.94 GiB free, E: has 3,391.19 GiB free, no model is loaded, and the LM Studio API server is stopped." ] }, { "title": "Qwen3.6-35B-A3B acquisition and integrity validation", "status": "Completed", "details": [ "Verified the official Qwen/Qwen3.6-35B-A3B base model and reviewed current GGUF options before downloading. Selected bartowski/Qwen_Qwen3.6-35B-A3B-GGUF because its single-file Q4_K_M is documented as the recommended default balance for most use cases.", "Confirmed LM Studio's downloadsFolder was D:\\AI\\Models\\LMStudio, D: had 896.94 GiB free, no matching local Qwen3.6-35B-A3B file existed, no model was loaded, and the local API server was stopped.", "Downloaded the exact Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf artifact through LM Studio from Hugging Face. LM Studio automatically placed it in the managed bartowski repository directory on D: and also downloaded the associated F16 multimodal projector.", "The complete transfer, finalization, and indexing took 1,015.95 seconds. The finalized directory contains a 22,285,080,192-byte main GGUF and an 899,283,296-byte projector, with no partial-download file remaining.", "Computed SHA-256 across the complete main GGUF in 285.48 seconds. The local digest B46FEDD33E0BFB0CAE308AA3C158D0A4B2C4A1D2185A1ED6F093CDAF39064772 exactly matches the publisher's recorded digest.", "LM Studio indexes the combined model package at 23,184,363,488 bytes and identifies it as qwen35moe, Q4_K_M, vision capable, trained for tool use, and supporting up to 262,144 tokens.", "Ran LM Studio's estimate-only path with maximum GPU offload, a 4,096-token context, and one parallel slot. Estimated GPU and total memory are both 22.38 GiB, within the RTX 5090's 32 GB VRAM.", "D: has 875.34 GiB free after the acquisition. No model was loaded and no API server was started; actual cold-load, inference-speed, quality, tool-use, and vision tests remain separate work." ] }, { "title": "Qwen3.6 8K runtime and Hiro evaluation", "status": "Completed", "details": [ "Ran LM Studio's 8,192-token estimate with maximum GPU offload and one parallel slot. Estimated GPU and total memory were both 22.51 GiB.", "Loaded the verified Q4_K_M model under a dedicated benchmark identifier. The load took 399.48 seconds from D:, LM Studio reported 21.59 GiB, and observed total GPU allocation reached 29,996 MiB from a 6,307 MiB pre-load baseline.", "Started the local API only after loaded-state confirmation and used the exact loaded identifier for every inference request.", "Discovered during warmup that short requests could exhaust their entire visible-output allowance in separated reasoning_content. OpenAI-compatible chat-template parameters, a prompt control marker, and the native reasoning-off control did not disable reasoning for this loaded GGUF; the native endpoint reported that the model did not expose reasoning configuration.", "Executed all 120 deterministic bake-off cases with a 1,024-token ceiling so most tasks could reach visible output. The primary run completed with zero request errors, averaged 78.37%, and passed 94 cases at the established 70% threshold.", "The category results were 86.29% average for evidence obedience, 74.55% for no-evidence refusal, 84.43% for simulated routing, 97.98% for structured JSON, 46.97% for troubleshooting, and 55.00% for self-improvement specifications.", "A second full sweep completed and reproduced the same material instability: reasoning-heavy cases frequently consumed all 1,024 tokens before producing a visible answer.", "Retried one troubleshooting and one self-improvement failure at 4,096 tokens. Both produced substantively correct visible answers, but only after more than 1,100 hidden reasoning tokens and roughly nine seconds per request.", "Validated real tool calling: the model returned finish_reason tool_calls, chose get_weather, and emitted the argument Seattle, Washington in 1.21 seconds without fabricating a weather result.", "Validated 8K behavior using a 7,623-token synthetic ledger. The response used 7,817 total tokens, returned the exact final marker, and completed in 2.46 seconds.", "Measured 170.32 raw completion tokens per second on a sustained generation request, while documenting that 89.3% of the 2,048-token output budget was hidden reasoning and the visible response stopped early.", "Validated multimodal inference with a known synthetic card containing a red circle, blue square, green triangle, and HIRO 8K text. The model identified every expected element correctly in 3.10 seconds.", "Unloaded the model, stopped the API server, removed temporary local test fixtures, and verified that LM Studio reported no loaded model." ] }, { "title": "Local Evaluation Observatory restoration", "status": "Completed", "details": [ "Confirmed that port 8765 had no listener, directly explaining why the remembered dashboard URL would not open.", "Verified that api/server.py still exposes the public /benchmarks route and serves static/benchmarks.html, and that the read-only /evaluations/api endpoints remain registered.", "Verified that hiro/evals/evaluations.sqlite remained intact at approximately 1.5 MB, with its SQLite shared-memory files present and no evidence of ledger deletion.", "Confirmed that 8765 is not Hiro's committed application port. The normal configuration remains 8000 for HTTPS plus 8001 for HTTP; 8765 was a standalone local dashboard launch.", "Found that the Windows Python launcher no longer resolved an installed interpreter and that no existing Hiro virtual environment was available.", "Created an isolated environment at .runtime/dashboard-env using the bundled Python 3.12 interpreter and installed Hiro's exact pinned requirements. The .runtime tree is ignored and no source or tracked configuration file changed.", "Started uvicorn with api.server:app on 127.0.0.1:8765. Binding only to loopback preserves the local-only scope of the remembered URL.", "Validated the live dashboard HTML and overview API. The API reported 100 runs, all 100 completed, 101 variants, and a latest weighted score of 1.0 for the resume-live-soak entry.", "Opened the restored URL in the system browser and intentionally left the service running for ongoing benchmark tracking. No local language model or LM Studio API server was started." ] }, { "title": "Matched Qwen3.6 Evaluation Observatory ledger run", "status": "Completed", "details": [ "Selected the existing frozen emergence-level1-public@1.1.0 suite: eight public cases with two repetitions each, for sixteen observations covering rule induction, structural and relational transfer, composition, distractor resistance, exception transfer, and calibration.", "Used the same full Hiro run_turn agent path and 512-token local completion ceiling as the compatible Gemma evaluation-isolation baseline. A temporary untracked runner redirected both local routing tiers to the exact loaded qwen36-35b-a3b-8k-ledger identifier without changing Hiro source or persistent configuration.", "Cold-loaded Qwen from D: in 419.63 seconds with maximum GPU offload, 8,192 context tokens, and one parallel slot. LM Studio reported a 21.59 GiB model allocation and the loaded-state check reported 29,495 of 32,607 MiB total GPU memory in use.", "The API preflight confirmed that the response model exactly matched qwen36-35b-a3b-8k-ledger. Even the 32-token READY request returned empty visible content and 104 reasoning characters, reproducing the forced-reasoning behavior before the ledger run.", "Preserved qwen36-35b-a3b-8k-public-20260720-r1 as an invalid infrastructure record. Restricted execution could not write Hiro's existing conversation database, so all sixteen observations stopped before inference with database errors; the result was not used as model evidence.", "Completed qwen36-35b-a3b-8k-public-20260720-r2 under Hiro's normal host permissions. The append-only resume mechanism retained six valid observations after the supervising command stopped watching and completed the remaining ten without duplication.", "The valid run produced a 0.854167 weighted score, 0.375 strict pass rate, zero invariant failures, and 3,545.67 ms p95 latency. Six of sixteen observations passed every assertion.", "Compared against three matched Gemma 4 12B runs, whose means were 0.954861 weighted score, 0.729167 pass rate, zero invariant failures, and 379.91 ms p95. Qwen was 0.100694 lower in weighted score, 0.354167 lower in pass rate, and 9.33 times slower at p95.", "The strongest Qwen category was relational transfer at 1.0. The largest category deficit against the selected Gemma r3 baseline was distractor resistance at -0.25, followed by exception transfer at -0.1667 and rule induction at -0.125.", "Inspected failures and found compact-answer violations and apparent reasoning fragments such as incomplete analysis rather than runtime errors. The ledger comparison reported no invariant regression, overlapping pass-rate confidence intervals, and statistically_clear=false.", "Verified the live dashboard returned HTTP 200 and its compare endpoint reproduced the ledger metrics. Unloaded Qwen, stopped only LM Studio's port-8080 API, removed temporary runners, and reverified that the port-8765 dashboard stayed online." ] }, { "title": "Self-improvement autonomy boundary diagnosis", "status": "Completed", "details": [ "Inspected the current proposal-only self-improvement v2 scheduler, the legacy patch-and-promote cycle, the evaluation runner and deterministic scorer, append-only evaluation and variant ledgers, external held-out provider, worktree manager, proposal-only promotion gate, dashboard routes, and the associated evaluation and laboratory tests.", "Confirmed that the nightly default is self_improvement_v2 in proposal_only mode. It analyzes response envelopes, produces structured ImprovementSpecs and curiosity items, writes research output, and runs lightweight probes, but its proposals are never auto-applied.", "Confirmed that the newer variant laboratory can pin paired public and held-out baselines, create candidate worktrees outside the source repository, audit declared path scope, compare repeated metrics, reject invariant and category regressions, and produce immutable proposals. Its CLI still requires an operator to sequence those actions and contains no candidate editor, automatic evaluation orchestration, merge, deployment, or cleanup authority.", "Contrasted the laboratory with the legacy run_cycle path. The legacy path can generate model-authored patches and directly apply low-risk changes to the live checkout before evaluating them. Its evaluator computes a post-change score but does not execute a true isolated baseline-versus-candidate comparison, and its experiment-level rollback command can require manual restoration.", "Identified the required staged migration: unify observations into evidence-backed opportunities; convert ImprovementSpecs into falsifiable experiments with declared scopes; create candidates only in isolated worktrees; evaluate stable and candidate variants on identical public and external held-out suites; apply statistical, invariant, scope, cost, and reliability gates; package eligible results for human approval; and only later consider bounded automatic promotion for narrowly reversible change classes.", "No Hiro files, runtime configuration, evaluation records, services, or model state were changed during this diagnosis. No Hiro tests or model evaluations were run; conclusions are based on source, documentation, and existing test definitions." ] }, { "title": "Qwen35 quality-first output budget and perfect ledger rerun", "status": "Completed", "details": [ "Traced the failure mechanism to core/router.py: the ordinary chat, non-streaming local call, and streaming local call all defaulted to 512 completion tokens regardless of model capability.", "Added a persistent qwen35 model profile targeting the standardized qwen36-35b-a3b LM Studio identifier on port 8080. Added per-profile default_max_tokens metadata while preserving 512 for Gemma, Qwen 4B, and Qwen 9B.", "Added a resolver that expands only the ordinary 512-token default for a matching profile. Explicitly short budgets such as the 120-token grounded-tool summarizer remain unchanged, and unknown models retain the caller's requested value.", "Applied the resolved budget to both OpenAI-compatible non-streaming and streaming local payloads so API and interactive paths cannot diverge.", "Added focused tests for Qwen expansion, explicit-short preservation, smaller-profile stability, unknown-model stability, and the actual max_tokens value transmitted in the HTTP payload. Fourteen targeted router and classifier tests passed in 0.65 seconds; pytest emitted only a future fixture-scope deprecation warning.", "Cold-loaded the same verified Qwen3.6 Q4_K_M model from D: at 8,192 context tokens, one slot, and maximum GPU offload. This load took 618.23 seconds; LM Studio again reported 21.59 GiB and observed total GPU use was 29,496 of 32,607 MiB.", "Verified the exact qwen36-35b-a3b identifier through Hiro's real local-call path. At a 4,096 effective budget, the model returned visible READY in 1.41 seconds instead of exhausting the allowance in separated reasoning.", "Ran qwen36-35b-a3b-8k-public-4096-20260720-r3 through emergence-level1-public@1.1.0. It scored 0.979167, passed 15/16 observations, had zero invariant failures, and measured 15,269.71 ms p95. The score gain over r2 was 0.125 and statistically clear.", "The sole r3 failure was a second relation-class repetition that returned an incomplete reasoning fragment after 23.73 seconds. Retesting that exact full-agent prompt with 7,168-token headroom returned the required LABEL: GLIM in 7.63 seconds.", "Raised the qwen35 profile to 7,168 tokens, leaving a nominal 1,024-token reserve within the 8K runtime context, and reran all regression tests successfully.", "Ran qwen36-35b-a3b-8k-public-7168-20260720-r4. All sixteen observations passed, weighted score and pass rate were both 1.0, every category score was 1.0, and invariant failures remained zero.", "The perfect run's latency distribution was 1,720.59 ms minimum, 4,824.30 ms median, 9,825.35 ms mean, 36,770.73 ms p95, and 37,363.94 ms maximum. Headroom does not force every response to consume the full budget, but two distractor cases and one calibration case used long reasoning paths.", "Verified r4 as the latest live dashboard entry, unloaded Qwen, stopped LM Studio's port-8080 API, removed all one-off runners, and rechecked that the port-8765 dashboard still returned HTTP 200." ] }, { "title": "Incremental self-improvement transition workflow", "status": "Designed", "details": [ "Established an initial division of responsibility: Hiro owns evidence collection, clustering, prioritization, experiment specifications, routine evaluation execution, comparison, and reporting; ChatGPT initially owns code implementation, judgment on ambiguous results, and integration review; the user retains priority and promotion decisions.", "Defined a six-stage delegation ladder: structured nightly triage, experiment design, evaluation orchestration, isolated candidate construction, bounded repair and retesting, and finally a narrow automatic-promotion lane.", "Specified one canonical experiment state machine from observed through triaged, specified, approved, baseline_ready, candidate_ready, evaluated, review_ready, promoted or rejected, and monitored. Each transition must append evidence rather than overwrite earlier records.", "Defined the handoff packet Hiro should produce for every review: source evidence, failure cluster, falsifiable hypothesis, target metric, allowed paths, baseline and candidate manifests, test and evaluation results, category regressions, diff summary, runtime cost, recommendation, and rollback instructions.", "Prioritized evaluation orchestration as the first substantial offload because it is repetitive, deterministic, already supported by the evaluation kernel, and easier to verify than autonomous code editing.", "Kept practical minimum boundaries rather than treating security hardening as the primary blocker: candidate changes remain in an external worktree, evaluation assets remain outside candidate write scope, only one experiment advances at a time, and integration continues to require review until rollback and post-promotion monitoring are demonstrated.", "No Hiro source, configuration, runtime state, evaluation ledger, model process, or scheduled task was changed while designing this workflow." ] }, { "title": "Cycle coordinator v0 implementation", "status": "Completed", "details": [ "Saved the agreed six-stage transition plan as docs/HIRO_SELF_IMPROVEMENT_TRANSITION.md, including responsibility transfer, the canonical state machine, stage graduation criteria, daily cadence, transition metrics, and coordinator usage.", "Added hiro/improvement/cycle_coordinator.py with a CLI that loads exactly one ImprovementSpec from JSON, a v2 run summary, or JSONL and requires explicit selection when several specs are present.", "Added a normalized v0 handoff contract requiring an identifier, title, problem, falsifiable hypothesis, source evidence, proposed change, expected benefit, rollback plan, valid risk classification, and, for code changes, an allowed path plus at least one planned test.", "Added deterministic cycle identifiers derived from the normalized spec, public and held-out suite hashes, and the exact baseline manifest. Repeating identical inputs reuses completed runs or resumes interrupted runs instead of duplicating model work.", "Added append-only ImprovementCycle and cycle-event records to the variant laboratory ledger. A separate pre-candidate cycle record is used because the existing VariantExperiment contract correctly assumes a candidate worktree already exists.", "The coordinator builds or reuses an immutable baseline manifest, runs public and held-out suites through the existing EvaluationRunner, creates an immutable BaselinePin, and records specified, baseline_started, per-suite completion, review_ready, and failure events.", "Evidence output is written atomically beneath the ignored .hiro_runtime/cycle_coordinator directory. Packets contain the normalized spec, cycle definition, manifest, baseline pin, and compact public and held-out metrics, while deliberately declaring candidate_created=false and promotion_considered=false.", "No live model evaluation, production ledger mutation, candidate worktree, patch generation, promotion, service restart, or scheduler activation was performed during implementation." ] }, { "title": "Nightly synthetic evaluation harvest", "status": "Completed", "details": [ "Added hiro/improvement/nightly_evaluation.py as a bounded public-development evaluation stage using the existing JsonlCaseProvider, EvaluationRunner, deterministic scorer, variant manifest, and append-only VariantLabLedger.", "Configured the default harvest to use rsi_cycle_v1.jsonl: eight public cases across identity, reasoning, instruction following, epistemics, and communication, with two repetitions per case and a sixteen-observation hard cap.", "Derived daily run and variant identifiers from the local date, suite identifier and version, and evaluated code hash. Re-entering the loop on the same day and code reuses a completed report without new model calls; interrupted reports resume only missing repetitions.", "Captured the most recent prior completed run for the same suite and version so a failure proposal can include pass-rate trend evidence rather than treating each night as an isolated event.", "Aggregated deterministic failures by category and weighted score impact. Only the highest-impact cluster becomes an ImprovementSpec, with failed case identifiers, top scorer reasons, the ledger run identifier, a falsifiable diagnostic hypothesis, expected benefit, and rollback language.", "Validated generated failure proposals against the cycle coordinator handoff contract. The diagnostic proposal declares that it does not yet authorize a code change; candidate scope is intentionally deferred until Hiro diagnoses the shared cause.", "Integrated the harvest as Phase 0 of self_improvement_v2. Run summaries, Markdown reports, console output, and Telegram briefings now expose the evaluation run, execution mode, observation count, failure count, and pass rate.", "Counted newly executed evaluation observations against the existing nightly model-call budget and limited combined conversation-derived plus evaluation-derived proposals to the configured maximum.", "Added configurable suite, ledger, case, observation, and enablement settings in both source defaults and the machine-local ignored runtime configuration.", "Added an in-process scheduler guard so the 22:00 two-minute polling window cannot overlap the self-improvement job or rerun it after a successful same-day completion. Failed launches remain eligible for a retry." ] }, { "title": "Independent nightly API startup and Observatory results tab", "status": "Completed", "details": [ "Added scripts/nightly_api_preflight.ps1 with idempotent readiness checks against the real /benchmarks page on ports 8001 and 8765. It does not treat a disconnected model as an API startup failure, because the API can be ready while its /health endpoint correctly reports degraded model state.", "Added scripts/run_hiro_api.cmd as a blocking wrapper around the pinned .runtime/dashboard-env Python and main.py. The wrapper keeps Task Scheduler attached to the actual API lifetime even though the Windows virtual-environment launcher hands execution to a base Python process.", "Normalized the inherited Windows process Path before Start-Process and enabled UTF-8 Python output. This handles launch environments that expose both Path and PATH without discarding other runtime variables Hiro needs.", "Added scripts/register_nightly_hiro_task.ps1. The task runs daily at 21:55, wakes the computer, starts when a trigger was missed, permits battery execution, ignores overlapping instances, retries failure three times at five-minute intervals, and has a ten-hour execution window.", "Registered the task as Hiro Nightly API Preflight for the current interactive Windows identity. Its stored action points to the checked-in preflight with -KeepAlive, and its verified next trigger is 2026-07-20 21:55:00 America/Los_Angeles.", "Extended main.py to serve the Evaluation Observatory on 127.0.0.1:8765 in the same process as ports 8000 and 8001. The existing application-level startup lock ensures the scheduler and monitor background loops are started only once across the three Uvicorn server threads.", "Added GET /evaluations/api/nightly. It filters completed nightly runs, loads full append-only reports, summarizes failed observations, case IDs, and categories, and deterministically reconstructs the same coordinator-compatible improvement signal produced by the nightly harvest.", "Added a Nightly Observatory tab with latest-run metric cards, a diagnostic proposal panel, and clickable longitudinal run history. Empty-state text explains that the enabled 22:00 cycle will supply the first run and does not suggest a manual command.", "Restarted Hiro through the registered task path and left that task actively keeping the API online. Live checks returned HTTP 200 from the nightly endpoint on both 8001 and 8765.", "Used the in-app browser to open the real loopback Observatory, select the Nightly tab, inspect its rendered empty state, and visually verify the complete desktop layout. The tab was left available for the user." ] }, { "title": "Nightly readiness monitoring and expansion criteria", "status": "Completed", "details": [ "Created an active daily Codex automation named Hiro nightly readiness review, scheduled for 07:30 America/Los_Angeles so the normal 22:00 run and its documented timeout envelope have time to finish.", "The review is read-only with respect to Hiro. It may inspect the Nightly Observatory data, append-only evaluation ledger, run summaries, and operational logs, but it may not alter code, configuration, services, schedules, cases, or recorded results.", "Each review classifies the latest night as a valid completed evaluation, infrastructure or model failure, incomplete run, or no run. Infrastructure failures are explicitly excluded from the successful-run count.", "Tracked evidence includes run identity, execution mode, duration when available, observation count, pass rate, failure categories, regression-probe status, API and model availability, errors, and whether Hiro's proposal is evidence-based and appropriately scoped.", "The monitor compares valid nights and recommends changes only after at least three clean completed runs, preferably five, with reliable completion, no recurring infrastructure failures, useful stable metrics, and sensible proposals.", "When expansion is justified, the policy prioritizes broader test diversity and adaptive focus on weak categories before adding repetitions, and may recommend a larger weekly deep run. It never applies recommendations without user approval." ] } ], "decisions": [ "Use two explicit E: destinations instead of merging unrelated directory layouts. LM Studio-managed models and Hiro's legacy files retain their existing organization and identifiers.", "Keep compatibility junctions even after updating LM Studio's direct download setting. Hiro contains pinned legacy paths, and junctions preserve behavior without coupling this operational migration to application-source edits.", "Copy and hash before activation rather than performing an unverified cross-volume move. Cross-volume failure must not leave Hiro with a partially populated live library.", "Keep C: backups until a real model loads and answers from E:. File equality proves copy integrity, while runtime validation proves the inference stack can consume the relocated model.", "Remove the large backups only after the real-load gate passes. The measured free-space result, not logical file totals, is the authoritative reclaimed-capacity value.", "Treat E: cold-load speed as a material limitation. The volume offers capacity, but the 3-minute-plus Gemma load means larger candidate startup times must be measured before operational adoption.", "Do not claim Hiro's launcher validation is sufficient when only the model-discovery endpoint responds. LM Studio can list installed models while none are loaded, so readiness must include loaded-state or completion evidence.", "Use direct uncached reads of the exact GGUF in addition to application load timing. This separates storage delivery from LM Studio initialization and Windows file-cache effects.", "Reject ambiguous application trials. A D:-routed load was not accepted until the preserved E: copy was outside LM Studio's scan tree and discovery returned exactly one Gemma candidate.", "Treat D: as the preferred hot-model volume based on the measured 2.45-times read advantage and 65.1% cold-load reduction. E: remains useful as higher-free-space bulk storage.", "Restore the permanent E: configuration after benchmarking because the request authorized measurement, not a second storage migration. Moving the active library to D: should be a separate intentional change.", "After receiving explicit authorization, move both the managed and legacy active libraries to D: rather than moving only Gemma. This makes performance consistent for every currently installed local model and consumes only about 51 GiB of D:'s available capacity.", "Keep separate D: directories for LM Studio-managed and legacy llama.cpp assets. Their layouts, identifiers, and consumers differ, and merging them would add operational risk without improving speed.", "Point the C: compatibility junctions directly to D: and retain E: junctions to D:. Direct C: links avoid an unnecessary hop, while E: links preserve scripts or tools that adopted the intermediate E: paths.", "Keep a small pre-D: LM Studio settings backup but remove the large E: model backups only after discovery, pinned-path checks, real loading, inference, unload, server shutdown, and final junction validation pass.", "Do not modify Hiro source paths during the storage move. Compatibility junctions preserve behavior while the model-path centralization work remains a separate, reviewable task.", "Use the base Qwen3.6-35B-A3B rather than an abliterated or instruction-altering derivative for the first trial. This keeps the evaluation anchored to the Qwen team's published model behavior.", "Select bartowski's Q4_K_M conversion because it is a standard, single-file, recommended default quant with a published SHA-256 and enough quality headroom while fitting the RTX 5090 with a practical context.", "Let LM Studio manage both the main GGUF and vision projector so discovery, multimodal pairing, identifiers, and future library maintenance remain native to the runtime.", "Download and verify now, but do not silently make the model Hiro's default. Resource estimation establishes feasibility; a controlled runtime and task-quality evaluation must precede routing changes.", "Evaluate Qwen3.6 at 8K rather than the earlier 4K estimate because 8K is the intended first operational context for Hiro and still fits the 32 GB GPU at one slot.", "Raise the suite ceiling from the historical 220 tokens to 1,024 for this checkpoint because separated hidden reasoning otherwise prevents many responses from reaching visible output. Record this as a comparability caveat rather than silently treating starved output as model quality.", "Do not compensate for the current reasoning configuration by making 4,096 output tokens Hiro's default. Representative tasks become answerable, but the latency and hidden-token overhead are excessive for routine compact responses.", "Keep Gemma 4 and Qwen 9B as the operational baselines until Qwen3.6 can run with a reliable reasoning-off chat template or runtime control and then pass the same compact-budget benchmark.", "Treat raw completion throughput and useful visible throughput as separate metrics for reasoning models. The measured 170.32 tokens per second is hardware throughput, not equivalent to 170 visible answer tokens per second.", "Restore the exact 127.0.0.1:8765 dashboard address as a standalone loopback service rather than changing Hiro's committed 8000/8001 application ports.", "Reconstruct the missing Python runtime under the already ignored .runtime directory. This restores execution without mixing generated environment files into the dirty source worktree.", "Use Hiro's existing FastAPI dashboard and evaluation-ledger API instead of creating a replacement static report, preserving all run, comparison, and lineage functionality.", "Leave the dashboard process running because the user requested an active tracking page; keep it independent from LM Studio so viewing results does not consume model VRAM.", "Use the existing Gemma 4 12B evaluation-isolation runs as the matched ledger baseline because they use the same suite and full Hiro agent path. The earlier Qwen 9B score remains useful broader context but is not represented by a directly compatible run in this ledger slice.", "Keep the infrastructure-only r1 record rather than deleting or rewriting append-only evidence. Exclude it explicitly from model comparison and use the completed r2 run as the valid candidate.", "Do not promote Qwen3.6 from this ledger result. It is materially worse on both compact-answer quality and latency under Hiro's current 512-token agent allowance, even though the small public suite does not make the score difference statistically clear.", "Treat reasoning control as the blocking experiment. A future rerun should keep the same frozen suite and ledger path while changing only a verified reasoning-off runtime or template, so any improvement is attributable.", "Treat self-improvement autonomy as an orchestration migration, not a transfer of unlimited authority. Hiro can own the repetitive research loop while protected evaluation assets and high-impact decisions remain independently governed.", "Retire or quarantine the legacy live-checkout mutation path before enabling autonomous candidate construction. Two competing pipelines with different safety and evidence semantics would make audit results ambiguous.", "Require candidate generation and execution to occur in disposable, resource-limited worktrees or equivalent sandboxes. Stable Hiro must supervise candidates; candidates must not supervise themselves or share writable evaluator state.", "Keep held-out cases, evaluator definitions, safety policy, permission boundaries, deployment credentials, and promotion policy outside candidate write scope. Automatic benchmark generation may add training and public-development cases, but it cannot manufacture its own promotion evidence.", "Use human approval as a permanent authority boundary for evaluator, safety, permissions, dependencies, schema migrations, deployment, and other high-blast-radius changes. A future automatic lane should be limited to predefined reversible scopes with proven rollback and health checks.", "Use a profile-aware budget instead of globally deleting completion limits. The stronger GPU makes Qwen's long reasoning affordable, but smaller models and intentionally compact tool summaries still benefit from bounded outputs.", "Set qwen35's 8K quality-first budget to 7,168 tokens after 4,096 passed 15/16 and an isolated 7,168-token retry fixed the remaining failure. This preserves a nominal prompt reserve while allowing the forced-reasoning GGUF to reach visible output.", "Treat the perfect r4 result as proof that output starvation, not core task capability, caused the earlier failure. Keep latency as a separate promotion dimension because the 36.77-second p95 is operationally significant.", "Do not silently change Hiro's active default profile as part of output tuning. The qwen35 profile is now supported and validated, while the final routing decision should explicitly balance perfect suite quality against cold-load and tail-latency costs.", "Transfer self-improvement responsibilities by task class, beginning with repetitive evidence handling and evaluation work rather than handing the complete cycle to Hiro at once.", "Make evaluation orchestration the first major autonomous responsibility after nightly triage because its outputs are structured and independently checkable.", "Limit each early cycle to one primary opportunity and one candidate so failures remain attributable and review packets stay compact.", "Keep ChatGPT as implementation fallback and reviewer during the first stages, then reduce its role after Hiro repeatedly produces complete experiment records and reproducible results.", "Use measured graduation criteria for each stage: complete artifacts, reproducible reruns, low duplicate or invalid proposal rates, zero worktree-scope violations, and successful rollback drills before authority expands.", "Treat automatic promotion as the final transition stage and initially restrict it to a small allowlist of reversible prompt, test, benchmark, or non-operational configuration changes.", "Use a distinct append-only ImprovementCycle record before candidate creation rather than fabricating a VariantExperiment with a nonexistent branch or worktree.", "Make cycle identity deterministic across the spec, suite contents, and baseline manifest so retrying the same work is idempotent and changing evidence creates a new cycle.", "Reject reuse of a baseline variant identifier when its immutable code, prompt, configuration, tool, model, or Git identity no longer matches; the operator must choose a new baseline identifier.", "Keep coordinator runtime packets outside version control under .hiro_runtime and write them atomically so an interrupted report write cannot leave a valid-looking partial packet.", "Stop coordinator v0 at baseline_ready. Candidate construction, candidate evaluation, promotion recommendation, merge, and deployment remain later transition stages.", "Summarize held-out evaluation outcomes in the evidence packet without copying held-out cases or assertions into the coordinator output.", "Use a synthetic public development suite nightly because it creates controlled, repeatable usage without consuming the held-out evidence needed to judge future candidates.", "Cap the first nightly harvest at eight cases and sixteen observations. This supplies broad evidence while reserving the remainder of the sixty-call self-improvement budget for conversation analysis, curiosity, journaling, and legacy probes.", "Use deterministic assertions rather than a model judge for nightly scoring. Hiro may generate the answers, but it does not get to reinterpret whether required formats, boundaries, tool restrictions, or content assertions passed.", "Create an improvement proposal only when deterministic failures exist. A perfect nightly run remains useful trend and stability evidence but does not create synthetic busywork.", "Generate a diagnostic, non-code-changing ImprovementSpec from the highest-impact cluster. The next transition stage can turn that diagnosis into allowed files and candidate tests after the shared cause is understood.", "Keep the existing lightweight probes temporarily as a separate compatibility signal. They are not treated as durable improvement evidence because their current integration still reports only a probe count.", "Run API preflight at 21:55 rather than replacing the 22:00 self-improvement scheduler. This separates process availability from evaluation orchestration and preserves the already-tested nightly path.", "Use the public /benchmarks page as readiness evidence. The existing /health endpoint includes model connectivity and may return 503 even when the API and scheduler are fully operational.", "Keep the API process attached to the scheduled task through the overnight window. A short-lived fire-and-forget launcher proved unreliable under Windows process-parent behavior.", "Serve the stable 8765 Observatory address from main.py instead of maintaining a second dashboard process. One process and one application startup lock avoid duplicate scheduler instances.", "Build the Nightly tab from the append-only evaluation ledger rather than scraping generated Markdown reports. Ledger data has explicit schemas, immutable run identity, exact case results, and durable historical comparisons.", "Keep the Nightly API and UI read-only. They expose public-development evidence and diagnostic proposals but do not add candidate creation, held-out access, promotion, or mutation controls.", "Use completed valid nights, not elapsed calendar nights, as the expansion gate. Model outages, API failures, interrupted cycles, and missing runs are operational evidence but do not establish evaluation stability.", "Require at least three valid completed nights and prefer five before changing workload size. This is long enough to expose basic scheduling, persistence, runtime, and proposal-quality problems without delaying small, reversible improvements indefinitely.", "Expand coverage before repetition count. New capability categories, multi-turn behavior, tool use, memory, and recovery scenarios provide more improvement information than repeatedly sampling the same deterministic prompts.", "Keep readiness review advisory. The monitor may recommend a bounded configuration or suite change, but code, schedule, evaluation, and authority changes still require explicit user approval." ], "validation": [ { "check": "Pre-migration process and port gate", "status": "passed", "result": "No relevant model process or inference listener was active before copy." }, { "check": "LM Studio library copy", "status": "passed", "result": "Five files totaling 19.392 GiB matched by relative path, byte count, and SHA-256." }, { "check": "Legacy llama.cpp library copy", "status": "passed", "result": "Thirteen files totaling 31.516 GiB matched by relative path, byte count, and SHA-256." }, { "check": "Junction targets", "status": "passed", "result": "Both legacy C: paths resolve to their exact E: destinations." }, { "check": "LM Studio setting", "status": "passed", "result": "downloadsFolder is E:\\AI\\Models\\LMStudio and the rewritten settings file parses as valid JSON." }, { "check": "LM Studio discovery", "status": "passed", "result": "Four language models and one embedding model were indexed from the relocated managed library." }, { "check": "Hiro pinned paths", "status": "passed", "result": "The Qwen 9B and Gemma 4 model_registry paths resolve through the compatibility junction." }, { "check": "Real Gemma load", "status": "passed", "result": "The 6.50 GiB Gemma model loaded from E: in 3 minutes 13.95 seconds with 4K context, maximum GPU offload, and one slot." }, { "check": "Warmup completion", "status": "passed", "result": "The local OpenAI-compatible endpoint returned READY in 0.363 seconds from the relocated model." }, { "check": "Unload and server restoration", "status": "passed", "result": "Gemma unloaded, the loaded-model list became empty, and the temporary API server stopped." }, { "check": "C: space recovery", "status": "passed", "result": "Measured free space increased by 44.42 GiB, from 20.83 GiB to 65.25 GiB." }, { "check": "Journal production build", "status": "passed", "result": "npm run build generated six dated journal pages, passed schema, sitemap, Atom, IndexNow, and noindex validation, compiled TypeScript, and completed the Vite production build." }, { "check": "Benchmark file identity", "status": "passed", "result": "The E: source, verified D: copy, and fresh D: cold-load copy were each 6,975,877,728 bytes and matched SHA-256 after testing." }, { "check": "Uncached sequential reads", "status": "passed", "result": "E: averaged 42.52 MiB/s across two trials; D: averaged 104.16 MiB/s across two trials, a 2.45-times advantage." }, { "check": "Cold LM Studio loads", "status": "passed", "result": "E: loaded in 199.73 seconds and the fresh-file D: trial loaded in 69.67 seconds with identical 4K-context, full-GPU, one-slot settings." }, { "check": "Warm-cache classification", "status": "passed", "result": "A subsequent D: reload completed in 5.25 seconds and was recorded separately rather than misreported as cold storage performance." }, { "check": "Cross-drive inference parity", "status": "passed", "result": "Both drives returned READY for all three prompts; first-response latency was 0.253 seconds on E: and 0.252 seconds on D:, with warm calls near 0.11 seconds." }, { "check": "Benchmark rollback and cleanup", "status": "passed", "result": "The original normal E: model directory and download setting were restored, two temporary D: directories totaling 12.994 GiB were removed, and LM Studio ended unloaded and stopped." }, { "check": "Journal rebuild after benchmark update", "status": "passed", "result": "The updated entry regenerated six dated pages, passed schema, sitemap, Atom, IndexNow, and noindex validation, compiled TypeScript, and completed the Vite production build." }, { "check": "D: migration preflight", "status": "passed", "result": "D: was healthy with 947.84 GiB free, destinations were absent, source directories were normal, and LM Studio was closed with no loaded model or running server." }, { "check": "Complete library copy", "status": "passed", "result": "Five managed files totaling 19.392 GiB and thirteen legacy files totaling 31.516 GiB copied to D: without robocopy failures." }, { "check": "Full migration hashes", "status": "passed", "result": "All eighteen files and 50.908 GiB matched by relative path, byte length, and SHA-256; mismatch count was zero." }, { "check": "D: activation paths", "status": "passed", "result": "Both C: compatibility junctions point directly to D:, both former E: library paths junction to D:, and LM Studio's downloadsFolder is the direct D: managed-library path." }, { "check": "Post-activation discovery", "status": "passed", "result": "LM Studio discovered four language models and one embedding model, with exactly one Gemma candidate; Hiro's pinned Qwen and Gemma files resolved at expected sizes." }, { "check": "Post-activation Gemma runtime", "status": "passed", "result": "Gemma loaded from D: in 69.76 seconds with full GPU offload, 4K context, and one slot; all three fixed prompts returned READY." }, { "check": "Final unload and cleanup", "status": "passed", "result": "Gemma unloaded, the server stopped, both E: backup directories were removed, E: reclaimed 50.91 GiB, and no temporary migration artifacts remained." }, { "check": "Journal rebuild after D: activation", "status": "passed", "result": "The expanded entry regenerated six dated pages, passed schema, sitemap, Atom, IndexNow, and noindex validation, compiled TypeScript, and completed the Vite production build." }, { "check": "Qwen acquisition preflight", "status": "passed", "result": "The direct D: LM Studio destination had 896.94 GiB free, no duplicate model was present, no model was loaded, and the API server was stopped." }, { "check": "Qwen download finalization", "status": "passed", "result": "LM Studio completed the 23.18 GB package download and indexing with no remaining .part file; the main GGUF and F16 projector are present." }, { "check": "Publisher hash verification", "status": "passed", "result": "The 22,285,080,192-byte Q4_K_M GGUF exactly matched the publisher's SHA-256 B46FEDD33E0BFB0CA...39064772." }, { "check": "Qwen LM Studio discovery", "status": "passed", "result": "LM Studio identifies one 35B-A3B Q4_K_M qwen35moe model with vision, tool-use training, the paired projector, and a 262,144-token maximum context." }, { "check": "Qwen resource estimate", "status": "passed", "result": "Estimate-only at 4K context, one slot, and full GPU offload reports 22.38 GiB GPU and total memory, within the 32 GB RTX 5090." }, { "check": "Journal rebuild after Qwen acquisition", "status": "passed", "result": "The expanded entry regenerated six dated pages, passed schema, sitemap, Atom, IndexNow, and noindex validation, compiled TypeScript, and completed the Vite production build." }, { "check": "Qwen 8K resource estimate", "status": "passed", "result": "At 8,192 context tokens, one slot, and full GPU offload, LM Studio estimated 22.51 GiB GPU and total memory." }, { "check": "Qwen 8K real load", "status": "passed", "result": "The Q4_K_M model loaded in 399.48 seconds; LM Studio reported 21.59 GiB and observed total GPU use reached 29,996 of 32,607 MiB." }, { "check": "Qwen core suite execution", "status": "passed", "result": "All 120 deterministic cases completed with zero API errors. The primary run averaged 78.37% and passed 94/120 cases at the 70% threshold." }, { "check": "Reasoning budget diagnosis", "status": "passed", "result": "All 15 troubleshooting cases failed at 1,024 tokens while almost entirely consuming the budget as hidden reasoning; two representative retries reached correct visible answers at 4,096 tokens." }, { "check": "Qwen function calling", "status": "passed", "result": "The model selected get_weather with location Seattle, Washington and returned finish_reason tool_calls in 1.21 seconds." }, { "check": "Qwen near-8K retrieval", "status": "passed", "result": "A 7,623-token prompt and 7,817-token total request returned the exact final marker in 2.46 seconds." }, { "check": "Qwen sustained generation", "status": "passed", "result": "The model generated 2,048 completion tokens in 12.02 seconds, or 170.32 tokens per second; 89.3% were classified as reasoning tokens and visible output was truncated." }, { "check": "Qwen vision inference", "status": "passed", "result": "The model correctly identified a red circle, blue square, green triangle, three-shape count, and HIRO 8K text in 3.10 seconds." }, { "check": "Qwen benchmark restoration", "status": "passed", "result": "The model unloaded, the API server stopped, temporary test fixtures were removed, and GPU use returned to 6,482 MiB." }, { "check": "Journal rebuild after Qwen 8K evaluation", "status": "passed", "result": "The expanded entry regenerated the dated journal pages, passed content and noindex validation, compiled TypeScript, and completed the Vite production build." }, { "check": "Dashboard route and ledger presence", "status": "passed", "result": "The /benchmarks route and static page remain in source, and the approximately 1.5 MB evaluation ledger remains present." }, { "check": "Dashboard runtime reconstruction", "status": "passed", "result": "An ignored Python 3.12 environment was created under .runtime and Hiro's pinned requirements installed successfully." }, { "check": "Loopback dashboard listener", "status": "passed", "result": "The existing FastAPI application is listening only on 127.0.0.1:8765 and serves /benchmarks with HTTP 200." }, { "check": "Live Evaluation Observatory data", "status": "passed", "result": "The overview API returned 100 completed runs, 101 variants, and the latest completed run at a 1.0 weighted score." }, { "check": "Dashboard restoration journal build", "status": "passed", "result": "The expanded entry regenerated the dated journal pages, passed content and noindex validation, compiled TypeScript, and completed the Vite production build." }, { "check": "Qwen ledger configuration", "status": "passed", "result": "LM Studio confirmed Q4_K_M, 8,192 context tokens, one slot, maximum GPU offload, the exact benchmark identifier, and a 21.59 GiB model allocation." }, { "check": "Ledger model-identity preflight", "status": "passed", "result": "The API response model exactly matched qwen36-35b-a3b-8k-ledger; the short probe also reproduced empty visible content with separated reasoning." }, { "check": "Invalid-run separation", "status": "passed", "result": "The r1 database-permission failure was identified as pre-inference infrastructure evidence and excluded from model comparison; it remains preserved for auditability." }, { "check": "Valid Qwen ledger run", "status": "passed", "result": "Run qwen36-35b-a3b-8k-public-20260720-r2 completed all 16 observations at 0.854167 weighted score, 0.375 pass rate, zero invariant failures, and 3,545.67 ms p95." }, { "check": "Matched Gemma comparison", "status": "passed", "result": "Three compatible Gemma runs averaged 0.954861 weighted score, 0.729167 pass rate, and 379.91 ms p95; Qwen was 10.07 score points and 35.42 pass-rate points lower and 9.33 times slower." }, { "check": "Evaluation Observatory ingestion", "status": "passed", "result": "The loopback dashboard returned HTTP 200, reported r2 as latest, and its compare endpoint reproduced the ledger result; the overview contained 130 completed runs and 103 variants." }, { "check": "Post-ledger runtime cleanup", "status": "passed", "result": "Qwen unloaded, LM Studio's API stopped, temporary runners were removed, and the Evaluation Observatory remained online on port 8765." }, { "check": "Journal rebuild after matched ledger evaluation", "status": "passed", "result": "The final expanded entry passed content and noindex validation, regenerated the dated journal pages, compiled TypeScript, and completed the Vite production build." }, { "check": "Self-improvement autonomy source diagnosis", "status": "passed", "result": "Relevant orchestration, proposal, mutation, evaluation, isolation, ledger, promotion, rollback, scheduler, dashboard, documentation, and test surfaces were inspected without modifying or executing Hiro." }, { "check": "Hiro implementation and evaluation tests for this diagnosis", "status": "not run", "result": "This session was explicitly read-only for Hiro. No pytest suite, model inference, evaluation run, candidate worktree, or improvement cycle was executed." }, { "check": "Journal build after self-improvement autonomy diagnosis", "status": "passed", "result": "The updated entry generated six dated journal pages, passed schema, sitemap, Atom, IndexNow, and noindex validation, compiled TypeScript, and completed the Vite production build." }, { "check": "Profile-aware budget unit tests", "status": "passed", "result": "Fourteen targeted tests passed, covering Qwen35 expansion to 7,168, explicit-short preservation, smaller and unknown model behavior, actual API payload propagation, and existing classifier behavior." }, { "check": "Qwen35 live router preflight", "status": "passed", "result": "Hiro resolved the profile to qwen36-35b-a3b with a 4,096-token interim budget and returned visible READY in 1.41 seconds." }, { "check": "Qwen35 4,096-token ledger run", "status": "passed", "result": "r3 completed all 16 observations at 0.979167 weighted score, 0.9375 pass rate, zero invariant failures, and 15,269.71 ms p95; only one observation failed." }, { "check": "Remaining-case 7,168-token probe", "status": "passed", "result": "The exact r3 relation-class failure returned LABEL: GLIM through the full Hiro agent in 7.63 seconds with 7,168-token headroom." }, { "check": "Qwen35 7,168-token ledger run", "status": "passed", "result": "r4 completed all 16 observations with 1.0 weighted score, 1.0 pass rate, every category at 1.0, zero invariant failures, and 36,770.73 ms p95." }, { "check": "Qwen35 improvement comparison", "status": "passed", "result": "r4 improved 0.145833 in weighted score and 0.625 in pass rate over starved r2, with non-overlapping pass intervals and statistically_clear=true; it also exceeded the three-run Gemma mean by 0.045139." }, { "check": "Quality-first latency characterization", "status": "passed", "result": "r4 measured 4,824.30 ms median, 9,825.35 ms mean, 36,770.73 ms p95, and 37,363.94 ms maximum latency; the p95 was 96.8 times the matched Gemma mean." }, { "check": "Post-tuning runtime cleanup", "status": "passed", "result": "Qwen unloaded, LM Studio's API stopped, one-off runners were removed, and the loopback Evaluation Observatory remained online with r4 as latest." }, { "check": "Journal rebuild after Qwen output tuning", "status": "passed", "result": "The final expanded entry passed content and noindex validation, regenerated the dated journal pages, compiled TypeScript, and completed the Vite production build." }, { "check": "Incremental transition workflow design", "status": "passed", "result": "The workflow was mapped to Hiro's existing proposal-only v2 loop, evaluation kernel, variant laboratory, worktree isolation, append-only ledgers, proposal gate, and current human-review boundary." }, { "check": "Hiro execution for transition workflow design", "status": "not run", "result": "This was a design-only session. No Hiro tests, evaluations, candidates, model calls, scheduler changes, or improvement cycles were executed." }, { "check": "Journal build after transition workflow design", "status": "passed", "result": "The expanded entry generated seven journal pages and timestamped aliases, passed timestamp schema, same-day identity, sitemap, Atom, noindex, and IndexNow validation, compiled TypeScript, and completed the Vite production build." }, { "check": "Cycle coordinator Python compilation", "status": "passed", "result": "py_compile completed for cycle_coordinator.py, lab_schema.py, lab_ledger.py, and test_cycle_coordinator.py." }, { "check": "Cycle coordinator CLI surface", "status": "passed", "result": "python -m hiro.improvement.cycle_coordinator --help loaded successfully and documented spec selection, public and held-out suites, immutable baseline identity, ledger, output, and repository options." }, { "check": "Initial focused pytest attempt", "status": "infrastructure failure", "result": "Collection succeeded and nine non-temp tests passed, but sixteen tests using tmp_path could not create pytest's default user-profile temporary directory. No Hiro assertion failure was reported in this attempt." }, { "check": "Focused coordinator and evaluation tests", "status": "passed", "result": "With --basetemp redirected to .hiro_runtime/pytest-cycle-coordinator, 25 tests passed in 1.43 seconds across test_cycle_coordinator.py, test_variant_laboratory.py, test_variant_lab_cli.py, and test_evaluation_kernel.py." }, { "check": "Journal build after cycle coordinator implementation", "status": "passed", "result": "The expanded entry generated seven journal pages and timestamped aliases, passed timestamp schema, same-day identity, sitemap, Atom, noindex, and IndexNow validation, compiled TypeScript, and completed the Vite production build." }, "Validated hiro/evals/public_cases/rsi_cycle_v1.jsonl through the evaluation CLI: valid=true, case_count=8, suite rsi-cycle-public@1.0.0.", "Added three nightly-harvest tests covering append-only failure evidence and coordinator-compatible proposal output, zero-call reuse of a completed same-day run, previous-run lookup, and observation-budget behavior. All three passed.", "Added two scheduler tests proving a successful nightly job runs only once per day within the process and an overlapping launch is rejected. Both passed.", "Ran the combined nightly, evaluation-kernel, variant-laboratory, cycle-coordinator, and scheduler selection: 28 tests passed in 1.93 seconds.", "Compiled the nightly evaluation module, self-improvement configuration and runner, scheduler, and new tests with py_compile successfully.", "Ran git diff --check across the nightly implementation and tests with no whitespace errors.", "Ran self_improvement_v2 in proposal-only dry-run mode with a one-minute ceiling and UTF-8 console mode. The nightly evaluation correctly reported skipped, the remaining dry-run phases completed, and zero model calls were made.", "The first dry-run command under the Windows CP1252 console failed before entering the loop because an existing emoji banner could not be encoded. Re-running with PYTHONUTF8=1 passed; this was a console-environment issue, not a nightly evaluation failure.", "Both PowerShell scripts parsed successfully and their -ValidateOnly modes returned the pinned runtime, entry point, runner, health endpoints, task name, 21:55 trigger, and KeepAlive action without changing external state.", "The first sandbox launch exposed duplicate Path and PATH entries that caused Start-Process to reject the inherited environment. The preflight now normalizes the process Path before launch.", "A trial using Start-Process -UseNewEnvironment removed runtime context needed by the bundled Python and failed during Windows networking initialization. The final implementation preserves the full environment after only normalizing Path.", "A real startup then showed that tracking the virtual-environment launcher PID was insufficient because it exited after handing execution to base Python. The blocking command wrapper corrected process ownership for Task Scheduler.", "Registered Hiro Nightly API Preflight successfully. Read-back confirmed state Ready before the manual verification run, a 2026-07-20T21:55:00-07:00 trigger, WakeToRun=true, StartWhenAvailable=true, a ten-hour limit, and the exact checked-in action.", "Started the registered task manually for end-to-end validation. It remained Running, the nightly endpoints on ports 8001 and 8765 returned HTTP 200, and ports 8000, 8001, and 8765 were all owned by the same Hiro Python PID.", "Ran 34 focused tests across startup validation, main server ports, Evaluation Observatory routes, nightly evaluation, duplicate scheduling, evaluation kernel, variant laboratory, and cycle coordinator. All 34 passed in 3.82 seconds.", "Compiled the changed Python modules and tests successfully, parsed both PowerShell scripts successfully, compiled the Observatory JavaScript with the Node function parser, and ran git diff --check with no whitespace errors.", "Browser validation confirmed a unique Nightly navigation tab, correct zero-run metric cards, the expected 22:00 empty-state explanation, no stray manual-command prompt, and a complete responsive desktop layout.", { "check": "Live pre-night readiness check", "status": "partial", "result": "At 18:56 America/Los_Angeles, the scheduled preflight was Running with its next trigger at 21:55, the persisted 22:00 self_improvement_v2 schedule was enabled, and the API and Observatory returned HTTP 200. Overall health returned HTTP 503 and the local model endpoint did not respond, so the system was not yet ready for a valid quality run." }, { "check": "Daily readiness-review automation", "status": "passed", "result": "The Codex app accepted and activated the Hiro nightly readiness review as a daily local-project automation at 07:30 America/Los_Angeles. An initial create request with an explicit UTC start anchor was rejected by automation validation; the successful definition uses a local wall-clock recurrence without that anchor." } ], "currentState": [ "LM Studio's direct download location and managed library are D:\\AI\\Models\\LMStudio.", "Hiro's active legacy GGUF library is D:\\AI\\Models\\llama.cpp.", "Both former C: model paths are direct compatibility junctions to D:, so existing Hiro configuration continues to resolve without source edits.", "Both former E: library paths remain as compatibility junctions to D: for tools that adopted the intermediate storage layout.", "C: has 65.25 GiB free after verified backup cleanup.", "No local model or LM Studio API server was left running.", "D: has 875.34 GiB free after receiving the active model estate and the Qwen3.6 package; E: has 3,391.19 GiB free after its verified backups were removed.", "D: is the measured performance winner for active models: 104.16 MiB/s uncached reads and a 69.67-second Gemma cold load, versus 42.52 MiB/s and 199.73 seconds on E:.", "Drive placement does not materially affect steady inference after the model is resident on the GPU.", "The launcher can confuse model discovery with loaded readiness when LM Studio's server lists installed models. A future readiness fix should require loaded-state or successful completion evidence.", "Qwen3.6-35B-A3B Q4_K_M and its F16 vision projector remain installed in LM Studio's D: library, hash verified, indexed, evaluated at 8K, and currently unloaded.", "The Qwen3.6 8K single-slot configuration fits the RTX 5090 but leaves limited VRAM headroom under the active Windows desktop workload: total observed use was 29,996 of 32,607 MiB.", "Qwen3.6 is not Hiro's default. Its current LM Studio chat template forces heavy hidden reasoning, producing a 78.37% primary core score and severe compact-output budget starvation despite fast raw generation.", "Qwen3.6 passed actual function calling, a 7,623-token context probe, and multimodal image understanding, so it remains a promising candidate once reasoning mode can be controlled reliably.", "The local Evaluation Observatory is available at http://127.0.0.1:8765/benchmarks and its loopback-only service is intentionally left running for benchmark tracking.", "The dashboard runtime lives under the ignored .runtime directory and does not change Hiro's configured 8000/8001 application ports or load an LLM.", "The valid Evaluation Observatory candidate is qwen36-35b-a3b-8k-public-20260720-r2: 0.854167 weighted score, 0.375 pass rate, zero invariant failures, and 3,545.67 ms p95 on emergence-level1-public@1.1.0.", "The matched three-run Gemma 4 12B mean remains substantially better at 0.954861 weighted score, 0.729167 pass rate, and 379.91 ms p95; Qwen3.6 remains unpromoted.", "The append-only ledger also contains an explicitly invalid r1 infrastructure run caused by restricted database permissions; it must not be interpreted as model quality.", "Hiro's scheduled self-improvement v2 loop is proposal-only: it can observe operational evidence and create structured improvement ideas, but it does not construct or promote candidates.", "The trusted evaluation and variant laboratory has the correct beginnings of an autonomy boundary, including external held-out cases, append-only evidence, worktree isolation, fingerprints, and proposal-only gates, but operator sequencing still supplies the orchestration now performed through ChatGPT-assisted development.", "The legacy improvement loop is not a safe shortcut to full autonomy because it mutates the live checkout before evaluation, lacks a rigorous isolated baseline comparison, and does not guarantee automatic restoration after a failed experiment.", "Hiro's router now includes a persistent qwen35 profile at qwen36-35b-a3b with a 7,168-token ordinary completion budget on both streaming and non-streaming paths.", "Gemma, Qwen 4B, Qwen 9B, explicitly short calls, and unknown models retain their previous output-budget behavior.", "The final r4 Qwen result is perfect on emergence-level1-public@1.1.0: 1.0 weighted score, 1.0 pass rate, every category at 1.0, and zero invariant failures.", "The quality-first Qwen profile has a substantial tail-latency cost: 9.83-second mean and 36.77-second p95 in r4, plus a measured 10-minute 18-second cold load in this session.", "The live dashboard exposes 132 completed evaluation runs across 105 variants and keeps qwen36-35b-a3b-8k-public-7168-20260720-r4 as its latest entry.", "The practical transition can begin without granting Hiro merge or deployment authority: its first responsibilities are producing a single ranked opportunity, a falsifiable experiment specification, and a reproducible evaluation packet.", "ChatGPT remains useful during the transition as a reviewer, difficult-change implementer, and diagnosis fallback rather than as the routine operator of every evaluation command.", "The desired end state is a Hiro-operated research loop with review-by-exception: routine cycles are collected, run, and summarized internally, while ambiguous, high-impact, or failing cycles are escalated.", "Cycle coordinator v0 now exists in Hiro but is not scheduled automatically. It is an explicit operator command that may invoke the configured model when given one accepted ImprovementSpec and matched evaluation suites.", "The coordinator's tested endpoint is baseline_ready: a durable cycle, resumable public and held-out runs, a baseline pin, and compact review artifacts, with no candidate or promotion action.", "The transition workflow is now preserved in the project documentation rather than only in conversation history.", "Hiro's persisted lab schedule is enabled, starts at 22:00 local time, and selects self_improvement_v2 in proposal_only mode.", "When the Hiro API process is running, its startup hook launches the lab scheduler and the next nightly job will execute the public rsi-cycle suite before conversation-derived analysis.", "The nightly public run writes full per-observation evidence to hiro/evals/evaluations.sqlite, while the self-improvement report includes a compact summary and any generated diagnostic proposal.", "No held-out evaluation, real model nightly run, candidate worktree, patch, promotion decision, or live runtime code change was executed during this implementation session.", "The older lightweight regression-probe phase still runs after the new harvest and still exposes only a count; it has not yet been migrated to the durable ledger.", "The Windows task Hiro Nightly API Preflight is registered for 21:55 local time and is currently Running after a successful manual end-to-end verification.", "One live Hiro process currently serves HTTPS on 8000, HTTP on 8001, and the loopback Evaluation Observatory on 8765. Both /evaluations/api/nightly endpoints return HTTP 200.", "The Observatory Nightly tab is live at http://127.0.0.1:8765/benchmarks and currently shows an intentional zero-run state pending the first 22:00 nightly evaluation.", "The task uses an Interactive logon principal. It can wake the computer and run while the user session is present, but it is not configured with stored credentials for operation after a full sign-out.", "API availability is now automatically handled before the nightly window; local model availability remains a separate dependency and is reported through evaluation failures or the existing model-health mechanisms.", "The active Hiro nightly readiness review runs daily at 07:30 America/Los_Angeles and reports whether the latest night was valid, operationally invalid, incomplete, or absent.", "No nightly evaluation has completed yet. The readiness counter therefore remains at zero valid completed nights, and no workload expansion is currently justified.", "The current nominal cycle targets 45 minutes with a 60-minute configured limit, but sequential evaluation and probe timeout blocks can allow a severely degraded run to extend to approximately 75 to 80 minutes.", "The local model endpoint was unavailable during the latest live check. Unless it is started before the cycle, tonight's data should be treated as infrastructure failure evidence rather than Hiro capability evidence." ], "nextSteps": [ "Use D: for new active model trials while reserving E: for archived models, datasets, and other capacity-oriented storage.", "Correct Hiro's Gemma launcher readiness check so an installed-but-unloaded model cannot satisfy the existing-server fast path.", "Record cold-load time, warm-load time, and steady-state inference separately for each model candidate.", "Keep the compatibility junctions until all Hiro model paths are intentionally centralized through configuration.", "Find a supported LM Studio runtime or model chat-template configuration that reliably disables Qwen3.6 reasoning for OpenAI-compatible requests, then repeat the 120-case suite at Hiro's historical 220-token ceiling.", "Update Hiro's benchmark reporting so visible answer tokens and hidden reasoning tokens are recorded separately for reasoning-capable checkpoints.", "If a reasoning-off rerun clears the compact-budget quality gate, add representative coding, repository-editing, and multi-turn tool tests before considering a routing change.", "Keep Qwen3.6 unloaded between experiments unless startup policy is intentionally changed; its measured D: load time was 6 minutes 39.48 seconds.", "If the Evaluation Observatory should survive reboots, add a reviewed launcher or scheduled startup entry for the loopback-only 8765 service instead of relying on an ad hoc process.", "Obtain a verified reasoning-off control for Qwen3.6, then rerun emergence-level1-public@1.1.0 through the same Evaluation Observatory path with the same 512-token agent ceiling and at least three repetitions.", "Add an explicit validity or exclusion annotation mechanism to the append-only ledger UI so infrastructure-only runs such as r1 remain auditable without appearing to be candidate model results.", "Use the Observatory for future model bake-offs whenever a frozen compatible suite and model-identity preflight are available; retain separate ad hoc probes for vision, tool calling, and long-context behavior until ledger suites cover them.", "Design one canonical self-improvement state machine that joins v2 observation and proposal generation to the isolated variant laboratory, and explicitly deprecate the legacy live-mutation path.", "Add a candidate-builder interface constrained by experiment-declared allowed paths, then run it inside the candidate worktree with fixed time, process, storage, model-call, and network budgets.", "Add an orchestrator that evaluates pinned stable and candidate manifests on the same public and external held-out suite versions, resumes interrupted runs, records infrastructure validity separately from model quality, and produces an independently verifiable evidence bundle.", "Before any automatic promotion lane, implement real transactional promotion and verified automatic rollback, post-deployment canaries, a kill switch outside Hiro's control, loop and budget limits, and durable alerts for stalled or repeatedly failing cycles.", "Measure evaluator reliability and contamination resistance over multiple generations before expanding authority. Start with Hiro generating proposals for review, then allow autonomous experiments, and only then consider automatic integration of a very small allowlisted class.", "Repeat the 7,168-token frozen-suite run at least twice more to measure stochastic stability and obtain a multi-run latency distribution before changing Hiro's default routing profile.", "Add visible-versus-reasoning token accounting to the Evaluation Observatory so output headroom and useful-answer efficiency can be compared directly across reasoning models.", "Evaluate a larger runtime context if routine Hiro prompts approach the nominal 1,024-token reserve left by the 7,168-token completion ceiling at 8K context.", "Decide whether qwen35 should be reserved for quality-critical requests or promoted more broadly after weighing its perfect suite score against 36.77-second p95 latency and long cold loads.", "Begin Stage 1 by standardizing the nightly output to one ranked opportunity plus a complete experiment specification and morning briefing; review ten cycles for usefulness, duplication, and evidence quality.", "In Stage 2, let Hiro pin or select a baseline, run matching public and held-out evaluations, resume interrupted runs, compare results, and produce the evidence packet without constructing a candidate.", "In Stage 3, connect approved experiment specifications to candidate worktree creation and allow Hiro to propose edits only inside declared paths, followed by syntax checks and targeted tests.", "In Stage 4, permit a bounded repair loop with a small attempt limit, then run the full stable-versus-candidate evaluation and prepare a promotion recommendation for review.", "After repeated successful reviewed promotions and rollback drills, define the first automatic-promotion allowlist and add post-promotion canary checks before expanding it.", "Track transition metrics separately from product-quality metrics: proposal acceptance rate, duplicate rate, invalid-run rate, human correction rate, time to evidence packet, scope violations, rerun reproducibility, and rollback success.", "Run coordinator v0 on a deliberately selected real ImprovementSpec and compatible public and external held-out suites when the configured local model is ready, using a new immutable baseline variant identifier.", "Review the first real evidence packet for completeness, useful failure classification, model and suite identity, and morning-briefing suitability before scheduling coordinator runs.", "Expose improvement-cycle status and evidence packet references in the read-only Evaluation Observatory after the first real coordinator cycle establishes the desired presentation.", "After repeated clean baseline packets, implement the approved-to-candidate transition that creates an external worktree and records a real VariantExperiment without adding merge authority.", "Allow the enabled 22:00 job to complete one real nightly public harvest and inspect its ledger report, runtime, pass-rate stability, and generated proposal quality.", "Add a morning analysis stage that reads the nightly report and enriches the top diagnostic signal into a falsifiable candidate specification with explicit allowed files and tests.", "Decide whether the legacy lightweight probes add distinct coverage; migrate useful cases into versioned suites and retire redundant count-only execution.", "Build suite rotation or staged expansion after several nights establish runtime and stability, while keeping a fixed anchor suite for longitudinal comparison.", "Add an operating-system wake or catch-up mechanism only if Hiro's API is not reliably running at 22:00; the current scheduler is process-resident and does not backfill a missed launch.", "Inspect the task state and first real Nightly tab entry after tonight's 22:00 evaluation, including whether the run was new, resumed, or reused.", "Confirm that the configured local model is available before 22:00 or add a separate bounded model preflight if unattended model startup is also desired.", "After the first run, verify that clicking its Nightly history row opens the expected per-case responses and deterministic scorer failures.", "Consider switching the Windows task to a non-interactive service principal only if overnight execution must continue after the user fully signs out.", "Add explicit startup-attempt and scheduler-launch events to the Observatory after several nights establish which operational details are useful rather than noisy.", "Let the morning readiness review classify and compare at least three valid completed runs, preferably five, before recommending any nightly workload change.", "When the valid-run gate is met, broaden the public suite with new capability categories before increasing repetitions, while retaining fixed anchor cases for longitudinal comparison.", "After nightly reliability is established, consider a separate weekly deep evaluation of approximately 100 to 200 observations rather than stretching every nightly run into a multi-hour job.", "Close the unattended local-model startup gap so the readiness monitor can distinguish real Hiro quality measurements from repeated connection failures." ], "disclosureNote": "This public entry describes non-sensitive storage architecture, measured capacities, validation outcomes, and runtime limitations. It excludes credentials, tokens, private user-profile paths, private data, and actionable security-sensitive details." }