{ "schemaVersion": 1, "date": "2026.07.19", "title": "RTX 5090 hardware review and local-model trial plan", "publicationStatus": "Published", "executiveSummary": [ "Hiro's host was inventoried after its GPU upgrade. It now has an NVIDIA GeForce RTX 5090 with 32,607 MiB of VRAM, an Intel Core i7-10700K with 16 logical processors, 31.84 GiB of physical RAM, and several terabytes of free space on E:.", "The practical target is fully GPU-resident models with 20B to 35B total parameters. The GPU can hold high-quality 4-bit or 5-bit versions with a useful context cache, while 32 GiB of system RAM makes aggressive CPU offload of 70B-plus models unattractive.", "The first recommended trials are Qwen3.6-35B-A3B Q4_K_M, Gemma 4 31B Q4_K_M, and gpt-oss-20b in native MXFP4. Gemma 4 26B-A4B, Qwen3.5-27B, and Devstral Small 2 24B form a second wave for efficiency, dense-model comparison, and coding specialization.", "No model was downloaded or benchmarked. This session produced an evidence-based acquisition and evaluation plan and leaves Gemma 4 12B as Hiro's baseline until a candidate passes compatibility, quality, latency, memory, and reliability gates." ], "workstreams": [ { "title": "Host hardware inventory", "status": "Completed", "details": [ "nvidia-smi identified an RTX 5090 with 32,607 MiB total VRAM, driver 591.86, CUDA 13.1 capability, compute capability 12.0, and a 575 W power limit.", "The Windows desktop used about 1,124 MiB of VRAM during inspection, leaving roughly 30.7 GiB before model allocation.", "The processor is an Intel Core i7-10700K at 3.80 GHz with 16 logical processors. The platform is an MSI MEG Z490 Trident X with an MEG Z490I UNIFY board.", "A Windows memory-status call reported 31.84 GiB physical RAM, 21.53 GiB available, and a 56.54 GiB total page-file commitment limit.", "Storage inspection found 24.0 GiB free on C:, 947.8 GiB on D:, and 3,391.2 GiB on E:. Future model files should live on E: rather than the current C: application-data path." ] }, { "title": "Hiro inference stack and baseline", "status": "Completed", "details": [ "Hiro uses an OpenAI-compatible local endpoint through LM Studio and llama.cpp. Its approved launcher loads Gemma 4 12B QAT Q4 with maximum GPU offload, a 4,096-token context, and one slot.", "The registry contains Qwen 9B and Gemma 4 12B. Configuration retains older Qwen 4B fast and Qwen 9B standard tiers while selecting Gemma as the active profile.", "The existing 120-case bakeoff covers evidence obedience, no-evidence refusal, self-improvement specifications, structured JSON, tool routing, and troubleshooting.", "The June 21 historical report recorded 80.0% quality for Qwen 9B and 83.1% for Gemma 4 12B QAT, with no recorded timeouts or crashes. These values were reviewed, not rerun.", "Port 8080 was not running. The LM Studio CLI reported commit 0b2a176, while runtime enumeration timed out after trying to wake the service." ] }, { "title": "Priority model shortlist", "status": "Completed", "details": [ "Priority 1 — Qwen3.6-35B-A3B: 35B total and 3B active parameters, 262K native context, multimodal input, agentic-coding improvements, and official llama.cpp support. Start with Q4_K_M, 8K context, one slot, and full GPU offload. Estimated weights are about 21 GiB before runtime and cache overhead.", "Priority 2 — Gemma 4 31B: a 30.7B dense model with up to 256K context. Q4_K_M should use about 18 to 19 GiB for weights and gives the most useful dense quality comparison with Qwen's sparse model.", "Priority 3 — gpt-oss-20b: 20.9B total and 3.6B active parameters, a 12.8 GiB native MXFP4 checkpoint, 128K context, adjustable reasoning effort, structured outputs, and strong tool-use training. OpenAI specifies a 16 GiB memory requirement.", "Priority 4 — Gemma 4 26B-A4B: 26B total with about 4B active. Try Q5_K_M if measured headroom permits, otherwise Q4_K_M. It is the efficiency-oriented Gemma alternative.", "Priority 5 — Qwen3.5-27B: a 27B dense model with 262K native context. It is a more established control for whether Qwen3.6's sparse architecture and newer runtime path improve Hiro. Q5_K_M should be practical.", "Priority 6 — Devstral Small 2 24B Instruct: a coding and software-engineering specialist with tools and vision. Mistral positions it for one RTX 4090 or a 32 GiB Mac. Evaluate it for Hiro's code and self-improvement workload, not as an automatic general default." ] }, { "title": "Runtime fit and compatibility", "status": "Completed", "details": [ "Total parameters determine resident weight memory for MoE models; active parameters mainly affect compute. A 35B-A3B model still needs VRAM and disk for roughly 35B quantized weights.", "Reserve 3 to 4 GiB of VRAM for Windows, CUDA allocations, and cache. Q4_K_M is the safe first quant for 31B to 35B models; Q5_K_M is a finalist experiment after measuring actual headroom.", "Do not begin at advertised 128K to 262K contexts. Establish 8K stability, then measure 16K and 32K because cache growth, prompt processing, and conversation reuse can dominate.", "Current llama.cpp supports the listed families, but each needs a pinned-runtime check. Open upstream Qwen hybrid-cache and Gemma cache-memory reports make long-conversation testing mandatory.", "Models needing roughly 60 to 80 GiB for quantized checkpoints, including gpt-oss-120b and similar 100B-plus MoE releases, are out of scope. The 32 GiB system RAM is not a reliable high-performance overflow tier." ] }, { "title": "Benchmark and promotion plan", "status": "Completed", "details": [ "Use a staged funnel: load and identity check, deterministic warmup, a balanced six-category smoke subset, the complete 120-case bakeoff, then multi-turn and real tool-call probes.", "Score by category, not only one aggregate. Evidence obedience, unsupported-claim refusal, valid JSON, and correct tool routing are promotion-critical.", "Record startup time, first-token latency, generation and prompt throughput, peak VRAM, peak RAM, timeout and crash rates, and cancellation recovery.", "Test non-thinking behavior separately from reasoning mode. Task content stays constant while each model receives its native supported chat template and controls.", "Compare Q4 and Q5 only for finalists. A full quant matrix would consume storage and time before establishing whether behavior is competitive.", "Keep Gemma 4 12B as rollback until a candidate beats its historical 83.1% result or has a justified category tradeoff with no reliability regression." ] }, { "title": "Journal indexing-policy alignment", "status": "Completed", "details": [ "Restored noindex, follow metadata to the journal index and generated pages to match current publication policy.", "Kept direct robots.txt access to /hiro/ and /llms.txt; the journal remains directly retrievable without search indexing.", "Did not add navigation or change Hiro application code, model configuration, credentials, or runtime services." ] } ], "decisions": [ "Try Qwen3.6-35B-A3B first because it combines the strongest current general, coding, multimodal, and agentic profile that fits as a fully resident 4-bit model.", "Use Gemma 4 31B as the dense quality control; only a dense peer reveals whether the MoE candidate gives up consistency on Hiro's evidence and formatting contracts.", "Keep gpt-oss-20b in the first wave because Hiro is a tool-using agent and its structured-output, reasoning-effort, and tool-use training may matter more than parameter count.", "Evaluate Devstral as a specialist. A coding win may qualify it for repository work without replacing Hiro's conversational model.", "Start large candidates at Q4_K_M and one slot to establish stability and VRAM reserve before maximizing quant fidelity.", "Move model storage to E: before downloads; C: cannot safely hold a 35B Q4 candidate plus temporary overhead.", "Do not pursue 70B-plus CPU offload; it would trade the 5090's advantage for pressure on 32 GiB RAM and an older eight-core CPU.", "Treat maximum context as a ceiling, not a deployment setting. Promote it only after cache, reuse, and multi-turn measurements.", "Pin and record the runtime for every bakeoff because family support and cache behavior change between builds.", "Do not change Hiro's default based on external benchmarks. Promotion requires Hiro's local suite and operational probes." ], "validation": [ { "check": "GPU inventory", "status": "passed", "result": "nvidia-smi reported the RTX 5090, 32,607 MiB VRAM, driver 591.86, CUDA 13.1 capability, and current use." }, { "check": "CPU and platform", "status": "passed", "result": "Registry and environment data identified the i7-10700K, 16 logical processors, and MSI Z490 platform." }, { "check": "Physical memory", "status": "passed", "result": "GlobalMemoryStatusEx returned 31.84 GiB total RAM and 21.53 GiB available." }, { "check": "Storage inventory", "status": "passed", "result": ".NET drive inspection returned 24.0 GiB free on C:, 947.8 GiB on D:, and 3,391.2 GiB on E:." }, { "check": "Hiro configuration review", "status": "passed", "result": "The launcher, settings, registry, and current 4K single-slot Gemma profile were inspected without reading or publishing secrets." }, { "check": "Historical bakeoff review", "status": "passed", "result": "The existing 120-case report was read as historical evidence and not represented as a new run." }, { "check": "Local model endpoint", "status": "not-running", "result": "http://127.0.0.1:8080/v1/models was unreachable during the final read-only check." }, { "check": "LM Studio runtime enumeration", "status": "incomplete", "result": "The CLI reported commit 0b2a176, but runtime listing timed out while waking the service." }, { "check": "New-model benchmark", "status": "not-run", "result": "No weights were downloaded, no server was started, and no new quality or speed claims were made." }, { "check": "Journal production build", "status": "passed", "result": "npm run build generated five dated journal pages, passed the schema, sitemap, Atom, IndexNow, and noindex validation gates, compiled TypeScript, and completed the Vite production build." } ], "currentState": [ "Hiro remains configured for Gemma 4 12B QAT through its existing launcher when the model lifecycle is started.", "The local model endpoint was offline at the end of review.", "The RTX 5090 creates a practical fully resident 20B to 35B tier, substantially above the former 12 GiB-class constraint.", "System RAM, C: free space, runtime pinning, and long-context cache behavior are the main constraints.", "No candidate has earned promotion; memory figures for GGUF choices remain estimates until exact files are loaded and measured." ], "nextSteps": [ "Move or reconfigure the LM Studio and GGUF library to E: before downloading any 20B-plus candidate.", "Pin a runtime that explicitly loads Qwen3.6, Gemma 4, gpt-oss, and Devstral GGUFs on Blackwell.", "Acquire Qwen3.6-35B-A3B Q4_K_M first; record its hash and size, then run load, identity, warmup, and VRAM checks at 8K with one slot.", "Run a balanced smoke subset and the full 120-case bakeoff against unchanged Gemma 4 12B.", "Repeat the gates for Gemma 4 31B Q4_K_M and native gpt-oss-20b before deciding whether second-wave models are needed.", "For finalists, test 16K and 32K conversations, cache reuse, cancellation, tools, and JSON before trying Q5.", "Promote by role if evidence supports it: general assistant, reasoning/tool specialist, and coding specialist need not be one model." ], "disclosureNote": "This public entry contains hardware classes, measured capacities, public model specifications, and non-sensitive architecture. It excludes credentials, tokens, private user data, and actionable security-sensitive details." }