Hiro development journal

Diagnosis of Hiro's idle local-model memory use

2026.07.22 at 18:45:19 America/Los_Angeles Published Machine-readable JSON

Executive summary

Diagnosed unexpectedly high memory use while Hiro appeared idle. The local inference server was still active with the Gemma 12B quantized model loaded in LM Studio, even though its status was idle and no evaluation was currently running.

The model was configured for maximum GPU offload and the GPU was actively allocating substantial VRAM. LM Studio itself had approximately 3 GB resident in physical RAM but about 16.5 GB of private committed address space. The system-wide 80-percent figure was therefore not physical RAM owned solely by LM Studio.

The current launcher intentionally leaves the model resident indefinitely because it specifies no idle time-to-live. This avoids a long cold start for scheduled evaluations but retains model mappings, CUDA buffers, and committed memory between runs. No service or configuration was changed during this diagnosis.

Follow-up configuration tracing found a migration gap: the active Hiro profile remains gemma and the nightly preflight explicitly invokes the Gemma launcher. A Qwen 3.6 profile exists in the router, but it is not selected and no Qwen 3.6 model is currently loaded.

Work completed

Process and memory inspection

Completed

The primary LM Studio process reported approximately 2.96 GB resident working set and 16.47 GB private committed memory. These metrics are different: committed address space is not equivalent to physical pages currently resident in RAM.
The workstation had approximately 31.84 GB physical memory with 6.65 GB available. Memory compression itself held approximately 8.21 GB resident, consistent with system-wide memory pressure from multiple applications and committed allocations.
Another active graphics application also had substantial private committed memory. The 80-percent system reading cannot be attributed to LM Studio alone from the resident-working-set evidence.

GPU and model residency

Completed

The GPU reported approximately 16.1 GB of 32.6 GB VRAM in use across the system. Per-process GPU attribution was unavailable without elevated permissions, but the local server configuration and LM Studio process list confirm that the Hiro model is loaded with maximum GPU offload.
LM Studio reported one loaded idle model, Gemma 4 12B QAT at approximately 6.98 GB with a 4096-token context and one parallel slot.
Maximum GPU offload places model computation and most weights on the GPU, but it does not imply zero system-memory use. The runtime can still memory-map the model file and allocate CPU metadata, pinned transfer buffers, context bookkeeping, and Windows shared-GPU or committed address space.

Persistence cause

Confirmed

Hiro's checked-in model launcher starts LM Studio on the local inference port and loads the model with maximum GPU offload, a 4096-token context, and one parallel slot.
The launcher does not supply LM Studio's supported idle TTL option. LM Studio therefore keeps the model loaded after evaluations finish; idle means not currently generating, not unloaded.
The recorded server startup took approximately 78 seconds in the existing metadata. Persistent residency was selected implicitly to avoid paying that cold-start delay before each evaluation.

Qwen 3.6 migration gap

Diagnosed; not changed

The effective configuration resolves active_model_profile to gemma. The standard and fast local tiers therefore both target the Gemma identifier on port 8080.
The nightly readiness preflight has a hard-coded Gemma starter path and invokes it before reporting readiness. The background launcher follows the same Gemma script.
The router contains an alternate profile keyed qwen35 that targets qwen36-35b-a3b on port 8080, but selecting that profile and providing a matching Qwen launcher/preflight identity check were never completed.
LM Studio reports only Gemma as loaded. Qwen 3.6 availability in the local model catalog does not make it the active Hiro runtime.

Decisions and reasoning

Do not stop or reconfigure the model server during a diagnosis-only request.
Distinguish resident physical memory, private committed memory, system memory pressure, and dedicated GPU allocation when reporting resource use.
Treat an idle loaded model as expected persistent server behavior, not evidence of active inference.
Recommend an idle TTL as the bounded remedy if lower idle resource use is preferred. A five-to-fifteen-minute TTL would unload after inactivity but reintroduce the measured cold-start delay for the next Hiro request.
Do not assume that maximum GPU offload eliminates all CPU-side mappings and buffers.
Treat the current Gemma process as a configuration migration defect relative to the intended Qwen 3.6 runtime, not as an unknown background job.

Validation and evidence

Check	Status	Result
LM Studio loaded-model inspection	completed	LM Studio reported one Gemma 12B quantized model loaded, idle, approximately 6.98 GB, 4096-token context, one parallel slot, local device, and no configured TTL.
Local inference endpoint	completed	The configured local inference endpoint responded successfully, confirming that the resident server remained available after the evaluation runs.
Memory and GPU counters	completed	System counters, process working sets, private committed bytes, and global GPU memory were inspected read-only. The evidence shows both GPU allocation and CPU-side committed memory, with system-wide pressure distributed across multiple processes and memory compression.
Hiro code or runtime changes	not-run-by-design	No process was stopped, no model was unloaded, and no launcher, TTL, configuration, service, schedule, or evaluation result was changed.
Journal tests and site build	passed	The timestamped-entry unit test passed. The production build generated and validated 29 journal entries, verified timestamp identity, same-day uniqueness, aliases, sitemap, Atom content, noindex metadata, and the IndexNow key, then completed TypeScript and Vite compilation.
Active-profile and launcher tracing	completed	The effective active profile resolved to gemma. The router contains Qwen 3.6 under an alternate qwen35 profile, while the nightly preflight and background launcher explicitly start and validate Gemma.

Current state

The local model server is running and the Gemma model is loaded but idle.
Hiro can respond without a cold model load, but the workstation retains CPU-side committed memory and GPU allocations while idle.
No idle TTL is configured, so residency continues until LM Studio unloads the model or the server is stopped.
The current resource state is a latency-versus-idle-memory tradeoff rather than evidence of an unknown active evaluation.
The intended Qwen 3.6 runtime is not currently active; Gemma remains selected by both configuration and operational launch paths.

Next steps

If the user prioritizes reclaiming idle resources, add an approved bounded LM Studio TTL to the checked-in launcher and test unload plus real automatic reload before the next evaluation.
If the user prioritizes immediate Hiro responses and frequent development cycles, retain persistent residency and close other high-memory graphics applications when necessary.
Add lightweight model-residency and memory-pressure visibility to operational status reporting so idle-loaded and actively-generating states are easy to distinguish.
With user approval, complete the Qwen 3.6 migration atomically: activate the correct router profile, add a pinned Qwen launcher, update preflight identity validation, verify a real child-process startup and model response, then unload the obsolete Gemma runtime.