Hiro development journal

Router performance telemetry added to Observatory

2026.07.23 at 15:37:20 America/Los_Angeles Published Machine-readable JSON

Executive summary

Added a Performance tab to Hiro's private Evaluation Observatory so routed development tasks can be reviewed by time, task label, selected model, route, status, duration, and token usage.

The new read-only API aggregates the router's append-only JSONL ledger and reports input, cached-input, output, reasoning, and total input-plus-output tokens without returning prompts or model output.

Restarted only Hiro's API process through its checked-in runner. Both local dashboard listeners returned the new endpoint successfully, while Qwen remained intentionally unloaded and the existing Cloudflare process was left unchanged.

Work completed

Performance dashboard

Completed

Added a first-class Performance navigation tab with summary cards, a chronological task-attempt table, and token allocation by model.
Each row identifies the safe task label, short task ID, task kind, risk, start time, exact cloud model or active local profile, reasoning level, route, attempt number, token breakdown, duration, and completion status.
The page handles an empty router ledger and refreshes on demand or when its tab is selected.

Prompt-free telemetry API

Completed

Added a read-only performance endpoint backed by the router's append-only ledger with a configurable ledger path.
Total token usage is defined as input plus output tokens. Cached input is shown as a subset of input and is not double-counted; reported reasoning tokens are displayed separately.
Cloud route names resolve to their configured model and reasoning profiles. Local attempts are identified by Hiro's active Qwen profile.

Safe task identity and ledger privacy

Completed

Added an optional safe task label to the router CLI and PowerShell launcher for readable dashboard attribution.
Unlabeled historical or future records fall back to task kind plus a short immutable task identifier.
Removed the truncated final model message from new summary-ledger records; detailed per-task artifacts remain local. The dashboard API never returns prompts or output.

Live service activation

Completed

Verified that the existing dashboard HTML immediately exposed the new tab, then restarted only the exact process owning Hiro's three expected API listeners so the new route module would load.
Used the checked-in blocking API runner with normalized Windows Path state and hidden process startup.
Confirmed the performance endpoint on both HTTP port 8001 and the loopback Observatory port 8765. The current ledger contained one task attempt and 13,525 total input-plus-output tokens.

Decisions and reasoning

Do not display task prompts or model output remotely. A safe operator label provides useful identity without expanding the dashboard's sensitive-data surface.
Count input plus output as total tokens and report cached input separately because cached tokens are already part of input usage.
Represent every attempt rather than collapsing escalations so model changes and the cost of failed lower-tier attempts remain visible.
Keep this view read-only and under the Observatory's existing private-access boundary.
Do not restart Cloudflare or load Qwen merely to activate a dashboard API change.

Validation and evidence

Check	Status	Result
Focused dashboard, API, router, and operational suite	passed	33 tests passed in 3.95 seconds, covering prompt-free output, exact token aggregation, model mapping, empty-ledger behavior, dashboard presence, existing evaluation APIs, remote access behavior, routing policy, one-escalation resume, output budgets, payloads, and operational boundaries.
Python compilation and router launcher startup	passed	The modified server, configuration, performance API, router CLI, and dispatcher compiled successfully. The normalized PowerShell launcher returned a ready self-check with all five routes.
Dashboard JavaScript parse	passed	The inline Observatory script was extracted and compiled with Node.js without a syntax error.
Live API activation	passed	After the scoped API restart, ports 8001 and 8765 both returned HTTP 200 from the new performance endpoint with one task, one attempt, and 13,525 total tokens.
Model shutdown preservation	passed	LM Studio reported that no models were loaded after activation, preserving the user's prior Qwen shutdown request.

Current state

The Performance tab is active in Hiro's running private Evaluation Observatory and reads the append-only router summary ledger.
New routed tasks can provide a safe label through the documented launcher; historical records remain identifiable through task kind and ID.
The current dataset contains the initial Luna Low router probe. The view will become comparative as real routed tasks accumulate.
The Cloudflare tunnel process and private-access configuration were not modified.

Next steps

Use a short safe label for each routed task so the Performance table remains readable without exposing prompts.
Accumulate a varied sample before changing route thresholds, then compare validated success, total tokens, cache share, duration, and escalation frequency by model.
Consider adding time-range and model filters once the ledger contains enough tasks to make filtering useful.
Keep per-task prompts, output, credentials, and detailed artifacts local rather than expanding the remote telemetry surface.