Hiro development journal

Dashboard recovery and verified workday startup

published Machine-readable JSON

Executive summary

Hiro's local dashboard was unreachable because no process was listening on its web ports. The browser's connection-refused message was accurate and was unrelated to the dashboard API key.

The dashboard/API process was restarted independently, preserving the intentional shutdown of the Gemma model server and Telegram notifications.

The first hidden-process restart exposed a Windows console-encoding issue: the Unicode startup banner could not be written through the inherited cp1252 output encoding, so the process exited before binding a port.

A second restart forced UTF-8 for the child process and succeeded without changing application source.

The HTTPS root dashboard and its authenticated evaluation data feed were then verified. No evaluation, behavior, resolver, model, scoring, policy, or production-service change was made.

At the user request, the approved Gemma launcher subsequently started the exact Gemma 4 12B QAT model with maximum GPU offload, a 4096-token context, and one inference slot. Startup took 20.599 seconds and the launcher warmup took 0.260 seconds.

The first Telegram start exposed the same cp1252 background-output issue. Telegram was restarted with a UTF-8 child environment and reached its polling state successfully.

A final local-only model request returned READY in 0.217 seconds. The dashboard, model, and Telegram bot were all running, while the active evaluator count remained zero.

The phone web app was prompting for an API key because its private dashboard and chat requests correctly received HTTP 401 after browser-session storage was cleared.

Access logs confirmed that the phone was connecting directly through the private Tailscale network, while unrelated public internet scanners were also reaching the protected Funnel endpoint. Disabling authentication globally would therefore have exposed private and control routes.

Hiro now trusts only direct loopback and Tailscale address ranges. Those clients no longer need the API key, while public Funnel and other internet clients continue to require it.

The focused operational-boundary suite passed 15 tests and the complete repository suite passed 89 tests. Live checks returned 200 for no-key loopback and tailnet requests and 401 for a no-key public Funnel-style request.

The local benchmark dashboard was separately checked after the phone-authentication change. Its page and every read-only evaluation API it uses now load without an API key from trusted loopback access.

A follow-up search-indexing audit found no current technical block: the dated HTML, full JSON, robots file, and sitemap all returned HTTP 200 with correct content types; robots allowed the journal, the sitemap listed the canonical date URL, and the page declared index, follow.

Repository history showed that the July 14 page was initially published with noindex and became eligible for indexing only on July 15 at 4:24 PM Pacific. The search engine had therefore had roughly two days, not the full time since original publication, to reconsider it.

Public search returned no results for the entire bballstatistics.com domain, not merely the Hiro page. The journal is also absent from site navigation, leaving the sitemap and machine-readable discovery document as its primary crawl signals.

Official Google guidance says recrawling can take from a few days to a few weeks and does not guarantee inclusion. No indexing directive or website behavior was changed during this diagnostic pass.

The bballstatistics.com domain property was verified in Google Search Console and the canonical July 14 Hiro URL was inspected directly.

Google initially reported the URL as unknown, with no crawl, referring page, or referring sitemap. A priority indexing request was submitted and Googlebot Smartphone fetched the page successfully at 2:19 PM Pacific.

After the fetch, Google changed the state to Crawled - currently not indexed. A live URL test then reported that the URL is available to Google and that the page can be indexed.

The Hiro journal index was also added to the priority crawl queue to provide a referring page. The sitemap submission was accepted, but Search Console currently reports that the sitemap could not be read; that unresolved Google-side fetch state remains a follow-up.

Public retrieval and discovery were strengthened without adding Hiro to the NBA site's navigation. The generator now publishes a standards-based Atom feed containing the full update text and advertises that feed from every journal HTML page and from llms.txt.

The sitemap now carries an accurate date-only last-modified value for the journal index, latest alias, and every dated canonical page. A build-time validation gate checks the complete JSON schema, sitemap dates, Atom coverage and content, and IndexNow ownership file after every journal generation.

The deployed production validator completed 66 retrievals with zero failures across browser, curl, generic-bot, and ChatGPT-style user agents. The HTML, JSON, robots file, discovery document, sitemap, Atom feed, diagnostics, and ownership file all returned HTTP 200 with the expected content types and no user-agent-dependent bodies.

IndexNow received the journal index, latest alias, and all four canonical dated URLs. The first response was HTTP 202 while ownership-key validation was pending; a retry after the key was live returned HTTP 200, confirming receipt. This notification does not guarantee crawl selection or indexing.

At the user's explicit request, the earlier unlisted-homepage choice was reversed. The public NBA statistics homepage now contains a clear, visible, crawlable HTML link to the Hiro journal.

Work completed

Connection-refused diagnosis

Completed
  • Confirmed that the standard dashboard and API ports had no active listeners.
  • Confirmed that the dashboard is served by Hiro's FastAPI process rather than by the model server.
  • Inspected the prior dashboard runtime record and logs, which showed that the previous process had served evaluation overview requests successfully before stopping.

Dashboard-only recovery

Completed
  • Started only Hiro's API/dashboard process in a hidden background session.
  • Left the Gemma endpoint on port 8080 stopped.
  • Left Telegram notifications disabled and the Telegram bot stopped.
  • Preserved the intentional-shutdown marker used by Hiro's lifecycle monitoring.

Windows background encoding diagnosis

Completed
  • The first restart attempt failed before application startup because the Unicode startup banner was encoded through a cp1252 output stream.
  • The retry explicitly selected UTF-8 for Python input/output and startup execution.
  • The UTF-8 retry reached application startup and brought up both the HTTPS dashboard endpoint and the companion HTTP API endpoint.

Dashboard and evaluation-feed validation

Completed
  • Verified that the canonical local dashboard root returned a successful HTML response over HTTPS.
  • Verified the evaluation overview through the normal API-key-protected path without exposing the key.
  • The evaluation overview reported 100 completed runs and retained proposal-only promotion mode.

Workday Gemma startup

Completed
  • Ran the approved lifecycle launcher after the user explicitly requested that Hiro be available through Telegram during the workday.
  • Verified the exact served model identifier, 4096-token context, one inference slot, and maximum GPU offload from startup telemetry.
  • The launcher measured 20.599-second startup latency and 0.260-second warmup latency with a READY response.
  • A separate final readiness request returned READY in 0.217 seconds.

Telegram startup recovery

Completed operationally
  • The lifecycle marker was initially enabled while the bot process had exited because its Unicode startup banner inherited cp1252 output encoding.
  • Restarted the Telegram lifecycle with UTF-8 forced in the child environment.
  • Verified that Telegram notifications were enabled, the managed bot process was alive, and the Telegram application had entered its started polling state.

Evaluation isolation

Completed
  • Inspected running Python command lines after startup.
  • Confirmed that no evaluation, regression-lite, emergence, or ordered-binding process was active.
  • No new evaluation run or promotion decision was created.

Phone web app authentication boundary

Completed and live
  • Diagnosed repeated no-key 401 responses from the phone chat, lab-status, and dashboard data requests.
  • Confirmed that the phone was a direct authenticated Tailscale client rather than a public Funnel client.
  • Added a narrow trusted-client classifier for loopback, Tailscale IPv4 CGNAT, and Tailscale IPv6 address ranges.
  • Kept API-key enforcement enabled for public and ordinary LAN addresses; the existing browser prompt remains available for untrusted clients.
  • Added an explicit configuration toggle that can disable tailnet trust and restore key requirements everywhere.
  • Restarted only the web API/dashboard; the model and Telegram services stayed online.

Benchmark dashboard trusted-access verification

Completed; no additional code change required
  • Confirmed that the benchmark page itself returned HTTP 200 over the local HTTP listener.
  • Confirmed that overview, run detail, comparison, and lineage requests each returned HTTP 200 without an API key from loopback.
  • Inspected the benchmark client logic and confirmed it prompts only after an HTTP 401; the remaining API-key button is for explicitly authenticating an untrusted public client.
  • Determined that the observed prompt came from a browser page session loaded before the trusted-client fix took effect.

Search indexing delay diagnosis

Completed; no active block found
  • Verified clean HTTP 200 responses and correct MIME types for the canonical HTML, full JSON, robots file, and sitemap.
  • Verified an index, follow page directive, an exact canonical URL, explicit robots permission, and inclusion of the dated URL in the sitemap.
  • Confirmed that exact-title, exact-URL, and site-restricted public searches still returned no result.
  • Reviewed publication history and identified the original noindex launch followed by the indexable-policy change on July 15.
  • Identified weak discovery signals: the domain has no visible indexed footprint and the unlisted journal has no normal site-navigation link.
  • Reviewed official Google, Bing Webmaster Tools, and IndexNow guidance for crawl requests, sitemap submission, URL inspection, and push notification.

Google Search Console inspection and crawl requests

Priority crawl submitted; indexing pending
  • Completed domain-property ownership verification without publishing account details.
  • Inspected the exact canonical July 14 URL and captured Google initial state: unknown URL, no last crawl, no referring sitemap, and no referring page.
  • Requested indexing once; Search Console confirmed that the URL was added to a priority crawl queue.
  • Confirmed a subsequent successful Googlebot Smartphone fetch with crawling allowed.
  • Ran the live URL test; Google reported that the URL is available and the page can be indexed.
  • Requested indexing once for the Hiro journal index so its dated-entry links can become a referring-page signal.
  • Submitted the full sitemap. Google acknowledged the submission, then displayed an unresolved could-not-fetch/could-not-read status with zero discovered pages.

Public retrieval and discovery hardening

Deployed and validated
  • Added a public Atom feed at /hiro/feed.xml with one entry per journal date and full plain-text content covering summary, workstreams, decisions, validation, current state, and next steps.
  • Advertised the Atom feed through an HTML alternate link, llms.txt, and the public retrieval diagnostics document.
  • Added date-only last-modified elements to every sitemap URL, using the newest journal date for the index and latest alias and each entry date for canonical dated pages.
  • Added a root-hosted IndexNow ownership file and a reusable submission command that reads the canonical URL set from the generated sitemap.
  • Added a deterministic generated-artifact validator to the normal journal generation and production build pipeline.
  • Extended the live retrieval validator to cover the feed, sitemap dates, ownership file, and full Atom content under four common user agents.

Homepage discovery link

Deployed and validated
  • Added a semantic navigation block directly to the static homepage document.
  • Added the plain anchor text Hiro development updates with href /hiro outside the React application root.
  • Applied minimal visible styling while keeping the link functional with JavaScript disabled.
  • Published the change through the existing Vercel deployment without changing the Hiro page URLs, schema, or retrieval behavior.

Decisions and reasoning

Validation and evidence

CheckStatusResult
Initial listener inspection passed No dashboard listener was present, matching the browser's ERR_CONNECTION_REFUSED result.
First hidden restart failed The process exited before binding because the Unicode startup banner could not be encoded with cp1252.
UTF-8 restart passed Both FastAPI application instances completed startup; HTTPS port 8000 and HTTP port 8001 were served.
Dashboard root passed GET https://127.0.0.1:8000/ returned HTTP 200 with text/html content.
Authentication boundary passed The protected evaluation route rejected an unauthenticated request with HTTP 401.
Authenticated evaluation overview passed The read-only overview returned HTTP 200, reported 100 completed runs, and reported proposal_only promotion mode.
Model isolation passed The local model endpoint remained unavailable after dashboard recovery; Gemma was not restarted.
Telegram isolation passed Telegram notifications remained disabled and the Telegram bot remained stopped.
Source and evaluation changes passed No Hiro source, evaluator, scoring, held-out data, policy, grounding, credential, certificate, or environment configuration was changed.
Gemma launcher passed The exact gemma-4-12b-it-qat-q4_0 model started with 20.599-second startup latency and a 0.260-second READY warmup.
Initial Telegram start failed The managed bot exited during its Unicode startup banner because the hidden child inherited cp1252 output encoding.
UTF-8 Telegram recovery passed The managed Telegram bot stayed alive, notifications were enabled, and the application log reported that polling had started.
Final local model readiness passed A local-only chat-completions request returned READY from gemma-4-12b-it-qat-q4_0 in 0.217 seconds.
Evaluator process audit passed Zero evaluator processes were active after the workday startup.
Phone request-path diagnosis passed Local access logs showed the phone requests arriving directly from the authenticated tailnet and receiving 401 before the change.
Focused operational-boundary suite passed 15 tests passed, covering trusted ranges, rejected public and ordinary LAN ranges, no-key tailnet access, key enforcement, and the disable switch.
Complete repository suite passed 89 tests passed in 9.46 seconds.
Live loopback no-key access passed The protected lab-status endpoint returned HTTP 200 without an API key from loopback.
Live tailnet no-key access passed The protected lab-status endpoint returned HTTP 200 without an API key over the direct Tailscale address.
Live public boundary passed A no-key Funnel-style request carrying a public client address returned HTTP 401.
Service continuity passed The Gemma model endpoint remained HTTP 200 and the Telegram bot remained running after the web API restart.
Benchmark page no-key load passed GET http://127.0.0.1:8001/benchmarks returned HTTP 200.
Benchmark overview no-key load passed The evaluation overview endpoint returned HTTP 200 from loopback without an API key.
Benchmark detail no-key load passed A real evaluation run detail endpoint returned HTTP 200 from loopback without an API key.
Benchmark comparison no-key load passed A real candidate-versus-baseline comparison returned HTTP 200 from loopback without an API key.
Benchmark lineage no-key load passed The evaluation lineage endpoint returned HTTP 200 from loopback without an API key.
Canonical HTML indexing audit passed The July 14 HTML returned HTTP 200, text/html, index and follow metadata, and the exact www canonical URL.
Machine-readable endpoint audit passed The full JSON endpoint returned HTTP 200 with application/json content.
Robots audit passed robots.txt returned HTTP 200, allowed the Hiro path, and referenced the sitemap.
Sitemap audit passed sitemap.xml returned HTTP 200 with application/xml content and included the canonical July 14 URL.
Public search discovery failed Exact-title, exact-URL, and site-restricted searches returned no result; no page from the domain was visibly indexed.
Indexing-policy history diagnostic The page launched with noindex and changed to index, follow on July 15 at 4:24 PM Pacific.
Search Console ownership passed The bballstatistics.com domain property was ownership-verified.
Initial canonical URL inspection diagnostic Google reported the July 14 URL as unknown, with no crawl, sitemap, or referring page.
July 14 priority crawl request passed Search Console confirmed that the canonical dated URL was added to the priority crawl queue.
Googlebot fetch passed Googlebot Smartphone crawled the page at approximately 2:19 PM Pacific; crawl was allowed and page fetch was successful.
Post-crawl index state pending Google changed the URL state to Crawled - currently not indexed.
Live URL eligibility test passed Search Console reported URL is available to Google and Page can be indexed.
Journal index priority request passed The canonical /hiro journal index was added to the priority crawl queue.
Sitemap submission partial Google accepted the sitemap submission but currently reports Sitemap could not be read and zero discovered pages.
Generated retrieval artifact gate passed The build-time validator passed for four journal entries, full latest JSON content, every sitemap last-modified value, full Atom feed coverage, and the public IndexNow ownership file.
Full website production build passed The TypeScript and Vite production build completed after the new generation and validation steps.
Live multi-user-agent retrieval suite passed 66 deployed requests completed with zero failures across browser, curl, generic-bot, and ChatGPT-style user agents; bodies were identical per endpoint and all expected status and content-type checks passed.
Live Atom and IndexNow ownership endpoints passed The deployed Atom feed returned HTTP 200 with application/xml, and the root ownership file returned HTTP 200 with text/plain.
IndexNow canonical URL notification passed After an initial HTTP 202 pending key validation, the retry returned HTTP 200 for the journal index, latest alias, and four canonical dated pages.
Homepage-link production build passed The complete TypeScript/Vite production build and generated Hiro artifact gate passed with the new static homepage navigation link.
Live raw homepage HTML passed The deployed homepage returned HTTP 200 text/html and contained the exact anchor <a href="/hiro">Hiro development updates</a> outside the React root.
JavaScript-independent discovery passed The Hiro link is present in the initial HTML response and does not require hydration, cookies, authentication, or client-side JavaScript.

Current state

Next steps