Hiro development journal

Turning Gmail review into one complete background scan

2026.07.26 at 18:03:32 America/Los_Angeles Published Machine-readable JSON

Executive summary

The earlier pagination repair exposed the next twelve unanalyzed conversations, but it still required the user to click Analyze once per twelve-conversation group.

Analyze now starts one server-owned background job that continues through the complete Gmail query until no eligible conversations remain. Twelve remains an internal safety batch size for local Qwen 9B and is no longer a user workflow limit.

The task dashboard polls compact job status and displays cumulative conversations reviewed, task and reading imports, ignore decisions, model failures, and completion state. Closing the dialog or losing the phone connection does not stop the server job.

The job retains the existing three-conversation Qwen request limit and individual-conversation fallback. A conversation that still cannot be classified is recorded in the job total and excluded from the current run so the remaining period continues.

Hiro was restarted and the private remote task page now serves the full-scan interface and background-job API. The real mailbox scan was intentionally not started during validation.

Work completed

Background scan orchestration

Completed

Added one active Gmail import job per task-list owner with queued, running, completed, completed-with-errors, authorization-required, and failed states.
The background worker repeatedly requests the next internal batch until the Gmail query is exhausted.
Cumulative progress includes batches, processed and analyzed conversations, imported tasks, imported reading items, ignores, failures, and conversions from prior ignore decisions.
A duplicate Analyze request while a job is active returns the current job instead of launching competing model work.

Forward progress and failure isolation

Completed

Extended the batch importer with transient exclusions used only within a background scan.
Normal scans temporarily exclude conversations that fail both batch and individual Qwen classification, preventing one difficult conversation from creating an infinite retry loop.
Ignored-item reanalysis temporarily excludes every source already handled in that job because an unchanged ignore remains eligible in persistent storage.
The worker stops safely if an importer response cannot demonstrate forward progress.

Remote progress experience

Completed

Changed the Gmail dialog copy to state that the complete query is reviewed in safe background batches.
Analyze and Reanalyze ignored now launch background jobs rather than holding one long browser request.
The dialog polls current status every two seconds and refreshes task and reading counts at completion.
Reopening the dialog reconnects to the current server job, including from the private phone-access page.

Runtime deployment

Completed

Restarted Hiro with the new job module and routes.
Verified the private remote page serves the complete-query copy and progress code.
Verified the API exposes job start, current-job, and job-status endpoints and reports no job before the user initiates the scan.

Decisions and reasoning

Keep twelve as the internal retrieval batch and three as the maximum structured Qwen request size because those bounds have passed prior reliability testing.
Move long-running orchestration to Hiro so browser disconnects do not cancel the scan.
Allow at most one active Gmail scan per owner to avoid duplicate imports and competing local-model load.
Continue through isolated model failures and report them at completion instead of repeatedly retrying the same conversation or stopping the remaining scan.
Do not start a full live mailbox scan as part of deployment validation because classification creates Review items and should begin from an explicit user action.

Validation and evidence

Check	Status	Result
Focused automated regression suite	passed	Eight tests passed. Coverage includes multi-batch completion, cumulative totals, continued scanning after an isolated classification failure, transient exclusion during ignored-item reanalysis, task and reading persistence, multiple tags, due dates, bounded Qwen batches, individual fallback, API round trips, and incremental source handling.
Task-page JavaScript syntax	passed	The embedded task-page script parsed successfully after the background polling changes.
Restarted Hiro runtime	passed	Hiro restarted in approximately 2.3 seconds and returned the task page normally.
Private remote page	passed	The private remote task page returned status 200 and contained the complete-query explanation and cumulative progress behavior.
Background job API availability	passed	The live OpenAPI contract exposed the job start route, and the current-job route returned an empty state before user initiation.
Public journal unit tests	passed	npm run test:hiro passed the timestamped-entry unit tests.
Public journal production build	passed	npm run build generated and validated 52 journal pages, then completed the TypeScript and Vite production build.

Current state

One Analyze click reviews the entire entered Gmail query, including the default thirty-day period.
The user can close the import dialog and later reopen it to see the active job status.
Tasks and reading items are still created in Review with Qwen-selected applicable tags and source-derived due dates.
Gmail remains read-only and the actual full scan has not been started during this deployment session.

Next steps

Initiate Analyze once from the refreshed task page and allow the background job to complete the thirty-day query.
Review created tasks and reading items after completion, correcting tags or wording where necessary.
Persist job progress to SQLite if restart-resilient live status becomes important beyond the existing source-level resume behavior.
Add pause and cancel controls only if real full-period runs show a practical need.