Hiro development journal

Real promotion activation and rollback proven end to end

2026.08.26 at 15:53:48 America/Los_Angeles Golden promotion and forced-failure rollback passed against the production promotion machinery Machine-readable JSON

Executive summary

Replaced open-ended promotion diagnosis with a deterministic process-level integrity harness for Hiro's current continuous-improvement promotion path.

Established the authoritative path from the continuous queue through the governor, durable promotion transaction, Git fast-forward, detached activation broker, exact process restart, loaded-revision verification, runtime probation, finalization, and additive rollback.

Proved a golden candidate reached queue state implemented and transaction state finalized while the isolated Hiro process actually ran the candidate revision.

Proved a second candidate was actually activated, deliberately failed deterministic post-activation verification, traversed compensating rollback, restarted at the additive rollback revision, and restored a tree identical to the pre-candidate baseline.

Changed only the harness module, its operating document, and its focused tests. No production governor, transaction, activation, rollback, queue, scoring, discovery, or candidate-generation implementation was changed.

Integrated the three-file harness revision and restarted the regular Hiro runtime. Hiro now reports status ok with matching loaded and checkout revision 1b44ff1bd2076e774138ab87f2c6a0d4a8bbbe53.

Work completed

Canonical production path and state authority

Completed before implementation

Confirmed that the live scheduler enters hiro.improvement.active_loop and then hiro.improvement.continuous_engine.run_continuous_improvement_tick.
Confirmed that candidate state is authoritative in the ideas table plus append-only events table in the continuous queue SQLite database.
Confirmed that promotion state is authoritative in promotion_transactions plus append-only promotion_transaction_events in that same database, with one active transaction enforced.
Traced the exact current call path through continuous_engine._advance_canary, PromotionTransactionStore.prepare, ContinuousGovernor.promote, Git fast-forward, launch_activation_broker, complete_hiro_activation, start_hiro, health revision verification, reconciliation, runtime probation, and finalization or rollback.
Explicitly excluded historical Stage 6 evidence promotion, the Stage 6C runtime controller, the older Stage 6 runner, Daylab, and Nightlab from the harness.

Deterministic process fixture

Completed

Added hiro/improvement/promotion_integrity_harness.py as an explicit command-line process proof rather than an automatically collected unit test.
Each scenario creates a fresh local clone, fresh candidate worktree, fresh SQLite database, and a harmless direct-child candidate that adds one reversible docs/hiro_worker_ probe artifact.
The harness persists the candidate through the real queue states queued, investigating, candidate, testing, and canary before entering the production promotion machinery.
The production default independent canary probe runs at logical minutes 0, 5, and 15, and the production governor runs its complete repository suite before allowing Git activation.
The isolated API runs with its background scheduler disabled so the deterministic driver cannot race another queue tick. The API process, startup helper, fixed ports, model connection, health endpoint, Git operations, broker, and restarts remain real.
The only activation seam supplies the isolated Hiro process ID to the real detached broker so the test orchestrator survives while the broker still terminates and replaces the actual isolated service process.
Runtime probation uses a deterministic process verifier in place of ambient assistant-lab and production-metric inputs. It makes real requests to health and tasks endpoints and checks loaded revision, checkout revision, revision match, active Git revision, and service readiness.
The logical clock advances through production probation checkpoints 0, 60, 240, and 480 without waiting eight wall-clock hours; actual runtime identity is checked at every checkpoint.

Golden-path process proof

Passed

The golden synthetic candidate revision was 9457d565e822aceb563464e166e200e062aeb32a over harness source revision 1b44ff1bd2076e774138ab87f2c6a0d4a8bbbe53.
Its production governor suite passed all 770 tests with four existing unknown-marker warnings in 431.43 seconds.
The active checkout fast-forwarded to the candidate, the detached broker stopped the exact baseline Hiro process, start_hiro launched the replacement, and the transaction recorded runtime_verified only after the health endpoint reported the candidate as both loaded and checked out.
Reconciliation placed both queue and transaction into probation. Deterministic process verification passed all four runtime checkpoints.
The queue finished at implemented, the transaction finished at finalized, Git HEAD remained the candidate revision, and the live health identity remained the candidate revision.

Forced verification-failure rollback proof

Passed

The rollback scenario candidate revision was febd058dc7864d4d6c61c12667dc9062f2df8d7d and independently passed all 770 governor tests with the same four existing warnings in 434.51 seconds.
The broker activated the candidate and health verification proved it was running before the deliberate failure was issued.
The deterministic verifier returned PROMOTION_HARNESS_FORCED_POST_ACTIVATION_FAILURE only after candidate runtime identity had passed.
The production path moved the queue to rollback and the transaction through compensating and rollback_restart_requested.
ContinuousGovernor.rollback created additive rollback revision fbd0a80df4e0b23e92bf72ce93ac25022cdbddc4. The real broker stopped the candidate process, restarted Hiro at that rollback revision, and recorded rolled_back after health verification.
The rollback revision's complete tree matched the pre-candidate tree and the candidate probe artifact was absent. The queue then finished at rejected while the transaction remained terminal at rolled_back.

Ordered transition evidence

Completed

Both scenarios produced one ordered trace combining persisted queue snapshots, append-only promotion transaction events, transaction identity, baseline revision, intended candidate revision, active runtime revision, and failure reason where applicable.
The golden trace contains 19 ordered records ending with queue implemented and transaction finalized.
The rollback trace contains 19 ordered records ending with transaction rolled_back and queue rejected.
The complete machine-readable result retains raw queue records, queue events, promotion transaction records, promotion transaction events, full-suite receipts, runtime identities, and the first-failure field, which is null for the successful run.

Bounded first-failure repairs

Completed

The first run stopped before candidate creation because the fresh isolated API lacked the local TLS certificate directory required by the real startup path. The harness was changed to link that required local runtime asset without copying or publishing it.
The next clean run reached the governor but its long generated fixture path caused path-sensitive Windows Git and pytest failures: 734 tests passed and 36 failed. The same retained candidate tests passed under a short temporary root.
The harness fixture root was shortened under _hpi, and both scenarios were restarted from fresh clones. No downstream speculative repair was made.
No defect in the production promotion, activation, verification, probation, or rollback implementation had to be changed for the final proof to pass.

Bounded integration and normal runtime restoration

Completed

Confirmed the regular live checkout was clean and had no active promotion transaction or active candidate lifecycle state before integration.
Fast-forwarded exactly docs/hiro_promotion_integrity_harness.md, hiro/improvement/promotion_integrity_harness.py, and tests/test_promotion_integrity_harness.py.
Restarted Hiro through the checked launcher and verified status ok, model connected, a clean checkout, and matching loaded and checkout revision 1b44ff1bd2076e774138ab87f2c6a0d4a8bbbe53.

Decisions and reasoning

Treat the process proof as the objective and refuse to substitute passing units, plausible code fixes, or mocked activation receipts for real runtime evidence.
Begin the harness with a deterministic acceptance candidate so discovery quality, external corroboration, candidate generation, and scoring cannot obscure the promotion boundary under test.
Use the current continuous queue and promotion transaction state machines as authoritative; retain historical Stage 6 implementations only as excluded context.
Keep the production governor full-suite gate intact even though it makes each scenario take about seven minutes.
Use a controlled logical clock for probation timing while retaining real process and revision verification at every production checkpoint.
Accept additive rollback semantics: the restored runtime uses a new revert SHA, and restoration is proven by both loaded-revision identity and exact tree equality with the pre-candidate baseline.
Repair only harness prerequisites at the first failing boundary. Do not alter downstream production promotion logic when no failing production transition has been observed.

Validation and evidence

Check	Status	Result
Focused harness, transaction, governor, and continuous-engine tests	passed	70 tests passed in 11.83 seconds after the final fixture-path repair.
Golden candidate governor gate	passed	770 tests passed with four existing unknown-marker warnings in 431.43 seconds.
Golden process activation and finalization	passed	Candidate 9457d565e822aceb563464e166e200e062aeb32a became the loaded and checked-out isolated runtime, completed all probation checkpoints, and finished as queue implemented plus transaction finalized.
Rollback candidate governor gate	passed	770 tests passed with four existing unknown-marker warnings in 434.51 seconds.
Forced post-activation failure and rollback	passed	Candidate febd058dc7864d4d6c61c12667dc9062f2df8d7d was verified active before the forced failure; rollback revision fbd0a80df4e0b23e92bf72ce93ac25022cdbddc4 was then verified running with a tree identical to baseline.
Normal Hiro restoration and harness integration	passed	The regular runtime is healthy and connected with matching loaded, checkout, and Git revision 1b44ff1bd2076e774138ab87f2c6a0d4a8bbbe53 and a clean working tree.
Public journal tests and build	passed	npm run test:hiro passed. npm run build generated and validated all 165 journal pages, compiled TypeScript, and completed the Vite production bundle.

Current state

Hiro is online and reports status ok with its configured Qwen model connected.
The live checkout is clean at 1b44ff1bd2076e774138ab87f2c6a0d4a8bbbe53, and the runtime reports that same loaded and checkout revision.
A reproducible process command now proves both the production golden path and production rollback path.
The successful machine-readable result has no first failure and retains 19 ordered trace records for each scenario.
No production promotion implementation was changed in order to obtain the passing result.

Next steps

Use the documented one-command harness whenever promotion-boundary integrity is questioned, and treat any nonzero exit as an exact first-transition failure rather than beginning another broad review.
If live Hiro still produces no promotions while this harness passes, investigate only the earliest upstream state where real candidates stop before canary eligibility; do not reopen the already-proven activation boundary without contrary evidence.
Preserve the successful trace format when future promotion states change so candidate identity, transaction identity, and active runtime revision remain jointly auditable.
Keep unrelated candidate-quality, scoring, source-discovery, and benchmark improvements outside promotion-integrity repair sessions.