Docs/Supervision (zirv ctx)

Cross-harness fallback and handover

Beyond waiting out a subscription window, zirv can steer work onto a different harness. fallback.rs connects the agent roster, usage windows, model-tier ladder, and delegation path: a new delegation (zirv ctx agent, a dashboard spawn) can be rerouted away from an exhausted or measured-low-headroom harness before it ever starts, while an already running supervised session can only move once that harness itself stops on a recognized usage-limit message — steering never interrupts a session that is still making progress. Either way the alternate harness must be enabled, ready, capacity-compatible, budget-compatible, and able to provide a verified equivalent model tier (cheap/standard/deep); zirv never guesses a tier translation for an operator-pinned model it cannot verify.

zirv ctx status reports the resolved policy and, for each harness in the fallback order, its readiness, capacity and current headroom:

fallback: enabled | order claude -> codex | steer below 20% headroom | candidate min 10% | unknown assumes 25%
  fallback claude: enabled / ready / full / 62% measured
  fallback codex: enabled / ready / small-only / 25% assumed

Each harness line is {enabled|disabled} / {ready|unavailable} / {small-only|full} / {headroom}: whether .settings.toml/ZIRV_AGENT_<NAME>_ENABLED has that adapter on, whether it currently resolves and is ready to launch, whether [agents] capacity-limits it to small tasks only, and the same measured/ assumed/opted-out headroom reading described above.

Configure it under [fallback] in ~/.zirv/ctx.toml:

[fallback]
enabled = true
order = ["claude", "codex"]
predictive_headroom_pct = 20.0        # steer new work below this headroom
min_candidate_headroom_pct = 10.0     # a candidate needs at least this much headroom to accept work
unknown_headroom_pct = 25.0           # assumed headroom when no reading exists (0 opts out)
small_task_max_tokens = 40000
small_task_max_tool_calls = 24
adaptive_delegation = true            # issue #358: route new delegations through the pure capacity allocator
auto_orchestrator_rollover = false    # issue #358: let the orchestrator seat itself roll over automatically (off by default)
orchestrator_rollover_headroom_pct = 20.0  # issue #358: threshold that arms a proactive rollover (defaults to predictive_headroom_pct)
rollover_cooldown_secs = 600          # issue #358: minimum gap between two automatic rollovers
reactive_force_after_secs = 120      # issue #401: grace before a confirmed block forces a mid-turn structural handover

[fallback.harness.codex]              # issue #358: per-harness overrides, both optional
max_active = 3
reserve_headroom_pct = 15.0

A repository checkout may only narrow these values (see Trust boundary above); ZIRV_CTX_FALLBACK* environment variables are the operator's final override. orchestrator_rollover_headroom_pct, rollover_cooldown_secs, and reactive_force_after_secs are repo-forbidden outright (see the table above) — tuning an already-enabled rollover's timing is an operator decision, the same as handoff.model.

Failed dashboard rollovers preserve the running session. An automatic harness-to-harness successor starts separately and acknowledges the handoff before the dashboard retires the source. Until then the original process, subagents, screen and seat generation remain intact. A successor that exits (including CLI exit 2), times out, or cannot commit its seat is discarded; the source is not relaunched or sent a continuation prompt. Codex readiness requires a completed assistant reply in its new rollout, not a quiet startup screen. Other harnesses must provide a turn-completion signal for this automatic transfer. New operator input cancels the pending transfer. Handoff lookup uses the harness's observed conversation ID; a missing user task prevents an automatic transfer.

Failed attempts persist last_rollover_at and a consecutive failure count. Automatic retries require newer usage evidence and wait at least max(rollover_cooldown_secs, 60) seconds, doubling after each failure up to one week or the binding window reset, whichever comes first. A successful transfer clears the count. These timing controls remain operator-only.

The displaced harness is parked, not closed. A successful rollover quits the source child, but quitting a harness does not destroy its conversation. The seat records the harness it was rolled off, together with that harness's OWN conversation id (observed at a turn boundary, which is the only place zirv's session uuid and the harness's conversation id are both visible), and keeps it until the seat comes home. When that harness reads healthy again — its window reset, its endpoint recovered — the reclaim path returns the seat to it and resumes that same conversation rather than starting the operator over, with the interim harness's own distilled handoff packet delivered as the resumed conversation's first message: continuous conversation, plus a record of everything that happened while it was parked. The first displacement wins, so a seat that hops twice still wants its original harness back; an operator's own zirv ctx handover owes no return. A return is only taken on a harness that can resume and accept a prompt in the same launch (claude's --resume <id> "<query>"); anything else keeps the previous behaviour — a cold launch carrying the packet — because resuming a conversation without telling it what happened in its absence would silently drop the interim's work.

A rollover can cross runtimes, and the seat keeps its identity. The same prepare/commit/abort transaction moves a seat between backends as well as between harnesses — harness→harness, harness→native, native→harness, native→native — and the seat's short id and generation lineage are exactly what does not change. What changes is which backend answers at that address, which zirv ctx status names inline (seat: claude opus [native] gen 3) alongside the harness or route the seat was displaced from and a one-line rollover record: trigger, direction, every route tried with why it was refused, and whether the seat committed, kept the original session (preparation failed, so the seat never moved), or parked honestly with all its durable state intact. The record lives at <state>/sessions/<short>.rollover.json.

Before a successor is prepared at all, a native source reaches a safe boundary: in-flight work is drained or explicitly cancelled, any tool effect that began and never reported is carried as outcome-unknown (never retried, never called failed), and one portable checkpoint — acknowledged input with whatever is still owed, task claims, completion receipts, outstanding tools, evidence — is committed to the journal. An outcome-unknown effect halts the successor until it is reconciled: the whole point is that a fresh model must not re-run it. Portable checkpoints stay strictly separate from a provider's own continuation envelope, and only a native successor on the identical route may keep that envelope; every other direction rebuilds a legal semantic history, and a coding-harness successor never sees another vendor's envelope at all.

Two rules are worth stating because they are refusals rather than fallbacks. A successor is validated — policy, capabilities, context room, billing authority, authentication, budget, startup — before the source is given up, and a failure keeps the original session. And a rollover is only authorized to move work onto the billing posture the seat already spends (plus local runtimes, which have no credential and no invoice): moving subscription work onto metered API credit is a decision an operator makes, never a silent consequence of capacity. Native subagents a rolling seat owns are finished, stopped, or retained under recorded ownership — zirv has no mechanism that migrates a running worker to another seat, so it never claims one.

Only one seat generation is write-capable at a time, and that is enforced rather than assumed: a stale generation is refused at native tool effects (the journal's own fence), at wrapped tool effects (the pre-tool hook), at zirv agent delegation, at the coordinator's plan graph and at the per-tree writer lease. In the window between prepare and commit the source still holds the seat, so a successor that has been validated but not committed cannot write either — which is what makes a crash at either boundary incapable of producing two writers.

Cross-harness capacity, at a glance. zirv ctx status (and zirv ctx status --json for machine-readable output) reports a pool section built on the same pure allocator: each harness's scheduling state (ready / draining / hard-blocked / unknown / disabled), its provider's headroom after outstanding reservations, and — when auto_orchestrator_ rollover is on — the orchestrator seat's own fencing generation. Provider token reservations are tracked in a small durable ledger (<state>/reservations/<provider>.json) so two admitted-but-unsettled delegations against the same billed account are never double-counted (capacity snapshot formula, the fenced rollover transaction, anti-flap rules).

Health-aware routing#

Usage headroom answers may this account spend more. It says nothing about whether the endpoint can be reached: a session that dies on API Error: Connection refused has full headroom and zero capacity. So alongside the headroom rules above, zirv keeps a small circuit breaker per harness, fed by the structured provider-error rows a transcript already carries. The harness is the whole identity: the failing hop in a transport or server failure is the connection or the endpoint, not the model, so an open breaker on one model would have to steer work away from the others anyway. The last model seen is recorded alongside it, as information for status only.

Six phases, per route:

1. healthy — routed to normally.
2. suspect — some transport/server failures inside window_secs, still routed to.
3. degraded — reachable, but measurably worse than it should be; still routed to, just ranked behind every healthy alternative.
4. open — open_after_failures reached: excluded from routing for cooldown_secs.
5. half-open — the cooldown elapsed; exactly one trial is admitted.
6. unavailable — an authentication, permission or model-not-found error; denied at once rather than after open_after_failures, then re-probed on the same cooldown_secs so a credential you have since fixed heals on its own.

Only transport (connection refused/reset, DNS, proxy, socket hang-up, timeouts) and server (overloaded, internal server error, service unavailable, bad or timed-out gateway) failures count towards opening a breaker. unavailable needs an explicit provider token (authentication_error, invalid_api_key, permission_error) or a structured 401/403/404 — never ordinary English, so a sandbox saying "permission denied" about a file cannot deny a vendor route. A rate limit is capacity, which the headroom rules above already own; a context overflow belongs to the session, which rot owns; anything unattributed changes nothing. Classification is gated on the structured rows only (Claude's isApiErrorMessage, codex's task_complete.error), so an agent merely writing about an outage can never trip a breaker.

Degraded is the soft half of the same idea, and the only phase that reduces a route rather than excluding it. Two signals can reach it, both read over window_secs:

• a rolling transport/server error rate at or above degrade_error_rate_pct, once the window holds at least degrade_min_samples dated turn outcomes (failures plus completed error-free turns);
• a first-token latency whose median is at or above degrade_ttft_ms, over the same minimum sample count. Opt-in: degrade_ttft_ms is unset by default.

Leaving degraded takes more than crossing back over the line: the error rate must fall below half degrade_error_rate_pct, or the latency median below three quarters of degrade_ttft_ms, or the evidence must age out of the window entirely. Without that hysteresis a route sitting on the threshold would flap in and out on every poll. A single completed turn never clears a degradation on its own — a rate is not a turn.

The latency signal is off by default because of what a transcript can actually measure. Neither harness records a first-token time, so it is derived from row timestamps: the gap between the user row that opened the turn and the first assistant row after it. That interval includes the model's own thinking, so a reasoning model reads "slow" on a perfectly healthy endpoint. Inter-token latency is not measurable from either transcript at all. So latency only ever reduces a route: it never opens a circuit, never marks a route unavailable, and never migrates a session.

A degraded route is reduced, never excluded. It loses every comparison to a healthy candidate regardless of headroom, and zirv agent steers a delegation off it — but only when a healthy alternative exists. If every route is degraded, the work stays where it was asked for: trading one degraded route for another is churn, not a reroute. A degraded seat route is not a rollover trigger either.

Shared endpoints. When [endpoint.claude] and [endpoint.codex] are both pointed at one gateway, the two routes have one dependency, and failing over between them just buys a second failure. So a breaker that opened on a transport or server failure also denies any other harness whose configured base_url resolves to the same host[:port], naming the route that actually failed. Only a configured endpoint override defines a shared dependency — native default accounts are always independent, because zirv knows nothing about the estate behind them. Authentication failures never propagate (a rejected credential belongs to one account, not to the host), and neither do rate limits, unavailable, or degradation. The dependency is only ever the host and port: no credentials, no paths, no full URLs reach a log or a status row. An alias denial excludes placements only — the seat sitting on the aliased route keeps working and never rolls over for its sibling's failure.

One trial at a time. A half-open route admits exactly one recovery probe, so two dispatches landing in the same cooldown cannot both "test" a broken endpoint. Whoever gets there first claims the trial, recorded as one health-trial line; anyone else is told who is holding it, and when it frees itself, and re-runs placement once with that route excluded. A dispatch that still has nowhere healthy to go is refused rather than launched onto the route that was already being probed. The claim expires by itself after cooldown_secs, so a probe that never reports back cannot hold a route shut. A claim attempt that finds the route's own health record momentarily locked by another process is refused the same way rather than waiting on it — the lock holder may be halfway through claiming the very same trial. A trial that succeeds recovers the route and clears the claim; one that fails reopens the breaker and clears it too.

An open route is excluded from zirv agent's automatic reroute, from the orchestrator seat's successor choice, and from the pool view's candidate ranking — carrying its own reason, not a bare hard-blocked. An open breaker on the seat's own route is itself a rollover trigger, and takes the reactive path, so the handoff is the host-computed structural packet: zirv never asks a route that just refused connections to distil a handoff. That structural packet also reconciles a partial stream (issue #455): a tool call whose result never arrived by the end of the scanned transcript is listed as UNRESOLVED rather than silently dropped -- with any file it claimed to modify marked (unconfirmed) -- and a reply cut short by the failure is withheld from Done and reported as PARTIAL instead of standing in as a finished answer, so the successor checks the real outcome before repeating or assuming away the side effect. Codex's own rollout carries no per-turn "stream ended mid-generation" marker, so its cut-tail detection only covers a task_complete that itself reports an error, not a raw stream cut with no completion event at all. If every route is denied, the seat parks exactly as it does when every account is exhausted, and the park reason names the health failures. State lives in <state>/health/<harness>.json — delete that file to reset a route by hand; there is no verb for it, and every phase heals on its own anyway. Each phase change is one health-open / health-half-open / health-recovered / health-unavailable line in the decision log, never one line per failure.

open_after_failures must be 1..=20 (the size of the per-route observation ring), and both window_secs and cooldown_secs must be above zero; a value outside that is refused at config load rather than leaving a breaker that can never trip. The same applies to the degrade knobs: degrade_error_rate_pct must be 1..=100, degrade_min_samples at least 2, and degrade_ttft_ms, when set at all, at least 1000.

Observations are stamped with the transcript row's own timestamp, not the clock, and de-duplicated by the row's own id — so re-reading a transcript from the start (a new supervisor, a rebuilt scoring checkpoint) can neither fold old failures into the current window nor count the same failure twice. That holds for every class, unavailable included: an authentication error older than window_secs is history and changes nothing, and healing a route clears its phase without forgetting which rows it has already counted. A row whose transcript states no timestamp at all is only ever counted when the poll that read it was reading genuinely new bytes. Both the interactive supervisor and the headless exec/loop supervisors feed the same records, each under the route's own lock, and a supervisor that finds the lock held skips the fold rather than waiting on another process.

zirv ctx status shows a line per route that is not healthy (and nothing at all when every route is fine), plus the exclusion reason in the pool section:

pool
  health: claude open (3 transport/server error(s), last model opus; next health check ~unix 1700000300 (estimate))
  health: codex degraded (2 transport/server error(s); codex: error rate 25% over 8 turns)
  excluded claude: claude: route health open: 3 transport error(s) in 10m; next health check in ~5m (estimate)

A half-open route names whoever holds its trial the same way (half-open (0 transport/server error(s); trial in flight: 7f3a2b1c (12s))).

The retry time is always an estimate: it is when the breaker admits its next trial, not when the endpoint is known to be back. zirv ctx status --json carries the same rows under the pool view's health field, omitted entirely while every route is healthy.

Configure it under [fallback.health]:

[fallback.health]
enabled = true               # issue #455: master switch (also off whenever fallback.enabled is off)
open_after_failures = 3      # transport/server failures inside the window that open a route
window_secs = 600            # the sliding window failures are counted over
cooldown_secs = 300          # how long an open route stays excluded before one trial
degrade_error_rate_pct = 25  # rolling error rate that ranks a reachable route last
degrade_min_samples = 8      # dated turn outcomes the window needs before either degrade signal fires
# degrade_ttft_ms = 20000    # opt-in first-token p50 that degrades a route; unset by default

--force and an explicit pin bypass health exactly as they bypass headroom: health only ever adds exclusions, it never overrules a route you asked for outright.

zirv ctx handover performs the swap directly, on demand, mid-session — the same mechanism the dashboard's Ctrl+A o picker and automatic fallback routing both use underneath:

zirv ctx handover --agent codex --model standard
zirv ctx handover --model deep      # same harness, a different model tier
zirv ctx handover --dry-run          # print the resolved swap; change nothing

--model accepts a literal model id or a generic tier (cheap/standard/ deep), resolved per target harness — claude's deep is opus, codex's is gpt-5.6-sol, for example, each overridable via [handover.<agent>] in ~/.zirv/ctx.toml or ZIRV_CTX_HANDOVER_<AGENT>_<TIER>. The swap carries a distilled handoff packet across so the successor session picks up task continuity; by default it waits for a verified-idle turn boundary, and --force swaps mid-turn instead. handover (and handover.<agent>.<tier>) is repo-forbidden: swapping the orchestrator seat's harness or model picks which vendor account gets spent, so only the operator's own ~/.zirv/ctx.toml, ZIRV_CTX_HANDOVER_*, or flags may set it.