Assistant execution recovery

The native HF, managed HF, and distributed Docker profiles run recurring recovery of abandoned assistant operations. This is a lifecycle service, not a script tied to one operation ID. A queued simulation can use an execution slot again only after recovery has proved that the assistant execution occupying it has stopped. An expired lease, timeout, cancellation request, or empty provider listing is insufficient proof.

Recovery marks an abandoned running assistant operation interrupted, clears its temporary compute and storage reservations, records the retained private byte count, and writes an audit event in the same database transaction. Unknown elapsed compute is charged conservatively at the reserved allowance. Recovery does not delete retained private files. A lost response can be retried: the current database state prevents a second settlement. Diagnostics distinguish a confirmed accounting commit from a later owner-release failure and omit exception text, SQL, credentials, and private content.

HF profiles

Native HF retains its controller fence in addition to the private profile lease and deployment-reservation lock. Managed HF uses shared PostgreSQL profile leases, reservation-row locking, and the durable HF child-job journal; it does not pretend to own the native single-controller fence. Both inspect exact provider job identities and fingerprints. Confirmed terminal evidence is retained so recovery does not depend on the provider keeping completed jobs forever.

Managed startup runs a bounded sweep before starting worker dispatch. The application lifespan continues scheduled fleet maintenance. The accounting administrator page exposes the same evidence-based recovery service; ordinary users and organization administrators cannot invoke platform-administration APIs. Retrying one operation applies the same checks as background recovery and has no force-refund switch.

Distributed Docker

Every new assistant execution reservation enrolls in the durable recovery protocol before submission. Each Docker child is journalled before launch and carries labels binding it to its exact reservation and child ID. The journal stores a hash of the Docker daemon identity. Another host cannot use its own absent-container result as proof for that execution.

The shared private directory must provide coherent POSIX flock, atomic rename, and fsync across every replica. A per-profile kernel lock complements the SQL publication lease. A still-running process retains this lock even when its SQL lease expires; recovery cannot reconcile its files concurrently. Do not delete .owner-fences files or replace their storage during execution. An object-storage mount without these filesystem guarantees is unsupported.

Once recovery holds the profile lock and SQL lease, it can inspect and remove an abandoned child's exact authenticated Docker object. The immutable container ID is committed before forced removal. A retry can therefore distinguish a known object's permanent absence from an unbound name that might still be created by a delayed Docker request. Charges stay reserved if the daemon is unavailable, identities disagree, or submission remains uncertain. Run a recovery-capable API replica on each Docker daemon with unfinished children; draining a host must retain that capability until its children settle.

Older ambiguous Docker reservations have no durable child journal. The upgrade preserves them and reports submission_unresolved; it does not invent execution evidence or silently refund them. An operator must establish stopped execution and use the deployment's audited reconciliation procedure. Existing finished/unsubmitted reservations retain their durable terminal/no-submission meaning.

Bounds and rollout

The shared settings PF_SAAS_ASSISTANT_RECOVERY_BATCH_SIZE (default 8, maximum 32), PF_SAAS_ASSISTANT_RECOVERY_MAX_SECONDS (default 15, maximum 60), and PF_SAAS_ASSISTANT_RECOVERY_INTERVAL_SECONDS (default 15) bound distributed and managed maintenance. A single in-flight provider call may extend a sweep past its selection deadline; every provider call has a separate timeout. Currently valid private leases are excluded in SQL as a candidate-selection optimization. Every selected operation still receives the full ownership and termination checks.

Apply the additive SaaS schema migration before starting distributed replicas. Stop or drain old API processes before starting processes that use the new filesystem lock protocol. Keep the same database, private storage, encryption keys, and Docker daemon identities during upgrade. Local protocol tests cover crash and retry behavior; they do not qualify production provider behavior, shared filesystem guarantees, or large-scale capacity. Include mixed-workload and failure-recovery testing on the intended infrastructure before public rollout.