Shared HF accounting and recovery

The hf_managed profile uses an external, writable PostgreSQL database for authoritative application state and encrypted Hugging Face objects for files. It is a separate deployment path from the native single-controller HF profile. These accounting changes do not convert a native Space into a replicated deployment. Use the managed migration and deployment guide first.

Defaults and online policy changes

The default remains conservative_lifetime. New writes collect immutable operation identities and durable write-closure receipts in either mode. In conservative mode, deletion or completion does not automatically refund the deployment storage allowance.

A platform administrator can select safe_lifecycle at /admin/hf-accounting. The policy is held in shared PostgreSQL and takes effect across replicas without a restart. Each change requires the observed policy generation and an audit reason. An organization owner or administrator does not receive platform administration authority. The HTML, script and every API use the existing authenticated platform role, same-origin checks and CSRF protection.

Safe lifecycle accounting:

• settles a future attempt's maximum output reservation only after its provider execution is durably terminal and its exact immutable output inventory is verified;
• releases temporary transfer storage only after the publication or discard authority, writer closure and exact physical absence have been established;
• releases artifact and private-state storage only after all live references have ended, the publisher is closed, physical deletion is verified and the database recovery protection described below allows deletion;
• updates the operation receipt and aggregate charge in one database transaction;
• retains a permanent SQL tombstone so a retried admission cannot recreate a settled or released immutable payload.

Switching back stops new credits; completed credits remain valid. Policy changes never recalculate usage from a bucket listing or reset the quota. A stale policy generation cannot authorize a new transition. Receipts with uncertain remote writes remain charged until independent writer-closure evidence is available.

The accounting ceiling measures admitted HF object bytes. The size, replication, indexes and backup retention of the external PostgreSQL database require their own provider capacity and monitoring. SQL audit/tombstone storage is not billed against the HF object meter; it grows with operation history and must be included in database sizing.

Migration and historical charges

Pause and drain the whole managed fleet before applying the release's schema upgrade. Serving startup validates existing schema and accounting state and does not create an empty replacement ledger. The explicit migration calls initialize_managed_accounting_schema(engine) after the imported meter exists.

The application database role must be able to inspect its PostgreSQL cluster identity and timeline. The database administrator grants only the needed functions, not superuser access, for example (replace pf_app with the actual application role):

GRANT EXECUTE ON FUNCTION pg_catalog.pg_control_system(),
  pg_catalog.pg_control_checkpoint() TO pf_app;

Serving startup verifies these reads before admitting users. A provider that does not expose these PostgreSQL identity functions is not qualified for this managed lifecycle implementation. Changing hosts, restoring a snapshot, or promoting a different timeline does not authorize interpreting an old publisher XID in a new timeline: affected private-state receipts stay held for explicit recovery reconciliation.

Before enabling managed artifact or private-state reclamation, generate an external recovery epoch and set the same value on every replica:

python -c "import uuid; print(uuid.uuid4().hex)"
# Set PF_HF_MANAGED_RECOVERY_EPOCH to that 32-character lowercase hex value.

This is an operator-controlled deployment setting, not a secret and not a value restored from the database. Without it the application can retain its normal conservative operation, but safe artifact/private-state credit remains blocked. Setting it before new private-state writes allows those receipts to be eligible later; old receipts with a missing or different epoch are never automatically rebound.

For every database restore, fork or replacement, stop and fence the entire old fleet, choose the supported matching database/object recovery point, generate a new epoch, and configure it on every replacement replica before any writer starts. Rotation invalidates the previous recovery-floor attestation and old private-state/deletion proofs. Review the provider's new recovery inventory and record a new floor before allowing reclamation. Existing charges stay held until their new authority is explicitly reconciled; rotation never refunds or resets usage. Do not copy an old epoch into a restored deployment.

A byte-for-byte physical clone using the same database identity, timeline and deployment settings cannot be distinguished from the original by application software alone. The external epoch rotation is therefore a required recovery runbook step, in addition to the automatic cluster/timeline checks.

Both native meter schema 1 and the Qualified lifecycle patch's schema 2 are accepted. Import preserves the exact total, ceiling, WAL reserve, every active reservation, lifecycle transition and the complete native meter provenance. Schema 2 policy and generation are preserved. Native encrypted retirement receipts and closed recovery-generation fences continue preventing replay after cutover. Preserve the original encrypted object namespace and state key.

An existing managed database is upgraded additively. Every pre-upgrade managed reservation is classified as legacy; no existing byte is automatically removed. Previously enabled recovery protection stays enabled when importing a native meter whose lifecycle mode has since been turned off. The same native snapshot may be imported again after an interrupted migration, but a different snapshot cannot replace an existing import. Audit and residual usage that cannot be attributed exactly remain charged.

The administrator's reconciliation preview matches historical operation identities, not equal byte amounts. The current verifier can reconcile retired compact output reservations with the historical :bounded-output-v2 identity, a durably stopped provider execution, publication/discard authority and verified absence of both exact output keys and the output prefix. Applying the reviewed batch recomputes the evidence and checks the review digest. Unverifiable entries, old per-file formats without retained inventories, and other unexplained charges remain held. This is intentional preservation of evidence, not a claim that all historical allowance can always be recovered.

Private-state receipts created on the native database retain their native transaction identity. Managed GC does not interpret those XIDs in the external database, or infer closure from coincidentally equal XID numbers. They remain held for separately verified migration reconciliation. New managed private-state receipts use the new shared database publication authority.

External database recovery protection

An older database backup can still reference a file that the current database has deleted. SQL commit alone is therefore insufficient evidence for deleting its encrypted object. Managed accounting records an immutable retirement time for each artifact allocation or private-state object and waits for a verified recovery boundary strictly later than that retirement.

The database backup coordinator or platform operator must:

1. Inspect the external provider's complete retained backup/PITR inventory and the native recovery roots retained during cutover.
2. Retire or make permanently unavailable every recovery point older than the proposed boundary. Fence/decommission any previous database authority.
3. Verify that the current external recovery epoch matches every active replica. Preserve the reviewed inventory as operational evidence and calculate its SHA256.
4. Use Database recovery protection in /admin/hf-accounting to record the oldest recoverable UTC timestamp, inventory reference, evidence SHA256 and reason, explicitly attesting to the retirement of all older roots.

PF binds this authenticated operator attestation to the current external epoch. It does not independently query a cloud backup provider, and it does not infer backup retirement from elapsed time, an empty object listing or a retention setting. An automated backup coordinator may use the same authenticated administrative operation only after independently verifying the provider inventory. Merely advancing a date to obtain a quota refund is unsafe.

The API is POST /api/admin/hf-accounting/recovery-floor and accepts:

{
  "floor_utc": "2026-10-09T12:00:00Z",
  "expected_generation": 0,
  "evidence_sha256": "64-lowercase-hex-characters-from-your-reviewed-inventory",
  "source": "Your provider inventory evidence reference",
  "native_roots_retired": true,
  "all_older_recovery_points_retired": true,
  "reason": "Explain the independently verified backup retirement"
}

The example hash is descriptive and must be replaced with a real SHA256. A future timestamp, stale generation, missing attestation or backwards boundary is rejected. First enabling protection also waits for every previously admitted conservative physical purge to confirm completion. The pending purge journal survives a process crash and is completed only by an exact-scope deletion retry. After a valid boundary advances, bounded background maintenance can delete and credit eligible objects. No boundary means no safe automatic artifact or private-state reclamation. Output settlement can still recover unused maximum-output allowance because it does not delete retained file data.

Recovery protection is permanent once enabled: switching quota policy back to conservative does not disable it. A never-enabled conservative deployment keeps its prior physical purge behavior and issues no lifecycle credits.

Never restore an older database snapshot against a namespace whose referenced objects have been reclaimed. Recovery must select a point at or after the recorded boundary with its matching object namespace. Restoring or replacing database authority is a controlled fleet-wide recovery/migration, not an online connection-string edit.

Administration and operational evidence

GET /api/admin/hf-accounting returns shared policy, charges, recovery boundary, receipt counts and execution recovery status. GET /api/admin/hf-accounting/audit provides bounded pages of policy, credit, reconciliation and recovery-boundary events. Use the returned next_cursor to traverse retained events; IDs are pagination identities rather than chronological sequence numbers.

Database admission, lifecycle changes and audit writes share a consistent lock order. Remote imported-receipt checks finish before the aggregate meter lock. Operator receipt counts use grouped SQL queries; immutable native provenance is not reread for every ordinary admission. Background cleanup uses bounded work and shared maintenance coordination across replicas.

The included tests cover exact accounting, policy fencing, legacy permits, platform access, migration preservation, writer/reference/recovery barriers and real PostgreSQL concurrent credit. Run the PostgreSQL suite with PF_TEST_POSTGRESQL_URL pointing to an isolated qualification database. It creates and removes only its own randomized schemas. Unit tests and these integration tests do not prove a particular public-user capacity or qualify a provider's backup inventory. Those require the release's workload and deployment qualification process against the actual provisioned environment.