Existing HF installation: upgrade and recovery

For an already running native HF deployment, follow section 11 of the native HF guide. It supplies the complete Windows CMD and HF website sequence. This companion summarizes the preservation and acceptance requirements. New installations start at section 1 of that guide and use the full r34 release. The concise current upgrade route is R34_NATIVE_HF_RELEASE_UPGRADE.md. The consolidated release includes all approved native fixes plus shared changes; there is no historical patch installation chain.

1. Record the current installation

Where: secure operator records and the HF website. Record the controller, private code and public runtime Space names, state Bucket and prefix, immutable runtime digest, private code commit and code SHA256, controller commit, and current configuration. Use the deployment record template (source package reference: HUGGING_FACE_DEPLOYMENT_RECORD.md). Keep existing database, session, state-encryption and BYOK/BYOS keys in your private secret store. HF does not reveal secret values after they are saved.

Preserve the existing Bucket, prefix and stable secrets throughout an upgrade. Keep PF_HF_BOOTSTRAP absent for an initialized service. Preserve the approved PF_HF_MAX_STORAGE_BYTES value (16 GiB is 17179869184) and encrypted quota reservations/usage. Never lower it to a fresh-install example or reset the meter. The new code restores and migrates its existing database; creating a new prefix or keys is a different installation and does not recover the old one.

2. Pause and make a verified recovery copy

Where: PF/HF website, then Windows CMD. Close admission to new work, allow active runs to finish or cancel them, and verify their terminal provider state. Confirm that there is no other controller using the same Bucket and prefix. Pause the controller and require PAUSED before copying durable state.

Where: HF website. Create a new, empty, private backup Bucket in the source Bucket's storage region. Choose a unique name for each maintenance window. pf-simulation-state-backup-20261001 is an example name, not a required name or a Bucket to reuse. Give the local HF token read access to the source and read and write access to this destination, while retaining its deployment scopes.

Where: Windows CMD. Use the guide's HF Bucket-to-Bucket copy and verification commands. Keep the controller paused until source, destination and a second source listing agree on every file path, byte length and Xet hash, the source is nonempty, and the destination is private. Record the file and byte counts. The backup uses HF authentication and requires no additional S3 configuration. Preserve the existing runtime storage credentials.

A verified copy is a recovery checkpoint. It does not prove that a restore into a separate environment has succeeded; record any restore drill separately.

3. Generate and qualify the replacement

Where: Windows CMD with the tools environment active. Verify the full archive against its accompanying SHA256 file and extract it into a clean directory. Keep the preceding release and its upload folders available. Use the packaged deploy\saas\huggingface_native\build_space.py with a new output directory; it generates runtime, jobs and space together. Check that the private code manifest and controller base-code manifest report the same generated SHA256.

Use the packaged deploy_release.py upload-code command for the generated jobs/ contents in the existing private code Space. Record its exact returned code_revision and code_sha256; it verifies the committed remote inventory. Use those exact identities with the compatible, qualified, immutable public runtime image. Use the generated runtime context to establish the dependency contract, including FMU and QuTiP. Build and qualify a changed public runtime and record its new digest. An unchanged compatible runtime may be retained only when its context and image provenance are verified as described in section 11.4.

Preview the pinned qualification specification and then submit the reviewed self-test. --submit starts a billable HF Job. If a Job ID is returned, inspect that existing Job before considering another submission. Require completion and both confinement_probe_passed=true and scientific_runtime_probe_passed=true for the newly uploaded code. A passing self-test is one acceptance item, not a full SaaS or restore qualification.

4. Apply matching controller settings and code

Where: HF website or the explicit HF CLI commands in section 11. Windows CMD set commands only configure the local terminal. Save the new private-code repository, commit and SHA256 in the controller's HF Variables, retain its qualified runtime image, and set or retain the bounded receiver-startup setting. The default PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS is 600 seconds; its allowed range is 30–3600. Verify the saved non-secret pins before uploading.

Keep the existing state binding and stable secrets, with PF_HF_BOOTSTRAP absent. Use the packaged deploy_release.py verify, prepare, apply, then status sequence for the matching generated bundle. Preparation verifies exact code pins; apply confirms pause before a parent-guarded commit, removes only recognized obsolete release-owned files and restarts. It refuses unexpected files or changed public Variables/revisions. Settings may themselves trigger provider restarts, so keep researcher access closed throughout maintenance. Inspect status before retrying an uncertain write; it can reconcile an exact committed bundle without repeating the mutation.

5. Verify recovery and reopen access

Require RUNNING, Application startup complete and the HTTP server startup message. Then open the actual direct .hf.space address in a normal browser tab. Sign in with the existing PF administrator account, open a previously saved project and download a known artifact. Run a small simulation through result collection, inspect its plots/report and verify a newly saved record survives a controlled restart. Test cancellation on a separate bounded run and apply the remaining authorization and operational acceptance checks in the guide.

Reopen researcher access only after the applicable checks pass. Use the guide's troubleshooting section if build or startup does not finish successfully; retain the state and configuration while identifying the failing phase.

Recovery and rollback

Retain the pre-upgrade backup and its matching code/configuration records. The consolidated shared application schema is 10, upgraded during native startup. A schema-9 or older controller cannot run against it. Reverting a controller commit alone does not reverse that migration. After startup has migrated data, use a reviewed forward repair or restore a separately verified pre-upgrade state with its complete matching old configuration/code. Account explicitly for writes after the backup; do not present code checkout as data rollback. Validate recovery in a separate isolated installation with the recorded compatible code and keys, without running a second writer against the production Bucket and prefix.

The full release includes those earlier compatibility fixes and the approved native latency, startup, response, runtime, WAL batching, ownership queue and persistence response recovery corrections. Follow the current full-release sequence rather than assembling those corrections separately.