# Existing HF installation: upgrade and recovery

For an already running native HF deployment, follow
[section 11 of the native HF guide](HUGGING_FACE_NATIVE_DEPLOYMENT.md#11-upgrade-an-existing-r34-deployment).
It supplies the complete Windows CMD and HF website sequence. This companion
summarizes the preservation and acceptance requirements. New installations start
at section 1 of that guide and use the full r34 release. The concise current
upgrade route is [R34_NATIVE_HF_RELEASE_UPGRADE.md](R34_NATIVE_HF_RELEASE_UPGRADE.md).
The consolidated release includes all approved native fixes plus shared changes;
there is no historical patch installation chain.

## 1. Record the current installation

**Where: secure operator records and the HF website.** Record the controller,
private code and public runtime Space names, state Bucket and prefix, immutable
runtime digest, private code commit and code SHA256, controller commit, and
current configuration. Use the [deployment record template](HUGGING_FACE_DEPLOYMENT_RECORD.md).
Keep existing database, session, state-encryption and BYOK/BYOS keys in your
private secret store. HF does not reveal secret values after they are saved.

Preserve the existing Bucket, prefix and stable secrets throughout an upgrade.
Keep `PF_HF_BOOTSTRAP` absent for an initialized service. Preserve the approved
`PF_HF_MAX_STORAGE_BYTES` value (16 GiB is `17179869184`) and encrypted quota
reservations/usage. Never lower it to a fresh-install example or reset the meter. The new code restores
and migrates its existing database; creating a new prefix or keys is a different
installation and does not recover the old one.

## 2. Pause and make a verified recovery copy

**Where: PF/HF website, then Windows CMD.** Close admission to new work, allow
active runs to finish or cancel them, and verify their terminal provider state.
Confirm that there is no other controller using the same Bucket and prefix.
Pause the controller and require `PAUSED` before copying durable state.

**Where: HF website.** Create a new, empty, private backup Bucket in the source
Bucket's storage region. Choose a unique name for each maintenance window.
`pf-simulation-state-backup-20261001` is an example name, not a required name or
a Bucket to reuse. Give the local HF token read access to the source and read
and write access to this destination, while retaining its deployment scopes.

**Where: Windows CMD.** Use the guide's HF Bucket-to-Bucket copy and verification
commands. Keep the controller paused until source, destination and a second
source listing agree on every file path, byte length and Xet hash, the source
is nonempty, and the destination is private. Record the file and byte counts.
The backup uses HF authentication and requires no additional S3 configuration.
Preserve the existing runtime storage credentials.

A verified copy is a recovery checkpoint. It does not prove that a restore into
a separate environment has succeeded; record any restore drill separately.

## 3. Generate and qualify the replacement

**Where: Windows CMD with the tools environment active.** Verify the full archive
against its accompanying SHA256 file and extract it into a clean directory.
Keep the preceding release and its upload folders available. Use the packaged
`deploy\saas\huggingface_native\build_space.py` with a new output directory;
it generates `runtime`, `jobs` and `space` together. Check that the private code
manifest and controller base-code manifest report the same generated SHA256.

Use the packaged `deploy_release.py upload-code` command for the generated
`jobs/` contents in the existing private code Space. Record its exact returned
`code_revision` and `code_sha256`; it verifies the committed remote inventory.
Use those exact identities with the compatible, qualified, immutable public
runtime image. Use the generated runtime context to establish the dependency contract,
including FMU and QuTiP. Build and qualify a changed public runtime and record
its new digest. An unchanged compatible runtime may be retained only when its context
and image provenance are verified as described in section 11.4.

Preview the pinned qualification specification and then submit the reviewed
self-test. `--submit` starts a billable HF Job. If a Job ID is returned, inspect
that existing Job before considering another submission. Require completion
and both `confinement_probe_passed=true` and
`scientific_runtime_probe_passed=true` for the newly uploaded code. A passing
self-test is one acceptance item, not a full SaaS or restore qualification.

## 4. Apply matching controller settings and code

**Where: HF website or the explicit HF CLI commands in section 11.** Windows
CMD `set` commands only configure the local terminal. Save the new private-code
repository, commit and SHA256 in the controller's HF **Variables**, retain its
qualified runtime image, and set or retain the bounded receiver-startup setting.
The default `PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS` is 600 seconds; its
allowed range is 30–3600. Verify the saved non-secret pins before uploading.

Keep the existing state binding and stable secrets, with `PF_HF_BOOTSTRAP`
absent. Use the packaged `deploy_release.py verify`, `prepare`, `apply`, then
`status` sequence for the matching generated bundle. Preparation verifies exact
code pins; apply confirms pause before a parent-guarded commit, removes only
recognized obsolete release-owned files and restarts. It refuses unexpected
files or changed public Variables/revisions. Settings may themselves trigger
provider restarts, so keep researcher access closed throughout maintenance.
Inspect `status` before retrying an uncertain write; it can reconcile an exact
committed bundle without repeating the mutation.

## 5. Verify recovery and reopen access

Require `RUNNING`, `Application startup complete` and the HTTP server startup
message. Then open the actual direct `.hf.space` address in a normal browser tab.
Sign in with the existing PF administrator account, open a previously saved
project and download a known artifact. Run a small simulation through result
collection, inspect its plots/report and verify a newly saved record survives
a controlled restart. Test cancellation on a separate bounded run and apply
the remaining authorization and operational acceptance checks in the guide.

Reopen researcher access only after the applicable checks pass. Use the guide's
troubleshooting section if build or startup does not finish successfully;
retain the state and configuration while identifying the failing phase.

## Recovery and rollback

Retain the pre-upgrade backup and its matching code/configuration records.
The consolidated shared application schema is **10**, upgraded during native
startup. A schema-9 or older controller cannot run against it. Reverting a controller
commit alone does not reverse that migration. After startup has migrated data,
use a reviewed forward repair or restore a separately verified pre-upgrade state
with its complete matching old configuration/code. Account explicitly for writes
after the backup; do not present code checkout as data rollback. Validate recovery in a
separate isolated installation with the recorded compatible code and keys,
without running a second writer against the production Bucket and prefix.

The full release includes those earlier compatibility fixes and the approved
native latency, startup, response, runtime, WAL batching, ownership queue and
persistence response recovery corrections. Follow the current
full-release sequence rather than assembling those corrections separately.