# PF Simulation r34 — complete Hugging Face deployment guide

**Current release — 16 October 2026:** the full r34 archive includes [workflow reliability](workflow-reliability.md) and [assistant consolidation](assistant-consolidation.md). Verify this archive with `python release_tools/package_r34_assistant_consolidation.py verify --package`. Follow the [native upgrade guide](native-upgrade.md) for an existing service. The dated sections below retain their original maintenance scope; their earlier verification commands are historical.

**2026-10-11 maintenance:** read the [tracking recovery guide](../R34_TRACKING_RECOVERY_GUIDE.md)
for the merged preparation/cache, log buffering and controller diagnostic fixes.
The [execution recovery guide](../R34_EXECUTION_RECOVERY_GUIDE.md) remains applicable
to the earlier Aer sandbox, live-log and Orbit Viewer corrections.
The package verifier for that retained 11 October revision was
`python release_tools/package_r34_tracking_recovery.py verify --package`.
Earlier verification commands below describe their original release revisions.

Current public-service and schema-10 upgrade instructions: [r34 public service deployment](R34_PUBLIC_SERVICE_DEPLOYMENT.md). This includes both 2026-10-09 lifecycle packages. Large active-user capacity requires the measured qualification described there.

Use the **complete r34 release supplied with this guide**. Extract it into a new
local project folder; this installation does not require the earlier compatibility
update ZIP or separately downloaded repair scripts. Compare the archive's SHA256
with its companion checksum before extraction. Keep the complete release and its
validation records.

For receiver startup and durability settings, also read
[WAL catch-up and batching](R34_HF_WAL_CATCHUP_AND_BATCHING.md).

This native mode runs the PF SaaS application, PostgreSQL process, scientific
Jobs and primary durable storage on Hugging Face. No separate Linux server is
required. Email delivery and users' BYOK/BYOS providers remain external services.
The standalone and ordinary Linux/Docker deployment instructions remain separate.

**If you already deployed:** a clean local extraction is not a remote reset.
Preserve your private Spaces, bucket, deployment prefix and all encryption,
session and database keys. Do not discard working state or substitute fresh keys.
Sections 1–9 cover a new installation. For an earlier r34 deployment that is
already running, use the complete [upgrade procedure in section 11](#11-upgrade-an-existing-r34-deployment).
The [recovery guide](HUGGING_FACE_R34_RECOVERY.md) explains the recovery boundaries.

### Where each action happens

| Label | Open this |
| --- | --- |
| **Browser — Hugging Face** | The HF website, signed in as the deployment operator. |
| **Windows CMD** | Start menu → type **cmd** → Command Prompt; ordinary privileges. |
| **Administrator PowerShell** | Start menu → PowerShell → **Run as administrator**; only explicitly labelled Windows setup/repair commands. |
| **Docker Desktop** | The Docker Desktop application on your PC; optional for the HF cloud-build route. |
| **Browser — email/DNS provider** | SMTP2GO and the provider managing your domain's DNS, such as Network Solutions. |

Paste each command as one line in the named shell. Do not copy a prompt such as
`D:\>`, PDF line numbers, or visual line wraps. Windows CMD uses `%NAME%` variables;
PowerShell uses different syntax. Commands here are CMD unless labelled otherwise.
Replace `YOUR_HF_USERNAME` and other `REPLACE_...` placeholders before running.
Run setup and upload commands **one at a time** and stop if a command fails.
Only the explicitly labelled read-only diagnostic batches may continue after
an individual failure.

**Milestones:** create resources → prepare Windows tools → build/upload the three
contexts → record immutable pins → run one confinement test → configure email and
controller → verify startup → qualify the complete service. A successful upload
or runner test is not proof that the complete SaaS deployment is ready.

## Consolidated r34 starting point

The October 2026 consolidated release incorporates the approved native Latency,
Application Latency (final corrected form), Startup Recovery, Response Correction,
Runtime Recovery, WAL Batching, Ownership Queue and Persistence Response Recovery
changes. It also contains the shared paging, bounded admission, database-pool,
lease/recovery and polling improvements. Do not replay the historical patch ZIPs.
User-reported live responsiveness of the prior native patch combination is
recorded separately from this release's offline regression results.

Native HF remains one PostgreSQL authority inside the private controller, with
its encrypted state and WAL in the existing Bucket. No external database service
is required. The optional managed PostgreSQL route is a separate explicit choice;
see [deployment choices](R34_CONSOLIDATED_DEPLOYMENT.md). Preserve
`PF_SAAS_DEPLOYMENT_PROFILE=single` for this guide.

The shared application schema is now **10**. Native startup restores the existing
cluster, applies its additive schema upgrade and initializes shared counters and
admission tables before serving. Existing users, keys and quota reservations are
retained. Take the section-11 backup first: returning to schema-9 or older code is not
a code-only rollback after migration. External PostgreSQL migrations are not
part of the native route.

For the shortest existing-Space route, see
[the consolidated native upgrade checklist](R34_NATIVE_HF_RELEASE_UPGRADE.md).

## 1. Create the HF resources and their credentials

**Where: Browser — Hugging Face.**

1. Create an account at <https://huggingface.co/join>, verify its email, and enable
   account security/recovery controls. Review [current eligibility and charges](https://huggingface.co/docs/hub/spaces-overview)
   for Docker Spaces, Jobs and Buckets before purchasing. HF currently requires
   a paid eligible account to create Docker Spaces; CPU Basic having no hourly
   Space charge does not make the whole deployment free.
2. At <https://huggingface.co/new-space>, create these **three Docker Spaces**.
   Select your username/organization as owner, **Docker** as SDK, and a blank
   Docker template if offered. The names below are used by the commands; use
   other names only if you change every matching reference.
3. At <https://huggingface.co/new-bucket>, create the private state Bucket below.
   Create it **before** configuring the token so it can be included in its scope.

| Resource name | Type / visibility | What will be uploaded |
| --- | --- | --- |
| `pf-runtime-r34` | **New Public Docker Space** | Only generated `runtime/`: public dependencies and generic bootstrap. |
| `pf-simulation-runner` | **Private Docker Space** | Generated `jobs/`: private PF code payload. |
| `pf-simulation-saas` | **Private Docker Space** | Generated `space/`: trusted SaaS controller. |
| `pf-simulation-state` | **Private Storage Bucket** | Encrypted service state/artifacts, written by the controller. |

**Never make a previously private runner/controller Space public.** Its Git
history may contain proprietary code. A new public runtime Space must contain
only the reviewed `runtime/` context, with no PF source, user files or secrets.
Keep the Bucket private. Existing private code/controller Spaces may be reused.

4. Open <https://huggingface.co/settings/tokens>. Create a **fine-grained** setup
   token, for example `pf-r34-setup`, with read/write access to those three Spaces,
   read/write access to **the state Bucket**, and permission to start/manage Jobs.
   Jobs management includes inspecting and cancelling the operator's Jobs.
   Space permissions do not implicitly include a separate private Bucket.
5. In the resource selector, search for the Bucket name and select the **Bucket**
   belonging to your owner, if it is offered there. Confirm its resource type and
   full owner/name; do not select a similarly named Space. If the UI cannot find
   it, check the Bucket's exact URL and available scope controls rather than
   granting every resource or recreating the Bucket.
6. Save the token securely. In that token's menu choose **Generate S3 credentials**
   and save both the `HFAK...` access-key ID and matching secret access key.
   The secret is displayed once. A separate narrower controller token may later
   replace the setup token: it needs Jobs management, private code access and
   read/write access to the deployment Bucket, not upload rights to all Spaces.

HF-native CLI access uses the **HF token**; the S3 API uses the **S3 pair** derived
from an HF token. S3 permissions inherit the parent token's scope. These are
different credentials; an S3 key is not an `hf auth login` token.
[HF S3 credential documentation](https://huggingface.co/docs/hub/storage-buckets-s3).

**Checkpoint:** all four resources exist with the stated visibility; the token's
scope includes the Bucket as well as the Spaces. Ordinary PF users must never
receive this operator token.

## 2. Prepare the full release and Windows tools

**Where: Windows CMD.** To check the download, replace the path in this example
and compare the result with the separately supplied checksum:

```bat
certutil -hashfile "FULL_PATH_TO_DOWNLOADED_RELEASE_ARCHIVE" SHA256
```

**Where: File Explorer / archive utility.** If the download ends in `.zip.xz`,
extract the outer `.xz` first to obtain the `.zip`, then extract that ZIP into a
fresh directory. [7-Zip](https://www.7-zip.org/) or another compatible utility can
open the outer layer; Explorer's **Extract All** can extract the resulting ZIP.
Do not run scripts inside an archive preview or overwrite an earlier project.
Locate
the project root containing `run_pf_saas.py`, `pf_hf_native` and `deploy`. All
helpers used below are already under `deploy\saas\huggingface_native` in that
release. Do not use earlier scripts from Downloads.

**Where: Windows CMD.** Set your nonsecret working values. Change the project path
to your actual extraction, and the owner to your HF username/organization:

```bat
set "PF_PROJECT_ROOT=D:\privileged-frame-simulation"
set "PF_HF_OWNER=YOUR_HF_USERNAME"
set "PF_HF_TOOLS=%USERPROFILE%\pf-hf-tools-r34"
set "PF_HF_UPLOAD=%USERPROFILE%\pf-hf-native-upload-r34-full"
set "PF_HF_DEPLOY_STATE=%USERPROFILE%\pf-hf-native-deploy-r34-full"
set "PF_HF_RUNTIME_REPO=%PF_HF_OWNER%/pf-runtime-r34"
set "PF_HF_CODE_REPO=%PF_HF_OWNER%/pf-simulation-runner"
set "PF_HF_APP_REPO=%PF_HF_OWNER%/pf-simulation-saas"
set "PF_HF_BUCKET_NAME=pf-simulation-state"
cd /d "%PF_PROJECT_ROOT%"
py -3.12 --version
```

These `set` values last only in this CMD window. If you reopen it, repeat this
block and the activation command below. They do **not** configure your HF Space.

Use 64-bit Python **3.12**. If it is missing, install it from [Python's official
Windows downloads](https://www.python.org/downloads/windows/) and confirm
`py -3.12 --version` works. The operator tooling was exercised with Python 3.12.10;
use a maintained compatible 3.12 installation and record the patch version.
Do not let an unrelated Python 3.13 environment silently run these tools.

Create the tools environment only when `PF_HF_TOOLS` names a new directory. An
existing correctly configured 3.12 environment can instead be activated; do not
overwrite an existing 3.13 environment.

```bat
py -3.12 -m venv "%PF_HF_TOOLS%"
call "%PF_HF_TOOLS%\Scripts\activate.bat"
python -m pip install "huggingface_hub==1.32.0" "truststore>=0.10,<1"
where python
where hf
python -c "import sys,huggingface_hub; print(sys.executable); print(sys.version.split()[0]); print(huggingface_hub.__version__)"
hf auth login
hf auth whoami
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"
```

In the login menu choose **Paste an access token**, if offered, and enter the setup
token only at the private prompt. Do not use `--token SECRET` in a command. The
first `python` and `hf` paths should be inside the tools environment; versions
should show Python 3.12 and Hub SDK 1.32.0. A successful Bucket information check
confirms this HF token can see it; it does not yet prove S3 writes/durability.
If any checkpoint fails, use section 10 before proceeding.

## 3. Choose the Docker tooling route

Both routes deploy Docker images on HF. **The cloud-build route used in sections
4–8 needs Python and the HF CLI, but no local Docker daemon.** HF builds the
uploaded Dockerfiles. The packaged registry helper reads image metadata without
Docker. It does not replace the subsequent live HF Job test.

### Standard option: install Docker Desktop for local Docker commands

**Where: your Windows PC, then Administrator PowerShell / Docker Desktop.**

1. Check [Docker's current Windows requirements](https://docs.docker.com/desktop/setup/install/windows-install/):
   supported 64-bit Windows, CPU SLAT support, at least 8 GB RAM, and hardware
   virtualization enabled in BIOS/UEFI. The WSL 2 backend requires WSL 2.1.5 or
   later. On a managed PC/virtual machine, confirm administrator/host support.
2. **Administrator PowerShell:** if WSL is absent, install it without a separate
   Linux distribution, then restart Windows when requested:

```powershell
wsl --install --no-distribution
```

If WSL is already installed, update it instead:

```powershell
wsl --update
```

3. Install Docker Desktop from its official page. Select the **WSL 2 backend**,
   start Docker Desktop, and complete its first-run prompts.
4. **Docker Desktop:** use **Linux containers** and, where shown, **Settings →
   General → Use the WSL 2 based engine**. Wait for the engine to run.
5. **Windows CMD:** check the local server:

```bat
wsl --version
docker version
docker info --format "{{.OSType}}"
```

Expected: both **Client** and **Server** sections, followed by `linux`. A client
version alone is not a working engine. A separate Ubuntu installation is not
needed for these CMD commands. [Microsoft WSL options](https://learn.microsoft.com/en-us/windows/wsl/basic-commands).

After the runtime has built in section 4, you can optionally authenticate your
local Docker client for registry operations. **Windows CMD:**

```bat
docker login registry.hf.space
```

At its prompts, enter your **personal HF username** and an HF token as the
password. An organization owner name is not the username of the person logging
in. Keep the token out of command arguments. This local login is optional and
does not replace section 5's anonymous public-image check or live HF Job test.

If local Docker cannot run, continue with the cloud-build route below. The
virtualization and named-pipe HTTP 500 troubleshooting in section 10 explains
local repair; neither problem requires changing PF's Linux/Docker server edition.


## 4. Build and upload the runtime and private code

**Where: Windows CMD, activated tools environment, project root.**

```bat
cd /d "%PF_PROJECT_ROOT%"
python deploy\saas\huggingface_native\build_space.py --output "%PF_HF_UPLOAD%"
```

`PF_HF_UPLOAD` must name a **new, nonexistent** output directory. If it already
exists, choose a new name, change that variable, and keep using it for every
upload. Do not mix an old `pf-hf-native-upload` folder with the current build.

The packaged compatibility-named alias below delegates to the same current
three-context builder; it is an alternative command, not an additional patch:

```bat
python deploy\saas\huggingface_native\build_space_windows_fix.py --output "%PF_HF_UPLOAD%"
```

Run **one** builder command, not both against the same output. Both accept
`--source` and `--output`. The builder produces `runtime/`, `jobs/`, `space/` and
matching ZIPs/manifests. It checks an explicit input allowlist and rejects symlinks.
The controller context includes `base-code-manifest.json`; keep it with its
matching `source/` tree and `jobs/pf-code.zip`. Native startup checks this inventory
against `PF_HF_CODE_SHA256` before admitting a new execution.
Verify that the generated runner and controller describe the same actual payload:

```bat
python -c "import hashlib,json,sys; from pathlib import Path; p=Path(sys.argv[1]); a=json.loads((p/'jobs'/'code-manifest.json').read_text(encoding='utf-8')); b=json.loads((p/'space'/'base-code-manifest.json').read_text(encoding='utf-8')); h=hashlib.sha256((p/'jobs'/'pf-code.zip').read_bytes()).hexdigest(); assert a['sha256']==b['sha256']==h, 'Generated contexts do not match'; print('VERIFIED_CODE_SHA256='+h)" "%PF_HF_UPLOAD%"
```

Require `VERIFIED_CODE_SHA256=` followed by the hash from this build. Do not
substitute the full release archive checksum for this private-code checksum.

Review `runtime/` before public upload. Upload only each context's contents,
never the full release, parent directory, live data or environment files.
Do not add any secrets to the public runtime Space.

```bat
hf upload "%PF_HF_RUNTIME_REPO%" "%PF_HF_UPLOAD%\runtime" . --repo-type space
python deploy\saas\huggingface_native\deploy_release.py --system-ca upload-code --bundle "%PF_HF_UPLOAD%" --repo "%PF_HF_CODE_REPO%" --fresh
python -c "import sys; from huggingface_hub import HfApi; print(HfApi().space_info(repo_id=sys.argv[1]).sha)" "%PF_HF_CODE_REPO%"
python -c "import json,sys; print(json.load(open(sys.argv[1], encoding='utf-8'))['sha256'])" "%PF_HF_UPLOAD%\jobs\code-manifest.json"
```

Save the last two outputs: the private upload's exact **40-character commit**
and the payload's **64-character SHA256**. They must come from **this build/upload**,
not a previous chat or release. In **Browser — HF**, wait for the public runtime
Space's build to succeed. The runtime/code pages are not the PF application.

**Checkpoint:** successful uploads, a built public runtime, and two recorded
private-code pins. A failed upload may leave uncertainty about remote commits;
check Files/history before deciding nothing changed. Xet 403 recovery is in
section 10. Do not delete repository files/history as a routine workaround.

## 5. Resolve the public image and qualify one HF Job

**Where: Browser — Hugging Face, public runtime Space.** Open its **Run with
Docker** / **Run locally** instructions (possibly in the page menu) and copy the
actual `registry.hf.space/...:latest` image reference. Do not guess it from the
Space URL, and do not use the private runner's old image.

**Where: Windows CMD.** Replace the placeholder with that image:

```bat
set "PF_HF_RUNTIME_TAG=REPLACE_WITH_ACTUAL_PUBLIC_REGISTRY_IMAGE_LATEST"
python deploy\saas\huggingface_native\hf_registry_digest.py --anonymous --system-ca --image "%PF_HF_RUNTIME_TAG%"
```

This packaged helper requires no Docker daemon. `--system-ca` uses Windows'
trusted certificate authorities via `truststore`; TLS verification remains on.
The anonymous check must succeed without reading/sending an HF token. Record its
complete `registry_image` value ending in `@sha256:` plus 64 hex characters.
Public visibility alone is not proof that a particular registry route is
anonymously accessible. An authenticated metadata lookup is not a substitute.

Set the three outputs below. The private commit and payload hash are lowercase
hexadecimal; no angle brackets or spaces belong inside these values.

```bat
set "PF_HF_JOB_IMAGE=REPLACE_WITH_FULL_REGISTRY_IMAGE_AT_SHA256_DIGEST"
set "PF_HF_CODE_REVISION=REPLACE_WITH_40_HEX_PRIVATE_COMMIT"
set "PF_HF_CODE_SHA256=REPLACE_WITH_64_HEX_PAYLOAD_SHA256"
python deploy\saas\huggingface_native\qualify.py --namespace "%PF_HF_OWNER%" --runtime-image "%PF_HF_JOB_IMAGE%" --code-repo "%PF_HF_CODE_REPO%" --code-revision "%PF_HF_CODE_REVISION%" --code-sha256 "%PF_HF_CODE_SHA256%" --flavor cpu-basic --system-ca
```

This is a **dry run**: no Job is submitted. Review the image, private revision,
hash, empty secrets and read-only `/pf-code` mount. The bootstrap verifies the
payload before extracting/importing it. When ready, the following command
explicitly submits **one paid Job**, with a five-minute provider timeout:

```bat
python deploy\saas\huggingface_native\qualify.py --namespace "%PF_HF_OWNER%" --runtime-image "%PF_HF_JOB_IMAGE%" --code-repo "%PF_HF_CODE_REPO%" --code-revision "%PF_HF_CODE_REVISION%" --code-sha256 "%PF_HF_CODE_SHA256%" --flavor cpu-basic --system-ca --submit
```

The helper supplies the short name `pf-r34-confinement-check`. Save the Job ID/URL. Acceptance requires
verified provider identity, a passing mandatory confinement probe, successful
confined FMPy Dahlquist and QuTiP decay probes, successful exit and provider
stage **COMPLETED**. Require `self_test_completed: true` and
`scientific_runtime_probe_passed: true`. These small dependency probes do not
qualify every scientific workflow or the complete SaaS deployment. Merely
starting or pulling an image is insufficient. Build the runtime context from
this release and record its new digest; updating private code while retaining
an earlier dependency image does not install the required scientific packages.

If the helper reports `RuntimeError`, a timeout or an uncertain response after
printing a Job ID, **do not submit again**. Inspect the existing Job in section
10 first. Completed logs can establish that execution ran, but do not by
themselves resolve a failed provider-identity check. A generic RuntimeError does
not identify its cause: resolve the helper error and compare the actual Job
with the expected identity before service admission. A cancellation request is not proof the VM
stopped. Failed confinement remains blocking; do not remove the probe or request
privileges.

The self-test runs outside PF's admission ledger and can consume HF credits
separately. Cloud Space operation/build resources are also outside that ledger.
Rebuilding code changes its commit/hash; rebuilding a runtime may change its
digest. Requalify new identities. Tags, `hf.co/spaces/OWNER/SPACE` shorthand and
digests copied from another deployment are not production pins.

**Checkpoint:** record this runtime digest, private repo/commit/hash, flavor and
successful Job evidence. These CMD variables are only local notes. **You must
enter their actual values into the controller's HF Variables in section 7.**

## 6. Set up email and generate stable secrets

### Email delivery

**Where: Browser — SMTP2GO, then your DNS provider.** Production local
authentication requires functioning account email even while self-registration
is disabled. The r34 mailer supports encrypted SMTP and optional Resend HTTPS.
This section retains the SMTP2GO path; a provider migration is not required.
See [public email delivery](PUBLIC_EMAIL_DELIVERY.md) for both transports and
their shared durable queue. HF Spaces currently permits outgoing
ports **80, 443 and 8080**; the usual SMTP ports 587/465 are not usable under
that published policy. [HF networking](https://huggingface.co/docs/hub/spaces-overview#networking).

A compatible documented setup is [SMTP2GO SMTP relay](https://developers.smtp2go.com/docs/smtp-relay):

1. Create the SMTP2GO account and review its plan/quota.
2. In **Sending → Verified Senders**, add your sending domain. In the domain's
   actual DNS manager, add the exact three CNAME records SMTP2GO displays.
   They handle return-path/SPF, DKIM and tracking. Use your displayed values,
   not another user's selector numbers.
3. For Network Solutions DNS: **Domains → your domain → Advanced Tools →
   Advanced DNS Records → Manage → Add Record → CNAME → Other Host**. Its form
   may append the domain; use the host part only when it does.
   [Network Solutions setup](https://support.smtp2go.com/hc/en-gb/articles/360022581174-DNS-Setup-for-Network-Solutions).
4. Add records without replacing website, MX, nameserver, SPF or existing DKIM
   records. SMTP2GO requires no change to existing SPF/MX. If `link` is already
   used, choose an unused tracking hostname in SMTP2GO. Preserving the existing
   hosting records keeps your Network Solutions website and mailbox in place.
5. In SMTP2GO click **Verify**. This checks DNS; it does not add records. Inspect
   each failure's tooltip. Propagation is a possible delay, not a substitute
   for checking the records were saved.
6. Create an **SMTP User** under **Sending → SMTP Users**. Save its username and
   password. These are not the website login, API key, mailbox password or PF
   administrator password. Open/click tracking can remain disabled.

For each DNS row, the **Host/Name** is the alias under your own domain; the
**Alias/Target** is SMTP2GO's supplied destination, such as `return.smtp2go.net`,
`dkim.smtp2go.net` or `track.smtp2go.net`. Targets are hostnames, without `https://`
or a path. Use the values displayed for your account. TTL is in seconds:
`3600` = 1 hour, `7200` = 2 hours, `14400` = 4 hours. Select the matching duration
if the DNS form uses a dropdown. TTL is caching behavior, not a guaranteed
verification waiting time.

Choose the actual sender mailbox you want recipients to see, for example
`support@YOUR_VERIFIED_DOMAIN`. Domain verification authorizes sending; it does
not choose or create that mailbox. Enter the chosen address as
`PF_SAAS_SMTP_FROM`; it is not the SMTP username or your HF account login.

Use `mail.smtp2go.com`, port **443**, **SSL=true**, **STARTTLS=false** in section
7. The public-readiness checker requires exactly one TLS mode and an allowed
outbound port for HF SMTP. Another provider's encrypted SMTP endpoint on an
allowed port can be used after verification. Changing Network Solutions' normal SMTP port to 443 alone
does not make its server support it.
[Verified sender details](https://support.smtp2go.com/hc/en-gb/articles/115004408567-Verified-Senders).

### Application and storage keys

**Where: Windows CMD, privately.** Generate these only for a genuinely new
deployment. Save each output in a password manager/offline recovery record,
labelled with its exact Secret name; then enter it in HF Secrets.
These commands print secrets, so do not share their output.

Run once for `PF_HF_STATE_KEY`:

```bat
python -c "import secrets; print(secrets.token_hex(32))"
```

Run the following **twice**, keeping separate outputs for
`PF_SAAS_ASSISTANT_CREDENTIALS_KEY` and `PF_SAAS_BYOS_CREDENTIALS_KEY`:

```bat
python -c "import base64,secrets; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())"
```

Run this separately for each of `PF_SAAS_SECRET_KEY`,
`PF_HF_PG_ADMIN_PASSWORD`, `PF_HF_PG_APP_PASSWORD` and
`PF_HF_PG_WAL_PASSWORD`:

```bat
python -c "import secrets; print(secrets.token_urlsafe(48))"
```

Choose/save a separate strong `PF_HF_ADMIN_PASSWORD` for your first PF login.
Do not reuse these values. Keep state, database, session and vault keys stable
across restarts/rebuilds: losing or replacing them can make saved data
unrecoverable. HF Secrets cannot be read back as a recovery method;
retain your own secure copy.

## 7. Configure the private controller Space

**Where: Browser — Hugging Face → your `pf-simulation-saas` Space → Settings →
Variables and secrets.** Add the following names to **Variables** or **Secrets**
as indicated. Enter literal values, without CMD `set` syntax, outer quotes, or
unexpanded `%PF_HF_OWNER%` placeholders. Every value is case-sensitive.

Get HF's actual assigned app host in **Windows CMD**:

```bat
python -c "import sys; from huggingface_hub import HfApi; print(HfApi().space_info(repo_id=sys.argv[1]).host)" "%PF_HF_APP_REPO%"
```

Use that host as an HTTPS origin, for example `https://ACTUAL-HOST.hf.space`,
with no trailing path; keep an existing `https://` prefix rather than adding it
twice. If empty, obtain it from the Space's open-app link after
its initial build. Do not guess it from the account name.


| Variable | Value/purpose |
| --- | --- |
| `PF_HF_NAMESPACE` | Your HF owner from `PF_HF_OWNER`; type the actual username/organization. |
| `PF_HF_BUCKET` | Bare bucket name, without namespace or `/`. |
| `PF_HF_PREFIX` | Stable deployment prefix, for example `pf-saas`. Never change it to bypass a full ledger. |
| `PF_HF_JOB_IMAGE` | Public runtime image reference ending in `@sha256:` and 64 hex characters, qualified in section 5. |
| `PF_HF_CODE_REPO` | Private Space repository containing the generated code payload, e.g. `YOUR_HF_USERNAME/pf-simulation-runner`. |
| `PF_HF_CODE_REVISION` | Exact 40-character Git commit of that private payload upload; never a branch/tag. |
| `PF_HF_CODE_SHA256` | SHA256 of the generated code archive, recorded by the builder. |
| `PF_HF_ASSISTANT_IMAGE` | Optional distinct public runtime digest for private workflows/assistant code; defaults to the job image. The same pinned private code payload is used. |
| `PF_HF_JOB_FLAVOR` | `cpu-basic` or `cpu-upgrade`, qualified for the actual workload. |
| `PF_HF_BOOTSTRAP` | `true` for the first empty deployment only; remove/set `false` after successful initialization. It never replaces an existing remote cluster. |
| `PF_HF_PG_START_TIMEOUT_SECONDS` | Bounded PostgreSQL startup/recovery wait, default `600`; use `30` to `3600` seconds as justified by restore qualification. |
| `PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS` | Separate bounded wait for the durable WAL receiver to finish catch-up and become synchronous; default `600`, integer range `30` through `3600`. It does not replace the PostgreSQL process-start timeout. |
| `PF_HF_MAX_STORAGE_BYTES` | Deployment lifetime byte-admission ceiling, e.g. `10737418240` for 10 GiB. Includes retained encrypted persistence. |
| `PF_HF_WAL_RESERVE_BYTES` | Capacity reserved for essential WAL/base backups, e.g. `268435456`. Must be positive and below the total ceiling. |
| `PF_HF_MAX_STATE_BYTES` | Bound per private state scope, e.g. `2147483648`. |
| `PF_HF_MAX_TRANSFER_FILES` | Maximum transfer/state inventory entries, e.g. `20000`. |
| `PF_SAAS_PUBLIC_URL` | Direct application origin, e.g. `https://YOUR-ACTUAL-SUBDOMAIN.hf.space`; copy the actual value from HF. |
| `PF_SAAS_DEPLOYMENT_BUDGET_ENABLED` | Required: `true`. |
| `PF_SAAS_DEPLOYMENT_MONTHLY_COMPUTE_SECONDS` | Finite shared monthly compute-admission ceiling, e.g. `3600` for an initial trial; committed charges include active reservations. Normal jobs reserve their timeout plus a 60-second launch/stop margin. See queued-run troubleshooting before adjusting it. |
| `PF_SAAS_WORKER_SLOTS` | Start with `1`; integer range `1` through `64`. This starts real internal dispatcher threads under the same controller; it does not create another database writer. |
| `PF_SAAS_DEPLOYMENT_MAX_CONCURRENT_JOBS` | Start with `1`; shared active execution-reservation cap, independent of worker slots. Increase only after load qualification and budget review. |
| `PF_SAAS_USER_MAX_CONCURRENT_JOBS` | Start with `1`; active scientific/slot-occupying assistant operations per user within an organization. Must be positive when worker slots exceed one. |
| `PF_SAAS_USER_MAX_QUEUED_JOBS` | Start with `20`; queued scientific jobs per user within an organization. Must be positive when worker slots exceed one. |
| `PF_SAAS_DEPLOYMENT_MAX_QUEUED_JOBS` | Shared queued scientific-job ceiling across all native HF tenants, default `10000`; set a smaller deliberate trial limit if desired. |
| `PF_SAAS_MAX_CONCURRENT_JOBS` | `1` initially; per-organization limit, separate from deployment concurrency. |
| `PF_SAAS_API_MAX_INFLIGHT_REQUESTS` / `API_MAX_INFLIGHT_UPLOADS` | Per-process HTTP/upload concurrency; defaults `64` and `2`. These protect request handling, not scientific execution capacity. |
| `PF_SAAS_DATABASE_POOL_SIZE` / `DATABASE_MAX_OVERFLOW` | Shared controller SQL pool bounds, defaults `5` and `5`; shared assistant pools are budgeted separately. |
| `PF_SAAS_DATABASE_POOL_TIMEOUT_SECONDS` / `DATABASE_LEASE_POOL_SIZE` | Main-pool wait in seconds (`5`) and independent lease connections (`1`). Main+overflow+lease+enabled assistant pools must not exceed `25` in this native profile. |
| `PF_SAAS_API_USER_REQUESTS_PER_MINUTE` | Authenticated per-user request admission; native HF uses a bounded in-memory counter to avoid a durable SQL write on each GET (initial example `600`). This is separate from in-flight concurrency; counters reset on restart. |
| `PF_SAAS_API_RATE_LOCAL_MAX_ACCOUNTS` | Native counter bound, default `10000`; a full active counter map returns 429 for new accounts. See [shared settings](R34_SHARED_CAPACITY_SETTINGS.md). |
| `PF_SAAS_AI_ADMISSION_ENABLED` | Keep `true` for hosted operator/BYOK/autonomous work; configure the shared deployment/organization/provider concurrency, RPM, TPM and estimated-cost bounds from the shared settings guide. |
| `PF_SAAS_MAX_JOB_SECONDS` | Per-invocation timeout, e.g. `600`; allow for CPU-intensive scientific needs. |
| `PF_SAAS_JOB_PREPARE_SECONDS` | Optional preparation deadline in seconds; default `300`. |
| `PF_SAAS_JOB_TRANSFER_SECONDS` | Optional input-transfer deadline in seconds; default `300`. |
| `PF_SAAS_JOB_SUBMIT_SECONDS` | Optional submission-handling deadline in seconds; default `60`. |
| `PF_SAAS_JOB_COLLECT_SECONDS` | Optional output-collection deadline in seconds; default `300`. |
| `PF_SAAS_JOB_CPUS` | `2` or less in this native profile. |
| `PF_SAAS_JOB_MEMORY_MB` | `4096` initially; adjust after qualification, never above `14336` in this profile. |
| `PF_SAAS_ASSISTANT_BYOK_ENABLED` | `true` to allow users' supported AI provider keys. |
| `PF_SAAS_BYOS_ENABLED` | `true` to allow supported user storage destinations. |
| `PF_SAAS_REGISTRATION_ENABLED` | Keep `false` during private qualification; enable only with the configured account/email flow. |
| `PF_SAAS_MAIL_MODE` | Required for default local authentication in production: `smtp`. |
| `PF_SAAS_SMTP_HOST` | `mail.smtp2go.com` for the documented SMTP2GO route. |
| `PF_SAAS_SMTP_PORT` | `443` for SMTP2GO implicit TLS; do not use blocked `587`/`465`. |
| `PF_SAAS_SMTP_FROM` | Your chosen real mailbox at the verified domain, e.g. `support@YOUR_VERIFIED_DOMAIN`; not the SMTP username. |
| `PF_SAAS_SMTP_STARTTLS` | `false` with the documented SMTP2GO port-443 setup. |
| `PF_SAAS_SMTP_SSL` | `true` with the documented SMTP2GO port-443 setup. |
| `PF_HF_ADMIN_EMAIL` | Your first PF administrator's email address; set it for first initialization. |
| `PF_HF_ADMIN_NAME` | Your first PF administrator's display name. |

| Secret | Purpose |
| --- | --- |
| `PF_HF_TOKEN` | Trusted Jobs control-plane token with access to the private code Space and required Bucket mounts. Never forwarded to scientific code. |
| `PF_HF_S3_ACCESS_KEY_ID` | Dedicated Bucket S3 access key. |
| `PF_HF_S3_SECRET_ACCESS_KEY` | Matching S3 secret. |
| `PF_HF_STATE_KEY` | Independent 32-byte key, encoded as 64 hex characters or standard Base64. |
| `PF_HF_PG_ADMIN_PASSWORD` | Persistent PostgreSQL administration password, at least 32 printable characters. |
| `PF_HF_PG_APP_PASSWORD` | Distinct application database password. |
| `PF_HF_PG_WAL_PASSWORD` | Distinct replication-role password. |
| `PF_SAAS_SECRET_KEY` | Stable independent SaaS session/signing key. |
| `PF_SAAS_ASSISTANT_CREDENTIALS_KEY` | Stable independent BYOK vault key: URL-safe Base64 encoding of 32 bytes. |
| `PF_SAAS_BYOS_CREDENTIALS_KEY` | Stable independent BYOS vault key: URL-safe Base64 encoding of 32 bytes. |
| `PF_SAAS_SMTP_USERNAME` | Username from SMTP2GO's SMTP Users page. |
| `PF_SAAS_SMTP_PASSWORD` | Matching SMTP User password, not an HTTP API key or your PF login password. |
| `PF_HF_ADMIN_PASSWORD` | First operator password; remove after successful first login. Existing accounts are never reset/promoted by this path. |



The launcher owns production mode, the local PostgreSQL connection and the
assistant/BYOS state directories. Do not set an external database URL, hybrid
gateway secret or separate worker for this native mode. Existing SaaS settings
still govern supported AI providers, endpoint allowlists, organizations and quotas.
The defaults above are starting limits, not validated capacity for every workload.

**Checkpoint:** the six runner values (namespace, image, code repository,
revision, hash and flavor) are only part of the configuration. Bucket access,
all stable keys/passwords, SMTP and deployment budgets must also be configured.
Keep `PF_SAAS_REGISTRATION_ENABLED=false` and the Space Private during validation.

## 8. Upload the controller and verify startup

**Where: Windows CMD.** Use the same current output directory from section 4:

```bat
python deploy\saas\huggingface_native\deploy_release.py verify --bundle "%PF_HF_UPLOAD%"
python deploy\saas\huggingface_native\deploy_release.py --system-ca prepare --bundle "%PF_HF_UPLOAD%" --space "%PF_HF_APP_REPO%" --state-dir "%PF_HF_DEPLOY_STATE%" --fresh
python deploy\saas\huggingface_native\deploy_release.py --system-ca apply --state-dir "%PF_HF_DEPLOY_STATE%"
```

Use `--fresh` only for a newly created empty controller Space. For an existing
native deployment, use section 11 without that flag. `prepare` verifies local
seals and private code pins without changing HF. `apply` confirms pause, commits
against the reviewed repository revision and requests restart. Keep the printed
revision and local preparation directory. It does not change configuration,
Secrets or durable data. Do not add blanket `--delete` flags.

After apply succeeds, wait for the controller:

```bat
hf spaces wait "%PF_HF_APP_REPO%" --timeout 25m
python deploy\saas\huggingface_native\deploy_release.py --system-ca status --state-dir "%PF_HF_DEPLOY_STATE%"
hf spaces logs "%PF_HF_APP_REPO%" --tail 150
```

Require `source_bytes_verified=true`, `running_revision_verified=true`, provider
stage `RUNNING` and application startup completion in the
runtime logs. A terminal build/runtime error ends the wait early; inspect its
logs even when the wait command returns an error. The CLI's `25m` wait is only
an observation window, separate from the controller's bounded startup settings.

**Where: Browser — Hugging Face.**

1. Open the controller's build logs, then its runtime/container logs. A completed
   Docker build is distinct from successful application startup.
2. The first empty deployment needs `PF_HF_BOOTSTRAP=true`. The supervisor creates
   PostgreSQL, persists a verified base backup, starts synchronous remote WAL,
   initializes ledgers, restores private state and then exposes port 7860.
   Existing remote state always takes the restore path. Failure keeps startup closed.
3. Open the **direct application URL** in its own browser tab. If the HF page's
   embedded frame is blocked, use the direct URL; do not weaken frame/CSRF/cookie
   protections. Private Space access still requires HF owner/collaborator access.
4. Log into **PF Simulation** with `PF_HF_ADMIN_EMAIL` and
   `PF_HF_ADMIN_PASSWORD`. Use this PF administrator password, not an HF token,
   S3 secret, PostgreSQL password or SMTP password. Existing users are never
   reset or promoted by bootstrap values.
5. After confirmed successful durable initialization and first login, remove
   `PF_HF_ADMIN_PASSWORD` and remove `PF_HF_BOOTSTRAP` (absence defaults to
   `false`). Preserve every other
   persistent key/password, bucket and prefix. Confirm the resulting restart
   restores the same application state. Use the account's normal password flow
   afterward, not bootstrap variables.

If a setting is missing or logs contain a generic `DurabilityError`, use section 10.
Do not generate new encryption keys or reset storage to get past recovery.

If HF remains **Starting** with sparse logs, use this **read-only Windows CMD**
batch and inspect the latest runtime logs in the browser:

```bat
python -c "import sys; from huggingface_hub import HfApi; print(HfApi().get_space_runtime(repo_id=sys.argv[1]).stage)" "%PF_HF_APP_REPO%"
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"
hf buckets list "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"
```

These checks do not reveal Secret values or prove recovery has finished.
`PF_HF_PG_START_TIMEOUT_SECONDS=600` bounds the PostgreSQL startup/recovery phase,
not the total Docker build or full controller initialization. The separate
`PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS` wait defaults to `600` seconds and
accepts integers from `30` through `3600`. Healthy WAL uploads can continue
while PostgreSQL reports `catchup` or `potential`; those states do not yet
satisfy the unchanged `streaming`/`sync` readiness checks. The receiver batches
contiguous data within a 100-millisecond collection window, up to 4 MiB; remote
publication time is additional. See the [maintenance note](R34_HF_WAL_CATCHUP_AND_BATCHING.md)
for the acknowledgement and startup boundaries. Sparse logs or an elapsed
interval alone are not grounds for a force restart or storage reset.

After completing section 9, a SaaS trial for ordinary PF users can use
**Protected** Space visibility: source stays private while the app is accessible.
Check current HF plan eligibility. PF users then authenticate to PF and retain
organization/project permissions; they do not need repository collaborator access
or the operator's HF token. Do not change the controller to Public.


## 9. Qualify the complete deployment before researcher access

Record the release hash, builder manifests, runtime image digest, private code
repository and exact commit, payload SHA256, controller Space revision, HF
hardware/flavor, package versions, test inputs, Job IDs, timestamps, and observed
results. Rebuilds and changes to those identities require qualification again. Required live gates include:

Run destructive fault tests in a **separate disposable validation deployment**,
with its own private Bucket/prefix and independently generated keys. Never
corrupt or remove WAL, database heads or artifacts, introduce a conflicting
controller, or exhaust storage against a deployment holding retained research
data. Ordinary restart/persistence checks should use known test records.

1. **Provider capabilities:** public digest image pull; exact-commit private
   repository mount; rejection of changed/missing payloads; dedicated Job
   execution; mandatory Landlock ABI 6/seccomp probe; read-only input and scoped
   output mounts; conditional Bucket create/update semantics and read-after-write
   visibility. An image download alone does not qualify execution.
2. **Research execution:** real representative relativity and quantum workloads,
   private Python workflows, active override layers, report rendering/export,
   literature attach/upload/remove, and autonomous investigation pause/recovery.
3. **Tenant separation:** two ordinary users in different organizations/projects;
   attempted cross-owner access to jobs, attachments, results, state, credentials,
   and BYOS destinations must be rejected. Verify execution cannot read controller
   files, another workload's data, controller secrets, or its Bucket credentials,
   or signal/trace the runner. Exercise direct sockets, inherited descriptors,
   symlink/hardlink escapes, subprocesses, and attempted privilege changes.
   Landlock is not a separate filesystem/PID namespace: some names and metadata
   may remain observable, so do not certify complete namespace invisibility.
4. **Durability:** acknowledge a known transaction/upload, forcibly replace the
   disposable controller and its local disk, and verify the acknowledged state,
   budget, credential references, artifacts, and private continuation checkpoints
   recover. Exercise WAL corruption, missing objects, and conflicting controllers:
   each must fail closed rather than serve older state.
5. **Faults and costs:** storage outage, expired token, provider API timeout around
   submission, cancellation, process/resource exhaustion, Job failure, controller
   restart while a Job is running, and exhausted global/storage budgets. Confirm
   no duplicate remote dispatch, false success, or unproven reservation refund.
   Attempt detached and double-forked children; verify termination of the entire
   dedicated Job before releasing its reservation. Per-process rlimits are not
   a delegated cgroup limit over a process tree; VM capacity and provider timeout
   remain the aggregate boundary.
6. **SaaS access:** secure cookies/CSRF at the canonical HTTPS origin, user account
   registration or invitations, SMTP, upload sizes, download integrity, BYOK/BYOS,
   and admission under concurrent users.

Passing local fixtures is not a substitute for these gates. If a gate fails,
keep registration and researcher access closed, preserve the failure evidence,
and fix/requalify the implementation. The same evidence standard applies to
Linux/Docker deployment validation.


## 10. Troubleshoot the failed checkpoint

Use the matching subsection; do not repeat the entire deployment. Read-only
checks can be run together. Review outputs before sharing: paths, computer names,
bucket names and application logs can be private. Never share HF tokens, S3
secrets, encryption keys, passwords or an unfiltered environment dump.

### Identify the controller's current stage

**Where: Windows CMD.** Inspect the same startup attempt before taking another
action:

```bat
python -c "import sys; from huggingface_hub import HfApi; print(HfApi(token=True).get_space_runtime(sys.argv[1]).stage)" "%PF_HF_APP_REPO%"
hf spaces logs "%PF_HF_APP_REPO%" --build --tail 150
hf spaces logs "%PF_HF_APP_REPO%" --tail 150
```

| Stage or finding | Action |
| --- | --- |
| `BUILDING` | Follow the current build logs; wait for that build to finish. |
| `BUILD_ERROR` | Locate the first build error. Verify the upload contains the entire current generated `space/` context, including `Dockerfile`, `.dockerignore`, `base-code-manifest.json` and its matching `source/`. The packaged builder includes the required controller build inputs. Correct the input or configuration identified by the error before another build. |
| `APP_STARTING` | Allow PostgreSQL restore and synchronous receiver catch-up to finish within their configured bounds. A completed Docker build alone does not establish database readiness. |
| `RUNTIME_ERROR` | Retain the controller commit, timestamps, runtime logs and the configured nonsecret pins. Check the specific failing prerequisite; do not infer a credential failure from a generic `OperationalError` or `RuntimeError`. |
| `RUNNING` with application startup complete | Proceed to application checks; this is not yet proof of a successful simulation or full service qualification. |

For database startup, verify unchanged bucket/prefix and persistent keys,
controller access to that storage, storage/WAL capacity, and the two independent
startup bounds. This release waits for PostgreSQL readiness and batches WAL
catch-up before admitting the application. Set
`PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS=600` for the normal receiver startup
budget; accepted values are `30`–`3600`. A longer CLI wait cannot change that
internal budget. Increase a bound only after establishing that recovery is
making healthy progress. Keep the synchronous readiness requirement intact.

If the cause remains uncertain, pause the failed controller, retain the verified
backup, and obtain a review of the sanitized logs and exact release/pins. Do not
change durable state to make an unexplained startup failure disappear.

### Run stays Queued with no attempt

`attempt_count=0` means the PF worker has not claimed the run. Check admission
before resubmitting. The Space being `RUNNING` only establishes application
startup; it does not establish that the dispatcher can reserve another run.

The deployment allowance is shared across organizations, simulations and admitted
assistant work. An organization's **Usage** page and `hf jobs ps` do not report
that ledger. A terminal HF Job list does not prove that all PF reservations have
settled. Retained charges can include conservatively accounted interrupted or
ambiguous work; do not delete attempts or ledger records to recover allowance.

In the **private configured service environment connected to the actual service
database**, an operator can inspect the packaged deployment control status:

```bash
python -m pf_saas.deployment_control status
```

This is not a command for an unconfigured Windows tools environment. Do not
expose the database or copy its credentials into a shared terminal to run it.
Read `usage.compute_committed_seconds`, `usage.compute_remaining_seconds`,
`usage.monthly_compute_seconds`, `usage.active_reservations` and `unresolved`.
Committed seconds already include active reservations: do not add them twice.
If operator database access is unavailable, obtain an authorized ledger check;
do not infer remaining allowance from tenant usage, elapsed wall time or HF Jobs.

For a normal job with a 600-second timeout, r34 requests **660 deployment
reservation seconds**: 600 execution seconds plus the 60-second launch/stop
margin. This is an admission reservation, not a claim that 660 seconds of
scientific computation occurred or an exact HF invoice charge. Different job
limits and assistant workflows can require different reservations.

As a conditional example, if the current UTC month's verified committed total
is **5,940 seconds**, the configured ceiling is **5,940**, and the intended next
reservation is **660**, a ceiling of **6,600** supplies that one reservation's
headroom. These are example values, not release defaults. Check global
concurrency, optional monetary admission and organization limits as well; raising
the compute ceiling does not resolve those separate blockers.

If the operator approves that additional allowance:

1. Stop new submissions across the deployment. Settle current executions and
   result collection. Review queued runs and investigations: increasing the
   ceiling can admit existing work automatically, so do not leave unintended
   work queued. Confirm provider termination where applicable.
2. Record the current nonsecret settings, immutable controller/runner pins and
   the verified monthly ledger values. Pause the controller as in section 11.2.
3. In **Browser — HF → controller Settings → Variables**, change only
   `PF_SAAS_DEPLOYMENT_MONTHLY_COMPUTE_SECONDS` to the deliberately chosen new
   ceiling. Keep `PF_SAAS_DEPLOYMENT_BUDGET_ENABLED=true`, the concurrency and
   monetary bounds, and the storage/state settings. Do not change the prefix,
   reset accounting, or alter the worker lease/durability requirements.
4. If the settings change starts the Space, pause it again. Read back the new
   ceiling and verify that the other settings and code pins are unchanged.
   For the example above, the operator record would state
   `DEPLOYMENT_CEILING_UPDATED=5940->6600` and
   `ADDITIONAL_ALLOWANCE_SECONDS=660`.
5. Restart once, require `RUNNING` and normal startup logs, then submit or retry
   only the intended bounded run. Require completion and accessible saved
   results as in section 11.7. Recheck admission before another run.

The ceiling is a total for the UTC billing month, not a refill balance. Raising
it preserves the existing charges. It also permits additional paid provider
work; it is not an account-wide HF billing cap. If admission has sufficient
headroom and the run remains queued, retain its job details and controller logs
for dispatcher investigation instead of repeatedly increasing the ceiling.

### Worker lease or result-publication failure

An interrupted attempt and an HF Job's final status describe different layers.
Read the PF attempt's phase, timestamps, `execution_started_at`, error and cleanup
state along with the matching provider identity. An absent execution timestamp
does not establish that scientific execution started. Total wall time includes
preparation, transfers and publication in addition to provider execution.

A `PF_WORKER_HEARTBEAT` controller log event identifies a failed worker renewal
with bounded timing and exception-category fields. Preserve it with the exact
controller and private-code pins. It is evidence for investigation, not proof of
a particular storage or database root cause. Keep lease expiry, synchronous WAL
acknowledgement and ownership fencing enabled; do not increase unrelated compute
or collection limits to hide lost ownership.

For collection or publication errors, use
[result publication and verification](R34_ARTIFACT_PUBLICATION_REPAIR.md).
A complete transfer byte counter or provider `COMPLETED` status is insufficient:
PF must commit its final attempt and make the saved results accessible.

### Measure the stages of a slow run

Read the authorized job/attempt `timings` record and the controller's structured
`PF_EXECUTION_TIMING` log. Older attempts can have no timing record. These
measurements use `controller_monotonic_wall` and distinguish preparation,
uploading, submission, provider queueing, observed execution and collection.
Collection stages distinguish transfer, import, publication and commit when
observed. Heartbeat, progress and publication operation aggregates can overlap
phases; do not add them to phase totals as additional elapsed time.

The database timing snapshot precedes its own final commit acknowledgement;
the final controller log can include that acknowledgement's duration. The
downloadable `timings.json` artifact is an earlier snapshot made before artifact
publication. `provider_runtime_seconds` is `null` when unavailable. An observed
executing phase is not an exact HF Job runtime or billing measurement, and
`accounted_compute_seconds` remains PF's separate internal accounting measure.
Use the matching provider record for provider runtime and billing. These
measurements locate delay without promising a particular completion time.

The workbench adapts automatic polling to activity, page visibility and errors.
Optional controller Variables are `PF_SAAS_UI_POLL_ACTIVE_MS` (5000),
`PF_SAAS_UI_POLL_IDLE_MS` (30000), `PF_SAAS_UI_POLL_HIDDEN_MS` (60000),
`PF_SAAS_UI_POLL_MAX_BACKOFF_MS` (120000) and
`PF_SAAS_UI_POLL_QUALIFICATION_MS` (60000). Values are milliseconds; keep active
at or below idle, idle at or below hidden, and hidden at or below maximum
backoff. Manual refresh and cancellation remain explicit immediate requests.
Polling settings change display request frequency, not execution capacity.

### Python 3.13 / Python 3.12 / package or PATH mismatch

**Where: Windows CMD.** The environment name in the prompt is not proof of the
interpreter actually running:

```bat
where python
where hf
python -c "import sys; print(sys.executable); print(sys.version.split()[0])"
python -m pip show huggingface_hub truststore
```

Activate the section-2 environment and repeat the checks. If an old environment
was built with Python 3.13, create a different new 3.12 environment; do not
overwrite/delete it as a repair. Use `python -m pip` inside the selected
environment. If `hf` still resolves elsewhere, invoke the expected executable:

```bat
call "%PF_HF_TOOLS%\Scripts\activate.bat"
"%PF_HF_TOOLS%\Scripts\hf.exe" auth whoami
```

Installing a package in one environment does not install it into another.
A Downloads copy of a helper is not necessarily the helper packaged in this release.

### Upload fails with Xet HTTP 403

**Where: Windows CMD.** First confirm the destination and current context path.
For a controller operation, first inspect the saved preparation and exact
remote revision. The following read-only check can reconcile an uncertain
commit response; it does not repeat the upload:

```bat
hf auth whoami
dir "%PF_HF_UPLOAD%\space\Dockerfile"
set "HF_HUB_DISABLE_XET=1"
python deploy\saas\huggingface_native\deploy_release.py verify --bundle "%PF_HF_UPLOAD%"
python deploy\saas\huggingface_native\deploy_release.py --system-ca prepare --bundle "%PF_HF_UPLOAD%" --space "%PF_HF_APP_REPO%" --state-dir "%PF_HF_DEPLOY_STATE%" --fresh
python deploy\saas\huggingface_native\deploy_release.py --system-ca apply --state-dir "%PF_HF_DEPLOY_STATE%"
```

Set `HF_HUB_DISABLE_XET` **before** launching the next CLI process, in the same
CMD window. It disables the optional Xet transfer path; it does not bypass access
control or guarantee every 403 is resolved. For an uncommitted preparation, resolve the network error and rerun the same
`apply`; its parent-revision guard prevents replacing a concurrent commit. For
runtime/code uploads, inspect the existing commit before deciding to retry their
matching section-4 command with this setting. Do not upload `space/` to the public
runtime. [HF environment-variable reference](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#hfhubdisablexet).

After the intended upload has succeeded and you have verified its commit, clear
the temporary override before later HF operations:

```bat
set "HF_HUB_DISABLE_XET="
```

If still denied, check that the active token has write access to that exact Space
and that the network can reach HF's transfer services. Keep certificate validation
enabled. A failed attempt does not prove no commit occurred: inspect **Files →
history** and compare its commit with your recorded upload. Align with the
current generated folder, without `--delete`.

### Registry TLS, authentication or missing Docker instructions

**Where: Windows CMD.**

```bat
python deploy\saas\huggingface_native\hf_registry_digest.py --version
python deploy\saas\huggingface_native\hf_registry_digest.py --diagnose --system-ca
python deploy\saas\huggingface_native\hf_registry_digest.py --anonymous --system-ca --image "%PF_HF_RUNTIME_TAG%"
```

The diagnostic reads no HF token. A registry-root 401 can be a normal anonymous
challenge; the selected image's anonymous manifest/digest check must still pass.
`CERTIFICATE_VERIFICATION_FAILED` calls for trusted-CA/clock/proxy repair or
`--system-ca`, not `verify=False`, insecure curl or disabled TLS checks.
If the helper reports that `--system-ca` conflicts with `SSL_CERT_FILE` or
`SSL_CERT_DIR`, retain your approved CA configuration and run the helper without
`--system-ca`. Do not clear an organization's approved CA settings or disable
verification merely to remove the error.

In **Browser — HF**, use the **public runtime** Space's page menu/run-locally
instructions after a successful build. HF calls the feature “Run with Docker”,
but its position may change. A private runner's registry login can verify metadata
yet still fail an HF Job pull. Do not make that private repository public.
If the current public image still gives 401, resolve its public access or publish
only the reviewed public runtime context to another anonymously pullable OCI
registry. The packaged `hf_registry_digest.py` accepts only `registry.hf.space`;
for another registry, use that registry's supported method to verify anonymous
access to the exact image digest, then run the same `qualify.py` qualification.

### Qualification reports RuntimeError or stops after a Job ID

**Where: Windows CMD.** Copy the already-created Job ID; these commands inspect
it and do not submit another paid Job:

```bat
set "PF_HF_TEST_JOB=REPLACE_WITH_EXISTING_JOB_ID"
hf jobs inspect "%PF_HF_TEST_JOB%" --namespace "%PF_HF_OWNER%" --format json
hf jobs logs "%PF_HF_TEST_JOB%" --namespace "%PF_HF_OWNER%"
```

Use this inspection for the credential-free qualification Job, and review it
privately before sharing. JSON avoids the human display truncating pins or mount
details. Check the same Job in **Browser — HF → Jobs**. Record terminal status, image,
command, revision/mount and confinement output. A generic `RuntimeError` has no
proven cause until the identity comparison is examined. Logs showing a passed
probe and **COMPLETED** are evidence for that run, not permission to ignore a
failed metadata check or open the full SaaS to users. Resolve the error's cause before
admission; retain reservations until actual provider termination is confirmed.
If no ID was printed, inspect the account's Jobs list before retrying.

The packaged qualifier supports HF's empty-secrets metadata representations
(`None`, `{}` and `[]`) while rejecting nonempty or malformed values. Use the
qualifier packaged in this release. Keep the public-runtime/private-payload
flow, mandatory confinement, empty secrets and read-only code mount intact.

### Bucket exists in the browser, but CLI says BucketNotFound / access denied

**Where: Windows CMD.**

```bat
hf auth whoami
if defined HF_TOKEN (echo HF_TOKEN override is set) else (echo HF_TOKEN override is not set)
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"
```

**Where: Browser — HF.** Compare the exact Bucket owner/name with its URL.
Then check the **active CLI token's** fine-grained scope: three selected Spaces
do not include the state Bucket. Add only the needed Bucket read/write grant
using the available resource/scope controls, save, and retry. Matching `whoami`
proves account identity, not resource permission.

An `HF_TOKEN` environment variable can override the saved login. If it is an
unintended old override, clear it only in this CMD window and select the intended
token interactively:

```bat
set "HF_TOKEN="
hf auth login --force
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"
```

Do not rotate S3 keys to fix a CLI HF-token error. Creating a Bucket after a token
does not itself require invalidation. S3 keys inherit their parent token's scope,
but do not assume an unverified permission change fixed existing S3 access:
test the controller/S3 path separately. If you replace/revoke that parent token,
update every affected HF token Secret and generate/update the corresponding S3
pair. Separate valid HF and S3 parent tokens need not be identical.
Preserve `PF_HF_STATE_KEY`, the Bucket, prefix and existing objects.

### Email sender stays Unverified / email delivery fails

**Where: Browser — SMTP2GO and DNS provider.** Open the domain's record details,
not just the summary row. Confirm all three CNAMEs exist at the authoritative DNS
provider, with no doubled domain suffix, wrong record type or replaced hosting
record. Read the failed record's tooltip. DNS propagation can take time, but
waiting does not correct a missing/wrong record. **Tracking Disabled** is a
separate setting, not evidence of the verification failure's cause.

**Where: Windows CMD.** Replace these placeholders with the three **full** CNAME
hostnames from SMTP2GO, then run this read-only DNS check:

```bat
nslookup -type=CNAME REPLACE_WITH_FULL_RETURN_PATH_HOSTNAME
nslookup -type=CNAME REPLACE_WITH_FULL_DKIM_HOSTNAME
nslookup -type=CNAME REPLACE_WITH_FULL_TRACKING_HOSTNAME
```

Each answer should point to its supplied SMTP2GO target. A missing answer may
reflect an unsaved/wrong record, the wrong DNS provider or a cached result;
check the provider's saved rows and SMTP2GO's error before deciding to wait.

For delivery, confirm `mail.smtp2go.com`, port `443`, SSL `true`, STARTTLS `false`,
a verified From address and the **SMTP User** credentials. Check SMTP2GO's activity
report for rejection/quota details. Do not put its HTTP API key in the SMTP
password field or switch to development outbox mode in production.

### Build succeeds, but startup reports DurabilityError

**Where: Browser — controller runtime logs and Settings.** A generic
`DurabilityError` is not enough to diagnose a single cause. Capture the time and
sanitized error context. Check the configured owner, bare Bucket name, prefix,
S3 credential pairing/permissions, unchanged state/database keys and available
storage/WAL allowance. A successful CLI Bucket lookup tests HF-token visibility,
not S3 writes, conditional updates, encryption or PostgreSQL restore.

Do not delete durable heads, WAL, ledgers or objects; change the prefix; generate
replacement state/vault keys; force bootstrap; or run a second controller to
bypass the failure. An existing deployment must recover its existing authenticated
state. Keep researcher access closed until the actual failure is resolved.

### App frame blocked, or administrator login fails

**Where: Browser.** Open the direct `https://...hf.space` origin obtained in
section 7; set `PF_SAAS_PUBLIC_URL` to that exact origin. HF private access and PF
login are separate. Do not disable frame, cookie or CSRF protections.

Use the PF administrator email/password from first initialization. Changing
`PF_HF_ADMIN_PASSWORD` later does not reset an existing account. If that password
is lost and SMTP works, use PF's **Forgot password?** flow. HF's masked Secrets
cannot recover the old value. Keep stable encryption/session/database keys.


### “Virtualization support not detected”

This is a local Windows/Docker startup problem. Treat three prerequisites separately: **CPU virtualization in BIOS/UEFI**, the **Windows WSL/Virtual Machine Platform components**, and **hypervisor startup**. Installing Python or changing HF tokens will not enable any of these.

**Where: ordinary Windows CMD.** Run these read-only checks. You can paste the complete block; one failed command does not invalidate the other results.

```bat
ver
systeminfo
wsl --status
wsl --version
wsl --list --verbose
powershell -NoProfile -Command "Get-CimInstance Win32_Processor | Select-Object Name,VirtualizationFirmwareEnabled,VMMonitorModeExtensions,SecondLevelAddressTranslationExtensions"
```

Also open **Task Manager → Performance → CPU** and inspect **Virtualization**. `systeminfo` may say a hypervisor has already been detected instead of listing each Hyper-V requirement; that is not evidence that virtualization is missing. Processor-property output is supporting evidence, especially on virtual machines, rather than the sole pass/fail test. [Docker virtualization troubleshooting](https://docs.docker.com/desktop/troubleshoot-and-support/troubleshoot/topics/#docker-desktop-fails-due-to-virtualization-not-working); [Microsoft processor-property reference](https://learn.microsoft.com/en-us/windows/win32/cimwin32prov/win32-processor).

| Finding | Next action |
|---|---|
| Task Manager shows virtualization disabled | Follow the PC manufacturer's BIOS/UEFI instructions to enable CPU virtualization, commonly named Intel VT-x/Virtualization Technology or AMD-V/SVM. Restart Windows. |
| WSL is missing or its modern version command is unavailable | Complete or update WSL using the standard setup in section 3, then restart when requested. |
| Firmware virtualization is enabled, but WSL reports a missing required feature | Check Windows components and hypervisor startup as below. |
| This Windows PC is itself a virtual machine or managed desktop | Ask its administrator whether nested virtualization is supported and enabled on the host. Guest settings alone cannot expose missing host capabilities. Use the HF-hosted alternative if unavailable. |

**Where: PowerShell as administrator.** Inspect the Windows feature states:

```powershell
Get-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
bcdedit /enum | findstr /i hypervisorlaunchtype
```

If WSL is not installed correctly, first use `wsl --install --no-distribution`. For a manual/older component-based WSL installation, the documented component repair is:

```powershell
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart
```

Restart Windows after enabling the components; `/norestart` only postpones that restart. Modern packaged WSL can differ from the older optional-component setup, so a disabled legacy WSL component alone is not proof that a working WSL 2 installation is broken. [Microsoft manual WSL installation](https://learn.microsoft.com/en-us/windows/wsl/install-manual); [modern WSL installation options](https://learn.microsoft.com/en-us/windows/wsl/basic-commands).

If the boot-configuration check explicitly reports **`hypervisorlaunchtype Off`**, and the computer's administrator permits the change, enable hypervisor startup:

```powershell
bcdedit /set hypervisorlaunchtype auto
```

Restart Windows, start Docker Desktop, and repeat `docker version`. An empty filtered result is not the same as an explicit `Off`; do not change boot settings merely because that line is absent. Firmware and corporate-managed settings should be handled by the machine's administrator. [Microsoft WSL startup troubleshooting](https://learn.microsoft.com/en-us/windows/wsl/troubleshooting#error-0x80370102-the-virtual-machine-could-not-be-started-because-a-required-feature-is-not-installed); [nested virtualization requirements](https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/enable-nested-virtualization).

### Docker login returns a named-pipe HTTP 500

Example of a local Docker engine endpoint:

```text
500 Internal Server Error ... dockerDesktopLinuxEngine/v1.56/auth
check if the server supports the requested API version
```

The named pipe identifies the local Docker Desktop engine connection. This message alone does not establish that HF rejected the token or that API version 1.56 is unsupported. Check the local engine before retrying authentication.

**Where: ordinary Windows CMD.** Read-only checks:

```bat
where docker
docker context show
docker context ls
docker version
docker info --format "{{.OSType}}"
if defined DOCKER_API_VERSION (echo DOCKER_API_VERSION is set) else (echo DOCKER_API_VERSION is not set)
if defined DOCKER_HOST (echo DOCKER_HOST is set) else (echo DOCKER_HOST is not set)
if defined DOCKER_CONTEXT (echo DOCKER_CONTEXT is set) else (echo DOCKER_CONTEXT is not set)
```

1. If the **Server** section is missing or errors, open Docker Desktop, inspect its startup error, and resolve the local engine/virtualization problem first. Restart Docker Desktop through its **Troubleshoot** menu when appropriate; do not use factory reset as the first repair.
2. If the selected context targets a different engine, choose the intended Docker Desktop Linux context shown by `docker context ls`. If it is named `desktop-linux`, this one-command check explicitly selects it:

   ```bat
   docker --context desktop-linux version
   ```

   Context names and environment overrides determine which engine receives a command; do not remove an intentional remote-engine configuration. [Docker contexts](https://docs.docker.com/engine/manage-resources/contexts/).
3. If `DOCKER_API_VERSION` was set for earlier troubleshooting, remove that override **in this CMD window** and retry:

   ```bat
   set "DOCKER_API_VERSION="
   docker version
   ```

   Docker normally negotiates a compatible API version. Do not guess an older API version merely because `/v1.56/` appears in the error. Use the Client/Server output to investigate a genuine version mismatch, and update Docker Desktop through its normal update mechanism if necessary. [Docker Engine API negotiation](https://docs.docker.com/reference/api/engine/).
4. Once Client and Server respond normally, retry `docker login registry.hf.space`
   interactively. Enter your personal HF username and token at its prompts; do
   not substitute an organization owner name or put the token in the command.
5. If Docker remains unavailable, use the packaged `deploy\saas\huggingface_native\hf_registry_digest.py` and the HF-hosted build path. A registry-metadata result establishes only metadata access; the subsequent HF Job test still establishes whether HF can pull and execute the image.

**Collecting diagnostics:** review system, context and log output before sharing
it: computer/domain names or network details may be private. Use the formatted
Docker check above rather than an unrestricted configuration/environment dump.
Never share HF tokens, S3 secrets, passwords or PF encryption keys.


## 11. Upgrade an existing r34 deployment

Use this route when an earlier r34 release is already deployed and running.
It upgrades the existing private controller and runner while retaining the
existing application data. It does not repeat first-time resource creation,
email setup, secret generation or database initialization. The complete release
contains the approved native response/durability corrections and the shared
application improvements. Historical patch installers are not required.

Keep the same state Bucket, deployment prefix, database passwords, state key,
session key and BYOK/BYOS vault keys. Preserve `PF_HF_MAX_STORAGE_BYTES`
at the already approved value: for the deployment raised to 32 GiB this is
`34359738368`. Keep the encrypted quota counter, reservation IDs and audit
history. Conservative lifetime admission remains the default; only explicit
safe-lifecycle policy and verified reconciliation can reclaim eligible charges. Keep the existing controller's valid HF
and Bucket S3 credentials. **This upgrade and its HF-to-HF backup require no
new S3 configuration.** Preserve the old release, its generated contexts, the
old controller revision and its qualified code/image pins for recovery review.
The new full archive and the newly generated private-code archive have distinct
SHA256 values; neither is the runner's Git commit or the runtime image digest.

### 11.1 Prepare the new release and record the running deployment

**Where: Windows CMD / File Explorer.** Compare the downloaded full archive
against its companion checksum, then extract `.xz` and `.zip` into a **fresh**
project directory as in section 2. Retain the earlier extraction separately.

```bat
certutil -hashfile "FULL_PATH_TO_DOWNLOADED_RELEASE_ARCHIVE" SHA256
```

Open the already configured Python 3.12 tools environment. Set the following
values for this upgrade. Replace every `REPLACE_...` value and use the actual
existing resource names if they differ from these examples:

```bat
set "PF_PROJECT_ROOT=D:\pf-r34-upgrade\privileged-frame-simulation"
set "PF_HF_TOOLS=%USERPROFILE%\pf-hf-tools"
set "PF_HF_OWNER=YOUR_HF_USERNAME"
set "PF_HF_APP_REPO=%PF_HF_OWNER%/pf-simulation-saas"
set "PF_HF_CODE_REPO=%PF_HF_OWNER%/pf-simulation-runner"
set "PF_HF_RUNTIME_REPO=%PF_HF_OWNER%/pf-runtime-r34"
set "PF_HF_BUCKET_NAME=pf-simulation-state"
set "PF_HF_BACKUP_BUCKET_NAME=pf-simulation-state-backup-REPLACE_WITH_UNIQUE_SUFFIX"
set "PF_HF_UPLOAD=%USERPROFILE%\pf-hf-native-upload-r34-REPLACE_WITH_UNIQUE_SUFFIX"
set "PF_HF_DEPLOY_STATE=%USERPROFILE%\pf-hf-native-deploy-r34-REPLACE_WITH_UNIQUE_SUFFIX"
call "%PF_HF_TOOLS%\Scripts\activate.bat"
cd /d "%PF_PROJECT_ROOT%"
where python
where hf
python -c "import sys,huggingface_hub; print(sys.executable); print(sys.version.split()[0]); print(huggingface_hub.__version__)"
```

Use Python 3.12 and the guide's Hub SDK version, `1.32.0`, with `truststore`.
If the existing environment needs preparation, use section 2; do not create a
new virtual environment over it. `PF_HF_UPLOAD` names a new generated-context
directory. These CMD variables do not update the remote Space.

**Where: Browser — HF → controller Settings.** Record the existing Variables,
resource visibility/hardware, and secure recovery references for its Secrets.
Specifically retain `PF_HF_JOB_IMAGE`, `PF_HF_ASSISTANT_IMAGE` if set,
`PF_HF_CODE_REPO`, `PF_HF_CODE_REVISION`, `PF_HF_CODE_SHA256`, job flavor,
namespace, Bucket and prefix, startup bounds and budget settings. HF cannot
show Secret values back to you; use the original secure recovery record.
Record the controller's current commit from **Files → history**. Keep these
old identities separate from the new values recorded later.

### 11.2 Stop new work, finish Jobs and pause the controller

**Where: PF application and Browser — HF.** Arrange a maintenance window and
stop users, API clients and scheduled automation from submitting work. Keep
registration disabled. Restrict researcher access for the upgrade using the
deployment's existing access controls; when necessary, use Private Space
visibility and coordinate with HF collaborators who retain access.

Let current PF simulations, private workflows, assistant work and autonomous
investigations finish and collect their results, or cancel them through PF
and verify cancellation in HF Jobs. Check the HF Jobs dashboard for any remaining deployment-owned Jobs,
including previously submitted qualification Jobs. A cancellation request
alone is not proof that a Job has stopped. Do not pause until active work and
its result collection have settled.

**Where: Windows CMD.** With new submissions still stopped:

```bat
hf spaces pause "%PF_HF_APP_REPO%"
python -c "import sys; from huggingface_hub import HfApi; s=HfApi(token=True).get_space_runtime(sys.argv[1]).stage; print(s); raise SystemExit(0 if s=='PAUSED' else 1)" "%PF_HF_APP_REPO%"
```

Require `PAUSED`. Pausing the controller does **not** cancel separately running
HF Jobs. Confirm that no other controller or writer uses this deployment's
Bucket/prefix. Keep the controller paused through backup, code qualification
and configuration. After any HF settings change that starts the Space, pause
it again before proceeding.

### 11.3 Create and verify a private HF backup

**Where: Browser — HF.** Open <https://huggingface.co/new-bucket>, signed in as
the operator for the deployment owner. Create a new **Private** Bucket in the
**same storage region** as the state Bucket. The storage region must match;
a prewarming region is a different setting. Use the exact unique name chosen
in `PF_HF_BACKUP_BUCKET_NAME`.

For example, `pf-simulation-state-backup-20261001` is an arbitrary name for one
maintenance window, not a required name or a Bucket to reuse for every upgrade.
Choose a new suffix for each backup. The destination must be empty before
copying. Keep the original state Bucket selected in the controller settings;
the backup is a separate recovery copy.

At <https://huggingface.co/settings/tokens>, select an operator token with its
existing Space upload/management and Jobs permissions plus these Bucket grants:

| Resource | Required access for this backup |
| --- | --- |
| Existing state Bucket | Read |
| New backup Bucket | Read and write |

Preserve any existing permissions the operator token needs for other deployment
actions. The controller's existing storage credentials are separate and stay
unchanged. Bucket grants are independent of Space grants; account identity
alone does not establish access to either Bucket.

**Where: Windows CMD.** Select that token explicitly for this maintenance
session. Clearing `HF_TOKEN` here removes the environment override of the saved
login. Enter the token only at the private login prompt:

```bat
set "HF_TOKEN="
hf auth login --force
hf auth whoami
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"
hf buckets info "%PF_HF_OWNER%/%PF_HF_BACKUP_BUCKET_NAME%"
python -c "import os; from huggingface_hub import HfApi; a=HfApi(token=True); s=os.environ['PF_HF_OWNER']+'/'+os.environ['PF_HF_BUCKET_NAME']; d=os.environ['PF_HF_OWNER']+'/'+os.environ['PF_HF_BACKUP_BUCKET_NAME']; assert s!=d, 'Source and backup must differ'; assert a.get_space_runtime(os.environ['PF_HF_APP_REPO']).stage=='PAUSED', 'Controller must be paused'; assert a.bucket_info(d).private is True, 'Backup must be private'; assert not list(a.list_bucket_tree(d,recursive=True)), 'Use a new empty backup Bucket'; print('BACKUP_DESTINATION_READY')"
```

Require `BACKUP_DESTINATION_READY`, then copy entirely within HF:

```bat
hf cp "hf://buckets/%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%/" "hf://buckets/%PF_HF_OWNER%/%PF_HF_BACKUP_BUCKET_NAME%/"
```

After `Copied`, verify file paths, sizes and Xet hashes. This reads the source,
the backup and the source again, checks private visibility and confirms that
the controller was paused before and after the comparison:

```bat
python -c "import os; from huggingface_hub import HfApi,BucketFile; a=HfApi(token=True); s=os.environ['PF_HF_OWNER']+'/'+os.environ['PF_HF_BUCKET_NAME']; d=os.environ['PF_HF_OWNER']+'/'+os.environ['PF_HF_BACKUP_BUCKET_NAME']; c=os.environ['PF_HF_APP_REPO']; read=lambda b:{f.path:(f.size,f.xet_hash) for f in a.list_bucket_tree(b,recursive=True) if isinstance(f,BucketFile)}; p=a.get_space_runtime(c).stage=='PAUSED'; x=read(s); y=read(d); z=read(s); ok=s!=d and p and a.get_space_runtime(c).stage=='PAUSED' and a.bucket_info(d).private is True and bool(x) and all(v[1] for v in x.values()) and x==y==z; print('BACKUP MANIFEST VERIFIED' if ok else 'BACKUP NOT VERIFIED'); print('Source files:',len(x),'Backup files:',len(y),'Bytes:',sum(v[0] for v in x.values())); raise SystemExit(0 if ok else 1)"
```

Require `BACKUP MANIFEST VERIFIED`. Save the source/destination names,
maintenance time, counts and command result with the upgrade record. Counts
and bytes depend on your deployment; do not require another deployment's
numbers. This establishes an unchanged matching copy manifest. It does not
constitute a restore drill or independently prove decryption/application
recovery. If it does not verify, keep the controller paused and resolve the
backup before changing deployed code.

### 11.4 Generate matching contexts and choose the runtime

**Where: Windows CMD, new project root.** Build once into the new output folder:

```bat
python deploy\saas\huggingface_native\build_space.py --output "%PF_HF_UPLOAD%"
python -c "import hashlib,json,sys; from pathlib import Path; p=Path(sys.argv[1]); a=json.loads((p/'jobs'/'code-manifest.json').read_text(encoding='utf-8')); b=json.loads((p/'space'/'base-code-manifest.json').read_text(encoding='utf-8')); h=hashlib.sha256((p/'jobs'/'pf-code.zip').read_bytes()).hexdigest(); assert a['sha256']==b['sha256']==h, 'Generated contexts do not match'; print('VERIFIED_CODE_SHA256='+h)" "%PF_HF_UPLOAD%"
```

Record the new `VERIFIED_CODE_SHA256`. Use this same `jobs/` and `space/` pair
through the rest of the upgrade. The complete release builds its required
controller inputs directly; no separately downloaded repair or diagnostic
wrapper is needed.

Earlier r34 builds added the supported FMU and QuTiP runtime dependencies.
Use the current generated context as the compatibility reference rather than
assuming any r34 runtime image matches. Qualification must report both
`confinement_probe_passed=true` and `scientific_runtime_probe_passed=true`.
The standalone edition retains its existing optional-dependency choices.

The public runtime can be reused when its generated context is unchanged and
the recorded digest belongs to the previously qualified deployment. Compare
the new `runtime.zip` with the retained `runtime.zip` used to build that image:

```bat
set "PF_HF_PREVIOUS_UPLOAD=FULL_PATH_TO_RETAINED_PREVIOUS_CONTEXTS"
python -c "import hashlib,sys; from pathlib import Path; digest=lambda p:hashlib.sha256(Path(p).read_bytes()).hexdigest(); a=digest(Path(sys.argv[1])/'runtime.zip'); b=digest(Path(sys.argv[2])/'runtime.zip'); assert a==b, 'Runtime contexts differ; build and qualify the current runtime'; print('RUNTIME_CONTEXT_UNCHANGED='+b)" "%PF_HF_PREVIOUS_UPLOAD%" "%PF_HF_UPLOAD%"
```

When that comparison passes and the retained context is tied to the old image
in your deployment record, use its existing pinned `PF_HF_JOB_IMAGE` digest.
The new code still needs a new qualification Job. If the runtime differs, the
reference context is unavailable, or its image provenance is uncertain, follow
sections 4–5 to upload only the current `runtime/`, wait for its build, verify
anonymous registry access and obtain its immutable digest. Keep both private
Spaces private. Never use a `:latest` tag as the production pin.

### 11.5 Upload the private payload and qualify its exact identity

**Where: Windows CMD.** Upload the generated private runner context:

```bat
python deploy\saas\huggingface_native\deploy_release.py --system-ca upload-code --bundle "%PF_HF_UPLOAD%" --repo "%PF_HF_CODE_REPO%"
```

Save the helper's `code_revision` and `code_sha256`; these identify the exact
verified upload. The helper uses a guarded parent commit and checks the remote
file inventory. Keep concurrent operator changes out of this maintenance window. Set the new commit, the code hash from section 11.4 and the selected
runtime digest. Do not reuse pins from another full archive or an earlier
qualification just because both releases are named r34:

```bat
set "PF_HF_CODE_REVISION=REPLACE_WITH_NEW_40_HEX_PRIVATE_COMMIT"
set "PF_HF_CODE_SHA256=REPLACE_WITH_NEW_64_HEX_PAYLOAD_SHA256"
set "PF_HF_JOB_IMAGE=REPLACE_WITH_FULL_REGISTRY_IMAGE_AT_SHA256_DIGEST"
python deploy\saas\huggingface_native\qualify.py --namespace "%PF_HF_OWNER%" --runtime-image "%PF_HF_JOB_IMAGE%" --code-repo "%PF_HF_CODE_REPO%" --code-revision "%PF_HF_CODE_REVISION%" --code-sha256 "%PF_HF_CODE_SHA256%" --flavor cpu-basic --system-ca
```

Review this **dry run**: exact image, repository, revision and hash; no secrets,
no exposed service/SSH, and the read-only `/pf-code` mount at that revision.
The following command deliberately submits **one paid qualification Job**:

```bat
python deploy\saas\huggingface_native\qualify.py --namespace "%PF_HF_OWNER%" --runtime-image "%PF_HF_JOB_IMAGE%" --code-repo "%PF_HF_CODE_REPO%" --code-revision "%PF_HF_CODE_REVISION%" --code-sha256 "%PF_HF_CODE_SHA256%" --flavor cpu-basic --system-ca --submit
```

Use the actual intended supported flavor instead of `cpu-basic` if needed, in
both commands, and retain the qualified flavor. Require provider stage
`COMPLETED`, `self_test_completed: true` and `confinement_probe_passed: true`.
Record the Job ID, URL and result. `full_saas_live_qualified: false` correctly
means application acceptance is still outstanding.

If a Job ID is printed before an uncertain helper response, inspect that same
Job; do not blindly submit another paid Job:

```bat
set "PF_HF_TEST_JOB=REPLACE_WITH_ALREADY_SUBMITTED_JOB_ID"
hf jobs inspect "%PF_HF_TEST_JOB%" --namespace "%PF_HF_OWNER%" --format json
hf jobs logs "%PF_HF_TEST_JOB%" --namespace "%PF_HF_OWNER%"
```

Compare its immutable identity and confinement evidence as in section 10.
If no Job ID was returned, check the HF Jobs dashboard before retrying. Resolve
any failed qualification check before bringing the upgraded controller online.

### 11.6 Set remote pins, upload the matching controller and restart

**Where: Browser — HF → controller Settings → Variables and secrets.** Keep all
existing storage, key, SMTP, origin and budget configuration. Remove
`PF_HF_BOOTSTRAP` if it is present, whether as a Variable or a Secret; absence
defaults to `false`. This existing deployment must restore its durable state.
Do not restore first-time administrator password settings.

Set these **remote Space Variables**, using the actual values just qualified:

| Variable | Value |
| --- | --- |
| `PF_HF_CODE_REPO` | Current private runner repository. |
| `PF_HF_CODE_REVISION` | New immutable upload commit from section 11.5. |
| `PF_HF_CODE_SHA256` | New verified private-code SHA256 from section 11.4. |
| `PF_HF_JOB_IMAGE` | Qualified immutable runtime digest, retained or newly built. |
| `PF_HF_JOB_FLAVOR` | Flavor used for the qualification. |
| `PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS` | `600`; separate from the PostgreSQL startup timeout. |

If `PF_HF_ASSISTANT_IMAGE` is explicitly configured to a different runtime,
qualify that image against the **new code commit and SHA256** before resuming
those workflows, even when the image digest itself is unchanged. Repeat section
11.5's dry run and paid self-test with that image as `--runtime-image`, keeping
the same new code pins, and record its result separately. Preserve the main
`PF_HF_JOB_IMAGE` pin. Local CMD `set` commands alone do not change remote Variables.
Check the saved values from CMD:

```bat
python -c "import os; from huggingface_hub import HfApi; a=HfApi(token=True); v=a.get_space_variables(os.environ['PF_HF_APP_REPO']); keys=('PF_HF_CODE_REPO','PF_HF_CODE_REVISION','PF_HF_CODE_SHA256','PF_HF_JOB_IMAGE'); assert all(k in v and v[k].value==os.environ[k] for k in keys), 'Controller variables do not match the qualified pins'; assert v.get('PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS') is not None and v['PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS'].value=='600', 'Unexpected receiver startup deadline'; assert 'PF_HF_BOOTSTRAP' not in v, 'Remove bootstrap from Variables'; print('CONTROLLER_CODE_PINS_VERIFIED')"
```

Require `CONTROLLER_CODE_PINS_VERIFIED`; also confirm bootstrap is absent from
Secrets in the HF settings. Verify the flavor matches the qualification.
If changing settings restarted the controller, pause it again. Upload only
the matching current `space/` context, then restart after the upload succeeds:

```bat
python deploy\saas\huggingface_native\deploy_release.py verify --bundle "%PF_HF_UPLOAD%"
python deploy\saas\huggingface_native\deploy_release.py --system-ca prepare --bundle "%PF_HF_UPLOAD%" --space "%PF_HF_APP_REPO%" --state-dir "%PF_HF_DEPLOY_STATE%"
python deploy\saas\huggingface_native\deploy_release.py --system-ca apply --state-dir "%PF_HF_DEPLOY_STATE%"
hf spaces wait "%PF_HF_APP_REPO%" --timeout 25m
python deploy\saas\huggingface_native\deploy_release.py --system-ca status --state-dir "%PF_HF_DEPLOY_STATE%"
hf spaces logs "%PF_HF_APP_REPO%" --tail 150
```

Run these individually. `prepare` is read-only and seals the current revision,
new code pins, configuration fingerprint and exact generated files in a new local
state directory. It schedules deletion of obsolete release-owned source and known
historical registry extras; unexpected unrelated files stop the preparation.
`apply` confirms pause, checks those pins again, submits one guarded commit and
requests restart. It never edits Space Variables, Secrets, Buckets or quota data.
If a write response is uncertain, run `status` first; do not repeat an upload.
A reconciled commit that remains `PAUSED` needs only the ordinary `hf spaces
restart` command, followed by wait and status. Require
`source_bytes_verified=true` and `running_revision_verified=true`. HTTP and
performance acceptance remain separate browser checks.

Save the controller commit. The native startup restores
durable state and runs the packaged application migrations; do not run a local
Windows migration against the hosted database. Require `RUNNING` plus
`Application startup complete` in the runtime logs. If the wait ends in an
error, still retrieve the logs and follow section 10; do not repeatedly restart
or change database state. Build logs are available separately:

```bat
hf spaces logs "%PF_HF_APP_REPO%" --build --tail 150
```

### 11.7 Verify retained data and execution before reopening access

See [result publication and verification](R34_ARTIFACT_PUBLICATION_REPAIR.md)
for the separate transfer, import and publication counters, safe failure codes
and saved-result acceptance checks.

Keep the maintenance window in place while checking the direct application URL:

1. Sign in with an **existing PF account**. Confirm its organizations, projects,
   retained simulation records, attachments and results. Confirm that expected
   BYOK/BYOS settings remain available without exposing credential values.
2. Submit one small representative simulation. Confirm one expected HF Job,
   completed result collection, and the expected plots/report/downloads in PF.
3. Run a separate small cancellation test. Cancel through PF, confirm provider
   termination and the application's final cancellation state, and check that
   it can admit the next permitted workload.
4. Verify normal durable writes and persistence using known test records.
   Schedule any destructive restore/fault qualification in a separate disposable
   validation deployment, as described in section 9.
5. Record outcomes against this full release checksum, new code hash, runner
   commit, runtime digest, controller commit, variables and Job IDs. Perform the
   remaining applicable service/authorization checks from section 9 before
   treating the complete deployment as qualified.

`RUNNING` and the confinement self-test are separate checkpoints; neither proves
that these application checks passed. Reopen the previous intended researcher
access only after the required checks succeed. Retain the private backup and
old deployment records for your recovery/retention policy; do not automatically
delete them on the first successful startup.

### 11.8 If the upgrade cannot be accepted

Stop new submissions and settle or cancel any test Jobs, then pause the
controller. Preserve the verified pre-upgrade backup, current state Bucket,
keys, old/new deployment identities and failure evidence. A controller startup
can claim durable ownership and publish recovery/WAL state even before the UI
opens, so failure does not establish that remote state was untouched.

Review compatibility before selecting rollback. Reverting only a code pin is
not a complete rollback: the controller source, private payload/revision/hash,
runtime and configuration must form a known matching qualified deployment.
An older controller is not automatically compatible with state written by a
newer one. If recovery requires the pre-upgrade Bucket copy, use a reviewed
restore procedure and explicitly account for changes made after that copy.
Do not copy it over the active Bucket, enable bootstrap, change keys/prefix,
or start a second controller as an improvised rollback. Resume service only
after restore and application acceptance have been established.


## Deployment-wide cost admission and storage maintenance

Compute admission is **deployment-wide**, in addition to existing user/org
quotas. Set finite compute seconds, concurrency, runtime, input/output, and
storage bounds before enabling researcher access. The optional monetary estimate
uses integer micro-USD settings:

- `PF_SAAS_DEPLOYMENT_BUDGET_MONTHLY_MICRO_USD`
- `PF_SAAS_DEPLOYMENT_BUDGET_FIXED_MICRO_USD`
- `PF_SAAS_DEPLOYMENT_BUDGET_SAFETY_MICRO_USD`
- `PF_SAAS_DEPLOYMENT_BUDGET_HOURLY_MICRO_USD`

Enter current provider rates for the selected hardware and a conservative
allowance for fixed costs. Reserving a Job includes bounded startup time. An
ambiguous dispatch is not retried blindly, and its reservation is not released
until provider termination can be established. A timed-out local HTTP request
does not prove that a remote Job stopped.

These controls stop **PF-controlled new work**. They are not a contractual cap
on the HF invoice: subscriptions, Space uptime, storage, API operations, provider
billing granularity, other account activity, and stale/orphaned resources can
still incur charges. User BYOK and BYOS services have their own billing controls.
Review HF billing and provider-level limits directly; do not describe the PF
estimate as an absolute maximum bill.

The storage meter conservatively retains reservations after ambiguous writes
and deletions. It does not automatically recycle capacity or delete backup/WAL
history. Near exhaustion, stop admitting work and conduct verified maintenance
before the reserved WAL capacity is exhausted. Do not reset/delete the meter,
change the prefix, or increase concurrency merely to bypass a stopped service.

To review or increase the byte ceiling deliberately, first stop the controller
and establish that provider Jobs are stopped. In a private operator environment
with the same bucket/prefix and credentials (not an ordinary unconfigured Windows shell), use:

```bash
python -m pf_hf_native.quota inspect
python -m pf_hf_native.quota raise-limit --new-limit-bytes NEW_CEILING --reason "Reviewed storage capacity increase" --confirm-supervisor-stopped
```

Use the existing ceiling in `PF_HF_MAX_STORAGE_BYTES` for this maintenance action;
after its successful conditional update, set the new ceiling in the Space and
restart. The operation retains usage/reservations and an audit record; it is an
explicit operator cost decision, not a reset. Keep the WAL reserve unchanged
unless a separately reviewed migration supports changing it.


## Increasing native capacity without replicating the controller

Concurrent users are not the same as simultaneous simulation runs. Many researchers can be signed in while only a bounded subset of jobs execute; queued jobs also consume admission/storage reservations. Configure all three rather than treating a user count as a worker count.

Start with one slot and one active run. After the existing single-run, cancellation, recovery and publication checks pass, a small two-run trial can use:

| Space Variable | Initial one-run deployment | Two-run qualification trial |
|---|---:|---:|
| `PF_SAAS_WORKER_SLOTS` | `1` | `2` |
| `PF_SAAS_USER_MAX_CONCURRENT_JOBS` | `1` | `1` |
| `PF_SAAS_USER_MAX_QUEUED_JOBS` | `20` | `20` |
| `PF_SAAS_MAX_CONCURRENT_JOBS` | `1` | `2` |
| `PF_SAAS_DEPLOYMENT_MAX_CONCURRENT_JOBS` | `1` | `2` |

This example allows two different users in the same organization to run concurrently, subject to current compute/storage reservations and provider capacity. A single user remains at one active run; increase that user's applicable cap deliberately if the trial specifically needs two runs by the same user. Raising these limits does not replenish consumed monthly allowance, storage admission or provider credits.

Set the Variables on the existing controller Space, retain `PF_SAAS_DEPLOYMENT_PROFILE=single`, and use a controlled restart after draining existing work. Do not create another controller or point an ordinary external worker at the native database. Submit three bounded jobs across authorized users: verify two actual provider executions overlap, the third remains queued, and it starts only after a slot is released. Then exercise cancellation, a controller interruption and concurrent downloads/private-state mutations; inspect attempt IDs, tenant isolation, lease gaps, immutable manifests and charged reservations. Each live trial consumes provider resources.

Increase to the next measured level only after recording response latency, queue delay, provider startup/runtime, upload/commit latency, controller memory/CPU, PostgreSQL connection/lock pressure and durable WAL latency. Native HTTP request admission and the isolated lease connection pool protect different resources. Start with the default SQL pools; increase lease capacity only within the profile's combined pool ceiling and after measuring commit delay. HF API/dispatcher threads share one main database client. Optional assistant stores and maintenance commands use additional connections; PostgreSQL remains capped at 60 connections. Keep one maintenance action at a time.

When API, database, durable-writer or provider limits require independently scalable tiers, use [the distributed deployment guide](../DISTRIBUTED_SAAS_DEPLOYMENT.md). That migration adds shared PostgreSQL/object storage, independent API replicas and dedicated worker hosts. It preserves the same user, organization and deployment admission concepts, while changing the infrastructure. The [capacity qualification guide](../deploy/saas/distributed/CAPACITY_QUALIFICATION.md) separates authenticated-user load measurements from scientific throughput. Neither a 64-slot setting nor a large distributed ceiling certifies hundreds of thousands of concurrent users or simulations.

## Architecture and execution limits

This native HF profile remains one controller, with configurable internal
dispatcher capacity. `PF_SAAS_WORKER_SLOTS` starts that many worker threads,
each responsible for at most one scientific attempt at a time. The default is
one and the software ceiling is 64; this is a bound, not a qualified load level.
Per-user, organization and deployment reservations still decide which work can
start. Increasing a quota without available slots and provider capacity does
not create additional execution throughput.
For separate API replicas and multiple Linux/Docker worker hosts sharing
PostgreSQL and object storage, use
[DISTRIBUTED_SAAS_DEPLOYMENT.md](../DISTRIBUTED_SAAS_DEPLOYMENT.md). That profile
provides configurable capacity and admission controls; neither profile claims
qualification for hundreds of thousands of concurrent simulations. Establish
capacity, provider quotas, cost controls and failure recovery for the actual
deployment before admitting that workload.

| Component | Location and responsibility |
| --- | --- |
| Trusted controller | One Docker Space runs the PF SaaS API, the configured pool of dispatcher threads, and one native supervisor/durable owner. |
| Database | PostgreSQL 15 on ordinary local Space disk; encrypted base backups and synchronous WAL are persisted directly through the HF Bucket S3 API. |
| Scientific execution | A separate HF Job VM for each invocation, with a public dependency image pinned by digest, a private code payload pinned by commit and SHA256, and mandatory Landlock/seccomp confinement. |
| Persistent artifacts | A private HF Bucket; application metadata references encrypted objects. |
| Private assistant/BYOS state | Restored before the API starts and persisted through the native state adapter. |
| Credentials | Space Secrets; user BYOK/BYOS credentials keep the existing owner binding and vault encryption. Jobs receive no controller credentials. |

Space disk is a working copy. PostgreSQL does **not** use `hf-mount` or a bucket
mount as its data directory. An acknowledged transaction depends on the remote
WAL receiver durably publishing its encrypted bytes and advancing a conditional
head. Loss of bucket access, a conflicting controller, or exhausted WAL capacity
closes admission; there is no asynchronous fallback. Crash recovery starts from
the remote base backup and authenticated WAL before exposing port 7860.

Only one controller is supported. Do not enable replicas or start a second
ordinary `run_pf_saas.py worker` against the native database. A replacement
controller fences the previous owner. The public API and all worker threads
share the same process, service database pools and ownership guard.

The default public runtime image contains the scientific and reporting dependencies
from the existing SaaS route and a generic checksum-verifying bootstrap. It contains
no PF source, credentials, user data, or private workflows. PF code is delivered
separately from a private repository at an exact commit and checked before use. Optional licensed/external research toolchains still require
their applicable software, configuration, and rights; HF hosting does not supply
MATLAB licenses, external hardware, or arbitrary remote services. Custom runtime
images must preserve the security contract and be requalified. A public runtime
image must contain only material that you have reviewed and may redistribute.

### Execution boundary and supported behavior

The HF runner uses one dedicated Job/VM per invocation and never pools mutually
untrusted researchers in one Job. Its unprivileged supervisor checks Landlock
ABI 6 or later, seccomp, `no_new_privs`, process cleanup, and the effective access
restrictions before executing a submitted program. Missing support or a failed
probe blocks execution. This profile does not require Bubblewrap, a Docker socket,
root, nested namespaces, mount privileges, or the experimental HF Sandbox API.

Within the Job, the submitted program can read the prepared `/source`, workload
helper files in `/trusted`, and explicitly allowed runtime dependencies, and
write its assigned work tree. The active bootstrap/runner source at
`/opt/pf/source` is not readable by the workload. The dedicated Job's initially
empty `/dev/shm` is writable for POSIX semaphores and shared-memory IPC; its
capacity is bounded by the VM's memory. Landlock controls access to content; it
does not make this a separate filesystem or PID namespace, and some path/process
metadata may remain visible. Metadata-changing operations such as `chmod`,
`utime`, and extended-attribute writes are blocked globally by seccomp because
Landlock does not mediate all those operations, including on the work tree.
Controller secrets and other tenants' inputs are not mounted into the Job.
Landlock signal/abstract-socket scoping and seccomp protect the supervisor from
the workload even though both use the image's unprivileged UID.

Network sockets are blocked for untrusted programs, including local listeners
and direct external API calls. Anonymous Unix socket pairs remain available for
in-workload communication. Existing trusted BYOK/BYOS orchestration remains in
the controller; do not place those credentials in the Job. Programs requiring a
local server, `multiprocessing.Manager`, or direct network access may need an
approved compatible workflow design. Ordinary multiprocessing pools, threads,
subprocesses, and new sessions/process groups are supported by the confinement
design. The supervisor acts as a subreaper and terminates/reaps detached
descendants before output collection. Test the actual scientific libraries and
external-tool adapters before promising support for a particular workflow.

Resource limits bound each process; the selected VM and provider timeout bound
the complete Job. Process-count limits are per UID, and output-directory checks
are admission/collection checks, not hard disk quotas or delegated cgroup limits.
The runner closes inherited descriptors, removes untrusted environment entries,
and reaps its children before output collection. The controller retains capacity
and budget reservations until the provider confirms the whole Job has terminated.
A passing probe is enforcement evidence, not a security certification or signed
attestation. The broader live checks in section 9 remain mandatory.

Administrator-approved research profiles can select additional immutable images
for optional scientific toolchains. Keep operator license files and service keys
out of images used for arbitrary private code; preserve the existing profile
policy and qualify each selected image. A license requiring unrestricted outbound
network access is not compatible with the native untrusted-workload network policy.

The application Space and Jobs can use different hardware. Choose Space memory
and local disk sufficient for PostgreSQL, private state restore, API requests,
and temporary transfers. Free/sleeping hardware is not a guarantee of continuous
availability. Avoid scheduled sleep during active investigations.

The initial native recovery implementation bounds the extracted database backup
at 8 GiB and stages compressed backup/WAL data in memory. Qualify the chosen
Space's RAM, disk, and recovery time against your retained data volume before
increasing storage allowances; this profile does not establish arbitrary
database scale. A larger storage ceiling alone does not raise the restore bound.


## Operation, restart, and recovery

- Start/restart only the native application Space. Do not manually start an
  additional controller or change PostgreSQL to asynchronous commit.
- The supervisor closes API admission if its worker, WAL receiver, or ownership
  guard fails. Provider Jobs may outlive a crashed controller; the durable
  journal retains their identity and unresolved budget reservations.
- Ordinary restart restores from the encrypted remote head. Never create a fresh
  empty cluster to conceal failed recovery. PostgreSQL major-version changes
  require a tested migration; do not point a different major binary at a backup.
- Before upgrades, retain the release/image identity, keys, and a verified
  recoverable backup. Stop new work, let active work finish/cancel with terminal
  confirmation, then replace the controller and perform the recovery checks.
- Monitor the HF Jobs list, account billing, local disk, durable byte meter, and
  remaining WAL reserve. HF bucket storage is mutable and non-versioned; the
  application's immutable encrypted objects are not protection from an operator
  deleting the bucket or losing the encryption key.
- Keep an independently retained, verified recovery copy for operational disaster
  recovery. A second HF Bucket can keep hosting within HF, but validate the copy's
  complete head/base/WAL/object dependency set before calling it a backup.
- An uncertain operation is an operator reconciliation case. Establish remote
  Job termination and durable receipt state before releasing reservations or
  removing transfer objects. Never infer termination solely from elapsed time.

From a private operator shell inside the running trusted controller environment,
inspect the durable reservation ledger with:

```bash
python -m pf_hf_native.control status
python -m pf_hf_native.control reconcile RESERVATION_ID
python -m pf_hf_native.control cancel RESERVATION_ID
```

Use a reservation identifier returned by `status`. `reconcile` checks provider
identity and terminal state without requesting cancellation; `cancel` requests
cancellation and then checks again. Neither command releases concurrency until
all recorded child Jobs are positively confirmed stopped. A successful check
retains the entire reserved compute charge. An uncertain result returns a
nonzero exit status and keeps the reservation; investigate provider state and
retry reconciliation after it becomes conclusive. Missing submission evidence
does not establish that a Job never launched. Do not edit database rows to
force a refund. Terminal transfer objects left by a crashed controller are
eligible for cleanup on the next normal native startup.

Storage metering is a lifetime admission bound, not measured current occupancy
or a monthly reset. Base backups and WAL are retained, and deleted/uncertain
objects remain charged. If the meter's reservation index fills, the offline
`pf_hf_native.quota compact-index` maintenance command can consolidate its
entries while preserving all charged usage; review its `--help` and stop the
supervisor first. Replayed old operations can then be charged again. There is
no automatic history garbage collection in this release.


## Compatibility and evidence

The native profile is selected by its own Dockerfile/launcher and the explicit
`huggingface` execution/storage settings. Existing standalone and Linux/Docker
launch commands, requirements files, and reference infrastructure remain their
existing routes. The optional hybrid gateway remains available. Native Python
dependencies are pinned separately in
`deploy/saas/huggingface_native/requirements.lock.txt`.

Local tests under `tests/test_hf_native_*.py` exercise contracts, fault handling,
confinement policy, persistence/restore rules, and deployment packaging. They
include checks that the public runtime context contains no private PF code and
that ordinary Linux/Docker and standalone entry points retain their defaults.
Local kernel execution is recorded separately from mocked provider contracts.
The local Linux 6.12.107 VM checks exercised Landlock ABI 6 enforcement with
Python 3.12.14, NumPy linear algebra, multiprocessing pools, threads, subprocesses,
and detached-descendant cleanup on timeout/cancellation. These checks did not
run in an HF Job or validate its image pull, repository mounts, or SaaS recovery.
The native Jobs serialization test uses the real pinned `huggingface_hub==1.32.0`
SDK with mocked HTTP transport; it validates client request/response contracts
without contacting or executing on HF.
They must be reported separately from actual HF execution, PostgreSQL crash
recovery, independent security assessment, scientific validation, and billing
qualification. No live deployment or provider purchase is performed by the
builder or by the release tests.

Use the [deployment record template](HUGGING_FACE_DEPLOYMENT_RECORD.md) to record
your own identities, commands, outcomes and remaining acceptance checks. Prior
successful Jobs qualify only their recorded image/code/hardware combination;
they are not reusable pins or full SaaS approval for a newly generated payload.


## Official platform references

- [Spaces overview and visibility](https://huggingface.co/docs/hub/spaces-overview)
- [Docker Spaces](https://huggingface.co/docs/hub/spaces-sdks-docker)
- [Space image access](https://huggingface.co/docs/hub/spaces-run-with-docker)
- [HF Jobs images](https://huggingface.co/docs/hub/jobs-images)
- [HF Jobs configuration and volumes](https://huggingface.co/docs/hub/jobs-configuration)
- [HF Jobs VM and nested-namespace limitations](https://huggingface.co/docs/huggingface_hub/v2.0.0/concepts/sandbox)
- [Linux Landlock security and limitations](https://docs.kernel.org/userspace-api/landlock.html)
- [Storage Buckets](https://huggingface.co/docs/hub/storage-buckets)
- [Bucket permissions](https://huggingface.co/docs/hub/storage-buckets-security)
- [HF CLI copy](https://huggingface.co/docs/huggingface_hub/guides/cli#copy-files)
- [Bucket S3 credentials and addressing](https://huggingface.co/docs/hub/storage-buckets-s3)
- [PostgreSQL synchronous replication](https://www.postgresql.org/docs/15/warm-standby.html#SYNCHRONOUS-REPLICATION)

- [HF CLI 1.32.0 commands](https://huggingface.co/docs/huggingface_hub/v1.32.0/package_reference/cli)
- [HF token scopes](https://huggingface.co/docs/hub/security-tokens)
- [Docker Desktop Windows installation](https://docs.docker.com/desktop/setup/install/windows-install/)
- [Microsoft WSL commands](https://learn.microsoft.com/en-us/windows/wsl/basic-commands)
- [SMTP2GO SMTP ports and credentials](https://developers.smtp2go.com/docs/smtp-relay)
- [SMTP2GO verified senders](https://support.smtp2go.com/hc/en-gb/articles/115004408567-Verified-Senders)
- [SMTP2GO setup at Network Solutions](https://support.smtp2go.com/hc/en-gb/articles/360022581174-DNS-Setup-for-Network-Solutions)

Review current account eligibility, prices, and platform behavior when deploying;
documentation and service capabilities can change independently of r34.


## 2026-10-14 HF Astropy reference-cache maintenance

See [R34_ASTROPY_CACHE_FIX_20261014.md](../R34_ASTROPY_CACHE_FIX_20261014.md) for the small native-worker cache-directory correction, targeted validation, and compact deployment patch. Each run still computes fresh results; scientific models and isolation restrictions are unchanged.


## 2026-10-15 private-workflow reliability maintenance

See [R34_WORKFLOW_RELIABILITY_FIXES_20261015.md](../R34_WORKFLOW_RELIABILITY_FIXES_20261015.md) for workflow refresh coalescing, private-authoring loading/error recovery, request-scoped storage reads, sanitized database diagnostics, and HF execution path/error corrections. The shared changes apply to standalone and SaaS components; native path staging applies to HF. The schema-32 release receipt records targeted checks only and preserves all historical validation evidence. Verify the full release with `python release_tools/package_r34_workflow_reliability.py verify --package`.