PF Simulation r34 — complete Hugging Face deployment guide

Current release — 16 October 2026: the full r34 archive includes workflow reliability and assistant consolidation. Verify this archive with python release_tools/package_r34_assistant_consolidation.py verify --package. Follow the native upgrade guide for an existing service. The dated sections below retain their original maintenance scope; their earlier verification commands are historical.

2026-10-11 maintenance: read the tracking recovery guide for the merged preparation/cache, log buffering and controller diagnostic fixes. The execution recovery guide remains applicable to the earlier Aer sandbox, live-log and Orbit Viewer corrections. The package verifier for that retained 11 October revision was python release_tools/package_r34_tracking_recovery.py verify --package. Earlier verification commands below describe their original release revisions.

Current public-service and schema-10 upgrade instructions: r34 public service deployment. This includes both 2026-10-09 lifecycle packages. Large active-user capacity requires the measured qualification described there.

Use the complete r34 release supplied with this guide. Extract it into a new local project folder; this installation does not require the earlier compatibility update ZIP or separately downloaded repair scripts. Compare the archive's SHA256 with its companion checksum before extraction. Keep the complete release and its validation records.

For receiver startup and durability settings, also read WAL catch-up and batching.

This native mode runs the PF SaaS application, PostgreSQL process, scientific Jobs and primary durable storage on Hugging Face. No separate Linux server is required. Email delivery and users' BYOK/BYOS providers remain external services. The standalone and ordinary Linux/Docker deployment instructions remain separate.

If you already deployed: a clean local extraction is not a remote reset. Preserve your private Spaces, bucket, deployment prefix and all encryption, session and database keys. Do not discard working state or substitute fresh keys. Sections 1–9 cover a new installation. For an earlier r34 deployment that is already running, use the complete upgrade procedure in section 11. The recovery guide explains the recovery boundaries.

Where each action happens

Paste each command as one line in the named shell. Do not copy a prompt such as D:\>, PDF line numbers, or visual line wraps. Windows CMD uses %NAME% variables; PowerShell uses different syntax. Commands here are CMD unless labelled otherwise. Replace YOUR_HF_USERNAME and other REPLACE_... placeholders before running. Run setup and upload commands one at a time and stop if a command fails. Only the explicitly labelled read-only diagnostic batches may continue after an individual failure.

Milestones: create resources → prepare Windows tools → build/upload the three contexts → record immutable pins → run one confinement test → configure email and controller → verify startup → qualify the complete service. A successful upload or runner test is not proof that the complete SaaS deployment is ready.

Consolidated r34 starting point

The October 2026 consolidated release incorporates the approved native Latency, Application Latency (final corrected form), Startup Recovery, Response Correction, Runtime Recovery, WAL Batching, Ownership Queue and Persistence Response Recovery changes. It also contains the shared paging, bounded admission, database-pool, lease/recovery and polling improvements. Do not replay the historical patch ZIPs. User-reported live responsiveness of the prior native patch combination is recorded separately from this release's offline regression results.

Native HF remains one PostgreSQL authority inside the private controller, with its encrypted state and WAL in the existing Bucket. No external database service is required. The optional managed PostgreSQL route is a separate explicit choice; see deployment choices. Preserve PF_SAAS_DEPLOYMENT_PROFILE=single for this guide.

The shared application schema is now 10. Native startup restores the existing cluster, applies its additive schema upgrade and initializes shared counters and admission tables before serving. Existing users, keys and quota reservations are retained. Take the section-11 backup first: returning to schema-9 or older code is not a code-only rollback after migration. External PostgreSQL migrations are not part of the native route.

For the shortest existing-Space route, see the consolidated native upgrade checklist.

1. Create the HF resources and their credentials

Where: Browser — Hugging Face.

1. Create an account at https://huggingface.co/join, verify its email, and enable account security/recovery controls. Review current eligibility and charges for Docker Spaces, Jobs and Buckets before purchasing. HF currently requires a paid eligible account to create Docker Spaces; CPU Basic having no hourly Space charge does not make the whole deployment free.
2. At https://huggingface.co/new-space, create these three Docker Spaces. Select your username/organization as owner, Docker as SDK, and a blank Docker template if offered. The names below are used by the commands; use other names only if you change every matching reference.
3. At https://huggingface.co/new-bucket, create the private state Bucket below. Create it before configuring the token so it can be included in its scope.

Never make a previously private runner/controller Space public. Its Git history may contain proprietary code. A new public runtime Space must contain only the reviewed runtime/ context, with no PF source, user files or secrets. Keep the Bucket private. Existing private code/controller Spaces may be reused.

4. Open https://huggingface.co/settings/tokens. Create a fine-grained setup token, for example pf-r34-setup, with read/write access to those three Spaces, read/write access to the state Bucket, and permission to start/manage Jobs. Jobs management includes inspecting and cancelling the operator's Jobs. Space permissions do not implicitly include a separate private Bucket.
5. In the resource selector, search for the Bucket name and select the Bucket belonging to your owner, if it is offered there. Confirm its resource type and full owner/name; do not select a similarly named Space. If the UI cannot find it, check the Bucket's exact URL and available scope controls rather than granting every resource or recreating the Bucket.
6. Save the token securely. In that token's menu choose Generate S3 credentials and save both the HFAK... access-key ID and matching secret access key. The secret is displayed once. A separate narrower controller token may later replace the setup token: it needs Jobs management, private code access and read/write access to the deployment Bucket, not upload rights to all Spaces.

HF-native CLI access uses the HF token; the S3 API uses the S3 pair derived from an HF token. S3 permissions inherit the parent token's scope. These are different credentials; an S3 key is not an hf auth login token. HF S3 credential documentation.

Checkpoint: all four resources exist with the stated visibility; the token's scope includes the Bucket as well as the Spaces. Ordinary PF users must never receive this operator token.

2. Prepare the full release and Windows tools

Where: Windows CMD. To check the download, replace the path in this example and compare the result with the separately supplied checksum:

certutil -hashfile "FULL_PATH_TO_DOWNLOADED_RELEASE_ARCHIVE" SHA256

Where: File Explorer / archive utility. If the download ends in .zip.xz, extract the outer .xz first to obtain the .zip, then extract that ZIP into a fresh directory. 7-Zip or another compatible utility can open the outer layer; Explorer's Extract All can extract the resulting ZIP. Do not run scripts inside an archive preview or overwrite an earlier project. Locate the project root containing run_pf_saas.py, pf_hf_native and deploy. All helpers used below are already under deploy\saas\huggingface_native in that release. Do not use earlier scripts from Downloads.

Where: Windows CMD. Set your nonsecret working values. Change the project path to your actual extraction, and the owner to your HF username/organization:

set "PF_PROJECT_ROOT=D:\privileged-frame-simulation"
set "PF_HF_OWNER=YOUR_HF_USERNAME"
set "PF_HF_TOOLS=%USERPROFILE%\pf-hf-tools-r34"
set "PF_HF_UPLOAD=%USERPROFILE%\pf-hf-native-upload-r34-full"
set "PF_HF_DEPLOY_STATE=%USERPROFILE%\pf-hf-native-deploy-r34-full"
set "PF_HF_RUNTIME_REPO=%PF_HF_OWNER%/pf-runtime-r34"
set "PF_HF_CODE_REPO=%PF_HF_OWNER%/pf-simulation-runner"
set "PF_HF_APP_REPO=%PF_HF_OWNER%/pf-simulation-saas"
set "PF_HF_BUCKET_NAME=pf-simulation-state"
cd /d "%PF_PROJECT_ROOT%"
py -3.12 --version

These set values last only in this CMD window. If you reopen it, repeat this block and the activation command below. They do not configure your HF Space.

Use 64-bit Python 3.12. If it is missing, install it from Python's official Windows downloads and confirm py -3.12 --version works. The operator tooling was exercised with Python 3.12.10; use a maintained compatible 3.12 installation and record the patch version. Do not let an unrelated Python 3.13 environment silently run these tools.

Create the tools environment only when PF_HF_TOOLS names a new directory. An existing correctly configured 3.12 environment can instead be activated; do not overwrite an existing 3.13 environment.

py -3.12 -m venv "%PF_HF_TOOLS%"
call "%PF_HF_TOOLS%\Scripts\activate.bat"
python -m pip install "huggingface_hub==1.32.0" "truststore>=0.10,<1"
where python
where hf
python -c "import sys,huggingface_hub; print(sys.executable); print(sys.version.split()[0]); print(huggingface_hub.__version__)"
hf auth login
hf auth whoami
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"

In the login menu choose Paste an access token, if offered, and enter the setup token only at the private prompt. Do not use --token SECRET in a command. The first python and hf paths should be inside the tools environment; versions should show Python 3.12 and Hub SDK 1.32.0. A successful Bucket information check confirms this HF token can see it; it does not yet prove S3 writes/durability. If any checkpoint fails, use section 10 before proceeding.

3. Choose the Docker tooling route

Both routes deploy Docker images on HF. The cloud-build route used in sections 4–8 needs Python and the HF CLI, but no local Docker daemon. HF builds the uploaded Dockerfiles. The packaged registry helper reads image metadata without Docker. It does not replace the subsequent live HF Job test.

Standard option: install Docker Desktop for local Docker commands

Where: your Windows PC, then Administrator PowerShell / Docker Desktop.

1. Check Docker's current Windows requirements: supported 64-bit Windows, CPU SLAT support, at least 8 GB RAM, and hardware virtualization enabled in BIOS/UEFI. The WSL 2 backend requires WSL 2.1.5 or later. On a managed PC/virtual machine, confirm administrator/host support.
2. Administrator PowerShell: if WSL is absent, install it without a separate Linux distribution, then restart Windows when requested:

wsl --install --no-distribution

If WSL is already installed, update it instead:

wsl --update

3. Install Docker Desktop from its official page. Select the WSL 2 backend, start Docker Desktop, and complete its first-run prompts.
4. Docker Desktop: use Linux containers and, where shown, Settings → General → Use the WSL 2 based engine. Wait for the engine to run.
5. Windows CMD: check the local server:

wsl --version
docker version
docker info --format "{{.OSType}}"

Expected: both Client and Server sections, followed by linux. A client version alone is not a working engine. A separate Ubuntu installation is not needed for these CMD commands. Microsoft WSL options.

After the runtime has built in section 4, you can optionally authenticate your local Docker client for registry operations. Windows CMD:

docker login registry.hf.space

At its prompts, enter your personal HF username and an HF token as the password. An organization owner name is not the username of the person logging in. Keep the token out of command arguments. This local login is optional and does not replace section 5's anonymous public-image check or live HF Job test.

If local Docker cannot run, continue with the cloud-build route below. The virtualization and named-pipe HTTP 500 troubleshooting in section 10 explains local repair; neither problem requires changing PF's Linux/Docker server edition.

4. Build and upload the runtime and private code

Where: Windows CMD, activated tools environment, project root.

cd /d "%PF_PROJECT_ROOT%"
python deploy\saas\huggingface_native\build_space.py --output "%PF_HF_UPLOAD%"

PF_HF_UPLOAD must name a new, nonexistent output directory. If it already exists, choose a new name, change that variable, and keep using it for every upload. Do not mix an old pf-hf-native-upload folder with the current build.

The packaged compatibility-named alias below delegates to the same current three-context builder; it is an alternative command, not an additional patch:

python deploy\saas\huggingface_native\build_space_windows_fix.py --output "%PF_HF_UPLOAD%"

Run one builder command, not both against the same output. Both accept --source and --output. The builder produces runtime/, jobs/, space/ and matching ZIPs/manifests. It checks an explicit input allowlist and rejects symlinks. The controller context includes base-code-manifest.json; keep it with its matching source/ tree and jobs/pf-code.zip. Native startup checks this inventory against PF_HF_CODE_SHA256 before admitting a new execution. Verify that the generated runner and controller describe the same actual payload:

python -c "import hashlib,json,sys; from pathlib import Path; p=Path(sys.argv[1]); a=json.loads((p/'jobs'/'code-manifest.json').read_text(encoding='utf-8')); b=json.loads((p/'space'/'base-code-manifest.json').read_text(encoding='utf-8')); h=hashlib.sha256((p/'jobs'/'pf-code.zip').read_bytes()).hexdigest(); assert a['sha256']==b['sha256']==h, 'Generated contexts do not match'; print('VERIFIED_CODE_SHA256='+h)" "%PF_HF_UPLOAD%"

Require VERIFIED_CODE_SHA256= followed by the hash from this build. Do not substitute the full release archive checksum for this private-code checksum.

Review runtime/ before public upload. Upload only each context's contents, never the full release, parent directory, live data or environment files. Do not add any secrets to the public runtime Space.

hf upload "%PF_HF_RUNTIME_REPO%" "%PF_HF_UPLOAD%\runtime" . --repo-type space
python deploy\saas\huggingface_native\deploy_release.py --system-ca upload-code --bundle "%PF_HF_UPLOAD%" --repo "%PF_HF_CODE_REPO%" --fresh
python -c "import sys; from huggingface_hub import HfApi; print(HfApi().space_info(repo_id=sys.argv[1]).sha)" "%PF_HF_CODE_REPO%"
python -c "import json,sys; print(json.load(open(sys.argv[1], encoding='utf-8'))['sha256'])" "%PF_HF_UPLOAD%\jobs\code-manifest.json"

Save the last two outputs: the private upload's exact 40-character commit and the payload's 64-character SHA256. They must come from this build/upload, not a previous chat or release. In Browser — HF, wait for the public runtime Space's build to succeed. The runtime/code pages are not the PF application.

Checkpoint: successful uploads, a built public runtime, and two recorded private-code pins. A failed upload may leave uncertainty about remote commits; check Files/history before deciding nothing changed. Xet 403 recovery is in section 10. Do not delete repository files/history as a routine workaround.

5. Resolve the public image and qualify one HF Job

Where: Browser — Hugging Face, public runtime Space. Open its Run with Docker / Run locally instructions (possibly in the page menu) and copy the actual registry.hf.space/...:latest image reference. Do not guess it from the Space URL, and do not use the private runner's old image.

Where: Windows CMD. Replace the placeholder with that image:

set "PF_HF_RUNTIME_TAG=REPLACE_WITH_ACTUAL_PUBLIC_REGISTRY_IMAGE_LATEST"
python deploy\saas\huggingface_native\hf_registry_digest.py --anonymous --system-ca --image "%PF_HF_RUNTIME_TAG%"

This packaged helper requires no Docker daemon. --system-ca uses Windows' trusted certificate authorities via truststore; TLS verification remains on. The anonymous check must succeed without reading/sending an HF token. Record its complete registry_image value ending in @sha256: plus 64 hex characters. Public visibility alone is not proof that a particular registry route is anonymously accessible. An authenticated metadata lookup is not a substitute.

Set the three outputs below. The private commit and payload hash are lowercase hexadecimal; no angle brackets or spaces belong inside these values.

set "PF_HF_JOB_IMAGE=REPLACE_WITH_FULL_REGISTRY_IMAGE_AT_SHA256_DIGEST"
set "PF_HF_CODE_REVISION=REPLACE_WITH_40_HEX_PRIVATE_COMMIT"
set "PF_HF_CODE_SHA256=REPLACE_WITH_64_HEX_PAYLOAD_SHA256"
python deploy\saas\huggingface_native\qualify.py --namespace "%PF_HF_OWNER%" --runtime-image "%PF_HF_JOB_IMAGE%" --code-repo "%PF_HF_CODE_REPO%" --code-revision "%PF_HF_CODE_REVISION%" --code-sha256 "%PF_HF_CODE_SHA256%" --flavor cpu-basic --system-ca

This is a dry run: no Job is submitted. Review the image, private revision, hash, empty secrets and read-only /pf-code mount. The bootstrap verifies the payload before extracting/importing it. When ready, the following command explicitly submits one paid Job, with a five-minute provider timeout:

python deploy\saas\huggingface_native\qualify.py --namespace "%PF_HF_OWNER%" --runtime-image "%PF_HF_JOB_IMAGE%" --code-repo "%PF_HF_CODE_REPO%" --code-revision "%PF_HF_CODE_REVISION%" --code-sha256 "%PF_HF_CODE_SHA256%" --flavor cpu-basic --system-ca --submit

The helper supplies the short name pf-r34-confinement-check. Save the Job ID/URL. Acceptance requires verified provider identity, a passing mandatory confinement probe, successful confined FMPy Dahlquist and QuTiP decay probes, successful exit and provider stage COMPLETED. Require self_test_completed: true and scientific_runtime_probe_passed: true. These small dependency probes do not qualify every scientific workflow or the complete SaaS deployment. Merely starting or pulling an image is insufficient. Build the runtime context from this release and record its new digest; updating private code while retaining an earlier dependency image does not install the required scientific packages.

If the helper reports RuntimeError, a timeout or an uncertain response after printing a Job ID, do not submit again. Inspect the existing Job in section 10 first. Completed logs can establish that execution ran, but do not by themselves resolve a failed provider-identity check. A generic RuntimeError does not identify its cause: resolve the helper error and compare the actual Job with the expected identity before service admission. A cancellation request is not proof the VM stopped. Failed confinement remains blocking; do not remove the probe or request privileges.

The self-test runs outside PF's admission ledger and can consume HF credits separately. Cloud Space operation/build resources are also outside that ledger. Rebuilding code changes its commit/hash; rebuilding a runtime may change its digest. Requalify new identities. Tags, hf.co/spaces/OWNER/SPACE shorthand and digests copied from another deployment are not production pins.

Checkpoint: record this runtime digest, private repo/commit/hash, flavor and successful Job evidence. These CMD variables are only local notes. You must enter their actual values into the controller's HF Variables in section 7.

6. Set up email and generate stable secrets

Email delivery

Where: Browser — SMTP2GO, then your DNS provider. Production local authentication requires functioning account email even while self-registration is disabled. The r34 mailer supports encrypted SMTP and optional Resend HTTPS. This section retains the SMTP2GO path; a provider migration is not required. See public email delivery for both transports and their shared durable queue. HF Spaces currently permits outgoing ports 80, 443 and 8080; the usual SMTP ports 587/465 are not usable under that published policy. HF networking.

A compatible documented setup is SMTP2GO SMTP relay:

1. Create the SMTP2GO account and review its plan/quota.
2. In Sending → Verified Senders, add your sending domain. In the domain's actual DNS manager, add the exact three CNAME records SMTP2GO displays. They handle return-path/SPF, DKIM and tracking. Use your displayed values, not another user's selector numbers.
3. For Network Solutions DNS: Domains → your domain → Advanced Tools → Advanced DNS Records → Manage → Add Record → CNAME → Other Host. Its form may append the domain; use the host part only when it does. Network Solutions setup.
4. Add records without replacing website, MX, nameserver, SPF or existing DKIM records. SMTP2GO requires no change to existing SPF/MX. If link is already used, choose an unused tracking hostname in SMTP2GO. Preserving the existing hosting records keeps your Network Solutions website and mailbox in place.
5. In SMTP2GO click Verify. This checks DNS; it does not add records. Inspect each failure's tooltip. Propagation is a possible delay, not a substitute for checking the records were saved.
6. Create an SMTP User under Sending → SMTP Users. Save its username and password. These are not the website login, API key, mailbox password or PF administrator password. Open/click tracking can remain disabled.

For each DNS row, the Host/Name is the alias under your own domain; the Alias/Target is SMTP2GO's supplied destination, such as return.smtp2go.net, dkim.smtp2go.net or track.smtp2go.net. Targets are hostnames, without https:// or a path. Use the values displayed for your account. TTL is in seconds: 3600 = 1 hour, 7200 = 2 hours, 14400 = 4 hours. Select the matching duration if the DNS form uses a dropdown. TTL is caching behavior, not a guaranteed verification waiting time.

Choose the actual sender mailbox you want recipients to see, for example support@YOUR_VERIFIED_DOMAIN. Domain verification authorizes sending; it does not choose or create that mailbox. Enter the chosen address as PF_SAAS_SMTP_FROM; it is not the SMTP username or your HF account login.

Use mail.smtp2go.com, port 443, SSL=true, STARTTLS=false in section 7. The public-readiness checker requires exactly one TLS mode and an allowed outbound port for HF SMTP. Another provider's encrypted SMTP endpoint on an allowed port can be used after verification. Changing Network Solutions' normal SMTP port to 443 alone does not make its server support it. Verified sender details.

Application and storage keys

Where: Windows CMD, privately. Generate these only for a genuinely new deployment. Save each output in a password manager/offline recovery record, labelled with its exact Secret name; then enter it in HF Secrets. These commands print secrets, so do not share their output.

Run once for PF_HF_STATE_KEY:

python -c "import secrets; print(secrets.token_hex(32))"

Run the following twice, keeping separate outputs for PF_SAAS_ASSISTANT_CREDENTIALS_KEY and PF_SAAS_BYOS_CREDENTIALS_KEY:

python -c "import base64,secrets; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())"

Run this separately for each of PF_SAAS_SECRET_KEY, PF_HF_PG_ADMIN_PASSWORD, PF_HF_PG_APP_PASSWORD and PF_HF_PG_WAL_PASSWORD:

python -c "import secrets; print(secrets.token_urlsafe(48))"

Choose/save a separate strong PF_HF_ADMIN_PASSWORD for your first PF login. Do not reuse these values. Keep state, database, session and vault keys stable across restarts/rebuilds: losing or replacing them can make saved data unrecoverable. HF Secrets cannot be read back as a recovery method; retain your own secure copy.

7. Configure the private controller Space

Where: Browser — Hugging Face → your pf-simulation-saas Space → Settings → Variables and secrets. Add the following names to Variables or Secrets as indicated. Enter literal values, without CMD set syntax, outer quotes, or unexpanded %PF_HF_OWNER% placeholders. Every value is case-sensitive.

Get HF's actual assigned app host in Windows CMD:

python -c "import sys; from huggingface_hub import HfApi; print(HfApi().space_info(repo_id=sys.argv[1]).host)" "%PF_HF_APP_REPO%"

Use that host as an HTTPS origin, for example https://ACTUAL-HOST.hf.space, with no trailing path; keep an existing https:// prefix rather than adding it twice. If empty, obtain it from the Space's open-app link after its initial build. Do not guess it from the account name.

The launcher owns production mode, the local PostgreSQL connection and the assistant/BYOS state directories. Do not set an external database URL, hybrid gateway secret or separate worker for this native mode. Existing SaaS settings still govern supported AI providers, endpoint allowlists, organizations and quotas. The defaults above are starting limits, not validated capacity for every workload.

Checkpoint: the six runner values (namespace, image, code repository, revision, hash and flavor) are only part of the configuration. Bucket access, all stable keys/passwords, SMTP and deployment budgets must also be configured. Keep PF_SAAS_REGISTRATION_ENABLED=false and the Space Private during validation.

8. Upload the controller and verify startup

Where: Windows CMD. Use the same current output directory from section 4:

python deploy\saas\huggingface_native\deploy_release.py verify --bundle "%PF_HF_UPLOAD%"
python deploy\saas\huggingface_native\deploy_release.py --system-ca prepare --bundle "%PF_HF_UPLOAD%" --space "%PF_HF_APP_REPO%" --state-dir "%PF_HF_DEPLOY_STATE%" --fresh
python deploy\saas\huggingface_native\deploy_release.py --system-ca apply --state-dir "%PF_HF_DEPLOY_STATE%"

Use --fresh only for a newly created empty controller Space. For an existing native deployment, use section 11 without that flag. prepare verifies local seals and private code pins without changing HF. apply confirms pause, commits against the reviewed repository revision and requests restart. Keep the printed revision and local preparation directory. It does not change configuration, Secrets or durable data. Do not add blanket --delete flags.

After apply succeeds, wait for the controller:

hf spaces wait "%PF_HF_APP_REPO%" --timeout 25m
python deploy\saas\huggingface_native\deploy_release.py --system-ca status --state-dir "%PF_HF_DEPLOY_STATE%"
hf spaces logs "%PF_HF_APP_REPO%" --tail 150

Require source_bytes_verified=true, running_revision_verified=true, provider stage RUNNING and application startup completion in the runtime logs. A terminal build/runtime error ends the wait early; inspect its logs even when the wait command returns an error. The CLI's 25m wait is only an observation window, separate from the controller's bounded startup settings.

Where: Browser — Hugging Face.

1. Open the controller's build logs, then its runtime/container logs. A completed Docker build is distinct from successful application startup.
2. The first empty deployment needs PF_HF_BOOTSTRAP=true. The supervisor creates PostgreSQL, persists a verified base backup, starts synchronous remote WAL, initializes ledgers, restores private state and then exposes port 7860. Existing remote state always takes the restore path. Failure keeps startup closed.
3. Open the direct application URL in its own browser tab. If the HF page's embedded frame is blocked, use the direct URL; do not weaken frame/CSRF/cookie protections. Private Space access still requires HF owner/collaborator access.
4. Log into PF Simulation with PF_HF_ADMIN_EMAIL and PF_HF_ADMIN_PASSWORD. Use this PF administrator password, not an HF token, S3 secret, PostgreSQL password or SMTP password. Existing users are never reset or promoted by bootstrap values.
5. After confirmed successful durable initialization and first login, remove PF_HF_ADMIN_PASSWORD and remove PF_HF_BOOTSTRAP (absence defaults to false). Preserve every other persistent key/password, bucket and prefix. Confirm the resulting restart restores the same application state. Use the account's normal password flow afterward, not bootstrap variables.

If a setting is missing or logs contain a generic DurabilityError, use section 10. Do not generate new encryption keys or reset storage to get past recovery.

If HF remains Starting with sparse logs, use this read-only Windows CMD batch and inspect the latest runtime logs in the browser:

python -c "import sys; from huggingface_hub import HfApi; print(HfApi().get_space_runtime(repo_id=sys.argv[1]).stage)" "%PF_HF_APP_REPO%"
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"
hf buckets list "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"

These checks do not reveal Secret values or prove recovery has finished. PF_HF_PG_START_TIMEOUT_SECONDS=600 bounds the PostgreSQL startup/recovery phase, not the total Docker build or full controller initialization. The separate PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS wait defaults to 600 seconds and accepts integers from 30 through 3600. Healthy WAL uploads can continue while PostgreSQL reports catchup or potential; those states do not yet satisfy the unchanged streaming/sync readiness checks. The receiver batches contiguous data within a 100-millisecond collection window, up to 4 MiB; remote publication time is additional. See the maintenance note for the acknowledgement and startup boundaries. Sparse logs or an elapsed interval alone are not grounds for a force restart or storage reset.

After completing section 9, a SaaS trial for ordinary PF users can use Protected Space visibility: source stays private while the app is accessible. Check current HF plan eligibility. PF users then authenticate to PF and retain organization/project permissions; they do not need repository collaborator access or the operator's HF token. Do not change the controller to Public.

9. Qualify the complete deployment before researcher access

Record the release hash, builder manifests, runtime image digest, private code repository and exact commit, payload SHA256, controller Space revision, HF hardware/flavor, package versions, test inputs, Job IDs, timestamps, and observed results. Rebuilds and changes to those identities require qualification again. Required live gates include:

Run destructive fault tests in a separate disposable validation deployment, with its own private Bucket/prefix and independently generated keys. Never corrupt or remove WAL, database heads or artifacts, introduce a conflicting controller, or exhaust storage against a deployment holding retained research data. Ordinary restart/persistence checks should use known test records.

1. Provider capabilities: public digest image pull; exact-commit private repository mount; rejection of changed/missing payloads; dedicated Job execution; mandatory Landlock ABI 6/seccomp probe; read-only input and scoped output mounts; conditional Bucket create/update semantics and read-after-write visibility. An image download alone does not qualify execution.
2. Research execution: real representative relativity and quantum workloads, private Python workflows, active override layers, report rendering/export, literature attach/upload/remove, and autonomous investigation pause/recovery.
3. Tenant separation: two ordinary users in different organizations/projects; attempted cross-owner access to jobs, attachments, results, state, credentials, and BYOS destinations must be rejected. Verify execution cannot read controller files, another workload's data, controller secrets, or its Bucket credentials, or signal/trace the runner. Exercise direct sockets, inherited descriptors, symlink/hardlink escapes, subprocesses, and attempted privilege changes. Landlock is not a separate filesystem/PID namespace: some names and metadata may remain observable, so do not certify complete namespace invisibility.
4. Durability: acknowledge a known transaction/upload, forcibly replace the disposable controller and its local disk, and verify the acknowledged state, budget, credential references, artifacts, and private continuation checkpoints recover. Exercise WAL corruption, missing objects, and conflicting controllers: each must fail closed rather than serve older state.
5. Faults and costs: storage outage, expired token, provider API timeout around submission, cancellation, process/resource exhaustion, Job failure, controller restart while a Job is running, and exhausted global/storage budgets. Confirm no duplicate remote dispatch, false success, or unproven reservation refund. Attempt detached and double-forked children; verify termination of the entire dedicated Job before releasing its reservation. Per-process rlimits are not a delegated cgroup limit over a process tree; VM capacity and provider timeout remain the aggregate boundary.
6. SaaS access: secure cookies/CSRF at the canonical HTTPS origin, user account registration or invitations, SMTP, upload sizes, download integrity, BYOK/BYOS, and admission under concurrent users.

Passing local fixtures is not a substitute for these gates. If a gate fails, keep registration and researcher access closed, preserve the failure evidence, and fix/requalify the implementation. The same evidence standard applies to Linux/Docker deployment validation.

10. Troubleshoot the failed checkpoint

Use the matching subsection; do not repeat the entire deployment. Read-only checks can be run together. Review outputs before sharing: paths, computer names, bucket names and application logs can be private. Never share HF tokens, S3 secrets, encryption keys, passwords or an unfiltered environment dump.

Identify the controller's current stage

Where: Windows CMD. Inspect the same startup attempt before taking another action:

python -c "import sys; from huggingface_hub import HfApi; print(HfApi(token=True).get_space_runtime(sys.argv[1]).stage)" "%PF_HF_APP_REPO%"
hf spaces logs "%PF_HF_APP_REPO%" --build --tail 150
hf spaces logs "%PF_HF_APP_REPO%" --tail 150

For database startup, verify unchanged bucket/prefix and persistent keys, controller access to that storage, storage/WAL capacity, and the two independent startup bounds. This release waits for PostgreSQL readiness and batches WAL catch-up before admitting the application. Set PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS=600 for the normal receiver startup budget; accepted values are 30–3600. A longer CLI wait cannot change that internal budget. Increase a bound only after establishing that recovery is making healthy progress. Keep the synchronous readiness requirement intact.

If the cause remains uncertain, pause the failed controller, retain the verified backup, and obtain a review of the sanitized logs and exact release/pins. Do not change durable state to make an unexplained startup failure disappear.

Run stays Queued with no attempt

attempt_count=0 means the PF worker has not claimed the run. Check admission before resubmitting. The Space being RUNNING only establishes application startup; it does not establish that the dispatcher can reserve another run.

The deployment allowance is shared across organizations, simulations and admitted assistant work. An organization's Usage page and hf jobs ps do not report that ledger. A terminal HF Job list does not prove that all PF reservations have settled. Retained charges can include conservatively accounted interrupted or ambiguous work; do not delete attempts or ledger records to recover allowance.

In the private configured service environment connected to the actual service database, an operator can inspect the packaged deployment control status:

python -m pf_saas.deployment_control status

This is not a command for an unconfigured Windows tools environment. Do not expose the database or copy its credentials into a shared terminal to run it. Read usage.compute_committed_seconds, usage.compute_remaining_seconds, usage.monthly_compute_seconds, usage.active_reservations and unresolved. Committed seconds already include active reservations: do not add them twice. If operator database access is unavailable, obtain an authorized ledger check; do not infer remaining allowance from tenant usage, elapsed wall time or HF Jobs.

For a normal job with a 600-second timeout, r34 requests 660 deployment reservation seconds: 600 execution seconds plus the 60-second launch/stop margin. This is an admission reservation, not a claim that 660 seconds of scientific computation occurred or an exact HF invoice charge. Different job limits and assistant workflows can require different reservations.

As a conditional example, if the current UTC month's verified committed total is 5,940 seconds, the configured ceiling is 5,940, and the intended next reservation is 660, a ceiling of 6,600 supplies that one reservation's headroom. These are example values, not release defaults. Check global concurrency, optional monetary admission and organization limits as well; raising the compute ceiling does not resolve those separate blockers.

If the operator approves that additional allowance:

1. Stop new submissions across the deployment. Settle current executions and result collection. Review queued runs and investigations: increasing the ceiling can admit existing work automatically, so do not leave unintended work queued. Confirm provider termination where applicable.
2. Record the current nonsecret settings, immutable controller/runner pins and the verified monthly ledger values. Pause the controller as in section 11.2.
3. In Browser — HF → controller Settings → Variables, change only PF_SAAS_DEPLOYMENT_MONTHLY_COMPUTE_SECONDS to the deliberately chosen new ceiling. Keep PF_SAAS_DEPLOYMENT_BUDGET_ENABLED=true, the concurrency and monetary bounds, and the storage/state settings. Do not change the prefix, reset accounting, or alter the worker lease/durability requirements.
4. If the settings change starts the Space, pause it again. Read back the new ceiling and verify that the other settings and code pins are unchanged. For the example above, the operator record would state DEPLOYMENT_CEILING_UPDATED=5940->6600 and ADDITIONAL_ALLOWANCE_SECONDS=660.
5. Restart once, require RUNNING and normal startup logs, then submit or retry only the intended bounded run. Require completion and accessible saved results as in section 11.7. Recheck admission before another run.

The ceiling is a total for the UTC billing month, not a refill balance. Raising it preserves the existing charges. It also permits additional paid provider work; it is not an account-wide HF billing cap. If admission has sufficient headroom and the run remains queued, retain its job details and controller logs for dispatcher investigation instead of repeatedly increasing the ceiling.

Worker lease or result-publication failure

An interrupted attempt and an HF Job's final status describe different layers. Read the PF attempt's phase, timestamps, execution_started_at, error and cleanup state along with the matching provider identity. An absent execution timestamp does not establish that scientific execution started. Total wall time includes preparation, transfers and publication in addition to provider execution.

A PF_WORKER_HEARTBEAT controller log event identifies a failed worker renewal with bounded timing and exception-category fields. Preserve it with the exact controller and private-code pins. It is evidence for investigation, not proof of a particular storage or database root cause. Keep lease expiry, synchronous WAL acknowledgement and ownership fencing enabled; do not increase unrelated compute or collection limits to hide lost ownership.

For collection or publication errors, use result publication and verification (source package reference: R34_ARTIFACT_PUBLICATION_REPAIR.md). A complete transfer byte counter or provider COMPLETED status is insufficient: PF must commit its final attempt and make the saved results accessible.

Measure the stages of a slow run

Read the authorized job/attempt timings record and the controller's structured PF_EXECUTION_TIMING log. Older attempts can have no timing record. These measurements use controller_monotonic_wall and distinguish preparation, uploading, submission, provider queueing, observed execution and collection. Collection stages distinguish transfer, import, publication and commit when observed. Heartbeat, progress and publication operation aggregates can overlap phases; do not add them to phase totals as additional elapsed time.

The database timing snapshot precedes its own final commit acknowledgement; the final controller log can include that acknowledgement's duration. The downloadable timings.json artifact is an earlier snapshot made before artifact publication. provider_runtime_seconds is null when unavailable. An observed executing phase is not an exact HF Job runtime or billing measurement, and accounted_compute_seconds remains PF's separate internal accounting measure. Use the matching provider record for provider runtime and billing. These measurements locate delay without promising a particular completion time.

The workbench adapts automatic polling to activity, page visibility and errors. Optional controller Variables are PF_SAAS_UI_POLL_ACTIVE_MS (5000), PF_SAAS_UI_POLL_IDLE_MS (30000), PF_SAAS_UI_POLL_HIDDEN_MS (60000), PF_SAAS_UI_POLL_MAX_BACKOFF_MS (120000) and PF_SAAS_UI_POLL_QUALIFICATION_MS (60000). Values are milliseconds; keep active at or below idle, idle at or below hidden, and hidden at or below maximum backoff. Manual refresh and cancellation remain explicit immediate requests. Polling settings change display request frequency, not execution capacity.

Python 3.13 / Python 3.12 / package or PATH mismatch

Where: Windows CMD. The environment name in the prompt is not proof of the interpreter actually running:

where python
where hf
python -c "import sys; print(sys.executable); print(sys.version.split()[0])"
python -m pip show huggingface_hub truststore

Activate the section-2 environment and repeat the checks. If an old environment was built with Python 3.13, create a different new 3.12 environment; do not overwrite/delete it as a repair. Use python -m pip inside the selected environment. If hf still resolves elsewhere, invoke the expected executable:

call "%PF_HF_TOOLS%\Scripts\activate.bat"
"%PF_HF_TOOLS%\Scripts\hf.exe" auth whoami

Installing a package in one environment does not install it into another. A Downloads copy of a helper is not necessarily the helper packaged in this release.

Upload fails with Xet HTTP 403

Where: Windows CMD. First confirm the destination and current context path. For a controller operation, first inspect the saved preparation and exact remote revision. The following read-only check can reconcile an uncertain commit response; it does not repeat the upload:

hf auth whoami
dir "%PF_HF_UPLOAD%\space\Dockerfile"
set "HF_HUB_DISABLE_XET=1"
python deploy\saas\huggingface_native\deploy_release.py verify --bundle "%PF_HF_UPLOAD%"
python deploy\saas\huggingface_native\deploy_release.py --system-ca prepare --bundle "%PF_HF_UPLOAD%" --space "%PF_HF_APP_REPO%" --state-dir "%PF_HF_DEPLOY_STATE%" --fresh
python deploy\saas\huggingface_native\deploy_release.py --system-ca apply --state-dir "%PF_HF_DEPLOY_STATE%"

Set HF_HUB_DISABLE_XET before launching the next CLI process, in the same CMD window. It disables the optional Xet transfer path; it does not bypass access control or guarantee every 403 is resolved. For an uncommitted preparation, resolve the network error and rerun the same apply; its parent-revision guard prevents replacing a concurrent commit. For runtime/code uploads, inspect the existing commit before deciding to retry their matching section-4 command with this setting. Do not upload space/ to the public runtime. HF environment-variable reference.

After the intended upload has succeeded and you have verified its commit, clear the temporary override before later HF operations:

set "HF_HUB_DISABLE_XET="

If still denied, check that the active token has write access to that exact Space and that the network can reach HF's transfer services. Keep certificate validation enabled. A failed attempt does not prove no commit occurred: inspect Files → history and compare its commit with your recorded upload. Align with the current generated folder, without --delete.

Registry TLS, authentication or missing Docker instructions

Where: Windows CMD.

python deploy\saas\huggingface_native\hf_registry_digest.py --version
python deploy\saas\huggingface_native\hf_registry_digest.py --diagnose --system-ca
python deploy\saas\huggingface_native\hf_registry_digest.py --anonymous --system-ca --image "%PF_HF_RUNTIME_TAG%"

The diagnostic reads no HF token. A registry-root 401 can be a normal anonymous challenge; the selected image's anonymous manifest/digest check must still pass. CERTIFICATE_VERIFICATION_FAILED calls for trusted-CA/clock/proxy repair or --system-ca, not verify=False, insecure curl or disabled TLS checks. If the helper reports that --system-ca conflicts with SSL_CERT_FILE or SSL_CERT_DIR, retain your approved CA configuration and run the helper without --system-ca. Do not clear an organization's approved CA settings or disable verification merely to remove the error.

In Browser — HF, use the public runtime Space's page menu/run-locally instructions after a successful build. HF calls the feature “Run with Docker”, but its position may change. A private runner's registry login can verify metadata yet still fail an HF Job pull. Do not make that private repository public. If the current public image still gives 401, resolve its public access or publish only the reviewed public runtime context to another anonymously pullable OCI registry. The packaged hf_registry_digest.py accepts only registry.hf.space; for another registry, use that registry's supported method to verify anonymous access to the exact image digest, then run the same qualify.py qualification.

Qualification reports RuntimeError or stops after a Job ID

Where: Windows CMD. Copy the already-created Job ID; these commands inspect it and do not submit another paid Job:

set "PF_HF_TEST_JOB=REPLACE_WITH_EXISTING_JOB_ID"
hf jobs inspect "%PF_HF_TEST_JOB%" --namespace "%PF_HF_OWNER%" --format json
hf jobs logs "%PF_HF_TEST_JOB%" --namespace "%PF_HF_OWNER%"

Use this inspection for the credential-free qualification Job, and review it privately before sharing. JSON avoids the human display truncating pins or mount details. Check the same Job in Browser — HF → Jobs. Record terminal status, image, command, revision/mount and confinement output. A generic RuntimeError has no proven cause until the identity comparison is examined. Logs showing a passed probe and COMPLETED are evidence for that run, not permission to ignore a failed metadata check or open the full SaaS to users. Resolve the error's cause before admission; retain reservations until actual provider termination is confirmed. If no ID was printed, inspect the account's Jobs list before retrying.

The packaged qualifier supports HF's empty-secrets metadata representations (None, {} and []) while rejecting nonempty or malformed values. Use the qualifier packaged in this release. Keep the public-runtime/private-payload flow, mandatory confinement, empty secrets and read-only code mount intact.

Bucket exists in the browser, but CLI says BucketNotFound / access denied

Where: Windows CMD.

hf auth whoami
if defined HF_TOKEN (echo HF_TOKEN override is set) else (echo HF_TOKEN override is not set)
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"

Where: Browser — HF. Compare the exact Bucket owner/name with its URL. Then check the active CLI token's fine-grained scope: three selected Spaces do not include the state Bucket. Add only the needed Bucket read/write grant using the available resource/scope controls, save, and retry. Matching whoami proves account identity, not resource permission.

An HF_TOKEN environment variable can override the saved login. If it is an unintended old override, clear it only in this CMD window and select the intended token interactively:

set "HF_TOKEN="
hf auth login --force
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"

Do not rotate S3 keys to fix a CLI HF-token error. Creating a Bucket after a token does not itself require invalidation. S3 keys inherit their parent token's scope, but do not assume an unverified permission change fixed existing S3 access: test the controller/S3 path separately. If you replace/revoke that parent token, update every affected HF token Secret and generate/update the corresponding S3 pair. Separate valid HF and S3 parent tokens need not be identical. Preserve PF_HF_STATE_KEY, the Bucket, prefix and existing objects.

Email sender stays Unverified / email delivery fails

Where: Browser — SMTP2GO and DNS provider. Open the domain's record details, not just the summary row. Confirm all three CNAMEs exist at the authoritative DNS provider, with no doubled domain suffix, wrong record type or replaced hosting record. Read the failed record's tooltip. DNS propagation can take time, but waiting does not correct a missing/wrong record. Tracking Disabled is a separate setting, not evidence of the verification failure's cause.

Where: Windows CMD. Replace these placeholders with the three full CNAME hostnames from SMTP2GO, then run this read-only DNS check:

nslookup -type=CNAME REPLACE_WITH_FULL_RETURN_PATH_HOSTNAME
nslookup -type=CNAME REPLACE_WITH_FULL_DKIM_HOSTNAME
nslookup -type=CNAME REPLACE_WITH_FULL_TRACKING_HOSTNAME

Each answer should point to its supplied SMTP2GO target. A missing answer may reflect an unsaved/wrong record, the wrong DNS provider or a cached result; check the provider's saved rows and SMTP2GO's error before deciding to wait.

For delivery, confirm mail.smtp2go.com, port 443, SSL true, STARTTLS false, a verified From address and the SMTP User credentials. Check SMTP2GO's activity report for rejection/quota details. Do not put its HTTP API key in the SMTP password field or switch to development outbox mode in production.

Build succeeds, but startup reports DurabilityError

Where: Browser — controller runtime logs and Settings. A generic DurabilityError is not enough to diagnose a single cause. Capture the time and sanitized error context. Check the configured owner, bare Bucket name, prefix, S3 credential pairing/permissions, unchanged state/database keys and available storage/WAL allowance. A successful CLI Bucket lookup tests HF-token visibility, not S3 writes, conditional updates, encryption or PostgreSQL restore.

Do not delete durable heads, WAL, ledgers or objects; change the prefix; generate replacement state/vault keys; force bootstrap; or run a second controller to bypass the failure. An existing deployment must recover its existing authenticated state. Keep researcher access closed until the actual failure is resolved.

App frame blocked, or administrator login fails

Where: Browser. Open the direct https://...hf.space origin obtained in section 7; set PF_SAAS_PUBLIC_URL to that exact origin. HF private access and PF login are separate. Do not disable frame, cookie or CSRF protections.

Use the PF administrator email/password from first initialization. Changing PF_HF_ADMIN_PASSWORD later does not reset an existing account. If that password is lost and SMTP works, use PF's Forgot password? flow. HF's masked Secrets cannot recover the old value. Keep stable encryption/session/database keys.

“Virtualization support not detected”

This is a local Windows/Docker startup problem. Treat three prerequisites separately: CPU virtualization in BIOS/UEFI, the Windows WSL/Virtual Machine Platform components, and hypervisor startup. Installing Python or changing HF tokens will not enable any of these.

Where: ordinary Windows CMD. Run these read-only checks. You can paste the complete block; one failed command does not invalidate the other results.

ver
systeminfo
wsl --status
wsl --version
wsl --list --verbose
powershell -NoProfile -Command "Get-CimInstance Win32_Processor | Select-Object Name,VirtualizationFirmwareEnabled,VMMonitorModeExtensions,SecondLevelAddressTranslationExtensions"

Also open Task Manager → Performance → CPU and inspect Virtualization. systeminfo may say a hypervisor has already been detected instead of listing each Hyper-V requirement; that is not evidence that virtualization is missing. Processor-property output is supporting evidence, especially on virtual machines, rather than the sole pass/fail test. Docker virtualization troubleshooting; Microsoft processor-property reference.

Where: PowerShell as administrator. Inspect the Windows feature states:

Get-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
bcdedit /enum | findstr /i hypervisorlaunchtype

If WSL is not installed correctly, first use wsl --install --no-distribution. For a manual/older component-based WSL installation, the documented component repair is:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Restart Windows after enabling the components; /norestart only postpones that restart. Modern packaged WSL can differ from the older optional-component setup, so a disabled legacy WSL component alone is not proof that a working WSL 2 installation is broken. Microsoft manual WSL installation; modern WSL installation options.

If the boot-configuration check explicitly reports hypervisorlaunchtype Off, and the computer's administrator permits the change, enable hypervisor startup:

bcdedit /set hypervisorlaunchtype auto

Restart Windows, start Docker Desktop, and repeat docker version. An empty filtered result is not the same as an explicit Off; do not change boot settings merely because that line is absent. Firmware and corporate-managed settings should be handled by the machine's administrator. Microsoft WSL startup troubleshooting; nested virtualization requirements.

Docker login returns a named-pipe HTTP 500

Example of a local Docker engine endpoint:

500 Internal Server Error ... dockerDesktopLinuxEngine/v1.56/auth
check if the server supports the requested API version

The named pipe identifies the local Docker Desktop engine connection. This message alone does not establish that HF rejected the token or that API version 1.56 is unsupported. Check the local engine before retrying authentication.

Where: ordinary Windows CMD. Read-only checks:

where docker
docker context show
docker context ls
docker version
docker info --format "{{.OSType}}"
if defined DOCKER_API_VERSION (echo DOCKER_API_VERSION is set) else (echo DOCKER_API_VERSION is not set)
if defined DOCKER_HOST (echo DOCKER_HOST is set) else (echo DOCKER_HOST is not set)
if defined DOCKER_CONTEXT (echo DOCKER_CONTEXT is set) else (echo DOCKER_CONTEXT is not set)

1. If the Server section is missing or errors, open Docker Desktop, inspect its startup error, and resolve the local engine/virtualization problem first. Restart Docker Desktop through its Troubleshoot menu when appropriate; do not use factory reset as the first repair.

2. If the selected context targets a different engine, choose the intended Docker Desktop Linux context shown by docker context ls. If it is named desktop-linux, this one-command check explicitly selects it:

   docker --context desktop-linux version
   

   Context names and environment overrides determine which engine receives a command; do not remove an intentional remote-engine configuration. Docker contexts.

3. If DOCKER_API_VERSION was set for earlier troubleshooting, remove that override in this CMD window and retry:

   set "DOCKER_API_VERSION="
   docker version
   

   Docker normally negotiates a compatible API version. Do not guess an older API version merely because /v1.56/ appears in the error. Use the Client/Server output to investigate a genuine version mismatch, and update Docker Desktop through its normal update mechanism if necessary. Docker Engine API negotiation.

4. Once Client and Server respond normally, retry docker login registry.hf.space interactively. Enter your personal HF username and token at its prompts; do not substitute an organization owner name or put the token in the command.

5. If Docker remains unavailable, use the packaged deploy\saas\huggingface_native\hf_registry_digest.py and the HF-hosted build path. A registry-metadata result establishes only metadata access; the subsequent HF Job test still establishes whether HF can pull and execute the image.

Collecting diagnostics: review system, context and log output before sharing it: computer/domain names or network details may be private. Use the formatted Docker check above rather than an unrestricted configuration/environment dump. Never share HF tokens, S3 secrets, passwords or PF encryption keys.

11. Upgrade an existing r34 deployment

Use this route when an earlier r34 release is already deployed and running. It upgrades the existing private controller and runner while retaining the existing application data. It does not repeat first-time resource creation, email setup, secret generation or database initialization. The complete release contains the approved native response/durability corrections and the shared application improvements. Historical patch installers are not required.

Keep the same state Bucket, deployment prefix, database passwords, state key, session key and BYOK/BYOS vault keys. Preserve PF_HF_MAX_STORAGE_BYTES at the already approved value: for the deployment raised to 32 GiB this is 34359738368. Keep the encrypted quota counter, reservation IDs and audit history. Conservative lifetime admission remains the default; only explicit safe-lifecycle policy and verified reconciliation can reclaim eligible charges. Keep the existing controller's valid HF and Bucket S3 credentials. This upgrade and its HF-to-HF backup require no new S3 configuration. Preserve the old release, its generated contexts, the old controller revision and its qualified code/image pins for recovery review. The new full archive and the newly generated private-code archive have distinct SHA256 values; neither is the runner's Git commit or the runtime image digest.

11.1 Prepare the new release and record the running deployment

Where: Windows CMD / File Explorer. Compare the downloaded full archive against its companion checksum, then extract .xz and .zip into a fresh project directory as in section 2. Retain the earlier extraction separately.

certutil -hashfile "FULL_PATH_TO_DOWNLOADED_RELEASE_ARCHIVE" SHA256

Open the already configured Python 3.12 tools environment. Set the following values for this upgrade. Replace every REPLACE_... value and use the actual existing resource names if they differ from these examples:

set "PF_PROJECT_ROOT=D:\pf-r34-upgrade\privileged-frame-simulation"
set "PF_HF_TOOLS=%USERPROFILE%\pf-hf-tools"
set "PF_HF_OWNER=YOUR_HF_USERNAME"
set "PF_HF_APP_REPO=%PF_HF_OWNER%/pf-simulation-saas"
set "PF_HF_CODE_REPO=%PF_HF_OWNER%/pf-simulation-runner"
set "PF_HF_RUNTIME_REPO=%PF_HF_OWNER%/pf-runtime-r34"
set "PF_HF_BUCKET_NAME=pf-simulation-state"
set "PF_HF_BACKUP_BUCKET_NAME=pf-simulation-state-backup-REPLACE_WITH_UNIQUE_SUFFIX"
set "PF_HF_UPLOAD=%USERPROFILE%\pf-hf-native-upload-r34-REPLACE_WITH_UNIQUE_SUFFIX"
set "PF_HF_DEPLOY_STATE=%USERPROFILE%\pf-hf-native-deploy-r34-REPLACE_WITH_UNIQUE_SUFFIX"
call "%PF_HF_TOOLS%\Scripts\activate.bat"
cd /d "%PF_PROJECT_ROOT%"
where python
where hf
python -c "import sys,huggingface_hub; print(sys.executable); print(sys.version.split()[0]); print(huggingface_hub.__version__)"

Use Python 3.12 and the guide's Hub SDK version, 1.32.0, with truststore. If the existing environment needs preparation, use section 2; do not create a new virtual environment over it. PF_HF_UPLOAD names a new generated-context directory. These CMD variables do not update the remote Space.

Where: Browser — HF → controller Settings. Record the existing Variables, resource visibility/hardware, and secure recovery references for its Secrets. Specifically retain PF_HF_JOB_IMAGE, PF_HF_ASSISTANT_IMAGE if set, PF_HF_CODE_REPO, PF_HF_CODE_REVISION, PF_HF_CODE_SHA256, job flavor, namespace, Bucket and prefix, startup bounds and budget settings. HF cannot show Secret values back to you; use the original secure recovery record. Record the controller's current commit from Files → history. Keep these old identities separate from the new values recorded later.

11.2 Stop new work, finish Jobs and pause the controller

Where: PF application and Browser — HF. Arrange a maintenance window and stop users, API clients and scheduled automation from submitting work. Keep registration disabled. Restrict researcher access for the upgrade using the deployment's existing access controls; when necessary, use Private Space visibility and coordinate with HF collaborators who retain access.

Let current PF simulations, private workflows, assistant work and autonomous investigations finish and collect their results, or cancel them through PF and verify cancellation in HF Jobs. Check the HF Jobs dashboard for any remaining deployment-owned Jobs, including previously submitted qualification Jobs. A cancellation request alone is not proof that a Job has stopped. Do not pause until active work and its result collection have settled.

Where: Windows CMD. With new submissions still stopped:

hf spaces pause "%PF_HF_APP_REPO%"
python -c "import sys; from huggingface_hub import HfApi; s=HfApi(token=True).get_space_runtime(sys.argv[1]).stage; print(s); raise SystemExit(0 if s=='PAUSED' else 1)" "%PF_HF_APP_REPO%"

Require PAUSED. Pausing the controller does not cancel separately running HF Jobs. Confirm that no other controller or writer uses this deployment's Bucket/prefix. Keep the controller paused through backup, code qualification and configuration. After any HF settings change that starts the Space, pause it again before proceeding.

11.3 Create and verify a private HF backup

Where: Browser — HF. Open https://huggingface.co/new-bucket, signed in as the operator for the deployment owner. Create a new Private Bucket in the same storage region as the state Bucket. The storage region must match; a prewarming region is a different setting. Use the exact unique name chosen in PF_HF_BACKUP_BUCKET_NAME.

For example, pf-simulation-state-backup-20261001 is an arbitrary name for one maintenance window, not a required name or a Bucket to reuse for every upgrade. Choose a new suffix for each backup. The destination must be empty before copying. Keep the original state Bucket selected in the controller settings; the backup is a separate recovery copy.

At https://huggingface.co/settings/tokens, select an operator token with its existing Space upload/management and Jobs permissions plus these Bucket grants:

Preserve any existing permissions the operator token needs for other deployment actions. The controller's existing storage credentials are separate and stay unchanged. Bucket grants are independent of Space grants; account identity alone does not establish access to either Bucket.

Where: Windows CMD. Select that token explicitly for this maintenance session. Clearing HF_TOKEN here removes the environment override of the saved login. Enter the token only at the private login prompt:

set "HF_TOKEN="
hf auth login --force
hf auth whoami
hf buckets info "%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%"
hf buckets info "%PF_HF_OWNER%/%PF_HF_BACKUP_BUCKET_NAME%"
python -c "import os; from huggingface_hub import HfApi; a=HfApi(token=True); s=os.environ['PF_HF_OWNER']+'/'+os.environ['PF_HF_BUCKET_NAME']; d=os.environ['PF_HF_OWNER']+'/'+os.environ['PF_HF_BACKUP_BUCKET_NAME']; assert s!=d, 'Source and backup must differ'; assert a.get_space_runtime(os.environ['PF_HF_APP_REPO']).stage=='PAUSED', 'Controller must be paused'; assert a.bucket_info(d).private is True, 'Backup must be private'; assert not list(a.list_bucket_tree(d,recursive=True)), 'Use a new empty backup Bucket'; print('BACKUP_DESTINATION_READY')"

Require BACKUP_DESTINATION_READY, then copy entirely within HF:

hf cp "hf://buckets/%PF_HF_OWNER%/%PF_HF_BUCKET_NAME%/" "hf://buckets/%PF_HF_OWNER%/%PF_HF_BACKUP_BUCKET_NAME%/"

After Copied, verify file paths, sizes and Xet hashes. This reads the source, the backup and the source again, checks private visibility and confirms that the controller was paused before and after the comparison:

python -c "import os; from huggingface_hub import HfApi,BucketFile; a=HfApi(token=True); s=os.environ['PF_HF_OWNER']+'/'+os.environ['PF_HF_BUCKET_NAME']; d=os.environ['PF_HF_OWNER']+'/'+os.environ['PF_HF_BACKUP_BUCKET_NAME']; c=os.environ['PF_HF_APP_REPO']; read=lambda b:{f.path:(f.size,f.xet_hash) for f in a.list_bucket_tree(b,recursive=True) if isinstance(f,BucketFile)}; p=a.get_space_runtime(c).stage=='PAUSED'; x=read(s); y=read(d); z=read(s); ok=s!=d and p and a.get_space_runtime(c).stage=='PAUSED' and a.bucket_info(d).private is True and bool(x) and all(v[1] for v in x.values()) and x==y==z; print('BACKUP MANIFEST VERIFIED' if ok else 'BACKUP NOT VERIFIED'); print('Source files:',len(x),'Backup files:',len(y),'Bytes:',sum(v[0] for v in x.values())); raise SystemExit(0 if ok else 1)"

Require BACKUP MANIFEST VERIFIED. Save the source/destination names, maintenance time, counts and command result with the upgrade record. Counts and bytes depend on your deployment; do not require another deployment's numbers. This establishes an unchanged matching copy manifest. It does not constitute a restore drill or independently prove decryption/application recovery. If it does not verify, keep the controller paused and resolve the backup before changing deployed code.

11.4 Generate matching contexts and choose the runtime

Where: Windows CMD, new project root. Build once into the new output folder:

python deploy\saas\huggingface_native\build_space.py --output "%PF_HF_UPLOAD%"
python -c "import hashlib,json,sys; from pathlib import Path; p=Path(sys.argv[1]); a=json.loads((p/'jobs'/'code-manifest.json').read_text(encoding='utf-8')); b=json.loads((p/'space'/'base-code-manifest.json').read_text(encoding='utf-8')); h=hashlib.sha256((p/'jobs'/'pf-code.zip').read_bytes()).hexdigest(); assert a['sha256']==b['sha256']==h, 'Generated contexts do not match'; print('VERIFIED_CODE_SHA256='+h)" "%PF_HF_UPLOAD%"

Record the new VERIFIED_CODE_SHA256. Use this same jobs/ and space/ pair through the rest of the upgrade. The complete release builds its required controller inputs directly; no separately downloaded repair or diagnostic wrapper is needed.

Earlier r34 builds added the supported FMU and QuTiP runtime dependencies. Use the current generated context as the compatibility reference rather than assuming any r34 runtime image matches. Qualification must report both confinement_probe_passed=true and scientific_runtime_probe_passed=true. The standalone edition retains its existing optional-dependency choices.

The public runtime can be reused when its generated context is unchanged and the recorded digest belongs to the previously qualified deployment. Compare the new runtime.zip with the retained runtime.zip used to build that image:

set "PF_HF_PREVIOUS_UPLOAD=FULL_PATH_TO_RETAINED_PREVIOUS_CONTEXTS"
python -c "import hashlib,sys; from pathlib import Path; digest=lambda p:hashlib.sha256(Path(p).read_bytes()).hexdigest(); a=digest(Path(sys.argv[1])/'runtime.zip'); b=digest(Path(sys.argv[2])/'runtime.zip'); assert a==b, 'Runtime contexts differ; build and qualify the current runtime'; print('RUNTIME_CONTEXT_UNCHANGED='+b)" "%PF_HF_PREVIOUS_UPLOAD%" "%PF_HF_UPLOAD%"

When that comparison passes and the retained context is tied to the old image in your deployment record, use its existing pinned PF_HF_JOB_IMAGE digest. The new code still needs a new qualification Job. If the runtime differs, the reference context is unavailable, or its image provenance is uncertain, follow sections 4–5 to upload only the current runtime/, wait for its build, verify anonymous registry access and obtain its immutable digest. Keep both private Spaces private. Never use a :latest tag as the production pin.

11.5 Upload the private payload and qualify its exact identity

Where: Windows CMD. Upload the generated private runner context:

python deploy\saas\huggingface_native\deploy_release.py --system-ca upload-code --bundle "%PF_HF_UPLOAD%" --repo "%PF_HF_CODE_REPO%"

Save the helper's code_revision and code_sha256; these identify the exact verified upload. The helper uses a guarded parent commit and checks the remote file inventory. Keep concurrent operator changes out of this maintenance window. Set the new commit, the code hash from section 11.4 and the selected runtime digest. Do not reuse pins from another full archive or an earlier qualification just because both releases are named r34:

set "PF_HF_CODE_REVISION=REPLACE_WITH_NEW_40_HEX_PRIVATE_COMMIT"
set "PF_HF_CODE_SHA256=REPLACE_WITH_NEW_64_HEX_PAYLOAD_SHA256"
set "PF_HF_JOB_IMAGE=REPLACE_WITH_FULL_REGISTRY_IMAGE_AT_SHA256_DIGEST"
python deploy\saas\huggingface_native\qualify.py --namespace "%PF_HF_OWNER%" --runtime-image "%PF_HF_JOB_IMAGE%" --code-repo "%PF_HF_CODE_REPO%" --code-revision "%PF_HF_CODE_REVISION%" --code-sha256 "%PF_HF_CODE_SHA256%" --flavor cpu-basic --system-ca

Review this dry run: exact image, repository, revision and hash; no secrets, no exposed service/SSH, and the read-only /pf-code mount at that revision. The following command deliberately submits one paid qualification Job:

python deploy\saas\huggingface_native\qualify.py --namespace "%PF_HF_OWNER%" --runtime-image "%PF_HF_JOB_IMAGE%" --code-repo "%PF_HF_CODE_REPO%" --code-revision "%PF_HF_CODE_REVISION%" --code-sha256 "%PF_HF_CODE_SHA256%" --flavor cpu-basic --system-ca --submit

Use the actual intended supported flavor instead of cpu-basic if needed, in both commands, and retain the qualified flavor. Require provider stage COMPLETED, self_test_completed: true and confinement_probe_passed: true. Record the Job ID, URL and result. full_saas_live_qualified: false correctly means application acceptance is still outstanding.

If a Job ID is printed before an uncertain helper response, inspect that same Job; do not blindly submit another paid Job:

set "PF_HF_TEST_JOB=REPLACE_WITH_ALREADY_SUBMITTED_JOB_ID"
hf jobs inspect "%PF_HF_TEST_JOB%" --namespace "%PF_HF_OWNER%" --format json
hf jobs logs "%PF_HF_TEST_JOB%" --namespace "%PF_HF_OWNER%"

Compare its immutable identity and confinement evidence as in section 10. If no Job ID was returned, check the HF Jobs dashboard before retrying. Resolve any failed qualification check before bringing the upgraded controller online.

11.6 Set remote pins, upload the matching controller and restart

Where: Browser — HF → controller Settings → Variables and secrets. Keep all existing storage, key, SMTP, origin and budget configuration. Remove PF_HF_BOOTSTRAP if it is present, whether as a Variable or a Secret; absence defaults to false. This existing deployment must restore its durable state. Do not restore first-time administrator password settings.

Set these remote Space Variables, using the actual values just qualified:

If PF_HF_ASSISTANT_IMAGE is explicitly configured to a different runtime, qualify that image against the new code commit and SHA256 before resuming those workflows, even when the image digest itself is unchanged. Repeat section 11.5's dry run and paid self-test with that image as --runtime-image, keeping the same new code pins, and record its result separately. Preserve the main PF_HF_JOB_IMAGE pin. Local CMD set commands alone do not change remote Variables. Check the saved values from CMD:

python -c "import os; from huggingface_hub import HfApi; a=HfApi(token=True); v=a.get_space_variables(os.environ['PF_HF_APP_REPO']); keys=('PF_HF_CODE_REPO','PF_HF_CODE_REVISION','PF_HF_CODE_SHA256','PF_HF_JOB_IMAGE'); assert all(k in v and v[k].value==os.environ[k] for k in keys), 'Controller variables do not match the qualified pins'; assert v.get('PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS') is not None and v['PF_HF_WAL_RECEIVER_START_TIMEOUT_SECONDS'].value=='600', 'Unexpected receiver startup deadline'; assert 'PF_HF_BOOTSTRAP' not in v, 'Remove bootstrap from Variables'; print('CONTROLLER_CODE_PINS_VERIFIED')"

Require CONTROLLER_CODE_PINS_VERIFIED; also confirm bootstrap is absent from Secrets in the HF settings. Verify the flavor matches the qualification. If changing settings restarted the controller, pause it again. Upload only the matching current space/ context, then restart after the upload succeeds:

python deploy\saas\huggingface_native\deploy_release.py verify --bundle "%PF_HF_UPLOAD%"
python deploy\saas\huggingface_native\deploy_release.py --system-ca prepare --bundle "%PF_HF_UPLOAD%" --space "%PF_HF_APP_REPO%" --state-dir "%PF_HF_DEPLOY_STATE%"
python deploy\saas\huggingface_native\deploy_release.py --system-ca apply --state-dir "%PF_HF_DEPLOY_STATE%"
hf spaces wait "%PF_HF_APP_REPO%" --timeout 25m
python deploy\saas\huggingface_native\deploy_release.py --system-ca status --state-dir "%PF_HF_DEPLOY_STATE%"
hf spaces logs "%PF_HF_APP_REPO%" --tail 150

Run these individually. prepare is read-only and seals the current revision, new code pins, configuration fingerprint and exact generated files in a new local state directory. It schedules deletion of obsolete release-owned source and known historical registry extras; unexpected unrelated files stop the preparation. apply confirms pause, checks those pins again, submits one guarded commit and requests restart. It never edits Space Variables, Secrets, Buckets or quota data. If a write response is uncertain, run status first; do not repeat an upload. A reconciled commit that remains PAUSED needs only the ordinary hf spaces restart command, followed by wait and status. Require source_bytes_verified=true and running_revision_verified=true. HTTP and performance acceptance remain separate browser checks.

Save the controller commit. The native startup restores durable state and runs the packaged application migrations; do not run a local Windows migration against the hosted database. Require RUNNING plus Application startup complete in the runtime logs. If the wait ends in an error, still retrieve the logs and follow section 10; do not repeatedly restart or change database state. Build logs are available separately:

hf spaces logs "%PF_HF_APP_REPO%" --build --tail 150

11.7 Verify retained data and execution before reopening access

See result publication and verification (source package reference: R34_ARTIFACT_PUBLICATION_REPAIR.md) for the separate transfer, import and publication counters, safe failure codes and saved-result acceptance checks.

Keep the maintenance window in place while checking the direct application URL:

1. Sign in with an existing PF account. Confirm its organizations, projects, retained simulation records, attachments and results. Confirm that expected BYOK/BYOS settings remain available without exposing credential values.
2. Submit one small representative simulation. Confirm one expected HF Job, completed result collection, and the expected plots/report/downloads in PF.
3. Run a separate small cancellation test. Cancel through PF, confirm provider termination and the application's final cancellation state, and check that it can admit the next permitted workload.
4. Verify normal durable writes and persistence using known test records. Schedule any destructive restore/fault qualification in a separate disposable validation deployment, as described in section 9.
5. Record outcomes against this full release checksum, new code hash, runner commit, runtime digest, controller commit, variables and Job IDs. Perform the remaining applicable service/authorization checks from section 9 before treating the complete deployment as qualified.

RUNNING and the confinement self-test are separate checkpoints; neither proves that these application checks passed. Reopen the previous intended researcher access only after the required checks succeed. Retain the private backup and old deployment records for your recovery/retention policy; do not automatically delete them on the first successful startup.

11.8 If the upgrade cannot be accepted

Stop new submissions and settle or cancel any test Jobs, then pause the controller. Preserve the verified pre-upgrade backup, current state Bucket, keys, old/new deployment identities and failure evidence. A controller startup can claim durable ownership and publish recovery/WAL state even before the UI opens, so failure does not establish that remote state was untouched.

Review compatibility before selecting rollback. Reverting only a code pin is not a complete rollback: the controller source, private payload/revision/hash, runtime and configuration must form a known matching qualified deployment. An older controller is not automatically compatible with state written by a newer one. If recovery requires the pre-upgrade Bucket copy, use a reviewed restore procedure and explicitly account for changes made after that copy. Do not copy it over the active Bucket, enable bootstrap, change keys/prefix, or start a second controller as an improvised rollback. Resume service only after restore and application acceptance have been established.

Deployment-wide cost admission and storage maintenance

Compute admission is deployment-wide, in addition to existing user/org quotas. Set finite compute seconds, concurrency, runtime, input/output, and storage bounds before enabling researcher access. The optional monetary estimate uses integer micro-USD settings:

• PF_SAAS_DEPLOYMENT_BUDGET_MONTHLY_MICRO_USD
• PF_SAAS_DEPLOYMENT_BUDGET_FIXED_MICRO_USD
• PF_SAAS_DEPLOYMENT_BUDGET_SAFETY_MICRO_USD
• PF_SAAS_DEPLOYMENT_BUDGET_HOURLY_MICRO_USD

Enter current provider rates for the selected hardware and a conservative allowance for fixed costs. Reserving a Job includes bounded startup time. An ambiguous dispatch is not retried blindly, and its reservation is not released until provider termination can be established. A timed-out local HTTP request does not prove that a remote Job stopped.

These controls stop PF-controlled new work. They are not a contractual cap on the HF invoice: subscriptions, Space uptime, storage, API operations, provider billing granularity, other account activity, and stale/orphaned resources can still incur charges. User BYOK and BYOS services have their own billing controls. Review HF billing and provider-level limits directly; do not describe the PF estimate as an absolute maximum bill.

The storage meter conservatively retains reservations after ambiguous writes and deletions. It does not automatically recycle capacity or delete backup/WAL history. Near exhaustion, stop admitting work and conduct verified maintenance before the reserved WAL capacity is exhausted. Do not reset/delete the meter, change the prefix, or increase concurrency merely to bypass a stopped service.

To review or increase the byte ceiling deliberately, first stop the controller and establish that provider Jobs are stopped. In a private operator environment with the same bucket/prefix and credentials (not an ordinary unconfigured Windows shell), use:

python -m pf_hf_native.quota inspect
python -m pf_hf_native.quota raise-limit --new-limit-bytes NEW_CEILING --reason "Reviewed storage capacity increase" --confirm-supervisor-stopped

Use the existing ceiling in PF_HF_MAX_STORAGE_BYTES for this maintenance action; after its successful conditional update, set the new ceiling in the Space and restart. The operation retains usage/reservations and an audit record; it is an explicit operator cost decision, not a reset. Keep the WAL reserve unchanged unless a separately reviewed migration supports changing it.

Increasing native capacity without replicating the controller

Concurrent users are not the same as simultaneous simulation runs. Many researchers can be signed in while only a bounded subset of jobs execute; queued jobs also consume admission/storage reservations. Configure all three rather than treating a user count as a worker count.

Start with one slot and one active run. After the existing single-run, cancellation, recovery and publication checks pass, a small two-run trial can use:

This example allows two different users in the same organization to run concurrently, subject to current compute/storage reservations and provider capacity. A single user remains at one active run; increase that user's applicable cap deliberately if the trial specifically needs two runs by the same user. Raising these limits does not replenish consumed monthly allowance, storage admission or provider credits.

Set the Variables on the existing controller Space, retain PF_SAAS_DEPLOYMENT_PROFILE=single, and use a controlled restart after draining existing work. Do not create another controller or point an ordinary external worker at the native database. Submit three bounded jobs across authorized users: verify two actual provider executions overlap, the third remains queued, and it starts only after a slot is released. Then exercise cancellation, a controller interruption and concurrent downloads/private-state mutations; inspect attempt IDs, tenant isolation, lease gaps, immutable manifests and charged reservations. Each live trial consumes provider resources.

Increase to the next measured level only after recording response latency, queue delay, provider startup/runtime, upload/commit latency, controller memory/CPU, PostgreSQL connection/lock pressure and durable WAL latency. Native HTTP request admission and the isolated lease connection pool protect different resources. Start with the default SQL pools; increase lease capacity only within the profile's combined pool ceiling and after measuring commit delay. HF API/dispatcher threads share one main database client. Optional assistant stores and maintenance commands use additional connections; PostgreSQL remains capped at 60 connections. Keep one maintenance action at a time.

When API, database, durable-writer or provider limits require independently scalable tiers, use the distributed deployment guide. That migration adds shared PostgreSQL/object storage, independent API replicas and dedicated worker hosts. It preserves the same user, organization and deployment admission concepts, while changing the infrastructure. The capacity qualification guide separates authenticated-user load measurements from scientific throughput. Neither a 64-slot setting nor a large distributed ceiling certifies hundreds of thousands of concurrent users or simulations.

Architecture and execution limits

This native HF profile remains one controller, with configurable internal dispatcher capacity. PF_SAAS_WORKER_SLOTS starts that many worker threads, each responsible for at most one scientific attempt at a time. The default is one and the software ceiling is 64; this is a bound, not a qualified load level. Per-user, organization and deployment reservations still decide which work can start. Increasing a quota without available slots and provider capacity does not create additional execution throughput. For separate API replicas and multiple Linux/Docker worker hosts sharing PostgreSQL and object storage, use DISTRIBUTED_SAAS_DEPLOYMENT.md. That profile provides configurable capacity and admission controls; neither profile claims qualification for hundreds of thousands of concurrent simulations. Establish capacity, provider quotas, cost controls and failure recovery for the actual deployment before admitting that workload.

Space disk is a working copy. PostgreSQL does not use hf-mount or a bucket mount as its data directory. An acknowledged transaction depends on the remote WAL receiver durably publishing its encrypted bytes and advancing a conditional head. Loss of bucket access, a conflicting controller, or exhausted WAL capacity closes admission; there is no asynchronous fallback. Crash recovery starts from the remote base backup and authenticated WAL before exposing port 7860.

Only one controller is supported. Do not enable replicas or start a second ordinary run_pf_saas.py worker against the native database. A replacement controller fences the previous owner. The public API and all worker threads share the same process, service database pools and ownership guard.

The default public runtime image contains the scientific and reporting dependencies from the existing SaaS route and a generic checksum-verifying bootstrap. It contains no PF source, credentials, user data, or private workflows. PF code is delivered separately from a private repository at an exact commit and checked before use. Optional licensed/external research toolchains still require their applicable software, configuration, and rights; HF hosting does not supply MATLAB licenses, external hardware, or arbitrary remote services. Custom runtime images must preserve the security contract and be requalified. A public runtime image must contain only material that you have reviewed and may redistribute.

Execution boundary and supported behavior

The HF runner uses one dedicated Job/VM per invocation and never pools mutually untrusted researchers in one Job. Its unprivileged supervisor checks Landlock ABI 6 or later, seccomp, no_new_privs, process cleanup, and the effective access restrictions before executing a submitted program. Missing support or a failed probe blocks execution. This profile does not require Bubblewrap, a Docker socket, root, nested namespaces, mount privileges, or the experimental HF Sandbox API.

Within the Job, the submitted program can read the prepared /source, workload helper files in /trusted, and explicitly allowed runtime dependencies, and write its assigned work tree. The active bootstrap/runner source at /opt/pf/source is not readable by the workload. The dedicated Job's initially empty /dev/shm is writable for POSIX semaphores and shared-memory IPC; its capacity is bounded by the VM's memory. Landlock controls access to content; it does not make this a separate filesystem or PID namespace, and some path/process metadata may remain visible. Metadata-changing operations such as chmod, utime, and extended-attribute writes are blocked globally by seccomp because Landlock does not mediate all those operations, including on the work tree. Controller secrets and other tenants' inputs are not mounted into the Job. Landlock signal/abstract-socket scoping and seccomp protect the supervisor from the workload even though both use the image's unprivileged UID.

Network sockets are blocked for untrusted programs, including local listeners and direct external API calls. Anonymous Unix socket pairs remain available for in-workload communication. Existing trusted BYOK/BYOS orchestration remains in the controller; do not place those credentials in the Job. Programs requiring a local server, multiprocessing.Manager, or direct network access may need an approved compatible workflow design. Ordinary multiprocessing pools, threads, subprocesses, and new sessions/process groups are supported by the confinement design. The supervisor acts as a subreaper and terminates/reaps detached descendants before output collection. Test the actual scientific libraries and external-tool adapters before promising support for a particular workflow.

Resource limits bound each process; the selected VM and provider timeout bound the complete Job. Process-count limits are per UID, and output-directory checks are admission/collection checks, not hard disk quotas or delegated cgroup limits. The runner closes inherited descriptors, removes untrusted environment entries, and reaps its children before output collection. The controller retains capacity and budget reservations until the provider confirms the whole Job has terminated. A passing probe is enforcement evidence, not a security certification or signed attestation. The broader live checks in section 9 remain mandatory.

Administrator-approved research profiles can select additional immutable images for optional scientific toolchains. Keep operator license files and service keys out of images used for arbitrary private code; preserve the existing profile policy and qualify each selected image. A license requiring unrestricted outbound network access is not compatible with the native untrusted-workload network policy.

The application Space and Jobs can use different hardware. Choose Space memory and local disk sufficient for PostgreSQL, private state restore, API requests, and temporary transfers. Free/sleeping hardware is not a guarantee of continuous availability. Avoid scheduled sleep during active investigations.

The initial native recovery implementation bounds the extracted database backup at 8 GiB and stages compressed backup/WAL data in memory. Qualify the chosen Space's RAM, disk, and recovery time against your retained data volume before increasing storage allowances; this profile does not establish arbitrary database scale. A larger storage ceiling alone does not raise the restore bound.

Operation, restart, and recovery

• Start/restart only the native application Space. Do not manually start an additional controller or change PostgreSQL to asynchronous commit.
• The supervisor closes API admission if its worker, WAL receiver, or ownership guard fails. Provider Jobs may outlive a crashed controller; the durable journal retains their identity and unresolved budget reservations.
• Ordinary restart restores from the encrypted remote head. Never create a fresh empty cluster to conceal failed recovery. PostgreSQL major-version changes require a tested migration; do not point a different major binary at a backup.
• Before upgrades, retain the release/image identity, keys, and a verified recoverable backup. Stop new work, let active work finish/cancel with terminal confirmation, then replace the controller and perform the recovery checks.
• Monitor the HF Jobs list, account billing, local disk, durable byte meter, and remaining WAL reserve. HF bucket storage is mutable and non-versioned; the application's immutable encrypted objects are not protection from an operator deleting the bucket or losing the encryption key.
• Keep an independently retained, verified recovery copy for operational disaster recovery. A second HF Bucket can keep hosting within HF, but validate the copy's complete head/base/WAL/object dependency set before calling it a backup.
• An uncertain operation is an operator reconciliation case. Establish remote Job termination and durable receipt state before releasing reservations or removing transfer objects. Never infer termination solely from elapsed time.

From a private operator shell inside the running trusted controller environment, inspect the durable reservation ledger with:

python -m pf_hf_native.control status
python -m pf_hf_native.control reconcile RESERVATION_ID
python -m pf_hf_native.control cancel RESERVATION_ID

Use a reservation identifier returned by status. reconcile checks provider identity and terminal state without requesting cancellation; cancel requests cancellation and then checks again. Neither command releases concurrency until all recorded child Jobs are positively confirmed stopped. A successful check retains the entire reserved compute charge. An uncertain result returns a nonzero exit status and keeps the reservation; investigate provider state and retry reconciliation after it becomes conclusive. Missing submission evidence does not establish that a Job never launched. Do not edit database rows to force a refund. Terminal transfer objects left by a crashed controller are eligible for cleanup on the next normal native startup.

Storage metering is a lifetime admission bound, not measured current occupancy or a monthly reset. Base backups and WAL are retained, and deleted/uncertain objects remain charged. If the meter's reservation index fills, the offline pf_hf_native.quota compact-index maintenance command can consolidate its entries while preserving all charged usage; review its --help and stop the supervisor first. Replayed old operations can then be charged again. There is no automatic history garbage collection in this release.

Compatibility and evidence

The native profile is selected by its own Dockerfile/launcher and the explicit huggingface execution/storage settings. Existing standalone and Linux/Docker launch commands, requirements files, and reference infrastructure remain their existing routes. The optional hybrid gateway remains available. Native Python dependencies are pinned separately in deploy/saas/huggingface_native/requirements.lock.txt.

Local tests under tests/test_hf_native_*.py exercise contracts, fault handling, confinement policy, persistence/restore rules, and deployment packaging. They include checks that the public runtime context contains no private PF code and that ordinary Linux/Docker and standalone entry points retain their defaults. Local kernel execution is recorded separately from mocked provider contracts. The local Linux 6.12.107 VM checks exercised Landlock ABI 6 enforcement with Python 3.12.14, NumPy linear algebra, multiprocessing pools, threads, subprocesses, and detached-descendant cleanup on timeout/cancellation. These checks did not run in an HF Job or validate its image pull, repository mounts, or SaaS recovery. The native Jobs serialization test uses the real pinned huggingface_hub==1.32.0 SDK with mocked HTTP transport; it validates client request/response contracts without contacting or executing on HF. They must be reported separately from actual HF execution, PostgreSQL crash recovery, independent security assessment, scientific validation, and billing qualification. No live deployment or provider purchase is performed by the builder or by the release tests.

Use the deployment record template (source package reference: HUGGING_FACE_DEPLOYMENT_RECORD.md) to record your own identities, commands, outcomes and remaining acceptance checks. Prior successful Jobs qualify only their recorded image/code/hardware combination; they are not reusable pins or full SaaS approval for a newly generated payload.

Official platform references

• Spaces overview and visibility

• Docker Spaces

• Space image access

• HF Jobs images

• HF Jobs configuration and volumes

• HF Jobs VM and nested-namespace limitations

• Linux Landlock security and limitations

• Storage Buckets

• Bucket permissions

• HF CLI copy

• Bucket S3 credentials and addressing

• PostgreSQL synchronous replication

• HF CLI 1.32.0 commands

• HF token scopes

• Docker Desktop Windows installation

• Microsoft WSL commands

• SMTP2GO SMTP ports and credentials

• SMTP2GO verified senders

• SMTP2GO setup at Network Solutions

Review current account eligibility, prices, and platform behavior when deploying; documentation and service capabilities can change independently of r34.

2026-10-14 HF Astropy reference-cache maintenance

See R34_ASTROPY_CACHE_FIX_20261014.md for the small native-worker cache-directory correction, targeted validation, and compact deployment patch. Each run still computes fresh results; scientific models and isolation restrictions are unchanged.

2026-10-15 private-workflow reliability maintenance

See R34_WORKFLOW_RELIABILITY_FIXES_20261015.md for workflow refresh coalescing, private-authoring loading/error recovery, request-scoped storage reads, sanitized database diagnostics, and HF execution path/error corrections. The shared changes apply to standalone and SaaS components; native path staging applies to HF. The schema-32 release receipt records targeted checks only and preserves all historical validation evidence. Verify the full release with python release_tools/package_r34_workflow_reliability.py verify --package.