PF Simulation r34 — Hugging Face gateway deployment

For the separate native deployment mode that hosts the SaaS controller, database process, execution Jobs and durable storage on HF, see HUGGING_FACE_NATIVE_DEPLOYMENT.md. The native guide's later existing-deployment upgrade section and recovery companion cover pausing, verified HF Bucket backups, qualified private-code replacement and controller recovery. Those steps apply to native HF execution. This guide continues to describe the optional hybrid gateway and its existing Linux/Docker execution backend.

Supported architecture and scope

This optional profile places a small Docker gateway in a Hugging Face Space and keeps the complete SaaS service on the existing dedicated Linux/Docker backend. Users open the Space's direct https://owner-space.hf.space application address. HTTPS requests reach the same SaaS API, worker, PostgreSQL database and durable state used by the full r34 reference deployment.

This is hybrid hosting. The Space does not run the scientific sandbox, PostgreSQL, private workflow programs or autonomous investigation workers. It does not use Hugging Face Jobs or the experimental HF Sandbox API. A separately operated Linux/Docker backend remains necessary. A single Space without that backend is not a full PF Simulation deployment.

No standalone startup path, standalone requirements file or default Linux deployment setting is replaced. Gateway enforcement is activated only when the operator sets PF_SAAS_HF_GATEWAY_SECRET. Leaving that setting absent retains the normal deployment profile. The gateway dependency lock is separate from the scientific runtime.

Before deployment

1. Provision and qualify the full backend using the Linux reference guide. Keep its rootless Docker worker, existing PostgreSQL and persistent state. A new managed PostgreSQL subscription is not required by this profile.
2. Use a dedicated, certificate-valid HTTPS backend hostname reachable from the Space on port 443, for example https://pf-backend.example.org. Preserve that hostname in DNS, Caddy and the TLS configuration. Do not expose the API's loopback port, PostgreSQL port, Docker socket or administrative shell.
3. Create a Docker Space and obtain its actual direct application origin. Copy the https://…hf.space address rather than deriving it from an account or repository name. Creating or upgrading a Space can incur charges; the generator below performs neither action.
4. Keep a private backup of the existing backend environment and durable data. Schedule the origin change because existing browser sessions and external callback URLs may need renewal. With gateway mode enabled, direct backend web requests without the shared gateway secret are intentionally rejected.

Only the gateway needs to be uploaded to HF. Public Space repositories expose their uploaded files. The bundle generator copies an exact allowlist and excludes the shared scientific source, release archives, databases, credentials, private layers and user outputs. Choose and qualify the appropriate Space visibility; PF account/project authorization remains enforced inside the application.

Generate the upload

Run from the unpacked r34 release with Python 3.12. The output parent must already exist; the output directory must not exist. Substitute your actual two origins:

python3.12 deploy/saas/huggingface/build_space.py build \
  --public-url https://owner-space.hf.space \
  --upstream-url https://pf-backend.example.org \
  --output /home/pfsim/pf-hf-upload

The command produces:

space/bundle-manifest.json records hashes of the upload inputs. A repeated build from the same source and origins has the same archive SHA-256. The generator refuses to overwrite existing output and rejects traversal and symlinks in its input/output paths. It does not read the process environment to populate configuration, and it never accepts a secret on the command line. URL validation at this stage is structural: DNS, TLS and connectivity require the later live preflight.

Upload only the contents of space/, or extract space.zip at the Space repository root. Do not upload the parent output directory or the full release. The Dockerfile explicitly runs as UID 1000 with writable user cache paths and serves port 7860. Python packages are pinned to locally exercised versions; the upstream Python base image tag remains mutable and should be reviewed or pinned to an approved digest by the operator before a controlled deployment.

Configure both ends

Generate a fresh high-entropy secret of at least 32 printable ASCII characters using your private secret manager. Use a distinct secret from database, session, BYOK and BYOS vault keys. Save it in the following two places only:

The reference service.env format uses single-line JSON-quoted values. Review and merge the two entries from backend.env.example; replace its placeholder privately. Do not replace the existing environment file with this fragment. Retain its database URL, vault keys, persistent paths, executor settings, SMTP, BYOS and other service settings. Keep its mode 0600 and never source it as shell code. The generator does not modify this file or the reference Caddyfile.

Restart API and worker using the existing reference service procedure, and let HF restart the Space after updating its runtime settings. Caddy must still accept TLS for the backend hostname; only the application's public origin changes. Gateway mode validates the shared secret and public-origin header before application routes, including health endpoints. It derives the effective application origin from configured values, so ordinary forwarded headers cannot select an arbitrary origin. The shared secret must not be exposed in proxy or access logs. There is no unauthenticated direct-backend bypass.

The secret is a gateway-to-backend boundary credential, not a PF user credential. Users still sign in to PF and retain all existing organization, membership, research policy and project restrictions. An operator with the gateway secret can reach the backend as the gateway; protect and rotate it accordingly. Rotate both ends together during a maintenance window. Administrative host access and the existing provisioning CLI remain available through your restricted SSH path.

Browser origin and external integrations

Use the direct Space application in a top-level browser tab. The embedded HF view offers a link to open the workbench; it does not relax the workbench's frame restrictions or rely on third-party session cookies. Keep Secure, HttpOnly, SameSite and CSRF protections enabled.

After an origin change, configure any existing external identity provider's approved callback/logout URLs for the new public origin. Check generated password-reset and invitation links and any BYOS NAS connector endpoint. Update external links/bookmarks that intentionally point to the old application origin. Do not change storage-provider credentials or backend Docker/database addresses merely because the public URL changed.

Gateway capacity and forwarding settings

The gateway streams HTTP request/response bodies; it does not store uploaded literature, download payloads or BYOK keys. It uses verified HTTPS to one fixed backend and does not follow upstream redirects automatically. Local health is not evidence of backend health. Proxy timeout/connection limits are admission and availability settings, not monthly spending limits. HF platform request limits can still apply; test representative large files on the target Space. The gateway's longer read timeout does not override HF edge timeouts. Confirm the actual HF edge and backend Caddy/proxy behavior for the longest legitimate assistant operation before claiming full target acceptance. A dropped browser connection does not prove the backend execution stopped or justify a budget refund.

Do not trust all forwarding addresses by setting 0.0.0.0/0 or ::/0. Leave trusted proxy networks empty until the actual HF ingress path and address ranges have been verified. Unverified forwarding headers are not trusted for client identity; IP-based sign-in throttling may aggregate users behind the same proxy. Qualify trusted client-address forwarding before exposing a multi-user service. Do not disable authentication throttling to work around an unverified proxy. Do not guess private-network ranges such as all of 10.0.0.0/8 as a substitute for verifying the actual ingress chain.

Costs and deployment-wide limits

This profile incurs the chosen HF account/Space charges plus the existing Linux backend, storage, backups and network costs. It creates no HF Jobs or automatic hardware upgrades. Previous estimates for HF Jobs workers do not quote this hybrid architecture. Choose fixed-price host resources and disable provider autoscaling/automatic storage expansion where available. BYOK and BYOS put designated AI/storage bills on users' accounts; they do not remove the operator's infrastructure and operational-storage costs.

PF's optional deployment-wide execution budget applies to aggregate admitted execution across SaaS users. Per-user limits do not enlarge this shared budget. Budget admission does not stop the HF subscription, an always-running Space, the Linux host, PostgreSQL, retained storage, bandwidth or third-party provider charges. It cannot guarantee a precise combined provider invoice. Configure provider-side spending/usage caps and capacity limits separately, with a reserve for fixed charges. Operator-funded model API calls, when enabled, also remain outside the isolated execution cost estimator and require the model provider's own account limits.

Activate the shared execution allowance

Merge these settings into the existing private backend service.env before restarting the services. These are example capacity choices, not a recommended monthly bill or per-user quota:

PF_SAAS_DEPLOYMENT_BUDGET_ENABLED="true"
PF_SAAS_DEPLOYMENT_MONTHLY_COMPUTE_SECONDS="36000"
PF_SAAS_DEPLOYMENT_MAX_CONCURRENT_JOBS="1"

This example permits 36,000 aggregate execution seconds per UTC calendar month and one admitted execution at a time across all users and organizations. The system reserves the maximum bounded execution time before launch; a task can therefore be refused when the remaining allowance is smaller than its timeout, even if the task might have finished sooner. Assistant operations with multiple bounded probes reserve their combined allowance. This is execution time, not a hardware-vCPU-second meter or a provider billing measurement. Each bounded execution also reserves 60 seconds for launch and termination verification; a four-part validation reserves that margin four times. Verified completion settles the observed time. Uncertain termination retains the full reservation and its execution slot for operator reconciliation.

The optional monetary estimate gate has these additional backend settings:

All monetary values are integers; 1 US dollar = 1,000,000 micro-USD. They are operator inputs, not prices fetched from HF or the backend provider. A zero hourly rate makes the monetary estimate zero and must not be interpreted as free hosting. Choose a conservative rate appropriate to the actual backend contract, retain the execution-seconds limit, and account for any fixed charges separately.

Create the additive budget tables explicitly before starting budget-enabled services. Run as the existing service account, from the active release directory, using the existing backend virtual environment:

cd /home/pfsim/pf-host/current
/home/pfsim/pf-host/venv/bin/python -m pf_saas.deployment_control \
  --env-file /home/pfsim/pf-host/config/service.env init
/home/pfsim/pf-host/venv/bin/python -m pf_saas.deployment_control \
  --env-file /home/pfsim/pf-host/config/service.env status

The private environment file must have owner-only permissions. The initialization creates only optional budget tables; normal core migrations and default Linux deployments do not create them. API and worker must use the same database and budget settings. Restart both after successful initialization. A missing budget schema prevents budget-enabled operation instead of silently bypassing the cap. The status command reports deployment-wide usage and unresolved reservations.

Before production admission, run the supplied PostgreSQL concurrency gate with PF_SAAS_TEST_DATABASE_URL set privately to a separate test database, using the same PostgreSQL major version and application driver as the deployment:

/home/pfsim/pf-host/venv/bin/python \
  deploy/saas/huggingface/validate_postgres_budget.py

It creates and removes one randomly named test schema and checks competing connections, cross-organization capacity, dispatch deduplication, transaction rollback and unresolved reservations crossing a UTC month. No configured test database produces not_run, never a pass. This gate does not establish Docker isolation, HF routing or provider billing behavior; those remain separate checks.

Each admitted execution attempt receives persistent accounting before dispatch. Confirmed termination settles actual measured duration conservatively; an uncertain launch, crash or unverified stop retains its reservation and capacity. Re-running a CLI, an HTTP timeout or requesting cancellation does not prove a container stopped. After independently inspecting the worker, container and retry state and confirming termination, the operator can reconcile an unresolved reservation by its ID:

For inspection, a normal job's reservation retry key is job:ATTEMPT_ID and its container is pf-job-ATTEMPT_ID. Budget-enabled assistant containers are named pf-ai-RESERVATION_ID-1 and, for subsequent validation children, -2 through -4. Inspect the relevant worker/API operation as well as those container names; a missing name alone does not rule out an interrupted launch still in flight. Do not reconcile while a worker could still start or retry that execution.

/home/pfsim/pf-host/venv/bin/python -m pf_saas.deployment_control \
  --env-file /home/pfsim/pf-host/config/service.env \
  reconcile RESERVATION_ID --confirmed-stopped

This conservatively charges the full reservation when duration is unknown. Use release-unsubmitted RESERVATION_ID only for reservations never claimed for execution; the CLI refuses that release after a dispatch claim. Unfinished work continues to consume allowance across a UTC month boundary. Conservative cross-month accounting can overcount; it does not create a fresh allowance for an unresolved earlier execution. Preserve these tables in ordinary backend backups and never delete them to reset a monthly allowance.

No HF token, account billing access or real deployment is needed to generate the bundle. This release does not publish the Space or purchase resources for you.

Preflight and target acceptance

After configuring both ends, provide the shared secret to a private shell environment using your secret manager. Do not include it as a command argument, commit it or capture the populated environment in logs. Then:

python3.12 deploy/saas/huggingface/build_space.py check \
  --public-url https://owner-space.hf.space \
  --upstream-url https://pf-backend.example.org

The check reads PF_HF_GATEWAY_SECRET, verifies HTTPS certificates, refuses redirects and ignores environment HTTP proxies. It reports status codes only: gateway health and proxied backend health must be 200; direct backend health without authentication must be 403; correctly authenticated backend health must be 200. No response bodies or secrets are printed. This check does not validate the scientific system or declare live deployment acceptance complete.

Record target acceptance before inviting users:

• Verify unauthorized direct backend requests and spoofed gateway/forwarding headers are rejected; confirm authenticated requests reach the expected PF release. Validate HF ingress client identity and sign-in throttling.
• Open the HF embed and then its direct application link. Test sign-in/out, session expiry, CSRF rejection, password reset, invitations, membership and external identity callbacks where enabled.
• Upload, associate, remove and download literature. Exercise all attachment routes and a file representative of the intended maximum size. Check error messages for oversized requests, interrupted streams and backend outages.
• Run a normal simulation, private workflow, active override stack, AI probe and autonomous ISAES investigation. Pause/resume, regenerate reports and download results. Confirm existing sandbox limits and cross-user isolation.
• Test BYOK and each enabled BYOS destination/connector without disclosing credentials. Verify institutional policy restrictions still apply.
• Restart/sleep the Space and restart API/worker independently. Confirm queued work, checkpoints, saved results and budgets remain on the durable backend; no science execution depends on the Space's ephemeral disk.
• Exercise simultaneous users at the deployment budget boundary, denied work, retries/timeouts, month rollover and process interruption. Reconcile admitted usage with operator records and separately inspect provider billing settings.
• Verify the backend backup/restore procedure and rollback to the previous public origin with gateway mode removed. Re-run ordinary Linux and standalone smoke tests independently of the gateway profile.

Local automated checks cover the gateway boundary, configuration and upload packaging; consult the release validation record for their exact results. Live HF deployment, provider billing controls and target-host isolation have not been exercised by the local release tests. They remain operational acceptance gates, not claimed passes. Commercial software, hardware instruments and other optional integrations keep their existing external prerequisites.

Rollback

Stop or drain admission, restore the previous PF_SAAS_PUBLIC_URL, remove the optional PF_SAAS_HF_GATEWAY_SECRET entry and restart services through the normal reference procedure. Restore external callback/connector URLs if changed. Retain existing PostgreSQL data, vault keys, Docker configuration and state; there is no data migration solely for changing the gateway. Disable or pause the Space separately through HF to end any continuing compute billing under your chosen plan. Do not remove a budget ledger to reset spending allowances. The execution allowance can remain enabled on the ordinary Linux deployment. To intentionally revert that optional policy as well, set PF_SAAS_DEPLOYMENT_BUDGET_ENABLED="false" and restart both services after draining work; this removes PF's admission protection, so review the consequences first. Keep its tables and records for reconciliation and later re-enablement.

Official platform references

Reviewed 2026-09-29. Platform capabilities, prices and account requirements may change; inspect these official sources before purchasing or deploying:

• Docker Spaces: port, runtime secrets, UID and persistence
• Space configuration metadata
• Space visibility, hardware, network and lifecycle
• Current Hugging Face pricing