Distributed SaaS deployment

Current public-service and schema-10 upgrade instructions: r34 public service deployment. This includes both 2026-10-09 lifecycle packages. Large active-user capacity requires the measured qualification described there.

Use this guide with the consolidated release entry guide and shared capacity settings. This profile retains shared PostgreSQL/private S3 and Docker workers; no HF account or external managed-database vendor is required. Hosted schema is 9.

The distributed profile runs independent API replicas and dedicated Linux execution hosts against one shared PostgreSQL database and private S3 storage. Start with one API process, one host and one execution slot. Add API replicas and worker hosts independently as measured demand grows. The standalone application continues to use run_pf_gui.py; it does not require these services.

This profile supplies safe shared admission and a functional worker fleet. The configured maximum is an operator limit, not evidence of tested throughput. Concurrent signed-in users, concurrent HTTP requests, queued simulations and simultaneously executing simulations are distinct capacities. Hundreds of thousands of active users or simultaneous simulations require provisioned infrastructure, performance testing and operating capacity. A Hugging Face PRO subscription does not provision this fleet or cover its infrastructure charges.

Supported profiles

Do not replicate the HF native controller or place a Docker worker unchanged in a generic Kubernetes pod. The native controller owns a fenced durable writer, while the Docker executor requires a local trusted Docker daemon and host-visible work paths. The existing AWS Batch deployment remains available through its separate deployment guide; the new distributed profile requires Docker so every claim uses the same shared deployment-budget execution contract.

Start with one and grow by measured demand

A researcher can remain signed in while no simulation runs. A user may submit several jobs; the queue and active-run caps determine how many are accepted and executed. For example, USER_MAX_CONCURRENT_JOBS=1, an organization cap of 4, and a deployment cap of 8 permit one active run per user, up to four in each organization and eight across the service, provided eight executor slots and provider resources are available. The user cap applies within an organization. Assistant operations occupying execution slots share these limits.

To support hundreds of thousands of simultaneous users, qualify the API/session workload separately from compute. If each of 100,000 active users polls once every five seconds, that endpoint alone receives about 20,000 requests per second before login, asset and report traffic. This arithmetic is a planning example, not a measured result. If those users also require one simulation running each, provision and qualify that distinct number of scientific execution slots. The same release supplies configuration and shared admission mechanisms for the growth path; the final usable capacity depends on the measured deployment.

1. Prepare the infrastructure

Provision a private PostgreSQL service, private S3 bucket, an HTTPS load balancer and dedicated Linux worker hosts. Protect the database and worker hosts from public ingress. Use PostgreSQL TLS and backups, S3 encryption and least-privilege instance identities. Do not place credential values in images, source control or simulation containers.

Install Python 3.12, the release's hosted requirements, Docker Engine and its CLI on worker hosts. Install the release at /opt/pf-simulation and its virtual environment at /opt/pf-saas-venv. Build Dockerfile.saas, perform the release's image checks and select the resulting immutable image digest. Pre-pull that digest on every worker host; network image pulls during a job increase startup latency. Host resources must cover all configured slots, each job's CPU/memory limits, temporary artifacts and the controller processes.

Create a dedicated pf-saas account and private /var/lib/pf-saas directory on each machine. Only trusted worker controllers receive Docker access: access to a rootful daemon is a host-administration privilege. Keep worker hosts separate from the API tier and other tenant workloads. Configure the fixed pf-jobs Docker network with the required egress policy. Scientific requests cannot choose an image, daemon, network or host path.

Use a direct PostgreSQL connection or a session-affine pooler. Shared resource locks retain session advisory locks; transaction/statement pooling is not a compatible connection mode. Install the PostgreSQL server's CA certificate at /etc/pf-saas/postgres-ca.pem, readable by the service account. The example uses sslmode=verify-full; its database hostname must match the certificate. Local self-managed PostgreSQL is an option when this trust configuration and all shared-resource requirements are satisfied.

Copy deploy/saas/distributed/fleet.env.example to /etc/pf-saas/fleet.env with mode 0600, then replace the placeholders through private configuration. Use identical database, identity, signing, storage and admission settings on every replica. Use a separate random metrics token. The example starts with one running job and one worker host; increasing these numbers is a separate capacity and spending decision.

To enable the assistant, BYOK or BYOS, apply the corresponding settings in deploy/saas/production.env.example, including independent credential encryption keys. Mount the same private assistant and BYOS state paths on every relevant API and worker host, set their *_SHARED_STATE=true flags, and allow these paths in the systemd units' ReadWritePaths. A per-machine directory does not become shared by setting the flag. Existing assistant tool execution also requires its configured trusted Docker executor on assistant-serving hosts; provision and isolate that execution capacity separately. The minimal fleet example leaves these optional services disabled until their shared state and executors are configured.

The shared filesystem must provide consistent cross-host read-after-write visibility, atomic rename and durable fsync, plus cross-host advisory flock semantics supported by its client/server configuration. Use consistent paths, UID/GID mappings and private permissions on every host. An ordinary object-store or FUSE mount is not automatically suitable. Qualify two-host locking, permission revocation, interrupted-write recovery and failover on the selected filesystem before enabling these optional services. Shared PostgreSQL metadata and *_SHARED_STATE=true do not establish filesystem conformance by themselves.

2. Initialize or migrate once

For an existing deployment, stop admission and drain all worker hosts before migrating. Preserve a verified database/object-store backup and the previous immutable image. Do not erase the existing usage ledger, job history or storage bindings.

Install the reviewed release and runtime on each host before initialization:

sudo useradd --system --create-home --home-dir /var/lib/pf-saas --shell /usr/sbin/nologin pf-saas
sudo install -d -o pf-saas -g pf-saas -m 0700 /var/lib/pf-saas
sudo install -d -o root -g pf-saas -m 0750 /etc/pf-saas
sudo install -o root -g root -m 0600 deploy/saas/distributed/fleet.env.example /etc/pf-saas/fleet.env
sudoedit /etc/pf-saas/fleet.env
sudo python3.12 -m venv /opt/pf-saas-venv
sudo /opt/pf-saas-venv/bin/python -m pip install -r /opt/pf-simulation/deploy/saas/requirements-runtime.lock.txt
sudo /opt/pf-saas-venv/bin/python -m pip check

These are fresh installation commands; retain an existing private environment on upgrade. /opt/pf-simulation must point at the complete reviewed release on all hosts. Populate the private environment and CA file before continuing. Do not put populated configuration in a ticket or print its secrets to a terminal. The environment file is read by the systemd manager; the process runs as the unprivileged service account.

Run initialization once on one operator host using that same environment:

sudo systemd-run --unit=pf-schema-init --wait --pipe --collect --uid=pf-saas --gid=pf-saas --property=WorkingDirectory=/opt/pf-simulation --property=EnvironmentFile=/etc/pf-saas/fleet.env /opt/pf-saas-venv/bin/python /opt/pf-simulation/run_pf_saas.py init

For an already configured operator shell, the equivalent application command is:

/opt/pf-saas-venv/bin/python /opt/pf-simulation/run_pf_saas.py init

Schema version 9 retains execution/worker/drain metadata and adds indexed asset references, trigger-maintained admission counters, durable AI admission/recovery state, and bounded maintenance records. It preserves existing jobs and attempts. Run this explicit migration before starting production processes with PF_SAAS_AUTO_CREATE_SCHEMA=false. Do not race multiple migrations or perform an index migration during live admission.

3. Start API replicas and worker hosts

Install deploy/saas/distributed/pf-api.service on API hosts and pf-workers.service on worker hosts. Adapt the fixed paths to the actual installation. For example, on the API host:

sudo install -o root -g root -m 0644 /opt/pf-simulation/deploy/saas/distributed/pf-api.service /etc/systemd/system/pf-api.service

On every worker host:

sudo install -o root -g root -m 0644 /opt/pf-simulation/deploy/saas/distributed/pf-workers.service /etc/systemd/system/pf-workers.service
sudo systemctl daemon-reload

Enable the appropriate service:

sudo systemctl daemon-reload
sudo systemctl enable --now pf-api.service

On each worker host:

sudo systemctl daemon-reload
sudo systemctl enable --now pf-workers.service

Each serve process is one API replica. Add replicas through the load balancer or deployment system; signed-in sessions do not consume one dedicated Python process or database connection. The API serves HTTP only; the worker service starts PF_SAAS_WORKER_SLOTS independent controller processes using the multiprocessing spawn method. Each process owns its database connections and runs at most one scientific job at a time. PostgreSQL row locks and the deployment-budget transaction prevent two workers from claiming the same attempt or exceeding the shared capacity/compute allowance. Requests using the same project-scoped Idempotency-Key and payload return the existing job; a different payload with that key is rejected.

API session state and authorization are in the shared database. Replicas use the same signing secret and public origin; sticky sessions are unnecessary for ordinary workbench requests. Configure the load balancer's health checks against /healthz and /readyz, and set trusted proxy CIDRs only when the network prevents direct spoofed ingress. Keep /internal/metrics private even though it requires a separate bearer token.

Confirm the service health, submit one bounded simulation, verify the selected attempt completes, inspect its outputs and download its ZIP. Then test cancellation, retry, permission revocation and a drained-worker restart before increasing capacity.

Upgrade to the consolidated r34 release

1. Put the load balancer in maintenance mode to stop new admissions. Disable scheduled PF scaler/maintenance clients. Stop API services on all API hosts, then request sudo systemctl stop pf-workers.service on every worker host. The worker supervisor drains; wait for it to exit. Confirm assistant work, Docker containers, leases and uncertain provider calls are stopped or resolved.
2. Take a consistent PostgreSQL backup and private S3/shared-state backup, including retained object versions and every encryption/session key. Record the paired database/storage restore point. Verify recovery on a separate environment. Retain previous immutable source, images and configuration.
3. Extract this complete release into a new versioned directory on each host. Build and test Dockerfile.saas, obtain its immutable digest, and pre-pull the same digest on every worker. Retain the original private environment and storage paths. Change PF_SAAS_DOCKER_IMAGE to the qualified new digest.
4. Add the new shared settings from fleet.env.example without replacing existing secrets, limits or quota history. Set DEPLOYMENT_API_PROCESSES to total API processes and DEPLOYMENT_WORKER_PROCESSES to all worker-slot processes. Count any additional database clients in the reserved budget. Keep these values identical across the fleet. An assistant-enabled fleet also needs its existing shared private paths and keys.
5. Point /opt/pf-simulation on all hosts at the new complete release, install the new locked runtime dependencies, and run the pf-schema-init command above once. Wait for successful schema-10 migration; keep all services stopped if it fails. Do not allow old services to write the upgraded schema.
6. Start the API service, then worker services. Check /healthz and /readyz through the load balancer and review logs. Perform the consolidated acceptance checks. Reopen admission and any previously authorized scaler only after acceptance.

A failure after migration requires forward repair or a coordinated restoration of the paired pre-upgrade database/object/private-state/configuration backup and old immutable images. A code-only rollback does not downgrade schema 10 or reverse its counters. Do not reset ledgers to get an older binary to start.

4. Configure admission separately from infrastructure

The scheduler rotates organizations, skips locked organization rows and skips users already at their running cap. Candidate selection is only a hint: the claim transaction rechecks current permissions, organization/user limits, storage reservations and the global budget. Additional workers cannot bypass these gates. Queue admission and usage history survive restarts.

Scientific jobs and assistant operations that occupy execution slots share both per-user and organization caps. Non-execution chat operations do not consume a simulation slot. An active or ambiguous provider reservation continues to occupy its user's and organization's capacity even if its controller operation has ended; only verified termination or a valid unsubmitted release frees it.

A queued job needs enough remaining deployment allowance for its reservation, including the execution margin. If an operator raises a ceiling from 5,940 to 6,600 seconds, the additional allowance is 660 seconds; it does not reset prior commitments. Inspect the ledger with python -m pf_saas.deployment_control status before changing limits. PF accounting is conservative admission accounting, not the provider invoice or a guarantee against non-execution infrastructure charges.

API and database capacity

HTTP admission occurs before native ownership checks, body buffering and application/database work. Saturation returns 503 with Retry-After; export saturation returns 429. Accepted requests retain normal authentication, CSRF, project permissions and native durability checks. Four separate bounded health/readiness slots remain available. Request and upload bounds are per process and protect memory; total upload limits are still enforced per request. Account for buffered bodies, parsing, multipart copies, reports and ZIP spools when choosing memory limits.

Synchronous authorization, storage and policy operations in async endpoints run in the bounded framework thread pool, leaving the event loop available to other requests. SQL connection pools are bounded separately; increasing HTTP admission does not automatically add threads, connections or execution slots. /internal/metrics includes per-process active request/upload counts and their capacities. Aggregate them across replicas without exposing tenant identifiers.

Budget database connections for every API process, worker-slot process, assistant store and maintenance client. A main service database client can use DATABASE_POOL_SIZE + DATABASE_MAX_OVERFLOW; a worker that renews leases also has its independent DATABASE_LEASE_POOL_SIZE. The optional assistant store now has a separately bounded pool (default 2, no overflow); DATABASE_ASSISTANT_POOL_SIZE and DATABASE_ASSISTANT_MAX_OVERFLOW control it. Keep spare database connections for administration, monitoring and migrations. Start with small pools; adding replicas must not silently exceed the database server's connection limit. Native HF API and dispatcher threads share one configured service database client, and the combined main/overflow/lease/assistant ceiling is capped at 25 to leave room under its fixed 60-connection database limit for assistant, WAL and control work. Do not run concurrent native maintenance clients as a way to increase service capacity.

Shared SQL sessions, OAuth login state, revocation and authentication rate limits make ordinary API requests independent of a particular replica. Qualification and metrics caches contain derived information only. BYOS and assistant files still require the shared-state contract in section 1. A connection pool or a cache does not replace durable authorization or cross-replica admission.

5. Observe and optionally automate worker capacity

Read an aggregate snapshot without modifying jobs or capacity:

python run_pf_saas.py scale-status
python run_pf_saas.py scale-status --prometheus

GET /internal/metrics with Authorization: Bearer <metrics token> exports low-cardinality queue, running-job, worker-slot and desired-host gauges. It includes no tenant labels, job identifiers, requests or credentials. Each API process caches the snapshot for ten seconds. The desired count is a bounded estimate using queued demand, per-user/organization room and the current deployment allowance; only claim-time admission is authoritative. Worker registration is observability metadata, never an execution lease.

An external infrastructure autoscaler can consume these metrics. A supplied AWS Auto Scaling Group adapter supports bounded scale-out and optional automatic scale-in:

1. Provision a fixed, unweighted, single-instance-type worker group. Its launch template must install the verified release, private configuration and pf-workers.service on each instance. Instance identities require only the deployment resources. The scaler identity requires DescribeAutoScalingGroups and SetDesiredCapacity for the selected group; it does not require permission to edit group bounds or launch templates.
2. Set PF_SAAS_SCALING_GROUP and explicit minimum, maximum and step settings. Provider bounds must stay within the PF operator bounds. Disable other desired-capacity controllers, scheduled scaling and scale-in policies for this group; the PF advisory lock serializes PF scaler processes, not unrelated provider administrators.
3. Preview with python run_pf_saas.py scale-asg. It makes no capacity changes.
4. After reviewing the target and associated infrastructure cost, use python run_pf_saas.py scale-asg --apply. By default it can only increase capacity within the fixed bounds and honors provider cooldown. Automatic removal also requires the opt-in configuration below.
5. For ongoing authorized scale-out, install pf-scale-out.service and its timer, then explicitly enable the timer. The supplied service includes --apply; enabling it authorizes paid capacity increases. Leave it disabled for the minimal deployment or an HF-only subscription.

With automatic scale-in disabled, remove a host manually: first stop pf-workers.service; SIGTERM stops new claims and lets current simulations, heartbeats and result publication finish. Wait for the service to exit and confirm the host has no active containers or leases before removing it. The service intentionally has no forced stop timeout. An unavailable host retains ambiguous reservations until termination is independently verified; a missing container on another host is not proof of termination. Use the existing stopped-execution reconciliation command only after obtaining that proof.

Optional automatic scale-in

Enable PF_SAAS_SCALING_AUTO_SCALE_IN=true on the scaler and its dedicated worker hosts. Keep assistant-serving APIs on separately provisioned hosts. Each worker obtains its instance ID from fixed, proxy-free IMDSv2 endpoints; it refuses a configured PF_SAAS_WORKER_INSTANCE_ID that does not match. IMDSv2 access is a worker-controller prerequisite, never a scientific-container permission. Use one supervisor service and identical private configuration per instance.

Configure the group with NewInstancesProtectedFromScaleIn=true, protect every existing member, and suspend AZRebalance. The scaler verifies these conditions before automatic removal. Disable unrelated desired-capacity controllers and instance refresh operations. Grant the scaler TerminateInstanceInAutoScalingGroup for this group in addition to its scale-out permissions. The supplied CLI disables SDK retries because exact-instance termination with capacity decrement has no application idempotency key. Preview remains read-only; --apply is required for both the database drain intent and the provider request.

The scaler selects one healthy, freshly registered member and commits its instance-specific drain latch. Job claims lock and check that same record, so a committed drain blocks new work. The supervisor finishes its children, their lease maintenance and result publication. It acknowledges drained only after all child processes exit, the local Docker daemon confirms no live PF containers, and database checks find no running attempts, active or ambiguous execution reservations, or unpurged unpublished local evidence on that host. It stays alive to provide fresh acknowledgements until retirement.

On a later pass, the scaler rechecks group membership, the same supervisor generation, the fresh acknowledgement and execution evidence. It commits a one-shot termination intent before calling TerminateInstanceInAutoScalingGroup for that exact instance. A surplus host is removed with ShouldDecrementDesiredCapacity=true; desired capacity is never reduced by selecting arbitrary members. If demand returns during drain, the same drained instance is replaced with ShouldDecrementDesiredCapacity=false, preserving requested capacity without reopening its latch.

A timeout, malformed reply, unavailable host or interrupted database acknowledgement never authorizes a second termination call or another host selection. The existing intent remains visible and blocks further automatic removals. A later observation of exact group-member disappearance and the corresponding desired-capacity outcome records group_removal_observed and a permanently fenced retired host. This confirms group membership, not EC2 terminal state or stopped infrastructure billing. Verify those separately through the provider. A permanently ambiguous or rejected request requires operator investigation; do not erase the record or replay a decrement based only on a timeout. Drained instances remain fenced across supervisor restarts and configuration refreshes. Unpublished local failure evidence must be published or explicitly purged before that host can retire.

AWS termination acceptance is asynchronous. Scale-in protection also does not prevent health-check replacement, Spot interruption or manual administrative termination. This mechanism makes PF-initiated scale-in drain safely; it does not turn an infrastructure failure into graceful completion. The ledger and attempt recovery retain ambiguous execution capacity independently.

Provider semantics: exact-instance termination and capacity decrement, scale-in protection and its limits.

Performance and qualification

Increasing concurrency improves throughput for independent jobs; it does not shorten one simulation. Pre-pulled worker images, asynchronous coalesced progress and independent lease renewal reduce controller overhead. Inspect per-attempt timing fields to distinguish preparation, input transfer, provider execution observation, result collection, verification, artifact publication and final commit. HF runtime and PF end-to-end duration describe different intervals; PF's execution observation is not the provider's actual start timestamp.

Workbench polling uses configurable active, idle, hidden-tab and retry-backoff intervals. Defaults are 5, 30, 60 and 120 seconds, with qualification checks every 60 seconds. Settings are PF_SAAS_UI_POLL_ACTIVE_MS, IDLE_MS, HIDDEN_MS, MAX_BACKOFF_MS and QUALIFICATION_MS. Tune them with API capacity and user experience measurements rather than multiplying aggressive polling across every connected user.

Use CAPACITY_QUALIFICATION.md for the staged workload contract and reproducible API probe. Measure authenticated user traffic separately from actual scientific execution; never label control-plane fixture traffic as simulation throughput.

Before production use of the distributed profile, run the PostgreSQL contention/admission gates against a disposable test database:

PF_TEST_POSTGRES_URL='postgresql+psycopg://.../disposable_test_db' python -m pytest -q tests/test_r34_validation_queue_postgres.py

Those tests can reset their test schema; never target production. Also run the release's hosted and standalone suites, verify the image and actual providers, and measure queue latency, submission throughput, CPU/memory, database lock/commit latency, storage throughput, lease gaps and cancellation under the intended load. Locally passing SQLite/fake-provider tests do not substitute for these PostgreSQL and real-infrastructure gates. No 100,000-user or 100,000-job capacity qualification is asserted by this release.