# r34 public service deployment and scale qualification This release integrates `PF_r34_HF_Lifecycle_Qualified_20261009` followed by `PF_r34_HF_Recovery_Correction_20261009` into the full release whose incoming SHA256 is `140cd2f3bbf55dc48f4095566c0dc9f367d85772666beb5fa60c87dac80d32de`. It also extends public registration, durable email, account-wide admission, distributed recovery, managed HF accounting and mixed-workload qualification. Do not install those patch ZIPs again over this release. The software can be configured from one user upward. **This archive does not certify hundreds of thousands of simultaneously active users.** That target requires a provisioned fleet, provider allocations and a successful measured campaign with the intended workload, data volume, regions and failure cases. Changing a numeric limit or accepting 200,000 credentials is not that evidence. ## Choose the architecture before opening registration | Deployment | Data authority and execution | Public-service scope | | --- | --- | --- | | Standalone | Local application and local files | Preserved; no SaaS infrastructure required | | Native HF, profile `single` | One fenced controller, supervised PostgreSQL, encrypted HF recovery, HF Jobs | Existing deployment upgrade; bounded pilot after readiness checks; never run two native controllers against one prefix | | Managed HF, profile `hf_managed` | Shared external PostgreSQL, encrypted HF objects, scoped leases, HF Jobs | Multiple coordinated API/worker replicas; requires external database, shared-state design, approved HF capacity and network connectivity | | Linux/Docker, profile `distributed` | Shared PostgreSQL, private S3, dedicated Docker workers, private shared assistant/BYOS files | Independent API and worker scaling behind a controlled HTTPS ingress | The existing `ignalum/pf-simulation-saas` can first receive the full native upgrade while retaining its identity, users, keys and storage. This does not turn its single controller into a fleet. A migration to managed HF or a distributed Linux/Docker deployment is a separate, explicit architecture step. Neither the release builder nor its readiness checker provisions resources. For native upgrade follow [the existing-service guide](R34_NATIVE_HF_RELEASE_UPGRADE.md). For managed HF follow [the managed migration guide](HUGGING_FACE_MANAGED_DEPLOYMENT.md). For Linux/Docker follow [the distributed guide](../DISTRIBUTED_SAAS_DEPLOYMENT.md). ## Maintain privacy while making the application reachable For HF, a **Protected** controller Space makes the running application reachable while retaining private repository source. Keep the scientific code Space, Bucket, backups and credential material private. Do not switch the code Space or the full source repository to Public. Protected visibility requires a supporting HF plan; see the [official visibility documentation](https://huggingface.co/docs/hub/spaces-overview#space-visibility). The deployment tools never change visibility for you. Use the actual HTTPS application origin as `PF_SAAS_PUBLIC_URL`, with no path. Verify direct and embedded access, cookie/CSRF behavior, origin checks and the exact forwarded-address contract before allowing untrusted users. See [public launch readiness](PUBLIC_LAUNCH_READINESS.md). Never enter `0.0.0.0/0` or an invented HF proxy network to make rate limits appear functional. If HF does not supply a verifiable ingress contract for the chosen topology, use a controlled ingress/gateway and prove that its origin cannot be bypassed. Only platform administrators may open HF accounting/recovery pages or APIs. Organization owners and organization administrators do not acquire this role. Registration creates a non-administrator account and requires email verification. Private assistant workspaces, project grants, BYOK and BYOS remain user scoped. ## Public registration and real account email Keep the existing `PF_SAAS_SECRET_KEY`: it protects sessions and encrypts queued account email. Retain the existing HF state key and BYOK/BYOS credential keys. Do not regenerate them during an upgrade. For local accounts configure: ```text PF_SAAS_AUTH_MODE=local PF_SAAS_REGISTRATION_ENABLED=true PF_SAAS_PUBLIC_URL=https://YOUR_APPLICATION_ORIGIN PF_SAAS_MAIL_DELIVERY_WORKER_ENABLED=true ``` Retain existing encrypted SMTP, or choose the optional Resend HTTPS provider. For SMTP2GO on HF, preserve `PF_SAAS_MAIL_MODE=smtp`, host `mail.smtp2go.com`, port `443`, `PF_SAAS_SMTP_SSL=true`, `PF_SAAS_SMTP_STARTTLS=false`, the verified `PF_SAAS_SMTP_FROM` and existing SMTP credentials in Secrets. No provider migration is required. For Resend use `PF_SAAS_MAIL_MODE=resend`, a verified `PF_SAAS_MAIL_FROM` and `PF_SAAS_MAIL_API_KEY` in the secret manager/HF Secrets, never a public Variable. The HTTPS provider uses a fixed endpoint and bounded timeouts. Both transports use the encrypted durable outbox. Registration, password-reset and invitation tokens are committed atomically with their queued email. Delivery is asynchronous; HTTP acceptance and provider acceptance do not prove arrival in an inbox. Follow [email setup, retry and diagnosis](PUBLIC_EMAIL_DELIVERY.md), including real delivery and single-use-token checks on an isolated staging account. HF documents egress on 80, 443 and 8080. The checker accepts SMTP on these ports with exactly one TLS mode; provider support and real delivery still need testing. Ordinary SMTP on 465/587 is not a reliable HF configuration. Linux/Docker may retain encrypted SMTP on its permitted ports. A managed PostgreSQL endpoint on 5432 likewise needs demonstrated permitted connectivity; choose a supported endpoint/port or provider-approved network arrangement. An HTTPS database administration URL is not a PostgreSQL protocol endpoint. See [HF networking](https://huggingface.co/docs/hub/spaces-overview#networking). ## Admission is shared across organizations The existing organization limits remain. Add explicit account caps to prevent one person multiplying their allowance by creating or joining organizations. These conservative example values are a starting policy, not a capacity claim: ```text PF_SAAS_ACCOUNT_MAX_CONCURRENT_JOBS=1 PF_SAAS_ACCOUNT_MAX_QUEUED_JOBS=20 PF_SAAS_ACCOUNT_MONTHLY_COMPUTE_SECONDS=36000 PF_SAAS_ACCOUNT_MAX_STORAGE_BYTES=5368709120 PF_SAAS_ACCOUNT_MAX_OWNED_ORGANIZATIONS=1 PF_SAAS_ACCOUNT_MONTHLY_AI_MESSAGES=1000 PF_SAAS_ACCOUNT_AI_MAX_CONCURRENT_REQUESTS=1 PF_SAAS_ACCOUNT_AI_REQUESTS_PER_MINUTE=30 PF_SAAS_ACCOUNT_AI_TOKENS_PER_MINUTE=250000 PF_SAAS_ACCOUNT_AI_MONTHLY_COST_MICRO_USD=0 PF_SAAS_ACCOUNT_ASSISTANT_MAX_QUEUED_OPERATIONS=4 PF_SAAS_ACCOUNT_ASSISTANT_MAX_CONCURRENT_OPERATIONS=1 ``` The example leaves monetary AI accounting disabled. To enable a cost cap, set both `PF_SAAS_AI_INPUT_COST_PER_MILLION_MICRO_USD` and `PF_SAAS_AI_OUTPUT_COST_PER_MILLION_MICRO_USD` to reviewed positive conservative tariffs for the allowed providers/models, then set a positive account cost cap. Zero tariffs with a positive cost cap are rejected at startup. New account caps default to zero (disabled) to preserve existing private deployments. Public readiness requires an explicit bounded policy. Configure the same policy on every API, mail and worker process. A billable provider may charge independently of PF's counters; set provider budgets as well. The migration attributes historical attempts from their jobs and profiles only where retained operations establish a unique user. Unknown ownership is not guessed. Enabling account caps fails closed on unresolved charged usage; retain the source records and resolve attribution through an audited maintenance procedure before public admission. Creating a new organization does not erase the account's existing reservations or retained bytes. ## Schema and lifecycle upgrade The SaaS schema is **10**. Drain all API writers, assistant continuations and workers; confirm corresponding executions terminal, then create and verify a consistent backup. Native HF applies the additive schema migration while its single controller starts. Distributed Linux/Docker runs the explicit existing `python -m pf_saas init` migration command with the same service configuration before any new application replica serves. Managed HF uses guarded maintenance and verifies authority and charged totals before activation. Do not mix old and new application processes during migration. A rollback is a coordinated restore of the pre-upgrade database, objects and code, with all writers stopped. Restoring only an older code image is not a schema rollback. The additive migration preserves retained files, source accounting records, users, organization/project grants and established execution authority. The HF storage policy remains `conservative_lifetime` by default. Platform administration can switch to `safe_lifecycle` with a generation check and an audit reason. Future output reservations settle from verified publication; physical deletion and refund require recorded ownership, terminal execution, recovery safety and idempotent receipts. Legacy charges require a reviewed reconciliation plan and its unchanged digest. Deleting an arbitrary object or `pf-code.zip` is not a valid quota reconciliation method. Keep the currently approved ceiling. For the deployment already raised to 32 GiB this is `34359738368` in both the encrypted meter and `PF_HF_MAX_STORAGE_BYTES`. Do not set it back to 16 GiB based on an expectation of refunds. First measure the reconciled retained usage and required reserve. Safe accounting prevents avoidable lifetime charges; it cannot guarantee that real retained data and recovery history fit within 16 GiB. Managed HF additionally coordinates reclamation with external database backup retention. Its audited recovery floor must establish that no older PITR point or native recovery root can still reference deleted data. See [managed accounting](HUGGING_FACE_MANAGED_ACCOUNTING.md). An expired timer alone does not prove backup safety. Execution recovery runs at startup and periodically. It acquires the owner's fence and proves that the corresponding execution has stopped before marking an operation interrupted and releasing temporary reservations. Retained files remain. Docker child executions receive durable launch/terminal journals; ambiguous legacy executions remain held until evidence resolves them. Recovery is a general lifecycle mechanism, not a hard-coded repair for one operation ID. ## Qualify and expand gradually 1. Build the complete native/managed contexts from this release and verify the exact private code ZIP and image pins. Check that new mail, recovery, admission and accounting modules are included in the generated contexts. 2. Run an isolated staging migration with a protected copy of representative state. Test registration, verification, reset, invitation, private workflows, permissions, concurrent users, cancellation, recovery and accounting. 3. Run the offline [public readiness checker](PUBLIC_LAUNCH_READINESS.md) against configuration and retained ingress/email/isolation/restore evidence. 4. Run the [mixed-workload qualification campaign](../deploy/saas/distributed/CAPACITY_QUALIFICATION.md) on an explicitly marked isolated target. Include real active workflows, provider capacity, BYOK/BYOS, failures, recovery, backlog drainage and a soak. Collect database, API, object-store, worker and provider measurements. 5. Approve the observed envelope, configure finite admission and expand public access in stages. At saturation the service must bound/deny work with retry guidance while preserving existing reservations and private results. The load generator is sharded; the intended user count is separate from executing-job concurrency. For example, 200,000 people browsing or editing is a different compute requirement from 200,000 simultaneous scientific jobs. Both must be represented explicitly in the campaign contract and provisioned within provider limits. The release's local tests and previous isolated HF qualification records cannot substitute for that campaign.