Optional Hugging Face deployment with managed PostgreSQL

This is an explicit alternative to native HF, not a prerequisite for its performance corrections. The consolidated full r34 release contains both options. Native HF still uses its supervised local PostgreSQL process and encrypted HF Bucket recovery. Managed HF uses a dedicated remote PostgreSQL database for durable application state and encrypted HF Bucket objects for artifacts. HF Jobs, private code, BYOK/BYOS, private workflows and the scientific runner remain supported.

Choose this guide only when adopting the external database architecture. To redeploy the already working native installation, use R34_NATIVE_HF_RELEASE_UPGRADE.md. Do not install historical patch ZIPs on top of the consolidated release.

1. Choose the operation

fresh refuses a nonempty database or object prefix; it must never reset an existing deployment. migrate checks the sealed deployed native controller, restores its database, verifies rows and referenced encrypted objects, imports its quota reservations and audits, applies schema 10, and retires native database authority before admitting the managed application. It requires stopped or explicitly paused work, not just expired leases.

upgrade is a guarded additive schema-9 to schema-10 upgrade (and an idempotent schema-10 refresh). Pause every replica before maintenance. The gate verifies stopped work, preserves the ready authority marker and quota totals, adds the accounting/recovery/mail tables and backfills attributable account usage. It does not reimport native state or reset charges. Older or future schemas are rejected by this managed upgrade route. See public service deployment and managed accounting.

All three operations use two guarded commits: maintenance first, application second. The tools preserve Space identity, access tokens, Variables and Secrets. They do not create or bill a database, create HF resources, configure Secrets, reset quota, or execute paid scientific/provider tests. The operator explicitly configures the optional new database.

2. Database prerequisites

For fresh and migrate, provision a dedicated PostgreSQL database whose operator account can create and migrate the application schema. Use PostgreSQL 15 or a compatible newer version, and verify compatibility with the bundled PostgreSQL 15 client tools. PostgreSQL 15 documentation describes forward loading of pg_dump output into newer server versions; the maintenance gate therefore accepts 15 and newer rather than excluding 16/17. This is a compatibility expectation, not a tested provider certificate. The application must connect to a writable primary with TLS, fsync=on and synchronous_commit=on (or remote_apply). Use a direct endpoint or a session affine pool; transaction pooling is unsupported because scoped session advisory locks must survive transaction boundaries.

For upgrade, retain the existing database and verify its backup first.

Place the database near the HF controller. Confirm actual PostgreSQL protocol connectivity from HF before migration; ordinary port 5432 may be blocked by HF egress policy. Use a provider-supported permitted endpoint/port or approved network arrangement, retaining verify-full TLS. An HTTPS API URL is not a PostgreSQL endpoint. Configure durable backups/PITR and exercise a restore before relying on it. Record connection capacity, retention, regional latency and failover behavior. Startup probes test protocol and server settings, not provider disk durability or restoration success.

The database URL must contain an explicit username/password/database, remote host, sslmode=verify-full and connect_timeout=1..30, for example:

postgresql+psycopg://USER:URL_ENCODED_PASSWORD@DB_HOST:5432/PF_DATABASE?sslmode=verify-full&sslrootcert=/etc/ssl/certs/ca-certificates.crt&connect_timeout=10

Use the provider's correct trusted CA configuration. A private provider CA must be securely provisioned at its configured path in the container; never disable certificate or hostname checks. The packaged public system CA bundle is appropriate only when it validates that provider's certificate. No DSN or password belongs in source, a public Variable, shell history, or a shared log. The validator rejects host overrides, options, duplicate TLS values and unbounded connection timeouts.

3. Extract the release and build the contexts

Verify the full release's .sha256, extract both the .xz and .zip layers into a new project directory, and retain the archive and qualification records. Existing HF installations retain all original resources and keys.

Prepare the Python 3.12 HF tools environment and login as described in native guide section 2. The operator tools use huggingface_hub==1.32.0. The following commands are Windows CMD, one line at a time. Replace the project path and owner before running. An existing correctly configured tools environment can be reused.

set "PF_PROJECT_ROOT=D:\privileged-frame-simulation"
set "PF_HF_OWNER=YOUR_HF_USERNAME"
set "PF_HF_APP_REPO=%PF_HF_OWNER%/pf-simulation-saas"
set "PF_HF_CODE_REPO=%PF_HF_OWNER%/pf-simulation-runner"
set "PF_HF_RUNTIME_REPO=%PF_HF_OWNER%/pf-runtime-r34"
set "PF_HF_MANAGED_UPLOAD=%USERPROFILE%\pf-hf-managed-r34-generated"
set "PF_HF_MANAGED_STATE=%USERPROFILE%\pf-hf-managed-r34-deployment"
cd /d "%PF_PROJECT_ROOT%"
call "%USERPROFILE%\pf-hf-tools-r34\Scripts\activate.bat"
hf auth whoami
python deploy\saas\huggingface_managed\build_space.py --output "%PF_HF_MANAGED_UPLOAD%"

The output and state directories must be new. The builder is offline and does not read operator credentials. It produces:

Do not upload the whole release or parent generated directory. Keep the controller Private during maintenance. After public readiness, Protected visibility may expose its application while retaining private source. Never make the scientific code repository Public. The managed contexts use the same scientific code digest as their private jobs/ payload.

For a new installation, create the private controller/code Spaces, private Bucket, scoped HF token and S3 credential pair using native guide section 1. Create a separate public dependency-only runtime Space if one is needed. For migration, keep the existing resources and keys. Build/upload the runtime and jobs contexts and obtain the exact immutable pins using native guide sections 4-6 with this managed-generated output directory in place of the native one. Do not run the native controller upload or PF_HF_BOOTSTRAP=true steps.

The key upload and identity checks are:

hf upload "%PF_HF_RUNTIME_REPO%" "%PF_HF_MANAGED_UPLOAD%\runtime" . --repo-type space
hf upload "%PF_HF_CODE_REPO%" "%PF_HF_MANAGED_UPLOAD%\jobs" . --repo-type space
hf spaces info "%PF_HF_CODE_REPO%" --expand sha
python -c "import json,sys; print(json.load(open(sys.argv[1], encoding='utf-8'))['sha256'])" "%PF_HF_MANAGED_UPLOAD%\jobs\code-manifest.json"

Use the native guide's registry helper to resolve the runtime image digest and run its confinement qualification. Record the 40-character code commit, 64-character code archive SHA256, and OCI image SHA256 digest. Updating code only is insufficient: the controller and private Jobs pins must agree. For migration, complete the drain and pause at the beginning of section 4 before changing any controller code pin or database setting; automatic restart of old controller source against new pins would fail its identity gate.

4. Preserve or configure the Space settings

Migration or managed upgrade: first drain scientific and AI/provider work, pause autonomous investigations at safe checkpoints, finish/reconcile pending uploads and BYOS operations, and pause the controller and any other application replicas. Keep a provider-independent recovery copy of native encrypted state and preserve every original decryption/database key. Do not treat an ambiguous provider timeout as proof that the job stopped. The maintenance program independently refuses unresolved work. For the native single-controller deployment, stop user admission before beginning this maintenance window.

All modes: in the controller Space's Settings, configure the required Secrets and Variables. Setting updates can trigger rebuilding/restarting; keep all application instances paused while preparing cutover. No managed profile Variable is required: the generated managed Dockerfile and launcher explicitly select it. Never change a currently serving native controller's deployment profile manually.

Retain positive existing limits rather than replacing working values with these examples. Remove PF_SAAS_HF_GATEWAY_SECRET: the managed/native direct controller is distinct from the legacy Linux-backend gateway. Do not reuse the HF state key for session, assistant or BYOS credential encryption.

For a fresh managed installation only, set PF_HF_ADMIN_EMAIL and PF_HF_ADMIN_PASSWORD Secrets to initialize the first operator after activation; remove the password Secret after verifying login. Never use these to reset or promote an existing account. Complete the native guide's full email, authentication, BYOK/BYOS and deployment-budget configuration rather than treating the shortened table above as a complete fresh-service credential setup.

Managed HF can retain SMTP2GO encrypted SMTP on port 443 with PF_SAAS_SMTP_SSL=true and PF_SAAS_SMTP_STARTTLS=false, its verified sender and existing credentials. Resend HTTPS is optional. Both transports require real account-email acceptance evidence before public launch; see public email delivery. The checker binds sender, SMTP endpoint and TLS policy into that evidence's configuration fingerprint.

Set connection capacity before preparing. Main-pool capacity must be at least PF_SAAS_WORKER_SLOTS + 4; default main pool 5 plus overflow 5 supports one worker. Include main, lease and assistant pools across all API and worker processes/replicas plus the administrator reserve in the provider's connection budget. The default one-process configuration estimates 23 total connections including ten reserved administration connections. Configured process counts must describe the actual maximum fleet. They do not create replicas or increase the database provider's cap. See the shared settings reference linked in R34_CONSOLIDATED_DEPLOYMENT.md.

5. Prepare and install maintenance

Choose one preparation command. All modes verify the newly uploaded scientific archive and exact remote controller bytes. prepare is read-only and saves a local source snapshot; the state directory contains private code and must remain private.

Fresh installation:

python deploy\saas\huggingface_managed\deploy_release.py prepare --bundle "%PF_HF_MANAGED_UPLOAD%" --space "%PF_HF_APP_REPO%" --mode fresh --state-dir "%PF_HF_MANAGED_STATE%"

Existing native installation to managed PostgreSQL:

python deploy\saas\huggingface_managed\deploy_release.py prepare --bundle "%PF_HF_MANAGED_UPLOAD%" --space "%PF_HF_APP_REPO%" --mode migrate --state-dir "%PF_HF_MANAGED_STATE%"

Already managed schema-9 or schema-10 installation:

python deploy\saas\huggingface_managed\deploy_release.py prepare --bundle "%PF_HF_MANAGED_UPLOAD%" --space "%PF_HF_APP_REPO%" --mode upgrade --state-dir "%PF_HF_MANAGED_STATE%"

Verify prepared_no_remote_changes, repository, mode and source revision. Then:

python deploy\saas\huggingface_managed\deploy_release.py maintenance --state-dir "%PF_HF_MANAGED_STATE%"
hf spaces wait "%PF_HF_APP_REPO%" --timeout 25m
python deploy\saas\huggingface_managed\deploy_release.py status --state-dir "%PF_HF_MANAGED_STATE%"

Maintenance pauses the Space before an expected-parent commit and requests a restart. RUNNING alone only means the maintenance server is reachable; it is not application readiness. Status must verify the runtime revision and an authenticated receipt with managed_fresh_ready, managed_migration_ready, or managed_upgrade_ready, the same deployment ID/source revision, maintenance_only=true and application_serving=false. A migration receipt additionally confirms native authority retirement. The upgrade guard requires existing_authority_preserved=true, schema_version=10, quota_charges_preserved=true and migration_scope=additive_schema_and_account_attribution. The upgrade operation emits data_changed=true because it can apply additive schema and attribution work; this is not a read-only gate, and that flag is not a row-by-row change count. Do not require data_changed=false for a schema-10 refresh. Do not change settings between preparation and activation.

The authenticated receipt request rejects redirects and uses verified HTTPS. For a managed Windows/system CA, install truststore in the tools environment and place --system-ca before each subcommand, for example:

python deploy\saas\huggingface_managed\deploy_release.py --system-ca status --state-dir "%PF_HF_MANAGED_STATE%"

The flag applies system certificate trust to Hub and receipt requests while retaining certificate/hostname verification. An enterprise proxy/403/TLS failure still blocks activation; fix approved network/CA access and rerun status. Do not bypass receipt verification or disable TLS.

6. Activate and qualify

Only after successful maintenance:

python deploy\saas\huggingface_managed\deploy_release.py activate --state-dir "%PF_HF_MANAGED_STATE%"
hf spaces wait "%PF_HF_APP_REPO%" --timeout 25m
python deploy\saas\huggingface_managed\deploy_release.py status --state-dir "%PF_HF_MANAGED_STATE%"
hf spaces logs "%PF_HF_APP_REPO%" --tail 200

Activation independently rechecks the maintenance receipt before switching the Docker entry point. Expect active_code_verified, the active commit as runtime_sha, and running_revision_verified=true. Then open the PF origin and verify authenticated login, organization/project archive and restore, private workflows, BYOK/BYOS access, a controlled scientific run, authenticated ZIP retrieval, persistence through a restart, and the provider database backup/restore procedure. Record the real requests' status, timing and results. Offline tests or an HF stage do not establish user-visible latency.

Managed serving checks the ready marker, schema 10, writable TLS PostgreSQL and imported/fresh quota. It never falls back to an empty local database. Startup reports phase and exception class without raw SQL, DSNs or private data. PF_HF_MANAGED_WORKERS_ENABLED=false makes an instance API-only; planning multiple replicas also requires accurate shared connection/provider admission budgets.

7. Failure, resume and upgrade boundaries

Keep the generated contexts and prepared state directory. If a commit call fails without a definite outcome, do not repeat it blindly:

python deploy\saas\huggingface_managed\deploy_release.py reconcile --state-dir "%PF_HF_MANAGED_STATE%"

The tool accepts only its exact expected child commit and bytes. A reconciled commit does not automatically restart; use hf spaces restart and repeat status. If no commit occurred, the Space remains paused and that stage can be retried. Do not delete prepared state to bypass a disagreement with remote history.

For a failed maintenance container, inspect its PF_MANAGED_MIGRATION phase, correct the external cause, and restart the same maintenance revision. Fresh initialization resumes only its exact marker/identity; it cannot adopt native state. Native migration resumes only an exact verified copy or staged target. A partial foreign target is never overwritten. Do not manually zero quota, clear ambiguous jobs, delete the native head, or promote a partly imported target.

Once native migration maintenance may have started, a code-only native rollback is not supported. The encrypted schema-2 retirement tombstone intentionally prevents old native binaries from becoming a second authority. Keep both stores, keys and receipts. Repair forward, or use an independently reviewed reverse data migration with both sides stopped and verified. The original native quota reservations, retry identities and limit history remain charged after migration; deleting artifact files does not refund conservative lifetime accounting.

For an already managed deployment, use --mode upgrade with new generated contexts and a new state directory. Back up the authoritative database, drain work and pause every application/worker replica before changing code pins. The guarded maintenance step must report managed_upgrade_ready. It performs the additive schema-9 to schema-10 migration, or verifies an already schema-10 deployment without reimporting native state or resetting charges. It preserves the authority marker and charged totals. Schema 8, future schema versions, a mismatched marker, changed quota policy or unresolved work are blocked. Do not switch to fresh or migrate to bypass that gate. Schemas other than 9 or 10 require a separately reviewed and qualified forward migration with database recovery available. Native full-release upgrades remain independently supported by the native guide.

Qualification status

This release includes local managed-runtime, migration-state, deployment-guard, shared-state, deadline and schema regression tests, including mocked Hub operations and SQLite semantic fixtures. The retained isolated HF Job evidence under validation_r34_public_service/isolated_postgres/ records 144 passing checks with no skips, including real PostgreSQL migration, accounting, admission, execution recovery, physical restore and a scientific worker regression. That Job used temporary test data and loopback test services; external mail, storage and AI-provider behavior remained fixtures. Ordinary local PostgreSQL-only tests still require an explicit test database and skip when it is absent.

No production managed-database cutover, live account-email delivery, production provider workload, regional failover or large-fleet load test is certified by this evidence. The migration runs actual bounded TLS/session-affinity/transaction probes on the operator's chosen target before moving authority; those probes do not replace that target's backup/restore or load qualification. Keep this optional mode separate from the user-confirmed native HF responsiveness improvement.