# Optional Hugging Face deployment with managed PostgreSQL

This is an explicit alternative to [native HF](HUGGING_FACE_NATIVE_DEPLOYMENT.md),
not a prerequisite for its performance corrections. The consolidated full r34
release contains both options. Native HF still uses its supervised local
PostgreSQL process and encrypted HF Bucket recovery. Managed HF uses a dedicated
remote PostgreSQL database for durable application state and encrypted HF Bucket
objects for artifacts. HF Jobs, private code, BYOK/BYOS, private workflows and the
scientific runner remain supported.

Choose this guide only when adopting the external database architecture. To
redeploy the already working native installation, use
[R34_NATIVE_HF_RELEASE_UPGRADE.md](R34_NATIVE_HF_RELEASE_UPGRADE.md).
Do not install historical patch ZIPs on top of the consolidated release.

## 1. Choose the operation

| Operation | Controller repository | Database and object state | Tool mode |
| --- | --- | --- | --- |
| New managed installation | New private Docker Space, or unused private blank Space | Empty dedicated PostgreSQL public schema and empty HF object prefix | `fresh` |
| Existing native HF to managed | Existing private native controller Space | Existing native Bucket/key/prefix, empty dedicated target PostgreSQL public schema | `migrate` |
| Existing managed HF code upgrade | Existing private managed controller Space | Existing authoritative schema-9 or schema-10 PostgreSQL, same Bucket/key/prefix | `upgrade` |

`fresh` refuses a nonempty database or object prefix; it must never reset an
existing deployment. `migrate` checks the sealed deployed native controller,
restores its database, verifies rows and referenced encrypted objects, imports
its quota reservations and audits, applies schema 10, and retires native database
authority before admitting the managed application. It requires stopped or
explicitly paused work, not just expired leases.

`upgrade` is a guarded additive schema-9 to schema-10 upgrade (and an idempotent
schema-10 refresh). Pause every replica before maintenance. The gate verifies
stopped work, preserves the ready authority marker and quota totals, adds the
accounting/recovery/mail tables and backfills attributable account usage. It
does not reimport native state or reset charges. Older or future schemas are
rejected by this managed upgrade route. See
[public service deployment](R34_PUBLIC_SERVICE_DEPLOYMENT.md) and
[managed accounting](HUGGING_FACE_MANAGED_ACCOUNTING.md).

All three operations use two guarded commits: maintenance first, application second.
The tools preserve Space identity, access tokens, Variables and Secrets. They do
not create or bill a database, create HF resources, configure Secrets, reset
quota, or execute paid scientific/provider tests. The operator explicitly
configures the optional new database.

## 2. Database prerequisites

For `fresh` and `migrate`, provision a dedicated PostgreSQL database whose operator account can create and
migrate the application schema. Use PostgreSQL 15 or a compatible newer version,
and verify compatibility with the bundled PostgreSQL 15 client tools.
[PostgreSQL 15 documentation](https://www.postgresql.org/docs/15/app-pgdump.html)
describes forward loading of `pg_dump` output into newer server versions; the
maintenance gate therefore accepts 15 and newer rather than excluding 16/17.
This is a compatibility expectation, not a tested provider certificate. The
application must connect to a writable primary with TLS, `fsync=on` and
`synchronous_commit=on` (or `remote_apply`). Use a direct endpoint or a **session
affine** pool; transaction pooling is unsupported because scoped session advisory
locks must survive transaction boundaries.

For `upgrade`, retain the existing database and verify its backup first.

Place the database near the HF controller. Confirm actual PostgreSQL protocol
connectivity from HF before migration; ordinary port 5432 may be blocked by
HF egress policy. Use a provider-supported permitted endpoint/port or approved
network arrangement, retaining verify-full TLS. An HTTPS API URL is not a
PostgreSQL endpoint. Configure durable backups/PITR and
exercise a restore before relying on it. Record connection capacity, retention,
regional latency and failover behavior. Startup probes test protocol and server
settings, not provider disk durability or restoration success.

The database URL must contain an explicit username/password/database, remote
host, `sslmode=verify-full` and `connect_timeout=1..30`, for example:

```text
postgresql+psycopg://USER:URL_ENCODED_PASSWORD@DB_HOST:5432/PF_DATABASE?sslmode=verify-full&sslrootcert=/etc/ssl/certs/ca-certificates.crt&connect_timeout=10
```

Use the provider's correct trusted CA configuration. A private provider CA must
be securely provisioned at its configured path in the container; never disable
certificate or hostname checks. The packaged public system CA bundle is
appropriate only when it validates that provider's certificate. No DSN or
password belongs in source, a public Variable, shell history, or a shared log.
The validator rejects host overrides, `options`, duplicate TLS values and
unbounded connection timeouts.

## 3. Extract the release and build the contexts

Verify the full release's `.sha256`, extract both the `.xz` and `.zip` layers
into a **new** project directory, and retain the archive and qualification
records. Existing HF installations retain all original resources and keys.

Prepare the Python 3.12 HF tools environment and login as described in
[native guide section 2](HUGGING_FACE_NATIVE_DEPLOYMENT.md#2-prepare-the-full-release-and-windows-tools).
The operator tools use `huggingface_hub==1.32.0`. The following commands are
**Windows CMD**, one line at a time. Replace the project path and owner before
running. An existing correctly configured tools environment can be reused.

```bat
set "PF_PROJECT_ROOT=D:\privileged-frame-simulation"
set "PF_HF_OWNER=YOUR_HF_USERNAME"
set "PF_HF_APP_REPO=%PF_HF_OWNER%/pf-simulation-saas"
set "PF_HF_CODE_REPO=%PF_HF_OWNER%/pf-simulation-runner"
set "PF_HF_RUNTIME_REPO=%PF_HF_OWNER%/pf-runtime-r34"
set "PF_HF_MANAGED_UPLOAD=%USERPROFILE%\pf-hf-managed-r34-generated"
set "PF_HF_MANAGED_STATE=%USERPROFILE%\pf-hf-managed-r34-deployment"
cd /d "%PF_PROJECT_ROOT%"
call "%USERPROFILE%\pf-hf-tools-r34\Scripts\activate.bat"
hf auth whoami
python deploy\saas\huggingface_managed\build_space.py --output "%PF_HF_MANAGED_UPLOAD%"
```

The output and state directories must be new. The builder is offline and does
not read operator credentials. It produces:

| Context | Purpose |
| --- | --- |
| `runtime/` | Public-safe third-party dependencies and generic verifier only |
| `jobs/` | Private scientific code archive and immutable identity |
| `maintenance-fresh/` | Empty-target initialization, no application or workers |
| `maintenance-migrate/` | Native restoration/import/retirement, no application or workers |
| `maintenance-upgrade/` | Existing managed schema-9/10 additive upgrade and authority/quota verification, no application or workers |
| `space/` | Managed application, admitted only after the database marker is ready |

Do not upload the whole release or parent generated directory. Keep the controller Private during maintenance. After public readiness,
Protected visibility may expose its application while retaining private source.
Never make the scientific code repository Public. The managed contexts use
the same scientific code digest as their private `jobs/` payload.

For a new installation, create the private controller/code Spaces, private
Bucket, scoped HF token and S3 credential pair using native guide section 1.
Create a separate public dependency-only runtime Space if one is needed.
For migration, keep the existing resources and keys. Build/upload the runtime
and jobs contexts and obtain the exact immutable pins using
[native guide sections 4-6](HUGGING_FACE_NATIVE_DEPLOYMENT.md#4-build-and-upload-the-runtime-and-private-code)
with **this managed-generated output directory** in place of the native one.
Do not run the native controller upload or `PF_HF_BOOTSTRAP=true` steps.

The key upload and identity checks are:

```bat
hf upload "%PF_HF_RUNTIME_REPO%" "%PF_HF_MANAGED_UPLOAD%\runtime" . --repo-type space
hf upload "%PF_HF_CODE_REPO%" "%PF_HF_MANAGED_UPLOAD%\jobs" . --repo-type space
hf spaces info "%PF_HF_CODE_REPO%" --expand sha
python -c "import json,sys; print(json.load(open(sys.argv[1], encoding='utf-8'))['sha256'])" "%PF_HF_MANAGED_UPLOAD%\jobs\code-manifest.json"
```

Use the native guide's registry helper to resolve the runtime image digest and
run its confinement qualification. Record the **40-character code commit**,
**64-character code archive SHA256**, and **OCI image SHA256 digest**. Updating
code only is insufficient: the controller and private Jobs pins must agree.
For migration, complete the drain and pause at the beginning of section 4
**before changing any controller code pin or database setting**; automatic
restart of old controller source against new pins would fail its identity gate.

## 4. Preserve or configure the Space settings

**Migration or managed upgrade:** first drain scientific and AI/provider work, pause autonomous
investigations at safe checkpoints, finish/reconcile pending uploads and BYOS
operations, and pause the controller and any other application replicas. Keep a
provider-independent recovery copy of native encrypted state and preserve every
original decryption/database key. Do not treat an ambiguous provider timeout as
proof that the job stopped. The maintenance program independently refuses
unresolved work. For the native single-controller deployment, stop user admission
before beginning this maintenance window.

**All modes:** in the controller Space's **Settings**, configure the required
Secrets and Variables. Setting updates can trigger rebuilding/restarting; keep
all application instances paused while preparing cutover. No managed profile
Variable is required: the generated managed Dockerfile and launcher explicitly
select it. Never change a currently serving native controller's deployment
profile manually.

| Setting | Where | Action |
| --- | --- | --- |
| `PF_SAAS_DATABASE_URL` | Secret | Verified remote PostgreSQL DSN; preserve the existing DSN for managed upgrades |
| `PF_HF_TOKEN`, `PF_HF_S3_ACCESS_KEY_ID`, `PF_HF_S3_SECRET_ACCESS_KEY` | Secrets | Preserve existing scoped credentials, or create for fresh installation |
| `PF_HF_STATE_KEY` | Secret | Preserve exactly on migration; independent 32-byte state key for fresh installation |
| `PF_SAAS_SECRET_KEY` | Secret | Preserve session-signing key; create independent key only for a new installation |
| `PF_SAAS_ASSISTANT_CREDENTIALS_KEY`, `PF_SAAS_BYOS_CREDENTIALS_KEY` | Secrets | Preserve existing encrypted BYOK/BYOS keys and enabled-provider configuration |
| `PF_HF_PG_ADMIN_PASSWORD`, `PF_HF_PG_APP_PASSWORD`, `PF_HF_PG_WAL_PASSWORD` | Secrets | Preserve original native credentials during migration; managed serving itself does not launch local PostgreSQL |
| `PF_HF_NAMESPACE`, `PF_HF_BUCKET`, `PF_HF_PREFIX` | Variables | Preserve exact native identity on migration; select an empty prefix for fresh mode |
| `PF_HF_CODE_REPO`, `PF_HF_CODE_REVISION`, `PF_HF_CODE_SHA256` | Variables | New private Jobs repository pins from section 3 |
| `PF_HF_JOB_IMAGE`, optional `PF_HF_ASSISTANT_IMAGE` | Variables | Qualified immutable public dependency image digest, with no private PF source |
| `PF_HF_MAX_STORAGE_BYTES` | Variable | Preserve the actual approved source meter limit; the existing service raised to 32 GiB uses `34359738368`. Historical 16 GiB installations use `17179869184` only while that remains their approved ceiling. |
| `PF_HF_WAL_RESERVE_BYTES` | Variable | Preserve source reservation policy; default 268435456 when applicable |
| `PF_HF_BOOTSTRAP` | Variable | Absent or `false` for every managed mode |
| `PF_SAAS_PUBLIC_URL` and email/authentication settings | Variables/Secrets as in native guide | Preserve configured origin and functioning authentication/email |
| `PF_SAAS_DEPLOYMENT_BUDGET_ENABLED` | Variable | `true`; retain configured compute/cost/queue ceilings |
| `PF_SAAS_USER_MAX_CONCURRENT_JOBS` | Variable | Explicit positive value; `1` is a conservative initial setting |
| `PF_SAAS_USER_MAX_QUEUED_JOBS` | Variable | Explicit positive value; `20` is a conservative initial setting |
| `PF_SAAS_AI_ADMISSION_ENABLED` | Variable | `true` |
| `PF_SAAS_API_USER_REQUESTS_PER_MINUTE` | Variable | Positive value; default `600` |
| `PF_SAAS_SCALING_AUTO_SCALE_IN` | Variable | `false`; the AWS scale-in controller is not an HF control plane |

Retain positive existing limits rather than replacing working values with these
examples. Remove `PF_SAAS_HF_GATEWAY_SECRET`: the managed/native direct controller
is distinct from the legacy Linux-backend gateway. Do not reuse the HF state key
for session, assistant or BYOS credential encryption.

For a fresh managed installation only, set `PF_HF_ADMIN_EMAIL` and
`PF_HF_ADMIN_PASSWORD` Secrets to initialize the first operator after activation;
remove the password Secret after verifying login. Never use these to reset or
promote an existing account. Complete the native guide's full email,
authentication, BYOK/BYOS and deployment-budget configuration rather than treating
the shortened table above as a complete fresh-service credential setup.

Managed HF can retain SMTP2GO encrypted SMTP on port 443 with
`PF_SAAS_SMTP_SSL=true` and `PF_SAAS_SMTP_STARTTLS=false`, its verified sender
and existing credentials. Resend HTTPS is optional. Both transports require
real account-email acceptance evidence before public launch; see
[public email delivery](PUBLIC_EMAIL_DELIVERY.md). The checker binds sender,
SMTP endpoint and TLS policy into that evidence's configuration fingerprint.

Set connection capacity before preparing. Main-pool capacity must be at least
`PF_SAAS_WORKER_SLOTS + 4`; default main pool 5 plus overflow 5 supports one
worker. Include main, lease and assistant pools across **all** API and worker
processes/replicas plus the administrator reserve in the provider's connection
budget. The default one-process configuration estimates 23 total connections
including ten reserved administration connections. Configured process counts
must describe the actual maximum fleet. They do not create replicas or increase
the database provider's cap. See the shared settings reference linked in
[R34_CONSOLIDATED_DEPLOYMENT.md](R34_CONSOLIDATED_DEPLOYMENT.md).

## 5. Prepare and install maintenance

Choose **one** preparation command. All modes verify the newly uploaded scientific
archive and exact remote controller bytes. `prepare` is read-only and saves a
local source snapshot; the state directory contains private code and must remain
private.

Fresh installation:

```bat
python deploy\saas\huggingface_managed\deploy_release.py prepare --bundle "%PF_HF_MANAGED_UPLOAD%" --space "%PF_HF_APP_REPO%" --mode fresh --state-dir "%PF_HF_MANAGED_STATE%"
```

Existing native installation to managed PostgreSQL:

```bat
python deploy\saas\huggingface_managed\deploy_release.py prepare --bundle "%PF_HF_MANAGED_UPLOAD%" --space "%PF_HF_APP_REPO%" --mode migrate --state-dir "%PF_HF_MANAGED_STATE%"
```

Already managed schema-9 or schema-10 installation:

```bat
python deploy\saas\huggingface_managed\deploy_release.py prepare --bundle "%PF_HF_MANAGED_UPLOAD%" --space "%PF_HF_APP_REPO%" --mode upgrade --state-dir "%PF_HF_MANAGED_STATE%"
```

Verify `prepared_no_remote_changes`, repository, mode and source revision. Then:

```bat
python deploy\saas\huggingface_managed\deploy_release.py maintenance --state-dir "%PF_HF_MANAGED_STATE%"
hf spaces wait "%PF_HF_APP_REPO%" --timeout 25m
python deploy\saas\huggingface_managed\deploy_release.py status --state-dir "%PF_HF_MANAGED_STATE%"
```

Maintenance pauses the Space before an expected-parent commit and requests a
restart. `RUNNING` alone only means the maintenance server is reachable; it is
not application readiness. Status must verify the runtime revision **and** an
authenticated receipt with `managed_fresh_ready`, `managed_migration_ready`, or `managed_upgrade_ready`,
the same deployment ID/source revision, `maintenance_only=true` and
`application_serving=false`. A migration receipt additionally confirms native
authority retirement. The upgrade guard requires `existing_authority_preserved=true`,
`schema_version=10`, `quota_charges_preserved=true` and
`migration_scope=additive_schema_and_account_attribution`. The upgrade operation
emits `data_changed=true` because it can apply additive schema and attribution
work; this is not a read-only gate, and that flag is not a row-by-row change
count. Do not require `data_changed=false` for a schema-10 refresh. Do not change
settings between preparation and activation.

The authenticated receipt request rejects redirects and uses verified HTTPS.
For a managed Windows/system CA, install `truststore` in the tools environment
and place `--system-ca` before each subcommand, for example:

```bat
python deploy\saas\huggingface_managed\deploy_release.py --system-ca status --state-dir "%PF_HF_MANAGED_STATE%"
```

The flag applies system certificate trust to Hub and receipt requests while
retaining certificate/hostname verification. An enterprise proxy/403/TLS failure
still blocks activation; fix approved network/CA access and rerun status. Do not bypass receipt verification or disable TLS.

## 6. Activate and qualify

Only after successful maintenance:

```bat
python deploy\saas\huggingface_managed\deploy_release.py activate --state-dir "%PF_HF_MANAGED_STATE%"
hf spaces wait "%PF_HF_APP_REPO%" --timeout 25m
python deploy\saas\huggingface_managed\deploy_release.py status --state-dir "%PF_HF_MANAGED_STATE%"
hf spaces logs "%PF_HF_APP_REPO%" --tail 200
```

Activation independently rechecks the maintenance receipt before switching the
Docker entry point. Expect `active_code_verified`, the active commit as
`runtime_sha`, and `running_revision_verified=true`. Then open the PF origin and
verify authenticated login, organization/project archive and restore, private
workflows, BYOK/BYOS access, a controlled scientific run, authenticated ZIP
retrieval, persistence through a restart, and the provider database backup/restore
procedure. Record the real requests' status, timing and results. Offline tests
or an HF stage do not establish user-visible latency.

Managed serving checks the ready marker, schema 10, writable TLS PostgreSQL and
imported/fresh quota. It never falls back to an empty local database. Startup
reports phase and exception class without raw SQL, DSNs or private data.
`PF_HF_MANAGED_WORKERS_ENABLED=false` makes an instance API-only; planning multiple
replicas also requires accurate shared connection/provider admission budgets.

## 7. Failure, resume and upgrade boundaries

Keep the generated contexts and prepared state directory. If a commit call fails
without a definite outcome, do not repeat it blindly:

```bat
python deploy\saas\huggingface_managed\deploy_release.py reconcile --state-dir "%PF_HF_MANAGED_STATE%"
```

The tool accepts only its exact expected child commit and bytes. A reconciled
commit does not automatically restart; use `hf spaces restart` and repeat status.
If no commit occurred, the Space remains paused and that stage can be retried.
Do not delete prepared state to bypass a disagreement with remote history.

For a failed maintenance container, inspect its `PF_MANAGED_MIGRATION` phase,
correct the external cause, and restart **the same maintenance revision**. Fresh
initialization resumes only its exact marker/identity; it cannot adopt native
state. Native migration resumes only an exact verified copy or staged target.
A partial foreign target is never overwritten. Do not manually zero quota,
clear ambiguous jobs, delete the native head, or promote a partly imported target.

Once native migration maintenance may have started, **a code-only native rollback
is not supported**. The encrypted schema-2 retirement tombstone intentionally
prevents old native binaries from becoming a second authority. Keep both stores,
keys and receipts. Repair forward, or use an independently reviewed reverse data
migration with both sides stopped and verified. The original native quota
reservations, retry identities and limit history remain charged after migration;
deleting artifact files does not refund conservative lifetime accounting.

For an already managed deployment, use `--mode upgrade` with new generated
contexts and a new state directory. Back up the authoritative database, drain
work and pause **every** application/worker replica before changing code pins.
The guarded maintenance step must report `managed_upgrade_ready`. It performs
the additive schema-9 to schema-10 migration, or verifies an already schema-10
deployment without reimporting native state or resetting charges. It preserves
the authority marker and charged totals. Schema 8, future schema versions, a
mismatched marker, changed quota policy or unresolved work are blocked. Do not
switch to `fresh` or `migrate` to bypass that gate. Schemas other than 9 or 10
require a separately reviewed and qualified forward migration with database
recovery available. Native full-release upgrades remain independently supported
by the native guide.

## Qualification status

This release includes local managed-runtime, migration-state, deployment-guard,
shared-state, deadline and schema regression tests, including mocked Hub
operations and SQLite semantic fixtures. The retained isolated HF Job evidence
under `validation_r34_public_service/isolated_postgres/` records 144 passing
checks with no skips, including real PostgreSQL migration, accounting, admission,
execution recovery, physical restore and a scientific worker regression. That
Job used temporary test data and loopback test services; external mail, storage
and AI-provider behavior remained fixtures. Ordinary local PostgreSQL-only tests
still require an explicit test database and skip when it is absent.

No production managed-database cutover, live account-email delivery, production
provider workload, regional failover or large-fleet load test is certified by
this evidence. The migration runs actual bounded TLS/session-affinity/transaction
probes on the operator's chosen target before moving authority; those probes do
not replace that target's backup/restore or load qualification. Keep this optional
mode separate from the user-confirmed native HF responsiveness improvement.