# PF Simulation r34: complete self-hosted research reference

For current r34 public-service and schema-10 instructions, follow the [public-service guide](public-service.md) and [deployment choices](deployment-choices.md). This page covers the Linux/Docker single-host option. Workflow and conversation deletion receipts, database state, private storage and encryption keys must be recovered consistently. Do not roll application code back against a migrated database; read the [lifecycle guide](r34-lifecycle.md) and [execution recovery](execution-recovery.md).

This reference provisions one dedicated Linux host for the API, durable worker,
PostgreSQL metadata, managed storage, private assistant state, BYOK vault, BYOS
journals and connector state, and isolated numerical execution. Caddy terminates
HTTPS. This is the reference for the **full r34 research feature set**; the earlier
AWS Fargate template remains available for its normal-run topology. Fargate does
not provide the local Docker daemon or durable assistant files required by this
reference. Do not enable the full assistant there by copying environment flags.

This is the single-host option in the [consolidated deployment guide](../../../docs/R34_CONSOLIDATED_DEPLOYMENT.md). It retains local PostgreSQL; the optional managed HF architecture is independent. Hosted database schema is **10** in the current public-service release.

## Trust and capability boundary

Use a dedicated Ubuntu 24.04/systemd host, Python 3.12, Docker Engine with the
Compose plugin and rootless support, cgroup v2 with systemd resource delegation,
and Caddy. The API and worker are native services under a dedicated unprivileged
account. Docker's daemon also runs under that account; the account must **not**
belong to the rootful `docker` group. API and worker can access that account's
state and rootless daemon; compromise of either exposes that account's data.
This topology does not claim isolation between those two trusted services.

Scientific and assistant workloads receive only their run files and an explicit
safe environment. They receive no database, BYOK/BYOS vault keys or Docker socket.
Assistant containers have no network, a read-only source/root filesystem,
capabilities removed, no privilege escalation, and CPU/memory/PID/time/output
limits. The normal worker defaults to network `none` too. Explicit live downloads
require an operator-reviewed outbound-network design; use supplied offline input
files in this default deployment. Provider calls and BYOS operations occur in the
trusted API, subject to project authorization and institutional policy.

The rootless executor uses container UID 0, which maps to the **non-root host
service account**, to access its private bind-mounted run folders. The runtime
checks the local per-user Docker socket and daemon rootless identity before this
mapping. It refuses to apply the mapping to rootful or remote daemons. Existing
rootful deployments retain the caller UID/GID mapping. This is namespace root,
not host root; deployments still require target-host isolation acceptance.

The default image includes the portable numerical research dependencies. MATLAB,
Swabian hardware/SDK and Einstein Toolkit installations remain explicit external
prerequisites. Configure their reviewed images using the existing research-profile
mechanism only after installing and qualifying those tools. This reference does
not pretend to provision commercial licences or physical instruments.

## Host preparation

Host-administrator steps, once (substitute the institution's account name):

```sh
sudo useradd --create-home --shell /bin/bash pfsim
sudo loginctl enable-linger pfsim
sudo apt-get update
sudo apt-get install python3.12-venv uidmap dbus-user-session slirp4netns
```

Install Docker Engine, `docker-ce-rootless-extras` and `docker-compose-plugin`
using the [official Ubuntu repository instructions](https://docs.docker.com/engine/install/ubuntu/).
Install Caddy using its [official service installation instructions](https://caddyserver.com/docs/install#debian-ubuntu-raspbian).
Use a proper login session as `pfsim` (SSH or `machinectl shell pfsim@`) so its user
systemd manager and `/run/user/UID` exist. Then:

```sh
dockerd-rootless-setuptool.sh install
systemctl --user enable --now docker.service
```

See Docker's [rootless prerequisites](https://docs.docker.com/engine/security/rootless/)
and [resource-limit guidance](https://docs.docker.com/engine/security/rootless/tips/).
The preflight rejects an absent rootless daemon, missing Compose, an unsupported
cgroup driver and unavailable resource controls. No rootful fallback is enabled.
DNS must point to the host. Permit inbound 80/443 to Caddy and an institutionally
restricted administrative login port. Do not expose ports 8000 or 15432.

## Configure and provision

Unpack the release to a versioned, administrator-reviewed directory such as
`/srv/pf-releases/r34-consolidated`. Keep it separate from private state.
Run every command below as `pfsim`, never as root. Paths must contain only letters,
digits, slash, dot, hyphen and underscore. The examples use a new installation at
`/home/pfsim/pf-host`; generation refuses to overwrite existing state/secrets.

```sh
python3.12 /srv/pf-releases/r34-consolidated/deploy/saas/reference/host.py \
  --installation /home/pfsim/pf-host configure \
  --source /srv/pf-releases/r34-consolidated \
  --domain research.example.edu \
  --smtp-host smtp.example.edu --smtp-from research@example.edu
```

The generated `config/service.env` uses JSON-quoted, single-line values with literal UTF-8. Do not use JSON `\u` escapes or control characters; this keeps CLI and systemd values identical. Edit
SMTP username/password if the institution's TLS SMTP relay requires them. The generated configuration declares one API and one worker process and bounded main, assistant and lease pools; review [shared capacity settings](../../../docs/R34_SHARED_CAPACITY_SETTINGS.md) before changing these counts. Keep
the file mode 0600 and installation/config directories mode 0700. Do not source
this file in a shell: the operator CLI and systemd parse it. Generated secrets
are independent session, assistant-vault, storage-vault, application-database
and DBA values. The PostgreSQL service reads its DBA secret before its official
entrypoint drops privileges; secrets are never arguments to a process.

```sh
python3.12 /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host install
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host build
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host migrate
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host bootstrap-admin --email operator@example.edu
```

`install` installs locked API dependencies and user systemd units. `build` builds
the research and assistant images, pins the resulting immutable local image IDs
in the private configuration, and actually probes rootless bind ownership and
memory/PID controls. New image identities are published only after a successful
probe. Startup requires its record to match the images, runtime helper, source
location, UID and kernel; rerun `build` after changing those inputs. Image downloads require outbound access. The PostgreSQL
major version is fixed at 17; `build` also pulls and pins its resolved repository
`@sha256` digest in `config/compose.json`. Review that image before admission.
Subsequent builds retain the pinned digest unless the operator explicitly
selects a reviewed replacement. Do not change its major version without a
PostgreSQL migration plan.

`migrate` starts PostgreSQL, provisions a non-superuser `pfapp` role, then invokes
PF's explicit ordered schema migration. The database lives in the rootless
Docker named volume; objects, work areas, assistant snapshots/evidence and BYOS
spool/journals live under `pf-host/state`. Both API and worker refer to these
same absolute host paths, so Docker bind mounts refer to the actual files.
`bootstrap-admin` prompts for the password; it does not write it into a command
line, logs or a delivered configuration file.

The host administrator installs the generated Caddyfile:

```sh
sudo install -o root -g caddy -m 0640 /home/pfsim/pf-host/config/Caddyfile /etc/caddy/Caddyfile
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl enable --now caddy
sudo systemctl reload caddy
```

Caddy obtains and renews the host certificate using its persistent service state.
The API binds only to loopback and trusts proxy client-address metadata only from
loopback. It derives secure cookies and canonical links from the configured
HTTPS origin. Do not add another upstream proxy without reviewing that trust
boundary. See [Caddy automatic HTTPS](https://caddyserver.com/docs/automatic-https).

```sh
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host start
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host status
curl --fail https://research.example.edu/healthz
journalctl --user -u pf-api -u pf-worker
```

Registration is disabled. The host operator first provisions each ordinary
researcher without granting global-administrator rights or creating an extra
organization. The operator must verify the account holder's email through the
institution's identity process and deliver the prompted password securely:

```sh
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host provision-user --email researcher@example.edu --name "Researcher Name"
```

Then an organization owner/admin sends that verified user an invitation through
the existing interface. The user signs in and accepts the invitation. Duplicate
accounts are rejected by provisioning; this command does not reset passwords or
promote users. Project access
controls are enforced by the application. The new institutional default
`PF_SAAS_RESEARCH_POLICY_DEFAULT=institutional` blocks external AI/provider/source
disclosure until an organization administrator explicitly configures policy.
Read `docs/SAAS_RESEARCH_POLICY.md` before authorizing a provider or disclosure.
Per-user BYOK does not override institution/project restrictions. BYOS is
available with its independent encrypted vault and durable connector journals;
custom compatible endpoints still require explicit operator configuration.

## Backup, restore and upgrades

Keep encrypted off-host backups of the entire backup directory. It contains
private research data and the keys needed to decrypt saved credentials. Caddy's
`/var/lib/caddy` and the host/rootless-Docker configuration require separate host
backups. Source releases and exact built images must also be retained; do not
rely on a mutable image tag for recovery. Avoid printing populated Compose or
service configuration in support tickets.

```sh
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host backup --output /home/pfsim/backups/r34-before-upgrade
```

Backup stops API/worker and orphaned PF execution containers, obtains a PostgreSQL
custom-format dump, archives the complete state and secret configuration, and
writes SHA-256 checksums. It leaves services stopped even on failure, preventing
a misleading partial backup from silently returning the service to operation.
Interrupted jobs/investigations are not reported as successful or automatically
replayed. Restore only a backup produced by this operator tool to the same
installation path (a new host must use that path and service UID):

```sh
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host restore --backup /home/pfsim/backups/r34-before-upgrade --confirm-replace
```

Restore checks hashes and rejects archive path traversal, links and special
files before modifying state. Existing state/config are renamed for recovery;
then only the application database is recreated empty and restored with errors
treated as failures. Restored database credentials are reset using the local
container DBA socket, so a previous credential rotation does not prevent
recovery. Schema checking/migration runs after restoring the dump. Services
remain stopped. Do not treat unkeyed checksums as authentication for backups from an
untrusted source. For orphaned assistant reservations/storage receipts, use the
existing `assistant-operations`, `assistant-reconcile --confirm-stopped`,
`assistant-storage-operations` and `reconcile-assistant-storage` operator commands
under the same service environment after confirming the old processes stopped.
For example, list unfinished operations without manually exporting secrets:

```sh
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host operator assistant-operations
```

Use `operator assistant-reconcile --operation ID --confirm-stopped` only after
verifying that operation's processes/containers are gone.

To upgrade, first verify the new release's package/evidence integrity, extract it
to a new directory and preserve the old source and Docker images:

```sh
/home/pfsim/pf-host/venv/bin/python /srv/pf-releases/r34-consolidated/deploy/saas/reference/host.py --installation /home/pfsim/pf-host upgrade --source /srv/pf-releases/r34-consolidated --backup /home/pfsim/backups/r34-before-update
```

Run the command from the **new** release, as shown. It takes a stopped backup,
switches the source link, adds only absent shared capacity settings while retaining
existing values, installs runtime dependencies, rebuilds/pins images, probes
execution and runs the current schema-10 migration. It does
not start public services. If an upgrade fails, keep them stopped: point
`pf-host/current` back to the old release, reinstall its runtime dependencies,
restore the paired pre-upgrade database/state/config backup, then preflight and
accept before starting. Database downgrade by simply reverting code is unsafe.
Version identity remains **r34** throughout this reference revision. Once migration
succeeds, run the documented `start` and `status` commands, check `/healthz` and
`/readyz`, then complete the target-host acceptance below before admitting users.
Do not run older-schema services against the upgraded database.

## Required target-host acceptance

These files and command contracts are implementation, not evidence of a live
institutional deployment. The development environment had no usable Docker
daemon or systemd privileges; it cannot certify this host's kernel isolation,
HTTPS issuance, SMTP, backups or provider connectivity. Before admitting users:

1. Run preflight/build/migration, HTTPS health and recovery-email checks.
2. Run and download a real numerical study; restart API/worker and verify durable
   jobs, artifacts, assistant records, BYOK credentials and BYOS routing persist.
3. Test private projects using two ordinary same-organization accounts and a
   separate organization; verify denial of every unauthorized artifact path.
4. Confirm policy-blocked external AI attempts send zero provider traffic; then
   explicitly permit and consent to a test account/provider and verify its path.
5. Exercise restricted Docker execution and resource exhaustion on the actual
   kernel, then perform a stopped backup/restore drill on a separate test host.
6. Qualify any optional external SDK, live-network profile, storage endpoint and
   the institution's concurrency/capacity target. Retain logs and run identities.

This remains the B4 deployment-acceptance gate; local unit tests do not close it.