PF Simulation r34: complete self-hosted research reference

For current r34 public-service and schema-10 instructions, follow the public-service guide and deployment choices. This page covers the Linux/Docker single-host option. Workflow and conversation deletion receipts, database state, private storage and encryption keys must be recovered consistently. Do not roll application code back against a migrated database; read the lifecycle guide and execution recovery.

This reference provisions one dedicated Linux host for the API, durable worker, PostgreSQL metadata, managed storage, private assistant state, BYOK vault, BYOS journals and connector state, and isolated numerical execution. Caddy terminates HTTPS. This is the reference for the full r34 research feature set; the earlier AWS Fargate template remains available for its normal-run topology. Fargate does not provide the local Docker daemon or durable assistant files required by this reference. Do not enable the full assistant there by copying environment flags.

This is the single-host option in the consolidated deployment guide. It retains local PostgreSQL; the optional managed HF architecture is independent. Hosted database schema is 10 in the current public-service release.

Trust and capability boundary

Use a dedicated Ubuntu 24.04/systemd host, Python 3.12, Docker Engine with the Compose plugin and rootless support, cgroup v2 with systemd resource delegation, and Caddy. The API and worker are native services under a dedicated unprivileged account. Docker's daemon also runs under that account; the account must not belong to the rootful docker group. API and worker can access that account's state and rootless daemon; compromise of either exposes that account's data. This topology does not claim isolation between those two trusted services.

Scientific and assistant workloads receive only their run files and an explicit safe environment. They receive no database, BYOK/BYOS vault keys or Docker socket. Assistant containers have no network, a read-only source/root filesystem, capabilities removed, no privilege escalation, and CPU/memory/PID/time/output limits. The normal worker defaults to network none too. Explicit live downloads require an operator-reviewed outbound-network design; use supplied offline input files in this default deployment. Provider calls and BYOS operations occur in the trusted API, subject to project authorization and institutional policy.

The rootless executor uses container UID 0, which maps to the non-root host service account, to access its private bind-mounted run folders. The runtime checks the local per-user Docker socket and daemon rootless identity before this mapping. It refuses to apply the mapping to rootful or remote daemons. Existing rootful deployments retain the caller UID/GID mapping. This is namespace root, not host root; deployments still require target-host isolation acceptance.

The default image includes the portable numerical research dependencies. MATLAB, Swabian hardware/SDK and Einstein Toolkit installations remain explicit external prerequisites. Configure their reviewed images using the existing research-profile mechanism only after installing and qualifying those tools. This reference does not pretend to provision commercial licences or physical instruments.

Host preparation

Host-administrator steps, once (substitute the institution's account name):

sudo useradd --create-home --shell /bin/bash pfsim
sudo loginctl enable-linger pfsim
sudo apt-get update
sudo apt-get install python3.12-venv uidmap dbus-user-session slirp4netns

Install Docker Engine, docker-ce-rootless-extras and docker-compose-plugin using the official Ubuntu repository instructions. Install Caddy using its official service installation instructions. Use a proper login session as pfsim (SSH or machinectl shell pfsim@) so its user systemd manager and /run/user/UID exist. Then:

dockerd-rootless-setuptool.sh install
systemctl --user enable --now docker.service

See Docker's rootless prerequisites and resource-limit guidance. The preflight rejects an absent rootless daemon, missing Compose, an unsupported cgroup driver and unavailable resource controls. No rootful fallback is enabled. DNS must point to the host. Permit inbound 80/443 to Caddy and an institutionally restricted administrative login port. Do not expose ports 8000 or 15432.

Configure and provision

Unpack the release to a versioned, administrator-reviewed directory such as /srv/pf-releases/r34-consolidated. Keep it separate from private state. Run every command below as pfsim, never as root. Paths must contain only letters, digits, slash, dot, hyphen and underscore. The examples use a new installation at /home/pfsim/pf-host; generation refuses to overwrite existing state/secrets.

python3.12 /srv/pf-releases/r34-consolidated/deploy/saas/reference/host.py \
  --installation /home/pfsim/pf-host configure \
  --source /srv/pf-releases/r34-consolidated \
  --domain research.example.edu \
  --smtp-host smtp.example.edu --smtp-from research@example.edu

The generated config/service.env uses JSON-quoted, single-line values with literal UTF-8. Do not use JSON \u escapes or control characters; this keeps CLI and systemd values identical. Edit SMTP username/password if the institution's TLS SMTP relay requires them. The generated configuration declares one API and one worker process and bounded main, assistant and lease pools; review shared capacity settings before changing these counts. Keep the file mode 0600 and installation/config directories mode 0700. Do not source this file in a shell: the operator CLI and systemd parse it. Generated secrets are independent session, assistant-vault, storage-vault, application-database and DBA values. The PostgreSQL service reads its DBA secret before its official entrypoint drops privileges; secrets are never arguments to a process.

python3.12 /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host install
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host build
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host migrate
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host bootstrap-admin --email operator@example.edu

install installs locked API dependencies and user systemd units. build builds the research and assistant images, pins the resulting immutable local image IDs in the private configuration, and actually probes rootless bind ownership and memory/PID controls. New image identities are published only after a successful probe. Startup requires its record to match the images, runtime helper, source location, UID and kernel; rerun build after changing those inputs. Image downloads require outbound access. The PostgreSQL major version is fixed at 17; build also pulls and pins its resolved repository @sha256 digest in config/compose.json. Review that image before admission. Subsequent builds retain the pinned digest unless the operator explicitly selects a reviewed replacement. Do not change its major version without a PostgreSQL migration plan.

migrate starts PostgreSQL, provisions a non-superuser pfapp role, then invokes PF's explicit ordered schema migration. The database lives in the rootless Docker named volume; objects, work areas, assistant snapshots/evidence and BYOS spool/journals live under pf-host/state. Both API and worker refer to these same absolute host paths, so Docker bind mounts refer to the actual files. bootstrap-admin prompts for the password; it does not write it into a command line, logs or a delivered configuration file.

The host administrator installs the generated Caddyfile:

sudo install -o root -g caddy -m 0640 /home/pfsim/pf-host/config/Caddyfile /etc/caddy/Caddyfile
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl enable --now caddy
sudo systemctl reload caddy

Caddy obtains and renews the host certificate using its persistent service state. The API binds only to loopback and trusts proxy client-address metadata only from loopback. It derives secure cookies and canonical links from the configured HTTPS origin. Do not add another upstream proxy without reviewing that trust boundary. See Caddy automatic HTTPS.

/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host start
/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host status
curl --fail https://research.example.edu/healthz
journalctl --user -u pf-api -u pf-worker

Registration is disabled. The host operator first provisions each ordinary researcher without granting global-administrator rights or creating an extra organization. The operator must verify the account holder's email through the institution's identity process and deliver the prompted password securely:

/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host provision-user --email researcher@example.edu --name "Researcher Name"

Then an organization owner/admin sends that verified user an invitation through the existing interface. The user signs in and accepts the invitation. Duplicate accounts are rejected by provisioning; this command does not reset passwords or promote users. Project access controls are enforced by the application. The new institutional default PF_SAAS_RESEARCH_POLICY_DEFAULT=institutional blocks external AI/provider/source disclosure until an organization administrator explicitly configures policy. Read docs/SAAS_RESEARCH_POLICY.md before authorizing a provider or disclosure. Per-user BYOK does not override institution/project restrictions. BYOS is available with its independent encrypted vault and durable connector journals; custom compatible endpoints still require explicit operator configuration.

Backup, restore and upgrades

Keep encrypted off-host backups of the entire backup directory. It contains private research data and the keys needed to decrypt saved credentials. Caddy's /var/lib/caddy and the host/rootless-Docker configuration require separate host backups. Source releases and exact built images must also be retained; do not rely on a mutable image tag for recovery. Avoid printing populated Compose or service configuration in support tickets.

/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host backup --output /home/pfsim/backups/r34-before-upgrade

Backup stops API/worker and orphaned PF execution containers, obtains a PostgreSQL custom-format dump, archives the complete state and secret configuration, and writes SHA-256 checksums. It leaves services stopped even on failure, preventing a misleading partial backup from silently returning the service to operation. Interrupted jobs/investigations are not reported as successful or automatically replayed. Restore only a backup produced by this operator tool to the same installation path (a new host must use that path and service UID):

/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host restore --backup /home/pfsim/backups/r34-before-upgrade --confirm-replace

Restore checks hashes and rejects archive path traversal, links and special files before modifying state. Existing state/config are renamed for recovery; then only the application database is recreated empty and restored with errors treated as failures. Restored database credentials are reset using the local container DBA socket, so a previous credential rotation does not prevent recovery. Schema checking/migration runs after restoring the dump. Services remain stopped. Do not treat unkeyed checksums as authentication for backups from an untrusted source. For orphaned assistant reservations/storage receipts, use the existing assistant-operations, assistant-reconcile --confirm-stopped, assistant-storage-operations and reconcile-assistant-storage operator commands under the same service environment after confirming the old processes stopped. For example, list unfinished operations without manually exporting secrets:

/home/pfsim/pf-host/venv/bin/python /home/pfsim/pf-host/current/deploy/saas/reference/host.py --installation /home/pfsim/pf-host operator assistant-operations

Use operator assistant-reconcile --operation ID --confirm-stopped only after verifying that operation's processes/containers are gone.

To upgrade, first verify the new release's package/evidence integrity, extract it to a new directory and preserve the old source and Docker images:

/home/pfsim/pf-host/venv/bin/python /srv/pf-releases/r34-consolidated/deploy/saas/reference/host.py --installation /home/pfsim/pf-host upgrade --source /srv/pf-releases/r34-consolidated --backup /home/pfsim/backups/r34-before-update

Run the command from the new release, as shown. It takes a stopped backup, switches the source link, adds only absent shared capacity settings while retaining existing values, installs runtime dependencies, rebuilds/pins images, probes execution and runs the current schema-10 migration. It does not start public services. If an upgrade fails, keep them stopped: point pf-host/current back to the old release, reinstall its runtime dependencies, restore the paired pre-upgrade database/state/config backup, then preflight and accept before starting. Database downgrade by simply reverting code is unsafe. Version identity remains r34 throughout this reference revision. Once migration succeeds, run the documented start and status commands, check /healthz and /readyz, then complete the target-host acceptance below before admitting users. Do not run older-schema services against the upgraded database.

Required target-host acceptance

These files and command contracts are implementation, not evidence of a live institutional deployment. The development environment had no usable Docker daemon or systemd privileges; it cannot certify this host's kernel isolation, HTTPS issuance, SMTP, backups or provider connectivity. Before admitting users:

1. Run preflight/build/migration, HTTPS health and recovery-email checks.
2. Run and download a real numerical study; restart API/worker and verify durable jobs, artifacts, assistant records, BYOK credentials and BYOS routing persist.
3. Test private projects using two ordinary same-organization accounts and a separate organization; verify denial of every unauthorized artifact path.
4. Confirm policy-blocked external AI attempts send zero provider traffic; then explicitly permit and consent to a test account/provider and verify its path.
5. Exercise restricted Docker execution and resource exhaustion on the actual kernel, then perform a stopped backup/restore drill on a separate test host.
6. Qualify any optional external SDK, live-network profile, storage endpoint and the institution's concurrency/capacity target. Retain logs and run identities.

This remains the B4 deployment-acceptance gate; local unit tests do not close it.