Measuring the configured SaaS capacity

The full public-release integration adds release_tools/saas_capacity_mixed.py for real authenticated writes, storage, scientific-job and assistant journeys. The original read-only probe remains available unchanged for cheap baseline measurement. Neither tool converts a configured user count into a capacity guarantee. This release has not been load-qualified for 200,000 concurrent users; that acceptance campaign requires the actual provisioned deployment and generators.

The deployment can start with one user and one execution slot. Grow the API replicas, shared PostgreSQL capacity and job worker fleet independently as the measured workload grows. A user count is not an execution-slot count: 200,000 signed-in users with one request every five seconds offer about 40,000 HTTP requests per second, while their scientific jobs have a separate arrival rate, runtime, queue, budget and concurrency limit.

release_tools/saas_capacity_probe.py provides a repeatable authenticated HTTP workload. It uses the installed httpx dependency and never creates accounts, submits jobs, starts assistants, uploads data, changes configuration, or calls paid scientific providers. GET requests still consume deployment resources and may update authentication activity timestamps. Run it against an authorized test deployment with representative data and monitored resource costs.

Workload and distinct users

Create and sign in distinct test accounts using the application's ordinary account workflow. Give each account access to its stated organization and project. Export only the existing pf_session cookie value and the corresponding IDs into a private JSON file. The application authenticates these routes through that cookie; a bearer token is not a substitute. This tool does not generate accounts, credentials or placeholder identities for a target population.

Example file shape, with illustrative values that must be replaced:

{
  "sessions": [
    {
      "cookie": "EXISTING_PF_SESSION_COOKIE_VALUE",
      "user_id": "existing_user_id",
      "project_id": "existing_project_id",
      "org_id": "existing_org_id",
      "job_id": "optional_existing_job_id"
    }
  ]
}

On Unix, the credential file must be owned by the running user with mode 0600 or 0400. Store it outside the release source and all published evidence. On Windows, use a private account directory and its access controls. Never paste cookies into shell arguments, source control, reports or shared logs. The tool rejects duplicate cookies and duplicate user IDs and verifies each account's identity using /api/me before continuing its journey. It does not print credentials, individual IDs, response bodies or HTTP exception messages.

Two fixed, read-only profiles are available:

Every request carries that user's cookie and project selection. The usage request uses that user's organization. Each user has at most one outstanding request. --cadence-seconds sets the interval between successive requests, not between five-request cycles; it represents an explicit browsing or polling assumption. Real browser background tabs, idle sessions, export traffic and job-specific polling can have different rates. Choose representative account, organization, project and job distributions; an empty demonstration account is not a representative large tenant.

Start small, then add independent generators

Run the command from the release root with its SaaS Python environment. Choose a new report filename for every run. The tool refuses existing output paths.

python release_tools/saas_capacity_probe.py \
  --base-url https://your-test-deployment.example \
  --credentials /private/pf-test-sessions.json \
  --users 1 --duration-seconds 180 --ramp-seconds 10 \
  --cadence-seconds 5 --max-rps 2 --max-in-flight 4 \
  --min-samples-per-route 5 \
  --output /private/capacity-one-user.json

Use stages such as 1, 100, 1,000, 10,000, 100,000 and 200,000 or more distinct users. --users N is the global target. Its numeric validation range is 1 through 1,000,000, but the complete credential file must also fit the tool's 64 MiB input limit. The numeric range does not guarantee that a million-user roster can be loaded. With this release's 64-character session cookies and 32-character IDs, a 200,000-user roster is approximately 41–49 MiB for browse, or 49–59 MiB for monitor, depending on JSON whitespace. At 250,000 users, a compact monitor roster is about 62 MiB and expanded whitespace can exceed the limit. Measure the actual file; omit unused job IDs for browse and unnecessary JSON whitespace.

Every generator parses the entire roster and checks global identity/cookie uniqueness before retaining its selected shard. Allow memory substantially above the file size for parsed JSON, session objects and uniqueness sets during startup. Sharding bounds active requests and retained session scheduling state; it does not remove that initial full-roster memory cost or the 64 MiB file limit. Larger credential sets require a separately qualified load generator; this tool does not accept independently supplied shard-local rosters.

For large targets, run multiple load-generator machines, each with the same ordered credential roster and a different zero-based --shard-index. Shard i selects roster indexes i, i + shards, i + 2*shards, ..., with no overlap between correctly configured shards. All generators must agree on --users, --shards, the roster, profile, cadence, ramp, duration and planned start time. The CLI starts immediately: coordinate launch externally and use the reported UTC start times to confirm overlapping measurement windows. Per-shard files are not an automatic global aggregate or proof of simultaneous activity.

For example, 200,000 users over 200 generators gives 1,000 users per generator. At a five-second cadence each shard's nominal demand is 200 RPS. This illustrative command is for shard 0; launch indexes 0 through 199 on the independently provisioned generators only after lower stages have passed:

python release_tools/saas_capacity_probe.py \
  --base-url https://your-test-deployment.example \
  --credentials /private/pf-test-sessions.json \
  --users 200000 --shards 200 --shard-index 0 \
  --duration-seconds 1200 --ramp-seconds 300 \
  --cadence-seconds 5 --max-rps 250 --max-in-flight 128 \
  --timeout-seconds 10 --min-samples-per-route 1000 \
  --max-error-rate 0.01 --max-p95-ms 1000 --max-schedule-lag-ms 1000 \
  --output /private/capacity-200000-shard-000.json

Those numbers are a workload specification, not an established hardware sizing recommendation or a claim that 200 generators are necessary or sufficient. Determine each generator's safe capacity from its own CPU, network, connection, memory and scheduling-lag measurements. The scheduler keeps small records per user and at most --max-in-flight asynchronous request tasks and reused connections (1–4,096 per generator). It does not start one thread or one running task per virtual user. --max-rps is a strict per-generator pacing ceiling; there is no catch-up request burst. A ceiling below the shard's nominal demand fails the observed-profile checks. Ramp time determines each user's first request, spread across the global target. Allow enough time after the last user arrives for multiple complete profiles and representative steady-state data.

Requests have an overall timeout and a bounded response size. Remote origins require HTTPS; plain HTTP is allowed only for local fixtures. Redirects are not followed, and environment proxy settings are not inherited. No automatic retries hide failed requests. A failed identity check or an HTTP 401/403 stops that user's journey and fails the session-coverage check; refresh expired sessions before a new independent run.

Interpret the report and correlate the deployment

The JSON records the exact target and shard configuration, users actually verified, users completing a successful full journey, route sample counts, status/error counts, bytes, mean/max/p50/p95/p99 latency, achieved RPS, peak outstanding requests, scheduling lag, missed cadences and requests already due but not started before the measurement deadline. It waits for bounded in-flight requests to finish and includes that drain time in achieved RPS. This is a paced closed-loop session workload; overload can suppress later requests. Suppressed cadences and horizon work are reported explicitly and make the profile fail. Percentiles are conservative histogram upper bounds with bins at most 5% wide; they must not be averaged across shards to manufacture global percentiles.

Exit code 0 means this shard met all declared checks: each target identity was verified, each user completed a successful journey, every route met the sample minimum and p95 threshold, error rate and scheduling lag stayed within bounds, the RPS ceiling covered nominal demand, and no due work or cadence was lost. Exit code 1 means one or more checks failed; inspect the report before increasing load. Exit code 2 means the run could not produce a completed report. The report always says capacity_qualified: false: a passing read-only profile is one piece of production qualification evidence, not a certification of that user count or scientific-job throughput.

Record the release hash, deployment topology and resource limits with every campaign. Correlate all shard windows with API p95/p99 and 503/429 rates, PostgreSQL connections/latency/locks, object-store latency, worker queues and claims, publication recovery, network/CPU/memory and generator saturation. Relevant controls in this release are:

Budget PostgreSQL connections across every API replica, worker process and enabled assistant store, rather than assigning each replica the database's entire connection budget. The assistant store has its own pool. HF native runs one durable controller with a bounded local database connection budget; the distributed deployment is the multi-replica/multi-host path. Follow the distributed deployment guide for shared state, worker admission, budget and scaling requirements.

Before declaring support for a specified concurrent-user population, also qualify representative job submission/queue/execution/publication, assistant and private-storage operations, uploads/downloads, sign-in/session renewal, tenant isolation, long-running soak behavior, safe scale-out/scale-in, and replica/worker/database failure recovery. These operations need a separate explicit test workload and cost budget; this read-only tool does not perform them. Preserve failed runs as well as successful runs, and qualify the complete campaign at the target population before changing any capacity claim.

Mixed workloads and isolated target guard

Use the mixed tool on a separate, disposable deployment with synthetic data, dedicated provider budgets, and test mailbox accounts. Configure PF_SAAS_CAPACITY_CAMPAIGN_ID to a unique 16–128-character identifier containing only letters, digits, underscores and dashes. /healthz advertises it in X-PF-Capacity-Campaign. The generator checks the exact header before sending any authenticated request. Keep this setting empty on the production service. The marker records an operator assertion that this is an isolated campaign; it cannot prove infrastructure separation. It grants no authentication or authorization bypass and is not a secret.

Mutations additionally require --allow-writes. Scientific job submission requires --allow-compute; provider messages require both --allow-provider and --allow-compute because an assistant can invoke execution tools. A prompt asking it not to execute is not a security boundary. Email requests require --allow-email and must address owned test mailboxes. Do not point any of these workloads at the current production HF controller. The harness does not change Space visibility, provision resources, stop workers or inject infrastructure faults automatically.

--mix browse=3,project=1,asset=1,isolation=1 is a deterministic weighted cycle of journeys. Account index offsets distribute the initial scenarios. Cadence is the interval between journey starts, unlike the original probe's per-request cadence. A journey can issue several HTTP requests. Choose weights and cadence from measured real user activity; they are not inferred from session count. Every target user must verify identity, complete a useful journey during the steady window, and complete isolation checks when that lane is enabled. Every configured scenario must meet --min-journeys-per-scenario on every shard.

Job/provider/email cost attempts have per-user hard ceilings (defaults one). Attempts consume the ceiling before the request, even if the response times out. Subsequent visits to an exhausted cost lane perform browsing and are counted as browsing, never as completed jobs/messages/emails. Set explicit larger ceilings to sustain a representative arrival rate throughout a soak. These are generator attempt limits, not provider billing guarantees: an assistant message may make several model calls or execute tools. Configure application/provider budgets separately and include them in the campaign record. A campaign of 200,000 users with one allowed job each can request 200,000 real jobs.

There are no automatic HTTP retries. An uncertain mutation disables that user, records uncertain_mutations, and fails the workload. Reconcile retained objects/jobs before a new run. Successful asset/attachment cleanup is performed through ordinary authorized APIs. The tool does not remove completed scientific jobs, existing assistant sessions, or project audit records; retained outputs are necessary for publication/accounting inspection. Dispose of the isolated deployment only after preserving evidence and reviewing running provider work. New mutations are forbidden once the measurement deadline passes, including requests delayed by generator pacing. Already-started work may finish bounded read-only polling. A successful HTTP write followed by unknown resource identity, failed readback or unfinished cleanup also stops that user and requires reconciliation; it is never automatically repeated on the next journey.

Shard-local roster and synchronized launch

Unlike the original probe, the mixed tool streams shard-local JSONL files. Each line is one account and contains its global index. Shard i must contain exactly indexes i, i + shards, ... in increasing order, below the global --users target. No generator parses a million-account global JSON document. The file limit is 512 MiB and each line is limited to 64 KiB; credentials remain private and permissions follow the original probe's ownership/mode rules.

Example for the first account of a two-account campaign:

{"index":0,"user_id":"actual_user_a","org_id":"actual_org_a","project_id":"actual_project_a","cookie":"actual_pf_session_value_a","job_id":"actual_retained_job_a","foreign_project_id":"actual_project_b","foreign_job_id":"actual_retained_job_b"}

Account B must own those foreign fixtures, and its own row points back at A's fixtures. Merely inventing unknown IDs that return 404 is insufficient: the aggregate verifies that every foreign project and job belongs to another tested account and that all accounts exercised their own fixture reads. Use separate organizations or properly restricted projects, not fixtures that are intentionally shared. For private assistant isolation add foreign_assistant_session_id; its existence and owner must also be recorded and reviewed in the external fixture inventory.

For login coverage, add email and password; a cookie can be omitted and the tool will authenticate normally before the first journey. For provider messages add assistant_session_id. Never publish these files, login cookies/passwords, private prompts or response bodies. Aggregate reports retain SHA256 account and fixture fingerprints to detect reused identities across rosters; protect those pseudonymous reports according to the campaign's data policy.

Prepare accounts and representative completed jobs through the ordinary test deployment/operator workflow. The load generator does not silently seed or mark unverified public registrations as verified. Registration, email verification, invitation and reset-token consumption need a separate mailbox acceptance campaign. Record its actual delivery receipts and token outcomes; HTTP 202 by itself is not delivery evidence.

Start with two accounts and the non-compute mix, using the exact SHA256 of the release actually installed on the target:

python release_tools/saas_capacity_mixed.py run \
  --base-url https://isolated-pf-test.example \
  --campaign-id pf-isolated-campaign-20261009 \
  --release-sha256 "$DEPLOYED_RELEASE_SHA256" \
  --credentials /private/campaign-shard-0.jsonl \
  --users 2 --allow-writes \
  --mix browse=3,project=1,asset=1,isolation=1 \
  --duration-seconds 300 --ramp-seconds 10 --min-steady-seconds 240 \
  --cadence-seconds 10 --max-rps 20 --max-in-flight 4 \
  --output /private/mixed-two-users.json

One-user functional scenarios work by selecting a mix without isolation; cross-account isolation and the aggregate acceptance gate require at least two real accounts. Increase populations in measured stages. At 200,000 users and 200 generators, every generator retains 1,000 users' scheduling records. Use the same --users 200000 --shards 200 and distinct --shard-index 0 through 199, with the corresponding private roster on each generator. Compute one future UTC epoch and distribute that identical --start-epoch to all generators; do not calculate a separate epoch independently on each host. Synchronize host clocks. Late-starting generators refuse to run. The aggregate permits at most one second of launch spread and requires overlapping steady measurement time.

Each generator has at most --max-in-flight active journeys and reused HTTP connections (maximum 4,096), rather than one task/thread per target user. --max-rps paces actual HTTP starts across all journeys with no catch-up burst. Network timeouts, whole-journey timeouts, bounded JSON responses and streamed download byte bounds cap work. Polling a scientific job occupies an active journey slot; budget generator slots and shards for the measured job duration. A saturated generator cannot pass merely because all accounts logged in: scheduling lag, missed journeys and users inactive in the steady window fail the checks. Ordinary auth/account/API limits remain enforced and 429/503 count as workload failures. Do not globally disable them to manufacture a result.

Choose --journey-timeout-seconds longer than the scientific job's timeout plus its allowed queue/publication delay. Use --job-payload with an actual validated PF request JSON and bounded resource_limits.timeout_seconds. Its SHA256 is recorded; mismatched scientific payloads cannot be aggregated. Capture separate campaigns for representative heavier workflows and larger exports. The small CSV lane does not qualify large-file ingestion. No single fixed workload mix represents every scientific workflow or assistant conversation.

Aggregate evidence, soak and failure recovery

python release_tools/saas_capacity_mixed.py aggregate \
  --report /private/mixed-shard-0.json \
  --report /private/mixed-shard-1.json \
  --output /private/mixed-campaign-aggregate.json

Supply every shard report exactly once. Aggregate rejects missing/duplicate shards, mismatched campaign configuration/workload, duplicate account identities, inconsistent counts/histogram totals, invalid observation windows and insufficient steady overlap. It merges histogram bins for global percentiles; it never averages p95 values. Each report includes sample/status/ error counts, response bytes, successful journeys by scenario, failed journey categories, actual steady request rate, scheduling lag, missed work, verified active users and bounded cost attempts. A write returning 200 with the wrong resource or failed readback is a failed journey. Job success requires terminal completion and matching result identity; a provider response with a partial or failed investigation does not pass. Exit codes are 0 for the declared observed workload passing, 1 for completed failing evidence, and 2 for an incomplete run. Output paths are exclusively created; preserve unsuccessful evidence too.

The supplied release SHA256 is an operator declaration. Independently verify the deployed code archive and image/source pins; the campaign marker verifies only the target's configured campaign identifier. Local report consistency checks do not authenticate evidence files or replace an independent review.

The aggregate still emits capacity_qualified: false. To accept a production capacity target, preserve and review all of the following for the same release, deployment topology, time window and workload mix:

1. Resource inventory: API replicas, database instance/pools, worker slots, provider quotas, object storage, shared assistant filesystem, mail service, generator hardware and network. Include configured budgets and provider account concurrency limits.
2. Sustained mixed workload at target active users, with a declared soak length, job arrival/completion rates, queue-delay limits, p95/p99 latency thresholds, request/error rates, object bytes and every shard's activity. Choose a soak long enough to expose session renewal, recurring maintenance and accounting retention; the CLI allows up to 24 hours per campaign.
3. Correlated API/database/lease/object/worker/generator CPU, memory, connections, latency, queue and saturation metrics. HTTP throughput alone cannot determine the capacity or headroom of a downstream scientific provider.
4. Scheduled, operator-controlled test-environment faults: terminate an API replica during writes, stop an executing worker, interrupt object access, and exercise database failover using the database operator's procedure. Timestamp each event, retain failed requests, and show subsequent recovery, bounded queue drain, no duplicate publication, and no premature refunds. The generator does not execute fault commands or certify recovery from a restored HTTP 200 alone.
5. Audited quota lifecycle evidence before/after successful, failed, interrupted and deleted attempts; conservative-mode compatibility; safe-mode settlement; historical charge reconciliation and retained file preservation.
6. Real registration/verification/reset/invitation mailbox receipts, replay and expired-token rejection, shared auth rate limits, ordinary-user denial of platform administration, BYOK/BYOS isolation/rotation, and retained artifact integrity/scientific validation for representative jobs.

HF native remains one durable controller and is not the topology to certify by replicating that writer. Multi-replica campaigns must use the documented distributed Docker or managed-HF architecture with external shared PostgreSQL and the required storage/lease contracts. Linux/Docker and HF campaigns need separate infrastructure evidence even when common application tests pass. Publish a capacity claim only for the tested workload, resources, limits and failure assumptions. A target of hundreds of thousands of concurrently active users is a deployment acceptance requirement, not a release-unit-test result.