ISAES completion: measured qualification and external acceptance

The original redesign requires software implementation, operational evidence, and representative-researcher usability. A passing automated suite does not complete every gate. The completion release keeps these results separate in a machine-readable qualification summary.

release_tools/isaes_completion_acceptance_contract.json freezes the supported workloads, budgets, participant population and task thresholds. It was written before acceptance measurement. Do not edit its thresholds to make a measured failure pass. A changed workload or threshold needs a separately identified contract and new qualification, preserving the earlier attempt.

Measured developer workloads

Run from the package root using the supported Python 3.12 research interpreter:

python release_tools/isaes_completion_qualification.py performance --output validation_r34_isaes_ai/performance.json
python release_tools/isaes_completion_qualification.py verify-performance validation_r34_isaes_ai/performance.json

The runner requires Linux/Python 3.12 for these frozen developer-executor budgets. It executes actual reporting calculations and exports in fresh subprocesses:

Elapsed wall time includes interpreter startup, numerical calculations and exports. Peak resident memory is the measured subprocess's lifetime high-water mark. Numeric libraries run with a recorded single-thread policy to avoid uncontrolled oversubscription. The receipt records runtime/package versions, hardware/CPU affinity, exact source inventory, contract hash, per-run commands, observed timings, peak memory, input/output identities and retained artifact hashes. Before and after checks reject source or runtime changes during the run.

The percentile is the nearest-rank statistic ceil(0.95*n) of sorted samples. With three or five observations it is the maximum. These small samples establish the frozen release workload check; they are not a broad production-latency SLA. No averaging hides a failing high-water memory or latency observation.

The runner refuses to overwrite an existing measurement directory. Choose a new output path when repeating an attempt. Failed subprocess logs remain available. The final release runner performs these measurements after the fresh software tests and checks the same frozen source afterward.

The live workbench browser runner supplies a separate pf.isaes.browser-performance/1 receipt with exactly three measurements of each:

Each browser receipt binds the contract and source identity and declares actual server execution with provider access disabled. Automated browser actions remain different from a real researcher study. The scope verifier recomputes percentiles from the observed runs rather than trusting a prefilled summary.

A supplementary large_array_navigation_100k gate renders two complete 100,000-point panels in an actual self-contained report served over local HTTP. It measures three fresh Chromium contexts, with initial chart display, real Plotly zoom, linked observation selection and saved-state download/restore. The same frozen 10,000/1,000/2,000 ms budgets apply to initial display, selection/zoom, and restore. Plotly drawing promises are awaited before stopping interaction clocks. All retained points and identities remain available; exact evidence values and unchanged payloads are checked. A sampled browser-reported JavaScript heap observation is retained separately from process peak RSS. This gate qualifies the standalone HTML route; the actual authenticated workbench gate remains separate. Its receipt is large-array-browser/isaes-large-array-browser.json under the validation folder.

Native Windows execution

After release packaging, collect all new external qualification inputs, receipts, logs and observations in a sibling directory such as ../isaes-external-qualification/, outside the extracted package. The sealed package inventory includes its existing validation evidence: adding receipts under validation_r34_isaes_ai or elsewhere inside the package changes that inventory. The helper accepts external paths and still binds the package's source and acceptance contract. Keep the delivered archive and its original qualification summary intact; retain later evidence as a separate supplement. Use a new receipt name for each attempt rather than overwriting an earlier one.

Native Windows qualification cannot run in a Linux container or WSL. Install the release's pinned supported dependencies, including every reporting/export dependency, in a native Windows Python 3.12 research environment. The selected tests also execute a real browser: Node.js must be on PATH, Playwright must be available to Node, and PF_BROWSER must name an installed Chromium executable. Set PF_PLAYWRIGHT_MODULE to its absolute module path when it is not available through ordinary Node resolution. Missing optional exporters or an unconfigured browser produce skips, which correctly fail this zero-skips qualification. From the extracted package root, replace the two example installation paths:

$env:PF_BROWSER = 'C:\Tools\chromium\chrome.exe'
$env:PF_PLAYWRIGHT_MODULE = 'C:\Tools\isaes-node\node_modules\playwright'
$env:PF_ISAES_PRESENTATION_DIR = [IO.Path]::GetFullPath('..\isaes-external-qualification\windows-browser-001')
Remove-Item Env:PF_ISAES_EXAMPLE_DIR -ErrorAction SilentlyContinue
py -3.12 release_tools/isaes_completion_qualification.py native-windows --output ../isaes-external-qualification/windows-001.json

The command refuses other operating systems or Python versions. It runs fixed scientific-method, quantum-protocol, presentation, lifecycle, actual workflow, SDK prerequisites, report-assistant/proposal contracts, HTTP authorization, launcher and installer tests. Assistant protocol tests use explicit local fixtures; this does not qualify live model accuracy. A passing receipt requires nonzero tests with zero failures, errors or skips. It retains JUnit and execution logs and binds source/runtime identities. The verifier checks the actual JUnit counts and fixed test list; a manually written status: passed is insufficient. This selected native route is separate from hosted containment or real BYOS qualification.

Do not describe Python 3.8 syntax parsing, native Windows launch screenshots or Linux test results as native Windows execution qualification. The legacy runtime continues to need its separately scoped validation if that deployment is used.

Real researcher task study

Use the uncompleted release_tools/isaes_completion_study_template.json as a collection schema. It contains no fabricated participants or passing outcomes. Export the current source/contract identity before collecting sessions:

python release_tools/isaes_completion_qualification.py identity --output ../isaes-external-qualification/qualification-identity.json

Copy the study template to that external collection directory and keep the completed manifest and its relative observation files there. Do not edit the template included in the sealed package.

The frozen study population is at least five researchers in relativity, quantum research or scientific computing with limited application-specific coding experience. Every participant must attempt all five frozen tasks:

1. Compare two compatible runs.
2. Investigate a failed scientific check.
3. Trace a literature comparison to its exact source.
4. Change and restore an appropriate view.
5. Export a report and presentation.

The manifest uses schema_version: "pf.isaes.researcher-study/1" and, only for real observed sessions, evidence_nature: "observed_real_researcher_sessions". Copy the source and contract hashes from the identity receipt. Record study, protocol and observer identities plus an ISO timestamp including timezone. Keep participants pseudonymous; each participant contains:

• participant_id: unique pseudonymous identity.
• participant_background: relevant scientific experience.
• research_area: relativity, quantum or scientific_computing.
• application_specific_coding_experience: limited for this population.
• tasks: exactly one attempt per frozen task, using its exact contract text.

Every task record contains the exact task, positive elapsed_seconds, integer assistance_count, errors, task_outcome and observation_artifact. Outcomes are completed_unassisted, completed_assisted or not_completed. Unassisted completion requires zero assistance; assisted completion records positive assistance. Record errors as objects with description and severity (critical_scientific_misinterpretation, operational or minor). An explicit empty errors array means no observed errors; omitting it is invalid.

observation_artifact names a retained relative path. The top-level evidence maps every observation artifact to its SHA256. Observations may be authorized session notes, task records or recordings. Keep identifiable personal information and unrelated confidential research out of the distributable receipt.

Acceptance requires at least 80% unassisted task completion overall and zero critical scientific misinterpretations. All attempted tasks count in the denominator. Per-task outcomes, times and errors remain visible rather than being hidden by the overall rate. An incomplete, simulated or unrepresentative study cannot satisfy the contract.

python release_tools/isaes_completion_qualification.py import-study ../isaes-external-qualification/collected-study.json --output ../isaes-external-qualification/study-import-001.json

The importer validates the schema, population declarations, task coverage, artifact hashes and threshold arithmetic, and copies the original manifest and observations into a retained evidence directory. Meeting arithmetic thresholds produces external_review_required, not automatic usability acceptance. Independent review must establish that the participants and observations are real, representative and correctly interpreted. A hash or a self-declared field cannot prove those facts. Scripts, agents and synthetic unit-test records are never substituted for representative researchers.

Hosted production and live BYOS

These gates are conditional on the actual deployment. Unknown deployment scope remains unqualified; it is not silently marked passed or not applicable. A standalone-only or regular-storage-only deployment can document why a conditional gate is outside its scope during release review, without claiming the omitted service was tested.

The uncompleted deployment template has schema pf.isaes.external-deployment/1. A real probe records execution_nature: "live_deployment_probe", a target deployment identity, operator, source/contract hashes and start/completion timestamps. Each check records an actual boolean passed, a concrete observation, and one or more evidence_artifacts named in the top-level SHA256 evidence inventory.

For production_container_isolation_if_hosted, the mandatory checks are:

• private_files_other_owner_denied
• host_filesystem_escape_denied
• unapproved_network_egress_denied
• resource_limits_enforced
• cancellation_stops_worker
• publication_rechecks_authorization
• rollback_preserves_readers_and_deletion_guards

For live_BYOS_if_deployed, the mandatory checks are:

• owner_scoped_write_read_hash_match
• cross_owner_access_denied
• credentials_not_exposed_in_artifacts
• interrupted_write_recovery
• retained_dependency_blocks_deletion
• authorized_deletion_recovery
• provider_failure_preserves_retained_evidence

Use isolated deployment test tenants and disposable, identified probe records. Do not delete or expose real retained research. The deployment operator must authorize the probe target, access, resource budget and destructive cleanup of the probe's own disposable records. An authorized deployment-specific Python driver can execute the probes and produce the manifest:

python release_tools/isaes_completion_qualification.py external-probe --driver ../isaes-external-qualification/authorized-deployment-probe.py --gate live_BYOS_if_deployed --output ../isaes-external-qualification/byos-probe-001.json

The driver receives --gate, --output, --source-identity and --contract-sha256. It owns the configured deployment authentication. The runner records driver hash, command and logs; it does not invent a production endpoint or use a mock service as production evidence. Credentials must not appear in logs/artifacts. The driver must place probe artifacts alongside its supplied output path, and any other temporary data outside the sealed package. Use --gate production_container_isolation_if_hosted with a separate output name for authorized hosted containment probes. Existing independently collected observations can be imported:

python release_tools/isaes_completion_qualification.py import-external ../isaes-external-qualification/collected-deployment.json --output ../isaes-external-qualification/deployment-import-001.json

Imports remain external_review_required after structural and hash checks. The operator's observations and deployment/containment adequacy require review. Failed observed behavior remains failed. Neither installed Docker nor local BYOS fixtures qualify a real deployed boundary by themselves.

Qualification summary and release integration

During the release build, before sealing the package:

python release_tools/isaes_completion_qualification.py summarize --performance validation_r34_isaes_ai/performance.json --browser validation_r34_isaes_ai/completion-browser/isaes-completion-browser.json --large-array validation_r34_isaes_ai/large-array-browser/isaes-large-array-browser.json --output validation_r34_isaes_ai/acceptance_status.json

After packaging, produce a separate supplemental summary instead. This example assumes the three named external receipts have been collected; omit any unavailable optional receipt argument so its gate remains pending:

python release_tools/isaes_completion_qualification.py summarize --performance validation_r34_isaes_ai/performance.json --browser validation_r34_isaes_ai/completion-browser/isaes-completion-browser.json --large-array validation_r34_isaes_ai/large-array-browser/isaes-large-array-browser.json --windows ../isaes-external-qualification/windows-001.json --study ../isaes-external-qualification/study-import-001.json --external ../isaes-external-qualification/byos-probe-001.json --output ../isaes-external-qualification/acceptance-status-001.json

Supply --windows, --study, and repeatable --external receipts when present. The developer automated workload result can pass while original_scope_status remains pending for missing native execution, real researcher observations or applicable deployment evidence. The summary lists every missing gate and never turns imported declarations into an all-scope approval. Final release review must retain these distinctions and record any independently established external acceptance separately; changing a receipt's status field is rejected.

The main release validator consumes verify_performance, verify_browser and the summary helper. Build-stage receipts and nested run/evidence artifacts belong to validation_r34_isaes_ai, outside the source inventory they bind. Post-release external supplements remain outside the entire sealed package; their evidence paths resolve relative to each supplemental receipt's directory. Historical 527-test ISAES evidence remains historical. Current measurements must come from the current frozen source, with current commands and runtime recorded.

Reporting 2.2.0 and optional AI scope

Current package verification uses python release_tools/validate_r34_isaes_ai.py verify --package. The preceding 2.1.0 receipts remain unchanged in validation_r34_isaes_completion; do not rerun its historical qualification in place. New receipts record the SDK correction, assistant evidence/proposal boundary, reviewed application and existing-provider compatibility. Protocol fixtures are explicitly test doubles, not live provider scientific evaluations.

The report assistant is disabled by default. An operator may set PF_ISAES_AI_ENABLED=1 for controlled evaluation with existing provider/BYOK settings and institutional controls. This opt-in is not an acceptance receipt. Broad activation still requires intended-environment qualification and representative researcher observations. Evaluate the actual selected provider/model on scientific grounding, unsupported claims, preserved negative findings and instruction-bearing literature; local structural checks do not establish semantic truth. Ordinary reporting and exports must continue to pass with AI access disabled and credentials absent.