PF research library · 5 min read
Autonomous ISAES: requirements and acceptance evidence
r34 documentation · isaes-autonomous-acceptance.md
This matrix covers the integrated autonomous investigation scope for PF Simulation r34. It separates implementation, executed software evidence, scientific judgments and deployment qualification. A planned capability or a passing protocol fixture is not an executed live-provider investigation.
The implementation baseline is archive SHA256 1987acf5712a7d2311744089bfb350885501bd925e18a2c31ee43447329d6a74. The contract is AUTONOMOUS_ISAES_IMPLEMENTATION_CONTRACT.md. Earlier ISAES and documentation validation trees remain historical evidence for their own source identities.
Integrated scope
| Requirement | Implementing surface | Required acceptance evidence |
|---|---|---|
| Unattended opt-in investigation from a completed scientific run | Standing workflow policy, completion trigger and durable controller | Disabled produces no provider operation; enabled workflow starts without a chat prompt; ownership and immutable base identity verified. |
| Independent deterministic operation preserved | Existing reporting and reviewed discussion routes | Ordinary analysis/export works without AI; existing reviewed proposals retain their approval/application rules. |
| Plan, execute, assess, present and refine | Provider tool loop and persisted investigation records | Ordered protocol fixtures retain decisions, actual tool receipts and a justified terminal outcome; unknown tools fail closed. |
| Associated literature and contextual online discovery | Literature tools and source records | Retained source reading, context-driven discovery, retrieval, source repair/expansion and exact identity/version binding. |
| Evidence-sensitive literature interpretation | Claims, citations and assessment tools | Distinguish support, challenge, mixed, incompatible and insufficient evidence; extraction gaps, approximate digitization and conflicts remain inspectable. |
| Open-ended scientific computation and presentation | Generated-code tools and isolated executor | Real numerical computation and output verification; code/execution/artifacts retained; unavailable packages or invalid outputs fail explicitly. |
| Re-analysis and actual follow-up runs | Reporting integration and child-run tools | Original results remain unchanged; a child has its own executed run identity and cumulative cost; generated prose cannot stand in for execution. |
| Consistent evidence across output formats | Investigation presentation and native report exports | Source identities, claims, decisions, methods, limitations and exact figure data remain traceable through JSON/HTML/PDF/PPTX/ZIP as applicable. |
| Scientific design safeguards | Sampling, preregistration, holdout and adaptive-decision records | Circular support is identified; declared holdouts restrict adaptive reads to explicit non-holdout datasets, including inherited child views and artifact subpointers; unclassified summaries are excluded; adaptive/post-hoc work is labeled exploratory. |
| Explicit delegation and privacy | Owner-scoped policy, institutional/provider admission and tool guards | No provider use without delegation; untrusted literature cannot grant permissions; credentials stay out of tool data and generated code. |
| Cumulative finite resource use | Controller budgets and provider/tool usage | Steps, calls, tokens, time, sources, downloaded bytes and child runs remain bounded across retries/resume. |
| Durable recovery and control | Job/checkpoint persistence, startup/periodic supervisor, pause/resume/cancel/revocation | Queued/running jobs resume after server restart without a browser session; current hosted identity/policy/metering is reauthorized; completed receipts are reused accurately; stopped states and unresolved admission remain protected. |
| Incomplete investigation reporting | Bounded deterministic partial JSON/HTML/ZIP snapshots | Paused, cancelled, budget-exhausted, failed, blocked and shutdown-requeued work retains its reason, progress, evidence references, unresolved findings and limitations without provider calls or new calculations. Stale presentations stay historical; partial artifacts cannot satisfy final completion. Storage failures remain visible. |
| Storage, authorization and lifecycle consistency | Standalone/SaaS routes, artifacts, BYOS and dependency checks | Cross-owner denial; integrity-checked artifact access; storage failures remain visible; retained dependencies prevent destructive deletion. |
| Complete operational documentation | Autonomous guide, main ISAES guide, developer guide and end-user manual | Exact shipped settings/routes and user controls documented; generated PDFs rendered and inspected; links checked. |
| Reproducible release identity | Root release qualification, hashes and archive readback | Frozen current source/test evidence; original bytes preserved; manifest and checksum inventories verified; complete archive readback. |
These requirements are cross-cutting. Testing a tool in isolation does not establish that the completion trigger, owner policy, tool execution, claim assessment, report export and recovery all work together. Integrated acceptance must exercise that entire chain with clearly identified input data and retained output identities.
Current evidence status
The current software evidence is recorded under validation_r34_autonomous, using test_execution.json for exact commands, counts, runtime and frozen source identity, reconciliation.json for the reviewed delta, and qualification_registry.json / research_integrations_registry.json for current applicability. MANIFEST.json identifies the delivered revision. This matrix intentionally carries no copied test count or blanket pass claim: inspect those source-bound records for the executed status.
Verify the installed autonomous evidence with:
python release_tools/validate_r34_autonomous.py verify --package
The command verifies recorded evidence; it does not execute tests or a live investigation. Missing, changed or source-mismatched receipts cannot establish acceptance. Archive identity is checked separately against the delivered SHA-256 file and package inventory.
The implementing modules are pf_reporting/autonomy/contracts.py, service.py, provider.py, literature.py, retrieval.py, extraction.py, science.py, science_worker.py and http.py, with integration through the existing workflows, reporting, assistant, storage/lifecycle and GUI layers. The executable suite inventory belongs to the release validator, so a subsequently renamed test does not silently change the evidence cited by this matrix.
The current runtime's live public-metadata probes could not resolve the Crossref, OpenAlex, DataCite or arXiv hosts. Consequently, implemented retrieval and fixture coverage do not establish live online research in this deployment environment. Successful live retrieval with the intended deployment network remains a separate gate; inspect the recorded probe result rather than treating it as a successful source fetch.
Every executed receipt must identify whether it uses actual scientific calculations, simulated provider protocol responses, adversarial fixtures, real browser behavior, or a live external service. Report their results separately. A fixture that requests a real local calculation can establish that calculation and the orchestration path; it cannot establish a live model's scientific reasoning quality.
External acceptance remains separate
| Gate | Evidence needed before acceptance |
|---|---|
| Live literature discovery/retrieval | Successful requests to intended scholarly services and source hosts in the deployed network, with actual retained retrieval/extraction identities and handled access/rate-limit failures. |
| Live provider/model scientific quality | Actual configured-provider investigations with retained requests/results, cost/usage, failure analysis and scientific review. |
| Independent expert and researcher usability | Representative people completing realistic research tasks; assessment of interpretations, provenance use, recovery, time and error rates. |
| Native Windows | Executed installation, provider/executor, generated-code, reporting, storage and recovery workflows on the supported Windows environment. |
| Applicable production containment | Deployed isolation, permissions, quotas, concurrency, interruption and operational recovery evidence for the intended host configuration. |
| Applicable production BYOS | Actual configured external-storage integrity, authorization, lifecycle and recovery evidence; local connector fixtures alone are insufficient. |
No external gate is closed merely because the application implements a control or its documentation describes it. Measured performance remains specific to the recorded workload and runtime; historical dense-view timings do not qualify autonomous investigation cost or latency.