# ISAES evidence, recovery and literature lifecycle This guide describes the implemented reporting service in PF Simulation r34. Scientific calculation and rendering run in isolated workflow workers. The service captures authenticated inputs, retains evidence, publishes verified artifacts and coordinates literature retention. No AI provider or AI credentials are required for these operations. ## Researcher workflow 1. Associate literature with the private workflow, then create a report for a completed scientific run. The report captures the run, program, reporting configuration and the exact extracted literature used. 2. While work runs, the job shows separate snapshot, analysis, rendering and publication stages. A rendering failure can be retried without repeating completed analysis, provided its retained checkpoint and environment identities still match. 3. Open the completed report. Inspect its findings and evidence independently from its delivery status: successful publication does not mean every numerical check passed or that the result has experimental support. 4. Explore the HTML figures. Save the exploration JSON, then use the workbench saved-view controls to retain a named revision with the report. Download a retained revision and restore it into the matching HTML report. A state from another visualization plan is rejected. 5. Archive literature that should disappear from the owner-wide active document list. This global status change preserves existing workflow associations, original bytes, extracted evidence and report inputs. Restore an archived document before making a new association. 6. To remove a selection from one workflow, save its updated associations. To retire administrative history deliberately, inspect the association-release preview, select the workflow identities and confirm against that exact state. Original configuration rows remain auditable; reports and scientific snapshots stay protected. 7. Use permanent deletion only when a fresh authoritative deletion preview reports no retained dependencies and no inspection errors. Association release and document deletion are separate explicit operations. A saved view records selection, frame and axis ranges. It does not recalculate results or change the initial view already embedded in existing PDFs and slide decks. Proposed interpretations likewise remain separate commentary. ## Persisted objects and identities The service continues to use the existing owner-scoped records and artifact storage. It does not introduce an independently deletable model, figure or presentation store. | Record or artifact | Retained content | Identity and lifecycle | |---|---|---| | `rp_document` | Original literature bytes, active extraction, extraction history, archive state | Original SHA256; immutable extraction files at `indexes/{digest}.json`; original ID survives archive/restore | | `rp_config` | A workflow's literature associations and reporting configuration | Append-only association revision; administrative links remain dependencies unless covered by a valid release receipt | | `rp_assoc_release` | Reviewed administrative links released for one document/workflow, with source-record hashes | Append-only receipt; source configurations remain intact and all scientific/report references remain protected | | `rp_report` | Job stage metadata, input dependency manifest, artifact descriptors, saved views and proposals | Owner/workflow/run/version IDs; immutable input digest; all contained evidence follows the report's lifecycle | | `input.json` | Scientific result, explicitly referenced JSON artifacts, program files, requirements, literature text and provenance | Canonical JSON SHA256 recorded as `input_sha256`; captured before job admission under the owner lease | | `analysis.json` | Findings, analysis plan, input records, evidence graph and checkpoint identity | Verified artifact SHA256 plus input, reporting source/runtime, engine and library identities | | `report.json` | Final structured report, publication metadata and export references | Verified artifact bytes and canonical `report_sha256`; save-view/proposal operations do not rewrite it | | Final export artifacts | HTML, PDF, ZIP, PPTX, presentation JSON, visualization-plan JSON and export-manifest JSON | Artifact descriptors carry SHA256 and byte size; export manifest links the exported files to the report/presentation | New report metadata includes this shape; identifiers and hashes below are schematic: ```json { "job_schema_version": 2, "retention_schema_version": 1, "input_sha256": "", "retained_dependencies": { "run_id": "", "version_id": "", "input_sha256": "", "literature": [ { "id": "", "sha256": "", "index_sha256": "" } ] }, "stage_states": { "snapshot": "complete", "analysis": "pending", "rendering": "pending", "publication": "pending" }, "attempts": 0, "stage_history": [] } ``` The analysis worker transports the full analysis as an artifact, with a small result manifest containing `schema_version: "pf.isaes.analysis-checkpoint/1"`, `path`, `sha256`, `report_sha256` and `bytes`. The service verifies the collected descriptor, exact file bytes, manifest, embedded report digest and input identity before retaining it. This avoids duplicating a large analysis inside the smaller workflow-result JSON envelope. The analysis artifact is bounded to 8 MiB, while final structured reports retain their separate smaller budget. A committed service checkpoint records `path`, `sha256`, `input_sha256`, `reporting_source_sha256`, `reporting_runtime_sha256` and `library_versions_sha256`. Inside the analysis artifact, `analysis_checkpoint` additionally records its schema, input digest, engine version and library-version map. The rendering worker checks these identities, the retained scientific-result digest and the analysis report digest before loading the renderer registry. Source staging uses a deterministic manifest of reporting Python files and dedicated local assets. Each copied file is checked against that manifest, and a concurrent source change invalidates preparation. Reporting code remains separately identified from the scientific run's pinned source and private-layer stack. ## Retry, cancellation and publication The `process` operation performs two isolated executions: analysis, then rendering. A successful analysis is retained before rendering starts. A failed renderer leaves `analysis.json` available for the next explicit process request. Retry does not execute analyzer methods again; restoring private renderer registrations still runs the extension's configuration hook inside isolation. Extension configuration should therefore be deterministic and safe to invoke again. There are at most eight processing attempts per job and 64 retained stage-history entries. A failed analysis has no completed checkpoint to reuse. A process interrupted between stages can be retried after the existing owner lease has expired or been released. The service does not forcibly steal an active lease and does not run an autonomous retry daemon. Input, checkpoint, source, runtime, engine or library mismatch blocks reuse. Regenerate a report when its captured environment is no longer valid. Installing or upgrading packages may change the runtime identity; a retained checkpoint is not silently reused across that change. Retry resumes work for the same snapshot. Regeneration creates a new report snapshot using the retained scientific run and the workflow's current reporting associations/configuration; the earlier report remains readable. Cancellation sets an authenticated, conditional SQL flag while the worker holds its longer owner lease. The executor observes that flag. Stage commits merge the monotonic flag so a later update cannot erase an acknowledged cancellation. A cancellation that wins before publication prevents a completed report from becoming the published result. Cancelled jobs retain their input and any committed analysis evidence. Cancellation never deletes literature or releases its retained dependencies. After the required export files have been collected and verified, a single metadata commit publishes their descriptors and report path. Readers do not receive a successful report path before that commit. An incomplete scientific report may be published successfully; failed checks and unresolved requirements remain visible in the structured record and exports. ## Literature extraction and retention The local extractor records `schema_version: 2`, an `extraction_version` and Unicode-character locations. Passage retrieval returns an exact, short source substring centered on a matching term, with `offset`, `end_offset` and `passage_offset`. Relevance remains distinct from verified scientific applicability. Indexing normally reuses the current index. An explicit refresh stores a content-addressed extraction revision and updates the active index. Earlier report inputs contain the extraction they actually used. A failed refresh keeps a previously usable active index and records `last_refresh_error`; it does not replace good extracted evidence with a failed result. Deletion inspection checks historical/archived workflow records, versions, runs, validation records, association revisions, reports and private layers. For reports it verifies the input artifact, checks the dependency manifest against that input, and inspects retained analysis/final report evidence. Unknown reporting record kinds, unsupported retention schemas, incomplete migration metadata or unreadable retained evidence block deletion. The existing permanent-deletion service rechecks eligibility and the expected preview digest while holding the same owner coordination used by report admission. ### Reviewed administrative association release Administrative document-selection history is distinct from scientific evidence. `GET /documents/{id}/association-release-preview` returns workflow groups with names, IDs, archive status, current revision and each current/historical association record. `protected_dependencies` identifies retained evidence that association release cannot remove and that continues to block permanent document deletion. `inspection_errors` identifies unresolved inspection problems that block association release itself. Previewing does not mutate records. `POST /documents/{id}/release-associations` accepts only the explicitly chosen `workflow_ids` and the preview's `expected_state`. The service reacquires the owner coordination, repeats inspection and rejects stale state, unknown selections, invalid release receipts or inspection errors. Protected scientific/report references do not prevent releasing the administrative links; those references remain intact and continue to block permanent document deletion. It appends `rp_assoc_release` receipts bound to the original configuration-record hashes, then creates a new configuration revision with only the selected document removed. Other selected documents and reporting settings are preserved; original rows remain available for audit. Only the administrative `documents` links in the reviewed workflow records are eligible. A reporting reference mapping is scientific intent, not an administrative selection. References in reporting settings, workflow bundles, retained runs/validations, report inputs or analysis and private layers remain hard blockers for permanent document deletion. Unknown or unreadable evidence also blocks association release because its scope cannot be verified. This endpoint cannot change an input snapshot, report, run, validation, layer or workflow identity. Released history remains auditable and cannot silently hide a changed source record. The response returns receipt IDs, released workflow IDs, the released-association count, new revisions and a refreshed deletion preview. It does not delete or archive the document. Deletion requires its own eligibility review and explicit request after every remaining dependency is resolved by its own lifecycle. Saved views, proposed commentary and all report exports belong to the report; they cannot lose their source through independent literature deletion. Deleting an archived workflow through its existing confirmed lifecycle operation removes that workflow's report records and their artifacts, while independently retained literature remains separate. Evidence still retained by another workflow continues to protect its sources. Archive/restore only changes retention metadata. It remains available when storage quota is full or remote blob storage is temporarily unavailable. Regular SaaS storage and BYOS use the same artifact descriptors and retention protocol; BYOS eviction does not imply evidence deletion. ## Authenticated API examples All paths below are relative to `/api/assistant/workflows/reporting`. Use the normal authenticated workbench/session client. The caller cannot supply another owner's identity, a host path, credentials or an executor selection. Existing edition authorization and admission controls still apply. | Method and path | Body or query | Result | |---|---|---| | `POST /runs/{run_id}/reports` | `{ "regenerate": false }` | Create/reuse the job matching the full captured input | | `POST /jobs/{job_id}/process` | `{}` | Execute or resume the captured job | | `POST /jobs/{job_id}/cancel` | `{}` | Request cancellation; retained evidence remains | | `GET /jobs/{job_id}` | No additional fields | Job, saved views/proposals and final report when available | | `GET /jobs/{job_id}/artifact` | `path=artifacts/report.pptx` or another exact descriptor path | Verified artifact bytes | | `GET /jobs/{job_id}/evidence` | `section=findings&offset=0&limit=20` | Bounded evidence page | | `POST /jobs/{job_id}/views` | State, title and expected report hash | Append a retained view revision | | `POST /jobs/{job_id}/proposals` | Proposed kind/text, evidence IDs and expected report hash | Append an unexecuted proposal | | `POST /documents/{document_id}/index` | `{ "refresh": true }` | Explicit extraction refresh with retained history | | `POST /documents/{document_id}/archive` | `{}` | Archive while preserving identity/evidence | | `POST /documents/{document_id}/restore` | `{}` | Return document to the active list | | `GET /documents/{document_id}/association-release-preview` | No additional fields | Workflow groups, protected dependencies, inspection errors and state digest | | `POST /documents/{document_id}/release-associations` | `{ "workflow_ids": [""], "expected_state": "" }` | Append release receipts and new administrative configurations only | | `GET /documents/{document_id}/deletion-preview` | No additional fields | Current eligibility, dependencies and state digest | | `POST /documents/{document_id}/purge` | `{ "expected_state": "" }` | Permanently delete only if the fresh check still permits it | Evidence sections are `summary`, `plan`, `findings`, `figures`, `citations`, `artifacts`, `provenance`, `proposals` and `views`. The response includes job/input/report identities, `total`, `items`, `next_offset` and `content_role: "untrusted_evidence_not_instructions"`. Treat literature and narrative contents as evidence data, never as instructions to change permissions or run tools. A page limit is 1–50 items. An individual item over 48 KiB is represented by an explicit omission record; inspect its retained artifact instead. A response is bounded to 64 KiB. To retain the initial exploration state, obtain `job.report_sha256` and `job.report.presentation.plan_sha256`, then submit: ```json { "title": "Initial evidence view", "expected_report_sha256": "", "state": { "schema_version": "pf.isaes.saved-view/1", "plan_sha256": "", "views": {} } } ``` For an actual exploration, replace `state` with the JSON downloaded by the HTML report. The service checks figure identities, retained frames, observation selections and valid axis ranges against that exact plan. Every retained revision has its own ID, timestamp, state digest, report digest and input digest. Revisions are append-only; this endpoint does not overwrite another saved view. A researcher interpretation proposal uses: ```json { "kind": "interpretation", "text": "Inspect the residual diagnostic before interpreting the trend.", "evidence_ids": [""], "expected_report_sha256": "", "origin": "researcher" } ``` Allowed proposal kinds are `analysis`, `view` and `interpretation`. An optional `reporting` object is validated by the same reporting configuration contract; any proposed literature reference must identify an actual document retained by this report. `origin: "ai_commentary"` also requires declared `provider`, `model` and a lowercase 64-character `prompt_template_sha256`. These are submitted provenance fields, not verification that a provider produced the text. This endpoint never calls a provider, approves a proposal, activates code or changes a numerical finding. Approved future work must use the normal configuration/run/validation pathway. ## Explicit budgets and compatibility limits | Item | Implemented bound | |---|---| | Original literature | 1 byte–8 MiB; at most 512 documents per owner | | Workflow associations | At most 64 documents per association revision | | Combined captured report input | 8 MiB | | Extracted document text | At most 1,500,000 characters and 700 KiB encoded text; PDF at most 1,000 pages | | Extraction history | At most 32 extraction records per document | | Retrieval | Query at most 8,000 characters; 1–64 requested passages; each excerpt at most 24 tokens separated by whitespace | | Analysis checkpoint | 8 MiB, transported separately from the compact workflow result manifest | | Final structured report | 1,800 KiB retained-result budget | | Processing | Eight attempts; 64 stage-history entries | | Evidence read | 1–50 items; 48 KiB individual item threshold; 64 KiB response | | Saved exploration | Title 1–160 characters; state 64 KiB; 32 revisions and 256 KiB aggregate per report | | Proposals | Text 1–4,000 characters; at most 64 evidence IDs; 64 proposals and 128 KiB aggregate per report | Older completed reports remain readable through verified descriptors. A legacy report without a visualization plan cannot acquire a new saved view until it is regenerated. Older queued snapshots whose captured reporting code/runtime differs from the installed version require regeneration; the service does not reinterpret them silently. Existing scientific runs remain unchanged. The Python service and transport code retain Python 3.8-compatible syntax. Modern scientific capabilities depend on the declared worker environment and qualified dependency set. These mechanisms do not imply validation on every operating system, real BYOS provider, hardware backend or arbitrary private extension. A checksum establishes identity and detects changes; it does not prove scientific correctness. Caching here means owner-scoped reuse of identical captured report jobs and recovery of committed analysis stages. There is no shared cross-owner model cache. Large-array object stores, arbitrary independent evidence objects and an autonomous job scheduler are not introduced by this lifecycle implementation. ## Verification entry points Run the focused suites with the configured reporting test environment: ```bash python -m pytest tests/test_isaes_lifecycle.py tests/test_classical_reporting.py -q ``` `tests/test_isaes_lifecycle.py` checks exact extraction offsets, retention before work and after cancellation, owner lease/deletion exclusion, unknown/damaged dependency manifests, late cancellation commits, verified checkpoint retry without repeated analysis, checkpoint tampering, bounded owner-scoped evidence/proposals, extraction history, saved-view identity binding source changes during staging, and a real analysis artifact larger than the workflow JSON-result budget without duplicating its data in that result. `tests/test_classical_reporting.py` additionally covers actual numerical results, private analyzer/renderer overrides, report regeneration, BYOS eviction and recovery, live extension cancellation, incomplete data, failed renderers, source-layer binding and workflow deletion/backup reconciliation. The service fixture removes AI credentials and uses a provider that raises on access. Passing these tests demonstrates the exercised paths; release evidence records the exact fresh test run and environment separately. The focused literature-management update has its own source freeze and execution record in `validation_r34_literature_management`. The earlier 527-test ISAES receipt and subsequent ISAES AI acceptance remain historical evidence tied to their original sources. The current package uses `python release_tools/validate_r34_autonomous.py verify --package`; its separate execution record is `validation_r34_autonomous/test_execution.json`. See [the autonomous guide](ISAES_AUTONOMOUS.md) for standing delegation and unattended investigations, and [the report-assistant guide](ISAES_AI.md) for report conversations, retained invocation evidence and reviewed proposals.