Scientific workflow and results

Scientific Agent is designed to leave checkable support for every research step, rather than merely asking a model for a long final answer. A run begins with a research contract and ends with metrics, artifacts, and complete provenance.

How a study advances

1. The research contract freezes the goal, acceptance criteria, budget, inputs, and allowed capabilities.
2. The Scientific Reasoner proposes the next step from current state, such as running an experiment, analyzing results, or finishing.
3. A plugin executes a named capability and returns metrics, observations, and file artifacts.
4. The Runtime persists immutable artifacts and events, then updates budget and criteria progress.
5. If criteria are not met and budget remains, the Reasoner continues from the new evidence. The run succeeds only after all criteria and deliverable requirements are met.

“Immutable” means a new result does not overwrite an old artifact. A better model, revised hypothesis, or repeated evaluation creates a new version and preserves the relationship between them.

Experiments, metrics, and observations

A successful plugin call commonly returns:

• Metrics: numbers evaluated by acceptance criteria, such as balanced_accuracy or quality_factor.
• Observations: structured summaries for people and the Reasoner.
• Artifacts: datasets, models, predictions, analyses, logs, figures, or reports.

Passing a metric does not necessarily make the study complete. If accuracy has passed but the contract also requires a verification result and scientific report, the run continues until those artifacts exist.

In the Control Room, check:

• current status and terminal failures under State;
• metric sources and remaining limits under Metrics and Budget;
• deliverables under Artifacts/Reports and in Lineage;
• execution and provider activity under Jobs and Model calls.

Analysis

scientific.analysis.analyze turns a dataset, trained model, and evaluation result into a framework-neutral analysis artifact. Public predictions and evaluation features enable additional threshold sweeps, regional performance, and spatial outcome analysis.

Analysis cannot read evaluator-private labels and does not replace trusted evaluation. Its role is to explain recorded results and provide a stable input for evidence extraction.

Query it through the API with:

curl -sS "http://127.0.0.1:8000/api/v1/runs/$RUN_ID/analyses" | jq

Evidence and claims

The Scientific Knowledge plugin turns result files into traceable knowledge:

• Evidence records an observation, source artifacts, confidence, scope, and limitations.
• A claim must reference supporting evidence and retains its own scope and limitations.

For example, “error rate is substantially higher in the boundary region” can be evidence extracted from an analysis. “The model needs more training samples near the decision boundary” is a supported claim that can still be tested or refuted.

The rule-based knowledge capabilities do not call another LLM and do not invent conclusions from nothing. Inspect their source chain in the Knowledge inspector or the /evidence and /claims endpoints.

Hypotheses and versions

A hypothesis contains a statement, supporting claims, confidence, and an explicit verification contract. Revising it creates a new version with a supersedes relationship instead of overwriting history.

A useful hypothesis includes:

• a falsifiable statement;
• its conditions and scope;
• supporting claims or evidence;
• observations that would support or reject it.

Submit one through a Scientific Command, or let a Reasoner schedule it when the workflow is enabled:

{
  "command_type": "SubmitHypothesis",
  "statement": "Boundary-focused samples improve boundary accuracy",
  "confidence": 0.65,
  "claim_refs": ["ARTIFACT_ID"],
  "conditions": [
    {"metric": "group.boundary.accuracy", "operator": ">=", "value": 0.85}
  ]
}

Verification plans and results

Verification has an explicit approval boundary:

1. Create a tool-independent verification plan for an existing hypothesis.
2. Review its objective, parameters, expected observations, and success conditions.
3. Explicitly approve the plan.
4. Request an authorized plugin to execute the verification experiment.
5. Record experiment artifacts and condition-by-condition outcomes as a verification result.
6. Update the hypothesis status as a new version.

The ordinary research loop requests verification actions only when enabled:

workflow:
  scientific_verification:
    enabled: true

Approval does not magically map an abstract experiment to any plugin. The run must already authorize a capability that can perform it. Otherwise, it stops with a clear missing-capability state for human review.

Trusted evaluation

A trusted evaluator differs from an ordinary research plugin. It may access evaluation-private targets, but those private artifacts never enter Reasoner context, normal artifact listings, or research-flow exports.

When a criterion contains:

require_trusted: true

an ordinary plugin cannot satisfy it by reporting a metric with the same name. A typical model-selection flow is:

1. A research plugin produces public evaluation inputs and predictions.
2. A trusted evaluator resolves matching private targets by evaluation_id.
3. It emits provenance-backed trusted metrics and an evaluation artifact.
4. The Criteria Engine decides whether the criterion passed.

Scientific reports

scientific.report.generate creates a Markdown report from the artifact graph and can add independent SVG figures. It cites existing experiment, dataset, model, evaluation, and analysis artifacts; it does not rerun the model or evaluator.

When reviewing a report, verify that:

• the goal and criteria match the research contract;
• each important number is traceable to an artifact;
• limitations and failed attempts remain visible;
• figures are independent files, not only screenshots embedded in a page.

When is a run complete?

Success requires all applicable conditions:

• every acceptance metric;
• every trusted-source requirement;
• the frozen predictor contract, if enabled;
• the scientific artifact-completeness contract, if enabled.

If the Reasoner proposes DONE while required artifacts are missing, the Runtime rejects early completion. Budget exhaustion, missing capability, pause, cancellation, and execution failure each have distinct states; none is dressed up as “success with a warning.”

Successful research state is frozen. To continue exploring, copy its configuration into a new run rather than editing old events.

Deliver results

There are two common handoff formats:

• A scientific export is reviewer-oriented and includes a report, model, and necessary metadata for a successful result.
• A Runtime backup is recovery-oriented and contains machine state and all run data for migration or disaster recovery.

They serve different purposes. An export is not a complete backup, and a backup is not a reader-ready final report.