CLI command reference

Runtime and Benchmark Suite use separate entry points:

scientific-agent        ordinary research runs and local Control API
scientific-agent-bench  external frozen-protocol evaluation

Global storage options

Runtime global options must precede the subcommand:

scientific-agent \
  --data-dir PATH \
  --cache-dir PATH \
  --config-dir PATH \
  <command>

scientific-agent new

Create the contract and RunCreated event without execution:

scientific-agent new CONFIG [REASONER_OPTIONS]

On success it emits creation logs and then the Run ID on its own line. This supports a two-stage script: create and retain the ID, review, then resume.

scientific-agent run

Create and immediately execute:

scientific-agent run CONFIG [REASONER_OPTIONS]

CONFIG may be .yaml, .yml, or .json. Final SUCCESS returns 0; another terminal or recoverable state returns 2 and prints:

DONE run_id=run_... status=<STATUS>

Reasoner overrides

Shared by new and run:

Example:

export OPENAI_API_KEY=...
scientific-agent run examples/beam_ml_llm.yaml \
  --reasoner real \
  --provider openai \
  --model <verified-model-id> \
  --temperature 0.2

Angle brackets denote a placeholder. Use a model ID currently available to the provider/account and compatible with the required structured-output mode.

scientific-agent resume

scientific-agent resume RUN_ID

Runtime replays the event stream and continues the same contract. It can:

• start a CREATED run made by new;
• continue PAUSED, BLOCKED, CAPABILITY_MISSING, or budget-adjusted BUDGET_EXHAUSTED runs;
• recognize a crash-stale RUNNING state and enter interrupted recovery;
• reuse committed source, model calls, and checkpoints instead of regenerating them.

SUCCESS, FAILED, and CANCELLED cannot continue. Resume uses the persisted plugin, provider, model, and capability contract and accepts no temporary model override.

scientific-agent status

scientific-agent status RUN_ID
scientific-agent status RUN_ID --events

JSON includes goal, status, metrics, budget usage, Reasoner/provider/model configuration, failures, experiment/job counts, latest compact result, and research-visible artifact references. --events adds sequence, timestamp, event type, and causing action without large artifact bytes or evaluator-private data.

For scripts:

scientific-agent status "$RUN_ID" | jq -r '.status'
scientific-agent status "$RUN_ID" | jq '.metrics, .budget_usage'

scientific-agent serve

Start the local Control API and bundled React UI:

scientific-agent serve \
  --host 127.0.0.1 \
  --port 8000 \
  --plugin-dir ./plugins

• --host defaults to 127.0.0.1; development mode is unauthenticated, so keep loopback.
• --port defaults to 8000.
• --plugin-dir is repeatable and names server-approved plugin roots. If omitted, a plugins/ directory under the current working directory is used when present.
• UI: http://127.0.0.1:8000/
• OpenAPI: http://127.0.0.1:8000/api/v1/docs

For browser-free operation see Headless Control API.

Benchmark commands

Benchmark global --data-dir defaults to .scientific-agent and also precedes its subcommand.

scientific-agent-bench run

scientific-agent-bench --data-dir .bench-data run CONFIG \
  --runs 3 \
  --output report.json \
  [REASONER_OPTIONS]

Runs independent Scientific Reasoner trials and emits a machine-readable report. Without output, the report is <data-dir>/benchmarks/latest-report.json.

scientific-agent-bench generalization

scientific-agent-bench generalization SPEC \
  --output benchmark-report.json \
  --markdown benchmark-report.md \
  [REASONER_OPTIONS]

Runs the frozen v0.6 protocol: after research success, unseen holdout data tests the frozen predictor and results are compared across baselines/budgets.

scientific-agent-bench difficulty

scientific-agent-bench difficulty SPEC --prepare-only
scientific-agent-bench difficulty SPEC \
  --output benchmark-report.json \
  --markdown benchmark-report.md \
  [REASONER_OPTIONS]

--prepare-only freezes private datasets and protocol identities without starting LLM runs. Re-running the same specification resumes unfinished slots and does not repeat completed research or holdout work.

Historical scientific-agent benchmark, benchmark-generalization, and benchmark-difficulty remain deprecated console-wrapper routes. New scripts should always use scientific-agent-bench.

Exit codes and errors

Errors go to stderr and normally start with error:. Automation should check both the process exit code and persisted run status, not only the last output line.