Benchmark Suite

The Benchmark Suite answers evaluation questions such as whether a Scientific Reasoner is stable under a frozen protocol, generalizes to unseen data, and is being tested at meaningful difficulty. It is separate from the ordinary Runtime: a normal study does not know it is under evaluation and cannot see private holdouts.

You do not need Benchmark to complete an ordinary study. Use it when comparing reasoners, model-routing choices, or regressions across versions.

Prepare the environment

Confirm the command is available:

scientific-agent-bench --help

The beam ML protocols require Xsuite and PyTorch. For a source checkout:

python -m pip install -e '.[beam-ml]'

To use a real LLM through the OpenAI API as well:

python -m pip install -e '.[beam-ml-llm]'

Configure the selected provider or complete codex login before running. Start with one small ordinary run to verify plugins and models before spending the full benchmark budget.

Repeat one configuration

The simplest evaluation creates isolated trials from one configuration:

scientific-agent-bench \
  --data-dir .bench-data \
  run examples/beam_ml_llm.yaml \
  --runs 3 \
  --output reports/repeated.json

Use it to inspect:

• success rate and failure categories;
• final trusted metrics per run;
• action, experiment, token, and time usage;
• whether each trial truly used an independent run directory.

Without --output, the report is written to <data-dir>/benchmarks/latest-report.json. The command returns a successful exit code only when there is a valid successful result.

Generalization protocol

The repository includes a runnable frozen specification at examples/beam_ml_benchmark.yaml:

scientific-agent-bench \
  --data-dir .bench-generalization \
  generalization examples/beam_ml_benchmark.yaml \
  --output reports/generalization.json \
  --markdown reports/generalization.md

It:

1. freezes an unseen Halton holdout before research begins;
2. runs studies at several training-sample budgets;
3. freezes the final predictor from each successful run;
4. performs holdout inference only after research is over;
5. compares a fixed 9-NN baseline and ablations;
6. audits private-data leakage, research freezing, and run isolation.

In-study and holdout metrics are not interchangeable. Focus on the generalization gap, trend across budgets, baseline deltas, and integrity status—not only the highest score.

Difficulty protocol

The difficulty protocol adds a boundary-enriched holdout, physics-parameter shifts, replications, and confidence intervals. Prepare private data first:

scientific-agent-bench \
  --data-dir .bench-difficulty \
  difficulty benchmarks/beam_ml_v07.yaml \
  --prepare-only

Review the generated freeze information, then run:

scientific-agent-bench \
  --data-dir .bench-difficulty \
  difficulty benchmarks/beam_ml_v07.yaml \
  --output reports/difficulty.json \
  --markdown reports/difficulty.md

Running the same spec again resumes incomplete replicate slots and reuses completed preparation that still verifies. It does not rerun completed slots merely for convenience.

Override model settings

All three subcommands accept Reasoner overrides similar to Runtime:

scientific-agent-bench run examples/beam_ml_llm.yaml \
  --runs 2 \
  --reasoner real \
  --provider codex-cli \
  --model YOUR_MODEL_ID \
  --temperature 0.2

The override is system-under-test metadata. It does not alter frozen datasets, goals, budgets, or the protocol hash. If a specification explicitly freezes provider/model identity, a conflicting setting is rejected or invalidates the evaluation.

Read a report

For every report, check:

• expected benchmark_version and specification hash;
• Runtime, Benchmark Suite, and prompt versions;
• valid, failed, recovered, and skipped run counts;
• trusted metrics and confidence intervals;
• baseline, ablation, boundary, and physics-shift results;
• all integrity, leakage, and freeze checks;
• original run IDs for event review in Control Room or CLI.

If integrity fails, do not present the benchmark as valid, regardless of its numerical score.

Keep comparisons fair

• Compare reasoners with the same frozen spec and fresh output roots.
• Do not repeatedly tune ordinary research against one private holdout result.
• Never copy benchmark-private data into Runtime inputs.
• When reporting a mean or median, retain individual runs and failure rates.
• Treat small-sample confidence intervals as uncertainty, not decoration.

Legacy scientific-agent benchmark* commands still forward for compatibility but print a deprecation warning. New scripts should use scientific-agent-bench so removal of the compatibility layer does not break them.