Run configuration files

A configuration freezes the goal, pass conditions, budget, and authorized tools for one study. Desktop creates the same kind of “research contract”; headless users can write it directly as YAML or JSON.

A minimal runnable configuration

title: "Damped oscillator check"
goal:
  description: "Confirm that the reference damped oscillator has quality factor >= 3"

acceptance:
  metric: quality_factor
  operator: ">="
  value: 3.0

budget:
  max_actions: 3
  max_experiments: 1
  max_failures: 1

plugins:
  - examples/plugins/oscillator_quality

capabilities:
  - tutorial.oscillator.evaluate

research_questions:
  - "What is the quality factor of the reference oscillator?"

The repository includes this file as examples/plugin_tutorial.yaml. Run it with:

scientific-agent run examples/plugin_tutorial.yaml

Required content

Explicitly list capabilities whenever possible. It is the run's capability allowlist. If omitted, all capabilities from the selected plugins are allowed for backward compatibility.

Acceptance criteria

A criterion consists of a metric, comparator, and value. Supported comparators are >=, >, <=, <, and ==.

acceptance:
  metric: balanced_accuracy
  operator: ">="
  value: 0.90
  require_trusted: true

With require_trusted: true, only a trusted evaluator may satisfy the criterion. This prevents training code from grading itself.

For several criteria, acceptance must equal the first item in acceptance_criteria, and metric names must be unique:

acceptance:
  metric: balanced_accuracy
  operator: ">="
  value: 0.90
  require_trusted: true

acceptance_criteria:
  - metric: balanced_accuracy
    operator: ">="
    value: 0.90
    require_trusted: true
  - metric: inference_seconds
    operator: "<="
    value: 2.0

Budgets

You do not need to specify every limit. Omitted optional limits are unbounded for that resource. Defaults are max_actions: 10, max_failures: 3, and max_repair_attempts: 1.

budget:
  max_actions: 12
  max_experiments: 8
  max_failures: 3
  max_wall_time: 3600
  max_tokens: 200000
  max_cpu_hours: 4
  max_gpu_hours: 1
  max_training_runs: 3
  max_training_samples: 50000
  max_solver_calls: 6
  max_reasoner_calls: 12
  max_reasoner_tokens: 240000
  max_model_calls: 20
  max_model_tokens: 300000
  max_code_generation_calls: 4
  max_code_repair_calls: 3
  max_repair_attempts: 1

The Control Room shows remaining budget, and an audited intervention can adjust it. To continue from BUDGET_EXHAUSTED, increase an exhausted limit or explicitly make the experiment limit unlimited. The Runtime never silently overspends.

Reasoner

A plugin may supply a deterministic reasoner, so tutorials and regression runs need no network access:

reasoner:
  type: deterministic

To let an LLM plan the study, name its provider and model:

reasoner:
  type: llm
  provider: openai
  model: YOUR_MODEL_ID
  temperature: 0.2
  max_output_tokens: 24000
  timeout_seconds: 180
  max_provider_retries: 1
  max_schema_repairs: 1
  prompt_version: scientific-reasoner-v1

The CLI options --reasoner, --provider, --model, and --temperature can temporarily override these reasoner settings.

Providers and secrets

Put credentials in environment variables, never in YAML:

export OPENAI_API_KEY="your key"

providers:
  openai:
    api_format: openai_responses
    auth_env: OPENAI_API_KEY
    structured_output_mode: json_schema

An OpenAI-compatible endpoint can be declared like this:

providers:
  lab_gateway:
    api_format: openai_chat_completions
    base_url_env: LAB_MODEL_BASE_URL
    auth_env: LAB_MODEL_API_KEY
    structured_output_mode: tool_call

API formats are openai_responses, openai_chat_completions, and anthropic_messages. Structured-output modes are json_schema, json_object, and tool_call; the Anthropic format accepts only json_schema or tool_call. A base_url must be an absolute HTTP(S) URL and may instead come from base_url_env.

Route models by role

One model is enough for a simple study. To control scientific reasoning, code generation, and code repair separately, define profiles:

models:
  profiles:
    reasoner:
      provider: codex-cli
      model: YOUR_REASONING_MODEL
      structured_output: true
    coder:
      provider: lab_gateway
      model: YOUR_CODING_MODEL
      coding: true
    repairer:
      provider: lab_gateway
      model: YOUR_REPAIR_MODEL
      coding: true
      fallback_profiles: [coder]

  routing:
    strategy: role_separated
    scientific_reasoning: reasoner
    code_generation: coder
    code_repair: repairer
    max_fallback_depth: 1

Routing strategies are strong_only, role_separated, and repair_escalation. These settings affect new runs only; an existing run keeps its original contract.

Capability constraints

Constraints can freeze evaluation data, sampling rules, or solver parameters so that the reasoner cannot move the benchmark during a run:

capability_constraints:
  beam.xsuite.generate_stability_dataset:
    - when:
        dataset_role: evaluation
      require:
        num_turns: 128
        dataset_role: evaluation
        evaluation_id: beam_survival_eval_v1

When a call matches when, the Runtime requires its inputs to satisfy require. Constraint fields are plugin-specific; inspect a capability in Desktop or at GET /api/v1/capabilities/{id} first.

Scientific workflow and final predictor

To require both passing metrics and complete scientific deliverables:

workflow:
  scientific_verification:
    enabled: true
  scientific_completion_contract:
    enabled: true
    required_artifacts:
      - scientific_report
      - verification_result

To require a frozen file-based predictor:

final_predictor:
  required: true
  interface: file-predictor-v1
  input_filename: predictor_input.json
  output_filename: predictor_output.json

Input and output names must differ and must be plain file names. A plugin must actually produce a conforming model package; enabling the switch does not generate one by itself.

Check a configuration

A reliable check sequence is:

1. Preview the research contract in Desktop, or call POST /runs/preview through the Control API.
2. Confirm that every acceptance metric appears in a capability's metric_outputs.
3. Start with a small budget and inspect scientific-agent status RUN_ID --events.

Parsing rejects unsupported comparators, invalid limits, embedded secrets, missing plugins, and invalid model routes. YAML and JSON expose the same fields.