Launching an environment

The server inside the image is a single binary. Your startup command names it and passes every option the task needs. There is no configuration file and no environment variable to set: the startup command that shipped with the environment is the whole configuration, and the values below come from that command rather than from you.

docker run -i --rm \
  --cpus 4 --memory 16g \
  --cap-add=SYS_PTRACE \
  --cap-add=SYS_ADMIN \
  --cap-add=NET_ADMIN \
  --security-opt seccomp=unconfined \
  --security-opt apparmor=unconfined \
  --network=none \
  --entrypoint bash \
  IMAGE \
  -c "/rlenv/mcp/server --tools=setup_problem,grade_problem,reproduce --grade-dimension=4 --weight-defect-matching=0.6 --weight-pov-quality=0.2 --weight-summary-quality=0.1 --weight-target-calls=0.1 --max-attempts=20 --output-directory=/tmp/output"

-i is required: the server speaks the protocol over standard input and output, so the container must keep its stdin open for the whole run. --rm removes the container when the server exits — normally, when your MCP client closes the pipe. Your client speaks to the container by spawning docker run as a subprocess and reading and writing its stdio directly. --network=none is not optional either: a graded episode with egress is a side channel, and Host requirements says why. See One episode, step by step for the sequence to run over it.

The one mistake to avoid

Pass the whole startup command as a single string to bash -c. The --tools= flag inside it is the entire agent contract — the server registers only the tools it names. If that flag doesn't reach the server intact, the server falls back to exposing all of its internal tools (bash, read_file, edit_file, verify, triage, validate, on top of the real ones), and the run is silently misconfigured.

Two ways to lose the flag:

1. Splitting the startup command into separate docker arguments. The image's own entrypoint is bash -c, so appending docker run IMAGE /rlenv/mcp/server --tools=... becomes bash -c "/rlenv/mcp/server" "--tools=..." inside the container — bash treats the first argument as its script and every remaining argument as a positional parameter ($0, $1, ...). The server starts with no options at all.
2. Falling back to the image's default CMD. Some images ship a partial default; treat the metadata's startup_command as authoritative.

Either failure passes the protocol handshake and hands the model tools it should never see. One episode, step by step step 4 is the tripwire: assert the server's advertised tool list equals the --tools= set on every launch.

A server started without an option set is not a supported mode

If the server is reached without a tool list, it exposes every operation it has, including two that exist for our own quality checks and must never be reachable by a model. That is a misconfigured launch, not a permissive default, and any score produced by such an episode is void. Assert the tool list, every time.

A reference implementation

scripts/examples/run_environment.py is the whole thing end to end: launch the container, run the agent loop, grade, tear down. One file, two dependencies. It reads two files:

• The problems metadata JSON (--metadata, required) — the per-image image, startup_command, required_tools, resource requirement, and problem id. This is the file that ships with your environments; the runner does not synthesise any of it.
• A provider/limits TOML (--config, optional) — model provider, model id, turn/time caps, docker resource caps. Everything else has a built-in default.

It is one file, but it is laid out as the components an orchestrator is made of — the protocol client, the model adapter, the episode controller, the transcript — with the same names and the same typed structures our own driver uses, so you can map it onto whatever you already have and replace one piece. The piece most people replace first is the model adapter: implement four methods and the rest stays put.

Two things in it are worth reading even if you write your own harness. It routes every agent tool call to the server rather than reaching into the container from outside, so the agent gets the persistent shell, the isolation and the tool timeout the environment intends. And it holds setup_problem and grade_problem apart from the model's tool surface in one place, refusing them at dispatch as well as hiding them from the tool list — because the server registers all of them on the same session and cannot tell your calls from your agent's.

Start with --check, which launches the environment, asserts the tool list and prints the task text, then exits. No model and no API key, so it tells you whether the environment is sound before you spend anything on an agent.

The same script drives Claude, an OpenAI-compatible endpoint (OpenAI itself, Azure, Ollama, vLLM, Together, Groq, Fireworks, LM Studio, …), or a new provider you add a shim for.

Requires Python 3.11+, docker on your PATH, pip install "mcp>=1.9", and the SDK for the provider you chose (anthropic or openai). Export that provider's API key before running.

python scripts/examples/run_environment.py <problem-id> \
    --metadata problems-metadata.json \
    [--config my-provider.toml] \
    [--transcript run.json] \
    [--artifacts-dir out/] \
    [--check]

scripts/examples/README.md documents the config sections and command-line flags in full.