BugcrowdBugcrowdSavant Environments

Custom

Name the server binary yourself and the startup command becomes the whole configuration — which tools the agent can reach, how the attempt is graded, how many attempts it gets. There is no configuration file and no environment variable to set.

docker run -d -i \
  --cap-add=SYS_PTRACE \
  --cap-add=SYS_ADMIN \
  --cap-add=NET_ADMIN \
  --security-opt seccomp=unconfined \
  --security-opt apparmor=unconfined \
  --network=none \
  --entrypoint /rlenv/mcp/server \
  inventory.forallsecure.com/<environment-name>:<tag> \
  --tools=setup_problem,grade_problem,reproduce \
  --max-attempts=20 \
  --output-directory=/tmp/output

The options above are the shape of the command, not a recommendation. The weights and the attempt budget belong to whoever is running the evaluation.

Where to go from here

  • Launching an environment — the full command with every option spelled out, and the one mistake to avoid: appending the startup command to the image's own entry point instead of passing --entrypoint, which silently starts the server with no options at all.
  • Option reference — every option, grouped by what it affects, and which ones are yours to change.
  • Sizing, timeouts and limits — the --cpus and --memory floors to pass through. Each environment declares a shape; treat it as a minimum rather than a suggestion.
  • Reading the grade — what grade_problem returns and how to read it.