One episode, step by step

An episode is nine steps. Steps 4 and 8 are yours to make; the rest are protocol.

1. Launch the container as in Launching an environment.
2. Open a protocol session over the container's stdin and stdout.
3. Initialise the session and wait for the acknowledgement.
4. Ask for the tool list, and assert it matches what you passed. If it does not, abort: this is the failure described in Launching an environment.1.
5. Call setup_problem. It returns the task text to give your agent.
6. Give that text to the agent, together with the operations from The tool surface.
7. Let the agent work, calling its action operation as its budget allows.
8. Decide the episode is over: the agent submitted, or you hit a limit.
9. Call grade_problem and record what it returns.

orchestrator                    environment
     |                               |
     |-- initialize ---------------->|
     |<-- capabilities --------------|
     |-- initialized (notify) ------>|
     |-- list tools ---------------->|
     |<-- tool list -----------------|   assert: equals what you passed
     |-- setup_problem ------------->|
     |<-- task text -----------------|
     |                               |
     |   ( agent loop: bash, and one action operation )
     |                               |
     |-- grade_problem ------------->|
     |<-- grade ---------------------|

The invariants worth asserting, by step:

• 3, initialise — the session negotiates protocol version 2024-11-05.

• 4, list tools — the returned set equals the set you passed. Anything else is a misconfigured launch.

• 5, setup_problem — called exactly once, before any action operation. Calling it again for the same task is harmless.

• 7, action operation — only the one belonging to this type.

• 9, grade_problem — called last. If you call it more than once, the last call is the authoritative one.