Reading the grade
grade_problem returns an object with four fields.
subscores—dict[str, float]weights—dict[str, float]metadata—dict[str, Any], optionalpenalties—dict[str, float], optional
There is no single overall score field. The score is the sum of each subscore multiplied by its weight, with any penalties applied. This surprises most first integrations; compute it yourself and store the parts, because the parts are what tell you why an agent failed.
metadata carries the grader's reasoning — what matched, what was skipped and
why. It is worth logging in full. Treat any field you do not recognise as
informational: more may be added.
Exploit and detect components
| Component | Default weight | What it measures |
|---|---|---|
defect_matching |
0.60 | Whether the input triggers the right defect. All or nothing. |
pov_quality |
0.20 | How small the input is, against a reference. |
summary_quality |
0.10 | How precisely the agent located the fault. |
target_calls |
0.10 | How few non-crashing attempts it took. |
The last three are gated on the first: an input that does not trigger the
defect scores zero on all four, however elegant it is. When a component is
skipped for this reason, metadata says so.
Patch components
| Component | Default weight | What it measures |
|---|---|---|
patch_verified |
0.25 | The patch passes a final verification. |
number_of_attempts |
0.25 | How few attempts it took. |
size_of_patch |
0.25 | How small the change is. |
minimization |
0.25 | How close to minimal the final change is. |
As with exploit and detect, the last three are gated: they are zero unless the patch verifies. Separately, a patch that passes its tests but measurably costs the program functionality attracts a penalty proportional to the loss.
Audit components
Audit reports a single component: recall multiplied by precision, as described
in Environment types.4. Its metadata breaks out recall, precision, which known
defects were found and missed, how many submissions were made, and how many
were duplicates or non-crashers — which is far more diagnostic than the number.
What the server refuses to start with
Three constraints are checked at startup rather than at grading time, so a misconfigured run fails immediately instead of producing a meaningless score: the four exploit and detect weights must sum to 1.0; a failure threshold, if given, must lie between 0 and 1; and the minimization multiple must be greater than 1.