Evaluators
Invoke a Custom Evaluator
Runs an evaluator that already exists in the workspace. Accepts either a conversation or the structured input and output fields; when both are present the conversation wins.
POST
Related guide: Evaluators guide. See the Evaluators guide for a walkthrough with examples.
timeout.call_timeout to control the timeout in milliseconds, up to 600000. The default is 120000 milliseconds unless configured otherwise. This uses the same request shape as the Responses API. Synchronous calls execute directly in graders-api and stop when the request deadline expires. Runtime failures return an error response; a valid negative verdict returns passed: false.
For LLM-as-a-judge Evaluators (llm_eval), set timeout.call_timeout to limit each provider attempt in milliseconds. Set retry.count to the number of retries after the initial attempt, from 0 to 5, and retry.on_codes to the HTTP status codes that allow a retry. Omitting on_codes uses the gateway’s default transient codes. Omitting retry preserves the existing retry policy. For LLM judges the timeout applies to each provider attempt; for other Evaluators it applies to execution. Caller cancellation still stops synchronous work, and the server caps each invocation at ten minutes including retries.
timeout.call_timeout is set, the gateway also includes HTTP 408 among the retryable codes, matching the Responses API.Authorizations
Body
application/json
Accepts two shapes. context names its fields after the template variables
they feed and is the one to use; the flat fields below are folded into
context when it is absent. Setting context wins.
Response
OK
The verdict. Its shape is fixed so existing consumers read the same JSON.