> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. # Create an Evaluator > Create a new evaluator in the workspace. **Related guide**: Evaluators guide. See the [Evaluators guide](/ai-studio/optimize/evaluators) for a walkthrough with examples. ## OpenAPI ````yaml post /v2/evaluators openapi: 3.1.0 info: title: orq.ai API version: '2.0' description: orq.ai API documentation servers: - url: https://my.orq.ai security: - ApiKey: [] tags: - name: Chunking description: Split text into smaller chunks for retrieval and generation workflows. - name: File Systems description: >- Create and manage persistent file systems that agents and MCP clients read from and write to. - name: Knowledge Bases description: Create and manage knowledge bases used by agents and retrieval workflows. - name: Memory Stores description: Create and manage memory stores, memories, and memory documents. - name: Evals description: Run an evaluator against a conversation and its result - name: Logs description: >- OpenTelemetry log query API. Search, filter, aggregate, and facet log records ingested via OTLP. - name: Reporting description: >- GenAI reporting API over canonical analytics rollups. Accepts a metric name, time range, grain, group-by, and filters; returns a typed time series and optional totals. - name: Traces description: >- Query and inspect ingested trace data: search trace summaries, aggregate metrics, and read individual traces and their spans. - description: List models available through the AI Router. name: Models - name: Policies - name: Alerts description: >- Alerts evaluate a Reporting API metric on a fixed interval and fire notifications through notifiers when the value breaches a threshold. Each breach opens a trigger that tracks the incident until the value recovers. - name: Annotation Queues description: Annotation queues collect spans for human review. - name: API keys description: >- API keys authenticate programmatic access to the workspace. They expose opaque tokens, per-domain access grants, and budget and rate-limit constraints. - name: Audit Logs description: Audit logs record workspace entity changes and access-relevant events. - name: Budgets description: >- Budgets govern spend, token usage, and request rate across six scopes: workspace, project, identity, API key, provider, and model. Every applicable budget is enforced, and the most restrictive limit applies per dimension. - name: Files description: File upload and retrieval operations. - name: Guardrail Rules description: >- Guardrail Rules conditionally enforce evaluators and plugins for AI Gateway traffic. Rules may be scoped to a project or the whole workspace. - name: Hub description: Hub items are reusable templates available to a workspace. - name: Identities description: >- Identities represent end users from your system for usage and engagement tracking. - name: Management keys description: >- Management keys are workspace-scoped credentials that authenticate programmatic access to workspace administration surfaces (API keys, budgets). Unlike project-scoped API keys, a management key always operates at the workspace level. - name: MCP Gateway description: >- Register upstream MCP servers, discover and sync their tools, and assemble gateways that expose a curated tool surface to MCP clients. - name: Model Catalog description: >- Browse the orq.ai model catalog: every model orq offers, across every provider, with pricing, capabilities and benchmark data. List endpoints only return models that are not deprecated. This API is public, requires no authentication, and is rate limited to 120 requests per minute per IP. Responses carry a 5-minute cache-control max-age. - name: Notifiers description: Notifier destinations used to send delivery and workflow notifications. - name: Projects description: Projects organize resources within a workspace - name: Routing Rules description: >- Routing Rules conditionally select models and enforce request plugins for AI Gateway traffic. Rules are evaluated by ascending priority and may be scoped to a project or the whole workspace. - name: Threads description: Threads group related trace invocations and their aggregate usage - name: Skills description: >- Skills are modular instructions you can use to codify processes and conventions - name: Smart Routers description: >- Create and manage workspace Smart Routers. A Smart Router selects a model from an eligible pool for each request according to a quality, balanced, or cost profile. - name: Webhooks description: >- Create and manage webhooks that deliver workspace events to external HTTPS endpoints. - name: Workspaces description: >- A workspace is the tenant. Create is called from a user session during onboarding; Get, List, and Update are the public management surface. - name: Workspace Security description: >- Workspace-level domain verification and IP allowlist controls. These operations are restricted to workspace administrators. - name: Workspace Settings description: >- Workspace-level settings managed with a workspace credential. A workspace is the tenant, so these settings are a singleton — there is nothing to create or delete, only read and update. - name: Responses - description: Run agents on a cron cadence. Minimum firing interval is 1 hour. name: Agent Schedules - name: Embeddings - name: Telemetry description: >- Unified query envelope for traces, metrics, and logs. One request shape, one filter dialect, and one response shape per source, validated by a per-source registry. - description: Beta. Run typed classification questions against a classify model. name: Classify - description: Search Gateway with managed credits or BYOK. name: Web Search externalDocs: url: https://docs.orq.ai description: orq.ai Documentation paths: /v2/evaluators: post: tags: - Evals summary: Create an Evaluator description: Create a new evaluator in the workspace. operationId: CreateEval requestBody: content: application/json: schema: oneOf: - oneOf: - type: object properties: guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator type: type: string enum: - llm_eval repetitions: type: - integer - 'null' minimum: 1 maximum: 3 prompt: type: string categories: items: type: string type: - array - 'null' categorical_labels: type: - array - 'null' items: type: object properties: value: type: string minLength: 1 description: type: string required: - value dataset_id: type: - string - 'null' path: type: string description: >- Legacy alternative to `project_id`. Storage path whose first segment names the project that owns the evaluator. Mutually exclusive with `project_id`. example: Default project_id: type: string minLength: 1 description: >- Unique identifier of the project that owns the evaluator, as returned by `GET /v2/projects`. Mutually exclusive with `path`. example: 01JMDPA3QW5C1V0NJ1PW34T4E5 description: type: string default: '' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ mode: type: string enum: - single model: type: string required: - type - prompt - key - mode - model title: LLM - type: object properties: guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator type: type: string enum: - llm_eval repetitions: type: - integer - 'null' minimum: 1 maximum: 3 prompt: type: string categories: items: type: string type: - array - 'null' categorical_labels: type: - array - 'null' items: type: object properties: value: type: string minLength: 1 description: type: string required: - value dataset_id: type: - string - 'null' path: type: string description: >- Legacy alternative to `project_id`. Storage path whose first segment names the project that owns the evaluator. Mutually exclusive with `project_id`. example: Default project_id: type: string minLength: 1 description: >- Unique identifier of the project that owns the evaluator, as returned by `GET /v2/projects`. Mutually exclusive with `path`. example: 01JMDPA3QW5C1V0NJ1PW34T4E5 description: type: string default: '' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ mode: type: string enum: - jury jury: type: object properties: judges: type: array items: type: object properties: model: type: string retry: type: object properties: count: type: integer minimum: 1 maximum: 5 default: 2 on_codes: type: array items: type: integer minimum: 100 maximum: 599 minItems: 1 default: - 429 - 500 - 502 - 503 - 504 fallbacks: type: array items: type: object properties: model: type: string required: - model default: [] required: - model minItems: 2 replacement_judges: type: array items: type: object properties: model: type: string retry: type: object properties: count: type: integer minimum: 1 maximum: 5 default: 2 on_codes: type: array items: type: integer minimum: 100 maximum: 599 minItems: 1 default: - 429 - 500 - 502 - 503 - 504 fallbacks: type: array items: type: object properties: model: type: string required: - model default: [] required: - model default: [] min_successful_judges: type: integer minimum: 2 default: 2 tie_value: type: string enum: - Tie default: Tie required: - judges required: - type - prompt - key - mode - jury title: LLM Jury - type: object properties: guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any output_type: type: string enum: - boolean - number code: type: string type: type: string enum: - python_eval path: type: string description: >- Legacy alternative to `project_id`. Storage path whose first segment names the project that owns the evaluator. Mutually exclusive with `project_id`. example: Default project_id: type: string minLength: 1 description: >- Unique identifier of the project that owns the evaluator, as returned by `GET /v2/projects`. Mutually exclusive with `path`. example: 01JMDPA3QW5C1V0NJ1PW34T4E5 description: type: string default: '' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - code - type - key title: Python responses: '200': description: Successfully created an evaluator content: application/json: schema: oneOf: - $ref: '#/components/schemas/EvaluatorResponseLlm' - $ref: '#/components/schemas/EvaluatorResponseJsonSchema' - $ref: '#/components/schemas/EvaluatorResponseHttp' - $ref: '#/components/schemas/EvaluatorResponsePython' - $ref: '#/components/schemas/EvaluatorResponseFunction' - $ref: '#/components/schemas/EvaluatorResponseRagas' - $ref: '#/components/schemas/EvaluatorResponseTypescript' discriminator: propertyName: type mapping: llm_eval: '#/components/schemas/EvaluatorResponseLlm' json_schema: '#/components/schemas/EvaluatorResponseJsonSchema' http_eval: '#/components/schemas/EvaluatorResponseHttp' python_eval: '#/components/schemas/EvaluatorResponsePython' function_eval: '#/components/schemas/EvaluatorResponseFunction' ragas: '#/components/schemas/EvaluatorResponseRagas' typescript_eval: '#/components/schemas/EvaluatorResponseTypescript' '403': description: Domain is not allowed '404': description: Workspace ID is not found on the request content: application/json: schema: type: object properties: message: type: string required: - message x-code-samples: - lang: typescript label: Node.js - Create an evaluator source: | import { Orq } from '@orq-ai/node'; const orq = new Orq({ apiKey: process.env.ORQ_API_KEY }); const evaluator = await orq.evals.create({ type: 'python_eval', key: 'answer_quality', path: 'Default', code: 'def evaluate(**kwargs):\n return True', outputType: 'boolean', }); console.log(evaluator.id); components: schemas: EvaluatorResponseLlm: type: object properties: _id: type: string description: type: string created: type: string default: '2026-08-05T15:12:51.947Z' updated: type: string default: '2026-08-05T15:12:51.947Z' updated_by_id: type: - string - 'null' project_id: type: string description: Unique identifier of the project owning this evaluator. guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any type: type: string enum: - llm_eval output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator repetitions: type: - integer - 'null' minimum: 1 maximum: 3 prompt: type: string categories: items: type: string type: - array - 'null' categorical_labels: type: - array - 'null' items: type: object properties: value: type: string minLength: 1 description: type: string required: - value dataset_id: type: - string - 'null' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ mode: type: string enum: - single - jury model: type: string jury: type: object properties: judges: type: array items: type: object properties: model: type: string retry: type: object properties: count: type: integer minimum: 1 maximum: 5 default: 2 on_codes: type: array items: type: integer minimum: 100 maximum: 599 minItems: 1 default: - 429 - 500 - 502 - 503 - 504 fallbacks: type: array items: type: object properties: model: type: string required: - model default: [] required: - model minItems: 2 replacement_judges: type: array items: type: object properties: model: type: string retry: type: object properties: count: type: integer minimum: 1 maximum: 5 default: 2 on_codes: type: array items: type: integer minimum: 100 maximum: 599 minItems: 1 default: - 429 - 500 - 502 - 503 - 504 fallbacks: type: array items: type: object properties: model: type: string required: - model default: [] required: - model default: [] min_successful_judges: type: integer minimum: 2 default: 2 tie_value: type: string enum: - Tie default: Tie required: - judges required: - _id - description - type - output_type - prompt - key - mode title: LLM EvaluatorResponseJsonSchema: type: object properties: _id: type: string description: type: string created: type: string default: '2026-08-05T15:12:51.947Z' updated: type: string default: '2026-08-05T15:12:51.947Z' updated_by_id: type: - string - 'null' project_id: type: string description: Unique identifier of the project owning this evaluator. guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any type: type: string enum: - json_schema output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator schema: type: string key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - type - output_type - schema - key title: JSON EvaluatorResponseHttp: type: object properties: _id: type: string description: type: string created: type: string default: '2026-08-05T15:12:51.947Z' updated: type: string default: '2026-08-05T15:12:51.947Z' updated_by_id: type: - string - 'null' project_id: type: string description: Unique identifier of the project owning this evaluator. guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any type: type: string enum: - http_eval output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator url: type: string method: type: string enum: - GET - POST headers: type: object additionalProperties: type: string payload: type: object additionalProperties: {} key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - type - output_type - url - method - headers - payload - key title: HTTP EvaluatorResponsePython: type: object properties: _id: type: string description: type: string created: type: string default: '2026-08-05T15:12:51.947Z' updated: type: string default: '2026-08-05T15:12:51.947Z' updated_by_id: type: - string - 'null' project_id: type: string description: Unique identifier of the project owning this evaluator. guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any code: type: string type: type: string enum: - python_eval output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - code - type - output_type - key title: Python EvaluatorResponseFunction: type: object properties: _id: type: string description: type: string created: type: string default: '2026-08-05T15:12:51.947Z' updated: type: string default: '2026-08-05T15:12:51.947Z' updated_by_id: type: - string - 'null' project_id: type: string description: Unique identifier of the project owning this evaluator. guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any type: type: string enum: - function_eval output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator function_params: oneOf: - type: object properties: type: type: string enum: - contains value: type: string required: - type - value - type: object properties: type: type: string enum: - contains_none keywords: type: array items: type: string required: - type - keywords - type: object properties: type: type: string enum: - contains_all keywords: type: array items: type: string required: - type - keywords - type: object properties: type: type: string enum: - contains_any keywords: type: array items: type: string required: - type - keywords - type: object properties: type: type: string enum: - contains_email required: - type - type: object properties: type: type: string enum: - contains_url required: - type - type: object properties: type: type: string enum: - contains_valid_link required: - type - type: object properties: type: type: string enum: - start_with value: type: string required: - type - value - type: object properties: type: type: string enum: - ends_with value: type: string required: - type - value - type: object properties: type: type: string enum: - exact_match required: - type - type: object properties: type: type: string enum: - length_less_than value: type: number required: - type - value - type: object properties: type: type: string enum: - length_between min: type: number max: type: number required: - type - min - max - type: object properties: type: type: string enum: - length_greater_than value: type: number required: - type - value - type: object properties: type: type: string enum: - one_line required: - type - type: object properties: type: type: string enum: - regex pattern: type: string required: - type - pattern - type: object properties: type: type: string enum: - is_valid_json required: - type - type: object properties: type: type: string enum: - moderations_openai required: - type - type: object properties: type: type: string enum: - moderations_google required: - type - type: object properties: type: type: string enum: - bert_score required: - type - type: object properties: type: type: string enum: - bleu_score required: - type - type: object properties: type: type: string enum: - rouge_n required: - type - type: object properties: type: type: string enum: - meteor_score required: - type - type: object properties: type: type: string enum: - cosine_similarity required: - type - type: object properties: type: type: string enum: - levenshtein_distance required: - type - type: object properties: type: type: string enum: - keywords_match keywords: type: array items: type: string required: - type - keywords - type: object properties: type: type: string enum: - most_repeated_words required: - type - type: object properties: type: type: string enum: - flesch_reading_ease required: - type - type: object properties: type: type: string enum: - gse_english_level required: - type - type: object properties: type: type: string enum: - words_count required: - type - type: object properties: type: type: string enum: - sentences_count required: - type - type: object properties: type: type: string enum: - lexical_repetition required: - type - type: object properties: type: type: string enum: - grammar_diversity required: - type key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - type - output_type - function_params - key title: Function EvaluatorResponseRagas: type: object properties: _id: type: string description: type: string created: type: string default: '2026-08-05T15:12:51.947Z' updated: type: string default: '2026-08-05T15:12:51.947Z' updated_by_id: type: - string - 'null' project_id: type: string description: Unique identifier of the project owning this evaluator. guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any type: type: string enum: - ragas output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator ragas_metric: type: string enum: - context_precision - context_recall - context_entities_recall - harmfulness - maliciousness - coherence - correctness - conciseness - response_relevancy - faithfulness - summarization - noise_sensitivity key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ model: type: string required: - _id - description - type - output_type - ragas_metric - key - model title: Ragas EvaluatorResponseTypescript: type: object properties: _id: type: string description: type: string created: type: string default: '2026-08-05T15:12:51.947Z' updated: type: string default: '2026-08-05T15:12:51.947Z' updated_by_id: type: - string - 'null' project_id: type: string description: Unique identifier of the project owning this evaluator. guardrail_config: anyOf: - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - type - value title: Boolean - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - type - values title: Categorical - type: object properties: enabled: type: boolean default: true alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - type - value - operator title: Number - type: 'null' x-speakeasy-type-override: any code: type: string type: type: string enum: - typescript_eval output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - code - type - output_type - key title: Typescript securitySchemes: ApiKey: type: http scheme: bearer bearerFormat: JWT ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.