> ## Documentation Index
> Fetch the complete documentation index at: https://docs.eu.linqalpha.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Get LLM Judge Evaluation

> Returns HTTP 200 in every state, including while still running. Branch on `status`, not on the status code. The response shape is identical across statuses — the same seven fields are always present.

## What it does

Returns the current state of one LLM Judge run, using the `evaluation_id` from
[Execute LLM Judge](/api-reference/evaluations/create_llm). The response shape is **identical in
every state** — the same six fields are always present — so a client reads `status` and never has
to branch on the body's shape.

<Warning>
  This returns HTTP `200` while the judge is still running. Branch on the `status` field, not on the
  status code.
</Warning>

## Statuses

| `status`    | Meaning                      | `verdict`                                                                                                             | `judge_model`              | `input_reference_count` / `verified_reference_count` | Retry?       |
| ----------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------------- | -------------------------- | ---------------------------------------------------- | ------------ |
| `pending`   | Queued or running.           | `null`                                                                                                                | `null`                     | `null`                                               | Keep polling |
| `completed` | Finished.                    | Fixed-rubric verdict — see below                                                                                      | The model that produced it | Both non-null                                        | No           |
| `failed`    | Something broke on our side. | Same shape as `completed`, but `scores` / `overall_score` are `null` and `reasoning` carries a customer-safe sentence | `null`                     | `null`                                               | Yes          |

A `GET` immediately after a `POST` returns `pending`. That is expected, not an error.

<Note>
  Failure detection: `verdict !== null && verdict.overall_score === null` is the failure
  marker. `reasoning` on such a verdict is the customer-safe sentence that used to be
  surfaced as a top-level `reason` field — that field was removed to keep the shape uniform.
</Note>

<Note>
  Unlike [Get Agent Judge Evaluation](/api-reference/evaluations/get), this endpoint does **not**
  currently return `excluded`. The LLM Judge has no exclusion pre-filter — the same status enum is
  exposed for consistency, but only `pending` / `completed` / `failed` are produced today.
</Note>

## Polling

Poll no more than **once every 10 seconds**. Typical runs settle in a few minutes; a sensible
client gives up after around 30 minutes and treats the run as failed.

```python theme={null}
import time
import requests

url = f"https://api.eu.linqalpha.com/v2/judge/llm/{evaluation_id}"
headers = {"X-API-KEY": "<your-api-key>"}

deadline = time.time() + 30 * 60          # give up after ~30 minutes
while time.time() < deadline:
    payload = requests.get(url, headers=headers).json()["payload"]

    if payload["status"] == "completed":
        print(payload["verdict"])
        # Coverage signal — see the note below.
        print(f"verified {payload['verified_reference_count']}/{payload['input_reference_count']}")
        break
    if payload["status"] == "failed":
        # `verdict.overall_score is None` is the failure marker; `reasoning` is the
        # customer-safe reason that used to live under a separate `reason` field.
        print("failed:", payload["verdict"]["reasoning"])   # safe to retry
        break

    time.sleep(10)                        # still pending
```

## Response

`verdict` uses one shape on every non-null run — four dimensions on a 1–5 scale, plus `reasoning`
and a server-computed `overall_score`. `completed` populates all three; `failed` uses the same
shape with `scores` / `overall_score` set to `null` and the customer-safe reason on `reasoning`.
Every successful LLM Judge run therefore returns scores directly comparable across callers and
time, and a caller reads one field to tell success from failure.

### `completed`

```json theme={null}
{
  "error": null,
  "payload": {
    "evaluation_id": "2fde560a-e7eb-45e0-8cd3-04e57c50d1d3",
    "status": "completed",
    "verdict": {
      "scores": {
        "Factuality": 4,
        "Completeness": 4,
        "Relevance": 5,
        "Grounding": 5
      },
      "reasoning": "The answer's core claims align with the supplied sources...",
      "overall_score": 4.5
    },
    "judge_model": "gpt-5.4-mini",
    "input_reference_count": 2,
    "verified_reference_count": 2
  }
}
```

### `failed`

```json theme={null}
{
  "error": null,
  "payload": {
    "evaluation_id": "2fde560a-e7eb-45e0-8cd3-04e57c50d1d3",
    "status": "failed",
    "verdict": {
      "scores": null,
      "reasoning": "The evaluation could not be completed.",
      "overall_score": null
    },
    "judge_model": null,
    "input_reference_count": null,
    "verified_reference_count": null
  }
}
```

## The response fields

| Field                      | Type           | Notes                                                                                                      |
| -------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------- |
| `evaluation_id`            | UUID           | The id from `POST /v2/judge/llm`.                                                                          |
| `status`                   | enum           | `pending` / `completed` / `failed`.                                                                        |
| `verdict`                  | object \| null | Non-null on `completed` **and** `failed`. See the two response examples above.                             |
| `judge_model`              | string \| null | The model that produced the verdict. Non-null only on `completed`.                                         |
| `input_reference_count`    | int \| null    | How many references you submitted. Non-null only on `completed`.                                           |
| `verified_reference_count` | int \| null    | How many references the verifier actually produced a verdict for. Non-null only on `completed`. See below. |

<AccordionGroup>
  <Accordion title="verified_reference_count vs input_reference_count — read them together">
    On a clean run the two are equal.

    When they differ (`verified_reference_count < input_reference_count`), a chunk of the verifier
    failed and only the surviving verdicts reached the judge — the verdict is still valid,
    but partly graded on incomplete verification. Downstream you may want to weigh those
    runs differently or resubmit.

    On non-`completed` statuses both are `null`.
  </Accordion>

  <Accordion title="Empty references — both counts are 0">
    Submitting `references: []` is a valid request; the verifier is skipped and the judge grades
    on `prompt` / `query` / `answer` only. In this case `input_reference_count` and
    `verified_reference_count` are both `0`, not `null`.
  </Accordion>
</AccordionGroup>

<Note>
  Per-reference verification verdicts, the evidence behind them, and internal cost/latency accounting
  are used to produce the verdict but are not part of this response.
</Note>

## Isolation

Judge runs are scoped to the organization that submitted them. An `evaluation_id` belonging to
another organization returns `404`, exactly as an id that does not exist — the two are
indistinguishable by design.

The two endpoints — this one and [Get Agent Judge Evaluation](/api-reference/evaluations/get) —
read from **disjoint** id spaces. An id from `POST /v2/judge/agent` returns `404` here, and vice versa.
Idempotency keys are scoped the same way, so the same key on the two endpoints yields two separate
evaluations.

A malformed `evaluation_id` is rejected with `400` before any lookup.


## OpenAPI

````yaml GET /v2/judge/llm/{evaluation_id}
openapi: 3.0.1
info:
  title: LinqAlpha API
  description: >-
    Linq helps finance professionals make informed decisions using
    Retrieval-Augmented Generation (RAG)-enhanced answers. By leveraging
    cutting-edge Large Language Models (LLM) and supplementary technology, Linq
    provides the most optimized responses based on your queries.
  version: 1.0.0
  license:
    name: MIT
servers:
  - url: https://api.eu.linqalpha.com
security:
  - ApiKeyAuth: []
tags:
  - name: Search
    description: Search and generate responses
  - name: Data
    description: Data retrieval and mapping
  - name: Feedback
    description: Conversation feedback
  - name: RMS
    description: Research Management System
  - name: Source Management
    description: Source batch and file management
  - name: MCP
    description: >-
      LinqAlpha MCP — Financial data tools for AI assistants via Model Context
      Protocol
  - name: Connectors
    description: Customer Connectors — customer-owned MCP connector management
  - name: Briefing
    description: Briefing Agent — automated market briefings with scheduling and delivery
  - name: Status
    description: Sync status — check organization, document, and container sync progress
paths:
  /v2/judge/llm/{evaluation_id}:
    get:
      tags:
        - Evaluations
      summary: Retrieve an LLM Judge evaluation
      description: >-
        Returns HTTP 200 in every state, including while still running. Branch
        on `status`, not on the status code. The response shape is identical
        across statuses — the same seven fields are always present.
      parameters:
        - name: evaluation_id
          in: path
          required: true
          description: >-
            The id returned by `POST /v2/judge/llm`. Ids from `POST
            /v2/judge/agent` return `404` here — the two endpoints read from
            disjoint id spaces.
          schema:
            type: string
            format: uuid
          example: 2fde560a-e7eb-45e0-8cd3-04e57c50d1d3
      responses:
        '200':
          description: The evaluation's current state. Same shape in every status.
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    nullable: true
                    example: null
                  payload:
                    $ref: '#/components/schemas/JudgeLlmResult'
              example:
                error: null
                payload:
                  evaluation_id: 2fde560a-e7eb-45e0-8cd3-04e57c50d1d3
                  status: completed
                  verdict:
                    scores:
                      Factuality: 4
                      Completeness: 4
                      Relevance: 5
                      Grounding: 5
                    reasoning: >-
                      The answer's core claims align with the supplied
                      sources...
                    overall_score: 4.5
                  judge_model: gpt-5.4-mini
                  input_reference_count: 2
                  verified_reference_count: 2
        '400':
          description: Malformed `evaluation_id`.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ApiErrorResponse'
              example:
                error:
                  code: INVALID_REQUEST_BODY
                  msg: evaluation_id must be a valid UUID
                  message: evaluation_id must be a valid UUID
                payload: null
        '401':
          description: Missing or invalid API key.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ApiErrorResponse'
              example:
                error:
                  code: UNAUTHORIZED
                  msg: Invalid API key
                  message: Invalid API key
                payload: null
        '404':
          description: >-
            No such evaluation in your organization. An id owned by another
            organization is indistinguishable from one that does not exist.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ApiErrorResponse'
              example:
                error:
                  code: JUDGE_EVALUATION_NOT_FOUND
                  msg: Evaluation not found
                  message: Evaluation not found
                payload: null
        '429':
          description: Organization rate limit exceeded.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ApiErrorResponse'
              example:
                error:
                  code: RATE_LIMIT_EXCEEDED
                  msg: Rate limit exceeded
                  message: Rate limit exceeded
                payload: null
        '502':
          description: Upstream failure while loading the evaluation.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ApiErrorResponse'
              example:
                error:
                  code: JUDGE_EVALUATION_FAIL
                  msg: Failed to load the evaluation
                  message: Failed to load the evaluation
                payload: null
        '504':
          description: Timed out loading the evaluation.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ApiErrorResponse'
              example:
                error:
                  code: JUDGE_EVALUATION_FAIL
                  msg: Evaluation lookup timed out
                  message: Evaluation lookup timed out
                payload: null
components:
  schemas:
    JudgeLlmResult:
      type: object
      required:
        - evaluation_id
        - status
        - verdict
        - judge_model
        - input_reference_count
        - verified_reference_count
      properties:
        evaluation_id:
          type: string
          format: uuid
        status:
          type: string
          enum:
            - pending
            - completed
            - excluded
            - failed
          description: >-
            `pending` means queued or running. The LLM judge does not currently
            produce `excluded`; the enum is shared with `POST /v2/judge/agent`
            for consistency.
        verdict:
          allOf:
            - $ref: '#/components/schemas/JudgeLlmDefaultVerdict'
          nullable: true
          description: >-
            The structured verdict. Non-null on both `completed` and `failed`.
            `completed` populates `scores` / `overall_score`; `failed` sets both
            to `null` and carries the customer-safe reason on `reasoning`.
            `pending` and `excluded` return `null`.
        judge_model:
          type: string
          nullable: true
          description: The model that produced the verdict. Non-null only on `completed`.
        input_reference_count:
          type: integer
          nullable: true
          description: >-
            How many references the request submitted. Non-null only on
            `completed`. `0` when `references: []` was sent.
        verified_reference_count:
          type: integer
          nullable: true
          description: >-
            How many references the verifier actually produced a verdict for.
            Equal to `input_reference_count` on a clean run; strictly less when
            a verifier chunk failed and only the surviving verdicts reached the
            judge. Non-null only on `completed`.
    ApiErrorResponse:
      type: object
      description: Standard error response wrapper
      properties:
        error:
          $ref: '#/components/schemas/ApiError'
        payload:
          description: Always null for error responses
          nullable: true
      required:
        - error
        - payload
    JudgeLlmDefaultVerdict:
      type: object
      description: >-
        The verdict shape returned on `completed` and `failed`. On `completed`,
        `scores` and `overall_score` are populated (fixed rubric: Factuality /
        Completeness / Relevance / Grounding, each 1-5; `overall_score` is the
        arithmetic mean, computed server-side). On `failed`, both `scores` and
        `overall_score` are `null` and `reasoning` carries a customer-safe
        failure sentence — that same sentence used to be surfaced as a separate
        top-level `reason` field, which was removed. Callers do not customise
        this shape.
      required:
        - scores
        - reasoning
        - overall_score
      properties:
        scores:
          nullable: true
          type: object
          properties:
            Factuality:
              type: integer
              minimum: 1
              maximum: 5
            Completeness:
              type: integer
              minimum: 1
              maximum: 5
            Relevance:
              type: integer
              minimum: 1
              maximum: 5
            Grounding:
              type: integer
              minimum: 1
              maximum: 5
          additionalProperties: false
          description: Per-dimension integer scores 1-5. `null` on `failed`.
        reasoning:
          type: string
          description: >-
            Short justification for the dimension scores on `completed`; on
            `failed` this is the customer-safe reason the evaluation did not
            complete.
        overall_score:
          nullable: true
          type: number
          description: >-
            Arithmetic mean of the dimension scores. Computed server-side on
            `completed`; `null` on `failed` — a caller can treat `overall_score
            === null` as the failure marker.
    ApiError:
      type: object
      properties:
        code:
          type: string
          description: >-
            Error code indicating the type of error. Common codes:

            - Authentication: `API_KEY_MISSING`, `INVALID_API_KEY`

            - Validation: `INVALID_REQUEST_BODY`, `ORGANIZATION_ID_MISSING`,
            `TICKERS_MISSING`, `CHAT_MESSAGE_ID_MISSING`, `DOCUMENT_ID_MISSING`

            - Service: `SEARCH_FAIL`, `TTS_FAIL`, `CREATE_CONV_FAIL`,
            `GET_STOCK_FAIL`, `CREATE_MSG_FAIL`, `ALPHA_COMP_FAIL`,
            `GET_REF_FAIL`

            - Connectors: `CONNECTOR_LIST_FAIL`, `CONNECTOR_NOT_FOUND`,
            `CONNECTOR_CREATE_FAIL`, `CONNECTOR_UPDATE_FAIL`,
            `CONNECTOR_DELETE_FAIL`, `CONNECTOR_TEST_FAIL`

            - Not Found: `NOT_FOUND`
          example: INVALID_REQUEST_BODY
        msg:
          type: string
          description: Error message (deprecated, use `message` instead)
          example: query is required and must be a string
        message:
          type: string
          description: Error message providing more details about the error
          example: query is required and must be a string
      required:
        - code
  securitySchemes:
    ApiKeyAuth:
      type: apiKey
      in: header
      name: X-API-KEY

````