> ## Documentation Index
> Fetch the complete documentation index at: https://docs.eu.linqalpha.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Execute LLM Judge
> Records the submission and queues it. Returns immediately; the judgement itself runs asynchronously and is retrieved with `GET /v2/judge/llm/{evaluation_id}`. Unlike `POST /v2/judge/agent` (Agent Judge), which returns a written assessment, this endpoint produces a **structured verdict** — dimension scores plus reasoning — on a fixed rubric (Factuality / Completeness / Relevance / Grounding, each 1-5) with a server-computed `overall_score`. Every run uses the same rubric so runs are comparable across callers and time.
## What it does
Submits an answer plus your judge prompt and the sources you want it graded against, and returns
an `evaluation_id` immediately. The judge verifies the sources within the selected source scope,
then a deterministic LLM grades the answer on a **fixed rubric** — Factuality / Completeness /
Relevance / Grounding, each 1–5 — and returns a **structured verdict** (dimension scores plus a
short reasoning, with a server-computed `overall_score` mean).
Unlike [Execute Agent Judge](/api-reference/evaluations/create), which returns a **written assessment**
against primary sources, this endpoint returns **numeric scores you can compare across runs**. The
rubric is the same for every run so scores are comparable across callers and across time.
That still takes **minutes, not seconds**, so this endpoint does not return the verdict. `202`
means the submission is recorded and will be evaluated — never that it has been.
```
POST /v2/judge/llm -> 202 { evaluation_id, status: "pending" }
|
| the judge runs on our side
v
GET /v2/judge/llm/{evaluation_id} -> the verdict, once it settles
```
`status` is `pending` for a new submission, but read it rather than assuming it. An
`Idempotency-Key` retry returns the evaluation that key already names — which may have
finished in the meantime — so it can come back `completed` or `failed`. The value set is the
same one [Get LLM Judge Evaluation](/api-reference/evaluations/get_llm) returns.
Retrieve the result with [Get LLM Judge Evaluation](/api-reference/evaluations/get_llm).
## Example
```python theme={null}
import requests
resp = requests.post(
"https://api.eu.linqalpha.com/v2/judge/llm",
headers={
"X-API-KEY": "",
"Content-Type": "application/json",
# Optional but recommended: makes a retry safe.
"Idempotency-Key": "run-2026-08-25-0001",
},
json={
# OPTIONAL. Your framing for the judge — delivered verbatim as the system message
# when supplied. The scoring dimensions themselves are fixed by the endpoint; use
# `prompt` to steer emphasis (e.g. "penalise unhedged numeric claims") rather than
# to change the rubric structure. Omit the field entirely to fall back to Linq's
# default judge prompt.
"prompt": (
"You are grading whether the ANSWER is supported by the SOURCES and by the "
"verification verdicts. Score each dimension in the fixed rubric strictly, "
"and justify the scores in a short reasoning."
),
"query": "How did Apple perform in Q3 FY2024?",
"answer": (
"Apple reported record Services revenue in Q3 FY2024 while iPhone revenue "
"slipped year-over-year."
),
# Inclusive search range. `start_time` is optional; `end_time` is required.
"time_window": {
"start_time": "2024-07-01T00:00:00Z",
"end_time": "2026-08-19T05:32:11Z",
},
# Optional. `external` is the default; use `rms` for RMS-only answers and
# `all` when the answer combines RMS and external sources.
"search_type": "external",
# Structured references match the agent-judge payload shape. `content` is what
# the final judge source-grounds against.
"references": [
{
"title": "Apple Q3 FY2024 Press Release",
"content": (
"Apple today announced financial results for its fiscal 2024 third "
"quarter. Services revenue reached an all-time high."
),
# Optional locator for the source the excerpt came from.
"url": "https://www.apple.com/newsroom/2024/08/apple-reports-third-quarter-results/",
"metadata": {
"published_at": "2024-08-01T13:30:00Z",
"source_type": "press_release",
},
},
{
"title": "Apple Investor Relations",
"content": (
"iPhone revenue was down slightly year-over-year while Services set "
"a new record."
),
},
],
},
)
evaluation_id = resp.json()["payload"]["evaluation_id"]
```
## The fields
These are the fields this endpoint reads; anything else in the body is rejected.
**Built for organization-bound API keys.** The organization and user are taken from your key.
The evaluation is filed under that organization, and RMS verification uses that user's
document visibility.
Platform-wide keys are not a supported configuration for this endpoint — it has not been
designed or tested against them, and the behaviour you get is whatever the shared
authentication layer does rather than something this endpoint guarantees. If you hold a
platform key, talk to your LinqAlpha contact before integrating.
| Field | Required | Notes |
| ------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `prompt` | No | Your framing for the judge. Delivered verbatim as the system message when supplied — steers emphasis; does **not** change the scoring dimensions. Omit the field to fall back to Linq's default judge prompt. If supplied it must contain non-whitespace text; an empty string is rejected. |
| `query` | Yes | The original user prompt that produced the answer. Lets the judge check whether the answer followed the instructions. |
| `answer` | Yes | The text being judged. Must contain non-whitespace text. |
| `time_window` | Yes | Inclusive search window. `end_time` is required and `start_time` is optional. Both use RFC 3339 timestamps **with a timezone offset**. `start_time` must be earlier than or equal to `end_time`; a future `end_time` is rejected. |
| `references` | No | Defaults to `[]`. Max 50 items. Each element is `{ "title"?, "content", "url"?, "metadata"? }`. `content` is required; the rest are optional. `url` must be HTTP(S). `metadata` accepts any JSON object, including nested objects and arrays. A bare URL string and undeclared reference-level fields are not accepted. |
| `search_type` | No | Verifier source scope: `rms`, `external`, or `all`. Defaults to `external`; use `all` when the answer combines RMS and external sources. |
**Fixed rubric.** Every LLM Judge run is scored on the same four dimensions
(Factuality / Completeness / Relevance / Grounding, each 1–5), with a server-computed
`overall_score` (arithmetic mean). Previously supported `scoring_rubric` and
`response_schema` fields were removed so that scores from any two runs are directly
comparable; sending either now returns a `400`.
`start_time` and `end_time` are inclusive bounds applied to the selected `search_type`.
When `search_type` is `all`, the same window applies to both RMS and external search.
Omit `start_time` to search everything up to and including `end_time`.
**The offset is required, and that is deliberate.** `2026-08-19T14:32:11` without one is
ambiguous, and reading it as UTC would move a Seoul timestamp nine hours. The result of that
is not an error you would see: it is a plausible verdict judged against the wrong instant.
Send `Z` or your own offset — both name the same instant and both are accepted.
`start_time` must be earlier than or equal to `end_time`. A future `end_time` is rejected;
a few minutes of clock skew is tolerated.
Each element is `{ "title"?, "content", "url"?, "metadata"? }`. The reference object only
accepts these four fields; any other undeclared field is rejected with `400`.
`content` is the passage itself. The verifier checks the answer against that text, so a
link alone would give it nothing to read; paste the passage you are relying on.
`url` is an optional HTTP(S) locator for that passage's source. It is a pointer used to
inspect the original, not evidence — neither the URL nor its domain is trusted as proof
that the excerpt, publisher, or date is correct.
`metadata` is an optional free-form JSON object. It can contain source dates such as
`published_at`, identifiers, tags, and arbitrary nested JSON without a predefined field list.
The `metadata` value itself must be an object.
## How the judgement is produced
The verifier runs each reference against Linq's tool set for the selected `search_type`,
producing a per-reference verdict (`supported` / `contradicted` / `unresolved`) with
evidence. When
`references: []` this step is skipped and no cost is incurred here.
A deterministic model (`temperature=0`, `seed=7`) reads your `prompt`, the query, answer,
references, and verification verdicts, and returns integer scores for each of the four
fixed dimensions plus a short reasoning. The server then computes `overall_score` as the
arithmetic mean of the four scores.
**Partial verification is preserved, not hidden.** If a verifier chunk failed, only the
surviving verdicts reach the judge, and the response reports `verified_reference_count`
**less than** `input_reference_count` so you can tell — see
[Get LLM Judge Evaluation](/api-reference/evaluations/get_llm).
## Idempotency
Send an `Idempotency-Key` header to make retries safe. Within your organization:
* Same key, same body → the **original** `evaluation_id`, no second judge run.
* Same key, different body → `409 Conflict`.
Use it whenever a network error leaves you unsure whether a submission landed. Without it, a retry
starts a second run and you are billed for both.
Idempotency is scoped per endpoint. Using the **same key** on
[Execute Agent Judge](/api-reference/evaluations/create) and this endpoint returns two distinct
evaluations — the two endpoints do not share the key space.
A retry still answers `202`, but the `status` it carries is the **original evaluation's current
status** — not necessarily `pending`. If that evaluation already finished, you get `completed` or
`failed` straight from the retry and there is nothing left to poll.
## Limits
| | Max |
| ---------------------- | ------------------------- |
| `prompt` | 50,000 UTF-16 code units |
| `answer` | 200,000 UTF-16 code units |
| `query` | 10,000 UTF-16 code units |
| `references` | 50 items |
| Whole body, serialized | 1,000,000 bytes (UTF-8) |
| Whole body, tokenized | \~100,000 tokens |
Oversized submissions are rejected with `400` at submission time — nothing is queued and nothing
is billed, so a request that is too large costs only the round trip.
Two of these are easy to trip without noticing.
**The character counts are UTF-16 code units**, which is what `"…".length` returns in
JavaScript. Characters outside the Basic Multilingual Plane — emoji, some rarer CJK — count as
two. If your text is plain prose the distinction never comes up.
**The token cap is separate from the byte cap**, and applies to the request as a whole. Dense
CJK text can pass 200,000 code units and still exceed 100,000 tokens, so a long Korean or
Japanese answer may be refused while a longer English one is not.
These may be raised as we see real usage. A raise never breaks a client that was within the old
figure, so code against them as minimums. If you are running close to one, tell us rather than
splitting a submission.
## OpenAPI
````yaml POST /v2/judge/llm
openapi: 3.0.1
info:
title: LinqAlpha API
description: >-
Linq helps finance professionals make informed decisions using
Retrieval-Augmented Generation (RAG)-enhanced answers. By leveraging
cutting-edge Large Language Models (LLM) and supplementary technology, Linq
provides the most optimized responses based on your queries.
version: 1.0.0
license:
name: MIT
servers:
- url: https://api.eu.linqalpha.com
security:
- ApiKeyAuth: []
tags:
- name: Search
description: Search and generate responses
- name: Data
description: Data retrieval and mapping
- name: Feedback
description: Conversation feedback
- name: RMS
description: Research Management System
- name: Source Management
description: Source batch and file management
- name: MCP
description: >-
LinqAlpha MCP — Financial data tools for AI assistants via Model Context
Protocol
- name: Connectors
description: Customer Connectors — customer-owned MCP connector management
- name: Briefing
description: Briefing Agent — automated market briefings with scheduling and delivery
- name: Status
description: Sync status — check organization, document, and container sync progress
paths:
/v2/judge/llm:
post:
tags:
- Evaluations
summary: Submit an LLM Judge evaluation
description: >-
Records the submission and queues it. Returns immediately; the judgement
itself runs asynchronously and is retrieved with `GET
/v2/judge/llm/{evaluation_id}`. Unlike `POST /v2/judge/agent` (Agent
Judge), which returns a written assessment, this endpoint produces a
**structured verdict** — dimension scores plus reasoning — on a fixed
rubric (Factuality / Completeness / Relevance / Grounding, each 1-5)
with a server-computed `overall_score`. Every run uses the same rubric
so runs are comparable across callers and time.
parameters:
- name: Idempotency-Key
in: header
required: false
description: >-
Makes a retry safe. Same key + same body returns the original
`evaluation_id`; same key + different body is a 409. Scope is
per-endpoint — the same key on `POST /v2/judge/agent` yields a
distinct evaluation.
schema:
type: string
maxLength: 255
example: run-2026-08-25-0001
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/JudgeLlmRequest'
responses:
'202':
description: >-
Accepted and durably recorded. **Not** evaluated yet — unless this
was an `Idempotency-Key` retry naming an evaluation that has since
finished, in which case `status` reports that evaluation's current
state. Read `status`; do not assume `pending`.
content:
application/json:
schema:
type: object
properties:
error:
nullable: true
example: null
payload:
$ref: '#/components/schemas/JudgeLlmAccepted'
example:
error: null
payload:
evaluation_id: 2fde560a-e7eb-45e0-8cd3-04e57c50d1d3
status: pending
'400':
description: >-
Invalid body, or a field past its size cap. Nothing is queued or
billed. `INVALID_REQUEST_BODY` and `JUDGE_EVALUATION_TOO_LARGE` come
from this API's own validation; `JUDGE_EVALUATION_INVALID_INPUT`
means the evaluation service rejected the payload on a further rule.
All three are the request's own problem — retrying it unchanged will
fail again.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiErrorResponse'
example:
error:
code: INVALID_REQUEST_BODY
msg: references must contain at most 50 items
message: references must contain at most 50 items
payload: null
'401':
description: Missing or invalid API key.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiErrorResponse'
example:
error:
code: UNAUTHORIZED
msg: Invalid API key
message: Invalid API key
payload: null
'409':
description: >-
`Idempotency-Key` already used by this organization on this endpoint
for a different body.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiErrorResponse'
example:
error:
code: JUDGE_EVALUATION_IDEMPOTENCY_CONFLICT
msg: Idempotency-Key was already used for a different request
message: Idempotency-Key was already used for a different request
payload: null
'429':
description: Organization rate limit exceeded.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiErrorResponse'
example:
error:
code: RATE_LIMIT_EXCEEDED
msg: Rate limit exceeded
message: Rate limit exceeded
payload: null
'502':
description: Upstream failure while recording the submission.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiErrorResponse'
example:
error:
code: JUDGE_EVALUATION_FAIL
msg: Failed to submit the evaluation
message: Failed to submit the evaluation
payload: null
'503':
description: Intake is temporarily disabled. Retry later.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiErrorResponse'
example:
error:
code: JUDGE_EVALUATION_UNAVAILABLE
msg: Evaluation service is temporarily unavailable
message: Evaluation service is temporarily unavailable
payload: null
'504':
description: Timed out recording the submission.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiErrorResponse'
example:
error:
code: JUDGE_EVALUATION_FAIL
msg: Evaluation request timed out
message: Evaluation request timed out
payload: null
components:
schemas:
JudgeLlmRequest:
type: object
additionalProperties: false
description: >-
Body for `POST /v2/judge/llm`. Unknown top-level fields are rejected
(public-api zod is `.strict()`). The previously supported
`scoring_rubric` and `response_schema` fields were removed to keep every
run comparable on the fixed default rubric; a caller still sending
either now gets a 400. Whole serialized body must be at most 1,000,000
bytes (UTF-8).
required:
- answer
- query
- time_window
properties:
prompt:
type: string
minLength: 1
maxLength: 50000
description: >-
Optional. Delivered verbatim as the judge's system message when
supplied. Omit the field to fall back to Linq's default judge
prompt. If supplied it must contain non-whitespace text; an empty
string is rejected.
answer:
type: string
minLength: 1
maxLength: 200000
description: The text being judged.
query:
type: string
minLength: 1
maxLength: 10000
description: >-
The original user prompt that produced the answer. Lets the judge
check whether the answer followed the instructions.
time_window:
$ref: '#/components/schemas/JudgeTimeWindow'
references:
type: array
items:
$ref: '#/components/schemas/JudgeLlmReference'
maxItems: 50
default: []
description: >-
Structured references the caller wants verified. At most 50 items.
Each element is `{ title?, content, url?, metadata? }`; `content` is
required, while `title`, an HTTP(S) `url`, and free-form JSON-object
`metadata` are optional. Bare URL strings and undeclared
reference-level fields are not accepted. `references: []` is allowed
— the verifier is skipped and the judge grades on `prompt` / `query`
/ `answer` only.
search_type:
type: string
enum:
- rms
- external
- all
default: external
description: >-
Verifier source scope: `rms`, `external`, or `all`. Defaults to
`external`; use `all` when the answer combines RMS and external
sources.
JudgeLlmAccepted:
type: object
required:
- evaluation_id
- status
properties:
evaluation_id:
type: string
format: uuid
status:
type: string
enum:
- pending
- completed
- excluded
- failed
description: >-
`pending` for a newly accepted submission. An `Idempotency-Key`
retry returns the evaluation that key already names, so it reports
that evaluation's current status, which may already be terminal.
Treat this as the same value set `GET /v2/judge/llm/{evaluation_id}`
returns rather than a constant.
ApiErrorResponse:
type: object
description: Standard error response wrapper
properties:
error:
$ref: '#/components/schemas/ApiError'
payload:
description: Always null for error responses
nullable: true
required:
- error
- payload
JudgeTimeWindow:
type: object
required:
- end_time
additionalProperties: false
properties:
start_time:
type: string
format: date-time
example: '2025-01-01T00:00:00Z'
description: >-
Optional inclusive lower bound. RFC 3339 with a timezone offset.
Must be earlier than or equal to `end_time`. Omit it when no lower
bound is needed.
end_time:
type: string
format: date-time
example: '2026-08-19T05:32:11Z'
description: >-
Required inclusive upper bound. RFC 3339 with a timezone offset and
must not be in the future.
description: >-
Inclusive bounds applied to the selected `search_type`. For `all`, the
same window applies to both RMS and external search. A bare date,
missing timezone offset, or nonexistent instant is rejected.
JudgeLlmReference:
type: object
required:
- content
properties:
title:
type: string
minLength: 1
description: Short label rendered alongside the citation. Optional.
content:
type: string
minLength: 1
description: >-
Verbatim excerpt from the source. What the final judge
source-grounds against.
url:
type: string
format: uri
pattern: ^[Hh][Tt][Tt][Pp][Ss]?://
description: >-
Optional HTTP(S) locator for the original source. A pointer for
inspecting the original, not evidence: neither the URL nor its
domain is trusted as proof that the excerpt, publisher, or date is
correct.
metadata:
type: object
additionalProperties: true
example:
published_at: '2024-08-01T13:30:00Z'
source_type: press_release
tags:
- earnings
- primary
description: >-
Optional caller-defined source attributes. Any JSON object is
accepted, including arbitrary keys and nested JSON values.
additionalProperties: false
description: >-
A source the answer relied on. Shape: `{ title?, content, url?,
metadata? }`; `content` is required. `url` is an optional HTTP(S)
locator and `metadata` an optional free-form JSON object. Bare URL
strings and undeclared reference-level fields are rejected.
ApiError:
type: object
properties:
code:
type: string
description: >-
Error code indicating the type of error. Common codes:
- Authentication: `API_KEY_MISSING`, `INVALID_API_KEY`
- Validation: `INVALID_REQUEST_BODY`, `ORGANIZATION_ID_MISSING`,
`TICKERS_MISSING`, `CHAT_MESSAGE_ID_MISSING`, `DOCUMENT_ID_MISSING`
- Service: `SEARCH_FAIL`, `TTS_FAIL`, `CREATE_CONV_FAIL`,
`GET_STOCK_FAIL`, `CREATE_MSG_FAIL`, `ALPHA_COMP_FAIL`,
`GET_REF_FAIL`
- Connectors: `CONNECTOR_LIST_FAIL`, `CONNECTOR_NOT_FOUND`,
`CONNECTOR_CREATE_FAIL`, `CONNECTOR_UPDATE_FAIL`,
`CONNECTOR_DELETE_FAIL`, `CONNECTOR_TEST_FAIL`
- Not Found: `NOT_FOUND`
example: INVALID_REQUEST_BODY
msg:
type: string
description: Error message (deprecated, use `message` instead)
example: query is required and must be a string
message:
type: string
description: Error message providing more details about the error
example: query is required and must be a string
required:
- code
securitySchemes:
ApiKeyAuth:
type: apiKey
in: header
name: X-API-KEY
````