soev_evals.judges

A Judge that asks a model, through a JudgeLLM adapter, for a label.

For each request, the system message is the context prefix and then the prompt of the request's task. The user message is the request's inputs as JSON. It asks the model for a JSON object with a label from the allowed labels and a detail. Then it validates the answer.

Make a judge that uses llm with prompts.

The model of an LLMJudge: an adapter that sends one chat request and returns the JSON answer.

Write an adapter for your model provider. complete must raise JudgeError for a failed call: a criterion records it as a verdict with the status "error", but any other exception stops the run.

from collections.abc import Mapping

import httpx
from pydantic import JsonValue

from soev_evals.core import JudgeError
from soev_evals.judges import LLMJudge, load_prompts


class ChatCompletionsLLM:
    context_window = 128_000

    def __init__(self, client: httpx.AsyncClient, model: str) -> None:
        self._client = client
        self._model = model

    async def complete(self, *, system: str, user: str, schema: Mapping[str, JsonValue]) -> str:
        schema_format = {"name": "verdict", "schema": dict(schema)}
        body = {
            "model": self._model,
            "messages": [{"role": "system", "content": system}, {"role": "user", "content": user}],
            "response_format": {"type": "json_schema", "json_schema": schema_format},
        }
        try:
            response = await self._client.post("chat/completions", json=body)
            response.raise_for_status()
            return response.json()["choices"][0]["message"]["content"]
        except (httpx.HTTPError, KeyError, IndexError, ValueError) as exc:
            raise JudgeError(f"judge call failed: {type(exc).__name__}") from exc


client = httpx.AsyncClient(base_url="http://localhost:8000/v1/")
judge = LLMJudge(ChatCompletionsLLM(client, "my-judge-model"), load_prompts())

Read the prompt of each built-in judge task.

The package ships one prompt file for each built-in judge kind: behaviour.expected, compare.equivalent, nuggets.coverage, privacy.names and rubric.judge.

Hash the exact prompt and rubric content used by a run.

The run manifest records the hashes, so a resumed run fails when a prompt changed.

A LanguageDetector with the bundled model of py3langid.

The model runs offline and gives the same answer for the same text. The CLI gives this detector to the reply.language criterion when the runtime provider gives none.

Load the bundled model once, so that the first detect call is not slow.