soev_evals.core

Cases and shared criteria from one logical suite.

A suite can span several YAML files: load_suites merges them. Validation fails when two cases have the same id, when a case names another suite, or when a case does not match id_pattern.

Load one suite from YAML text. The function reads no file.

The registry checks the kind and the parameters of each criterion, so an unknown kind or a wrong parameter fails here, before a run starts.

from soev_evals.core import load_suite
from soev_evals.criteria import builtin_registry

text = '''
name: smoke
cases:
  - id: greet
    lineage: greet
    category: smalltalk
    turns:
      - prompt: Hallo, wie ben jij?
    criteria:
      - kind: reply.language
        language: nl
'''
suite = load_suite(text, builtin_registry())

Raises:

• SuiteError: The text is not valid YAML or not a valid suite, or a criterion has an unknown kind or invalid parameters.

Merge the files of one suite into one suite.

The merged suite has the cases of all documents, in the order of documents.

Raises:

• SuiteError: documents is empty, a document is not a valid suite, the documents differ in name, defaults or id_pattern, or two documents have a case with the same id.

A conversation and its project-independent evaluation definition.

A case is one entry of a suite file. Each attempt of a case runs its turns on a new conversation, and each variant on a new conversation of its own. Then the criteria grade the evidence.

One user prompt in a conversation.

The expected behaviour, content and source roles for a case.

A reference fact and the ids of the sources that support it.

The nuggets.coverage criterion asks the judge whether the reply covers each nugget.

Required alternative groups, supporting sources and forbidden sources.

Each entry is a source id, as CitationMapping resolves it from a citation. The citation criteria compare these ids with the cited ids. With the default match "hierarchical", an id also matches the ids below it in the / path: law/1 matches law/1/2, and law/1/2 matches law/1.

Generation and review history carried with a case.

This package stores the history with the case, but does not read it.

Source coverage assessment and non-authoritative hints for expert review.

This package stores and validates the assessment, but no built-in criterion reads it.

The label of an Answerability assessment: how well the sources cover the question of a case.

Only the label "limited" has reasons, as a list of Limitation values.

A reason for the Answerability label "limited".

Suite content is invalid or inconsistent.

load_suite and load_suites raise it.

Project-owned arm, judge, suites, hooks and invariant definitions.

A project is one YAML file. It names the system under test, the suites and the criteria that every case shares. load of soev_evals.cli reads it with its suites, its hooks and its corpus lookup.

The settings of the arm: the server, the agent and the names of the environment variables of its credentials.

The project file holds only the names of the environment variables, never their values. The arm reads the values when it opens its HTTP client.

The judge settings of a project.

soev_evals records endpoint in the manifest, but connects no judge to it. A RuntimeProvider reads it to make its judge.

The requested repetition count for each case.

A dotted element field path and optional canonical-id extraction regex.

The arm uses it to turn each element that a reply cites into the canonical source id of a Citation.

Where the corpus criteria find the ids and the text of the corpus.

The project loader reads the file when it exists. Each line is a JSON object with an id and a text. Without the file, the criteria that need the corpus give a verdict with the status "error".

Incremental lineage split targets and suite-specific strata.

The settings of a dataset split. No command of soev_evals reads them.

The case field that groups the lineages of a suite into strata for a dataset split.

The split puts the same share of each stratum on the test side. The value comes from the root case of a lineage. "rule_direction" combines the rule with the expected behaviour: "pass" for "answer", "fire" otherwise.

Project content is invalid or inconsistent.

The project loader of soev_evals.cli raises it, for example when two suites of a project have a case with the same id.

A named criterion with scoring policy and kind-specific parameters.

It is one entry of criteria, defaults or invariants in a YAML file. The parameters of the kind can stand next to kind, or under params. A key next to kind replaces the same key under params:

- kind: reply.max_length
  max: 2000
  gate: true

A criterion implementation with a validated parameter model.

A criterion in a suite names its kind in kind. Builtin of soev_evals.criteria makes a kind of a typed async function. Register your own kinds in the CRITERIA list of the project's hooks module.

Resolve and evaluate an explicitly supplied collection of kinds.

builtin_registry of soev_evals.criteria returns a registry with the built-in kinds.

Make a registry of kinds.

Raises:

• DuplicateCriterionError: Two kinds have the same name.

The services and the project lookup data that each criterion gets.

Each criterion gets the same context. A criterion does no I/O of its own: what it needs from outside comes from here. A service or lookup that the project did not configure is None or empty, and a criterion that needs it gives a verdict with the status "error".

One criterion outcome with the scoring policy used for the attempt.

A criterion name is absent from the registry.

CriterionRegistry raises it for a spec whose kind it does not hold. load_suite reports it as a SuiteError.

Two implementations claim the same criterion name.

CriterionRegistry raises it when it is made with two kinds of the same name, for example a project hook that reuses the name of a built-in kind.

A criterion has invalid parameters.

CriterionRegistry.validate and CriterionRegistry.evaluate raise it when the parameters of a spec do not fit the parameter model of its kind. load_suite reports it as a SuiteError.

The evidence of one attempt: its turns in order, its failure and the evidence of its variants.

The runner builds it for each attempt, and the criteria grade it. The archive stores it, so a rescore can grade it again without a new conversation.

The reply, citations, tool calls and measurements from one turn.

A Conversation returns it for each turn that it sends. A turn with a failure ends the conversation: the runner sends no more turns of the case.

A cited source resolved to its canonical id.

A tool invocation and a compact result description.

Measured token and time consumption of a turn.

A case's verdicts from one attempt.

An arm, which starts a new conversation with the system under test for each attempt.

The arm is the connection to the system under test. SoevThreadArm of soev_evals.arms is the built-in arm.

A single isolated multi-turn interaction with the system under test.

An Arm starts it. The runner sends the turns of the case in order, then closes the conversation.

The setup of a project, which manages project-specific fixtures and artifacts around an attempt.

The runner calls prepare before the conversation, collect after it, and cleanup at the end. It does this for the turns of the case and again for each variant. Give a setup to the CLI as SETUP in the project's hooks module.

A judge, which gives a label for a grading request.

The judge owns the prompts. A criterion gets a judge from CriterionContext.judge. LLMJudge of soev_evals.judges is the built-in judge.

The task and the inputs that a criterion sends to a judge.

A criterion sends the task and the inputs. The judge adds the prompt of the task.

A judge's chosen label and explanation.

A language detector, which identifies the language of a single sentence.

The reply.language criterion calls it for each sentence of the reply. LangidDetector of soev_evals.judges is the built-in detector.

A run archive, which persists and retrieves complete runs for resume and rescore.

A run has a manifest, a journal of completed attempts, the evidence of each attempt and a report. The runner calls these synchronous methods in a worker thread. FileRunArchive of soev_evals.run is the built-in archive.

A tracker, which publishes run records to an optional tracking destination.

The runner logs a warning for each exception of a tracker and continues, so a tracker cannot make a run fail.

The system under test could not complete a conversation.

An Arm or a Conversation raises it. The runner records it as the failure of the attempt's evidence, and the run continues.

Case preparation, collection or cleanup failed.

A CaseSetup raises it. The runner records it as the failure of the attempt's evidence, and the run continues.

The judge could not produce a usable result.

A Judge raises it. LLMJudge raises it when it has no prompt for the task, and when the model's answer is not valid JSON or has a label that the request does not allow. It also raises it when a request with a context prefix does not fit the model's context window. A built-in criterion records it as a verdict with the status "error".

The detector could not identify a sentence's language.

A LanguageDetector raises it, for example for an empty sentence. The reply.language criterion records it as a verdict with the status "error".

Run archive data could not be read or written.

A RunArchive raises it. The runner also raises it when a resumed run does not match its archive.

A frozen value object that rejects unexpected fields.

It is the pydantic base of the public models of soev_evals. A field name that the model does not declare fails validation, and so do the float values inf and nan. An instance cannot change after it is made. Subclass it for the parameter model of your own CriterionKind, so that a typo in a suite file fails when the suite loads.

An evaluation operation could not be completed.

It is the base class of the errors of soev_evals. Catch it to handle any of them.