soev for developers
soev-evalsReference

soev_evals.cli

Load an evaluation project for the soev-evals command, and give the command its services.

The CLI is soev-evals [--debug] <command>, also python -m soev_evals. Each command takes --project PATH, the path of project.yaml. The commands run, report and rescore keep the runs in runs/ beside it, unless you give --runs-dir DIR.

  • run [--suite NAME]... [--split dev|test] [--k N] [--resume RUN_ID | --run-id NAME] [--runtime NAME] runs the cases and archives their evidence.
  • report RUN prints the archived report of a run.
  • rescore RUN [--offline] [--runtime NAME] evaluates the archived evidence of a run with the current criteria.
  • seal [--append] records the digests of the test cases in sealed-test.yaml. After that, a changed test case stops each command that loads the suites.

When a selected case is a test case, the CLI prints aggregates only, and an error shows no sealed content.

Projects

A project file names the system under test, the judge, the suites and an optional hooks module.

load

functionload(path, *, definitions=True, suite_files=True, allow_test_additions=False, project=None) -> LoadedProject

Load a project file with its hooks, its suites and its corpus lookup.

The project setting hooks names a Python module that load imports. Its attribute CRITERIA is a list of criterion kinds that the suites can use beside the built-in kinds. Its attribute SETUP is the case setup of the project. Each suite of the project is the merge of the files that match its glob pattern. When the project directory has a sealed-test.yaml, load checks that the test cases did not change since the seal.

parampathPath

The path of the project file, project.yaml.

paramdefinitionsbool
= True

If false, load reads only the project file: no hooks module, no suites, the built-in criterion kinds, and an empty context. The report command uses this.

paramsuite_filesbool
= True

If false, load imports the hooks module and validates the project invariants, but reads no suite files. The context then has the messages only.

paramallow_test_additionsbool
= False

If true, the seal check accepts test cases that the seal does not have yet. Each sealed test case must still be the same. seal --append uses this.

paramprojectProject | None
= None

The project to use instead of the content of the project file. path still sets the project directory.

Returns

LoadedProject

The loaded project.

Raises:

  • ProjectError: A suite file declares another suite than its project entry, two suites have a case with the same id, a sealed test digest changed, or the corpus lookup is invalid.
  • SealChangedError: A sealed test case changed or was removed.
  • SuiteError: A suite has no file, or a suite file is invalid.
  • UnknownCriterionError: A criterion names a kind that the registry does not have.
  • CriterionParamsError: A criterion has invalid parameters.
  • DuplicateCriterionError: A kind of the hooks module has the name of another kind.
  • ImportError: The hooks module cannot be imported.
  • OSError: The project file or a rubric file cannot be read.
  • yaml.YAMLError: The project file is not valid YAML.
  • pydantic.ValidationError: The project file is not a valid project.

LoadedProject

dataclassLoadedProject(path, project, registry, suites, setup, context)

A project as load read it: its definitions, and what its hooks module adds.

parampathPath

The absolute path of the project file. Suite patterns, rubrics, the corpus lookup and the seal are relative to its directory.

paramprojectProject

The project.

paramregistryCriterionRegistry

The built-in criterion kinds, and the CRITERIA of the hooks module.

paramsuiteslist[Suite]

The suites of the project. Empty when load did not read the suite files.

paramsetupCaseSetup | None

The SETUP of the hooks module, or None.

paramcontextCriterionContext

What the criteria get from the project: the messages, the rubric texts by path, and the ids, texts and law names of the corpus lookup. It has no judge and no language detector.

SealChangedError

errorSealChangedError(changed_ids)

Sealed test cases changed or were removed since the seal.

load raises it when a test case of the file sealed-test.yaml has another digest now, or is gone. The CLI prints aggregate_message, which has no case ids, so the ids of sealed cases do not reach the console.

Make the error for the test cases changed_ids.

paramchanged_idslist[str]

The ids of the changed or removed test cases. The message of the error lists them.

attributeaggregate_messagestr

The message with the number of changed cases only, without their ids.

Runtimes

A runtime provider gives the run and rescore commands their arm, judge, language detector, setup and tracker.

RuntimeProvider

protocolRuntimeProvider

A provider of the services that the run and rescore commands use.

Register a factory of providers in the entry point group soev.evals.runtimes of your package. The CLI calls the factory without arguments. Select the provider with --runtime NAME, or with the project setting runtime: NAME. Without either, the CLI uses its built-in provider: the run command gets a SoevThreadArm for the system of the project, with a client from create_http_client, and no judge.

The CLI raises ProjectError when two entry points have the same name, or when no entry point has the selected name. An offline rescore opens no provider.

Example

from contextlib import asynccontextmanager

from soev_evals.arms import SoevThreadArm, create_http_client
from soev_evals.cli import RuntimeBindings
from soev_evals.judges import LLMJudge, load_prompts

llm = ...  # a JudgeLLM, for example the ChatCompletionsLLM of the JudgeLLM example


class Runtime:
    @asynccontextmanager
    async def open(self, loaded, *, operation):
        async with create_http_client(loaded.project.system) as http:
            system, citations = loaded.project.system, loaded.project.citations
            arm = SoevThreadArm(http, system, citations) if operation == "run" else None
            yield RuntimeBindings(arm=arm, judge=LLMJudge(llm, load_prompts()))
[project.entry-points."soev.evals.runtimes"]
mine = "my_package.runtime:Runtime"
methodopen(loaded, *, operation) -> AbstractAsyncContextManager[RuntimeBindings]

Return a context manager that gives the services of one command, and releases them after it.

The command runs inside the context manager, so the services can hold open clients.

paramloadedLoadedProject

The loaded project.

paramoperationLiteral['run', 'rescore']

"run" for the run command, "rescore" for a rescore command that is not offline.

RuntimeBindings

dataclassRuntimeBindings(arm=None, judge=None, language_detector=None, setup=None, tracker=None)

The services that a RuntimeProvider gives to the run and rescore commands.

paramarmArm | None
= None

The system under test. The run command needs it. The rescore command does not use it.

paramjudgeJudge | None
= None

The judge for the judge criteria. If None, these criteria cannot call a judge and get the status error.

paramlanguage_detectorLanguageDetector | None
= None

The language detector for the language criteria. If None, the CLI uses a LangidDetector of soev_evals.judges.

paramsetupCaseSetup | None
= None

The fixtures of each case. If None, the CLI uses the SETUP of the project hooks module. The rescore command does not use it.

paramtrackerTracker | None
= None

Where the run command also publishes the run. If None, it publishes nothing. The rescore command does not use it.

soev_evals.arms

System-under-test adapters: run the cases of an evaluation as conversations with an agent.

soev_evals.core

Pure evaluation models, ports, loading and scoring.

On this page

ProjectsloadLoadedProjectSealChangedErrorRuntimesRuntimeProviderRuntimeBindings