soev for developers
soev-evalsReference

soev_evals.run

Execution, persistence and reporting of evaluation runs: run, archive, rescore and observe them.

A run freezes its inputs in a manifest, makes k attempts of each case with an arm, and evaluates the evidence of each attempt with the criteria of the case. An archive keeps the manifest, the evidence, a journal of the attempts and the report, so a run can resume after it stops, and a rescore can evaluate the same evidence again.

Runs

A runner makes the attempts of a manifest and returns the report.

Runner

dataclassRunner(arm, registry, ctx, archive, setup=None, tracker=None, observer=None)

A runner that runs the attempts of a manifest, evaluates them and archives them.

Each attempt runs the case on a fresh conversation of the arm, and each variant of the case on another fresh conversation. Then the criteria of the manifest evaluate the evidence. The archive saves the evidence first, and then appends the attempt to the journal of the run. So a run that stops can resume: run with resume=True runs only the attempts that the journal does not have.

Example

from pathlib import Path

from soev_evals.cli import load
from soev_evals.judges import load_prompts, prompt_shas
from soev_evals.run import FileRunArchive, Runner, build_manifest

loaded = load(Path("project.yaml"))
arm = ...  # an Arm, for example a SoevThreadArm of soev_evals.arms
archive = FileRunArchive(Path("runs"))
manifest = build_manifest(
    loaded.project, loaded.suites, run_id="baseline", revision="unknown", shas=prompt_shas(load_prompts())
)
report = await Runner(arm, loaded.registry, loaded.context, archive).run(manifest)
paramarmArm

The system under test.

paramregistryCriterionRegistry

The criterion kinds that evaluate the criteria of the manifest.

paramctxCriterionContext

What the criteria get: the judge, the language detector, the messages and the corpus lookup.

paramarchiveRunArchive

Where the runner keeps the manifest, the evidence, the journal and the report of each run.

paramsetupCaseSetup | None
= None

The fixtures of the project. If set, the runner prepares them before each conversation, collects their artifacts after it and cleans them up. If None, conversations run without fixtures.

paramtrackerTracker | None
= None

Where the runner also publishes the run, each attempt with its evidence, and the report. A failure of the tracker is logged as a warning and does not stop the run. If None, nothing is published.

paramobserverEvaluationObserver | None
= None

What records the run and its attempts, for example as spans. If None, the runner uses an OtelObserver of soev_evals.track when the [otel] extra is installed, and records nothing when it is not.

methodrunasync (manifest, *, resume=False) -> Report

Execute unfinished attempts and persist the aggregate report.

The run makes manifest.k attempts of each active case of the manifest, at most manifest.project.concurrency at the same time. An ArmError of the arm or a SetupError of the setup does not stop the run. It becomes the failure of the evidence, and the criteria evaluate that evidence. Any other exception of the arm or the setup stops the run (see ExceptionGroup).

parammanifestManifest

The manifest of the run, for example from build_manifest.

paramresumebool
= False

If false, the run is new, and the archive creates it. If true, the run continues the archived run manifest.run_id: the runner checks the archived manifest and journal, and then runs only the attempts that the journal does not have.

Returns

Report

The report over all attempts of the journal. The archive saves it too, and replaces an earlier report.

Raises:

  • ArchiveError: The archive cannot create or read the run before the attempts start. With resume, also: the archived manifest differs from manifest, or the journal has a duplicate or unexpected attempt.
  • ExceptionGroup: An attempt raised an error, for example an ArchiveError when the archive cannot save the attempt. The other attempts stop too. The attempts that the journal has stay archived.

build_manifest

functionbuild_manifest(project, suites, *, run_id, revision, shas, k=None, split=None, case_ids=None) -> Manifest

Freeze selected cases, suite digests and all effective criteria.

The manifest holds everything that a run needs to repeat or resume: the project, the selected cases, a digest of each suite, and the criteria of each case. The criteria of a case are the project invariants, then the suite defaults, then the criteria of the case. The manifest names the credentials of the system by their environment variables only, so it holds no secret.

paramprojectProject

The project of the run.

paramsuitesSequence[Suite]

The suites to select cases from. The manifest has a digest of each of them.

paramrun_idstr

The id of the run in the archive.

paramrevisionstr

The revision of the code under test, for example a git commit.

paramshasMapping[str, str]

A hash of each judge prompt and rubric, for example from prompt_shas of soev_evals.judges.

paramkint | None
= None

The number of attempts of each case. If None, the project setting runs.k.

paramsplitstr | None
= None

If set, only the cases with this split, "dev" or "test". If None, the cases of each split, and the cases without a split.

paramcase_idsfrozenset[str] | None
= None

If set, only the cases with these ids, for example the cases of an archived run.

Returns

Manifest

The manifest. Only active cases are selected.

Raises:

  • ProjectError: No case is selected, or two selected cases have the same id.

Manifest

modelManifest

The inputs of one run, frozen when the run starts.

It holds the reproducibility inputs without resolved environment values. build_manifest makes it. The archive keeps it as manifest.json, and a resume must give an equal manifest.

attributerun_idstr

The id of the run in the archive.

attributegit_shastr

The revision of build_manifest: the revision of the code under test, or "unknown" when the CLI cannot read it from git.

attributeprojectProject

The project of the run.

attributesuite_digestsdict[str, str]

A digest of the content of each suite, by suite name.

attributetest_digestsdict[str, str]
= dict()

A digest of the test cases and the defaults of each suite that has test cases, by suite name.

attributekint

The number of attempts of each case.

attributejudge_endpointstr

The judge endpoint of the project.

attributeprompt_shasdict[str, str]

A hash of each judge prompt and rubric, by name.

attributecaseslist[Case]

The selected cases.

attributecriteriadict[str, list[CriterionSpec]]

The criteria of each case, by case id: the project invariants, then the suite defaults, then the criteria of the case.

Report

modelReport

The metrics of a run, over all its attempts.

The report holds run metrics with private per-case detail and an adjudication queue. The archive keeps it as report.json. A set of metrics has the keys cases, attempts, pass_k (the share of all cases that have k attempts and passed each of them, so a case with fewer attempts counts as not passed), pass_any (the share of cases with a passed attempt), mean_score (the mean score of the attempts), flaky (the number of cases that passed and failed) and undersampled (the number of cases with fewer than k attempts).

attributeoveralldict[str, JsonValue]

The metrics over all requested cases, also the cases without attempts.

attributegroupsdict[str, dict[str, dict[str, JsonValue]]]

The metrics of each group, by grouping and group name. The groupings are suite, category, rule and criterion. The criterion grouping scores each criterion kind on its own verdicts.

attributecaseslist[dict[str, JsonValue]]

One row for each case: its id, k, its attempts, its passed attempts, its mean score, and the flags pass_k, pass_any, flaky and undersampled.

attributeadjudicationlist[dict[str, JsonValue]]
= list()

One row for each source id in the unseen of an attempt: the case id, the attempt number and the source id. A person decides whether the source is correct.

attributerefusalsdict[str, JsonValue]
= dict()

The results of the refusal.code criteria: a confusion matrix of the expected and the detected refusal code, the miss rate and the false refusal rate of each rule, and the number of excluded error verdicts.

attributeconsistencydict[str, JsonValue]
= dict()

How consistent the attempts of a case are, over the cases with at least two attempts that all record cited articles. mean_article_agreement is the mean share of attempts that cite the most common set of articles. score_spread gives the min, max and mean over cases of the min, max and population standard deviation of the scores of each case. It names no case.

attributesealedbool

Whether a requested case is a test case. The CLI then prints the aggregates only, without cases and adjudication.

Attempt

modelAttempt

One evaluated attempt of a case.

It is one durable completion in the run journal: a line of attempts.jsonl.

attributenumberint

The number of the attempt, from 1.

attributescoredScoredAttempt

The case and the verdicts of its criteria, in the order of criteria.

attributecriterialist[CriterionSpec]

The criteria that the attempt was evaluated with.

attributejudgedlist[int]
= list()

The positions in criteria of the criteria that called the judge, and of the built-in judge criteria. An offline rescore reuses the verdicts at these positions.

attributecited_articleslist[str] | None
= None

The source ids that the evidence cites, each without its paragraph part, sorted. None in a record without them.

attributeunseenlist[str]
= list()

The cited source ids that match no source role of the case, for each citations.unseen criterion. The report lists them for adjudication.

methodas_json() -> dict[str, JsonValue]

Return a JSON-compatible archive record.

Archives

An archive keeps runs for a resume, a report and a rescore.

FileRunArchive

classFileRunArchive(root)

A RunArchive that keeps each run as files in a directory below one root directory.

It persists complete runs beneath one injected root directory. A run <run> has these files:

  • <run>/manifest.json: the manifest, written once.
  • <run>/attempts.jsonl: the journal, one JSON object for each completed attempt.
  • <run>/evidence/<case>/<n>.json: the evidence of attempt n of a case.
  • <run>/report.json: the latest report.

The run id and the case id are percent-encoded, so each is one path component. Each JSON file is replaced as a whole: the archive writes a temporary file, flushes it to disk and renames it. The journal survives an interrupted write: a read ignores an incomplete last line, and the next append removes it.

Make an archive below root. The archive creates the directory with the first run.

paramrootPath

The directory that holds the runs.

methodcreate(run_id, manifest) -> None

Create a new run and write its manifest.

paramrun_idstr

The id of the new run.

parammanifestMapping[str, JsonValue]

The manifest as JSON.

Raises:

  • ArchiveError: The run exists already, the id is empty, . or .., or the archive cannot write the manifest.
methodappend(run_id, attempt) -> None

Append one attempt to the journal.

The archive flushes the line to disk before it returns.

paramrun_idstr

The id of the run.

paramattemptMapping[str, JsonValue]

The attempt as JSON.

Raises:

  • ArchiveError: The archive cannot write the journal.
methodattempts(run_id) -> list[dict[str, JsonValue]]

Read the attempts of the journal.

paramrun_idstr

The id of the run.

Returns

list[dict[str, JsonValue]]

The attempts of the journal as JSON, in the order of their completion. Empty when the run has no journal yet.

Raises:

  • ArchiveError: The archive cannot read the journal, or a line is not a JSON object.
methodsave_evidence(run_id, case_id, *, attempt, evidence) -> None

Write the evidence of one attempt.

paramrun_idstr

The id of the run.

paramcase_idstr

The id of the case.

paramattemptint

The number of the attempt, from 1.

paramevidenceEvidence

The evidence of the attempt, with the evidence of its variants.

Raises:

  • ArchiveError: attempt is below 1, or the archive cannot write the file.
methodload_evidence(run_id, case_id, *, attempt) -> Evidence

Read evidence for one archived attempt.

paramrun_idstr

The id of the run.

paramcase_idstr

The id of the case.

paramattemptint

The number of the attempt, from 1.

Returns

Evidence

The evidence of the attempt.

Raises:

  • ArchiveError: attempt is below 1, or the archive cannot read the file.
methodmanifest(run_id) -> dict[str, JsonValue]

Read the manifest of a run.

paramrun_idstr

The id of the run.

Returns

dict[str, JsonValue]

The manifest as JSON.

Raises:

  • ArchiveError: The archive cannot read the manifest, for example because the run does not exist.
methodsave_report(run_id, report) -> None

Replace the derived report atomically.

paramrun_idstr

The id of the run.

paramreportMapping[str, JsonValue]

The report as JSON.

Raises:

  • ArchiveError: The archive cannot write the report.

Rescores

A rescore evaluates the archived evidence of a run with the current criteria, without the arm.

rescore

functionrescoreasync (archive, run_id, *, cases, criteria, registry, ctx, offline=False, observer=None) -> Report

Evaluate the archived evidence of a run again with the current criteria, and replace its report.

A rescore does not run the arm. It evaluates the archived evidence of each attempt of the journal again, with the criteria that you give. Then the archive replaces the report of the run. The manifest, the journal and the evidence stay as they are.

paramarchiveRunArchive

The archive that holds the run.

paramrun_idstr

The id of the archived run.

paramcasesSequence[Case]

The current definitions of the cases. Each attempt of the journal needs its case here. The report includes each case, also a case without attempts. A case that was a test case in the archived run stays sealed in the report.

paramcriteriaMapping[str, Sequence[CriterionSpec]]

The criteria of each case, by case id. Each case of the journal needs an entry.

paramregistryCriterionRegistry

The criterion kinds that evaluate the criteria.

paramctxCriterionContext

What the criteria get: the judge, the language detector, the messages and the corpus lookup.

paramofflinebool
= False

If true, the rescore calls no judge. A criterion takes the archived judge verdict of the same kind and parameters, when the archived attempt has one. A built-in judge criterion without such a verdict gets the status error.

paramobserverEvaluationObserver | None
= None

What records the rescore and its attempts. If None, the rescore uses an OtelObserver of soev_evals.track when the [otel] extra is installed, and records nothing when it is not.

Returns

Report

The new report, with the k of the archived manifest.

Raises:

  • ArchiveError: The archive cannot read the run or write the report, or a case of the journal is not in cases.
  • KeyError: criteria has no entry for a case of the journal.

Observation

An observer records a run and each of its attempts, for example as OpenTelemetry spans.

EvaluationObserver

protocolEvaluationObserver

An observer of runs and rescores, for example to record traces.

Runner and rescore call it around the whole run, and around each attempt. Without an observer, they use an OtelObserver of soev_evals.track when the [otel] extra is installed, and NoObservation when it is not.

methodrun(manifest, *, operation, resume) -> AbstractContextManager[None]

Return the context manager that wraps a whole run or rescore.

parammanifestManifest

The manifest of the run.

paramoperationOperation

Whether this is a run or a rescore.

paramresumebool

Whether the run continues an archived run. Always false for a rescore.

methodattempt(manifest, case, number) -> AbstractContextManager[AttemptObservation]

Return the context manager that wraps one attempt, and gives its observation.

In a run, the block covers the conversation, the criteria and the archive writes of the attempt. In a rescore, it covers the evidence read and the criteria.

parammanifestManifest

The manifest of the run.

paramcaseCase

The case of the attempt. In a rescore, a case that was a test case in the archived run has the split test.

paramnumberint

The number of the attempt, from 1.

AttemptObservation

classAttemptObservation

What an observer records about one attempt while the attempt runs.

EvaluationObserver.attempt gives one for each attempt. The methods of this class record nothing: subclass it and override the methods that you need.

methodcriterion(spec, index, verdict, *, cached, judged) -> None

Record the verdict of one criterion of the attempt.

paramspecCriterionSpec

The criterion.

paramindexint

The position of the criterion in the criteria of the attempt, from 0. A manifest lists the project invariants first, then the suite defaults, then the criteria of the case.

paramverdictVerdict

The verdict of the criterion.

paramcachedbool

Whether an offline rescore took the verdict from the archived attempt, without a judge call.

paramjudgedbool

Whether the criterion called the judge in this evaluation.

methodcomplete(attempt, evidence) -> None

Record the result of the attempt, after all its criteria.

In a run, this happens after the archive saved the attempt. In a rescore, it happens after the attempt got its new verdicts.

paramattemptAttempt

The scored attempt.

paramevidenceEvidence

The evidence of the attempt, with the evidence of its variants.

Operation

type aliasOperation = Literal['run', 'rescore']

What an observed run does: "run" runs the cases, "rescore" evaluates the evidence of an archived run.

NoObservation

classNoObservation

An EvaluationObserver that records nothing.

Give it to Runner or rescore to turn off the default OpenTelemetry observer.

methodrun(manifest, *, operation, resume) -> AbstractContextManager[None]

Return a context manager that does nothing.

parammanifestManifest
paramoperationOperation
paramresumebool
methodattempt(manifest, case, number) -> AbstractContextManager[AttemptObservation]

Return a context manager that gives an AttemptObservation that records nothing.

parammanifestManifest
paramcaseCase
paramnumberint

soev_evals.judges

An LLM judge with the prompts of the built-in judge criteria, and an offline language detector.

soev_evals.track

Record runs and attempts as OpenTelemetry spans.

On this page

RunsRunnerbuild_manifestManifestReportAttemptArchivesFileRunArchiveRescoresrescoreObservationEvaluationObserverAttemptObservationOperationNoObservation