Overview

soev-evals runs the cases of an evaluation project against an agent, scores the answers with criteria and model judges, and archives each run on disk. It reaches the agent over HTTP, so it imports no other soev package. The command soev-evals runs, reports, rescores and seals a project. A model judge needs a runtime provider: see --runtime in the commands, or set runtime: in project.yaml. To seal a project is to record a digest of each test case, so that a later change to a sealed test case stops each command.

InstallCopy Anchor Link

soev-evals needs Python 3.12 or later.

pip install soev-evals

Where to startCopy Anchor Link

• Commands: soev-evals run, report, rescore and seal.
• Reference of soev_evals: soev_evals.core holds the models of a project and the scoring, and soev_evals.arms sends the cases to an agent.