soev-evalsReference
Commands
Run, report, rescore and seal evaluation projects.
soev-evals
soev-evals [-h] [--debug] {run,report,rescore,seal} ...| Argument | Default | Description |
|---|---|---|
--debug | Show a traceback (sealed content remains redacted) |
Installed packages can add commands through the entry-point group soev.evals.commands.
soev-evals run
Run the cases and keep their evidence
soev-evals run [-h] --project PROJECT [--runs-dir RUNS_DIR] [--suite SUITE]
[--split {dev,test}] [--k K] [--resume RUN_ID] [--run-id RUN_ID]
[--runtime RUNTIME]| Argument | Default | Description |
|---|---|---|
--project | required | Path to project.yaml |
--runs-dir | The directory of the run archives. The default is runs/ beside project.yaml. | |
--suite | The name of a suite. Repeat it to select more suites. | |
--split | Only the cases of this split. One of dev, test. | |
--k | The number of attempts of each case. It replaces runs.k of project.yaml. | |
--resume | Continue the archived run RUN_ID | |
--run-id | Optional name for a new run | |
--runtime | The name of a runtime provider in the entry-point group soev.evals.runtimes. Without it and without runtime: in project.yaml, the run has no judge, and each judge criterion gives an error verdict. |
soev-evals report
Print the archived report
soev-evals report [-h] --project PROJECT [--runs-dir RUNS_DIR] RUN| Argument | Default | Description |
|---|---|---|
--project | required | Path to project.yaml |
--runs-dir | The directory of the run archives. The default is runs/ beside project.yaml. | |
RUN | required | The id of the archived run |
soev-evals rescore
Score the archived evidence again
soev-evals rescore [-h] --project PROJECT [--runs-dir RUNS_DIR] [--offline]
[--runtime RUNTIME]
RUN| Argument | Default | Description |
|---|---|---|
--project | required | Path to project.yaml |
--runs-dir | The directory of the run archives. The default is runs/ beside project.yaml. | |
RUN | required | The id of the archived run |
--offline | Keep archived judge verdicts | |
--runtime | The name of a runtime provider in the entry-point group soev.evals.runtimes. Without it and without runtime: in project.yaml, the rescore has no judge, and each judge criterion gives an error verdict. |
soev-evals seal
Record a digest of each test case, so that a later change to a sealed test case stops each command
soev-evals seal [-h] --project PROJECT [--append]| Argument | Default | Description |
|---|---|---|
--project | required | Path to project.yaml |
--append | Accept new test cases. Each sealed test case must stay the same. |