soev_evals.criteria

Return a new registry with the 22 built-in criterion kinds.

Each call returns a new registry. The CLI adds the kinds of the CRITERIA list of the project's hooks module to the built-in kinds. Each criterion also takes the weight and gate of CriterionSpec. The built-in kinds, with their parameters and defaults:

• reply.language: language, min_share (default 1.0). Passes when at least min_share of the detected sentences of the reply are in language.
• reply.contains: substrings and regex (default empty lists), case_sensitive (default false). Passes when each pattern matches the reply.
• reply.not_contains: the parameters of reply.contains. Passes when no pattern matches the reply.
• reply.max_length: max. Passes when the reply has at most max Unicode characters.
• privacy.none: target (reply or prompt, default reply), kinds (default bsn, iban, email, nl_phone). Passes when the target holds none of the kinds. kinds replaces the default list. The other kinds are postcode, birthdate, id_number and license_plate.
• privacy.names: target. Passes when a judge finds no personal name in the target (label none).
• behaviour.expected: no parameters. Passes when a judge labels the reply with the expected behaviour of the case: answer, refuse or abstain.
• behaviour.message: no parameters. Passes when the reply holds the expected project message of the case, after whitespace and case normalization.
• refusal.code: no parameters. Passes when the detected refusal code is the expected message code of a refuse or abstain case, or none for another case.
• citations.required: match (hierarchical or exact, default hierarchical), min (default 1.0). Passes when the citations cover at least min of the required source groups.
• citations.forbidden: match. Passes when no citation matches a forbidden source.
• citations.in_corpus: no parameters. Passes when each cited id is in the corpus lookup.
• citations.unseen: match. Always passes, and lists each cited id without a source role for adjudication.
• citations.quotes_verbatim: no parameters. Passes when each double-quoted span of the reply occurs in one corpus unit. A reply without quotes is not applicable.
• citations.mentions_in_corpus: no parameters. Passes when each article that the reply mentions resolves to a corpus unit.
• nuggets.coverage: min (default 1.0). Passes when a judge finds at least min of the vital nuggets in the reply. A partial nugget counts 0.5.
• rubric.judge: rubric (a path), labels, pass_labels. Passes when a judge gives the reply a label of pass_labels.
• compare.equivalent: no parameters. Passes when a judge finds the reply of each variant equivalent to the reply of the base case.
• compare.same_citations: no parameters. Passes when each variant cites the same set of articles as the base case.
• usage.max_tokens: max. Passes when the prompt and completion tokens of all turns are at most max.
• usage.max_seconds: max. Passes when the durations of all turns are at most max seconds.
• run.no_failure: no parameters. Passes when no turn recorded a failure.

The five kinds behaviour.expected, privacy.names, nuggets.coverage, rubric.judge and compare.equivalent need a judge, and reply.language needs a language detector. citations.in_corpus, citations.quotes_verbatim and citations.mentions_in_corpus need the corpus lookup data. When the service or the data is not in the CriterionContext, such a kind returns a verdict with the status error for each case that it applies to.

A criterion kind made of a name, a parameter model and a typed evaluator.

Builtin adapts a typed evaluator to the public CriterionKind protocol. The built-in kinds are Builtin instances, and you can make your own kind in the same way. A project registers its own kinds in the CRITERIA list of its hooks module. A suite then names the kind in the kind of a criterion and gives its parameters next to it.

from soev_evals.core import Case, CriterionContext, Evidence, FrozenModel, Verdict
from soev_evals.criteria import Builtin, error, outcome


class ThanksParams(FrozenModel):
    word: str = "dank"


async def thanks(case: Case, evidence: Evidence, *, params: ThanksParams, ctx: CriterionContext) -> Verdict:
    if evidence.reply is None:
        return error("missing final reply")
    return outcome(passed=params.word in evidence.reply.casefold(), detail=f"word={params.word}")


CRITERIA = [Builtin("custom.thanks", ThanksParams, thanks)]

The async function that grades the evidence for a Builtin kind.

An evaluator is an async function that grades the evidence of one attempt of a case. Builtin gives it the parameters of the criterion as an instance of its own parameter model P. An evaluator does no I/O of its own: it gets the judge, the language detector and the project lookup data through the CriterionContext.

The parameter model of a criterion kind that accepts no kind-specific parameters.

The model has no fields and refuses unknown fields. So when a criterion of such a kind gives a parameter, the registry raises CriterionParamsError when it validates the criterion. The weight and gate of a criterion are not kind-specific parameters, so they stay allowed.

Return a pass or fail verdict with a useful explanation.

Use it when the criterion can grade the evidence.

Return a verdict with the status error.

Use it when the criterion cannot grade the evidence, for example because the reply is missing or the judge is not configured. The score of the attempt leaves the verdict out, but the attempt does not pass.

Return a verdict that counts neither in the score nor in the pass of the attempt.

Use it when the criterion does not apply to the case, for example when the case declares no expected message.

Validate injected judge output and expose unusable results as errors.

The function asks the judge of ctx to label request. It catches the failures of the judge, so an evaluator can return the verdict when it gets one:

result = await judge_label(request, ctx)
if isinstance(result, Verdict):
    return result