soev_evals.track
Record runs and attempts as OpenTelemetry spans.
This package needs the [otel] extra: import soev_evals.track raises ImportError without it. Without the extra,
Runner and rescore of soev_evals.run record no spans. With it, they use an OtelObserver with the global
tracer provider, unless you give them another observer.
OpenTelemetry
An observer records spans without content, so you can export them from a run on sealed test cases.
OtelObserver
classOtelObserver(tracer=None, *, tracer_provider=None)An EvaluationObserver that records each run and each attempt as an OpenTelemetry span.
The spans hold no content: no prompts, no replies and no judge output. They hold ids, counts, statuses, scores,
token usage and the ids of the threads. A sealed test case gets a keyed hash of the run id and the case id as
soev.eval.case_key, instead of its id. The key is random for each observer, so the hash links the spans of
one run only. Your application owns the tracer provider and its exporters.
If a span call fails, the observer logs a warning and the evaluation continues. An exception in the code inside
a span sets error.type and the error status on the span, and is raised again.
The span soev.evals.run has the attributes soev.eval.run_id, .operation, .project, .revision, .k,
.resume and .suites. Each span soev.evals.attempt is a child of it, with soev.eval.run_id, .operation,
.case_id or .case_key, .suite, .split, .attempt and .duration_s. A rescore adds
soev.eval.source_run_id and .source_attempt. Each criterion adds the event soev.evals.criterion to the
attempt span. When the attempt completes, the span gets the result: soev.eval.passed, .error_count,
.gate_failed, .score, the thread ids soev.thread.ids and soev.thread.id, and the token usage
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. A failure of the conversation sets the error type
ExecutionError, and a verdict with the status error sets GradingError.
This class needs the [otel] extra.
Example
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from soev_evals.run import Runner
from soev_evals.track import OtelObserver
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
# arm, registry, ctx and archive: as in the example of Runner.
runner = Runner(arm, registry, ctx, archive, observer=OtelObserver(tracer_provider=provider))Make an observer that starts its spans with tracer.
paramtracerTracer | None= NoneThe tracer for the spans. If None, the observer gets the tracer soev_evals from
tracer_provider.
paramtracer_providerTracerProvider | None= NoneThe provider of the tracer, when tracer is None. If None too, the observer uses
the global tracer provider of OpenTelemetry.
Raises:
ValueError: Bothtracerandtracer_providerare given.
methodrun(manifest, *, operation='run', resume=False) -> AbstractContextManager[None]Record the span soev.evals.run for the code inside the with block.
parammanifestManifestThe manifest of the run.
paramoperationOperation= 'run'"run" for a run, "rescore" for a rescore of an archived run.
paramresumebool= FalseWhether the run continues an archived run.
Returns
AbstractContextManager[None]A context manager. The attempt spans inside it are children of the run span.
methodattempt(manifest, case, number) -> AbstractContextManager[AttemptObservation]Record the span soev.evals.attempt for one attempt of a case.
parammanifestManifestThe manifest of the run.
paramcaseCaseThe case of the attempt. A case with the split test gets a keyed hash instead of its id.
paramnumberintThe number of the attempt, from 1.
Returns
AbstractContextManager[AttemptObservation]A context manager that gives the observation of the attempt. The observation adds an event to the span for each criterion, and the result of the attempt when it completes.