soev_evals.arms
System-under-test adapters: run the cases of an evaluation as conversations with an agent.
An arm starts one conversation for each attempt of a case, and each conversation returns the evidence of each turn.
This package has the arm for an agent on the threads API of a soev server. To test another system, write your own
Arm of soev_evals.core.
Threads API
The arm sends each turn to a thread of the server, and reads the evidence of the turn from that thread.
SoevThreadArm
classSoevThreadArm(client, config, mapping)The arm for an agent on the threads API of a soev server.
It opens a fresh logical conversation for every attempt or variant. Each conversation runs on a new thread of
the agent config.agent.
Example
from pathlib import Path
from soev_evals.arms import SoevThreadArm, create_http_client
from soev_evals.cli import load
loaded = load(Path("project.yaml"))
project, case = loaded.project, loaded.suites[0].cases[0]
async with create_http_client(project.system) as client:
arm = SoevThreadArm(client, project.system, project.citations)
conversation = await arm.start(case)
evidence = await conversation.send(case.turns[0])
await conversation.close()Make an arm that sends its requests with client.
paramclienthttpx.AsyncClientThe client for the server, for example from create_http_client. The arm does not close it.
paramconfigSystemConfigThe system configuration: the agent, the model, the knowledge keys and the timeout.
parammappingCitationMappingHow to find the canonical source id of a cited element.
methodstartasync (case) -> SoevConversationPrepare a new conversation whose first send creates its thread.
paramcaseCaseThe case of the attempt. The arm does not read it.
Returns
SoevConversationA conversation without a thread. The first SoevConversation.send creates the thread.
SoevConversation
classSoevConversation(client, config, mapping)One conversation of a SoevThreadArm, on one thread of the server.
It retains one thread id and its last observed event position. SoevThreadArm.start makes it.
Make a conversation without a thread.
paramclienthttpx.AsyncClientThe client for the server. The conversation does not close it.
paramconfigSystemConfigThe system configuration: the agent, the model, the knowledge keys and the timeout.
parammappingCitationMappingHow to find the canonical source id of a cited element.
methodsendasync (turn) -> TurnEvidenceSend a turn, read its event stream to the end, and return the evidence of the turn.
The first send creates a thread of the agent of the system configuration with
POST v1/chat/threads, and reads the thread id from the X-Soev-Thread-Id header. Each later send posts to
the inputs of that thread. The request asks for an event stream, and the send reads the stream to its end.
Then the send reads the thread and takes the events of the root stream that came after the previous turn.
The evidence has the content of the last model output without tool calls as its reply, the citations of
each model output, and the tool calls with a summary of their outputs. A citation without a canonical id
gets the source id unresolved:<source>. The usage is set only when each model output of the turn reports
its tokens. With the [otel] extra, the requests carry the trace context of the current span.
paramturnTurnThe turn to send.
Returns
TurnEvidenceThe evidence of the turn. failure is set when the stream reported an error, a request or a thread read
failed, the thread is not idle or finished, a citation has no canonical id, the turn has no final model
output, or the turn recorded a failure, a cancellation or an exceeded budget.
Raises:
ArmError: The conversation is closed.
methodcloseasync () -> NoneClose the conversation. The thread stays on the server.
After close, send raises ArmError.
create_http_client
functioncreate_http_client(config) -> httpx.AsyncClientMake the HTTP client for the server of config, from three environment variables.
The function reads three environment variables, whose names config gives: the base URL of the server,
the bearer token, and the subject that the server records for each thread. The values stay in the client only,
so they never reach a manifest or an archive.
paramconfigSystemConfigThe system configuration of the project.
Returns
httpx.AsyncClientA client with the base URL, the Authorization and X-Soev-Subject headers, and the timeout
config.timeout_s. The caller owns the client and must close it.
Raises:
ArmError: One of the three environment variables is not set.