soev for developers
soev-agentsReferencesoev_agents.testing

soev_agents.testing.http

Conformance checks for the threads and public HTTP profiles.

The suite sends HTTP requests to a running deployment and checks the HTTP responses of the API: the event streams, the refusals, the failure and , the , the and the delete. The deployment runs ConformanceAgent, so the suite needs no live model. Run it from Python with run_conformance, or from the command line with python -m soev_agents.testing.http --url <base URL>.

The command expects ConformanceAgent under the name http-conformance (--agent). --mount public checks the routes of Mount.public(), and --profile public runs the public profile. --auth-provider module:object gives an AuthProvider.

The agent under test

The deployment serves this scripted .

ConformanceAgent

classConformanceAgent()

The scripted that the HTTP conformance suite runs against. It calls no live model.

Register it in the deployment under the name of ConformanceTarget.agent, and give create_app the argument models=["scripted"]. The suite checks that the deployment refuses another model with 422. Each fresh input is one command:

  • "output": record a conformance_note with the text "é世界", then the output of a scripted model.
  • "request": record a Request of the kind clarification that expects an integer. A response continues the as "output" does.
  • "slow": record nothing and wait until the is cancelled.
  • "fail": raise a TransientError, so that the fails. A then continues as "output" does.
  • An integer: continue as "output" does.

The scripted model gives the text "scripted output é世界" and the reasoning "scripted reasoning". A streamed call yields both as fragments first. The output names the model of the latest input, or "scripted".

Make the , with slow_started clear.

attributeinput
= …

The . It accepts the four commands and any integer.

attributeelement_typesSequence[type[Element]]
= ()

No types.

attributeevent_typesSequence[type[Event]]
= …

The type of the conformance_note .

attributeslow_started

An asyncio.Event that is set while a "slow" command waits. A ConformanceSignals can wait on it in wait_running.

methodbuild(resources, turn, model) -> Graph

Build the that runs the command that opened the .

paramresourcesobject

Not read.

paramturn_Command

The command that opened the .

parammodelstr | None

Not read. The scripted model names the model of the latest input.

Returns

Graph

The that runs the command.

Running the suite

A target says where the deployment is and which policies it applies. The report says which checks passed.

run_conformance

functionrun_conformanceasync (target, *, profile='threads') -> ConformanceReport

Run the conformance suite against target, and delete each that it opened.

The threads profile checks the API: the contract and capabilities, history and , and refusals, failure and , a running and its , and delete, the declared policies, and the OpenAPI document. The public profile adds the , model and lists, and the completions, for each of these routes that target.mount enables. The suite stops at the first check that fails. It skips no check of an operation that the target enables.

paramtargetConformanceTarget

The deployment and its policies.

paramprofileLiteral['threads', 'public']
= 'threads'

"threads" for the API, or "public" to add the lists and the completions.

Returns

ConformanceReport

The report: the profile, the checks that passed and the number of HTTP requests.

Raises:

  • AssertionError: A check failed. For a wrong , the message names the operation and both .
  • TimeoutError: An HTTP request, or a wait for a signal, took longer than target.timeout.
  • httpx.HTTPError: The suite could not reach the deployment.

ConformanceTarget

dataclassConformanceTarget(base_url, mount, auth, agent, model=None, timeout=10.0, transport=None, unauthorized_status=None, conceal_foreign=False, error_media_type='application/json', signals=None, task_error=None)

The deployment that the suite checks, and the policies that it applies.

parambase_urlstr

The base URL of the deployment.

parammountMount

The routes that the deployment serves. The suite checks only the routes that the Mount enables. Mount.public() gives the routes of the public profile.

paramauthAuthProvider

The provider of the headers of each HTTP request.

paramagentstr

The name under which the deployment serves ConformanceAgent.

parammodelstr | None
= None

The model that each opened names, or None for the deployment's default model.

paramtimeoutfloat
= 10.0

The seconds that one HTTP request, or one wait for a signal, can take.

paramtransporthttpx.AsyncBaseTransport | None
= None

The httpx transport, for example an httpx.ASGITransport for an app in the same process, or None for the network.

paramunauthorized_statusint | None
= None

The that the deployment gives an unauthorized open, or None to skip the admission check.

paramconceal_foreignbool
= False

Whether the deployment returns 404 for a of another identity. True adds the ownership check.

paramerror_media_typestr
= 'application/json'

The media type of the deployment's error bodies.

paramsignalsConformanceSignals | None
= None

The readiness signals of a deployment in the same process, or None: the suite then polls the until it runs, and reads the event stream of the .

paramtask_errorMapping[str, str] | None
= None

The error object of the last of a failed streamed completion, or None to check only that it has a code and a message.

ConformanceReport

dataclassConformanceReport(profile, checks, requests)

The result of a conformance run that passed.

paramprofilestr

The profile that ran: "threads" or "public".

paramcheckstuple[str, ...]

The name of each group of checks that passed, in run order.

paramrequestsint

The number of HTTP requests that the suite sent, cleanup included.

ConformanceSignals

protocolConformanceSignals

The readiness signals of a deployment in the same process, which the suite waits on.

The suite needs them when an ASGI transport buffers the event streams. Give them when the deployment runs in the same process as the suite. The suite then waits on these signals, and not on the event stream, before it checks a running or deletes a during a .

methodwait_runningasync (thread_id) -> None

Return when the of a "slow" command started on the .

paramthread_idstr

The of the .

methodwait_eventsasync (thread_id, after) -> None

Return when the event stream of the , read after the after, sent its first .

paramthread_idstr

The of the event stream.

paramafterint

The that the event stream was read after.

Authentication

An auth provider gives the headers of each HTTP request that the suite sends.

AuthProvider

protocolAuthProvider

The source of the authentication headers of each HTTP request that the suite sends.

methodheadersasync (request, *, identity) -> Mapping[str, str]

Return the headers that authenticate request as identity.

The suite calls it once for each HTTP request, cleanup included, after the body is serialized. So a provider can sign the exact body. Read secrets from the provider's own environment.

paramrequestTestRequest

The HTTP request that the suite is about to send.

paramidentitystr

Who sends it: "owner" for the normal checks, "unauthorized" for the admission check, and "other" for the ownership check.

Returns

Mapping[str, str]

The headers to add to the HTTP request.

NoAuth

classNoAuth

The auth provider of an anonymous deployment: it adds no headers.

methodheadersasync (request, *, identity) -> Mapping[str, str]

Return no headers, for any HTTP request and any identity.

paramrequestTestRequest

The HTTP request that the suite is about to send.

paramidentitystr

Who sends it.

Returns

Mapping[str, str]

An empty mapping.

TestRequest

dataclassTestRequest(operation, method, path, body, thread_id, after=0)

One HTTP request that the suite is about to send, as its auth provider sees it.

paramoperationstr

What the HTTP request does: open, read, inputs, resume, events, fork, cancel, delete, health, capabilities, openapi, tools, models, list or task.

parammethodstr

The HTTP method.

parampathstr

The path, with its query string, below the base URL.

parambodybytes

The exact body that the suite sends, as JSON bytes. Empty for an HTTP request without a body.

paramthread_idstr | None

The that the HTTP request is about, or None.

paramafterint
= 0

The that an events request sends as Last-Event-ID, else 0.

soev_agents.testing

Check that an agent keeps the determinism and no-hidden-state rules, and that a store keeps its contract.

Glossary

The terms that this reference uses, defined once.

On this page

The agent under testConformanceAgentRunning the suiterun_conformanceConformanceTargetConformanceReportConformanceSignalsAuthenticationAuthProviderNoAuthTestRequest