soev for developers
soev-agentsTutorials

Tutorial 6: serve an agent and chat with it

Serve the agent of tutorial 3 over HTTP, call the server with curl, and write a command-line chat client that streams the answer.

In this tutorial, you serve the agent of tutorial 3 over HTTP and talk to it from the command line. You start a server, call it with curl to see what it sends, and then write a small chat client that shows the answer while the model writes it. The server gives each request the library of its user, as the script of tutorial 3 did.

Do tutorial 3 first: you serve its agent.

Set up

Use the repository and the environment variables of tutorial 1. Make a folder for this tutorial:

mkdir -p my_tutorials/06-serve

The server serves the agent of tutorial 3, so my_tutorials/03-many-users must hold its agent.py, prompt.yaml, library.py and tools.py. You need two terminals: one for the server, and one for the client.

Step 1: write the server

Make my_tutorials/06-serve/server.py:

server.py
import os

import uvicorn
from agent import LIBRARIES, assistant
from fastapi import Request
from library import Library
from openai import AsyncOpenAI

from soev_agents.adapters import InMemoryTraceStore, OpenAIChatModel
from soev_agents.entrypoints.http import create_app
from soev_agents.runtime import Agent
from soev_agents.toolkit import Text

model = OpenAIChatModel(AsyncOpenAI(), model=os.environ["OPENAI_MODEL"])


async def library(request: Request, agent: Agent[Library, Text]) -> Library:
    user = request.headers.get("x-user", "resident")
    return LIBRARIES.get(user, LIBRARIES["resident"])


app = create_app(
    InMemoryTraceStore(),
    {"assistant": assistant(model)},
    version="tutorial-6",
    resources=library,
    default_model=model.model,
)

if __name__ == "__main__":
    uvicorn.run(app, host="127.0.0.1", port=8000)

create_app makes a FastAPI application, a web application that serves the thread API of soev-agents:

  • The store and the agent are the same as in tutorial 3. One agent serves all requests.
  • resources gets the resources for each request. It receives the HTTP request, so it can choose the library of the user who asks. Here, it reads the user from the header X-User. A request without a known user gets the library of a resident.
  • default_model is the model for an input that names none. The input records it, so the trace always shows which model answered.

A header such as X-User is only for this tutorial: a client can write any value in it. A real server reads the user from its authentication, never from a value that the client chooses.

from agent import ... imports the agent of tutorial 3. PYTHONPATH in the next command tells Python to look for modules in the folder of tutorial 3.

uvicorn is the web server that runs the application. Start it in the first terminal:

PYTHONPATH=my_tutorials/03-many-users uv run python my_tutorials/06-serve/server.py

The server has no authentication. Run it only on your own computer.

Step 2: call the server with curl

In the second terminal, open a thread with a first question:

curl -N http://127.0.0.1:8000/threads \
  -H 'Content-Type: application/json' -H 'Accept: text/event-stream' \
  -d '{"agent": "assistant", "input": "How much is a parking permit?"}'

Accept: text/event-stream asks for server-sent events: a response that sends each event when the server records it. The output is similar to this, shortened:

id: 1
event: opened
data: {"id":"root:0","position":1,...,"type":"opened","payload":{"agent":"assistant"}}

id: 2
event: input
data: {"id":"root:1","position":2,...,"type":"input","payload":{"elements":[],"payload":"How much is a parking permit?","model":"gpt-4.1-mini"}}

event: calling
data: {"model":"gpt-4.1-mini","node":"llm","produces":"model_output"}

id: 3
event: model_output
data: {"id":"root:2","position":3,"type":"model_output","payload":{"content":"","tool_calls":[{"id":"call-2","name":"search","arguments":{"query":"parking permit"}}],...}}

id: 4
event: tool_output
data: {"id":"root:3","position":4,"type":"tool_output","payload":{"elements":[{"type":"document","id":"parking-permits",...},...]}}

event: calling
data: {"model":"gpt-4.1-mini","node":"llm","produces":"model_output"}

event: delta
data: {"text":"A resident parking permit costs 120 euros a year"}

event: citation
data: {"at":48,"status":"resolved","source":"parking-permits#0","document":"parking-permits","reason":null}

event: delta
data: {"text":"."}

id: 5
event: model_output
data: {"id":"root:4","position":5,"type":"model_output","payload":{"content":"A resident parking permit costs 120 euros a year.","citations":[...],...}}

id: 6
event: finished
...

event: status
data: {"state":"finished","pending":[],"position":6}

There are two kinds of frames:

  • A frame with an id: line is an event of the trace. The id is its position in the log of the thread, counted from 1. It is one more than the seq that you printed in tutorial 1. These are the same events that you printed there.
  • A frame without an id: line is not recorded. calling comes before each model call. delta carries a piece of the answer while the model writes it. citation says where a citation stands. status ends the response and says where the thread stands.

The answer text has no markers. The citation frame and the citations of the model_output give the position (at) and the passage (source) of each citation. A client shows a citation in its own way.

The response has a Location header, such as /threads/d9568d51efd44af8b6a73057859b53b1. To see it, add -i to the command. Send a follow-up question to that thread:

curl -N http://127.0.0.1:8000/threads/<id>/inputs \
  -H 'Content-Type: application/json' -H 'Accept: text/event-stream' \
  -d '{"input": "And the dog tax?"}'

To read the whole thread, use curl http://127.0.0.1:8000/threads/<id>. The server also serves its API documentation at http://127.0.0.1:8000/docs.

Now ask as an employee. The header X-User names the user:

curl -N http://127.0.0.1:8000/threads \
  -H 'Content-Type: application/json' -H 'Accept: text/event-stream' -H 'X-User: employee' \
  -d '{"agent": "assistant", "input": "Is there an exemption from the dog tax?"}'

The tool_output frame of this thread holds the internal document dog-tax-exemptions. Send the same question without the header: the search finds only the public documents.

Step 3: read the frames in Python

Make my_tutorials/06-serve/chat.py. Start with a function that reads server-sent events:

chat.py
import json
import os
from collections.abc import Iterator
from typing import Any

import httpx

SERVER = os.environ.get("SOEV_URL", "http://127.0.0.1:8000")
USER = os.environ.get("SOEV_USER", "resident")


def frames(response: httpx.Response) -> Iterator[tuple[str, dict[str, Any]]]:
    event, data = "", ""
    for line in response.iter_lines():
        if line.startswith("event:"):
            event = line.removeprefix("event:").strip()
        elif line.startswith("data:"):
            data += line.removeprefix("data:").strip()
        elif not line and event:
            yield event, json.loads(data)
            event, data = "", ""

USER is the user that the client sends in X-User. To chat as an employee, set SOEV_USER=employee.

A frame is a group of lines that ends with an empty line. frames returns the type of each frame and its data. It ignores the id: lines and the keepalive comments that the server sends on an idle connection.

Step 4: send a message and show the answer

chat.py
def send(client: httpx.Client, path: str, body: dict[str, Any]) -> str | None:
    texts: dict[str, str] = {}
    cited: list[str] = []
    print("assistant> ", end="", flush=True)
    headers = {"Accept": "text/event-stream", "X-User": USER}
    with client.stream("POST", path, json=body, headers=headers) as response:
        response.raise_for_status()
        for event, data in frames(response):
            if event == "delta":
                print(data["text"], end="", flush=True)
            elif event == "citation" and data["status"] == "resolved":
                if data["source"] not in cited:
                    cited.append(data["source"])
                print(f" [{cited.index(data['source']) + 1}]", end="", flush=True)
            elif event == "model_output":
                for call in data["payload"]["tool_calls"]:
                    print(f"({call['name']} {json.dumps(call['arguments'])}) ", end="", flush=True)
            elif event == "tool_output":
                texts.update({each["id"]: each["text"] for each in data["payload"]["elements"] if each["text"]})
    print()
    for number, source in enumerate(cited, start=1):
        print(f"  [{number}] {texts[source]}")
    return response.headers.get("location")

send posts one message and prints the frames that it needs:

  • delta: it prints the text at once, so you see the answer while the model writes it.
  • citation: it prints a marker. The client numbers the passages of each answer from 1, in its own way.
  • model_output: it prints the tool calls of the model, such as the search query.
  • tool_output: it keeps the text of each passage, to print the cited passages at the end.

It returns the Location header: the address of the thread.

Step 5: write the chat loop

chat.py
def main() -> None:
    location = None
    with httpx.Client(base_url=SERVER, timeout=None) as client:
        while True:
            try:
                text = input("you> ")
            except EOFError:
                print()
                return
            if location is None:
                location = send(client, "/threads", {"agent": "assistant", "input": text})
                print(f"(thread {location})")
            else:
                send(client, f"{location}/inputs", {"input": text})


if __name__ == "__main__":
    main()

The first message opens a thread, and the client prints the address of the thread. Each next message goes to the same thread, so the agent has the whole conversation. Run the client in the second terminal:

uv run python my_tutorials/06-serve/chat.py

Type a question, and press Enter. To stop, press Ctrl+D. A conversation is similar to this:

you> How much is a parking permit?
assistant> (search {"query": "parking permit"}) A resident parking permit costs 120 euros a year [1].
  [1] A resident parking permit costs 120 euros a year.
(thread /threads/d9568d51efd44af8b6a73057859b53b1)
you> And the dog tax?
assistant> (search {"query": "dog tax"}) The dog tax is 98 euros a year for the first dog [1].
  [1] The dog tax is 98 euros a year for the first dog and 130 euros for each other dog.
you>

The client keeps nothing but the address of the thread. The server has the whole conversation. Stop the client and run curl http://127.0.0.1:8000<address>, with the address that the client printed: all events are there. If you stop the server, the conversation is gone, because the store is in memory. For a store that keeps threads, use PostgresTraceStore.

What you built

  • A server for your agent, made with create_app, that gives each request the library of its user.
  • Calls to the thread API with curl: open a thread, send a follow-up question, read the thread, and ask as another user.
  • A chat client that streams the answer and shows its citations.

The complete code is in examples/tutorials/06-serve. The HTTP API guide is in packages/soev-agents/src/soev_agents/entrypoints/http/docs.

Tutorial 5: a refund agent with approval and a safe payment

Write a graph of your own for a refund agent that asks an employee for approval and pays once, also after a failure.

How-to guides

Recipes for one task each, for readers who know the basics of soev-agents.

On this page

Set upStep 1: write the serverStep 2: call the server with curlStep 3: read the frames in PythonStep 4: send a message and show the answerStep 5: write the chat loopWhat you built