Tutorial 4: control what the model sees

The model knows only what it receives at each call. When a model gives a strange answer, the first question is: what did it receive? In this tutorial, you print exactly what one model call received. Then you give the assistant a context policy: rules that keep a long conversation small enough for the model. You see an old search result shrink, and the model get a passage back with the tool reopen.

Do tutorial 2 first: this agent uses its tools.

Set upCopy Anchor Link

Make a folder for this tutorial, and copy the library and the tools of tutorial 2 into it:

mkdir -p my_tutorials/04-context
cp my_tutorials/02-citations/library.py my_tutorials/02-citations/tools.py my_tutorials/04-context/

This tutorial does not change library.py or tools.py.

Step 1: understand the context builderCopy Anchor Link

At each model call, the agent makes a request: the system prompt, the messages of the conversation, and the tools that the model can call. The part of the agent that makes the request is the context builder. It is a function from the trace to the request.

• It builds the request again at each call, from the trace alone. It stores nothing.
• The trace keeps everything that happened. The request is a view of the trace: the context builder chooses what the model sees, and in which form.
• So the context builder can make a request smaller, and the trace still has the full text.

In ReActAgent, the setting context of ReActConfig holds the context builder. By default, it is MessageBuilder(): the builder of the toolkit, with its default settings. MessageBuilder has its own settings:

In tutorial 2, the <document> and <passage> tags were the work of show.

ReActAgent gives the builder the parts of each turn: the model, the system prompt, the tools of the turn, and the budget of tool calls. Some settings of ReActConfig also reach the request: prompt, render_input (how the model sees an input of the user), and limits (the context window of the model).

In this tutorial, you change the policy of MessageBuilder.

Step 2: give the agent a context policyCopy Anchor Link

Make my_tutorials/04-context/agent.py. These are all its imports, the prompt, and the questions that you ask:

import asyncio
import json
import os

from library import LIBRARY
from openai import AsyncOpenAI
from tools import Document, Passage, read_document, search

from soev_agents.adapters import InMemoryTraceStore, OpenAIChatModel
from soev_agents.domain import Event, Input, ModelOutput, ToolOutput, Trace
from soev_agents.ports import AssistantMessage, ChatRequest
from soev_agents.runtime import AgentCatalog, accepted, replay
from soev_agents.toolkit import BRACKETS, ContextPolicy, MessageBuilder, ReActAgent, ReActConfig, Text, Tokens

PROMPT = """You are the assistant of the town of Springfield.
Answer questions from the town documents. If the documents do not answer the question, say so. Never guess.
After each fact, cite the passage that gives it with its number, like [1]."""

QUESTIONS = (
    "How often can I have bulky waste picked up?",
    "How much is a parking permit?",
    "And when is residual waste collected?",
)

Now make the agent. It is the agent of tutorial 2, with one new setting, context:

def assistant(model: OpenAIChatModel) -> ReActAgent[None, Text]:
    config = ReActConfig(
        model=model,
        prompt=PROMPT,
        tools=[search.bind(library=LIBRARY), read_document.bind(library=LIBRARY)],
        citations=BRACKETS,
        context=MessageBuilder(ContextPolicy(start_at=Tokens(500), target=Tokens(250), min_gain=None)),
    )
    return ReActAgent(config, element_types=(Document, Passage))

context=MessageBuilder(ContextPolicy(...)) reads: the request is built by a MessageBuilder, with this policy. Each agent already has a policy: ContextPolicy() with its defaults. Before each model call, the policy measures the request. If the request is larger than start_at, the policy makes older tool results smaller until the request is below target.

The defaults suit a real context window and real documents:

The documents of this tutorial are small, so with the defaults nothing shrinks. The small values here make the policy act in a conversation of three questions. Do not use them for a real agent.

The policy makes a tool result smaller in steps. By default:

1. Results of all turns but the last two are compressed: each passage loses its text, and its tag stays.
2. Results of all turns but the last four are stubbed: the result becomes one line.
3. When the request is above 85% of the window, the results of the last turns are also stubbed.

A passage that an answer cited keeps its text in each step. The model can still read the facts that it gave to the user.

Step 3: print what a model call receivedCopy Anchor Link

Add a function that gets the requests of a thread:

async def requests(agent: ReActAgent[None, Text], trace: Trace) -> list[ChatRequest]:
    """What each model call of the thread received, built again from the events of the thread."""
    calls = await replay(trace.by_type(Event), lambda input: agent.build(None, accepted(Text, input)[1], input.model))
    return [call.request for call in calls if call.request is not None]

replay builds each request of the thread again, from its events. It runs nothing and calls no model. For each turn, it needs the graph of that turn, so you give it a function that builds the graph from the input of the turn. accepted(Text, input) parses the input into Text, as the catalog does. The first argument of build, None, is the resources: this agent has none.

replay works because the context builder stores nothing. The same trace and the same code give the same request. You can use replay on any thread in a store, also a thread of last week.

Next, add a function that prints a request:

def show(request: ChatRequest) -> None:
    print("Tools:", ", ".join(tool.name for tool in request.tools))
    for message in request.messages:
        text = message.content if isinstance(message.content, str) else "(images)"
        if isinstance(message, AssistantMessage) and message.tool_calls:
            text = " ".join(f"{call.name}({json.dumps(call.arguments)})" for call in message.tool_calls)
        print(f"[{type(message).__name__}] {text}")

It prints the tools, and then each message with its type. For a tool call, it prints the name of the tool and its arguments.

Step 4: ask three questionsCopy Anchor Link

Add main, and the lines that run it:

async def main() -> None:
    model = OpenAIChatModel(AsyncOpenAI(), model=os.environ["OPENAI_MODEL"])
    agent = assistant(model)
    agents = AgentCatalog(InMemoryTraceStore(), {"assistant": agent}, version="tutorial-4")

    first = Input(payload=QUESTIONS[0], model=model.model)
    thread, _ = await agents.open("assistant", first)
    trace = await thread.resume()
    for question in QUESTIONS[1:]:
        follow_up = Input(payload=question, model=model.model)
        thread = await agents.thread(thread.id, input=follow_up)
        trace = await thread.send(follow_up)
    print("Answer:", trace.by_type(ModelOutput)[-1].content)

    show((await requests(agent, trace))[-1])

    found = trace.by_type(ToolOutput)[0]
    print("In the trace:", [element.text for element in found.elements if isinstance(element, Passage)])


if __name__ == "__main__":
    asyncio.run(main())

main asks the three questions in one thread, prints the last answer, and prints the request of the last model call. At the end, it prints the passages of the first search result, as the trace holds them.

Run the agent:

uv run python my_tutorials/04-context/agent.py

The output is similar to this:

Answer: Residual waste is collected every Tuesday morning.
Tools: search, read_document, reopen
[SystemMessage] You are the assistant of the town of Springfield.
Answer questions from the town documents. If the documents do not answer the question, say so. Never guess.
After each fact, cite the passage that gives it with its number, like [1].
[UserMessage] How often can I have bulky waste picked up?
[AssistantMessage] search({"query": "bulky waste pickup"})
[ToolResult] [r1: text left out to save space]
<document id="1" title="Waste collection">
<passage id="2" cite="[2]">Bulky waste, such as furniture, is picked up after an appointment. Each household can have it picked up for free four times a year. Each extra pickup costs 25 euros.</passage>
<passage id="3" cite="[3]"/>
</document>
[AssistantMessage] Four times a year, for free [2].
[UserMessage] How much is a parking permit?
[AssistantMessage] search({"query": "parking permit"})
[ToolResult] <document id="4" title="Parking permits">
<passage id="5" cite="[5]">A resident parking permit costs 120 euros a year.</passage>
<passage id="6" cite="[6]">Each household can have at most two resident parking permits.</passage>
<passage id="7" cite="[7]">Apply for a permit on the town website. The permit starts on the day after the payment.</passage>
</document>
[AssistantMessage] A resident parking permit costs 120 euros a year [5].
[UserMessage] And when is residual waste collected?
[AssistantMessage] reopen({"elements": ["3"]})
[ToolResult] <document id="1" title="Waste collection">
<passage id="3" cite="[3]">Residual waste is collected every Tuesday morning. Put the container on the street before 7:30.</passage>
</document>
In the trace: ['Bulky waste, such as furniture, is picked up after an appointment. Each household can have it picked up for free four times a year. Each extra pickup costs 25 euros.', 'Residual waste is collected every Tuesday morning. Put the container on the street before 7:30.']

Read the request from the top:

1. The first search result is compressed. At the third question, the request was larger than start_at, so the policy compressed the results of the first turn. That is all turns but the last two. The result starts with its handle, r1: a name for the whole result.
2. Passage [2] kept its text, because the first answer cited it.
3. Passage [3] lost its text. Its tag stays, so the model knows that the passage exists, and can name it by its number.
4. The tools now include reopen. The context builder offers it from the first call that shrinks a result. reopen takes element numbers, or result handles such as r1.
5. The model called reopen for passage [3], because the third question is about residual waste. The result of reopen shows the passage again. The answer cites [3]: the same number as before. The printed answer has no marker, because the citations are data (tutorial 2).
6. The trace still holds the full text of both passages. The policy changed only the request.

Your model can also search again, instead of reopen: both give it the passage. A model does not always choose the same way.

With these small documents, the policy saves almost nothing. The description of reopen is longer than the text that the policy left out. With real documents, one search result often holds thousands of tokens, and the policy frees much more room.

Step 5: know the other settingsCopy Anchor Link

ContextPolicy has more settings. You do not need them for this tutorial, but they are what you change when the requests of a real agent do not fit in the window:

• result_cap: the largest tool result that the model reads, by default 8,000 tokens. The context builder cuts a larger result, and says how much it left out.
• output_reserve: the room kept free for the answer, by default 8,192 tokens. A call whose request leaves less room fails before it reaches the model.
• tools: a form for each tool, by name. For example, tools={"read_document": Forms(full=sources(cap(Tokens(500))))} shows each passage of read_document with at most 500 tokens. Forms, sources, cap and Tokens import from soev_agents.toolkit.
• compact: the model writes a summary of older turns, and the summary replaces them in the request. Each summary is a Compaction event in the trace.
• reopen: the reopen tool. None switches it off.

Compaction is a different mechanism from the shrink steps. A shrink step changes only the request, but compaction makes an extra model call and records the summary in the trace. It is still part of the same context management, and it depends on the context builder in three ways:

1. Compaction starts when the request that the builder made, after its shrink steps, is larger than compact.when.
2. The summary call receives the request of the builder, with the instructions for the summary after it.
3. The summary helps only if the builder shows it in place of the turns that it replaced. MessageBuilder does this.

That is why compact is in the policy of the builder, and not in a separate setting. If a summary does not change the next request, the builder ignored it, and the turn fails with CompactionError.

The reference for each setting is in ContextPolicy and Compact.

Step 6: choose what the model seesCopy Anchor Link

The context policy decides how much of the conversation the model sees. Sometimes you want a different view. Choose the smallest change that does what you need:

A forgetful agentCopy Anchor Link

Some agents must forget each earlier turn. For example, a kiosk in the town hall where each visitor asks one question. select of MessageBuilder chooses the events that the model sees. Make my_tutorials/04-context/forgetful.py:

import asyncio
import os

from library import LIBRARY
from openai import AsyncOpenAI
from tools import Document, Passage, read_document, search

from soev_agents.adapters import InMemoryTraceStore, OpenAIChatModel
from soev_agents.domain import Event, Input, ModelOutput, Trace
from soev_agents.runtime import AgentCatalog, turn_starts
from soev_agents.toolkit import BRACKETS, MessageBuilder, ReActAgent, ReActConfig

PROMPT = """You are the assistant of the town of Springfield.
Answer questions from the town documents. If the documents do not answer the question, say so. Never guess.
After each fact, cite the passage that gives it with its number, like [1]."""


def last_turn(trace: Trace) -> list[Event]:
    """The events of the running turn: the model forgets each earlier turn."""
    events = trace.by_type(Event)
    return list(events[turn_starts(events)[-1] :])

last_turn is a function from the trace to the events that the model sees. turn_starts gives the position of each input that starts a turn, so the function keeps the events from the last of them.

Now give last_turn to MessageBuilder as select, and ask a question with a follow-up question:

async def main() -> None:
    model = OpenAIChatModel(AsyncOpenAI(), model=os.environ["OPENAI_MODEL"])
    config = ReActConfig(
        model=model,
        prompt=PROMPT,
        tools=[search.bind(library=LIBRARY), read_document.bind(library=LIBRARY)],
        citations=BRACKETS,
        context=MessageBuilder(select=last_turn),
    )
    agent = ReActAgent(config, element_types=(Document, Passage))
    agents = AgentCatalog(InMemoryTraceStore(), {"assistant": agent}, version="tutorial-4")

    first = Input(payload="When is residual waste collected?", model=model.model)
    thread, _ = await agents.open("assistant", first)
    print("Answer:", (await thread.resume()).by_type(ModelOutput)[-1].content)

    follow_up = Input(payload="And paper?", model=model.model)
    thread = await agents.thread(thread.id, input=follow_up)
    trace = await thread.send(follow_up)
    print("Answer:", trace.by_type(ModelOutput)[-1].content)
    print("In the trace:", [event.payload for event in trace.by_type(Input)])


if __name__ == "__main__":
    asyncio.run(main())

Run it:

uv run python my_tutorials/04-context/forgetful.py

The output is similar to this:

Answer: Residual waste is collected every Tuesday morning.
Answer: What do you want to know about paper?
In the trace: ['When is residual waste collected?', 'And paper?']

In tutorial 1, the model understood "And paper?", because it got the whole conversation. This agent sends only "And paper?", so the model does not know what the question is about. The trace still holds both questions: select changes only the request. Each element keeps its number in the whole thread, so a citation in a later turn still names the right passage.

A context builder of your ownCopy Anchor Link

When the layout of the request itself must change, write a context builder of your own, and give it as context. ReActAgent asks three things of it, which the protocol ContextStrategy in soev_agents.toolkit describes:

1. for_turn: make the context builder of one turn from the parts of the turn. The parts are the model, the prompt, the offered tools, the budget, how an input renders, the citation strategy and the limits.
2. added_tools: the tools that the builder offers itself, such as reopen. The tool step of the agent runs them.
3. wrap: what the builder adds to the model step, such as compaction. A builder without compaction returns the step unchanged.

MessageBuilder does all three from its policy. A builder of your own must offer the tools that it gets, and stop when the budget says so. If it does not, the turn stays correct: the tool step still checks each call, and the agent still stops tool calls after passes. But the model gets worse instructions.

What you builtCopy Anchor Link

• A function that prints exactly what a model call received, built again from the trace with replay.
• A context policy that shrinks older tool results, and keeps the cited passages.
• A conversation in which the model used reopen to read a passage again.
• A forgetful agent, with select of MessageBuilder.

You also saw the rule behind it: the trace keeps everything, and the context builder makes each request as a view of the trace. The policy decides how much the model sees, the settings of MessageBuilder decide what it sees, and a context builder of your own decides how the request is laid out.

The complete code is in examples/tutorials/04-context.

Next, in tutorial 5, you build a graph yourself, for an agent that is not a ReAct loop.