Tutorial 2: cite the documents

In tutorial 1, the assistant answered from the town documents, but its answers named no source. A reader cannot check such an answer. In this tutorial, the assistant cites the passage behind each fact. The tools return elements instead of text, the model cites them by number, and you read the citations of an answer. You also test the tools without a model.

Do tutorial 1 first: this tutorial does not explain again what it explained.

Set upCopy Anchor Link

Use the repository and the environment variables of tutorial 1. Make a folder for this tutorial, and copy the document store of tutorial 1 into it:

mkdir -p my_tutorials/02-citations
cp my_tutorials/01-first-agent/library.py my_tutorials/02-citations/

This tutorial does not change library.py.

Step 1: define the elementsCopy Anchor Link

An element is a thing that a tool shows to the model, and that the model can refer to again. Why the framework needs elements: it can act only on ids that it finds as data, not on ids inside a text. With elements, the framework gives you five things:

1. Short numbers. Each element gets one number for the whole thread, and a new event never changes an old number. A model copies 5 correctly. It does not always copy waste-collection#2 correctly.
2. Safe tool arguments. A tool can take an element number as an argument. The framework changes the number back to the real id, and it refuses a number that names no element of the right type, before the tool runs (step 4).
3. Citations as data. When the model writes [2], the framework records which passage [2] names. A client can show that passage. A number that names nothing is recorded too, so a false citation is visible.
4. Read or only named. A citation resolves only to content that the model read. A citation of a thing that the model only saw by name resolves to nothing.
5. Selection by type. A tool can ask for the elements of one type that the turn showed (step 3).

There are two kinds of elements:

• A Source is content that the model read, such as a passage. The model can cite it.
• A Reference is a thing that the model can name, such as a document, but not cite. The model did not read its content.

You make your own element types: subclass Source or Reference. A subclass can add fields, and the model sees each field. Make my_tutorials/02-citations/tools.py:

from typing import Annotated

from library import Library, Section
from pydantic import Field

from soev_agents.domain import Reference, Source, ToolOutput
from soev_agents.runtime import ElementId, Recorded, at_most, tool


class Document(Reference):
    title: str


class Passage(Source):
    pass

• Document adds the field title, so the model sees the title of each document.
• Passage adds nothing. A Source already has a text, and a ref: the id of the thing that it is part of.

The class name is the type that the model sees: <document> and <passage>. The store keeps the class name too, in kebab case. So if you rename a class, the stored threads do not load. To rename a class safely, give the new class the old name as its type, with a type annotation: type: str = "document".

Step 2: change sections into elementsCopy Anchor Link

The library has its own type, Section. What the model sees are elements. Two small functions change one into the other:

def as_document(section: Section) -> Document:
    return Document(id=section.document, title=section.title)


def as_passage(section: Section) -> Passage:
    return Passage(id=section.id, ref=section.document, text=section.text)

The id of an element is the real id from the library. The model never sees it: the model sees the number of the element.

Step 3: write the search toolCopy Anchor Link

@tool(
    name="search",
    description="Search the town documents for passages. Returns the passages and the documents that they are part "
    "of. Use it to find the facts for each answer.",
    limit=at_most(2, calls="search", error="You searched twice. Answer with what you found."),
)
async def search(
    query: Annotated[str, Field(min_length=1, description="Words to look for in the town documents.")],
    *,
    library: Library,
    seen: Recorded[Passage],
) -> ToolOutput:
    shown = {passage.id for passage in seen}
    found = [section for section in library.search(query) if section.id not in shown]
    if not found:
        return ToolOutput(error="No new passages for these words. Try other words, or answer with what you found.")
    documents = {section.document: as_document(section) for section in found}
    return ToolOutput(elements=(*documents.values(), *map(as_passage, found)))

The names query, library and seen are your choice. The framework reads only where a parameter is, and what type it has:

Field adds a rule (at least one character) and a description for the model.

Why Recorded exists: one tool object serves all threads, and a thread can continue in another process after a crash. So "which passages did the model already see" cannot be a variable in the tool. It must come from the trace. The type Recorded[Passage] tells the framework three things: fill this parameter, with the passages that this turn recorded, and hide the parameter from the model. Recorded takes an element type, not an event type. Here, the tool uses seen to leave out passages that this turn already showed.

A value that you give with bind must not change. bind refuses a list, a dict, a set and other mutable built-in values, because such a value can keep state between calls. bind does not check other objects, such as a dataclass that is not frozen. Make sure that a bound value does not change: here, Library is a frozen dataclass with tuples.

The tool has two limits:

• limit=at_most(2, ...) is a limit for this tool in each turn: the model can search two times. A third call gets the error text, and the tool does not run. calls names what to count: the calls of a tool. at_most can also count the calls of another tool, or the elements of one type that the turn recorded.
• Each agent also has a limit for all tools together: ten model calls with tool calls in each turn. After that, the agent tells the model to answer with what it has. Give passes in ReActConfig to change it.

The tool returns the passages and their documents as elements. It has no text: the framework shows the elements to the model.

Step 4: write a tool that takes an element numberCopy Anchor Link

The second tool reads a whole document:

@tool(
    name="read_document",
    description="Read all passages of one document. Use it when the passages that search found are not enough.",
)
async def read_document(
    document: Annotated[str, ElementId(Document), Field(description="The number of a document that search found.")],
    *,
    library: Library,
) -> ToolOutput:
    sections = library.document(document)
    return ToolOutput(elements=(as_document(sections[0]), *map(as_passage, sections)))

The model sees each element under a number, never under its real id such as dog-tax. ElementId(Document) tells the framework that document holds the number of a Document. The framework changes the number to the real id before the tool runs. A number that names no document never reaches the tool: the model gets an error.

Step 5: make the agentCopy Anchor Link

Make my_tutorials/02-citations/agent.py. These are all its imports, and the prompt:

import asyncio
import os

from library import LIBRARY
from openai import AsyncOpenAI
from tools import Document, Passage, read_document, search

from soev_agents.adapters import InMemoryTraceStore, OpenAIChatModel
from soev_agents.domain import Input, ModelOutput, Trace, inserted
from soev_agents.runtime import AgentCatalog
from soev_agents.toolkit import BRACKETS, ReActAgent, ReActConfig

PROMPT = """You are the assistant of the town of Springfield.
Answer questions from the town documents. If the documents do not answer the question, say so. Never guess.
After each fact, cite the passage that gives it with its number, like [1]."""

The prompt tells the model how to cite: with the number in brackets. Next, add a function that prints an answer with its citations:

def show(trace: Trace) -> None:
    answer = trace.by_type(ModelOutput)[-1]
    print("Answer:", inserted(answer.content, [(marker.at, marker.text) for marker in answer.markers]))
    cited = {number.written: number.source for marker in answer.markers for number in marker.cited}
    for written, source in cited.items():
        print(f"  [{written}]", "names no passage" if source is None else LIBRARY.section(source).text)

When the model answers, the framework takes the markers, such as [2], out of the text. A client gets clean text, and the position of each citation as data. In the ModelOutput:

• content is the answer without markers.
• markers holds each marker: its position at, the text that was cut out, and the numbers that it cited.
• Each number.source is the real id of the passage, or None if the number named no passage.

inserted puts the markers back, to print the answer as the model wrote it.

Now add main:

async def main() -> None:
    model = OpenAIChatModel(AsyncOpenAI(), model=os.environ["OPENAI_MODEL"])
    config = ReActConfig(
        model=model,
        prompt=PROMPT,
        tools=[search.bind(library=LIBRARY), read_document.bind(library=LIBRARY)],
        citations=BRACKETS,
    )
    assistant = ReActAgent(config, element_types=(Document, Passage))
    agents = AgentCatalog(InMemoryTraceStore(), {"assistant": assistant}, version="tutorial-2")

What is new since tutorial 1:

• search.bind(library=LIBRARY) gives the tool its library. bind, a method of FunctionTool, returns a new tool, with the same name, description and limit. The tool takes the library as a parameter, and does not read LIBRARY itself, for two reasons. A test can give the tool its own library (step 7). In tutorial 3, each user gets a different library. A tool with a parameter that nothing binds is refused when the agent starts a turn, before the model runs.
• citations=BRACKETS is the citation strategy: the model writes [2], and the framework reads the number back.
• element_types lists the element types that the threads of this agent record. The store keeps each element as JSON, with its type name. To read a thread back, the framework needs the class for each name.

Step 6: ask two questionsCopy Anchor Link

Add the questions at the end of main, and the lines that run main:

    question = Input(payload="How often can I have bulky waste picked up, and what does it cost?", model=model.model)
    thread, _ = await agents.open("assistant", question)
    show(await thread.resume())

    follow_up = Input(payload="And how much is a parking permit?", model=model.model)
    thread = await agents.thread(thread.id, input=follow_up)
    show(await thread.send(follow_up))


if __name__ == "__main__":
    asyncio.run(main())

Run the agent:

uv run python my_tutorials/02-citations/agent.py

The output is similar to this:

Answer: You can have bulky waste picked up for free four times a year [2]. An extra pickup costs 25 euros [2].
  [2] Bulky waste, such as furniture, is picked up after an appointment. Each household can have it picked up for free four times a year. Each extra pickup costs 25 euros.
Answer: A resident parking permit costs 120 euros a year [5].
  [5] A resident parking permit costs 120 euros a year.

This is what the model read as the result of its first search:

<document id="1" title="Waste collection">
<passage id="2" cite="[2]">Bulky waste, such as furniture, is picked up after an appointment. Each household can have it picked up for free four times a year. Each extra pickup costs 25 euros.</passage>
<passage id="3" cite="[3]">Residual waste is collected every Tuesday morning. Put the container on the street before 7:30.</passage>
</document>

• id is the number of the element, not its real id. The model uses it in a tool argument, such as the document of read_document.
• cite is the form to use in the answer. Only a passage has it. The document has no cite, because it is a Reference: the model did not read it.
• title is the field that Document added.

The tool did not write this text. It returned elements, and a part of the agent, the context builder, made the text from the trace. The context builder builds the whole request again at each model call: the system prompt, the messages and the tools. Tutorial 4 shows how to see each request, and how to control it.

The numbers go in order of first appearance in the thread. A new event never changes an earlier number, so the parking permit passage of the second turn is [5], after the three elements of the first turn and the parking document.

Step 7: test the toolsCopy Anchor Link

A tool is still a function that you can call, so a test calls it directly, without a model or a framework. Make my_tutorials/02-citations/test_tools.py:

import asyncio

from library import LIBRARY
from tools import Passage, read_document, search


def test_search_returns_passages_and_their_documents():
    output = asyncio.run(search("bulky waste", library=LIBRARY, seen=()))
    assert [element.id for element in output.elements] == [
        "waste-collection",
        "waste-collection#2",
        "waste-collection#0",
    ]


def test_search_leaves_out_passages_that_the_turn_already_showed():
    first = asyncio.run(search("parking permit", library=LIBRARY, seen=()))
    seen = tuple(element for element in first.elements if isinstance(element, Passage))
    again = asyncio.run(search("parking permit", library=LIBRARY, seen=seen))
    assert again.error is not None


def test_read_document_returns_all_passages_of_the_document():
    output = asyncio.run(read_document("dog-tax", library=LIBRARY))
    assert [element.id for element in output.elements] == ["dog-tax", "dog-tax#0", "dog-tax#1"]

The test gives library and seen itself, because no agent binds or fills them here. In a test, a Recorded parameter is a plain tuple. Run the tests:

uv run pytest my_tutorials/02-citations/test_tools.py

What you builtCopy Anchor Link

• Two element types: Document, which the model can name, and Passage, which it can cite.
• A search tool with a library that the agent binds, a Recorded parameter and a limit for each turn.
• A tool that takes an element number, which the framework changes to the real id.
• An agent that cites, and a function that reads the citations of an answer.
• Tests for the tools, without a model.

The complete code is in examples/tutorials/02-citations.

All users of this agent search the same library. In tutorial 3, each user gets the documents that the user can read.