soev for developers
soev-indexing

soev_indexing

Parse, chunk, embed and index documents, and read what you indexed back as a soev-retrieval store.

Indexer does each step in one call. The subpackages hold the steps: soev_indexing.parsing parses a file into markdown, soev_indexing.backends extracts the text of a PDF, and soev_indexing.chunking splits the markdown into chunks. soev_indexing.contracts holds the records inside a chunk: its rectangles and its table positions. The command soev-index index indexes a file from a terminal.

Modules

Indexing

An indexer parses, chunks and embeds a document, and writes its chunks to a sink.

Indexer

dataclassIndexer(embedder, sink, backend='pymupdf')

A pipeline that parses, chunks and embeds a document, and writes its chunks to a sink.

The command soev-index index runs the same pipeline with a PgSink.

Example

from psycopg_pool import AsyncConnectionPool
from soev_inference import InferenceClient

from soev_indexing import Embedder, Indexer, PgSink

client = InferenceClient(config_roots=[config_root])
async with AsyncConnectionPool(postgres_uri, open=False) as pool:
    sink = PgSink(pool)
    await sink.setup()
    indexer = Indexer(Embedder(client, "local--embedding"), sink)
    document, chunks = await indexer.index(
        data, filename="report.pdf", collection="reports", subject="key-holder"
    )
await client.aclose()
paramembedderEmbedder

What embeds the chunks.

paramsinkSink

Where the embedded chunks go.

parambackendstr
= 'pymupdf'

The PDF backend that parse_and_chunk uses: "pymupdf" or "liteparse".

methodindexasync (data, *, filename, collection, subject, document=None) -> tuple[str, int]

Index one document into a collection of a subject.

The parse and the chunking run in a thread (see parse_and_chunk). The sink then replaces the chunks that the document had in the collection. The title of the document is the name part of filename. An error of the embedding client or of the sink passes through unchanged.

paramdatabytes

The bytes of the file.

paramfilenamestr

The name of the file. Its extension selects the parser.

paramcollectionstr

The name of the collection. It must not be blank.

paramsubjectstr

The owner of the collection. It must not be blank.

paramdocumentstr | None
= None

The id of the document, or None (or an empty string) for the SHA-256 hex digest of data. Index a file again with the same id to replace its chunks.

Returns

tuple[str, int]

A tuple (document, chunks): the id of the document and the number of chunks that the sink received.

Raises:

  • UnsupportedFormat: The extension of filename is not supported.
  • ValueError: collection or subject is blank, the document has no text, the PDF backend cannot open the PDF, the embeddings are not valid, or the sink refuses the chunks (see PgSink.upsert).
  • ImportError: The extra of the PDF backend is not installed.

parse_and_chunk

functionparse_and_chunk(data, filename, *, backend='pymupdf') -> list[ChunkResult]

Parse a file into markdown and split it into chunks of fixed size.

The extension of filename, in any case, selects the parser:

  • pdf: PdfProcessor with the backend that backend names.
  • txt, text, md, markdown, rst and json: TextProcessor. A JSON file becomes a fenced json code block.
  • tsv: TsvProcessor.
  • csv: OfficeProcessor.

MarkdownChunker with its defaults then splits the markdown. The function is synchronous and uses the CPU, so an async caller runs it in a thread, as Indexer.index does.

paramdatabytes

The bytes of the file.

paramfilenamestr

The name of the file. The function reads only its extension.

parambackendstr
= 'pymupdf'

The PDF backend: "pymupdf" (PyMuPDFBackend, extra [pdf]) or "liteparse" (LiteParseBackend, extra [liteparse]). Only a PDF file uses it.

Returns

list[ChunkResult]

The chunks in document order. The list is empty when the file has no text.

Raises:

  • UnsupportedFormat: The extension is not one of the list above.
  • ValueError: backend names no backend, or the PDF backend cannot open the PDF.
  • ImportError: The extra of the PDF backend is not installed.

UnsupportedFormat

errorUnsupportedFormat

A file has an extension that parse_and_chunk cannot parse.

parse_and_chunk and Indexer.index raise it. The message names the supported formats.

Embedding

An embedder turns the text of each chunk into a vector, with an embedding endpoint of soev-inference.

Embedder

dataclassEmbedder(client, endpoint)

An embedder that sends chunks to one embedding endpoint of a soev-inference client.

paramclientEmbedClient

The client that embeds the texts.

paramendpointstr

The name of the embedding endpoint, for example "local--embedding". Indexer gives it to the sink, which stores it with the collection.

methodembedasync (chunks) -> list[EmbeddedChunk]

Embed the content of each chunk, in one request to the endpoint.

paramchunkslist[ChunkResult]

The chunks to embed.

Returns

list[EmbeddedChunk]

One embedded chunk for each chunk, in the order of chunks. An empty list makes no request and gives an empty list.

Raises:

  • ValueError: The response does not hold one vector for each chunk, or the vectors are empty, differ in length or hold a value that is not finite.

EmbeddedChunk

dataclassEmbeddedChunk(chunk, vector)

A chunk with its embedding vector, ready for a sink.

paramchunkChunkResult

The chunk. Its content is the text that the vector embeds.

paramvectortuple[float, ...]

The embedding vector of the chunk. Its values are finite.

Sinks

A sink keeps the embedded chunks of each collection. PgSink also reads them back for a search.

Sink

protocolSink

A store that keeps the embedded chunks of each document, by subject and collection.

Indexer writes to it. PgSink is the stock sink. Write your own sink for another store.

methodupsertasync (*, subject, collection, document, title, chunks, endpoint) -> None

Write the chunks of one document to a collection, in place of the chunks that it had.

paramsubjectstr

The owner of the collection, for example the subject of a request's identity.

paramcollectionstr

The name of the collection. Each subject has its own collections.

paramdocumentstr

The id of the document in the collection.

paramtitlestr

The title of the document, for example its file name.

paramchunkslist[EmbeddedChunk]

The embedded chunks of the document, in document order. Indexer never passes an empty list.

paramendpointstr

The name of the embedding endpoint that made the vectors. A search of the collection must embed its query with the same endpoint.

PgSink

dataclassPgSink(pool, schema='soev_indexing')

A sink that keeps chunks and their vectors in two tables of a PostgreSQL schema.

The table collections holds one row for each collection of a subject, with the vector length and the embedding endpoint of the collection. The table chunks holds one row for each chunk, with its text, a JSON object of metadata and its vector. Call setup once before the first upsert.

setup creates no index on the vectors, so a vector search compares the query with each row that the filter keeps. The table has no tsvector column, so the keyword side of a search parses the text of each row for each query. It uses the English text search configuration, also for Dutch text.

parampoolAsyncConnectionPool[Any]

The connection pool. The caller opens and closes it.

paramschemastr
= 'soev_indexing'

The PostgreSQL schema of the two tables.

attributetablePgTable

The table of the chunks, as soev-retrieval reads it.

attributecollections_tablesql.Identifier

The qualified name of the table of the collections.

methodsetupasync () -> None

Create the pgvector extension, the schema and the two tables, when they do not exist.

The database user needs the right to create the extension, or the extension must exist already.

methodupsertasync (*, subject, collection, document, title, chunks, endpoint) -> None

Write the chunks of one document to a collection, in place of the chunks that it had.

All writes run in one transaction. The first write of a collection records the vector length and endpoint. Each chunk gets the id <document>-<chunk_index>, and its metadata holds the text, the title, the pages, the character offsets and the rectangles of the chunk.

paramsubjectstr

The owner of the collection.

paramcollectionstr

The name of the collection.

paramdocumentstr

The id of the document in the collection.

paramtitlestr

The title of the document.

paramchunkslist[EmbeddedChunk]

The embedded chunks of the document. Their vectors have the same length.

paramendpointstr

The name of the embedding endpoint that made the vectors.

Raises:

  • ValueError: chunks is empty, or the collection has a different embedding endpoint or vector length.
methodlistasync (subject) -> list[Collection]

Return the collections of a subject that hold chunks.

paramsubjectstr

The owner of the collections.

Returns

list[Collection]

One collection for each collection name, in name order.

methodreader(subject, collections) -> PgVectorStore

Return a soev-retrieval store that searches collections of a subject.

Each collection is a target of the store, with the fields text and title. A search reads only the rows of subject and of its target. The store does not embed the query: embed it with the endpoint that the collection uses.

paramsubjectstr

The owner of the collections.

paramcollectionsSequence[str]

The names of the collections that the store can search.

Returns

PgVectorStore

A store over the same pool.

Collection

dataclassCollection(name, documents, chunks)

One collection of a subject, with its counts.

paramnamestr

The name of the collection.

paramdocumentsint

The number of documents in the collection.

paramchunksint

The number of chunks in the collection.

Results

A parser returns a parse result, and a chunker returns a list of chunk results.

MarkdownParseResult

modelMarkdownParseResult

Result of parsing a document into markdown.

The processors of soev_indexing.parsing return it, and the chunkers of soev_indexing.chunking read it. It is a frozen pydantic model.

attributemarkdownstr

The extracted markdown text. It can hold table placeholders (for example ⟦tbl-3.html⟧) when a parser extracts tables separately. The parsers of this package leave tables inline as markdown and give an empty tables map.

attributepage_rangeslist[tuple[int, int, int]]

List of (start_char, end_char, page_number).

attributetotal_pagesint

Total number of pages in the source document.

attributepages_needing_ocrlist[int]
= []

The pages with fewer characters than the quality_threshold of PdfProcessor. The other parsers of this package leave it empty.

attributetablesdict[str, str]
= {}

Map of placeholder id to table HTML (or markdown), for tables that a parser extracted separately. The chunkers of this package do not put them in the chunks. Empty when the parser leaves tables inline.

attributeblockslist[LayoutBlock]
= []

The layout blocks of markdown, in order and without gaps between them. A block has a rectangle when the format has geometry. Empty when the parser gives no blocks.

attributepage_dimensionsdict[str, tuple[float, float]]
= {}

Map of page number to (width, height) in points, for consumers of the rectangles. Empty when the format has no geometry. The keys are page numbers as strings, so that a document store that refuses integer keys can store the output of model_dump().

ChunkResult

modelChunkResult

Result of chunking a document into a text segment.

The chunkers of soev_indexing.chunking return a list of it, in document order. It is a frozen pydantic model.

attributecontentstr

The chunk text that a user sees, and that Embedder embeds.

attributechunk_indexint

Position of this chunk in the document (0-based).

attributestart_charint
= 0

Start character position of the chunk in the markdown that the chunker read.

attributeend_charint
= 0

Exclusive end character position of the chunk in the markdown that the chunker read.

attributepageslist[int]
= []

Source page numbers this chunk spans.

attributebboxeslist[BboxRect]
= []

Rectangles (page + bbox in points, top-left origin, 1-based pages) of the layout blocks overlapping this chunk's span. Empty when the parse carried no geometry — consumers fall back to page-level attribution.

attributetableslist[TableSpan]
= []

Character ranges in content where a table was put in place of a placeholder. The chunkers of this package leave it empty.

attributecontext_headerstr
= ''

The heading path of the chunk, A > B > C, which AdaptiveChunker fills. Show it with the chunk. Empty from MarkdownChunker.

attributeunit_idstr
= ''

The stable id of the unit of a structured document. The chunkers of this package leave it empty.

attributeunit_chunk_indexint
= 0

The position of the chunk in its unit of a structured document. The chunkers of this package leave it 0.

attributeunit_refslist[str]
= []

The references to the sub-units in the chunk, for a structured document. The chunkers of this package leave it empty.

Overview

Parse and chunk PDF, text and delimited documents, and index them with a small pgvector pipeline.

Commands

Index a file into a pgvector collection.

On this page

ModulesIndexingIndexerparse_and_chunkUnsupportedFormatEmbeddingEmbedderEmbeddedChunkSinksSinkPgSinkCollectionResultsMarkdownParseResultChunkResult