soev-indexing
Overview
Parse and chunk PDF, text and delimited documents, and index them with a small pgvector pipeline.
soev-indexing parses a document into markdown, splits it into chunks, embeds each chunk, and writes the chunks
into a pgvector collection. You can read the collection back as a soev-retrieval store. Indexer does each step
in one call, and the command soev-index index indexes a file from a terminal.
The command embeds through soev-inference, so it needs an inference config root with an embedding endpoint. The soev-inference README describes a config root.
Install
soev-indexing needs Python 3.12 or later.
pip install "soev-indexing[pdf]"| Extra | Adds |
|---|---|
pdf | PyMuPDF and pymupdf4llm, for the PDF backend pymupdf, the default of --pdf-backend |
liteparse | liteparse, for the PDF backend liteparse |
Where to start
- Commands: the arguments of
soev-index index. - Reference of
soev_indexing:Indexer, and the subpackages for each step:soev_indexing.parsing,soev_indexing.backendsandsoev_indexing.chunking.