soev_indexing.chunking
Split parsed markdown into chunks, by structure or at a fixed size.
Call a chunker with a MarkdownParseResult and a dict of job_parameters, the configuration of one job.
It returns the chunks as a list of ChunkResult, in document order. Its supported_types is
{"markdown_document"}, and its output_type is "chunked_document". The docstring of each chunker lists the
keys of job_parameters that it reads.
Chunkers
AdaptiveChunker keeps sections and sentences whole. MarkdownChunker cuts the text at a fixed size.
AdaptiveChunker
classAdaptiveChunker(*, target_tokens=…, max_tokens=None, min_tokens=None)A chunker that normalises markdown, finds its structure and packs it into chunks.
It is the structure-aware chunker for Dutch and English documents:
- It reflows hard-wrapped lines into paragraphs. It joins a word that a line break split when the joined word
is a known word of the language. In a document of three pages or more, it drops running headers, footers
and page numbers at the edges of the pages. It recognises unmarked headings, such as
4.3 TitelorHOOFDSTUK 2. - It packs the blocks of a section up to the target size. A block over the target splits at sentence ends, with Dutch and English abbreviations known. A table splits at rows, and each piece repeats the header row.
- A heading starts a new chunk, unless the chunk so far is a whole section below the minimum. A chunk below the minimum joins the chunk before it in the same top section, when the two fit under the maximum.
ChunkResult.context_headerholds the last three headings of the path of the chunk,A > B > C, at most 300 characters. The content of the chunk holds only the headings inside the chunk.
Sizes are estimated tokens: the number of characters divided by 3.
Example
from soev_indexing.chunking import AdaptiveChunker
from soev_indexing.parsing import TextProcessor
parsed = TextProcessor()(data, {})
chunks = AdaptiveChunker()(parsed, {"chunker": {"target_tokens": 800}})paramtarget_tokensint= …Size to pack towards, in estimated tokens. Per
job: job_parameters.chunker.target_tokens, or the legacy
top-level chunk_size in characters. The default is 1200,
about 1024 real tokens of Dutch prose.
parammax_tokensint | None= NoneHard ceiling; defaults to 1.5 x target.
parammin_tokensint | None= NoneA section boundary does not close a chunk below this; defaults to target / 4.
attributesupported_typesfrozenset[str]The input types that the chunker accepts: {"markdown_document"}.
attributeoutput_typestrThe type of the output: "chunked_document".
method__call__(input_data, job_parameters) -> list[ChunkResult]Chunk one parsed document.
Each chunk gets the pages whose range overlaps it, and the rectangles of at most 40 layout blocks that overlap it. A chunk that overlaps no page range gets the last page of the chunk before it, or page 1.
paraminput_dataMarkdownParseResultThe parse result to chunk.
paramjob_parametersdict[str, Any]Per-job configuration. A value here replaces the value of the constructor. Recognized keys:
chunker(dict): Holdstarget_tokens,max_tokensandmin_tokens(int), in estimated tokens.chunk_size(int): The legacy target size in characters. It counts only whenchunker.target_tokensis not given.
Returns
list[ChunkResult]Chunks in document order; empty when the markdown is blank.
MarkdownChunker
classMarkdownChunker(*, chunk_size=…, chunk_overlap=None)A chunker that splits markdown into chunks of fixed size, with recursive character splitting.
It first removes null bytes and control characters, but keeps line breaks and tabs. It then splits at
blank lines, then at line breaks, then after . , then at spaces, and as the last resort between any two
characters. parse_and_chunk uses it with its defaults. It ignores headings: use AdaptiveChunker to keep
sections and sentences whole.
The start_char and end_char of a chunk can be wrong when the markdown holds control characters,
because they count the text after their removal. They can also be wrong when the markdown repeats a text,
because the chunker looks for each chunk from the start of the previous chunk.
paramchunk_sizeint= …The maximum length of a chunk in characters. The default, 2816, is about 940 estimated tokens.
paramchunk_overlapint | None= NoneThe characters that two neighbouring chunks share, or None for a tenth of
chunk_size.
attributesupported_typesfrozenset[str]The input types that the chunker accepts: {"markdown_document"}.
attributeoutput_typestrThe type of the output: "chunked_document".
method__call__(input_data, job_parameters) -> list[ChunkResult]Split markdown into chunks with page tracking.
Each chunk gets the pages whose range overlaps it, or page 1 when none does, and the rectangles of at most 40 layout blocks that overlap it.
paraminput_dataMarkdownParseResultParse result containing markdown and page ranges.
paramjob_parametersdict[str, Any]Per-job configuration. Recognized keys:
chunk_size(int): Target chunk size in characters.chunk_overlap(int): Overlap between chunks in characters.
Returns
list[ChunkResult]List of chunk results. Empty if input is blank.