soev_indexing.parsing
Parse a file into markdown, with page ranges and positioned layout blocks.
Call a processor with the bytes of a file and a dict of job_parameters, the configuration of one job. It
returns a MarkdownParseResult. Its supported_types names the formats that it accepts, and its output_type is
"markdown_document". Warren, the message-driven processing runtime, calls processors in this shape, with the
job_parameters of each message. The docstring of each processor lists the keys of job_parameters that it reads.
PdfProcessor builds the markdown from the pages that a backend extracts. soev_indexing.backends holds the
stock backends.
PdfProcessor
classPdfProcessor(*, backend)A processor that parses PDF bytes into markdown with a PdfBackend.
parambackendPdfBackendThe backend that extracts the text and the geometry of each page.
attributesupported_typesfrozenset[str]The input types that the processor accepts: {"pdf"}.
attributeoutput_typestrThe type of the output: "markdown_document".
method__call__(input_data, job_parameters) -> MarkdownParseResultParse PDF bytes into markdown with page tracking.
The processor finds the geometry blocks of a page in its text, in reading order. When the found blocks cover at least 60% of the characters of the page, each found block becomes a layout block with its rectangle. Otherwise the page becomes one layout block without a rectangle.
paraminput_databytesRaw PDF bytes.
paramjob_parametersdict[str, Any]Per-job configuration. Recognized keys:
quality_threshold(int): Minimum chars per page. Pages below this are flagged for OCR. Default: 20.
Returns
MarkdownParseResultParse result with markdown, page ranges, and OCR flags.
Raises:
Exception: The error of the backend when it cannot open or parse the PDF.
PdfBackend
protocolPdfBackendProtocol for PDF text extraction backends.
A backend takes the bytes of a PDF and returns a PdfExtractionResult. It reads no job_parameters.
PyMuPDFBackend and LiteParseBackend implement it. Give a backend to PdfProcessor.
methodextract(raw_bytes) -> PdfExtractionResultExtract text from PDF bytes.
paramraw_bytesbytesRaw PDF file bytes.
Returns
PdfExtractionResultExtraction result with per-page text.
Raises:
Exception: The error of the PDF library when it cannot open or parse the PDF.
PdfExtractionResult
dataclassPdfExtractionResult(pages, total_pages, page_dimensions=dict(), geometry=dict())Raw extraction output from a PDF backend.
A PdfBackend returns it, and PdfProcessor builds the markdown and the blocks from it.
parampageslist[tuple[str, int]]Per-page text with 1-based page numbers
(page_text, page_number).
PdfProcessor joins the page texts with nothing between them, so a page text ends with its own
separator.
paramtotal_pagesintTotal number of pages in the source document.
parampage_dimensionsdict[int, tuple[float, float]]= dict()Map of page -> (width_pt, height_pt).
paramgeometrydict[int, list[tuple[str, BBox, str]]]= dict()Map of page -> ordered (text, bbox, kind) blocks from the backend's positional text extraction, used to attach bboxes.
Text, tables and office files
These processors parse plain text and markdown, delimited tables, and office files.
TextProcessor
classTextProcessor()A processor that parses plain-text and markdown bytes into markdown.
It decodes the bytes as UTF-8, with a replacement character for each byte that does not decode. The text becomes the markdown as it is. The result has one page and one layout block without a rectangle.
attributesupported_typesfrozenset[str]The input types that the processor accepts: txt, text, md, markdown, rst and json.
attributeoutput_typestrThe type of the output: "markdown_document".
method__call__(input_data, job_parameters) -> MarkdownParseResultParse plain-text/markdown bytes into markdown.
paraminput_databytesRaw text bytes (UTF-8 with replacement on errors).
paramjob_parametersdict[str, Any]Per-job configuration. Recognized keys:
format(str):"json"puts the text in a fencedjsoncode block. Other values have no effect.
Returns
MarkdownParseResultParse result containing the text as markdown.
TsvProcessor
classTsvProcessor()A processor that parses tab-separated bytes into one markdown table.
The first row is the header of the table, and the processor drops rows without text. A line break in a cell becomes
<br>. The result has one page and one layout block of kind table.
attributesupported_typesfrozenset[str]The input types that the processor accepts: {"tsv"}.
attributeoutput_typestrThe type of the output: "markdown_document".
method__call__(input_data, job_parameters) -> MarkdownParseResultParse tab-separated bytes into a markdown table.
paraminput_databytesThe bytes, decoded as UTF-8 with a replacement character for each byte that does not decode.
paramjob_parametersdict[str, Any]The per-job configuration. The processor reads no keys.
Returns
MarkdownParseResultThe parse result with the table as markdown. A file without rows gives its text as the markdown.
OfficeProcessor
classOfficeProcessor(*, exclude=frozenset())A processor that parses DOCX, XLSX, PPTX and CSV bytes into markdown.
Each format gives its own markdown:
- DOCX and DOTX: one page. Headings become
#headings, tables stay HTML, and the markdown has no images. - XLSX and XLTX: one page for each sheet, under the heading
# Sheet: <name>. Rows before the first row with two values become lines of text, and the other rows become a markdown table. - PPTX: one page for each slide that has content, under the heading
# Slide <n>. Each text, table and speaker note becomes a layout block. A text or a table has the rectangle of its shape. - CSV: one page with one markdown table.
The parsers need packages that no extra of soev-indexing declares: mammoth and markdownify for DOCX,
openpyxl for XLSX and python-pptx for PPTX. Without them, a call raises ImportError.
paramexcludefrozenset[str]= frozenset()The formats that the processor does not claim, so that another processor can take them. A runtime that refuses two processors for one format then sees separate formats.
attributesupported_typesfrozenset[str]The accepted formats: docx, dotx, xlsx, xltx, pptx and csv, less those in exclude.
attributeoutput_typestrThe type of the output: "markdown_document".
method__call__(input_data, job_parameters) -> MarkdownParseResultParse office bytes into markdown.
A ZIP package must declare at most 10,000 members and at most four times PARSER_MAX_DOCUMENT_BYTES (an
environment variable, default 200 MiB) of content. The check reads the declared sizes before any member.
paraminput_databytesThe bytes of the file.
paramjob_parametersdict[str, Any]The per-job configuration. Recognized keys:
format(str): The format of the bytes, one ofsupported_types. Without it, the processor detects the format: a ZIP package with aword/,xl/orppt/folder is DOCX, XLSX or PPTX, and other bytes are CSV.
Returns
MarkdownParseResultThe parse result with the markdown, a page range for each page and the layout blocks.
Raises:
ValueError: The ZIP package is over the limits, or it is not a DOCX, XLSX or PPTX package.KeyError: The format is not one ofsupported_types.ImportError: The package that the format needs is not installed.
Layout
A layout block is one span of the markdown, with its page and, when the format has geometry, its rectangle.
LayoutBlock
dataclassLayoutBlock(text, start_char, end_char, page, kind=None, bbox=None)One positioned span of parsed markdown.
MarkdownParseResult.blocks holds the blocks of a document in order, and they tile its markdown. A
chunker gives a chunk the rectangle of each block that overlaps it.
paramtextstrThe block's text as it appears in the markdown.
paramstart_charintStart offset into the concatenated markdown.
paramend_charintExclusive end offset into the concatenated markdown.
parampageint1-based page/sheet/slide number.
paramkindstr | None= NoneOptional label (heading/text/table/sheet/...).
parambboxBBox | None= NoneOptional (x0, y0, x1, y1) in points, top-left origin, in the
page's coordinate space. None when the format has no geometry.
BBox
type aliasBBox = tuple[float, float, float, float]A rectangle (x0, y0, x1, y1) in points, with the origin at the top left of the page.