soev_retrieval.engine

Search every group's scopes for query and return one ranking from combine, best first.

Every search names one target of one store. Scopes in one target are one search, which matches any of their filters, so the store scores their hits against each other. An unfiltered scope makes its whole target unfiltered. Lists from different targets or stores meet only in combine, which receives one list per search, in group and scope order. Each search returns at most limit hits. The query is embedded once per group, with the group's own embedder. A group with no scopes embeds and searches nothing. Errors are the store's, as hybrid_search raises them.

from soev_retrieval.domain import DeclaredFilter, SearchScope
from soev_retrieval.engine import Rrf, SearchGroup, search_scopes

hits = await search_scopes(
    [
        SearchGroup(
            store=policy_store,
            scopes=[SearchScope("policies", where=DeclaredFilter("year", "gte", 2020))],
            embed=embed_policy_query,
        ),
        SearchGroup(store=case_store, scopes=[SearchScope("cases")], embed=embed_case_query),
    ],
    "pension transfer rules",
    limit=10,
    combine=Rrf(),
)

Raises:

• InvalidQueryError: A store rejected a search.
• StoreUnavailable: A store gave no answer.
• Unconfigured: A group has no embedder, and its store cannot embed the query.

The scopes one store serves, with the embedder its index was built with (None leaves it to the store).

Return at most limit chunks of target matching query, best first.

The query vector is vector when given, else embed(query) when an embedder is given, else left to the store, which raises Unconfigured if it cannot embed. alpha and where go to the store unchanged. See VectorStore.hybrid for their meaning and the errors raised.

Raises:

• InvalidQueryError: The store rejected the call.
• StoreUnavailable: The store gave no answer.
• Unconfigured: No vector and no embedder are given, and the store cannot embed the query.

An async function that embeds one query text into one vector.

Give a search the embedder that the index of its store was built with, so that the query vector and the stored vectors compare.

A strategy that turns the query and one ranked list of hits per search into one ranking, best first.

Scores from different searches do not compare (see Hit), so a strategy works from ranks, or rescores every hit on one scale. Hits with the same id are the same chunk and appear once.

search_scopes calls it. Rrf is the stock strategy.

Reciprocal rank fusion: a hit scores the sum of 1 / (k + rank) over the lists it appears in, rank from 1.

Uses ranks only, so lists whose scores do not compare can be fused. Each fused hit carries its fused score and the properties of its first appearance.

VectorStore over another, calling it up to attempts times while it raises StoreUnavailable.

Waits backoff seconds before the second attempt, doubling before each further one, and raises the last StoreUnavailable when every attempt failed. A rejected call (InvalidQueryError) or an Unconfigured store is raised at once: calling again would get the same answer.

Raises:

• ValueError: When the instance is made: attempts is less than 1.

VectorStore over another, calling it with every predicate of the filter replaced by what rewrite returns.

rewrite sees the target and each predicate of the filter of each method (the three searches and fetch), wherever it sits in the expression, and the expression around it is kept. It is for an operator the store cannot serve itself: a folder scope (under) resolved through a catalog into the document ids it covers, a name looked up into a key.

A predicate rewritten to NOTHING matches nothing, so its branch of an AnyOf drops out, its AllOf matches nothing, and its Not matches everything. When the whole filter comes to match nothing, the call answers an empty sequence without the inner store being called. Every other argument reaches the inner store unchanged, results come back untouched, and fields passes through. PostFiltered acts on the results instead. The two compose in either order.

The rule that gives the filter to put in the place of one predicate of the caller's filter.

It gets the target and one predicate, and returns the predicate itself, a replacement the inner store can serve, or NOTHING when no chunk could match it.

PreFiltered calls it once for each predicate of each filter.

VectorStore over another, returning only what keep returns.

keep sees every result the inner store answered with (Hits from a search, Chunks from a fetch) and returns those to pass on. It runs after the inner store applied limit, so a call can return fewer results than asked for. The store does not fetch more results to replace them. It is for a rule the index cannot apply itself: a revocation the index does not know about yet, a blocklist, a score floor, deduplication. Every argument reaches the inner store unchanged, and fields passes through untouched.

The rule that picks, from the results a store answered with, the ones to return.

It gets the target and the results, in order, and returns the ones to pass on.

PostFiltered calls it with the hits of each search and with the chunks of each fetch.