soev for developers
soev-retrievalReference

soev_retrieval.testing

The contract every VectorStore implementation must pass, and a scripted double for what is built on it.

Subclass VectorStoreContract in your own test tree and implement make_store (or override the store fixture) so that the target TARGET holds exactly the chunks in CORPUS, each stored with its vector and properties, under a schema matching SPEC. Every test then runs against your store. ScriptedVectorStore is for tests of code that calls a store: it answers from a script and records what it was asked.

This package imports pytest. Install the testing extra of soev-retrieval to use it.

Contract suite

The suite tests a store against the contract of VectorStore, with a fixed corpus.

VectorStoreContract

classVectorStoreContract

The test suite that every VectorStore implementation must pass.

Subclass it in your own test tree, with a name that pytest collects. Implement make_store, or override the store fixture, so that TARGET holds exactly the chunks of CORPUS under a schema that matches SPEC. Each test then runs against your store. The tests are async and carry no asyncio marker, so run them with pytest-asyncio in auto mode.

Example

from soev_retrieval.testing import CORPUS, TARGET, VectorStoreContract


class TestMyStore(VectorStoreContract):
    async def make_store(self):
        store = MyStore()  # your own VectorStore implementation
        for record in CORPUS:
            # `add` is the write API of your own store: `VectorStore` has no write method.
            await store.add(TARGET, record.id, record.document_id, record.vector, record.properties)
        return store
methodmake_storeasync () -> VectorStore

Return the store under test, with the chunks of CORPUS stored in TARGET.

Returns

VectorStore

The store to test.

Raises:

  • NotImplementedError: The subclass implements neither this method nor the store fixture.
methodstoreasync () -> VectorStore

Give each test the store that make_store returns.

Override it with a pytest.fixture named store to build the store from other fixtures.

Returns

VectorStore

The store under test.

methodtest_near_vector_orders_by_similarityasync (store)

Check that near_vector returns the most similar chunks first.

paramstore
methodtest_near_vector_respects_limitasync (store)

Check that near_vector returns at most limit hits.

paramstore
methodtest_hits_carry_id_document_and_propertiesasync (store)

Check that a hit carries the chunk id, the document id and the stored properties.

paramstore
methodtest_hit_properties_are_only_declared_fieldsasync (store)

Check that a hit holds only the properties that fields declares.

paramstore
methodtest_fields_names_the_filterable_fieldsasync (store)

Check that fields marks exactly the filterable fields of SPEC filterable, with their types.

paramstore
methodtest_unknown_target_is_refusedasync (store)

Check that a search of an unknown target raises InvalidQueryError.

paramstore
methodtest_bm25_returns_only_chunks_with_the_termasync (store)

Check that bm25 returns no chunk without a term of the query.

paramstore
methodtest_bm25_finds_every_chunk_with_the_termasync (store)

Check that bm25 returns every chunk with a term of the query.

paramstore
methodtest_hybrid_ranks_the_chunk_matching_text_and_vector_firstasync (store)

Check that hybrid ranks first the chunk that matches both the text and the vector.

paramstore
methodtest_hybrid_respects_limitasync (store)

Check that hybrid returns at most limit hits.

paramstore
methodtest_hybrid_alpha_one_is_pure_vectorasync (store)

Check that hybrid with alpha=1.0 ranks by the vector alone.

paramstore
methodtest_hybrid_alpha_zero_is_pure_keywordasync (store)

Check that hybrid with alpha=0.0 ranks by the keyword match alone.

paramstore
methodtest_filter_eq_restricts_a_searchasync (store)

Check that an eq filter restricts the candidates of a search.

paramstore
methodtest_filter_in_matches_any_listed_valueasync (store)

Check that an in filter matches a chunk with any of the listed values.

paramstore
methodtest_filter_range_operatorsasync (store)

Check that the gt and lte filters compare a number field.

paramstore
methodtest_filter_contains_any_on_a_list_fieldasync (store)

Check that a contains_any filter matches a list field that holds any of the listed values.

paramstore
methodtest_filters_apply_to_hybrid_tooasync (store)

Check that a filter restricts the candidates of hybrid.

paramstore
methodtest_unknown_field_is_refused_and_namedasync (store)

Check that a filter on an unknown field raises InvalidQueryError that names the field.

paramstore
methodtest_unfilterable_field_is_refused_and_namedasync (store)

Check that a filter on an unfilterable field raises InvalidQueryError that names the field.

paramstore
methodtest_unknown_operator_is_refused_and_namedasync (store)

Check that an unknown operator raises InvalidQueryError that names the field of the filter.

paramstore
methodtest_operand_shape_is_refusedasync (store)

Check that an operand of the wrong shape for its operator raises InvalidQueryError.

paramstore
methodtest_a_value_of_the_wrong_type_is_refused_and_namedasync (store)

Check that an operand of the wrong type for its field raises InvalidQueryError that names it.

paramstore
methodtest_fetch_orders_by_a_fieldasync (store)

Check that fetch sorts on order_by, in ascending and in descending order.

paramstore
methodtest_fetch_respects_limitasync (store)

Check that fetch returns limit chunks when more chunks match.

paramstore
methodtest_fetch_returns_chunksasync (store)

Check that fetch returns chunks with the chunk id, the document id and the stored properties.

paramstore
methodtest_filter_gt_on_a_date_fieldasync (store)

Check that a gt filter compares a date field with ISO 8601 text.

paramstore
methodtest_every_filter_must_holdasync (store)

Check that AllOf matches only the chunks for which every term holds.

paramstore
methodtest_any_of_matches_either_branchasync (store)

Check that AnyOf matches the chunks for which any term holds.

paramstore
methodtest_not_excludes_what_its_term_matchesasync (store)

Check that Not excludes the chunks that its term matches.

paramstore
methodtest_not_eq_matches_a_chunk_with_no_value_for_the_fieldasync (store)

Check that Not of an eq filter matches a chunk without a value for the field.

paramstore
methodtest_an_expression_nestsasync (store)

Check that a nested filter expression restricts a search.

paramstore
methodtest_an_or_inside_a_search_filters_the_candidatesasync (store)

Check that an AnyOf filter restricts the candidates of hybrid.

paramstore
methodtest_nothing_matches_nothingasync (store)

Check that NOTHING, alone or inside an AllOf, matches no chunk.

paramstore
methodtest_an_empty_all_of_matches_everythingasync (store)

Check that an AllOf without terms matches every chunk.

paramstore
methodtest_a_bad_predicate_inside_an_expression_is_refused_and_namedasync (store)

Check that a bad predicate inside an AnyOf raises InvalidQueryError that names its field.

paramstore

TARGET

constantTARGET = 'docs'

The target that holds the contract corpus. A store under test must serve CORPUS under this name.

CORPUS

constantCORPUS: tuple[Record, ...]

The four chunks that a store under test must hold in TARGET, each with its vector and its properties.

Chunk c2 has no published value, so the suite checks how a filter treats a missing value.

Record

dataclassRecord(id, document_id, vector, properties)

One chunk of the contract corpus as it must be stored.

paramidstr

The id of the chunk. A hit or a chunk of the store must carry it as id.

paramdocument_idstr

The document that the chunk came from.

paramvectortuple[float, ...]

The embedding to store with the chunk.

parampropertiesMapping[str, object]

The stored fields of the chunk. A chunk can lack a field of SPEC.

SPEC

constantSPEC: FieldSpec

The stored fields of TARGET. A store under test must return a field spec for TARGET that matches it.

text is the only field that a filter cannot name. tags holds a sequence of keywords.

Scripted store

A scripted store tests code that calls a store, without a backend.

ScriptedVectorStore

classScriptedVectorStore(hits=(), *, chunks=(), errors=(), fields=None)

A VectorStore that answers every search with hits and every fetch with chunks.

errors are raised one per call, in order, before any answer is given. fields is what fields() returns for any target. calls records (verb, target, keyword arguments) for every call made.

A call that raises an error of errors is recorded in calls too.

paramhitsSequence[Hit]
= ()

The hits that each search returns. Change it to change the next answers.

paramchunksSequence[Chunk]
= ()

The chunks that each fetch returns. Change it to change the next answers.

paramerrorsSequence[Exception]
= ()

The errors to raise, one for each call, before the store answers again.

paramfieldsFieldSpec | None
= None

The field spec that fields() returns, or None for an empty spec.

attributecallslist[tuple[str, str, dict[str, Any]]]

One (verb, target, keyword arguments) tuple for each call, in call order.

methodhybridasync (target, *, query, vector=None, limit, where=None, alpha=None) -> Sequence[Hit]

Record the call, then raise the next error of errors, or return hits.

paramtargetTarget

The target, recorded in calls.

paramquerystr

Recorded in calls.

paramvectorSequence[float] | None
= None

Recorded in calls.

paramlimitint

Recorded in calls. The store does not apply it.

paramwhereFilter | None
= None

Recorded in calls. The store does not apply it.

paramalphafloat | None
= None

Recorded in calls.

Returns

Sequence[Hit]

A copy of hits.

Raises:

  • Exception: The next error of errors, while any are left.
methodnear_vectorasync (target, *, vector, limit, where=None) -> Sequence[Hit]

Record the call, then raise the next error of errors, or return hits.

paramtargetTarget

The target, recorded in calls.

paramvectorSequence[float]

Recorded in calls.

paramlimitint

Recorded in calls. The store does not apply it.

paramwhereFilter | None
= None

Recorded in calls. The store does not apply it.

Returns

Sequence[Hit]

A copy of hits.

Raises:

  • Exception: The next error of errors, while any are left.
methodbm25async (target, *, query, limit, where=None) -> Sequence[Hit]

Record the call, then raise the next error of errors, or return hits.

paramtargetTarget

The target, recorded in calls.

paramquerystr

Recorded in calls.

paramlimitint

Recorded in calls. The store does not apply it.

paramwhereFilter | None
= None

Recorded in calls. The store does not apply it.

Returns

Sequence[Hit]

A copy of hits.

Raises:

  • Exception: The next error of errors, while any are left.
methodfetchasync (target, *, where=None, limit, order_by=None, descending=False) -> Sequence[Chunk]

Record the call, then raise the next error of errors, or return chunks.

paramtargetTarget

The target, recorded in calls.

paramwhereFilter | None
= None

Recorded in calls. The store does not apply it.

paramlimitint

Recorded in calls. The store does not apply it.

paramorder_bystr | None
= None

Recorded in calls. The store does not sort.

paramdescendingbool
= False

Recorded in calls.

Returns

Sequence[Chunk]

A copy of chunks.

Raises:

  • Exception: The next error of errors, while any are left.
methodfieldsasync (target) -> FieldSpec

Record the call, then raise the next error of errors, or return the scripted field spec.

paramtargetTarget

The target, recorded in calls. Every target gets the same spec.

Returns

FieldSpec

The fields that the store was made with.

Raises:

  • Exception: The next error of errors, while any are left.

soev_retrieval.ports

Implement a vector store: the protocol that every store implements, its errors, and its filter rule.

soev_retrieval.adapters

The stock stores: one `VectorStore` for each vector database.

On this page

Contract suiteVectorStoreContractTARGETCORPUSRecordSPECScripted storeScriptedVectorStore