soev for developers
WarrenReference

warren.storage

Storage: the store errors, request scoping and the helpers that the stores share.

Each store type is in its own subpackage, such as warren.storage.jobs and warren.storage.results.

Modules

Errors

ResourceNotFoundError

errorResourceNotFoundError

A requested resource does not exist in the store.

ResourceAlreadyExistsError

errorResourceAlreadyExistsError

A resource with the given identity already exists.

TransientStoreError

errorTransientStoreError

A store operation failed for a transient, retryable reason.

Signals a temporary backend condition (connection blip, failover, server-selection timeout) rather than a permanent fault. Callers that cannot tolerate a dropped operation (e.g. the JobStatusWorker observer, where a lost write strands a job) can retry on it without depending on the concrete backend.

The job store and the job-results store raise this type for transient failures, and other errors from these two stores are permanent. The document, results and publishing stores raise the backend error unchanged.

Retry on transient errors

run_with_transient_retry

functionrun_with_transient_retryasync [R](operation, *, attempts=5, base_delay=1.0, max_delay=8.0, on_retry=None) -> R

Run operation, retrying TransientStoreError with exponential backoff.

Permanent errors propagate immediately — only TransientStoreError is retried. After attempts transient failures the last error is re-raised, leaving the caller to decide whether to propagate or drop.

Defaults (5 attempts, 1s base, 8s cap → ~15s total) are sized to ride a MongoDB replica-set failover/election (electionTimeoutMillis defaults to 10s); a longer outage is treated as a real one.

paramoperationCallable[[], Awaitable[R]]

Zero-argument async callable. Re-invoked from scratch on each attempt, so it must be idempotent.

paramattemptsint
= 5

Maximum number of attempts (must be >= 1).

parambase_delayfloat
= 1.0

Base backoff in seconds; before the try after attempt n it sleeps base_delay * 2 ** (n - 1).

parammax_delayfloat
= 8.0

Upper bound on a single backoff sleep, so the exponential growth cannot run away.

paramon_retryCallable[[int, TransientStoreError], None] | None
= None

Optional callback on_retry(attempt, error) invoked before each backoff sleep — e.g. for contextual logging.

Returns

R

The result of operation.

Raises:

  • TransientStoreError: if every attempt fails transiently.

classify_transient

functionclassify_transient[**P, R](func) -> Callable[P, Awaitable[R]]

Re-raise transient pymongo errors from func as TransientStoreError.

Permanent errors propagate unchanged. Wrap async store methods so callers can retry on the standard TransientStoreError type.

paramfuncCallable[P, Awaitable[R]]

classify_transient_methods

functionclassify_transient_methods(cls) -> type

Class decorator: apply classify_transient to every public async method.

It wraps only the methods that the class itself defines, not inherited methods.

Applies the transient-error contract store-wide, so the whole implementation honours "transient backend failures are raised as TransientStoreError" without per-method boilerplate. Private (_-prefixed) methods are left untouched — public methods that call them still classify at the public boundary.

paramclstype

Scopes

current_scope

constantcurrent_scope: ContextVar[str | None] = ContextVar('warren_scope', default=None)

The scope of the current handler, set by the consumer for each message.

get_current_scope

functionget_current_scope() -> str | None

Return the raw scope of the current handler.

require_scope

functionrequire_scope(*, required=True) -> str | None
paramrequiredbool
= True

valid_scope

functionvalid_scope(scope) -> TypeGuard[str]

Whether a raw scope is a valid namespace.

A valid scope is a string of 1 to 40 characters that matches [a-z0-9][a-z0-9-]{0,39}.

paramscopeobject

scope_fields

functionscope_fields(enabled, scope=None) -> dict[str, str]

Fields for a control row, without rejecting invalid envelopes.

paramenabledbool
paramscopeobject
= None

scoped_cache_key

functionscoped_cache_key(key, *, required=True) -> str
paramkeystr
paramrequiredbool
= True

ScopedDatabase

classScopedDatabase(client, prefix='wr_', *, required=True, default_database=None)

Resolve content collections and prepare indexes once per namespace.

Calling the instance returns the database prefix + scope for the scope of the current handler.

paramclientAsyncMongoClient

Async MongoDB client.

paramprefixstr
= 'wr_'

Prefix of each scope database name.

paramrequiredbool
= True

Raise HardFailureException when the handler has no scope. False: use default_database.

paramdefault_databasestr | None
= None

Database to use when the handler has no scope and required is False. If it is None, a missing scope raises HardFailureException.

Raises:

  • HardFailureException: On call, if the scope is malformed, or missing with no fallback database.
method__call__() -> AsyncDatabase
methodcollectionasync (name, prepare) -> AsyncCollection
paramnamestr
paramprepareCallable[[AsyncCollection], Awaitable[None]]

ScopeClients

protocolScopeClients
attributemongo_clientAsyncMongoClient | None
attributeredis_clientRedis | None

erase_scope

functionerase_scopeasync (infra_or_clients, scope, *, control_database, prefix) -> dict[str, int]

Erase scoped content, cache keys and control rows after traffic is stopped.

It deletes three things:

  • the scope database prefix + scope (or each of its collections, if the user cannot drop the database)
  • the Redis keys that match s:{scope}:*
  • the rows with {"scope": scope} in each collection of control_database
paraminfra_or_clientsScopeClients | tuple[AsyncMongoClient, Redis]

An object with mongo_client and redis_client, or a (mongo_client, redis_client) tuple.

paramscopestr

The scope to erase.

paramcontrol_databasestr

Database of the control rows. It must differ from the scope database.

paramprefixstr

Prefix of the scope database name, as in ScopedDatabase.

Returns

dict[str, int]

Counts: {"collections", "redis_keys", "control_rows"}.

Raises:

  • HardFailureException: If the scope is malformed.
  • ValueError: If the scope database equals control_database, or a MongoDB or Redis client is missing.

Shared stores and helpers

CachedDocumentStore

classCachedDocumentStore(store, cache, *, cache_ttl_seconds=3600, name=None)

Document store with read-through/write-through cache.

Wraps any DocumentStoreInterface with a CacheInterface[Dict] for fast reads. All writes go to both store and cache (write-through). Reads check cache first, falling back to store on miss (read-through).

Cache failures are logged as warnings and do not propagate, except HardFailureException (a missing or malformed scope on a scoped cache). The underlying store is the source of truth.

paramstoreDocumentStoreInterface

Persistent document store.

paramcacheCacheInterface[dict]

Cache for fast reads.

paramcache_ttl_secondsint
= 3600

Expiry for cached entries in seconds.

paramnamestr | None
= None
methodinsertasync (doc, overwrite_existing=False) -> str

Insert document into store and cache.

paramdocdict

Document to insert.

paramoverwrite_existingbool
= False

Allow overwrite if True.

Returns

str

Document ID.

Raises:

  • DocumentAlreadyExistsError: If overwrite_existing is False and a document with the same ID already exists.
methodupsert_manyasync (docs) -> list[str | None]
paramdocslist[dict]
methodupdateasync (doc_id, updates) -> None

Update document in store and invalidate cache.

paramdoc_idstr

Document ID.

paramupdatesdict

Dictionary with updates.

Raises:

  • DocumentNotFoundError: If document does not exist.
methoddeleteasync (doc_id, *, expected=None) -> bool

Delete document from store and cache.

paramdoc_idstr

Document ID.

paramexpecteddict | None
= None

Additional field values required for deletion.

Returns

bool

True if document existed and was deleted.

methodexistsasync (doc_id) -> bool

Check if document exists, trying cache first.

paramdoc_idstr

Document ID.

Returns

bool

True if document exists.

methodget_documentasync (doc_id) -> dict

Get document, trying cache first (read-through).

paramdoc_idstr

Document ID.

Returns

dict

Document dict.

Raises:

  • DocumentNotFoundError: If document does not exist.
methodquery(params, *, sort_by=None) -> AsyncGenerator[dict, None]

Query document store (cache is bypassed).

paramparamsdict

Query parameters.

paramsort_bystr | None
= None

Returns

AsyncGenerator[dict, None]

Async generator of documents found.

methodget_document_type() -> str

Return document type from underlying store.

methodget_doc_id_field() -> str

Return doc ID field name from underlying store.

methodhas_unique_indexasync (index_spec) -> bool

Check unique index on underlying store.

paramindex_specIndexSpec

MemoryStoreRegistry

classMemoryStoreRegistry()

Hands out in-process stores so co-hosted runners share state.

A database shares state by name: two workers that open the collection "chunks" see the same documents. In one process that role is played by this registry. Document stores, results stores and caches are get-or-create on a name, so the worker that writes "chunks" and the worker that reads it get the same store. The job store and the job-results store are one per registry. Build one per process (or one per test, for a fresh world) and pass it to whatever constructs the runners.

The first call for a name fixes that store's configuration; later calls return the existing store and ignore their keyword arguments.

methoddocument_store(collection_name, *, doc_id_field='doc_id', unique_indexes=None) -> MemoryDocumentStore
paramcollection_namestr
paramdoc_id_fieldstr
= 'doc_id'
paramunique_indexeslist[IndexSpec] | None
= None
methodresults_storeasync (collection_name) -> DefaultResultsStore

A real DefaultResultsStore over a memory document store.

Same doc_id_field and unique index create_default_results_store uses on MongoDB, and no cache: reads already come from memory.

paramcollection_namestr
methodjob_store() -> MemoryJobStore
methodjob_results_store() -> MemoryJobResultsStore
methodcache(name, *, default_ttl_seconds=None) -> MemoryCache
paramnamestr
paramdefault_ttl_secondsint | None
= None

configure_ttl_index

functionconfigure_ttl_indexasync (collection, field, seconds) -> None

Configure a positive TTL; leave indexes untouched when unset.

Drops an existing single-field index on field with other options, then creates the TTL index.

paramcollectionAsyncCollection
paramfieldstr
paramsecondsint | None

Raises:

  • ValueError: If seconds is not positive.

On this page