Overview

Warren is a message-driven framework for distributed processing. Independent items flow through a pipeline of workers. Each worker selects the messages that it handles, stores its results, and publishes references for the next workers. The messages stay small, because the payloads are in storage. To scale a worker type, add replicas that share its queue or consumer group.

Warren gives the worker base classes and the interfaces for pubsub and storage. The runtime connects them to RabbitMQ or Kafka, with MongoDB and Redis, or to in-memory implementations for local runs and tests.

InstallCopy Anchor Link

Warren needs Python 3.12 or later. Install the extra of the transport that you use:

pip install "warren[rmq]"

The memory transport needs no extra. If you select a transport or a document resolver without its extra, Warren raises OptionalDependencyError.

Where to startCopy Anchor Link

• The Quickstart in the repository runs a three-stage example pipeline on RabbitMQ, MongoDB and Redis.
• warren.runtime: compose a pipeline with PipelineSpec and WorkerSpec, and load the settings with RuntimeConfig. For a local run without infrastructure, set backend: memory in the RuntimeConfig, and run the runners in one process with run_in_process.
• warren.workers: the worker base classes, such as AsyncProcessingWorkerBase, and the message that a worker receives, ProcessingMessage.
• warren.pubsub: the publisher and consumer contracts and the routing. The transports are warren.pubsub.rabbitmq, warren.pubsub.kafka and warren.pubsub.memory.
• warren.storage: the stores for documents, results, jobs and the cache, each in its own subpackage.