Queues

Bizon is queue-system agnostic—you can use what you already run, or start simple and scale later.

Why Queues?

Queues provide a buffer between extraction and loading:

• Decoupling: Sources and destinations work independently
• Reliability: Data is preserved if destinations are temporarily unavailable
• Throughput: Enable parallel processing and backpressure handling

Available Queue Options

Python Queue

Best for: Development and testing

queue:
  type: python

• No external dependencies
• In-memory only (data lost on restart)
• Single-process execution

RabbitMQ

Best for: Production with high throughput

queue:
  type: rabbitmq
  config:
    host: localhost
    port: 5672
    username: guest
    password: guest

Install the optional dependency:

pip install bizon[rabbitmq]

Features:

• Persistent messaging
• Acknowledgment-based delivery
• Horizontal scaling with multiple consumers

Kafka / Redpanda

Best for: Production with persistence and replay

queue:
  type: kafka
  config:
    bootstrap_servers: localhost:9092
    topic: bizon-pipeline

Install the optional dependency:

pip install bizon[kafka]

Features:

• Log-based persistence
• Message replay capability
• High throughput at scale
• Works with Kafka or Redpanda

Choosing a Queue

Queue	Persistence	Throughput	Complexity	Use Case
Python Queue	No	Low	None	Development
RabbitMQ	Yes	High	Medium	Production
Kafka	Yes	Very High	High	High-volume production

Migration Path

Start simple and scale as needed:

1. Development: Use Python Queue for quick iteration
2. Staging: Move to RabbitMQ for realistic testing
3. Production: Use Kafka/Redpanda for maximum throughput

The pipeline configuration stays the same—only the queue section changes.

Next Steps

• See the Configuration Reference for all options
• Learn about Architecture for the full picture