Skip to content
Bizon Bizon

Configuration Reference

This page documents all available configuration options for Bizon pipelines.

Pipeline Configuration

Section titled “Pipeline Configuration”

A complete pipeline configuration file:

name: my-pipeline


source:
  type: <source_type>
  config:
    # Source-specific configuration


destination:
  type: <destination_type>
  config:
    # Destination-specific configuration


queue:
  type: python | rabbitmq | kafka
  config:
    # Queue-specific configuration


backend:
  type: sqlite | postgres | bigquery
  config:
    # Backend-specific configuration


runner:
  mode: thread | process | stream
  workers: 4  # Number of parallel workers


metrics:
  enabled: true
  provider: datadog | opentelemetry
  config:
    # Provider-specific configuration

Source Options

Section titled “Source Options”

Common Options

Section titled “Common Options”
OptionTypeDescription
typestringSource type identifier
configobjectSource-specific configuration

Destination Options

Section titled “Destination Options”

BigQuery

Section titled “BigQuery”
destination:
  type: bigquery
  config:
    project_id: your-gcp-project
    dataset: your_dataset
    table: your_table
    credentials_path: /path/to/credentials.json  # Optional

Queue Options

Section titled “Queue Options”

Python Queue

Section titled “Python Queue”
queue:
  type: python
  config:
    max_size: 10000  # Maximum queue size

RabbitMQ

Section titled “RabbitMQ”
queue:
  type: rabbitmq
  config:
    host: localhost
    port: 5672
    username: guest
    password: guest
    virtual_host: /
    queue_name: bizon

Kafka

Section titled “Kafka”
queue:
  type: kafka
  config:
    bootstrap_servers: localhost:9092
    topic: bizon-pipeline
    consumer_group: bizon-group

Backend Options

Section titled “Backend Options”

SQLite

Section titled “SQLite”
backend:
  type: sqlite
  config:
    path: :memory:  # Or path to file

PostgreSQL

Section titled “PostgreSQL”
backend:
  type: postgres
  config:
    connection_string: postgresql://user:pass@host:5432/db
    checkpoint_interval: 1000

BigQuery

Section titled “BigQuery”
backend:
  type: bigquery
  config:
    project_id: your-gcp-project
    dataset: bizon_state

Runner Options

Section titled “Runner Options”
runner:
  mode: thread  # thread, process, or stream
  workers: 4    # Number of parallel workers
  batch_size: 1000  # Records per batch

Metrics Options

Section titled “Metrics Options”

Datadog

Section titled “Datadog”
metrics:
  enabled: true
  provider: datadog
  config:
    api_key: your-api-key
    app_key: your-app-key

OpenTelemetry

Section titled “OpenTelemetry”
metrics:
  enabled: true
  provider: opentelemetry
  config:
    endpoint: http://localhost:4317
    service_name: bizon-pipeline

Environment Variables

Section titled “Environment Variables”

Configuration values can reference environment variables:

destination:
  type: bigquery
  config:
    project_id: ${GCP_PROJECT_ID}