Pipelines API

POST /api/pipelines

Create a new pipeline.

Request

{
  "name": "HubSpot to BigQuery",
  "config": {
    "name": "hubspot-contacts",
    "source": {
      "name": "hubspot",
      "stream": "contacts",
      "authentication": {
        "type": "api_key",
        "params": { "token": "pat-xxx" }
      }
    },
    "destination": {
      "name": "bigquery",
      "config": {
        "project_id": "my-project",
        "dataset": "raw_data"
      }
    }
  },
  "schedule": "0 */6 * * *",
  "enabled": true,
  "domain_id": "uuid"
}

Response

{
  "id": "uuid",
  "name": "HubSpot to BigQuery",
  "schedule": "0 */6 * * *",
  "enabled": true,
  "domain_id": "uuid",
  "created_at": "2024-01-15T10:00:00Z",
  "updated_at": null
}

Note: The config field is encrypted at rest. The full config is only returned on GET /api/pipelines/{id}.

GET /api/pipelines

List all pipelines in the organization.

Query Parameters:

Parameter	Type	Default	Description
limit	int	100	Maximum results
offset	int	0	Pagination offset
domain_id	uuid	null	Filter by domain

[
  {
    "id": "uuid",
    "name": "HubSpot to BigQuery",
    "schedule": "0 */6 * * *",
    "enabled": true,
    "domain_id": "uuid",
    "domain_name": "Marketing",
    "last_run_status": "success",
    "last_run_at": "2024-01-15T16:00:00Z",
    "created_at": "2024-01-15T10:00:00Z"
  }
]

GET /api/pipelines/{id}

Get a pipeline by ID, including full config.

{
  "id": "uuid",
  "name": "HubSpot to BigQuery",
  "config": {
    "name": "hubspot-contacts",
    "source": { ... },
    "destination": { ... }
  },
  "schedule": "0 */6 * * *",
  "enabled": true,
  "domain_id": "uuid",
  "created_at": "2024-01-15T10:00:00Z",
  "updated_at": null
}

PUT /api/pipelines/{id}

Update a pipeline.

{
  "name": "HubSpot Contacts (Production)",
  "schedule": "0 */4 * * *",
  "enabled": false
}

DELETE /api/pipelines/{id}

Delete a pipeline and all its runs.

Returns 204 No Content.

POST /api/pipelines/{id}/run

Trigger a pipeline run manually.

{
  "id": "uuid",
  "pipeline_id": "uuid",
  "status": "pending",
  "triggered_by": "manual",
  "created_at": "2024-01-15T10:30:00Z"
}

POST /api/pipelines/{id}/duplicate

Create a copy of a pipeline.

{
  "name": "HubSpot to BigQuery (Copy)"  // optional, auto-generated if omitted
}

Pipeline Runs

GET /api/pipelines/{id}/runs

List runs for a pipeline.

Query Parameters:

Parameter	Type	Default	Description
limit	int	100	Maximum results
offset	int	0	Pagination offset
status	string	null	Filter by status

[
  {
    "id": "uuid",
    "pipeline_id": "uuid",
    "status": "success",
    "triggered_by": "schedule",
    "started_at": "2024-01-15T16:00:00Z",
    "finished_at": "2024-01-15T16:05:23Z",
    "error": null,
    "created_at": "2024-01-15T16:00:00Z"
  }
]

GET /api/pipelines/runs/{run_id}

Get run details.

{
  "id": "uuid",
  "pipeline_id": "uuid",
  "status": "success",
  "triggered_by": "manual",
  "started_at": "2024-01-15T16:00:00Z",
  "finished_at": "2024-01-15T16:05:23Z",
  "error": null,
  "output_file": "outputs/uuid/output.parquet",
  "output_file_size": 1048576
}

GET /api/pipelines/runs/{run_id}/logs

Get run logs.

{
  "logs": "2024-01-15 16:00:00 INFO Starting pipeline...\n..."
}

GET /api/pipelines/runs/{run_id}/download

Download run output file. Returns file with appropriate Content-Type.

POST /api/pipelines/runs/{run_id}/cancel

Cancel a running pipeline.

{
  "status": "cancelled"
}

Run Status Values

Status	Description
pending	Queued, waiting for worker
running	Currently executing
success	Completed successfully
failed	Failed with error
cancelled	Manually cancelled