# george-souza/carbon-intensity-pipeline architecture

> Architecture diagram of the GitHub repository george-souza/carbon-intensity-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/george-souza/carbon-intensity-pipeline
- Repository: https://github.com/george-souza/carbon-intensity-pipeline

## Overview

This is an early-stage UK carbon-intensity ETL pipeline. Its principal workflow is a CLI-selected backfill or incremental run: the runner creates an ingestion record, requests data from the public API, stores raw responses, and finalizes run status. A separate migration command prepares the database schema. The README describes validation, transformation/loading into the half-hourly table, and a future REST API, but those stages are not implemented in the shown runtime. Edges reflect calls shown in excerpts or the README; unsampled schema behavior is not inferred.

Main entry points: [`src/extract/__main__.py`](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/extract/__main__.py) (Extraction CLI), [`src/db.py`](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/db.py) (Database operations), [`src/migrate.py`](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/migrate.py) (Migration runner), [`src/extract/client.py`](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/extract/client.py) (API client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Extraction"]
  node_extract_cli["Extraction CLI<br/>[__main__.py]"]
  node_runner["Run orchestration<br/>[runner.py]"]
  node_client["API client<br/>[client.py]"]
  node_validation["Validation"]
  node_transform_load["Transform and load"]
end

subgraph group_persistence["Persistence"]
  node_db_helpers["Database operations<br/>[db.py]"]
  node_postgres[("PostgreSQL")]
  node_sql_migrations["SQL migrations"]
  node_half_hourly[("Half-hourly data")]
end

subgraph group_operations["Operations"]
  node_settings["Configuration<br/>[config.py]"]
  node_logging["Logging setup<br/>[logging_setup.py]"]
  node_migrator["Migration runner<br/>[migrate.py]"]
  node_rest_api["Future REST API"]
end

node_operator(("Operator"))
node_api{{"Carbon Intensity API"}}

node_operator -->|"runs"| node_extract_cli
node_extract_cli -->|"loads settings"| node_settings
node_extract_cli -->|"configures"| node_logging
node_extract_cli -->|"dispatches run"| node_runner
node_extract_cli -->|"creates client"| node_client
node_extract_cli -->|"creates engine"| node_db_helpers
node_client -->|"reads settings"| node_settings
node_client -->|"requests data"| node_api
node_runner -->|"fetches data"| node_client
node_runner -->|"records run and responses"| node_db_helpers
node_db_helpers -->|"gets database URL"| node_settings
node_db_helpers -->|"reads and writes"| node_postgres
node_operator -->|"runs"| node_migrator
node_migrator -->|"gets engine"| node_db_helpers
node_migrator -->|"applies"| node_sql_migrations
node_migrator -->|"tracks schema"| node_postgres
node_api -.->|"planned input"| node_validation
node_validation -.->|"planned stage"| node_transform_load
node_transform_load -.->|"planned load"| node_half_hourly
node_half_hourly -.->|"future serving"| node_rest_api

click node_extract_cli "https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/extract/__main__.py"
click node_settings "https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/config.py"
click node_logging "https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/logging_setup.py"
click node_runner "https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/extract/runner.py"
click node_client "https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/extract/client.py"
click node_db_helpers "https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/db.py"
click node_migrator "https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/migrate.py"
click node_sql_migrations "https://github.com/george-souza/carbon-intensity-pipeline/tree/main/sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extract_cli,node_runner,node_client,node_validation,node_transform_load toneBlue
class node_db_helpers,node_postgres,node_sql_migrations,node_half_hourly toneAmber
class node_settings,node_logging,node_migrator,node_rest_api,node_api toneMint
class node_operator toneIndigo
```

## Components

### Extraction

- [Extraction CLI](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/extract/__main__.py): `src/extract/__main__.py`
- [Run orchestration](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/extract/runner.py): `src/extract/runner.py`
- [API client](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/extract/client.py): `src/extract/client.py`
- **Validation**
- **Transform and load**

### Persistence

- [Database operations](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/db.py): `src/db.py`
- **PostgreSQL**
- [SQL migrations](https://github.com/george-souza/carbon-intensity-pipeline/tree/main/sql): `sql`
- **Half-hourly data**

### Operations

- [Configuration](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/config.py): `src/config.py`
- [Logging setup](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/logging_setup.py): `src/logging_setup.py`
- [Migration runner](https://github.com/george-souza/carbon-intensity-pipeline/blob/main/src/migrate.py): `src/migrate.py`
- **Future REST API**

### Other components

- **Operator**
- **Carbon Intensity API**

## Connections

- Operator → Extraction CLI: runs (evidence: `README.md`)
- Extraction CLI → Configuration: loads settings (evidence: `src/extract/__main__.py`)
- Extraction CLI → Logging setup: configures (evidence: `src/extract/__main__.py`)
- Extraction CLI → Run orchestration: dispatches run (evidence: `src/extract/__main__.py`)
- Extraction CLI → API client: creates client (evidence: `src/extract/__main__.py`)
- Extraction CLI → Database operations: creates engine (evidence: `src/extract/__main__.py`)
- API client → Configuration: reads settings (evidence: `src/extract/client.py`)
- API client → Carbon Intensity API: requests data (evidence: `src/extract/client.py`)
- Run orchestration → API client: fetches data (evidence: `src/extract/runner.py`)
- Run orchestration → Database operations: records run and responses (evidence: `src/extract/runner.py`)
- Database operations → Configuration: gets database URL (evidence: `src/db.py`)
- Database operations → PostgreSQL: reads and writes (evidence: `src/db.py`)
- Operator → Migration runner: runs (evidence: `README.md`)
- Migration runner → Database operations: gets engine (evidence: `src/migrate.py`)
- Migration runner → SQL migrations: applies (evidence: `src/migrate.py`)
- Migration runner → PostgreSQL: tracks schema (evidence: `src/migrate.py`)
- Carbon Intensity API → Validation: planned input (evidence: `README.md`)
- Validation → Transform and load: planned stage (evidence: `README.md`)
- Transform and load → Half-hourly data: planned load (evidence: `README.md`)
- Half-hourly data → Future REST API: future serving (evidence: `README.md`)
