# charity-kat/my-first-data-pipeline architecture

> Architecture diagram of the GitHub repository charity-kat/my-first-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/charity-kat/my-first-data-pipeline
- Repository: https://github.com/charity-kat/my-first-data-pipeline

## Overview

This is a small Python crypto-price data pipeline. A user runs the orchestrator, which invokes extraction from CoinGecko, transformation of the raw CSV, and loading of the clean CSV into SQLite. The transform adds an extraction timestamp and calculated USD/ZAR rate; loading also queries the table to verify its contents. The README documents no further capabilities, so this map follows the sampled runtime source.

Main entry points: [`load.py`](https://github.com/charity-kat/my-first-data-pipeline/blob/main/load.py) (Database loading), [`pipeline.py`](https://github.com/charity-kat/my-first-data-pipeline/blob/main/pipeline.py) (Pipeline runner), [`extract.py`](https://github.com/charity-kat/my-first-data-pipeline/blob/main/extract.py) (Price extraction), [`transform.py`](https://github.com/charity-kat/my-first-data-pipeline/blob/main/transform.py) (Price transformation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Pipeline runtime"]
  node_pipeline["Pipeline runner<br/>[pipeline.py]"]
end

subgraph group_stages["Data stages"]
  node_extract["Price extraction<br/>[extract.py]"]
  node_transform["Price transformation<br/>[transform.py]"]
  node_load["Database loading<br/>[load.py]"]
end

subgraph group_storage["Data storage"]
  node_rawcsv["Raw prices CSV"]
  node_cleancsv["Clean prices CSV"]
  node_sqlite[("Crypto prices DB<br/>[crypto_pipeline.db]")]
end

node_user(("User"))
node_coingecko["CoinGecko API"]

node_user -->|"runs"| node_pipeline
node_pipeline -->|"calls first"| node_extract
node_pipeline -->|"calls second"| node_transform
node_pipeline -->|"calls third"| node_load
node_extract -->|"requests prices"| node_coingecko
node_extract -->|"writes raw data"| node_rawcsv
node_transform -->|"reads raw data"| node_rawcsv
node_transform -->|"writes clean data"| node_cleancsv
node_load -->|"reads clean data"| node_cleancsv
node_load -->|"appends rows"| node_sqlite
node_load -->|"queries table"| node_sqlite
node_sqlite -->|"returns rows"| node_load

click node_pipeline "https://github.com/charity-kat/my-first-data-pipeline/blob/main/pipeline.py"
click node_extract "https://github.com/charity-kat/my-first-data-pipeline/blob/main/extract.py"
click node_transform "https://github.com/charity-kat/my-first-data-pipeline/blob/main/transform.py"
click node_load "https://github.com/charity-kat/my-first-data-pipeline/blob/main/load.py"
click node_rawcsv "https://github.com/charity-kat/my-first-data-pipeline/blob/main/data/raw_crypto_prices.csv"
click node_cleancsv "https://github.com/charity-kat/my-first-data-pipeline/blob/main/data/clean_crypto_prices.csv"
click node_sqlite "https://github.com/charity-kat/my-first-data-pipeline/blob/main/data/crypto_pipeline.db"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_user toneBlue
class node_extract,node_transform,node_load toneAmber
class node_rawcsv,node_cleancsv,node_sqlite,node_coingecko toneMint
```

## Components

### Pipeline runtime

- [Pipeline runner](https://github.com/charity-kat/my-first-data-pipeline/blob/main/pipeline.py): `pipeline.py`

### Data stages

- [Price extraction](https://github.com/charity-kat/my-first-data-pipeline/blob/main/extract.py): `extract.py`
- [Price transformation](https://github.com/charity-kat/my-first-data-pipeline/blob/main/transform.py): `transform.py`
- [Database loading](https://github.com/charity-kat/my-first-data-pipeline/blob/main/load.py): `load.py`

### Data storage

- [Raw prices CSV](https://github.com/charity-kat/my-first-data-pipeline/blob/main/data/raw_crypto_prices.csv): `data/raw_crypto_prices.csv`
- [Clean prices CSV](https://github.com/charity-kat/my-first-data-pipeline/blob/main/data/clean_crypto_prices.csv): `data/clean_crypto_prices.csv`
- [Crypto prices DB](https://github.com/charity-kat/my-first-data-pipeline/blob/main/data/crypto_pipeline.db): `data/crypto_pipeline.db`

### Other components

- **User**
- **CoinGecko API**

## Connections

- User → Pipeline runner: runs
- Pipeline runner → Price extraction: calls first
- Pipeline runner → Price transformation: calls second
- Pipeline runner → Database loading: calls third
- Price extraction → CoinGecko API: requests prices
- Price extraction → Raw prices CSV: writes raw data
- Price transformation → Raw prices CSV: reads raw data
- Price transformation → Clean prices CSV: writes clean data
- Database loading → Clean prices CSV: reads clean data
- Database loading → Crypto prices DB: appends rows
- Database loading → Crypto prices DB: queries table
- Crypto prices DB → Database loading: returns rows