# mxavier-dev/countries-etl-pipeline architecture

> Architecture diagram of the GitHub repository mxavier-dev/countries-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mxavier-dev/countries-etl-pipeline
- Repository: https://github.com/mxavier-dev/countries-etl-pipeline

## Overview

This repository is a small country-data ETL application. Its user-to-result workflow is to run the pipeline, fetch country records from the REST Countries API, save the raw response, transform selected fields, save the processed records, and insert them into MySQL. The graph also shows the database connection configuration and pipeline logger. Pipeline sequencing is visible in the sampled entry point; schema setup is documented in the README. The graph does not assume further runtime behavior from unsampled files.

Main entry points: [`src/pipeline.py`](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/pipeline.py) (Pipeline orchestrator), [`src/load.py`](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/load.py) (Country persistence), [`src/extract.py`](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/extract.py) (Country extraction), [`src/connect/connection.py`](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/connect/connection.py) (MySQL connection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["ETL workflow"]
  node_pipeline["Pipeline orchestrator<br/>[pipeline.py]"]
  node_extract["Country extraction<br/>[extract.py]"]
  node_transform["Country transformation<br/>[transform.py]"]
  node_load["Country persistence<br/>[load.py]"]
end

subgraph group_storage["Data storage"]
  node_raw["Raw JSON archive"]
  node_processed["Processed JSON archive"]
  node_schema["Countries schema<br/>[schema.sql]"]
  node_database[("MySQL database")]
end

subgraph group_support["Runtime support"]
  node_connection["MySQL connection<br/>[connection.py]"]
  node_config["Database configuration<br/>[config.py]"]
  node_logger["Pipeline logger<br/>[logger.py]"]
end

node_operator(("Pipeline operator"))
node_api{{"REST Countries API"}}

node_operator -->|"runs"| node_pipeline
node_pipeline -->|"calls"| node_extract
node_extract -->|"fetches records"| node_api
node_pipeline -->|"saves response"| node_raw
node_pipeline -->|"calls"| node_transform
node_pipeline -->|"saves transformed data"| node_processed
node_pipeline -->|"calls insert"| node_load
node_load -->|"connects via"| node_connection
node_connection -->|"reads settings"| node_config
node_load -->|"inserts records"| node_database
node_schema -->|"defines schema"| node_database
node_pipeline -->|"uses"| node_logger

click node_pipeline "https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/pipeline.py"
click node_extract "https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/extract.py"
click node_transform "https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/transform.py"
click node_raw "https://github.com/mxavier-dev/countries-etl-pipeline/tree/main/data/raw"
click node_processed "https://github.com/mxavier-dev/countries-etl-pipeline/tree/main/data/processed"
click node_load "https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/load.py"
click node_connection "https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/connect/connection.py"
click node_config "https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/connect/config.py"
click node_schema "https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/db/schema.sql"
click node_logger "https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/utils/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_extract,node_transform,node_load toneBlue
class node_raw,node_processed,node_schema,node_database toneAmber
class node_connection,node_config,node_logger,node_api toneMint
class node_operator toneIndigo
```

## Components

### ETL workflow

- [Pipeline orchestrator](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`
- [Country extraction](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/extract.py): `src/extract.py`
- [Country transformation](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/transform.py): `src/transform.py`
- [Country persistence](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/load.py): `src/load.py`

### Data storage

- [Raw JSON archive](https://github.com/mxavier-dev/countries-etl-pipeline/tree/main/data/raw): `data/raw`
- [Processed JSON archive](https://github.com/mxavier-dev/countries-etl-pipeline/tree/main/data/processed): `data/processed`
- [Countries schema](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/db/schema.sql): `db/schema.sql`
- **MySQL database**

### Runtime support

- [MySQL connection](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/connect/connection.py): `src/connect/connection.py`
- [Database configuration](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/connect/config.py): `src/connect/config.py`
- [Pipeline logger](https://github.com/mxavier-dev/countries-etl-pipeline/blob/main/src/utils/logger.py): `src/utils/logger.py`

### Other components

- **Pipeline operator**
- **REST Countries API**

## Connections

- Pipeline operator → Pipeline orchestrator: runs (evidence: `README.md`)
- Pipeline orchestrator → Country extraction: calls (evidence: `src/pipeline.py`)
- Country extraction → REST Countries API: fetches records (evidence: `src/extract.py`)
- Pipeline orchestrator → Raw JSON archive: saves response (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Country transformation: calls (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Processed JSON archive: saves transformed data (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Country persistence: calls insert (evidence: `src/pipeline.py`)
- Country persistence → MySQL connection: connects via (evidence: `src/load.py`)
- MySQL connection → Database configuration: reads settings (evidence: `src/connect/connection.py`)
- Country persistence → MySQL database: inserts records (evidence: `src/load.py`)
- Countries schema → MySQL database: defines schema (evidence: `README.md`)
- Pipeline orchestrator → Pipeline logger: uses (evidence: `src/pipeline.py`)
