How mxavier-dev/countries-etl-pipeline works
This repository is a small country-data ETL application. Its user-to-result workflow is to run the pipeline, fetch country records from the REST Countries API, save the raw response, transform selected fields, save the processed records, and insert them into MySQL. The graph also shows the database connection configuration and pipeline logger. Pipeline sequencing is visible in the sampled entry point; schema setup is documented in the README. The graph does not assume further runtime behavior from unsampled files.
Main entry points: pipeline.py Pipeline orchestrator · load.py Country persistence · extract.py Country extraction · connection.py MySQL connection
Components 13
ETL workflow
- Pipeline orchestrator
src/pipeline.py - Country extraction
src/extract.py - Country transformation
src/transform.py - Country persistence
src/load.py
Data storage
- Raw JSON archive
data/raw - Processed JSON archive
data/processed - Countries schema
db/schema.sql - MySQL database
Runtime support
- MySQL connection
src/connect/connection.py - Database configuration
src/connect/config.py - Pipeline logger
src/utils/logger.py
Other components
- Pipeline operator
- REST Countries API
Connections 12
- Pipeline operator → Pipeline orchestrator: runs
README.md - Pipeline orchestrator → Country extraction: calls
src/pipeline.py - Country extraction → REST Countries API: fetches records
src/extract.py - Pipeline orchestrator → Raw JSON archive: saves response
src/pipeline.py - Pipeline orchestrator → Country transformation: calls
src/pipeline.py - Pipeline orchestrator → Processed JSON archive: saves transformed data
src/pipeline.py - Pipeline orchestrator → Country persistence: calls insert
src/pipeline.py - Country persistence → MySQL connection: connects via
src/load.py - MySQL connection → Database configuration: reads settings
src/connect/connection.py - Country persistence → MySQL database: inserts records
src/load.py - Countries schema → MySQL database: defines schema
README.md - Pipeline orchestrator → Pipeline logger: uses
src/pipeline.py