How peddiaravind/weather_api_etl works
This repository is a weather-data ETL application. Running main.py creates the MySQL table, fetches current conditions for the default city, transforms the API JSON into a one-row Pandas DataFrame, loads it into MySQL, and exports the DataFrame to CSV. Configuration and logging support the runtime stages. The workflow is based on main.py and the sampled stage implementations; README-listed capabilities such as CSV export are included, while unsampled behavior is not inferred.
Main entry points: main.py Pipeline runner · extract.py Weather extraction · load.py MySQL loading · connection.py Database connection
Components 13
ETL orchestration
- Pipeline runner
main.py - CSV export
main.py
Weather data stages
- Weather extraction
extract.py - Data transformation
transform.py - MySQL loading
load.py
Persistence
- Table setup
db/create_tables.py - Database connection
db/connection.py
Shared services
- Runtime configuration
config/config.py - ETL logger
utils/logger.py
Other components
- Pipeline operator
- WeatherAPI
- MySQL database
- Environment variables
Connections 19
- Pipeline operator → Pipeline runner: runs
- Pipeline runner → Table setup: creates table
- Pipeline runner → Weather extraction: requests data
- Weather extraction → WeatherAPI: fetches JSON
- Weather extraction → Data transformation
- Pipeline runner → Data transformation: transforms data
- Pipeline runner → MySQL loading: loads DataFrame
- Pipeline runner → CSV export: exports CSV
- Table setup → Database connection: requests connection
- MySQL loading → Database connection: requests connection
- Database connection → Runtime configuration: reads settings
- Runtime configuration → Environment variables: reads variables
- Database connection → MySQL database: connects
- Table setup → MySQL database: creates table
- MySQL loading → MySQL database: inserts rows
- Weather extraction → Runtime configuration: reads API key
- Weather extraction → ETL logger: logs events
- Data transformation → ETL logger: logs events
- MySQL loading → ETL logger: logs events