How abhisheksingh1180/fakerapi-elt-pipeline works
Project type and purpose
• This is a data‐engineering ELT pipeline project that uses Apache Airflow to orchestrate the movement and transformation of synthetic “product” data (from FakerAPI) into Snowflake via Google Cloud Storage (GCS).
• Purpose: Extract product data from an HTTP API, stage it in GCS, load into Snowflake tables (stage → refined → confirmed), then run Snowflake stored procedures to apply business logic.
File structure
• Top level
• README.md – project overview
• docker-compose.yaml, dockerfile – containerize Airflow environment with local executor.
• dags/product_pipeline.py – Airflow DAG definition (tasks: extract, load, transform).
• packages/fakerAPI.py – Python module to call the Faker API and write CSV.
• sql/ddl/* – Snowflake DDL scripts to create raw, refined, confirmed schemas and views.
• sql/scripts/* – SQL to move data between stages inside Snowflake.
• temp/*.csv – sample extract output.
README for additional insights
• Confirms key components: Airflow, Faker API, GCS staging, Snowflake stored procedures.
• Describes workflow: extract → stage → load → transform, all orchestrated by Airflow.
a. Main components
1. FakerAPI module (Python) – HTTP client that produces product CSVs
2. Apache Airflow – scheduler & executor running in Docker containers
3. Google Cloud Storage (GCS) – staging bucket for raw CSVs
4. Snowflake – cloud data warehouse hosting stage, refined, confirmed tables + stored procedures
5. Local filesystem or temp folder – intermediate storage in container
6. Docker network – connects Airflow services to GCS and Snowflake
b. Relationships & interactions
• The Airflow DAG’s first task calls FakerAPI.py → writes CSV to temp/ or directly to GCS.
• Next Airflow tasks upload CSVs to GCS buckets.
• Subsequent Airflow tasks issue Snowflake COPY INTO commands to load staging tables.
• Final DAG tasks invoke Snowflake stored procedures (via SnowflakeHook) to transform data in place.
• Airflow’s webserver and scheduler monitor & log each task.
c. Architectural patterns & design principles
• ELT pattern: extract raw, load raw, then transform in-database.
• Containerized microservices: Airflow components (scheduler, webserver, worker) run in Docker for isolation & reproducibility.
• Modular code: separate packages/fakerAPI and DAG definitions.
• Infrastructure as code: Docker and Docker Compose for environment consistency.
d. Technologies & frameworks
• Apache Airflow (Python)
• Docker / Docker Compose
• Google Cloud Storage (GCS)
• Snowflake (SQL DDL + stored procedures)
• FakerAPI (Python library)
Detailed but not over‐engineered
• How Airflow’s DAG steps map to GCS uploads and Snowflake loads.
• Orchestration flow; show monitoring (Airflow WebUI) if helpful.