How feritbulut/zomato-ai-data-pipeline-with-aws-snowflake-airflow-dbt works
This repository implements a scheduled food-delivery analytics platform: CSV data is staged in S3, loaded to Snowflake, transformed through dbt staging and marts, and exposed through AI-powered review and warehouse-query apps. The Airflow DAG orders the daily load, core build, review enrichment, and AI build. RAG and text-to-SQL are interactive paths alongside the scheduled pipeline. Some capabilities and model lineage are documented in the README but their source was not sampled, so only documented connections are shown.
Main entry points: zomato_batch.py Daily DAG · enrich_reviews.py Review enrichment · rag_chat.py Review RAG app · text_to_sql.py Text-to-SQL app
Components 15
Ingestion and warehouse
- AWS IAM access
aws - Snowflake setup
snowflake/01_setup.sql - External stage
snowflake/03_stage_and_formats.sql - Snowflake RAW
snowflake/04_raw_table.sql
dbt transformations
- dbt staging
zomato/models/staging - dbt marts
zomato/models/marts
Daily orchestration
- Daily DAG
airflow/dags/zomato_batch.py
AI capabilities
- Review enrichment
ai/enrich_reviews.py - Enriched reviews
ai/enrich_reviews.py
Interactive analytics
- Review RAG app
ai/rag_chat.py - Text-to-SQL app
ai/text_to_sql.py
Other components
- CSV data
- Amazon S3
- Gemini API
- Analytics user
Connections 23
- CSV data → Amazon S3: uploaded as
- AWS IAM access → External stage: authorizes access
- Snowflake setup → External stage: sets up
- External stage → Amazon S3: reads files
- Daily DAG → Snowflake RAW: runs COPY INTO
- Amazon S3 → Snowflake RAW: loads CSVs
- Daily DAG → dbt staging: runs core build
- Snowflake RAW → dbt staging: dbt reads
- dbt staging → dbt marts: dbt transforms
- Daily DAG → Review enrichment: runs task
- Review enrichment → Snowflake RAW: reads reviews
- Review enrichment → Gemini API: classifies reviews
- Review enrichment → Enriched reviews: writes labels
- Daily DAG → dbt marts: runs AI build
- Enriched reviews → dbt marts: feeds AI models
- Analytics user → Review RAG app: asks review question
- Review RAG app → dbt staging: samples reviews
- Review RAG app → Gemini API: embeds and answers
- Review RAG app → Analytics user: returns answer and sources
- Analytics user → Text-to-SQL app: asks data question
- Text-to-SQL app → Gemini API: generates SQL
- Text-to-SQL app → dbt marts: executes guarded query
- Text-to-SQL app → Analytics user: returns results