How andreichiro/data_engineer_end2end works
Project Type and Purpose
• This is an end-to-end data‐engineering and inference platform—a modern data-stack pipeline that ingests YouTube metadata and transcriptions, processes and models it (Medallion architecture), builds a data mart for analytics, then surfaces it via a Django-based inference service (LLM fine-tuned with Whisper transcripts).
File‐Structure Analysis
• handlers-airflow: Airflow DAG definitions, Docker-Compose, Terraform/Ansible for spinning up Airflow, design-pattern based extractors (YouTube, AWS hooks/operators), Kinesis hooks.
• databricks: Spark notebooks and Spark SQL for staging layer transformations.
• app_dbt: dbt Cloud project implementing intermediate and data-mart layers with CTEs and Snowflake schema.
• aws: AWS Lambda scripts (Kinesis trigger, S3 trigger), logging samples.
• application: Terraform + Docker setup for AWS ECS + Django app (inference API + Streamlit front-end assets), GitHub Actions, Django settings, static assets, urls/views.
• infra: Terraform modules for ECS, ECR, VPC, ALB, Security Groups, Roles.
• data: locally checked-in raw, staging, mart parquet files (mirror of S3/Delta).
• samples: logs and artifact samples from each subsystem.
Key Insights from README
• ELT approach: Extract with Airflow → Load to S3/Kinesis → Transform in Databricks → Model in dbt → Serve via Django + LLM.
• Event-driven architecture: S3 triggers OR Kinesis triggers launch downstream jobs (Databricks staging layer, Lambdas).
• Technologies: YouTube API, Whisper/OpenAI, Airflow, Docker, Terraform, Ansible, AWS (Glue, Lambda, API Gateway, Kinesis, Athena, ECS, SageMaker), Databricks, dbt, Django, Streamlit, LLM (longchain).
Main Components & Interactions
a. Data Ingestion Layer
• Airflow (Docker-Compose) runs DAGs using YouTube operators → writes raw JSON to S3.
• AWS Lambda (lambda_kinesis.py) ingest streaming transcripts from Kinesis → writes raw transcripts to S3.
• S3 bucket with “raw/” prefix triggers Lambda or Databricks job.
b. Staging & Transformation Layer
• Databricks Spark jobs consume raw S3 data, flatten JSON, write Delta Lake under “staging/”.
• AWS Glue & Athena can catalog the staging data.
c. Modeling Layer
• dbt Cloud project reads Delta/staging tables → builds intermediate tables → Snowflake schema marts → writes “mart/” layer.
• dbt tests for freshness, surrogate keys, data quality.
d. Serving & Inference Layer
• Dockerized Django application deployed on AWS ECS behind an ALB → exposes REST/Streamlit UI.
• Django loads mart data via Athena or Redshift (?) and loads fine-tuned LLM (SageMaker endpoint) for question answering.
• OpenAI Whisper used offline to generate transcripts; longchain for LLM workflows.
e. Infrastructure & Orchestration
• Terraform + Ansible modules define VPC, ECS cluster, ECR repos, roles, security groups, ALB.
• GitHub Actions CI for Django tests across OS.
• Terraform in handlers-airflow spins up Airflow infra; playbook.yml for Ansible.
Architectural Patterns & Design Principles
• Medallion (bronze→silver→gold) layering.
• Event-driven triggers (S3/Kinesis→Lambda/DATABRICKS).
• Factory and abstract-class patterns in Airflow hooks/operators.
• Infrastructure as Code with Terraform/Ansible.
• CI/CD pipelines (GitHub Actions, Docker).
• Separation of concerns: ingestion, transformation, modeling, serving.
Essential Details to Capture
• Sequence: Extract → Load → Transform → Model → Serve.
• Event triggers: S3 and Kinesis events.
• Technologies at each step.
• CI/CD and IaC governing infrastructure and code deployment.