# andreichiro/data_engineer_end2end architecture

> Architecture diagram of the GitHub repository andreichiro/data_engineer_end2end, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/andreichiro/data_engineer_end2end
- Repository: https://github.com/andreichiro/data_engineer_end2end

## Overview

##### Project Type and Purpose
- This is an end-to-end data‐engineering and inference platform—a modern data-stack pipeline that ingests YouTube metadata and transcriptions, processes and models it (Medallion architecture), builds a data mart for analytics, then surfaces it via a Django-based inference service (LLM fine-tuned with Whisper transcripts).

##### File‐Structure Analysis
- handlers-airflow: Airflow DAG definitions, Docker-Compose, Terraform/Ansible for spinning up Airflow, design-pattern based extractors (YouTube, AWS hooks/operators), Kinesis hooks.
- databricks: Spark notebooks and Spark SQL for staging layer transformations.
- app_dbt: dbt Cloud project implementing intermediate and data-mart layers with CTEs and Snowflake schema.
- aws: AWS Lambda scripts (Kinesis trigger, S3 trigger), logging samples.
- application: Terraform + Docker setup for AWS ECS + Django app (inference API + Streamlit front-end assets), GitHub Actions, Django settings, static assets, urls/views.
- infra: Terraform modules for ECS, ECR, VPC, ALB, Security Groups, Roles.
- data: locally checked-in raw, staging, mart parquet files (mirror of S3/Delta).
- samples: logs and artifact samples from each subsystem.

##### Key Insights from README
- ELT approach: Extract with Airflow → Load to S3/Kinesis → Transform in Databricks → Model in dbt → Serve via Django + LLM.
- Event-driven architecture: S3 triggers OR Kinesis triggers launch downstream jobs (Databricks staging layer, Lambdas).
- Technologies: YouTube API, Whisper/OpenAI, Airflow, Docker, Terraform, Ansible, AWS (Glue, Lambda, API Gateway, Kinesis, Athena, ECS, SageMaker), Databricks, dbt, Django, Streamlit, LLM (longchain).

##### Main Components & Interactions
a. Data Ingestion Layer
- Airflow (Docker-Compose) runs DAGs using YouTube operators → writes raw JSON to S3.
- AWS Lambda (lambda_kinesis.py) ingest streaming transcripts from Kinesis → writes raw transcripts to S3.
- S3 bucket with “raw/” prefix triggers Lambda or Databricks job.
b. Staging & Transformation Layer
- Databricks Spark jobs consume raw S3 data, flatten JSON, write Delta Lake under “staging/”.
- AWS Glue & Athena can catalog the staging data.
c. Modeling Layer
- dbt Cloud project reads Delta/staging tables → builds intermediate tables → Snowflake schema marts → writes “mart/” layer.
- dbt tests for freshness, surrogate keys, data quality.
d. Serving & Inference Layer
- Dockerized Django application deployed on AWS ECS behind an ALB → exposes REST/Streamlit UI.
- Django loads mart data via Athena or Redshift (?) and loads fine-tuned LLM (SageMaker endpoint) for question answering.
- OpenAI Whisper used offline to generate transcripts; longchain for LLM workflows.
e. Infrastructure & Orchestration
- Terraform + Ansible modules define VPC, ECS cluster, ECR repos, roles, security groups, ALB.
- GitHub Actions CI for Django tests across OS.
- Terraform in handlers-airflow spins up Airflow infra; playbook.yml for Ansible.

##### Architectural Patterns & Design Principles
- Medallion (bronze→silver→gold) layering.
- Event-driven triggers (S3/Kinesis→Lambda/DATABRICKS).
- Factory and abstract-class patterns in Airflow hooks/operators.
- Infrastructure as Code with Terraform/Ansible.
- CI/CD pipelines (GitHub Actions, Docker).
- Separation of concerns: ingestion, transformation, modeling, serving.

##### Essential Details to Capture
- Sequence: Extract → Load → Transform → Model → Serve.
- Event triggers: S3 and Kinesis events.
- Technologies at each step.
- CI/CD and IaC governing infrastructure and code deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources
    YT["YouTube API"]:::external
    Whisper["OpenAI Whisper"]:::external
    Longchain["Longchain LLM"]:::external

    %% Ingestion Layer
    subgraph "Ingestion Layer" 
        direction TB
        Airflow["Airflow (Docker-Compose)"]:::ingestion
        KinesisLambda["Lambda (Kinesis Trigger)"]:::ingestion
        S3Raw["S3 Bucket (raw/)"]:::storage
    end

    %% Handlers-Airflow Components
    subgraph "Airflow Project" 
        direction TB
        DAG["youtube_aws_dag.py"]:::ingestion
        AbstractHook["abstract_hook.py"]:::ingestion
        AbstractOp["abstract_operator.py"]:::ingestion
        HookFactory["hook_factory.py"]:::ingestion
        OpFactory["operator_factory.py"]:::ingestion
        YTHook["youtube_hook.py"]:::ingestion
        AWSHook["aws_hook.py"]:::ingestion
        YTOp["youtube_operator.py"]:::ingestion
        AWSOp["aws_operator.py"]:::ingestion
        Processors["processors/"]:::ingestion
        Compose["docker-compose.yaml"]:::ingestion
    end

    %% Streaming Ingestion
    subgraph "AWS Lambda" 
        direction TB
        LambdaKinesis["lambda_kinesis.py"]:::ingestion
        LambdaS3["lambda.py"]:::ingestion
    end

    %% Storage Layer - Bronze
    S3Raw -->|Triggers| LambdaS3
    S3Raw -->|Triggers| Databricks

    DAG -->|"writes raw JSON to"| S3Raw
    YTHook --> YTOp
    AWSHook --> AWSOp
    HookFactory --> YTHook
    HookFactory --> AWSHook
    OpFactory --> YTOp
    OpFactory --> AWSOp
    DAG --> AbstractHook
    DAG --> AbstractOp
    DAG --> Processors
    Airflow --> DAG
    Airflow --> Compose

    LambdaKinesis -->|"writes raw transcripts to"| S3Raw

    %% Transformation Layer
    subgraph "Staging & Transformation" 
        direction TB
        Databricks["Databricks Spark"]:::compute
        DeltaLake["Delta Lake (staging/)"]:::storage
    end

    S3Raw -->|Consumes raw data| Databricks
    Databricks -->|Writes flattened data| DeltaLake

    %% Modeling Layer
    subgraph "Modeling Layer" 
        direction TB
        dbtProject["dbt Cloud"]:::modeling
        Intermediate["intermediate/*.sql"]:::modeling
        Marts["marts/*.sql"]:::modeling
        Macros["macros/"]:::modeling
        Sources["sources.yml"]:::modeling
        Tests["tests/"]:::modeling
    end

    DeltaLake -->|Source for models| dbtProject
    dbtProject --> Intermediate
    dbtProject --> Marts
    dbtProject --> Macros
    dbtProject --> Sources
    dbtProject --> Tests
    Marts -->|"writes to"| DataMart["Data Mart (gold/)"]:::storage

    %% Serving & Inference Layer
    subgraph "Serving & Inference Layer" 
        direction TB
        ECS["AWS ECS + ALB"]:::serving
        Django["Django App"]:::serving
        Streamlit["Streamlit UI"]:::serving
        Athena["Athena/Redshift"]:::storage
        SageMaker["SageMaker Endpoint"]:::compute
    end

    DataMart -->|Queried by| Athena
    Athena --> Django
    SageMaker --> Django
    Longchain --> Django
    Whisper --> Longchain
    Django --> Streamlit
    ECS --> Django

    %% Infrastructure & CI/CD
    subgraph "Infrastructure & CI/CD" 
        direction TB
        TerraformAirflow["handlers-airflow/infra/"]:::infra
        Playbook["handlers-airflow/playbook.yml"]:::infra
        TerraformApp["application/infra/"]:::infra
        ProdTF["application/env/prod/"]:::infra
        CI["GitHub Actions"]:::infra
    end

    TerraformAirflow --> Airflow
    Playbook --> Airflow
    TerraformApp --> ECS
    ProdTF --> ECS
    CI --> Django

    %% Reference Data & Samples
    subgraph "Reference Data & Samples"
        direction TB
        RawData["data/raw/"]:::storage
        StagingData["data/staging/"]:::storage
        MartData["data/mart/"]:::storage
        Samples["samples/"]:::storage
    end

    RawData --> S3Raw
    StagingData --> DeltaLake
    MartData --> DataMart
    Samples --> Airflow

    %% Click Events
    click DAG "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/dags/youtube_aws_dag.py"
    click AbstractHook "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/abstract_classes/abstract_hook.py"
    click AbstractOp "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/abstract_classes/abstract_operator.py"
    click HookFactory "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/factories/hook_factory.py"
    click OpFactory "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/factories/operator_factory.py"
    click YTHook "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/hooks/youtube_hook.py"
    click AWSHook "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/hooks/aws_hook.py"
    click YTOp "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/operators/youtube_operator.py"
    click AWSOp "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/airflow-project/operators/aws_operator.py"
    click Processors "https://github.com/andreichiro/data_engineer_end2end/tree/main/handlers-airflow/airflow-project/processors/"
    click Compose "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/docker-compose.yaml"
    click TerraformAirflow "https://github.com/andreichiro/data_engineer_end2end/tree/main/handlers-airflow/infra/"
    click Playbook "https://github.com/andreichiro/data_engineer_end2end/blob/main/handlers-airflow/playbook.yml"
    click LambdaKinesis "https://github.com/andreichiro/data_engineer_end2end/blob/main/aws/lambda_kinesis.py"
    click LambdaS3 "https://github.com/andreichiro/data_engineer_end2end/blob/main/aws/lambda.py"
    click Databricks "https://github.com/andreichiro/data_engineer_end2end/blob/main/databricks/Databricks - Spark Transformations (staging layer) 2023-06-27 14_59_34.html"
    click dbtProject "https://github.com/andreichiro/data_engineer_end2end/blob/main/app_dbt/dbt_project.yml"
    click Intermediate "https://github.com/andreichiro/data_engineer_end2end/tree/main/app_dbt/models/intermediate/"
    click Marts "https://github.com/andreichiro/data_engineer_end2end/tree/main/app_dbt/models/marts/"
    click Macros "https://github.com/andreichiro/data_engineer_end2end/tree/main/app_dbt/macros/"
    click Sources "https://github.com/andreichiro/data_engineer_end2end/blob/main/app_dbt/models/intermediate/sources.yml"
    click Tests "https://github.com/andreichiro/data_engineer_end2end/tree/main/app_dbt/tests/"
    click Django "https://github.com/andreichiro/data_engineer_end2end/blob/main/application/application/settings.py"
    click Streamlit "https://github.com/andreichiro/data_engineer_end2end/tree/main/application/application/static/"
    click ProdTF "https://github.com/andreichiro/data_engineer_end2end/tree/main/application/env/prod/"
    click TerraformApp "https://github.com/andreichiro/data_engineer_end2end/tree/main/application/infra/"
    click RawData "https://github.com/andreichiro/data_engineer_end2end/tree/main/data/raw/"
    click StagingData "https://github.com/andreichiro/data_engineer_end2end/tree/main/data/staging/"
    click MartData "https://github.com/andreichiro/data_engineer_end2end/tree/main/data/mart/"

    %% Styles
    classDef external fill:#f9f,stroke:#333,stroke-width:1px
    classDef ingestion fill:#bbdefb,stroke:#1e88e5,stroke-width:2px
    classDef storage fill:#cfd8dc,stroke:#546e7a,stroke-width:1px
    classDef compute fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
    classDef modeling fill:#ffe0b2,stroke:#fb8c00,stroke-width:2px
    classDef serving fill:#e1bee7,stroke:#8e24aa,stroke-width:2px
    classDef infra fill:#e0e0e0,stroke:#757575,stroke-width:1px
```
