# ioaviator/de_hackathon_usa_college_info architecture

> Architecture diagram of the GitHub repository ioaviator/de_hackathon_usa_college_info, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ioaviator/de_hackathon_usa_college_info
- Repository: https://github.com/ioaviator/de_hackathon_usa_college_info

## Overview

##### Main Components
- Data Ingestion: A module that extracts data from the external USA College Scorecard API. (This is supported by parts of the ELT scripts in include/ELT/extract.py.)
- Data Transformation & ETL Pipeline: Represent the transformation and processing steps – including authentication, configuration, extraction, transformation, and loading. These are detailed in the include/ELT folder (modules like transform.py, load.py, and database utilities).
- Storage Layers:
- Raw Data Store: Azure Blob Storage is used to store raw/parquet files.
- Processed Data Store: Azure PostgreSQL holds the structured and analytic-ready data.
- Orchestration & Scheduling: Apache Airflow is used for orchestration (with DAGs defined in the dags directory plus the include/dag_context logic).
- Containerization and CI/CD:
- Docker: Dockerfile indicates containerization, with the project built into a container image.
- Azure Container Registry: Managed via Terraform configuration under terraform/container_registry.
- GitHub Actions: Automated testing, linting, Terraform deployment, and image build workflows (files under .github/workflows).
- Infrastructure as Code: Terraform scripts (located in terraform/) provisioning multiple Azure resources including the storage backend, PostgreSQL database, and container registry.
- Dev Environment and Deployment Tools:
- Astro CLI for Apache Airflow management.
- Python virtual environment setup and package management (requirements.txt, packages.txt).

##### Relationships and Data Flow
- The pipeline is triggered either manually (via main.py) or through Airflow orchestrated DAGs (from the dags directory).
- Data is extracted from the API, transformed by ETL modules and then:
- Stored in the raw data layer on Azure Blob Storage (for archival and later processing).
- Loaded into Azure PostgreSQL for analytics (visualized in Power BI as noted in the README).
- Docker and Terraform allow packaging of the application and automated infrastructure deployments.
- GitHub Actions continuously integrate code changes, perform linting/tests, build containers, and deploy using Terraform to Azure.

##### Architectural Patterns & Design Principles
- Separation of Concerns: The ETL process is split among several clear modules (auth, extract, transform, load, and database utilities).
- Infrastructure as Code: All cloud resource provisioning is automated via Terraform.
- Containerization & CI/CD: The use of Docker combined with GitHub Actions simplifies deployment and ensures consistency.
- Orchestration: Apache Airflow (managed with Astro CLI) orchestrates the workflow, allowing modularity and scheduling.
- Cloud-Native Elements: Integration with Azure services (Blob Storage, PostgreSQL, Container Registry) is central to the design.

##### Technologies and Frameworks to Label
- Python and its libraries for ETL processing.
- Apache Airflow for orchestration.
- Terraform for infra provisioning.
- Docker for containerization.
- Azure services: Blob Storage, PostgreSQL, Container Registry.
- GitHub Actions for continuous integration and deployment.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Orchestration (Airflow)
    subgraph "Orchestration (Airflow)"
        A1["Airflow DAGs"]:::orchestration
        A2["Airflow Context"]:::orchestration
    end

    %% Dev Environment
    subgraph "Dev Environment"
        D1["Requirements"]:::dev
        D2["Packages"]:::dev
        D3["Python Environment"]:::dev
        D1 --> D3
        D2 --> D3
    end

    %% ETL Process
    subgraph "ETL Process"
        EI["API Ingestion"]:::etl
        EX["Extract Module"]:::etl
        TR["Transform Module"]:::etl
        LD["Load Module"]:::etl
        AU["Auth Module"]:::etl
        CF["Config Module"]:::etl
        DB["Database Utils"]:::etl
        EI --> EX
        EX --> TR
        TR --> LD
        AU --> EX
        CF --> EX
        DB --> EX
    end

    %% Data Storage (Azure)
    subgraph "Data Storage (Azure)"
        RS["Raw Data Storage (Blob)"]:::storage
        PS["Processed Data Storage (PostgreSQL)"]:::storage
    end

    %% CI/CD & Containerization
    subgraph "CI/CD and Containerization"
        GH["GitHub Actions"]:::cicd
        DC["Docker Container"]:::container
    end

    %% Infrastructure (Terraform)
    subgraph "Infrastructure (Terraform)"
        TF["Terraform Scripts"]:::infra
        TFCR["Terraform Container Registry"]:::infra
        TFP["Terraform Postgres"]:::infra
        ACR["Azure Container Registry"]:::infra
    end

    %% Data Flow Connections
    A1 -->|"triggers"| EX
    A2 -->|"provides_context"| EX
    D3 -->|"supports"| EX
    EX -->|"saves_raw"| RS
    LD -->|"loads_data"| PS
    GH -->|"builds_image"| DC
    DC -->|"deployed_via"| TF
    TF -->|"provisions"| RS
    TF -->|"provisions"| PS
    TF -->|"provisions"| ACR
    TF --> TFCR
    TF --> TFP

    %% Click Events
    click EX "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/include/ELT/extract.py"
    click TR "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/include/ELT/transform.py"
    click LD "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/include/ELT/load.py"
    click AU "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/include/ELT/auth.py"
    click CF "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/include/ELT/config.py"
    click DB "https://github.com/ioaviator/de_hackathon_usa_college_info/tree/main/include/ELT/database"
    click A1 "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/dags/top_1000_usa_schools.py"
    click A2 "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/include/dag_context/main.py"
    click DC "https://github.com/ioaviator/de_hackathon_usa_college_info/tree/main/Dockerfile"
    click GH "https://github.com/ioaviator/de_hackathon_usa_college_info/tree/main/.github/workflows"
    click TF "https://github.com/ioaviator/de_hackathon_usa_college_info/tree/main/terraform/"
    click TFCR "https://github.com/ioaviator/de_hackathon_usa_college_info/tree/main/terraform/container_registry"
    click TFP "https://github.com/ioaviator/de_hackathon_usa_college_info/tree/main/terraform/postgresdb"
    click D1 "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/requirements.txt"
    click D2 "https://github.com/ioaviator/de_hackathon_usa_college_info/blob/main/packages.txt"

    %% Styles
    classDef etl fill:#ffcccc,stroke:#333,stroke-width:2px;
    classDef storage fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#ccccff,stroke:#333,stroke-width:2px;
    classDef container fill:#ffcc99,stroke:#333,stroke-width:2px;
    classDef cicd fill:#ffff99,stroke:#333,stroke-width:2px;
    classDef infra fill:#ccffff,stroke:#333,stroke-width:2px;
    classDef dev fill:#ff99ff,stroke:#333,stroke-width:2px;
```
