# loinguyen3108/sportify-music-analysis architecture

> Architecture diagram of the GitHub repository loinguyen3108/sportify-music-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/loinguyen3108/sportify-music-analysis
- Repository: https://github.com/loinguyen3108/sportify-music-analysis

## Overview

##### Main Components Identification
- Data Ingestion:
- Spotify Crawler: Located under src/crawler, this component interacts with the Spotify Web API to retrieve artist, album, playlist, and track data.
- Kafka: Under both containers/kafka and src/infratructure/kafka, it serves as the message streaming backbone.
- Data Storage:
- PostgreSQL: Scripts in containers/postgres indicate its use as a staging area.
- Clickhouse: Containers/clickhouse represents an analytical database for real‑time metrics.
- Data Lake / GCS: Mentioned in the README for storing raw and intermediate data.
- BigQuery: For star schema analytics as noted in the README.
- Data Processing:
- Apache Spark: Containers/spark and src/infratructure/spark map out usage of Spark for distributed ETL and transformations.
- Orchestration:
- Airflow: Under containers/airflow, responsible for scheduling and managing pipeline workflows.
- Auxiliary Components:
- CLI Utilities: Found in src/cli for tasks such as authentication, ingestion, consumption, and monitoring.
- Pipelines: src/pipelines holds ETL pipeline scripts, queries, and transformations.
- Terraform: Infrastructure code under terraform for provisioning cloud resources.
- Migration scripts: Directories under migration show SQL scripts for schema changes.

##### Component Relationships & Interactions
- Data Flow:
- Data is first acquired through the Spotify crawler.
- Freshly fetched data is published to Kafka, then staged into PostgreSQL.
- Data is simultaneously stored into a data lake (GCS/MinIO) for further processing.
- Apache Spark processes and transforms the data, loading it into analytical stores (BigQuery and Clickhouse).
- Airflow orchestrates and triggers these pipeline stages, ensuring dependencies are maintained.
- Communication Patterns:
- Messages flow from producers (crawlers or ingestion processes) to Kafka.
- Consumers (Spark jobs or ETL scripts in pipelines) retrieve data from Kafka.
- SQL migrations update the underlying schema in PostgreSQL or Clickhouse.
- Deployment Architecture:
- The entire system is containerized using Docker & managed with Docker Compose.
- Terraform defines infrastructure resources externally for scaling in a cloud environment.

##### Architectural Patterns & Key Design Principles
- Modularity: Each service (crawling, processing, storage, orchestration) is clearly defined in its own container or module.
- Scalability: Leveraging Kafka and Spark supports both real‑time and batch processing.
- Extensibility: The use of CLI modules and pipelines suggests that future expansion or modification of ingestion and processing modules is considered.
- Infrastructure as Code: Terraform scripts proclaim a commitment to scalable and reproducible cloud resource management.

##### Technologies & Frameworks to Highlight
- Docker & Docker Compose for containerization.
- Apache Kafka for streaming pipelines.
- Apache Airflow for orchestrating ETL workflows.
- Apache Spark for distributed processing.
- PostgreSQL for initial staging and Clickhouse for real‑time analytics.
- BigQuery for cloud‑based dimensional modeling (star schema) and analytics.
- Terraform for managing provisioning and infrastructure.
- Python with various utility modules in src for ETL and CLI interactions.

a. An outline that categorizes the project into major layers:
- Ingestion (Spotify API, Crawler, Kafka)
- Storage (PostgreSQL, Data Lake, Clickhouse, BigQuery)
- Processing (Spark ETL, Data Pipelines)
- Orchestration & Management (Airflow, Terraform)
b. Sketch how each component connects:
- Flow from the Spotify Web API, processed by the crawler (src/crawler) which sends messages into Kafka.
- Kafka feeding data into both the immediate landing zone (PostgreSQL) and the Spark ETL pipeline.
- Transformation in Spark and then the storage in BigQuery (for batch analytics) and Clickhouse (for real‑time queries).
- Incorporate the orchestration by showing Airflow’s control over these processes.
c. Underlying infrastructure:
- Docker containers (as seen in the containers/ directory) representing each service.
- Terraform-managed resources to achieve cloud scalability.
d. Labels and visual markers:
- Deployment-time aspects (e.g., Docker Compose and configuration flow).

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        SPA["Spotify API"]:::ingestion
        SC["Spotify Crawler"]:::ingestion
        Kafka["Kafka Broker"]:::ingestion
    end

    %% Data Storage Layer
    subgraph "Data Storage"
        PST["PostgreSQL (Staging)"]:::storage
        GCS["Data Lake (GCS)"]:::storage
        CH["Clickhouse (Real‑Time Analytics)"]:::storage
        BQ["BigQuery (Batch Analytics)"]:::storage
    end

    %% Data Processing Layer
    subgraph "Data Processing"
        Spark["Apache Spark Processing"]:::processing
    end

    %% Orchestration & Management Layer
    subgraph "Orchestration & Management"
        Airflow["Airflow Orchestrator"]:::orchestration
        Terraform["Terraform (IaC)"]:::orchestration
    end

    %% Auxiliary Tools Layer
    subgraph "Auxiliary Tools"
        CLI["CLI Utilities"]:::auxiliary
        Pipelines["ETL Pipelines"]:::auxiliary
        Migration["Migration Scripts"]:::auxiliary
    end

    %% Data Flow Connections
    SPA -->|"calls"| SC
    SC -->|"publishes"| Kafka
    Kafka -->|"feeds"| PST
    Kafka -->|"feeds"| GCS
    Kafka -->|"triggers"| Spark
    Spark -->|"loads"| CH
    Spark -->|"loads"| BQ

    %% Orchestration Connections
    Airflow -->|"orchestrates"| SC
    Airflow -->|"orchestrates"| Kafka
    Airflow -->|"orchestrates"| Spark
    Terraform -->|"provisions"| Airflow
    Terraform -->|"provisions"| Kafka
    Terraform -->|"provisions"| Spark

    %% Auxiliary Connections
    CLI -->|"manages"| Pipelines
    Pipelines -->|"executes"| Spark
    Migration -->|"updatesSchema"| PST
    Migration -->|"updatesSchema"| CH

    %% Click Events
    click SC "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/src/crawler"
    click Kafka "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/containers/kafka"
    click Kafka "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/src/infratructure/kafka"
    click PST "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/containers/postgres"
    click CH "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/containers/clickhouse"
    click GCS "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/src/helpers/gcs"
    click BQ "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/migration/bigquery"
    click Spark "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/containers/spark"
    click Spark "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/src/infratructure/spark"
    click Airflow "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/containers/airflow"
    click CLI "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/src/cli"
    click Pipelines "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/src/pipelines"
    click Terraform "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/terraform"
    click Migration "https://github.com/loinguyen3108/sportify-music-analysis/tree/main/migration"

    %% Class Definitions
    classDef ingestion fill:#d1e7dd,stroke:#0f5132,stroke-width:2px;
    classDef processing fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef storage fill:#cfe2ff,stroke:#084298,stroke-width:2px;
    classDef orchestration fill:#f8d7da,stroke:#842029,stroke-width:2px;
    classDef auxiliary fill:#e2e3e5,stroke:#41464b,stroke-width:2px;
```
