# damipop7/finalbigdata architecture

> Architecture diagram of the GitHub repository damipop7/finalbigdata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/damipop7/finalbigdata
- Repository: https://github.com/damipop7/finalbigdata

## Overview

##### Project Type and Purpose
- This is an end‐to‐end soccer analytics data pipeline written in Python, orchestrated with Apache Airflow.
- It pulls raw data from external sources (Kaggle, web scraping via FBRef), performs ETL (ingestion, cleaning, feature engineering), trains ML models, evaluates them, and produces visual reports/dashboards.

##### Main Components
a. External Data Sources
- Kaggle API
- FBRef (web scraping)
b. Orchestration Layer
- Airflow Scheduler & Web UI
- DAG definition (`soccer_pipeline_dag.py`)
c. Data Storage
- Raw data folder (`data/raw/`)
- Processed data folder (`data/processed/`)
- Model artifacts (`models/trained/`)
d. Compute/Processing Modules
- Data Ingestion (`src/data/data_ingestion.py`)
- Data Cleaning & Processing (`src/data/data_cleaning.py`, `feature_engineering.py`)
- Model Training (`src/models/model_training.py`)
- Model Evaluation (`src/models/model_evaluation.py`)
e. Configuration & Deployment
- Pipeline configuration (`config/pipeline_config.yaml`)
- Environment setup (`.env`, `requirements.txt`, `setup.py`)
- Entry point CLI (`run_pipeline.py`)

##### Relationships and Data Flow
- Airflow Scheduler triggers the DAG, which executes tasks in sequence.
- Task “Data Ingestion” pulls from Kaggle API & FBRef → writes CSV/Parquet into `data/raw/`.
- Task “Data Cleaning/Feature Engineering” reads `data/raw/` → produces cleaned datasets in `data/processed/`.
- Task “Model Training” reads processed features → serializes model files in `models/trained/`.
- Task “Model Evaluation” takes trained models + test data → outputs evaluation metrics.
- Task “Visualization” reads processed data + evaluation results → writes charts/dashboards to `reports/` or serves via web.

##### Architectural Patterns & Design Principles
- Layered ETL architecture: Ingestion → Processing → Modeling → Visualization.
- DAG-based orchestration for clear dependencies and retries.
- Separation of concerns: each Python module handles a single responsibility.
- Config‐driven (YAML) to allow parameter changes without code edits.

##### Putting It All Together
- To the right, show the Reports/Visualization output.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources
    subgraph "External Sources"
        direction TB
        Kaggle["Kaggle API"]:::external
        FBRef["FBRef Scraper"]:::external
    end

    %% Storage Layers
    Raw["Raw Data (data/raw/)"]:::storage
    Processed["Processed Data (data/processed/)"]:::storage
    ModelArtifacts["Model Artifacts (models/trained/)"]:::storage
    EvaluationResults["Evaluation Metrics"]:::storage
    Reports["Reports/Dashboards (reports/)"]:::storage

    %% Orchestration Layer
    CLI["Entry Point CLI (run_pipeline.py)"]:::orchestration
    subgraph "Airflow Orchestration"
        direction TB
        Scheduler["Scheduler & Web UI"]:::orchestration
        Scheduler --> Ingestion["Data Ingestion Task"]:::compute
        Ingestion --> Cleaning["Data Cleaning & Feature Engineering Task"]:::compute
        Cleaning --> Training["Model Training Task"]:::compute
        Training --> Evaluation["Model Evaluation Task"]:::compute
        Evaluation --> Visualization["Visualization Task"]:::compute
    end
    CLI --> Scheduler

    %% Data Flow
    Kaggle --> Raw
    FBRef --> Raw
    Raw --> Cleaning
    Cleaning --> Processed
    Training --> ModelArtifacts
    Evaluation --> EvaluationResults
    Visualization --> Reports

    %% Configuration & Environment
    subgraph "Configuration & Environment" 
        direction TB
        Config["Pipeline Config (config/pipeline_config.yaml)"]:::config
        EnvDot[".env"]:::config
        EnvReq["requirements.txt"]:::config
        EnvSetup["setup_project.py"]:::config
    end
    Config -.-> Scheduler
    EnvDot -.-> Scheduler
    EnvReq -.-> Scheduler
    EnvSetup -.-> Scheduler

    %% Package Init Modules
    subgraph "Package Init Modules"
        direction TB
        PI1["src/__init__.py"]:::config
        PI2["src/data/__init__.py"]:::config
        PI3["src/models/__init__.py"]:::config
        PI4["src/visualization/__init__.py"]:::config
    end
    PI1 -.-> Scheduler

    %% Click Events
    click Ingestion "https://github.com/damipop7/finalbigdata/blob/main/src/data/data_ingestion.py"
    click CLI "https://github.com/damipop7/finalbigdata/blob/main/run_pipeline.py"
    click Config "https://github.com/damipop7/finalbigdata/blob/main/config/pipeline_config.yaml"
    click EnvDot "https://github.com/damipop7/finalbigdata/blob/main/.env"
    click EnvReq "https://github.com/damipop7/finalbigdata/blob/main/requirements.txt"
    click EnvSetup "https://github.com/damipop7/finalbigdata/blob/main/setup_project.py"
    click PI1 "https://github.com/damipop7/finalbigdata/blob/main/src/__init__.py"
    click PI2 "https://github.com/damipop7/finalbigdata/blob/main/src/data/__init__.py"
    click PI3 "https://github.com/damipop7/finalbigdata/blob/main/src/models/__init__.py"
    click PI4 "https://github.com/damipop7/finalbigdata/blob/main/src/visualization/__init__.py"

    %% Styles
    classDef compute fill:#add8e6,stroke:#000
    classDef storage fill:#90ee90,stroke:#000,stroke-width:1px
    classDef external fill:#ffcc99,stroke:#000
    classDef orchestration fill:#d3d3d3,stroke:#000
    classDef config fill:transparent,stroke:#666,stroke-dasharray: 5 5
    classDef orchestration fill:#ffe4b5,stroke:#000
    class Legend fill:#fff,stroke:#000,stroke-width:1px

    %% Legend
    subgraph Legend["Legend"]
        direction TB
        A["Blue: Compute Tasks"]:::compute
        B["Green: Storage Layers"]:::storage
        C["Orange: External Sources"]:::external
        D["Gray Dashed: Config & Env"]:::config
        E["Light Orange: Orchestration"]:::orchestration
    end
```
