# jesuisroti/mpg_data_analytics architecture

> Architecture diagram of the GitHub repository jesuisroti/mpg_data_analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jesuisroti/mpg_data_analytics
- Repository: https://github.com/jesuisroti/mpg_data_analytics

## Overview

##### Project Type and Purpose
- This project is a data analysis tool combined with a full-stack web app. Its primary function is to extract, clean, and display football player data for a fantasy football game (MPG).
- It comprises both a standalone Python script and a web application (Flask back-end coupled with a Next.js front-end) for data visualization.
- An ETL pipeline is implemented using Python and Pandas (with the cleaning function in utils/general_functions.py), and background tasks (with Celery) keep auxiliary data (mercatoPR) updated.

##### Analyzing the File Structure
- The top-level directories and files give clear hints about the separation of concerns:
- The “flask_api” directory covers the back-end with app.py (the main Flask app initializing Celery) and routes.py (defining application routes). It also includes a subfolder “players_data” that handles database creation and extra data processing.
- The “front” directory hosts the Next.js application with its own configuration (e.g., next.config.js, package.json) and the UI components.
- The “utils” directory contains common functions (e.g., data cleaning and plotting).
- The standalone “mpg_main.py” script uses backend utilities (from utils/general_functions.py) to produce pyplot visualizations.
- .github and docs directories indicate continuous integration, deployment workflows, and documentation (including screenshots of plots and the Flask app), reinforcing the multi-faceted nature of the project.

##### Key Components
a. Data Extraction & Cleaning:
- MPG API (external data source) → Data Extraction module (likely implemented in the main script and later processed in Flask routes).
- ETL Pipeline: Extraction → Clean transformation (using Pandas functions in utils/general_functions.py) → Loading.

b. Visualization Options:
- Standalone Python Script: mpg_main.py that uses matplotlib/pyplot to display data.
- Flask Web App: A back-end providing endpoints (routes.py) that serve data for visualization.

c. Celery-based Background Processing:
- Within flask_api/app.py, Celery tasks are set to periodically extract additional player stats (mercatoPR) from external endpoints.
- Their results are stored in a SQLite database (set up via the script in flask_api/players_data/create_database.py).
- Dependency on a Redis server (as a Celery broker) which should be represented as an external service.

d. Front-End (Next.js Application):
- The “front” directory contains a Next.js based UI that communicates with the Flask API.
- It uses environmental configuration (NEXT_PUBLIC_API_URL) to connect to the backend.
- Components such as charts and layouts illustrate the mindful use of UI libraries (e.g., Tailwind CSS as suggested by tailwind.config.js).

##### Relationships and Data Flow
- The cleaned data then flows into two parallel paths: one for generating plots via the standalone script and another for serving via the Flask API.
- The Flask API directly interacts with:
- The Celery service (for background tasks) which periodically enrich data.
- The SQLite database where enriched data is stored.
- The Next.js front-end makes HTTP calls to the Flask API endpoints for dynamic, user-friendly data presentation.
- Use arrowed connectors to illustrate periodic task scheduling from Celery to the data extraction/transformation modules.

##### Architectural Patterns and Technologies
- Clearly denote separation between the front-end (Next.js, tailored with React components and Tailwind) and back-end (Flask for API routing, Celery for asynchronous tasks).
- ETL process: external API → Python/Pandas cleaning → data flows into visualization components.
- Mention the use of Docker/GitHub workflows if relevant (represented in .github/workflows/deploy_master.yml) though they may be abstracted out as deployment configuration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    MPG_API["External Data Source (MPG API)"]:::external
    ETL["ETL Pipeline (Data Extraction & Cleaning)"]:::etl
    Standalone["Standalone Python Plotting Script (mpg_main.py)"]:::etl

    MPG_API -->|"extracts"| ETL
    ETL -->|"feeds"| Standalone

    subgraph "Flask Web App"
        FlaskApp["Flask App Initialization"]:::backend
        APIRoutes["API Routes"]:::backend
    end

    ETL -->|"serves_cleaned_data"| FlaskApp
    FlaskApp -->|"defines_endpoints"| APIRoutes

    Celery["Celery Background Processing"]:::backend
    FlaskApp -->|"triggers"| Celery
    Celery -->|"updates"| SQLite["SQLite Database (Enriched Data)"]:::backend
    Celery -->|"uses"| Redis["Redis Broker"]:::external

    subgraph "Next.js Front-End"
        NextApp["Next.js Front-End Application"]:::frontend
        Chart["Chart Component"]:::frontend
        Layout["Layout Component"]:::frontend
    end

    NextApp -->|"HTTP_calls"| APIRoutes
    NextApp -->|"renders"| Chart
    NextApp -->|"includes"| Layout

    click ETL "https://github.com/jesuisroti/mpg_data_analytics/blob/master/utils/general_functions.py"
    click Standalone "https://github.com/jesuisroti/mpg_data_analytics/blob/master/mpg_main.py"
    click FlaskApp "https://github.com/jesuisroti/mpg_data_analytics/blob/master/flask_api/app.py"
    click APIRoutes "https://github.com/jesuisroti/mpg_data_analytics/blob/master/flask_api/routes.py"
    click Celery "https://github.com/jesuisroti/mpg_data_analytics/blob/master/flask_api/app.py"
    click SQLite "https://github.com/jesuisroti/mpg_data_analytics/blob/master/flask_api/players_data/create_database.py"
    click NextApp "https://github.com/jesuisroti/mpg_data_analytics/tree/master/front/"
    click Chart "https://github.com/jesuisroti/mpg_data_analytics/blob/master/front/app/components/chart.js"
    click Layout "https://github.com/jesuisroti/mpg_data_analytics/blob/master/front/app/layout.js"

    classDef external fill:#aaffaa,stroke:#333,stroke-width:2px,font-family:monospace;
    classDef etl fill:#ffe4b5,stroke:#333,stroke-width:2px,font-family:monospace;
    classDef backend fill:#ffcccb,stroke:#333,stroke-width:2px,font-family:monospace;
    classDef frontend fill:#add8e6,stroke:#333,stroke-width:2px,font-family:monospace;
```
