How jesuisroti/mpg_data_analytics works
Project Type and Purpose
• This project is a data analysis tool combined with a full-stack web app. Its primary function is to extract, clean, and display football player data for a fantasy football game (MPG).
• It comprises both a standalone Python script and a web application (Flask back-end coupled with a Next.js front-end) for data visualization.
• An ETL pipeline is implemented using Python and Pandas (with the cleaning function in utils/general_functions.py), and background tasks (with Celery) keep auxiliary data (mercatoPR) updated.
Analyzing the File Structure
• The top-level directories and files give clear hints about the separation of concerns:
• The “flask_api” directory covers the back-end with app.py (the main Flask app initializing Celery) and routes.py (defining application routes). It also includes a subfolder “players_data” that handles database creation and extra data processing.
• The “front” directory hosts the Next.js application with its own configuration (e.g., next.config.js, package.json) and the UI components.
• The “utils” directory contains common functions (e.g., data cleaning and plotting).
• The standalone “mpg_main.py” script uses backend utilities (from utils/general_functions.py) to produce pyplot visualizations.
• .github and docs directories indicate continuous integration, deployment workflows, and documentation (including screenshots of plots and the Flask app), reinforcing the multi-faceted nature of the project.
Key Components
a. Data Extraction & Cleaning:
• MPG API (external data source) → Data Extraction module (likely implemented in the main script and later processed in Flask routes).
• ETL Pipeline: Extraction → Clean transformation (using Pandas functions in utils/general_functions.py) → Loading.
b. Visualization Options:
• Standalone Python Script: mpg_main.py that uses matplotlib/pyplot to display data.
• Flask Web App: A back-end providing endpoints (routes.py) that serve data for visualization.
c. Celery-based Background Processing:
• Within flask_api/app.py, Celery tasks are set to periodically extract additional player stats (mercatoPR) from external endpoints.
• Their results are stored in a SQLite database (set up via the script in flask_api/players_data/create_database.py).
• Dependency on a Redis server (as a Celery broker) which should be represented as an external service.
d. Front-End (Next.js Application):
• The “front” directory contains a Next.js based UI that communicates with the Flask API.
• It uses environmental configuration (NEXT_PUBLIC_API_URL) to connect to the backend.
• Components such as charts and layouts illustrate the mindful use of UI libraries (e.g., Tailwind CSS as suggested by tailwind.config.js).
Relationships and Data Flow
• The cleaned data then flows into two parallel paths: one for generating plots via the standalone script and another for serving via the Flask API.
• The Flask API directly interacts with:
• The Celery service (for background tasks) which periodically enrich data.
• The SQLite database where enriched data is stored.
• The Next.js front-end makes HTTP calls to the Flask API endpoints for dynamic, user-friendly data presentation.
• Use arrowed connectors to illustrate periodic task scheduling from Celery to the data extraction/transformation modules.
Architectural Patterns and Technologies
• Clearly denote separation between the front-end (Next.js, tailored with React components and Tailwind) and back-end (Flask for API routing, Celery for asynchronous tasks).
• ETL process: external API → Python/Pandas cleaning → data flows into visualization components.
• Mention the use of Docker/GitHub workflows if relevant (represented in .github/workflows/deploy_master.yml) though they may be abstracted out as deployment configuration.