# ayushd64/pandas_dpa architecture

> Architecture diagram of the GitHub repository ayushd64/pandas_dpa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayushd64/pandas_dpa
- Repository: https://github.com/ayushd64/pandas_dpa

## Overview

1. **Repository purpose**
   - This is a **Python/Pandas-based offline data pipeline** for soccer/FIFA player analysis.
   - It appears to ingest yearly FIFA CSV datasets, clean and transform them, train a model, run optimization/analysis, and generate plots and derived output folders.

2. **Primary inputs**
   - The repository includes yearly source files: `2009.csv` through `2019.csv`, plus `raw_data.csv`.
   - The README says `raw_dats.csv` is the 2020 FIFA dataset, so the project likely spans multiple seasons/years of player data.
   - These CSVs are the main data boundary into the system; there is no sign of a database or external API dependency at runtime.

3. **Data acquisition layer**
   - `Web_Scrapping.py` suggests a scraper that likely gathers FIFA/player data from an external website and produces raw tabular data.
   - This is the first stage in the pipeline and likely feeds the CSV inputs used by the rest of the system.
   - If the scraper is used, it is a separate acquisition boundary from the main analysis pipeline.

4. **Cleaning and normalization**
   - `Data_Clean.py` is the cleanup stage, likely handling missing values, inconsistent formats, and column normalization across yearly files.
   - This stage probably converts the scraped/raw data into a consistent schema suitable for analysis and modeling.
   - The presence of many year-specific CSVs implies schema harmonization is important here.

5. **Feature processing / transformation**
   - `Data_Process.py` likely performs feature engineering and prepares the cleaned dataset for modeling.
   - This is where the pipeline probably derives analysis-ready fields from raw player attributes and performance indicators.
   - It acts as the bridge between cleaning and ML training.

6. **Train/test partitioning**
   - `Train_Test_Split.py` isolates the modeling dataset into training and evaluation subsets.
   - This is a distinct boundary before model fitting, which suggests the project treats evaluation as a separate step rather than doing it inline.

7. **Modeling subsystem**
   - `ML.py` is the main machine-learning component.
   - It likely trains one or more predictive models using the processed training data and produces prediction outputs.
   - The folders `before_prediction` and `after_prediction` suggest the pipeline stores intermediate datasets or comparison artifacts around the model inference step.

8. **Optimization subsystem**
   - `Optimization.py` likely tunes model parameters or computes an optimized assignment/ranking based on predictions or player metrics.
   - This suggests the project is not just descriptive analytics; it includes a decision/optimization layer on top of the model outputs.

9. **Analysis and visualization**
   - `Data_Analyze.py` and `Plots.py` are the reporting/interpretation layer.
   - `Plots.py` likely generates charts, while `Data_Analyze.py` computes summary statistics or comparative insights.
   - The `player_performance` directory probably contains analysis outputs specific to player-level results.

10. **Pipeline orchestration**
   - `Master.py` is the top-level orchestrator.
   - The README instructs running it from Jupyter with `%run Master.py`, which implies the whole workflow is executed as a scripted batch pipeline rather than through a service or UI.
   - This script likely sequences the steps: load data → clean → process → split → train → optimize → analyze/plot.

11. **Output/artifact boundaries**
   - `before_prediction`, `after_prediction`, and `player_performance` look like generated artifact directories rather than source code.
   - These directories likely hold intermediate and final outputs for model comparison, prediction results, and performance summaries.
   - This is useful for diagramming because it separates transient processing stages from persisted results.

12. **Tooling/runtime**
   - The repo is Python 3–based and depends on packages listed in `requirements.txt`.
   - Execution is intended from a Jupyter notebook environment, but the actual work is in reusable `.py` modules.
   - The included PDF report indicates the code supports an offline analysis/reporting workflow, not a production API or web app.

13. **Overall architecture**
   - The system is a **linear data pipeline** with clear phases: acquisition → cleaning → transformation → split → model/optimization → analysis/plots.
   - `Master.py` coordinates the flow across these modules, and the CSV files plus output directories form the main data stores.
   - For an architecture diagram, the most important distinction is between the **raw yearly data inputs**, the **processing/modeling stages**, and the **generated analysis artifacts**.

Main entry points: [`Master.py`](https://github.com/ayushd64/pandas_dpa/blob/master/Master.py) (Master), [`ML.py`](https://github.com/ayushd64/pandas_dpa/blob/master/ML.py) (ML), [`Data_Analyze.py`](https://github.com/ayushd64/pandas_dpa/blob/master/Data_Analyze.py) (Analyze), [`Data_Clean.py`](https://github.com/ayushd64/pandas_dpa/blob/master/Data_Clean.py) (Clean Data)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs"]
  node_yearly_csvs[("Yearly CSVs<br/>dataset")]
  node_raw_data[("Raw Data<br/>dataset<br/>[raw_data.csv]")]
end

subgraph group_pipeline["Pipeline"]
  node_master["Master<br/>orchestrator<br/>[Master.py]"]
  node_web_scraping["Web Scrape<br/>acquisition<br/>[Web_Scrapping.py]"]
  node_data_clean["Clean Data<br/>cleaning<br/>[Data_Clean.py]"]
  node_data_process["Process Data<br/>feature-engineering<br/>[Data_Process.py]"]
  node_train_test_split["Split Data<br/>partitioning"]
  node_ml["ML<br/>model-training<br/>[ML.py]"]
  node_optimization["Optimize<br/>optimization<br/>[Optimization.py]"]
  node_data_analyze["Analyze<br/>analysis<br/>[Data_Analyze.py]"]
  node_plots["Plots<br/>visualization<br/>[Plots.py]"]
end

subgraph group_artifacts["Artifacts"]
  node_before_prediction["Before Predict<br/>artifact-store"]
  node_after_prediction["After Predict<br/>artifact-store"]
  node_player_performance["Player Perf<br/>report-store"]
end

node_web_scraping -->|"produces"| node_raw_data
node_yearly_csvs -->|"inputs"| node_data_clean
node_raw_data -->|"inputs"| node_data_clean
node_data_clean -->|"prepared data"| node_data_process
node_data_process -->|"features"| node_train_test_split
node_train_test_split -->|"train/eval"| node_ml
node_ml -->|"stores"| node_before_prediction
node_ml -->|"stores"| node_after_prediction
node_ml -->|"predictions"| node_optimization
node_optimization -->|"results"| node_data_analyze
node_data_analyze -->|"summaries"| node_plots
node_plots -->|"publishes"| node_player_performance
node_data_analyze -->|"publishes"| node_player_performance
node_master -.->|"runs"| node_web_scraping
node_master -.->|"runs"| node_data_clean
node_master -.->|"runs"| node_data_process
node_master -.->|"runs"| node_train_test_split
node_master -.->|"runs"| node_ml
node_master -.->|"runs"| node_optimization
node_master -.->|"runs"| node_data_analyze
node_master -.->|"runs"| node_plots

click node_master "https://github.com/ayushd64/pandas_dpa/blob/master/Master.py"
click node_web_scraping "https://github.com/ayushd64/pandas_dpa/blob/master/Web_Scrapping.py"
click node_data_clean "https://github.com/ayushd64/pandas_dpa/blob/master/Data_Clean.py"
click node_data_process "https://github.com/ayushd64/pandas_dpa/blob/master/Data_Process.py"
click node_train_test_split "https://github.com/ayushd64/pandas_dpa/blob/master/Train_Test_Split.py"
click node_ml "https://github.com/ayushd64/pandas_dpa/blob/master/ML.py"
click node_optimization "https://github.com/ayushd64/pandas_dpa/blob/master/Optimization.py"
click node_data_analyze "https://github.com/ayushd64/pandas_dpa/blob/master/Data_Analyze.py"
click node_plots "https://github.com/ayushd64/pandas_dpa/blob/master/Plots.py"
click node_raw_data "https://github.com/ayushd64/pandas_dpa/blob/master/raw_data.csv"
click node_before_prediction "https://github.com/ayushd64/pandas_dpa/tree/master/before_prediction"
click node_after_prediction "https://github.com/ayushd64/pandas_dpa/tree/master/after_prediction"
click node_player_performance "https://github.com/ayushd64/pandas_dpa/tree/master/player_performance"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_yearly_csvs,node_raw_data toneBlue
class node_master,node_web_scraping,node_data_clean,node_data_process,node_train_test_split,node_ml,node_optimization,node_data_analyze,node_plots toneAmber
class node_before_prediction,node_after_prediction,node_player_performance toneMint
```

## Components

### Inputs

Raw and yearly FIFA CSV data

- **Yearly CSVs**: dataset · FIFA season datasets from 2009 through 2019
- [Raw Data](https://github.com/ayushd64/pandas_dpa/blob/master/raw_data.csv): dataset · Combined raw tabular dataset · `raw_data.csv`

### Pipeline

Cleaning, modeling, optimization, and reporting stages

- [Master](https://github.com/ayushd64/pandas_dpa/blob/master/Master.py): orchestrator · Top-level batch workflow runner · `Master.py`
- [Web Scrape](https://github.com/ayushd64/pandas_dpa/blob/master/Web_Scrapping.py): acquisition · Collects raw player data from the source site · `Web_Scrapping.py`
- [Clean Data](https://github.com/ayushd64/pandas_dpa/blob/master/Data_Clean.py): cleaning · Normalizes and prepares raw CSV data · `Data_Clean.py`
- [Process Data](https://github.com/ayushd64/pandas_dpa/blob/master/Data_Process.py): feature-engineering · Transforms cleaned data into model-ready features · `Data_Process.py`
- [Split Data](https://github.com/ayushd64/pandas_dpa/blob/master/Train_Test_Split.py): partitioning · Separates training and evaluation sets · `Train_Test_Split.py`
- [ML](https://github.com/ayushd64/pandas_dpa/blob/master/ML.py): model-training · Trains predictive models and generates predictions · `ML.py`
- [Optimize](https://github.com/ayushd64/pandas_dpa/blob/master/Optimization.py): optimization · Tunes or ranks results using model outputs · `Optimization.py`
- [Analyze](https://github.com/ayushd64/pandas_dpa/blob/master/Data_Analyze.py): analysis · Computes summaries and comparative insights · `Data_Analyze.py`
- [Plots](https://github.com/ayushd64/pandas_dpa/blob/master/Plots.py): visualization · Builds charts and visual reports · `Plots.py`

### Artifacts

Generated prediction and analysis outputs

- [Before Predict](https://github.com/ayushd64/pandas_dpa/tree/master/before_prediction): artifact-store · Intermediate inputs before prediction · `before_prediction`
- [After Predict](https://github.com/ayushd64/pandas_dpa/tree/master/after_prediction): artifact-store · Outputs after prediction · `after_prediction`
- [Player Perf](https://github.com/ayushd64/pandas_dpa/tree/master/player_performance): report-store · Player-level analysis and performance artifacts · `player_performance`

## Connections

- Web Scrape → Raw Data: produces · Scraped raw data is written to the raw dataset
- Yearly CSVs → Clean Data: inputs · Season CSVs are cleaned and harmonized
- Raw Data → Clean Data: inputs · Raw combined data enters cleaning
- Clean Data → Process Data: prepared data · Cleaned records are transformed into features
- Process Data → Split Data: features · Model-ready dataset is partitioned
- Split Data → ML: train/eval · Split datasets feed model training
- ML → Before Predict: stores · Pre-prediction artifacts are captured
- ML → After Predict: stores · Prediction outputs are captured
- ML → Optimize: predictions · Model outputs feed the optimization layer
- Optimize → Analyze: results · Optimized results are analyzed
- Analyze → Plots: summaries · Analysis results are visualized
- Plots → Player Perf: publishes · Charts and reports are stored for player performance
- Analyze → Player Perf: publishes · Analysis artifacts are stored alongside visuals
- Master → Web Scrape: runs · Master orchestrates the acquisition step
- Master → Clean Data: runs · Master orchestrates cleaning
- Master → Process Data: runs · Master orchestrates feature processing
- Master → Split Data: runs · Master orchestrates splitting
- Master → ML: runs · Master orchestrates model training
- Master → Optimize: runs · Master orchestrates optimization
- Master → Analyze: runs · Master orchestrates analysis
- Master → Plots: runs · Master orchestrates plotting