How damipop7/finalbigdata works
Project Type and Purpose
• This is an end‐to‐end soccer analytics data pipeline written in Python, orchestrated with Apache Airflow.
• It pulls raw data from external sources (Kaggle, web scraping via FBRef), performs ETL (ingestion, cleaning, feature engineering), trains ML models, evaluates them, and produces visual reports/dashboards.
Main Components
a. External Data Sources
• Kaggle API
• FBRef (web scraping)
b. Orchestration Layer
• Airflow Scheduler & Web UI
• DAG definition (soccer_pipeline_dag.py)
c. Data Storage
• Raw data folder (data/raw/)
• Processed data folder (data/processed/)
• Model artifacts (models/trained/)
d. Compute/Processing Modules
• Data Ingestion (src/data/data_ingestion.py)
• Data Cleaning & Processing (src/data/data_cleaning.py, feature_engineering.py)
• Model Training (src/models/model_training.py)
• Model Evaluation (src/models/model_evaluation.py)
e. Configuration & Deployment
• Pipeline configuration (config/pipeline_config.yaml)
• Environment setup (.env, requirements.txt, setup.py)
• Entry point CLI (run_pipeline.py)
Relationships and Data Flow
• Airflow Scheduler triggers the DAG, which executes tasks in sequence.
• Task “Data Ingestion” pulls from Kaggle API & FBRef → writes CSV/Parquet into data/raw/.
• Task “Data Cleaning/Feature Engineering” reads data/raw/ → produces cleaned datasets in data/processed/.
• Task “Model Training” reads processed features → serializes model files in models/trained/.
• Task “Model Evaluation” takes trained models + test data → outputs evaluation metrics.
• Task “Visualization” reads processed data + evaluation results → writes charts/dashboards to reports/ or serves via web.
Architectural Patterns & Design Principles
• Layered ETL architecture: Ingestion → Processing → Modeling → Visualization.
• DAG-based orchestration for clear dependencies and retries.
• Separation of concerns: each Python module handles a single responsibility.
• Config‐driven (YAML) to allow parameter changes without code edits.
Putting It All Together
• To the right, show the Reports/Visualization output.