How ravikumarchavva/cricket-prediction works
Overall Structure & Main Components
• Data Sources & Ingestion:
• External data sources like Cricsheet and ESPN cricket stats (as indicated by modules in “include/a_data_sources”).
• Role of Apache Airflow (the “dags” folder with t20.py and the .astro folder for additional Airflow configuration) in triggering the pipeline.
• Data Processing Pipeline:
• Sequential stages:
• Data Preprocessing: Represent the cleaning and transformation stage (data_manuplation/1_preprocessing and tasks in “include/b_data_preprocessing”).
• Exploratory Data Analysis (EDA): Capture the profiling and analysis phase (data_manuplation/2_eda).
• Data Merging: Combine various datasets (data_manuplation/3_mergers and “include/c_data_merging”).
• Data Filtering: Filter and finalize the dataset (data_manuplation/4_filteringData and “include/d_data_filtering”).
• Machine Learning Modeling:
• Divide into subsections (present in the “ml_modeling” directory):
• Data Preparation (stage 1): Prepares the dataset with Spark and notebooks.
• Model Training:
• Naive training (2_naivetraining)
• Augmented training (3_augumented_training)
• Hyperparameter tuning (4_hptuning, with guidance from WandB as noted in the README)
• Final model selection and conversion to ONNX (5_selecting_best_model_to_onnx)
• Subcomponents for each model type: note that the system uses an RNN for sequential data (ball-by-ball info), a CNN for player stats, and a DNN for team stats; these are later combined in an ensemble architecture.
• Supporting Components:
• Utilities: Represent the “utils” folder which contains helper scripts (architecture, data utilities, model utilities, spark_utils).
• Experiment Tracking & Storage: Note the use of Weights & Biases and Hadoop HDFS (listed in the README).
• Visualization & Documentation: Include the “public” folder with drawings (architectureOverview.drawio, images for pipeline, tuning, results, win/loss examples).
Connection & Data Flow
• That Apache Airflow orchestrates the ETL tasks by connecting the ingestion modules to processing tasks (each staged as an independent Airflow task).
• How the output from data processing flows into multiple ML modules (RNN, CNN, DNN) and finally into the ensemble model.
Architectural Patterns & Design Principles
• Separation of concerns:
• Data extraction, preprocessing, merging, and filtering are clearly segmented in different directories and modules.
• Model training stages are modular, allowing for experimentation (naive training, augmented training, hyperparameter tuning).
• Use containerization:
• Docker encapsulating the entire application, ensuring consistency across development and deployment.
• Fault tolerance & scalability considerations through Airflow’s DAG orchestration, Apache Spark’s distributed processing, and modular design that allows for future enhancements (like live API integration).
Final Advice for the Principal Software Engineer
1. Data Ingestion and Preprocessing
2. ETL Pipeline orchestrated by Airflow
3. ML Modeling (with sub-blocks for each model type)
4. Deployment Environment (Docker, etc.)
5. Supporting modules (configs, utilities, experiment tracking)
• Clearly label every block and annotate with technology stacks and responsibilities.