How sawlachintan/cricket_data works
Project’s Purpose and Type
• The project focuses on the ingestion, processing, and analysis of cricket data from various leagues/series.
• It follows a multi-stage pipeline that categorizes data into Bronze, Silver, and Gold engineering layers, and finally into an analysis layer where visualizations (dashboards) and prediction models run.
• The purpose is twofold: to transform raw cricket data (stored in CSVs and Parquet files under data/bronze and data/util directories) into refined datasets, and to enable further analysis for insights and predictions.
File Structure
• Key directories are:
• data/bronze: Contains raw data segregated by cricket leagues/series (such as bbl, bpl, cpl, etc.), with each league having multiple files (event_df, info_df, innings_df, etc.).
• data/util: Contains utility CSV files summarizing or supporting information across leagues.
• The presence of Python scripts like fetch_data.py and json_to_csv.py indicates parts of the pipeline for data extraction and transformation.
• The existence of requirements.txt suggests dependency management.
• There is no explicit frontend or backend directory, which reinforces that this is a data engineering pipeline project primarily.
Utilize the README Insights
• A high-level engineering process showing the flow from Bronze to Silver to Gold, and eventually connecting to the Analysis components (visualizations and prediction).
Main Components
a. Data Ingestion:
• Web Layer/Endpoint represented by fetch_data.py.
• Process that converts JSON to CSV (json_to_csv.py), tying into the data processing pipeline.
b. Data Processing Pipeline (Engineering Layer):
• Bronze: Raw data ingestion from various leagues (as structured under data/bronze).
• Silver and Gold: Intermediate and final stages of data processing/aggregation, even if the physical directories are not present, they are implied by the pipeline flow.
c. Data Analysis:
• Visualization (dashboards) module.
• Prediction or modeling module.
Relationships and Interactions Between Components
• The Web layer (fetch_data) initiates data collection and then passes data to json_to_csv, which converts data from JSON into CSV.
• Converted data feeds into the Bronze stage where raw datasets are organized by different cricket leagues.
• Processed data then move organically to the Silver stage and eventually to the Gold stage where refined “basic_stats” and other metrics are computed.
• Finally, the refined data is used by the Analysis layer for visualizations and predictive analytics.
Architectural Patterns and Design Principles
• Pipeline pattern: Data moves in a sequential manner through distinct stages (Bronze → Silver → Gold → Analysis).
• Separation of concerns: Ingestion, transformation, and analysis are clearly separated.
• Layered Architecture: Different layers of processing; engineering stages and analysis functions are visually and conceptually divided.