How chinmayee421/ipl_data_analysis_spark works
Main Components
• Data Storage: AWS S3 bucket containing the CSV files (Ball_By_Ball.csv, Match.csv, Player_match.csv, Player.csv, Team.csv).
• Processing Engine: Apache Spark cluster that is used for distributed data processing.
• Notebook Environment: The Databricks environment where the IPL_SPARK_PROJECT.ipynb notebook is executed.
• Programming Environment: Python/PySpark as the language and API used for the analysis.
• (Optional) External Interfaces: Although not explicitly mentioned, you could also represent potential visual outputs or result exports that might be used for further reporting or dashboarding.
Architectural Patterns and Relationships
• That the system leverages a cloud-based data lake (AWS S3) for storing structured CSV files.
• Demonstrate the decoupled nature between data storage and data processing—data is loaded from S3 into the Spark cluster, processed, and then results can be visualized or exported.
• That the architecture leverages distributed processing via Apache Spark to efficiently handle EDA on potentially large datasets.
• Dependency lines between the notebook environment and the Apache Spark cluster to represent that the notebook is the orchestrator of the analysis tasks.