How dininduviduneth/reddit-explore-uu works
Project Type and Purpose
• This is a data engineering project enabling experiments on large datasets (Reddit comments) using a containerized Spark-HDFS cluster.
• The project involves using Docker (via docker-compose) to set up both Spark and HDFS environments, and running various analysis notebooks and scripts (found in the “computational-experiments” and “data-analysis” directories).
File Structure
• Top-level directories include:
• computational-experiments — contains Python notebooks and scripts for running Spark jobs/experiments.
• data-analysis — includes additional notebooks focused on analyzing datasets and correlations.
• data — holds sample dataset files.
• docker-config — contains the docker-compose and Dockerfiles for configuring Spark master/worker and HDFS containers.
• The presence of Docker configuration files indicates a microservices-like containerized architecture.
• Multiple Jupyter notebooks point to the use of interactive data analysis environments.
Extract Key Insights from the README
• The README outlines the setup and configuration for a multi-container Spark-HDFS cluster deployed with Docker. It also shows the steps for uploading the data into HDFS and running Jupyter notebooks in the “data-analysis” directory.
• It also details scaling the workers via docker-compose, highlighting potential dynamic adjustments in system resources.
a. Main Components:
• Docker Environment: A container orchestration layer managing all components.
• Spark Cluster:
• Spark Master (holds orchestration logic).
• Spark Worker Nodes (multiple containers that scale with docker-compose).
• HDFS Cluster:
• Hadoop Namenode (central management).
• Hadoop Datanodes (distributed storage containers).
• Data Ingestion:
• Data upload process from external download into HDFS.
• Sample dataset stored in the “data” directory.
• Analysis Layer:
• Jupyter Notebook Environment that accesses the Spark cluster for running data analysis.
• Notebooks and scripts in “computational-experiments” and “data-analysis” directories that define various analysis pipelines.
b. Relationships and Interactions:
• That analysis code (Jupyter notebooks) interacts with the Spark cluster, which in turn accesses HDFS data.
c. Architectural Patterns and Design Principles:
• Containerized Microservices: Each major component (Spark, HDFS) runs in its own container, scalable via Docker.
• Distributed Computation: Spark master/worker architecture enabling parallel processing.
• Separation of Concerns: Clear delineation between cluster management (Docker), data storage (HDFS), data processing (Spark), and analysis (Jupyter notebooks).
d. Relevant Technologies, Frameworks, and Libraries:
• Apache Spark and PySpark for distributed data processing.
• Apache Hadoop (HDFS) for distributed data storage.
• Docker and docker-compose for container orchestration.
• Jupyter Notebook for interactive data analysis.