# dininduviduneth/reddit-explore-uu architecture

> Architecture diagram of the GitHub repository dininduviduneth/reddit-explore-uu, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dininduviduneth/reddit-explore-uu
- Repository: https://github.com/dininduviduneth/reddit-explore-uu

## Overview

##### Project Type and Purpose
- This is a data engineering project enabling experiments on large datasets (Reddit comments) using a containerized Spark-HDFS cluster.
- The project involves using Docker (via docker-compose) to set up both Spark and HDFS environments, and running various analysis notebooks and scripts (found in the “computational-experiments” and “data-analysis” directories).

##### File Structure
- Top-level directories include:
- computational-experiments — contains Python notebooks and scripts for running Spark jobs/experiments.
- data-analysis — includes additional notebooks focused on analyzing datasets and correlations.
- data — holds sample dataset files.
- docker-config — contains the docker-compose and Dockerfiles for configuring Spark master/worker and HDFS containers.
- The presence of Docker configuration files indicates a microservices-like containerized architecture.
- Multiple Jupyter notebooks point to the use of interactive data analysis environments.

##### Extract Key Insights from the README
- The README outlines the setup and configuration for a multi-container Spark-HDFS cluster deployed with Docker. It also shows the steps for uploading the data into HDFS and running Jupyter notebooks in the “data-analysis” directory.
- It also details scaling the workers via docker-compose, highlighting potential dynamic adjustments in system resources.

a. Main Components:
- Docker Environment: A container orchestration layer managing all components.
- Spark Cluster:
- Spark Master (holds orchestration logic).
- Spark Worker Nodes (multiple containers that scale with docker-compose).
- HDFS Cluster:
- Hadoop Namenode (central management).
- Hadoop Datanodes (distributed storage containers).
- Data Ingestion:
- Data upload process from external download into HDFS.
- Sample dataset stored in the “data” directory.
- Analysis Layer:
- Jupyter Notebook Environment that accesses the Spark cluster for running data analysis.
- Notebooks and scripts in “computational-experiments” and “data-analysis” directories that define various analysis pipelines.

b. Relationships and Interactions:
- That analysis code (Jupyter notebooks) interacts with the Spark cluster, which in turn accesses HDFS data.

c. Architectural Patterns and Design Principles:
- Containerized Microservices: Each major component (Spark, HDFS) runs in its own container, scalable via Docker.
- Distributed Computation: Spark master/worker architecture enabling parallel processing.
- Separation of Concerns: Clear delineation between cluster management (Docker), data storage (HDFS), data processing (Spark), and analysis (Jupyter notebooks).

d. Relevant Technologies, Frameworks, and Libraries:
- Apache Spark and PySpark for distributed data processing.
- Apache Hadoop (HDFS) for distributed data storage.
- Docker and docker-compose for container orchestration.
- Jupyter Notebook for interactive data analysis.

## Diagram (Mermaid)

```mermaid
graph TD
    A["Docker Environment"]:::docker
    subgraph "Spark Cluster"
        SM["Spark Master"]:::spark
        SW["Spark Workers (Scalable)"]:::spark
    end
    subgraph "HDFS Cluster"
        NN["Hadoop Namenode"]:::hdfs
        DN["Hadoop Datanodes"]:::hdfs
    end
    subgraph "Analysis Layer"
        JA["Jupyter Analysis Server"]:::analysis
        CE["Computational Experiments"]:::analysis
        DAN["Data Analysis Notebooks"]:::analysis
    end
    E["Reddit Comments Data"]:::data

    %% Docker Environment manages containers
    A --> SM
    A --> SW
    A --> NN
    A --> DN

    %% Data ingestion flows into HDFS
    E --> NN

    %% Spark accesses data from HDFS
    NN --> SM
    NN --> SW

    %% Analysis components interact with the Spark cluster
    JA --> SM
    CE --> SM
    DAN --> SM

    %% Click Events
    click A "https://github.com/dininduviduneth/reddit-explore-uu/tree/main/docker-config/"
    click SM "https://github.com/dininduviduneth/reddit-explore-uu/tree/main/docker-config/spark-master/"
    click SW "https://github.com/dininduviduneth/reddit-explore-uu/tree/main/docker-config/spark-worker/"
    click E "https://github.com/dininduviduneth/reddit-explore-uu/tree/main/data/"
    click CE "https://github.com/dininduviduneth/reddit-explore-uu/tree/main/computational-experiments/"
    click DAN "https://github.com/dininduviduneth/reddit-explore-uu/tree/main/data-analysis/"

    %% Styles
    classDef docker fill:#FFCCCC,stroke:#CC0000,stroke-width:2px;
    classDef spark fill:#CCFFCC,stroke:#009900,stroke-width:2px;
    classDef hdfs fill:#CCCCFF,stroke:#0000CC,stroke-width:2px;
    classDef analysis fill:#FFFFCC,stroke:#CCCC00,stroke-width:2px;
    classDef data fill:#FFE0B3,stroke:#FF8000,stroke-width:2px;
```
