# yeisme/hadoop_bigdata_analysis architecture

> Architecture diagram of the GitHub repository yeisme/hadoop_bigdata_analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yeisme/hadoop_bigdata_analysis
- Repository: https://github.com/yeisme/hadoop_bigdata_analysis

## Overview

##### Main Components Identification
The project appears to be a data processing pipeline with the following key components:

a) Data Processing Components:
- Dataset Management System
- Extraction Service
- Validation Service
- Preprocessing Pipeline
- Hadoop Service Integration

b) Deployment Infrastructure:
- Kubernetes Cluster (1-node and 3-node configurations)
- Docker Container System

##### Component Relationships

Use the following connections:
- Double lines for service boundaries

##### Specific Elements

a) Infrastructure Components:
**[Kubernetes Cluster]**
- 1-Node Configuration
- 3-Node Configuration
- Docker Container
- Hadoop Service

b) Data Processing Flow:
**[Datasets] → [Extract Service] → [Validation] → [Preprocessing] → [Output]**

c) Support Systems:
- Script execution environment
- Environment configuration
- Make automation system

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Infrastructure
        K8S[Kubernetes Cluster]
        K1N["1-Node Config"]:::k8s
        K3N["3-Node Config"]:::k8s
        DOC[Docker Container]:::docker
        HAD[Hadoop Service]:::hadoop
    end

    subgraph "Data Processing"
        DMS[Dataset Management]:::data
        EXT[Extraction Service]:::process
        VAL[Validation Service]:::process
        PRE[Preprocessing Pipeline]:::process
    end

    subgraph "Output Processing"
        DBLP[DBLP Processor]:::output
        FOR[Forest Processor]:::output
    end

    subgraph "Configuration & Build"
        ENV[Environment Config]:::config
        MK[Build System]:::config
        INIT[Project Init]:::config
    end

    %% Infrastructure Relationships
    K8S --> K1N
    K8S --> K3N
    K3N --> DOC
    DOC --> HAD

    %% Data Processing Flow
    DMS --> EXT
    EXT --> VAL
    VAL --> PRE
    PRE --> DBLP
    PRE --> FOR

    %% Configuration Dependencies
    ENV -.-> DMS
    ENV -.-> HAD
    MK -.-> K8S
    INIT -.-> ENV

    %% Component Mappings
    click DMS "https://github.com/yeisme/Hadoop_bigdata_analysis/tree/main/datasets/"
    click EXT "https://github.com/yeisme/Hadoop_bigdata_analysis/tree/main/datasets/extract/"
    click VAL "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/datasets/validate.py"
    click PRE "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/script/data_preprocessing.sh"
    click HAD "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/script/start_hadoop_service.sh"
    click K1N "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/deploy/k8s/value-1node.yaml"
    click K3N "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/deploy/k8s/value-3node.yaml"
    click DOC "https://github.com/yeisme/Hadoop_bigdata_analysis/tree/main/deploy/k8s/Dockerfile"
    click DBLP "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/output/DBLP/preprocessing.sh"
    click FOR "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/output/forest/preprocessing.sh"
    click ENV "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/.env"
    click MK "https://github.com/yeisme/Hadoop_bigdata_analysis/tree/main/Makefile"
    click INIT "https://github.com/yeisme/Hadoop_bigdata_analysis/blob/main/script/init_project.sh"

    %% Styling
    classDef k8s fill:#326CE5,color:white
    classDef docker fill:#2496ED,color:white
    classDef hadoop fill:#FF6600,color:white
    classDef data fill:#4CAF50,color:white
    classDef process fill:#81C784,color:white
    classDef output fill:#FFD700,color:black
    classDef config fill:#9E9E9E,color:white

    %% Legend
    subgraph Legend
        L1[Kubernetes]:::k8s
        L2[Docker]:::docker
        L3[Hadoop]:::hadoop
        L4[Data Management]:::data
        L5[Processing]:::process
        L6[Output]:::output
        L7[Configuration]:::config
    end
```
