# 527515025/bigdata architecture

> Architecture diagram of the GitHub repository 527515025/bigdata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/527515025/bigdata
- Repository: https://github.com/527515025/bigdata

## Overview

##### Project Type Analysis
This is a data processing pipeline project consisting of three main components:
- Flume plugins for data ingestion
- Kafka messaging system for data streaming
- Spark processing for data analytics

##### Main Components

A. Flume Components:
- TailDirSource: Data ingestion from log files
- File monitoring system
- Log4j integration
- Checkpoint mechanism for resume capability
- MysqlSink: Data output component
- MySQL database connection
- HTTP request capability
- Custom Interceptor: Data transformation/filtering

B. Kafka Components:
- Producer systems
- Consumer systems
- Streaming components
- Word count processing

C. Spark Components:
- Spark SQL functionality
- Spark Streaming
- Direct Kafka integration
- Various processing components (WordCount, SQL operations)

**Data Sources → Flume → Kafka → Spark → Output Systems**

##### Detailed Component Breakdown

a) Flume Section:
**[Log Files] → [TailDirSource] → [Interceptor] → [MysqlSink]**
- ↓
- [MySQL DB]
- ↓
- [HTTP Endpoint]

b) Kafka Section:
**[Producers] → [Kafka Brokers] → [Consumers]**
- ↓
- [Kafka Streams]
- ↓
- [Word Count Processing]

c) Spark Section:
**[Spark Streaming] → [Direct Kafka Consumer]**
- ↓
**[Spark SQL] → [Data Processing]**
- ↓
**[Output Storage]**

##### Key Relationships to Highlight

- Data flow between Flume and MySQL
- Kafka producer/consumer relationships
- Spark's direct integration with Kafka
- Stream processing pipelines
- Data transformation points

##### Technical Details

- File monitoring capabilities in TailDirSource
- MySQL connection details in MysqlSink
- Kafka stream processing configurations
- Spark SQL and streaming integration points
- Checkpoint and recovery mechanisms

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and classes
    classDef flume fill:#b3d1ff,stroke:#333
    classDef kafka fill:#c2f0c2,stroke:#333
    classDef spark fill:#ffd9b3,stroke:#333
    classDef storage fill:#d9d9d9,stroke:#333
    classDef external fill:#f2f2f2,stroke:#333

    %% External Systems
    LogFiles[("Log Files")]:::external
    MySQL[("MySQL DB")]:::storage
    HTTP["HTTP Endpoint"]:::external

    %% Flume Components
    subgraph Flume
        TailDir["TailDirSource"]:::flume
        Interceptor["Custom Interceptor"]:::flume
        MysqlSink["MySQL Sink"]:::flume
    end

    %% Kafka Components
    subgraph Kafka
        KafkaProducer["Kafka Producer"]:::kafka
        KafkaBroker["Kafka Brokers"]:::kafka
        KafkaConsumer["Kafka Consumer"]:::kafka
        KafkaStream["Kafka Streams"]:::kafka
        WordCount["Word Count Processing"]:::kafka
    end

    %% Spark Components
    subgraph Spark
        SparkStream["Spark Streaming"]:::spark
        DirectKafka["Direct Kafka Consumer"]:::spark
        SparkSQL["Spark SQL"]:::spark
        DataProc["Data Processing"]:::spark
        SQLDataSource["SQL Data Source"]:::spark
    end

    %% Connections
    LogFiles --> TailDir
    TailDir --> Interceptor
    Interceptor --> MysqlSink
    MysqlSink --> MySQL
    MysqlSink --> HTTP

    KafkaProducer --> KafkaBroker
    KafkaBroker --> KafkaConsumer
    KafkaBroker --> KafkaStream
    KafkaStream --> WordCount

    KafkaBroker --> DirectKafka
    DirectKafka --> SparkStream
    SparkStream --> SparkSQL
    SparkSQL --> DataProc
    SparkSQL --> SQLDataSource

    %% Click events
    click TailDir "https://github.com/527515025/bigdata/blob/master/flume/tailDirSource/src/main/java/com/us/source/taildir/TaildirSource.java"
    click MysqlSink "https://github.com/527515025/bigdata/blob/master/flume/flumeMysqlSink/src/main/java/com/us/flume/MysqlSink.java"
    click Interceptor "https://github.com/527515025/bigdata/blob/master/flume/flumeInterceptor/src/main/java/com/us/MyInterceptor.java"
    click HTTP "https://github.com/527515025/bigdata/blob/master/flume/flumeMysqlSink/src/main/java/com/us/flume/HttpRequest.java"
    click KafkaProducer "https://github.com/527515025/bigdata/blob/master/kafka/src/main/java/com/us/kafka/KafkaProducerDemo.java"
    click KafkaConsumer "https://github.com/527515025/bigdata/blob/master/kafka/src/main/java/com/us/kafka/KafkaConsumer.java"
    click KafkaStream "https://github.com/527515025/bigdata/blob/master/kafka/src/main/java/com/us/kafka/Stream/KafkaStreamWordCount.java"
    click WordCount "https://github.com/527515025/bigdata/blob/master/kafka/src/main/java/com/us/kafka/Stream/WordCountProcessorDemo.java"
    click SparkSQL "https://github.com/527515025/bigdata/blob/master/spark/mySpark/src/main/scala/SparkSQLDemo.scala"
    click DirectKafka "https://github.com/527515025/bigdata/blob/master/spark/mySpark/src/main/scala/DirectKafkaWordCountDemo.scala"
    click DataProc "https://github.com/527515025/bigdata/blob/master/spark/mySpark/src/main/scala/WordCount.scala"
    click SparkStream "https://github.com/527515025/bigdata/blob/master/spark/mySpark/src/main/scala/QueueStreamDemo.scala"
    click SQLDataSource "https://github.com/527515025/bigdata/blob/master/spark/mySpark/src/main/scala/SQLDataSourceDemo.scala"
```
