# big-data-manning/big-data-code architecture

> Architecture diagram of the GitHub repository big-data-manning/big-data-code, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/big-data-manning/big-data-code
- Repository: https://github.com/big-data-manning/big-data-code

## Overview

##### Project Type and Purpose
- This is a Lambda‑Architecture–style real‑time analytics pipeline (as described in “Big Data: Principles and best practices of scalable real‑time data systems”).
- It implements three key layers:
a. Batch Layer (batchlayer) – Periodic, large‑scale processing over immutable raw data.
b. Speed Layer (speedlayer) – Real‑time, low‑latency processing for up‑to‑the‑moment views.
c. Serving Layer – Merges results from batch and speed layers for query responses.
- It uses Java, Apache Thrift (schema.thrift), Apache Pail (for flat files on HDFS), Hadoop MapReduce, Apache Storm/Trident, and Cassandra.

##### File Structure Analysis
- Top‑level: genthrift.sh, pom.xml (Maven), schema.thrift (Thrift schema)
- src/java/manning/tap: custom Pail “Taps” for ingesting and serializing data (DataPailTap, ThriftPailStructure, etc.)
- src/java/manning/batchlayer: BatchWorkflow.java – orchestrates MapReduce jobs reading Pail data, producing batch views.
- src/java/manning/speedlayer: TridentSpeedLayer.java, CassandraState.java, UniquesOverTime.java – real‑time Storm topology, writes to Cassandra.
- src/java/manning/schema: Java data models generated from Thrift (DataUnit, PageViewEdge, ExternalDataSystem, etc.).
- No explicit “servinglayer” directory, but CassandraState acts as the serving view for real‑time data; batch views are stored back in Pail/HDFS.

##### Key Architectural Patterns
- Lambda Architecture: separate batch and speed paths, unified query interface.
- Data serialization via Thrift for cross‑component data contracts.
- Storage abstraction via Apache Pail for HDFS files.
- Real‑time processing via Storm Trident and Cassandra (stateful topologies).

##### Components
A. Data Sources & Ingestion
- External Data Systems (e.g., clickstreams, page‑view events)
- Thrift Pulldown: genthrift.sh → Java schema models
- Pail Taps (DataPailTap/SplitDataPailTap) writing to HDFS (Master Dataset)
B. Batch Layer
- BatchWorkflow (MapReduce jobs)
- Reads Master Dataset (Pail on HDFS)
- Writes Batch Views (Pail/HDFS)
C. Speed Layer
- TridentSpeedLayer topology (Storm)
- Real‑time streams (Kafka or other ingestion – implied)
- UniquesOverTime computation
- Writes to Cassandra via CassandraState
D. Serving Layer / Query Interface
- Consolidates Batch Views (HDFS) + Real‑time Views (Cassandra)
- Exposes RPC/API for downstream consumers

##### Technologies & Frameworks
- Apache Thrift (schema.thrift → Java)
- Apache Hadoop & Pail (HDFS storage abstraction)
- Apache Storm Trident (real‑time processing)
- Apache Cassandra (real‑time state store)
- Maven (pom.xml)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top-level Schema & Build
    subgraph "Schema & Build"
        direction TB
        ThriftScript["genthrift.sh"]:::schema
        ThriftSchema["schema.thrift"]:::schema
        GenSchema["Java Schema Models"]:::schema
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        ExternalData["External Data Systems"]:::ingestion
        DataPailTap["DataPailTap"]:::ingestion
        SplitDataPailTap["SplitDataPailTap"]:::ingestion
        DataPailStructure["DataPailStructure"]:::ingestion
        SplitDataPailStructure["SplitDataPailStructure"]:::ingestion
        ThriftPailStructure["ThriftPailStructure"]:::ingestion
        MasterDataset["Master Dataset (Pail/HDFS)"]:::storage
    end

    %% Batch Layer
    subgraph "Batch Layer"
        direction TB
        BatchWorkflow["BatchWorkflow"]:::processing
        BatchViews["Batch Views (Pail/HDFS)"]:::storage
    end

    %% Speed Layer
    subgraph "Speed Layer"
        direction TB
        TridentSpeed["TridentSpeedLayer"]:::processing
        UniquesOverTime["UniquesOverTime"]:::processing
        CassandraState["CassandraState"]:::storage
    end

    %% Serving Layer
    subgraph "Serving Layer"
        direction TB
        QueryAPI["Query API / Dashboard"]:::api
    end

    %% Data Flows
    ThriftScript -->|generates| GenSchema
    ThriftSchema -->|defines schema| GenSchema
    ExternalData -->|writes raw events| DataPailTap
    ExternalData -->|writes raw events| SplitDataPailTap
    DataPailTap -->|uses| DataPailStructure
    SplitDataPailTap -->|uses| SplitDataPailStructure
    DataPailTap -->|uses| ThriftPailStructure
    SplitDataPailTap -->|uses| ThriftPailStructure
    DataPailTap --> MasterDataset
    SplitDataPailTap --> MasterDataset

    MasterDataset -->|batch read| BatchWorkflow
    BatchWorkflow -->|writes| BatchViews

    ExternalData -.-|real-time stream| TridentSpeed
    TridentSpeed -->|computes| UniquesOverTime
    UniquesOverTime -->|updates| CassandraState

    BatchViews -->|batch view| QueryAPI
    CassandraState -->|real-time view| QueryAPI

    %% Click Events
    click ThriftSchema "https://github.com/big-data-manning/big-data-code/blob/master/schema.thrift"
    click ThriftScript "https://github.com/big-data-manning/big-data-code/blob/master/genthrift.sh"
    click GenSchema "https://github.com/big-data-manning/big-data-code/tree/master/src/java/manning/schema/"
    click DataPailTap "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/tap/DataPailTap.java"
    click SplitDataPailTap "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/tap/SplitDataPailTap.java"
    click DataPailStructure "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/tap/DataPailStructure.java"
    click SplitDataPailStructure "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/tap/SplitDataPailStructure.java"
    click ThriftPailStructure "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/tap/ThriftPailStructure.java"
    click BatchWorkflow "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/batchlayer/BatchWorkflow.java"
    click TridentSpeed "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/speedlayer/TridentSpeedLayer.java"
    click UniquesOverTime "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/speedlayer/UniquesOverTime.java"
    click CassandraState "https://github.com/big-data-manning/big-data-code/blob/master/src/java/manning/speedlayer/CassandraState.java"
    click Pom "https://github.com/big-data-manning/big-data-code/blob/master/pom.xml"
    click README "https://github.com/big-data-manning/big-data-code/blob/master/README.md"

    %% Styles
    classDef schema fill:#FFA07A,stroke:#333,stroke-width:1px
    classDef ingestion fill:#90EE90,stroke:#333,stroke-width:1px
    classDef processing fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef storage fill:#FFD700,stroke:#333,stroke-width:1px
    classDef api fill:#DDA0DD,stroke:#333,stroke-width:1px
```
