How big-data-manning/big-data-code works
Project Type and Purpose
• This is a Lambda‑Architecture–style real‑time analytics pipeline (as described in “Big Data: Principles and best practices of scalable real‑time data systems”).
• It implements three key layers:
a. Batch Layer (batchlayer) – Periodic, large‑scale processing over immutable raw data.
b. Speed Layer (speedlayer) – Real‑time, low‑latency processing for up‑to‑the‑moment views.
c. Serving Layer – Merges results from batch and speed layers for query responses.
• It uses Java, Apache Thrift (schema.thrift), Apache Pail (for flat files on HDFS), Hadoop MapReduce, Apache Storm/Trident, and Cassandra.
File Structure Analysis
• Top‑level: genthrift.sh, pom.xml (Maven), schema.thrift (Thrift schema)
• src/java/manning/tap: custom Pail “Taps” for ingesting and serializing data (DataPailTap, ThriftPailStructure, etc.)
• src/java/manning/batchlayer: BatchWorkflow.java – orchestrates MapReduce jobs reading Pail data, producing batch views.
• src/java/manning/speedlayer: TridentSpeedLayer.java, CassandraState.java, UniquesOverTime.java – real‑time Storm topology, writes to Cassandra.
• src/java/manning/schema: Java data models generated from Thrift (DataUnit, PageViewEdge, ExternalDataSystem, etc.).
• No explicit “servinglayer” directory, but CassandraState acts as the serving view for real‑time data; batch views are stored back in Pail/HDFS.
Key Architectural Patterns
• Lambda Architecture: separate batch and speed paths, unified query interface.
• Data serialization via Thrift for cross‑component data contracts.
• Storage abstraction via Apache Pail for HDFS files.
• Real‑time processing via Storm Trident and Cassandra (stateful topologies).
Components
A. Data Sources & Ingestion
• External Data Systems (e.g., clickstreams, page‑view events)
• Thrift Pulldown: genthrift.sh → Java schema models
• Pail Taps (DataPailTap/SplitDataPailTap) writing to HDFS (Master Dataset)
B. Batch Layer
• BatchWorkflow (MapReduce jobs)
• Reads Master Dataset (Pail on HDFS)
• Writes Batch Views (Pail/HDFS)
C. Speed Layer
• TridentSpeedLayer topology (Storm)
• Real‑time streams (Kafka or other ingestion – implied)
• UniquesOverTime computation
• Writes to Cassandra via CassandraState
D. Serving Layer / Query Interface
• Consolidates Batch Views (HDFS) + Real‑time Views (Cassandra)
• Exposes RPC/API for downstream consumers
Technologies & Frameworks
• Apache Thrift (schema.thrift → Java)
• Apache Hadoop & Pail (HDFS storage abstraction)
• Apache Storm Trident (real‑time processing)
• Apache Cassandra (real‑time state store)
• Maven (pom.xml)