How akstrov/bigdatalabs works
Project type and purpose
• This is an educational “Big Data Labs” repository, not a production-grade system.
• Purpose: provide hands-on labs on distributed storage (HDFS/HBase), batch processing (MapReduce/Spark), and streaming (Kafka + Hadoop streaming).
File structure
• Top-level directories:
• lab0/ – Docker Compose environment setup for Hadoop, HBase, Spark, Kafka, Jupyter.
• lab2/ – A Maven-based Java project exercising basic HDFS operations (write, read, status).
• lab3_mapreduce/ – A Maven Java MapReduce WordCount example plus Python streaming mapper/reducer scripts.
• Configuration files under lab0/image/config → one container per service (config for core-site.xml, hdfs-site.xml, etc.).
• Build artifacts (lab2/target, lab3_mapreduce/target) indicate the compile-then-package workflow using Maven.
README for additional insights
• Confirms tech stack: Apache Hadoop (HDFS, YARN), HBase, Spark, Kafka, Docker, Maven, Java, Python.
• Describes lab progression:
• Lab 0: environment bootstrapping
• Lab 2: HDFS API exercises
• Lab 3: MapReduce (Java + Python streaming)