How beingmani/mbd-assignment works
Project type and purpose
• This is an educational Hadoop/MapReduce project containing:
– Assignment 1: word-count, unique-word pipelines (single job, chain mapper, two jobs)
– Assignment 2: a friend-recommender MapReduce job and a Python/Notebook exercise
– A MkDocs-generated documentation site
Main components
a. Developer workstation(s)
– Cloudera VM
– MacOS + Docker
– Python environment (for Jupyter/Ex1)
b. Build/artifact creation
– Java source (src/) → compiled .class → JARs under out/artifacts or jars/
– Python scripts / Jupyter notebooks
c. Hadoop cluster (real or Docker)
– HDFS (input directories; output directories)
– YARN/MapReduce framework
– JobManager (StubDriver.jar, WordCount.jar, Custom*.jar)
d. MapReduce execution
– Mapper classes
– Reducer classes
– (Optional) ChainMapper or two sequential jobs
– MRUnit tests (unit test harness)
e. Data flow
– Raw text files in HDFS → Map tasks → Shuffle & Sort → Reduce tasks → output files
f. Documentation site
– MkDocs (mkdocs.yml) → static HTML/CSS/JS under docs/site
Relationships & interactions
• Developer → builds JARs & Python scripts
• Developer → submits jobs (via StubDriver on JVM or python runner) → YARN
• YARN → schedules Map tasks on cluster nodes → read input blocks from HDFS
• Mappers → emit key-value pairs → shuffle across network → sorted by key
• Reducers → aggregate per key → write output files back to HDFS
• MRUnit tests run locally against mapper/reducer logic (no cluster)
• MkDocs builds documentation site; user browses via static web server
Architectural patterns / design principles
– Batch ETL pipeline (map → reduce → output)
– Single-responsibility: separate Mapper & Reducer classes
– Job orchestration: single job vs. chaining vs. multi-job workflows
– Environment abstraction: same code runs on Cloudera VM or MacOS/Docker
– Test harness decouples logic from Hadoop cluster
Technologies & frameworks
– Java 8+, Hadoop MapReduce API
– MRUnit for unit testing
– Python 3 + Jupyter for Ex1 in Assign2
– MkDocs (YAML config, Markdown → HTML)
– Clear labels for each component (e.g., “StubDriver.java” → “CustomWordCount.jar”)
– Optional: annotate key intermediate files (part-r-00000)