How san-deep-reddy/map-reduce works
Project Type and Purpose
• This is a hybrid, distributed data‐processing framework implementing the MapReduce programming model.
• Python drives orchestration (Master, task distribution, fault tolerance, user‐defined map/reduce functions).
• C++ modules (via pybind11 and nlohmann‐json) provide performance‐critical data parsing, key‐value emission, sorting, and JSON I/O.
File Structure
• Top‐level:
• setup.py, CMakeLists.txt, requirements.txt – for build & installation (Python & C++).
• src/python/ – core orchestration code:
– main.py (Master initialization & workflow)
– map.py, reduce.py (wrapper/abstractions for user‐defined functions + integration with C++ modules)
• src/cpp/ – performance‐critical implementations:
– mapper.cpp, reducer.cpp (fast parsing, KV generation, sorting logic)
– json_utils.hpp (JSON read/write helpers)
• No external microservices folder – it’s a single process (Master) for coordination plus spawned worker processes.
Key Insights from the README
• Three logical components: Master, Mappers, Reducers.
• Fault tolerance: master detects/restarts failed mappers.
• Dynamic scaling: configurable mapper/reducer counts.
• Hybrid fallback: can run pure‐Python if C++ modules unavailable.
a. Main Components
• Master (Python process)
• Mapper Workers (process pool)
• Reducer Workers (process pool)
• C++ Modules (pybind11 interfaces loaded by mappers/reducers)
• External Storage I/O (input file, intermediate JSON files or memory queues, output files)
b. Relationships & Interactions
1. Master reads configuration → spawns N mapper processes.
2. Each Mapper loads C++ “mapper” module for fast parse & emit.
3. Mappers write intermediate KV “shards” (JSON or in‐memory) to staging area.
4. Master triggers shuffle: groups intermediate keys by reducer index.
5. Master spawns M reducer processes.
6. Each Reducer loads C++ “reducer” module for sorting & final emit.
7. Reducers write final output.
8. Master monitors worker health → on failure, respawns worker and replays its input split.
c. Architectural Patterns & Principles
• Master–Worker pattern
• Pipeline stages (Map → Shuffle → Reduce)
• Hybrid cross‐language integration (pybind11 bridge)
• Fault‐tolerance via process monitoring & restart
• Parallelism via Python multiprocessing
d. Key Technologies
• Python 3.6+ multiprocessing library
• pybind11 for C++/Python binding
• nlohmann‐json C++ library for JSON I/O
• CMake & setup.py for build & install
Detail
• Break the pipeline into as many sub‐components as make sense (e.g., “JSON Reader” vs. “KV Sorter” in the C++ reducer).
• Configuration inputs (JSON config file) as a separate element feeding into Master.
• Call out external dependencies: Python, C++ compiler, pybind11, nlohmann‐json library.