# san-deep-reddy/map-reduce architecture

> Architecture diagram of the GitHub repository san-deep-reddy/map-reduce, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/san-deep-reddy/map-reduce
- Repository: https://github.com/san-deep-reddy/map-reduce

## Overview

##### Project Type and Purpose
- This is a hybrid, distributed data‐processing framework implementing the MapReduce programming model.
- Python drives orchestration (Master, task distribution, fault tolerance, user‐defined map/reduce functions).
- C++ modules (via pybind11 and nlohmann‐json) provide performance‐critical data parsing, key‐value emission, sorting, and JSON I/O.

##### File Structure
- Top‐level:
- setup.py, CMakeLists.txt, requirements.txt – for build & installation (Python & C++).
- src/python/ – core orchestration code:
– main.py (Master initialization & workflow)
– map.py, reduce.py (wrapper/abstractions for user‐defined functions + integration with C++ modules)
- src/cpp/ – performance‐critical implementations:
– mapper.cpp, reducer.cpp (fast parsing, KV generation, sorting logic)
– json_utils.hpp (JSON read/write helpers)
- No external microservices folder – it’s a single process (Master) for coordination plus spawned worker processes.

##### Key Insights from the README
- Three logical components: Master, Mappers, Reducers.
- Fault tolerance: master detects/restarts failed mappers.
- Dynamic scaling: configurable mapper/reducer counts.
- Hybrid fallback: can run pure‐Python if C++ modules unavailable.

a. Main Components
- Master (Python process)
- Mapper Workers (process pool)
- Reducer Workers (process pool)
- C++ Modules (pybind11 interfaces loaded by mappers/reducers)
- External Storage I/O (input file, intermediate JSON files or memory queues, output files)
b. Relationships & Interactions
1. Master reads configuration → spawns N mapper processes.
2. Each Mapper loads C++ “mapper” module for fast parse & emit.
3. Mappers write intermediate KV “shards” (JSON or in‐memory) to staging area.
4. Master triggers shuffle: groups intermediate keys by reducer index.
5. Master spawns M reducer processes.
6. Each Reducer loads C++ “reducer” module for sorting & final emit.
7. Reducers write final output.
8. Master monitors worker health → on failure, respawns worker and replays its input split.
c. Architectural Patterns & Principles
- Master–Worker pattern
- Pipeline stages (Map → Shuffle → Reduce)
- Hybrid cross‐language integration (pybind11 bridge)
- Fault‐tolerance via process monitoring & restart
- Parallelism via Python multiprocessing
d. Key Technologies
- Python 3.6+ multiprocessing library
- pybind11 for C++/Python binding
- nlohmann‐json C++ library for JSON I/O
- CMake & setup.py for build & install

##### Detail
- Break the pipeline into as many sub‐components as make sense (e.g., “JSON Reader” vs. “KV Sorter” in the C++ reducer).
- Configuration inputs (JSON config file) as a separate element feeding into Master.
- Call out external dependencies: Python, C++ compiler, pybind11, nlohmann‐json library.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration and Documentation
    config["Config (JSON file)"]:::external
    doc["Project README"]:::external
    click doc "https://github.com/san-deep-reddy/map-reduce/blob/main/README.md"

    %% Build & Installation
    subgraph "Build & Installation"
        setup["setup.py"]:::external
        cmake["CMakeLists.txt"]:::external
        req["requirements.txt"]:::external
    end
    click setup "https://github.com/san-deep-reddy/map-reduce/blob/main/setup.py"
    click cmake "https://github.com/san-deep-reddy/map-reduce/blob/main/CMakeLists.txt"
    click req "https://github.com/san-deep-reddy/map-reduce/blob/main/requirements.txt"

    %% Core Components
    config --> master
    doc --> master

    master["Master (main.py)"]:::orchestration
    click master "https://github.com/san-deep-reddy/map-reduce/blob/main/src/python/main.py"

    %% Worker Pools
    subgraph "Map Stage"
        mappers["Mapper Workers (pool)"]:::orchestration
        click mappers "https://github.com/san-deep-reddy/map-reduce/blob/main/src/python/map.py"
        cppMapper["C++ Mapper Module"]:::cpp
        click cppMapper "https://github.com/san-deep-reddy/map-reduce/blob/main/src/cpp/mapper.cpp"
        fallbackMap["Python Fallback"]:::orchestration
    end

    subgraph "Reduce Stage"
        reducers["Reducer Workers (pool)"]:::orchestration
        click reducers "https://github.com/san-deep-reddy/map-reduce/blob/main/src/python/reduce.py"
        cppReducer["C++ Reducer Module"]:::cpp
        click cppReducer "https://github.com/san-deep-reddy/map-reduce/blob/main/src/cpp/reducer.cpp"
        fallbackReduce["Python Fallback"]:::orchestration
    end

    %% Data Stores
    inputData["Input Data"]:::data
    intermediateData["Intermediate Shards"]:::data
    outputData["Output Data"]:::data

    %% Orchestration Flow
    master -->|spawn| mappers
    inputData -->|read| mappers
    mappers -->|"emit KV"| intermediateData
    mappers -->|load| cppMapper
    mappers -.->|fallback| fallbackMap
    fallbackMap -.->|execute| mappers

    intermediateData -->|"shuffle"| master
    master -->|spawn| reducers
    intermediateData -->|partition| reducers
    reducers -->|"emit result"| outputData
    reducers -->|load| cppReducer
    reducers -.->|fallback| fallbackReduce
    fallbackReduce -.->|execute| reducers

    %% Fault Tolerance
    master -->|monitor heartbeat| mappers
    master -->|monitor heartbeat| reducers
    master -->|respawn failed| mappers
    master -->|respawn failed| reducers

    %% Relationships
    master --> setup
    master --> cmake
    master --> req

    %% Classes
    classDef orchestration fill:#dae8fc,stroke:#6c8ebf,color:#000;
    classDef data fill:#d5e8d4,stroke:#82b366,color:#000;
    classDef cpp fill:#f8cecc,stroke:#b85450,color:#000;
    classDef external fill:#f5f5f5,stroke:#999,color:#000;

    %% Legend
    subgraph Legend
        A["Master / Workers (Python)"]:::orchestration
        B["Data Stores"]:::data
        C["C++ Modules"]:::cpp
        D["External / Config"]:::external
    end
```
