# arpbansal/distgrep architecture

> Architecture diagram of the GitHub repository arpbansal/distgrep, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arpbansal/distgrep
- Repository: https://github.com/arpbansal/distgrep

## Overview

##### Project type and purpose
- This is a Go-based distributed system that implements the “Distributed Grep” example from Google’s MapReduce paper (Section 2.3).
- It exposes an HTTP API for clients to submit grep queries, coordinates map/reduce jobs across worker nodes, uses Raft for consensus/leader election, and includes service discovery and logging.

##### File structure
- Top-level:
- main.go – application entrypoint, bootstraps all subsystems
- go.mod/go.sum, Makefile – build/test scripts
- internal/http – HTTP server module (client API)
- internal/coordinator – orchestrates Map and Reduce tasks
- internal/discovery – service registry for worker node lookup
- internal/grep – the map function (grep logic)
- internal/mapreduce – framework tying map + reduce phases
- internal/raft – Raft consensus layer (cluster membership, leader election)
- internal/logger – centralized logging
- internal/types – shared data structures (job definitions, state)
- test – chaos tests for distributed behavior and Raft tests

##### README for technical stack
- Go 1.24, Makefile targets for build/run/test/lint

##### Capturing essential details
- Inter-module dependencies (e.g., Coordinator imports MapReduce and Discovery, Service Discovery relies on Raft).
- That each Worker runs both map (grep) and reduce functions.
- How the Makefile orchestrates local build/run/test for development.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    subgraph "Client"
        direction TB
        Client["Client (CLI/REST)"]:::client
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        HTTP["HTTP Server\n(internal/http/server.go)"]:::control
    end

    %% Control Plane
    subgraph "Control Plane" 
        direction TB
        Coord["Coordinator Service\n(internal/coordinator/coordinator.go)"]:::control
        Disc["Service Discovery\n(internal/discovery/discovery.go)"]:::control
        RaftCluster["Raft Cluster\n(cluster.go, fsm.go)"]:::control
    end

    %% Data Plane
    subgraph "Worker Pool"
        direction TB
        MapReduceFW["MapReduce Framework\n(internal/mapreduce/mapreduce.go)"]:::data
        GrepWorker["Grep Worker\n(internal/grep/grep.go)"]:::data
    end

    %% Infrastructure
    subgraph "Infrastructure"
        direction TB
        StateStore["Metadata Store\n(internal/types/state.go, types.go)"]:::infra
        Logger["Logger Service\n(internal/logger/log.go)"]:::infra
    end

    %% Entry & Build
    subgraph "Bootstrap & CI"
        direction TB
        Main["main.go"]:::infra
        Build["Makefile\n(go.mod, go.sum)"]:::infra
        ChaosTest["Chaos Test\n(test/chaos_test.go)"]:::infra
        RaftTest["Raft Test\n(test/raft_test.go)"]:::infra
    end

    %% Data Flows
    Client -->|"submit grep job"| HTTP
    HTTP -->|"forward job request"| Coord
    Coord -->|"query workers"| Disc
    Disc -->|"member list"| RaftCluster
    RaftCluster -->|"cluster membership"| Disc
    Coord -->|"assign map tasks"| GrepWorker
    GrepWorker -->|"map results"| Coord
    Coord -->|"invoke reduce"| MapReduceFW
    MapReduceFW -->|"reduce results"| Coord
    Coord -->|"return final result"| HTTP
    HTTP -->|"HTTP response"| Client

    %% Logging
    Client -->|logs| Logger
    HTTP -->|logs| Logger
    Coord -->|logs| Logger
    Disc -->|logs| Logger
    RaftCluster -->|logs| Logger
    GrepWorker -->|logs| Logger
    MapReduceFW -->|logs| Logger

    %% Metadata Store
    RaftCluster -->|persist log state| StateStore
    Disc -->|read/write metadata| StateStore
    Coord -->|read/write metadata| StateStore

    %% Bootstrap & CI Links
    Main --> HTTP
    Main --> Coord
    Main --> Disc
    Main --> RaftCluster
    Build --> Main
    Build --> ChaosTest
    Build --> RaftTest

    %% Click Events
    click HTTP "https://github.com/arpbansal/distgrep/blob/main/internal/http/server.go"
    click Coord "https://github.com/arpbansal/distgrep/blob/main/internal/coordinator/coordinator.go"
    click Disc "https://github.com/arpbansal/distgrep/blob/main/internal/discovery/discovery.go"
    click GrepWorker "https://github.com/arpbansal/distgrep/blob/main/internal/grep/grep.go"
    click MapReduceFW "https://github.com/arpbansal/distgrep/blob/main/internal/mapreduce/mapreduce.go"
    click RaftCluster "https://github.com/arpbansal/distgrep/blob/main/internal/raft/cluster.go"
    click RaftCluster "https://github.com/arpbansal/distgrep/blob/main/internal/raft/fsm.go"
    click Logger "https://github.com/arpbansal/distgrep/blob/main/internal/logger/log.go"
    click StateStore "https://github.com/arpbansal/distgrep/blob/main/internal/types/state.go"
    click StateStore "https://github.com/arpbansal/distgrep/blob/main/internal/types/types.go"
    click Main "https://github.com/arpbansal/distgrep/blob/main/main.go"
    click Build "https://github.com/arpbansal/distgrep/tree/main/Makefile"
    click ChaosTest "https://github.com/arpbansal/distgrep/blob/main/test/chaos_test.go"
    click RaftTest "https://github.com/arpbansal/distgrep/blob/main/test/raft_test.go"

    %% Styles
    classDef client fill:#d3d3d3,stroke:#333,stroke-width:1px;
    classDef control fill:#cce5ff,stroke:#004085,stroke-width:1px;
    classDef data fill:#d4edda,stroke:#155724,stroke-width:1px;
    classDef infra fill:#fff3cd,stroke:#856404,stroke-width:1px;
```
