How arpbansal/distgrep works
Project type and purpose
• This is a Go-based distributed system that implements the “Distributed Grep” example from Google’s MapReduce paper (Section 2.3).
• It exposes an HTTP API for clients to submit grep queries, coordinates map/reduce jobs across worker nodes, uses Raft for consensus/leader election, and includes service discovery and logging.
File structure
• Top-level:
• main.go – application entrypoint, bootstraps all subsystems
• go.mod/go.sum, Makefile – build/test scripts
• internal/http – HTTP server module (client API)
• internal/coordinator – orchestrates Map and Reduce tasks
• internal/discovery – service registry for worker node lookup
• internal/grep – the map function (grep logic)
• internal/mapreduce – framework tying map + reduce phases
• internal/raft – Raft consensus layer (cluster membership, leader election)
• internal/logger – centralized logging
• internal/types – shared data structures (job definitions, state)
• test – chaos tests for distributed behavior and Raft tests
README for technical stack
• Go 1.24, Makefile targets for build/run/test/lint
Capturing essential details
• Inter-module dependencies (e.g., Coordinator imports MapReduce and Discovery, Service Discovery relies on Raft).
• That each Worker runs both map (grep) and reduce functions.
• How the Makefile orchestrates local build/run/test for development.