How liuyuyan2717/6824lab works
Project Type and Purpose
• This is an educational, multi‑lab distributed systems project (MIT 6.824 labs 1–4).
• It implements:
• A Raft consensus library (src/raft)
• A replicated key/value store on Raft (src/kvraft)
• A simple MapReduce framework and apps (src/mr & src/mrapps)
• A sharded key/value service with dynamic configuration (src/shardctrler & src/shardkv)
• RPC & serialization utilities (src/labrpc, src/labgob)
• A linearizability checker/visualizer (src/porcupine)
• The README confirms this is a student lab repo, not a production system.
File‑Structure Analysis
Top‑level “src” subdirs:
• labrpc & labgob: lightweight RPC and Go‑object‐serialization packages used by all distributed modules.
• raft: core consensus algorithm with persister, state machine interface, RPC definitions.
• kvraft: a key/value server that uses raft to replicate commands.
• shardctrler: controller service that uses raft to manage shard configurations.
• shardkv: sharded key/value servers coordinated by the shard controller, also using raft for each group.
• mr & mrapps: MapReduce coordinator, workers, and sample applications (wc, indexer, etc.).
• porcupine: a post‑hoc checker to validate linearizability of Raft‑based operations.
• main: various test drivers (diskvd, lockd/c, viewd, MapReduce tests, etc.) and sample text files.
README Insights
• The project is strictly for lab exercises; no one central “production” executable.
• Each lab adds a new distributed service or client harness.
• KVStateMachine (kvraft)
• Config State Machine (shardctrler)
• ShardKV StateMachine (shardkv)
• MR Coordinator
• MR Workers
• lockd / lockc (lab1)
• diskvd (lab1)
• viewd (lab1)
• pbd / pbc (lab3 backup / primary)
b. Relationships & Data Flows
• Clients → labrpc → Service endpoints (Coordinator, KVServer, ShardKVServer, ShardController)
• Service endpoints → labrpc → Worker or Peer RPC calls
• Services use labgob for encoding/decoding commands & snapshots
• Each raft node persists state via Persister component → stable storage
• ShardController assigns shards → ShardKV servers pull configs via RPC
• MR Coordinator issues tasks → MR Workers fetch data (text files) → workers send back intermediate/results
• Porcupine taps into logs emitted by RPC calls & Raft commands to validate linearizability offline.
c. Architectural Patterns & Principles
• Modular service layering: RPC & serialization libraries at bottom; consensus core next; then business‑logic state machines; then client/harness at top.
• Stateful services built atop a generic consensus abstraction (Raft).
• Dynamic configuration and sharding: a control plane (shardctrler) drives data‑plane services (shardkv).
• Separation of concerns: MapReduce layer isolated from Raft layer, communicating only via RPC.
• Test harnesses simulate failure & recovery (lock servers, view service, crash modes).
d. Key Technologies & Libraries
• Go modules (go.mod/go.sum)
• labrpc (in‑process RPC simulation over Go channels)
• labgob (Go object marshaling)
• porcupine (linearizability checker in Go)
• Built‑in Go testing framework for automated lab tests.
Details to Capture
• The dual‑role of labrpc & labgob across every service.
• The generic Raft interface vs. concrete state‑machine implementations (kvraft, shardctrler, shardkv).
• Dynamic config flow: ShardController → ShardKV groups.
• MapReduce control/data flows (Coordinator → Worker → File system).
• Verification via porcupine reading RPC logs.
Putting it all together:
– Above each cluster, attach its respective StateMachine (kv, shardctrler, shardkv)