# liuyuyan2717/6824lab architecture

> Architecture diagram of the GitHub repository liuyuyan2717/6824lab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/liuyuyan2717/6824lab
- Repository: https://github.com/liuyuyan2717/6824lab

## Overview

##### Project Type and Purpose
- This is an educational, multi‑lab distributed systems project (MIT 6.824 labs 1–4).
- It implements:
- A Raft consensus library (src/raft)
- A replicated key/value store on Raft (src/kvraft)
- A simple MapReduce framework and apps (src/mr & src/mrapps)
- A sharded key/value service with dynamic configuration (src/shardctrler & src/shardkv)
- RPC & serialization utilities (src/labrpc, src/labgob)
- A linearizability checker/visualizer (src/porcupine)
- The README confirms this is a student lab repo, not a production system.

##### File‑Structure Analysis
Top‑level “src” subdirs:
- labrpc & labgob: lightweight RPC and Go‑object‐serialization packages used by all distributed modules.
- raft: core consensus algorithm with persister, state machine interface, RPC definitions.
- kvraft: a key/value server that uses raft to replicate commands.
- shardctrler: controller service that uses raft to manage shard configurations.
- shardkv: sharded key/value servers coordinated by the shard controller, also using raft for each group.
- mr & mrapps: MapReduce coordinator, workers, and sample applications (wc, indexer, etc.).
- porcupine: a post‑hoc checker to validate linearizability of Raft‑based operations.
- main: various test drivers (diskvd, lockd/c, viewd, MapReduce tests, etc.) and sample text files.

##### README Insights
- The project is strictly for lab exercises; no one central “production” executable.
- Each lab adds a new distributed service or client harness.

- KVStateMachine (kvraft)
- Config State Machine (shardctrler)
- ShardKV StateMachine (shardkv)
- MR Coordinator
- MR Workers
- lockd / lockc (lab1)
- diskvd (lab1)
- viewd (lab1)
- pbd / pbc (lab3 backup / primary)

b. Relationships & Data Flows
- Clients → labrpc → Service endpoints (Coordinator, KVServer, ShardKVServer, ShardController)
- Service endpoints → labrpc → Worker or Peer RPC calls
- Services use labgob for encoding/decoding commands & snapshots
- Each raft node persists state via Persister component → stable storage
- ShardController assigns shards → ShardKV servers pull configs via RPC
- MR Coordinator issues tasks → MR Workers fetch data (text files) → workers send back intermediate/results
- Porcupine taps into logs emitted by RPC calls & Raft commands to validate linearizability offline.

c. Architectural Patterns & Principles
- Modular service layering: RPC & serialization libraries at bottom; consensus core next; then business‑logic state machines; then client/harness at top.
- Stateful services built atop a generic consensus abstraction (Raft).
- Dynamic configuration and sharding: a control plane (shardctrler) drives data‑plane services (shardkv).
- Separation of concerns: MapReduce layer isolated from Raft layer, communicating only via RPC.
- Test harnesses simulate failure & recovery (lock servers, view service, crash modes).

d. Key Technologies & Libraries
- Go modules (go.mod/go.sum)
- labrpc (in‑process RPC simulation over Go channels)
- labgob (Go object marshaling)
- porcupine (linearizability checker in Go)
- Built‑in Go testing framework for automated lab tests.

##### Details to Capture
- The dual‑role of labrpc & labgob across every service.
- The generic Raft interface vs. concrete state‑machine implementations (kvraft, shardctrler, shardkv).
- Dynamic config flow: ShardController → ShardKV groups.
- MapReduce control/data flows (Coordinator → Worker → File system).
- Verification via porcupine reading RPC logs.

Putting it all together:
– Above each cluster, attach its respective StateMachine (kv, shardctrler, shardkv)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Infrastructure Layer
    subgraph "Infrastructure"
        labrpc["labrpc"]:::infra
        labgob["labgob"]:::infra
    end

    %% Consensus Layer
    subgraph "Consensus Layer (Raft Clusters)"
        subgraph "KV Raft Cluster"
            kv1["Server1"]:::consensus
            kv2["Server2"]:::consensus
            kv3["Server3"]:::consensus
        end
        subgraph "ShardCtrler Raft Cluster"
            sc1["Server1"]:::consensus
            sc2["Server2"]:::consensus
            sc3["Server3"]:::consensus
        end
        subgraph "ShardKV Raft Cluster"
            sk1["Server1"]:::consensus
            sk2["Server2"]:::consensus
            sk3["Server3"]:::consensus
        end
    end

    %% State Machines
    KVStateMachine["KVStateMachine"]:::state
    ConfigStateMachine["ConfigStateMachine"]:::state
    ShardKVStateMachine["ShardKVStateMachine"]:::state

    %% MapReduce Layer
    subgraph "MapReduce Layer"
        MRCoordinator["MR Coordinator"]:::mr
        MRWorker1["Worker1"]:::mr
        MRWorkerN["WorkerN"]:::mr
        DataStore["Text File Store"]:::external
    end

    %% Test Harnesses
    subgraph "Test Harnesses & Utilities"
        diskvd["diskvd"]:::test
        lockd["lockd/lockc"]:::test
        viewd["viewd"]:::test
        pbdc["pbd/pbc"]:::test
        mrMain["MR Drivers"]:::test
    end

    %% Verification Tool
    porcupine["Porcupine"]:::verify

    %% Infrastructure to Services
    labrpc -->|"RPC" | KVStateMachine
    labrpc -->|"RPC" | ConfigStateMachine
    labrpc -->|"RPC" | ShardKVStateMachine
    labgob -.-> KVStateMachine
    labgob -.-> ConfigStateMachine
    labgob -.-> ShardKVStateMachine

    %% Raft to StateMachines
    kv1 & kv2 & kv3 -->|"Apply(command)"| KVStateMachine
    sc1 & sc2 & sc3 -->|"Apply(command)"| ConfigStateMachine
    sk1 & sk2 & sk3 -->|"Apply(command)"| ShardKVStateMachine

    %% Shard Controller to ShardKV
    ConfigStateMachine -->|"ShardConfig RPC"| ShardKVStateMachine

    %% MapReduce flows
    labrpc -->|"RPC" | MRCoordinator
    MRCoordinator -->|"AssignTask"| MRWorker1
    MRCoordinator -->|"AssignTask"| MRWorkerN
    MRWorker1 -->|"Read"| DataStore
    MRWorkerN -->|"Write"| DataStore

    %% Test Harness interactions
    diskvd -->|"Test KV raft"| kv1
    lockd -->|"Test lock service"| KVStateMachine
    viewd -->|"Test view service"| ConfigStateMachine
    pbdc -->|"Test primary-backup"| KVStateMachine
    mrMain -->|"Run MR"| MRCoordinator

    %% Verification hooking
    porcupine -.->|"Logs"| labrpc
    porcupine -.->|"Logs"| kv1
    porcupine -.->|"Logs"| sc1
    porcupine -.->|"Logs"| MRCoordinator

    %% Click Events
    click labrpc "https://github.com/liuyuyan2717/6824lab/blob/main/src/labrpc/labrpc.go"
    click labgob "https://github.com/liuyuyan2717/6824lab/blob/main/src/labgob/labgob.go"
    click KVStateMachine "https://github.com/liuyuyan2717/6824lab/blob/main/src/kvraft/KVStateMachine.go"
    click ConfigStateMachine "https://github.com/liuyuyan2717/6824lab/blob/main/src/shardctrler/configStateMachine.go"
    click ShardKVStateMachine "https://github.com/liuyuyan2717/6824lab/blob/main/src/shardkv/server.go"
    click MRCoordinator "https://github.com/liuyuyan2717/6824lab/blob/main/src/mr/coordinator.go"
    click MRWorker1 "https://github.com/liuyuyan2717/6824lab/blob/main/src/mr/worker.go"
    click porcupine "https://github.com/liuyuyan2717/6824lab/blob/main/src/porcupine/porcupine.go"
    click diskvd "https://github.com/liuyuyan2717/6824lab/blob/main/src/main/diskvd.go"
    click lockd "https://github.com/liuyuyan2717/6824lab/blob/main/src/main/lockd.go"
    click viewd "https://github.com/liuyuyan2717/6824lab/blob/main/src/main/viewd.go"
    click pbdc "https://github.com/liuyuyan2717/6824lab/blob/main/src/main/pbd.go"
    click mrMain "https://github.com/liuyuyan2717/6824lab/blob/main/src/main/mrcoordinator.go"

    %% Styles
    classDef infra fill:#CCCCCC,stroke:#333,stroke-width:1px
    classDef consensus fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef state fill:#90EE90,stroke:#333,stroke-width:1px
    classDef mr fill:#FFA500,stroke:#333,stroke-width:1px
    classDef test fill:#FFFF00,stroke:#333,stroke-width:1px
    classDef verify fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#333,stroke-width:1px
```
