# ghassen-fatnassi/fast architecture

> Architecture diagram of the GitHub repository ghassen-fatnassi/fast, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ghassen-fatnassi/fast
- Repository: https://github.com/ghassen-fatnassi/fast

## Overview

##### Project type and purpose
- This is a monorepo of educational labs for two domains:
a. MIT 6.5840 (Distributed Systems) labs under MIT6.5840/src
b. Stanford CS 149 (Parallel Computing) labs under StanfordCS149
- Each subdirectory is a self-contained lab or utility (e.g., kvraft, raft, MapReduce, tasksys, ISPC renderers).

##### File structure
- Top-level directories: MIT6.5840 and StanfordCS149
- MIT6.5840/src contains:
- kvraft, kvsrv, shardctrler, shardkv – key/value stores & sharding controllers
- raft – consensus algorithm
- mr & main – MapReduce coordinator/worker pipelines
- labrpc, labgob – RPC and serialization libraries
- porcupine – consistency checker/visualizer
- StanfordCS149/LAB1 and LAB2 contain multiple progN_xxx directories with C++ and ISPC code, Makefiles, and object artifacts.

##### README
- Confirms two course tracks: distributed systems (labs 1–5) and parallel computing (labs 1–5)
- Author’s goal: learn-by-doing; each lab implements a self-contained system or library.

a. Main components
- Distributed-systems track:
1. RPC library (labrpc) & serialization (labgob)
2. Raft servers + persister
3. Key/Value servers (kvraft & kvsrv)
4. Shard controller cluster (shardctrler)
5. Sharded KV clusters (shardkv)
6. MapReduce: Coordinator, Worker, Client scripts
7. Consistency checker (porcupine)
- Parallel-computing track:
1. Task execution library (tasksys cpp)
2. Mandelbrot/CUDA/ISPC renderers (prog1–prog3)
3. Math kernels (sqrt, saxpy)
b. Relationships & interactions
- Clients → RPC layer → Servers (raft, kv, shardctrler)
- Coordinator ↔ Workers via RPC
- Raft servers replicate logs → persister
- Shard controller issues shard maps → KV servers
- Parallel labs: main app → tasksys → worker threads / ISPC kernels
c. Architectural patterns & principles
- Modular service layers: RPC / serialization separated from business logic
- Consensus (leader election, log replication)
- Sharding for scalability
- Master-worker for MapReduce
- Thread pool and data parallel kernels for compute labs
d. Key technologies/frameworks
- Go modules for MIT labs (go.mod)
- C++/ISPC + Makefiles for Stanford labs
- Shell scripts for test harnesses

##### Elements
- Labels: component name + primary language/runtime
- Rectangles for services/binaries
- Distinct hues for MIT vs. Stanford domains

##### Detailed capture of essential elements
- Within Distributed lane, sub-lanes for RPC, consensus, KV, sharding, MapReduce, and checker
- Within Parallel lane, sub-lanes for tasksys, renderer apps, and math kernels
- Lab folder as a subsystem, with input/output artifacts (e.g., .ppm images)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Distributed Systems Track
    subgraph "Distributed Systems Track"
        direction TB
        %% RPC & Serialization
        subgraph RPC_Serialization
            direction TB
            LabRPC["labrpc (Go)"]:::rpc
            LabGOB["labgob (Go)"]:::rpc
        end

        %% Consensus (Raft)
        subgraph Consensus_Cluster
            direction TB
            Raft["Raft Server"]:::consensus
            Persister["Persister (Log Storage)"]:::consensus
        end

        %% Key/Value Services
        subgraph Key_Value_Services
            direction TB
            KVraftServ["kvraft Server"]:::kv
            KVraftClient["kvraft Client"]:::kv
            KVSrvServ["kvsrv Server"]:::kv
            KVSrvClient["kvsrv Client"]:::kv
        end

        %% Sharding
        subgraph Sharding_Cluster
            direction TB
            ShardCtlServ["shardctrler Server"]:::sharding
            ShardCtlClient["shardctrler Client"]:::sharding
            ShardKVServ["shardkv Server"]:::sharding
            ShardKVClient["shardkv Client"]:::sharding
        end

        %% MapReduce
        subgraph MapReduce_Workflow
            direction TB
            MRCoord["Coordinator"]:::mapreduce
            MRWorker["Worker"]:::mapreduce
            MRSeq["Sequential Client"]:::mapreduce
            MRTest["Test Harness"]:::mapreduce
        end

        %% Consistency Checker
        subgraph Consistency_Checker
            direction TB
            Porcupine["Porcupine Checker"]:::checker
            Visualization["Visualization"]:::checker
        end
    end

    %% Parallel Computing Track
    subgraph "Parallel Computing Track"
        direction TB
        %% Task Execution Library
        subgraph TaskSys_Lib
            direction TB
            TaskSysCPP["tasksys.cpp"]:::parallel-lib
            CycleTimer["CycleTimer.h"]:::parallel-lib
        end

        %% Renderer Applications
        subgraph Renderers
            direction TB
            MandelThreadMain["mandelbrotThread main.cpp"]:::renderer
            MandelThreadSrc["mandelbrotThread.cpp"]:::renderer
            VecIntrinCPP["CS149intrin.cpp"]:::renderer
            VecIntrinH["CS149intrin.h"]:::renderer
            ISPCMain["mandelbrot.ispc"]:::renderer
            ISPCApp["prog3 main.cpp"]:::renderer
        end

        %% Kernels
        subgraph Math_Kernels
            direction TB
            SqrtSerial["sqrtSerial.cpp"]:::kernel
            SqrtISPC["sqrt.ispc"]:::kernel
            SaxpySerial["saxpySerial.cpp"]:::kernel
            SaxpyISPC["saxpy.ispc"]:::kernel
        end
    end

    %% Data Flows & Relationships
    ClientApps(["Client Apps"]):::external
    ClientApps -->|uses RPC| LabRPC
    LabRPC -->|calls| Raft
    Raft -->|persists| Persister
    LabRPC -->|RPC| KVraftServ
    KVraftClient -->|RPC| LabRPC
    LabRPC -->|RPC| KVSrvServ
    KVSrvClient -->|RPC| LabRPC
    ShardCtlClient -->|RPC| LabRPC
    LabRPC -->|RPC| ShardCtlServ
    ShardCtlServ -->|shard map| ShardKVServ
    ShardKVClient -->|RPC| LabRPC
    MRSeq -->|invokes| MRCoord
    MRCoord -->|assign tasks| MRWorker
    MRWorker -->|report results| MRCoord
    Porcupine -->|analyzes logs| Persister
    MRTest -->|runs tests| MRCoord

    TaskSysCPP -->|spawns threads| MandelThreadMain
    MandelThreadMain -->|launch threads| MandelThreadSrc
    TaskSysCPP -->|calls| ISPCApp
    TaskSysCPP -->|calls| VecIntrinCPP
    SqrtSerial -->|compares| SqrtISPC
    SaxpySerial -->|compares| SaxpyISPC

    %% Click Events
    click LabRPC "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/labrpc/labrpc.go"
    click LabGOB "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/labgob/labgob.go"
    click Raft "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/raft/raft.go"
    click Persister "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/raft/persister.go"
    click KVraftServ "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/kvraft/server.go"
    click KVraftClient "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/kvraft/client.go"
    click KVSrvServ "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/kvsrv/server.go"
    click KVSrvClient "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/kvsrv/client.go"
    click ShardCtlServ "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/shardctrler/server.go"
    click ShardCtlClient "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/shardctrler/client.go"
    click ShardKVServ "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/shardkv/server.go"
    click ShardKVClient "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/shardkv/client.go"
    click MRCoord "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/main/mrcoordinator.go"
    click MRWorker "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/main/mrworker.go"
    click MRSeq "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/main/mrsequential.go"
    click MRTest "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/main/test-mr.sh"
    click Porcupine "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/porcupine/porcupine.go"
    click Visualization "https://github.com/ghassen-fatnassi/fast/blob/main/MIT6.5840/src/porcupine/visualization.go"
    click TaskSysCPP "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/common/tasksys.cpp"
    click CycleTimer "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/common/CycleTimer.h"
    click MandelThreadMain "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog1_mandelbrot_threads/main.cpp"
    click MandelThreadSrc "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog1_mandelbrot_threads/mandelbrotThread.cpp"
    click VecIntrinCPP "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog2_vecintrin/CS149intrin.cpp"
    click VecIntrinH "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog2_vecintrin/CS149intrin.h"
    click ISPCMain "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog3_mandelbrot_ispc/mandelbrot.ispc"
    click ISPCApp "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog3_mandelbrot_ispc/main.cpp"
    click SqrtSerial "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog4_sqrt/sqrtSerial.cpp"
    click SqrtISPC "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog4_sqrt/sqrt.ispc"
    click SaxpySerial "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog5_saxpy/saxpySerial.cpp"
    click SaxpyISPC "https://github.com/ghassen-fatnassi/fast/blob/main/StanfordCS149/LAB1/prog5_saxpy/saxpy.ispc"

    %% Styles
    classDef rpc fill:#D0E8FF,stroke:#3B82F6,color:#1E40AF,stroke-width:1px;
    classDef consensus fill:#DCFCE7,stroke:#10B981,color:#065F46,stroke-width:1px;
    classDef kv fill:#FEF3C7,stroke:#F59E0B,color:#92400E,stroke-width:1px;
    classDef sharding fill:#F5F3FF,stroke:#A78BFA,color:#4C1D95,stroke-width:1px;
    classDef mapreduce fill:#E0FCFF,stroke:#06B6D4,color:#065F46,stroke-width:1px;
    classDef checker fill:#FFE4E6,stroke:#F43F5E,color:#881337,stroke-width:1px;
    classDef parallel-lib fill:#FEF2F2,stroke:#EF4444,color:#7F1D1D,stroke-width:1px;
    classDef renderer fill:#F0FDF4,stroke:#10B981,color:#065F46,stroke-width:1px;
    classDef kernel fill:#F7FEE7,stroke:#84CC16,color:#365314,stroke-width:1px;
    classDef external fill:#F3F4F6,stroke:#9CA3AF,color:#374151,stroke-width:1px,stroke-dasharray: 5 5;
```
