# casys-kaist/neupims architecture

> Architecture diagram of the GitHub repository casys-kaist/neupims, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/casys-kaist/neupims
- Repository: https://github.com/casys-kaist/neupims

## Overview

##### Project type and purpose
- This project, “NeuPIMs Simulator,” is a cycle-accurate heterogeneous simulation tool combining an open-source NPU simulator (ONNXim) and an in-house PIM (Processing-in-Memory) simulator based on DRAMsim3.
- Its primary goal is to model batched large-language-model (LLM) inference on a hybrid NPU+PIM architecture, evaluating performance under various configurations.

##### File structure
Top-level files and folders:
- .gitignore, .gitmodules, CMakeLists.txt, Dockerfile, build.sh, brun.sh, conanfile.txt
- configs/
– memory_configs/ (e.g., neupims.json)
– model_configs/ (e.g., gpt3-7B.json, gpt3-175B.json)
– system_configs/ (e.g., sub-batch-off.json, systolic_ws_128x128_dev.json)
- extern/
– FlameGraph/
– NewtonSim/ (full DRAM simulation submodule: configs, src, include, scripts, tests)
– booksim/
- request-traces/ (pre-generated CSV workloads under clb/ and rr/)
- src/
– allocator/ (ActivationAllocator.cc, WeightAllocator.cc, KVCacheAllocator.cc, AddressAllocator.h)
– client/ (Client.cc, Client.h)
– helper/ (CommandLineParser.cc/h, HelperFunctions.h)
– operations/ (Add, Attention, MatMul, Softmax, fused ops, NeuPIMS-specific ops, etc.)
– scheduler/ (NeuPIMScheduler, OrcaScheduler, Scheduler core)
– tensor/ (BTensor, NPUTensor, PIMTensor, NPUTensor2D, PIM inner)
– main.cc and core modules (Simulator.cc/h, SimulationConfig.h, RequestGenerator.cc/h, ModelProgram.cc/h, NeuPIMSCore.cc/h, etc.)
- trace-generator/ (Python Jupyter/test scripts, subdirs alpaca/, share-gpt2/, share-gpt-prev/)
- README.md

##### README for additional insights
- Documents build methods (Docker vs. manual), dependencies (torch, conan, onnxruntime, cmake), compilation steps, and run scripts (build.sh, brun.sh).
- Describes three modes/branches: npu-only, npu+pim, and full NeuPIMs with dual row‐buffer and sub-batch interleaving.

a. Main components (each maps to one or more folders):
1. User Interface & CLI (src/client, src/helper, main.cc)
2. Configuration Manager (configs/* + src/SimulationConfig.h + trace-generator configs)
3. Trace Generator (trace-generator/ Python scripts → generates request CSVs or in-memory streams)
4. Request Generator (src/RequestGenerator.cc/h)
5. Simulation Core (src/Simulator.cc/h, src/Core.cc/h)
6. Scheduler Layer (src/scheduler/)
7. NPU Engine (ONNXim integration under src/operations, extern/ONNXim)
8. PIM Engine & Memory System
- PIM logic: src/NeuPIMSCore.cc/h, src/NeuPIMSystolicWS.cc/h, src/SystolicWS.cc/h, src/SystolicOS.cc/h
- DRAMsim3 integration: extern/NewtonSim/ (configs, src, include)
9. Interconnect (src/Interconnect.cc/h)
10. Allocators (src/allocator/)
11. Tensor Abstractions (src/tensor/)
12. Statistics & Logging (src/simple_stats, src/Stats.cpp, extern/FlameGraph scripts)
13. Build & Deployment (CMakeLists.txt, conanfile.txt, build.sh, brun.sh, Dockerfile)

b. Relationships and data flow
- The CLI parses flags (src/helper) → invokes the Simulator with a chosen mode.
- The Configuration Manager loads JSON/INI (configs/* or NewtonSim/configs) → populates SimulationConfig.
- Trace Generator (optional) produces request sequences → RequestGenerator ingests them.
- RequestGenerator produces batched requests → forwarded to Scheduler.
- Scheduler dispatches operations to NPU Engine (ONNX operations) or PIM Engine (tensor-in-memory operations).
- PIM Engine delegates DRAM requests to NewtonSim (extern/NewtonSim) via Interconnect.
- NPU Engine and PIM Engine both use Tensor & Allocator modules for data layout and memory.
- All components emit statistics → Stats & Logger produce output and optional FlameGraphs.

c. Key architectural patterns
- Modular layers (CLI, Config, Tracing, Scheduling, Execution engines)
- Plugin-style extern modules (ONNXim, NewtonSim)
- Separation of concerns: Core simulation vs. memory vs. compute
- Config-driven behavior (JSON / INI files)
- Modes as branches (npu-only, npu+pim, neuPIMs)

d. Technologies & frameworks
- C++ simulation core (CMake + Conan)
- Python scripts and Jupyter notebooks for trace generation
- DRAMsim3 (via extern/NewtonSim)
- ONNX runtime integration (extern/ONNXim)
- Docker for reproducible builds

##### Final recommendations
- Folder (configs, extern, request-traces, src/[allocator, client, helper, operations, scheduler, tensor], trace-generator) is represented as a component or subcomponent.
- Essential interactions without overcrowding: group related subfolders (e.g., src/tensor and src/allocator under a “Memory Abstraction” layer).

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Build & Deployment"
        B1["Build & Deployment\n(CMakeLists.txt, conanfile.txt,\nbuild.sh, brun.sh, Dockerfile)"]:::build
    end

    subgraph "Configuration & Tracing"
        CMI["Configuration Manager\n(configs/, src/SimulationConfig.h)"]:::config
        TG["Trace Generator\n(trace-generator/)"]:::config
    end

    subgraph "User Interface & CLI"
        CLI["CLI & Helper\n(src/client, src/helper, src/main.cc)"]:::client
    end

    subgraph "Core Simulation"
        RG["Request Generator\n(src/RequestGenerator.cc, src/RequestGenerator.h)"]:::core
        SC["Simulation Core\n(src/Simulator.cc, src/Simulator.h,\nsrc/Core.cc, src/Core.h)"]:::core
        SCH["Scheduler Layer\n(src/scheduler/)"]:::core
        IC["Interconnect\n(src/Interconnect.cc, src/Interconnect.h)"]:::core
        ST["Statistics & Logging\n(src/Stats.cpp, src/Stat.h,\nsrc/Logger.h)"]:::core
    end

    subgraph "Compute Engines"
        NPU["NPU Engine\n(src/operations/)"]:::compute
        PIM["PIM Engine & Memory System\n(src/NeuPIMSCore*, src/NeuPIMSystolicWS*,\nsrc/SystolicWS*, src/SystolicOS*)"]:::compute
    end

    subgraph "Memory Abstraction"
        AL["Allocators\n(src/allocator/)"]:::memory
        TA["Tensor Abstractions\n(src/tensor/)"]:::memory
    end

    subgraph "External Modules"
        E1["NewtonSim (DRAMsim3)\n(extern/NewtonSim/)"]:::external
        E2["ONNXim\n(extern/ONNXim)"]:::external
        FG["FlameGraph\n(extern/FlameGraph/)"]:::external
    end

    %% Data Flow
    B1 --> CLI
    CLI --> CMI
    CLI --> SC
    CMI --> SC
    TG --> RG
    RG --> SCH
    SC --> SCH
    SCH --> NPU
    SCH --> PIM
    NPU --> AL
    NPU --> TA
    PIM --> IC
    PIM --> AL
    PIM --> TA
    IC --> E1
    NPU --> E2
    SC --> ST
    NPU --> ST
    PIM --> ST

    %% Click Events
    click CLI "https://github.com/casys-kaist/neupims/tree/main/src/client"
    click CLI "https://github.com/casys-kaist/neupims/tree/main/src/helper"
    click CLI "https://github.com/casys-kaist/neupims/blob/main/src/main.cc"
    click CMI "https://github.com/casys-kaist/neupims/tree/main/configs/"
    click CMI "https://github.com/casys-kaist/neupims/blob/main/src/SimulationConfig.h"
    click TG "https://github.com/casys-kaist/neupims/tree/main/trace-generator/"
    click RG "https://github.com/casys-kaist/neupims/blob/main/src/RequestGenerator.cc"
    click RG "https://github.com/casys-kaist/neupims/blob/main/src/RequestGenerator.h"
    click SC "https://github.com/casys-kaist/neupims/blob/main/src/Simulator.cc"
    click SC "https://github.com/casys-kaist/neupims/blob/main/src/Simulator.h"
    click SC "https://github.com/casys-kaist/neupims/blob/main/src/Core.cc"
    click SC "https://github.com/casys-kaist/neupims/blob/main/src/Core.h"
    click SCH "https://github.com/casys-kaist/neupims/tree/main/src/scheduler/"
    click IC "https://github.com/casys-kaist/neupims/blob/main/src/Interconnect.cc"
    click IC "https://github.com/casys-kaist/neupims/blob/main/src/Interconnect.h"
    click NPU "https://github.com/casys-kaist/neupims/tree/main/src/operations/"
    click PIM "https://github.com/casys-kaist/neupims/blob/main/src/NeuPIMSCore.cc"
    click PIM "https://github.com/casys-kaist/neupims/blob/main/src/NeuPIMSCore.h"
    click PIM "https://github.com/casys-kaist/neupims/blob/main/src/NeuPIMSystolicWS.cc"
    click PIM "https://github.com/casys-kaist/neupims/blob/main/src/NeuPIMSystolicWS.h"
    click PIM "https://github.com/casys-kaist/neupims/blob/main/src/SystolicWS.cc"
    click PIM "https://github.com/casys-kaist/neupims/blob/main/src/SystolicWS.h"
    click PIM "https://github.com/casys-kaist/neupims/blob/main/src/SystolicOS.cc"
    click PIM "https://github.com/casys-kaist/neupims/blob/main/src/SystolicOS.h"
    click E1 "https://github.com/casys-kaist/neupims/tree/main/extern/NewtonSim/"
    click E2 "https://github.com/casys-kaist/neupims/tree/main/extern/ONNXim"
    click FG "https://github.com/casys-kaist/neupims/tree/main/extern/FlameGraph/"
    click AL "https://github.com/casys-kaist/neupims/tree/main/src/allocator/"
    click TA "https://github.com/casys-kaist/neupims/tree/main/src/tensor/"
    click ST "https://github.com/casys-kaist/neupims/blob/main/src/Stats.cpp"
    click ST "https://github.com/casys-kaist/neupims/blob/main/src/Stat.h"
    click ST "https://github.com/casys-kaist/neupims/blob/main/src/Logger.h"
    click B1 "https://github.com/casys-kaist/neupims/blob/main/CMakeLists.txt"
    click B1 "https://github.com/casys-kaist/neupims/blob/main/conanfile.txt"
    click B1 "https://github.com/casys-kaist/neupims/blob/main/build.sh"
    click B1 "https://github.com/casys-kaist/neupims/blob/main/brun.sh"
    click B1 "https://github.com/casys-kaist/neupims/tree/main/Dockerfile"

    %% Styles
    classDef build fill:#cccccc,stroke:#333,stroke-width:1px
    classDef config fill:#cce5ff,stroke:#3385ff,stroke-width:1px
    classDef client fill:#e6ffe6,stroke:#33cc33,stroke-width:1px
    classDef core fill:#d9f2d9,stroke:#28a428,stroke-width:1px
    classDef compute fill:#fff9c4,stroke:#ffeb3b,stroke-width:1px
    classDef memory fill:#ffecb3,stroke:#ffa000,stroke-width:1px
    classDef external fill:#ffe0b2,stroke:#ff9800,stroke-width:1px
```
