How casys-kaist/neupims works
Project type and purpose
• This project, “NeuPIMs Simulator,” is a cycle-accurate heterogeneous simulation tool combining an open-source NPU simulator (ONNXim) and an in-house PIM (Processing-in-Memory) simulator based on DRAMsim3.
• Its primary goal is to model batched large-language-model (LLM) inference on a hybrid NPU+PIM architecture, evaluating performance under various configurations.
File structure
Top-level files and folders:
• .gitignore, .gitmodules, CMakeLists.txt, Dockerfile, build.sh, brun.sh, conanfile.txt
• configs/
– memory_configs/ (e.g., neupims.json)
– model_configs/ (e.g., gpt3-7B.json, gpt3-175B.json)
– system_configs/ (e.g., sub-batch-off.json, systolic_ws_128x128_dev.json)
• extern/
– FlameGraph/
– NewtonSim/ (full DRAM simulation submodule: configs, src, include, scripts, tests)
– booksim/
• request-traces/ (pre-generated CSV workloads under clb/ and rr/)
• src/
– allocator/ (ActivationAllocator.cc, WeightAllocator.cc, KVCacheAllocator.cc, AddressAllocator.h)
– client/ (Client.cc, Client.h)
– helper/ (CommandLineParser.cc/h, HelperFunctions.h)
– operations/ (Add, Attention, MatMul, Softmax, fused ops, NeuPIMS-specific ops, etc.)
– scheduler/ (NeuPIMScheduler, OrcaScheduler, Scheduler core)
– tensor/ (BTensor, NPUTensor, PIMTensor, NPUTensor2D, PIM inner)
– main.cc and core modules (Simulator.cc/h, SimulationConfig.h, RequestGenerator.cc/h, ModelProgram.cc/h, NeuPIMSCore.cc/h, etc.)
• trace-generator/ (Python Jupyter/test scripts, subdirs alpaca/, share-gpt2/, share-gpt-prev/)
• README.md
README for additional insights
• Documents build methods (Docker vs. manual), dependencies (torch, conan, onnxruntime, cmake), compilation steps, and run scripts (build.sh, brun.sh).
• Describes three modes/branches: npu-only, npu+pim, and full NeuPIMs with dual row‐buffer and sub-batch interleaving.
a. Main components (each maps to one or more folders):
1. User Interface & CLI (src/client, src/helper, main.cc)
2. Configuration Manager (configs/* + src/SimulationConfig.h + trace-generator configs)
3. Trace Generator (trace-generator/ Python scripts → generates request CSVs or in-memory streams)
4. Request Generator (src/RequestGenerator.cc/h)
5. Simulation Core (src/Simulator.cc/h, src/Core.cc/h)
6. Scheduler Layer (src/scheduler/)
7. NPU Engine (ONNXim integration under src/operations, extern/ONNXim)
8. PIM Engine & Memory System
• PIM logic: src/NeuPIMSCore.cc/h, src/NeuPIMSystolicWS.cc/h, src/SystolicWS.cc/h, src/SystolicOS.cc/h
• DRAMsim3 integration: extern/NewtonSim/ (configs, src, include)
9. Interconnect (src/Interconnect.cc/h)
10. Allocators (src/allocator/)
11. Tensor Abstractions (src/tensor/)
12. Statistics & Logging (src/simple_stats, src/Stats.cpp, extern/FlameGraph scripts)
13. Build & Deployment (CMakeLists.txt, conanfile.txt, build.sh, brun.sh, Dockerfile)
b. Relationships and data flow
• The CLI parses flags (src/helper) → invokes the Simulator with a chosen mode.
• The Configuration Manager loads JSON/INI (configs/* or NewtonSim/configs) → populates SimulationConfig.
• Trace Generator (optional) produces request sequences → RequestGenerator ingests them.
• RequestGenerator produces batched requests → forwarded to Scheduler.
• Scheduler dispatches operations to NPU Engine (ONNX operations) or PIM Engine (tensor-in-memory operations).
• PIM Engine delegates DRAM requests to NewtonSim (extern/NewtonSim) via Interconnect.
• NPU Engine and PIM Engine both use Tensor & Allocator modules for data layout and memory.
• All components emit statistics → Stats & Logger produce output and optional FlameGraphs.
c. Key architectural patterns
• Modular layers (CLI, Config, Tracing, Scheduling, Execution engines)
• Plugin-style extern modules (ONNXim, NewtonSim)
• Separation of concerns: Core simulation vs. memory vs. compute
• Config-driven behavior (JSON / INI files)
• Modes as branches (npu-only, npu+pim, neuPIMs)
d. Technologies & frameworks
• C++ simulation core (CMake + Conan)
• Python scripts and Jupyter notebooks for trace generation
• DRAMsim3 (via extern/NewtonSim)
• ONNX runtime integration (extern/ONNXim)
• Docker for reproducible builds
Final recommendations
• Folder (configs, extern, request-traces, src/[allocator, client, helper, operations, scheduler, tensor], trace-generator) is represented as a component or subcomponent.
• Essential interactions without overcrowding: group related subfolders (e.g., src/tensor and src/allocator under a “Memory Abstraction” layer).