# project-hami/hami-core architecture

> Architecture diagram of the GitHub repository project-hami/hami-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/project-hami/hami-core
- Repository: https://github.com/project-hami/hami-core

## Overview

##### Project Type and Purpose
- HAMi-core is a native C hook-library for CUDA environments.
- Its purpose is to virtualize device memory, enforce per-GPU memory and SM-utilization limits, and provide real-time monitoring by intercepting CUDA Runtime → CUDA Driver API calls (via LD_PRELOAD).

##### File Structure Analysis
- Top-level build: CMakeLists.txt, Makefile, build.sh, dockerfiles/
- src/allocator: custom GPU memory allocator
- src/cuda: hooks for CUDA Runtime API (context, device, stream, event, graph, memory)
- src/nvml: NVML API hooks for monitoring and driver queries
- src/multiprocess: auxiliary processes for enforcement (memory_limit, utilization_watcher, shrreg_tool)
- src/include: shared headers (hook interfaces, utilities, config)
- src/libvgpu.c, src/utils.c: library entry, initialization, common logic
- test/: unit tests for allocators, host/device allocations, Python scripts for integration tests

##### README Insights
a) Overall HAMi architecture (container-level)
b) HAMi-core position between libcudart.so and libcuda.so
- Shows LD_PRELOAD, environment variables, cache directory, docker usage, logging levels

##### Components
a) Client Application
- A CUDA application or framework (e.g., TensorFlow, PyTorch, MXNet)
b) Hook Library (libvgpu.so)
- Initialization / Entry (libvgpu.c)
- Module: allocator (virtual memory management)
- Module: SM limiter (time-slice enforcement)
- Module: monitor (NVML hook + utilization watcher)
- Module: multiprocess helpers (spawned daemons)
c) CUDA Runtime Library (libcudart.so)
d) CUDA Driver Library (libcuda.so)
e) NVML Library (libnvidia-ml.so)
f) Shared Cache / Lock Directory (/tmp/vgpulock, /tmp/cudevshr.cache)
g) Docker Container / Host OS / GPU Device

##### Relationships and Data Flows
- Client App → (CUDA API call) → libcudart.so
- libcudart.so → (hook) → libvgpu.so
- Intercept cudaMalloc, cudaFree, cudaLaunchKernel, etc.
- Custom logic (check memory limits, queue time-slice, log)
- Forward call → libcuda.so
- libvgpu.so → NVML hooks → libnvidia-ml.so for real-time utilization
- multiprocess_memory_limit & multiprocess_utilization_watcher ↔ shared memory or cache files to coordinate between processes
- Host OS → GPU driver → Physical GPU

##### Architectural Patterns and Principles
- LD_PRELOAD hooking (dynamic link preemptive overrides)
- Modular C components (allocator, hook layers, monitor)
- Daemonized enforcement processes for isolation
- Shared‐cache / file‐lock for cross‐process synchronization

##### Specific Items
- Docker context (optional): show how container mounts libraries and devices

- Two interception points: cuda runtime → hook → driver; NVML hooks
- Inter‐module interactions within libvgpu.so (allocator ↔ monitor)
- Flow of memory allocation requests vs. monitoring events

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Layer: Client Application
    ClientApp["Client Application\n(e.g., TensorFlow, PyTorch)"]:::user

    %% Layer 1: CUDA Runtime
    CUDART["libcudart.so"]:::external

    %% Layer 2: Hook Library with Modules
    subgraph "Hook Library (libvgpu.so)" 
        direction TB
        Init["Initialization / Entry\n(libvgpu.c)"]:::hook
        Allocator["Allocator Module\n(virtual memory)"]:::hook
        SMLimiter["SM Limiter Module\n(time-slice enforcement)"]:::hook
        Monitor["Monitor Module\n(NVML hooks)"]:::hook
        subgraph "Multiprocess Helpers"
            direction TB
            MemDaemon["multiprocess_memory_limit"]:::aux
            UtilDaemon["multiprocess_utilization_watcher"]:::aux
            ShrReg["shrreg_tool"]:::aux
        end
        Utils["Common Utilities\n(utils.c)"]:::hook
    end

    %% Layer 3: CUDA Driver
    LCuda["libcuda.so"]:::external
    GPUDriver["GPU Driver"]:::external
    GPUHardware["Physical GPU"]:::external

    %% Side: NVML Library & Shared Cache
    NVML["libnvidia-ml.so\n(NVML API)"]:::external
    SharedCache["Shared Cache / Lock Dir\n(/tmp/vgpulock, /tmp/cudevshr.cache)"]:::aux

    %% Environment Note
    EnvVars["Env Vars: LD_PRELOAD,\nCUDA_DEVICE_MEMORY_LIMIT"]:::note

    %% Connections: API Flow
    ClientApp -->|"CUDA API call"| CUDART
    CUDART -->|"hooked calls"| Init
    Init --> Allocator
    Init --> SMLimiter
    Init --> Monitor
    Init --> Utils
    Init --> MemDaemon
    Init --> UtilDaemon
    Init --> ShrReg
    Allocator -->|"forward call"| LCuda
    SMLimiter -->|"forward call"| LCuda
    Monitor -->|"runtime util"| NVML
    LCuda --> GPUDriver
    GPUDriver --> GPUHardware

    %% IPC and Monitoring Flows
    Monitor -.->|"async monitor"| SharedCache
    MemDaemon -.->|"IPC via files"| SharedCache
    UtilDaemon -.->|"IPC via files"| SharedCache
    ShrReg -.->|"register updates"| SharedCache
    EnvVars --- Init

    %% Click Events
    click Init "https://github.com/project-hami/hami-core/blob/main/src/libvgpu.c"
    click Utils "https://github.com/project-hami/hami-core/blob/main/src/utils.c"
    click Allocator "https://github.com/project-hami/hami-core/blob/main/src/allocator/allocator.c"
    click Allocator "https://github.com/project-hami/hami-core/blob/main/src/allocator/allocator.h"
    click MemDaemon "https://github.com/project-hami/hami-core/blob/main/src/multiprocess/multiprocess_memory_limit.c"
    click MemDaemon "https://github.com/project-hami/hami-core/blob/main/src/multiprocess/multiprocess_memory_limit.h"
    click UtilDaemon "https://github.com/project-hami/hami-core/blob/main/src/multiprocess/multiprocess_utilization_watcher.c"
    click UtilDaemon "https://github.com/project-hami/hami-core/blob/main/src/multiprocess/multiprocess_utilization_watcher.h"
    click ShrReg "https://github.com/project-hami/hami-core/blob/main/src/multiprocess/shrreg_tool.c"
    click NVML "https://github.com/project-hami/hami-core/blob/main/src/nvml/hook.c"
    click NVML "https://github.com/project-hami/hami-core/blob/main/src/nvml/nvml_entry.c"
    click CUDART "https://github.com/project-hami/hami-core/blob/main/src/cuda/hook.c"
    click CUDART "https://github.com/project-hami/hami-core/blob/main/src/cuda/context.c"
    click CUDART "https://github.com/project-hami/hami-core/blob/main/src/cuda/device.c"
    click CUDART "https://github.com/project-hami/hami-core/blob/main/src/cuda/event.c"
    click CUDART "https://github.com/project-hami/hami-core/blob/main/src/cuda/graph.c"
    click CUDART "https://github.com/project-hami/hami-core/blob/main/src/cuda/memory.c"
    click CUDART "https://github.com/project-hami/hami-core/blob/main/src/cuda/stream.c"

    %% Styling
    classDef user fill:#cce5ff,stroke:#004085,color:#004085;
    classDef hook fill:#d4edda,stroke:#155724,color:#155724;
    classDef external fill:#fff3cd,stroke:#856404,color:#856404;
    classDef aux fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef note fill:#f8d7da,stroke:#721c24,color:#721c24;
```
