How project-hami/hami-core works
Project Type and Purpose
• HAMi-core is a native C hook-library for CUDA environments.
• Its purpose is to virtualize device memory, enforce per-GPU memory and SM-utilization limits, and provide real-time monitoring by intercepting CUDA Runtime → CUDA Driver API calls (via LD_PRELOAD).
File Structure Analysis
• Top-level build: CMakeLists.txt, Makefile, build.sh, dockerfiles/
• src/allocator: custom GPU memory allocator
• src/cuda: hooks for CUDA Runtime API (context, device, stream, event, graph, memory)
• src/nvml: NVML API hooks for monitoring and driver queries
• src/multiprocess: auxiliary processes for enforcement (memory_limit, utilization_watcher, shrreg_tool)
• src/include: shared headers (hook interfaces, utilities, config)
• src/libvgpu.c, src/utils.c: library entry, initialization, common logic
• test/: unit tests for allocators, host/device allocations, Python scripts for integration tests
README Insights
a) Overall HAMi architecture (container-level)
b) HAMi-core position between libcudart.so and libcuda.so
• Shows LD_PRELOAD, environment variables, cache directory, docker usage, logging levels
Components
a) Client Application
• A CUDA application or framework (e.g., TensorFlow, PyTorch, MXNet)
b) Hook Library (libvgpu.so)
• Initialization / Entry (libvgpu.c)
• Module: allocator (virtual memory management)
• Module: SM limiter (time-slice enforcement)
• Module: monitor (NVML hook + utilization watcher)
• Module: multiprocess helpers (spawned daemons)
c) CUDA Runtime Library (libcudart.so)
d) CUDA Driver Library (libcuda.so)
e) NVML Library (libnvidia-ml.so)
f) Shared Cache / Lock Directory (/tmp/vgpulock, /tmp/cudevshr.cache)
g) Docker Container / Host OS / GPU Device
Relationships and Data Flows
• Client App → (CUDA API call) → libcudart.so
• libcudart.so → (hook) → libvgpu.so
• Intercept cudaMalloc, cudaFree, cudaLaunchKernel, etc.
• Custom logic (check memory limits, queue time-slice, log)
• Forward call → libcuda.so
• libvgpu.so → NVML hooks → libnvidia-ml.so for real-time utilization
• multiprocess_memory_limit & multiprocess_utilization_watcher ↔ shared memory or cache files to coordinate between processes
• Host OS → GPU driver → Physical GPU
Architectural Patterns and Principles
• LD_PRELOAD hooking (dynamic link preemptive overrides)
• Modular C components (allocator, hook layers, monitor)
• Daemonized enforcement processes for isolation
• Shared‐cache / file‐lock for cross‐process synchronization
Specific Items
• Docker context (optional): show how container mounts libraries and devices
• Two interception points: cuda runtime → hook → driver; NVML hooks
• Inter‐module interactions within libvgpu.so (allocator ↔ monitor)
• Flow of memory allocation requests vs. monitoring events