# k-wu/flashtrain architecture

> Architecture diagram of the GitHub repository k-wu/flashtrain, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/k-wu/flashtrain
- Repository: https://github.com/k-wu/flashtrain

## Overview

##### Main components
- Core FlashTrain Module: Begin with the “flashtrain” package as the central component. This module initializes the framework and contains shared utilities (for example, flashtrain/__init__.py and flashtrain/logger.py).
- Malloc Hook / Memory Allocation: Highlight the “flashtrain/malloc_hook” directory. This is a custom C++ module built via a shell script (make.sh) that wraps CUDA memory allocations to register them with the cuFile system. Note that this component is critical for optimizing pinned GPU memory transfers.
- Tensor Cache Subsystem: Represent the “flashtrain/tensor_cache” folder as a major subsystem. Within it, include:
– The core tensor caching logic (e.g., tensor_cache.py, pipeline_tensor_cache.py)
– Adapters, configuration scripts (configs directory), and utilities for managing namespaces and RAID setups for stored tensors.
– The reevaluator submodule, which further optimizes/reevaluates tensor offloading (flashtrain/tensor_cache/reevaluator).
– The custom hooks and monkey patched DeepSpeed checkpointing support in this module.
– Megatron-DeepSpeed, which must be installed separately and is referenced in the README.
– Customized scripts (found in third_party/customized_scripts) and delta scripts for distributed training.
– Optionally, Transformer-Engine for further optimizations.
- Testing & Validation: Include a component for the “flashtrain/tests” repository that covers multiple testing modules (unit tests for custom allocators, integrated tests with HuggingFace models, multiprocessing tests, etc.).

##### Diagramming the relationships and interactions
- Center the flashtrain core module and illustrate how it calls/initializes both the malloc hook (for optimized memory allocation) and the tensor cache system.
- Malloc hook to the GPU direct storage and show its relationship to asynchronous file I/O (supported by components in flashtrain/docs such as ASYNC_FILE_IO.md and GPUDIRECT_STORAGE.md). This indicates how raw memory allocations transition into enhanced I/O performance.
- Testing components as ancillary blocks that interact with nearly every module—illustrating that tests validate functionalities like distributed parallelism, memory allocation, and tensor caching behaviors.

##### Architectural patterns and design principles
- Modular design: Each major function (memory allocation, tensor caching, external integrations) is neatly partitioned in its own directory.
- Asynchronous/Parallel processing: With support for multiprocess parallelism (as evident in both docs and tests), note the importance of async file I/O and multiprocessing in the design.
- Customization and extensibility: The extensive set of configuration scripts in tensor_cache/configs and the third-party scripts exemplify extensibility and system customization tailored to high-performance computing environments.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% FlashTrain Core Module
    subgraph "FlashTrain Core Module"
        CoreInit["FlashTrain Core - __init__"]:::core
        CoreLogger["FlashTrain Logger"]:::core
    end

    %% Custom Malloc Hook
    subgraph "Custom Malloc Hook"
        MallocCPP["Malloc Hook - hook.cpp"]:::malloc
        MallocMake["Malloc Hook - make.sh"]:::malloc
    end

    %% Tensor Cache Subsystem
    subgraph "Tensor Cache Subsystem"
        TensorCache["Tensor Cache - tensor_cache.py"]:::tensor
        PipelineCache["Tensor Cache - pipeline_tensor_cache.py"]:::tensor
        TensorAdapters["Tensor Cache Adapters & Utils"]:::tensor
        TensorConfigs["Tensor Cache Configs"]:::tensor
        TensorReevaluator["Tensor Cache Reevaluator"]:::tensor
    end

    %% Third Party Integration
    subgraph "Third Party Integration"
        ThirdPartyMeg["Megatron-DeepSpeed"]:::external
        ThirdPartyCustom["Customized Scripts"]:::external
        ThirdPartyDelta["Delta Scripts"]:::external
        ExternalExtras["Kvikio & Transformer-Engine"]:::external
    end

    %% Testing & Validation
    Testing["Tests & Validation"]:::testing

    %% Connections between components
    CoreInit -->|"register_allocations"| MallocCPP
    CoreInit -->|"initialize_tensor_cache"| TensorCache
    TensorReevaluator -->|"deepspeed_API_call"| ThirdPartyMeg
    CoreInit -->|"external_integration"| ExternalExtras

    %% Testing relationships
    Testing ---|"unit_tests"| CoreInit
    Testing ---|"allocator_tests"| MallocCPP
    Testing ---|"tensor_cache_tests"| TensorCache
    Testing ---|"integration_tests"| ThirdPartyMeg

    %% Styles
    classDef core fill:#AED6F1,stroke:#000,stroke-width:2px;
    classDef malloc fill:#F9E79F,stroke:#000,stroke-width:2px;
    classDef tensor fill:#D6EAF8,stroke:#000,stroke-width:2px;
    classDef external fill:#ABEBC6,stroke:#000,stroke-width:2px;
    classDef testing fill:#F5B7B1,stroke:#000,stroke-width:2px;

    %% Click Events for FlashTrain Core Module
    click CoreInit "https://github.com/k-wu/flashtrain/blob/main/flashtrain/__init__.py"
    click CoreLogger "https://github.com/k-wu/flashtrain/blob/main/flashtrain/logger.py"

    %% Click Events for Malloc Hook
    click MallocCPP "https://github.com/k-wu/flashtrain/blob/main/flashtrain/malloc_hook/hook.cpp"
    click MallocMake "https://github.com/k-wu/flashtrain/blob/main/flashtrain/malloc_hook/make.sh"

    %% Click Events for Tensor Cache Subsystem – Core Logic
    click TensorCache "https://github.com/k-wu/flashtrain/blob/main/flashtrain/tensor_cache/tensor_cache.py"
    click PipelineCache "https://github.com/k-wu/flashtrain/blob/main/flashtrain/tensor_cache/pipeline_tensor_cache.py"

    %% Click Events for Tensor Cache Subsystem – Adapters & Utilities
    click TensorAdapters "https://github.com/k-wu/flashtrain/blob/main/flashtrain/tensor_cache/adapters.py"
    click TensorAdapters "https://github.com/k-wu/flashtrain/blob/main/flashtrain/tensor_cache/scope_tree.py"
    click TensorAdapters "https://github.com/k-wu/flashtrain/blob/main/flashtrain/tensor_cache/uniqueid_utils.py"
    click TensorAdapters "https://github.com/k-wu/flashtrain/blob/main/flashtrain/tensor_cache/utils.py"

    %% Click Events for Tensor Cache Subsystem – Configuration Scripts
    click TensorConfigs "https://github.com/k-wu/flashtrain/tree/main/flashtrain/tensor_cache/configs"

    %% Click Events for Tensor Cache Subsystem – Reevaluator Submodule
    click TensorReevaluator "https://github.com/k-wu/flashtrain/tree/main/flashtrain/tensor_cache/reevaluator"

    %% Click Events for Third Party Integration
    click ThirdPartyMeg "https://github.com/k-wu/flashtrain/tree/main/third_party/Megatron-DeepSpeed"
    click ThirdPartyCustom "https://github.com/k-wu/flashtrain/tree/main/third_party/customized_scripts"
    click ThirdPartyDelta "https://github.com/k-wu/flashtrain/tree/main/third_party/delta_scripts"

    %% Click Event for Testing & Validation
    click Testing "https://github.com/k-wu/flashtrain/tree/main/flashtrain/tests"
```
