# rwkv/rwkv-lm architecture

> Architecture diagram of the GitHub repository rwkv/rwkv-lm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rwkv/rwkv-lm
- Repository: https://github.com/rwkv/rwkv-lm

## Overview

##### Main System Components
- Model Core (src): In every version, the src folder contains key files: model.py (defining the network architecture, including time-mix and channel-mix operations, custom recurrent formulations, and various initialization tricks), trainer.py (the training routine and integration with PyTorch Lightning), and utils.py (helper functions for data processing, state management, etc.). Represent these as a “Model Core” subcomponent that implements the RWKV’s unique RNN-style / transformer-parallelizable design.
- CUDA Accelerated Modules: Notice that several versions (v2, v3, v4, v4neo, v5, and v7) include a dedicated “cuda” directory with custom CUDA kernels (e.g., timex_cuda.cu, timex_op.cpp, wkv_cuda.cu, etc.). These modules are critical to speeding up core computations. They should be diagrammed as an “Accelerated Computation Layer” linked to the model’s core compute routines.
- Training & Inference Scripts: Files like train.py, run.py, and demo scripts (rwkv_demo*, verify.py, chat.py) serve as the entry points for training and inference. They orchestrate the use of the model core and launch the corresponding routines. Represent a “Training/Inference Orchestration” component that interfaces with the core model and also triggers CUDA kernels when available.

##### Relationships and Data Flow
- The Data Pipeline (tokenizers and preparation scripts) feeds training data (large text corpora, e.g., enwik8 or jsonl files) into the Training/Orchestration system.
- The training scripts (train.py, demo-training-run.sh, etc.) pass data to the Model Core, which is implemented in model.py using custom layers (e.g., time-mix and channel-mix) and the special recurrent formulation (exploiting time-decay factors and “receptance” gates) as detailed in the README.
- For performance, when running on GPU, the system “offloads” or integrates with the specialized CUDA kernels (visible in the cuda folders) to accelerate sequential operations in the model (e.g., efficient computation of time-weighted summations).
- Outputs from the model then flow into inference scripts (run.py, chat.py, etc.) and evaluation routines, with feedback loops (as in training via trainer.py) managing state propagation across tokens.
- Optionally, showing how the model’s hidden state is handled (passed/reused between tokens) would illustrate the constant-space, recurrent aspect of RWKV.

##### Architectural Patterns and Design Principles
- Pipeline Structure: There is a clear separation between data pre-processing, model architecture (core computation), training/inference orchestration, and acceleration (CUDA). This follows a typical deep learning system’s layered design.
- Custom Optimizations: The design exploits specialized initialization tricks, new sampling methods, and state tuning. You could note (in annotations) the presence of “tricks” like token-shift mixing, innovative initialization, and RK techniques that optimize the training dynamics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% RWKV Versions Cluster
    subgraph "RWKV Versions"
        V1["RWKV-v1"]:::versions
        V2["RWKV-v2-RNN"]:::versions
        V3["RWKV-v3"]:::versions
        V4["RWKV-v4"]:::versions
        V4n["RWKV-v4neo"]:::versions
        V5["RWKV-v5"]:::versions
        V6["RWKV-v6"]:::versions
        V7["RWKV-v7"]:::versions
    end

    %% Common Components
    DataPrep["Data Preparation (Tokenizer & JSON to BinIDX)"]:::data
    TrainingInfer["Training/Inference Orchestration (train.py, run.py, chat.py)"]:::training
    ModelCore["Model Core (RWKV – time mix, channel mix, state handling)"]:::modelcore
    CudaLayer["Custom CUDA Kernel Layer (Kernel Acceleration)"]:::cuda
    External["External Services / Libraries"]:::external

    %% Data Flow Connections
    DataPrep -->|"feeds"| TrainingInfer
    TrainingInfer -->|"triggers"| ModelCore
    ModelCore -->|"accelerates"| CudaLayer
    ModelCore -->|"state_updates"| TrainingInfer
    External -->|"supports"| TrainingInfer

    %% Representative connections from RWKV Versions
    V1 -->|"ex: training_scripts"| TrainingInfer
    V1 -->|"ex: model_core"| ModelCore

    %% Styles
    classDef versions fill:#d9ead3,stroke:#000,stroke-width:1px;
    classDef data fill:#ffcccc,stroke:#000,stroke-width:1px;
    classDef training fill:#ccffcc,stroke:#000,stroke-width:1px;
    classDef modelcore fill:#cce5ff,stroke:#000,stroke-width:1px;
    classDef cuda fill:#fff2cc,stroke:#000,stroke-width:1px;
    classDef external fill:#e6ccff,stroke:#000,stroke-width:1px;

    %% Click Events for RWKV Versions
    click V1 "https://github.com/rwkv/rwkv-lm/tree/main/RWKV-v1"
    click V2 "https://github.com/rwkv/rwkv-lm/tree/main/RWKV-v2-RNN"
    click V3 "https://github.com/rwkv/rwkv-lm/tree/main/RWKV-v3"
    click V4 "https://github.com/rwkv/rwkv-lm/tree/main/RWKV-v4"
    click V4n "https://github.com/rwkv/rwkv-lm/tree/main/RWKV-v4neo"
    click V5 "https://github.com/rwkv/rwkv-lm/tree/main/RWKV-v5"
    click V6 "https://github.com/rwkv/rwkv-lm/tree/main/RWKV-v6"
    click V7 "https://github.com/rwkv/rwkv-lm/tree/main/RWKV-v7"

    %% Click Events for Common Components
    click DataPrep "https://github.com/rwkv/rwkv-lm/blob/main/RWKV-v5/make_data.py"
    click TrainingInfer "https://github.com/rwkv/rwkv-lm/blob/main/RWKV-v5/train.py"
    click ModelCore "https://github.com/rwkv/rwkv-lm/blob/main/RWKV-v1/src/model.py"
    click CudaLayer "https://github.com/rwkv/rwkv-lm/blob/main/RWKV-v4/cuda/wkv_cuda.cu"
    click External "https://github.com/rwkv/rwkv-lm/blob/main/RWKV-v5/requirements.txt"
```
