How rwkv/rwkv-lm works
Main System Components
• Model Core (src): In every version, the src folder contains key files: model.py (defining the network architecture, including time-mix and channel-mix operations, custom recurrent formulations, and various initialization tricks), trainer.py (the training routine and integration with PyTorch Lightning), and utils.py (helper functions for data processing, state management, etc.). Represent these as a “Model Core” subcomponent that implements the RWKV’s unique RNN-style / transformer-parallelizable design.
• CUDA Accelerated Modules: Notice that several versions (v2, v3, v4, v4neo, v5, and v7) include a dedicated “cuda” directory with custom CUDA kernels (e.g., timex_cuda.cu, timex_op.cpp, wkv_cuda.cu, etc.). These modules are critical to speeding up core computations. They should be diagrammed as an “Accelerated Computation Layer” linked to the model’s core compute routines.
• Training & Inference Scripts: Files like train.py, run.py, and demo scripts (rwkv_demo*, verify.py, chat.py) serve as the entry points for training and inference. They orchestrate the use of the model core and launch the corresponding routines. Represent a “Training/Inference Orchestration” component that interfaces with the core model and also triggers CUDA kernels when available.
Relationships and Data Flow
• The Data Pipeline (tokenizers and preparation scripts) feeds training data (large text corpora, e.g., enwik8 or jsonl files) into the Training/Orchestration system.
• The training scripts (train.py, demo-training-run.sh, etc.) pass data to the Model Core, which is implemented in model.py using custom layers (e.g., time-mix and channel-mix) and the special recurrent formulation (exploiting time-decay factors and “receptance” gates) as detailed in the README.
• For performance, when running on GPU, the system “offloads” or integrates with the specialized CUDA kernels (visible in the cuda folders) to accelerate sequential operations in the model (e.g., efficient computation of time-weighted summations).
• Outputs from the model then flow into inference scripts (run.py, chat.py, etc.) and evaluation routines, with feedback loops (as in training via trainer.py) managing state propagation across tokens.
• Optionally, showing how the model’s hidden state is handled (passed/reused between tokens) would illustrate the constant-space, recurrent aspect of RWKV.
Architectural Patterns and Design Principles
• Pipeline Structure: There is a clear separation between data pre-processing, model architecture (core computation), training/inference orchestration, and acceleration (CUDA). This follows a typical deep learning system’s layered design.
• Custom Optimizations: The design exploits specialized initialization tricks, new sampling methods, and state tuning. You could note (in annotations) the presence of “tricks” like token-shift mixing, innovative initialization, and RK techniques that optimize the training dynamics.