# seal-rg/recurrent-pretraining architecture

> Architecture diagram of the GitHub repository seal-rg/recurrent-pretraining, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/seal-rg/recurrent-pretraining
- Repository: https://github.com/seal-rg/recurrent-pretraining

## Overview

##### Project Type and Purpose
- Key indicators include the presence of a train.py orchestrator, multiple scripts in the scripts/ folder for data preparation, tokenizer generation, and dataset shuffling, along with configuration files (launch_configs/) for the specialized GPU cluster environment.

##### Analysis of the File Structure
- Top-level folders like "evaluate_raven", "recpre", "scripts", and "launch_configs" show a separation of concerns:
- The "recpre" directory holds the core components of the model architecture, dynamic model registration (recpre/model_registry.py), and the model definition (recpre/model_dynamic.py), as well as parallelism implementations (find "SimpleFabric" in recpre/utils.py).
- "evaluate_raven" contains evaluation scripts and configuration files (YAMLs) for converting checkpoints and running quick evaluation comparisons.
- "scripts" is dedicated to data preparation routines: downloading raw datasets, tokenization (using scripts/tokenizer_generation.py), data cleaning and shuffling. These steps are fundamental before training.
- "launch_configs" and "launch_frontier.py" manage environment-specific configurations and command-line orchestration for launching training jobs on a GPU cluster.
- Notice also the supporting files like README.md, LICENSE, and pyproject.toml which provide context and dependency management.

##### Important Architectural Patterns and Design Elements
- Separation of Preprocessing and Training: Data preparation scripts run separately to tokenize, deduplicate, and prepare datasets before training.
- Configuration-Driven Execution: Config files (YAML) in launch_configs allow one to define environment parameters, while train.py takes these configurations as input, making the system flexible for different cluster setups.
- Parallelism and Communication Strategy: The usage of a custom "SimpleFabric" implementation in recpre/utils.py (handling allreduce communication) shows a key part of the architecture for GPU cluster operations.
- Model Registry and Dynamic Selection: The directory recpre/model_registry.py indicates that models (and possibly variants) are dynamically loaded or switched, implying modularity.

##### Main Components
a. Data Processing Pipeline:
- Data Ingestion: Represent scripts like scalable_data_download.py and parquets processing (dedup, shuffler, tokenizer conversion).
- Tokenizer: Highlight the tokenization process in scripts/tokenizer_generation.py.
- Data Staging Area: Implicit staging folder where raw data is downloaded and then processed.

b. Model Definition and Training:
- Parallelism Layer: A module within recpre/utils.py (SimpleFabric) to handle GPU communication and scaling across AMD clusters.

c. Evaluation and Deployment:
- Inference Pipeline: Reference to recpre/raven_modeling_minimal.py which contains the inference version of the model architecture.

d. Infrastructure & Configuration:
- Launch Configurations: A block representing the files in launch_configs/ and the entry-point script launch_frontier.py. These show how the system is configured for multi-node AMD GPU clusters.
- Environment Setup & Flags: Mention and visually denote the overall environment configuration steps needed per the README.

##### Relationships and Data Flow
- From Data Ingestion → Preprocessing (tokenization, cleaning/shuffling) → Tokenized Dataset.
- From Tokenized Dataset → Training Orchestration (train.py) → Model Training.
- Internal flow within training: Model Definition (model_dynamic.py) → Forward/backward propagation, with inter-node communication via SimpleFabric.
- To Evaluation: Trained Checkpoints from training pass to evaluation scripts/converters for performance assessment and eventual deployment.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        direction TB
        A1["Data Download (scripts/scalable_data_download.py)"]:::preproc
        A2["Data Download (scripts/download_stream_to_local_copy.py)"]:::preproc
        B1["Data Cleaning (scripts/parquet_to_parquet_dedup.py)"]:::preproc
        B2["Data Cleaning (scripts/parquet_to_parquet_sentence_dedup.py)"]:::preproc
        B3["Data Shuffling (scripts/parquet_to_parquet_shuffler.py)"]:::preproc
        C["Tokenizer Generation (scripts/tokenizer_generation.py)"]:::preproc
    end

    %% Model Training Layer
    subgraph "Model Training Layer"
        direction TB
        D["Core Model (recpre/model_dynamic.py)"]:::training
        E["Model Registry (recpre/model_registry.py)"]:::training
        F["Parallelism (SimpleFabric) (recpre/utils.py)"]:::training
        G["Training Orchestrator (train.py)"]:::training
    end

    %% Evaluation & Inference Layer
    subgraph "Evaluation & Inference Layer"
        direction TB
        H["Evaluation Scripts (evaluate_raven/)"]:::eval
        I["Inference Model (recpre/raven_modeling_minimal.py)"]:::eval
    end

    %% Infrastructure & Configuration Layer
    subgraph "Infrastructure & Config Layer"
        direction TB
        J["Launch Configurations (launch_configs/)"]:::config
        K["Cluster Orchestration (launch_frontier.py)"]:::config
    end

    %% Data Processing Flow
    A1 -->|"download"| B1
    A2 -->|"download"| B1
    B1 -->|"cleaning"| B2
    B2 -->|"cleaning"| B3
    B3 -->|"preprocessed"| C

    %% Training Flow
    C -->|"tokenized data"| G
    G -->|"uses"| D
    G -->|"selects"| E
    G -->|"communicates via"| F

    %% Evaluation Flow
    G -->|"produces checkpoints"| H
    H -->|"inference"| I

    %% Configuration Influence
    J -->|"configures"| G
    J -->|"influences"| F
    K -->|"orchestrates"| G
    K -->|"orchestrates"| F

    %% Click Events for Preprocessing Layer
    click A1 "https://github.com/seal-rg/recurrent-pretraining/blob/main/scripts/scalable_data_download.py"
    click A2 "https://github.com/seal-rg/recurrent-pretraining/blob/main/scripts/download_stream_to_local_copy.py"
    click B1 "https://github.com/seal-rg/recurrent-pretraining/blob/main/scripts/parquet_to_parquet_dedup.py"
    click B2 "https://github.com/seal-rg/recurrent-pretraining/blob/main/scripts/parquet_to_parquet_sentence_dedup.py"
    click B3 "https://github.com/seal-rg/recurrent-pretraining/blob/main/scripts/parquet_to_parquet_shuffler.py"
    click C "https://github.com/seal-rg/recurrent-pretraining/blob/main/scripts/tokenizer_generation.py"

    %% Click Events for Model Training Layer
    click D "https://github.com/seal-rg/recurrent-pretraining/blob/main/recpre/model_dynamic.py"
    click E "https://github.com/seal-rg/recurrent-pretraining/blob/main/recpre/model_registry.py"
    click F "https://github.com/seal-rg/recurrent-pretraining/blob/main/recpre/utils.py"
    click G "https://github.com/seal-rg/recurrent-pretraining/blob/main/train.py"

    %% Click Events for Evaluation & Inference Layer
    click H "https://github.com/seal-rg/recurrent-pretraining/tree/main/evaluate_raven/"
    click I "https://github.com/seal-rg/recurrent-pretraining/blob/main/recpre/raven_modeling_minimal.py"

    %% Click Events for Infrastructure & Configuration Layer
    click J "https://github.com/seal-rg/recurrent-pretraining/tree/main/launch_configs/"
    click K "https://github.com/seal-rg/recurrent-pretraining/blob/main/launch_frontier.py"

    %% Styles
    classDef preproc fill:#FFD1DC,stroke:#CC3366,stroke-width:2px;
    classDef training fill:#C1E1C1,stroke:#2E8B57,stroke-width:2px;
    classDef eval fill:#BDD7EE,stroke:#1F4E79,stroke-width:2px;
    classDef config fill:#EAD1DC,stroke:#993366,stroke-width:2px;
```
