# danielbrodrick1995/coco-llm-official architecture

> Architecture diagram of the GitHub repository danielbrodrick1995/coco-llm-official, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/danielbrodrick1995/coco-llm-official
- Repository: https://github.com/danielbrodrick1995/coco-llm-official

## Overview

##### Project Type and Purpose
- This is an open-source machine-learning training pipeline for a domain-specific large language model (COCO LLM).
- It provides modules for data ingestion/preprocessing, model definition and tokenization, alignment/fine-tuning, distributed training configurations, and release scripts.

##### Main Architectural Components
a. Data Layer
- ‘src/data’ module: data loaders, utilities, raw data ingestion.
- External data sources: COBOL codebases, benchmarks (COCO bench).
b. Model Layer
- ‘src/model’ module: tokenizer registration, model wrappers.
- Base model checkpoint (DeepSeek-Coder 7B).
c. Alignment & Fine-Tuning Layer
- ‘src/alignment’: decontamination, alignment data preparation, model_utils.
- train_raw.py: raw LM training.
- train_instruct.py: instruction-tuning pipeline.
d. Configuration & Orchestration
- ‘recipes/accelerate_configs’: distributed training (DeepSpeed, FSDP, Multi-GPU).
- ‘recipes/deepseek’: hyperparameter recipes for LoRA, SFT, full fine-tuning.
- scripts/*.sh: convenience wrappers to launch training with chosen configs.
e. External Services & Infrastructure
- Compute cluster nodes with GPUs/TPUs.
- Accelerate library, DeepSpeed, PyTorch Lightning or plain PyTorch.
f. Release & Benchmark
- alignment/release.py to package model artifacts.
- COCO bench (external evaluation suite).

##### Relationships and Data Flow
- Data Layer → Alignment Layer: raw data passes through decontamination → yields aligned examples.
- Alignment Layer → Model Layer: alignment examples used by train_instruct.py to fine-tune model_utils.
- Model Layer ↔ Configuration Layer: training entry points (train_raw.py, train_instruct.py) read accelerate_configs and deepseek recipes to spawn distributed jobs.
- Training Jobs → Release Layer: upon convergence, release.py packages and uploads artifacts.
- Release Artifacts → External Services: evaluation on COCO bench, model registry.

##### Architectural Patterns & Design Principles
- Modular layering: separation of data, model, alignment, and orchestration.
- Code-first ML pipeline: python entry-points (train_*.py) call modules rather than monolithic scripts.
- Configuration-as-code: YAML recipes for reproducibility.
- Extensibility: new recipes under ‘recipes/’, new data pipeline under ‘src/data/’, new alignment strategies under ‘src/alignment/’.

##### Elements
- Clear labels for each component (e.g., “Data Loader,” “Alignment Module,” “Train_Instruct.py”).
- Rectangles for code modules/directories.
- Clouds/servers for compute cluster or external services.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    COBOL["COBOL Codebases"]:::external
    BenchExt["COCO Bench (Source)"]:::external

    %% Data Layer
    subgraph "Data Layer"
        DL1["data.py"]:::data
        DL2["utils.py"]:::data
    end

    %% Alignment Layer
    subgraph "Alignment Layer"
        AL1["decontaminate.py"]:::alignment
        AL2["data.py"]:::alignment
        AL3["model_utils.py"]:::alignment
        AL4["configs.py"]:::alignment
    end

    %% Model Layer
    subgraph "Model Layer"
        Tokenizer["tokenizer.py"]:::model
        ModelInit["__init__.py"]:::model
        BaseModel["DeepSeek-Coder 7B Checkpoint"]:::model
    end

    %% Orchestration & Configurations
    subgraph "Training Entry Points"
        TrainRaw["train_raw.py"]:::orchestration
        TrainInst["train_instruct.py"]:::orchestration
    end

    subgraph "Recipes: Accelerate" 
        DS3["deepspeed_zero3.yaml"]:::orchestration
        DS3L["deepspeed_zero3_lora.yaml"]:::orchestration
        FSDP["fsdp.yaml"]:::orchestration
        FSDPQ["fsdp_qlora.yaml"]:::orchestration
        MGPU["multi_gpu.yaml"]:::orchestration
    end

    subgraph "Recipes: Deepseek"
        Full["full.yaml"]:::orchestration
        FullInst["full_instruct.yaml"]:::orchestration
        LoraInst["lora_instruct.yaml"]:::orchestration
        LoraSFT["lora_sft.yaml"]:::orchestration
    end

    subgraph "Helper Scripts"
        FTsh["ft.sh"]:::orchestration
        Instsh["instruct.sh"]:::orchestration
    end

    %% Release & Evaluation
    subgraph "Release & Evaluation"
        Release["release.py"]:::release
        BenchEval["COCO Bench (Eval)"]:::external
    end

    %% Data Flow
    COBOL --> DL1
    BenchExt --> DL2
    DL1 --> AL1
    DL2 --> AL1
    AL1 --> AL2
    AL2 --> TrainInst
    TrainInst --> BaseModel
    AL3 --> TrainRaw
    AL3 --> TrainInst
    TrainRaw --> BaseModel

    %% Config & Control Flow
    TrainRaw -.-> DS3
    TrainRaw -.-> DS3L
    TrainRaw -.-> FSDP
    TrainRaw -.-> FSDPQ
    TrainRaw -.-> MGPU
    TrainRaw -.-> Full
    TrainRaw -.-> FullInst
    TrainRaw -.-> LoraInst
    TrainRaw -.-> LoraSFT

    TrainInst -.-> DS3
    TrainInst -.-> DS3L
    TrainInst -.-> FSDP
    TrainInst -.-> FSDPQ
    TrainInst -.-> MGPU
    TrainInst -.-> Full
    TrainInst -.-> FullInst
    TrainInst -.-> LoraInst
    TrainInst -.-> LoraSFT

    FTsh --> TrainRaw
    Instsh --> TrainInst

    %% Release Flow
    TrainRaw --> Release
    TrainInst --> Release
    Release --> BenchEval

    %% Click Events
    click DL1 "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/data/data.py"
    click DL2 "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/data/utils.py"
    click Tokenizer "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/model/tokenizer.py"
    click ModelInit "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/model/__init__.py"
    click AL1 "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/alignment/decontaminate.py"
    click AL2 "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/alignment/data.py"
    click AL3 "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/alignment/model_utils.py"
    click AL4 "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/alignment/configs.py"
    click TrainRaw "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/train_raw.py"
    click TrainInst "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/train_instruct.py"
    click DS3 "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/accelerate_configs/deepspeed_zero3.yaml"
    click DS3L "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/accelerate_configs/deepspeed_zero3_lora.yaml"
    click FSDP "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/accelerate_configs/fsdp.yaml"
    click FSDPQ "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/accelerate_configs/fsdp_qlora.yaml"
    click MGPU "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/accelerate_configs/multi_gpu.yaml"
    click Full "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/deepseek/full.yaml"
    click FullInst "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/deepseek/full_instruct.yaml"
    click LoraInst "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/deepseek/lora_instruct.yaml"
    click LoraSFT "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/recipes/deepseek/lora_sft.yaml"
    click FTsh "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/scripts/ft.sh"
    click Instsh "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/scripts/instruct.sh"
    click Release "https://github.com/danielbrodrick1995/coco-llm-official/blob/main/src/alignment/release.py"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#069,stroke-width:1px;
    classDef model fill:#D4FDD4,stroke:#080,stroke-width:1px;
    classDef orchestration fill:#FFE6B3,stroke:#E68A00,stroke-width:1px;
    classDef alignment fill:#E8D0FF,stroke:#A020F0,stroke-width:1px;
    classDef external fill:#E0E0E0,stroke:#888,stroke-width:1px;
    classDef release fill:#FFD0D0,stroke:#D00,stroke-width:1px;
```
