# observerw/chisellm architecture

> Architecture diagram of the GitHub repository observerw/chisellm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/observerw/chisellm
- Repository: https://github.com/observerw/chisellm

## Overview

##### Project Type and Purpose
- This is a machine-learning research project focusing on building reasoning LLMs for the Chisel hardware-construction language.
- It consists of two major phases: Data Preparation (dataset pipelines) and Model Training (DeepSpeed configurations).

##### Main Components
a. Input Sources
– Raw Chisel RTL code (public repositories, internal corpora)
b. Data Processing Pipelines
– Completion Pipeline (chisellm/dataset/completion)
– Decompile Pipeline (chisellm/dataset/decompile)
– Template Engine (chisellm/utils/template.py + Jinja2 templates in templates/ folders)
c. Datasets Output
– Processed “Completion” dataset
– Processed “Decompile” dataset
– (Optionally) local storage or direct upload to Hugging Face Datasets
d. Training Configuration
– YAML configs (chisellm/train_config/ChiseLLM-7B.yaml, ChiseLLM-32B.yaml)
– DeepSpeed configs (chisellm/train_config/ds/*.json)
e. Model Training
– Training runner (invokes DeepSpeed + PyTorch on GPU/cluster)
– Utilities (chisellm/utils) for logging, checkpointing, template rendering
f. Resulting Models
– ChiseLLM-7B and ChiseLLM-32B artifacts
– Upload to Hugging Face Model Hub
g. External Services & Tools
– Hugging Face Datasets & Model Hub
– DeepSpeed
– Jinja2 (templating)
– PyTorch/transformers

##### Relationships and Data Flow
- Inside each pipeline, show “pipeline.py” invoking “templates/…”
- “Processed Datasets” + “train_config” → “Training Runner”
- “Training Runner” uses “DeepSpeed configs” and “utils” → produces “Model Artifacts”
- “Model Artifacts” → upload to “Hugging Face Model Hub”

##### Architectural Patterns and Design Principles
- Clear separation of concerns:
– Dataset ingestion & transformation
– Training orchestration & configuration
– Utility functions abstracting template logic
- Modular pipelines for different tasks (completion vs. decompile)
- Configuration-driven training (YAML + JSON) for reproducibility
- Use of templating for prompt engineering (Jinja2)

##### Essential Elements
– Clear component labels matching your file tree.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Project Entry & Metadata
    subgraph "Project Entry & Metadata"
        direction TB
        README["README.md"]:::util
        Pyproject["pyproject.toml"]:::util
    end

    %% Raw Input Sources
    RawCode["Raw Chisel RTL Code"]:::data

    %% Data Preparation Pipelines
    subgraph "Data Preparation Pipelines"
        direction TB
        CompletionPipeline["chisellm/dataset/completion/pipeline.py"]:::pipeline
        CompletionTemplates["Completion Templates"]:::pipeline
        DecompilePipeline["chisellm/dataset/decompile/pipeline.py"]:::pipeline
        DecompileTemplates["Decompile Templates"]:::pipeline
    end

    %% Utility Components
    subgraph "Utilities"
        direction TB
        TemplateEngine["chisellm/utils/template.py"]:::util
        Utils["chisellm/utils/"]:::util
    end

    %% Processed Data Store
    ProcessedDatasets["Processed Datasets"]:::data

    %% Training Configuration
    subgraph "Training Configurations"
        direction TB
        YAML7B["ChiseLLM-7B.yaml"]:::config
        YAML32B["ChiseLLM-32B.yaml"]:::config
        DSConfig["DeepSpeed Configs"]:::config
    end

    %% Model Training
    TrainingRunner["Training Runner"]:::compute

    %% Model Artifacts & External Services
    ModelArtifacts["Model Artifacts"]:::compute
    subgraph "External Services"
        direction TB
        HFDatasets["Hugging Face Datasets"]:::external
        HFModelHub["Hugging Face Model Hub"]:::external
        DeepSpeed["DeepSpeed"]:::external
        PyTorch["PyTorch/Transformers"]:::external
        Jinja2["Jinja2"]:::external
    end

    %% Data Flows
    RawCode -->|"ingest raw RTL"| CompletionPipeline
    RawCode -->|"ingest raw RTL"| DecompilePipeline

    CompletionPipeline -->|"uses templates"| CompletionTemplates
    DecompilePipeline -->|"uses templates"| DecompileTemplates

    CompletionPipeline -->|"calls template util"| TemplateEngine
    DecompilePipeline -->|"calls template util"| TemplateEngine

    CompletionPipeline -->|"produces JSON dataset"| ProcessedDatasets
    DecompilePipeline -->|"produces JSON dataset"| ProcessedDatasets

    ProcessedDatasets -->|"feeds data"| TrainingRunner
    YAML7B -->|"configures training"| TrainingRunner
    YAML32B -->|"configures training"| TrainingRunner
    DSConfig -->|"DeepSpeed settings"| TrainingRunner

    TrainingRunner -->|"uses utils"| Utils
    TrainingRunner -->|"runs on GPU cluster"| DeepSpeed
    TrainingRunner -->|"runs on GPU cluster"| PyTorch

    TrainingRunner -->|"outputs checkpoints"| ModelArtifacts
    ModelArtifacts -->|"upload models"| HFModelHub
    ModelArtifacts -->|"publish datasets"| HFDatasets

    CompletionTemplates -->|"powered by"| Jinja2
    DecompileTemplates -->|"powered by"| Jinja2

    %% Click Events
    click README "https://github.com/observerw/chisellm/blob/main/README.md"
    click Pyproject "https://github.com/observerw/chisellm/blob/main/pyproject.toml"
    click CompletionPipeline "https://github.com/observerw/chisellm/blob/main/chisellm/dataset/completion/pipeline.py"
    click CompletionTemplates "https://github.com/observerw/chisellm/tree/main/chisellm/dataset/completion/templates/"
    click DecompilePipeline "https://github.com/observerw/chisellm/blob/main/chisellm/dataset/decompile/pipeline.py"
    click DecompileTemplates "https://github.com/observerw/chisellm/tree/main/chisellm/dataset/decompile/templates/"
    click TemplateEngine "https://github.com/observerw/chisellm/blob/main/chisellm/utils/template.py"
    click Utils "https://github.com/observerw/chisellm/tree/main/chisellm/utils/"
    click YAML7B "https://github.com/observerw/chisellm/blob/main/chisellm/train_config/ChiseLLM-7B.yaml"
    click YAML32B "https://github.com/observerw/chisellm/blob/main/chisellm/train_config/ChiseLLM-32B.yaml"
    click DSConfig "https://github.com/observerw/chisellm/tree/main/chisellm/train_config/ds/"

    %% Styles
    classDef pipeline fill:#cce5ff,stroke:#004085,color:#004085
    classDef config fill:#d4edda,stroke:#155724,color:#155724
    classDef compute fill:#fff3cd,stroke:#856404,color:#856404
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef util fill:#f8d7da,stroke:#721c24,color:#721c24
    classDef data fill:#d1ecf1,stroke:#0c5460,color:#0c5460
```
