# rhqo/toy-models-of-superposition architecture

> Architecture diagram of the GitHub repository rhqo/toy-models-of-superposition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rhqo/toy-models-of-superposition
- Repository: https://github.com/rhqo/toy-models-of-superposition

## Overview

##### Project Type Analysis
This is a research-oriented project implementing mathematical models to study neural network behavior, specifically focusing on dimensional projection and feature superposition. The project consists of multiple Jupyter notebooks that explore different aspects of the phenomenon.

##### Main Components

A. Input Layer Components:
- Feature Space Representations (varying dimensions: 3D, 4D, 5D, 8D)
- Sparsity Parameters
- Feature Characteristics (uniform/non-uniform)

B. Processing Components:
- Dimension Reduction Models
- ReLU Model
- Linear Model
- Projection Mechanisms (h=Wx)
- Training Components

C. Output Layer Components:
- Lower Dimensional Projections (2D, 3D)
- Visualization Systems
- Analysis Metrics

**[Input Layer] → [Processing Layer] → [Output Layer]**
- ↓               ↓                    ↓
**[Feature Space]  [Model Types]     [Projections]**
**[Sparsity]      [Training]        [Visualizations]**
**[Parameters]     [Transformations] [Analysis]**

##### Specific Elements

- Blue: Input components
- Green: Processing components
- Orange: Output components
- Purple: Data flow paths

b) Component Relationships:

##### Key Aspects to Emphasize

a) Dimensional Transformation Pipeline:
- Clear paths from higher to lower dimensions
- Where sparsity parameters influence the process
- Different model variants (ReLU vs Linear)

b) Analysis Components:
- Feature geometry analysis
- Sparsity impact visualization
- Non-uniform superposition effects

##### Technical Details

- Input dimension ranges (3D-8D)
- Output dimension ranges (2D-3D)
- Sparsity parameter ranges (0-0.999)
- Model types and their connections
- Visualization output types

##### Implementation Notes

- Clear hierarchical representation
- Multiple connection types
- Proper labeling and annotations
- Ability to show mathematical expressions (h=Wx)

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Legend
        L1[Input Components]:::input
        L2[Processing Components]:::processing
        L3[Output Components]:::output
        L4[Data Flow]-- Flow -->L5
        L5[Component]
    end

    subgraph Input
        FS[Feature Space]:::input
        SP[Sparsity Parameters]:::input
        FC[Feature Characteristics]:::input
    end

    subgraph Processing
        BM[Base Model]:::processing
        click BM "https://github.com/Rhqo/Toy-Models-of-Superposition/blob/main/toy_models.ipynb"
        
        subgraph Models
            RM[ReLU Model]:::processing
            LM[Linear Model]:::processing
        end
        
        subgraph "Experimental Processing"
            IS[Initial Setup]:::processing
            click IS "https://github.com/Rhqo/Toy-Models-of-Superposition/blob/main/KBH_Toy_Models_0.ipynb"
            FSA[Feature Space Analysis]:::processing
            click FSA "https://github.com/Rhqo/Toy-Models-of-Superposition/blob/main/KBH_Toy_Models_1.ipynb"
            MP[Model Processing]:::processing
            click MP "https://github.com/Rhqo/Toy-Models-of-Superposition/blob/main/KBH_Toy_Models_2.ipynb"
        end
        
        TF[Theoretical Framework]:::processing
        click TF "https://github.com/Rhqo/Toy-Models-of-Superposition/blob/main/Study_Toy-models-of-superposition.pdf"
        
        EM[Exact Mathematical Models]:::processing
        click EM "https://github.com/Rhqo/Toy-Models-of-Superposition/blob/main/Exploring_Exact_Toy_Models.ipynb"
    end

    subgraph Output
        DP[Dimensional Projections]:::output
        VS[Visualization System]:::output
        AM[Analysis Metrics]:::output
        OA[Output Analysis]:::output
        click OA "https://github.com/Rhqo/Toy-Models-of-Superposition/blob/main/KBH_Toy_Models_3.ipynb"
    end

    %% Input Connections
    FS --> BM
    SP --> BM
    FC --> BM

    %% Processing Connections
    BM --> Models
    BM --> EM
    Models --> IS
    IS --> FSA
    FSA --> MP
    EM --> MP
    TF -.-> MP

    %% Output Connections
    MP --> DP
    MP --> VS
    MP --> AM
    DP --> OA
    VS --> OA
    AM --> OA

    classDef input fill:#a8d1ff,stroke:#333,stroke-width:2px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:2px
    classDef output fill:#ffb366,stroke:#333,stroke-width:2px
```
