# pbloem/former architecture

> Architecture diagram of the GitHub repository pbloem/former, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pbloem/former
- Repository: https://github.com/pbloem/former

## Overview

##### Main Components

a. Core Components (Central Layer):
- Transform these directories into architectural components:
- `former/modules.py` - Core transformer building blocks
- `former/transformers.py` - Main transformer implementation
- `former/util` - Utility functions and helpers

b. Experiment Layer (Top Layer):
- `experiments/classify.py` - Classification pipeline
- `experiments/generate.py` - Generation pipeline

c. Data Layer (Bottom Layer):
- Data handling components for:
- IMDb dataset (mentioned in README)
- Wikipedia data (enwik8.gz)

##### Component Relationships

**[Data Sources] → [Data Processing] → [Transformer Core] → [Experiments]**

##### Specific Implementation Details

**┌─ Experiments Layer ─────────────┐**
- Classification Pipeline       │
- Generation Pipeline          │
- ┬────────────────┘
- ↓
**┌─ Transformer Core ──────────────┐**
- ┌─ Modules ─────┐ ┌─ Utils ──┐ │
- Attention   │ │ - Helper │ │
- Layers      │ │   Methods│ │
- ┘ └──────────┘ │
- ┬────────────────┘
- ↓
**┌─ Data Layer ──────────────────┐**
- IMDb Dataset               │
- Wikipedia Corpus (enwik8)  │
- ┘

##### Important Elements to Highlight

a. Single Stack Architecture:
- That this is a single-stack transformer implementation (not encoder-decoder)

b. PyTorch Integration:
- PyTorch as a foundational layer or dependency
- How it interfaces with the transformer implementation

##### Testing Components
A side panel showing:
**┌─ Tests ────────────────┐**
- Module Tests        │
- Compression Tests   │
- Utility Tests       │
- ┘

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef data fill:#2ecc71,stroke:#27ae60,color:white
    classDef experiment fill:#e67e22,stroke:#d35400,color:white
    classDef test fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef util fill:#9b59b6,stroke:#8e44ad,color:white

    %% Experiment Layer
    subgraph ExperimentLayer
        ClassifyPipe["Classification Pipeline"]:::experiment
        GeneratePipe["Generation Pipeline"]:::experiment
    end

    %% Core Layer
    subgraph CoreLayer
        subgraph Modules
            Attention["Attention Mechanism"]:::core
            Layers["Neural Layers"]:::core
        end
        
        subgraph Utils
            UtilFuncs["Helper Methods"]:::util
        end
        
        MainTransformer["Main Transformer"]:::core
    end

    %% Data Layer
    subgraph DataLayer
        IMDb["IMDb Dataset"]:::data
        Wiki["Wikipedia Corpus"]:::data
    end

    %% Testing Components
    subgraph TestComponents
        ModuleTests["Module Tests"]:::test
        CompTests["Compression Tests"]:::test
        UtilTests["Utility Tests"]:::test
    end

    %% Relationships
    IMDb --> MainTransformer
    Wiki --> MainTransformer
    Attention --> MainTransformer
    Layers --> MainTransformer
    UtilFuncs --> MainTransformer
    MainTransformer --> ClassifyPipe
    MainTransformer --> GeneratePipe
    
    ModuleTests -.-> Modules
    CompTests -.-> MainTransformer
    UtilTests -.-> Utils

    %% Click Events
    click Attention "https://github.com/pbloem/former/blob/master/former/modules.py"
    click MainTransformer "https://github.com/pbloem/former/blob/master/former/transformers.py"
    click UtilFuncs "https://github.com/pbloem/former/blob/master/former/util/util.py"
    click ClassifyPipe "https://github.com/pbloem/former/blob/master/experiments/classify.py"
    click GeneratePipe "https://github.com/pbloem/former/blob/master/experiments/generate.py"
    click Wiki "https://github.com/pbloem/former/blob/master/data/enwik8.gz"
    click ModuleTests "https://github.com/pbloem/former/blob/master/tests/test_modules.py"
    click CompTests "https://github.com/pbloem/former/blob/master/tests/compression.py"
    click UtilTests "https://github.com/pbloem/former/blob/master/tests/test_util.py"

    %% Legend
    subgraph Legend
        CoreComp["Core Component"]:::core
        DataComp["Data Component"]:::data
        ExpComp["Experiment Component"]:::experiment
        TestComp["Test Component"]:::test
        UtilComp["Utility Component"]:::util
    end
```
