# smolorg/smolgrad architecture

> Architecture diagram of the GitHub repository smolorg/smolgrad, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/smolorg/smolgrad
- Repository: https://github.com/smolorg/smolgrad

## Overview

##### Main Components
- Core Engine (smolgrad/core/engine.py): The heart of the system. This is where the Tensor class is implemented. It is responsible for:
- Constructing the computational graph as operations are performed on tensors.
- Tracking “grad_fn” functions that define the backward pass for both unary and binary operations.
- Executing the backward method which initiates backpropagation after topologically sorting the graph.
- Neural Network Module (smolgrad/nn/): This directory houses many modules essential to building neural networks including:
- Activation functions (e.g., in _activations.py).
- Layers (e.g., linear, dropout, layernorm, attention, embedding).
- Loss functions and module containers (e.g., _losses.py, _module.py, sequential.py).
- Optimizers (smolgrad/optims/): Implements various optimization algorithms (e.g., Adam, RMSprop, SGD with momentum) that use gradients computed by the engine to update model parameters.
- Utility Functions (smolgrad/utils.py): Miscellaneous helper functions that support core operations or debugging.
- Examples and Tests: Notebooks (such as example_linear.ipynb, example_modules.ipynb) and test scripts (test.py, test_backward.py, etc.) demonstrate usage and validate functionality.

##### Relationships & Interactions
- The Tensor Class (in the core) is the building block. All operations on tensors (such as arithmetic, activation functions, etc.) will create nodes in the computational graph where each node knows its “grad_fn”.
- The NN modules use the Tensor class to build layers and composite operations. They act as clients to the core engine and define how data flows forward through the network.
- Upon a forward pass, the network layers process tensor data, and the computational graph is built up accordingly.
- During the backward pass, the Tensor’s backward method traverses the graph in a backpropagation manner, using defined backward functions from each operation.
- Optimizers take the computed gradients from the Tensor objects and adjust parameters to minimize a loss function.
- The notebooks and tests serve as both validation and a living demonstration of how the engine, NN modules, and optimizers come together.

##### Important Architectural Patterns & Design Principles
- Modularity: Separation between the core autograd engine, neural network layers, and optimization logic.
- Backpropagation Algorithm: Uses topological sorting and chain rule mechanics to update gradients in the correct order.
- Extensibility: The structure allows for the addition of more operations, layers, or optimizers without modifying the core engine.

##### Relevant Technologies & Libraries
- Numpy: Likely used for low-level numerical computations.
- Apple’s MLX: Mentioned in the README as an inspiration or possibly for underlying acceleration.
- Python: The entire project is written in Python, with Jupyter notebooks for demonstration and learning purposes.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Core Engine"
        ENGINE["Tensor Class & Computational Graph"]:::core
    end

    subgraph "Neural Network Modules"
        ACT["Activation Functions"]:::nn
        ACT_MOD["Activation Modules"]:::nn
        LOSS["Loss Functions"]:::nn
        MODULE["Module Base & Containers"]:::nn
        LAYERS["Specific Layers"]:::nn
    end

    subgraph "Optimizers"
        ADAM["Adam Optimizer"]:::optimizers
        RMS["RMSprop Optimizer"]:::optimizers
        SGD["SGD_with_Momentum"]:::optimizers
        OPT_BASE["Optimizer Base"]:::optimizers
    end

    subgraph "Utilities & Validation"
        UTIL["Utility Functions"]:::utils
        EX["Examples & Demonstrations"]:::examples
        TEST["Testing Modules"]:::tests
    end

    subgraph "External Dependencies"
        NUMPY["NumPy"]:::external
        MLX["Apple's MLX"]:::external
    end

    NUMPY -->|"supports"| ENGINE
    MLX -->|"supports"| ENGINE

    ACT -->|"graph-building"| ENGINE
    ACT_MOD -->|"graph-building"| ENGINE
    LOSS -->|"graph-building"| ENGINE
    MODULE -->|"graph-building"| ENGINE
    LAYERS -->|"graph-building"| ENGINE

    ENGINE -->|"backpropagation"| ADAM
    ENGINE -->|"backpropagation"| RMS
    ENGINE -->|"backpropagation"| SGD
    ENGINE -->|"backpropagation"| OPT_BASE

    UTIL -->|"helpers"| ENGINE
    EX -->|"demonstrates"| ENGINE
    EX -->|"demonstrates"| ADAM
    TEST -->|"tests"| ENGINE

    click ENGINE "https://github.com/smolorg/smolgrad/blob/master/smolgrad/core/engine.py"
    click ACT "https://github.com/smolorg/smolgrad/blob/master/smolgrad/nn/_activations.py"
    click ACT_MOD "https://github.com/smolorg/smolgrad/blob/master/smolgrad/nn/_acts_modules.py"
    click LOSS "https://github.com/smolorg/smolgrad/blob/master/smolgrad/nn/_losses.py"
    click MODULE "https://github.com/smolorg/smolgrad/blob/master/smolgrad/nn/_module.py"
    click LAYERS "https://github.com/smolorg/smolgrad/blob/master/smolgrad/nn/linear.py"
    click ADAM "https://github.com/smolorg/smolgrad/blob/master/smolgrad/optims/adam.py"
    click RMS "https://github.com/smolorg/smolgrad/blob/master/smolgrad/optims/rmsprop.py"
    click SGD "https://github.com/smolorg/smolgrad/blob/master/smolgrad/optims/sgd_momentum.py"
    click OPT_BASE "https://github.com/smolorg/smolgrad/blob/master/smolgrad/optims/_optimizer.py"
    click UTIL "https://github.com/smolorg/smolgrad/blob/master/smolgrad/utils.py"
    click EX "https://github.com/smolorg/smolgrad/blob/master/example_linear.ipynb"
    click TEST "https://github.com/smolorg/smolgrad/blob/master/test.py"

    classDef core fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef nn fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef optimizers fill:#FFDAB9,stroke:#000,stroke-width:2px;
    classDef utils fill:#E6E6FA,stroke:#000,stroke-width:2px;
    classDef examples fill:#FFC0CB,stroke:#000,stroke-width:2px;
    classDef tests fill:#F08080,stroke:#000,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
