How smolorg/smolgrad works
Main Components
• Core Engine (smolgrad/core/engine.py): The heart of the system. This is where the Tensor class is implemented. It is responsible for:
• Constructing the computational graph as operations are performed on tensors.
• Tracking “grad_fn” functions that define the backward pass for both unary and binary operations.
• Executing the backward method which initiates backpropagation after topologically sorting the graph.
• Neural Network Module (smolgrad/nn/): This directory houses many modules essential to building neural networks including:
• Activation functions (e.g., in _activations.py).
• Layers (e.g., linear, dropout, layernorm, attention, embedding).
• Loss functions and module containers (e.g., _losses.py, _module.py, sequential.py).
• Optimizers (smolgrad/optims/): Implements various optimization algorithms (e.g., Adam, RMSprop, SGD with momentum) that use gradients computed by the engine to update model parameters.
• Utility Functions (smolgrad/utils.py): Miscellaneous helper functions that support core operations or debugging.
• Examples and Tests: Notebooks (such as example_linear.ipynb, example_modules.ipynb) and test scripts (test.py, test_backward.py, etc.) demonstrate usage and validate functionality.
Relationships & Interactions
• The Tensor Class (in the core) is the building block. All operations on tensors (such as arithmetic, activation functions, etc.) will create nodes in the computational graph where each node knows its “grad_fn”.
• The NN modules use the Tensor class to build layers and composite operations. They act as clients to the core engine and define how data flows forward through the network.
• Upon a forward pass, the network layers process tensor data, and the computational graph is built up accordingly.
• During the backward pass, the Tensor’s backward method traverses the graph in a backpropagation manner, using defined backward functions from each operation.
• Optimizers take the computed gradients from the Tensor objects and adjust parameters to minimize a loss function.
• The notebooks and tests serve as both validation and a living demonstration of how the engine, NN modules, and optimizers come together.
Important Architectural Patterns & Design Principles
• Modularity: Separation between the core autograd engine, neural network layers, and optimization logic.
• Backpropagation Algorithm: Uses topological sorting and chain rule mechanics to update gradients in the correct order.
• Extensibility: The structure allows for the addition of more operations, layers, or optimizers without modifying the core engine.
Relevant Technologies & Libraries
• Numpy: Likely used for low-level numerical computations.
• Apple’s MLX: Mentioned in the README as an inspiration or possibly for underlying acceleration.
• Python: The entire project is written in Python, with Jupyter notebooks for demonstration and learning purposes.