# lionelmessi6410/neural-networks-from-scratch architecture

> Architecture diagram of the GitHub repository lionelmessi6410/neural-networks-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lionelmessi6410/neural-networks-from-scratch
- Repository: https://github.com/lionelmessi6410/neural-networks-from-scratch

## Overview

1. **Repository purpose**
   - This is a small, self-contained **NumPy implementation of a neural network from scratch**.
   - The repo is instructional rather than production-oriented: it demonstrates the full training pipeline for **MNIST digit classification** without TensorFlow/PyTorch.

2. **Main execution path**
   - The primary entry point is **`train.py`**.
   - Running `python train.py` starts the end-to-end workflow: load/prepare MNIST data, build the model, train for multiple epochs, and report validation/test accuracy on CPU.

3. **Core model subsystem**
   - **`model.py`** contains the neural network implementation, likely centered around a `DeepNeuralNetwork` class.
   - The model stores:
     - layer parameters (`W`, `b`)
     - intermediate activations/cache for backpropagation
     - computed gradients for the optimizer step
   - The network is a **fully connected MLP** with one hidden layer in the runnable tutorial flow, mapping **784 input pixels → 64 hidden units → 10 class outputs**.

4. **Forward-pass data flow**
   - Input images are **flattened 28×28 grayscale MNIST images**.
   - The forward pass is a sequence of:
     - linear transform via matrix multiplication
     - hidden activation using **ReLU or sigmoid**
     - final **softmax** to produce class probabilities
   - The cache keeps intermediate tensors so the backward pass can reuse them without recomputing.

5. **Backward-pass / gradient subsystem**
   - The backpropagation logic computes gradients for the loss with respect to each learnable parameter.
   - The final layer uses the softmax-cross-entropy gradient shortcut pattern (`output - y` style), then propagates errors backward through the hidden activation derivative.
   - This is the central boundary where the repo turns forward activations into parameter updates.

6. **Optimization subsystem**
   - The repo supports two update rules:
     - **SGD**
     - **Momentum**
   - The optimizer updates the weight/bias parameters directly from gradients after each batch.
   - Momentum state is maintained separately from the parameters, which matters for the architecture diagram because it is an additional training-time state, not part of inference.

7. **Training loop structure**
   - Training is organized as nested loops:
     - outer loop over **epochs**
     - inner loop over **mini-batches**
   - Each batch goes through:
     - shuffle
     - forward pass
     - backward pass
     - parameter update
   - Accuracy is evaluated on a validation/test split after training iterations, so the training script is both the orchestrator and the evaluator.

8. **Configuration boundary**
   - `train.py` exposes a small CLI for runtime behavior:
     - activation: `relu` or `sigmoid`
     - optimizer: `sgd` or `momentum`
     - batch size
     - learning rate
     - momentum beta
   - This means the architecture is parameterized at startup, not hard-coded into separate model variants.

9. **Data boundary**
   - The only real external dependency is the **MNIST dataset**.
   - Data flow is:
     - raw digit images and labels
     - preprocessing/flattening
     - one-hot encoded labels for training
     - batches fed into the model
   - There is no evidence of a database, service backend, or model-serving component; everything is local, in-memory, and CPU-based.

10. **Notebook and documentation layer**
   - **`NN-from-Scratch.ipynb`** mirrors the tutorial content and likely serves as the explanatory/interactive version of the same code path.
   - **`README.md`** is the main architectural guide and usage document.
   - **`figs/`** contains explanatory diagrams used by the tutorial; these are documentation assets, not runtime dependencies.

11. **Repository shape for diagramming**
   - A good architecture diagram would show a single process with four major blocks:
     - MNIST data preparation
     - model forward/backward computation in NumPy
     - optimizer update step
     - evaluation/accuracy reporting
   - The boundaries are simple: **CLI/config → training loop → model math → metrics**.

12. **Overall architectural character**
   - This is a **monolithic educational ML implementation** with no microservices, no distributed training, and no persistence layer.
   - The main architectural idea is to expose the mechanics of neural networks explicitly: parameters, activations, gradients, and optimizer state all live in Python/NumPy structures inside the same process.

Main entry points: [`train.py`](https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/train.py) (train.py), [`model.py`](https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/model.py) (model.py), [`NN-from-Scratch.ipynb`](https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/NN-from-Scratch.ipynb) (Notebook), [`README.md`](https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/README.md) (README)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Training runtime"]
  node_train_py["train.py<br/>entrypoint<br/>[train.py]"]
  node_cli_config["CLI config<br/>runtime config"]
  node_metrics(("Accuracy<br/>evaluation"))
end

subgraph group_model["Neural net core"]
  node_model_py["model.py<br/>nn core<br/>[model.py]"]
  node_network["MLP<br/>dense network"]
  node_params[("Parameters<br/>weights")]
  node_forward(("Forward pass<br/>inference"))
  node_activation(("Activation<br/>relu/sigmoid"))
  node_cache[("Cache<br/>intermediate state")]
  node_backprop(("Backprop<br/>gradient calc"))
  node_optimizer["Optimizer<br/>sgd/momentum"]
  node_mom_state[("Momentum state<br/>training state")]
end

subgraph group_data["Data boundary"]
  node_mnist_load[("Load MNIST<br/>data ingest")]
  node_preprocess["Preprocess<br/>data prep"]
  node_batcher["Mini-batches<br/>batching"]
end

subgraph group_docs["Docs and tutorial"]
  node_readme["README<br/>guide<br/>[README.md]"]
  node_notebook["Notebook<br/>tutorial"]
  node_figures["Figures<br/>assets"]
end

node_readme -->|"describes"| node_train_py
node_notebook -.->|"mirrors"| node_model_py
node_figures -.->|"illustrates"| node_notebook
node_train_py -->|"parses"| node_cli_config
node_train_py -->|"loads"| node_mnist_load
node_mnist_load -->|"feeds"| node_preprocess
node_preprocess -->|"batches"| node_batcher
node_batcher -->|"drives"| node_train_py
node_train_py -->|"builds"| node_model_py
node_model_py -->|"contains"| node_network
node_network -->|"stores"| node_params
node_network -->|"executes"| node_forward
node_forward -->|"uses"| node_activation
node_forward -->|"records"| node_cache
node_cache -->|"supports"| node_backprop
node_backprop -->|"produces"| node_optimizer
node_optimizer -->|"updates"| node_params
node_optimizer -->|"maintains"| node_mom_state
node_train_py -->|"evaluates"| node_metrics
node_metrics -->|"reports"| node_train_py
node_cli_config -->|"selects"| node_activation
node_cli_config -->|"selects"| node_optimizer

click node_train_py "https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/train.py"
click node_model_py "https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/model.py"
click node_readme "https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/README.md"
click node_notebook "https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/NN-from-Scratch.ipynb"
click node_figures "https://github.com/lionelmessi6410/neural-networks-from-scratch/tree/main/figs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train_py,node_cli_config,node_metrics toneBlue
class node_model_py,node_network,node_params,node_forward,node_activation,node_cache,node_backprop,node_optimizer,node_mom_state toneAmber
class node_mnist_load,node_preprocess,node_batcher toneMint
class node_readme,node_notebook,node_figures toneRose
```

## Components

### Training runtime

Single-process NumPy training and evaluation flow

- [train.py](https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/train.py): entrypoint · CLI entry and end-to-end training orchestrator · `train.py`
- **CLI config**: runtime config · Activation, optimizer, batch size, learning rate, momentum beta
- **Accuracy**: evaluation · Validation and test accuracy reporting

### Neural net core

Forward, backward, and parameter update mechanics

- [model.py](https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/model.py): nn core · Neural network implementation in NumPy · `model.py`
- **MLP**: dense network · 784→64→10 fully connected classifier
- **Parameters**: weights · Layer weights and biases stored in NumPy arrays
- **Forward pass**: inference · Linear transforms, activation, and softmax
- **Activation**: relu/sigmoid · Hidden-layer nonlinearity selected at startup
- **Cache**: intermediate state · Stored activations and tensors for backprop
- **Backprop**: gradient calc · Softmax-cross-entropy shortcut and hidden-layer derivatives
- **Optimizer**: sgd/momentum · Updates parameters from gradients using SGD or momentum
- **Momentum state**: training state · Velocity buffers maintained separately from parameters

### Data boundary

MNIST loading, preprocessing, and batching

- **Load MNIST**: data ingest · Read raw digit images and labels
- **Preprocess**: data prep · Flatten images and one-hot encode labels
- **Mini-batches**: batching · Shuffle and split data into batches

### Docs and tutorial

Notebook, README, and figures for explanation

- [README](https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/README.md): guide · Usage and architectural overview · `README.md`
- [Notebook](https://github.com/lionelmessi6410/neural-networks-from-scratch/blob/main/NN-from-Scratch.ipynb): tutorial · Interactive walkthrough of the same concepts · `NN-from-Scratch.ipynb`
- [Figures](https://github.com/lionelmessi6410/neural-networks-from-scratch/tree/main/figs): assets · Illustration assets for the tutorial · `figs`

## Connections

- README → train.py: describes · Documentation points to the main executable
- Notebook → model.py: mirrors · Interactive tutorial uses the same core mechanics
- Figures → Notebook: illustrates · Diagram assets support the tutorial narrative
- train.py → CLI config: parses · Startup arguments define runtime behavior
- train.py → Load MNIST: loads · Main script begins by acquiring MNIST data
- Load MNIST → Preprocess: feeds · Raw data is prepared for learning
- Preprocess → Mini-batches: batches · Prepared arrays are shuffled and grouped
- Mini-batches → train.py: drives · Mini-batches flow through the training loop
- train.py → model.py: builds · Training script instantiates the neural network
- model.py → MLP: contains · Model module defines the dense classifier
- MLP → Parameters: stores · Weights and biases are part of model state
- MLP → Forward pass: executes · Forward computation produces class probabilities
- Forward pass → Activation: uses · Hidden layer nonlinearity is configurable
- Forward pass → Cache: records · Intermediate tensors are cached for gradients
- Cache → Backprop: supports · Backward pass reuses saved activations
- Backprop → Optimizer: produces · Gradients are handed to the update rule
- Optimizer → Parameters: updates · SGD or momentum mutates learnable parameters
- Optimizer → Momentum state: maintains · Momentum keeps separate velocity buffers
- train.py → Accuracy: evaluates · Training script reports validation and test accuracy
- Accuracy → train.py: reports · Evaluation results are surfaced by the orchestrator
- CLI config → Activation: selects · Activation choice is set at startup
- CLI config → Optimizer: selects · Optimizer choice is set at startup
