How lionelmessi6410/neural-networks-from-scratch works
1. Repository purpose
- This is a small, self-contained NumPy implementation of a neural network from scratch.
- The repo is instructional rather than production-oriented: it demonstrates the full training pipeline for MNIST digit classification without TensorFlow/PyTorch.
2. Main execution path
- The primary entry point is train.py.
- Running python train.py starts the end-to-end workflow: load/prepare MNIST data, build the model, train for multiple epochs, and report validation/test accuracy on CPU.
3. Core model subsystem
- model.py contains the neural network implementation, likely centered around a DeepNeuralNetwork class.
- The model stores:
- layer parameters (W, b)
- intermediate activations/cache for backpropagation
- computed gradients for the optimizer step
- The network is a fully connected MLP with one hidden layer in the runnable tutorial flow, mapping 784 input pixels → 64 hidden units → 10 class outputs.
4. Forward-pass data flow
- Input images are flattened 28×28 grayscale MNIST images.
- The forward pass is a sequence of:
- linear transform via matrix multiplication
- hidden activation using ReLU or sigmoid
- final softmax to produce class probabilities
- The cache keeps intermediate tensors so the backward pass can reuse them without recomputing.
5. Backward-pass / gradient subsystem
- The backpropagation logic computes gradients for the loss with respect to each learnable parameter.
- The final layer uses the softmax-cross-entropy gradient shortcut pattern (output - y style), then propagates errors backward through the hidden activation derivative.
- This is the central boundary where the repo turns forward activations into parameter updates.
6. Optimization subsystem
- The repo supports two update rules:
- SGD
- Momentum
- The optimizer updates the weight/bias parameters directly from gradients after each batch.
- Momentum state is maintained separately from the parameters, which matters for the architecture diagram because it is an additional training-time state, not part of inference.
7. Training loop structure
- Training is organized as nested loops:
- outer loop over epochs
- inner loop over mini-batches
- Each batch goes through:
- shuffle
- forward pass
- backward pass
- parameter update
- Accuracy is evaluated on a validation/test split after training iterations, so the training script is both the orchestrator and the evaluator.
8. Configuration boundary
- train.py exposes a small CLI for runtime behavior:
- activation: relu or sigmoid
- optimizer: sgd or momentum
- batch size
- learning rate
- momentum beta
- This means the architecture is parameterized at startup, not hard-coded into separate model variants.
9. Data boundary
- The only real external dependency is the MNIST dataset.
- Data flow is:
- raw digit images and labels
- preprocessing/flattening
- one-hot encoded labels for training
- batches fed into the model
- There is no evidence of a database, service backend, or model-serving component; everything is local, in-memory, and CPU-based.
10. Notebook and documentation layer
- NN-from-Scratch.ipynb mirrors the tutorial content and likely serves as the explanatory/interactive version of the same code path.
- README.md is the main architectural guide and usage document.
- figs/ contains explanatory diagrams used by the tutorial; these are documentation assets, not runtime dependencies.
11. Repository shape for diagramming
- A good architecture diagram would show a single process with four major blocks:
- MNIST data preparation
- model forward/backward computation in NumPy
- optimizer update step
- evaluation/accuracy reporting
- The boundaries are simple: CLI/config → training loop → model math → metrics.
12. Overall architectural character
- This is a monolithic educational ML implementation with no microservices, no distributed training, and no persistence layer.
- The main architectural idea is to expose the mechanics of neural networks explicitly: parameters, activations, gradients, and optimizer state all live in Python/NumPy structures inside the same process.
Main entry points: train.py train.py · model.py model.py · NN-from-Scratch.ipynb Notebook · README.md README
Components 18
Training runtime
Single-process NumPy training and evaluation flow
- train.py · CLI entry and end-to-end training orchestrator
train.py - CLI config · Activation, optimizer, batch size, learning rate, momentum beta
- Accuracy · Validation and test accuracy reporting
Neural net core
Forward, backward, and parameter update mechanics
- model.py · Neural network implementation in NumPy
model.py - MLP · 784→64→10 fully connected classifier
- Parameters · Layer weights and biases stored in NumPy arrays
- Forward pass · Linear transforms, activation, and softmax
- Activation · Hidden-layer nonlinearity selected at startup
- Cache · Stored activations and tensors for backprop
- Backprop · Softmax-cross-entropy shortcut and hidden-layer derivatives
- Optimizer · Updates parameters from gradients using SGD or momentum
- Momentum state · Velocity buffers maintained separately from parameters
Data boundary
MNIST loading, preprocessing, and batching
- Load MNIST · Read raw digit images and labels
- Preprocess · Flatten images and one-hot encode labels
- Mini-batches · Shuffle and split data into batches
Connections 22
- README → train.py: describes · Documentation points to the main executable
- Notebook → model.py: mirrors · Interactive tutorial uses the same core mechanics
- Figures → Notebook: illustrates · Diagram assets support the tutorial narrative
- train.py → CLI config: parses · Startup arguments define runtime behavior
- train.py → Load MNIST: loads · Main script begins by acquiring MNIST data
- Load MNIST → Preprocess: feeds · Raw data is prepared for learning
- Preprocess → Mini-batches: batches · Prepared arrays are shuffled and grouped
- Mini-batches → train.py: drives · Mini-batches flow through the training loop
- train.py → model.py: builds · Training script instantiates the neural network
- model.py → MLP: contains · Model module defines the dense classifier
- MLP → Parameters: stores · Weights and biases are part of model state
- MLP → Forward pass: executes · Forward computation produces class probabilities
- Forward pass → Activation: uses · Hidden layer nonlinearity is configurable
- Forward pass → Cache: records · Intermediate tensors are cached for gradients
- Cache → Backprop: supports · Backward pass reuses saved activations
- Backprop → Optimizer: produces · Gradients are handed to the update rule
- Optimizer → Parameters: updates · SGD or momentum mutates learnable parameters
- Optimizer → Momentum state: maintains · Momentum keeps separate velocity buffers
- train.py → Accuracy: evaluates · Training script reports validation and test accuracy
- Accuracy → train.py: reports · Evaluation results are surfaced by the orchestrator
- CLI config → Activation: selects · Activation choice is set at startup
- CLI config → Optimizer: selects · Optimizer choice is set at startup