How chenyuxyz/teenygrad works
Project type and purpose
• This is a minimal deep‑learning framework + MNIST training script (“teenygrad”).
• It provides a tensor API with autograd/realtime graph compilation and a small training example (mnist.py).
File structure
• Top level: configuration (pre‑commit, mypy, CI), LICENSE, README
• Scripts:
• mnist.py — sample training entry point
• import_from_tinygrad.py, sz.py — utility scripts
• extra/datasets: local MNIST data files
• teenygrad/ (core library):
• tensor.py — Tensor class + frontend autograd API
• mlops.py — high‑level layers, loss functions, data utilities
• nn/optim.py — optimizer implementations (SGD, etc.)
• lazy.py — lazy evaluation graph builder (backend interface)
• ops.py — primitive tensor operations (add, matmul, etc.)
• realize.py — graph executor mapping nodes to numpy calls
• helpers.py — utility routines (e.g., random, broadcasting)
• test/: unit tests for dtype, ops, and optim
• .github/workflows/test.yml: CI runs tests on push
README for additional insights
• Single dependency: numpy (+ tqdm for progress bars)
• Usage: “PYTHONPATH='.' python mnist.py”
• tensor.py + mlops.py are frontends, lazy.py is the backend
a. Main components
• Training script (mnist.py)
• Dataset storage (extra/datasets/MNIST files)
• Core library packages:
• Frontend API layer: tensor.py, mlops.py
• Optimizers: nn/optim.py
• Lazy graph builder: lazy.py
• Primitive ops: ops.py
• Graph executor: realize.py
• Helpers: helpers.py
• External dependencies: numpy, tqdm
• Test suite (test/) and CI pipeline (.github/workflows)
b. Relationships and interactions
1. mnist.py loads MNIST files → constructs network via mlops & tensor APIs
2. Forward pass calls Tensor methods → lazy.py builds computational graph of ops
4. At execution, realize.py traverses the graph → issues numpy calls via helpers
5. Backward pass (autograd) is wired into Tensor → gradients flow through the same graph
6. nn/optim.py reads/writes Tensor parameters to perform updates
7. tqdm wraps loops in mnist.py for progress bars
8. test/ invokes tensor.py, ops.py, nn/optim.py directly to verify correctness
9. CI pipeline triggers test suite on every push
c. Architectural patterns & design principles
• Frontend/backend separation: user‑facing tensor/mlops vs. lazy evaluation & realizer
• Lazy evaluation & computational graph for automatic differentiation
• Modular layering: primitives (ops.py) → graph builder (lazy.py) → executor (realize.py)
• Thin dependency surface (only numpy, tqdm) for portability
d. Key technologies and libraries
• Python 3.x
• numpy (core tensor ops)
• tqdm (progress bars)
• pytest (via CI)