How pbloem/former works
Main Components
a. Core Components (Central Layer):
• Transform these directories into architectural components:
• former/modules.py - Core transformer building blocks
• former/transformers.py - Main transformer implementation
• former/util - Utility functions and helpers
b. Experiment Layer (Top Layer):
• experiments/classify.py - Classification pipeline
• experiments/generate.py - Generation pipeline
c. Data Layer (Bottom Layer):
• Data handling components for:
• IMDb dataset (mentioned in README)
• Wikipedia data (enwik8.gz)
Component Relationships
[Data Sources] → [Data Processing] → [Transformer Core] → [Experiments]
Specific Implementation Details
┌─ Experiments Layer ─────────────┐
• Classification Pipeline │
• Generation Pipeline │
• ┬────────────────┘
• ↓
┌─ Transformer Core ──────────────┐
• ┌─ Modules ─────┐ ┌─ Utils ──┐ │
• Attention │ │ - Helper │ │
• Layers │ │ Methods│ │
• ┘ └──────────┘ │
• ┬────────────────┘
• ↓
┌─ Data Layer ──────────────────┐
• IMDb Dataset │
• Wikipedia Corpus (enwik8) │
• ┘
Important Elements to Highlight
a. Single Stack Architecture:
• That this is a single-stack transformer implementation (not encoder-decoder)
b. PyTorch Integration:
• PyTorch as a foundational layer or dependency
• How it interfaces with the transformer implementation
Testing Components
A side panel showing:
┌─ Tests ────────────────┐
• Module Tests │
• Compression Tests │
• Utility Tests │
• ┘