How amaldo123/ngc-pc-transformers works
NGC-PC-Transformers is a predictive-coding language-model implementation. Its user workflow tokenizes text, prepares next-token batches, trains the model, then evaluates or generates text. The model combines embedding, attention, MLP, and output components with a projection subsystem used in predictive-coding updates; attention and embedding utilities provide internal computation. The README documents the tokenize/train/evaluate/generate commands. Some connections are based on the SOURCE INDEX imports rather than complete excerpts, so unsampled implementation details are not expanded.
Main entry points: model.py Predictive transformer · blocks.py Transformer block · proj_block.py Projection block · train.py Training
Components 23
Language workflow
- Tokenizer
data_preprocess/tokenizer.py - Token batches
data_preprocess/data_loader.py - Training
train.py - Evaluation
eval.py - Text generation
generation.py - Model configuration
config.py
Predictive model
- Predictive transformer
model.py - Token embedding
layers/embedding.py - Transformer block
layers/blocks.py - Self-attention
layers/attention.py - Feed-forward MLP
layers/mlp.py - Vocabulary output
layers/output.py
Projection and learning
- Predictive projection
projection/projection.py - Projection block
projection/proj_block.py
Model internals
- Embedding synapse
utils/embed_utils.py - Attention computation
utils/attention_utils.py - Attention rate cell
utils/attn_ratecell.py - Model utilities
utils/model_util.py - Residual operations
utils/residual_util.py - RMS normalization
utils/rms_norm_util.py - RoPE utilities
utils/rope_utils.py
Other components
- User
- Text corpus
data_preprocess/data
Connections 34
- User → Tokenizer: runs tokenization
README.md - Text corpus → Tokenizer: tokenizes
- Tokenizer → Token batches: produces token data
- User → Training: starts training
README.md - Token batches → Training: supplies batches
train.py - Training → Predictive transformer: trains
train.py - Training → Evaluation: invokes evaluation
train.py - User → Evaluation: runs evaluation
README.md - Token batches → Evaluation: supplies batches
eval.py - Evaluation → Predictive transformer: evaluates
eval.py - User → Text generation: runs generation
README.md - Text generation → Predictive transformer: generates with
generation.py - Text generation → Tokenizer: uses tokenizer
generation.py - Model configuration → Training: configures
train.py - Model configuration → Predictive transformer: configures
model.py - Model configuration → Tokenizer: configures
data_preprocess/tokenizer.py - Predictive transformer → Token embedding: uses
model.py - Predictive transformer → Transformer block: uses
model.py - Predictive transformer → Vocabulary output: uses
model.py - Predictive transformer → Predictive projection: uses
model.py - Transformer block → Self-attention: constructs
layers/blocks.py - Transformer block → Feed-forward MLP: constructs
layers/blocks.py - Transformer block → Residual operations: uses
layers/blocks.py - Transformer block → RMS normalization: imports
layers/blocks.py - Token embedding → Embedding synapse: uses
layers/embedding.py - Self-attention → Attention computation: uses
layers/attention.py - Self-attention → Attention rate cell: uses
layers/attention.py - Attention computation → Model utilities: uses softmax utility
utils/attention_utils.py - Attention computation → RoPE utilities: uses positional encoding
utils/attention_utils.py - Predictive projection → Projection block: constructs
projection/projection.py - Projection block → Attention computation: uses attention
projection/proj_block.py - Projection block → Attention rate cell: uses dynamics
projection/proj_block.py - Projection block → Feed-forward MLP: uses
projection/proj_block.py - Projection block → Model utilities: uses reshape utility
projection/proj_block.py