How rodrigomaulen/attention_based_pca works
This repository is a collection of standalone JAX experiments for studying attention-based PCA. The user-facing workflow is to run an experiment script and obtain a paper figure. The scripts cover linear risk, softmax attention, in-context learning (ICL), and comparisons between finite-prompt SGD and infinite-prompt gradient descent; several vary dimension or sequence length. The excerpts show experiment and plotting logic, but not every script’s output details, so the figure relationship is documented at the collection level rather than as a detailed runtime pipeline.
Main entry points: linear_finite.py Finite Linear · linear_finite_change_L.py Linear vs Length · linear_analytic.py Analytic Linear · linear_analytic_change_dim.py Linear vs Dimension
Components 15
Linear Risk
- Finite Linear
linear_finite.py - Linear vs Length
linear_finite_change_L.py - Analytic Linear
linear_analytic.py - Linear vs Dimension
linear_analytic_change_dim.py
Softmax Attention
- Finite Softmax
softmax_finite.py - Softmax vs Length
softmax_finite_alignment_vs_L.py - Infinite Softmax
softmax_infinite.py - Softmax vs Dimension
softmax_infinite_change_dim.py
In-Context Learning
- Finite ICL
icl_finite.py - ICL vs Length
icl_finite_alignment_vs_L.py - Infinite ICL
icl_infinite.py - ICL vs Dimension
icl_infinite_change_dim.py
Prompt Comparisons
- Softmax Comparison
combined_softmax_infinite_finite.py - ICL Comparison
combined_icl_infinite_finite.py
Other components
- Paper Figures
Connections 14
- Finite Linear → Paper Figures: creates figure
README.md - Linear vs Length → Paper Figures: creates figure
README.md - Analytic Linear → Paper Figures: creates figure
README.md - Linear vs Dimension → Paper Figures: creates figure
README.md - Finite Softmax → Paper Figures: creates figure
README.md - Softmax vs Length → Paper Figures: creates figure
README.md - Infinite Softmax → Paper Figures: creates figure
README.md - Softmax vs Dimension → Paper Figures: creates figure
README.md - Finite ICL → Paper Figures: creates figure
README.md - ICL vs Length → Paper Figures: creates figure
README.md - Infinite ICL → Paper Figures: creates figure
README.md - ICL vs Dimension → Paper Figures: creates figure
README.md - Softmax Comparison → Paper Figures: creates figure
README.md - ICL Comparison → Paper Figures: creates figure
README.md