How josejuan98/transformer-attention-with-hdc-binding works
Project Type and Purpose
• This is an open-source research framework and library for experimenting with Transformer models augmented by Hyperdimensional Computing (HDC) binding for positional encodings, specifically targeting time-series classification tasks (with potential NLP extensions).
• It combines:
• A reusable experiment framework (data loading, preprocessing, runner, metrics, error handling)
• A modular model library (embedding layers, binding operations, positional encodings, Transformer components)
• Scripts and configuration for orchestrating experiments, visualizations, and hardware backends
High-Level Components
a. Experiment CLI & Orchestration
• Makefile & shell scripts (scripts/)
• Poetry/pyproject for dependency management
• GitHub Actions workflows for linting and testing
b. Experiment Framework (src/experiment_framework)
• Config parsers & factories (config/)
• Data pipeline & datamodule (data/)
• Runner (training loop, metrics handling, error handling) (runner/)
c. Model Library (src/models)
• Architectures (time-series transformer classifier)
• Embedding layers (linear projection, convolutional)
• Binding methods (additive, multiplicative, circular convolution)
• Positional encodings (sinusoidal variants, random, fractional, HDC binding integration)
• Transformer internals (attention, feed-forward, encoder)
• Callbacks (learning rate finder, fine-tuning)
d. Experiment Definitions & Scripts (src/experiments)
• Time series experiment configurations and launcher (run.py)
• Metric aggregation, layer & similarity visualizations
e. Utilities (src/utils)
• Logging, config utilities, plot helpers, cache management
f. Data & Artifacts
• External UCR/UEA datasets (downloaded via datamodule)
• Local artifacts/ directory for datasets, model checkpoints, logs, metrics
• docs/experiment_results for aggregated CSVs and plots
g. Testing (src/test)
• Unit tests for framework, models, transformer modules
Interactions and Data Flow
1. User runs make run-ts or invokes scripts/install.sh / setup_cuda_server.sh.
2. Makefile calls the experiment script (src/experiments/time_series/run.py), passing experiment and model configuration.
3. Runner loads config via experiment_framework.config, instantiates datamodule and model via factories.
4. Data pipeline (experiment_framework.data) fetches and preprocesses UCR/UEA datasets.
5. Model library constructs embedding + binding + positional encoding + Transformer encoder layers.
6. Training loop executes forward/backward passes, invoking callbacks and metrics handlers.
7. Metrics and checkpoints are saved under artifacts/models and logged via utils/log.py.
8. Post-processing scripts aggregate metrics (src/experiments/aggregate_metrics.py) and visualization modules generate plots in docs/plots.
Architectural Patterns and Principles
• Factory Pattern: for configs, models, data modules, binding and embedding choices.
• Separation of Concerns: clear layers—config, data, model, runner, scripts, utilities.
• Modular & Extensible: adding a new binding method only requires updating models/binding/factory.py and implementing the new class.
• Reproducibility: fixed random seeds, config-driven experiments, automated CI.