How thuml/itransformer works
Main Components
• Data Provider: Represented by the “data_provider” directory (data_factory.py and data_loader.py). This component is responsible for preparing, loading, and possibly preprocessing the time series data.
• Core Model & Layers: Split into two parts:
• Model Implementations in the “model” directory (e.g., Flashformer.py, Flowformer.py, Informer.py, etc. and their inverted counterparts). These are the various forecasting model definitions.
• Lower-level Transformer Layers located in the “layers” directory (e.g., Embed.py, SelfAttention_Family.py, Transformer_EncDec.py) which implement the core building blocks or modules of the Transformer architectures.
• Experimentation Framework: Represented by the “experiments” directory (exp_basic.py, exp_long_term_forecasting.py, etc.), which shows how different configurations or experiments are set up to test the forecasting models.
• Scripts for Reproducibility & Performance Boosting: The “scripts” directory is heavily organized into subfolders for various experiment runs:
• Performance boosting (e.g., “boost_performance”) for comparing different models,
• Lookback increasing (e.g., “increasing_lookback”),
• Multivariate forecasting, variate generalization, and model efficiency.
• Utility Modules: The “utils” directory provides helper functions (masking, metrics, timefeatures, etc.) that assist various components of the system.
• Main Execution Entry: The “run.py” file, which likely serves as a high-level entry point tying some of these components together.
Relationships and Interactions
• The Data Provider component supplies data to the Core Models.
• The Core Models use the Layers (i.e., specific implementations of attention and embedding modules) and are integrated with utility functions.
• Experiment scripts (and run.py) orchestrate the training, evaluation, and comparison of these models based on provided datasets.
• Scripts in subdirectories further encapsulate run configurations (for different performance boosting scenarios or forecasting setups) by invoking the models with particular settings and datasets.
• Dependencies flow from high-level experiment execution (scripts) down to core modeling (model and layers) and then to data handling and utilities.
Architectural Patterns and Design Principles
• Modularity: The project cleanly separates data ingestion (data_provider), model building (model and layers), and evaluation/experimentation (experiments and scripts).
• Reusability: The existence of utility and helper functions (utils) and modular design in defining different Transformer variants (both standard and inverted).
• Experiment-Driven: The scripts and experiments directories indicate that the architectural design focuses on reproducibility and comparative analysis following research best practices.
• Clear Separation of Concerns: Input/output handling, core model definitions, and experimental validations are all compartmentalized.
Relevant Technologies, Frameworks, and Libraries
• PyTorch: As inferred from the README instructions (“pip install -r requirements.txt”) and typical usage in Transformer implementations.
• External Codebases: References to FlashAttention, and inspiration from projects like Reformer, Informer, Autoformer, etc.
• Shell Scripting: Scripts for executing experiments in Linux environments (e.g., bash scripts in the “scripts” folders).