# lucidrains/tab-transformer-pytorch architecture

> Architecture diagram of the GitHub repository lucidrains/tab-transformer-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucidrains/tab-transformer-pytorch
- Repository: https://github.com/lucidrains/tab-transformer-pytorch

## Overview

##### Project Type & Purpose
- This is a deep learning library focused on tabular data. Its core functionality is to provide two transformer-based modules (TabTransformer and FTTransformer) that process categorical and continuous data.
- The README emphasizes usage examples in PyTorch, installation via pip, and comparisons between transformer methods, so it’s designed as a tool/plugin for data scientists and ML engineers.

##### Key Components
a. Core Transformer Modules:
- A module for TabTransformer (implemented in tab_transformer_pytorch/tab_transformer_pytorch.py) that embeds categorical data, processes continuous inputs (normalized optionally via provided mean and std), and then passes them through multi-head attention layers, feedforward networks, and final MLP layers.
- A module for FTTransformer (in tab_transformer_pytorch/ft_transformer.py) as an alternative architecture that simplifies the processing of continuous data, inspired by the referenced paper from Yandex.

b. External Dependencies:
- Standard library and configuration artifacts (for example, the setup.py which ensures the package can be installed properly).

c. Data Flow / Interactions:
- Input data: Show a block that details the input types (categorical data and continuous/numerical values).
- Embedding Layers: Illustrate the embedding of categorical data and potentially the normalization for continuous values.
- Transformer Layers: Use a clear block that represents the multi-head self-attention, transformer blocks (stacked “depth” times), and associated dropout layers.

d. Alternative Execution Paths:
- Unsupervised training routine: Although not a full module in this repository, the README mentions an unsupervised training approach with Electra on the transformer, so include a note/branch for this potential pipeline.

##### Architectural Patterns
- Data Preprocessing (standalone block where continuous values might be normalized)
- Transformer-based Feature Extraction (split into TabTransformer and FTTransformer branches)
- Prediction Head (the final layers outputting predictions)
- These modules interact in a sequential manner (data flows from inputs → embedding/normalization → transformer blocks → output layer).

## Diagram (Mermaid)

```mermaid
flowchart TD
    Config["Package Configuration & Installation"]:::config
    Input["Input Data (Categorical & Continuous)"]:::data
    CE["Categorical Data Encoder"]:::preprocess
    CN["Continuous Data Normalizer"]:::preprocess
    Merge["Merged Preprocessed Data"]:::preprocess
    Fx["Feature Extraction Choice"]:::transformation
    TT["TabTransformer Module<br/>(heads:8,depth:6)"]:::transformer
    FT["FTTransformer Module<br/>(simplified continuous processing)"]:::transformer
    PH["MLP/Prediction Head"]:::prediction
    Output["Prediction Output"]:::output
    Torch["PyTorch"]:::external
    Electra["Optional Unsupervised Training (Electra)"]:::optional

    Config -->|"installs"| Input
    Input --> CE
    Input --> CN
    CE --> Merge
    CN --> Merge
    Merge --> Fx
    Fx --> TT
    Fx --> FT
    TT --> PH
    FT --> PH
    PH --> Output

    Torch --> TT
    Torch --> FT
    TT -.-> Electra

    %% Click Events
    click TT "https://github.com/lucidrains/tab-transformer-pytorch/blob/main/tab_transformer_pytorch/tab_transformer_pytorch.py"
    click FT "https://github.com/lucidrains/tab-transformer-pytorch/blob/main/tab_transformer_pytorch/ft_transformer.py"
    click Config "https://github.com/lucidrains/tab-transformer-pytorch/blob/main/setup.py"

    %% Classes and Styles
    classDef data fill:#D0E1F9,stroke:#1D3557,stroke-width:2px;
    classDef preprocess fill:#CFFFD3,stroke:#2E7D32,stroke-width:2px;
    classDef transformation fill:#FFF3BF,stroke:#F57F17,stroke-width:2px;
    classDef transformer fill:#FFE0B2,stroke:#E65100,stroke-width:2px;
    classDef prediction fill:#F8BBD0,stroke:#AD1457,stroke-width:2px;
    classDef output fill:#D1C4E9,stroke:#5E35B1,stroke-width:2px;
    classDef external fill:#E1F5FE,stroke:#0277BD,stroke-width:2px;
    classDef optional fill:#B2DFDB,stroke:#00796B,stroke-width:2px;
    classDef config fill:#FFECB3,stroke:#F9A825,stroke-width:2px;
```
