# qiuzh20/gated_attention architecture

> Architecture diagram of the GitHub repository qiuzh20/gated_attention, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qiuzh20/gated_attention
- Repository: https://github.com/qiuzh20/gated_attention

## Overview

##### Project Type and Purpose
- This is a research‐oriented Python codebase that extends the Qwen3 transformer architecture with gated attention mechanisms and provides tools for visualizing the resulting attention maps and training benchmarks. It is neither a web service nor a microservice ecosystem, but rather a self‐contained library plus demo/analysis scripts with accompanying data assets.

##### Main Components
a. Core Modules
- configuration_qwen3.py – defines the `Qwen3Config` extensions to toggle headwise or elementwise gating.
- modeling_qwen3.py – implements the modified `Qwen3Attention` class with gating logic inserted after SDPA.
b. Demo & Visualization
- demo.py (demoe.py) – demo script that:
– loads a pretrained Qwen3 variant via HuggingFace Transformers
– injects the gating configuration
– runs a forward pass to extract attention weights
– uses Matplotlib to render attention‐map PNGs
c. Analysis Notebooks and Figures
- figs/plot_compare.ipynb, plot_curve.ipynb – Jupyter notebooks that load TensorBoard curve files (`.pt`) or benchmark data to produce PDF/PNG plots.
- figs/tb_curves/*.pt – serialized training curves used by the notebooks.
- figs/*.pdf – pre‐rendered figures included in the paper.
d. External Dependencies
- PyTorch (torch)
- HuggingFace Transformers
- NumPy, Matplotlib

##### Relationships and Data Flow
1) configuration_qwen3.py → modeling_qwen3.py
– The config flags (`headwise_attn_output_gate`, `elementwise_attn_output_gate`) are read by the `Qwen3Attention` constructor.
2) demo.py → (configuration_qwen3, modeling_qwen3)
– demo.py imports both modules, builds a model with gating enabled, and runs inference.
3) demo.py → HuggingFace Transformers API
– Loads pretrained weights from the HuggingFace Hub.
4) demo.py → Matplotlib → output `{model_name}_selected_layer_attention_maps.png`
5) figs notebooks → `.pt` curves → Matplotlib → PDF/PNG plots for benchmarks

##### Architectural Patterns and Design Principles
- Modular design: separation of configuration, model logic, and demo/visualization.
- Plugin‐style gating: gating options are purely configurable, no hardcoding of variants.
- Data‐pipeline pattern in notebooks: deserialization of TB curves → processing → visualization.
- Reuse of external ML infrastructure (Transformers, TensorBoard, Matplotlib).

a. Top-Level View
– Group code modules on the left: configuration_qwen3.py → modeling_qwen3.py → demo.py.
– Place external libraries (Transformers, PyTorch) as clouds to the right of demo.py.
b. Visualization Subsystem
– Show demo.py feeding attention matrices to Matplotlib → output PNG.
– Show figs notebooks reading tb_curves `.pt` → Matplotlib → output PDF.
c. Data Layer
– Position `figs/tb_curves/*.pt` and prebuilt `figs/*.pdf` below the notebooks.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Code Modules
    subgraph "Core Modules"
        direction TB
        Config["configuration_qwen3.py"]:::code
        Modeling["modeling_qwen3.py"]:::code
        Demo["demoe.py"]:::code
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        PyTorch["PyTorch"]:::external
        Transformers["HuggingFace Transformers"]:::external
        Matplotlib["Matplotlib"]:::external
        HFHub["HuggingFace Hub"]:::external
    end

    %% Visualization & Analysis
    subgraph "Visualization & Analysis"
        direction TB
        CompareNotebook["plot_compare.ipynb"]:::data
        CurveNotebook["plot_curve.ipynb"]:::data
        subgraph "Training Curve Data"
            direction TB
            CurveData1["3T-baseline.pt"]:::data
            CurveData2["3T-gate.pt"]:::data
        end
        subgraph "Pre-rendered Figures"
            direction TB
            PreFig1["benchmark_ppl_comparison.pdf"]:::output
            PreFig2["lm_loss.pdf"]:::output
        end
        NotebookOutput["benchmarks_plots.pdf/png"]:::output
    end

    %% Outputs
    AttentionPNG["attention_maps.png"]:::output

    %% Connections
    Config -->|"import flags"| Modeling
    Demo -->|"import"| Config
    Demo -->|"import"| Modeling
    Demo -->|"load_weights()"| HFHub
    Demo -->|"uses"| Transformers
    Demo -->|"uses"| PyTorch
    Demo -->|"uses"| Matplotlib
    Demo -->|"forward() → attn maps"| AttentionPNG
    AttentionPNG -->|"savefig()"| AttentionPNG

    CurveData1 -.->|"load .pt"| CompareNotebook
    CurveData2 -.->|"load .pt"| CurveNotebook
    CompareNotebook -->|"plot()"| NotebookOutput
    CurveNotebook -->|"plot()"| NotebookOutput
    CompareNotebook -->|"reads"| PreFig1
    CurveNotebook -->|"reads"| PreFig2

    %% Click Events
    click Config "https://github.com/qiuzh20/gated_attention/blob/main/configuration_qwen3.py"
    click Modeling "https://github.com/qiuzh20/gated_attention/blob/main/modeling_qwen3.py"
    click Demo "https://github.com/qiuzh20/gated_attention/blob/main/demoe.py"
    click CompareNotebook "https://github.com/qiuzh20/gated_attention/blob/main/figs/plot_compare.ipynb"
    click CurveNotebook "https://github.com/qiuzh20/gated_attention/blob/main/figs/plot_curve.ipynb"
    click CurveData1 "https://github.com/qiuzh20/gated_attention/blob/main/figs/tb_curves/3T-baseline.pt"
    click CurveData2 "https://github.com/qiuzh20/gated_attention/blob/main/figs/tb_curves/3T-gate.pt"
    click PreFig1 "https://github.com/qiuzh20/gated_attention/blob/main/figs/benchmark_ppl_comparison.pdf"
    click PreFig2 "https://github.com/qiuzh20/gated_attention/blob/main/figs/lm_loss.pdf"

    %% Styles
    classDef code fill:#cfe2f3,stroke:#0b5394,color:#073763
    classDef external fill:#f9cb9c,stroke:#b45f06,color:#783f04
    classDef data fill:#d9ead3,stroke:#38761d,color:#274e13
    classDef output fill:#eeeeee,stroke:#666666,color:#333333
```
