# jianjieluo/scd-net architecture

> Architecture diagram of the GitHub repository jianjieluo/scd-net, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jianjieluo/scd-net
- Repository: https://github.com/jianjieluo/scd-net

## Overview

##### Main Components
- Input/Data Preparation: Represent data inputs like the bottom-up attention visual features and datasets (e.g., MSCOCO) which are prepared via scripts in the “tools” directory (e.g., create_feats.py, msrvtt_preprocess.py). Also note the “data” folder and external dataset instructions from the README.
- Configuration & Scripts: The “configs” directory organizes training scripts and YAML configuration files for both Stage1 and Stage2 training (with separate scripts for cross-entropy (XE), RL training, and inference). This shows a clear separation of training phases.
- Diffusion Captioning Pipeline: Illustrate the cascaded nature of the network: Stage1 (baseline captioning training with XE and RL) and Stage2 (refinement using guided self-critical sequence training and knowledge distillation updates through the “tools/update_kd_sents” scripts).
- Core Model and Engine: Use the “xmodaler” folder which contains the essential modeling, datasets, engine, evaluation modules, and components for the diffusion model architecture. This module encompasses:
- Datasets (image and video submodules)
- Engine classes (train loops, different trainer types including RL and diffusion trainers)
- Modeling components (encoders, decoders, embedding layers, and meta architectures that implement the diffusion process with semantic conditioning)
- Evaluation and scoring (for CIDEr-D and retrieval evaluation)
- External Dependencies & Libraries: Acknowledge the use of external modules such as bottom-up attention feature extractors and components borrowed from bit-diffusion and X-Modaler for model loading, distributed training, and so on.

##### Relationships and Interactions
- Data flows from raw image and feature extraction (input features extraction) into the dataset pipeline, then to the model.
- There is an initial Stage1 training process (further split into XE and RL training) where the model learns caption generation with a diffusion process.
- Outputs from Stage1 (captions/inference results) are fed into Stage2 training as additional semantic guidance via the guided self-critical sequence training strategy.
- Tools for post-processing and KD update (located in tools/update_kd_sents) merge predictions from autoregressive teacher models with diffusion outputs for the improved training signal during Stage2.
- The training loop and inference routines in xmodaler/engine orchestrate the necessary calls to model components, while evaluation modules (xmodaler/evaluation) process the outputs.

##### Architectural Patterns & Key Design Principles
- Cascaded and staged training: Emphasize the two-stage approach that isolates initial caption generation and subsequent refinement.
- Modular design: The extensive use of separate modules (configs, tools, xmodaler libraries) suggests an emphasis on reusability and separation of concerns.
- Diffusion with Semantic Conditioning: Focus on how the diffusion process is enriched with semantic priors and updated via guided self-critical training.
- Integration of External Features: The system relies on pre-extracted bottom-up attention features for visual representation.

##### Final Recommendations
- Two-stage cascaded training strategy with clear separations between Stage1 and Stage2 components.
- Incorporate any external or integrated components (e.g., bottom-up attention extraction and external diffusion codebases) to complete the overall architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing Subgraph
    subgraph "Data & Preprocessing"
        DP1["Feature Extraction (tools/create_feats.py)"]
        DP2["MSRVTT Preprocessing (tools/msrvtt_preprocess.py)"]
        DP3["Data Directory (data)"]
    end

    %% Configuration & Scripts Subgraph
    subgraph "Configuration & Scripts"
        CS1["Stage1 Configs (configs/image_caption/scdnet/stage1)"]
        CS2["Stage2 Configs (configs/image_caption/scdnet/stage2)"]
    end

    %% Stage 1 Training Pipeline Subgraph
    subgraph "Stage 1 - Baseline Training"
        T1_XE["XE Training"]
        T1_RL["RL Training"]
        T1_INF["Inference"]
        T1_XE -->|"feeds"| T1_INF
        T1_RL -->|"feeds"| T1_INF
    end

    %% KD Update Subgraph
    subgraph "KD Update"
        KD["KD Update (tools/update_kd_sents)"]
    end

    %% Stage 2 Training Pipeline Subgraph
    subgraph "Stage 2 - Refinement Training"
        T2["Stage2 Training"]
    end

    %% X-Modaler Core Engine Subgraph
    subgraph "X-Modaler Core Engine"
        CE1["Engine & Datasets (xmodaler/engine/train_loop.py)"]
        CE2["Evaluation (xmodaler/evaluation)"]
        CE3["Modeling (xmodaler/modeling)"]
        DS_T["Diffusion Transformer (xmodaler/modeling/bit_diffusion_transformer_enc_dec.py)"]
        DS_S["Diffusion Sampler (xmodaler/modeling/decode_strategy/cascaded_diffusion_sampler.py)"]
        CE3 -->|"contains"| DS_T
        CE3 -->|"contains"| DS_S
    end

    %% Data Flow Connections
    DP1 -->|"prepares"| CS1
    DP2 -->|"prepares"| CS1
    DP3 -->|"provides"| CS1

    CS1 -->|"triggers"| T1_XE
    CS1 -->|"triggers"| T1_RL
    CS2 -->|"configures"| T2

    T1_INF -->|"guides"| KD
    KD -->|"updates"| T2

    T1_INF -->|"executes"| CE1
    T2 -->|"executes"| CE1

    %% Optional direct link from Config to Core Engine for orchestration
    CS1 ---|"orchestrates"| CE1

    %% Styling Classes
    class DP1,DP2,DP3 dpStyle
    class CS1,CS2 configStyle
    class T1_XE,T1_RL,T1_INF,T2 trainingStyle
    class KD kdStyle
    class CE1,CE2,CE3,DS_T,DS_S coreStyle

    %% Click Events for Data & Preprocessing
    click DP1 "https://github.com/jianjieluo/scd-net/blob/main/tools/create_feats.py"
    click DP2 "https://github.com/jianjieluo/scd-net/blob/main/tools/msrvtt_preprocess.py"
    click DP3 "https://github.com/jianjieluo/scd-net/tree/main/data"

    %% Click Events for Configuration & Scripts
    click CS1 "https://github.com/jianjieluo/scd-net/tree/main/configs/image_caption/scdnet/stage1"
    click CS2 "https://github.com/jianjieluo/scd-net/tree/main/configs/image_caption/scdnet/stage2"

    %% Click Event for KD Update
    click KD "https://github.com/jianjieluo/scd-net/tree/main/tools/update_kd_sents"

    %% Click Events for Core Model Engine
    click CE1 "https://github.com/jianjieluo/scd-net/blob/main/xmodaler/engine/train_loop.py"
    click CE2 "https://github.com/jianjieluo/scd-net/tree/main/xmodaler/evaluation"
    click CE3 "https://github.com/jianjieluo/scd-net/tree/main/xmodaler/modeling"
    click DS_T "https://github.com/jianjieluo/scd-net/blob/main/xmodaler/modeling/bit_diffusion_transformer_enc_dec.py"
    click DS_S "https://github.com/jianjieluo/scd-net/blob/main/xmodaler/modeling/decode_strategy/cascaded_diffusion_sampler.py"

    %% Class Definitions (colors)
    classDef dpStyle fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef configStyle fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef trainingStyle fill:#FFFF99,stroke:#000,stroke-width:2px;
    classDef kdStyle fill:#E6E6FA,stroke:#000,stroke-width:2px;
    classDef coreStyle fill:#FFC0CB,stroke:#000,stroke-width:2px;
```
