# yechanjung/semantic_fusion architecture

> Architecture diagram of the GitHub repository yechanjung/semantic_fusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yechanjung/semantic_fusion
- Repository: https://github.com/yechanjung/semantic_fusion

## Overview

##### Project type and purpose
- This is a research-oriented example implementation of the “Semantic Fusion” approach in reinforcement learning (RL).
- It demonstrates how to treat perception outputs and physical state as separate modalities, encode each independently, and fuse them via cross-modal attention inside a PPO agent.

a. Environment (e.g., OpenAI Gym)
b. Perception Model (external vision or detector producing semantic outputs)
c. Physical State Extractor (reads raw state: positions, velocities)
d. PhysicalStateEncoder (nn.Module)
e. SemanticOutputEncoder (nn.Module)
f. CrossModalAttention (nn.Module)
g. SemanticFusionPPO agent (policy & value networks)
h. Training Loop / Optimizer (PPO update mechanism)

##### Relationships and data flows
- The Environment emits two streams each time step:
- Raw physical state → PhysicalStateEncoder → physical feature vector
- Sensor/perception outputs → SemanticOutputEncoder → semantic feature vector
- Both feature vectors flow into CrossModalAttention → fused latent representation
- Fused representation feeds into the SemanticFusionPPO policy head (action probabilities) and value head (state value)
- Selected action is sent back to the Environment
- Rewards, next state, and next perception outputs loop back into the encoders, and gradients flow back through all modules during the PPO update

##### Architectural patterns and design principles
- Modularity: separate encoder modules for each modality
- Clear separation of concerns: perception vs. physical state vs. fusion vs. policy
- Attention-based fusion: adaptive weighting of modalities rather than naive concatenation
- End-to-end differentiability: entire pipeline—from encoders through PPO update—is a single computational graph

##### Essentials to capture
- Two independent encoder paths and their inputs
- Cross-modal attention as a separate fusion stage
- The PPO agent consuming fused output to produce actions
- The closed-loop interaction between agent and environment
- The end-to-end nature of the training process

## Diagram (Mermaid)

```mermaid
flowchart TB
    Environment["Environment (OpenAI Gym)"]:::env
    PhysicalStateExtractor["Physical State Extractor"]:::encoder
    PhysicalStateEncoder["PhysicalStateEncoder (nn.Module)"]:::encoder
    PerceptionModel["Perception Model"]:::encoder
    SemanticOutputEncoder["SemanticOutputEncoder (nn.Module)"]:::encoder
    CrossModalAttention["CrossModalAttention (nn.Module)"]:::fusion
    SemanticFusionPPO["SemanticFusionPPO Agent\n(policy & value networks)"]:::agent
    TrainingLoop[/"Training Loop & Optimizer\n(PPO update mechanism)"/]:::training

    Environment -->|"raw physical state"| PhysicalStateExtractor
    PhysicalStateExtractor -->|"physical_state"| PhysicalStateEncoder
    Environment -->|"perception outputs"| PerceptionModel
    PerceptionModel -->|"semantic data"| SemanticOutputEncoder
    PhysicalStateEncoder -->|"physical_features"| CrossModalAttention
    SemanticOutputEncoder -->|"semantic_features"| CrossModalAttention
    CrossModalAttention -->|"fused_representation"| SemanticFusionPPO
    SemanticFusionPPO -->|"action"| Environment
    SemanticFusionPPO -->|"gradients"| TrainingLoop

    TrainingLoop -.-> PhysicalStateEncoder
    TrainingLoop -.-> SemanticOutputEncoder
    TrainingLoop -.-> CrossModalAttention
    TrainingLoop -.-> SemanticFusionPPO

    click Environment "https://github.com/yechanjung/semantic_fusion/blob/main/semantic_fusion.py"
    click SemanticOutputEncoder "https://github.com/yechanjung/semantic_fusion/blob/main/semantic_fusion.py"
    click PhysicalStateEncoder "https://github.com/yechanjung/semantic_fusion/blob/main/semantic_fusion.py"
    click CrossModalAttention "https://github.com/yechanjung/semantic_fusion/blob/main/semantic_fusion.py"
    click SemanticFusionPPO "https://github.com/yechanjung/semantic_fusion/blob/main/semantic_fusion.py"
    click TrainingLoop "https://github.com/yechanjung/semantic_fusion/blob/main/semantic_fusion.py"

    classDef env fill:#FFA500,stroke:#333;
    classDef encoder fill:#ADD8E6,stroke:#333;
    classDef fusion fill:#90EE90,stroke:#333;
    classDef agent fill:#D8BFD8,stroke:#333;
    classDef training fill:#D3D3D3,stroke:#333;
```
