How yechanjung/semantic_fusion works
Project type and purpose
• This is a research-oriented example implementation of the “Semantic Fusion” approach in reinforcement learning (RL).
• It demonstrates how to treat perception outputs and physical state as separate modalities, encode each independently, and fuse them via cross-modal attention inside a PPO agent.
a. Environment (e.g., OpenAI Gym)
b. Perception Model (external vision or detector producing semantic outputs)
c. Physical State Extractor (reads raw state: positions, velocities)
d. PhysicalStateEncoder (nn.Module)
e. SemanticOutputEncoder (nn.Module)
f. CrossModalAttention (nn.Module)
g. SemanticFusionPPO agent (policy & value networks)
h. Training Loop / Optimizer (PPO update mechanism)
Relationships and data flows
• The Environment emits two streams each time step:
• Raw physical state → PhysicalStateEncoder → physical feature vector
• Sensor/perception outputs → SemanticOutputEncoder → semantic feature vector
• Both feature vectors flow into CrossModalAttention → fused latent representation
• Fused representation feeds into the SemanticFusionPPO policy head (action probabilities) and value head (state value)
• Selected action is sent back to the Environment
• Rewards, next state, and next perception outputs loop back into the encoders, and gradients flow back through all modules during the PPO update
Architectural patterns and design principles
• Modularity: separate encoder modules for each modality
• Clear separation of concerns: perception vs. physical state vs. fusion vs. policy
• Attention-based fusion: adaptive weighting of modalities rather than naive concatenation
• End-to-end differentiability: entire pipeline—from encoders through PPO update—is a single computational graph
Essentials to capture
• Two independent encoder paths and their inputs
• Cross-modal attention as a separate fusion stage
• The PPO agent consuming fused output to produce actions
• The closed-loop interaction between agent and environment
• The end-to-end nature of the training process