How mohebabdelhafidh/memory-in-action-conditioned-world-models-gru-vs.-transformer-vs.-xlstm works
This is a small research system for comparing recurrent sequence cores in action-conditioned world models on a partially observable cue-recall task. Episodes are generated from the simulator and loaded as padded sequences; training fits the model to predict next frames and rewards. The trained model is then evaluated and used for imagination-based planning/visualization. The shared model encodes frames and actions, runs a selectable dynamics core, and decodes predictions. Some evaluation and planning modules are unsampled, so their internal connections are omitted or marked unverified.
Main entry points: world_model.py Action-conditioned model · visualize.py Imagination visualization · train.py Model training · dataset.py Episode dataset
Components 18
Task and data
- Cue-recall simulator
envs/cue_recall.py - Episode collection
data/collect.py - Episode dataset
data/dataset.py
World model
- Action-conditioned model
models/world_model.py - Frame encoder
models/encoder.py - Frame decoder
models/decoder.py - Dynamics core interface
models/cores/base.py - GRU core
models/cores/gru.py - Memoryless control
models/cores/memoryless.py - Transformer core
models/cores/transformer.py - xLSTM core
models/cores/xlstm_core.py
Training and evaluation
- Model training
train.py - Model evaluation
eval - Efficiency tests
eval/efficiency.py - Gap sweeps
eval/gap_sweep.py
Planning and analysis
- CEM model planning
planning/cem_mpc.py - Imagination visualization
eval/visualize.py
Other components
- Researcher
Connections 20
- Episode collection → Cue-recall simulator: runs episodes
data/collect.py - Episode dataset → Episode collection: loads episodes
data/dataset.py - Model training → Episode dataset: loads batches
train.py - Model training → Action-conditioned model: trains
train.py - Action-conditioned model → Frame encoder: encodes frames
models/world_model.py - Action-conditioned model → Frame decoder: decodes predictions
models/world_model.py - Action-conditioned model → Dynamics core interface: runs dynamics
models/world_model.py - Action-conditioned model → GRU core: builds core
models/world_model.py - Action-conditioned model → Memoryless control: builds core
models/world_model.py - Action-conditioned model → Transformer core: builds core
models/world_model.py - Action-conditioned model → xLSTM core: builds core
models/world_model.py - Model evaluation → Action-conditioned model: evaluates model
eval/visualize.py - Efficiency tests → Action-conditioned model: tests efficiency
eval/efficiency.py - Gap sweeps → Episode dataset: loads episodes
eval/gap_sweep.py - Gap sweeps → Action-conditioned model: sweeps gaps
eval/gap_sweep.py - Researcher → Model training: starts training
- CEM model planning → Action-conditioned model: plans by imagination
- Imagination visualization → Cue-recall simulator: builds context
eval/visualize.py - Imagination visualization → Action-conditioned model: imagines outcomes
eval/visualize.py - Imagination visualization → Frame decoder: converts frames
eval/visualize.py