How jakobovski/decoupled-multimodal-learning works
Main Components to Capture
• Data Sources & Preprocessing:
– Indicate the MNIST image data (found in MNIST_data, utils/MNIST_data, and examples/MNIST_data) and the FSDD audio data.
– Mark preprocessing pipelines or encoding generation scripts (see utils/mnist_encoding_generator.py, fsdd_encoding_generator.py).
• Autoencoder Modules:
– Represent the autoencoder components (modules/autoencoder and modules/cortex/autoencoder) that reduce raw sensory data (both image and audio) into 64-dimensional encodings.
– Label these “Visual Autoencoder” and “Audio Autoencoder.” (Even if the same code isn’t repeated, note their roles from the architectural description.)
• Clustering Components:
– Include the clustering algorithms that form on top of the autoencoder outputs. These are implemented in modules such as modules/cdz/cluster_correlation.py and modules/cdz/cdz.py plus associated files (e.g., db/basic_table.py, db/one_to_many_table.py).
– Indicate that clustering is done per modality, transforming each autoencoder’s output into clusters that naturally represent classes.
• Convergence-Divergence Zone (CDZ):
– Represent the CDZ as a central “hub” that couples clusters across modalities.
– Highlight the one-way (asymmetric) temporal correlation learning that the CDZ provides.
• Shared Utilities and Supporting Modules:
– Note supporting modules like brain.py and config.py that might coordinate the overall execution.
Relationships and Interactions
• Data Flow:
– From autoencoders, direct the “encodings” flow into modality-specific clustering modules.
• Interdependencies:
– Also indicate the feedback loop for generative output (as hinted by the “stream-of-thought” reference in the README).
Architectural Patterns / Design Principles
• Decoupling of modalities. Make clear that each modality (audio vs. visual) has a self-contained pipeline (autoencoder + clustering), then bridged by the CDZ.
• Unsupervised nature: no explicit supervised label handling; instead, operational focus on temporal and spatial co-occurrences.
Relevant Technologies and Files
• That autoencoders use stacked, denoising architectures (layer sizes stated in the README).
• Mention that clustering relies on a novel algorithm (as described in the readme and implemented in modules/cdz/cluster_correlation.py).