# jakobovski/decoupled-multimodal-learning architecture

> Architecture diagram of the GitHub repository jakobovski/decoupled-multimodal-learning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jakobovski/decoupled-multimodal-learning
- Repository: https://github.com/jakobovski/decoupled-multimodal-learning

## Overview

##### Main Components to Capture
- Data Sources & Preprocessing:
– Indicate the MNIST image data (found in MNIST_data, utils/MNIST_data, and examples/MNIST_data) and the FSDD audio data.
– Mark preprocessing pipelines or encoding generation scripts (see utils/mnist_encoding_generator.py, fsdd_encoding_generator.py).

- Autoencoder Modules:
– Represent the autoencoder components (modules/autoencoder and modules/cortex/autoencoder) that reduce raw sensory data (both image and audio) into 64-dimensional encodings.
– Label these “Visual Autoencoder” and “Audio Autoencoder.” (Even if the same code isn’t repeated, note their roles from the architectural description.)

- Clustering Components:
– Include the clustering algorithms that form on top of the autoencoder outputs. These are implemented in modules such as modules/cdz/cluster_correlation.py and modules/cdz/cdz.py plus associated files (e.g., db/basic_table.py, db/one_to_many_table.py).
– Indicate that clustering is done per modality, transforming each autoencoder’s output into clusters that naturally represent classes.

- Convergence-Divergence Zone (CDZ):
– Represent the CDZ as a central “hub” that couples clusters across modalities.
– Highlight the one-way (asymmetric) temporal correlation learning that the CDZ provides.

- Shared Utilities and Supporting Modules:
– Note supporting modules like brain.py and config.py that might coordinate the overall execution.

##### Relationships and Interactions
- Data Flow:
– From autoencoders, direct the “encodings” flow into modality-specific clustering modules.

- Interdependencies:
– Also indicate the feedback loop for generative output (as hinted by the “stream-of-thought” reference in the README).

##### Architectural Patterns / Design Principles
- Decoupling of modalities. Make clear that each modality (audio vs. visual) has a self-contained pipeline (autoencoder + clustering), then bridged by the CDZ.
- Unsupervised nature: no explicit supervised label handling; instead, operational focus on temporal and spatial co-occurrences.

##### Relevant Technologies and Files
- That autoencoders use stacked, denoising architectures (layer sizes stated in the README).
- Mention that clustering relies on a novel algorithm (as described in the readme and implemented in modules/cdz/cluster_correlation.py).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    subgraph "Input Layer"
        mnist["MNIST Data"]:::data
        fsdd["FSDD Audio Data"]:::data
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        mnist_enc["MNIST Encoding Generator"]:::preprocessing
        fsdd_enc["FSDD Encoding Generator"]:::preprocessing
    end

    %% Visual Processing
    subgraph "Visual Processing"
        v_auto["Visual Autoencoder"]:::autoencoder
        v_cluster["Visual Clustering"]:::clustering
        basicTable["Basic Table"]:::clustering
        oneToMany["One-To-Many Table"]:::clustering
    end

    %% Audio Processing
    subgraph "Audio Processing"
        a_auto["Audio Autoencoder"]:::autoencoder
        a_cluster["Audio Clustering"]:::clustering
    end

    %% Integration Layer
    subgraph "Integration Layer"
        cdz["Convergence-Divergence Zone (CDZ) Engine"]:::cdz
    end

    %% Shared Utilities
    subgraph "Shared Utilities"
        sharedData["Data Packet Utility"]:::shared
        brainCtrl["Brain Controller"]:::shared
        configMod["Config Module"]:::shared
    end

    %% Data Flow Connections
    mnist -->|"data_flow"| mnist_enc
    fsdd -->|"data_flow"| fsdd_enc

    mnist_enc -->|"encoding"| v_auto
    fsdd_enc -->|"encoding"| a_auto

    v_auto -->|"encoding"| v_cluster
    a_auto -->|"encoding"| a_cluster

    v_cluster -->|"cluster"| cdz
    a_cluster -->|"cluster"| cdz

    %% Feedback / Excitation Loop
    cdz -->|"excitation"| v_cluster
    cdz -->|"excitation"| a_cluster

    %% Clustering Support Connections
    v_cluster -->|"support"| basicTable
    v_cluster -->|"support"| oneToMany

    %% Shared Utilities and Coordination
    brainCtrl -->|"coordination"| v_auto
    brainCtrl -->|"coordination"| a_auto
    configMod -->|"configuration"| brainCtrl

    %% Styles
    classDef data fill:#f6d186,stroke:#333,stroke-width:2px;
    classDef preprocessing fill:#c7f464,stroke:#333,stroke-width:2px;
    classDef autoencoder fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef clustering fill:#ffcccb,stroke:#333,stroke-width:2px;
    classDef cdz fill:#d3d3d3,stroke:#333,stroke-width:2px;
    classDef shared fill:#ffebcd,stroke:#333,stroke-width:2px;

    %% Click Events
    click mnist "https://github.com/jakobovski/decoupled-multimodal-learning/tree/master/MNIST_data"
    click fsdd "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/data/fsdd_train_encodings_labels.npy"
    click mnist_enc "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/utils/mnist_encoding_generator.py"
    click fsdd_enc "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/utils/fsdd_encoding_generator.py"
    click v_auto "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/modules/autoencoder/autoencoder.py"
    click a_auto "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/modules/cortex/autoencoder.py"
    click v_cluster "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/modules/cdz/cluster_correlation.py"
    click a_cluster "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/modules/cortex/cluster.py"
    click basicTable "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/db/basic_table.py"
    click oneToMany "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/db/one_to_many_table.py"
    click cdz "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/modules/cdz/cdz.py"
    click sharedData "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/modules/shared_components/data_packet.py"
    click brainCtrl "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/brain.py"
    click configMod "https://github.com/jakobovski/decoupled-multimodal-learning/blob/master/config.py"
```
