# markvdw/gparml architecture

> Architecture diagram of the GitHub repository markvdw/gparml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/markvdw/gparml
- Repository: https://github.com/markvdw/gparml

## Overview

##### Main Components
a. Entry­points / Orchestration Layer
– parallel_GPLVM.py
– gd-example.py, scg-example.py, scg_adapted-example.py (sequential runs)
b. Core Computation Modules
– partial_terms.py (computes variational partial sums)
– kernels.py / kernel_exp.py (covariance functions)
– scg.py, scg_adapted.py, gd.py (optimisers)
c. Parallelization Engines (MapReduce Pattern)
– local_MapReduce.py (local multithreading/process)
– SGE_MapReduce.py (Sun Grid Engine integration)
– (Hadoop streaming via simplejson/jar optional)
d. Supporting Utilities
– supporting_functions.py (data checks, sanity tests)
– nputil.py, pre_process.py, predict.py
– tools/ (easy_dataset.py, split_data.py, show_embeddings.py)
e. Data Storage
– Input folder (CSV files of Y)
– Embeddings folder (X initial or updated)
– Statistics folder (intermediate stats per iteration)
– Tmp folder (shared scratch for MapReduce)

##### Out Relationships & Data Flow
– Start: User invokes parallel_GPLVM.py with arguments (–i, –e, –p, –T, etc.).
– Initialization: pre_process.py loads inputs; tools/pca or random init writes seed embeddings into “embeddings/”.
– Iteration Loop (T times):
- Stage (per node or thread):
– Load subset of Y, X from input/emeddings.
– Compute partial_terms (calls kernels, kernel_exp).
– Optionally run gd/scg/scg_adapted on local partials.
– Write partial outputs to “tmp/” or SGE scratch.
- Reduce Stage (master):
– supporting_functions.py / local_MapReduce.py collects all partials.
– Aggregates sums, updates global parameters (X, kernel hyperparameters, beta).
– Write new embeddings and statistics to respective folders.
- Loop back for next iteration.
– End: final embeddings in “embeddings/”, stats in “statistics/”. Optionally call predict.py.

##### Architectural Patterns & Principles
– MapReduce: clear Map (compute local partial) vs. Reduce (aggregate).
– Plug-in optimisers: gd, scg, scg_adapted follow same interface.
– Modular kernels: can add new covariance functions.
– Isolation of I/O vs. compute logic: pre_process & predict separate.
– Extensibility for new parallel backends (SGE, Hadoop, local).

– [ ] Clear labels for each component and folder.
– [ ] Containers grouping Core Computation, Parallel Engines, Utilities, Data Stores.
– [ ] An annotation of the Map (per-node computation) vs. Reduce (aggregation) stages.
– [ ] Loop indication for iterative inference (–T iterations).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry & Orchestration Layer
    subgraph "Entry & Orchestration"
        PG["parallel_GPLVM.py"]:::entry
        GD_EX["gd-example.py"]:::entry
        SCG_EX["scg-example.py"]:::entry
        SCGA_EX["scg_adapted-example.py"]:::entry
    end

    %% Utilities
    subgraph "Utilities"
        PRE["pre_process.py"]:::util
        NPU["nputil.py"]:::util
        SUP["supporting_functions.py"]:::util
        PRED["predict.py"]:::util
    end

    %% Tools
    subgraph "Tools"
        TD01["tools/__init__.py"]:::util
        TD02["tools/easy_dataset.py"]:::util
        TD03["tools/split_data.py"]:::util
        TD04["tools/show_embeddings.py"]:::util
    end

    %% Core Computation
    subgraph "Core Computation"
        PT["partial_terms.py"]:::core
        KN["kernels.py"]:::core
        KE["kernel_exp.py"]:::core
        GD["gd.py"]:::core
        SCG["scg.py"]:::core
        SCGA["scg_adapted.py"]:::core
    end

    %% Parallel Layer
    subgraph "Parallel Layer"
        subgraph "Map Stage"
            PT --> KN
            PT --> KE
            PT --> OPT{"choose optimizer"}:::core
            OPT --> GD
            OPT --> SCG
            OPT --> SCGA
            MAP_OUT["Write partials to tmp/"]:::data
        end
        subgraph "Reduce Stage"
            SUP --> AGG["Aggregate partials"]:::core
            AGG --> UPD["Update global parameters"]:::core
            REDUC_OUT1["Write embeddings"]:::data
            REDUC_OUT2["Write statistics"]:::data
        end
        LOCAL_MR["local_MapReduce.py"]:::parallel
        SGE_MR["SGE_MapReduce.py"]:::parallel
    end

    %% Data Stores
    subgraph "Data Stores"
        IN["Input folder (Y CSV)"]:::data
        EMB["Embeddings folder"]:::data
        STAT["Statistics folder"]:::data
        TMP["Tmp folder"]:::data
    end

    %% External Services
    subgraph "External Systems"
        SGE["SGE Cluster"]:::external
        HADOOP["Hadoop Streaming"]:::external
    end

    %% Validation
    TEST["test.py"]:::util

    %% Data Flow & Control
    PG -->|arguments| PRE
    GD_EX --> PRE
    SCG_EX --> PRE
    SCGA_EX --> PRE
    PRE --> IN
    PRE --> EMB
    IN -->|Y partitions| PT
    EMB -->|initial X| PT
    PT --> MAP_OUT
    MAP_OUT --> TMP
    TMP --> LOCAL_MR
    TMP --> SGE_MR
    LOCAL_MR --> SUP
    SGE_MR --> SUP
    SUP --> AGG
    AGG --> UPD
    UPD --> EMB
    UPD --> STAT
    STAT --> PRED
    PRED -->|predictions| EMB
    SGE_MR --> SGE
    LOCAL_MR --> HADOOP
    TEST --> SUP

    %% Loop for iterations
    UPD -.->|loop T iterations| PT

    %% Click Events
    click PG "https://github.com/markvdw/gparml/blob/public-master/parallel_GPLVM.py"
    click GD_EX "https://github.com/markvdw/gparml/blob/public-master/gd-example.py"
    click SCG_EX "https://github.com/markvdw/gparml/blob/public-master/scg-example.py"
    click SCGA_EX "https://github.com/markvdw/gparml/blob/public-master/scg_adapted-example.py"
    click PT "https://github.com/markvdw/gparml/blob/public-master/partial_terms.py"
    click KN "https://github.com/markvdw/gparml/blob/public-master/kernels.py"
    click KE "https://github.com/markvdw/gparml/blob/public-master/kernel_exp.py"
    click GD "https://github.com/markvdw/gparml/blob/public-master/gd.py"
    click SCG "https://github.com/markvdw/gparml/blob/public-master/scg.py"
    click SCGA "https://github.com/markvdw/gparml/blob/public-master/scg_adapted.py"
    click LOCAL_MR "https://github.com/markvdw/gparml/blob/public-master/local_MapReduce.py"
    click SGE_MR "https://github.com/markvdw/gparml/blob/public-master/SGE_MapReduce.py"
    click SUP "https://github.com/markvdw/gparml/blob/public-master/supporting_functions.py"
    click NPU "https://github.com/markvdw/gparml/blob/public-master/nputil.py"
    click PRE "https://github.com/markvdw/gparml/blob/public-master/pre_process.py"
    click PRED "https://github.com/markvdw/gparml/blob/public-master/predict.py"
    click TD01 "https://github.com/markvdw/gparml/blob/public-master/tools/__init__.py"
    click TD02 "https://github.com/markvdw/gparml/blob/public-master/tools/easy_dataset.py"
    click TD03 "https://github.com/markvdw/gparml/blob/public-master/tools/split_data.py"
    click TD04 "https://github.com/markvdw/gparml/blob/public-master/tools/show_embeddings.py"
    click TEST "https://github.com/markvdw/gparml/blob/public-master/test.py"

    %% Styles
    classDef entry fill:#cce5ff,stroke:#1f78b4,stroke-width:2px
    classDef util fill:#fff2cc,stroke:#b58900,stroke-width:2px
    classDef core fill:#d4f4dd,stroke:#2a9d8f,stroke-width:2px,stroke-dasharray: 5 5
    classDef parallel fill:#ffdab9,stroke:#e07a5f,stroke-width:2px
    classDef data fill:#e0e0e0,stroke:#4a4a4a,stroke-width:2px,stroke-dasharray: 2 2
    classDef external fill:#e0bbff,stroke:#9d4edd,stroke-width:2px,stroke-dasharray: 4 2
```
