How markvdw/gparml works
Main Components
a. Entrypoints / Orchestration Layer
– parallel_GPLVM.py
– gd-example.py, scg-example.py, scg_adapted-example.py (sequential runs)
b. Core Computation Modules
– partial_terms.py (computes variational partial sums)
– kernels.py / kernel_exp.py (covariance functions)
– scg.py, scg_adapted.py, gd.py (optimisers)
c. Parallelization Engines (MapReduce Pattern)
– local_MapReduce.py (local multithreading/process)
– SGE_MapReduce.py (Sun Grid Engine integration)
– (Hadoop streaming via simplejson/jar optional)
d. Supporting Utilities
– supporting_functions.py (data checks, sanity tests)
– nputil.py, pre_process.py, predict.py
– tools/ (easy_dataset.py, split_data.py, show_embeddings.py)
e. Data Storage
– Input folder (CSV files of Y)
– Embeddings folder (X initial or updated)
– Statistics folder (intermediate stats per iteration)
– Tmp folder (shared scratch for MapReduce)
Out Relationships & Data Flow
– Start: User invokes parallel_GPLVM.py with arguments (–i, –e, –p, –T, etc.).
– Initialization: pre_process.py loads inputs; tools/pca or random init writes seed embeddings into “embeddings/”.
– Iteration Loop (T times):
• Stage (per node or thread):
– Load subset of Y, X from input/emeddings.
– Compute partial_terms (calls kernels, kernel_exp).
– Optionally run gd/scg/scg_adapted on local partials.
– Write partial outputs to “tmp/” or SGE scratch.
• Reduce Stage (master):
– supporting_functions.py / local_MapReduce.py collects all partials.
– Aggregates sums, updates global parameters (X, kernel hyperparameters, beta).
– Write new embeddings and statistics to respective folders.
• Loop back for next iteration.
– End: final embeddings in “embeddings/”, stats in “statistics/”. Optionally call predict.py.
Architectural Patterns & Principles
– MapReduce: clear Map (compute local partial) vs. Reduce (aggregate).
– Plug-in optimisers: gd, scg, scg_adapted follow same interface.
– Modular kernels: can add new covariance functions.
– Isolation of I/O vs. compute logic: pre_process & predict separate.
– Extensibility for new parallel backends (SGE, Hadoop, local).
– [ ] Clear labels for each component and folder.
– [ ] Containers grouping Core Computation, Parallel Engines, Utilities, Data Stores.
– [ ] An annotation of the Map (per-node computation) vs. Reduce (aggregation) stages.
– [ ] Loop indication for iterative inference (–T iterations).